NAT'L INST. OF STAND & TECH

AlllOb QS3EM7

N!ST

% PUBLICATIONS

3C

100

J57

Vo. 952

:ooi

2

il institute of Standards an

y Administration, U.S. Departn^enfo
The research for this report was conducted under the auspices of the National Bureau of Economic Research (NBER). The

NBER is a private, non-profit non-partisan organization engaged in quantitative analysis of the American economy. This

report has not undergone the review accorded official NBER publications; in particular, it has not been submitted for approval

by the Board of Directors. The views expressed in the report are those of the authors.

/

/
NIST Special Publication 952

Papers and Proceedings of the

Advanced Technology Program's

International Conference on the

Economic Evaluation of

Technological Change

June 15-16, 1998

Georgetown University Conference Center

Washington, D.C

Richard N. Spivack, Ph.D.

Editor

Economic Assessment Office

Advanced Technology Program

National Institute ofStandards and Technology

Gaithersburg, MD 20899-4710

Issued July 2001

U.S. Department of Commerce

Donald L. Evans, Secretary

National Institute of Standards and Technology

Karen H. Brown, Acting Director
Certain commercial entities, equipment, or materials may be identified in this document in order to describe an experimental procedure or concept adequately. Such

identification is not intended to imply recommendation or endorsement by the National Institute of Standards and Technology, nor is it intended to imply that the

entities, materials, or equipment are necessarily the best available for the purpose.

National Institute of Standards and Technology Special Publication 952

Natl. Inst. Stand. Technol. Spec. Publ. 952, 155 pages (July 2001)

CODEN: NSPUE2

U.S. GOVERNMENT PRINTING OFFICE

WASHINGTON: 2001

For sale by the Superintendent of Documents, U.S. Government Printing Office

Internet: bookstore.gpo.gov — Phone: (202) 512-1800 —Fax: (202) 512-2250

Mail: Stop SSOP, Washington, DC 20402-0001
Papers and Proceedings of the

Advanced Technology Program's International Conference on the

Economic Evaluation ofTechnological Change

Table of Contents

Editor's Introduction

Advanced Technology Program

ATP Evaluation

Papers and Proceedings

Acknowledgements

CONFERENCE OVERVIEW

Purpose

Selection Process

Panel Sessions

Keynote Speakers

Conference Theme

Theme 1: The Case for Public Policy: Market-based Problems Affecting the

Innovation Process

Theme 2: Policy Goals and Program Design Issues Theme 3: Evaluation of National Programs

Theme 4: Metrics of Evaluating Public R&D Programs

Conclusion

A. THE CASE FOR PUBLIC POLICY: MARKET-BASED PROBLEMS AFFECTING

THE INNOVATION PROCESS

Capital Formation and Investment in Venture Markets: An Assessment of Market Imperfections

R&D Spillovers, Appropriability and R&D Intensity: A Survey Based Approach

Comments on R&D Spillovers, Appropriabihty, and R&D Intensity: A Survey-based Approach ..

B. POLICY GOALS AND PROGRAM DESIGN ISSUES

Orchestrating Technology Policies - German Lessons for Evaluations

ATP and the U.S. Innovation System

Comments on ATP and the U.S. Innovation System

C. EVALUATION OF NATIONAL PROGRAMS

R&D, Government Support, and Firm Productivity in Israeli Industry

Comments on R&D, Government Support, and Firm Productivity In Israeh Industry

From New Growth Theory and Coordination Problems to Technology Policy

Are Swiss Government Progranunes of Promotion of Advanced Manufacturing

Technologies (Amt) Effective? An Economic Analysis Based on Micro-level Survey Data ., Comments on Are Swiss Programmes Of Promotion Of Advanced Manufacturing

Technologies (Amt) Effective? An Economic Analysis Based On Micro-level Survey Data ..

iii
D. METRICS OF EVALUATING PUBLIC R&D PROGRAMS 97

Citations from Patents to Papers: A Measure of Public Research Spillover into Private Industry 99

Comments on Citations from Patents to Papers: A Measure of Public Research

Spillover Into Private Industry 108

A Quality-adjusted Cost Index for Estimating Future Consumer Surplus from Innovation:

A Case Study for The Advanced Technology Program Ill Socio-economic Effects of Collaborative R&D - European Experiences 121

Comments on Socio-economic Effects of Collaborative R&D - European Experiences 129

OPENING SESSION 131

Opening Remarks: Overview of the Advanced Technology Program and Its Evaluaton Program 133

Keynote: R&D and Productivity: Some Historical Reflections 135

The Solow Productivity Paradox: What Do Computers Do to Productivity? 138

The Evaluation of Private Sector Research 139

ATP CONFERENCE 141

International Conference on the Economic Evaluation of Technical Change 143

Publicly Financed Research Consortia 144

Public-private Partnerships In R&D: Lessons Learned 155

Five Concluding Remarks From the Presentations and Discussions 160

iv
Editor's Introduction

This publication contains the Papers and Proceedings of

the International Conference on the "Economic Evaluation of

Technological Change." This collection of Papers relates to the evaluation of the Advanced Technology Program (ATP) and se- lected foreign public investment programs in technology research and development (R&D). The ATP is the nation's civilian tech- nology program charged with improving the competitiveness of

U.S. businesses by providing funding for promising high-risk

technology R&D projects that are deemed to be too risky to be

supported by the private sector. The common element of all Papers and presentations is an emphasis on the empirical as- sessment of the effects of government investment in science and

technology on advanced industrial economies.

Presented papers cover a wide range of topics focusing specifically on the assessment of the effects of government technol- ogy investment programs on advanced industrial economies,

and identification of outstanding issues in evaluation. The conference succeeded in bringing together from across the U.S. and

the rest of the industrialized world government R&D policy-makers, analysts, R&D managers, and academic and other researchers who study technology evaluation issues. The conference

advanced the state of the art of evaluating the impact of govern- ment investments of the type funded by the U.S. Department of

Commerce's Advanced Technology Program in a number of ways, and furthered the dialogue on government's support for industrial R&D to achieve national and regional economic goals. As

anticipated, the presentations faciUtated comparisons ofthe evaluation methods and practices of different countries, including the

role of government in the support of industrial R&D to achieve

national and regional economic goals. Papers presented discussed

programs in the European Union (EU), Israel, Switzerland, Norway, and Germany.

While the programs share common ground, their individual

areas offocus are quite distinct. For example, the ETJs Fifth Frame- work Program for Research and Technological Development

(RTD) incorporates research themes which are grouped by socio- economic objectives rather than by technology topics as was

the case for focused programs in the ATP. Switzerland's program

for funding industrial R&D is focused upon fostering collaborations between academic institutions and the private sector, enabling Swiss companies to participate in international R&D pro- grams, providing training for small- and mid-size businesses, and

supporting specific industry sectors. The ATP also encourages

collaboration, but technology transfer from universities to in- dustry is not the central thrust of the program. Norway's emphasis is on involving users of technology in directing the research

projects, while the ATP's focus is upon the innovator. Germany's

Science and Technology program underscores the "environment

shaping conditions" (market institutions that influence innovation) of its programs while the ATP's institution-building role is more direct.

Advanced Technology Program

The Advanced Technology Program (ATP) is a cost-shar- ing program designed to partner the federal government with the

private sector to further both the development and dissemination of "high-risk"' technologies which offer the potential for

significant, broad-based economic benefits for the nation. ATP

awards are made to individual companies as well as to research

joint ventures for the purpose of encouraging industry to accel- erate the creation and commercialization of enabling technologies that are expected to yield large economic benefits extending

significantly beyond the direct benefits to award recipients.

Rosalie Ruegg, Director of the Economic Assessment Office

ofATP, in her opening remarks, describes ATP as "

. . .a relatively recent component among United States strategies to foster innovation in the civilian sector. It is unique in having as its main

long-term goal that of economic growth. In contrast, the U.S. mission agencies, such as the Department of Defense and the Department of Energy, often also call out the importance of the economic effects of research they fund, but their first priority is, respectively, defense and energy."

Ruegg identifies ATP's focus as being centered on tech- nologies that offer the potential for substantial increases in productivity and competitiveness of firms, provide consumers with new, better, and lower-cost products and services, and increase

high-wage employment in the United States. To accomplish its mission, ATP relies on the presence of expected private returns

to induce companies to be willing to plan, propose, and cost- share research with the ATP, and, if the research is successful in overcoming the technical hurdles, to pursue commercial devel- opment of the new technology with private capital. The ATP

applies the same criteria to the many proposals it receives from

companies. It selects for public funding those projects whose

potential social rate of return is expected to far exceed the private

rate of return on investment. Furthermore, ATP funds only those

projects which the private sector is either not going to undertake

at all by itself or is not going to undertake at a scale, speed, and

scope necessary to realize the large social benefit potential.

ATPEvaluation

The ATP initiated an evaluation program at its outset, as a management tool to enable the program to meet its mission better and operate more efficienUy, and to meet the many external re- quirements and requests for ATP program results as well.

V
Requests for evaluation outcomes arrive from many directions,

from individual members of Congress and their staff, from Congressional subcommittees, the General Accounting Office, the Executive Office of the President, the Office of Management and

Budget, the Office of the Inspector General, the press, think tanks,

industry groups, and others. In addition, ATP, like other federal programs, is subject to the evaluation requirements of the 1993 Government Performance

and Results Act (GPRA). The GPRA resulted from a bipartisan

effort to improve accountability, productivity, and effectiveness of federal programs through strategic planning, goal setting, and

performance assessment. ATP/NIST is developing assessment

plans and techniques, producing measures of program perfor- mance and carrying out evaluation studies in compliance with

GPRA.

Since its inception in 1990, through 1998 ATP has held 39

competitions resulting in 43 1 awards to single companies and

joint ventures, involving over 1 ,000 project participants. The ATP

has awarded approximately $ 1 ,386 million, and industry has pro- vided approximately $1,397 million in matching funds during this same period.

Papers and Proceedings

The Proceedings include remarks made in the opening ses- sion and by the Luncheon and Keynote Speakers, and the re- sponses prepared by Discussants who participated in the two

conference panels, as well as papers presented in the panel sessions. The Papers constitute the greater part of the volume.

It should be noted that not all of the material presented at the conference is encompassed in these pages. In the case of the conference Papers as well as the Panel Sessions, several of the

discussants' comments were not made available to the editor. Only outlines of the presentations of the luncheon speakers are

available.

Acknowledgements

Putting together a first-time conference that includes experts in the field from around the world requires a great deal of

assistance. I am indebted to a group of highly talented individuals for their help in bringing this conference to fruition. Lee

Branstetter, the conference co-chair, participated in every stage

of the planning process from the development of the conference

theme to the selection of papers for presentation. Janet Brumby

and Cindy Smith capably handled the voluminous correspon- dence associated with the travel arrangements. Toni Nashwinter

and her staff met the challenge of mailing the conference an- nouncements. Kathy Kilmer and Tammie Grice of the NIST Conference Department were instrumental in guaranteeing the smooth operation of the conference. Finally, I would like to thank

Rosalie Ruegg, Maryellen Kelley and Connie Chang for providing editorial assistance.

vi
CONFERENCE ONTHEECONOMICEVALUATION

OF TECHNOLOGICAL CHANGE

C

O

N

F

E

R

E

N

C

O

V

E

R

V

I

E

W

CONFERENCE OVERVIEW

Purpose

The purpose ofthe conference is to present papers that deal with the assessment of the effects of government investment in enabling technologies on advanced industrial economics

including an examination ofthe theoretical underpinnings ofthe need for public investment in R&D efforts as well as a review of

selected national programs. The intention is to advance the

state ofthe art ofevaluating the impact ofgovernment investments

of the type funded by ATP. The conference brought together

from across the U.S. and the rest of the mdustrialized world,

govenmient R&D policy-makers, analysts, R&D managers, and

academic and other researchers who smdy technology evaluation

issues. It is anticipated that the presentations will facilitate comparisons ofthe evaluation methods and practices ofdifferent coxmtries as well as facilitate learning across national borders. The conference is one of a set of activities comprising ATP's

economic evaluation program. Other evaluation activities include

conducting case studies, data compilation, statistical and

econometric analyses, and model building.

Selection Process

Abstracts of papers addressing the following subject areas were soUcited: measurement of R&D spillovers, collaborative research and development, the impact of subsidies and tax mcentives on private sector research and development activities, capital market imperfections and implications for private sector funding of R&D, substitution and complementarity between

private and public R&D, and issues concerning the

conunerciahzation of science. A Call for Papers was issued

through a variety of sources including the Commerce Business

Daily (CBD), the web page ofATP, the web page ofthe National

Bureau ofEconomic Research (NBER), as well as web pages of

various universities and international organizations. Priority was

placed upon empirical research, although applied theoretical papers were also considered. Selection was made on the basis of

abstracts of no more than 500 words in length.

Fifty abstracts were submitted from government agencies

and academic institutions from around the world including France,

Germany, Italy, Switzerland, Norway, Australia, Romania, and

China. In addition to the issues identified in the soUcitation notice, several absfracts described foreign public investment programs

similar in nature to the ATP. Five abstracts were selected for frill paper development and presentation at the conference. An

additional five papers were selected for presentation from studies

funded by the Advanced Technology Program's Economic

Assessment Office (ATP/EAO).' Selections were made by the conference co-chau-s, Dr. Richard N. Spivack of the ATP/EAO

and Dr. Lee Branstetter ofthe University of California at Davis

and the NBER. Discussants were identiBed to offer critical remarks

following the presentation of each paper. Panel Sessions

Two panel sessions consisting of both academics and

government representatives from several countries who have

had years of experience in either researching government

technology programs or in administering them were tasked with

addressing the following topics. The panel titled "Publicly

Financed Research Consortia" was chaired by Rosalie Ruegg,

Director ofthe ATP/EAO, and included: Dr. Luke Georghiou of

PREST, UK; Dr. Francois Sand of the EUREKA Secretariat,

Belgium; Dr. Giovanni Abramo ofthe Consiglio Nazionale delle Recerche, Italy; and. Dr. Mariko Sakakibara formerly of the Ministry of International Trade and Industry (MITI), Japan,

currently on the faculty ofUCLA.

The panel that addressed the topic of "Public-Private

Partnerships m R&D: Lessons Learned" was chaired by Maryellen

Kelley, Senior Economist at the ATP/EAO, and consisted of: Mr. Liam O'Sullivan ofthe European Commission, Belgiimi; Dr. Phillipe

Laredo ofthe Centre de Sociologies de 1' Innovation, France; Mr. Ralph Lattimore ofthe Productivity Commission, Australia; and. Dr. Adam Jaffe ofBrandeis University

Each member ofthe respective panel sessions was provided

with the same set ofquestions in advance ofthe conference m an

attempt to focus the discussion. The questions for the first panel,

"Publicly Financed Research Consortia," were as follows.- 1) What

sorts of technologies, e.g., infra structural, should be targeted

in research consortia? 2) What is the appropriate mix of

participating firms in research consortia in terms of size, technological strength, etc? 3) To what extent should the funding government agency attempt to actively participate in the project?

The questions for the second panel, "Public-Private

Partnerships in R&D: Lessons Learned," were as follows: 1) Why

are we having this conference and how can one benefit from it. 2)

Which methodologies appear most promising for current usage

and which ones need further development? 3) What are the multiple purposes and rationales for such partnerships from a public policy perspective and from the perspective of private

industry? 4) Consider how these rationales and the foci of

' The subject areas presented do not include the assessment of ATP's impact, per se, nor do they address ATP's operating

procedures. Instead, authors were asked to write about the theoretical approaches of assessing economic impact.
ATP PAPERSANDPROCEEDINGS

technology policy initiatives vary among countries: related to economic development, policy priorities, strengths/weaknesses

of the private sector. Each discussant was allowed to review

their initial responses in light of the ensuing panel discussion

before submitting them for publication.

Keynote Speakers

Three keynote speakers recognized as leadmg authorities in

their field offered iosights on the following areas of interest: the impact ofcomputers on worker productivity, the decision-making

process of selected U.S. corporations regarding the funding of

R&D, and an oral history of the evolution of the economics of

technological development as an economics subfield. The

speakers are: Professor Rebecca Henderson ofthe Massachusetts

Institute ofTechnology and the NBER, Professor Zvi Griliches of

Harvard University and the NBER, and Dr. Jack Triplett of the Brookmgs Institution.

Conference Theme

The over-arching conference theme is the empirical

assessment of the effects of science and technology promotion

policies in advanced industrial societies. The intention is to bring

together policy practitioners from advanced industrial economies

with academic researchers working on evaluation issues in order

to present and compare the experience in policies and policy

evaluation methods and practices in different countries. The papers are grouped into four themes. The first theme;

"The Case for Public Policy: Market-based Problems Affectmg

the Innovation Process," examines the private capital markets'

role in providmg sufficient funds for high-risk technology R&D

as well as the ability of developers to receive sufficient

compensation when their efforts result in social benefits far exceeding private benefits. The second theme; "Policy Goals

and Program Design Issues," traces the effect of differences in program design and objectives on the evaluation approach. The

third theme; "Evaluation of National Programs," discusses

differing measures of success and failure in terms of selected outcomes measures. Lastly, the fourth theme; "Metrics of

Evaluating Public R&D Programs," presents different

methodologies for assessing program impact.

Theme 1: The Casefor Public Policy: Market-based

Problems Affecting the Innovation Process.

An economic rationale for public sector support of

developing high-risk technologies is that there exists an madequate supply ofprivate sector funds available for high-risk

enabling technology projects. This deficiency could be

attributable to a variety of factors including the risky emd long- term nature ofmost R&D projects, and the substantial opportunity

costs associated with long-term investments, as well as problems

in appropriability, whereby "spillovers" occur and the returns from the innovation are not sufficiently captured by the innovator

to warrant investment. Where the social rate ofreturn is greater

than the private rate of return to the innovating firm

underinvestment tends to occur. The existence of spillovers

imphes that private firms will invest less than is socially desirable

in R&D, with the result that some desirable research projects will not be undertaken, and others will be undertaken more slowly or on a smaller scale than is socially desirable. Problems with the flow of fimding which create "financing gaps" also can result in underinvestment in projects with potentially high social benefits. This economic rationale underlies the very foundation of the ATP.

The papers synopsized below address the rationale for public

fimding of enabling technologies by discussing topics such as the "financial gaps" that exist in the debt and equity markets, and

the impact of spillovers on the appropriability of rents as an

unportant determinant of innovative activity.

Private Cavital Markets in the U.S.

Lerner and Gompers focus on the adequacy ofprivate sector financing ofyoung entrepreneurial firms, as well as the ability of

public programs to supplement private fimding. Rather than

considering the whole range offinancial alternatives available to small high-technology firms, they concentrate on one financial intermediary: the venture capital organization. Citing the recent

literature regarding the growing concern about the adequacy of

private sector mechanisms in providing financing of young

entrepreneurial firms, especially technology intensive ones, they

note that capital constraints appear to limit R&D expenditures,

especially in smaller firms. They claim that problems in the debt

and equity markets are contributing to "financial gaps" which

appear to be a consequence of the information gaps that exist between entrepreneurs and mvestors. The authors deduce that as a source of financing for small high-technology firms, venture capital has certain limitations. Because of the imevenness (over time) ofthe flows into venture

fluids and the concenfration ofinvestments by venture capitalists in a narrow array of technologies, there continues to be a gap

between the willmgness and capability of these private sector sources to fiind the broad array of industrial R&D activities that are socially beneficial. Concluding that a "financial gap" continues to exist despite the growth in venture capital fluids , Lerner and Gompers endorse

the need for policies and programs such as the ATP to fill these gaps.

4
CONFERENCE OVERVIEW

Appropriabilitv Conditions and R&D Investment

The impact of spillovers on the appropriability ofeconomic

rents from R&D is seen by Cohen and Walsh as an important

determinant of innovative activity and performance as well as a key concept motivating policy interventions in support of

industrial R&D. Appropriability is characterized as the degree to which different appropriability mechanisms (e.g., secrecy, patents) or strategies increase the returns to the firm from its R&D

investments. While there is now strong evidence supporting the argument that R&D spillovers have important effects on

innovative performance and productivity growth, empirical

analyses ofthe direct effects of appropriability on the conduct of

R&D has proven elusive. In an effort to address this short-coming, the authors test an

empirical model relating R&D information flows, appropriabiUty

and R&D intensity to one another. Drawing on the results of

their 1994 survey ofR&D directors oflarge U.S. companies Cohen

and Walsh find that the more effective the appropriability

mechanisms in protecting the profits attributable to invention, the greater are the incentives to conduct R&D.

The authors present a methodology equipped with measures ofappropriabiUty that may be used to rank industries with respect

to their appropriability conditions. The authors conclude that these measures can provide a basis for identifying industries where appropriabihty is weakest, and, hence, where the need for ATP is greatest. Theme 2: Policy Goals and Program Design Issues.

The evolution of government science and technology

programs entails a natural shift m focus brought about by

changing political and economic environments. The following papers focus upon the significance of these changes in program

design in shaping the approach to program evaluation. ATP,

while still in its formative years, has focused on "spillovers" from

technology diffusion as an important mechanism by which

innovation contributes to economic growth. By contrast,

Germany's science and technology policies have only very

recently begun emphasizing the importance of spillovers.

Advanced Technolosy Prosram

In a program such as ATP, spillovers may hold the key to most ofthe economic benefits generated as a result ofthe program

but in the economic Uterature few methodologies exist to measure

their impact. In his dinner speech Griliches, discussing his earlier work in the field of the economics of science and technology

R&D, observed that "...it was obvious that what was missmg

was some accounting for spillovers of R&D results and other

sources ofnew knowledge across firms, industries, and countries.

The overall conclusion was that spillovers seem to be there and

are important." While assessing private and social benefits, the ATP has focused its evaluation efforts on assessing the extent and nature of the spillovers from public-private partnering m

R&D activities.

German Science and Technolosy Policy

Over the past two decades the German Science & Technology

(S&T) programs have evolved from a general scientific and basic research orientation into a focused policy instrument for industrial innovation. The current rationale for the programs rests upon

issues ofmarket failure and the role ofthe government in "shaping

conditions" conducive to technology R&D with minunal state intervention. Adopting the term "diffusion-oriented" the German

S&T programs incorporate the notion that the invention and

initial commercial exploitation of new products and processes matter less than the rapid and widespread diffusion of that technology.

Dreher and Kuhlmann present an overview ofthe German

S&T programs including a history ofprogram evaluation and an overview ofthe many evaluation instruments with an assessment ofhow they could be utilized by future users. The experience of

nearly two decades of program evaluation have established an

evaluation studies research community in Germany m the fields of economics and social sciences covering a relatively broad

range ofmethodological approaches and evaluation instruments. As the German S&T policy has evolved from a scientific and

basic research orientation towards use as an instnmient for

indusfrial innovation, there has been a growing interest in evaluation metrics. Advocates of public sector support for the private

development of high risk technologies base their support on

theories ofmarket failure. On the other hand, opponents discount

such theories of market failure and focus on perceived

discrepancies between the decision making rationales ofpassive

bureaucrats and those of active entrepreneurs. Such

discrepancies, they argue, lead bureaucrats to a less than optimal

selection process for both projects and technologies. As a consequence, administrators of government programs are presumed to lack the information necessary for proper decision

making and therefore lack sufficient information necessary for proper decision makmg. The result is a German S&T policy best described as 'diffusion-oriented' rather than 'mission-oriented'

with an orientation towards an environment shaping policy in which there appears to be the lack ofa comprehensive theoretical model oftechnological change which includes action by the state as an active determinant, other then simply 'shaping conditions'

The German S&T "diffusion-oriented" policy fits well within

the overall objective of having the state act as the guarantor of a

5
ATP PAPERSAND PROCEEDINGS

high level of basic research and a sound technology base for industry while not being regarded as a 'director' of industry.

Arvanitis, et al, observed a similar premise in their investigation

of the Swiss technology programs. ATP may also be classified as "diffusion-oriented" rather than "mission-oriented," but fiirther similarities with the German

and Swiss programs beyond this classification scheme are quite limited especially when the discussion turns to the "envu-on- ment shaping conditions" of these programs.

Spillovers from Innovation

Fogarty, Sinha, and Jaffe present an analysis of technical

interrelations using patent data that they suggest may provide a

useful framework for developing ATP strategies. Specifically, they

suggest that clusters of interrelated technologies may indicate

industries with large spillover potentials. The authors conclude

that ATP's fimded projects may produce more influential spillovers

if selection is based upon knowledge of the network of ties between R&D organizations. Technologies that involve such

links are taken by Fogarty et al. to indicate the influence of

enabling technologies. Finally, the methodology could be useful m providing a possible approach to measuring the "spillover

gap."

Fogarty, Sinha, and Jaffe' s paper addresses a deficiency in existing methodologies that utilize patent information to measure

spillovers in which weights are assigned to patents simply by

the quantity ofcitations. In these models it is often assumed that

'all patent citations are equal' (assigned a value of 1 ). The authors improve upon this analysis by applying a fuzzy 'systems'

methodology which allows them to develop indicative

membership measures between 0 and 1 representing the strength

of the mteraction between any pau^ise combmation of R&D

labs, specific to organization, technology, and region. They

believe that this approach renders new, deeper insights into innovation as well as allowing important tests ofR&D spillover

hypotheses with relevance to ATP and similar government

support programs. Using the R&D lab as the basic unit of measurement their fuzzy system methodology represents R&D networks as a

fimction of the influences of R&D labs among one another's patenting activity. By a 'system' they refer to the existence of a hierarchy of technologies, R&D organizations, and regions

cormected by a communication network. System effects occur when a change in any component diffuses throughout the network. They reason that the notion of an innovation system is

intuitively appealing and is consistent with existing evidence on R&D spillovers.

From their analysis, Fogarty, Sinha, and Jaffe suggest that there is an underlying 'spillover network' structure. Further

research is needed to identify the spillover mechanisms and the

factors affecting progress along various diffusion pathways.

Theme 3: Evaluation ofNational Programs.

Several papers were presented in which selected foreign

programs' evaluation practices were discussed, highlighting the

similarities as well as differences between the various national programs. Some ofthe variance in programs may be ascribed to the differences in the cultural and political environments existing

in the countries, while other discrepancies may be attributable to a mix oftechnology specific factors. For the most part, conference

participants shared the view that market structures lead to sub- optimal results in selected areas in which governments are attempting to increase innovative activity.

Public investment programs in technology R&D can be foxind

in many countries ofvarying sizes and economic maturity aroimd

the world. These government programs emphasize the importance

of mnovation to economic prosperity by providing financial assistance to firms and other entities and fostering collaboration

as a mechanism for solving larger, more complex research

problems. From a policy standpoint, several programs which

started out with a scientific and basic research orientation have

migrated into policy for industrial innovation. O'Sullivan,

commenting on the European Union programs, observed that R&D policy was once viewed mainly as an instrument of

microeconomic policy, but is now viewed as an economic policy

tool which also has macroeconomic significance. Dreher and

Kuhlmann concur with this view for the case ofGerman Science

and Technology programs. The evaluation of the European Union's Framework

Programs^ consists ofcontinuous monitoring ofproject activities, annual reporting, and five-year assessments, carried out midway

through program implementation. The criteria include relevance of objectives, efficiency of operation and effectiveness in terms of objectives achievement, "European added value" and

dissemination/exploitation of results. Applying these

methodologies to the new challenges brought about by the introduction of the Fifth Framework Program has required

evaluation metrics of regional economic and social benefits. Evaluation studies are now expected to deliver assessments of

multiple goals, including evidence ofthe degree to which technical

^ The EU Framework Programs for Research and Technological

Development was created in 1984 and are structured to set strategic goals for community research over five-year time horizons.

6
CONFERENCE OVERVIEW

objectives (defined as industrial competitiveness) are achieved

and broad social objectives are met (e.g., creation ofemployment,

distributional issues, etc.). These assessment criteria are relatively new. Progress in meeting objectives under the core goal of

improving the competitiveness of European industry is still an

important evaluation criterion as well. This is in stark contrast to the evaluation requirements of

ATP, whose overall goal is the diffusion oftechnologies and the creation of broad-based economic benefits through

commercialization activities of the funded firms and through

various spillover pathways. The Fifth Framework Programmes'

objectives are expressed in terms of socio-economic problem

solving. Laredo noted that a keyword in this conference for justifying public intervention in innovation was in his opinion

"spillovers". He noted that nearly all ofthe U.S. presentations at the conference emphasized the importance of spillovers in assessing the benefits from the ATP, whereas references to spillovers were completely absent from the presentations

concerning other countries. Sand, drawing upon his experiences from the EUREKA^

program, discussed the importance of the "partnering" or "risk sharing" aspects of the European public support programs

whereby public funds are provided to permit the participants to take a risk that they would not have taken alone. Georghiou

notes that there is also a rationale for targeting specific technical areas in certain circumstances. These include those areas where

co-ordination ofthe R&D activities can improve the pace at which

imovations are commercialized. ATP's "focused" programs'*, in which industry groups identify a cluster oftechnologies, or a set ofproblems that may be attacked with a number ofdifferent (but

related) approaches, may be seen as an example of Georghiou'

point. In a focused program, multiple projects are funded m the same area.

Estimating Social Welfare Benefits: Israeli Industrial

Programs

Griliches and Regev discuss the various support institutions and programs that were created over the past 20 years by the IsraeU government in a concerted, large scale effort to develop

and promote science-based, high-tech industrial sectors. For

Israel, the scarcity of natural resources, the small size of the

' Based in Brussels, the EUREKA Initiative was launched in 1985, providing a framework for international collaboration in the area of advanced technologies among firms, research institutes and

universities with the aim of strengthening the productivity and

competitiveness of Europe's industries.

* Beginning with the 1999 Competition, ATP replaced focused program competitions it had since 1994 and the General Competition

with an Open Competition consisting of five "technology boards" to assess project proposals.

economy and the presence of a highly educated workforce, led policy makers to emphasize economic policies where innovation

and technical change could lead to a high growth path.

Previous research by Griliches and Lichtenberg' on U.S.

defense-related R&D expenditures showed a negative effect of

public R&D investment on industry productivity. Public funding

of industry R&D undertaken by firms for purposes other than in response to the needs of defense agencies or some other

government mission is relatively new in the United States. Hence,

previous research showing an absence of social benefits from

public investment in industry R&D is not informative about the

effects of public R&D spending m a program such as ATP.

Griliches and Regev explore the impact of a program in Israel that has provided funding to firms for their own R&D activities for more the 20 years. In this paper, the authors examine the

effects of an Israeli government-supported R&D program on the productivity of those firms. They estimate the effects of the Israeli government's R&D

program on Israeli firms' productivity (measured as the firm's output per person year) in econometric models using four different estimation procedures. In all regressions, the effect ofgovernment

funded R&D is always positive and statistically significant m

aknost all ofthe estimated models. Griliches and Regev consider

these estimates to be higher than what they expected. They

attribute the results to two possible features of public funding programs. The first explanation is that there is a selection effect imderlying these results, meaning that the firms who apply to the government and are awarded funds have better R&D projects than those who don't apply and don't win. The second explanation

is that these effects reflect heterogeneity among firms in their capabilities and that the government office responsible for

selection is able to discern these differences. Both explanations

support the conclusion that there is a substantial program effect on Israeli firm performance. These estimates may be refined in future work to include measures ofthe "permanent" attributes of

firms related to their R&D capabilities and other variables. Griliches and Regev conclude "that the mechanisms that

allocate R&D dollars seem to be doing their work properly in most cases, and the more they manage to 'pick winners' the

better." Development of methodologies that will assist in the

selection process to improve the ability ofprograms to correctly

' Griliches, Zvi and Frank Lichtenberg, "R&D and Productivity at the Industry Level: Is There Still a Relationship?" In Z. Griliches (ed.), R&D. Patents and Productivity. Chicago: University of Chicago Press, pp. 465-496. 1984.

7
ATP PAPERSANDPROCEEDINGS

diagnose a firm's and a technology's potential may, therefore,

increase the continued success of such programs.

Estimatins Social Welfare Benefits: Norwegian Industrial Programs

Klette and Moen reach a conclusion contrary to that of

Griliches and Regev regarding the effectiveness of R&D public

support programs. They examme the Norwegian government's

initiative implemented in the 1980's for the Information

Technology (IT) industry and test a hypothesis that public policy

interventions occur when generic or general purpose technologies

(GPT's) are involved. (GPT's are defined as having wide

applicability and possessing the potential for significant economic

growth). They present a quantitative analysis of the IT-related technology programs in which they compare the performance of

targeted firms to non-targeted firms in the same industries and

then consider the development ofthe IT-industry and the related high-tech manufacturing sectors relative to the performance of

the manufacturing sector at large. Fmally, they compare the performance of these sectors in Norway to their performance in other OECD economies.

Citing the current status ofeconomic models, including most

theoretical growth models, that tend to "treat all forms oftechnical change in the same, diffuse manner," the authors contend that there has been little economic analysis suggestmg that research and iimovation associated with 'generic' technologies such as mformation technology require particular attention. Based on

the theoretical arguments related to GPT's, and presented

throughout theu- paper, they conclude that one would expect the IT programs and the coordination efforts which accompany them

to stimulate economic performance in the targeted firms and

therefore are justified in receiving government subsidies. But

their research provides little evidence that industry has benefited

significantly from the financial stimulation and the coordination

effort of the support programs. These findings lead to the conclusion that the Norwegian government's efforts to stimulate and coordmate the development ofIT-products and applications

have not met with much success. The authors' explanation for their findings can be summed

up in the conception of the industrial iimovation process where

coordination problems and "market failure" issues are pervasive.

PoUcy makers and bureaucrats may lack the necessary information

and know-how to design programs that improve upon the existing

market conditions. In contrast, ATP's approach views public

policy as offering market corrections through the collaborative

efforts of a "public/private partnership."

Development and Diffusion of Technolosv: Swiss Technolosv Programs

Arvanitis, Hollenstein and Lenz evaluate the Swiss

government's program to promote a more rapid and broad

diffusion of Advanced Manufacturing Technologies (AMT) by

examining the degree to which the program achieved a level of

success greater than what would have been attained by firms

acting alone. Their investigation ofthe issue of 'additionality' is particularly interesting in light of the fact that Swiss technology

policy places a relatively low emphasis upon direct measures for fostering innovation in the economy, preferring instead to create a favorable "environment" for the introduction ofnew products

and production techniques. This "framework-oriented" policy is supplemented, though, by the use of particular initiatives to support rapid diffusion of selected basic technologies which are considered to be relevant for a broad spectrum of industrial

activities. Their results demonsfrate a positive impact from the policy

in the case of small firms and/or those firms adopting AMT for the first tune or characterized by a low intensity of AMT use. There was no demonstrated influence on the adoption strategy

oflarge firms. Regarding the issue ofadditionality, in the case of

small firms they conclude that it is quite probable that subsidized

R&D projects in the field of AMT, in general, might just allow

firms to substitute government fimds for their own. Evidence

surrounding the impact ofpublicly supported technology R&D,

in regards to the issue of additionality, often remains inconclusive. Mariko, citing her work on the Japanese economy, observed that the provision of government subsidies could contribute to the enhancement of technological opportunities firms

face. Subsidies can reduce the effective cost ofconducting R&D

which, in turn, could encoiu-age firms to undertake more R&D.

But on the contrary, she also notes that though governmentsponsored R&D consortia will allow firms to conduct research

projects which firms would have done on a much smaller scale without government subsidies, government subsidies could be

used as a substitute for a firm's spendmg on R&D. This makes it quite difficult to gauge the overall impact.

Theme 4: Metrics ofEvaluating Public R&D

Programs.

The following papers focus on the methodological aspects

of evaluation using a variety of measures of outcomes and impacts, e. g., patent analysis, product quality and performance

analysis, and an examination ofthe issue of "additionality."

As Ruegg pointed out in her opening remarks, evaluation

has been practised by the ATP since the inception of the pro- gram in order to respond to both external and internal requests

8
CONFERENCE OVERVIEW

and needs regarding program performance. Externally, ATP has

faced demands for performance measures which have been driven

by policy issues. Specific requests for evaluation results have come from individual members ofCongress and their staff, from

Congressional subcommittees, the General Accoimting Office, the Executive Office ofthe President, the Office ofManagement

and Budget, the Office of Inspector General, the Press, think

tanks, industry, and others. ATP has also received inquiries that are focused more on evaluation tools and methodologies than on specific evaluation results, from counterpart programs in other

coimtries, from other Federal agencies, State and regional gov- ernment agencies, universities, businesses, and consultants. In response to these increased demands for performance

results, along with the passage ofthe Government Performance

and Results Act (1993), the Economic Assessment Office (EAO)

of ATP was created in 1994 and tasked with measuring the impacts of ATP's funding of high-risk, enabling technologies, as well as investigating underlying relationships between technological change and economic phenomena.

The EAO also funds the development of new methodologies. Two ofthe papers discussed at the conference on the theme

of metrics provide examples of new approaches to evaluation

metrics (Mowery and Austin and MaCauley) were commissioned

by the EAO.

Patent Analysis

Patent citations have often been used to frace technology

diffusion. Narin and Olivastro believe citations from patents to papers provide direct evidence of spillovers from basic research

to industrial technology and in their paper attempt to measure

the spillover effects of publicly supported research onto industrial technology by analyzing more than 750,000 non-patent ref- erences listed as "other references cited" on the front pages of

the U.S. patents. It is inferred that these citations are direct evi- dence of a massive spillover from basic research to industrial technology. By applymg this technique they also seek to test the hypothesis that publicly supported science acts as a driving

force behind high technology and economic growth.

The results from the Narin and Olivastro study indicate a high probability that public science does play an essential role in supporting U.S. industry, with the intensity of the linkage be- tween patented industrial technology and public science having

increased significantly over the last decade. This result is prevalent across all the science-linked areas ofindustry, amongst companies large and small, and appears to be a fundamental pillar for the advance of U.S. technology. The underlying hypothesis -

that public science is a drivmg force behind high technology -

seems to be strongly supported by their data analysis. Further- more, they show that the science influencing mdustry is mainstream focusing upon issues of importance to the disciplme. In other words the research is quite basic, quite recent, published in highly influential journals, authored at major universities and

laboratories, and is supported by pubUc and charitable institutions. While the use of patent data as a measure of spillovers is quite common throughout the literature, Mowery, as well as others, questions how much information of use to other inventors

patents convey, especially in the absence of methods for rival inventors to access know-how. He suggests that patents alone

are insufficient to spark widespread imitation. He also observes

that even if one assumes that patents do convey information, the often-lengthy lead times in their publication means that sectors

in which lead time is especially important should find patents

particularly unhelpful sources of information. While the precise

utility of patent data as a measure of spillovers continues to stir debate it remains one ofthe most used measures and is often the subject of varied mterpretation.

Product Quality Improvements and Performance as Measures of Program Achievement

Austin and Macauley present a method for estimating con- sumer surplus from planned new product innovations focusing

upon the improved price-performance characteristics and benefits of those innovations for existmg services. These new tech- nologies are claimed to have leapfrogged beyond current best

practice and therefore their benefits are assessed with respect to a baseline ofexisting, state-of-the-art technologies that are themselves continually being improved. The authors present their fmdings in terms of the economic value to consumers of result- ing quality improvements, compared to advances that would have

been expected to occur in the absence of the program.

They employ a cost-index approach using changes in quality-adjusted prices ofnew technologies which, under certam as- sumptions, provides a measure ofconsmner surplus. The ability to perform this calculation depends upon the structure of the downstream market in which the technology is to be applied, and

is appropriate to the kinds of new technologies sponsored by

ATP whose demand is typically mediated by producers using

the innovations in theu" production processes. As long as the downstream market is competitive, or is a government agency,

the producer acts as an agent for consiuners when it uses an upstream technology. This activity generates a classic supply

and demand curve iatersect rendering the area under the "de- rived-demand" curve, calculated by the cost-index method, as a measure of consumer surplus.

The approach suggested by Austin and Macauley avoids

the need for econometric estimation and makes it possible to perform the estunation in sectors for which output quantity and

quality-adjusted output price are unobservable. Since ATP fimds

9
ATP PAPERSAND PROCEEDINGS

the R&D of technical innovations that have not yet reached the market this evaluation technique allows for predicting consumer

surplus on the basis of what one currently knows about these technologies.

Additionality

Georghiou addresses the issue of "additionality," a term used in reference to whether a particular public expenditure is used 'in addition to' those funds which would have been ex- pended by a firm m its absence or used in place of those funds. He presents a brief overview ofthe evaluation efforts applied to two ofEurope's principal programs for the support ofcollaborative R&D, the European Commission's Framework Programmes

and the inter-govermnental EUREKA Initiative. In his assess- ment of these programs he focuses upon the use of evaluation

metrics as a policy instrument noting their limitations. In the case ofmeasuring the impact ofpublic funding upon the economy as a whole, he discusses the importance of "additionality" as a policy concern. The theme ofadditionality has, at times, emerged

in the U.S. at the center ofthe debate surrounding ATP. There is some concern among policy makers that funds provided by the government are a substitute for investments that the private sec- tor should undertake on their own. In Eiu-ope the issue of

"additionality" is currently at the center of a dispute between

different branches of the European Commission whereby one

branch advocates the demonstration ofthe impact ofpublic funding on a project-by-project basis while another finds it sufficient to demonstrate the results of public funding as an overall in- crease in spending on R&D.

Georghiou' s investigation ofcompanies involved in collaborative R&D projects shows a substantial degree of agreement

among managers about the importance of the govenmient pro- gram in stunulating additional R&D mvestment by the private

sector. His survey results indicate that twice as many companies

would not have carried out the R&D projects as compared to those who said they would have continued even if no govern- ment funding was received. He concludes that it is not obvious

whether projects with high additionality will produce greater or smaller impacts. On the one hand, one would expect firms to cover their highest priority projects with their own resources and

hence put forward marginal projects for funding. This reasoning

is not confirmed by the evidence fi"om his evaluations though as

there are several examples ofprojects with both high additionality

and high subsequent impact. This result could be explained by

the assumption that public fundmg motivates firms to undertake

projects with a higher risk but also a potentially higher pay-off.

"Crowding out" refers to the phenomenon that only a limited amount of R&D can be undertaken and that when government

funds R&D there will be less private-sector investment in R&D.

The issue of "crowding out" was ofmajor importance to each one of the public support programs discussed at this conference.

In regards to whether public support programs stimulate additional R&D, Lerner and Gompers caution that public funding in areas with extensive private financing may lead to a "crowding

out" phenomenon whereby public funds may replace private

expenditures for research. The total amount spent on R&D and

resulting innovation may consequently not increase much in these cases, thus limitmg social returns. Dreher and Kuhlmann address the issue of additionality

through their discussion of two public support programs aimed

at SMEs, the R&D Personnel Costs Subsidies Program of the Federal Ministry for Economic Affairs and the Promotion ofResearch Personnel Growth. They conclude that - although there were some fi-ee-rider effects (crowding out)- the sponsored SMEs

fi-om the manufacturing sector greatly increased their R&D efforts.

Conclusion

A conmion theme expressed by many of the participants

during the conference concerned the nature and the scope ofthe challenge in developing and implementing evaluation metrics of

R&D programs. Griliches, in his keynote address, identified some

ofthese challenges, ". . .aggregate productivity numbers are only

dimly and possibly misleadingly related both to the measure- ment oftrue technical change and the impact ofR&D on it, especially federally supported R&D. This is due, in part, to difficulties in productivity measurement per se, and second to the particular location of most of R&D in the mdustrial spectrum." The measurement difficuhies are attributable to the application ofcurrent productivity measures to "...the 'uimieasurable' sectors, such

as services and construction," and the growth in importance of

these sectors in the overall economy.

Despite these daunting challenges, the evaluation efforts discussed at the conference offered an opportunity to highlight those tools and methodologies which offer great promise and are proving to be capable of investigating more thoroughly such

themes as the "financial gaps" that result from a lack offunds for high-risk technology R&D and the calculation ofreturns on public investment, including those that offer more "complete" measurements by incorporating "spillovers" into the net social benefits.ATP's goal of achieving economic benefits through tech- nology development was shared by all of the programs dis- cussed at the conference. How each program measured success towards reaching this goal differed significantly especially for smaller countries such as Norway, Switzerland, and Israel.

10
CONFERENCE OVERVIEW

Future work must view the evaluation oftechnology R&D

as an all-mclusive event—one in which a portfoUo ofmethodologies exist and are appUed in several areas. The Economic Assessment OfBce of ATP is developing such a portfoUo offering a balanced approach to evaluation by covering a variety ofthemes

believed to be essential to gaining insight into the program's

impact upon the U.S. economy. The portfolio contains case studies of funded projects, analytical studies of spillovers and

spillover pathways, studies identifying imperfections in the fi- nancial markets which may lead to insufficient funding for technology R&D, studies of the role of ATP in forming collaborations, and theoretical papers examming such themes as "crowding ouf and "additionality."

To complement this work it is necessary to collect new

types of data while at the same time develop "more appropriate"

metrics capable of providing better evaluation measures. It is believed that only through such a balanced approach one may

expect to gain a true understandmg of the economic impacts

offered by such complex programs as ATP.

11
I

i

I
THE CASE FOR PUBLIC POLICY: MARKET-BASED

PROBLEMSAFFECTING THEINNOVATIONPROCESS

A

Capital Formation and Investment in Venture Markets:

An Assessment of Market Imperfections

by Josh Lemer and Paul A. Gompers^

The recent economic literature suggests several reasons for concern about the adequacy of private sector mechanisms to finance small high-technology firms. A growing body ofempirical research on capital constraints (reviewed in Hubbard [1998])

documents that an inability to obtain external fmancing limits many forms ofbusiness investment. Particularly relevant are works

by Hall [1992], Hao and Jaffe [1993], and Himmelberg and Petersen [1994]. These show that capital constraints appear to limit re- search-and-development expenditures, especially in smaller firms. A related body of literature documents that investments in R&D

yield high private and social rates ofreturn (e.g., Griliches [ 1 986],

Mansfield, et al. [1977]). These fmdings similarly suggest that a higher level of R&D spending would be desirable.

Why are Investments in Entrepreneurial Firms Problematic?

Entrepreneurial firms often develop products and ideas that require substantial capital during the formative stages of the companies' life. Many entrepreneurs do not have sufficient fimds

to finance projects themselves, and they must therefore seek

outside fmancing. But because the nature ofthe entrepreneurial

setting, the process of raising fmancing can often be a troubled

one. To briefly review the types of conflicts that can emerge in these settings, Jensen and Meckling [1976] demonstrate that agency conflicts between managers and investors can affect the willingness of both debt and equity providers to invest capital. If the firm raises equity from outside investors, the manager has an incentive to engage in wasteftil expenditures (e.g., lavish offices) because he may benefit disproportionately from these but

does not bear their entire cost. Similarly, if the firm raises debt, the manager may increase risk to undesirable levels. Because

providers of capital recognize these problems, outside investors demand a higher rate of return than would be the case if the

fiinds were internally generated.

Even if the manager is motivated to maximize shareholder

value, informational asymmetries may make raising external capital more expensive or even preclude it entirely. Myers and Majluf

[1984] and Greenwald, Stiglitz, and Weiss [1984] demonstrate

that equity offerings of fums may be associated with a "lemons"

problem. If the manager is better informed about the firm's in- vestment opportunities and acts in the interest of current share- holders, then managers only issue new shares when the

company's stock is overvalued. Indeed, numerous studies have

documented that stock prices decline upon the announcement

of equity issues largely because of the negative signal that it sends to the market.

These information problems have also been shown to exist

in debt markets. Stiglitz and Weiss [1981] show that ifbanks fmd

it difficult to discriminate among companies, raising interest rates can have perverse selection effects. In particular, high interest rates discourage all but the highest-risk borrowers, so the quality of the loan pool declmes markedly. To address this problem,

banks may restrict the amount of lending rather than increasing

interest rates. These problems in the debt and equity markets are a conse- quence of the uiformation gaps between the entrepreneurs and

investors. If the information asymmetries could be eliminated,

fmancing constraints would disappear. Financial economists ar- gue that venture capital organizations can address these problems. By intensively scrutinizing firms before providing capital

and then monitoring them afterwards, they can alleviate some of

the information gaps and reduce capital constraints.

HowDo Venture Capitalists Address These Problems?

In particular, a series of academic studies (Gompers [1995,

1997], Lemer [1994, 1995], and Sahlman [1990] are empirical ex- amples; see Barry [1994] for an overview of the extensive theoretical literature) have documented the mechanisms that venture

capitalists employ to address these challenges. We will highlight six of these responses below.

The fu-st set relates to the fmancing of firms. First, from

whom a firm acquires capital is not always obvious. Each sourceprivate equity investors, corporations, and the public marketsmay be appropriate for a firm at different points in its life. Further- more, as the firm changes over time, the appropriate source of

fmancing may change. Because the firm may be very different in the fiiture, investors and entrepreneurs need to be able to antici- pate change.

' Josh Lemer and Paul A. Gompers are on the faculty of the Harvard Business School and are members of the NBER.
ATP PAPERSAND PROCEEDINGS

Second, the form of financing plays a critical role in reducing

potential conflicts. Financing provided by private equity investors can be simple debt or equity, or it can involve hybrid securities like convertible preferred equity or convertible debt. These

financial structures can potentially screen out overconfident or under-qualified entrepreneurs. The structure and timing of fi- nancing can also reduce the impact of uncertainty on future re- turns.A third element is the division of the profits between the entrepreneurs and the mvestors. The most obvious aspect is the

pricing ofthe investment: for a given cash infusion, how much of

the company does the private equity investor receive? Compensation contracts can be written that align the incentives of managers and investors. Incentive compensation can be in the form

of cash, stock, or options. Performance can be tied to several measures and compared to various benchmarks. Carefully de- signed incentive schemes can avert destructive behavior.

The second set of activities of private equity investors relates to the strategic control of the firm. Monitoring is a critical

role. Both parties must ensure that proper actions are taken and

that appropriate progress is being made. Critical control mechanisms-e.g., active and qualified boards of directors, the right to approve important decisions, and the ability to fire and recruit key managers-need to be effectively allocated in any relation- ship between an entrepreneur and investors. Venture capital investors can also encourage firms to alter the nature of their assets and thus obtain greater financial flexibility. Patents, trademarks, and copyrights are all mechanisms to protect firm assets. Understanding the advantages and limita- tions of various forms of intellectual property protection, and

coordinating financial and intellectual property strategies are

essential to ensuring a young firm's growth. Investors can also shape firms' assets by encouraging certain strategic decisions,

such as the creation of a set of "locked-in" users who rely on the

firm's products.

Evaluation is the final, and perhaps most critical, element of

the relationship between entrepreneurs and private equity in- vestors. The ultimate control mechanism exercised by the private

equity investors is to refuse to provide more financing to a firm. In many cases, the investor can-through direct or indirect ac- tions-even block the firm's ability to raise capital from other sources. The importance ofthese mechanisms is underscored by the success of venture-backed firms. A variety of evidence suggests that venture-backed firms are more successful than their peers:

• One illustration of this difference is in the share of the companies making the transition from private to public

ownership through initial public offerings (IPOs), which

typically include many ofthe most successful firms. In recent years, fiilly 30% of the IPOs have been of venture-backed firms. (Detailed summary statistics are avail- able in Gompers and Lemer [ 1 997] .) This is much greater

than the share ofyoimg firms receiving venture financing.

• Venture-backed firms are also more successful after going public. Brav and Gompers [1997] show that in the

five years after going public, IPOs that had previously

received equity financing from venture capitalists out- perform other offerings.

• Venture capital appears to contribute to technological

innovation. In a panel study of twenty industries over

three decades, Kortum and Lemer [1998] demonstrate a

relationship between the extent of venture financing in particular industries and their rate of patents. The pat- tern appears to be robust to a variety of controls for reverse causality and alternative explanations.

What Are the Limitations of Venture Capital Invest- ment?

At the same time, venture capital appears to have important

limitations as a source of financing for small high-technology

firms. Both the uneveimess of the inflows into venture funds

and the concentration of investments within a few narrow tech- nologies may limit its effectiveness as a source of financing.

The first ofthese limitations relates to the supply ofventure

capital. During the past twenty years, commitments to the U.S.

venture capital industry have grown dramatically. This growth

has not been uniform: peaks in fundraising have been followed

by major retrenchments. Despite the importance of and interest

in the venture capital sector, the underlying causes ofthese dra- matic movements in venture fundraising are little understood.

In a paper titled "What Drives Venture Capital Fundraising?,"^ we analyze these patterns systematically. We find that regulatory changes have had an important impact on commitments to venture capital funds. The Department ofLabor's 1978 clarifica- tion of the prudent man rule, which enabled pension fimds to freely invest in venture capital, had a generally positive effect on commitments to the industry by increasing the supply of fiinds.

^ This paper and another titled "Money Chasing Deals?" were

prepared in support of this study and are available upon request from

the authors.

16
MARKET-BASED PROBLEMSAFFECTING THEINNOVA TIONPROCESS

Capital gains tax rates also appear to have an important ef- fect on fundraising at both the industry and venture organization level. Decreases in the capital gains tax rates are associated

with greater venture capital commitments. Rate changes, however, affect both taxable and tax-exempt investors almost identically. Decreases in capital gains tax rates appear to mcrease commitments to venture capital fimds not through increases in the desire for contributions to new funds by taxable investors, but rather through increases in the demand for venture capital

investments when workers have greater mcentives to become

entrepreneurs.

A key point to emerge from this analysis is the importance of

economic policies m shaping the supply ofventure capital. These

shifts are largely exogenous to the nature of technological opportunities. This pattern suggests that it is by no means clear

that the optimal number of small high-technology fmns are re- ceiving financing from the venture sector or that there is no role

for public venture capital programs. There is also the possibility,

however, that the level of venture capital funding may be too

high. Furthermore, if public policy can more readily affect the demand for venture capital than the supply of fianding, then it unclear whether direct grant programs such as the ATP can af- fect fundraising levels. The optimal level ofventure capital fimding and the manner in which it can be affected by the ATP pro- gram remain open research questions.

A second paper examines the narrow focus ofventure capital investment. Venture capital investments are tremendously concentrated, whether measured by the technological span of the firms backed, the location ofthe firms, or the size ofthe investment. One

of the key implications of "Money Chasing Deals?" is the potentially detrimental impact that this concentration can have. Panels B and C ofTable 1 document this pattern, showing the

distribution of early-stage venture financings by state and Stan- dard Industrial Classification (SIC) code in 1995. The concentration of awards in California and Massachusetts, as well as in computer

software and communications sectors, is apparent.

This concentration may be problematic, whether we examine

its impact on social or private returns. On the one hand, several models argue that institutional investors frequently engage in "herding": making investments that are too similar to one another. These

models suggest that a variety of factors-for instance, when perfor- mance is assessed on a relative, not an absolute, basis-can lead to

investors obtaining poor performance by making too similar invest- ments. (Much ofthe theoretical literature is reviewed in Devenow

and Welch [1996]; Sahlman and Stevenson [1986] present a case

study suggesting such behavior by venture capitalists.) As a result, social welfare may suffer because value-creating investments

in less popular technological and geographic areas may have been

ignored.

The concentration in technological investment areas may

be a matter of particular concern. An extensive literature on

technology races (reviewed in Reinganum [1989]) shows how a

small initial technological advantage can franslate into a sus- tained lead. If venture capital organizations neglect making small investments into a wide variety of technologies, the long-run

defrimental impact on America's competitive position may be

substantial. "Money Chasing Deals?" examines the pattern of invest- ment during the most recent period of growth (between 1987 and

1995). As venture fundraising climbed, investments remained

narrowly concentrated on healthcare and information technologies. Rather than diversifying their investments, venture groups

bid up the price paid for individual investments.

We employ a dataset of over 4000 venture investments between 1987 and 1995 as well as detailed information on capital inflows. Because gaps of one to two years between refmancings

ofventure-backed firms are typical, a price index based purely on

the changes in valuations between financings for the same company would be incomplete and misleading. We consequently

employ a hedonic approach, regressing the valuation of firms on

their characteristics such as age, stage of development, and in- dustry, as well as inflow of fiinds into the venture capital industry. We also control for public market valuations through in- dexes ofpublic market prices offirms in the same industries and

average book-to-market and eamings-to-price ratios. In this way, we seek to disaggregate whether movements in valuations reflect the flow of funds into the private equity industry or ahematively the changing composition of transactions or

shifts in public market values. We find a strong relation between

the valuation of venture capital investments and capital inflows. While other variables also have significant explanatory powerfor mstance, the marginal impact of a doubling in public market

values was between a 15% and 35% mcrease in the valuation of

private equity transactions-the inflows variable is significantly

positive. A doubling of inflows into venture funds led to be- tween a 7% and 21% increase in valuation levels. The results are robust to the use of a variety of specifications and control variables. These results corroborate practitioner claims that increas- ing capital inflows have led to higher security prices, or colloquially, "too much money chasing too few deals." (Three representative accounts over the decades are Noone and Rubel [1970],

Sahhnan and Stevenson [1986], and Asset Alternatives [1996].)

We also fmd that firms located in geographical areas where

venture capitalists tend to concentrate and in industries that are

particularly sought after increase in price even more in response

to venture inflows. This suggests that attractive, underflmded

opportunities exist in overlooked areas and technologies.

17
ATP PAPERSAND PROCEEDINGS

It is also worth noting that there is another way in which

venture capital investments are concentrated: the similarities in investment size. In particular, venture funds tend to make quite

substantial investments, even in young firms. For instance, the mean venture investment in a start-up or early-stage business

between 1 96 1 and 1 992 was $1.8 million (in 1 992 dollars) (Gompers

[1995]). The substantial size of these investments is largely a conse- quence of the demands of institutional investors. The typical venture organization raises a fiind (structured as a limited part- nership) every few years. Because investments in partnerships

are often time-consuming to negotiate and monitor, institutions prefer making relatively large investments in venture funds (typically $ 1 0 million or more). Furthermore, governance and regulatory considerations lead investors to limit the share of the fiind that any one limited partner holds. (The structure of venture

partnerships is discussed at length in Gompers and Lemer [1996,

1999].) These pressures lead venture organizations to raise substantial funds. As the venture industry has grown, the average

fund size has increased, from $30 million in 1 985 to $80 million in 1995 (VentureOne [1996]). Because each firm in his portfolio

must be closely scrutinized, the typical venture capitalist is re- sponsible for no more than a dozen investments. Venture organizations are consequently unwilling to invest in very young firms

that only require small capital inftisions. Panel A ofTable 1 compares seed and early-stage investments by venture funds with

the total amount raised by these funds.

Are Alternative Financing Sources Adequate?

It may be wondered why these inefficiencies in the venture

capital market should be a source of general concern, much less public intervention. A natural question is why entrepreneurial

firms do not rely on the several alternative capital sources that also finance entrepreneurial furns. Can small high-technology

firms raise capital from other financing sources, most notably

individual investors or banks?

Both of the leading alternative sources of fmancing for entrepreneurial firms, however, have substantial limitations. These

limitations are particularly critical in small high-technology in- dustries that are particularly interesting to policy-makers.

The informal risk capital market consists of individuals com- monly referred to as "angels." These "angels" are wealthy busi- nessmen, doctors, lawyers, and others who are willing to take an

equity stake in a fledgling company in return for seed capital.

Firms that require substantial amounts of money, however, may

not be able to receive sufficient capital from the "angel" network

because the market is dispersed with little information sharing

and the amount of invested capital tends to be small. The amount

that a firm can raise from individual investors is usually much

less than the minimum financing round that a venture fund will consider providing. Freear and Wetzel [ 1 990] report that median

fmancing round raised by private high-technology firms from

individual investors was about $200,000. 82% ofthe rounds from

individuals were under $500,000.

Similarly, bank fmancing is unlikely to fill the gap for tech- nology-based firms. Companies that lack substantial tangible

assets and have very uncertain prospects are unlikely to receive

substantial bank loans. These firms face many years ofnegative

earnings and are unable to make interest payments on debt obli- gations. This characterization applies to many, ifnot most, tech- nology-based young firms. Thus, a substantial gap exists between the resources that firms can raise from individual investors and from venture capitalists. Bank loans may also not be able to address this problem.

Awards from programs such as Advanced Technology Program

may partially fill these gaps, as well as addressing the concerns about the geographic and industry concenfration of awards dis- cussed above.

What Are Implicationsfor the Advanced Technology

Program?

These analyses have two primary implications for the adminisfrators of the Advanced Technology Program. In this final section, we highlight these implications for program management.

First, the administrators should be sensitive to the importance of the venture capital sector as a source of fmancing. In many cases, funds from the Advanced Technology Program can- not carry the technology all the way to the marketplace. At some

point, additional resources will be required. Furthermore, as dis- cussed above, venture capitalists provide a range of services in addition to their capital. These may be difficult to duplicate

through other means. Thus, venture capital is an important-and

in many instances, the best-fmancing source as high-technology firms move new products or services from conception to the market. A second insight is the need to tailor the Advanced Technology Program's awards to reflect the dynamics ofventure capital market. This awareness is likely to lead to opportunities to maximize the return from public funds. One example is the industry concentration of venture funds discussed above. It probably

makes little sense to target awards in technologies that have

recently attracted heavy backing from venture investors, such

as human genomics or Internet tools. Public fimding in areas

18
MARKET-BASED PROBLEMSAFFECTING THEINNOVATIONPROCESS

with extensive private financing may lead to a "crowding out" phenomenon: public funds may replace private expenditures for research. The total amount spent on R&D and resulting iimovation may consequently not increase much in these cases (for a

discussion ofthis problem, see Wallsten [1997]). Rather, it seems more sensible to target the broad array of technologies not at- tracting much interest from the venture community.

Similarly, it may make sense to adjust the Program's strategy

during periods when venture investors are experiencing difficulties raising new funds (e.g., much of the 1970s, the late 1980s,

and the early 1990s). A critical mechanism in the venture

capitalist's tool-kit is the staging of investments. Giving entre- preneurs only part of the money they need and tying the possibility ofrefinancing to reaching a particular technological milestone helps limit venture capitalists' losses by allowing the venture capitalist to cut off funding to underperforming firms. (By

way of contrast, corporations, which usually lack such disciplin- ary mechanisms, have been known to spend hundreds of millions of dollars on new ventures before terminating them.) During sudden fundraising droughts, however, this method can lead

to firms with promising technology being cut off from fiirther funding. As our case studies indicate. Advanced Technology

Program funds have, in some cases, allowed small companies

with promising technologies but which were experiencing tech- nological delays to reach a stipulated milestone and obtain additional financing. This may well be an attractive strategy to pursue during these periods.

References

Asset Alternatives, 1996, "Private Equity Managers Struggle to Avoid Hubris in Hot Market," Private Equity Analyst, 6 (September), 1,28-31.

Barry, Christopher B., 1 994, "New Directions in Research on Venture Capital Finance," Financial Management, 23 (Autumn), 3-15.

Brav, Alon, and Paul A. Gompers, 1 997, "Myth or Reality?: LongRun Underperformance ofInitial Public Offerings; Evidence from

Venture Capital and Nonventure Capital-Backed IPOs," Journal ofFinance, 52,1791-1821.

Devenow, Andrea, and Ivo Welch, 1 996, "Rational Herding in Financial Economics," European Economic Review, 40, forthcoming.

Freear, John, and William E. Wetzel, Jr., 1990, "Who Bankrolls

High-Tech Entrepreneurs?," Journal of Business Venturing, 5, 77-89.

Gompers, Paul A., 1995, "Optimal Investment, Monitoring, and

the Staging of Venture Capital," Journal of Finance, 50, 1461-

1489.

, 1 997, "An Examination ofConvertible Securities in Venture Capital." Unpublished working paper. Harvard Business

School.

, and Josh Lemer, 1 996, "The Use ofCovenants: An

Analysis of Venture Partnership Agreements," Journal of Law

and Economics, 39, 463-498.

, and , 1997, "Venture Capital and the Creation ofPublic Companies: Do Venture Capitalists Really Bring

More than Money?," Journal ofPrivate Equity, 1 (Fall), 15-32.

, and , 1999, "An Analysis of Compensation in the U.S. Venture Capital Peirtnership," Journal of

Financial Economics, 5 1

, 3-44.

Greenwald, Bruce C, Joseph E. Stiglitz, and Andrew Weiss, 1984,

"Information Imperfections in the Capital Market and Macroeconomic Fluctuations," American Economic Review Papers and

Proceedings, 74, 194-199.

Griliches, Zvi, 1986, "Productivity, R&D, and Basic Research at the Firm Level in the 1970's," American Economic Review, 76,

141-154.

Hall, Bronwyn, H., 1992, "Investment and Research and Development: Does the Source ofFinancmg Matter?," Working Paper

No. 92- 1 94, Department ofEconomics, University ofCalifornia at Berkeley.

Hao, Kenneth Y., and Adam B. Jaffe, 1993, "Effect ofLiquidity on

Firms' R&D Spending," Economics ofInnovation and New Technology, 2, 275-282.

Himmelberg, Charles P., and Bruce C. Petersen, 1994, "R&D and

Internal Finance: A Panel Study of Small Firms m High-Tech

Industries," Review ofEconomics and Statistics, 76, 38-51.

Hubbard, R. Glenn, 1998, "Capital-Market Imperfections and In- vestment," Journal ofEconomic Literature, forthcoming.

Jensen, Michael C, and William H. Meckling, 1976, "Theory of

the Firm: Managerial Behavior, Agency Costs and Ownership

Structure," Journal ofFinancial Economics, 3, 305-360.

19
ATP PAPERSAND PROCEEDINGS

Kortum, Samuel, and Josh Lemer, 1998, "Does Venture Capital

Spur Innovation?," National Bureau ofEconomic Research Working Paper No. 6846.

Lemer, Josh, 1994. "The Syndication ofVenture Capital Invest- ments," Financial Management, 23 (Autumn) 16-27.

, 1995, "Venture Capitalists and the Oversight of

Private Firms," Journal ofFinance, 50, 301-318.

, 1 999, "The Government as Venture Capitalist: The

Long-Run Impact of the SBIR Program," Journal of Business,

forthcoming.

Liles, Patrick, 1977, Sustaining the Venture Capital Firm, Cambridge, Management Analysis Center.

Mansfield, Edwin, John Rapoport, Anthony Romeo, Samuel

Wagner, and George Beardsley, 1 977, "Social and Private Rates

of Return from Industrial Innovations," Quarterly Journal ofEconomics, 91, 221-240.

Myers, Stewart C, and Nicholas Majluf, 1984, "Corporate Fi- nancing and Investment Decisions When Firms Have Information That Investors Do Not Have," Journal ofFinancial Economics, 13, 187-221.

Noone, Charles M., and Stanley M. Rubel, 1970, SBICs: Pioneers

in Organized Venture Capital, Chicago, Capital Publishing Company.

Reinganum, Jennifer R., 1989, "The Timing of Innovation: Research, Development and Diffusion," in Richard Schmalensee

and Richard D. Willig, eds., The Handbook ofIndustrial Organization, New York: North-Holland.

Sahhnan, William A., 1990, "The Stnicture and Governance of

Venture Capital Organizations," Journal ofFinancial Economics,

27,473-521.

, and Howard Stevenson, 1986, "Capital Market

Myopia," Journal ofBusiness Venturing, 1, 7-30.

StigUtz, Joseph E., and Andrew Weiss, 1981, "Credit Rationing in Markets with Incomplete Information," American Economic Review, 71, 393-409.

U.S. Small Business Administration, 1996, http://

www.sbaonline.sba.gov/Research_And_Development.

VentureOne, 1 997, National Venture Capital Association Annual

Report, San Francisco, VentureOne.

Vincenti, Lisa, 1996, "Trio Preps Unusual Effort to Raise Capital

for Early-Stage Funds," Venture Capital Journal, 36 (February),

10

Wallsten, Scott, 1997, "Rethinking the Small Business Innovation Research Program," in Lewis M. Branscomb and James H.

Keller, editors, Investing in Innovation, Cambridge, MIT Press.

20
MARKET-BASED PROBLEMSAFFECTING THEINNOVATIONPROCESS

Table I: Volume of Venture Capital Activity. The table provides an overview of investment activity by U.S. venture capital

organizations. Panel A indicates the total amount raised by venture capital funds and the amount of early-stage investment, all expressed in 1994 dollars. No data are available on the number ofearly-stage investments prior to 1981. Panels B and C display

the amount ofventure investments in 1995, disaggregated by the leading states and industries. The sources are VentureOne [1996]

and unpublished databases of Venture Economics and VentureOne.

Panel A: Amount of Venture Activity Venture Capital Early-Stage Investments by Venture Funds

Year Raised in Year $ ofFinancings Number ofFinancings

1977 91 414

1978 442 520

1979 503 755

1980 1260 802

1981 1713 806 227

1982 2061 813 343

1983 5516 1707 413

1984 4931 1689 568

1985 4240 1194 529

1986 4429 1478 716

1987 5550 1440 796

1988 3822 1272 674

1989 3858 1119 623

1990 2173 705 571

1991 1569 458 335

1992 2822 646 435

1993 3008 765 368

1994 4596 1005 499

1995 4536 1438 611

Panel B: Leading; States, Venture Financing, 1995

State $ ofFinancings % of Total # ofFinancings

California 2274 30.6 437

Massachusetts 772 10.4 131

New Jersey 724 9.7 36

Texas 352 4.7 40

Illinois 340 4.6 29

Panel C: Leading Industries, Venture Financing, 1995

Industry (SIC Code) $ ofFinancings % ofTotal # ofFinancings

Communications & networking 1376 18.5 180

Software & information services 1239 16.7 291

Retailing & consumer products 1207 16.2 90

Medical compounds 716 9.6 113

Medical devices & equipment 607 8.2 108

21
R&D Spillovers, Appropriability and R&D Intensity:

A Survey Based Approach

by Wesley M. Cohen and John P. Walsh' ^

Abstract

This paper uses recently collected datafrom the Carnegie

Mellon Survey on Industrial R&D to build an industry-level

empirical model of R&D spillovers, appropriability and R&D

intensity, in order to evaluate the quality of our survey-based measures and to deepen our understanding of appropriability

andspillovers and their impacts on R&D spending. Wefind that

the more appropriable are the rents to R&D, the higher the R&D intensity of an industry. We also find that intraindustry

R&D-related information flows negatively affect some

appropriability mechanisms, but lead to greater R&D intensity. This last result indicates that, controlling for the effect of

intraindustry information flows on appropriability,

intraindustry R&D information flows complement firms ' own R&D efforts. We also use these newly-developed measures of

appropriability to rank industries with respect to their

appropriability conditions. Wefind that our survey-based measures provide a basis for identifying industries where

appropriability is weakest, and, hence, where one might expect

R&D spending to be particularly subject to marketfailure.

Introduction

R&D spillovers and the appropriability ofrents due to R&D

are seen as key determinants of innovative activity and perfor- mance and are key concepts motivating policy interventions in support of industrial R&D. Unfortunately, direct measurement of

these two notions has proven elusive. Moreover, while there is now strong evidence supporting the argument that R&D

spillovers have important effects on innovative performance and

productivity growth, empirical analyses of the direct effects of

appropriability on the conduct of R&D have not yielded a consensus. This paper uses recently collected data from the Carnegie

Mellon Survey on Industrial R&D to build an industry-level empirical model of R&D spillovers, appropriability and R&D intensity, in order to evaluate the quality of our survey-based measures and to deepen our understanding of appropriability and

spillovers and their impacts on R&D spending.

' Wesley M. Cohen. Department of Social and Decision Sciences,

Carnegie Mellon University, Pittsburgh, PA and John P. Walsh.

Department of Sociology, University of Illinois at Chicago.

^ We thank Zvi Griliches, Adam Jaffe and other participants in the NBER workshop on R&D Spillovers and technology policy sponsored

by the Advanced Technology Program, Dept. of Commerce, for

We find that the more appropriable are the rents to R&D, the higher the R&D intensity of an industry. We also find that intraindustry R&D-related information flows negatively affect some appropriability mechanisms, but lead to greater R&D intensity. This last result indicates that, controllmg for the effect of

intraindustry information flows on appropriability, intraindustry

R&D information flows complement firms' own R&D efforts. We also use these newly-developed measures of

appropriability to rank industries with respect to their

appropriability conditions. Although within- industry heterogeneity ofresponses limits our ability to discriminate across industries, we still find that our survey-based measures provide a basis for identifying industries where appropriability is weakest,

and, hence, where one might expect R&D spending to be particularly subject to market failure. Section 2 ofthe paper describes the industry-level model of

appropriability and R&D that we intend to test. In Section 3, we

describe the data. Section 4 presents the model and the construction of measures. In Section 5, we present the empirical

results from the industry level model. In Section 6, we present

rankings of industries intended to suggest where R&D market

failures are most likely to occur. We conclude in Section 7.

Conceptual Approach and Model

Infraindustry R&D spillovers are the uncompensated benefits derived by firms from their rivals' R&D activities. Conceived

as such, there is little distinction between R&D spillovers and

appropriability, which refers to the degree to which firms can

appropriate the profits from their own R&D activity. At times,

however, the notion of spillovers are identified with intraindustry

R&D-related information flows. While R&D information flow may

be a key determmant of R&D spillovers and appropriability, it may not be the only determinant. Conflating R&D spillovers (and

hence appropriability) with intramdustry information flow has

led some scholars to claim that the empirical effect of

appropriability on R&D spending is ambiguous (Cf. Cohen and

Levin [ 1 989]). These ambiguous findings may resuh from failing to examine the effect of appropriability on R&D spending while

controlling for the independent and direct effect of infraindustry

comments, and thank Marco Ceccagnoli, Scott Jaspan and Scott Bricker

for research assistance. The co-investigator with Professor Cohen on

the data collection for the Carnegie Mellon Survey of Industrial Research and Development is Professor Richard Nelson. We also thank the Advanced Technology Program for supporting this project.
MARKET-BASED PROBLEMSAFFECTING THEINNOVA TIONPROCESS

R&D information flows. With separate measures ofappropriability

and intraindustry R&D information flows available from the Carnegie Mellon Survey of Industrial R&D, we can control for the effect of intraindustry R&D information flows on

appropriability, and then separately observe the possibly

countervailing effect ofthese information flows on R&D.

There are two ways to think about appropriability. The most

straightforward way is to think of it as simply the profits due to invention that firms within an industry are able to retain for themselves. A second way builds on institutional observations of

Levin, et al. [ 1 987] that firms protect those rents in different ways,

with patents, secrecy, lead time, complementary capabilities, and

so on. This observation suggests a more multidimensional view

of appropriability in which it is conceived of as the rents due to invention that firms retain through the use ofeach ofthese mechanisms, recognizing that in some industries, only one such mechanism may be emphasized, and in others, combinations of these mechanisms may be used. Explicitly controlling for the particular

mechanisms employed can capture the differential effects of dif- ferent mechanisms on R&D information spillovers and, indirectly, on R&D incentives. For example, heavy reliance on secrecy may

produce significantly less information spillover than use of patents, even ifboth are equally effective in conferring appropriability. In this paper, we test an empirical model relating R&D infor- mation flows, appropriability and R&D intensity to one another,

where appropriability is characterized as the degree to which

different appropriability mechanisms or strategies increase the

rents due to R&D. Diagram 1 characterizes the central relation- ships that we posit.' In this framework, we suggest that the more

effective are appropriability mechanisms such as secrecy or patents in protecting the profits due to invention, the greater are the

incentives to conduct R&D. At the same time, we suggest that there is a reciprocal relationship between the effectiveness ofthe different appropriability mechanisms and the extent of

intraindustry information flows. To the degree that R&D-related

information flows are stronger within an industry, the more diffi- cult appropriation of rents to R&D will be, notwithstanding the

particular mechanism employed. However, use of specific

appropriability mechanisms may, at the same time, condition the

extent of intraindustry information flows. For example, if firms

rely on secrecy, we would expect information flows to be less than if they rely on patents, which disclose information. As

argued above, we expect that once the negative effect of

intraindustry information flows on appropriability is controlled,

we are then in a position to see whether the direct effect ofthose

flows on R&D is negative or positive. We would expect the

direct effect to be positive to the degree that information from

rivals complements own R&D effort and negative to the degree

that it substitutes for own R&D. We also expect that the greater the R&D intensity of an industry, the greater the information

flows. Moreover, to the degree that firms in an industry conduct more R&D, the more able they are to exploit those flows. Both of

these arguments imply a positive effect ofR&D on intraindustry

R&D information flows.

Data

The proposed empirical analysis relies principally upon sur- vey data from the 1994 Carnegie Mellon Survey (CMS) of Industrial R&D in the United States. These survey data provide measures of the extent of knowledge flows both across competitors

and fi-om outside the industry, measures of the effectiveness of

appropriability mechanisms and measures of the R&D activity

and performance of firms and industries, among other variables. Building on prior empirical research showing that there are important cross-mdustry differences in the factors affecting technical advance (e.g. Nelson etal. [1967]; Cohen [1995]), data were

collected at the business unit level rather than at the level of the

enterprise as a whole. The respondents were R&D unit directors

for manufacturing firms. Our sample frame was drawn largely

from the Directory of American Research and Technology

(DART). The U.S. survey effort yielded 1489 completed questionnaires, representing an unadjusted response rate of 46% and

an adjusted response rate of 54%". For this analysis, our sample

is aggregated to 54 3-digit manufacturing industries with more

than six cases. We also use archival sources to collect data on fum sales and employment and industry sales, number of firms

and concentration ratios. Table 1 provides the descriptive statistics on firm and busi- ness unit size and R&D intensity for the sample. The average

firm and business unit sales revenues are $4.4 billion and $1.7

billion respectively. As the figures for the first and third quartiles

indicate, the business imit and firm size distributions are quite

broad, including numerous small firms. The sample mean R&D

intensity, defmed as business unit R&D divided by business

unit sales, is 2.3%.

' Diagram 1 omits the exogenous variables hypothesized to affect each ofthe endogenous variables.

" Our response rate was adjusted for respondent ineligibility inferred from our nonrespondent survey.

23
ATP PAPERSAND PROCEEDINGS

Table 1. Descriptive Statisticsfor CMS sample

Variable

Business Unit Employees (lOOO's)

Business Unit Sales ($ millions)

Firm Employees (lOOO's)

Firm Sales ($ millions)

Business Unit R&D Intensity (%)

N Mean Median 1st Quartile 3rd Quartile

4.40 0.45 0.12 2.10 959

1720.00 120.00 20.00 650.00 833

20.00 3.30 0.30 17.00 1115 4440.00 550.00 40.00 2750.00 1129

2.33 1.92 0.67 4.61 700

Measures And Model

In this section, we will discuss our measures of spillover,

appropriability and R&D intensity and our model for estimating

the relations among these variables, based on Diagram 1 . Our

unit of analysis for this model is the industry, and thus all measures are constructed at the industry level. We begin by dis- cussing our measures of appropriability, and then describe our measures of spillover and R&D intensity.

Appropriability, as suggested above, is a multidimensional

concept. Our survey contained twelve items asking respondents

about the percentage of innovations for which a given

appropriability mechanism was effective for protecting the firm's competitive advantage from those innovations (i.e., the extent to which this mechanism is important). The key mechanisms considered in our survey include secrecy, patents, other legal mechanisms (e.g., copyright or design registration), complementary

manufacturing facilities and know-how, complementary sales and

service, and being first to market, asked separately for product

and process innovations. Using a respondent-level exploratory

factor analysis of these twelve items, we find three distinct di- mensions to appropriability. Employing Bartlett's factor score (Mardia, Kent and Bibby [1979]), which uses the factor loadings

of each measure as weights, we construct for each respondent

normalized factor scores for each of the three factors. We then

use simple averages for respondents in each industry to construct the following industry-level measures of appropriability:

• CAPABILITIES/LEAD: For product and process iimo

vations, the extent to which complementary manufac

turing facilities and know-how, complementary sales and

service and being first to market are effective in protect

ing innovations.

• LEGAL: For product and process innovations, the extent to which patents and other legal mechanism are

effective in protecting iimovations.

• SECRECY: For product and process irmovations, the extent to which secrecy is effective in protecting innovations.

In addition to these three measures of appropriability, we

also have the following measures for spillover and R&D intensity:

• INFO-RIVALS: Our measure of mtraindustry R&D in- formation flows is the percentage of respondents in an

industry saying that information from rivals either suggested new R&D projects or contributed to R&D project

completion in the prior three years.

• R&D-INTENSITY: We use a sales weighted average of

the R&D intensities of the business units in each in- dustry, where R&D intensity is business unit R&D di- vided by business unit sales. The major focus ofthis paper is to develop a model ofR&D

intensity that distinguishes between the possibly offsetting ef- fects of spillovers and appropriability on R&D intensity by controlling for the reciprocal effects of each on the other and measuring separately their effects on R&D intensity. The work of

Spence [1984], Levin and Reiss [1984] and Cohen and Levinthal

[ 1 989] all highlight offsetting effects ofR&D information spillovers on R&D spending. Although, at the industry level, such infor- mation flows diminish the appropriability of firms' profits due to R&D, they may also complement firms' R&D and thus yield off- setting effects on industry R&D spending. Because they did not

benefit from data permitting separate measurement ofintraindustry

information flows and appropriability, prior studies have claimed

that these offsetting effects account for the ambiguous effect on R&D intensity of their operationalizations of industry

appropriability (e.g., Cohen and Levinthal [1 989]). Following prior

work, we consider industry R&D intensity to be a fiinction of

three industry- level variables, namely technological opportunity,

demand, and the degree to which firms expect to appropriate

returns to their innovations (cf Cohen [1995]). We mclude our

three appropriability mechanism effectiveness scores, namely

LEGAL, CAPABILITIES/LEAD and SECRECY, along with our measure for intraindustry information flows, INFO-RIVALS and

our control for market mediated information flows. Our model

structure and measures should allow us to discern the

countervailing effects ofintraindustry information flows on R&D

24
MARKET-BASED PROBLEMSAFFECTING THE INNOVATIONPROCESS

intensity. To the extent that intraindustry spillovers influence

R&D spending either by complementing firms' R&D efforts or by making those efforts more efficient, they are considered to

reflect a dimension oftechnological opportimity. Thus, our variable, INFO-RIVALS should reflect one dimension oftechnological opportunity. In our specification, building on prior empirical

work, we include variables representing other dimensions oftech- nological opportunity as well. We follow Levin, Cohen and

Mowery [1985] and Cohen, Levin and Mowery [1987] by includ- ing a survey-based variable representing the vitality of the underlying scientific and engineering knowledge base. Knowledge

flows from supplying firms have also been considered a dimension of technological opportunity since they might have the ef- fect of making firms' R&D more efficient, and so we include a measure of information flows from suppliers as well. We also

control for the effect of industry demand by including a measure

of each industry's average annual rate of growth.

Following Diagram 1, we use these variables to estimate a system of five equations using two stage least squares.^ In

addition to the endogenous variables in the diagram, our model

includes several exogenous variables that measure extraindustry

information flows (such as those from universities, suppliers and

customers), ties between R&D and production units, the emphasis on product versus process innovation, the importance of

university research, and demand growth (for details, see Cohen

and Walsh [1998]). We also control for market-mediated channels of intraindustry information flows (to make our measure INFO-RIVALS more closely tied to nonpecuniary spillovers).

There is at least one predetermined variable in each equation, so

the entire model is identified. Since we are examining industrylevel effects, all variables are expressed as industry averages. To

control for sampling error in these estimates, and as a partial control for the heterogeneity of firms within our sample industries, we weight each case by the square root of the number of

observations in that industry.

Results

Table 2 presents the results of the two-stage least squares

estimates for the model shown in Diagram 1 . We will discuss the

results for each endogenous variable.

' Although perhaps more appropriate for a larger sample, three

stage least squares regressions are also employed for estimation, partly

to exploit its efficiency properties and partly as a robustness check on

the two stage least squares results. The results are largely similar, with some changes in significance levels for certain effects. Three-stage least squares results available from the authors.

' Given a small number of industry observations in our sample

(N=54), we will use a cutoff of a .10 confidence level to qualify a

coefficient estimate as "significant."

Intraindustry R&D Information Flows [INFORIVALS]

The first column of Table 2 gives the results for the predictors of spillovers (INFO-RIVALS). The results suggest that the

particular appropriability mechanisms that tend to be used in industries indeed affect information flows. The clearest and most

robust effect is exercised by SECRECY, which has a negative and

significant effect on information spillovers.* A negative effect of

secrecy in general is quite sensible and conforms to priors. An

interesting result here is the fact that LEGAL apparently exercises little or no effect on intraindustry R&D information flows. The reason might be that patents, for example, are designed to have two effects. While they are supposed to confer

appropriability, they also diffuse information. Thus, while patents may deny rivals from exploiting some information, at least in some forms, they at the same may provide information that benefit rival R&D. The absence of any clear sign for the effect of

LEGAL is consistent with these offsetting effects. As expected,

higher industry R&D intensity (R&D-INTENSITY) appears to contribute significantly to intraindustry information flows.

Appropriability Strategies [SECRECY, CAPABILITIES/LEAD, LEGAL]

The next three columns give the results for our measures of

appropriability. Across the three appropriability mechanism equations, our measure for infraindustry information flows (INFORTVALS) exercises a significant, negative effect only on SECRECY.

Otherwise, the qualitative effects are typically very small.'' R&DIntensity

The last column gives the resuhs for our featured R&D in- tensity equation. We see that our infraindustry information flow

variable [INFO-RIVALS] has a significantpositive effect on R&D

intensity. We also see that the variables representing the effec- tiveness ofthe three appropriability sfrategies [SECRECY, CAPABILITIES/LEAD and LEGAL] all have additional positive ef- fects on R&D intensity. In the three-stage least squares equation, all three effects are significant. This result suggests that

'' If, however, we drop the number of R&D competitors from the model, with which INFO-RIVALS is somewhat collinear, the sign on INFO-RIVALS becomes consistently negative and the statistical significance of the INFO-RIVALS variable strengthens in all three appropriability mechanism equations, although rarely exceeding

conventional thresholds.

25
ATP PAPERSAND PROCEEDINGS

Table 2. Two-Stage Least Squares Regression ofIntraindustry Information Flows, Appropriability and R&D Intensity

VARIABLE INFO RIVALS SECRECY

CAPABILITIES

/LEAD LEGAL

R&D

INTENSITY

INFO_RIVALS + + +*** SECRECY +

CAPABILITIES/ + + LEAD

LEGAL

R&D INTENSITY

+

r2 50 42 37 51 53

Notes: N=54

***p<.01

** p<.05

* p<.10

The model includes exogenous variables measuring extraindustry information flows, technological opportunity, demand growth

and controls for market-based information flows. Full model and three-stage least squares results availablefrom authors.

once one controls for the influence of intraindustry information

flows on appropriability, the direct influence of intraindustry R&D

information flows on R&D intensity is strong and positive. This

result suggests a strong complementarity between own R&D

and information spilled out from rivals' R&D. Thus, we appear to have distinguished a direct effect of intraindustry information

spillovers from that of appropriability. In contrast to prior empirical results in the literature, our

results provide evidence that greater appropriability encourages

industry R&D intensity. We have obtained this result, however,

after controlling for the effect of intraindustry information flows on appropriability itself. That effect turns out to be negative and

significant only in the case of SECRECY. We also observe a strong positive impact of intraindustry R&D information flows on industry R&D intensity. These results, and particularly those

in the R&D intensity equation, conform with predictions grounded

in prior theory and empirical work. Not only do they, therefore,

support the way economists tend to see the role ofappropriability

and spillovers, they also elicit some confidence in the survey- based measures that we have employed.

Industry Level Indicators OfAppropriability And

Intraindustry Information Flows

The fact that our industry-level measures of appropriability

and intraindustry R&D information flows are performing in a

sensible way in our model suggests that these measures may

actually reflect what we think they do. If we take this proposition

seriously, too seriously perhaps, these measures may allow us to discriminate across industries according to the extent of

intraindustry R&D information flows and appropriability. By doing so, we should be able to identify industries where the spillovers

from R&D are greatest, and R&D investment is most subject to market failure.

To probe whether our industry-level measures allow us to distinguish industries by level of R&D spillovers and

appropriability, we present for the fifteen most R&D intensive

industries in our sample the industry means for our measure of

intraindustry R&D information flows and the factor scores repre- senting the effectiveness of the three core appropriability strate- gies that we identified. In addition to presenting industry means

in Figures I through 4, each ofthese figures also present ninetyfive percent confidence intervals around those means, which is particularly important ifwe wish to use these measures to claim,

for example, that secrecy or legal mechanisms are more effective

in one industry than in another. For the fifteen most R&D intensive industries in our sample.

Figure 1 presents our measure of intraindustry information flows,

namely the percentage of respondents reporting that information from rivals suggested new R&D projects or confributed to the completion ofexisting projects. The ninety-five percent confidence intervals suggest that although the percentages vary from 26% for plastic materials and synthetics to 1 00% for agricultural chemicals, these differences are not very discriminating. If we rely on the confidence intervals, we make, at best, coarse

distinctions. For example, we can claim that the information flows

in plastic materials and synthetics, industrial organic chemicals

and communications equipment are lower than those in agricultural chemicals, or that those in plastic materials and synthetics

are lower than those in motor vehicles and equipment, computers or semiconductors, but it is not possible to discriminate the numerous industries in the middle from those either at the top or bottom.

Figures 2 through 4 present the factor-based indexes for the effectiveness of the three appropriability strategies involving,

respectively: the exploitation of complementary capabilities and

lead time in Figure 2; the exploitation of secrecy in Figure 3; and

the exploitation of legal mechanisms, particularly patents, in Fig26
MARKET-BASED PROBLEMSAFFECTING THEINNOVATIONPROCESS

ure 4. In each ofthese figiires, an industry factor score index of

zero signifies that the industry is average with regard to the

appropriability strategy in question. As in the case of Figure 1, the overriding impression conveyed by the confidence intervals shown in Figures 2 through 4 is that it is difficult to fmely dis- criminate across industries. We do fmd, however, that we can

usually distinguish groups of industries with regard to the effec- tiveness of each appropriability strategy. For example. Figure 2

suggests that the two top industries, photographic equipment

and supplies and electronic components, appear to benefit from

greater effectiveness of complementary capabilities and lead

time, and agricultural chemicals less so than other industries. With regard to the factor score for the effectiveness of secrecy considered in Figure 3, four industries, namely miscellaneous

chemical products, semiconductors, drugs and plastic materials

and synthetics appear to rank significantly higher than the other eleven industries. With regard to patents and other legal mechanisms considered in Figure 4, six industries appear to rank higher

than the other nine, mcluding agricultural chemicals, medical in- struments, computers, special purpose machinery, construction

machinery and equipment and drugs.

The question is how might we exploit these coarse distinc- tions to identify industries where R&D spending might be particularly subject to market failure. One approach might be to ex- clude those industries where at least one of the three sets of

mechanisms appears to work particularly well. On this basis, we

can eliminate the eleven industries named above, leaving communications equipment, industrial organic chemicals, motor vehicles and equipment, and measurement and control devices as

industries where R&D may be particularly subject to market fail- ure. Since our regression analyses above suggest that the link between intraindustry R&D information flows and

appropriability—at least as measured in terms of the strength of

appropriability strategies—is not that close, one cannot conclude that there is some close negative monotonic relationship

between the two. Motivated by both theory and our other empirical fmding that intraindustry information flows positively

contribute to industry R&D activity, it would be useful therefore

to consider those information flows in any consideration of in- dustries to which government might provide R&D support. Consider, therefore, the four industries which our results would suggest may be most susceptible to a market failure in their R&D

decisions, namely communications equipment, industrial organic

chemicals, motor vehicles and equipment, and measurement and

control devices. Presenting our featured measure of intraindustry

information flows. Figure 1 shows that of these four industries,

two—industrial organic chemicals and communication equipment—rank thirteenth and fourteenth with respect to these flows,

and rank significantly below the top ranked industry, though not

significantly below the large group of mdustries m the middle

which includes the measurement and control devices and the motor vehicles and equipment industries. Thus, our measure of

intraindustry information flows does not offer a strong basis for discriminating across industries for the purpose of public support of R&D, unless one is willing to accept a lower confidence

level in drawing such distinctions. Additional information can be brought to bear. For example,

recall from our regression analyses above that, to reflect the

intraindustry information flows associated with nonpecuniary

R&D spillovers, it was useful to confrol for the importance of

market-mediated channels ofinformation flow across rivals. Thus,

consider Figure 5, which ranks the fifteen most R&D intensive

industries by our composite factor index reflecting the importance of these market-mediated channels. We observe that the importance of such channels in motor vehicles and equipment is not significantly distinguishable from their importance in the drug

industry, which is the top ranked industry on this dimension.

This suggests that while intraindustry information flows are considerable in motor vehicles and equipment, they may be marketmediated to an important extent and may thus not reflect nonpecuniary R&D spillovers. In confrast, the importance of market

mediated information channels is quite low for the measurement

and control device industry, and significantly below that of the communications equipment and motor vehicles and equipment

indusfries. This finding might suggest that if one had to choose

between these three indusfries as targets for government support sfrictly on the basis of this very crude evaluation of the likelihood of generating social gains from R&D that are not privately captured, the measurement and confrol device industry

would be a sensible candidate.

At this point, we are reluctant to recommend public support

for industrial R&D in any industry too seriously. Clearly, there are numerous substantive and methodological assumptions underpiiming the analysis that need to be subject to considerable

scrutiny. For example, it is a leap of faith, although one to which we have subscribed, to assume that our crude measures of the effectiveness of appropriability sfrategies accurately index the share ofsocial returns to R&D that are privately garnered. Moreover, any decisions to allocate public funds in support of R&D

across indusfries need to consider other factors that condition

the absolute level of social gains from R&D, such as technological opportunity and demand conditions. In this regard, any guid- ance offered by our measures of appropriability reflects a pre- sumption that all else is being held equal—but all else is not held

equal, and these other factors must therefore be considered. For

example, while our measures might commend the measurement

and control device and industrial organic chemical industries as candidates for government support, one might argue that the

27
A TP PAPERSAND PROCEEDINGS

technological opportunities facing the former may be much greater than those facing the latter. We suggest, though, that however

tentative our conclusions may be for the moment, survey-based measures of appropriability and intraindustry information flows

may be of some use in guiding our thinking about what industries might be the most fruitful beneficiaries ofgovernment R&D

support from a social welfare perspective. Thus, while the utility

of these measures for guiding policymakers' allocation of R&D

support across industries is compromised by their coarseness and particularly by the heterogeneity of responses within industries, the measures may be of use even within these constraints.

Conclusion

We began with several objectives: to create industry-level measures of appropriability and spillovers and validate these with a model of R&D intensity, spillovers and appropriability;

and to use these measures to identify industries where market

failures may be more likely. We used the Carnegie Mellon Survey

of Industrial R&D to provide the data with which we constructed measures for these various concepts. Factor analysis suggests that appropriability can be partitioned into three distinct components, representing complementary capabilities and lead time advantages; secrecy; and patents

and other legal mechanisms. Our industry-level model of R&D

intensity, spillovers and appropriability suggests that the measures of spillover and appropriability derived from the CMS may

be reasonably valid indicators ofthese difficult-to-operationalize

concepts. The overall model fits well and most of the results reported in the model reflect prior understandings ofthe expected

relationships. All three of the appropriability dimensions are

positively related to R&D intensity, as is R&D information flows

from rivals. In addition, intraindustry R&D information flow is negatively related to the effectiveness of secrecy as a strategy of

appropriation. Finally, higher R&D intensity is associated with

greater information flows. Thus, we see a complementarity be- tween R&D and spillovers, with each positively related to the

other, controlling for appropriability (and opportunity and de- mand growth).

We then use these measures to rank industries in terms of

information flows and appropriability, focusing on the more R&D

intensive industries. Here, we find that while industries differ, within-industry variance is substantial. We can point to certain

industries as being near the top or the bottom, but we are reluctant to make distinctions among most of the industries on any

given measure. We suspect small numbers at the industry level (with fewer than 10 cases in some industries), measurement error, and firm-level differences all contribute to this within-industry

variance. Although within-industry heterogeneity of responses

limits our ability to discriminate across industries, we still fmd

that our survey-based measures provide a basis for identifying

industries where appropriability is weakest, and, hence, where

one might expect R&D spendmg to be particularly subject to market failure. Overall, our results suggest that survey-based measures of

appropriability and spillovers may be useful in modeling the rela- tionships between R&D intensity, spillovers and appropriability. In addition, appropriability may be best thought of as a multidi- mensional concept, as firms and indusfries emphasize different

strategies for appropriating the returns to innovation. These strat- egies have differential effects on information flows and, indirectly, on R&D intensity. One question that needs further re- search is how these various dimensions are related to a firm's or

industry's total appropriability of rents due to mnovation. In

addition, further research might focus on how to refine our measures to increase their discriminatory power. Finally, our results suggest that R&D intensity is positively related to appropriability,

when confrolling for the effects of spillovers, and that spillovers

have a net positive effect on R&D intensity.

References

Cohen, WM [1995] "Empirical studies ofinnovative activity." In

P. Stoneman, ed. Handbook ofthe Economics ofInnovation and

Technological Change. Oxford: Blackwell.

Cohen, WM and RC Levin [ 1 989] "Empirical studies of irmovation and market structure." In R. Schmalensee and R. Willig, eds. Handbook of Industrial Organization. Amsterdam: North Holland.

Cohen, WM and DA Levmthal [1989] "Innovation and learning: the two faces ofR&D." Economic Journal 99: (397) 569-596.

Cohen, WM, RC Levin and DC Mowery [1987] "Firm size and

research-and-development intensity: a reexamination." Journal ofIndustrial Economics 35: (4) 543-565.

Cohen, WM and JP Walsh [1998] "R&D spillovers,

appropriability and R&D intensity: a survey based approach."

Paper presented at the NIST Advanced Technology Program

International Conference on the Economic Evaluation of Technological Change, Washington, DC, June 15-16.

Levin, RC, WM Cohen and DC Mowery [1985] "R&D

appropriability, opportunity, and market structure: new evidence on some Schumpeterian hypotheses." American Economic Re- view Proceedings 75:20-24.

28
MARKET-BASED PROBLEMSAFFECTING THE INNOVATIONPROCESS

Levin, RC and PC Reiss [ 1 984] "Tests ofa Schumpeterian model

ofR&D and market structure." In Z. Griliches, ed. R&D, Patents and Productivity. Chicago: University of Chicago Press for the >©ER

Levin, RC, AK Klevorick, RRNelson and SG Winter [1987] "Appropriating the returns from industrial research-and-development." Brookings Papers on Economic Activity 3 :783-83 1.

Mardia, KV, JT Kent and JM Bibby [ 1 979] Multivariate Analysis.

New York: Academic Press.

Nelson, RR, MJ Peck and ED Kalachek [ 1 967] Technology, Eco- nomic Growth and Public PoUcy. Washington: Brookings Institution.

Industry R&D Intensity

R&D Information Flows

+/-

Appropriability Mechanism

Effectiveness

Diagram 1. Relationships across Industry R&D Intensity,

Intraindustry R&D Information Flows and Appropriability

Mechanism Effectiveness.

Spence, AM [1984] "Cost reduction, competition, and industrial performance." Econometrica 52:101-121.

29
Comments On "R&D Spillovers, Appropriability, And R&D Intensity:

A Survey-based Approach

by David C. Mowery'

This paper presents some very interesting preliminary empirical results from the Carnegie-Mellon University survey of

industrial R&D managers. The bulk ofthe paper is devoted to an

analysis ofthe determinants ofR&D intensity, however, and the

links between this analysis and program evaluation or design

raise a number ofmany questions. I first consider the core analysis and findings ofthe paper and then briefly discuss its implications for program design and evaluation. Cohen & Walsh use their survey data to disentangle the

significance of Cohen & Levinthal's "2 faces of R&D"—^how

significant, relative to one another, are the discouragements to R&D investment resulting from limitations on firms'

appropriability of the returns to their R&D, vs. the encourage- ment to R&D investment that may result from any

complementarities between intra-industry knowledge flows (nonpecuniary spillovers) and own-frrm R&D investment? As the authors note, these competing influences are difficult to disentangle, not least because the firm's choice of instruments to protect the results of its R&D may directly affect the nonpecuniary

spillovers associated with that R&D. For example, patents may

have very different results from secrecy, if one believes that the information disclosed in patents is useful to rival would-be in- ventors (assumes that knowhow is of little importance; note the abundant historical evidence that patents alone are insufficient).

The authors use a system of equations to estimate the deter- minants of( 1 ) the importance of intra-industry knowledge flows;

(2) the importance of secrecy, legal protection, and lead time/

capabilities as mechanisms for appropriating retums; and (3) drawing on the results ofthe first two specifications, the determinants

of R&D intensity. All of these analyses are conducted at the

"line of business" level of analysis, relying on survey respondents' estimates of R&D spending and other business conditions in their unit of larger, diversified corporations. The most

important results in this section are (1) the fmding that spillovers

do indeed promote higher levels of R&D investment; and (2) the finding that spillovers are not related in any very well-behaved

fashion to the operation of mechanisms of appropriability, i.e., one can't find any robust, monotonically negative relationship

between spillovers and the mechanisms of appropriability ana- lyzed in the paper. The authors rely exclusively on survey data for their analysis, and should incorporate some data from other, non-survey

sources into this analysis and as a validity check on the original survey data. For example, some ofthe R&D intensity variables.

and a number ofthe variables that attempt to measure the role of

"external" sources of technical knowledge might be measured

with data from other sources, including data on patents.

Among their detailed results, several are ofparticular interest. The authors find that legal means of appropriating retums have little/no effect on intra-industry flows. This result raises several questions: (1) how much information of use to other inventors do patents convey, especially in the absence of methods for rival inventors to access knowhow? Considerable evi- dence suggests that patents alone are insufficient to spark widespread imitation; they are rights to exclude, rather than docu- ments with commercially relevant information. This raises question (2): Even assuming that patents do convey information, the often-lengthy lead times in their publication means that sectors

in which lead time is especially important should find patents

particularly unhelpfiil sources ofinformation. This question merits

further exploration.

Although the authors highlight this point for their measure

of the importance of secrecy, all oftheir measures ofthe importance of different channels for capturing the retums to R&D

confound some concept of the frequency with which they are used with a notion oftheir importance; in the case of patents, the only one ofthese mechanisms for which we have good data, the

distribution of importance is enormously skewed within a large

portfolio. And in other work, Cohen & Walsh report a decline in the effectiveness ofpatents for capturing retums simultaneously

with an upsurge in their use. The authors' empirical results on the use by firms of secrecy

to appropriate the retums to R&D seem especially problematic.

Note the inconsistency between the statement that "...public

channels of information flow across competitors are sfronger

when similar channels between universities and industry are

stronger." (p. 15) and "... as the amount of information conveyed

through public channels increases, firms will again become more concemed about keeping their own fmdings secret, even while it may be more difficult to do so." Another anomalous result is in the equation for R&D intensity, where "pecuniary" spillovers, defmed by the authors as high ratings of importance for collaborative R&D, reverse engineering of competitors' products, and inward licensing of tech- nology, has a negative coefficient. Since many of the activities included by the authors in this category presumably are sup- ported by firms' in-house R&D investments, the negative coeffi- cient seems strange. Moreover, it implies that R&D-intensive

' Dr. David Mowery is a Professor of Economics at the Haas

School of Business, U.C. Berkeley.
MARKET-BASED PROBLEMSAFFECTING THEINNOVATIONPROCESS

firms invest little in categories that theyjudge as important channels of inward spillovers. This result, suggesting that contract R&D, collaboration, and other types ofcollaboration are of lower

importance in more R&D intensive industries, also seems to contradict casual empiricism.

Moving on to consider links between their results and pro- gram design or evaluation criteria, the authors next look at the

extent to which their measures differentiate among individual 3-

digit industries, focusing on the 15 most R&D-intensive, in an

attempt to determine where one would identify the most finitful targets for public subsidies. Interestingly, most oftheir measures

provide little basis for distinguishing sharply among industries within these top 15. Survey resuhs, according to the authors, thus far don't provide much guidance to the policymaker seeking economically defensible targets for R&D subsidies. Moreover, the authors' groups of industries that display similar levels of reliance on various devices for appropriating the returns to R&D display very little similarity in other dimensions, suggesting that these results must be treated with caution. The authors also examine the distribution of ATP-fiinded

projects across their industries, to determine whether the selection of projects is related in any meaningful way to the level of

spillovers, finding that R&D intensity predicts the number of

fiinded projects, while higher appropriability is associated with

smaller numbers of fimded projects. But this result is not very

significant, since, the authors note that they do not include the

total number of proposals. In other words, this finding may

simply mean that more R&D-intensive industries with weaker

mechanisms for capturing the returns from R&D write more pro- posals.

In conclusion, this paper presents a number of very intriguing results, and its use of survey data for collecting detailed information on inter-industry differences is a model for other

exercises in such areas as "iimovation surveys." But the guid- ance from this paper for program design and evaluation is limited. Such guidance as does emerge is most usefiil for broader

assessments of choice among policy tools or design, rather than

for the analysis of specific programs such as ATP. For example,

Are R&D subsidies the most desirable instrument for "highspillover" industries? How does the payofffrom R&D subsidies

to furns compare with those from R&D grants or subsidies to other "sectorally relevant" institutions, such as government labs, universities? This paper's survey results could be used to address some of these questions. But the results say little about

the utility R&D subsidies for furns or groups of firms, the ATP

policy design. Another challenge to the design ofATP concerns

the paper's finding of significant knowledge spillovers among

firms within an industry? In view ofthe fact that survey respondents do not view patents as an important vehicle for these knowledge spillovers, the encouragement for patenting of research by

ATP and other "R&D partnership" programs (e.g., CRADAs)

sponsored by the federal government may in fact limit, rather than promote, knowledge spillovers.

31

POLICY GOALSAND PROGRAMDESIGNISSUES

B

Orchestrating Technology Policies - German Lessons for Evaluations

by Dr. Carsten Dreher and Dr. Stefan Kuhlmann*

Abstract

Debates in Germany on technology policy suffer from the absence of a comprehensive theory of technological change and

from not taking into account the evidence and lessons ofevaluation experiences in order to allow a more differentiated picture on policy making, instruments and technologies. This paper offers a closer look on these issues. It provides (1) an over- view on German S&T-policy, (2) summarizes the experiences of

evaluations of German S&T-policies, and, (3) elaborates the

roles ofdifferent instruments in the development and diffusion process ofnew technologies. The conclusion will be derived as

lessonsfor S&T-policy evaluationsfrom a Germanperspective.

Introduction

In the seventies the German Science and Technology (S&T)

policy made a forward leap from a scientific and basic research

orientation towards a policy for industrial umovation. Then - and

partly still today - in Germany the advocates of 'Strukturpolitik'

and 'Ordnungspolitik' fought a serious battle.'

• Advocates of state technology policy measures use an

argumentation based on the theory of market failure.^ According to this theory, market structures in a national economy may lead to sub-optimal market results due,

for instance, to insufficiently market sizes or high barri- ers to market entry. Or, due to the influence of external

effects, the market mechanism may not create paretooptimal results. Under such circumstances the state may

be justified in intervening - or may even have to inter- vene.

• On the other hand, 'opponents' of state technology

policy generally advance two arguments,' firstly, that the rationales by policy decision-makers differ funda-

"Head of department Innovations in Production of the FraunhoferISI and lecturer for economics at the Technical University Karlsruhe.

Contact: Fraunofer Institute for Systems and Innovations Research

(ISI), Breslauer Strasse 48, D-76139 Karlsruhe, Phone: ++49-721-6809-

305, Fax: ++49-721-6809-I3 1, e-mail: cd@isi.fhg.de.

' cf overview on the German debates GIELOWu.a 1985, EWERS

1990, BOGELEIN 1990, FRITSCH u.a. 1993, OBERENDER (Hrsg.) 1994, KLODT 1995.

' e.g. HAUFF, SCHARPF 1975; GAHLEN, STABLER 1986; MEISSNER, ZINN 1989.

mentally from the rationales of enterprises active in the market place and, secondly, that because of their dis- tance from the market policy-makers do not have suffi- cient information at hand compared with enterprises

actively engaged in the market."* Hence, the debates about the sense or nonsense of industrial innovation policies in the U.S. with their rigidity and sometimes

emotional intensity (Branscomb, Keller 1998) sound familiar to Ger- man ears. This debate suffers from two deficits,' the first being that there is no comprehensive theory oftechnological change which

includes action by the state as an active determinant in models,

rather than ' shaping conditions' . Secondly, after decades ofpracticing technology policy, it would seem appropriate, possibly even essential, for the debate to take into account the experience

gathered so far, and to evidence a more sophisticated differentia- tion between technology policy instruments. Here, a valuable contribution can be made by discussion of

the results derived from the examination and evaluation ofmany

individual state technology policy measures. Evaluation research has now acciunulated a considerable wealth of experience with

regard to the practice and adminisfration of technology policy measures, and in their optimization.^ Thus evaluation research could put more detail into policy debate, thus endowing it with

practical usefulness for policy.

As contribution to this problem the paper offers a closer look on the different roles of S&T-policy instruments and their

affinity to the subject of state intervention - the technologies.

Considering the procedural aspects of evaluations could be helpful, too. This allows conclusions on the governance of S&Tpolicies in general. This paper relies on two recently published

books (Dreher 1997, Kuhhnann 1998) which conclusions were combined for this conference. For this aim the paper will (1) provide an overview on Geman

S&T-policies, (2) summarize the experiences of evaluations of

German S&T-policies and (3) elaborate instruments in the devel

3 e.g. KIRCHGASSNER 1979; STAUDT 1986; FELS 1989; EWERS 1 990; OBERENDER, STREIT (Hrsg.) 1 99 1

.

" e.g. STREIT 1984, BEIRAT 1985.

5 cf. SOETE 1991, MEYER-KRAHMER, KUNTZE 1992 and as again highlighted by the OECD-conference on 'Policy Evaluation in Innovation and Technology: Towards Best Practice' in Paris 1997.

^ e.g. BECHER, KUHLMANN 1995.
ATP PAPERSAND PROCEEDINGS

opment and diffusion process of new technologies. The conclu • sion will be derived as lessons for S&T-policy evaluations from

a German perspective.

Science and Technology Policy in Germany

In an international comparison, the German science and tech-

*

nology system is considered to be well developed: in 1 993, 475,000

persons (full-time equivalents) performed research and development (R&D). Altogether, the expenditure on research in Germany

in 1 995 amounted to 79 billion DM. As a percentage ofthe gross domestic product (1988: 2.9; 1995: 2.3) this still puts Germany

among the world leaders, but in the last few years it has fallen somewhat behind Japan (1988: 2.9; 1995: 3.0) andthe USA (1988:

2.8; 1995: 2.6) (see OECD, April 1995; BMBF 1998). The R&D

infrastructure is regarded as highly differentiated (cf. Reger/

Kuhlmann 1995, 1 Ipp, data from BMBF 1996, see figure 1): • Industry accounts for the major share of R&D in Ger- • many: in 1995, almost 70 per cent ofmainly applied re- search and experimental development. Only relatively

few large multinational enterprises, especially in the chemical and electrotechnical industries, conduct longterm, application-oriented basic research.

• The higher education institutes (i.e. universities, technical colleges; 1 1 0,000 R&D staff) have the second highest research expenditure. They concentrate on basic research and long-term application-oriented research, • largely funded by the federal states and by the Deutsche

J Forschungsgemeinschaft (DFG), a highly independent

funding body financially supported by the federal and

the state governments. In the course ofthe last 1 0 years, additionally, there has been a marked increase in the share ofthe roles ofdifferent share of industrial R&D in the research budget of several universities, especially technical ones. Figure 1: The German Research Landscape 1995

a? Hdrwhote 14,9 1,3 0,8 52J_

State-*- — Proximity -* Industry

t Herrnann xxn Helrchcite Aisecialioa of fierrt*! Ffesfearch C*r»fei SHSF), i.e- tt* rvjUonal rtsisarch certlrei I 5o(ifn«ii-\'/'rih«lm-{jeibr«te SotsIv J Wta.}, formeatv Blue List 3 AiF, Confjtferaliori dt Ir-dustnal Fleae»ch Associi^iarjs 'Qlo von Guericke' ftV.

A group of federal and state research establishments

( 1 1 ,000 R&D staff) carries out research which is more or

less directly associated with the tasks ofthe public authorities that finance it, usually federal or state ministries (e.g. the development ofnorms and standards, test- ing procedures, certification, etc.). The 1 6 national research centers (22,500 R&D staff, now

acting jointly as Helmholtz Society) perform particularly activities with a long-term orientation which ap- pear to have a high degree of risk, require relatively

large research teams and incur high costs. In recent years the national research centers have diversified their fields of activity considerably and are now active in a wide range of areas, especially in high energy physics, space technology, medicine, biotechnology, applied

mathematics, software technology and environmental

technology.

The mstitutes of the Max Planck Society (MPG; 9,000

R&D staff) mainly perform basic research in selected

fields of natural sciences and the arts. They concenfrate on new research topics of potential future importance which either have not yet found an established place at higher education institutes, or - because of

their interdisciplinary character or the resources they

require - cannot find a place there. The institutes ofthe Fraunhofer Society (FhG; ca. 4,000

R&D staff) have the task of promoting the practical

application ofscientific knowledge through long-term application-oriented and applied research. The FhG performs

mainly contract research which is financed partly by in- dustry and partly by the public sector. The organization

regards itself as the 'interface' between science and in- R&D expenditure bn DM

36
POLICY GOALSAND PROGRAMDESIGN ISSUES

dustry in Germany. No other German R&D institution has grown so rapidly over the last 1 5 years.

• The laboratories of the Confederation of Industrial Research Associations (AiF) perform tasks in applied re- search and experimental development for the sector-specific needs ofindustrial enterprises. Their work, which is financed partly fi-om public sources and partly by industry, particularly benefits small and medium-sized enterprises (SMEs) which are grouped into sector-specific re- search organizations.

• The institutes ofthe Leibnitz Society (formerly known as

'blue list') form aremainingcategory ofinstitutions jointly

founded by the federal and state level (ca. 10,000 R&D

staff). After the unification many East German institutes were promoted this way. One ofthe main reasons for the extent ofinstitutional differentiation in the German research 'landscape' is the fact that there is no

strong, central policy body responsible for science, research and

technology: within the federal system ofGermany, it is essentially the 16 states {Lander) that are responsible for science and aca- demic research. Although in the course oftime the Federal Govern- ment has, in agreement with the states, taken on many areas of

responsibility, there is always a certain degree of competition be- tween the central authorities and the states, and also between the

various states. The states are running nearly all higher education

institutes; they maintain (to a varying extent) non-university re- search institutions and - also to a varying extent - launched their own technology pohcy programs. This does lead to some redundancies in the capacities of the research and innovation system,

but it also guarantees a decentralized, 'autonomous' structure of

research capacities, even outside the large cities and agglomerations.Against this background, it is not surprising that German S/T

policy relating to industry over the last decade was described as

'diffusion-oriented' rather than 'mission-oriented' (cf Ergas 1987,

192). A strategic industrial or technology policy with definite missions, such as is pursued in France, can hardly be implemented in a

decentralized structured research system. Another reason for the

stronger 'diffusion orientation' of German S/T policy is the basic

attitude prevalent in public policy that the state does not wish to be

regarded as a 'director' of industry, but only as the guarantor ofan

effective academic training system (higher education institutes), a

high level of basic research and a sound technology base for in- dustry (industrially oriented R&D programs). However this basic

position, as critics continually point out, only works to a limited

extent in practice (cf e.g. StarbattyA^etterlein 1990).

Since the 1 950s, German S/T policy - as in other westem Euro- pean industrialized countries - has continually extended the scope of its activities; this can be illustrated using a 'shell model' (cf

Meyer-Krahmer/Kuntze 1992; Braunling and Maas 1988). In this model, the core area of S/T support in the 1950s comprised the

(basic) imiversity research and 'special area research' carried out in federal and state research establishments.

• A first ' layer' is characterized by big technology programs

fi-om the mid 1950s onwards, showing a meirked orienta- tion towards goals similar to those ofthe USA, mainly in the fields ofnuclear technology, aerospace and data processing, and later microelectronics (see e.g. Weyer 1994)

• A second research support 'layer' was developed fi-om the beginning ofthe 1 970s, in order to create and support

the conditions necessary for the export of technologyintensive goods. Public fianding flowed into research

projects of industry and institutes for applied research to promote cross-sectoral technologies (e.g. materials), key

technologies (e.g. microelectronics) and technological

systems (e.g. transport systems).

• In the 1 970s, the reform policies ofthe Social Democratic

government triggered the formation of a third research support 'layer', complementing the goals ofthe first and

second layers by research activities in the areas of envi- ronment, public health and the employment market whose

aims related primarily to social policy.

• From the end ofthe 1 970s, a fourth ' layer' emerged, using

the instruments ofinnovation policy: their aim is the diffii- sion ofinnovative or improved technologies, also among

SMEs and in less developed geographical regions. This

includes the support of activities in R&D as well as the building up and strengthening ofan infi-astructure for the support of technology transfer fi-om the science system

into industry. This layer grew during the 1980s but has

lost some importance since the early 1 990s. The shell model clearly shows that over the last 40 years these

focal orientations of policy have not succeeded one another, but have accumulated! The result has been the formation ofa differentiated range ofS/T policy instruments, extending fi-om institutional support measures, over financial research support programs and

the creation of institutions for technology transfer, to regulative

policy measures. Today, roughly speaking, halfthe S/T fiinding of

the Federal Ministry for Education, Science, Research and Technology (BMBF)'' is spent in the form of institutional support and

the other half in the form of research, technology or iimovation

programs (i.e. project fiinding).

' in 1994 the Federal Ministry for Research and Technology

(BMFT) was merged with the Ministry for Education and Science. The

official acronym is now BMBF.

37
A TP PAPERSAND PROCEEDINGS

A remarkable turbulence shook the R&D infrastructure after the German unification: Between 1990 and 1994 the S/T infra- structure ofthe former German Democratic Republic experienced

a drastic and painful process of shrinkage - e.g. the industrial R&D personnel decreased from 70,000 to 12,000 -, and the re- search capacities of the East were basically restructured (see

Holland/Kuhlmann 1 995). Today, they constitute a generally compatible part ofthe total German research landscape, and are beginning to participate in European collaborative research. Looking

back critically, there was practically no alternative to this highspeed, pro-active, and to some extent authoritarian adaptation process in the East. Nevertheless, unique opportunities for cre- ating a modernized S/T infrastructure in Germany as a whole

were not taken; the persevering interests of the long-standing

and politically well equipped West German R&D institutions dominated this process of change (Mayntz 1993 and 1994), al- though serious and qualitatively high standing evaluation pro- cedures were applied.

Evaluation of S&T Programs in Germany

The functions and the use of evaluation procedures in the German S/T system are wide-ranging,* and the expectations and

requirements of the evaluation users vary significantly

• from the provision of legitimization for the disfribution of public money and the demonstration of adequate

and effective use of the fiinding,

• via targeting and 'controlling' in the sense of improved

management and 'fine tuning' of S/T policy programs,

• to an attempt to improve transparency in the rules of

the game and the profusion of research funding and

subsidies, and to enhance the information basis for shaping S/T policies, in the sense of a 'moderation' between

diverging and competing interests of various players

within the S/T system.

When taking a historical view, we can differentiate two basic development lines of evaluation - similar to other indusfrialized countries - in the field of S/T in Germany. They can be sketched - like the S/T policy measures described above - as a shell model:

• The 'core' consists of the 'peer review' procedures

and later additionally the measurement ofresearch per- formance of single researchers or groups (using

bibliomefrics etc.) as science-internal instruments for the allocation of funds to research institutions. These

instruments are widely used in the German S/T system,

in particular in the reahn ofbasic and long-term applied

research.

this section of the paper follows mainly Kuhlmann 1995.

• An 'outer layer' of this shell model consists of evaluation studies in the sense of 'impact analyses ' of S/T

policy programs. This evaluation concept could be understood as science-external, initiated prevalently by

policy-makers in order to prove the achievement ofpolitically set scientific, technological, economic or societal targets. Impact analyses spread since the late 1 970s

in particular in the areas of innovation and strategic

technology programs.

It is important to note that evaluation efforts of the 'impact

analysis' type did not supplant the peer review approach: with

the historical emergence oftechnology and innovation programs,

the impact analysis practice was added to the existing 'system of

evaluation'. Compared to other national S/T systems, though, it

is remarkable that significant parts ofthe German S/T system, in particular the field of university research institutions (let alone

university teaching) have not been evaluated at all so far! (Kuhhnann 1997) Considering the background of this confer- ence we are focusing on S&T program evaluations in this paper.

In Germany evaluation studies in the sense of impact analyses of S/T policy programs have been increasingly carried out

since the end ofthe 1 970s (cf. Kuhlmarm/Meyer-Krahmer 1995; Meyer-Krahmer/Montigny 1 989), usually performed by outside

independent research institutes on behalf ofthe policy adminisfration system.

An early 'landmark evaluation ' that served as a model for many other program impact studies in Germany was conducted

in the field of support measures for small and medium-sized enterprises (SMEs): from the end of the 1970s to the end of the 1980s, particular attention was paid in West Germany to SMEs

within the framework of innovation policy. The most important measures were the 'R&D Personnel Costs Subsidies Program

(PCSP)' of the Federal Ministry for Economic Affairs and the 'Promotion of Research Personnel Growth (GP)' which was

launched later by the Federal Ministry for Research and Technology. Between 1 979 and 1 988 ahnost 20,000 SMEs were spon- sored under the PCSP, involving a total amount ofDM 3.2 thousand million. SMEs which had taken on new R&D personnel

between 1984 and 1987 were entitled to apply under the GP;

about 6,000 enterprises were sponsored, most ofwhich had also taken part in the PCSP; the program volume is estimated at DM

230 million. Both programs have been subject to a joint evaluation by the German Instimte for Economic Research (DIW) and

the Fraunhofer Institute for Systems and Irmovation Research

(ISI) (cf Meyer-Krahmer et al. 1 984; Kuntze/Homschild 1 995). In Germany, only a few programs have been evaluated in such depth

and over such a long time period. The evaluation concept for the

38
POLICY GOALSAND PROGRAMDESIGNISSUES

PCSP was established even before the start of the program; this meant that useful basic information for the evaluation could be

ascertained from the application forms of all applying companies. A monitoring evaluation was conducted during the fu-st phase ofthe program between 1 979 and 1 984, which was divided

into 2 sections covering short- and medium-term effects. The

overall conclusion of the evaluation efforts was that - although

there were some free-rider effects - the sponsored SMEs from the manufacturing sector greatly increased their R&D efforts since

the end ofthe 1 970s, and that the two programs have contributed

to this development. Viewed against its historical background

and taking into account the results of the evaluations, the decision to discontinue all personnel subsidies at the end of the 1980s did not seem justified. At least for the GP, the results ofthe

evaluation showed that a further strengthening of the innovation potential of SMEs could be expected (Kuntze/Homschild

1995,52).

Since this 'landmark' effort most SME-oriented programs

have been evaluated comparatively often, m particular m the

fields of application-oriented industrial research and develop- ment (e.g. BecherAVolff 1 995), technology transfer (e.g. P*firrmann/

Schroeder 1995; Hafkesbrink/Horst 1995), and technology diffu- sion (for manufacturing technologies, e.g. Lay 1989; Lay 1993; Wengel et al. 1995) - i.e. those measures with a wide public appeal. Since the end ofthe 1980s also strategic programs aiming at strengthening R&D in key technology lines have been increas- ingly analyzed (e.g. laser technology: Reinhard 1995;

Microsystems technology: Becker 1995; Eschenbach et al. 1995; Lorenzen 1990).

This experience of nearly two decades of program evaluation helped to establish a certain 'evaluation scene' in Germany,

consisting of a group of experts and institutes in the fields of

economics and social sciences covering a relative broad range of

methodological approaches and evaluation instruments (see contributions in Becher/Kuhknann 1995).

In 1992/93 the German Ministry for Research and Technology (BMFT) initiated a 'Metaevaluation' ofthis gathered experi- ence (cf Kuhhnann/HoUand 1995). The task ofthe study was to suggest improvements in the methodological instruments and

the management of program evaluations of the BMFT on the

basis of a comparative analysis and assessment of methods and

results of 50 program evaluations which the ministry has commissioned since 1985. In the following we will report on the results of this 'Metaevaluation'. The 50 evaluation studies were

characterized according to a set of simple categories, and the

following pictiire emerged hereby:

• Type of evaluation: three evaluation types could be

differentiated: retrospective (ex post) evaluations, (monitoring) studies and prospective (ex ante) analyses. Ex

post evaluations aim mainly to analyze the effects achieved in order to check the (further) need for public

promotion measures. Monitoring evaluation should

support the program management and make corrections

in the course possible, if required. 'Strategic' ex ante analyses should assess the effects of alternative tech- nology policy approaches and interventions

prognostically; 'operational' ex ante studies examine

the attainability of already set program targets and the probable effects, depending on the chosen program design. The elements ofthis typology mostly appeared in a mixed form: the ex post element was most frequently

represented, however the monitoring type appeared al- most as often, and mainly as a product ofmonitoring or ex post evaluations, the operational ex ante element;

strategic ex ante analyses were conducted more fre- quently than expected.

• Target group analyses: almost all studies analyzed the

situation or the behavior of potential or actual target groups for S/T policy measures relatively comprehensively. In most cases these analyses were based on oral and/or written questionings.

• Analysis of the status of research and technology in the target area of the program under evaluation, in national and in international terms: this can be of great

significance for the assessment of the necessity and

appropriateness ofthe policy interventions. In approximately half of the studies, in which such an analysis

would have been sensible, examinations were carried out in relative detail; for the other half, such analyses were conducted either only to a lunited extent or not at

all. Technology foresight concepts (e.g. Delphi ap- proach) are only beginning to be utilized in connection

with evaluation studies.

• Analysis oftechnology diffusion achieved: in more than

halfthe cases in which analyses ofthe actually realized (ex post) or the anticipated (ex ante) technology diffu- sion would have been useful, such examinations were

conducted either only to a limited extent or not at all; in

less than half of the studies comprehensive diffusion analyses were carried out.

• Data basis: only few studies base their analyses on a combination of standard S/T indicators (e.g. R&D,

patent, trade statistics from 'official' sources) and per- ceptual data (e.g. results of own surveys, data on applicants). A smaller group of studies used both types of

data scarcely or not at all. The majority of the studies was based exclusively, or predominantly, on the results of own inquiries (mainly perceptive data).

39
ATP PAPERSAND PROCEEDINGS

• Policy recommendations: most studies contain rela- tively comprehensive conclusions for S/T policy action. Less than half contain only poor recommendations or none at all (however it must be remembered that policy papers could have been submitted separately

from the final report).

• Duration ofevaluation: The major part of the studies had a duration of approximately two years. Some very

short-term studies (under one year) by their very nature

contain only very limited questions and can scarcely be

described as thorough evaluations.

• Publication: the major part of the studies is virtually inaccessible to the public; the final reports are frequently

available only as 'gray literature' or only as an mtemal

document. A mere quarter of all studies have been published as books (and appear in the catalogues of libraries and booksellers).

The program evaluation practice ofthe BMFT was concentrated on a limited number of promotion fields in the past years. More than 80 per cent of the financial expenditurefor evaluation studies was spent in the areas of innovation promotion,

health research, technology and working conditions, information technology, materials research, and transportation research,

which make up 40 per cent ofthe total research promotion ofthe BMFT. The majority of the promotion fields, measured in the spending on evaluation, show only few evaluation activities; however, since the late 1980s they have been slowly but steadily

spreading to other fields. By comparison with other mdustrial

nations, it appears that the German expenditure on program evaluation in proportion to fimding budgets is in midfield (i.e. between

0.5 and 1.0 percent).

Did all these evaluation efforts concerning S/T policy pro- grams finally have any influence on the practical policy-making! In a few cases, in fact, drastic consequences were drawn:

e.g. in 1989 a planned but not yet realized program was stopped:

a DIW evaluation recommended to hold back a program supporting the dissemination of compound materials in SMEs which the BMFT had been considering - neither the technology development nor industry were estimated to be 'ripe' enough at that time

for such a promotion. The evaluation saved many million Marks

of public money from being wasted. Nevertheless, normally the impacts of evaluation studies are not so immediately apparent,

but they can have a lasting effect: in many cases the results of

The promotion has focused on big FMS with 5-10 connected machine tools. But manufacturing conditions emphasized small systems because flexibility requirements in German

manufacturing (not the same mass production as in Japan or

U.S.) would have increased the costs for informatjion technologies and material flow automation exfremely. These underlying

mainly on the interest and the orientation ofthe individual policymakers in charge. There is no regular public forum for debates on

evaluation results. The goal of the 'Metaevaluation' was to identify typical

problem zones and significant requirements for fixture evaluation

studies, which should be taken into consideration in the discus- sion about the fiirther development of evaluation practice. We

identified several crucial lessons:

• Although ab-eady an old requirement of evaluation re- search, the systematic scrutiny of the underlying as- sumptions ofeconomic or technological problems which

make a state intervention necessary has repeatedly

proved to be difficult. This important analysis was of- ten only very insufficiently carried out, which is in itself an expression of unsolved methodological problems.

• Evaluation practice must always be developed in the context of the specific conditions in the relevant pro- motionfield (ranging e.g. from health research to sfra- tegic technologies, and to pilot programs for technol- ogy-based start-up companies). Especially the methodological approach (e.g. the use of adequate indicator systems for evaluations on diverse program levels) must

meet the specific demands ofthe promotion field. Highly

standardized procedures are imsuitable.

• There were only few horizontal evaluation studies of

different but related policy initiatives used in the process ofresearch plaiming. The inclusion of activities of

other actors in the field of research and technology

promotion, and the increasing overlapping and fiision of S/T fields could have been taken into account here. Horizontal evaluation studies could be ofgreat interest to various user groups.

• The significance of assessment processes along the

lines ofpeer review must not be underestimated. The

designing of the interface with evaluation studies is a conceptual problem which has not been solved satis- factorily yet.

• In several larger evaluation studies a strained (at times

explosive) relationship developed between the detached

(as required) evaluation activity on the one hand and a

certain involvement in the management ofa program on

the other. Such an evaluation concept 'is in danger of

deviating too far from the ideal scene of extensive inde- pendence, since its close involvement in the innovation support program and the ensuing allegiance to the program aims can prevent an objective evaluation'

(Eschenbach et al. 1995, 192). Double fiinctions must

be picked up at the outset and organized transparently,

in order to guarantee the independence ofthe evaluators.

40
I

POLICY GOALSAND PROGRAMDESIGN ISSUES

• A more intensified user orientation of evaluation studies reduces possible conflicts due to diverging expectations of the performance and results. Ideally, different

interests in an evaluation study are deliberately included

in the concept and possible contradictions are thus proj ductively utilized.

j • Evaluation studies are carried out in order to support S/

' T policy decisions, as well as in the implementation of

running programs and when designing new ones. Therefore, studies should also contain policy recommenda-

. tions as a rule. If the results of the empirical survey

j seem to point to other political conclusions than the commissioner of the study expects or wishes, then it

1 must be guaranteed that these deviating votes can still be explicitly presented and also published - otherwise

the credibility of evaluation processes is basically at stake.

It would doubtless be premature at this stage to speak of an

'evaluation culture' in the field of German S/T policies: up till now only 'peer review' evaluations are widely accepted, while

evaluation studies have raised some interest but not yet occupied an assured position in the policy-making system.

Using evaluation procedures as one source ofcreating trans- parency m S/T policy 'games in multiactor networks' (Scharpf

1993) would imply an analysis of the frequently contradictory

. rationales of S/T policy players in order to facilitate mutual cri1 tique and learning processes. This would also involve the as- sessment of indirect and unintended individual and interactive impacts of S/T policy measures in societal, economic and ecological spheres. However, the mterest of policy-makers in such

j advanced evaluation concepts is limited: in Germany the authoriI ties commissioning evaluation efforts generally tend to be adf ministrators with departmental orientations and do not, as the

I system stands, have a strong interest in studies with a far-reachi ing perspective.

S&T-Policy Instruments and the Role ofTechnology

Whereas the observation of German evaluation practice rec- ommends an assurance of basic and agreed principles and an

I actor oriented input into the public discussion about the impacts

and future of S&T policy actions, for policymakers within the public arenas the following questions arise:

' mainly following DREHER 1 997.

'« MEYER-KRAHMER, KUNTZE 1 992, p. 106 using a productlife-cycle model and VDI/VDE (LUTZ 1992, p. 33) referring to a not

specified 'maturity degree'. " cf. SCHWITALLA 1993; GRUPP 1992; SCHMOCH u.a. 1988.

• Which different instruments are available?

• and where and when is the right time to use them?

Answers to these questions allow a useful input into the way role evaluations should play in a national S&T-system.

The process of identifying the tasks of different technology

policy instruments and defining the prerequisites for their appli- cation leads to directly plausible ideas about possible areas of

application.^ Arnold, Guy 1992, Meyer-Krahmer, Kuntze 1992 and

Liitz 1992 attempt a classification of technology policy instru- ments based on stylized paths oftechnology development which

are not operationalized any further, either qualitatively or quantitatively'". A classification based on measuring the activity levels of scientific research, industrial R&D and turnover is undertaken by Grupp (Ed.) 1993, p.218." Estimations of the possible

economically realizable potentials through numbers of possible

users or the absolute turnovers to be expected is not undertaken

in this model.

However, as emphasized by Jacobs 1990, it is the problems

associated with diffiision which are particularly relevant for the

political 'steering' of the development and spread of technology.'^ Jacobs emphasizes that income from technological innovations can only be realized at the point of use or sale. Molina

1994 analyzes the diffusion paths of new technologies and their dependency upon agreements and constructs. In doing so, he

emphasizes that technology diffusion flourishes best in a situa- tion where account is taken ofthe specific nature of demand for the technology. Only in this way can the demand orientation in technology policy, advocated e.g. by Meyer-Krahmer 1996, p. 19, be realized. Thus a description oftechnological development and diffu- sion has to define the number of possible users (adopter potentials) in absolute terms, so as to provide technology policy deci- sion-makers with information on the size of beneficial effects when selecting instruments. '^ It is also necessary to precisely

determine the state of the technology and its compatibility with

(ftiture) user structures, and to identify the attainable adopter

potentials in detail. A reference model, elaborated in Dreher 1997,

offers a characterization of the development and diffusion process using the above criteria. These criteria have to be reflected

in the prerequisites for technology policy instruments and in their predefmed tasks. Despite the nimierous different mathematical formulations, theories of the diffiision of new technologies

have one feature in common: diffiision curves are 'S curves'. The

'2 cf. ARNOLD, GUY 1992.

'3 on the necessity see also STONEMANN, DIEDEREN 1994,

MEYER 1995.

41
ATP PAPERSAND PROCEEDINGS

S-curve also reappears for various adopter categories (Rogers

1983). The reference model is based on the product life cycle

model of Abemathy, Utterback 1978, the diffusion phases of

Rogers 1893'" and on diffusion patterns of firms gained from

empirical diffusion research, particularly Lay, Michler 1989. Four

phases are distinguished, each having different characteristics with regard to the state of technology and the characteristics of

adopter firms (more details in Dreher 1997, pp. 54-86, see

figure 2): in Phase I, the technology is used for the first time

in Phase II, the first imitators use the technology

in Phase III, a dominant design emerges and diffusion

accelerates

in Phase IV, imitating firms cause diffusion to progress to the

saturation limit. These characteristics could be easily connected to certain instruments oftechnology policy. By focusing on different types

of financial subsidies for a selective technology policy (for the

fill! range ofpossible instruments see e.g. Meyer-Krahmer, Kuntze

1992), the following tasks of the instruments have to be considered as well as the necessary conditions which have to be ful- filled before using them (see figure 3).

Figure 2: Phases in the Development

and Diffusion ofa New Technology

Figure 3: Tasks and Prerequisites for

Use ofGermanTechnology Policy

Instruments (financial subsidies only)

Phase

State of

technology

Technology

development

process

Central competitive advantage

Compatibility with

adopter

structures

Size of

adopting

enterprises

Adopter

category^'

Max. proportion of

adopter

potential

I many variants evolve

Uncertainty, searching new or better functionality Low no distinction Innovator 0.025 N

n

selection of

variants Stabilization

quality and

reliability of

offer Medium tend to be

large firms

early adopter 0.16N

m

formation of dominant

design Optimization price

tend to be

large and

medium-sized

early majority adopters

0.5 N

IV

dominant

design plus market niche

offers

Optimization and special adaptation developments

price plus customer-specific adaptations and

after-sales services

Complete

compatibility, individual adaptation

mediiun and

small firms

late majority

adopters and

laggards

N

Prerequisites for use

Instrument Task State of technology Compatibility with adopter structures

Target group of promotion

measures

Category of

possible adopters

Supply-oriented

project promotion

Development of new

variants, stabilizing suppliers

open; need not be mature for application

Not necessary

Tend to be larger

suppliers, research

institutes

innovators

Demandoriented project promotion

Development of new

variants, stabilizing suppliers, opening up new application fields, demonstration projects

open, but should be

able to be 'tried out' Not necessary Tend to be larger suppliers iimovators

Promotion of

consortia ('Verbundprojekte')

Transfer of new

technology to concrete

applications combining

suppliers, users and

research institutions

open, but the prototype stage of

application should

already have been performed

elsewhere

Not necessary

initially

Tend to be larger

suppliers, users of all categories, research

institutes

innovators, early adopters

Supplyoriented,

indirect-specific promotion

Selecting or stabilizing individual options, increasing numbers of

suppliers by financial subsidy

definable; stable, 'masterable'

variants. Medium Suppliers eaiiy adopters

Demandoriented,

indirect-specific promotion

Selecting or stabilizing a dominant design, accelerating diffusion by financial subsidy to users

clearly definable, low complexity, dominant design should be at least in sight

High Adopters

early majority adopters

overview in e.g. KORTMANN 1995, p. 24-29.

42
POLICY GOALSAND PROGRAMDESIGN ISSUES

The importance of this adjustment process is imderlined by

Stoneman, Diederen 1994 who, taking as reference an optimally

defined diffusion path, point to the possibility that through the

intervention of policy, diffusion may take place too strongly or

too soon considering optimal welfare effects. Empirical confirmation from environmental policy is provided for example by

Holzschumacher 1 994 and Kortmann 1 995 . Ifthe determinants of

technological development and the prerequisites for use coincide, then the instruments can be used effectively. If they do not

coincide, sub-optimal results or even failure will be the outcome.

This postulation gives rise to the resulting assignment of instru- ments to specific phases (see figure 4):

• Direct project promotion should take place in phase I.

• The promotion of consortia combining suppliers and

users can be initiated at end of phase I and also possibly at the beginning of phase II.

• Supply-oriented, indirect-specific promotion is appropriate in phase II.

• User-oriented indirect-specific promotion does not ap- pear to make sense before the beginning of phase III (detailed discussion in Dreher 1997).

Figure 4: Affinity of Technology Policy Instruments

(fmancialsubsidies) to Diffusion Phases

Adopter potential

A

Phase 1

The empuical test of this taxonomy used the promotion of

advanced manufacturing technologies in Germany as an example

(cf Dreher 1997). These technologies were promoted by the Ger- man Federal Mmistry of Research and Technology from 1971 -

1 992, with over 1 .5 billion DM. The development and diffusion of these technologies were observed together with the used

technology policy instruments and the success ofthe individual promotion scheme. Hence, the success ofthe individual scheme

could be assessed whether or not it has considered the above

identified preconditions. As empirical sources were used: (1) a survey on the diffusion of advanced manufacturing technologies in (Western) Germany, (2) the analysis of the 20 years of

promoting AMT and (3) results of independent evaluations of

the individual schemes. In total 7 technologies with 25 promotion schemes were observed. The empirical results confirmed

the above developed taxonomy as the examples illustrate.

The Promotion ofComputer Aided Design (CAD)

CAD was promoted in Germany since 1974 starting with

projects elaborating sector-specific software solutions. Until 1983

technology policy focused from 1984 to 1988 on subsidies for adopters within an indirect-specific promotion scheme. In total ca, 460 million DMwere spent fi-om 1 974 to 1 988. The diffusion of CAD started in the seventies, had reached

1.5 per cent in 1980 in the target group of manufacturing tech- nologies suppliers. In 1984 7 per cent of the firms used CAD,

1 989 over 50 per cent were reached. Evaluations of the schemes showed that in the first phase

the promotion strategically opened up adopter potentials by of- fering supported solutions (phase I of the model). After these attempts a demand-oriented promotion pushed (cf Lay, Wengel

:1989) the diffusion successfully, fulfilling the prerequisites of

phase III ofthe model.

The Promotion of Flexible Manufacturing Systems

(FMS)

The promotion ofFMS started m Germany in 1 97 1 . Exploring basic principles and solving fundamental information tech- nology problems were the first projects. Special applications were

the topic of a second round in the eighties using consortia of

suppliers, users and research institutions. The diffusion never took off to the extent policy and engi- neers had predicted. In the target group of manufacturing tech- nologies suppliers in 1 983 I per cent used FMS, 1 989 7 per cent of the sample used FMS.

I 43
ATP PAPERSAND PROCEEDINGS

conditions were correctly identified by prospective studies of

the late seventies. But interests of big suppliers and research

institutions in order to fulfill their technical 'dreams' predominated policy decisions. In the end ahnost 1 million DM of subsidy went into each ofthe implemented FMS in the target group. Compared to CAD this seems to be no success story. The results highlighted that planning and evaluation ofnew

policy actions should consider the characteristics of the tech- nology and the state ofdiffusion together with target-group characteristics and administration procedures. This may enhance the

efficiency of the technology policy actions and the indicators may support better ex-ante evaluations.

Conclusions

The paper illustrated the diversity and difficulties of gover- nance of the German S&T-system, the history of S&T program

evaluations and the roles and differences of instruments if tech- nology - the issue all is about - is considered in its specificity's

and how it will or should be used by future adopters.

The lessons learned for German S&T policies are manifold:

• The requirements the metaevaluation of the German

S&T evaluation experience has identified have to be

ensured in the fiiture in order to deliver facts, information and sound policy recommendations.

• Considering the diversity of actors and interests an

evaluation is not the fmal statement but part of an arena where it has to take some sort of a moderating task. Identifying the actors and their rationales has to be a

central task of an evaluation as well in order to ensure

the necessary transparency in a democratic society. A

multi-perspective evaluation may offer such an opportunity.

• But if evaluation is part of a political process with no

clear-cut right or wrong not only transparency is essential. For achieving a constructive role focusing on the next steps in the future instead of blaming the wrongdoers has its merits. Hence, an outlook of the developments is important which could use the methodologies

oftechnology foresight, technology assessment, diffii- sion studies, patent analysis, bibliometrics etc. in order

to paint a picture of the future.'' To combine these ap- proaches with evaluation procedures the European

Commission is supporting a network ofevaluation, tech- nology assessment and foresight experts (e.g. Kuhhnann

1997).

• Technologies differ, instruments differ in thek tasks. Orchestrating both may save money. This requires de- veloped and empirical elaborated taxonomies (like the one presented) which could be easily obtained by exante evaluations. Thus, proof of the possibility of empirical foresight esti- mates for areas of application gives more emphasis to a promotion that is based on consistent ex-ante evaluation'^ - performed

before a technology policy intervention is defmed and planned.

These prognoses also make it easier for other societal groups to

participate in formulating the aims and means oftechnology policy.

Thus these and other tools improve the overall information

base of policy makers for a technology policy, weakening the

argument'"' ofpublic policy from information deficits when compared with companies. Particularly in the early development phase

of a technology, which is associated with high uncertainty, enterprises are by no means more knowledgeable about the broad

development ofthe technology supply, although they obviously

have more knowledge of their own user context per se. The fact that companies themselves cooperate with external research mstitutions underlines the possibility of the state having access to relevant information ofthis kind in principle, since most research

institutes are run as public agencies.

Hence, more decisive are the ways ofimplementation ofstra- tegic knowledge in real policy, and the fact ofthe independence | of state decisions dedicated to the furtherance of economic and

societal interests. Bringing in actors that are external to policy

administration - despite the involvement of individual interests > - into the designmg of concrete measures does facilitate implementation in day-to-day terms. Technology policy remains - first and foremost - policy.

'5 for methodologies see e.g. from Germany CUHLS " compare arguments ofMEYER-KRAHMER 1989, pp. 222.

1997, GRUPP 1997.

'^ KUHLMANN, HOLLAND 1994.

44
POLICY GOALSAND PROGRAMDESIGNISSUES

References

Arnold, E.; Guy, K. (1992): Diffusion Policies for IT - The way

forward. Summary Paper ofOECD/ICCP-Expert Group-Meeting.

Paris.

Becher, G. ; Wolff, H. (1995): Evaluation ofthe Promotion ofR&D

Cooperations ofSmall and Medium-sized Enterprises. In: Becher,

G./Kuhlmann, S. (eds.): Evaluation of Technology Policy

Programmes in Germany, Boston/ Dordrecht/London (Kluwer

Academic Publishers), 55-80.

Becher, G.; Kuhlmann, S. (eds.) (1995): Evaluation ofTechnology

Policy Programmes in Germany. Dordrecht, Boston, London.

Becker, C. (1995).- Evaluation of the Promotion of Information

Technology: Microelectronics; Microperiphery and

Microsystems Technology. In: Becher, G./Kuhlmann, S. (eds.): Evaluation ofTechnology Policy Programmes in Germany, Boston/Dordrecht/London (Kluwer Academic Publishers), 137-164.

Beirat (1985): Strukturwandel fur Wachstum und mehr

Beschaftigung. In: Wissenschaftlicher Beirat beim

Bundesminsterium fur Wirtschaft. Bonn. BMBF (1996): Bundesbericht Forschung. Bonn. BMFT (1993a), Federal Ministry for Research and Technology

1 993. Report ofthe Federal Government on Research, Abridged

Version. Bonn.

Bogelein, M. (1990): Ordnungspolitische Ausnahmebereiche - Marktwirtschaftliche Legitimation und wirtschaftspolitische

Konsequenzen. Wiesbaden.

Branscomb, L.M.; Keller, J.H. (1998): Investing in Innoation. Creating a Research and Innovation Policy That Works. The MIT

Press, Cambridge, Massachusetts, and London, England.

Braunling G.; Maas, M. (1988): Nutzung der Ergebnisse aus

offentlicher Forschung und Entwicklung, Forschungsbericht an

die EG. Karlsruhe.

Cuhls, K. (1997): Retrospektive auf die Technikvorausschau und

die technologiepolitischen Anwendungen des Delphi-Verfahrens

in Japan. Heidelberg (Physica/Springer, Reihe: Technik, Wirtschaft

undPolitik, Bd. 29).

Dreher, C. (1997): Technologiepolitik und Technikdiffusion,

Auswahl und Einsatz von Forderinstrumenten am Beispiel der

Fertigungstechnik. Baden-Baden.

Ergas, H. (1987): Does Technology Policy Matter? hi: Guile, B.R./

Brooks, H.: Technology and Global Industry. Companies and

Nations m the World Economy, Washington D.C., 1 9 1 -245

.

Eschenbach, R.; Hafkesbrink, J.; Lutz, S. (1995): Evaluation ofthe

Promotion ofPre-competitive Research and Development Cooperation in the Case of Microsystems Technology (MST). In: Becher, G./Kuhbnann, S. (eds.): Evaluation ofTechnology Policy

Programmes in Germany, Boston/Dordrecht/London (Kluwer

Academic Publishers), 165-196.

Ewers, H.J. (1990): Marktversagen und Politikversagen als Legitimation staatlicher Forschungs- und Technologiepolitik. In: Krupp, H. (Hrsg.): Technologiepolitik angesichts der

Umweltkatastrophe. Heidelberg, S. 147-160.

Ewers, H.J.; Becker, C; Fritsch, M. (1990): Wirkungen des Emsatzes computergestutzter Techniken in Industriebetrieben,

In der Reihe: Arbeitsmarktwirkungen modemer Technologien,

Bd. 6. Berlin/New York.

Pels, G. (1989): Die Rahmenbedingimgen marktwirtschaftlicher

Technologiepolitik: Stellungnahme aufdem IW/ZVEI Forum. In: Europaische Technologiepolitik fiir den Binnenmarkt. Koln.

Fritsch, M.; Wein, Th; Ewers, H.-J. (1993): Marktversagen und

Wirtschaftspolitik - Mikro-okonomische Grundlagen staatlichen Handelns. Milnchen.

Gahlen, M.; Stadler, M. (1986): Marktstruktur und Innovationen:

Eine modelltheoretische Analyse. Institut fur

Volkswirtschaftslehre der Universitat Augsburg.

Gielow, G.; Krist, H.; Meyer-Krahmer, F. (1985): Industrielle Forschungs- und Technologieforderung - Diskussion

theoretischer Ansatze und ihrer empirischer Evidenz. Endbericht

an das BMFT, FhG-ISI. Karlsruhe.

Grande, E.; Hausler, J. (1994): Industrieforschung und

Forschungspolitik - Staatliche Steuerungspotentiale in der

Informationstechnik. Frankfurt a.M.

Grupp, H. (1997): The Links Between Competitiveness, Firm's

Innovative Activities and Public R&D Support in Germany: An

Empirical Analysis. In: Technology Analysis & Strategic Management, Vol. 9, No. 1, 19-33.

Haflcesbrink, J.; Horst, H. (1995): Evaluation ofTechnology Transfer Initiatives in the Framework of Innovation Support

Programmes Instruments of Technology Transfer Evaluation -

45
ATP PAPERSAND PROCEEDINGS

The Case ofMicrosystems Technology .In: Becher, G./Kuhlmann,

S. (eds.): Evaluation of Technology Policy Programmes in Ger- many, Boston/Dordrecht/London (Kluwer Academic Publishers),

197-213.

Hauff, v.; Scharpf, F.W. (1975): Modernisierung der

Volkswirtschaft: Technologiepolitik als Strukturpolitik.

Europaische Verlagsanstalt. Frankfurt/Main, Kob.

Holland, D.;Kuhlmann, S. (1995): Systemwandel und industrielle Innovation. Studien zum technologischen und industriellen Umbruch in den neuen Bundeslandem, Heidelberg (Physica/

Springer Publishers, Series "Technik, Wirtschaft und Politik",

vol. 16).

Holschumacher, R. (1994): Determinanten der Entwicklung und

Ausbreitung von Emissionsminderungstechnologien fiir fossil befeuerte Kraftwerke - ein intemationaler Vergleich. Dissertation. Universitat TH Karlsruhe.

KirchgaBner, C. (1979): Zur Struktur politisch-okonomischer

Konjunkturzyklen. In: C.Ch.v. Weizsacker (Hrsg.) Staat und

Wirtschaft. Berlin.

Klodt, H. (1995): Grundlagen der Forschungs- und

Technologiepolitik. Milnchen.

Kortmann, W. (1995): Diffusion, Marktentwicklung und

Wettbewerb. Eine Untersuchung uber die Bestimmungsgriinde

zu Beginn des Ausbreitungsprozesses technologischer Produkte.

Frankfurt a.M.

Kuhlmann, S. (1998): Politikmoderation - Evaluationsverfahren in der Forschungs- und Technologiepolitik. Baden-Baden.

Kuhlmann, S. (1997): The Potential for Integration ofTechnology

Foresight, Technology Assessement and S+T-Policy Evaluation

in Europe, ASTPP-Progress Report, Spring.Bruxelles.

Kuhlmann, S. (1995): Patterns of Science and Technology Policy - Evaluation Germany. In: Research Evaluation, Vol. 5, April 1995,

pp. 22-33.

Kuhlmann, S.; Holland, D. (1994): Evaluation von

Technologiepolitik in Deutschland - Konzepte, Anwendung,

Perspektiven. Technik, Wirtschaft und Politik No. 9. Schriftenreihe des Fraunhofer-Instituts fiir Systemtechnik und

Innovationsforschung, Karlsruhe, Physica-Verlag. Heidelberg.

Kuhlmann, S.; Holland, D. (1995a): Evaluation von

Technologiepolitik in Deutschland, Konzepte, Anwendung,

Perspektiven, Heidelberg (Physica/Springer Publishers, Series "Technik, Wirtschaft und Politik", vol. 12).

Kuhlmann, S.; Holland, D. (1995b).- Erfolgsfaktoren

wirtschaftsnaher Forschung, Heidelberg (Physica/Springer Publishers, Series "Technik, Wirtschaft und Politik", vol. 15).

Kuntze, U. ; Homschild, K. ( 1 995).- Evaluation ofthe Promotion of

R&D Activities in Small and Medium-sized Enterprises. In: Becher, G. / Kuhlmann, S. (eds.): Evaluation of Technology Policy

Progranmies in Germany, Boston/Dordrecht/London (Kluwer Aca- demic Publishers), 33-54.

Lay, G. (1 993): Government support ofcomputer integrated manufacturmg in Germany: first results of an impact analysis. In: Technovation 13 (5), 283-297.

Lorenzen, H.-P. (1990): Innovative Programme Design - When are Integrated Approaches Worthwhile? In: Krupp, H. (ed.):

Technikpolitik angesichts der Umweltkatastrophe, Heidelberg

(Physica/Springer Publishers), 1 72- 1 80.

Liitz, S. (1992): Die staatliche Steuerung von Kooperationen in der Industrieforschung-Funktionsweisen und

Erfolgsbedingungen des Forderinstruments Verbundforschung.

(Dissertation in Vorbereitung). Koln.

Mayntz, R. (1993): Networks, Issues, and Games:

Multiorganizational Interactions in the Restructuring of a National Research System, in: F.W. Scharpf(ed.): Games and Hierar- chies and Networks, Frankfurt / Boulder (Campus / Westview),

pp. 189-209.

Mayntz, R. (1994): Academy of Sciences in Crisis: A Case Study

ofa Fruitless Struggle for Survival. In: Schimank, U./Stucke, A.

(eds.): Coping with Trouble. How Science Reacts to Political Disturbances of Research Conditions, Frankfiirt/New York (Campus/St. Martins 's Press), 163-188.

Meiliner, W.; Zinn, K.G. (1984): Der neue Wohlstand. C. Bertelsmann Verlag. Munchen.

Meyer, M. (1995): Forschungssubventionen aus.

wettbewerbspolitischer Sicht. Bd. 16 der Veroffentlichungen des HWWA-Instituts fur Wirtschaftsforschung. Hamburg, BadenBaden.

46
POLICY GOALSAND PROGRAMDESIGN ISSUES

Meyer-Krahmer, F. (1989): Der EinfluB staatlicher

Technologiepolitik auf industrielle Innovationen. Nomos

Verlagsgesellschaft. Baden-Baden.

Meyer-Krahmer, F.; Gielow, G.; Kuntze, U. (1984):

Wirkungsanalyse der Zuschiisse fur Personal in Forschung und

Entwicklung. Gutachten im Auftrage des Bundesministeriums

fiir Wirtschaft, Bonn. Fraunhofer Institute for Systems and Innovation Research (ISI), Karlsruhe, 1984 (mimeo).

Meyer-Krahmer, F.; Montigny, P. (1989): Evaluations of innovations programmes in selected European countries. In: Research

Pohcy 1 8 Elsevier Science PubUshers B.V., North Holland, S. 3 1 3-

332.

Meyer-Krahmer, F.; Reiss, Th. (1992): Ex-ante Evaluation and Technology Assessment - Two Emergmg Elements of Technology

Policy Evaluation. In: Research Evaluation, Vol. 2, No. 1, April 1 992, S. 47-54.

Oberender, P. (Hrsg.) (1994): Industriepolitilc im Widerstreit mit

der Wettbewerbs-politk. Berlin.

Oberender, P.; Streit, M. (Hrsg.) (1991): Marktwutschaft und In- novation. Baden-Baden.

Pfirrmann, O.; Schroeder, K. (1995): Problems and Implications of

an Institution-based Impact Analysis of the Berlin Model of

Technology and Know-how Transfer. In: Becher, G. /Kuhknann,

S. (eds.): Evaluation of Technology Policy Programmes in Ger- many, Boston/Dordrecht/London (Kluwer Academic PubUshers),

215-241.

Reger, G.; Kuhhnann, S. (1995): European Technolgy Policy in Germany. The Impact ofEuropean Community Policies upon Sci- ence and Technology in Germany, Heidelberg (Physica/Springer

Publishers, Series "Technology, Innovation, and Policy", Vol. 2).

Reinhard, M (1959).: Diffusion and Potential Analyses as Part of

Programme Evaluations: Exemplified in the Promotion ofLaser Technology and the Promotion of Thin Film Technology. In: Becher, G./Kuhlmarm, S. (eds.): Evaluation ofTechnology Policy

Programmes in Germany, Boston/Dordrecht/London (Kluwer

Academic Publishers), 117-136.

Scharpf, F.W. (1993) (ed.): Games and Hierarchies and Networks.

Analytical and Empirical Approaches to the Study of Gover- nance Institutions, Frankfurt/ Boulder (Campus / Westview).

Schmoch, U. et al. (1988): Technikprognosen mit

Patentindikationen. Koln.

Schwitalla, B. (1993): Messung und Erklarung industrieller Innovationsaktivitaten mit emer empirischen Analyse fur die westdeutsche Industrie. Heidelberg.

Soete, L. (1991): Technology and Economy m a changing world: Conference on Technology and the global Economy. MERIT,

Faculty ofEconomics University ofLimburg (The Netherlands).

Starbatty, J.; Vetterlein, U. (1990): Die Technologiepolitik der Europaischen Gemeinschaft. Baden-Baden.

Stoneman, P.; Diederen, P. (1994): Technology Diffusion and Public Policy. In: The Economic Journal 104 (July), Oxford, S. 918-

930.

Streit, M.E. (1984): Innovationspolitik zwischen Unwissenheit

und Anmaiiung von Wissen. In: Hamburger Jahrbuch fur

Wirtschafls- und Gesellschaflspolitik, Bd. 29, S. 35-54.

Wengel, J.; Lay, G.; Dreher, C (1995): Evaluation ofthe Indirectspecific Promotion ofManufacturing Technology. In: Becher, G./

Kuhlmann, S. (eds.): Evaluation of Technology Policy

Programmes in Germany, Boston/Dordrecht/

London (Kluwer Academic PubHshers), 8 1-99.

Weyer, J. ( 1 994): Space Policy in West Germany 1 945- 1 965 : Stra- tegic Action and Actor Network Dynamics. In: Schimank, U./

Stucke, A. (eds.): Coping with Trouble. How Science Reacts to

Political Disturbances of Research Conditions, Frankfurt / New

York (Campus / St. Martins's Press),333-355.

47
ATP and the U.S. Innovation System

by Michael S. Fogarty, Amit K. Sinha, and Adam B. Jaffe.'

Introduction

Evidence indicates that R&D spillovers cause a large gap

between social and private rates of return - a difference on the order of 50 to 100%.^ A large 'spillover' gap creates a rationale

for the Advanced Technology Program (ATP) which seeks to maximize net social benefits by supporting pre-competitive, enabling technologies. ATP's challenge is to identify and fund

projects with a high social rate of return that would otherwise be

underfunded, delayed, or inadequately pursued in the absence

of ATP support (i.e., fund projects with a large 'spillover gap').

Our analysis indicates that ATP's funded projects will produce more influential spillovers if selection is based on knowledge of

networks of R&D organizations that are influential sources of

enabling technologies.

This paper has two primary purposes. The first is to de- velop a new fuzzy systems methodology for analysis of R&D

spillovers.' The new methodology yields new, deeper insights

into innovation and suggests important R&D spillover hypotheses with relevance to ATP. A second purpose is to illustrate the usefulness of the method for evaluating and improving the se- lection of projects by ATP. We do this by focusing on one

specific technology: MEMS (microelectro-mechanical systems),

which is an emerging technology being discussed by ATP as a Focused Program candidate."*

Methodology

Researchers studying spillovers have been constrained by

not having a method that allows them to discover and analyze

patterns in the complex patent data set. If the 'spillover pool'

metaphor is the right one, then statistical methods and interpre-

' Michael S. Fogarty is a Professor of Economics at Case Western

Reserve University and a member of the NBER, Amit K. Sinha is a

graduate student at Case Western University, Adam B. Jaffe is a Professor of Economics at Brandeis University and a member of the NBER.

^ Two types of R&D spillovers are particularly relevant to this study: (1) "knowledge spillovers" —These occur because knowledge

created by one firm is typically not contained within that firm, and

thereby it creates value for other firms and other firms' customers; and

(2) "network spillovers" - These result from the profitability of a set of

interrelated and interdependent technologies that may depend on

achieving a critical mass of success so that each firm pursuing one or more of these related technologies creates economic benefits for other

firms and their customers. See Adam B. Jaffe, "Economic Analysis of Research Spillovers: Implications for the Advanced Technology

Program," October 1996.

tations found in the literature make sense. However, if there is an

underlying spillover network structure with diffusion pathways

and specific spillover mechanisms embedded in the patent data,

then existing methods miss crucial information. For example,

R&D networks will influence the rate of invention and innovation, technology diffusion, and the geography of spillovers (i.e., the location of social benefits). The fuzzy methodology gives us a new framework and tool for analyzing long standing spillover

questions, interpreting spillovers, and simulating policy inter- ventions.

The basic unit in our model is the R&D lab. Our fuzzy system

methodology gives us a way to represent the interrelationships

forming R&D networks. The R&D networks are constructed from

interactions between R&D labs. We analyze these interactions using patent citations, which are interpreted as a form of communication. Communication takes many forms, including reading of

papers, attendance at conferences, hiring of leading consultants, word-of-mouth, analysis of patent database, hiring of university graduates, and personal conmiunication.

By a system we mean that there exists a hierarchy of tech- ! nologies, R&D organizations, and regions connected by a communication network. System effects exist when a change in any

component diffuses throughout the network. The notion of an

innovation system is intuitively appealing and is consistent with

existing evidence on R&D spillovers: our interviews suggest

that inventors and R&D directors behave as if there is a system;

patent citations are more likely to be external to a region when

there exists a high level of local inventive activity; and the social value of CT-scanner patents increases nonlinearly with the number of patents (i.e., the information content increases disproportionately with the number of citations).

' Lofti Zadeh developed fuzzy set methods in the mid-1960s as a ; way to incorporate the "vagueness" or imprecision of data and other | information in our lives and decisions. Statistical imprecision involves < outcomes that are random (a flip of a coin); fuzzy imprecision is nonstatistical (it's about time we begin our descent for a landing). The

difference applies to sets. For example, "crisp" (conventional) sets involve precise conditions for membership in a set. The answer is yes

or no, 1 or 0. Fuzzy sets satisfy imprecise conditions with a membership

function between 0 and 1 . The membership function can be used to describe"truth values" -X belongs to Y with a truth of 0.8. For a good i introduction to fuzzy methods, see James C. Bezdek and Sankar K. Pal, ] Fuzzy Models for Pattern Recognition (Piscataway, NJ: IEEE Press, 1991), Chapter 1. Also see Amit Sinha, "A Study of Fuzzy Logic

Based Pattern Recognition Models," (mimeo), February 19, 1998.

^ * ATP's Focused Program involves multi-year projects with well- , defined technologies and clear business goals. See http://

www.atp.nist.gOv/atp/focusprg.htm#Ongoing.
POLICY GOALSAND PROGRAMDESIGNISSUES

Existing methods simply weight patents by quantity ofcita- tions to measure spillovers and the significance of the technology.' In other words, they assume ' all patent citations are equal'

(assign a value of 1 to each citation); our fuzzy methodology

allows us to develop indicative membership measures between 0

and 1 representing the strength of interaction between any

pairwise combination ofR&D labs, specific to organization, tech- nology, and region. Our systems model builds the system itera- tively incorporating the first, second, and then the third level of

diffusion ofspillovers. The result is a hierarchical R&D network

system.

We can then determine the influence ofparticular R&D labs by technology and region. The sum of the interactions "truths"

(values between 0 and 1) across an R&D network is the lab's

total system value (SV) for the technology. The SV is a function

of the number of links in a network and the strength of each

pairwise interaction. Therefore, an individual R&D lab's SV is

f| high when there are a large number of intensive links with other

influential labs. For each pair of R&D labs in a network we have three measures: (1) relative importance of the unit (system value or

strength); (2) strength of interaction (truth value); and (3) the

relative systems influence of the link between the two units (the

system value of the source multiplied by the truth of interaction between the source and learning R&D labs). The results can be

aggregated across the three dimensions by year - technology,

I organization, and region.

Our methodology represents an advance for several reasons. First, it allows us to search for and identify patterns or

structure in the complex patent data. This was made possible by

our new fuzzy systems algorithm that makes 'relational' simulta- neous clustering of patents in four spaces possible with a UNIX

I workstation. The four spaces are: technology, organization,

I region, and time. Second, the methodology suggests new hyj potheses yielding new insights about invention and the innovaj tion system. For example, because R&D networks serve as a

source as well as a user of R&D spillovers, an individual firm's value as a source of new technology depends on its ability to

1 learn from its external environment. Third, the methodology offers a framework for developing ATP strategies to maximize

spillovers and suggests an approach to evaluating ATP projects. For example, ATP should look for enabling R&D networks - not

enabling technologies. The methodology can assist in the selection of ATP projects by providing measures of the significance

^ For example, see Zvi Griliches, "Patent Statistics as Economic

Indicators: A Survey," Journal ofEconomic Literature, Vol. 92 (1990), 630-653; Adam Jaffe, "Technological Opportunity and Spillovers of R&D: Evidence from Firms' Patents, Profits and Market Value,"

American Economic Review (1986); Adam Jaffe, Rebecca Henderson,

oftechnologies and by exploring the likely influence ofproposed

project participants, including joint ventures. The methodology

also permits analysis ofMEMS R&D spillovers by broad industry-based technologies (automotive, aerospace, advanced materials, auto, information technology, and biomedical devices) as well as regions. Finally, the methodology also suggests a possible approach to measuring the "spillover gap."

Figure 1 illustrates one segment ofa specific R&D organizational network for MEMS centered on MIT over the period 1 985-

95. The fiill network is much larger and is discussed in the fol- lowing section (see Figure 4 for the top twenty-five MEMS organizations. This segment is centered on MIT, which is ranked 20"'

in the full network.) It illustrates that R&D networks operate as both sources and learners. One implication is that a firm's value

as a source depends on its ability to learn fi om its external envi- ronment. The nodes in the R&D network correspond to the R&D

labs and arrows indicate spillover flows between labs. The colour bar indicates importance of the R&D lab and the systems influ- ence of interactions. The most influential node is brown; the

least (not shown) is dark blue. Arrows follow the same scheme.

The analysis, which is based on real data, identifies AT&T,

Honeywell and Xerox as the most important MEMS organizations in this part ofthe network. The full network contains more

nodes, incorporating interactions involving third-order diffiision of spillovers. Next are GTE, MIT and the US Army. The figure

incorporates the most important interactions - i.e., those closest members of MIT's R&D MEMS neighborhood, including first and second-order diffusion spillovers. For example, flow from

MIT to Xerox is about 0.5 . This means that about 50% of all R&D

pairwise interactions in the fiill MEMS network are less intense than that between MIT and Xerox. Also shown, GTE has a second-order influence on MIT which occurs through its direct influence on AT&T.

ATP and the Case of MEMS

Our paper focuses on one application: microelectro-mechanical systems (MEMS), which is an important emerging technol- ogy being consider by ATP for its Focused Program. Our analysis is not intended to be a formal evaluation ofMEMS as an ATP

focused program.

MEMS combines computation, sensing, and actuation with

miniaturization to make mechanical and electrical components.

The bulk of applications are pressure sensors, optical switching,

and Manuel Trajtenberg, "Geographic Localization of Knowledge

Spillovers as Evidence by Patent Citations," Quarterly Journal of

Economics, Vol. 108 (1993), 577-598; and Joel Podolny and Andrea

Shepard, "When Are Technology Spillovers Local?: Citation Patterns

in the Semiconductor Industry," (mimeo), 1997.

49
ATP PAPERSAND PROCEEDINGS

Figure 1 : MEMS R&D Network Around Massachusetts Institute of Technology

Table 1 highlights systems hypotheses with implications for ATP.

Table 1: A Summary ofSelected Spillover Hypotheses and

Implicationsfor ATP

Spillover Hypotheses Implications for ATP

1. Spillovers are becoming increasing^ important to

firms' performance. A firm's system position affects

its rate of invention and innovation as well as the speed and geographical extent of technology

difiiisioa Firms will decline if they become less effective learners.

ATP will maximize spillovers by selecting projects with enabling networks. These projects invoh'e organizations positioned in strong R&D networks,

incorporating the influence of all three spaces. Joint ventures involving small firms offer a way to magnify

R&D spillovers by connecting small firms' R&D to more influential R&D partners.

2. Interactions in the system focused on strong R&D

networks cause increasing returns to R&D.

Geographic spillovers will become increasing^ concentrated by regioa In the pre-competitive,

enabling stage, spillovers are highly geographically concentrated and successful regions increasingly become their own source of the technology.

ATP will maximize spillovers by funding projects that draw heavify on strong R&D networks. The

outcome will likely reinforce patterns of concentration, even though spillovers will be widefy shared across

organizations, industries, and regions. This strategy gives the U.S. an earfy advantage in emerging

technobgjes.

3. Firms in the network of top universities are more

influential sources of enabling technology.

ATP will maximize spillovers by selecting projects

involving technologies that draw extensive^ fi-om

universities and government labs. Drawing on strong GUI networks leverages federa research support.

50
POLICYGOALSAND PROGRAMDESIGNISSUES

inertial sensors, flmd regulation and control, and mass data stor- age. These cut across a number of manufacturing industries,

including: sensors, industrial and residential controls, electronic components, computer peripherals, automotive and aerospace

electronics, analytical instruments, and office equipment. The

industry list suggests a potential for generating a large, broadbased volimie ofR&D spillovers.

Our analysis suggests that R&D networks generating pre- competitive, enabling technologies may have certain characteristics, such as: (1) universities and government labs play signifi- cant roles as sources; (2) the network is sparse and evolvmg; (3) the technology is new (cited patents are relatively current); (4)

total system spillovers increase significantly and technology gets

diffused rapidly; (5) influential companies perform significant basic research; and (6) technologies become geographically concentrated in important regions serving as incubators.* ^ MEMS is an emerging technology worldwide.* About 200

firms are actively engaged in MEMS R&D - roughly 80 are US

firms; Japan is the second major player. According to DoD, the MEMS market was $1 billion in 1994. Projections for the year 2000 range from $8 to 14 bilhon. It is likely that MEMS interacts extensively with other technologies. U.S. industry investment in MEMS so far has been fairly

modest (about $ 120 million in 1 995). In contrast, in the same year

federal R&D support of MEMS was a large component (about

$35 million), $30 million ofwhidi came from DoD (mainly DARPA).'

' Serious R&D organizations invest a portion of R&D to actively

acquire external knowledge and aggressively search what Stuart & Podolny

call the technological landscape for ideas, knowledge of competitors'

technology, etc. See W. Cohen and D. Levinthal, "Absorptive Capacity:

ANew Perspective on Learning and Innovation,'M(im/«Js^ra//ve Science

Quarterly, Vol. 35, 128-152. Clearly, a firm's learning process isn't random. Evidence from our interviews suggest that firms are focusing more resources on acquiring technology from external sources. See Adam B. Jaffe, Michael S. Fogarty, and Bruce A. Banks, "Evidence

from Patents and Patent Citations on the Impact of NASA and other Federal Labs on Commercial Innovation," V. XLVI, No. 2, Journal of

Industrial Economics QwnQ 1998), 183-205.

' Although influential R&D organizations search in a global R&D

network, in a pre-competitive, incubation phase, geography plays a

critical function —the accumulation of a critical mass of strong network

connections that speed growth ofthe enabling technology. In general, as

local activity increases, the volume of important spillovers grows but an

increasingly large fraction becomes external as a region develops higher

order R&D organizations with world-wide connections. Because R&D

labs have a specific location, an agglomeration ofstrong R&D networks

serves a dual function —good regional sources are also good learners. Local R&D networks are strong only if they are solidly linked to the

global network.

' For a description of MEMS, see Department of Defense,

"Microelectromechanical Systems: A DoD Dual Use Technology

Suggestive of the technology's emergmg character, about 30

universities and government labs are actively pursuing

MEMStechnologies. NSF's MEMS support was $3 million. Labs

contributed about $2 million. Since 1 989 the NSF has sponsored

124 MEMS-related projects at 6 1 organizations (mamly universities), with fimding ofabout $25 million. Approximately $ 1 .4 million consists of SBIR grants.'"

Data and MEMS Patents

The data are drawn from the universe of patents granted by

the US Patent Office from 1963 through 1995." Information on

patent citations beguis in 1977. Electronic data on assignee is available beginning in 1969. We geographically locate patents

using the inventor's address, which means that location in our

analysis is the R&D lab's location - not the headquarter (as- signee) location. In addition to country and state, inventors

have been sorted first iato coimties and then mefropolitan areas. As a foundation for analyzing MEMS, we developed a core database of about 1,200 MEMS patents, startmg with a short list of key inventors and federally-ftmded MEMS projects. Cita- tions to these initial patents were used to identify additional MEMS candidate patents. Each candidate patent absfract and

exemplary claims was read to ensure that the patent was a MEMS

technology.'^

Industrial Assessment (Final Report)," December 1995. For an

interesting description of MEMS, see Discover (March 1998).

' Our research shows that DARPA presently hinds 62 projects at 48 organizations (17 universities, 5 government labs, 1 8 large companies, and 8 small firms). DARPA is currently funding 5 SBIR projects at 4

companies; they previously fimded an additional 5 SBIR projects. The

Army has funded 17 MEMS-related projects at 14 firms through its SBIR program. The projects amount to nearly $2 million. NASA has sponsored 20 MEMS-related SBIR projects. (No dollar amount was

available.) However, the MEMS working group at NASA-Lewis in Cleveland supported $2.5 million MEMS R&D by 17 S&Es. Moreover,

Ohio MEMS-Net has funded $2.4 million for capital investments m

1995 and 1996. Ranked by total NSF support of MEMS projects, the top ten

institutions include: Stanford, UC Berkeley, University of Michigan,

Cornell, University of Utah, University of Pennsylvania, University of

Illinois Chicago, Case Western Reserve University, University of

Minnesota, and University of Hawaii. Most of the MEMS university

projects are associated with fairly extensive patenting. The 61 MEMS

universities currently account for 312 MEMS patents. " The study uses the comprehensive patent database developed

jointly by REI-NBER. REI is the Center for Regional Economic Issues

at Case Western Reserve University.

The anaysis of MEMS patents was done by David Hochfelder, a research assistant on the ATP project. Hochfelder has a Masters

degree in electrical engineering.

51
ATP PAPERSAND PROCEEDINGS

MEMS — The Top Five Countries

There exists considerable international competition involving MEMS technology. The technology is concentrated in a few

countries. As shown in Figure 2, our systems analysis ofMEMS

technologies ranks the US as first, with Japan second, followed

by Germany, France, and Great Britain. Ranks are based on each

country's systems value as a MEMS source (i.e., our fuzzy esti- mate of each country's contribution to MEMS technologies).

Spillovers occur across international boundaries. FigureS shows

the balance ofMEMS spillover flow for seven countries with the

largest MEMS concentration (flow in minus flow out). What the data shows for MEMS is that higher-order countries are net exporters of the technology. The balance of MEMS knowledge

flows between the US and Japan favors the U.S.

Figure 2: World's Top 5 Sources ofMEMS

Suics Britain

Figure 3: Balance ofInternational Flows in MEMS

Sues Brilaiq

The MEMS R&D Network - A Brief Description

There exists potential for ATP to use the methodology to be- come more strategic in its project support by incorporating knowledge of spillover networks. For example, one important consideration for ATP would be to use knowledge of R&D networks to obtain a more favorable balance ofknowledge spillover trade flows. We can describe networks in a great deal ofdetail - about 400

technologies (patent classes), individual R&D labs by organization, and metropoUtan region location. Only selected findings are shown here. The original paper includes additional MEMS R&D

network findings by technology, organization, and region. The

data cover the period 1985-95. Each member's position is simulta- neously located in the three spaces. For example, only the findings

for each network' s most influential members as influential sources

are shown. These particular examples characterize the R&D net- work sources ofMEMStechnologies. Aparallel network exists for 'spillover use' networks.

Technolozies

The paper illustrates the system importance of leading tech- nologies in the MEMS network. The top five MEMS technologies

are: Semiconductor Device Manufacturing Process, Metal Working, Electricity: Electrical Systems & Devices, Incremental Printing

ofSymboUc Information, Optics: Systems (including Communication) and Elements. Our analysis shows that each of the top five MEMS technologies plays an influential role in both auto and aero- space industry technologies.

Organizations ^

I Figure 4 shows the top MEMS organizations ranked on sys- ; tem influence (i.e., spillovers). An organization's position on the

list is determined by the spillover generated by its associated R&D ^ network. The analysis shows that IBM ranks first as a source,

^ followed by U.S. PhilUps, NEC, GE, Texas Instnmients and United

Technologies. Some of the key universities are (in order of their influence) MIT, Stanford, and Berkeley. The fixll organization Ust (not shown) shows that federal labs and universities are prominent

MEMS sources. j Each R&D network's spillovers can be analyzed across all

^ three dimensions. This capability would provide ATP with the

^ means to select projects associated with enabling MEMS R&D [ networks. One hypothesis would be that ATP's funding will be

^ more "enabling" —draws on more basic, early-stage research, and . stimulates broader, more influential spillovers across a wider specj, trum of important uses — if it leverages other unportant federal R&Dflmding.

52
POLICY GOALSAND PROGRAMDESIGNISSUES

Figure 4: Fuzzy Network ofMEMS Organizations

2°T : — :—

Regions

When ATP selects a project, it also selects the geography of

R&D spillovers and, therefore, the benefits ofATP-flinded R&D.

Previously we saw that MEMS is highly concentrated in a small

set of leading countries. An analysis of US regions shows that development of MEMS is also geographically concentrated in a few top regions: San Francisco, Boston, LA, NY, Chicago,

andDallas account for the lion's share of the technology. Our

hypothesis is that an important characteristic of enabling net- works is location in a successful regional agglomeration supportive of new technology development. Both San Francisco

and Boston are important MEMS sources. In other words, both

regions are influential external sources ofMEMS. However, only

San Francisco appears to be developing significant local

spillovers. Based on this particular enabling characteristic of

networks, ATP investments in projects with strong connections

to San Francisco's MEMS network may produce more spillovers

and faster development of the technology. Investing in ATP

projects located in these large R&D agglomerations will likely

produce a higher social rate of return because the investment

builds from critical mass, creating increasing spillover returns to R&D. ATP's investments in leading regions may also resuh in capture of a larger share of spillovers by the US. The reason

stems from the expected increasing returns to R&D coupled with

a faster rate oftechnology diffusion and commercialization. Even

though ATP's investments are concentrated in leading regions,

spillover benefits will be shared by a much larger set ofUS industries and regions that draw from incubator regions.

Conclusions

Our new fiizzy systems methodology offers a new, potentially powerful way to analyze and interpret R&D spillovers, yielding much deeper knowledge of R&D spillovers than existing

methods. Our analysis indicates that ATP's funded projects will produce more influential spillovers if selection is based knowledge ofnetworks ofR&D organizations that are influential sources ofenabling technologies. MEMS provides an important example.

The key confribution of this study is the identification of

R&D networks as the main spillover mechanism. Our system

model is built on the assumption that patent citations between

two R&D organizations occurring over a period oftime identify

communication and measure interactions. However, the evi- dence shows that patent citations are a noisy indicator ofknowledge spillovers. Supportive evidence comes from a number of

quantitative studies relating patent citations to measures of

value; interviews ofR&D directors, patent attorneys, and inventors; and at least one case study. The evidence is clearly more

supportive at the R&D lab level over time, which is the basic unit ofthis study, than at the individual patent level. But more valida- tion is clearly needed. The authors have begun a pilot national survey of inventors that will seek to determine the conditions

under which patent citations can be used to measure communication and knowledge spillovers. Importantly, our survey will also be used to validate the system structure identified by the new model. In addition, there is a need for more R&D lab inter- views and case studies focused on the structure of R&D net- works. These would help to identify the specific spillover mechanisms that ATP's projects can influence to maximize social benefits by funding projects with a large 'spillover gap.'

Our methodology may provide a way to evaluate the

spillover gap associated with projects. ATP seeks projects with

both a high social and private rate of return. The following is only suggestive. First, set a minimum private rate ofreturn and

then calculate the total system value of each project across the network (such as MEMS). Second, separate the system value

into two components: (a) internal to the project's participants

(private gain) and (b) external to the project's participants (social

gain). An approximation ofthe spillover gap is simply the differ- ence between estimated social gain and private gain. The calculation requires taking into account the fact that ATP project participants draw spillovers (learn) from the full network. Where

multiple projects are funded, we also need to evaluate inter-project

spillovers. These are specific to organization, technology, re- gion and time. The last step is to rank projects by the spillover gap53
ATP PAPERSAND PROCEEDINGS

A comparison ofspillover ranks based on the existing method

and our method shows that while the ranks are correlated there

are significant differences that stem from differences in the

strength of the networks. An important next step in developing

the methodology is to develop a fuzzy measure theoretic basis

for our method. Statistical hypothesis testing requires that the

underlying model satisfies the conditions ofmeasurability. Since

fuzzy measures do not necessarily satisfy classical measure properties, fuzzy models carmot be readily used for standard statisti- cal hypothesis testing. One of the authors is currently developing the theoretical basis that would make such testing possible.

54
Comments on ATP and the U.S. Innovation System:

APreliminary Report

by James D. Adams'

This paper is an ambitious attempt to measure elements of

system-wide R&D spillovers. By system-wide I mean the effect of spillovers throughout the economy, as opposed to the immediate effects among firms in the same locality and industry. The

paper employs a new approach to patent citations in order to reach this goal. The method used by Sinha, Fogarty, and Jaffe

(hereafter SFJ) is based on the mathematics of fuzzy sets. Fuzzy

set theory is concerned with the assigimient of "truth" values,

which indicate the likelihood that an object belongs to a set when membership in that set is not known with certainty. In the

specific context of this paper the objects are R&D laboratories,

and the question is whether the laboratories belong to a specific

citation network.

The principal reason for the emphasis on defining the likely

limit ofthe citation network is that the influence wielded by cita- tions presumably depends on the importance of the citing R&D

laboratory in the network. Previous papers have treated all cita- tions as equally valuable, effectively visualizing the citation net- work or system as a citation "sphere" in many dimensions sur- rounding the R&D laboratory being cited. The promise offered by this paper lies in its potential to define a broader citation system, so that the influence of a citation is not rigidly set equal

to 1 .0, but instead depends on how often, and by whom, the

citing R&D laboratories are themselves being cited. In this way

the multi-dimensional citation sphere turns into a much larger

and more irregular citation "solid". In addition the secondary

citations are themselves weighted by the influence of tertiary

and higher order citing R&D laboratories, and so on to the edges

of the network. If the methodology succeeds in this goal, then a

picture ofthe importance ofpatents and the issuing R&D laboratories emerges that is closer to capturing the influence of ideas than the older, single stage citation methodology.

The methodology would have obvious potential relevance

to ATP, since ATP is interested in high risk, enablmg technologies. Network externalities, and by inference citation networks,

can be expected to accompany such technologies. Thus the

social benefits of technologies in which ATP is interested are

likely to exceed the firm level benefits. It follows that techniques

for identifying networks could help ATP m selecting its portfolio

of projects.

If the potential of the method is this large, does it have any

shortcomings? I believe that there are some, but m makmg this assessment I am driven to say that the mathematics of the paper

are not carefully spelled out in the version of the paper that was

available to me. It follows that the interpretation that I place on

the methodology of SFJ may differ somewhat fi^om the formulae

that are actually being used to derive the results.

First, it is my understanding that the measures of citation influence are based on citation fractions to and from pairs of

laboratories A and B, normalized relative to a population of cita- tion fi-actions. If this is correct—and page 14 of SFJ says that it

is—then this aspect of their measurements takes out scale ofthe

laboratories. In my view this biases the influence of larger laboratories downward compared with smaller laboratories. Ofcourse,

in their empirical application to the MEMS technology (MicroElectro-Mechanical Systems), SFJ fmd that larger geographic

centers of research have larger influence, partly because of the muhiple layers of citation involvmg such centers. It would ap- pear that their systems methodology makes up in breadth of

citation some of what it loses in scale of the R&D laboratory or research center. Nevertheless, the direct effect of size of large laboratories should be fully taken into account. Second, SFJ should demonstrate whether their corrected

citation influence measure outperforms the traditional single stage

citation measure in explaining performance ofthe citing laboratories. That is, in an econometric framework, they should demonstrate whether their system-wide spillovers have an incremental

effect over and above single-stage spillovers in explaining pat- enting, stock market value, total factor productivity, and other indexes of industrial success. Third, in their paper SFJ apply their methodology to two

technologies that are already known to have large network externalities. What would be more helpful to ATP is a predictive

method that would catch the technologies at an early stage and

based on their characteristics at that time, allow us to predict

which technologies are likely to develop into fiill-blown networks.

I submit that some ofthese characteristics are abeady known

at this time. Candidate technologies in some sense would have

to be beyond the ability of the private sector to commercialize.

Candidate technologies would often originate in basic science; they would be high risk, and their applications would be largely

unknown; they would be subject to standardization, once applied, thereby rendering them more useful; and their external benefits would be immense. It remains to be seen how these literary characteristics translate into the fuzzy set mathematics of this paper, applied to technologies bom in the past few years or so at

the time of the analysis.

' Dr. Adams is Professor of Economics at the University of Florida and

is a Research Associate of the NBER.

EVALUATIONOFNATIONAL PROGRAMS

N

C

R&D, Government Support, and Firm Productivity in Israeli Industry

by Zvi Griliches and Haim Regev'

In the last two decades or so, Israel has engaged in concerted wide-ranging efforts to develop and promote its science- based high technology sectors. The premise was that given its lack of natural resources on the one hand and the availability of

a highly educated workforce on the other, economic policies aimed

at overcoming market failures usually associated with innovation and technical change would lead to higher economic grow1:h.

To implement this overall policy, various support institu- tions and programs were created over the years. The main ones

are:

1. R&D subsidies granted by the Chief Scientist of the Ministry ofIndustry and Trade to support commercial

R&D projects performed by the private sector. This is the largest (and oldest) support program and it has been

widely credited with propelling the tremendous growth

of Israel's high technology sector in the last decade.

2. The "Masnef program, also from the Ministry of In- dustry and Trade. This program supports consortia en- gaged in developing "generic" technologies expected

to be relevant to the members of the consortium, and

possibly also to others in the same field. That is, it supports precompetitive R&D, and does so by encouraging ( and then subsidizing) the creation ofcollaborative

projects that include firms operating in similar or re- lated areas and sometimes also academic mstitutions.

3. The "National S&T Infrastructure" program, backed

by the Ministry of Science and the National Council for Research and Development. This program identifies criti- cal areas of science and technology that have potential

commercial applications, in which Israel may have (or

may develop) a comparative advantage, and supports

projects conducted primarily by consortia of academic

research groups and also some members from industry.

4. The various Binational programs (such as with the US),

involving support to collaborative R&D projects be- tween Israeli and American researchers and companies.

' Harvard University and NBER and The Central Statistical Bureau of Israel respectively.

5. R&D projects supported by the Ministry of Defense, having potential civilian applications.

6. Support to basic research at universities and other aca- demic institutions. This paper examines the overall role of government-supported R&D in Israeli industrial firms, especially its contribution

to growth in their productivity.^ Subsequent work will focus more

specifically on the project-level data available in the records of

the Chief Scientist. The analysis of the Israeli data may also be

helpfiil in understandmg the working ofR&D support programs

in other counfries, especially in the US, where the Advanced

Technology Program (ATP) is operating in a somewhat similar manner. Most OECD countries have govemment-ftmded R&D support programs that encourage and complement private invest- ment in civilian R&D. There seems to be a rather wide consensus among economists in the OECD countries that government action to promote research and development is well advised. The

analysis of Israeli data may be also helpfiil in understanding the operation of R&D support programs in other counfries because

efforts to evaluate such projects have been rather scanty, at least

in the sense of economefric studies based on firm-level data (see

Klette, Moen, and Griliches, 1 998, for a recent review).

The mission of ATP is relatively new in the history of gov- ernment support to R&D in the US. Most ofthe previous federal support to R&D went either dkectly to universities and similar

institutions or was largely confract research ftinded by national defense and space agencies. Direct support of indusfrial R&D

with no quid pro quo is something new in the US, though it has

been a long established practice in Israel, Japan and other countries. Evidence ofthe social effectiveness offederal R&D expenditures in US industry is rather scant and largely negative (see Griliches and Lichtenberg (1984) and Griliches (1986)), and its defenders have been reduced to looking for possible inducement effects on private R&D expenditures (Archibald and Peirera,

1996). But that evidence is largely irrelevant for assessing the

potential benefits ofthe Israeli support programs, since that support is given to companies to help them pursue their own goals.

^This article is part of a more extensive research project that in- cludes a study (by Dr. Saul Lach of the Hebrew University of Jerusa- lem) on additionality in R&D and studies on subsidization of

physical capital (Bregman, Fuss, Regev, 1998) and labor (forthcoming) in Israeli industry.
ATP PAPERSAND PROCEEDINGS

It should be noted that we will not be able to address one of

the major topics of interest in such studies: the role of spillovers

from government-supported R&D. We are limited to confidential individual firm data within the Israeli CBS, though we can also look a bit more broadly at industry-wide effects. Given that most

ofthe output of these firms is exported, we have no good way to

trace the effects ofthis R&D further downstream. But analyzing

the first-order effects of government-supported R&D on productivity at the firm level is of interest in itself So far, the results obtained for US data have been largely negative. Since the goal

of R&D support has been different in Israel—^much closer to what ATP is trying to achieve in its own program—^the results for

Israeli data are also likely to be different, and possibly much more

positive.

The Period in Review

The two decades covered in this study were a period of

political and economic turbulence in Israel. The period began

with the energy crisis following the 1973 Yom Kippur War, which

set off an inflationary process that peaked in the mid- 1 980s. The

salient events that left their imprint on this period include:

• The change in ruling party, as the Likud acceded to power in 1 977 and introduced economic liberalization.

• The inflation process, which peaked at triple-digit levels in the first half of the 1980s.

• The 1985 economic stabilization plan, which drasti- cally reduced inflation and began an era of economic

restructuring.

• The exposure ofIsraeli manufactures to competing imports. Practically speaking, the process began with the Free Trade Area Agreement with the European Com- mon Market, concluded in 1 975, and continued with the signing of an FTA accord with the United States in 1 985

and the elimination ofnontariff import barriers between

1985 and 1990.

• Mass immigration in the first half ofthe 1 990s, mostly

from the former Soviet Union, which led to a significant increase in trained manpower that was integrated into

Israeli industry and made a major contribution to the perceptible growth of that industry during this time.

• The peace process that began at the end of the period,

affecting Israel' s international trade, foreign investments

in its industry, and the overall economy.

'In fact, the price will not decrease by the full 50 percent if the firm has to remit royalties to the government if the R&D succeeds—as is the case in Israel—but the idea is still the same.

The world economy also underwent far-reaching changes

during this period. Globalization and greater openness led to a

freer flow of goods, information, and people; greater competition; and a rising share of high-tech industry in many countries.

Israeli industry seems to have adjusted well to these changes.

The specific question explored in this paper concerns the effects of government-supported R&D on productivity at the level of

the firm and also the national economy as a whole.

Economic Examination ofR&D Support Programs

A Conceptual Perspective

There cire several ways of looking at govenmient support of

private R&D. The simplest way is just to think of it as a decline in the real price (cost) of R&D facing a firm. If the government

contributes 50 percent of the cost of a project, then the price of

doing this project is half of what it would be otherwise. If the production function is ofthe Cobb-Douglas form, including also

its R&D capital component, then a decline of 50 percent in the

price of R&D will lead to a doubling of R&D.^ That is, the fiill effect of the R&D support is the mcrease in total R&D that it mduces. There is nothing further to look for in the production

function. All the other effects are R&D effects in general, includ- ing its various spillover externalities. The only question here is by how much did the grant mcrease total R&D? Was it complementary to the firm's own expenditures or did it substitute for them? That is the issue of additionality. In terms of the production function for output, this assumes that only total R&D matters and that privately financed and government-supported R&D

are perfect substitutes." An alternative view denies that the source of funding does

not matter and looks for differences in the effectiveness with ' which such ftinds are used by firms. Govemmentally supported = R&D may be used less efficiently if it is subject to various constraints or if the entrepreneurs do not treat grants as "their" ] money. It could, on the other hand, yield a higher rate ofreturn if \ both the application and the selection processes choose the • more promising projects, i.e., if the agencies can actually "pick

winners." The simplest way of formulating this view is via the concept of''effective " R&D capital, where certain R&D expendi- I tures may create more (or less) capital than is indicated by the budgeted or reported figures. ^ "Effective" R&D capital can be written as:

I (I) R = R +(1 + B) R =R (1 +bs) ^ £0 g T

''Our inquiry found that one dollar of government subsidy for R&D j expands the firm's own R&D by $0.83 and that the difference relative to j,

1 is not statistically significant. As stated above, this matter is being

^ explored in an additional study.
EVALUA TION OFNATIONAL PROGRAMS

where and are own and government-granted R&D capital

respectively; 6 is the effective premium or discount on supported

R&D; R^ + Rg = R^ is the total reported R&D; and s = R^ / Rj. is the share of R&D grants in total R&D expenditures. If effective R&D enters the production function logarithmically, then we can

rewrite its logarithm approximately as: log R^ = log Rj.(l+5 s) s

logRj.+ 5s, provided the last term is small enough.

Even if it is not small, one can still think of adding the share of govemmentally supported R&D to the list of the included

variables in an estimated production function as a Lagrange multiplier test, since under the null hypothesis of equal effective- ness its coefficient should be zero. Thus the first view leads to looking at fums' own R&D expenditures and asking what happened to them as a result of the

availability ofgovernmental support, while the second looks for

differential productivity effects between own and governmentsupported R&D. One can, of course look for other effects at the

Ij firm level, such as export performance and employment expansion. But the most interesting other effects are the externalities generated by such R&D expenditures, in terms of information

made available to other firms in industry, in terms of helping to

create a newly trained labor force, and in terms ofproviding new,

higher quality, and cheaper goods and services to consumers

both in the original country and abroad. While the last aspect, that of externalities, may be the most

important in evaluating the success of such support programs,

our data will not allow us to pursue it. We shall focus mainly on

the second aspect, the question of the relative efficiencies of

different sources ofR&D for increasing the measured productivity of the "receiving" firms. The first topic, that of the effective- ness of the inducement process, will be taken up only briefly in the more detailed report of our research, since a more complete

analysis is currently being pursued by us and by Saul Lach.

Thus, the effect of government support of R&D, R , on a

' firm's output is a function of the extent to which this R<S?D en- hances output (is effective). We may view this by setting it within

a Cobb-Douglas production function, in which we mclude R&D

capital services as an mput:

' (2) log>^ = 2>,/?^ + YlogR^ + Ylog(l+6s)

k

where y expresses the effectiveness (elasticity) of total R&D.

! Obviously we cannot perform these and other computations

without estimates of 6 and y. We obtain these estimates by

I estimating the production function (2). First we used a standard method (OLS) to estimate the log

linear production fimction, adding firm fixed-effects and dummy

i variables and the percent-of-support variable, using the approximation: log (1 + 5s) « 6s. This approximation works when 5s is

"small." Since we obtained rather large estimates of5, we cannot assume that this approximation is entirely suitable in our case. But when we used non-linear least-squares to estimate equation

(2) directly, the results were rather similar.

The Data

This work is based on a unique data set that brings together

statistics from various sources on output, inputs (intermediates,

labor, and physical capital), and R&D (human resources, expenditures, grants, capital stock, etc.). All the data are at the firm

level and are based mainly on two sources: surveys performed

by Israel's Central Bureau of Statistics (CBS) and data obtained

from the Chief Scientist of the Ministry of Industry and Trade,

whose office provides the bulk of industrial R&D support in

Israel.This unique data set organizes the large amount of information assembled by the CBS fi-om industrial firms over many years

into long-term panels that allow us not only to examine the struc- ture of industrial firms at a point in time but also to follow their development over the years. The use ofpanels seems especially

appropriate in a study on the implications of R&D support for firms' performance, because it reveals correspondences between

productivity at different points in time and R&D investments,

activity, and fiinding type in previous periods.

The basic data—derived from CBS annual surveys (on in- dustry and R&D) and periodic surveys (every 5 years on skilled manpower and every 10 years on capital stock)—were used to estimate the variables needed to calibrate a production-function

model. This study placed special emphasis on calculating the R&D variables, i.e., R&D capital stock and the percent of gov- ernment support to it. R&D Variables

The CBS has been performing annual surveys of industrial R&D since 1970. Until 1980, they included all enterprises reported

in the records of the Chief Scientist of the Ministry of Industry

and other sources as having engaged in research and development. Since 1980, they have included a sample of firms engaging

in R&D. These samples are replaced every few years, and then

"R&D censuses" are collected. The censuses are used, among

other things, as a framework for the drawing ofnew samples. One

ofthe main sources from which the sample is selected is a list of

firms that applied to the Chief Scientist ofthe Ministry of Industry and Trade for R&D support. Another important source is the annual Industry and Crafts Survey, collected by the CBS since 1955, which occasionally examines R&D activity.

61
ATP PAPERSAND PROCEEDINGS

To calculate the R&D variables that we needed to estimate the production function, we gathered data on each firm's armual R&D

expenditure and grants in four periods, corresponding to the years

in which the samples were replaced.

To calculate the R&D variables that we needed to estimate the production function, we gathered data on each firm's annual R&D

expenditure and grants in four periods, corresponding to the years

in which the samples were replaced.

1990-1994—five annual surveys; 1985-1998—three annual surveys (1985, 1987, 1988);

1979-1982—three annual surveys (1979, 1980, 1982);

1975-1977—three annual surveys (1975, 1976, 1977).

On the assumption that an R&D project has a seven-year

lifetime,' annual estimates were computed for gross capital stock, net capital stock (gross stock less cumulative depreciation), and

capital services (computed as annual depreciation plus 5 percent of

net capital) in total R&D capital, with a distmction drawn between

R&D capital originating from the firms' own sources or from R&D

grants. As stated, to estimate R&D capital stock, seven consecutive years ofdata are needed for each firm. For firms that appear in the R&D samples during more than one period, all the requisite data were usually available. For firms that appear only intermittently, the missing data were estimated (for total R&D only, irrespective of

support) on the basis ofdata from the R&D censuses performed in

1 970-1 979, 1 984, and 1 990, and on the basis ofthe ChiefScientist's records. All the estimates were expressed in 1990 dollars using the Consumer Price Index to deflate current values.*

Productivity and Economic Performance Data

Most ofthe basic economic data were taken from the regular

industry surveys of the CBS. These surveys provide conventional data on domestic sales and exports; expenditures on inter- mediate inputs; employment, labor hours, and labor costs; changes

in stocks; annual mvestments in buildings, equipment, and motor vehicles; and characteristics of firms such as industry and

sector. These data were used to generate estimates of gross output, intermediates, and labor input.

Assembling the Data into Panels

The copious data obtained from the industry surveys were

organized into consistent panels. The industry surveys use a sample of approximately 2,000 enterprises that employ five per- sons or more. The sample is usually replaced every five years, although in the 1980s it took more than a decade to make the replacement and an update took place in the middle of that period. Usually firms that had seventy-five employees or more were

retained in the samples and not replaced. During the period of

concern in this paper, the samples were replaced in 1979, 1990,

and 1995. By the nature ofthings, one may construct full panels

only for periods in which a fijll sample was used. This is shown in detail in Tables 1 and 2, which present the numbers of firms and

observations of the three panels on which this study is based. Table 1 presents summary data for each period while Table 2

presents the average figure per survey.

• The full panel consists of24,775 observations that were

included in the twelve industry surveys between 1 975

and 1994. Information about R&D activity was provided

in 14 percent of the observations, and information on R&D support was obtained for 5 percent of firms in the panel (averaging about 100 firms aimually).

• The early 1 990s panel includes the five armual industry surveys performed between 1990 and 1994. The sur- veys investigated 2,920 firms, ofwhich 1,661 operated

throughout the whole period, and contained 11,158 observations. Supported R&D was reported in approximately 5.5 percent of the observations; unsupported

R&D was reported in an additional 1 0 percent.

• The 1980s panel spans the 1979-1988 decade and in- cludes about 8,000 observations, of which 4.5 percent

reported R&D support.

• The panel for the second half ofthe 1 970s includes find- ings from three mdustry surveys performed between

1975 and 1977. There were 5,730 observations, ofwhich

4 percent reported R&D support.

Table 1. Firms and Observations in Panels, by Periods

Period Surveys Number of

observations

Number of firms Firms active throughout period

-1- -2- -3- -4- -5-

1990-1994 1990, 1991, 1992,

1993, 1994

11,158 2,920 1,661

1979-1988 1979, 1982, 1985, 1988 7,887 2,819 1,297

1975-1977 1975, 1976, 1977 5,730 2,324 1,703

Total, 1975-1994 24,775

62
EVAL UATION OFNATIONAL PROGRAMS

Table 2. Firms in Panels, by Periods and by R&D Activity and Support

Surveys All firms Firms engaging Firms engaging Firms not

m supporiea 111 unsuppurit^u engagmg in ivocif R&D R&D

-1- -2- -3- -4- -5- -6- 1990-1994 Firms 2,232 124 224 1,884

Percent 100% 5.6% 10.0% 84.4%

1979-1988 Firms 1,973 91 189 1,693 Percent 100% 4.6% 9.6% 85.8%

1975-1977 Firms 1,911 75 90 1,746 Percent 100% 3.9% 4.7% 91.4%

Findings

In this section we present our main empirical findings. First we present some descriptive data on the extent of R&D and

support in Israeli industry, as reflected in the R&D surveys ofthe CBS (Figures 1 and 2). Secondly we show (m Figures 3-4) the

share and performance of 3 groups of firms: R&D-Supported Firms. This group includesfirms that reported conducting R&D and receiving grant(s).

R&D-Unsupported Firms. Includesfirms that reported

R&D expenditures but didn 't report any grant.

NORD Firms that didn't report any formal R&D

activity.

Finally we present in Tables 3-5 the production function

estimates. R&D Expenditures and Grants. In the twenty-year period

covered by our study Israeli industrial firms increased their annual investment in R&D by a factor of4.4 (an annual rate of7.7%)

which came close to 700 million in constant 1990 dollars in 1996. Figure la which presents the data also shows that during most of

the period reviewed R&D support fluctuated within a range of

18-20 percent of industrial firms' R&D outlays. There were several outliers—23 percent in 1976-1977 and 1 1 percent in 1984-

1985.

Figure la: R&D Expenditures and Grants 1976-1996

800 -

Share ofR&D in Industrial Sales. Figure lb shows that in the late 1970s about 1 percent of industrial sales was invested in industrial R&D. R&D expenditure continued to grow rapidly until the mid-1980s to more than 2 percent of sales. The watershed

year was 1985, when R&D expenditure began to decrease—in a process that coincided with the termination of development of

the Lavi aircraft—and stabilized at 1 .6-1.8 percent ofsales.

Figure lb. R&D Expenditure as a Percentage ofSales

2.00 I

o 1.00

J? 0.50

Concentration in R&D. Figure 2a shows that R&D is quite concentrated. Ten firms carry out 46 percent ofR&D and 45 firms do more than 80 percent.

Figure 2a. Concentration ofR&D Expenditures, 1996

R&D Support. Figure 2b shows that 13 percent of R&D is done by firms that receive no grants and that 3 1 percent ofR&D

is done by firms that cover 3 1 percent or more of their R&D

outlays with grants. An additional 25 percent of R&D is per- formed by firms that are supported at the rate of 21-30 percent.

63
ATP PAPERS AND PROCEEDINGS

Figure 2b. Distribution ofR&D Expenditures by Rate ofSupport, 1996

40+% no support 31-40% (7.8%) (13.5%)

(23.6%)^ ^

21-30°J

(25.6%) 20%

The Contribution (Share) of the R&D-Supported Firms. The

R&D-supported firms' share in employment was around 10 per- cent at the beginning of the period, it grew substantially during

the first decade to 20 percent in 1985 and declined during the second decade to 14.5 percent in 1994 (Figure 3). Yet they ac- counted for a majority of R&D capital services, reaching about

80 percent in 1994, despite the decrease in labor share. Figure 3

also reveals that from 1982 onward R&D-supported firms ac- counted for 35 to 48 percent of Israeli industrial exports.

Figure 3. Share of Labor, Capital R&D and Exports of R&DSupported Firms

Output (Figure 4). Throughout the period reviewed (except

for 1976), output per person-year was 20-50 percent higher in firms that received support than in firms that engaged in R&D

without support. This effect was especially strong in 1985 and

1988, with the disparity reaching some 50 percent. In 1994, the difference contracted to 30 percent. Throughout this period out- put per worker was usually 20-30 percent higher in unsupported

R&D firms than in non-R&D-performing firms.

Figure 4. Outputper Person- Year by R&DSupport Status ofFirms

Findings of the Production Function. Table 3 presents our

production function estimates for Israeli industrial firms. The

explained variable is output per person-year, and the explanatory

variables are intermediate inputs, labor, capital input (physical

and R&D), quality of manpower, and dummy variables for firm

size, industry, sector, survey year, R&D activity, and method of

estimating physical capital (imputation).

Columns 2 and 3 present the results of estimating the pro- duction function for the entire period from 1975 to 1994, using all available observations.^ They are based on an unbalanced panel

of 24,775 observations culled from eleven industry surveys. Col- umn 2 presents the simple version that was used, for example, by

Griliches and Regev (1995), which assumes that R&D support

has no differential effect on productivity, i.e., that different sources of R&D funding are perfect substitutes.

Table 3. Production Functions for Total Output, 1975-1994

and Intermediate Periods Dependent Variable: Output per

Person-Year, Logarithms ofAbsolute Values

Period 1975-1994 1975-1994 1990-1994 1979-1988 1975-1977

excL gov. grant incl. gov. grant incl. gov. grant ind. gov, grant ind. gov. grant

var. var. var. 1-

-2- -3- .4- -5- .6-

Olwprvniiniis 24,115 ?4TTg 7 ftB7 5.730 r' O.Sg7 O.S8g 0.890 0.842 0.S50 RMSF, 0.296 0.294 0.253 0.317 0.324

Inlermfriiafp Input

0.659 0.660 0.664 0.677 0.624

(294) (294) (194) (15.?) (149)

PhysimI Papital 0.078 0.077 0.106 0.057 0.055

(29) (29) (27) (10.9) (7.4)

BAn rapiral SiTviri.s 0.034 0.02S 0.036 0.028 0.023

(9.7) (7.6) (7.2) (4.4) (2.1) R&DGninK a^a 0.183 0.134 0J91 -0.180 Frartinn nfthpTnlal (4.4) (2.7) (4.2) (-1.01 Estimated t ratios in parentheses.

The regression includes additional control variables: industry, size of firm, year, and sector. The coefficients are very similar to those reported by

Griliches and Regev (1995) and other studies that based themselves on similar data. In Column 3, we add the variable R&D grants as a percent of

total R&D capital services and obtain a positive and statisti- cally significant (t=4.4) coefficient of 0.18. In Columns 4-6, we

present the same regression for interim periods and obtain positive and significant coefficients for 1990-1994 and 1979-1988.

However, the coefficient for the 1970s period is not statistically

significant.

If the null hypothesis of perfect substitutability is correct, we would expect to find a zero coefficient for this variable. If government-supported R&D is used less efficiently, we would

expect its coefficient to be negative. In fact, the estimated coeffi- cient is positive and, in most sub-periods, statistically signifi- cant. Thus, our first result shows that such support is not wasted

in the Israeli economy and may even have a higher rate of re- turn than privately-financed R&D.

64
EVAL UATION OFNATIONAL PROGRAMS

These production-function results are subject to a variety

of left-out variable biases. Firms that perform R&D and receive

government assistance are not a random sample in the population of all "appropriate" firms. Although a lengthy list ofcontrol

variables was included, significant aspects may have been omitted in the characterization of these firms—e.g., quality of management and other resources which caimot be measured by the econometrician but are related to present success and present

R&D policy. We can deal with this problem in two ways. First, we

may assume that such attributes of firms that were left out are

largely constant, at least over a period of five to ten years, and

may apply methods that try to solve such problems, such as "fixed effects" or changes between the beginning of the period

and its end. This eliminates the unobserved fixed firm characteristics and doesn't allow them to contaminate the estimated coefficients of the relevant variables. Second, we can estimate a parallel model offirms' decisions

to perform R&D and to apply for grants and incorporate its implications into the estimation ofthe production function to account

for the selectivity problem. In this paper, we focus on the first choice. We intend to pursue the second possibility (the more

elaborate one) in a subsequent paper. "Fixed Ejfects " and "Long Dijferences " Production Functions. Table 4 provides an estimate of the production flmction

using the "fixed effects" method. These estimates reflect the "within" dimension of the data and eliminate the influence of

fixed differences among fmns on the estimated coefficients. The

functions were estimated for two main periods, the 1 980s and the

1990s, in balanced panels (firms that were active throughout that time) and unbalanced panels. We also estimated production functions for changes between the beginning and the end of each

period (long differences). These methods show that both the

coefficient of total R&D and the implicit benefit of its governmental component are positive and have become statistically more significant in the 1 990s. As for changes between the beginning and the end of the period, the picture resembles that elicited by the fixed-effects regressions, although the statistical signifi- cance of the R&D-support variable declines. Estimating the Contribution ofSupport to Firm Productivity. Table 5 gathers the estimates of d—^the effectiveness ofR&D

support—as obtained by the various estimation methods: ordi- nary least-squares, fixed-effects, and changes from beginning to end of period.

These different estimation methods yield similar results: in the 1990s, d falls within the range of2^. Inthe 1980s, d is very

high—in the range of 7-10. In the 1970s, it is not statistically

significant. Although the coefficient of the percent-subsidized

' A fuller version of these figures and tables will be given in the more extensive report on this study (in preparation).

variable, d g, is positive and statistically significant, d, the pre- mium on government-supported R&D is difficult to estimate accurately. This happens because d is estimated as a ratio of two

coefficients, each of which is estimated with reasonably large

standard deviations. We attempted to examine the distribution of d by using the bootstrap sampling technique. For example, in a balanced panel

for 1990-1994, in a sampling of 100 samples, estimates of stan- dard deviation were obtained that pointed to a confidence inter- val for d of about 0.4 to 7. Similar results were obtained when

Feiler limits were used, as shown in Griliches ( 1 967), pp. 1 09- 1 i 0.

Table 4. Estimated Fixed Effects and Long Differences

Production Functions, 1990-1994 and 1979-1998 Dependent

Variable: Outputper Person-Year, Logarithms ofAbsolute Values

1979-88

Period

1990-94 Unbalan

ccd

1990-94 Balan- txA1990-94 Long

DifG£i^.ence3

1979-88 Unbalanced1979-88

Balanced

Long

Diffei^ enccs

Observatioiu 11,158 8.305 1.654 7387 5,188 1.297 0.6S8 0.65« 0.685 0.619 0.61 0.59

Intennediale Input

0.610

(110)

0.60«

(98)

0.671

(55)

0.651

(78)

0.64

(68)

0.63

(40)

Ffivsical Capital 0.0«9 0.0«7 0.047 0.076 0.084 0.087 Servko (13) (12) (4-3) 02) (7-S) &1)

R&D Capital Service! 0.09«

(13.7)

0.086

(11-6)

0.050

(4.0)

0.031

(4.4)

0.034

(4-1)

0.036

(3.6)

Grants as a */• of CapitaJ O.I4« a-7)

0.173 OS)0.149 (15)0.129

(1.3)

0.131

(1.9)

0.165

(1-5) R&D Service

Estimated t-ratios in parentheses.

Table 5. Summary ofFindingsa: Estimated Coefficients

Period T b

-2- -3- (1/2) Indirect Measur^eot

1. Production Function Full period 0.183 0.028 6.5 1990-1994 0.134 0.036 3.7 1979-1988 0.290 0.028 103

1975-1977 -0-180' 0.023 -7.g«

2. rned EfTects, balanced 1990-1994 0.173 0.086 2.0 1979-1988 0.131 0.043 3.0

3. LoDg DifTcrmcci 1990-1994 0.149 0050 3.0 1979-1988 0165 0,036 4.6

4. Direct (non-linear) 1990-1994

197>-1988

4.0 73

* Not statistically significant.

Consequently, the premium on support is indeed positive,

but at the present stage of the study we cannot estimate it with

much precision.

Conclusion

As we have seen, the coefficients ofthe grants variables are positive and statistically significant in abnost all ofthe estimated

models. However, they are much higher than one would have

predicted. It is hard to believe that government support is at least twice as effective, and in other periods seven times as effective,

than the firms' own R&D investments. This fmding indicates

65
ATP PAPERSAND PROCEEDINGS

that the specification of our model may be incorrect or may exclude some relevant variables. There may well be an upward bias and a selectivity effect that originates in the way grant-receiving

projects or firms are chosen. Ifthis selectivity is based on permanent attributes of the firms, it can be dealt with, in principle, by

including individual fixed effects variables or using long-term

changes. However, if the selection process successfully identi

fies and predicts future success—and allocates resources ac- cordingly—then even "within" estimates may be upward-biased.

Another possible interpretation is that firms are heteroge- neous in terms of their technological potential (g) and that the Chief Scientist's apparatus is able to pinpoint these differences,

albeit only partly.

Importantly, these reservations do not diminish our main

finding: that the mechanisms that allocate R&D dollars seem to be doing their work properly in most cases, and the more they

manage to "pick winners" the better. However, our estimates

should be treated cautiously and should not be used in an at- tempt to predict what will happen if R&D support budgets are increased or cut. Success may hinge on the ability to contmue

diagnosing correctly a firm's potential.

We should also note that our study used R&D grant data as obtained from CBS surveys but did not use the detailed database

of the Chief Scientist of the Ministry of Industry and Trade,

which was made available to the CBS. By adding these detailed data to our panels and calibrating these models and others, we

may shed further light on the topics at issue. Also, the Bank of

Israel is doing research on additionality, using CBS data, and we

also intend to investigate our data further.

Bibliography

Archibald, R. B. and A. M. Pierera (1996). "On the Effects of

Public and Private R&D." Williamsburg, VA: College ofWilliam

and Mary. Mimeograph.

Griliches, Z. (1986) "Productivity, R&D, and Basic Research at the Firm Level in the 1970s," American Economic Review, 76(1),

141-54.

(1961). "Distributed Lags: A Survey," Econometrica, 35 (No.

1): 108+109.

and F. Lichtenberg (1984). "R&D and Productivity at the Industry Level: Is There Still a Relationship?". In Z. Griliches

(ed.), R&D, Patents and Productivity. Chicago: University of

Chicago Press, pp. 465+496.

and H. Regev (1995). "Firm Productivity in Israeli Industry:

1979+1988," Journal ofEconometrics, 65 (No. 1): 175+204.

Israel Central Bureau of Statistics (CBS) (1998). "Survey ofR&D

m Industry, 1996". Survey of Innovation in Industry 1997. Current Statistical Findings No. 24. (Hebrew)

(various years). Survey of Industry and Manufacturing.

(Hebrew)

(1998). Survey of Fixed Gross Capital Stock in Industry,

1 . 1 . 1 992. Publication No. 1 098. (Hebrew)

Klette, T. J. and J. Moen (1998). "R&D Responses to R&D Subsidies: A Theoretical Analysis and aMicroeconomic Study." Mimeograph. Presented at the NBER Summer Institute.

and Z. Griliches. "Do Subsidies to Commercial R&D Re- duce Market Failures? Microeconomic Evaluation Studies,"

Memorandum No. 13/99, University ofOslo. Also, Working Pa- per No. 6947, NBER, Cambridge, MA. Forthcoming ui Research

Policy.

66
Comments on R&D, Government Support,

and Firm Productivity in Israeli Industry

byFran9oisSand^

Mr. Regev started his presentation by giving us an over- view of the context of R&D support in Israel. He briefly intro- duced the various institutions and programmes which aim at supporting R&D policy in his country. He quoted six ofthem:

• R&D subsidies granted by the Chief Scientist of the MIT

• The "Magnet" ^ro^don

• The "National S& T infrastructure " - consortia of aca- demic research and members from industry

• Binational programs (USA and Israeli) - support col- laborative R&D

• R&D projects supported by the Ministry ofDefense

• Support to basic research at Universities

He addressed the Method of Analysis outlining the following elements:

• Estimating Production Function, including R&D

Capital Measures.

• Calculating Rates of Return to the various Capital Measures.

• Descriptive Statistics by R&D activity and government

support.

The production Function Variables concerned were also pre- sented as follows:

• Dependent: Production per person-year

• Inputs: Intermediate, Capital Services. R&D Capital Ser- vices (All per Person-Year)

• Control Variables: Quality ofLabor, Size, Branch, Sector, Age, Cohort, Shadow-of-death (continuity).

Mr Regev addressed the goal of the paper as the examination of the overall role of government supported R&D and its contribution to the growth of productivity in related industries. He reported on the practical methodology based on the "Longitudinal Panels," documenting the sources of data used by them,

and the types and time span concerned (every year over the 1990/94 period, and less frequently before, with up to a 40 year period covered in some cases).

He then discussed the R&D capital variables and services and the maimer m which the analysis is both currently performed

and will be performed in future developments. In the presentation, I was impressed by the volume of data involved in the study

and made available not only by the OCS but also by the participants. It would be a dream to have just a small fraction of it in our

case, regarding participants in EUREKA projects.

He then proceeded to talk about levels of funding and at this point, I have a question for Mr Regev. On the Web, in "Israel

Start ups and Innovations: Overviews", Dr. Oma Berry, the Chief

Scientific officer quotes the figure of $1.25 million as the total overseas investment in Israel in 1 996. How does it compare with

the total national support and could this affect the results of

your study in any way?"

Then statistics were presented indicating a high concentration of R&D. It was said that 14% were not supported: 14% of

what (firms, total expenditure?). A high concentration exists also

in certain sectors. Then the production account by R&D activity was considered. It resulted that the return on investment was

higher for firms with R&D. At that point, I was not well seated and could not read the figures related to the effect of subsidies?

Was it positive or negative?

Professor Zvi Griliches presented some conceptual considerations. I was very interested, following the talk that Professor Zvi Griliches gave the evening before, to hear the conclusions on this matter. He stressed the differences that exist between

ftmding by government investment and private investment, questioning how worthwhile the first one could be. One could wonder

indeed if the money allocated by government is really a full in- crease in the R&D budget or whether there is a partial displacement of capital towards other types of investment within the firm? Regarding the matching of figures, a study is underway.

' Dr. Sand is with the EUREKA Secretariat where he is responsible for the Continuous and Systematic Evaluation of the EUREKA

Initiative.
ATP PAPERSAND PROCEEDINGS

He then addressed Secondary effects. He particularly underlined two opposite effects depending on whether people in- terested consider the subsidies as "their money" or not? The

question that entrepreneurs treat public support as if it were their own money or not can be addressed, but how could we measure

such a "subjective" behaviour? He referred to the previous studies (the U.S. case in the 70s and 80s) on the rather negative effect of U.S. federal R&D expenditures on industry (the 1986 results looks different). It would be interesting to see if such an effect is comparable in the context of Israel. Apparently, it looks more

favourable. It even looks more positive than the Norwegian results that were reported earlier. Is it a matter of chance?

He went on with Selection related problems. I shall at this point ask a question which I consider of basic importance in this matter: You said that "supported R&D could yield a higher rate of return, if the application and the selection processes 'pick the winners.'" I am wondering whether there is not some type of

vicious/disturbing approach here. Indeed, should public money

be invested in companies that could possibly be as efficient themselves without public support? This type of question must

be addressed clearly by any governmental agency. I think that in Germany, for example, the most successful furns are not necessarily the ones that are supported by government funding.

Several questions relating to the model to be addressed: Could bottom up not be advisable? Is money the critical factor? How to control the average conditions ofthe firm? What is happening to the productivity of the firm over time? A "Change"

metric?

It appears a very high premium of300%? How realistic is the resuh? Could Instrumental Variable Techniques be used in the

future?

In the future, new approaches are to be considered such as:

1) Impact on R&D: what are the differences between sup- ported and not supported projects.

2) Back payment of successful projects: the corresponding effects have not yet been analyzed.

It is wise to underline how important the large data set from

NO, U.S. (ATP) etc., will be in connection with many methodological questions.

In conclusion, before letting the other participants ask their questions, I would like to remind you oftwo ofmine, i.e.: "How

do you discriminate the effects of the overseas investments in Israel ($ 1 ,2 1 5M in 1996) and those ofthe national support?" and

"Is it the role of the National support schemes to pick the winners?"

68
From;Growth Theory to Technology Policy - Coordination Problems in Theory and Practice

by Tor Jakob Klette and Jarle Moen'

Abstract

Economists, in particular Bresnahan and Trajtenberg (1995),

have recently drawn attention to the importance ofgeneric or

general purpose technologies (GPT's) and their significance

for economic growth. An interesting part ofthis research identifies coordination problems in the introduction of GPT's, and

the potentially large benefits in coordinating research andproduct development. Thinking about information technology as a

GPT, with the associated coordination problems, seems to fit well with the motivation behind governmental support schemes

to IT and related high-tech industries in Norway. Thefirst part

of our studyfocuses on a series ofsuch ITprograms that have

been implemented in Norway from the early 1980s with the

objective ofcoordinating the development of information tech- nology and its application throughout the economy. We examine in some detail the largest of these IT-programs through its planning and implementation stages and emphasize how closely

it is connected to recent economic analysis of GPT's. The sec- ondpart ofour study examines to what extent these governmental plans and subsidy schemes have been successful in creating

economic results in terms ofgrowth andprofits in the ITandITrelated industries. In thefinal part ofthe paper we discuss some

of the lessons about the problems with technology policy at a

practical level.

Introduction

Information technology has been recognized as a 'generic

technology' with 'strategic importance' for economic developI ment by many commentators and governments. In this spirit a

' number of countries, including Norway, have implemented governmental programs to promote the production and application

of information technology. Economists have had a hard time

making sense of terms such as a 'generic technology' and a

ii technology being of 'strategic importance', at least until Bresnahan and Trajtenberg introduced the notion of 'General

Purpose Technologies', and examined their potential importance

j for economic growth. GPT's are characterized by their wide appliI

cability, their potential for development and what Bresnahan and

Trajtenberg called iimovative complementarities. By innovative

complementarities they had in mind positive pecuniary extemali

ties between the development ofthe basic general purpose tech- nology and innovations in the sectors using this technology.

Such externalities tend to create coordination problems and

Bresnahan and Trajtenberg argued that due to the pervasive

applicability ofGPT's, these coordination problems might be large even in a macroeconomic perspective.

As we explain in detail below, the analysis of coordination

problems associated with 'general purpose technologies' seems

to capture quite well the motivation behind the substantial effort and money spent by governmental agencies in Norway to pro- mote the production and utilization of information technology,

and also the many attempts to coordinate the various policy

tools involved in this effort. The dominating part of these IT- programs became targeted directly at promoting the manufacturing ofIT-products. The IT-programs were implemented throughout the 1980s and 1990s, and their considerable size is indicated by the total expenditures amounting to NOK4.4 billion ($620 M)

for the largest of the programs implemented over the four year

period 1987-1990. Having discussed the theory and the programs in the first two sections, we present a quantitative analysis of the impact of

the IT-related technology programs on the manufacturing part of

the IT-industry including closely related high tech manufacturing sectors. In the first part of this analysis we compare the performance of targeted firms to other firms in the same industries. Next, we consider the development ofthe IT-industry and

the related high tech manufacturing sectors relative to the per- formance of the manufacturing sector at large, and finally we

compare the performance of these sectors in Norway to their performance in other OECD economies.

The general conclusion is that the IT-programs, while well

justified according to economic principles, seem to have failed in promoting the development of the IT manufacturing sector in Norway. In the last part ofthe paper we discuss various explanations for the failure ofthese programs such as informational problems and institutional inertia in the governmental agencies heading their unplementation.

' Tor Jakob Klette is with Statistics Norway, Research Department.

Jarle Moen is with the Norwegian School of Economics and Business

Administration.
ATP PAPERSAND PROCEEDINGS

From New Growth Theory and Coordination Problems to Technology Policy.

Innovation, Economic Growth and Technology Policy.

Externalities associated with R&D, learning and innovation

have been emphasized in recent developments in growth theory,

and it has been widely recognized that these externalities create coordination problems and possibly scope for welfare improving

government interventions. Theoretical work on economic devel- opment and growth has emphasized that the development of new industries in the presence of such externalities tend to create multiple equilibria where one equilibrium corresponds to the new industry never reaching a 'critical mass' or never 'taking

off while other equilibria correspond to the industry 'taking

off and starting on a cumulative growth process.' It is the complementarity in activities across independent firms, e.g. in innovation activities, that give rise to multiple equilibria with

high and low levels of growth. There are several policy tools available to deal with externalities and coordination problems in innovative activities as discussed by Romer (1993) and many

others. In theory, external effects can be corrected for by tax

credits, grants, public production and extending property rights

through patents or copyrights. All these means have been used

by the OECD countries to promote R&D and innovation. However, the issue of optimal design ofR&D and innovation poHcies

are far from settled, and the practice of technology policy vary

substantially across countries, technological fields and various

stages of the innovation process."

A particular coordination problem that we want to focus on

arises when the technology in question is 'generic'. Information

technology is one example ofthis, and it is a technology that has

been actively promoted by most OECD governments.

An Economic Analysis of 'Generic ' or 'General Purpose ' Technologies

According to Bresnahan and Trajtenberg (1995), economic

models, including most growth theoretical models, tend to "treat

all forms of technical change in the same, diffuse manner," and

there has been little economic analysis suggesting that research and innovation associated with "generic' technologies such as information technology require particular attention. This motivated Bresnahan and Trajtenberg (1995) to introduce the notion

^ See the appendix in Da Rin and Hellman( 1997) for a formal discussion of the notion of critical mass and take off problems in the presence of

positive externalities and complementarities.

of' General Purpose Technologies' ^ (hereafter GPT's), which they

characterized by: (i) pervasiveness, (ii) potential for technical improvements, and (iii) innovational complementarities. Drawing

on studies by economic historians on the role of the steam engine, the factory system and electricity, they argue that GPT's

may be essential to understand the importance of innovation for economic growth. With respect to recent history, Bresnahan and

Trajtenberg focus on the development of semiconductors and

rr. There are two features of GPT's that we should emphasize.

First, generality of purpose which means that a GPT potentially can be applied in several application sectors. Second, that such

applications require complementary iimovations. That is, there

is complementarity between iimovations in the GPT and innovations in the related application sectors. An innovation in an application sector will make the GPT more useful and thereby ex- tend its market. A larger market means that further innovations in the GPT will be profitable. A better GPT will in turn widen its usefiilness in the application sectors and thereby make further complementary innovations in the application sectors profitable.

This complementarity between innovations in the GPT and an

associated application sector involves pecuniary externalities which tend to create a coordination problem.

There is a second type of complementarity associated with

GPT's. An irmovation in one application sector will, as we just

have explained, create incentives to develop further improvements in the GPT. Improvement of the GPT will benefit other

application sectors associated with the GPT, and hence, there is complementarity not only between the GPT and each application

sector, but also between iimovations in different application sectors. This creates further pecuniary externalities, and a need for coordinating innovations both between the GPT and each appli- cation sector and between different application sectors associ- ated with the same GPT.

Bresnahan and Trajtenberg (1995) argue that the development of a GPT and its applications have a sequential order. Specific innovations in the application sectors can only be implemented profitably when the GPT has reached a certain stage of

development. This sequential aspect of innovations m the GPT

and innovations in the application sectors reinforce the desirability ofcoordinating R&D and innovative activities. Bresnahan

and Trajtenberg point to the current complaints of software de- velopers against Microsoft as an illustration of the coordination

problems that might arise. Software developers argue that Microsoft 'excessively' exploits its coordination advantage as

' See e.g. Murphy, Shleifer, and Vishy (1989), Milgrom, Qian,

and Roberts (1991), and for a survey, Matsuyama (1995).

" See Mowery (1995).

' See also the subsequent work in Helpman (1998).

70
EVAL UATION OFNATIONAL PROGRAMS

the developer of both Windows and other software, by not dis- closing as soon as possible features in new versions of Windows. The general point is that there might be a significant ad- vantage for the developers of various applications to have detailed insights into the research and development of the basic

technology, i.e. the GPT itself Bresnahan and Trajtenberg conclude that arm-length market transactions between the GPT and its users will give 'too

little, too late' innovation. Difficulties in forecasting and coordinating the technological developments in the GPT or in the vari- ous application sectors can lower the rate of technical advance,

diffusion and development of new as well as old sectors of the economy. Economists, when recognizing these coordination problems and their undesirable consequences for economic growth,

tend to point out the scope for welfare improving government

intervention.

Technology Policy andrras a General Purpose Technology

Information technology at several levels can be characterized as a GPT. First, at a basic technological level, the development of semiconductors and integrated circuits have served as a GPT for a vast number of application sectors, and there have

been strong iimovational complementarities between the devel- opment ofthe integrated circuits and innovations in various kinds

of computers, telecommunication equipment and a whole range

of other electronic devices. Second, if we focus on the development ofthe computer, in particular the PC, this represents a GPT

in itself, having e.g. different pieces of software serving as appli- cation sectors. Thinking ftirther about various kinds of software

associated with the PCs, we can recognize e.g. the worksheet or word processors as GPT's at a new level.

I Our point is that the introduction of various parts of inforI mation technology often involve innovative complementarities

I and might therefore create some of the coordination problems

that we discussed above. This perceived need for coordination seem to capture quite well the motivation behind the policy ini- tiatives related to production and application of information techijj nology made by the Norwegian government in the 1980s and

»' 1990s. Similar initiatives were launched by the governments in

I other OECD economies.

' Introducing the National Program for Information Technology for the period 1987-90, the government wrote in its budget

report:* The motivation for the program is information

technology's role as a strategically important field for

' 'Statsbudsjettet 1986/87', St.prp. nr. 1, p. 40. Our translation.

' Harlem et al. p. 235. Our translation.

manufacturing growth, and fiirthermore its general sig- nificance for increasing productivity and growth in other

industries and services. This argument was elaborated on in the report fi-om the offi- cial commission evaluating the program, where the following as- pects of information technology were emphasized:^

Information technology has broad industrial and

economy wide applications, but this is not entirely exceptional. More basic for this type oftechnology is the need not only to develop the technology itself, but to adopt the technology to the needs in quite different applications; in manufacturing, the public sector and in the economy at large. In this situation there are two

essential factors relevant for the development of a co- ordinated technology policy: The applications repre- sent the market for the manufacturers while the manufacturers are problem solvers for the users. This creates a demand for an IT-policy reflecting the integration be- tween researchers, users and producers.

The report fi-om the official commission then goes on to discuss to what extent the targeted program for information tech- nology was an appropriate policy tool, and we will return to their conclusions below.

To sum up, the Norwegian policy initiatives on information

technology in the 1980s and 1990s were motivated by an understanding of the broad set of potential applications for IT and the

inter-action between the basic innovations and the adoption and

development of these innovations in the applications sectors. This motivation for a coordinated plan and a govenmient initia- tive targeted at information technology, is in our interpretation

congruent to the analysis of GPT's and the coordination problems emphasized by Bresnahan and Trajtenberg (1995).

Coordination Problems and the Norwegian IT-Programs.

The Technology Programs related to Information Technology in the 1980s and 1990s

In Norway in the 1980s there were some widely held concerns about the state of the domestic mformation technology

industries with an emphasis on the following three sets of problems: (i) Fragmentation ofpublic fimds for R&D, innovation and

utilization ofIT-technology, (ii) too many small and independent

firms, and (iii) little long term planning and originality in product

development.* The promotion of the IT-industry in the period we

« See e.g. Murphy, Shleifer, and Vishy (1989), Milgrom, Qian, and

Roberts (1991), and for a survey, Matsuyama (1995).

71
ATP PAPERSAND PROCEEDINGS

consider from 1 982 to 1 995 was organized and coordinated through

a number ofplans and programs ofvarious size.' The largest plan in

this period was the aforementioned National Program for Information Technology,'" lasting from 1987 to 1990. This program had a

total budget ofNOK4.4 billion' ' and included a number of ' subprograms' (see below).

Before 1987, the Royal Norwegian Council for Scientific and

Industrial Research (NTNF) had implemented several funding

schemes which were predecessors to the National Program for In- formation Technology,'^ and the industrial part ofthe National Pro- gram for Information Technology was succeeded by the 'National Plan for Improved Utilization of Information Technology in the Norwegian Industry 1 992-95 '

. ' ' This last program was small in terms ofits independent budget, and its main objective was to coordinate

ongoing public support schemes related to information technology.

In the rest of this paper we will refer to the various support

schemes for industrial applications of information technology as

the 'IT-programs'. Before we turn to an overall evaluation of the economic impact ofthe IT-programs, we will discuss more closely

the National Program for Information Technology. As stated, this was the most important and ambitious of the programs, and its implementation and organization are extensively documented in Harlem et al., Buland (1996) and other publications.

A closer look at the National Programfor Information

Technology, 1987-90

The National Program for Information Technology was a broad

plan to coordinate activities aimed at promoting the production and

applications of information technology. The plan covered basic

research, education, production of integrated circuits and comput-

' The R&D subsidy programs have been administered by various

research councils and governmental funds. With respect to the high tech

industries the Royal Norwegian Council for Scientific and Industrial Research and the Fund for Industry were the most prominent agencies. In the early 1990s the various research councils were merged into the Norwegian

Research Council, and most governmental industry fiinds were merged

into the Norwegian Industry and Regional Development Fund. Besides

these agencies, R&D grants have also been awarded directly through

ministries. 'Den nasjonale handlingsplan for informasjonsteknologi'. See Harlem et al. (1990) and Buland (1996) for detailed documentation.

" Approximately $ 620 Mill. This is the size of the formal budget, while the 'fresh money' amounted to NOK 2. 1 billion, see Harlem et al. (1990), ch. 7.2.3. These included: (i) 'Nyskapningsplanen 1977-82', see Gronhaug

and Fredriksen (1984). (ii) 'NTNFs Handlingsprogram for Mikroelektronikk og Databehandling 1982-85', see Klette and Sognen

(1986). (iii) 'Nyskapning i naeringslivet' which started in 1984. (iv) 'NTNF's sepsial-program for mikroelektronikk' which started in 1985 . All these activities were related and the last two programs were

ers, and applications of information technology throughout the economy including the public sector.''' Even though the original

plan had a very broad scope, the actual implementation ofthe pro- gram focused heavily on manufacturing of elecfronics and other IT-products. According to Harlem et al. (1990):'^ The program's focus on manufacturing can be ob- served in the distribution of project grants by institution; 48 percent ofthe budget went to firms [which were mainly

firms in electronics and related high tech industries], while

another 33 percent went to government labs which in practice also were focused on applied research for the manufacturing sector. The project fiinds were very imevenly distributed across firms,

with the ten largest recipients receiving 35 percent of the fiinds. These firms were producing electronic products, telecommunication equipment, instruments and computers.'* The largest recipient, Norsk Data, received by itself more than 12 percent of the budget allocated to firms.'^ Table 1 presents the expenditures for the National Program for Information Technology 1987-90. To illustrate the considerable

magnitude of the numbers in Table 1, one should notice that, e.g., publicly fiinded technological and scientific R&D in universities and governmental labs in 1989 in total amounted to NOK 2542

Mill."As can be seen from Table I , a significant part ofthe National

Program for Information Technology's budget went to education

and to a lesser degree also to basic research related to IT. At least the educational part of the program has been considered successfiil by Harlem et al. (1990) and others, but our focus is on the substantially larger parts of the IT-programs that were targeted more

directly at industrial production and appUcations of information

technology.

continued within the National Program for Information Technology

from 1 987. The research councils also sponsored a number of individual research projects related to IT. See 'Stortings prp. nr. 133, 1977/78' for

details. IT-plan for naeringslivet 1992-95", see Olsen et al. (1997) for

details.

'" See Harlem et al. ( 1 990), chs. 4 and 7.2 . " P. 64, our translation. The ten largest recipients were Norsk Data, Autodisplay, EB

Nera, Nordic VLSI, EB, LCD Vision, Seatex, Micron, Simrad Subsea

and Alcatel/STK. The order reflects the size of the fimding. " This percentage does not include the so-called FUNN-project.

See Harlem et. al. (1990), especially ch. 4.1.1 for further details on Norsk Data's projects within the National Program for Information Technology. " See NIFU ( 1 99 1 ), Table T6 and N2. Publicly funded technological

R&D in universities and governmental labs in total amounted to NOK

1245 Mill, while the public funding for scientific research in universities was NOK 1297 Mill. Publicly funded R&D in private firms was NOK

465 Mill, in 1989.

72
EVALUATION OFNATIONAL PROGRAMS

Table 1: Expenditure within the "National Program for

Information Technology 1987-1990" broken down byfield and

year. Million NOK

1987 1988 1989 1990 Total

EducaEon 306 373 426 427 1 532

Rfsearch 138 132 135 130 534

Produa development 134 ISl 239 221 745

Applications 329 369 398 474 1 570

Total 907 1 025 1 197 1 252 4381

Source: Harlem et al. (1990).

A Quantitative Assessment ofthe Economic Results

of High Tech Support in the 1980s and 1990s.

Expectations about the Effects ofthe IT-policy.

Based on the theoretical arguments related to GPT's, one would expect the IT programs and the coordination effort to stimulate economic performance in the targeted furos and industries. Such expectations were most clearly stated by the committee heading the implementation of the National Program for In- formation Technology from 1988-90, which anticipated an annual

growth of 1 5 percent in sales and 20 percent in exports from IT

manufacturing as a result ofthe Program; see Harlem et al. ( 1 990),

pp. 173-4.

It is not obvious how one could test such predictions, since we do not know what would have happened if the program had

not been initiated. We have confronted the predictions with observed outcomes in a number of ways. Our first approach is based on comparing the performance ofthe firms receiving R&D

support to other firms operating in the same industries, and the

prediction we consider is that the supported firms performed

better than other firms. The hypothesis is that the supported

fums belong to targeted technology groups which will benefit more from the IT programs and are more able to exploit the iimovative opportunities related to IT than other fums in the IT in- dustry.

One can argue that the comparison between supported and

other firms in the same industry is too narrow a view and that the IT-programs have created benefits for all firms in IT-related in- dustries. As a second approach we therefore consider the per- formance of the supported industries relative to the rest of the manufacturing sector, and fmally, we also compare the perfor- mance ofthe high tech industries in Norway to their performance

in other OECD economies. The last comparison must be inter- preted with caution since the IT industry has been strongly supported in other OECD economies, as we will discuss below.

" In a previous version of this paper (Klette and Moen, 1998), we also presented an analysis based on a sample for the more narrowly defined

IT industry consisting of ISIC 3825 and 3832.

The Magnitude ofthe R&D support to the High Tech

Industries.

We define the IT or information technology industry as consisting ofthe manufacture of office machinery and communication equipment, i.e. ISIC 3825 and 3832. This is the kind of pro- duction most intensely promoted by the governmental programs

described above, and consequently the sectors where we should

expect to see the main effects. However, related sectors also received significant support, and many companies have both

production and research activities covering a broader class of

products. Due to this and due to the associated classification problems and possible spillovers between closely related pro- duction activities within companies, we have in our econometric

work decided to use R&D data aggregated to the three digit line of business level. Our sample, therefore, covers more general

high tech mdustries than IT, namely the manufacture of machinery, electrical equipment and technical instruments, i.e. ISIC 382,

383 and 385."

The R&D support most relevant for our discussion is the subsidies administered by the research coimcils and industry

funds, and this R&D support has on average been about 80

million NOKa year, having a maximum of 123 miUionNOK in 1987. Since then the support has decreased by 46 percent in nominal

terms or by 58 percent ifthe figures are deflated by the consumer

price index. In 1 995 the support was about 67 million kroner which was about 1,250 kroner per employee in the high tech industries.^" The research councils and industry funds financed about

6 percent of the total R&D investments in these industries in 1987 and about 3 percent in 1995. Including the grants awarded

directly through ministries, the shares increased to about 24 per- cent and 1 1 percent respectively.

Microeconometric evidence on Subsidized versus nonSubsidized Firms: Short and Medium run effects of

Public R&D support

It is difficuh to fmd one variable that defines the success of

a firm. We therefore study the effect of receiving public R&D

support on a variety of different performance measures. Further- more, as there is no theoretical model predicting how a particular

level of subsidy will affect these different measures, we use a simple dummy variable approach, following Irwin and Klenow

(1996). Our basic idea is to compare subsidized and non-subsidized firms to clarify whether subsidized firms on average have

^° Looking at the IT-industry in isolation, the support per employee

from the Research Council and the Industry Fund was three times larger.

73
ATP PAPERSAND PROCEEDINGS

performed better then the others. The advantage of doing this within a regression framework, is that it enables us to control for other variables that might be correlated both with performance

and with the probability of receiving a subsidy.

Based on the time series files ofthe Norwegian manufacturing statistics collected by Statistics Norway, we have constructed

eight performance measures containing information on four dif- ferent aspects of firm success. Information on R&D and R&D

subsidies is merged together from the R&D surveys conducted

by the Royal Norwegian Council for Scientific and Industrial Research (NTNF) in the years 1982-1989 and by Statistics Norway in the years 1991-1995.

The R&D subsidy dummies are based on the share of subsidies to total R&D over the three years prior to the year of observation. We do not expect a small subsidy to have much effect on

performance, and therefore we do not distinguish between zero and less than a five percent subsidy share. On the other hand, a

large subsidy might affect a firm differently than a medium subsidy, and to test this hypothesis we have one dummy indicating more than a 5 percent subsidy share and an additional dummy

indicating more than a 25 percent subsidy share.^' Using these

definitions, there are 84 1 observations with more than a 5 percent

subsidy share, and 357 of these have more than a 25 percent

subsidy share. There are 1,958 observations with positive R&D

in at least one of the three years prior to the year of observation,

and altogether our sample consists of about 6,000 plant-year

observations spanning ISIC 382, 383 and 385 in the years 1983 to 1995. The appendix gives further details on sample and variable construction.

We have regressed each performance measure on the two

subsidy dummies and all regressions include time and industry

dummies. It is possible that significant coefficients on the subsidy dummies are due to reversed causality, i.e. that successful, or possibly unsuccessfiil, firms have a better chance ofreceiving

subsidies. This can, at least partially, be controlled for by intro- ducing plant specific fixed effects, which is equivalent to measuring all variables as deviations from the firm specific means.

Unfortunately, this comes at a cost, as the downward bias on the estimated coefficients due to measurement errors, is likely to increase."

It should be emphasized that the units of observation in the regressions are manufacturing plants, while the R&D statistics

^' Firms with a subsidy share exceeding 25 percent are quite similar

to other firms with respect to size, capital intensity and profit margins.

However, they receive 70 percent of total R&D support, but only 39

percent of the R&D support from the research councils. These firms account for 33 percent of total R&D in the high tech industries we

consider.

for these plants are based on the R&D activity at the level ofthe business unit within the firm which they belong to. With plants

as units of observation we are able to keep track ofthe history of

production activities that belong to restructured firms. This is essential since several ofthe largest IT firms, e.g. Norsk Data and

Kongsberg Vapenfabrikk, were restructured within the period

covered by our sample. To keep the terminology simple we will, however, refer to R&D firms and other firms in the discussion of

our results, rather than more precise terms such as plants be- longing to R&D performing firms. We start out by analyzing the effect of subsidies on firm growth, and the results are given in the first two columns in Table

2. Table 2.A reports results from ordinary OLS regressions, while

Table 2.B reports results from regressions that incorporate plant

fixed effects. In column 1 , the growth measure is based on manhours, and m column 2 the growth measure is based on sales. No

matter which measure is used, there do not appear to be important differences between subsidized and non-subsidized firms. The point estimates are negative but statistically insignificant for firms receiving between 5 and 25 percent subsidies, and positive or close to zero (but statistically insignificant) for firms re- ceiving more than 25 percent subsidies.^^ In passing, we notice

that the resuUs in Table 2 also show that R&D firms have on

average grown more slowly than non-R&D firms, both in terms of man-hours and sales.^" The effect of subsidies on profitability are examined in col- umn 3 and 4 in Table 2. We measure profitability both as return to assets and by the profit margin. One might argue that return to assets is the more relevant measure ofthe two, but the reliability

ofthis measure is reduced by the large measurement errors associated with the capital variable. This is evident from the small Rsquare and the large root mean square error in column 3, and

there are no significant coefficients emerging from these regressions, whether estimated with or without fixed effects. Neither

does column 4 show any significant difference in the profit margins between firms with and without R&D subsidies. However,

there seems to be a general characteristic ofall R&D performing

firms that they have higher profit margins than firms without

R&D, as shown by the positive and significant coefficient for the diunmy for firms reporting R&D.

Turning to the effect of subsidies on productivity, the re- gression results are reported m columns 5 and 6. We have used

" Cf Griliches and Hausman (1986). This effect is given by the sum of the two coefficients. Testing

robustness, we have found that the results presented in Table 2 are

largely unchanged if we neglect the firms receiving large, defense related R&D contracts. ^"This is consistent with the findings reported in Klette and Forre

(1995).

74
EVALUATION OFNATIONAL PROGRAMS

both labor productivity, column 5, and total factor productivity,

column 6, as the dependent variable. Our results show that the subsidized firms have a lower level of productivity, and the dif- ferences are statistically highly significant when fixed effects are

included. The effect of subsidies on the investment intensity is re- ported in column 7 in Table 2. The investment intensity is defined

as investments in machinery and buildings relative to sales, and we consider this measure as a proxy for expected growth in sales. Furthermore, we believe that expected growth in sales is positively correlated with the success of the firm's R&D projects,

particularly after industry differences have been controlled for. Looking at column 7, we find that there are no systematic differ- ences between subsidized and non-subsidized firms in this re- spect.Private R&D expenditure could also be considered a proxy

for past R&D success, and besides this, stimulating R&D expenditure has been an explicit aim ofthe technology programs. From

column 8 we see that there are no significant differences between

the intensity of privately financed R&D in subsidized and nonsubsidized firms. In an ongoing companion study, Klette and

Moen (1997), we examine the effect ofR&D subsidies on private

R&D expenditure in more detail, applying various econometric

approaches. Preliminary results from that study confirm that subsidies do have a little effect on private R&D expenditure.

OLS estimates based on yearly data from ISIC 382, 383 and

385 in 1982-1995. The sample is moderately trimmed, cf the data

Table 2. The effect ofR&D subsidies onfirm performance

appendix. Robust standard errors in parenthesis. Time dummies

are included in all regressions. Industry dummies at the five digit

SIC level, are included in the OLS regression. The R&D subsidy

share is the sum of deflated R&D subsidies over the three years

prior to the year of observation divided by the corresponding sum oftotal R&D investments. If only one or two years prior to the year of observation is available, the subsidy share is based on this information alone. The R&D dummy is one ifthe firm has

reported R&D in one of the the three years prior to the year of

observation.

* * * Significant at the 1 % level

* * Significant at the 5% level

* Significant at the 1 0% level

Longer Run Effects

Studying the effect of R&D withm the high tech industries,

it is customary to assume a one year lag between the R&D in- vestments and the first effect on production. This is justified by

the short-term nature of much commercial R&D, but it seems

likely that the peak of the impact has more than a one year lag. For this reason we defined our subsidy dummy in the last section using a three year 'window'. However, it could be that R&D

projects supported by public agencies have a particularly long- term nature, and it has been argued that the effect of the subsi- dies given in the late 1980s has not been visible imtil lately."

Against this, one might argue that the growth experienced durGrowth in Growth in Return on Return on Labour Total factor Investment Inten. in priv. manhours sales assets sales productivity productivity Intensity fman. R&D

A: OLS estimates Dum. For R&D sub. share >0.05 -0.007 -0.021 0.049 0.033 -0.027 0.001 0.0027 0.029

(0.018) (0.030) (0.051) (0.035) (0.022) (0.010) (0.0036) (0.030) Dum. For R&D sub. share >0.25 0.044 0.083 0.049 -0.045 0.017 -0.0003 0.0025 -0.026

(0.030) (0.060) (0.094) (0.041) (0.031) (0.013) (0.0061) (0.036) Dum. For reporting R&D -0.041 -0.019 -0,11 0.024*** 0.083*** 0.061*** -0.0032

(0.011) (0.020) (0.12) (0.007) (0.012) (0.006) (0.0027)

No. of observations 5622 5622 6020 6041 6041 5874 6041 1958

R-squared 0.03 0.02 0.004 0.03 0.12 0.13 O.OI 0,11 Root mean square error 0.40 0.61 7.11 0.28 42.5 0,17 0.09 0,37

B: Fixed effects estimates Dum. For R&D sub. share >0.05 -0.019 -0.063* -0.075 0.011 -0.063*** -0,023** 0.0013 0,021

(0.020) (0.034) (0.012) (0.031) (0,022) (0,009) (0.049) (0,030) Dum. For R&D sub. share >0.25 0.018 0.094 0.017 -0.063 0.005 0,013 -0.0037 -0.049

(0.037) (0.070) (0.013) (0.067) (0,030) (0,012) (0.0071) (0,069) Dum For reporting R&D -0.023 0.011 0.035 0.023*** 0.029* 0,030*** -0.0051

(0.020) (0.034) (0.17) (0.009) (0.016) (0,008) (0.0043)

No. of observations 5622 5622 6020 6041 6041 5874 6041 1958 Root mean square error 0.40 0.61 7.11 0.28 42.5 0.17 0.09 0,37

See e.g. the front page in Computer World,No. 38, 1997.

75
ATP PAPERSAND PROCEEDINGS

ing the last years, is more likely to be an ordinary business cycle

effect than an effect of previous technology programs, as there has been strong growth in all sectors ofthe Norwegian economy.

In order to investigate this issue closer, we have compared the growth of subsidized and non-subsidized firms that existed m

1985, over the entire decade 1985 to 1995. We have defined subsidized firms as firms who had more than five percent of their R&D expenses over the years 1 985 to 1 993 financed by the gov- ernment and we have aggregated across all firms in each group.^*

The results are reported in Table 3. Once again we have used

several different performance measures, and we have deliberately chosen measures that are easy to interpret.

Looking at Table 3, we may first note that subsidized firms

have a higher R&D intensity than non-subsidized firms. This

indicates that the chance of getting R&D subsidies has been

greater for the R&D intensive firms. However, we see that the growth in private R&D investments as well as in R&D intensity

have been greater for the non-subsidized firms, and consequently

the subsidies do not seem to have stimulated R&D investments.

With respect to growth, whether in employees or sales, we see a

similar pattern as the non-subsidized firms have performed better than the subsidized ones. Looking at labor productivity, we find

that both the level and the grow1:h rate were of about the same

magnitude for the two groups. However, as the subsidized firms

started out with a higher capital intensity and had a stronger

growth in the capital intensity, they seem to have performed worse than the non-subsidized firms with respect to total factor productivity. Turning to profitability which might be considered

the most important measure, the non-subsidized firms were the most profitable both in the beginning and in the end of the period, and the subsidized firms had by 1995 not even caught up

with the 1985 level of the non-subsidized firms. On the other hand, the subsidized firms did have a stronger growth in profitability than the non-subsidized ones. Finally, looking at the exit

rate given in the last row, we see that there is no significant

difference between the two groups.

Industrial Growth

The aim ofthe technology programs have been to promote

the entire Norwegian IT industry, and in addition to R&D subsidies, relevant education and academic research have also been

supported. One way to evaluate the totality of these efforts is to compare the experience ofthe Norwegian high tech industries to

total Norwegian manufacturing and to the IT industries in other OECD countries. We have performed international comparisons

using data fi-om the OECD STAN, ANBERD and BERD data- bases.

Table 3. The aggregate developmentfor R&Dfirms established in ISIC 382, 383 or 385 not later than 1985

R&D firms with R&D subsidy share less than R&D firms with R&D subsidy share greater 5% than or equal to 5%

1985 1995 Growth 1985 1995 Growth

Private R&D investments 990 850 -14% 810 660 -18%

-average 8.8 10.5 19% 8.4 9.9 17%

R&D intensity 4.1% 4.8% 15% 8.1% 6.7% -17%

Employment 22280 14940 -33% 16480 9400 -43%

-average 199 184 -8% 172 140 -19%

Sales 14530 18080 24% 10380 12370 19%

-average 130 223 72% 108 185 71%

Labor productivity 151 253 68% 146 253 74%

Capital intensity 0.46 0.66 44% 0.61 0.97 60%

Return on assets 19.1% 24.7% 30% 12.4% 18.0% 45%

Return on sales 13.4% 13.5% 0.5% 11.9% 13.2% 11%

No. of plants 112 81 -28% 96 67 -30%

The subsidy share is the part ofthe firm 's deflated R&D investments in 1985-1993 which wasfinanced by public grants.

R&D investments are deflated by a wage index and given in millions of 1995 NOK. Sales are given in nominal millions NOK.

Labor productivty is value addedper manhour in nominal NOK. Capital intensity is assets per employee, given in nominal

millions NOK. The calculations are based on plant level data.

In an earlier version of this paper, Klette and Moen (1998), we there is some evidence that the subsidized firms have performed better

also considered the performance of the median firm in each group, and than the non-subsidized ones, but the evidence is not very strong, we examined differences in performance within the more narrowly

defined IT-industry. Considering the IT industry narrowly defined,

76
EVALUATION OFNATIONAL PROGRAMS

R&D which in 1 987 is twice as large as m other OECD countries. Despite this reservation about the OECD numbers, we believe it is interesting to compare the performance ofthe Norwegian IT industry to the IT industry in other OECD countries as we do in Figure 1 which displays the relationship between R&D

intensity and production.^' Not surprisingly, it is evident that Norway has a very small share ofthe world market. At the same

time, the R&D intensity in the Norwegian IT industry is very

high, and only Sweden had a comparable increase in the R&D

Table 4: The importance ofhigh technology and IT relative to ttal manufacturing

1983 1987 1991 1995

Norway OECD Norway OECD Norway OECD Norway OECD

Employment 19% 24% 21% 25% 20% 25% 19%

Value added 19% 22% 20% 21% 19% 22% 19%

Total R&D including R&D institutes 54% 41% 54% 43% 47% 43%

Total intramural R&D 60% 37% 54% 40% 51% 40%

Total subsidy to intramural R&D 80% 48% 85% 34% 76% 39%

Subs, as share of tot. intramural R&D 12% 1 1% 20% 10% 15% 8%

Starting out looking at Table 4, we can see that in Norway

the share of IT and general high tech in total manufacturing is smaller than the OECD average. Furthermore, from 1 983 to 1 995

,

these shares do not change significantly.^'' Despite these industries being less important in Norway than overall in the OECD,

Norway is conducting more ofits total manufacturing R&D within

these industries. The reason for this is most likely the composition of Norwegian manufacturing, its major sectors having low

R&D intensities.

ISIC 382, 383. and 385. The OECD columns give the aggregate of 13 major industrialized countries for which we have

complete data. These are Norway, Sweden, Finland, Denmark, Germany, UK, France, Italy, Spain, USA, Canada, Australia and

Japan. All variables, except subsidy as share oftotal intramural R&D, are measured in percent ofall manufacturing industries. The distribution of subsidies is given in the last two rows. In Norway, the ratio between the share of R&D subsidies received

by high tech industries and these industries' share oftotal R&D,

is higher than the OECD average. The Norwegian high tech in- dustries also have a higher share of their R&D financed by subsidies than the corresponding OECD average. The difference is most significant in 1987 when Norway laimched the National

Program for Information Technology as described above. The

Norwegian industry received about the same amount of R&D

support (in relative terms) at the beginning of the time period

studied, but by 1987 this had changed as the Norwegian IT in- dustry at that time received significantly more support than the OECD average. One should, however, notice that international comparisons of public R&D support are problematic, as it is hard

to identify with much precision how much ofe.g. defense related research that benefits the IT industry. Furthermore, in several OECD countries significant amounts ofpublic R&D support are given in terms of tax relief, and such tax allowances are not re- flected in the numbers reported in Table 4.^* In this perspective, one should not take the OECD mmibers presented in Table 4 at face value and conclude that Norway had a subsidy share in

^' Defining the manufacturing IT-industry as most ofNACE sectors 30-33, gives the same conclusion. " See Bloom, Griffith, and Van Reenen (1997) for an analysis of R&D tax subsidies in a number of OECD countries.

In Klette and Moen (1998), we also examine the differences

across OECD countries in terms of R&D, employment growth, labor

intensity. Despite the increased R&D intensity, in the years 1 988

to 1992, Norway was the only country with a fall in production.

This fall in production is obviously related to the severe reces- sion experienced in Norway during these years, but ifthe Norwegian IT industry had been internationally competitive, the condition on the domestic market should not have been too severe an

obstacle in a period of growth in the international market.^" Summary ofEconomic Results

Most coimtries support IT and related high tech industries. In Norway, the R&D subsidies were particularly large in the sec- ond half of the 1980s, both in a national and probably also in an

international perspective. In this section we have mvestigated

the effect of these subsidies, using several different approaches

and data sources. First, comparing subsidized and non-subsidized firms within the high tech industries, there is little evidence

in favor of the subsidized firms being more successfiil. Second,

looking at these industries relative to aggregate Norwegian manufacturing, their importance have not increased. Third, comparing

the development ofthe Norwegian IT industry to the IT industry

productivity and export performance for the IT-industry. Notice that Figure 1 is based on the IT-industry narrowly defined.

^" Further discussion ofthe magnitude ofthe IT program in Norway

compared to other OECD countries can be found in Buland (1996, ch. 2) and Harlem et al. (1990, ch. 2).

77
ATP PAPERSAND PROCEEDINGS

Figure 1. R&D intensity andproduction in the IT industry (ISIC 3825 and 3832). Norway compared to other OECD countries.

a

o

t33

o

50

48

46

44

42

40 -

38

Ja93 ^ US93

Ja88 US88

Ja83

Ge92 UK92 UK88,

UK83

^ ^Ne88

Ne83 Sw88

10 12

-Sw93

NORWAY92

14 16 20 R&D Intensity

Production is measured as the log ofgross output in 1985 dollars. R&D intensity is R&D investments in percent ofgross output. Source: OECD, DSTI(STANandANBERD andBERD).

of other OECD countries, the Norwegian industry does not per- form particularly well. Obviously, if someone claims that the subsidized firms and the entire Norwegian IT industry would have

performed a lot worse without the support, we cannot prove him

or her wrong. ^'Nonetheless, we believe a reasonable interpretation ofour results is that the public financial support to R&D and

innovation in the IT industry did not create a substantial stimulus to its performance, in contrast to what one would expect from

the arguments made by the promoters of the IT-programs and

from the theoretical arguments presented above.

Coordination Problems And Technology Policy In

Practice

The IT-prosrams: coordination failures at the policy

level

We have pointed out that GPT's often create co-ordination

problems that will tend to slow down the development of the GPT's and thereby the emergence of new indusfries and eco-

^' In that case, however, it would still be difficult to argue in favor of the subsidies, as the rate of return on invested capital in technology

industries has been lower than the rate of return in other manufacturing

nomic growth more generally. We have also argued that it is reasonable to interpret the Norwegian IT-programs as govern- mental efforts to overcome these coordination problems and

thereby encourage R&D, umovation and utilization ofIT-related products.

Oiu" empirical analysis of the economic performance in the firms and sectors targeted by the IT-programs revealed few results suggesting that they have benefited significantly from the

financial stimulation and the coordination effort ofthe programs. These findings lead to the conclusion stated above that the Norwegian governmental effort to stimulate and coordinate the de- velopment of IT-products and applications have not been very

successful. We are, however, not the first evaluation study to recognize the failure ofthe coordination activities in the IT-programs; this aspect has been emphasized in all previous evaluation reports. A report evaluating the part ofthe National Program

for Information Technology organized by the Industry Fund,

concluded that they found few concrete results with respect to the creation of 'strategic alliances' or 'coordinated groups' which was an explicit and major objective ofthis part ofthe program.^^

industries, according to the Federation of Norwegian Engineering

Industries (1998).

32 See Hervik and Guvag ( 1 989).

78
EVALUATION OFNATIONAL PROGRAMS

In the overall evaluation a year later, Harlem et al. (1990) concluded that the plan has undoubtedly failed in improving coordination and integration of policy towards information tech- nology."" The difficulties involved in implementing coordinating activities could clearly be recognized during the operation of

the program as the committee heading the implementation was

entirely reorganized twice during the program's four years of

existence. The reorganization ofthe heading committees was to a

large extent due to dissatisfaction in the Ministry of Industry

with the way the various activities were organized and the lack of

broader co-ordination, as described in Harlem et al. (1990), ch.

Two years later, in the government's report to the Parliament on the research activity in the Norwegian economy, reference was made to this negative conclusion by Harlem et al. (1990) and

the report elaborated on it." The main conclusion is that [the research programs including the research activities within the National Program for Information Technology] did not lead to the intended coordination for the programs as a whole, not in the

relationship between the government agencies and the private

agents, nor between the various government agencies." Further- more, the research programs have not been successful as policy

tools, neither with respect to organization, planning or information. Research activities have to a large extent remained as fragmented as before the programs were implemented." These conclusions were based on an assessment of 9 research programs,

including research programs on biotechnology, offshore and other

activities, in addition to information technology which was by

far the largest among them.

Given these clearly recognized problems with the coordination efforts up to 1992, it is a bit depressing to read the main

conclusions of the report on the evaluation ofthe 'National Plan

for Improved Utilization of Information Technology in the Norwegian Industry 1992-95' presented in Olsen et al. (1997):^*

[The plan] never became an instrument for coordination ofgovernmental institutions and means.... The plan never managed to mobilize any strategic use of other resource and means present in governmental institutions... To explain this poor coordinating performance,

several factors ought to be mentioned. First, it appears

as very unclear exactly what the plan was going to co-

" P. 233, our translation. We recognize that the focus on coordination

failures in this and other evaluation reports often refers to problems in coordinating institutional arrangements rather than the projects directly. However, it seems likely that poor coordination at an institutional level

will show up as poor coordination also at a project level and our empirical

findings confirm this expectation by showing that the coordination at the

project level was not very successful. ^ See also Buland, especially chs. 9 and 10.

ordinate, and why coordination was important. Second,

institutional resistance ... never produced a clunate con- ducive for cooperation and coordination among the rel- evant institutions. The explanatory factors emphasized in this quote from Olsen

et al. (1997) deserve further attention and we will return to them

below. First, we want to point out that the two important questions of what the plans were supposed to coordinate, and why

coordination was important, were only considered in general and

superficial terms in the evaluation reports. The evaluation re- ports unanimously complain about poor coordination, but there

is a striking omission of analysis at a practical level of what the plans were supposed to coordinate, and why. For instance, none

ofthe reports identified or examined concrete examples ofopportunities for beneficial coordination that were missed. One inter- pretation of this omission is that a carefiil discussion of such

specific opportunities would require a lot of detailed information

and therefore would be too difficuh or time consuming, even

with the benefits of hindsight. The amount of mformation re- quired to identify coordination opportunities is the issue that we

want to consider next. Two Pessimistic And One Optimistic View OfCoordination Problems

Coordination Beyond Stylized Models

Above we have tried to link the IT-programs to recent theoretical work on mnovative complementarities, GPT's and coordi- nation problems in order to identify more clearly the basic principles. However, understanding the basic principles ofcoordination problems does not take one very far in the direction of useful, practical conclusions about how to construct technology

policy. Understanding the basic problems, one is lead to a new

but not sunpler set ofquestions: What activities in what firms are complementary and need to be coordinated, and in what way?

An appropriate choice of policy tools requb^es a detailed understanding ofthe externalities and the iimovative complementarities

mvolved, as well as the nature of the firms' behavior and constraints.

" Cf Ministry of Church, Education and Research (1 992), p. 92-94, Cf Olsen et al. (1 997), p.vii. One should keep in mind that when

the Norwegian research councils were completely reorganized in 1992 by the establishment of the Norwegian Research Council, it was largely based on the hope that this should promote coordination of related but poorly coordinated activities that previously had been organized by

different research councils.

79
ATP PAPERSAND PROCEEDINGS

Matsuyama ( 1 997) and others have emphasized that the in- formational requirements at a practical level raises serious questions about the possibilities for government policy to correct coordinating problems in the real world. Matsuyama argues that coordination problems are pervasive phenomena and he emphasizes that economists' illustration of coordination problems by

means of simplistic game theoretic models are useful to illustrate co-ordination problems as a possibility. But such game theoretic models tend to trivialize the coordination difficulties that face policy makers; in real coordination problems, the nature of 'the game', the pay-off structure, the identity ofthe players and even

their number are often unknown to the policy makers. Further- more, the nature ofthe game can change rapidly and dramatically

due to outside influences. These problems might be particularly relevant in a rapidly developing technological field such as infor- mation technology and in a small open economy such as the Norwegian.

Consider as an example the case of Norsk Data which was

one of the largest, and no doubt the leading manufacturing firm

in the Norwegian IT-industry in the 1980s. Norsk Data's production of minicomputers with its integrated software was highly

successful until the mid 1980s and it was recognized as the fast- est growing and third most profitable computer firm in the world

in 1 986." However, the situation was entirely different two years

later when it became clear that so-called open standards, in particular the UNIX operating system, eliminated the need for tight

integration between production of the computer hardware and

the software. Norsk Data was running large deficits at the end of

the decade and heading fast towards bankruptcy. It was finally

dissolved and partly sold to the German firm Siemens/Nixdorf in 199 1 . As mentioned above, Norsk Data was the largest recipient

of project support within the National Program for Information

Technology, something which perhaps illustrates the information problem emphasized by Matsuyama (1997).

Institutional Inertia as a Barrier to Coordination

Bresnahan and Trajtenberg ( 1 995) have made a related point

in their analysis of co-ordination problems associated with general purpose technologies. They argue that the institutions de- signed to correct the coordmation problems display much more

inertia than the leading technologies. When a GPT era approaches

its end and a new GPT emerges, the old institutions will resist change and the economy might 'get stuck' with the wrong insti- tutions, namely those that have been designed to solve the coordination problems associated with the previous GPT.

This argument is consistent with what Olsen et al. (1997)

noted, that "institutional resistance never produced a climate

3' See Steine(1992), p. 11

conducive for cooperation and co-ordination among the relevant

institutions" within the 'National Plan for Improved Utilization of

Information Technology in the Norwegian Industry 1992-95'. Institutional resistance and inertia was also a basic problem in the implementation ofthe National Program for Information Technology and an important reason why the heading committee of the program was reorganized twice during the four years it lasted. The previously mentioned report to the Parliament discussing research programs more generally,'* suggests that the problem

of sluggish institutional changes in new technological and sci- entific fields have been quite pervasive. The problems and dis- cussions leading up to the recent establishment of the Norwegian Research Council underscores this point, cf footnote 35. In other terms, even though coordination problems suggest

that Pareto improvements are possible, widespread institutional resistance show that policy reforms create 'winners', but also

'losers' which, although they could be compensated in principle,

makes it difficult to implement desirable policy changes even when we disregard the information problem discussed above.

Coordination through the Market: the Optimistic View

Coordination problems illustrated by game theoretic analysis are based on non-cooperative behavior as an assumption. However, it is not obvious that firms in the same industry or firms that are vertically related are unable to implement cooperative solutions through negotiations and contractual relationships. This view has

been most forcefully stated in the classical paper by Coase ( 1 960),

where he claimed that coordination problems associated with

complementary activities often will be solved through such market

mechanisms. This optimistic view appears to be orthogonal to Matsuyama (1997) and the cited argument in Bresnahan and

Trajtenberg (1995), but it leads to a similar conclusion about the

limited role for governments to act as a coordinator. Coase has

argued that the market mechanism will tend to incorporate or compensate for external effects if transaction costs are not high.'' His

point is that, in the presence of positive extemal effects, there are strong incentives to sign a contract or organize a compensation

arrangement between e.g. a firm receiving a positive extemal effect and a firm providing the source of this effect. Coase also argued

that economists tend to ignore such options for compensation

through the market. A rhetorical remark by Matsuyama (1997) ech- oes this argument: "Ifthe coordination problem were simple enough

for even the outsider, such as the economists or the bureaucrat, to know how to solve it, it would have been taken care of a long time

ago by those directly involved with the problem."

The ability ofthe market itselfto facilitate coordination, has

" Cf. Ministry of Church, Education and Research (1992).

'' See Coase (1988) where he has elaborated on this argument.

80
EVALUATION OFNATIONAL PROGRAMS

to a large extent been ignored in economic studies of technical change and in recent research on 'new' growth theory.'^ However, when we examine the Norwegian IT-industry, it is clear that the firms are involved in a large set ofcoordinating arrangements

organized through contracts and other private institutions. According to Aakvaag et al. (1 996), about 60 percent ofthe Norwegian electronics firms report that they participate in technological cooperation schemes. Partner firms often have a partly inte- grated ownership structure, which is one important market ar- rangement to internalize this type of externalities. A different example of coordination through private institutions is given by

Sterne (1992), who argues that an important contribution to the

early success ofNorsk Data was its close contact with demanding customers. Norsk Data organized a formal user group in order

to coordinate the development of their minicomputers and soft- ware with organizational and other irmovations developed by its customers. Similar user groups and other coordinating relation- ships are well known throughout the computer industry. Formal

contracts coordinating the development of new technologies in the primary iimovating firm and 'partner' firms using the new

technology are regularly announced in the business press. To

take a very recent case, the Norwegian electronics company MRT

Micro, which has developed PC-cards to digitalize pictures, has

just armounced that they have signed collaboration contracts with four firms using these PC-cards."" These four firms are quite

different; one is e.g. making identification system for the police

and defense, while another is making measurement instruments

for opticians and eye-doctors.

Industry associations are another set of private institutions which are important in facilitating co-ordination of innovative

activities.''^ In a theoretical study, Romer (1993) has examined new institutional arrangements to improve the co-ordinating function of such organizations. However, it must be left for future research to examine the empirical performance of such organizations in co-ordinating R&D activities and privately funded re- search joint ventures more generally. Our point here is only to

illustrate the widespread co-ordination of complementary iimovative activities across independent firms through contracts and

other private institutions.

Conclusions

The motivation for the IT programs in Norway in the 1 980s

and 1990s seem to a large extent to accord well with the coordination problems identified in the new growth theory and especially the recent theory on 'General Purpose Technologies' introduced

by Bresnahan and Trajtenberg (1995). Having studied the Norwegian IT industry, we have no reason to doubt that irmovative

complementarities associated with such technologies can be

pervasive phenomena, and that these complementarities create a nimiber of coordination problems. A major question we have

addressed in this study is to what extent the considerable public

funds spent on coordinating and promoting the R&D activities in the Norwegian IT industry have been successful in overcoming such coordination problems and stimulated the performance

of this industry and closely related industries. Our findings suggest that the results have been very modest and that the IT programs were largely unsuccessfiil.''^ Why did not these technology programs succeed, despite

their appeal ex ante and according to economic theory? In contrast to the situation with illustrative and simplistic game theoretic models, in real coordination problems, information is a seri- ous obstacle; what is the nature of the game, which players are involved, what do the pay-off structure look like and how rapidly

is it likely to change? Or in less formal terms; exactly which firms and what activities should be coordinated and in what way?

These serious questions are very hard to answer in a rapidly

developing field such as information technology and might be

particularly hard to solve in a small open economy where a large

majority of the innovations take place abroad. We believe that

industrial irmovation is an activity where co-ordination problems and 'market failure' often are pervasive, but it is probably

also an activity where policy makers and bureaucrats often lack

the information needed to improve on the market solution. The coordination problems created by complementary innovative activities across different firms seem in many cases to be

at least partly resolved by private institutions such as industry

associations, privately funded research joint ventures and other cooperative research agreements. In future research it could be

*° See, however, the recent literature on research joint ventures, e.g. Kamien, Muiler, and Zang (1992). DagensNaeringsliv, 13.1 1.97, p.8.

*^ The industry association for IT firms in Norway (ITF) reports

a large number ofcoordinated research projects and research joint ventures

in its annual report (The IT-Industry's Association, 1996).

"•^ Wicken (1994, pp. 271-2), summarizing a number of studies on

the history ofNorwegian technology policy from World War II onwards,

draw a similar conclusion.

81
ATP PAPERSAND PROCEEDINGS

interesting to examine more directly the role ofsuch cooperative

activities.*"

Data Appendix

Our econometric analysis uses merged data from R&D sur- veys and time series files of the manufacturing statistics. The

manufacturing statistics of Statistics Norway is an annual census of all plants in the Norwegian manufacturing industry. From

this source we use information on output and other inputs than

R&D. We have only used plants with more than five employees,

as there is limited information on the smaller ones. See Halvorsen,

Jensen, and Foyn (1991) for documentation. For reasons given in section 4.2, we have aggregated the R&D expenditures to the

three digit (ISIC) line of busmess level before merging these

variables to the manufacturing statistics. If a firm has several

plants with the same three digit ISIC classification, the R&D

expenditures are distributed according to sales before fiirther aggregation to the industry level. Note, however, that 64 percent

ofthe plants with a positive R&D variable are single plant firms. R&D surveys are available for the years 1982-85, 1987, 1989,

1 99 1, 1 993 and 1 995 . These surveys were carried out by the Royal

Norwegian Council for Scientific and Industrial Research (NTNF)

imtil 1989 and by Statistics Norway from 1991. See Skorge, Foyn,

and frengen ( 1 996) for definitions and industry level figures. Since

the surveys had a broad coverage in the industries studied, we

believe the totals given by that figure are close to the correct numbers. The merged data set used in the economefric analysis

includes fewer R&D units due to matchmg problems.

The international comparisons of aggregate industry per- formance are based on the STAN, ANBERD and BERD data- bases, prepared by the OECD. At the core of our analysis is the AN-BERD (Analytical Business Enterprise R&D) database, which

contains information on business enterprise R&D defmed in a

consistent way across the main OECD countries. The BERD da- tabase includes information about business enterprise R&D fi- nanced by the government through research contracts and di- rect grants. The STAN (Structural Analysis) industrial database

contains internationally comparable information on input, output, exports, investments and value added in fixed and nominal

prices by countries and sectors.

"" Dixit and Olson (1997) have recently studied some difficulties in getting economic agents to participate in bargaining and negotiations

leading up to cooperative solutions.

82
Are Swiss Gk)vernment Programmes of Promotion of Advanced Manufacturing Technologies (AMT)

Effective? An Economic Analysis Based on Micro-level Survey Data

By Spyros Arvanitis, Heinz Hollenstein and Stephan Lenz'

Summary

This paper describes the approach applied and the results of an evaluation of the effects of the Swiss programme to pro- mote the diffusion of Advanced Manufacturing Technologies

(AMT) in terms of velocity, broadness and intensity of AMT

adoption in the period 1990-1996. This evaluation study was

undertaken on behalf of the Swiss Ministry of Economic Affairs. The analysis is based on (mainly qualitative) firm data col- lected in the course of the Swiss Innovation Survey 1996. The

available information refers to the time profile of the introduction of nineteen AMT elements covering all relevant fields of manufacturing activity (design, production planning, production tech- niques, handling, quality control, conamunication/control) and

includes also a whole series of factors which presumably deter- mine the firms' adoption behaviour. The final data set contains

667 firms representing all manufacturing industries as well as firm

size classes. About 80% of them were using AMT in 1996 (with a median AMT-intensity of seven technology elements), and 20%

of the users have been promoted by some component of the governmental support programme (information/training, consultation as well as R&D-projects). However, our data suffer from an

important shortcoming: the time dimension is accounted for only

for the adoption variables because we could not collect timeindexed data for the explanatory variable (reflecting the limits of

retrospective data collection).

The method used is based on estimating an adoption equation which, in addition to the main explanatory variables as pro- posed by the theory of technology diffusion, takes account of

variables depicting policy intervention (promotion yes/no, type

and intensity of promotion). For the econometric estimation we

used probit and ordered probit models. Special attention was

given to the size dependence of policy effectiveness by estimat- ing the adoption equation separately for two different firm size

classes. The most delicate problem to be solved at this first stage

of analysis has been the construction of an adoption variable

specific to the time profile of the promotion programme. We constructed several variables, but we report here mainly on a vari- able measuring the change of adoption intensity between 1990

and 1996. In a next step we also estimated a policy equation with a set of firm characteristics as regressors. As a result of the

"single equations approach" (i.e. separate estimation of the adoption and the policy equation) we found that promotion - primarily

through information/training as well as consultancy - did lead to a faster, broader and more intensive adoption of AMT especially

for small firms (less than 200 employees). Even if we succeeded

in extracting a significant positive impact of the policy variable(s) there still remained the problem of causality between adoption

and promotion to be ultimately resolved only in a simultaneous

equation framework. Our main finding of the single equation ap- proach was confirmed by estimating simultaneously the adoption and the policy equation (simultaneous probit model).

Inb'oduction

The main feature of Swiss technology policy is the low

weight it places on direct measures for fostering innovation in the economy. It is primarily oriented towards creating a favourable environment for the introduction of new products and production techniques, whether such innovations rest on firm-internal research and development or on the adoption of novelties generated by other firms or institutions. This framework-oriented policy is supplemented by a number of specific measures to encourage, first, cooperation between

public research institutions and private enterprise through joint

projects either by a bottom-up approach (without restrictions with regard to the field of technology) or by the top-down support of "oriented research in the pre-competitive stage" in specific technology areas such as optoelectronics, material sciences,

etc. Second, there are some policy initiatives to support rapid

diffusion of selected basic technologies which are considered to be relevant for a broad spectrum of industrial activities. To be

mentioned are primarily two "programmes of action" in the field of "Advanced Manufacturing Technologies" (AMT) and "Micro electronics" respectively, which offer to firms information

and training services as well as subsidies for consultancy and

development projects; the latter are in most cases based on joint ventures between firms or between firms and research institu- tions embedded in regional networks. The working of these programmes is restricted to a couple of years, i.e. the policy concept is to give an initial support for technological adjustments

which, as in the case of AMT or microelectronics, pose a chal- lenge for many firms and/or industries. The policy measures rest on the hypothesis of the existence of significant bottlenecks in an early phase of diffusion; secondly, and in a more general

perspective, it is assumed that a strong linking of a firm's knowledge base with external know-how is a key factor to secure a high

' Spyros Arvanitis, Heinz Hollenstein and Stephan Lenz are with KOF/ETH, Institute for Business Cycle Research, Swiss Federal Institute of Technology, Zurich.
ATP PAPERSANDPROCEEDINGS

innovation performance, a proposition supported by empirical

work with data for Swiss manufacturing firms (see Arvanitis and

Hollenstein, 1994, 1996, 1998a). In addition, these programmes

are based on an approach which, in a balanced way, takes ac- count ofhuman, technical and organizational factors and restric- tions (in Switzerland labelled as MTO approach "MenschTechnik-Organisation").

In this paper we address the problem of evaluation of tech- nology policy with regard to the Swiss program ofpromoting the diffusion ofAMT which has been initiated in 1990 and ended in 1996. In section 2 we briefly characterize our approach and point

to some differences to other methods of evaluation; we also introduce the model of technology diffusion we used in this investigation. In section 3 we describe shortly our data base and

the econometric methods appUed in the empirical part ofthe work.

In section 4 the specification of the adoption and policy equation respectively are presented in some detail. Section 5 contains

the empirical results. We conclude by an assessment ofthe pro- posed procedure and some recommendations for evaluating specific policy measures.

Theoretical Background

Evaluation Concept

The majority of evaluations of public support for technol- ogy diffusion, not only in Switzerland, is primarily directed to the efficiency of such programmes in a rather narrow sense. The

topics covered by such evaluations are typically the following:

Is the target group well informed on the support measures avail- able? Do firms to which policy is targeted take part to a satisfac- tory degree? Are management and procedures of a programme

efficient? In what way do firms decide whether they should commit themselves, and what may be concluded fi"om this regarding

the effectiveness of the programme? What are the motives for participating? Are barriers of diffusion as perceived by firms

sufficiently addressed by the policy measures? This type of

evaluation, though useful, does not yield an assessment of the economic effects ofdiffusion-oriented measures, because it concentrates on 'programme-immanent' performance measures and

does not take account offirms not participating (no control group

^ Typical examples for Switzerland are Freiburghaus et al. (1990) who evaluated the promotion of joint research of universities and firms, and Balthasar et al. (1997) who assessed the effectiveness of

the Swiss participation in technology programmes ofthe EU; for Austria

see, for example, Polt et al. (1994) who assessed the public support measures for the introduction of AMT. A recent report by OECD

(1995) with respect to the diffusion of information technologies in SMEs, drawing on a number of country studies, is also primarily based on this type of evaluation, which does not sufficiently take account of economic performance measures or changes oftechnology adoption (an exception, to some extent, is the country study „Canada"). See OECD

analysis).^ The same type of critique is put forward by Stoneman

and Diederen (1994)^ (see also Karshenas and Stoneman 1995). In contrast, the present mvestigation is focussed on the

results ofpoUcy intervention. More specifically, it is asked whether

the primary goal ofpublic support, i.e. a more rapid and broader

diffusion ofAMT compared to firms standing aside (i.e. the control group), is actually attained. In addition, we try to identify the

relative effectiveness of the main elements of the support

programme, i.e. information/training, consultancy projects as well

as R&D projects. The analysis concentrates on a single policy

programme without taking account ofthe role it plays within the national system of innovation, although the results of a specific measure presumably are mfluenced by such 'systemic conditions'. It does also not (directly) contribute to 'policy

benchmarkmg' because the necessary conditions for such an

exercise, i.e. the application of comparable methods to similar policy measures in several countries, are not (yet) given.

The envisaged type of evaluation requu-es an econometric

analysis based on the theory of technology diffusion usmg micro-level firm data for a (representative) sample which contams

subsidized firms as well as not supported ones. More specifically, we estunate equations of technology adoption based on

general factors determming the use ofnew production techniques

as well as pohcy variables to identify the marginal effect ofpoUcy

mtervention.

Formally, the evaluation approach may be summarized by

(1) A = a+ aX + aP + aA*

0 1 2 3 with A standing for the adoption of AMT, X as a vector ofvari- ables determinmg technology adoption, P as a vector of policy mstruments and A* representmg the mtensity of AMT use m

the year the supporting measures became effective. A difficult problem in any evaluation ofpolicy effectiveness

is the identification of the dhection of causality. Is it promotion

that leads to a more rapid technology diffusion, or is the opposite true, i.e. are the firms which are taking up the new technology

in an early stage anyway those lookmg successfiilly for public

support? In a cross-section analysis there is an endogeneity

problem unless the direction ofcausality is fixed a priori."* Even in

(1997) for an assessment ofthe state ofthe art in technology evaluation. 'Stoneman and van Diederen (1994, 928): „The DTI in the United

Kingdom, for example, legitimates its diffusion policy with reference to market failure but evaluates its diffusion programmes predominantly in terms of the efficiency of their management, the accuracy oftargeting, the appropriateness oftheir tool mix and the appreciation ofthe recipients of information."

•This problem could be circumvented if we were able to introduce

for P a lead in relation to A. However, this procedure is not feasible because we do not dispose of time-indexed information for the policy

variables.

84
EVALUATION OFNATIONAL PROGRAMS

the case of a longitudinal analysis the identification of the causality might be difficult because of the known problem of 'post

hoc ergo propter hoc'

.

In this perspective, we intend to tackle the problem, though

not to solve it, by estimating two equations, one for technology

adoption and a second one for government promotion, (i) separately and (ii) simultaneously (simultaneous probit model) and

then compare the results for both procedures. Formally spoken:

(2a) A = a+ aX + aP + aA*

0 1 2 3

(2b) P = b +b Y + b A

0 1 2 (vector Y stands for factors which determine whether a firm gets

government support).

An Eclectic Model ofTechnology Diffusion

The main element of our procedure is the estimation of an

equation explaining the adoption of AMT based on a set of

"general" factors determining the adoption decision and suit- able measures ofpolicy intervention. Given the state ofthe art of

theoretical and empirical research of technology diffusion, the

theoretical approach used in deriving the explanatory variables

is eclectic in nature. It is based on the general framework pro- posed by Karshenas and Stoneman (1995).' According to these

authors the main (neo-classical) models of diffusion may be cat- egorized in four groups (epidemic, rank, stock and order model)

which can be fused to a "higher order model" containing the

specific features ofthe different approaches. Economically spoken, the general framework is given by factors reflecting antici- pated benefits and costs of the adoption of a new technology

which are taken into consideration by the adopting firm when

assessing the profitability of an eventual technology adoption.

Within this general framework we distinguish a whole series of explanatory variables (vector X in the equations above) cap- turing both benefit and cost factors (see Arvanitis and

Hollenstein 1998b). These variables refer to:

• the technology market: transparency; uncertainty as to the effectiveness of the technology, information costs; general range ofappUcation (technological opportunities);

market concentration; (expected) price ofthe technology;

• the product market: demand perspectives, intensity of

price and non-price competition, market concentration;

• the factor markets: availability of qualifications impor-

' See Sarkar (1998) for a survey on the theoretical development in

this field.

' See Ewers et al. (1990) and Fritsch (1991) for an example for this type of approach.

tant for technology adoption; internal and external fi- nancial restrictions;

• adoption-specific firm characteristics: know-how in the

field ofAMT; firm-specific potential for the application

of this technology depending on the type of products

and production processes; compatibility of AMT with

the existing physical, human and knowledge capital as well as the organizational and managerial structures in- ducing AMT-specific adjustment costs. Some of these variables cannot be directly measured and

should be considered as latent variables. However, survey infor- mation related to specific 'bottlenecks/obstacles to adopt AMT,'

the 'objectives pursued by introducing such technologies,' the

'factors enforcing the adoption of AMT' and information refer- ring to a firm's characteristics in general as well as to its innovative activities may reflect such 'latent variables' to a sufficient degree and could be used as proxies.^

Data and Method

The analysis is based on firm data collected in the course of

the Swiss Innovation Survey 1996 as a supplement to the stan- dard questionnaire, thus allowing the combination of AMT specific information with basic data on innovation and technology

use. The available variables are to a large extent qualitative in nature, i.e. categorical or ordinal measures. As far as AMT is concerned the questionnaire yields data on the time profile ofthe

introduction ofnineteen AMT-elements, the linking ofthese tech- nologies, the assessment of a whole series of objectives pursued

by introducing AMT as well as the significance of factors impeding its application, the impact of AMT on competitiveness, employment requu-ements and organizational structure and, fmally,

information on government promotion ofAMT.'' The survey has been addressed to manufacturing firms based on a sample (mnovation panel) stratified by industry (17 2-digit

industries) and fmn size (3 industry-specific size classes) with

full coverage of the upper size class in each industry. Additionally, the questionnaire was sent out to fums not contained in this panel which have been involved in the programme of AMT pro- motion. The response rate has been about 34% with a somewhat

higher percentage for non-panel firms. 80% of the respondents

used at least one AMT element in 1996 with a median AMTintensity of seven technologies and 20% have got promoted by

one or more element(s) of the government support programme.

^ The questionnaire is available in German, French and Italian and

can be handed out at request.

85
ATP PAPERSAND PROCEEDINGS

The final data set used in the econometric estimations contains

463 firms fairly representative for all manufacturing industries as well as firm size classes in the original sample (see table A. 1 in the

annex).^

Table 1 contains some information on the adoption rates of

19 elements ofAMT in Swiss manufacturing since 1993 (including planned application of these technologies till 1999). The de- gree of diffusion (percentage of firms using a certain technol- ogy) in 1999 and the diffusion velocity (increase ofthe percent- age firms using a certain technology element in the period 1993-

1 999) varies quite strongly among the technology elements listed

in table 1 . For example, the degree of diffusion ofCNC machines

being akeady an 'old' technology was quite high and changes

only slightly between 1993 and 1999. On the other hand, 'new'

technologies such as 'simulation' and 'rapid prototyping' or FMS

are used only by a small share of firms but this share is going to double until 1999. Owing to limited resources we could not perform a nonresponse analysis, so we cannot exclude that some kind of selectivity bias as to the adoption behaviour of the responding firms

may exist in our data. However, we minimize the risk of being

confronted with this kind ofbias by building our empirical analysis not on the (presumably biased) information on adopting AMT

in general, but on the specific use of some AMT elements for firms already being an adopter of AMT (see subsection 4.1). Further, the estimates ofprogramme impact may be biased due to

self selection (see e.g. Maddala 1983). Self selection arises when

firms are not randomly assigned to policy-promoted and nonpolicy-promoted (control) groups. However, the sknultaneous

consideration of a policy equation permits us to also address

this kind of econometric problem. For the estimation of the single equations we used a probit

(for dichotomous dependent variables) or an ordered probit model

(for ordinally scaled dependent variables with more than two

measurement levels). A two-step procedure (first, estimation of

the coefficients ofthe reduced form ofthe original simultaneous

equation system, then estimation of the structural parameters

based on the covariance matrix ofthe reduced form coefficients) was applied to estimate the simultaneous probit model according

to the algorithm implemented in the MECOSA software devel- oped by Schepers and Arminger ( 1 992).

Specification ofthe Empirical Model.

Adoption Equation.

Dependent Variable

Throughout the paper we use an ordinal measure of the change ofthe AMT intensity between 1990 and 1996 (DAMTINT)

as adoption variable. Thus, we implicitly presume that govern- ment promotion becomes effective within 1 year. The AMT in- tensity is measured as the number of technology elements (out

of a list of 19 such elements in table 1) akeady m use in a firm m

a certain year. We constructed a three-level ordinal variable which

contains the following categories: no change or change by 1 AMT element (29.9% of the firms which had adopted at least 1 AMP element till 1996); change by 2 to 4 elements (45.2% ofthe adopting firms); change by more than 4 elements (24.9% of the adopting firms). By defining the adoption variable as a measure

of intensity change we avoid to some extent identification problems which arise, first, because we do not know which specific

technology is supported in every single case of promotion and,

secondly, we do not have data for the explanatory variables which

are differentiated by technology and time.

Explanatory Variables

We distinguish six groups ofexplanatory variables (besides

two adoption-specific control variables and industry dummies;

see table 2). A first group of determining factors refers to objectives of and motives for the adoption ofAMT which we interpret as proxies for anticipated revenue increases due to the use of new technology. Such an interpretation can bejustified on ground

of some evidence on the degree ofthe attainment ofthe pursued

firm objectives related to AMT adoption (measured on a five point Likert scale): about 48% ofthe firms having adopted AMT

reported a very high degree of attainment of their objectives; some 40% of them reported a middle degree of attainment and

only 12% of the AMT adoptors answered that their degree of

goal attainment had been very low.' The six metric variables listed in table 1 under the heading

'objectives/motives' are the factor scores resulting from a principal component factor analysis of 26 single objectives of AMT

' In addition to the data collected by this survey we dispose of some information made available by the institutions responsible for the implementation of the programme, that is, in the first place, seven

regional AMT centres set up in 1990. These data refer primarily to some quantitative measures indicating the intensity ofpromotion (number

of projects, amount involved, etc) but without differentiation as to the

type of support. However, the quality of the information does not come up to the requirements ofan evaluation exercise and is not presented here (see Arvanitis et al. 1998, ch. 6).

' Small firms show in general a higher degree of goal attainment than large ones (see Arvanitis and HoUenstein 1 998b).

86
EVALUATION OFNATIONAL PROGRAMS

included in the questionnaire (see Arvanitis and Hollenstein 1 998b

for details). Two ofthese factors are related primarily to product

improvements (DEV, QUAL), two further ones seem to be connected more closely to changes in the production technology

(COST, FLEX). A fifth factor, TOP, refers generally to anticipated revenue potentials due to technology lead. For all five variables we expect to find a positive influence on the adoption variable. The sixth variable, FINCOMP, has a somewhat different character and cannot be linked directly to revenue increases. It contains elements of a 'defensive' adoption strategy (favourable

financing conditions; competition pressure). It is not a priori clear whether these factors lead to an early or late technology

adoption; thus, the sign of the coefficient of this variable is undetermined.

A second group of variables is related to factors impeding

the AMT adoption. The six metric variables TECH, KNOWPERS,

RESIST, INVCOST, UTILIZ and COMPAT, which were also constructed by a principal component factor analysis of 26 single

adoption impediments, can be interpreted as proxies for several types of adjustment costs caused by an introduction of ATP

which may lead to a late technology adoption. Thus, we expect a

negative sign for these variables. A third category of explanatory variables represents the market conditions under which the firms are operating: medium

term expected change of demand (D; measured on a five point

Likert scale for the period 1993-1999);'° intensity of price and

non-price competition (IPC and INPC; measured on a five point

Likert scale); product market concentration (measured by three dummies for firms confronted with markets with varying numbers ofcompetitors). Favourable demand conditions leading to a quicker pay back of investment in AMT may have a positive influence on the propensity to adopt; thus, we expect a positive

sign for this variable. The same holds for the intensity of nonprice competition being closely related to demand shifts. Not

clear is the direction ofthe effect ofthe intensity ofprice competition: the introduction of cost-reducing AMT may be favoured

by strong price pressure; on the other hand a profit squeeze

caused by sharp price competition may severely limit the invest- ment options of the firm. The sign of the effect of market struc- ture on adoption is also not quite clear. Hypotheses for both a

positive (Schumpeter effect) and a negative effect (fi-ee competition effect) can be found in literature, so the problem has to be

resolved at the empirical level. A fiirther group ofvariables serves to characterize the products and the production technology ofthe firms. The main idea is that the product type and the existing production technology of

It is obvious that this variable is irrelevant for AMT adoption

before 1993 unless it can be shown that differences in the demand

potential among firms are persistent.

a firm play an important role in identifying the extent and the

limits ofAMTuse in this firm (see Ewers et al. 1 990). For example, we expect that the benefits for applying AMT in producing customer-specific (PDUSER) or otherwise highly differentiated products (PDMARKT) would be rather small, so that a negative sign

for this variable is quite probable. On the other hand, we postulate a positive influence on the adoption variable for large-batch

(LBATCH) as well as for mass or continuous flow production

(CONTFLOW), where scale effects are possible and AMT can

unfold its fiill potential; analogously, we expect no effect for small-batch production (SBATCH). However, the use ofsophisticated AMT permits not only the realization of scale economies

but also an increase of production flexibility, thus enabling the

firm to adapt quickly and efficiently to extemal or internal changes.

This second effect works in the opposite direction to the first one, in this way weakening somewhat the overall influence of

product and process technology on adoption behaviour. All these

variables are (0, 1 )-measures.

Table 1: Use ofAdvanced Manufacturing Technologies

(AMT) in Swiss Manufacturing (Percentage of responding

firms; 1999: planned application)

Technology Element 1993 1996 1999

Design Computer-aided design and/or engineering (CAD/CAE) 506 57.0 60 8 Computer-aided design/manufacturing (CAD/CAM) 344 42.5 47.8

Simulation, rapid prototyping 5.0 7.7 11.4

Planning

Digital firm data representation 40.1 48 7 64 2

Computer-aided (manufacturing) planning (CAP) 40 8 50.7 62.9

Production Computer numerical controlled machines (CNC/CND) 48 9 52.8 55.0

Materials working lasers 8.8 107 13.1

Pick-and-place robots 19.9 23.2 26.7 Complex robots 8.1 11.8 13 4

Flexible manufacturing cells (FMC) 10.1 13 8 20.0

Flexible manufacturing systems (FMS) 8.6 13.1 18.8

Handling

Automatic storage and retrieval systems 25.0 32.2 41.5

Transport systems 10.8 13 2 16 5

Quality control

Quality control (CAQ) on material 22.4 28.7 41.2

Quality control (CAQ) on final products 21.9 300 42 8

Communication

Local area network for technical data (LAN-T) 25.9 362 47.1

Local area network for factory use (LAN-F) 22.2 318 44.1

Production planning systems (PPS) 41 9 54.8 67.3

Interfirm networks 6.6 15 8 37.3

87
A TP PAPERSAND PROCEEDINGS

A firm's ability to absorb knowledge from external sources and exploit it for its own innovative activities is a major determi- nant of innovation performance (see e.g. Cohen and Levinthal

1989). We use two variables to proxy the capacity to absorb

external knowledge, first the share ofemployees with high vocational qualification (HUMCAP) and second the bmary variable 'R&D cooperation activities yes/no' (COOP). The latter is a measure of a firm's embedding in existing knowledge networks,

the first one reflects the overall ability to assess technological

opportunities; both seem to enhance considerably the absorptive capacity of a firm. Thus, we expect a positive sign of the

coefficients of these variables in the adoption equation.

Firm size is another important factor which influences adoption behaviour and is included as explanatory variable in most

studies of adoption behaviour (see e.g. Colombo and Mosconi

1995; Karshenas and Stoneman 1993; Stoneman and Tiovanen

1997). We specified this variable either as a series of dummy

variables (see table 4) or as a polynom with respect to the number

of employees (linear and quadratic term; see tables 7a and 7b).

We expect a positive effect of firm size. However, there is a dis- turbing aspect to this approach: one does not know exactly what

is measured by this variable; firm size can be used as a proxy for various economic effects connected with the iimovation or tech- nology adoption process, but there is no comprehensive theory

available to explain the observed size effects. In our case we

found statistically significant correlations at the 5%-level with

firm size (number ofemployees) for some ofthe explanatory vari- ables listed in table 2. Positively correlated with firm size are the

variables HUMCAP, COOP, PDMARKET; also positively but somewhat weaker are correlated the variables FLEX, INPC,

LBATCH and CONTFLOW. Further, negative correlations were

found for PDUSER, SBATCH (and FIRMAGE). For these vari- ables the effect on adoption behaviour is itself size-dependent.

Therefore, in a first step we chose to estimate our model both

vvith and without firm size as an independent variable in order to assess the extent of interdependency of size with the other variables. In a further step, we also estimated the model (together

with the policy equation) for a subsample of small firms, thus

taking fiill account of the size-dependency of the independent

variables. Finally, we included some industry dummies, firm age

(FIRMAGE) and the AMT intensity in year 1990 (start of the promotion programme; INT90) as control variables in the adoption equation. FIRMAGE should capture the effect of an older firm having a larger chance to adopt a new technology than a younger firm; thus we expect a positive correlation of this vari- able with the adoption variable. We hypothesize that the negative adoption effect for firms with a high AMT intensity m 1990

being near a 'saturation point' with regard to the application of

AMT would dominate over the positive effect of 'knowledge

accumulation' (or ' learning effect') due to the early use ofAMT,

so we expect a negative sign for this control variable.

Table 2: Specification ofthe Explanatory Variables

(Adoption Equation)

Variable Description Sign

/. Objectives of/motive (Scores of a principal FTNCOMP COST

FLEX DEV QUAL

BEST

5 for the adoption ofAMT

omponent factor analysis of 26 objectives of AMT; six factors) Favourable financial conditions; competitive pressure Cosl reduction

tfigher flexibiUty Improving product development

Better product quality Securing technological lead/best practice

?

+

+

++

2. Impediments to the (Scores of a principal TECH KNOWPERS

RESIST INVCOST

UTILIZ COMPAT

tdoption ofAMT

omponent factor analysis of 26 barriers to AMT ; six factors) High technological costs / uncertainties Lack of knowledge / lack of adequately qualified personell

Intnifirm resistance to new technology High investment costs Uncertain capacity utilization Compatibility problems (e.g. with installed machinery, etc)

-

3. Market conditions D

IPC INPC CONC 16-50 CONCl 1-1

5

CONCl-10

Mediuin-lerm (e>^ected) change of donand

Intensity of price competition in the product market

Intensity of non-price comptlition in the product market Three durruny variables for market concentration based on the number of

Principal competitors in the world (product) marktf

(16-50, 1 1 to 15, 1 to 10 competitors; 50+ as reference group)

+

?

?

?

4. Type ofproduction/ Product characteristics PDMARKET

PDUSER

Process characteristics SBATCH LBATCH CONTFLOW

products (dummy variables with Standardized product:{' as reference group)

Product differentiation Products according to user specifications (durrmiy variables with 'single-piece production' as reference group)

Small-batch production Medium- / large-batch production Continous (low / mass production

+

S. Absorptive capacity HUMCAP

COOP

Percentage share of highly qualified persormel Cooperation in R&D activities (dummy variable) +

+

Four dummy variables (50-99. 100-199,200-499. 500+; 1-49 employees as reference group) +

6. Control variables FIRMAGE

INT90 Number of years

Intensity of use of AMT 1 990

+

Policy equation

Policy Variables

Table 3 contains some information on govenunent support

of AMT in the period 1990-1996. About 20% ofthe firms in the

final data set have been supported by some element of the gov- ernment AMTpromotion programme. Subsidies for R&D projects

connected with the introduction and/or extension of AMT has

been the type of support most frequently applied (about 60% of

the firms); training and consulting services were claimed by about

two fifths ofthe firms. A more detailed view ofthis information

shows that there are considerable differences among industries and firm size classes with respect to promotion frequency and

mode of promotion. Not surprisingly, mechanical engineering/

vehicles and electrical engineering/electronics having a very high

potential for AMT use had benefited more-than-average from

government support. For these indusfries promotion was concenfrated primarily in R&D projects, whereas in metalworking

and other industries the support was focussed on consulting

and, somewhat lesser, training schemes. Very small (less than 50

employees) and large firms (more than 500 employees) have more

55
EVALUATION OFNATIONAL PROGRAMS

frequently received AMT support than medium-sized firms. Finally, there is a close relationship between mode of promotion

and firm size: very small and large firms received more-than-aver- age support in R&D, medium-sized ones in training; firms with

up to 200 employees were stronger promoted through the offer- ing of consulting services. The promotion of small firms may be

a reasonable policy goal, but it is difficult to find some justifica- tion for the support of large firms being for himselfalready mtensive users of AMT. Is it a policy failure? This is an important

point to be clarified by the analysis below.

Table 3: Government Promotion ofAh4T

NumbcT of fimu N

Govemmoit-supponed rums N •/.

Type of govenunenl ^uppon

Training Consulting Development (%of govemmcnt-.'.uppoiteil firm.s)

Tolat ineimijacturing 463 96 20.7 37J 43.1 60J

Groups cifIndustries 99 23 23J 435 62J 43J

Macfaiiw^/vdiklcs 101 28 27.7 25.8 34.5 76.7

ElectrjnachinVelectninics 95 25 26.3 32.1 25.0 714

OthiT iattusUKS 168 20 11,9 54.6 571 40.9

Firm size (numbCT of onployces) tenUun SO 83 22 26.5 18.5 53.9 63.0 50-99 87 13 H.9 50.0 61.5 66.7 100-199 127 20 1S.8 55.0 57.9 45.0 20^499 100 20 20.0 55.0 31.6 31.6 SOO and more 66 21 318 24.0 20.0 88.0

To model the impact ofgovernment AMT promotion (policy

effect) we constructed two types of variables: simple dichoto- mous variables 'promotion yes/no', overall and differentiated by

type of support (information/training, subsidies for consultancy

and for R&D projects), and variables based on a firm's assess- ment of the stimulus from promotion (overall, by type) on a five point Likert scale. These stimulus measures were subsequently

transformed to a binary variable (value 0 for stimulus of 1 and 2, value 1 for stimulus of 3 to 5 on the original five point scale); this binary variable has been used throughout in this study." We

report the estimates for both types of policy variables in the

'single equation' framework and only for the stimulus variable (CIMTHM) m the 'simultaneous equation system' framework.

Explanatory Variables

A policy equation was specified on grounds of 'ad hoc'

plausibility arguments. Specifically, several firm-specific factors were taken into consideration. First, firm size and industry dummies were inserted in the equation to control for general characteristics ofthe firm influencing the probability of getting govern- ment support. Because of limited financial resources small firms

" 26% of promoted firms report an overall 'high' policy

stimulus; the corresponding figures for training, consulting and R&D are 28.9%, 22.7% and 15.0% respectively.

One could also argue the other way round stating that large firms have better chances to get government subsides than small ones

because they can spend more in lobbying and have possibly greater

have a larger incentive to claim such support as large ones. Moreover, the promotion of small firms has been an explicit goal ofthe Swiss AMT progrzimme.'^ We expect thus a negative correlation of the policy variable with firm size. Casual observation of the data shows that the frequency ofAMT promotion varies considerably among indusfries thus justifying the inclusion of industry

dummies as regressors in the policy equation. Second, we considered two mstitutional characteristics which may be related to the probability ofbeing supported by a government programme,

i.e. status as affiliate or parent-house and affiliate of a foreign

enterprise. However, it is not a priori clear in which direction these variables could influence the policy variable. Third, firms

confronted with serious problems of financing innovation

projects should be more inclined to claim government support

than other firms. We constructed a proxy for 'limited financial resources for developing/adopting new technology' by calculating the means of the statements of the responding firms with

respect to five impediments of innovative activity related to fi- nancial problems (measured on a five point Likert scale). We

expect a positive sign for this variable. Fourth, we included a dummy variable measuring the previous experience ofa firm with

other government technology promotion programmes (positive

sign). We also experimented with variables such as measures of

intensity of innovation activities, labour productivity and export

share which capture several dimensions of firm performance;

however, these variables were omitted m the final estimates of

the policy equation because they showed no effect on the policy

variable in all estimates.

Empirical Results

Estimates ofthe Single Equations

Adoption Equation

Table 4 contains the estimates of the adoption equation

(DAMINT as dependent variable) based on 461 observations for firms having adopted at least one AMT element. The coefficients ofan estimate ofthe fiiU technology adoption model as specified in section 4 are listed in column 1 . Colimm 2 contains only those

coefficients which were statistically significant at the 10%-level

(backward elimination). An estimate ofthe model without firm size dummies containing only the variables with statistically significant coefficients at the 1 0%-level is found in column 3

.

experience in this field. However, we use a specific variable to cover 'promotion experience', thus we expect that with respect to firm size the 'small firm' effect will dominate.

' ^ These impediments are: high innovation costs, long pay-back

period for innovation projects, lack of internal and external financial resources, high tax burden.

89
ATP PAPERSAND PROCEEDINGS

The main contribution to the explanation of adoption

behaviour (in addition to the control variables) comes from three groups of variables: objectives of adoption of AMT reflecting

anticipated revenue increases related to the application ofAMT,

knowledge absorptive capacity and firm size. Four out ofsix variables representing the mfluence of adoption objectives have positive signs, one of them a negative one;

all signs are in the expected direction. The variables FLEX ('higher

flexibility') and BEST ('remaining on top of technological improvements') seem to have the largest weight among these variables. We could not find any significant effect for the sixth vari- able in this group DEV ('unproving product development').

As expected, human capital (HUMCAP) and the propensity

to cooperate in R&D (COOP) are also very important in explaining adoption behaviour. This is a result similar to that obtained in most innovation studies. A comparison of the coefficients of

these variables in columns 1, 2 and 3 confirms our correlation

calculations between firm size and HUMCAP, COOP mentioned

in section 4 which showed a considerable size dependency of

these variables. All size dummies in column 1 are positive and of increasing

magnitude indicating a positive and monotonous relationship

between adoption and firm size. We could not fmd any signifi- cant effect related to the variables for various possible cost- increasing hindrances to AMT adoption except for INVCOST

('high investment costs') having the right negative sign. In accordance with earlier results for the innovation

behaviour of Swiss manufacturing firms the variables reflecting

market conditions show no significant influence on adoption

behaviour. However, we report some reserve with respect to the

result for the variable D (demand expectations) being possibly

not properly measured (see also section 4). The modes of exist- ing production technology and existing types of products do

not seem to play a major role for explaining the intensity ofAMT

use; only firms with medium-/large-batch production adopt, as expected, AMT more intensively than other firms. There are two

contrary effects (scale economies vs. flexibilty possibilities) af- fecting the overall influence ofthese variables which, according

to our results, counterbalance each other.

Finally, we obtain also for the control variables the expected

signs, the coefficient ofFIRMAGE being positive but not statis- tically significant probably due to the short period covered by

their change variable DAMINT.

In sum, the anticipated new revenue potentials are seemingly much more relevant for the firm's decision to adopt (or

intensify the use) of AMT than the costs which are associated

with the introduction and adaptation of these new technologies.

A high capacity to efficiently absorb and apply new knowledge

supports an earlier and/or more intensive adoption of AMT. Fi- nancing the investment for the new technology is found to be a problem especially for small firms causing a postponing of the

adoption ofAMT. Large firms seem to have a general advantage

in adopting AMT earlier than small ones. Finally, previous use of

AMT does not lead to a more intensive application of this tech- nology later on, that is the 'saturation effect' is dominating the

'learning effect'.

Policy Effects in the Adoption Equation

Table 5 contains estimates of the adoption equation includ- ing two alternative policy variables (government support: yes/ now; stimulus) as additional explanatory factors. The estimation

of Equation 1 was based on all available observations ('total sample'; N=463). For the estimation of equation 2 we used a sample of those firms which at the launching of the promotion

programme in 1990 did not yet apply AMT in production ('re- duced sample'; N=330). Thus, testing for the policy effect in equation 2 is more restrictive than in equation 1. We estimated

the adoption equation also for the subsamples of small (less than 200 employees) and large (200 and more employees) firms. All coefficients ofthe policy variables in table 2 are positive.

However, for all firms, i.e. independent ofsize, we find only in one

case a significant coefficient of the policy variables at the 10%-

level (reduced sample; stimulus variable). The policy effect is significant in all estimates based on the subsample of small firms

for both types ofpolicy variables with and without 'before- 1 990-

AMT adoptors'. On the contrary, we carmot find a significant

policy effect for subsample ofthe large firms. In sum, as to the 'single equation framework' there is an unequivocal positive influence of small firms getting govern- ment support on the probability of increasing the intensity of

AMT use. This is a first hint that policy does reach its specific

target of supporting the diffusion ofAMT in small firms.'''

Policy Equation

Estimates of the policy equation are found in table 6. According to these results there exists a negative relationship be- tween firm size and the probability of being a government-supported firm (see columns 1 and 3); this relationship is stronger

when the 'total sample' (N=463) is used (see column 1). Taken

together with the findings that, first, AMT diffusion and firm size are positively correlated and, second, that there is a positive

effect of policy variables on adoption, this result can be inter- preted as an additional indirect hint that the causality between

adoption and policy intervention runs from policy to adoption

and not the other way round.

More detailed estimates not presented here show that the

overall positive policy effect can be traced back mainly to the effects of

consulting and training, whereas R&D promotion does not seem to contribute to a faster adoption.

90
EVAL UATION OFNATIONAL PROGRAMS

Table 4: Determinants ofthe Change ofthe AAfT 1990-96 Adoption Intensity

(Ordered Probit Estimates of the Adoption Equation)

Explanatory Variables DAM UN 1

1 z

1

Objectives rUNv^WlVLr 1 J

T 9** -.ID

(.U6J (.05) COST .09 .10* .12**

(.06) (.05) FLEX .21** .23** .23**

(.06) (.06) (.06) DEV .04

(.06) QUAL .13** .11** .10*

(.06) (.06) (.05) BEST .19** .19** .20**

(.06) (06) (.06)

Impediments

Mi

1 r\c\ (.06) KNOWPERS .02

(.06) RESIST -.03

/ c\c\ (.06) INVCOST -.10 -.10* -.12**

(.06) (.06) (.06) UTILIZ .06

(.06) COMPAT -.06

(.05)

Market conditions D .02 (.06) IPC .08

(.06) INPC -.06

(.06) CONC 16-50 .08

(.17) CONCll-15 .03 (.15) CONGO 1-10 .06

(.15)

Explanatory Variables IJAM 1 IJN 1 2J

Type ofproduction

PDMARKET -.01 (.12) PDUSER .00

(.14) SBATCH .05

(.12) LBATCH .29** .29** .36**

(.12) (.11) (.11) CONTFLOW -.12

(.16) Knowledge

Absorptive capacity HUMCAP .01 .009*

(.01) (.004) COOP .19 .21* .30**

(.12) (.12) (.12)

Firm size 850-99 .33* -

52**

(.19) (.14) S200-499 .57** .28*

(.21) (.16) S500+ 1.2** .85**

(.26) (.20)

Control variables FIRMAGE .00

(.00) INT90 -.18** -.16** -.13**

(.02) (.02) (.02) N 461 461 461 McFadden r2 .129 .116 .092

LR-statistic 127* 114* 90* % concordance 73.3 72.3 70.2 Equal slope test 44 25* 10

Thefirst equation contains all model variables; the second and the third equation (without dummiesforfirm size) contain only the variables with statistically significant coefficients at the 10%-level (backward elimination). Standard errors are included in brackets under the estimatedparameters (**, * indicate statistical significance at the 5%-level and 10%-level resp.) Intercepts and

the coefficients ofindustry dummies have been omitted.

91
ATP PAPERSAND PROCEEDINGS

Further important factors which correlate positively with the

probability of government support are the lack of financial re- sources and, partly, previous experience with government support in other programmes and the status ofbeing an affiliate of a

foreign enterprise. The effect of industry dummies is negligible

(except for the metalworking industry).

To test for the direction of the interrelationship between

adoption and policy we also inserted the adoption variable

DAMINT as additional regressor in the policy equation (col- umns 2 and 4). We obtain a significant positive coefficient for the adoption variable when using the 'reduced sample' (N=330; col- umn 3). This is a hint that also the opposite causality works;

thus, we need a simultaneous equation fi-amework to clarify this issue. Table 5: The influence ofPolicy Variables (Ordered

Probit Estimates ofthe Adoption Equation)

Adoption Variable

Policy Variable DAMINT

N=463

1 N=330

2

Government

Support: yes/now

all Firms .17 .24 (.14) (.28) Small .33* . .51*

(.19) (.22) Large .18 .04

(.24) (.36)

Government

Support: stimulus

all Firms .24 .46*

(.17) (.22) Small .53* .67*

(.21) (.25) Large .18 .29

(.34) (.68)

The table contains only the coefficients of the two alternative policy variables used in the estimations; the standard errors are included in brackets under the coefficients (* indicates

statistical significance at the 10%-level). Both policy variables are binary; the stimulus variable was originally measured as an ordinal variable on afive point Likert-scale andsubsequently

transformed to a binary one (value 0 for stimuli of 1 and 2, value 1 for stimuli of 3 to 5 on the original scale). Equation 1 was estimatedfor allfirms. For the estimation ofequation 2 we

used a sample ofthosefirms which at the start ofthe government

promotion programme in 1990 did not yet apply AMT in production. 'Small' refers to a sample containing only firms

with less than 200 employees, 'large' to its complement

containingfirms with 200 employees and more.

Table 6: Probit Estimates ofthe Policy Equation

Explanatory CIMTHM (stimulus) Variables N_463

N= 330 Firm size 1 2 3 4

S20-50 -.58 -.56 -.08 -.08

(.35) (.35) S50-99 -.95* -.89* -.56 -.50

(.37) (.36) SlOO-199 -.61* -.64* -.15 -.15

(.35) (.35) (.34) (.34) S200-499 -.94* -.93* -.72* -.75*

(.37) (37) (40) (40)

S500-999 -.86* -81* -1.09* -1.00*

(.43) (.43) (60) (.60) S1000+ -61 .58

(.45) (45)

Groups ojindustries Metalworking .44* .45* ,30 .29

(.22) (23) tTT\

N'lachineryA'ehicles 20 .21 .31 .32

(.24) (.24) (.29) (.28)

Electrical machinery/ 15 18 -.25 -.15 Hectronics (.25) (25) (.34) (.33) Chemical^plastics -.43 -.37 -.03 .08

(.42) (.41) (.47) (46)

Ownershp status

Affiliate company -04 -.02 -.01 .05 (18) (17) (.22) (.21) Firm in foreign ,39* .39* -.01 .03 Ownei3hip (23) (23) (34) (.33)

Financing<£fficulties .28* .27* . .39* .38* (16) (.16) (.20) (.20)

Previous experience .28 .28 .39* .38*

stilh gavemm sttpport (18) (.17) (22) (.22)

Adoption variable .14 .27* DAMINT (11) (14) N 463 463 330 330

IktFaddenR' .078 073 .105 .088

LR-statistic 27* 25* 24* 20* %concordance 70.7 68.9 72.9 70.4

Simultaneous Estimates

We estimated a simultaneous probit model with an adoption

and a policy equation each of them containing as additional ex- planatory variable the dependent variable of the other one. We

conducted the simultaneous estimation only with the variables which in the single equations showed statistically significant coefficients at the 1 0%-level. Table 7a contains the estimates for both the 'total' and the 'reduced' sample for all firms, i.e. inde- pendent of firm size, table 7b shows the corresponding estimates

for the subsample of small firms (less than 200 employees). We

applied the same specification as in the 'single equation' framework for all independent variables except for firm size. In this case we used a polynom with respect to the number of employees (linear and quadratic term) for the estimates in table 7a and a

linear term for the estimates in table 7b. The basic pattern of the model estimates for both equations

is the same indicating to a certain robustness of the underlying

relationships. The results are quite clear with respect to the

92
EVALUATION OFNATIONAL PROGRAMS

influence of the adoption variable on the policy variable: the

coefficient of DAMINT (adoption variable) is not statistically

significant in all four estimates in tables 7a and 7b. The hypothesis of the causality runnmg from the policy to the adoption

variable is confirmed for the 'reduced sample' (more restrictive condition) both for all firms and the subsample of small firms.

Table 7a: Simultaneous Probit Estimates ofthe Adoption and

the Policy Equation (all Firms)

Explanatory LIAMIIN 1 K,lvA 1 MM UAMIIN 1 LIM ItiM

Vanabiss total sample reduced sample FINCOMP -.I4* -. 14*

(.07) (.08) COST .11* .11 (.05) (.07) 23* 27*

(.06) (.07) QUAL .10* 13*

t.uoj BEST .19* * .20*

(.07) INVCOST -12* -.06 (06) (.07) LBATCH .33» .26*

(.11) (13) COOP 25* 22

(.12) (.14) L .33' -03 1.53* -1.68 (number of employees) (.16) (.27) (.63) (1.30) L2 -.02 .01 -.42 .08

(.02) (.04) (.54) (1.70) INT90 -.15« -.20*

(.02) (.03) Fimi in foreign .46* 2%

Ownerehip (.23) (.34) Ftnancing problems .13 .17

(.16) (.20) Previous experience .4I* .51* with govern, support (.17) (.22) CIMTHM 08 28*

(policy variable) (12) (.16) DAMINT 07 -.09 (adoption variable) (.13) (.15) N 463 330 QT<S: N2=21.1;df=14) 24.5 15,8 P 101 .213*

R2

(.09) (12)

0.250 0087 0 239 0 087 Conclusions

The paper applies a procedure of analyzing the impact ofpublic

promotion of AMT on the diffusion of such technologies based on the estimation of an adoption equation within the framework

ofdiffusion theory by using (primarily qualitative) firm data. The

approach takes account of the interdependency of adoption and

goveniment promotion and brings to light some econometric

evidence on the direction of the casuality between these two

variables, thus helping to assess the economic impact of the AMT promotion programme.

According to the technology diffusion model on which our

analysis builds the most important factors influencing the adoption behavjior ofa firm are: positively, a series ofanticipated new

revenue potentials (e.g. through cost reduction, higher flexibility, etc.); the high capacity to efficiently absorb and apply new

technology; firm size; negatively, adjustment costs which are associated with the introduction and adaption ofthese new tech- nologies (primarily investment costs). Has the Swiss government programme to promote AMT

been effective? Our evidence shows that the policy goal to foster the diffusion ofAMT was clearly attained in the case of small

firms (with less than 200 employees) and/or for firms adopting

AMT for the first time or characterized by a low intensity of

AMT use with some overlap between these two categories. The

promotion instruments have, however, no influence on the adoption strategy of large firms. The support through training

programmes and consultation seems to be more effective than

by subsidizing R&D projects. Especially for large firms it is quite

doubtful whether subsidized R&D projects in the field of AMT

are more than just a substitute for a firm's own activities. We are convinced that this type of a analysis should be a

core element of policy evaluation. An improvement of the data base could significantly raise its reliability because the weaknesses ofthe approach lie primarily at the empirical level. Therefore, it is indispensable that the preconditions for an evaluation

should be secured from the very beginning ofpolicy implementation, i.e. already in its preparatory stage. This means, among

other things, that the institutions responsible for policy implementation should be obliged to gather the necessary data and

get the competence to enforce the promoted firms to deliver the information needed.

However, we are aware of the shortcomings of a procedure

based only on 'hard' evidence which does not account suffi- ciently of "soft factors" relevant to the adoption of AMT (e.g. the organizational environment). Presimiably these are of particular importance in the case of AMT because the introduction

of such technologies often has a significant impact on the sys- tem of production as a whole. Thus, we propose as a general

strategy the parallel use of several methods to evaluate a certain policy measure. For example, in the case presented in this paper we are in favour of supplementing our approach, first, by undertaking a number of in-depth interviews at the level of firms with

specific characteristics which seem important according to the present analysis.'^ Secondly, we find it useful to conduct comparative studies of the promotion of AMT in different countries

using a similar methodological approach. The latter work could

contribute to "policy benchmarking" found nowadays useful to improve policy making at the national or regional level.

' ' Two further studies in this direction were effectively conducted

parallel to our work.

93
ATP PAPERSAND PROCEEDINGS

Table 7b: Simultaneous Probit Estimates of the Adoption and

the Policy Equation

(small Firms; less than 200 employees)

Explanatory DAMINT CIMTHM DAMINT CIMTHM

Variables reduced sample COST .15* .22* -

(.07) (.08) FLEX .20* - .22* -

(.08) (.09) DEV .29* - .33* -

(.07) (.08) QUAL .17* - .18* -

(.06) (.07) BEST .21* - .16* - (.07) (.08) INVCOST -.11 - -.06 - (.08) (.62) COMPAT -.15* - -.26* -

(.13) (.07) IPC .15* - .17* -

(.08) (.09) LBATCH .39* - .20 - (.15) (.17) COOP .22* - .49* -

(.15) (.17) L 2.71* .69 1.52 -.09 (number of employees) (1.40) (1.90) (1.60) (2.1) INT90 -.18* - -.20* -

(.03) (.04) Firm in foreign .62* .48 Ownership (.33) {.39) Financing problems .19 .08

(.20) (.23) Previous experience .24 .36 with govemm. support (.22) (.25) CIMTHM .20 .34*

(policy variable) (.14) (.19) DAMINT -.04 -.09 (adoption variable) (.13) (.14) N 297 227 Qt(©: k2 =22.3; df^l5) 33.9 20.9 P

.233* .331*

R2

(.13) (.14) 0.306 0.058 0.295 0.062

See table 7a. References

Arvanitis S., Donze L., HoUenstein H. und S. Lenz (1998) Wirksamkeit

der diffusionsorientierten Technologieforderung des Bundes. Eine

Analyse anhand von Firmendaten, hrsg. vom Bundesamt fiir Berufsbildung und Technologic, Berne.

Arvanitis S. and H. HoUenstein (1994) Demand and Supply Factors in Explaining the Innovative Activity of Swiss Manufacturing Firms, An

Analysis Based on Input-, Output- and Market-oriented Innovation

Indicators, Economics ofInnovation and New Technology, 3, 1 5-30.

Arvanitis S. and H. HoUenstein (1996) Industrial Innovation in Switzerland: A Model-based Analysis with Survey Data, in: A.

Kleinknecht (ed.) Determinants ofInnovation andDiffusion. The Message

from New Indicators, Macmillan, London.

Arvanitis S. and H. HoUenstein (1998a) Firm Performance, Innovation and Technological Spillovers: A Cross-Section Analysis with Swiss

Firm Data, in: G. Eliasson, C. Green and C. McCann (eds.) The

Microfoundation ofEconomic Growth: A Schumpeterian Perspective,

University of Michigan Press, Ann Arbor (forthcoming).

Arvanitis S. and H. HoUenstein (1998b) The Determinants ofDiffusion ofAdvanced Manufacturing Technologies - An Empirical Analysis Based on Firm-level Data, mimeo, Zurich.

Balthasar A., Reger G., Battig C. und S. Btihrer (1997): Evaluation der Schweizer Beteiligung an den FTE-Rahmenprogrammen der Europaischen Union, vdf Hochsulverlag AG, Zurich.

Cohen W.M. and D. Levinthal ( 1 989) Innovation and Learning: The two

Faces ofR&D, Economic Journal, 99, 569-596.

Colombo M.G. and R. Mosconi (1995) Complementarity and

Cumulative Learning Effects in the Early Diffusion of Multiple

Technologies, Journal ofIndustrial Economics, 43, 13-48.

Freiburghaus, D., W. Zimmermann und A. Balthasar (1990) Evaluation der Forderung praxisorientierter Forschung (KWF), Studienreihe des Bundesamtes fiirKonjunkturfragen, StudieNr. 12, Bern.

Ewers H.-J., Becker C. und M. Fritsch (1990) Wirkungen des Einsatzes

computergestiitzter Technologien in Industriebetrieben, Walter de

Gruyter, Berlin.

Fritsch M. (1991) Die Ubernahme neuer Techniken durch

Industriebetriebe. Empirische Beftinde und Schlussfolgerungen fur die weitere Forschung, IFO-Studien, 37, 1 - 1 8. OECD (1 995) Information Technology (IT) Diffusion Policies for Small and Medium-Sized Enterprises (SMEs), OECD Working Papers, No.

60, Paris. OECD (1997) Policy Evaluation in Innovation and Technology. Toward

Best Practices, Paris.

Polt, W., E. Buchinger, L. Jorg, A. Kopcsa, H. Leo, L. Mustonen, F. Ohler, S. Patsios (1994) Evaluierung des ITF-Forderschwerpunktes

„Flexible Computerintegrierte Fertigung" (FlexCIM), Forschungszen- trum Seibersdorf, Seibersdorf, Februar.

Karshenas M. and P. Stoneman (1 993) Rank, Stock Order and Epidemic

Effects in the Diffusion of New Process Technologies. An empirical

Model, Rand Journal ofEconomics, 21, 27-44.

Karshenas M. and P. Stoneman (1995) Technological Diffusion, in: P. Stoneman (ed.), Handbook of the Economics of Innovation and

Technological Change, Blackwell, Oxford.

94
EVALUATION OFNATIONAL PROGRAMS

Madalla G.S. (1983) Limited Dependent Variables and Qualitative Variables in Econometrics, Cambridge University Press, Cambridge.

Sarkar, J. (1998), Technological Diffusion: Alternative Theories and

Historical Evidence, Journal ofEconomic Surveys, 12, 13 1-176.

Schepers, A. and G. Arminger (1992) MECOSA, A Programme System

for the Analysis of General Mean- and Covariance Structures with Non-Metric Variables, User Guide (SLI-AG, CH-8500 Frauenfeld).

Stoneman, P. and P. van Diederen (1994) Technology Diffusion and

Public Policy, The Economic Journal, 104,918-930.

Stoneman, P. and O. Toivanen (1997) The Diffusion of Multiple

Technologies: An Empirical Study, Economics ofInnovation and New

Technology,5, 1-17.

Table A. J: Description ofthe Used Data Set N %

Total manufacturing 463 100

Industry

Food, beverage 19 4.1

Textiles 14 3.0 Clothing, leather 5 1.1 Wood 10 2.2 Paper 15 3.2

Printing 13 2.8 Chemicals 14 3.0 Pubber, plastics 24 5.2

Glass, stone, clay 16 3.5 Metals 99 21.4

Machinery 84 18.1

Elektrical machinery 27 5.8

Elektronics, instrum. 83 17.9 Vehicles 17 3.7 Other manufacturing 23 5.0 Firm size (number of employees)

5-49 83 17.9 50-99 87 18.8 100-199 127 27.4

200-499 100 21.6

500 and more 66 14.3

95
Comments on Are Swiss Programmes OfPromotion OfAdvanced Manufacturing Technologies (Amt)

Effective? An Economic Analysis Based On Micro-level Survey Data

ByRonJarmin'

Data Issues Methodological Issues

Retrospective data Respondents can not remember, some evidence in the U.S. ofde-adoption (McGuckin, Streitwieser and Doms

Economics of Innovation and New Technology, (1998,

forthcoming)).

Can respondents answer all questions accurately?

Can the same person accurately answer questions on

market structure and technology adoption?

For what period are the control variables measured?

I assume they are measured at the end. This is after any

impacts that may have altered their values (e.g., size market conditions)

Sample selection?

Two types of survey non-response that determine final sample

34% response rate leads to N=667

What yields regression Ns of 463, 330? Item non-re- sponse? If so, for what questions? How did both types

of non-response vary across treated and non-treated

firms? Also, a panel was drawn. Then some treated plants were added. Do you re-weight the responses?

' Dr. Jarmin is currently a research economist at the Center for Economic Studies of the U.S. Bureau of the Census.

1. Need to know more about the policy interventions. We want to know how firms are selected into the pro- gram. Do they approach the program of visa versa? What is the nature of the interventions and how much

variation is there?

2. Consider changing the rationale and interpretation of the two equation model.

Adoption: A = aO + alX + a2P + e

PoUcy: P = bO + blY + u

This is the basic selection model in Maddala 1983. Need m

instrument in Y that ?s not in X. Or panel data if the unobserved characteristic in e that is related to P is fixed over time

(e.g. managerial quality).

3. Is there evidence adoption of AMT ?s improves the perfor mance of Swiss firms?

US evidence (Mcguckin et al and Doms, Durme and Troske

QJE 1 997) suggests that adopters performed better both pre- and post adoption.

4. Provide a table of descriptive statistics that compare treated and untreated firms.
METRICS OFEVALUATINGPUBLICR&D PROGRAMS

s

E

C

T

I

O

N

D

Citations from Patents to Papers: A Measure of Public Research Spillover into Private Industry

By Francis Narin and Dominic Olivastro^

Abstract

Recent research carried out for the National Science

Foundation, and other agencies has measured the direct

spillover effects ofpublicly supported research onto industrial technology by tracing hundreds ofthousands ofreferencesfrom

recent U.S. patents to the underlying scientific papers they cite. For example, it was found that the citation of U.S.-authored

papers by U.S. patents had tripled between 1987/88 and 1993/

94. This increase is continuing with more than 8,000 nonpatent references per week appearing on thefrontpages ofU.S. patents in late 1997.

It was also shown that close to 75 percent of the papers

cited by U.S. industrial patents had their origin in publicly

supported research institutions, at universities, medical schools,

andgovernment supported laboratories and similar institutions

in the U.S. and abroad. It was alsofound that the cited research,

especially in the biotechnology area tends to be very basic,

quite recent, with the papers originating at the mostprestigious

universities and laboratories, including NIST, and heavily

supported by NSF, NIH, and other organizations.

This paper will both bring the previously reported data

up-to-date by adding two additional years to the basic statistics, andadd new information on the localized linkages andspillover

effects. In the previously reported data it was shown that each

country's inventors in the U.S. patent system tend to

preferentially cite their own country 's papers by afactor of2 or

more. Datajust emergingfrom this research indicates that there

are, infact, similar local effects ofthis spillover, and thatpatents

from inventors in a given state show a substantial preferential

citing ofpapers in their own state and surrounding geographic

areas. This identifies, for thefirst time, a very local component

of the spillover from basic research conducted in public

institutions, to local industrial technology.

Introduction

Among both scientists and economists it is widely accepted

that public science •— scientific research that is performed in academic and governmental research institutions and supported

by governmental and charitable agencies — is a driving force

behind high technology and economic growth. Our recent paper

(Narin, Hamilton & Olivastro, 1997), from which much of this paper is drawn, provided quantitative evidence of both the magnitude and the direction of that force, based on tracing tens of thousands of references from recent United States patents to the scientific research papers they cite. Those citations from

patents to papers are du^ect evidence of a massive spillover from

basic research to industrial technology.

The premise that academic research makes an important

confribution to economic grovil:h is well accepted across the

political spectrum. For example, in his statement on Technology

for America's Economic Grovyth in February 1993, President

Clinton stated that "Scientific advances are the well-spring of

technical innovations. The benefits are seen in economic growth,

improved health care, and many other areas." The economic impact of science has, of course, long been a motivation for the goverrmient's support of academic research. During a period of tight research budgets some 30 years ago, Weinberg (1963) eloquently discussed the criteria for scientific choice, mentioning "technological merit" as one ofthe important

factors in determining the support of a scientific field. The

importance of this support was recognized even earlier in the fundamental assessments of science by Vannevar Bush (1960,

first edition 1 945), at a time when science was just emerging as a major area of governmental concern, and has recently been

reviewed by Martin and his colleagues (1996) in a report to the U.K. Government.

Furthermore, Rosenberg and his colleagues (1990) have

attributed much of the "Western Miracle" to the ability of the

free western economies to absorb and utilize scientific knowledge;

they specifically assert that "close links between the growth of

scientific knowledge and the rise of technology have permitted

the market economies of the Western nations to achieve

unprecedented prosperity."

The notion that technology springs from a scientific base was originally embedded in the "linear model" of innovation:

from basic research through applied research continuing into technology and resultant economic benefit. This simple linear model has been supplanted by much more complex views ofthe process, with many feedback loops and major confributions of

technology to science, but the origins of research knowledge in basic research still lie at the core of the process (Tumey, 1991).

' Dr. Francis Narin is President of CHI Research, Inc. Dominic Olivastro

is on the staff of CHI Research, Inc.
ATP PAPERS AND PROCEEDINGS

This paper will review our large-scale evidence detailing a

massive, contemporary linkage between industrial technology

and public science, with a tripling of the knowledge links from

U.S. technology to U.S. science in just six years. That analysis of more than 100,000 patent-to-science references provides evidence

that public science plays a crucial role in patented industrial technology. For example, the data showed that only 1 7 percent of

the science references on the front pages of drugs & medicine

patents are to U.S. industry papers, while 50 percent are to U.S.

public science, and 33 percent are to foreign science, most of

which is also public. The data also show that (1) the patent-toscience linkage has a sfrong national component, with U.S.- authored scientific papers particularly heavily cited by U.S.-invented patents; (2) the linkage is appropriately subject specific,

with chemical patents citing chemistry papers, drugs & medicine

patents citing biomedical papers, and so forth; and (3) that the linkage is quite contemporaneous and quite basic, with the refer- ences on U.S. patents citing recent papers situated at the basic

end of the research specfrum. The present paper shows that these linkages are still increasing at a rapid rate, and adds two

major new dimensions: evidence for the broadening ofthis link- age, and evidence that this linkage has a localized state-by-state

spillover effect.

Methodology

This paper is based on an analysis ofthe more than 750,000

non-patent references (NPR's) which were listed as "other

references cited" on the front pages of the U.S. patents issued

since 1989. For regular U.S. patents issued in 1993 and 1994, on

average there were 8.6 references/patent to U.S. patents, 1.9 references/patent to foreign patents, and 1.5 non-patent

references/patent.

The "references cited" on U.S. patents are a fimdamental

requirement of patent law. When a U.S. patent is issued it has to

satisfy three general criteria: it must be useful, it must be novel,

and it must not be obvious. The novelty requirement is the primary

factor leading to the references which appear on the front page

of the U.S. patent, since it is the responsibility of the patent

applicant and his attorney, and ofthe patent examiner, to identify,

through various references cited, all of the important prior art upon which the issued patent improves. These references are chosen and/or screened by the patent examiner, who is "not

called upon to cite all references that are available, but only the

best" (Patent and Trademark Office, 1995).

Non-patent references (NPR's), as they appear on the front pages of U.S. patents, are a mixed set of references to scientific journal papers, meetings, books, and many non-scientific sources such as industrial standards, technical disclosures, engineering

manuals, and every other conceivable kind ofpublished material.

All ofthe NPR's underwent a standardization process to put

them into various categories, such as science references,

abstracts, and books. Roughly 55 percent ofthe NPR's are science references —that is, citations to scientific journal papers, scientific meetings and other scientific publications; roughly 75 percent of

these science references are citations to papers published in the 4,000 or so journals covered by the Science Citation Index (SCI),

and 75% of these are complete and accurate enough to match to the cited paper. Most ofthe unmatched references were either incomplete or

erroneous; for example, a reference might omit the author's name,

or misspell it, or give the wrong volume or page numbers.

Three fmal methodological notes are important. First, the data are extremely complex, in that there are many different ways

counts can be made. A given paper may be cited in several patents and a given patent may cite a number of different papers. In addition, a paper often has authors from more than one

institution, and may be supported by more than one agency. This combination ofmany patents citing many papers with many

different institutional addresses and support sources gives rise to various complexities in counting and presenting data. The second methodological note concerns the limitation of

this analysis to the science references on the front pages of the U.S. patents, omitting any analysis of references in the body of

the patents. The reasons for this are both theoretical and practical.

Theoretically, the front page references should be the most

important ones on a patent, since they are the ones relied upon,

as mentioned previously, by the examiner in establishing the

patent's novelty. Furthermore, from a practical viewpoint it is far more difficult to exfract the non-patent references scattered through the text of a patent. In one brief study conducted to evaluate the

representativeness ofthe front page science references, we found

that approximately half of all the science references are on the

front page, and that there was great similarity between front page

and fiill text science references (Narinetal., 1988). As a result it

is quite reasonable to assume that the science references analyzed

here are representative of all of the science references in the patent and probably underestimate the dependence oftechnology

upon science by a significant factor. The increasing dependence

ofmodem technology on science revealed in this paper is, almost

certainly, a very conservative estimate ofthat important linkage.

Finally, the data presented herein measure linkages in codified knowledge, as reflected in patents and papers, and understates

the important confributions that public science makes to industry

through research fraining, which predominates in technologybased industries such as automobiles and aircraft.

100
METRICS OF EVALUATING PUBLIC R&D PROGRAMS

Overall Linkage Characteristics

The U.S. patent system is quite representative ofthe world's

technology. Approximately half ofthe inventors ofU.S. patents

are foreign, and each country's inventors patent in the U.S. in rough proportion to their country's Gross Domestic Product (GDP)

(Narin, 1991). In addition, the patent system covers the whole

range oftechnology, from old but still active classes representing

such basic mechanical technologies as railroads, to the most

modem human genetics technology. Across all of these countries and technologies there has

been a steady increase in science linkage for at least two decades. The last decade ofthis increase is shown in Figure 1 on a countryby-country basis, and in Table 1 on a country-by-technology

basis. Figure 1 shows that science linkage is increasing fastest for U.S. and U.K.-invented patents, but also increasing steadily

in U.S. patents with French, German, and Japanese inventors, as

it is for patents from almost all other inventor countries. The

highly science-linked position ofthe U.S. and the U.K., compared

to the other three major countries, is the result of a combination

of two factors: a tendency within all technologies for U.S. and

U.K. patents to be more science-linked, and the comparatively

high level of U.S. and U.K. patenting activity in drugs and

medicine, biotechnology, and medical instrumentation, all ofwhich

are highly science-linked.

Table 1: Patents by Industrial Technology

Inventor Chemical £>rug and Medicine Electrical Prof & Sci Patents Patents Component Patents Instruments Patents Country 1985 1990 1995 1985 1990 1995 1985 1990 1995 1985 1990 1995

US. 0.94 1. 85 4.63 3.05 5.48 11 61 0 53 0.93 1.28 058 0.84 1.72

U.K. 068 1.05 2.50 1.33 2.66 5.26 0.44 0.70 1 20 039 0.76 1 35

Japan 0.44 067 1.28 1.06 1.57 3.26 0 31 0.46 0.69 0 13 0 17 0.42

France 0.32 0.63 1 05 1 24 1 27 249 0,54 0.64 0.79 0.31 0.43 1.02 Germany 0.44 0.63 1.34 0,97 1 67 3 54 0.44 0.64 0.98 0.24 0.33 0 55

All Countries 0.74 1 30 3.18 2.17 3 78 866 0,45 0.73 1.00 041 0.58 1.27

Another important characteristic of science linkage is that it

is very subject specific. Drugs & medicine patents invented in all countries cite almost exclusively to papers in the scientific fields ofclinical medicine and biomedical research. Similarly, chemical

patents cite chemistry papers heavily, and electronics patents

cite to physics and engineering papers, and there is throughout

the linkage phenomenon a subject-specific couple between the technology ofthe patent and the science upon which it is building.

The linkage is also quite recent; the peak cited year for papers

cited in drugs & medicine patents is 4-6 years prior to patent

grant, only a year or two slower than citing from biomedical papers

to biomedical papers.

Figure 1: Science Referencing. The average number of

science references per U.S. patent are consistently higherfor

U.S. and U.K. invented U.S. patents.

2.5

o

E

1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996

Another important characteristic ofthis linkage is the sfrong

national component. Each country's inventors in the U.S. patent

system cite their own country's papers two to four times as often

as expected, when adjusted for the size of a country's scientific publication rate. For example, approximately 7 percent of all the papers in the SCI are authored by scientists at German institutions. Compared to that, approxunately 1 7 percent of all ofthe science papers cited on the front pages of German-invented U.S. patents

are German-authored papers. The ratio of those two percents

represents a national citation ratio ofapproximately 2.4, in Germanto-German patent-to-paper citation. This is a completely general

phenomena, and is apparent for all the five major countries in Figure 2. If each country's inventors were equally likely to cite any of the world's science, then the height of each column in Figure 2 would be 1 . The fact that the diagonal columns are

significantly greater than unity, ranging from 2 to 4, indicates the

strong domestic component that exists in science linkage, showing

that each country's inventors are preferentially building upon

their own domestic science. Science linkage has a distinctly national component.

Figure 2: International Inventors in the U.S. Patent System

Country of Citing Country of

Patent cited Paper

1993-94 1981-93

101
ATP PAPERS AND PROCEEDINGS

It should be noted that the same national preference is apparent when looking at cross-national citations from papersto-papers and patents-to-patents. In both of those cases there is just as strong a national component in the linkage; all of these

citing phenomena show that there are still very strong national

ties between scientists within a country and inventors within a

country, and between national inventors and scientists, implying

that a strong scientific base is necessary for a strong national

technology, in science dependent areas of technology.

The final general point about this linkage is that the papers

cited in patents are published in prestigious, mainstream, basic

scientific research journals. Table 2 shows the 25 journals most

heavily cited by 1 993- 1 994 patents for the four most heavily cited

scientific categories: biomedicine, combining clinical medicine

and biomedical research, chemistry, physics and engineering &

technology.

Table 2: Numbers ofCitations and Cited Papersfor Top 25 Journals.

thirteenth on the physics list, led by more applied journals. As

would be expected in engineering & technology, many of the journals are electrical engineering journals, since there is such a very large amount ofpatenting in the electronics field, and many

of these patents cite to papers in IEEE and other electronics- related journals.

Cited U.S. Science

This section will describe the U.S.-authored scientific papers

cited in 1987-88 and 1993-94 U.S. patents. Because of the tendency for U.S.-invented patents to cite U.S. papers, the high

activity ofthe U.S. in science-linked areas oftechnology, and the

large size of the U.S. scientific establishment, approximately 58

percent of the ^CZ-covered papers cited in patents in the four

BIOMEDICAL JOURNALS CHEMISTRY JOURNALS PHYSICS JOURNALS ENGINEERING & TECH JOURNALS

a Cites # Papers # Cites # Papers # Cites # Papers # Cites # Papers

P NAS us 3835 2134 J AM CHEM S 1394 811 APPL PHYS L 2210 1356 lEEET 1812 1206 NATURE 2398 1175 TETRAHEDR L 991 673 JPN J A PHY 773 535 ELECTR LETT 721 521 SCIENCE 2344 1129 J ORG CHEM 945 632 J APPL PHYS 771 569 IEEE ELEC D 345 220

J BIOL CHEM 1874 1243 J CHEM S 590 412 J VAC SCI T 670 435 IEEE COMMUN 233 143

J MED CHEM 1136 646 ANALYT CHEM 472 324 APPL OPTICS 472 372 J AM CERAM 181 144 CELL 1121 644 J ELCHEM so 460 346 PHYS REV L 349 168 PIEEE 166 91 NUCL ACID R 1072 669 J CHROMAT 379 270 OPTICS LETT 347 242 IEEE COMPUT 164 103

J IMMUNOL 979 686 MACROMOLEC 314 194 IEEE J Q EL 299 195 IEEE J SEL 163 104 BIOC BIG? R 922 524 TETRAHEDRON 307 188 J LIGHTW T 298 204 SENS ACTUAT 162 no

BIOCHEM 821 543 J CRYST GR 277 216 THIN SOL n 286 174 J MATER SCI 152 99 GENE 111 406 J POL SC 241 179 REV SCI INS 228 156 IEEE CONS E 122 95 CANCER RES 679 485 ANGEW CHEM 224 114 NUCL INSTRU 222 182 IBM J RES 122 78 EMBO J 679 420 MACROMOL CH 218 149 PHYS REV 203 146 IEEE NUCL S 93 75

J VIROLOGY 593 449 INORG CHEM 212 140 OPT ENG 18S 120 J NON-CRYST 92 70 JBACT 496 381 CHEM LETT 204 149 SOL ST TECH 181 122 COMPUTER 89 65 ANALYT BIOC 490 308 J APPL POLY 168 101 J MAGN RES 172 108 COMM ACM 87 59 MOL CELL B 482 335 POLYMER 166 90 IEEE ACOUST 172 103 TAPPI J 86 59

J EXP MED 476 310 J PHYS CHEM 164 98 IEEE PHOTON 154 108 IEEE PATT A 73 50 INFEC IMMUN 472 380 SYNTHESIS-S 151 109 SYNTH METAL 152 77 J MATER RES 70 44 BIOC BIOP A 422 300 B CHEM S J 126 101 J PHYS 136 98 METALLURG T 70 44

J CLIN INV 409 288 J HETERO CH 112 85 J OPT SOC 134 94 J METAL 70 28 FEBS LETTER 400 266 J CATALYSIS 112 76 MOLEC CRYST 114 70 AT&T TECH J 67 43 BLOOD 380 284 POLYM BULL 112 54 PHYSICA C 102 82 J ALLOY COM 62 47

J CELL BIOL 365 256 HETEROCYCLE 100 78 APPL PHYS 92 61 IND ENG RES 60 37 N ENG J MED 362 251 ORGANOMETAL 99 66 OPT COMMUN

SOL ST COMM

80

80

67

56

J ELEC MAT 56 41

The journals shown on Table 2 are clearly prestigious and

influential, and for the biomedical and chemistry papers, quite

basic.The physics journals, however, are not as basic, with much

of the physics cited in patents published in applied physics

journals, rather than the more basic theoretical and high energy

physics journals. The Physical Review, for example, the largest

and one of the most prestigious physics journals, is only

years had at least one U.S. author, a total of 44,765 papers. An

attempt was made to locate each ofthese papers in the library, in order to verify the institutional addresses of the authors, and to tabulate the research support acknowledgements.

Figure 3 shows the most dramatic finding: that there has

been remarkable increase in linkage between U.S. patents and

U.S.-authored scientific papers; in just six years the number of

U.S.-authored papers cited in patents has more than doubled.

102
METRICS OF EVALUATING PUBLIC R&D PROGRAMS

the number of citations to these papers aknost tripled, and the number of research support acknowledgements on the papers more than tripled. Since ober this period of time the number of

patents in th U.S. system has only grown by 28 percent, an

increase in science linkage of200 percent is truly remarkable, and

indicative of a rapidly increasing dependence of patented

technology upon contemporary science.

Figure 3

1987-88 citing 1975-85

Number of Papers Number of Citations Number of Support

Cited Sources Cited

Figure 4 shows that this dramatic increase continued through

1996, increasing another 50 percent in the 1995-96 time period.

Although not yet matched, preliminary observatjions indicate

this increase continued through 1997.

Figure 4

1967 1988 IS© 1990 1991 1992 1993 1954 1995 1996

OtingraertYeaNote: The data is based on a 12-year window with a 3-year lag. For example: 1996patients are citing 1982-1993papers. 1995patents

are citing 1981-1992 papers, and so on.

A second important point is that these cited research papers

are authored at prestigious universities and laboratories, just as they were published in prestigious journals. This is shown in Table 3, which lists the top 25 author institutions for the four

categories. Note that in Table 3 we have combined the universities and their medical schools. Most of the biomedical papers listed there as Harvard University are papers which are published at Harvard Medical School. Note also that those are whole counts of citations. In a whole count, if a paper has authors at two

institutions, say Harvard and Stanford, it is counted as one

citation to each institution. In a fractional count, used in some of

the data later on, a cited paper with authors at Harvard and

Stanford would be counted as one-half a citation to each. For the biomedical papers it was not surprising that the National Cancer Institute (NCI) appears very high on the list, since NCI has a large Intramural research program. The high

ranking of the Veterans Administration (VA) was somewhat

surprising. However, almost all ofthese VA papers are co-authored with scientists at the medical schools associated with the VA

hospitals, indicating a close and productive linkage between the VA and the academic commimity.

A number ofthe major private companies are high on the list for chemistry papers, with DuPont listed as fourth, Lucent's Bell Labs sixth and IBM seventh. For both physics papers, and

engineering & technology papers, the top two cited organizations

are Bell Labs and IBM, both of which have large numbers of

patents, and many Bell Labs and IBM patents cite their own

company papers. However, there is also much citation from other patents to the Lucent and IBM papers, and this is by no means a

reflection ofexcessive company self-citation. For example, only

19 percent ofthe patent citations to Bell Labs papers come from

Lucent itself, while only 2 1 percent of the cites to IBM papers come from IBM patents. Note also that in physics and engineering some ofthe major

national laboratories also appear quite high on the list, as well

the private companies and major universities; this list clearly

shows a high degree of interdependence between the academic,

industrial and governmental R&D communities.

The Science Base of U.S. Industry

This section will look at the patent-to-paper linkage data from the perspective of U.S. industry patents. The data show

that even when the data are restricted to patents assigned to U.S.

industry, the dependence on U.S. public science is very

substantial. In this section the set of citing patents was resfricted to those patents whose assignees were U.S. non-governmental,

non-academic organizations, almost all ofwhich are U.S. industry.

Within these lists there may be a few small clinics or hospitals;

however, almost all the patents are from U.S. industrial companies.

103
ATP PAPERS AND PROCEEDINGS

Table 3. Numbers ofCitations and Cited Papersfor Top 25 Cited Journals.

BIOMEDICAL PAPERS CHEMISTRY PAPERS PHYSICS PAPERS

OalVdlU (JIUV 2506 MIT 171

Mflt'l l^all CanrpT \^(Ui^ci 11191 Tnct 1279 U Texas Austin 171

Veteran's Administration 1033 Harvard Univ 160 Univ Cal San Francisco 930 Dupont Co 142

Stanford Univ 920 u Cal Derkeiey 139 Univ Washington 845 Lucent Bell Labs 130 MIT 756 IBM Corp 122

Scripps Clin and Res Fdn 690 Merk & Co Inc 102 Univ Cal Los Angeles 642 Cornell Univ 96 Mass Gen Hosp 625 Texas A&M Univ 95

Johns Hopkins Univ 610 Penn State Univ 89

Washington Univ 588 Univ Wisconsin 87

Univ Cal San Diego 534 Purdue Univ 83 Univ Pennsylvania 517 Univ Illinois 83 Merk & Co Inc 484 U Cal Los Angeles 79

Yale Univ 463 Univ Massachusetts 72

Nat'l Inst Allergy & 456 Va Polytech Inst 71

Infectious Dis Univ Wisconsin 448 Rice Univ 70

Univ Michigan 447 Univ N Carolina 70

Cornell Univ 432 Dow Chemical Co 67

Genentech Inc 432 Stanford Univ 67 Columbia Univ 428 Univ Florida 66 Duke Univ 413 Univ Nebraska 66

Univ Colorado 409 U Cal San Diego 65

Rockefeller Univ 405 Iowa State Univ 63

The citing data consists of papers published in 1988, the peak cited year for science references in 1 993-1 994 patents. The

reason for restricting the data to one cited year is that, while

sector and institution had already been assigned to all the U.S.

institutions, the same had not been done for papers with foreignauthored addresses, and therefore a time-consuming assignment

of an organizational sector to the cited foreign papers had to be

carried out. Confining the data to one cited year kept the effort within reasonable bounds.

The overall data are shown in Table 4, both for all industry

patents, and for industry patents in three subsets: drugs &

medicine (including biotechnology), chemicals (excluding drugs

& medicine), and electrical components & accessories &

communication equipment. The table shows the strong reliance of U.S. industry patents on public sector science; overall, only

20.4 percent ofthe cited papers are from U.S. industry, 6.3 percent

from foreign industry, and the remaining 73.3 percent are from

public science, in the U.S. and overseas. This implies that U.S.

industry is far from self-sufficient in science. The great majority

ofthe science base ofU.S. industry comes from the public sector; public science appears to be crucial to the advance of U.S.

industrial technology.

As shown in Table 4, the area with the largest private science

base is electrical components, for which approximately half of

the cited papers are from industry. In the case of electronics, the

large electronics companies, IBM, Hitachi, AT&T, and others.

ENGINEERING & TECH PAPERS

854 1 itf^Pnt' Rf^tl T aVic 471 IBM Corp 566 IBM Corp 428

Stanford Univ 300 U Cal Berkeley 189

Bellcore 174 MIT 174 DSN, Kes Lab 167 Stanford Univ 162

Lincoln Labs 150 General Electric Co 1 1

1

MIT 133 Tex Instruments Inc 96 Univ Illinois 120 UbN, Res Lab 88 U Cal Santa Barbara 1 10 N Carolina State Univ 84

Cornell Univ 106 Bellcore 78 U Cal nerkeley 100 Xerox Corp 69 Xerox Corp 95 Univ Illinois 64 Univ Pennsylvania 93 Penn State Univ 60 N Carolina State Univ 90 U Cal Los Angeles 59

Caltech 87 Lincoln Labs 57

Sandia Natl Labs 83 Sandia Natl Labs 54 Lawr Livermore Nat'l Labs 80 Carnegie Mellon Univ 53

General Electric Co 74 U Texas, Austin 53 Lawr Berkeley Lab 74 Hewlett Packard Co 51 Gen Motors Corp 65 Motorola Inc 51

Jet Prop Lab.Caltech 64 Cornell Univ 48

Los Alamos Nat'l Labs 63 Univ Michigan 48

Natl Bur Standards 62 Univ Southern Cal 48

Hewlett Packard Co 60 Univ Washington 44 U Cal San Diego 56 Oak Ridge Natl Lab 43 Univ Houston 56

pubUsh heavily, and this private sector component ofthe scientific literature is quite heavily utilized m patents. Nevertheless, even

in this case halfofthe science base is public. For drugs & medicine,

chemicals, and abnost all ofthe rest ofthe database more than 75

percent of the science base of U.S. industry, as expressed by

these citations, has its origin in public science. The U.S. public

science component is larger than the foreign, but foreign public

science is still an important contributor.

Table 4: Institutional Origin ofPapers Cited in 1993-1994

U.S. Industry Patents

All Patents Drug and Medicine

Patents

Chemical

Patents

Electrical Component

Patents

us Industry 20.4% 16.7% 18.3% 37.5%

US Public 43.9% 50.3% 42.7% 29.6%

Foreign Industry 6.3% 4.2% 6.0% 13.3%

Foreign Public 29-4% 28.8% 33.0% 19 6%

100.0% 100.0% 100.0% 100.0%

# of Matched Cites 5217 1584 1784 585

Data from Fran (12/30/96) which he got from refsec41.xls These are percentages of references.

104
METRICS OF EVALUATING PUBLIC R&D PROGRAMS

The Spreading of Science Linkage

In the previous sections we have shown that the intensity

of linkage between patented industrial technology and public

science has increased markedly over the last decade. In this section we will show that simultaneous with the increase there has been a broadening of the science dependence.

The conservative nature of our approach is reflected both in the type of references we chose, and in how we culled down to the ones actually matched and analyzed.

As mentioned previously, we chose to use only the

references on the front page of the patent, those that had been

placed there with the acquiescence of the examiner; we omitted

the science references in the body of the patent, even though an

earlier study showed that there were as many science references

in the body of the patent as there are in the front page, and that the characteristics of the references - the fields, subfields, and

so forth, to which they referred were rather similar. Figure 5 shows the selection process used just for those

references from the front page, the ones upon which all the previous data was based. The data there are for the years 1993

and 1994, and they show that in that year 3 8.6 percent ofthe U.S.-

invented, assigned. Type 1 (regular) U.S. patents had at least one

non-patent reference (NPR).

Figure 5: Selection Process

Patents that refer to at least one Non-Patent Reference

(l*R). which is ...

... to a scientific fFR, and ...

pubished in a post-73 SO journal, and ...

. matched to the SCI, and .

. authored at a US-pubic instlution, and .

...pubished in 1981-1991

10 15 20 25 30 35 40 45

As mentioned earlier, we standardized those non-patent

references, and identified those which were to scientific papers, including papers in journals, presented at meetings, and so forth. Twenty-three percent of the patents had at least one of those

science references. We then limited our analysis to those papers published in a

Science Citation Index fSC/j-covered journal dated after 1973,

since that is the science database to which we were matching the science references; this reduces the percent of patents citing science papers to 18 percent.

After the matching process, we were reduced to 14.9 percent

of the patents that had an NPR that was matchable to the SCI

post- 1973. The 3.25 percent difference is basically due to incomplete or incorrect references in the patents. Sometimes the author name is missing, sometime the year or page are missing or

incorrect, there are spelling errors all through the data, and so

forth.We then looked at what percentage of those matched

references which were authored at U.S. public institutions, which

again reduces it significantly. The other cited papers were

authored at U.S. private companies or at foreign institutions. Finally, for the detailed library lookup we considered only an

1 1-year window which was, for patents issued in 1993/94, the years 1981 to 1991. Thus, the data upon which the analysis is based are, in a sense, very conservative.

We will now take a closer look at the second bar of Figure 5, the percent of U.S.-invented, assigned, regular patents which

cite to at least one scientific non-patent reference. This percentage

has been steadily increasing over the last decade as shown in Figure 6 for all areas of patenting, and in Figure 7 for drugs and

medicines, the most scientifically linked area of patenting. In

particular. Figure 6 shows that in 1985, 15 percent of all those

patents cited at least one scientific reference, rising to 29 percent

by 1 996, just 1 1 years later. Similar rises are shown in the number

of patents that have multiple science references: in 1985 only 3 percent ofthe patents had 5 or more science references, rising to 12 percent by 1996 and, in fact, by 1996, 3 percent of all patents

had 20 or more science references on their front pages. Most of

those patents, of course, are in the extremely highly science- linked areas ofbiomedicine, such as genetics, although there are some very highly science-linked patents in electronics and other advanced areas of technology.

Percent of US-invented, Assigned, Type-I (Invention) Patents

at different levels ofciting to Scientific Non-Patent References

1985 1990 1996

1985 1990

1996

# pats # sNPR # NPR

>=1

14 655 2013 2830 366

14 1079 6100 7883 692

14 2058 30872 34860 1725

# pats # sNPR # NPR

>=1

14

14

14

655

1079 2058

2013 6100

30872

2830

7883 34860

56

64

84

# References

=5 >=10 >=15 >=20

60 18 3 1 197 63 26 14 939 510 296 185

# References =10 >=20 >=30 >=40

9 3 0 0

18 6 2 1 46 25 14 9

105
ATP PAPERS AND PROCEEDINGS

Figure 6 and 7. Percent ofUS-Invented, Assigned Type-1 Patents

in Durgs & Medicines iwth NReferences to Science Papers.

>=1 >=10 >=20 >=30 >=40

Number of References to Scientific NPR

Figure 7 shows similar data, but in this case restricted to the broad area ofdrugs & medicine, which is the most science-linked

of all the broad areas of technology. In the case of drugs and

medicine the increase is still dramatic, although by 1 985, 56 percent

of the patents had at least one science reference, rising to 84

percent by 1996. There is also a marked rise in the patents with

large numbers of non-patent references: for example, 9 percent

had 1 0 or more in 1 985, but 46 percent did in 1 996, a remarkably

strong dependence on, as we have demonstrated, contemporary,

basic mainstream scientific research. A full 9 percent ofthe drugs

and medicine patents had 40 or more science references on their front pages in 1996. Thus, in this section we have shown that the growing linkage

between patented technology and basic research has also been

a spreading linkage, to include a wider and wider percentage of

patents.

Within State Linkage

In an earlier section of this paper we showed that there was

a very strong within country linkage preference, with each

country's patents citing papers from their own country two to three times as frequently as expected, adjusted for the number of

papers published in each country. In this section we will show

that there is analogous geographic preference operating within

the United States, with a noticeable tendency for industry patents

within a given state to cite to papers originating at institutions within that same state. We will illustrate the results for basic biomedicine, from a study we did for New York State. In that study we defined basic

biomedicine as all journals in our two fields of clinical medicme

and biomedical research, which were categorized as being in Level

4, Basic Research, or Level 3, Clinical Investigation (Narin, Pinski

&. Gee, 1976). Thus, this set ofjournals represents the most basic

kinds of scientific research, produced, of course, predominately

at universities, colleges, and medical schools, and such

governmental basic research institutions as the National

Institutes of Health. For this illustration we have removed government and

university owned patents, so that the data reflect the crucial spillover from public science to private industry.

Figure 8 shows the raw citing data for the seven largest

patenting states, tracing all patent cites to their basic biomedicine

papers. The graph shows the number of linkages from patents

invented in the given state, to papers authored at institutions in the cited paper state, for patents issued in 1983-96 and papers

published in 1 1 -year windows, for example, 1 983 patents citing to 1973-83 papers. Note the prominence of the within state citing.

Ofcourse, this is very strongly affected by the number ofpatents

and the number of papers within the state; a state with large

number of biomedical patents and papers, such as California

will, of course, heavily cite to California papers.

Figure 8 and 9

Citing Patent State (1983-96 Patents)

Figure 9 adjusts for this withm the data set itself, showing

papers in the database of papers cited by patents, which is not

perfectly related to number of papers from the state, in general.

However, if a given state's patents were citing papers in a geographically random manner within the set of papers cited by

patents, then the height of each of those bars would be 1 . That

is clearly not the case; the average withing state bar is close to twice as high as would be expected, showing a within state preferential utilization of science by that state's industrial technol- ogy, of a factor of2.

Conclusions

Therefore, we conclude that public science plays an essential | role in supporting U.S. industry, across all the science-linked

106
METRICS OF EVALUATING PUBLIC R&D PROGRAMS

areas of industry, amongst companies large and small, and is a fundamental pillar of the advance of U.S. technology. The

underlying hypothesis described in the beginning of this paper - that public science is a driving force behind high technology -

is clearly supported by the data shown herein. Furthermore, the data shows that the science that is spilling over into high

technology is mainstream; it is quite basic, quite recent, published

in highly influential journals, authored at major universities and

laboratories, and supported by public and charitable institutions.

Acknowledgements

The authors wish to acknowledge the encouragement and

guidance of Ms. Jennifer S. Bond, Program Director, Science and

Engineering Indicators Unit, the National Science Foundation,

throughout the course of the work, and for many years before. Work supported by NSF Grant No. SRS-9411 378 and NSF Contract No.SRS-9301815.

References

Bush, v., 1945 (reprinted 1960), Science: The Endless Frontier,

National Science Foundation, Washington.

Clinton, W. L. and A. Gore, Jr., 1993, Technology for America's Economic Growth, ANew Direction to Build Economic

Strength, Office of the President of the United States.

Martin, B., and A. Salter, 1996, The Relationship between

Publicly Funded Basic Research and Economic Performance, H.M.

Treasury, London.

Narin, R, K. Hamilton, D. Olivastro, 1997, The increasing

linkage between U.S. technology and public science. Research

Policy (26.3), 317-330. Reprinted in the AAAS Science and

Technology Yearbook, 1998.

Narin, F., 1991, Globalization of Research, Scholarly

Information and Patents —Ten Year Trends. Proceedings of the North American Serials Interest Group (NASIF) 6th Annual

Conference June 14-17, 1991, The Serials Librarian21, 2-3.

Narin, F., and D. Olivastro, 1988, Patent Citation Analysis:

New Validation Studies and Linkage Statistics, in: A.F.J, van

Raan, AJ. Nederhoff and H.F. Moed (Editors), Science Indicators: Their Use in Science Policy and Their Role in Science Studies,

(DSWOPress, The Netherlands) pp. 1 4- 1 6.

Narin, R, G. Pinski, and H. Gee, 1976, Structure of the Biomedical Literature, Journal of the American Society for Information Science, 27, (1), 25-45.

Rosenberg, N. and L. E.Birdzell, Jr., 1990, Science, technology

and the western miracle. Scientific American 263, (5), 42-54.

Tumey, J., 1991, What drives the engines of innovation?.

New Scientist 40.

Patent and Trademark Office, U.S. Department ofCommerce,

Manual ofPatent Examing Procedures, 6th Edition, 1995, Section

904.02.

Weinberg, A.M., 1963, Criteria for Scientific Choice, Minerva

1 , 2, and 1964, Criteria for Scientific Choice 11: The Two Cultures,

Minerva 3, 3.

707
Comments on Citations from Patents to Papers: A Measure of Public Research Spillover

Into Private Industry

by Sylvia Kraemer'

The passage in 1 993 of the Government Performance and

Results Act (GPRA) put federal Agencies on notice that, beginning with their FY 1999 budget requests, they would have to

project for the Congress what they intended to accomplish in the next year, in concrete and measurable terms, and be held ac- countable for those results. The ramifications of this legislation

for the pervasive culture of the Federal bureaucracy can not be

overestimated.

First, for 30 years both the existence and the funding, of numerous Federal agencies have been the principal means by

which the Congress and the White House have attempted to demonstrate to their various constituencies that they would make

good on their campaign promises. Create an agency, increase its budget — constituencies would accept those "inputs" as proof

that something would be done. Whether something was actually done —whether cancer got cured, poverty eradicated, schools

got better, or the welfare-dependent discovered the joys ofwork —was and remains very difficult to ascertain, much less to measure.Secondly, the GPRA put Federal R&D agencies on special notice that they, too, would have to demonstrate returns on the

public's investment in research and development. However, unlike the agencies whose mission has been to somehow aher or improve the behavior ofmillions ofAmericans, science agencies

have had a ready-made ideology, the constant incantation of

which might seem to relieve them of the need to measure the

returns on Federal funding for research. That ideology, ofcourse, was best captured in Vannevar Bush's 1945 report. Science —

The Endless Frontier. Bush maintained that science was the bedrock of technological progress and its associated economic

returns. Science —at least, basic science —was done largely in university laboratories. Thus the Bush paradigm attracted a powerful constituency that combined the friends of large research

universities like M.I.T. with the thousands of individual and in- stitutional beneficiaries of the post-WW II "GI bill" education

benefits.

' Dr. Sylvia D. Kraemer is Director of Special Studies Office of

Policy and Plans NASA.

^ See, for example, Daniel Sarewitz, Frontiers ofIllusion: Science,

Technology, and the Politics ofProgress (Philadelphia, PA: Temple

University Press, 1996).

' Schmookler selected his 934 inventions on the basis of their descriptions in technical and trade journals and technological and economic histories. Jacob Schmookler, Invention and Economic Growth

(Cambridge, MA: Harvard University Press, 1966), pp. 31-32.

* These studies deemphasized the importance of scientific research when compared to markets, business and research management, financial

incentives, communication, and simple chance. See, for examples, Richard

R. Nelson, 'The Economics of Invention: A Survey of the Literature', The Journal ofBusiness, Vol. 32, No. 2 (April 1959), 101-27; Chalmers

A measure ofthe power ofthe higher-education and university research constituencies has been the persistence ofthe Bush

paradigm in our discussions ofR&D investment and policy, not- withstanding that it was seriously undermined in the early 1 960's and continues to be challenged as an ideology that has outlived

its tune^. Bush's argument rested on two assertions. One was

the quintessentially American equation oftechnology with wealth

and happiness. The other was the view, traceable to antiquity,

that those who work with their minds operate at a higher level than those who work with their hands. The modem variant of

this view is that basic science is the font oftechnological progress. The first of these two assertions took an almost fatal beating

fi-om the human catastrophe that was the bombing of Hiroshima

and Nagasaki. The second assertion was also seriously chal- lenged when the economist Jacob Schmookler tried to identify the inspiration behind 934 "important inventions" made during

the period 1 800- 1 957. Schmookler found that: In the majority of cases, no stimulus to the mvention was identified. But in a significant minority ofcases, the stimulus is identified, and for ahnost all of these that stimulus is a technical problem or opportunity conceived

by the inventor largely in economic terms. . . . When the inventions themselves are examined in their historical context, in most instances either the inventions contain no identifiable scientific component, or the science that they embody is at least twenty years old.^ Schmookler was by no means alone with his fmding. Indeed,

there was no shortage during the 1960's and 1970's of studies questioning the necessity of basic research to technological

progress as a general rule, mcluding studies sponsored by such

"mainstream" research organizations as the National Science

Foundation and the Department of Defense." The emergence of

the History of Technology as an independent discipline during

the 1960's gave eloquent testimony to the growing body of re- search demonstrating that, while basic science sometimes contributed intellectual content to new technologies, that content

W. Sherwin and Raymond S. Isenson, 'Project Hindsight: A Defense Department Study of the Utility of Research', Science, Vol. 156 (23 June 1967), 1571-77; James M. Utterback, 'Innovation in Industry and

the Diffusion of Technology', Science, Vol. 183 (15 February 1974), 620-26; Illinois Institute ofTechnology Research Institute, Technology

in Retrospect and Critical Events in Science, Report to the National

Science Foundation (Chicago: Illinois Institute ofTechnology Research

Institute, 1968); John Walsh, 'Scientific Opportunities Syndrome:

Invoking the British Experience', Science, Vol. 190 (24 October 1975), 364-66; and Alan G. Mencher and Michael Beesley, 'Lessons for American Policymakers From the British Labour Government's 1964- 1970 Experience in Applying Technology to Economic Objectives',

reprinted in the Congressional Record (6 November 1975), 35428-30.
METRICS OF EVALUATING PUBLIC R&D PROGRAMS

was not necessarily essential to the emergence and success of new technology. Put another way, a profitable investment portfoUo for technological innovation could contain many things

besides basic research performed on university campuses.

However, the question of public support for basic research

continues, especially in a world of shrinking Federal budgets.

And the issue continues typically to be focused more on "how

much" support (as in dollars and percentages) should we give,

rather than how most effectively can we give it? Which brings

us to the importance of the work being done by Francis Narin

and CHI Research, Inc. As constituencies of Federal science agencies struggle to come to grips with the requirements of "The Results Acf, it becomes apparent how dependent the discussion about the importance of basic science to technology has been on anecdotal

evidence. Comprehensive, systematically collected and objectively verifiable data describing productivity in government re- search and invention is limited, to the best of my knowledge, to patent data and the Science Citation Index (SCI). Making sense ofthis data requires painstakmg work, and, before the availability of fast computers, was practically impossible. Thus, we owe a

large debt to Dr. Narin and his colleagues at CHI, Inc. Anyone

purporting to comment on the productivity of research or innovation in the United States must now be familiar with their detailed analyses of massive amounts of patent and citation data. To appreciate the magnitude of this task, keep in mind that between 1976 and 1996 the number ofpatents issued by the U.S.

Patent and Trademark Office (USPTO) to U.S. entities alone,

annually, increased steadily from 37 thousand to 55 thousand.

(One should note that these figures represented only between

41% and 45% of all patents issued by the USPTO annually.)

During 1989-1 996, the period examined by Narin, the USPTO is- sued well over 750,000 patents in all. All of these patents have references to other patents ("prior

art") and to research papers which are typically supplied by patent

applicants, patent attorneys, and patent examiners. Narin has

examined more than 750,000 [sic] ofthe "non-patenf references appearing on the front pages of these patents, determining that about 55% of these non-patent references were to science journal papers, scientific meetings, and other science publications.

About 75% of these references were to papers published in the SCI's roughly 4,000 journals, and about 75% of these citations were "complete and accurate enough to match to the cited paper. Thus 41%) of the non-patent references' are to papers appearing in the SCI. Or, roughly 41%) ofU.S. patents issued during

1989-1996 cite at least one SCI science reference. On the basis of

these citations Narin concludes that there is a significant "link-

' Or, 750,000 x 55% - 412,500 x 75% = 309,375, or 41% of 750,000.

age" between science and patented inventions, and the heart of

his paper describes the salient characteristics of these linkages.

From this Narin makes a courageous leap to the conclusion that these linkages are "indicative of a rapidly increasing dependence

of patented technology upon contemporary science". This conclusion, however, is premature. To begin with, "dependence" is a powerful causal statement, and statistical link- ages tell us only about numerical correlations, not causal rela- tionships among the items being enumerated. Such statements

require an understanding of the dynamics, or underlying processes, that the relationships share. Because analyses of patent

and citation data can play such an important role in our ability to understand the processes of invention, we should fiilly exploit

this data to help us arrive at more persuasive generalizations

about the causal relationships between invention and its many

sources —sources that include economic incentive, exposure to a rich envkonment of informed and creative discussion, ample

resources and facilities, and so on. A good place to start might be with the other references —

those references to other patents and "prior art". Let's agree, for the time being, that a citation "linkage" on the face of a patent

constitutes a declaration of dependence (so to speak). If the 750,000 non-patent references in patents issued 1989-1996 repre- sent only an average of 1.5 non-patent references per patent, while the same number of patents during the same period have

an average of 8.5 references to U.S. and foreign patents per patent

issued, does this constitute evidence of "significant spillover

effects of publicly supported research onto industrial technol- ogy?" I think not. However, building on Dr. Narin's demonstration ofthe usefiilness ofpatent and science citation data, it should

prove even more valuable, for our understanding of where in- ventions come from, to analyze the much larger number ofpatent references per patent.

Secondly, I am certainly not the fu^st —nor will I be the last — to point out that neither patents nor science citations have

uniform value. Twice the number of patents does not give us

twice the number ofusefiil or commercially profitable inventions. Twice the number of science citations does not give us a twofold increase in our understanding of a natural phenomenon. At

the very most, numbers of patents, numbers of citations, and

numbers of references, give us indicators of productivity. And

thus all ofus who collect and manipulate patent and similar data need to be extremely cautious about the kinds ofgeneralizations we draw from our scrutiny ofthat data. Finally, while there are limits to what we can learn by counting patents, science citations, and patent references, they can

serve as extremely useful "markers" to help us design qualitative

inquiries into the nature of innovation and the dissemination of

109
ATP PAPERS AND PROCEEDINGS

scientific understanding. For example, an analysis of the patent

classes to which NASA's 2,385 patents were assigned during

1976-96 reveals a much greater heterogeneity of invention than

most of us, I thmk, would expect. Of those 2,385 patents, only

132 were assigned to "Aeronautics". If we ask, in what patent

classes was NASA assigned three or more patents, we find that three or more NASA patents were assigned to as many as 58

different patent classes. Our own clustering of those same pat- ents into technology application clusters (rather than industry

product clusters) shows a broad distribution ofinvention in NASA

across many diverse technologies, from mechanical devices to biology and human medicine, as well as from electrical, light, and

nuclear energy to motors pumps and engines. The largest such

cluster consists of five patent classes in the general area of measuring and testing devices, to which 282 NASA patents were

assigned.

Because patent data gives us a comprehensive catalogue of

inventions — comparable, let's say, to a telephone book — we

can do reasonably reliable sampling. Randomly sampled patented

inventions are a mountain full of precious metals waiting to be

mined. Studies of randomly sampled (rather than anecdotally

sampled) cases of patented inventions will enable us to form

reliable generalizations about who invents, under what circumstances, with what results. Combined with licensmg and commercialization information, we may begin to understand, with

much greater certainty than before, where the greatest potential

for R&D investment return lies waiting, whether we are investing

40 1 (k) plans, our firm's capital, or the Federal budget. My guess

is that we will find an ancient truism to be true: We will want to carry our eggs in several baskets, of which fundamental science

will be only one.

110
A Quality-adjusted Cost Index for Estimating Future Consumer Surplus from Innovation:

A Case Study for the Advanced Technology Program

by David Austin and Molly Macauley

Introduction

This paper describes a method for estimating future con- sumer surplus from planned new product iimovations. Such a

capability is attractive to policy-makers and govenmient agencies involved in supporting technology research and development (R&D), both in the private and public sectors.^ We have

used an earlier version of this model to estimate taxpayer benefits from a program to develop new technologies for space sci- ence, ' and are deploying a newer version ofthe model in a study

of digital data storage technologies supported by the Advanced

Technology Program (ATP) ofthe Department of Commerce. For ATP, and other programs that invest in R&D, identifying con- sumer benefits from these taxpayer-supported investments is a

natural way to account for an important element of their perfor- mance.The Advanced Technology Program seeks to generate benefits from iimovations enabling new medical, communications,

computing, and other services, as well as from the improved

price-performance characteristics ofthose innovations for exist- ing services. It is the benefits for existing services that are the focus of our analysis. The new technologies are claimed to have

leapfrogged beyond current best practice. Therefore we assess

their benefits with respect to a baseline of existing, state-of-theart technologies that are themselves continually being improved.

We express our findings in terms of the economic value to con- sumers ofresulting quality improvements, compared to advances

that would have been expected to occur in the absence of the program. Our focus on input performance will not capture all of the

potential consumer benefits from government-sponsored R&D.

These include new services enabled by ATP's advanced tech- nology investments, which are an unportant motivating factor in the creation of that program. ATP investments will also create

' Dr. Molly K. Macauley is a Senior Fellow at Resources for the

Future. Dr. David H. Austin is a Fellow in the Quality ofthe Environment

Division, Resources for the Future.

^ This method supports much of what the Government Performance

and Results Act (GPRA) requires of government agencies.

^ The primary "consumers" ofspace science is actually space scientists, rather than ordinary taxpayers. Since NASA's agency ultimate

responsibility is to taxpayers, however, the distinction is only apparent. The point is these are benefits enjoyed by users rather than producers

ofa technology.

unmeasured private benefits, appropriated by the innovator or enjoyed by other firms in the form ofknowledge spillovers. The

public benefit from the new services is potentially enormous, and may result in increases in demand for the new technologies

compared to existing ones. Prospective estimation ofthis benefit

is problematic, however. More so than for improvements in price

and input performance, forecasting the growth in demand due to quality improvements in the outputs (e.g., true video-on-demand,

or virtual real-estate touring) is fraught with uncertainty. Measuring benefits ofnew services calls for predictions ofthe market's response to the new products, and the analyst is not always on

very firm ground in doing this." All types ofbenefits, public and private, are of equal importance from an economic efficiency perspective. However, it may

be sufficient to assess public benefits, just from improvements in the delivery of existing services.' These benefits are an important goal ofATP, and may suffice to demonstrate favorable agency

performance. In fact, were its investments to yield only private

benefits ATP would not succeed in producing market spillovers.*

To account fully for net benefits, the development and opportunity costs of achieving the benefits should also be accounted

for. Full cost estimates would be problematic if the technology

has akeady undergone significant initial development before ATP

got involved. In any case, cost estimation goes beyond the scope of this study. Our analysis provides a rigorous and defensible

estimate of one important type of likely fiiture benefits, those accrumg directly to consumers in the form ofincreased service at lower cost. This approach requires, along with forecasts of demand

changes resulting from quality changes, estimates of shadow

values, or consumer willingness to pay for those quality changes.

We use hedonic econometric techniques to estimate values for unprovements in the most important ofa technology's "generic"

performance dunensions. By way of example, for ATP's digital

By contrast, forecasting demand for new pharmaceuticals, for example, can be simply a matter of knowing the size of the population

affected by the condition addressed by the new drug.

' This formulation is intended to include even many new technologies

sponsored by ATP. In the archetypical example of ATP's flat-panel display project, new display screens would be coupled with existing

computing services, for example.

' It may produce knowledge spillovers, benefits enjoyed by other innovating firms.
ATP PAPERS AND PROCEEDINGS

data storage program, we estimate the shadow values ofthe new

tape units' improved storage capacity, data rate, and file access

time. Hedonics involves explaining changes in price by examining changes in performance capabilities over time. Consumers'

implicit willingness to pay for these quality changes are esti- mated as the relationship between each quality dimension and

the product's market price.

As already noted, our approach is only to measure returns from individual projects, exclusive ofR&D costs, although these could be accounted for with some extensions to the model. A full accounting of ATP's performance would, in addition to taking

such costs into account, examine its record over a portfolio of

investments. Because returns are an ex post measure, a finding

of negative expected consumer surplus would not necessarily

indicate unsatisfactory agency performance. All investments

have, ex ante, a risk of failure or ofunder-performance. Indeed, a number of ATP investments have failed to reach fruition. It is only in the context of the collective returns to the program, and

the market spillovers it has created, that the investment program

can be judged.

In future research, we plan to extend our model to accommodate a portfolio approach to R&D investment. With slight modifications, our approach can be adapted for use in planning pri- vate R&D investment strategies as well. At the planning stage,

this model can be used to assess the potential of proposed R&D

investments before committing to them.

Background

We employ a cost-mdex approach pioneered in Bresnahan

(1986). Bresnahan estimated the consumer surplus from advances

in general piupose, mainframe computers between 1958 and 1972. He demonstrated the applicability of a cost-index approach

(Caves, et al. 1 982) to measuring consumer surplus using changes

in quality-adjusted prices of new technologies. Under certain key assumptions this approach permits the estimation of the

relevant area under the demand curve for the new technology,

without having to estimate the demand curve by itself This obviates the need for econometric estimation and, in particular, makes

it possible to perform the estimation in sectors for which output

quantity and quality-adjusted output price are unobservable. As

' These are, as Bresnahan ( 1 986) notes, "the downstream sectors

such as services, government, health care, etc.—[which] lack sensible measures of real output." This is a particular problem for measuring

digital data storage benefits ofthe high-end, high-capacity tape "library"

systems that are our focus, and where the using sectors tend to produce

outputs with unobservable quality.

' The other "key assumption" necessary for this method's accuracy

is that the price index is correct. No index perfectly satisfies all of the

tenets of demand theory or conforms to all ofthe desirable properties of

Bresnahan (1986) and Griliches (1979) point out, this unobservability tends to characterize sectors in which the benefits from important technological advances have been realized.' The ability of the cost-index approach to return an estimate

of consumer surplus depends on the structure ofthe downstream

market in which the technology is applied.* Demand for the kinds

ofnew technologies sponsored by ATP typically has been mediated by producers using the innovations in their production processes. It is this derived demand, in the satisfaction of demand

for output services, that is the focus of the cost-index method.'

As long as the downstream market is competitive, or is a govern- ment agency such as NASA or the Department ofDefence, then

the producer acts as an agent for consumers when it uses an upsfream technology. This agency aligns the producer's profit

maximization with consumer's expenditure minimization, which

renders the area under the derived-demand curve, calculated by

the cost-index method, a measure of consumer surplus.

Ifthe using market is concentrated, the index would measure what is being maximized by the producer, /. e., profit, rather than

total (producer and consumer) surplus. When downsfreeim pro- ducers have market power, the derived demand curve for the innovation is shifted inward relative to the competitive case. This is not a difficulty with the method, since the appropriate

quantity to measure would be this consumer surplus given the producer's market power. Rather, as Bresnahan (1986, p. 745)

shows, in a concenfrated market, integration of the area under

the demand curve, which the index approach is implicitly doing,

yields a measure that understates total surplus by some positive amount. Heuristically, since even monopoly producers typically cannot appropriate all consumer surplus, the resulting measure

of profit undercounts total benefits.'" Bresnahan applied this methodology to the fmancial ser- vices sector (FSS), one of the leading users of mainframe computers at that time. The fmal output, financial services, is not observable in terms of quantity and quality-adjusted price. That

the FSS could be freated as highly competitive allowed it to be

freated as an agent for the end consumer, particularly with re- spect to the purchase of computers. The derived demand for computers, as an intermediate good in the provision of financial

services, is mediated through fmal demand for those services. The market for financial services was, at that time, a competitive

index numbers, such as transitivity, scaleability, and so forth. The index

selected by Bresnahan and used in this paper satisfies most ofthe more

important properties. See Diewert and Nakamura (1993).

' The notion is that the innovation is an intermediate good, and the

services it provides (refrigeration, space science, data storage) are the

final good. Demand for the innovation is derived from the demand for

final goods. Note we—and Bresnahan—make no assumptions about potential

gains from a competitive market structure in the innovating sector. The

cost index approach measures only the expected actual gains.

112
METRICS OF EVALUATING PUBLIC R&D PROGRAMS

one, which allows Bresnahan to treat the derived demand as if it was generated directly by consumers of those services. Therefore

the area under the derived demand curve provides a measure of consumer surplus from the development of these computers.

Bresnahan 's approach was retrospective: he applied his model to past innovations. Our contribution is to apply this methodology prospectively, to innovations that have not yet reached

the market. We developed our approach in our prior research on

space technologies—propulsion, communication, solar energy, imaging, and navigation systems—selected for trial and flight validation under the auspices ofNASA's New Milleimium Pro- gram (NMP). NASA is both the consumer of these technologies—literally, and as agent for the taxpaying public—and the producer of the downstream product, space science. The downstream market in this case provides a public good, and the con- sumer "agency" requirement is satisfied, not by a competitive

market structure but literally by NASA's being a government

agency. The technologies of the NMP are scheduled for imminent

launch on the first mission ofthe program. Deep Space I, set for October 1998. As such, their performance and quality-adjusted

prices are already fairly well understood and provide a reasonable basis on which to predict likely values and uncertainties for several years out beyond their initial use. The ATP technologies

are not as far along, so our evaluation involves predicting con- sumer surplus on the basis of what we currently know about

these technologies.

Our data comprise the stated expectations ofengineers, product managers, technologists, and other persons familiar with the

innovation. For each technology we elicit their beliefs concerning the most likely values of current and near-future "off-the- shelf prices," performance—in each of several dimensions, the

size ofthe market, and the rate ofmarket acceptance. Our analysis compares these data against the same attributes of the best and most comparable of existing technologies. The change in consumer well-being resulting from each innovation is captured

by a Tomqvist cost index that estimates consumers' hypothetical willingness to pay for them in a "counterfactual" world in which they were not invented. We do not attempt a comprehensive accounting of con- sumer surplus in all markets in which the technology is to be

used. Bresnahan focused on the major downstream marketfor

mainframe computers, and our purpose is to identify the most

important market or markets for our technologies. In this way

'

' As already noted, these do not include development costs. Many of

the space technologies, and presumably the ATP technologies as well, have been under development in one form or another for many years, and a full accounting oftheir development costs over the years probably

would be impossible to achieve.

we will capture a representative, ifnot dominating, portion of

total consumer spillover, while avoiding details about the tech- nologies ' penetration in minor markets. Because our data consist of expectations about the fiiture, our explicitly incorporating uncertainty, and making specific para- mefric assumptions, is a critical distinguishmg feature of this analysis. Bresnahan used single-point values for expenditure

shares and costs. Although those numbers are no doubt ap- proximations and probably reflect some accounting error, his data do not support an analysis of uncertainty beyond a sensitivity

analysis over a pair ofdivergent, quality-adjusted computer price

indices. In this way Bresnahan bounds the estimates. By contrast, since our technologies have no record of performance at

all, we are obtaining from our subjects both their point estimates

of expected values and then- associated uncertainties. We can

therefore express these data in probabilistic terms and implement

the cost-index calculation in a decision-modelling framework. In

this setting, analysis of the relative influence of each input, and

ofthe joint implications ofthe many assumptions that inform the

experts' forecasts, is straightforward.

The result is a flexible model which simulates the empirical

probability density of consumer surplus outcomes implied by

the input uncertainties. The structure of the model eases the

tasks of isolating the inputs that most drive the uncertainty in the results, and analyzing the sensitivity of mean output values

to fluctuations in the values of the inputs. As noted earlier, be- cause it combines uncertainties and expected values across a range of performance attributes and adoption rates, this modelling framework may prove particularly valuable to R&D plarmers,

public and private, whose analytical methods have to now been more piecemeal.'^

The chief contribution of this prospective cost index ap- proach is it examines in a proper context the implications of as- sumptions—about costs, performance, price, and sales—that are made in planning an R&D project. To be sure, some type of

projection ofexpected returns is a commonplace among iimovators. Within this unified framework, however, point estimates

and uncertainties can all be modelled simultaneously, and compared to expectations about best-available technologies. The

result is a flexible, experimental platform upon which can be con- ducted "what if sensitivity analyses that can provide a fuller picture ofthe likelihood of different levels of success or failure, in a way the conventional one-dimensional analyses are likely to miss.

In our analysis we perform, for each model input, a sensitivity

analysis—^the effects of changing input levels on mean consumer surplus. We also carry out an importance analysis for each input, or the effect of

input uncertainty on uncertainty in the estimate of consumer surplus.

113
ATP PAPERS AND PROCEEDINGS

Model

The cost index indicates how much more expensive an

equivalent level of services would have been in the absence of

the new technology." We use the index to compare utility in the expected world of data services employing ATP innovations and

a "defender-technology" (DT) world using best-available, non- ATP technologies. For instance, we compare a high-capacity,

high-density LINEAR SCAN tape library to a currently-available

line ofhelical-scan-based technologies with lower-densities. The

performance of the ATP technologies is intended to leapfrog

conventional technology capabilities. We assume innovation

continues over time in both technologies, but that DT innovations come at a slower pace because the technologies have been

available for awhile. The cost index, multiplied by the share oftotal expenditures

devoted to the technology, gives the consumer surplus, in dollars, resulting from the outward shift in the technology supply

curve. This shift represents mcreases in output that can be supplied at a given price, because the technology has reduced costs. The defender technology's supply curve will also shift outward

over time. As long as the initial shift in the ATP technology's

supply curve is larger than that of the DT curve, the cost index

will be greater than unity. Ignoring the cost of the R&D, the measured consumer surplus would indicate how much better off taxpayers are than in the absence of ATP. Where the shift in the DT supply curve is greater, the index will be less than one and

consumers will be worse offthan if the ATP technology had not been adopted.

Figure 1 illustrates the consumer surplus from the ATP iimovation, or consumers' willingness to pay to move from the de- fender technology to the expected iimovation. The shift in the ATP-technology supply curve can be due to a combination of

cost reductions and quality improvements. The purpose of a

quality-adjusted cost index is to account for both.

Cost Index Formula

Following Bresnahan, we use the Tomqvist cost index pre- sented in Caves etal. (1982). This index is the arithmetic mean of

two monetized measures of change in consumer utility. These

measures, a pair of Konus cost indices, are ratios of "minimum

Bresnahan calculates a "cost of living and of providing financial services" index, based on computer expenditures for financial services

as a share of the total personal consumption expenditures (PCE). If quality-adjustments and expenditures on digital data storage are too

small a share of PCE, we would base the cost index on expenditures as

a share of sector expenditures only. We do this for the space technologies

to distinguish the cost index from unity. Since NASA serves the space

Figure 1. Derived Demandfor New Technologies:

Illustration ofNet Surplus Change.

q:" q Qo'qo.q.""' q

Since NASA serves the space science community as well as the

public, an index ofthe "cost ofproviding space science and

ofmaking propulsion systems " is relevant.

expenditure fimctions" for achieving two given levels of con- sumer utility. For consumers facing quality-adjusted prices Wor W' for the technology mputs'" to production in the adopting

sector (e.g., tape backup as an input to provision of information

services), and aggregate prices P' for everything else in their consumption bundle, C''' in the expression below is the relative cost of achieving utility i/' in the non-ATP world, compared to

its cost given /, the ATP iimovation. Similarly, Cgives the rela- tive cost of achieving utility i/ given ATP innovation / rather than the DT:

*dt _ E (u ,p ,W ) *i _ E (u ,p ,W ) C — * fit J I ^ ~ * 1 1 f E{u\py) E{u\p,w')

Utility u!" and u! are the best an optimizing consumer will achieve given, respectively, DT or ATP technology. Quality ad- justments to prices W are expressed relative to baseline quality

for the defender technology. Prices P are assumed to be the same

under both regimes, which simplifies the problem considerably.'^

Economists term C'"^' consumers' compensating variation, the relative income requu^ed to keep consumer utility constant following a change in price. After a change in W, C** is the rela- tive gain or loss in income that would just compensate consumers so they are indifferent between DT and the / innovation.

science community as well as the public, an index of the "cost of

providing space science and of making propulsion systems" is relevant. Expenditure functions normally take output prices and utility levels as their arguments. The assumption of perfect competition or agency

allows substitution of input prices for output prices. Only the very weak assumption that substitution of /for dt does not

affect general price levels is required.

114
METRICS OF EVALUATING PUBLIC R&D PROGRAMS

Similarly, Cis called the equivalent variation, the amount

DT consimiers would be willing to pay, in terms of relative in- come, to achieve the optimum utility levels available in an ATP

world. Both C'" and C'will be less than unity ifthe ATP innovation is inferior, on a quality-adjusted price basis, to what will be

available at the same time from the DT technology. The indices

will be greater than one if the ATP technology is superior.

As measures of consumer surplus, both C and C'"^' have

advantages and disadvantages, and neither is ideal for all applications.'^ The Tomqvist index is a composite measure that gives

each equal weight, by averaging the two m logarithms. As is well known from the theory of index numbers, no single index satisfies all "desirable" properties or tests (e.g., tests related to

scalability, transitivity, symmetry, proportionality). The Tomqvist

index satisfies many ofthe tests (see Diewert and Nakamura, 1 993).

We assume, following Caves, et al. and Bresnahan, that consumer expenditures E' can be represented by a translog functional form.''' In this case the Tomqvist index simplifies to:

}{ ln(c X C )= j4 ^'dt + ^/ )

• In KWiJ

(1)

The s- denote factor expenditure shares for the defender

technology dt or innovation /, as a fraction oftotal expenditures

in some appropriate setting. For NMP, we used the U.S. space

science budget as our denominator, for reasons already described. Beyond swapping observable input price and quality for unobservable output price and quantity, an additional advantage of

this approach is that, because the two "time periods" are con- temporaneous, prices and expenditure shares for "other" goods,

and quality-adjusted prices for other inputs in the adoptmg sector, are unchanging and cancel out of the equation.

Changes in relative prices can lead to changes in the mix of

input factors in production of final output, and in the demand for

that final output. Bresnahan points out that the translog fiinc- tional form places no restrictions on elasticities of substitution between the new technology and other factors, or on the income

or price elasticities of demand for the final good. Moreover, the

translog allows for arbitrary shifts in demand, say, (in the case of

space exploration) due to technical progress in unrelated computer technologies, or from taste-driven changes, not attribut- able to ATP, in the budget for computer technologies, as long as demand elasticities are unaffected. '* We do not believe this latter difficulty impinges on our analysis of ATP case studies in the near term. We restrict our attention to the short- to medium-term

See Varian (1992) for details. " The translog, aside from having desirable properties exploited

by this procedure, is a flexible functional form, able to approximate well many production and expenditure functions.

future of 5-8 years. To the extent this issue is a concern, later years should be increasingly discounted.

The cost index describes how much higher (or possibly lower)

costs would have been in the absence ofthe irmovation. For any

relative price ^dt/ the index will be closer to unity the smaller /Wj

the share of the total budget (or of total private consumption

expenditures) is spent on the technology. The index for an innovation that offers only small savings over the defender technol- ogy, but which is a significant share of total expenditure, would

be larger.

Inputs and Output

The inputs to the model include the elements of our cost

indices, as well as expectations about changes in these elements over time, and the adoption rates of the new technologies. We

assume these change over time in a way that tends to increase consumer surplus. We incorporate expectations about leamingby-doing and returns to further R&D that tend to favor the new

innovation over the existing defender technology, which we treat as already having benefited from lower-cost improvements. We

also project the adoption of the new technology over the exist- ing DT as a monotonically increasing fiinction of time. Finally, we discount fiiture price expectations to present-value terms,

which favors neither technology in particular. Expected benefits

reflect off-the-shelf prices (and quality adjustments) but not the

cost of the R&D, and the shape of the final density function is determined by our assumptions about the uncertainties in the model.

Simulating the model a large number oftimes forms an empirical density fianction, which is the output of the model. This

probability density fiinction for consumer surplus receives contributions from each ofthe uncertain inputs, which all have their ovm specified probability densities. In each iteration ofthe model, we sample independently from each input distribution, and combine the values according to the cost-index expression (1). The

final density function comprises the outcomes of the individual

simulations.

Market Size and Expenditure Share

In the application of our model to the ATP case studies, the expenditure shares s, in the cost index formula refer to the total outlay on digital data storage devices by sectors using those de-

" This paragraph paraphrases remarks in Bresnahan (1986) p. 751.

115
ATP PAPERS AND PROCEEDINGS

vices to produce final outputs. To calculate these numbers it is necessary to know both the expected sales, in dollars, of digital data storage devices—for the defender and the innovator; and the

size ofthe downstream industry buying the devices. In Bresnahan's

application, he further divides the technology expenditure share by

the fraction oftotal private consumption expenditures (PCE) spent on fmancial services. This step is not strictly necessary, and will be

troublesome for small expenditure shares." Carrying out that divi- sion produces a "cost of living and ofmaking computers" index in the Bresnahan paper, where the fmancial services sector is treated

as buying computers "for" consumers under the assumption that price-cost margins had been competed away in financial services. In the case of digital data storage, the "using" sector is the information services sector. Digital data storage devices are pur- chased to produce data back-up services, as part of a larger set of

computer network services. Since this sector is also relatively competitive, it is possible to employ Bresnahan's device oftreating the information sector as purchasing digital data storage devices, and

providing information services, "for" consumers. In other words,

because market-power distortions may be small in this sector, it is

fair to think offirms acting as "agents" for consumers, as Bresnahan assumes in his paper. Were this assumption to be faulty, the result would be an wwdferstatement of potential consumer surplus. Because ofthis agency, one can speak ofconsumers' "cost of

living and ofmaking digital data storage devices" (COLAMDDSD).

This is the result of a calculation where the share of expenditures on digital data storage devices in the information services sector is

itself divided by the share ofPCE spent on information services. If the share ofPCE expenditures on information services is too small, or digital data storage expenditures as a share of the information

services sector, the expenditure shares 5. will be dominated by rounding error.^° In this case it is reasonable to calculate a "cost ofproviding information services and of making digital data storage devices" index. This differs from the COLAMDDSD index in that the shares have not been divided through by PCE. The result will be an

index that is farther from unity, and which in conversion to dollars is to be applied to information services expenditures rather than PCE.

Dynamic Trends in Prices and Adoption ofNew Technology

All of the adjustments and uncertainties we have presented

take place in a dynamic setting. The information we have sought

from the technology experts concerns the expected timing and characteristics of the new technology as it is expected to be when

" Small shares will be dominated by rounding error.

^" In the extreme, expenditures on digital data storage devices would

represent a zero share of PCE and the COLAMDDSD index would

equal unity.

initially introduced. However, we are also concerned with rates of

change in expected values, over the following several years. Past experience shows technology prices, for a given level of quality, decline rather markedly. At the same time, the fiirther into the future a forecast is made, the more imcertainty there must be about all values.We institute these dynamic elements into the model by putting

time frends on some of the parameter values in the model. We

expect prices to decline over time both for the innovation and for the DT. We also expect that, wherever the innovation offers positive consumer surplus, the DT will gradually lose market share as the new technology is increasingly adopted.

It is natural to assume prices for the new technology will ini- tially decline more rapidly than for the DT, because ofleaming-bydoing effects. In the relatively near term for which we conduct our

analysis, we assume linear price declines. This is a conservative

assumption, especially with respect to the iimovation. We will re- port sensitivities to the unequal price declines, but because we

assume the differences will be small, we do not expect this parameter to affect the results very much.

We assume a variety of formulae for the adoption rate. We

work with the family ofWeibuU distributions, which have a characteristic 'S' shape. Figures 3 and 4 depict two possible adoption

rates. The flatter curve, figure 3 , depicting a more protracted period

of adoption, is the more conservative assumption. We can choose

between faster and slower adoptions on the basis of quality-adjusted price differences. We perform sensitivity analyses on these

functions.

Uncertainty

Our approach to modelling uncertainties is based on the concept of "subjective probabilities." This "Bayesian" approach re- jects the notion that probability necessarily derives from frequencies that would be realized from idealized "infinite sequences" of

repeated outcomes. For repeatable events the contrasting

"frequentist" approach is serviceable: in this view the 50% odds of

"heads" from an unbiased coin-flip is based on the expected out- come of an idealized, infmite sequence of coin tosses. However,

such formulations are not well suited to non-repeatable events, such as probable future costs or adoption rates. In our model, for

instance, cost uncertainty stems more from what is not known about

the technology than from randomness in the data-generating process ofsome idealized, repeated iimovation experiment from which

actual costs will be just one draw.^'

^' This is a subtle but important distinction. Some ofthe uncertainty

in expert opinion does arise from randomness in real manufacturing

costs over time.

116
METRICS OF EVALUATING PUBLIC R&D PROGRAMS

For space technologies, our probability distribution of likely off-the-shelf costs blends the subjective beliefs of technology

experts with our own beliefs about the likely biases of these

experts. For ATP, the uncertainty is less over expected market

price than it is over the tuning and performance of the innovation. In both cases, rather than taking engmeers' guesses at face

value, we assume that they are over-optimistic.^^ From scientists and managers familiar with the technologies, we have solicited opinions about the "most likely" outcomes, as well as about the

full range of possible expected values, based on perceptions of

uncertainties and past outcomes. We use the high and low points

from this range to formulate the expert's prior cost distribution.^ To properly answer questions about uncertainties, the experts must have in mind their own personal subjective probability distribution from which their expectations are drawn. Our in- terview is structured to elicit this type of information. We

operationalize our assumption about optimism by embedding

the expert expectations in a probability distribution that is skewed

to the left (in other words, having a long right-hand tail). We

assume that the actual costs will be distributed asymmetrically

about their expected mean value, with a greater than 50% chance

that costs will exceed expectations. This assumption is based on

a study indicating that ex ante cost expectations for "pioneer

projects" are biased downwards (Quirk and Terasawa, 1986). In

other words, expectations on costs for new technologies tend to be overly optimistic.

Specifically, we take the experts' mean expectation, /7/m5 50%

ofthe experts ' uncertainty parameter, and locate this at the median of a lognormal distribution fiinction.^" In practice, this tends

to place the expert mean around the SO*" percentile ofthe lognormal, with the actual location depending, by construction, on the amount of expressed uncertainty. The greater the ex ante uncertainty, the lower is the percentile which the expert's expectation

will occupy in the density function. This approach identifies only one of the two parameters of

the lognormal family of distributions, the median. We select the

other parameter, the geometric standard deviation, by assuming

a value of 1.5. This value is chosen arbitrarily to achieve the We base this on Quirk and Terasawa ( 1 986) about which more

follows. " There are better ways to elicit uncertainties, and one must be

careful combining expert prior distributions. We address the former

concerns with our sensitivity analysis.

The uncertainty parameter is the spread between best and worst

expected costs. There is no "best" distribution, and our procedure is ad

hoc. We also experimented with a less conservative approach which

places the experts' means precisely at the 30* percentile of a lognormal, and their "worst case" outcome at the 90* percentile. We find that our

estimates of net benefits, and of the uncertainties associated with those

skewed shape we wish to impose. A value close to 1 .0 produces

a symmetric distribution, while values much larger than 1 .5 pro- duce unrealistically high estimates. We subject this parameter to

sensitivity analysis.

Figure 2, below, contrasts this type of distribution function

with the one we employ for experts' forecasts concemmg the defender technology. For the DT, experience and direct observation inform the responses. Since we are asking about expected

future costs, we still are implicitly seeking information from their subjective prior probability disfributions.^'

Figure 2. Simulated Adoption Rate

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Years

The production-cost literature offers little guidance on the form our prior probability density functions should take. We use

the lognormal family to model "pioneer" project cost overruns because it is skewed in the desired direction, and because these curves have simple parametric representations.

Figure 2 contrasts sample cost density functions for the DT

and for the iimovation. The "lognormal" density function for the innovation is asymmetric, reflecting greater uncertainty than what

is known about the DT, as modelled with the normal density

function. If the experts are less over-optimistic than we assume,

so that the lognormal distribution is less skewed than we make it, we will underestimate consumer surplus. Given the prospective

benefits, are rather insensitive to our choice ofapproach, at least among

the distributions we have considered. A technical paraphrase of our approach would be "thinking about the probability distribution of all possible costs for this technology, with your median cost estimate being at the 50th percentile of this distribution, what ("upper bound") cost estimate would be at about the 85th percentile?" In practice, we interpret their upper-bound responses

in several ways. We assume normality for DT; for new technologies the interpretation is fundamentally similar—we seek quantiles of the underlying distribution—but we make stronger assumptions about the location and shape of the expectations. We find that our results are reassuringly insensitive to "reasonable" treatment of the responses.
ATP PAPERS AND PROCEEDINGS

nature of the analysis, though, it is appropriate to make conservative assumptions so as to minimize concerns the findings stem from best-case outcomes.

Figure 3. Simulated Adoption Rate.

15 Years

Quality Adjustment

To put expected prices ofthe DT and new technologies on a

strictly comparable footing, we must account for quality differ- ences in the embodying products. For instance, in the case of

digital data storage devices, these differences include file access

time, data transfer rate, and capacity. These dimensions are not

the only potential sources of quality change, but they are widely

agreed to be the most important.-^^

Consumers are generally willing to pay more for higher per- formance, but it is not immediately obvious how much more they

are willing to pay. To estimate this, we perform a hedonic statis- tical analysis to explain the contribution to market price of each

relevant quality characteristic. The outputs of the model, which

is fitted to price-performance data on existing technologies, are a

set of estimated coefficients—one for each characteristic^—^that can be interpreted as consumer values. These "shadow prices"

represent the average consumer's willingness to pay for incre- mental increases in the various quality dimensions.

To calculate cost indices, we adjust expected unit prices by

netting the estimated values of the expected quality improve- ments out of the forecast final good price.^' The result is the

quality adjusted ^^(the adjustment ofthe new technology prices

To a person, the technology managers we have interviewed in the

digital data storage field agree these are the most important performance

dimensions. " These shadow values enter the model as uncertain parameters, using

estimated standard errors from the hedonic regressions.

relative to the DT) for the cost mdex formula. Where the new

technologies represent a substantial gain over DT, beyond that observed in the data used in the hedonic regression, we subject our estimates to sensitivity analysis.

Finally, new technologies do not necessarily improve in all dimensions. Our model does not preclude a defender technology's

having a smaller real unit price J^^than the real price Wj Where the new technology is adopted it would, in this case, produce negative consumer surplus relative to that available for the DT. Pre- sumably the market would reject the innovation in this case.

Figure 4. Illustration ofCost Under-Estimation

Lognomal

(new technology)

Expert's / \ Normal

best guess / I (defending technology)

(pegged al 1

30th percentile) /

1 /.

Cost ($) Summary

By way of summarizing our discussion of the model, we

present a series of figures and tables to portray important model

elements and outputs. Figure 5 illustrates schematically the rela- tionships between the key model input steps described in this paper. The output ofthe model, an empirical distribution of esti- mated consumer surplus over time, is the result of a sfraightforward calculation carried out in each iteration of the model. It is the product ofthe cost index times total expenditures, either PCE

or in the appropriate sector of the economy. Table 1 displays some of the indices we have estimated for the space technology

case studies.^' Technologies with index values greater than one

will increase consumer surplus, while those with index numbers

less than one would reduce consumer surplus if adopted. The

dollar values for these contributions are given in table 2.

Careftil readers may have noted that one must actually exponentiate

expression (1) to get cost indices such as given in table 1. The index

actually described in the text produces "percent change in surplus" values which are close to zero. One gets virtually the same result by first exponentiating and subtracting 1 , which is the correct calculation and is what we actually do. In the body of the paper we focus on "unitary" index numbers for expository reasons.

118
METRICS OF EVALUATING PUBLIC R&D PROGRAMS

Figure 5. Model Inputs, Intermediate Calculations

Technology

Unit Prices

Quality Adjustments

ReaJIJnit

Prices

UNCERTAINTY:

Probability

Distributions on

Input Parameters

Annual

Expenditures

Demandfor

Technology

Market Penetration

-Tecbnoloigy

ExpeiKiilure

. Sliares. IT

Cost Index

Key:

\2 Input Data Values

Istomediate ;j

I Calculations Net of Defensler Technology

Figure 6. Net Benefit ofNew Space Technology.

We also estimate a "bottom line", both in that study and for

the ATP cases. This is an estimate oftotal expected surplus over

time. In the space study this is expressed as an "effective augmentation" of the public space science budget (since net cost savings on the space technologies can be used elsewhere in the space science program). Figure 8 shows how surplus grows over

time. The shape of the curve depends on the rate of "market

penetration" (or adoption, for space technologies), and on the budget for space missions requiring either the innovation or the DT. Figure 6 also depicts uncertainty as growing over time, a

natural resuh of the model's parameters. This figure illustrates the way the results for the ATP study will be expressed.

Conclusions and Extensions

Our model provides a sound empirical basis for assessing re- turns to investment in new technologies. By taking explicit account of alternative technologies and the fact that innovation proceeds apace in their development—^not just in the new technologies

and by accounting for uncertainties in the timing and quality of

innovation, we derive defensible estimates of expected consumer

surplus.

Our approach also represents a first formalization of the ten- dency towards cost-esthnation bias in new technologies. We think

the explicit treatment ofuncertainty and the modelling ofdefender

Table 1. "Cost-of-Performing-Space-Science-and-of-Producing-New-Technology" Index.

Year New technology #1 New technology #2

0 1 1

1 1.01335 1 2 1.01271 1 3 1.02638 0.99959 4 1.02674 0.99955

5 1.02709 0.99950

Table 2. Benefit of Using the NMP Technology, Relative to Defending Technology (U.S. dollars)

Year New technology #1 New technology #2

0 0 0

1 27,040,000 0

2 27,400,000 0

3 58,550,000 -900,500 4 59,360,000 -1,002,000

5 60,140,000 -1,101,000

119
ATP PAPERS AND PROCEEDINGS

technologies, together with cost-estimation bias accounting, ren- der our model a very useful tool for government agencies and pri- vate firms to use in assessing their funding of new technologies.

We envision several extensions ofour model in future research.

It can be applied to private sector investments in new technologies,

where estimation and valuation of quality improvements can be

helpful in predicting the likelihood ofa new technology' s succeeding in the marketplace. The model can also be adapted to the esti- mation ofprivate returns. With some modification, the model can be used to identify the investment rate and technology selections most likely to yield the highest returns among alternatives. In other words, the model can help designate investment portfolio strate- gies. Such an extension ofthe model would include the "drawing

board" phase of the innovation process, where agencies and pri- vate firms consider optimal investment strategies given competing

opportunities for use of the R&D capital.

Another extension we are planning is to consider technologies that generate externalities, such as ones that confer marginal

social benefits on top of private benefits (for instance, "environmentally friendly" technologies). For this application, we would

include shadow values of social benefits among a technology's

"quality" adjustments.

References

Bresnahan, Tim. 1986. "Measuring the Spillovers from Technical

Hausman, Jerry. 1997. "Cellular Telephone, New Products and

the CPI." NBER Working Paper 5982 (Cambridge, MA: National

Bureau of Economic Research, March).

Quirk, James and Katsuaki Terasawa. 1982. "Divergent Expectations and R&D Expenditures," in James Quirk, Katsuaki Terasawa,

and David Whipple (eds.) Coal Models and Their Use in Government Planning (New York: Praeger).

Quirk, James and Katsuaki Terasawa. 1986. "Sample Selection

and Cost Underestimation Bias in Pioneer Projects," Land Economics vol. 62, no. 2 (May), pp. 192-200.

Terasawa, Katsuaki, James Quirk, and Keith Womer. 1989. "Turbulence, Cost Escalation and Capital Intensity Bias in Defense

Contracting," in Institute of Cost Analysis, National Conference

Cost Analysis Applications of Economics and Operations Research (Springer-Verlag).

Trajtenberg, Manuel. 1990. Economic Analysis ofProduct Innovation: The Case of CT Scanners (Cambridge, MA: Harvard

University Press).

Varian, Hal R. 1992. Microeconomic Analysis. W.W. Norton &

Company, Inc., New York, New York.

Advance: Mainframe Computers in Financial Services," American Economic Review vol. 76, no. 4 (September), pp. 742-755.

Caves, Douglas W., Laurits R. Christensen, and W. Erwin Diewert. 1982. "The Economic Theory of Index Numbers and the Measurement of Input, Output, and Productivity," Econometrica vol. 50, no. 6 (November), pp. 1393-1414.

Chambers, Robert G. 1988. Applied Production Analysis: The

Dual Aproach (Cambridge, UK: Cambridge University Press).

Diewert, W. Erwin and Alice O. Nakamura (eds.). 1993. Essays in Index Number Theory, Vol. 1 (Amsterdam: North-Holland).

Gordon, Robert J. 1990. The Measurement ofDurable Goods

Prices (Chicago: University of Chicago Press).

Griliches, Zvi. 1979. "Issues in Assessing the Contribution of

R&D to Productivit>' Growth," BellJournal ofEconomics vol. 10

(Spring), pp. 92-1 16.

120
Socio-economic Effects of Collaborative R&D - European Experiences

by Luke Georghiou^

Introduction

Since the early 1980s, governments in Europe, singly or collectively, have been supporting collaborative R&D between firms, also with the participation of universities and other research or- ganizations. The prime rationale for this support has always been

to strengthen industrial competitiveness, although other social and policy goals are also cited, notably creation of employment.

These collaborative programmes have been subject to mtensive

scrutiny by a succession of evaluations of all types.

European evaluation practice has been well documented^

sometimes in a wider context. At a national level there has been

considerable diversity driven by differences in the state of scientific development, the organization of science and most importantly fi"om broad practices of governance in the countries concerned.-' Nonetheless there has also been commonality ofexperience, driven in part by participation in pan-European evaluations and methodological programmes, and to some degree a

calling upon teams fi-om other countries to provide the desired

expertise of independence, hi this paper the experience of evaluation at a European level is reviewed and in particular the success, or lack of it, in addressing socio-economic effects. After a

brief review ofthe history and composition ofEurope's two main

instruments for the support of collaborative R&D, the European

Commission's Framework Programmes and the inter-governmental EUREKA Initiative, some issues arising from the evaluation

of these are considered in turn. After a discussion of the type of

outputs and impacts to be expected, the issues of additionality

and policy persistence are considered before conclusions are drawn.

Collaborative R&D in Europe and its Evaluation.

Collaborative R&D Programmes

European collaboration in R&D has a long history, origmating for the European Communities in the nuclear coal and steel

' Luke Georghiou is Professor of Science and Technology Policy and Management at the University of Manchester and Director of PREST

(Policy Research in Engineering, Science and Technology) an institute of

that University.

^ Gibbons M. and Georghiou L., Evaluation ofResearch - A Selection of Current Practices, Paris:OECD, 1987; Meyer-Krahmer F. and

Montigny P., Evaluations of innovation programs in selected European

countries, 7?ejearc/2Po/zcy7S(6):pp 3 13-32, 1989; Ormala E., Nordic

experiences ofthe evaluation oftechnological research and development.

fields and later extending to other areas of science and technol- ogy. However, programmes which aimed to promote inter-firm collaboration (as well as the involvement ofuniversities and public research organizations) as an instrument of industrial policy did not emerge imtil the 1980s. A key development was a 'Round

Table' ofthe then twelve largest European information technol- ogy companies which in 1982 produced a proposal for the pilot

phase of ESPRIT, the first of a succession of IT support

programmes.'' In parallel there were similar national initiatives such as the United Kingdom's Alvey Programme for Advanced

Information Technology. All were justified by the concept of

'pre-competitive research', that is to say shared generic research which would be followed by participants competing in the market. The origins ofthis rationale were explicitly a response to the perceived success of collaborative projects in Japan, notably the VLSI project.' That this perception overestimated the importance

of collaboration and underestimated the degree of internal competition in Japan did not matter; pre-competitiveness provided a convenient label to present to the prevailing conservative ide- ologies in government. Somewhat ironically, one of the most

consistent evaluation findings in the ensuing years was that collaboration between direct competitors was relatively rare, ex- cept in the field of standards, and that complementarity between

the knowledge bases of participants was the driving force for cooperation.

The European Commission proceeded to organize its R&D

promotion activities into a more comprehensive multi-annual plan

with common objectives, to be called the Framework Programme.

This has continued through four iterations with a fifth at an advanced state of preparation. The budget of the Fourth Framework Programme was 13.1 billion ECU. Industrial participants

receive up to 50% funding and academic participants 100% of

their marginal costs. Framework Programmes have been made up

a series of Specific Programmes corresponding to particular objectives or technological areas. They share the overall goal of

strengthening the science and technology base of Community

industry enabling it to become more competitive at the intemaResearch Policy 18 (6) pp.343-59, 1989; CallonM., Laredo P. and Mustar P., La gestion strategique de la recherche et de la technologic,

Paris:Economica, 1995, Georghiou L., Issues in the evaluation of

innovation and technology policy, Evaluation Vol. Np. 1 , January 1 998

^ Georghiou L., Research evaluation in European national science and

technology systems, Research Evaluation Vol.5, No.l, April 1995

•Guzzetti L., A brief history of European Union research policy.

Commission of the European Communities, October 1995, pp.76-83

^ Oakley B. and Owen K., Alvey - Britain 's strategic computing

initiative,Mn, 1989
ATP PAPERS AND PROCEEDINGS

tional level. Since the Treaty on European Union came into force

in 1993 the rationale has extended to the promotion of all re- search actions which fiirther other Community policies. In the

Fifth Framework Programme the objectives are evolving further

to be expressed in terms of socio-economic problem solving.

The second form of European collaborative R&D addressed

in this paper is that supported under the EUREKA Initiative. Though concerned exclusively with civil technologies, this be- gan as a response to the US Strategic Defense Initiative which

had prompted fears ofa new technology gap opening.* EUREKA

has been positioned nearer to the market than the Framework

Programme and is organized on a bottom-up principle, whereby

there are no work plans and industrial participants have wide

scope to propose projects. The central mechanisms ofEUREKA

approve projects for 'labelling' but there is no budget; public

funding of participants is left to national governments which

may use existing schemes, have a dedicated budget, or if they

choose, offer no fiinding. By 1996 EUREKA has approved over 1200 projects involving 5,600 participating organizations. Public

ftinding in recent years has been of the order of 300-500 million

ECU per year, very roughly 20% of project costs. Membership

includes several Eastern European countries and Russia.

Evaluation ofthe Framework Programme

The European Commission's Framework Programmes have

provided a focal point of interest for evaluators from their outset, building upon approaches developed in the late 1970s and early

1 980s for the evaluation of Community R&D programmes.^ The

Framework Programmes have been evaluated in a variety of dif- ferent ways, addressing the Framework as a whole, its component programmes (so-called vertical evaluations of Specific

Programmes), cross-programme issues (horizontal evaluations)

and impacts on individual Member States. Variety has also been

experienced in types of evaluator (panels or evaluation experts),

client groups (the Commission, Member States or the European

Parliament), and in the methods used (principally surveys and

interviews with programme managers and participants, but also extending to bibliometrics, network analysis and economic analysis). In the present era, the system rests principally upon a scheme

introduced in 1995 which consists of continuous monitoring,

Peterson J., 'EUREKA: a historical perspective' in Krige J. and

Guzzzetti L. (edsj History of European Scientific and Technological

Cooperation, Firenze 9-11 Novemmber 1995, Commission of the European Communities 1997

' For historical analyses of the development of evaluation of the Framework Programme see Georghiou L., Assessing the Framework

Programmes - a meta-evaluation, Evaluation Vol. 1, No. 2, October 1995; Olds B.J., Technological Eur-phoria? An analysis ofEuropean

reporting annually, and five-year assessments, carried out midway through programme implementation but including within

the frame ofanalysis the preceding programme. This timing fea- ture has proved the best compromise between the requirement

for results in time for decision-making about the successor programme and allowmg some time for effects to be manifested.

It is self-evident that this is dependent upon continuity from one programme to another and would not work where there are sharp

changes ofstrategy. The system operates for Specific Programmes

in parallel and in theory the results are input to an overall evaluation of the Framework Programme. In the first iteration of the

five year assessments the Framework Programme level evaluation tended to proceed under its own momentum without close connection to the underpinning exercises. Both monitoring and

assessment involve panels of external experts (3-4 in the first case and 6-10 in the second). Monitoring is a relatively light

activity aimed at quick course correction, while the assessments

take a broader view.

The terms of reference are extensive. For the assessments,

criteria cover relevance ofobjectives, efficiency ofoperation and

effectiveness in terms of objectives achievement, "European

added value" and dissemination/exploitation of resuhs. However, in a historical context the present system can be seen as a move away from previous practice of panels taking longer and

being supported by a variety of studies designed to cast light on

a particular programme, and towards an approach where monitoring and standardised surveys replace much ofthe data collec- tion previously devolved to panels. The resulting panel reports

may contain good ideas but they are based more upon impressions than hard data. Much then depends upon the efficacy of

the monitoring system.

Arguments in favour of the current approach rest upon the assumption that the Framework Programme is now a well understood policy instrument from which surprising lessons are unlikely to emerge. In this situation, there could be diminishing

returns in the constant repetition of in-depth evaluations. In- stead the requirement is for a system which allows consistent measurement and comparison with past performance. There is some evidence that the rise ofmonitoring approaches is part of a more general trend towards performance indicators as a complement to or even a substitute for in-depth evaluation. The risk is that performance indicators inevitably reflect a particular view of

Community science and technology programme evaluation reports, Rotterdam: Beliedsstudies Technologie Economie, 1992; Guzzetti L., A

BriefHistory ofEuropean Union Research Policy, Luxembourg: European

Commission, October 1995, pp. 101- 108; and Commission of the European Communities, Second European Report on Science and

Technology Indicators, Luxembourg: European Commission, pp. 562-

585, 1998

122
METRICS OF EVALUATING PUBLIC R&D PROGRAMS

the world and may miss developments which do not conform to

that perspective. For example publication counts may miss the development of new forms of scientific communication. Developments such as new or extended networks, or tacit knowledge,

which are inherently difficuh to measure may pass out of the

analytical framework. Furthermore, there is the well-known risk

that researchers will begin to perform to the indicators rather than to their true objectives, with divergence from the main

objectives and a risk of loss of variety in the system.

Looking at both the past and present evaluation arrange- ments it is clear that success has been mixed. On the positive

side the independence ofthe expert panels has rarely if ever been

challenged and their scientific judgements if predictable have

not been questionable. Programme management has also been

heavily scrutinised, though the main barriers to improvement

have often been more fundamental than those responsible for

the operation of R&D programmes are able to deal with. The

constant lacuna, frequently repeated by the panellists themselves, has been in the ability to engage with the extent of socio- economic effects. There are reasons why a panel-based system

is likely to encounter difficulties in this domain. The very seniority ofpanellists chosen to give a strategic (and legitimated) view

prevents them from becoming engaged in extensive contact with

projects or the firms in which they are located, particularly on the

short time scale of current evaluations (which can be as little as

three months). It is also rare for more than two or three members

to have an economic, evaluation or even industrial R&D management background. The majority are eminent in the relevant

technological field. This creates an expertise gap which was noted

in a high level review ofthe Commission's evaluation practices a few years ago.* This affects the ability of panels to absorb infor- mation which may be provided to them.

Outside the mandated evaluations discussed above, socio- economic impact has been addressed in studies commissioned

through research programmes or by the management of indi- vidual programmes.' Perhaps the highest profile for such works

has been achieved by a series of studies which have calculated an "impact ratio", that is to say an average measure of the eco- nomic effects associated with a programme which gives the potential economic gains as a ratio of the investment of public

' Chabbal R., Organisation ofresearch evaluation in the Commission

ofthe European Communities, EURl 1 545, 1 988

' A further type of evaluation has looked at the impact ofEU R&D on

individual member states. The best of these studies have identified important issues in terms of the organisational response to the programmes, eg. Lar6do P., Gallon M. et al L 'impact de programmes

communitaires sur le tissu scientifique et technique frangais, Paris:La Documentation Franfaise, 1 990; Reger G. and Kuhlmann S., European

technology policy in Germany, Heidelberg:Physica-Verlag, 1995; Luukkonen T. and Niskanen P., Learning through collaboration,

Helsinki:VTT, 1998

resources in the programme. In the best known of these ap- proaches, from the BETA group at the University of Strasbourg,

the calculation of "added value" is based upon both direct ef- fects and four categories of indkect effects (resuhing from learn- ing benefits to the participant arising from the project) and attrib- uted to the research project by asking firms to agree on conservative estimates for coefficients which atfribute a proportion of

commercial effects to the project in question.'" Over a period of

time ratios for the Commission's industrial technologies

programme (BRITE-EURAM) from studies by BETA and others were in the range of4.4 to 7.0." It should be strongly emphasised

that these are not rates ofreturn and could in principle represent

unprofitable investments if the non-R&D expenditure on innovation were sufficiently high. The need for careful interpretation

is clear - freatment of these figures as a sign of spectacular suc- cess for the programme would seriously question the rationale

for fiirther public funding in the light of such demonstrated benefits. While the scale of effects could be disputed, undoubtedly

these methodologies have established the positive role of this type ofcollaborative R&D.

For the future, as noted above, the Fifth Framework

Programme is re-positioning itself away from its predecessors'

original self-styled pre-competitive origins to a position where it

is 'sold' to its political audience as a means by which European

economic and social problems may be solved. This has raised the stakes for evaluation which is now expected to deliver evi- dence ofthe degree to which such objectives are being achieved

in a more concrete way than before and with the added complication of addressing broadly cast social objectives in addition to the previous core goal of improving the competitiveness of European industry. Currently solutions are being sought but evaluation faces a major challenge to meet these expectations.

Evaluation ofEUREKA

The EUREKA Initiative has had a separate tradition ofevaluation from the Framework Programme.'^ The management style

ofthis Initiative is very different, with a very small central secretariat co-ordinating decisions taken by representatives of the

BETA, Economic evaluation ofthe effects of the BRITE-EURAM

Programmes on European Industry, Final Report, BETA, Strasbourg, 1993

'

' Second European Report on Science and Technology Indicators,

Luxembourg: European Commission, pp. 575, 1998 The exception to this was a transfer of expertise when a senior

official from the European Commission's evaluation unit moved to the EUREKA Secretariat. Evaluation culture often depends upon the presence

or absence of key individuals, reflecting perhaps a lack of depth in

institutionalisation.

123
ATP PAPERS AND PROCEEDINGS

Member States who choose whether or not, and by how much, to fund their own nationals' participation in collaborative projects.

Each year the Chairmanship of EUREKA rotates among members, and it is traditional for a country to put forward a programme

of activities for that year. Evaluation has had its roots in such

programmes, in that several countries saw fit to put forward evaluation as a theme for their year. It should also be noted that, given

its embedment in national innovation systems, EUREKA has

been the subject of numerous national evaluation studies. These

are not covered here. The first international exercise took place in

1 990-9 1 under Dutch leadership and was concerned mainly with

procedures."

A much larger scale exercise with a specific emphasis on

economic effects was undertaken in 1992-93 when France, the country which had been pro-active in the launch of EUREKA,

took the chair. Teams fi-om fourteen countries worked together in a commonly defined methodology involving a whole population survey and interviews with multiple participants in a selection of

projects.'''-'' Instruments developed for that evaluation, notably

the questionnau-e, have been replicated in a large number of European evaluations subsequently. Since the great majority of

projects were still in progress at that time, the emphasis of the evaluation was on expected benefits. It was already possible to extract several key lessons for successful irmovation in this context, for example the relative success of vertical as opposed to horizontal collaborations, the correlation between success and

the project being core to the firm's strategy, and the common

failure, particularly among smaller firms, to carry out adequate

advanced market research. The next episode in the history ofEUREKA evaluation came

under the Swiss chair in 1994-95.'^ This was explicitly a follow- up to the French chair exercise, to examine the emerging picture

after completion of a significant tranche of projects. The struc- ture in this case was similar to the earlier panel evaluations ofEC

programmes, though the expertise base was broader in the ab- sence of a single technological domain to be exammed. This

evaluation confirmed the earlier findings but noted some drift in the Initiative away fi-om large strategic initiatives co-ordinating

entire sectors, towards being an instrument principally directed towards small firms. Efforts have since been made to ensure that both fijnctions are maintained. The more longitudinal perspective possible fi-om this exercise also demonstrated that success

is not something which can be defined at a single point in time as

Dekker W. et al, Report of the EUREKA Assessment panel,

EUREKA Secretariat, 1991 Ormala E. et al, Evaluation of EUREKA Industrial and Economic

Effects, EUREKA Secretariat, 1993

the fortunes of projects and participants ebb and flow often for reasons beyond the scope of the Initiative or even of R&D.

This finding, together with a realisation that evaluation

should not be seen as an ad hoc activity, led the following chair country, Belgium to initiate a new approach, which came to be

called Continuous and systematic evaluation. As its name implies, this procedure requires that information is collected sys- tematically every year, by sending a standardised questionnaire

to all projects which finish durmg that year. Continuity comes

both from the flow of information about commercial and employment impacts, and because the approach ensures continuity of

method from year to year.

The evaluation employs three main instruments:

• Final Report: a four page questioimaire sent to participants at the end of the project R&D phase;

• Market Impact Reports: a 2-page questiormaire repeating the parts of the Final Report dealing with commercial exploitation of the R&D results and their employment impacts and sent to participants who have previously recorded commercial effects after one, three and

five years from the end ofthe project;

• Semi-structured interviews: collecting more detailed and qualitative information, normally on projects which were completed three years previously, and also seek- ing to validate the questiormaires.

To give an idea ofthe scale ofthe exercise, in 1997 the analysis was based upon 434 Final Reports, 265 of which were from

firms, the rest from non-industrial partners. This represented re- sponse rates of 79% of projects, 77% of main partners and 25%

of all participants. In addition, 34 Market Impact reports were

received and 30 face-to-face interviews conducted.

The approach is unusual in following up projects some time

after completion, an aspect which has proved particularly re- warding. After a pilot year it was decided to convene an Expert

Advisory Group to oversee the process, and to recommend an "Annual Impact Report" report to the country holding the chair. This report presents the findings, conclusions and reconmiendations and is presented formally to the Ministerial Conference

that governs EUREKA. The need for the expert group arose for several reasons, including the need to interpret data, to have

" Dale A. and Barker K., The evaluation ofEUREKA: a pan-European

collaborative evaluation of a pan-European collaborative technology programme, Research Evaluation 4(2): 66-74

'* Airaghi A. et al, EUREKA Evaluation Report, EUREKA Secretariat, 1995

124
METRICS OF EVALUATING PUBLIC R&D PROGRAMS

suitably qualified people to perform interviews, and to provide

an independent validation of the findings and methodology.

To give some flavour of the results which have emerged,

and which for the purposes of this paper have implications for evaluation practice more generally, some findings from the Annual Impact Report of 1997 are discussed in the following paragraphs.

While 78% ofparticipants were expecting commercialization

at the end ofthe project (and 45% had achieved it by that time), one year later one third ofthese had failed to commercialize. The

conclusion is that initial expectations based upon technological

success may not be realized in the market and hence fmdings are

sensitive to the timing of evaluation. There is a skewed distribution of success, defined in terms ofimpact upon turnover. As Figure 1 shows, just over half ofthe

projects achieved little or no effect (3% oftotal turnover effects) while conversely 2% of projects accounted for 54% ofturnover

generated. This is not untypical for an R&D portfolio but it also shows the necessity for large ifnot comprehensive samples when

a handful of "blockbusters" are so important in assessing the

overall impact.

Interviews have confirmed that routes to commercial success are often more complex than those which can be captured in a questionnaire. Typical examples include firms that have devel- oped new equipment, improved their knowledge base, acquired new skills, changed concepts or market strategy and entered new networks. Even where the original project objectives have

not been achieved some combination of the above has led to

significant economic benefits for the firm, causing it to report a high degree of satisfaction with the Initiative. These participants can be taken more seriously than some m public programmes

because not all received public funding.

Questions on employment effects were often not answered

(a question response rate of c.43%) and interviews showed that participants were uneasy even where they did answer. Answers

received showed that the principal effect had been to safeguard

jobs in the organisation concerned. Other recorded effects were

usually small but positive. Firms were generally even more un- comfortable with the idea of assessing effects outside their organisation. Such effects included jobs created or destroyed as

a result of the application of a product or process. None were

able to volunteer information on displacement effects in competing firms. The reasons for difficulty were generally that employment decisions involved a wider range of considerations. The

only unambiguous cases ofemployment directly associated with

" Georghiou L. et al. The continuous and systematic evaluation of

EUREKA. Annual Impact Report 1997, EUREKA Secretariat, Brussels, 1997

a project came when a new firm, factory or line was established to exploit the output ofthe project. Negative effects often required

qualification; they could be replacing dirty or dangerous jobs

where there was difficulty in fmding labour. Although the evaluation panel is well aware of the difficulties involved in dealing

with the employment issue, there is immense political pressure in Europe for information on this topic and so the attitude has been

that an imperfect measure is better than none if it is treated with

due caution. Figure 1. Impact on Participant 's Turnover ofEUREKA

Projects.

gl Connpany g % Turnover

Source: EUREKA Annual Impact Report 1997.

The continuous and systematic evaluation is itself a learn- ing process, with minor changes introduced each year, within the

constraint of not losing the contmuity and comparability which

it offers. The database ofproject effects is cumulative, gaining in analytical power each year. At present the analysis focuses on

projects finishing in the year in question but as cohorts are completed, more detailed longitudinal analysis will also become feasible.

It is recognised that from time to time fiirther strategic evaluations will be necessary which look at issues beyond the scope ofproject impacts. In the meantime this system provides a mechanism which holds the confidence of its prmcipal client community (the national administrations and politicians) and points them

towards ways in which the commercialization ofresults could be

improved.

Outputs, Impacts and Effects^^

Implicit in the discussion above is the existence of a clear view ofthe menu of outputs and impacts which may arise from a

'* The discussion in this section draws heavily upon Bach L. and Georghiou L., The nature and scope of RTD impact measurement,

discussion paper for International Worksiiop on Measurement of RTD

Results/Impact, Brussels, May 28/29, 1998

125
ATP PAPERS AND PROCEEDINGS

Table 1. Outputs and Effects

iHipaClS/CllcCls — Intermediate prototypes Competitiveness sales outputs technological sub-systems market share demonstrations open up markets models/simulators create new markets

integration of technologies lower cosis tools/techniques/methods faster time to market

intellectual property license income

decisions on further RTD Employment jobs created new products jobs in regions of high unemployment

Products improved products jobs secured new processes jobs lost Processes improved processes Organization formation of new firm new services joint venture to exploit results Services improved services new technological networks/contacts

processes for delivering new new market networks/contacts

services improved capacity to absorb knowledge

de facto standard core competence improvement

OJ I ] — Standards de jure standard further RID

reference change in strategy conformance reorganization of firm to exploit results memoranda of understanding increased profile common functional Quality 01 hie healthcare

specification safety code of practice social development & services

identified need for regulatory improved border protection & policing

change support for cultural heritage

management & organization Control & care of reduced pollution

Knowledge technical the environment improved information on pollution & hazards and skills training activities reduced raw material use workshops/seminars/ reduced energy consumption

Dissemination conferences positive impact upon global climate technology transfer activities decrease in pollutants

knowledge & skills transfer

• Cohesion employment in LFRs

publication/documentation infrastructure of LFRs

participation of LFRs

lurtner KlU in LrKs

regulation and policy in LFRs

Development of transport

infrastructure telecommunications

urban development

rural development

Production & energy savings

rational use of renewable sources energy IlUClCcu SalcLy assurance of future supply

distribution of energy development of internal market

Industrial development of SME sector development development of large organizations support for trade EU regulations or policy

Regulation & national regulations or policy

policy world-wide regulations or policy co-ordination between national &

Community RTD programmes

126
METRICS OF EVALUATING PUBLIC R&D PROGRAMS

programme. This is not necessarily the case. While to some degree it is possible to distinguish between outputs from RTD,

(ranging from scientific outputs such as publications, through

"intermediate" outputs such as patents and prototypes, to "fi- nal" outputs such as new or improved products, processes or

services) and impacts or effects which arise from the interaction between the outputs and the economy or society (for example

sales, improved competitive position, or policies/regulations

which improve quality of life). Several other dimensions also

exist for these measures. A fundamental distinction is that be- tween artefacts, knowledge and skills, raising in turn the distinc- tion between tangible and intangible, codified and tacit outputs.

The models of innovation which are explicitly or implicitly

used to underpin an evaluation methodology condition the out- puts and effects selected for examination. Hence a linear or se- quential model of innovation would be represented crudely by a

research phase in which scientific knowledge is produced, fol- lowed by a transfer to a development environment and then a

transfer of a prototype or its equivalent to a context of commercialization. Under these conditions the ultimate monetary benefits associated with the end of the sequence would be used at least conceptually as a base from which the sequence of contri- butions of the preceding phases would be calculated on an in- put-output basis. More recent approaches to innovation, grounded in interactive, evolutionary and systemic contexts, identify the linear model

as an extreme special case. It has been demonstrated that by

neglecting feedbacks such a model systematically underestimates

the benefits of R&D." An interactive approach focuses attention upon learning benefits, structural changes including new

networks, and the establishment of standards. Under such conditions it becomes clear that the types of effects and the relations between them differ according to the technology and market

involved. Such differences may be founded in the appropriability

of knowledge in the sector. It is clear that there is a large differ- ence between the ways in which biotechnology, software and

nanotechnology research are likely to manifest themselves in a socio-economic context.

Earlier work by author and a collaborator summarized the range of effects to be addressed.^" These may be seen in an expanded form in an extract from a taxonomy of outputs and

effects developed for a project level management and evaluation

system being developed by another part of the European Commission for application to the Framework Programme (see Table

1). It should be emphasised that these are not intended as a

" Swann P., The Economic Benefits of Basic Research, Report to Department of Trade and Industry, PREST, 1997

checklist but rather as elements from which a logical model of

outputs and effects appropriate to a particular project (or

programme) can be constructed.

Additionality and Policy Persistence

A final dimension to consider in examining the European experience of evaluating collaborative R&D programmes is that ofthe

effects ofthese as a poUcy instrument. Ultimately, however sophisticated an analytical scheme can be created for the identification and measurement ofeffects, there is the limitation ofthe degree to which that effect can be attributed to the intervention. The first barrier is one of measurement. Attribution of an effect, as the dis- cussion ofEC and EUREKA experiences above indicates, is dependent upon separating the effect due to the programme from any

other influences which contribute to or hinder the manifestation of

that effect. Such influences may be at the technological level through

results being combined with those from other research projects, inside or outside the progremme, or through acquisition oftechnol- ogy by non-R&D means. Beyond technology, a multitute ofother

factors impinge upon successful innovation including marketing

capability, investment capacity and factors outside the firm such as the regulatory environment.

There is a great temptation for the policy-maker to see the

contractual entity which he/she funds as a discrete entity, when for the recipients it is a further contribution to a raft ofassociated work

from which the firm's ultimate deliverables are expected. In this situation, it is not surprising that the other evaluation question

which inevitably receives a very low response rate is that asking

the firm to indicate the rate of return on the project (or on their participation in it). This 'project fallacy' implies assumptions about

the way in which support is used by the firm and enters the realm of

additionality, another minefield for evaluators. In essence, the

additionality issue is based upon the difference made by the inter- vention. Initially the term was primarily applied to inputs and raised

issues about whether the expenditure on was 'additional' to that which would have been incurred by the firm in the absence of the subsidy. This dimension remains of interest because it has emerged

as the centre of a dispute between different branches of the Euro- pean Commission with respect to competition law and restrictions on state aid. In essence, the competition commissioner considers

that the fimding should demonstrably be spent on the identified project while the research and industry commissioners consider it sufficient to demonstrate that the result ofthe subsidy is increased spend on R&D.^'

^"Georghiou L. and Meyer-Krahmer F., Evaluation ofsocio-economic

effects of European Community R&D programmes in the SPEAR

NetW0lk,7?e5'e(3rc/2 Evaluation 2(1), pp.5- 15, 1992

2' Research Europe, 1998

127
ATP PAPERS AND PROCEEDINGS

Nevertheless, collaborative R&D is a more complex instrument than that implied by such questions. Various surveys have shown

a substantial degree ofagreement in the response received to questions about additionality.^^ While there is inevitably some "dead

weight" in the funding (firms which would have done the research without funding), twice as many claim that they would not have

done it at all. However, the most common grouping (and one which

to a lesser degree would probably encompass most members ofthe

other two groups) is the response that as a result ofthe programme

they did it differently - faster, with a wider range of options, and

above-all collaboratively, thus releasing the specific benefits of

collaboration. For small firms, particularly start-ups, it has been

shown that participation can provide formative management routines, through the requirements for clear project plans, etc.^' All of

these effects are summarized in the term behavioural additionality.

It is not obvious whether projects with high additionality will produce greater or smaller impacts. On the one hand one would

expect firms to cover their highest priority projects with their own

resources and hence put forward marginal projects for fimding.

This rationalistic analysis is not confirmed by the evidence from

evaluations - there are ample examples of projects with both high

additionality and high subsequent impact. One explanation could

be that public funding motivates firms to take undertake projects

with a higher risk but potential higher pay-off. The question about

whether the RTD [Research Technology Development] would have

been done differently raises a fiirther point, that ofthe persistence

of effects. If behavioural changes are stimulated by the intervention are these maintained beyond the period of the intervention. Success in the structural objectives of European programmes are dependent upon achieving a lasting shift and all other effects are increased to the extent that the firm has learned from its experience.

The issues raised here are an essential component ofan evaluation.

Conclusions

In this brief review of European experiences it has been possible only to touch upon some ofthe issues arising from the collective experience of evaluation. The rich variety of national experi- ences and the interaction with evaluation ofpolicy measures 'dovra- stream' from support for collaborative research are both important

exclusions deserving further consideration. Comparative study with

the experiences of the USA and Japan with this instrument are another relatively untapped vein which this conference will help to

exploit. However, to summarize and conclude, some important is- sues emerge. To re-iterate, the development of R&D programme

evaluation in Europe has been associated with the rise ofcollaborative research as an instrument and to some extent their fates are inter-mingled. Ofcourse, other dimensions ofresearch and iimovation policy also deserve evaluation but few offer the complexity of

challenges that arise when organizations of different types, sizes and competencies seek to combine their skills and resources to achieve mutually beneficial goals.

The durability ofthe European experience shows that collaboration works - participants keep coming back for more and often maintain their links outside the programme context. It also shows

that it is very hard to measure just how well it works in terms that would be comfortable to an investment banker or a finance ministry. In part this is because the language ofinvestment deals with simple

inputs and outputs and with what is monetisable. Experience indi- cates that this restricted view of research risks leading to a serious under-valuation that excludes the large range of options which it opens for those who take the risks. A tendency for evaluation and monitoring to converge has both positive and negative aspects. A well-managed approach to the systematic collection ofinformation increases the efficiency of

evaluations that make use of it, as well as providing a resource for programme managers. It is also best that evaluation resources should be concentrated upon those areas where learning is most likely and

away from routine activities. While evaluation may earn its keep

through providing the evidence to sceptics that R&D does pay, its prime function should always be one of learning and co-evolution

with the policies whose impacts it seeks to measure.

" Buisseret T., Cameron H. and Georghiou L., What difference does " Kastrinos N., The EC Framework Programme and the technology

it make? Additionality in the public support of R&D in large firms, strategies ofEuropean firms. Commission ofthe European Communities,

International Journal ofTechnology Management, 10(4-6) pp.587-600, EUR 15784, 1994

1995

128
Comments on Socio-economic Effects of Collaborative

R&D - European Experiences

by Giovanni Abramo'

The origins ofthe discussant date back at the end ofthe 1 3*

century when the Roman Catholic Church estabHshed the Inquisition, a general tribunal aimed at the discovery and suppression

ofheresy and the punishment ofheretics. The role ofthe discussant, at that time named the inquisitor, was to prosecute anyone

professing a religious, philosophical or scientific belief opposed

to the orthodox doctrines or official views. Worldwide renown

philosophers/scientists, such as Galileo, Giordano Bruno, etc. were condemned for heresy by the Inquisition (the statue of

Giordano Bruno was erected in the same place where he was put

to stake: "Campo de' Fiori", one ofthe most beautiful squares in Rome). The obvious effect of Inquisition was the slowdown of

the pace of social and technical development. Fortimately times

have changed as it is shown in this specific case, although certain subtle, hidden forms of inquisition still persist m certain

contexts. Professor Georghiou, one of the protagonists of European R&D programs evaluations for over 1 5 years, that is one of

the most influent representatives of the evaluation "doctrines"

in Europe, is now playing the role ofthe defendant, while myself,

a heretic almost by nature, has the unusual role of the "inquisitor".Because Italy is the country where Inquisition was bom and

developed, as an Italian and often heretic I need state upfront

that my objective is not to send Prof. Georghiou to the stake. My

hope is to be able to provide, through my comments, an additional perspective, a managerial one, to the problem of evaluation, which may inspire new insights in those, hke Prof Georghiou,

deeply involved in the subject.

I will skip the usual identification of and elaboration on the good points ofthe paper, which the art of communication would

recommend to start with (my heretical nature comes to surface),

and pass directly to pose few, hopefully challenging, questions.

The paper cogently traces the evolution of the approaches

and methodologies to evaluate European publicly fiinded R&D

programs, in particular the Framework Programme and Eureka.

The description is mainly focused on the difficulties generally

encountered by the different approaches.

Constraints regarding paper length have probably inhibited

the presentation of the results of evaluation studies. In fact, evidence ofthe socio-economic effects is hardly shown, with the only exception of the evaluation exercise by the BETA Group.

Then, the first question which comes to mind is: "After 15 years of evaluation studies in Europe are there figures which provide

evidence to skeptics that R&D does pay (citing the author)? If yes, it would be beneficial to include them in the paper. Ifnot, as

I suspect, then "why have similar studies in the U.S. (Griliches,

Mansfield, Nadiri, etc.) have provided quantitative answers? Is it because the Americans are better, or because we are trying to measure the "unmeasureable" (what does "social" mean to Europeans differently from Americans?)?

Passing from the results of the evaluation studies to the use of such results, if ...the primefunction ofevaluation should be one of learning and co-evolution with the policies whose impacts it seeks to measure... (citing the author), "What has been

the impact of such evaluation studies on the EU R&D policies?

How have the evaluation findings contributed to improve EU

R&D policies?"

The paper would have greatly benefited from elaborating on

such issue, otherwise the population of skeptics, I am afraid,

would grow larger. In fact, most ofthose who would not need to be convinced that R&D does pay, would start questioning

whether investments in the evaluation of EU R&D programmes

do pay. Evaluation studies findings, in addition to facilitating policy

makers in orienting their decisions, are an important tool for project management as well. This elicits a further question: "How

have evaluation studies contributed to improve the management

of the various EU R&D programmes? Within this context the author cites a study by Chabbal of 1988, which points out the

significant problem of experts' background in evaluation panels

(mainly technical). My own experience, ten years later, as an EU

evaluator for the selection ofprojects, shows that the problem is

still there. This lead to my final comment. My impression is that the EU

has placed more emphasis on ex-post evaluation than on ex-ante. Ex-post evaluation tends to see Programme Management as a black box: policy objectives and money are the inputs; socio- economic effects are the outputs to be measured. The correlation between project management and outputs (effectiveness and

efficiency) is very sfrong. Then, if, after all, to obtain evidence of

economic returns proves to be so formidable, why not focusing

resources in optimizing the processes within the black box (i.e. programme management)? Probably, at the end we would not still know the exact public rate ofreturn to EU R&D investments, but

at least, whatever it might be, we would know for sure that it is the highest rate that we could achieve.

' Dr. Abramo is with the Consiglio Nazionale delle Ricerche, Direzione

Centraie Attivita Scientifiche, Rome.

INTERNATIONAL CONFERENCE ON THEECONOMIC

EVALUATIONOF TECHNOLOGICAL CHANGE

o

p

E

N

I

N

G

S

E

S

S

I

o

N

I

Overview of the Advanced Technology Program and Its Evaluation Program

by Rosalie Ruegg, Director, ATP Economic Assessment Office

ATP is a relatively recent component among United States

strategies to foster innovation in the civilian sector. It is unique

in having as its main long-term goal that of economic growth. In

contrast, the "U.S. mission agencies," such as the Department of

Defense and the Department of Energy, often also call out the importance of the economic effects of research they fund, but

their first priority is, respectively, defense and energy. Its mission is to foster the rapid development and commercialization of

pre-competitive, generic technologies with potential for significant, diffuse impact on the nation's economy. The program shares

part of the research costs with industry of technology development projects conceived, proposed, and led by U.S. companies -

projects that are selected for award by ATP through a competitive peer-review process that evaluates the technical and busi- ness/economic merit ofproposals. Each project selected for funding has specific technical and business goals, funding allocations, and completion dates that are established at the outset. The projects are monitored by ATP staff and can be terminated

for cause before completion. Funding is on a cost-reimbursable

basis. Eligibility extends both to single-company projects, most

of which show extensive collaborative activity through subcontracting and informal alliances, and to joint ventures which must

include at least two for-profit companies and often include other companies, universities, and non-profit organizations as additional members, as well as subcontracting and informal alliance arrangements. Figure 1 highlights some of ATP's distinctive

features. The summary statistics in figure 2 show through 1998,

the cumulative number of proposals received from companies,

the amount of money offered by industry and requested from the ATP, the number of organizations involved, together with the number of awards made by ATP, the associated funding, and the number of participating organizations.'

Figure 2 - Summary Statistics for ATP (1990-1998)

Proposals Received Number of proposals 3,585

Total research proposed (U.S. $ millions) $14,794

Industry share (U.S. $ millions) $7,103

ATP funding requested (U.S. Smillions) $7,691

Number of oroDosine organizations 6,104 Awards Made

Number of awards 431

Total research funded (U.S. $ millions) $2,783

Industry share (U.S. $ millions) $1,997

ATP share (U.S. $ millions) $1,386

Number of particiDating organizations 1.010 ATP has an integrated set of strategies for accomplishing its mission. It funds projects that are desirable for their social benefits, essentially buying down technical risk that exceeds the

level acceptable to private investors. It encourages companies,

universities, and other organizations to undertake multi-disci- plinary collaboration to solve complex problems of broad importance. It focuses on multi-year partnerships needed to address

problems that tend to be neglected when short-term perspectives predominate in investment decisions. Through its selection criteria, it fosters the structuring of partnerships that inte- grate objectives and approaches across the areas of research,

commercial interests of the innovator(s), and the national

economy. Short of compromising the incentive to conunercialize

the technology, ATP encourages companies to share the result- ing scientific and technical knowledge with others. In short, ATP aims to induce industry to undertake challenging research needed to develop enabling technologies with potentially highpayoff potential for the nation, research that businesses other- wise would not undertake at all or not with the scale, scope, or timing needed to realize the potentially large societal benefits. Because ATP's mission is economic in nature, its evaluation

emphasizes economic impacts of the program. But there are a number of sub-objectives and constraints to the program that condition the program and its evaluation. For example, in addition to providing economic benefits, projects must entail highrisk research. Hence, one aspect of ATP's evaluation concerns

the scientific and technical contributions of funded research. Because the research is high-risk, it is understood that not all projects will be fully successful. Only a fraction will likely acFigure 1 - Key Features ofATP

• Technology development for national economic benefit

• Industry leadership in planning and implementing projects

• Project selection criteria on technical and economic merit

• Annual rigorous competitions based on peer review

• Positioned after basic science and before product

development

• Not "entitlement" funding - all projects have end dates

' More information about ATP is available on the World Wide Web:

http://www.atp.nist.gov; by e-mail: atp@nist.gov; by phone 1-800- ATP-FUND (1-800-287-3863); by written request: Advanced Technology Program, National Institute of Standards and Technology, 100

Bureau Drive, Stop 4701, Gaithersburg, MD 20899-4701.
ATP PAPERSAND PROCEEDINGS

complish all the goals — scientific knowledge creation, timely

commercialization of products and processes, and widespread

diffusion of the technology leading to large spillover benefits. Most will be at least partial successes given that scientific knowledge often is gained even from research failures. Many will hkely

yield a sufficient return to pay back their costs. A few likely will be "home runs." Since accelerating the development and commercialization

oftechnology is a program mission, ATP's evaluation tracks the degree of speed up of technology development and the rate of

commercial progress ofaward recipients. It also investigates the economic value ofaccelerating technology development projects.

Since the program is for national benefit, the evaluation is concerned with generating and measurmg spillover effects be- yond the direct benefits to innovators — including market

spillovers, knowledge spillovers, and network spillovers. It seeks

to fund technology development projects for which the spillovers are large, as reflected in gaps between the resuhing social rates of return and the private rates of return to the innovators. Because ATP is charged with promoting the formation of

research joint ventures, collaborative research is another topic

of particular interest for ATP's evaluation. The objectives and

progress ofthe individual joint-venture members are tracked, as well as those ofthe overall entity. Efficiency issues, the internal- ization of spillover effects, and technology diffusion effects are examples of research topics of interest to ATP that concern col- laborations.

Because it is critically important to the realization ofbenefits from the program that ATP make a net contribution to the nation'

s

economy —leveragmg rather than displacing private sources of

capital — ATP's evaluation seeks to measure the differential impacts attributable to ATP, m addition to the overall impact of

projects and groups of projects. With- and without-ATP sce- narios are posed to help get at the effects attributable to ATP.

This entails the use of counterfactuals and the attendant uncertainties thereof

The ATP initiated evaluation at the outset of the program,

first, to develop a management tool to make the program better meet its mission and operate more efficiently; and, second, to meet the many external requirements and requests for ATP pro- gram results. Demands for performance measures for ATP are

intense. Requests for evaluation results come frequently from

individual members of Congress and their staff, from Congressional subcommittees, the General Accounting Office, the Executive Office of the President, the Office of Management and

Budget, the Office ofthe Inspector General, the Press, think tanks,

industry groups, and others.

Title II of the 7\merican Technology Preeminence Act of 1991

(P.L. 102-245), enacted in 1992, directed that a comprehansive

report on the results of the ATP be submitted to each House of

the Congress and the President not later than 1996. This report was delivered in April 1996 (the Advanced Technology Program

1996).In addition, ATP, like other federal programs, is subject to the evaluation requirements ofthe 1993 Government Performance

and Results Act (GPRA). The GPRA resulted from a bipartisan

effort to improve accountability, productivity, and effectiveness of federal programs through sfrategic planning, goal setting, and

performance assessment. ATP/NIST is developing assessment

plans and techniques, and carrying out evaluation studies in compliance with the GPRA. ATP receives many inquires about

its evaluation tools and methodologies from other agencies, as well as from similar programs in other countries. To square the often urgent demands in the short run for evaluation results with the reality that patience is required to

realize and validate empirically long-run program outcomes, ATP

has adopted a multicomponent evaluation sfrategy. Its main

components include (1) descriptive (statistical) profiling of applicants, projects, participants, technologies, and target apphcations; (2) progress measures derived principally from surveys and ATP's "Business Reporting System;" (3) real-time monitoring ofproject developments by ATP's staff; (4) "status reports" on completed projects; (5) microeconomic and macroeconomic

case studies of project impacts; (6) methodological research to improve the tools of longer term evaluation; (7) special-issues

studies to inform program structure and evaluation; and (8) economefric and statistical analyses of the impacts of projects and

focused programs.

Early evaluation studies carried out by ATP economists and

contractors suggest that ATP is on track, meeting its objectives,

and delivering results for the U.S. economy. But at this time only

rough quantitative and qualitative projections ofproject impacts

are possible due to limited information and uncertainties about

the ultimate outcomes. These measures will become better in- formed as commercialization and diffusion activities progress. By tracking developments as they unfold, we expect over time to be able to reduce the estimating errors, extend the scope ofanalysis, and provide better measures. Over the coming years, the ATP expects to contribute significantly to the body of work on

technology impact assessment and to build towards a more comprehensive view ofthe impacts ofATP.

134
R&D and Productivity: Some Historical Reflections

byZviGriliches'

(Keynote Address)

Dedicated to the memory ofEdwin Mansfield, one ofthe great

pioneers ofthis subject

I want to note first that studies oftechnology and patenting

have a long history and predate the more recent interest in aggregate productivity and the residual. For example, the 1 960 Minnesota Conference on The Rate and Direction ofInventive Activity, organized imder the auspices of Simon Kuznets, with the

participation ofmost ofthe next generation ofscholars who would

matter in this field: Mansfield, Nelson, Scherer, Schmookler, and

myself among others, did not really relate to aggregate productivity measures, though Nelson in his introduction to the 1962

published volume tries to make a connection. The conference

and the book were focussed primarily on the micro-economics

and econometrics of invention and innovation. But as the evidence on large residuals in the accounting for output growth percolated through the consciousness of the eco- nomic community (as the result of the work of Schmookler,

Abramovitz, Kendrick, and the synthesis by Solow), it became common to interpret this residual as reflecting largely technical change and think of it as being caused primarily by public and

private R&D efforts. Akeady earlier, in 1953 and 1958 respectively, Theodore Schultz (recently deceased) and I made calculations relating the two in the agricultural sector and implying rather high rates of return to the publicly supported R&D there. Very early on researchers turned to regression analysis using various constructs of R&D capital as their independent variable. The pioneers here were Terleckyj, Minasian, Mansfield,

myself, and others. We quickly established that R&D seemed to be an important "statistically significanf variable in explaining

productivity differences, but the estimated coefficients were not

large enough to provide a major explanation ofthe large growth

in TFP observed in those years.

It was obvious that what was missing was some accounting

for spillovers of R&D results and other sources of new knowledge across firms, industries, and countries. So a number of

studies, starting with the work of Evenson and Kislev (1973) and

Terleckyj ( 1 974), looked for some ways of capturing them in the

data. The most influential of the more recent studies are Jaffe (1986) and Coe and Helpman (1995). Many more have been done

since. Though each one is subject to some reservations, the

overall conclusion was that spillovers seem to be there and are important. The eventual justification for the activities of our

host, the ATP program, is ultimately based on the perceived

importance of such spillovers and the need to support them expUcitly.

So while one can say that this line of research was very

successful, both substantively and in its impact on policy, the world was changing while we were collecting our laurels. The

unexplained growth ofproductivity, which we were attributing to science and industrial R&D, came close to disappearing! (See

Figure 1.) Two questions arose: If R&D is so important "where is the meat"? Where is the promised productivity growth? Ifone pointed

to a decline in the aggregate rate of growth of R&D, the answer was that most of that decline was in federally supported R&D

which earlier studies had found less productive (Figure 2). But

then why was there no spurt in productivity growth as the result of the spurt in private R&D investment in the mid-eighties?

Either R&D didn't matter as much as we had claimed, or it's impact had declined, or it was running into sharply diminishing

returns. One example of such an argument was based on patent data which were largely stagnant in spite ofthe growth in R&D.

(See Figure 3.) But the decline in patenting per R&D dollar was

happening also in the good old days!

A second piece of disquieting evidence is the decline m the apparent valuation of R&D by the stock market. (See Figure 4, from B.H. Hall (1996).)

I want to admit the possibility that the rate of TFP growth

that was achieved during the 1950's and 1960's may not be sus- tainable in the long run, but I am not as pessimistic about our

future (at least from this direction). The future knpact ofour (and

the world's) political leaders on the lives of our children is rather scary to contemplate. First, there is a large reservoir of still undiffused technical and organization knowledge that could and

should improve productivity for years to come. Also, there is much technology "in the making" that has not yet hit the markets

significantly. Biotech is a fine example. But the main point that I want to make in closing, a point that was clear to the participants of the 1960 Minnesota Conference,

is that aggregate productivity numbers are only dimly and possibly misleadingly related both to the measurement oftrue technical change and the impact of R&D on it, especially federally

supported R&D. This is due, in part, to difficulties in productivity measurement per se, and second to the particular location of

most ofR&D in the industrial spectrum. There are two questions

here: (a) is the productivity slowdown real? (b) can we see the

effects ofR&D on it?

' Dr. Griliches is a Professor of Economics at Harvard University and a member of the National Bureau of Economic Research.
A TP PAPERSAND PROCEEDINGS

The answer to the first question is clouded by the fact that much ofthis slowdown occurred in what I call the "unmeasurable"

sectors, such as services and construction, and that these sectors also grew greatly in unportance in the total economy. This is documented in Figiu"e 5 and Table 1 . Looking at Table 1 and the numbers for the non-farm, non-manufacturing sectors, do we

really believe that two-thirds of the economy has been retro- gressing for the last 20 years?

Table 1 . Multifactor Productivity Growth

1949-73 1973-79 1979-90 1990-96

Non-farm business 1.87 0.40 0.00 0.20 Excluding labor quality adjustment 2.07 0.40 0.30 0.60 Manufacturing 1.74 -0.40 1.00 1.90 Implied non-farm non-manufacturing 1.77 0.82 -0.17 -0.29

Source: USDL 95-518m95187. and Lysko (1995) A. Based on gross sectoral output measure rather tan GDP.

B. The first number is computed as 2.07-.32 x 1 .747.65) 1-

.32), where .32 is the approximate share ofmanufacturing in total non-farm business during the first period

and .65 is the ratio of value-added to sectoral output in manufacturing. These two numbers are approximately

.23 and .53 respectively, for the last two periods.

It is also the case that most of the R&D in the economy has

been directed to the "unmeasurable" sectors: space, defense,

health, and IT, and in none of these sectors will a major technological breakthrough show up easily in the productivity numbers

as they are currently computed. This is not entirely true of computers and communication advances, but they too, as you are

likely to hear from Tripplett tomorrow, are largely consumed in the "unmeasurable" sectors: primarily finance, health, education,

and other services. And much ofthe direct R&D has also moved

into these sectors. NSF estimates that a quarter of all industrial R&D is now performed in non-manufacturing industries, up from

only 3 percent in 1995, where its output is again unlikely to be

measured.

So I want to conclude by saying that I am cautiously opti- mistic about the immediate future, especially as I observe the recent rise in patenting and the up-tic in manufacturing productivity. (See Figure 6.) But we, as a research community, must

develop better tools for measuring the contribution ofPv&D. Aggregate productivity statistics just won't do it. That's what you

are here for, and I am encouraged by the quality ofthe new work

by a new generation ofresearchers that is reflected at this meeting.

I am worried, however, by the continued decline, both relative to GDP and in absolute terms, in federal support to the national

R&D effort. It has fallen from more than a halfofthe total R&D in the 1970'stojust 30 percent in 1997. Some ofthe cuts may have

been warranted, but I doubt that this is the investment in the

fixture that we should be cutting, on the margin. And Francis

Narin's evidence presented today only strengthens my worries. While I do not think that these declines have had or will have

noticeable effects in the near fiiture, they may come back to haunt us later on, especially if we figure out better ways of mea- surmg their impact and realize what has happened. But it may be

too late then. The sooner we can have a better quantitative

understanding of these processes the better. The fixture of our

children may be at stake here.

Figure 1: Growth Outputper Hour

1960-1973 1973-1990 1990-1994

Education and Experience ^Capital Intensity |BR&D BResldual

Efficiency

Figure 2: Expenditures as a Percentage ofGDP By Funding

Source, 1960-1995

Other B Indattrr

136
OPENING REMARKSAND PANEL SESSIONS

Figure 3: Domestic Patent Applicatins per Company

Financed R&D in Industry (1972 $'s) an per Scientist and

Engineer

Figure 4: Implied Valuatins ofPhysical and R&D Capital in the Stock Market. Co-efficients from log Q equations,

unbalanced annual cross-sections, 1976, 1995.

76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95

Basic model: V = q(a)A + q(r)K Est eq.: log (V/A) = log q(a) + log[1 + q(r)K / q(a)A] QRD —EQ2

Figure 5:Log ofGDPper hour: 1948-1996 Figure 6: Patent Applications in the U.S. numbers, 1948-

1996

120,000 -

100,000 -

80,000 -

60,000 - Domestic

40,000 -

20,000 - Foreign,—.- ^

n

48 53 58 63 68 73 78 83 88 93 FRAPPL —DAPPL

year

737
The Solow Productivity Paradox:

What Do Computers Do to Productivity?

Jack E.

"You can see the computer age everywhere but in the productivity statistics.

Robert Solow, New York Review ofBooks, July 12, 1987

Dr. Triplett referred to Solow's often repeated aphorism, now

more than ten years old, in his discussion of the impact of computers on worker productivity. The presentation reviewed and

assessed the most common "explanations" for this paradox:

You don 't see computers "everywhere, " in a meaningful economic sense. Computers and information pro- cessing equipment are a relatively small share of GDP

and of the capital stock.

You only think you see computers everywhere. Government hedonic price indexes for computers fall "too

fast," according to this position, and therefore real computer output growth is also "too fast."

You may not see computers everywhere, but in the in- dustrial sectors where you most see them, output is poorly measured. Examples are finance and insurance,

which are heavy users of information technology and

where even the concept of output is poorly specified.

iplett'

Even ifyou do see computers everywhere, some ofwhat

they do is not counted in economic statistics. Examples

are consumption on the job, convenience, better userinterface, and so forth.

You don 't see computers in the productivity statistics

yet, but wait a bit and you will. This is the analogy

with the diffusion of electricity, the idea that the productivity implications of a new technology are only vis- ible with a long lag.

You see computers everywhere but in the productivity

statistics because computers are not as productive as you think. Here, there are many anecdotes. The technological demands of networking are also a factor.

There is no paradox: Some economists are counting

innovations and new products on an arithmetic scale when they should count on a logarithmic scale.

' Dr. Triplett is a Fellow at the Brookings Institution in Washington, D.C.
The Evaluation ofPrivate Sector Research

Rebecca Henderson'*

Dr. Henderson discussed the often complex issues faced by

private sector managers seeking to evaluate the productivity of

their research programs. She reviewed a variety of techniques

used within the private sector, focusing particularly on the ten- sions between ex ante and ex post evaluation, between individual versus team or group evaluation, between long and short range outcomes and between qualitative and quantitative metrics. She concluded with a discussion of the possible implications of

private sector experience for public sector work.

* Dr. Henderson is a Professor in the Sloan School of Business at M.I.T and a member of the National Bureau of Economic Research.

INTERNATIONAL CONFERENCE ON THEECONOMIC

EVALUATIONOF TECHNOLOGICAL CHANGE

International Conference on the

Economic Evaluation ofTechnical Change

Panel Sessions

Two panel sessions were held consisting ofboth academics

and government representatives from several countries who have

had years of experience in either researching government tech- nology programs or in administering them. The panel titled "Publicly Financed Research Consortia" was chaned by Rosalie

Ruegg, Director ofATP/EAO, and included: Dr. Luke Georghiou

ofPREST, UK; Dr. Francois Sand of the EUREKA Secretariat,

Belgium; Dr. Giovanni Abramo ofthe ConsigUo Nazionale delle Recerche, Italy; and. Dr. Mariko Sakakibara formerly ofthe Ministry of International Trade and Industry (MITI), Japan. Each

member ofthe panel received a list ofquestions in advance ofthe conference in an effort to focus the discussion. The panel that addressed the topic of"Public-Private Part- nerships in R&D: Lessons Learned" was chaired by Maryellen

Kelley, Senior Economist at ATP/EAO, and consisted of: Liam

O' Sullivan ofthe European Commission, Belgium; Dr. Phillipe

Laredo of the Centre de Sociologies de I'lnnovation, France; Ralph Lattimore ofthe Productivity Commission, AustraUa; and, Dr. Adam Jaffe ofBrandeis University.
PANEL SESSION: Publicly Financed Research Consortia

Professor Luke Georghiou

Director, PREST

University ofManchester, UK

1) What sorts oftechnologies, e.g., infra structural, should be

targeted in research consortia?

The first question is whether any technologies should be

targeted - if the benefits are intended to be manifested in the improved competitive performance of individual firms then it should be up to those firms to come forward with proposals for areas of advanced technology which they wish to pursue. This

case is reinforced by the convergence and inter-dependence of

many advanced technologies, for example the importance of

bioinformatics in progressing biotechnology and conversely

the prospect ofthe use ofbiological processes in advanced computing.

However, I would also concede that there are arguments for targeting in certain circumstances. These include those areas where co-ordination at the R&D phase improves the speed and

scope of the market phase. A good example here would be vehicle information systems, which require coordinated R&D across a variety of firms m the electronic, automobile and service sectors, together with an input from regulators and standard-setting

bodies. Effects m these circumstances are greater than in stand- alone project assistance. A similar argument could apply in a very new area where a focus of activity could create a critical mass of researchers and again amplify the benefits.

2) What is the appropriate mix ofparticipatingfirms in research

consortia in terms ofsize, technological strength, etc? A first point is that the European experience has shown that the appropriate mix includes not only firms but also a significant

contribution from universities and public research bodies. The

"golden triangle" project feamres a producer who takes the output to market, a user who gives early feedback and benefits from

early access, and an academic partner who raises the technological level ofthe activity as a whole. European experiences encompass firms of all sizes but their needs are different. Small firms

(unless they are specialized research performers for whom the program is a type ofbusiness) normally have a much shorter time

horizon and seek to go straight to the market after the project

(though many fmd that the ensuing "productization" phase is costly and time-consuming to the point where they need external

help). Large fums, on the other hand tend to see the programs as a window on the world where they can monitor academic devel- opments and the activities of other firms while building their competence base.

hi terms ofthe direct mix the key point is to structure projects

around technological and market complementarity - few projects

succeed containing direct competitors (other than for standards

or non-competitive areas). It is the differences between participants which lead to success.

3) To what extent should the funding government agency at- tempt to actively participate in the project?

A well run selection process increases confidence in the initia- tive. Passing on knowledge gleaned from previous work is also useful for participants, for example, that concerning intellectual property rights agreements. During the project the aim should be

to interfere as little as possible except to ensure that resources

spent are matched by work done. Help should be on hand if things go wrong. At the end, small firms may need further help in securing access to private venture capital or to find a partner able to offer a better route to the market.
Dr. Francois Sand

Eureka Secretariat

Brussels, Belgium

Introduction

I would like to start with a short introduction of myself: my

name is Fran9ois Sand and I am from the EUREKA Secretariat based in Brussels (Belgium). I was a nuclear physicist (I still am

I suppose!). Eleven years ago I joined the Evaluation service at the European Commission (DG XII, Science, Research & Development) where I had the opportunity to coordinate many evaluation exercises for specific Community R&D programs Uke BRITE,

STD, EURAM, etc. Five years ago, I was seconded to the EUREKA Secretariat. One of my main tasks was to set up the CSE

(Continuous and Systematic Evaluation process) which assesses

the impact that EUREKA projects have had on the market in the

years after completion. For three years now, an Annual Impact

Report has been presented to the yearly Ministerial Conference.

I have attended all the presentations that have been given

so far today. They gave me the impression of being on my bi- cycle on the ground and looking at you in the blue sky flying the

ultimate jet model: it was very impressive! At my ordinary level, however, I consider that one ofthe roles of evaluation is to start correcting a somewhat too simplistic view that some of our politicians sometimes have in mind regarding outputs ofR&D public

funding.

Being the last speaker this evening, I shall not repeat many

of the views that the previous panelists expressed and which I share with many of them. Furthermore, relying on EUREKA's

experience for my contribution, I shall point out several difficulties specifically connected to the international cooperation framewoik.

Let's now address the first question:

1) What sorts of technologies, e.g., infra structural, should be

targeted in research consortia?

In the EUREKA initiative, due to the basic principles in its defmition, especially flexibility and bottom-up initiative, this problem oftargeting doesn't occur directly at the central level (inter- national). The national support process, however, which is part

of the labeling procedure introduces de facto a type of hidden

selection amongst the project proposals which nevertheless asserts that clear targets are usefiil for national operators of the

initiative. Supportmg collaborative research to produce structural results is, for itself, meaningless. Indeed the target must not merely

be in R&D but in longer term perspectives for which R&D is to be

considered as a tool, i.e., supporting vector. But if it should help

in improving societal structures, it becomes worthwhile to address it. In a case like PROMETHEUS, for example, a project which

aims at a global approach ofroad traffic problems, it will be based on complementary expertise of the industrial consortium set up

by the main European car industries as well as on related devel- opment. The adaptation of the infra-structural technologies

needed for the project will be at the core ofthe development. We

can pick as examples the communication tools with global sys- tems (IR, VHF, GPS/GSM etc), communication between cars (in- cluding information on the status of the surface etc.), autono- mous detection systems (radar, vision, detectors, etc). Each of

these examples have been developed in view of the specific application and simultaneously extend the basic know-how avail- able for other fields of application.

Another example ofa project having a strong component of

infra-structural technologies is HDTV (High Definition Television). It could be considered in principle as a failure regarding

the official target (the system involved the use of analog signals)

but it was actually a success regarding all the infra-structural development (standards) and fall out (digital development). Fur- thermore, it later permitted the partners to relaunch quickly an- other project, ADTT (Advanced Digital Television Technologies) for a broader prospect in the use of digital TV signals. We

noticed however that in this case an advisory board at the political/administrative level could possibly have helped to refocus

the project much earlier as underlined in the Airaghi evaluation

report.The experience proves that all the technologies are not similarly successful in the framework of collaborative projects and

does not appeal in the same way to different types of participants.
ATP PAPERSAND PROCEEDINGS

2) What is the appropriate mix ofparticipatingfirms in research

consortia in terms ofsize, technological strength, etc?

The past and current evaluation exercises put in light the following findings regarding consortium structures:

• The best structures involve a chain of qualification with

partnership extending from the producers to the final

clients.

• This supposes a clear distribution of tasks from the very beginning and an agreement on the fixture exploitation: i.e., how to share the results.

• The projects with several competitors are only very sel- dom successful. This is, however, not a rule and de- pends very much on the type of target considered by

the project. Common definition for new standards

(HDTV) or improvement of safety on the road

(Prometheus) or organization of a worldwide market

(Gallileo & Amadeus) are examples ofpositive collaboration between competitors. In most of these positive

examples, it is to be noticed that a global reorganization

ofthe corresponding market was at stake. Consequently,

the participating organizations were well aware of the importance for them to be part of the effort, in order to preserve their fiiture share of the market.

• As a rule the leaders of a consortium play a crucial role. In most cases, it must be an industrialist, even if there

are some few examples of success with non-indusfrial

leadership. When such a situation occurs however, it is advisable to attach to the consortium a type of advi- sory board, mainly constituted with industrialists, to help the leader when strategic decisions regarding the products and/or processes are to be taken.

• Expectations from participants are very much dependant upon size: * SME's are aiming at short term market applications * Large companies are more attentive to the evolu

tion ofknow-how and in controlling market evolu

' tion; * Service companies, which are increasingly more

active in such projects, are interested in gaining

good adaptation (and accurate information) vis-avis market needs and networking (for new clients).

• No general rules can be derived from EUREKA's experi- ence regarding the technological strength. One of the

possible results from analysis on a number ofcase studies is that, when large numbers of participants are col- laboratuig together despite their competitive status, the

possibility of success can be better if the organization

ofthe project is more "a la carte", i.e. when the development of the project reaches a stage where more precise

and applied outputs are concerned, some clustering

between participants occurs which permits a better control of mformation fluxes and preserves the IPR. Another interesting frend, in the results so far, indicates

that success often occurs in projects where a dominant

partner works with potential subcontractors or operators.

7>) To what extent should the funding government agency at- tempt to actively participate in the project?

When interviewing the participants, the following argument came

regularly:

There is a clear basic idea: if the industrialists knew from the beginning that the chances of success are great and the potential market large, they would never look for (international) cooperation framework with the related problems. The main goal of

public support is consequently to permit the participants to take a risk that they would not have taken alone. The level should be

adjusted accordingly. This should take into account the distance

to the market as presented in figure 2. Regarding public fimding,

the figure indicates that the agencies should not intervene in Domain A where industries can take over on their own because

the risk is acceptable. On the other hand, public fimding can help

industries to start projects in Domain B where the risk is too large

for individual initiative until the moment when the risk decreases

sufficiently to be accepted by the firms. In this framework, an active participation of the funding

agency could also be encouraged at the following levels to:

• Force the participants to properly prepare the files of

the project (especially market analysis and forecast).

However, the services involved in this matter must keep

in mind to limit as much as possible their bureaucracy.

• Push the participants to follow-up project development

by an accurate monitoring system and a sound analysis

of these developments. The agency has also to put the

participants in a position to adapt in real time to the evolution ofthe project (results obtained versus evolution ofthe market) and to recommend a preliminary feasibility phase when the project is ambitious.

146
OPENING REMARKSAND PANEL SESSIONS

• Encourage the set up of an industrial advisory board,

especially, as previously said, if the leader is a nonindustrialist.

• Ensure a more active role in mediating and facilitating

collaboration (as requested by participants - Annual

Impact Report '97')- It should never take over the participants responsibilities and should keep the level of

control a reasonable one (i.e. taking the opportunity to help when needed but avoiding any form of dirigisme).

• Help to identify and solve the problems linked to commer

cialization not for but with the participants.

• Ensure the required supportive measures (norms and

standards).

' The successive Annual Impact Reports are available from the EUREKA Secretariat on request by e-mail to : hilde.iemants@es.eureka.be.

147
Giovanni Abramo

Consiglio Nazionale Delle Recerche

Rome, Italy

Premise

As soon as I began elaborating on the intellectually chal- lenging questions to be raised in this panel, I immediately real- ized that no answer, however generic it may be, would be appropriate to every environment. Being Italian (Italy is a relatively small economic entity among major industrialized countries) and

a European Union citizen at the same time, unavoidably induces

me to take an international perspective any time I face problems

of the kmd discussed here (especially if I have to do it in English

- the good side ofbeing "small" and not English mother-tongue!).

Every time I tried to figure out the sorts of technologies to be targeted or the appropriate mix of participating firms in re- search consortia in Italy, I realized that the answer would have

not necessarily been the same at the European Union level. As a consequence, in order to avoid potential subsequent misunderstanding I find it useful to make from the very beginning a few

considerations which may elicit the intrinsic limitations of and

needed assumptions for the problem at hand.

First of all, the answers to the above questions are strongly

dependent on the context where research consortia operate. The

economic, industrial and research policies of a country, its industrial structure and technology infrastructure, its level oftechnol- ogy specialization, its business culture, as well as its geopolitical

role will all, more or less, affect the decision on the appropriate

mix of participating firms or the sorts of technology to be tar- geted.

Second, at a more micro level, the answers to the above

questions may be different depending on the specific publicly

financed R&D program we are referring to. Also, they will de- pend on the content and objectives of other co-existing pro- grams. A consortium engaging in basic research, for example,

may entail a different set of partners than a consortium engaging

in technology development. So the answers will very likely de- pend on the scope and objectives of the program, unless we

assume that a research consortium is an end per se rather than a mean.

Third, in case we are able to achieve commonly agreed upon more or less defmitive answers, it is important to ask ourselves from the very start how we are going to use such answers: Simply to enhance awareness of people involved in the matter? Or,

possibly to revise and improve current publicly supported R&D

programs? Ifthe latter is the case, as I presume, it is important to keep in mind that implementation issues should be considered

from the very early stage of the program formulation/revision process in order to reduce the likelihood of subsequent operations problems. The adoption of this "concurrent engineering"

perspective will affect for sure the process of elaboration on the

issues at hand, and very likely its outcome as well. The foregoing considerations elicit the need to defme more

precisely our field of investigation, in order to try to provide answers not too generic in kind. Then, an assumption needs to be made, regarding the reference publicly financed R&D pro- grams. We may frame the questions in the context of ATP-like

programs (at European Community level one of the programs

which more closely resembles ATP, in terms of broadness of

technology fields covered and market orientation, seems to be

the INNOVATION Program ofDGXIII, as far as the Technology

Validation Projects and the Technology Transfer Projects are considered). Furthermore, the approach I take is to consider, in my elaboration, whenever deemed appropriate, also the implications for implementation ofpossible solutions to the above questions. As for the geographical context, I will avoid focusing on

Italy, which would be as less difficult to me as of little interest to the audience, but I will try to leave it open, although with an unavoidable emphasis on Europe, which may synergistically

complement the U.S. focus of local participants in the panel.

Given the above assumptions and limitations it is important

then, that the following discussion and conclusions be treated with proper caution and that the potential bias of the European

perspective of the panelist be borne in mind, especially by the non-European audience.

Following on the above preliminary considerations common

to all the questions raised, I now turn to elaborate on each single

question separately.

1) What is the appropriate mix ofparticipatingfirms in terms of

size, technological strength, etc. ?

Needless to repeat that the answer to such a question is strongly linked to the context and underlying philosophy of the program at hand and the policies it is aimed to realize.
OPENING REMARKSAND PANEL SESSIONS

With reference to size, the attitude towards fostering the

participation ofsmall- and medium-sized enterprises (SME's) in publicly financed R&D consortia varies among nations, depending on the context and priorities deriving from it. The European

Community, for example, has emphasized special attention on

SME's, since the very beginning ofits R&Dprograms. The United

States ATP, instead, only recently, I understand, has set the explicit goal to increase participation in ATP by SME's. This is probably due to the different needs arising from the very differ- ent industrial structures ofthe two economies. 92.4% ofthe 1 1 .6 million European enterprises employ below 100 employees, and

only 0. 1% above 500 employees. SME's in Europe represent 70%

oftotal revenue and contribute to 75% oftotal employment. When

you deal with such proportions, probably there is little need of

referring to economic or organizational theories to justify inter- ventions in favor of SME's. If, on the one hand, SME's, with their high specialization, flexibility and dynamism may be considered one of the main potential strengths of the EC, on the other hand

their small size may penalize them and make them vulnerable when it comes to face the complex challenge oftechnology-based

global competition. Then, ad hoc policies may be deemed necessary to favor SME's growth and networking and to foster the

diffusion oftechnological innovations among SME's competing

in traditional sectors. Anyway, there are a number ofreasons, more or less directly

based on economic theory, why the participation in R&D consortia by SME's should be fostered. Many of these reasons may

apply also to those nations, such as the U.S. and Japan, whose

industrial structures may not make the attention to SME's so

vital.With reference to one ofthe objectives common to all ATPlike programs, i.e. the maximization of the gap between social returns and private returns to R&D investment, a direct conse- quence follows with regard to SME's. As compared to large companies, SME's in general tend to be less able to capture the same number of and to the same extent the potential externalities aris- ing from R&D investments. In fact, the likelihood that SME's

have market power in the relevant markets, the needed complementary assets, the ability to protect the innovation and to create and sustain long-lasting first-mover's advantages is comparatively lower than large companies. This would have a positive impact on public returns, making SME's participation in publicly financed R&D consortia, all others equal, more appealing

than large companies.

With reference to another objective common to all ATP-like

programs, which is minimizing displacement or, in other words,

selecting those projects that would be "under funded," delayed,

or, otherwise inadequately pursued, the likelihood that the latter would happen with SME's is comparatively higher than with

large companies. This is due to the increasingly high costs of

R&D, the comparatively higher financing barriers to SME's (es- pecially where financial markets are not so developed as in the

U.S.), and the comparatively more negative impact of potential

project failure on SME's.

It seems also that SME's tend to benefit more from R&D

consortia to establish their competitive position. A possible ex- planation for this is that smaller firms have more potential to grow than larger firms which have ah^eady established competitive positions. Furthermore, as organizational literature explains,

through R&D consortia firms internalize the externalities created through spillovers (this statement would open an interesting

question on the preference between single-company's R&D

project proposals vs R&D consortia's, from the financing agency

standpoint). Since SME's are less capable than large firms in general to appropriate externalities, it is likely that they may benefit more, all things equal, from participating in R&D consortia. If one also considers the learning process involved in joint-R&D

activities, in addition to the economic potential of their output,

SME's tend to have more to learn than (from) large companies,

which makes SME/large company partnerships more appealing.

The factors presented above belong on the good side of the

coin. A comprehensive analysis though, should also investigate

potentially less favorable aspects connected with SME's participation in R&D consortia. The latter regard mainly the implications for R&D project execution and program implementation.

Among the former, the relatively lower bargaining power of

SME's participatmg in R&D consortia together with large companies, raises the likelihood of moral hazard especially with re- gard to intellectual property rights issues and exploitation of

project results. As a consequence organization costs, such as those associated with monitoring opportunistic behavior of participants and aligning interests among participants, increase. Risk

ofproject failure should increase as well. Among the latter, less sophisticated information processing

systems and weaker capabilities by SME's to provide project

proposal assessors with all data and information needed to ad- equately carry out ex-ante project evaluation and selection, are

to be expected and counteracted (in my own experience with the EC as evaluator of around twenty R&D project proposals, I have

never come across a financial or economic cost-benefit analysis

correctly carried out, nor was I able to carry out one myselfwith

the information provided. This would open an interesting question about the interface losses between scientific-level debate,

ofthe kind we are having here, program formulation and program

implementation).

The positive reasons identified above whichjustify the pres- ence of SME's in publicly financed R&D consortia are based

essentially on economic potential issues. The ATP-like financ149
ATP PAPERSAND PROCEEDINGS

ing agency, though, is interested above all in the realization of

such economic potential. More particularly though, in those

projects where the gap between the expected social and private

rates of remms to the nation is particularly high.

From this perspective, we may rephrase the original question in this way: "What is the appropriate mix of partners in a consortium that assure the realization of the economic potential

of a project?" Excluding not-for-profit organizations as lead pro- ponents, ATP in a sense, has already given a partial answer to the question (although there may be other reasons unknown to me underlying such constraints).

A well balanced partnership in ATP-like R&D projects is probably one involving all or most, accordmg to the specific

project, of the following: technology providers; technology users/intermediaries; end-users; and venture capitalists. Equally

important is the convergence of the interest of all partners and a congruent resources/commitment participation of each partner. In Europe in particular, where the ratio of public R&D to private R&D investments is higher and the integration ofUniversities/public research institutions with industry is less strong

than in the United States, consortia composed of entities from

both sides may be worth promoting.

Partaerships which involve trans-sectoral dimensions should

be highly regarded as well, for the high potential oftrans-sectoral knowledge spillovers. Trans-regional dimension, instead, appreciated as a positive factor in the above mentioned EC program

(fostering cohesion), may be open to objections. While from a

social point of view, it may be regarded as positive to reward the

participation of less industrialized regions, the merit-based se- lection process would be unavoidably distorted. It is a matter of

the underlying philosophy. Someone would say: "Teach them

how to fish rather than giving them a fish". Others would reply:

"Give them some fish while teaching them how to fish".

2. What sort of technologies, e.g. infra structural, should be

targeted in publicly financed research consortia?

The most direct answer to such a question would be: those

technologies which are more likely to contribute to the achievement of the ends of the publicly financed program in question.

Then, for ATP-like programs, technologies which entail high

market and knowledge spillovers, such as: • infra technologies, i.e. those technologies that leverage

industry's ability to conduct every stage of technol- ogy-based activity;

• "systemic" technologies, i.e. those technologies whose

commercial value is strongly dependent on the devel- opment of other related technologies.

• "generic" technologies, i.e. those technologies with a wide range of potential market applications, which go

beyond the scope of corporate sfrategies of firms participating in the consortium.

• "bandwagon effect" technologies, i.e. those technologies whose development may ehcit significant developments of other technologies/applications.

• "breakthrough" technologies, i.e. those technologies

which may open up new fields of activity, whose embedded benefits are unlikely to be wholly appropriated

by the inventor.

• "gap filling" technologies, i.e. those technologies whose

development induces significant advances in the underlying knowledge.

• technologies whose appropriability regime is weak, be- cause ofthe inefficacy of the legal mechanisms ofprotection or the intrinsic nature of the technology (i.e. technologies whose embedded knowledge is codified

rather than tacit);

• technologies requiring certain complementary capabilities or assets, other than those possessed by the in- ventor.

Ifthe public financed R&D programs explicitly pursue social returns which go beyond the purely economic component of

social returns, such as the above said EC programs, additional considerations need to be made. Probably, the most urgent problem that Europe faces, and indeed has proclaimed it will immediately deal with, is unemployment (it is sufficient to compare EU

unemployment rate: 12%, with the U.S. rate: 5%, and Japan's rate: below 3% to realize the relevance ofthe problem). In this context EU publicly financed R&D programs in the fixture will most likely

stress even more the importance of "employment generation"

technology development.

Another aspect which may be considered from a social per- spective is the distribution of economic benefits arising from

publicly financed R&D projects. Two opposite positions confront themselves in the current debate: whether and to what extent one should rely on market forces alone for a fair distribution of economic benefits. The answer may depend on the particular

environment. In fact, another element ofdifferentiation between

Europe and the U.S. and Japan is that regional development is less homogeneous in the EU. Probably, market forces alone would

not be sufficient in Europe to reduce the gap among regions, or

at least to do it at an adequate pace. In the absence of the EC

programs specifically targeted to such problems, R&D programs

would favor the development of those technologies oriented

towards the needs of the industrial base of less developed re- gions.

150
OPENING REMARKSAND PANEL SESSIONS

The sort of technologies to be targeted in the R&D pro- grams we are discussing here may depend also on the level of

industrial specialization of and the industrial policy pursued by

nations. If we compare international specialization indexes of

high- and low-tech industries (in 1992, high-tech: U.S., 151; Ja- pan, 144; Europe, 82. Low tech: U.S., 74; Japan, 46; Europe, 113)

and their evolution along time, Europe competitive position in high-tech industries results relatively weak and deteriorating.

This awareness may well lead to specific industrial and technol- ogy poHcies, which will determine the sorts oftechnologies to be

piu^ued in R&D programs. Furthermore, such targeted technologies may vary across industrial sectors and along time. While in the past EC R&D policies seemed to emphasize the shifting of

the best practice production possibility frontier, more and more

diffusion concerns (moving industrial production units closer to the frontier) are being considered and affecting the formulation

and implementation of R&D programs (see the up coming EC

Fifth R&D framework program). Japan is another case in point.

R&D programs' ends and technology pursued therein, have

changed along time parallel to the shifting of Japanese industry

form a simple follower stage to one of leadership in an increas- ingly number of sectors.

3. To what extent should the funding government agency at- tempt to actively participate in the project?

In order to answer this question it should be defined what

"actively participate" means. In the world, it is possible to find examples of institutions whose mission is both to carry out re- search and act as funding agency as well. The National Research

Council of Italy, to which I belong, is a case in point. In these

contexts potential synergies between research and fimding activities may be realized. At the same time the risk of opportunistic

behavior gets higher, whereby projects evaluators may have more

or less direct vested interests in the projects under exammation.

For those government funding agencies which do not carry out research themselves, their more appropriate role is probably

that of "sponsor". As far as it falls within their power domain,

they should act as facilitators, smoothing obstacles, in order to improve the likelihood of success of the project. They should

also try to streamline the bureaucratic burden of project monitoring and make it functional also to better management ofthe project

itself If necessary, they should try to protect the project from

unnecessary distractions, such as the overlapping and redundancy ofbureaucratic requirements by different administrations. The most finitful participation in a project would be, in my

view, making the external environment as suitable as possible to allow for project success. This would entail to identify the critical factors of success of a project and the external variables which

mostly affect them, in order to intervene on them, if within the agency's capabilities.

The search for complementarities and synergies with other

similar programs is to be searched for, in order to avoid inefficien- cies or duplication of efforts. Finally, a way to transfer the learning curve ofparticipating

to such programs to new participants should also be devised, to enhance cost efficiency (it must be kept in mind that the larger the number of firms participating in a consortium, the more complex its management, therefore management costs increase. Fur- thermore, SMEs may have more problems than large companies

in adequately fiilfilling all program's requirements).

Concluding Remarks

The main conclusion that we may draw from the foregoing

discussion is that there are no standard answers to the problem.

Answers need to be nation-specific, reflecting context, policy,

and culture ofthe country at stake. Differences can be identified even at the very basic level of objective formulation ofprograms.

Is there a publicly financed R&D program in the US, EC, Japan, or anywhere else which does not pursue high social returns? Still, what does "social" mean to an American, a European and a Japa- nese? As can be extrapolated from the content, ends and eligibility/selection criteria of different nations' programs, the concept

social is open to a quite broad range of interpretations. Employment generation, geographical distribution ofbenefits, etc. seem

to be more highly valued by Europeans than by Americans or Japanese. Here, it is not a matter of somebody being right and

somebody else being wrong. Each one is right, m his/her own

context/culture.

I do not know if the results of this workshop are going to contribute to affect in any way future formulation of publicly

financed R&D programs. If this is the case, it should be bom in mind that considering at a very early stage the implications of

program changes on implementation is of utmost importance.

The implementation ofa publicly fmanced R&D program is nothing else than an optimization problem. Changes to a program may

affect either the eligibility criteria or the selection criteria or both.

Eligibihty criteria turn our optimization problem into a constrained optimization problem. The field ofpossible solutions to the problem may be more or less large depending on the number of constraints embedded in the eligibility criteria. We must be aware

then, that any time we add eligibility criteria we restrict the field of possible solutions. Eligibility criteria should be consistent with a prioritization ofobjectives. For example, the EC R&D pro- grams' eligibility criterion oftransnational consortia can be trans- lated mto "select those projects with the highest expected public

returns for the EC, as long as they are proposed by transnational

151
ATP PAPERSAND PROCEEDINGS

consortia". Contrary to what most people would expect, an inevi- table interpretation of the underlying message is that the main

objective is not strengthening European industry competitiveness, especially SMEs, as it is often emphasized. Rather, it is fostering European member states integration (or, at least it is a

prerequisite to the former). Nor can be thought that the objective

is to encourage setting up R&D consortia, as this could be done

as well with a national scope. Changes may affect also selection criteria. The larger the number of selection criteria, the more complex the selection process. The relative weight ofeach criterion and oftheir correlation should be defined with respect to the objectives of the program.

Also, it must be borne in mind that the task of an evaluator is extraordinary difficult. In addition to assessing the public potential (think ofall the factors which determine it) ofa project, he has

to assess then the likelihood of success (think again of all the

factors which may determme it), and finally the proportion of

benefits which are likely to be appropriated by domestic industry

vis-a-vis foreign competitors. Moreover, he has to do it under

time pressure and relying on data provided by parties with vested

interests and sometimes inadequate. At EC level these difficulties are fiuther amplified (I just remember evaluating a project few

months ago where, in addition to myself from Italy, there were two other evaluators from Germany and France, the EC officer in charge ofthe project from Belgium or, rather, Flemish (never confiise a Flemish with a Valloon), and two partners of the consortium, one from Denmark and one from Greece. You may wonder:

"was Britain missing?". Do not worry, we all spoke and read

documents in English. Probably, British English.

152
Professor Mariko Sakakibara

University ofCalifornia, Los Angeles

I would like to discuss some issues regarding the design of

publicly-funded research consortia based on my research on

government-sponsored R&D consortia in Japan. When the government is involved in the formation ofresearch consortia, gov- ernment planners or coordinators can determine the design of

consortia to achieve desired goals. The design of consortia

includes the types of companies that participate, characteristics of target technology, and the degree of government involve- ment. The goal the government wants to achieve through consortia formation partly depends on the national development

stage, and partly on the national innovation systems countries

possess. I want to discuss the issue of the design of consortia in a broader context surrounding R&D in general, including some

review ofthe historical development of R&D consortia in Japan.

Research consortia can change or enhance three major determinants of private R&D intensity, namely: technological opportunities; expected market size; and appropriability conditions.

First, research consortia can contribute to or increase the tech- nological opportunities companies face. For example, if the out- comes of research consortia efforts diffuse among and beyond

participants, they will stimulate a new round of innovative activities. Also, within research consortia, fums can bring different expertise together in order to create new technology. The provision of government subsidies might contribute to the enhancement of technological opportunities firms face because subsidies can reduce the effective cost of conducting R&D, which

might encourage firms to undertake more R&D.

Second, expected market size can be increased if uncertainty

regarding buyer preferences are resolved. Also, when several

standards exist and it is unclear which standard will become dominant, firms cannot focus then research direction on one stan- dard; this uncertainty can discourage furns' efforts toward innovation. If research consortia involve vertical cooperation, suppliers would have a better idea about the characteristics of the expected market because buyers can communicate with potential

suppliers about the specifications of products they might want.

Third, appropriability conditions can be improved through

the formation of research consortia because firms can set clear cost-sharing rules before R&D is executed and outcomes are

shared. Thus, the incentive to conduct R&D, which could be

hindered by potential spillovers of R&D outcomes, can be re- stored through the occurrence of R&D consortia. Different countries face different degrees of problems on

these three determinants of R&D intensity, and so the design of

research consortia should be set to address major problems a country wants to solve. When a country is in a developing

stage, for example, or behind in a certain technology area, a major

problem that type of country faces is the lack of R&D capability

by domestic firms to produce original R&D results. Research

consortia will be used to transfer and digest technology devel- oped by advanced countries. In such a case, participating fums

in a consortium will come from a narrow set of industries, be- cause the goal of the project is clear, almost set by other coimtries, and the major task is to take the most efficient research path

by dividing tasks and avoiding wastefiil duplications. I call this the VLSI project model. For these countries, large government

subsidies will be warranted against other countries. There will be a limited moral hazard problem in research consortia in this case, because there will be no significant difference

in R&D capabilities among firms. This implies that cooperation

among companies with different technological levels will be possible because no one firm will feel that it suffers from the opportunistic behavior of other companies to absorb as much as possible while keeping their own technology secret. Even if a fum

feels that some proprietary knowledge may be threatened, the

benefit to participating in research consortia, namely the govern- ment subsidies and access to advanced technology, will out- weigh such a potential loss. When a country is one of the technology leaders, the desired goal has to be changed from the model I just described. The major problem for a country will now be to generate original

technology which will be a core ofnext-generation technologies.

The goal to be achieved through research consortia formation

should be to increase technological opportunities available to companies, and to facilitate spillovers among companies. To

achieve this goal, governments should promote the formation of

consortia which consist of firms from diverse industries. In this case, the participants will face learning opportunities fi-om firms

with different backgrounds, and so it might be the case that firms

will have an incentive to increase then- R&D efforts in order to absorb from other members. Knowledge or skill sharing, rather than cost sharing, should be the main purpose of such consortia,

and government subsidies should play a less critical role compared with the previous model. In my analysis, I found evidence

that the knowledge sharing opportunity brought by diverse participants has a positive effect on participants' R&D efforts. Once a country achieves technological progress and firms

accumulate technological capabilities, it will be more difficult to ask technology leaders to participate in research consortia, be- cause potentially they can lose more than they can gain from

other participants. The target technology should be basic, be- cause when near-commercial technology is targeted, the conflict of interest will be greater among participants, which will exacerbate moral hazard problems.
ATP PAPERSAND PROCEEDINGS

Japanese consortia confirm the propositions above. I col- lected data on government-sponsored R&D consortia in Japan.

In my data set, I identified 237 R&D consortia occurring between

1959 and 1992. 1 171 companies participated in these consortia

during this period. I conducted a questionnaire to participants

of these R&D consortia in 1993. There were 398 useable re- sponses from 67 companies concerning 86 projects.

Questionnaire results showed that the breadth of participation in a consortium has increased over the years ofthe study. It

is also shown that the target R&D outcome has gone from near- commercial in the 1960s to more basic by the late 1980s. In my

sample, government-sponsored R&D consortia represent only a

small fraction ofthe total Japanese R&D expenditure, and the per

project budget peaked in the 1970s.

In the 1960s and 1970s, Japanese firms were desperate to catch up with Western technology leaders, facing direct competition from them after the trade and foreign direct investment

liberalization in the early 1970s. Japanese firms at that time did

not possess enough R&D capability to produce original R&D

results, and there was no significant difference in R&D capability among firms. Their financial resources were also limited. Government funding for R&D consortia did matter for participants,

and the subsidy provided to a typical consortium in the 1970s was eight times larger than the subsidy provided in the 1980s. As for the composition and the goal of research consortia,

participants tended to come from a narrow set of industries, and

the target technology was relatively near-commercial. Participation was a signal ofa firm's recognized R&D capability to outsiders such as buyer industries and banks, which encouraged the

participation of technology leaders. This model no longer holds in the 1990s. Differences in capability among firms has grown larger, and in interviews some

government officials noted that there are signs that technology

leaders are now hesitant to participate in R&D consortia. The

size ofgovernment funding per project has become smaller, firms have become less capital constrained, and so government funding is not a primary inducement of participation. Questionnaire

results suggest that firms can only agree on projects which are

distant from commercialization, and which are associated with

less conflict of interest. They can only agree on the projects

which require complementary knowledge, and ones which are impossible to conduct by themselves.

What is the role ofpublicly-fimded research consortia in the 1990s? The 1970s model of horizontal cooperation which was

driven by the motive of sharing fixed costs with heavy govern- ment involvement, as seen in the VLSI project or SEMATECH, is not an answer. A general recommendation for government policy

to developed countries is that government should set criteria to promote cooperative R&D which achieves its social goals and

minimizes risks to competition: encourage participation from dif- ferent industries; promote cooperation in basic research; pro- mote cooperation in emerging industries; and give partial funding to make government fiinding complementary. My analysis

suggests that though government-sponsored R&D consortia

will allow firms to conduct research projects which firms would

have done on a much smaller scale without government subsidies, government subsidies could work as a substitute to a firm's

total R&D spending.

Governments should note that cooperative R&D cannot substitute for other policy tools to stimulate private R&D. For ex- ample, cooperative R&D might have only a limited role in in- creasing technological opportunities and technological diffusion; fiinding of basic research, especially funding ofresearch in universities might be a more effective means for this purpose. For

countries with high researcher mobility, researchers contribute

to the dissemination oftechnological information, and so such a

role by R&D consortia might be limited. Also, cooperative R&D

does not solve demand uncertainty unless user industries are

also participating; public procurement is a more direct means for

this purpose, though it should be used only in the cases in which

the government can be a sophisticated and cost conscious buyer.

In conclusion, the design of research consortia, and tech- nology policy in general, should be considered in the context of

national innovation systems. Research consortia should be

treated as one of many tools govenmients can use to promote

R&D.

154
PANEL SESSION: Public-private Partnerships In R&D: Lessons Learned

LiamO'Sullivan

European Commission, Belgium

1. Why are we having this conference and how can one benefit from it?

I think the importance ofthis conference derives from a number of factors. First, I think it reflects the increased standing of

R&D in economic policy formation. From a European Union perspective, this is reflected in the design of the evaluation criteria for the Fifth Framework Program, which has put specific emphasis on economic factors, particularly economic growth and employment performance. These are largely macroeconomic considerations and are a measure of the concern felt by policy makers over poor economic growth performance in Europe and the need to cast the net wider than the traditional instruments to find

the means of solving the problem. The US, for its part, is enjoying a period of strong growth, which has depended in no small way on the contribution of high-tech industries. Thus across the world economy, there is an increasing appreciation and expectation ofthe economic role ofR&D investment. However, the specifics of this role have never been satisfactorily identified and

events such as this help to clarify how the task can best be

addressed. Second, this new emphasis on economic objectives implies

that we now have to measure research performance in economic

terms. We can no longer adopt the 'research is good' attitude which underpinned earlier approaches to evaluation. We are asked

to choose between alternative research programs and are thereby

confronted with the classic economic problem of resource allo- cation and therefore need a way of measuring impact. However, we are also confronted with a serious measurement problem.

This was more or less the conclusion of the Workshop we orga- nized on measuring RTD results/impact in Brussels last month

and the discussion here helps us in addressing the problem of

developing appropriate methodologies for measuring economic

impact.

Third, the classic benefit ofconferences and the reason those of us in the knowledge generation business encourage them is to

facilitate dissemination. Much of the material presented here in the last two days opens up promising avenues of exploration

which can potentially be applied in Europe to our own problems

of impact measurement. Perhaps my presence here will be justi- fied by the generation of a sort of transatlantic spillover effect. Fourth, there is no doubt a more skeptical political climate

underpinning the support for public research programs. Part of

the need to demonstrate results, and particularly economic results, is to fulfil a defensive purpose. The political motivation for

intervention to support research is increasingly based on the premise that it boosts economic performance - this is no doubt

part of the motivation for the increased reliance on partaerships

between public and private sector. However, as we have seen,

this support for public RTD is fragile and anything which can be

done, therefore, to demonstrate the positive impact of our efforts can only be helpful to those of us who believe that there is a role

for the government to play in the economic domain. Compared to the political hostility recently experienced in the US, the political

climate in Europe is in some respects more supportive ofgovern- ment support for R&D. However, the atmosphere is certainly more skeptical than heretofore and this is reflected in the adoption of the sfricter new selection criteria and although European

political attitudes remain solidly in favor of intervention, there is nonetheless evidence of a hardening of attitudes which means a more uncertain future for public research, particularly research in areas where the results are less tangible.

2. Which methodologies appear most promisingfor current us- age and which ones needfurther development?

Measurement ofeconomic impact is, as we have discovered

at European level, an area which poses serious methodological

problems. Part of the problem can be attributed to a mismatch

between the political objectives set for R&D performance which

are largely macroeconomic in character (improving economic

growth or alleviating the unemployment problem) and the means

available to tackle the measurement problem (at this stage, the only scientifically reliable methodologies available are microeconomic in character. Ofthese, the one most successfully

employed at European Framework Program level is that of the BETA institute at the University of Strasbourg which has re- sulted in the calculation of impact ratios for given levels ofRTD

investment in various European research programs. Perhaps the

best publicized BETA fmding was in relation to the 1993 smdy of

the BRITE-EURAM program which indicated a 7: 1 ratio of eco- nomic benefits accruing from the initial EU investment. The ap- peal of this methodology derives from its theoretical rigor and

empirical validity. However, the difficulty with such

microeconomic methodologies is their limited applicability to the macroeconomic objectives outlined above. Using such

microeconomic results in a political context can only give rise to serious problems of interpretation regarding the economic prowess of technology policy.
ATP PAPERSAND PROCEEDINGS

Of other approaches, the one which I found most striking

(even if its fmdmgs were not what supporters of intervention one would have wished for) was that used by Scott Wallsten and

documented in his testimony before Congress on the impact of

the SBIR program. Apart from its theoretical plausibility, the at- tractiveness of this methodology I think derives from its direct focus on the additionality issue and is therefore deserving of

wider application into areas where the results are likely to be more favorable to proponents of R&D support.

Other methodologies which are promising but are in need of

further refmement include some of those which have been pre- sented at this conference. From the viewpoint ofdesigning policy,

I was particularly interested in approaches which sought to establish a clear link between policy and subsequent performance. - of

particular note in this regard were those adopted by Klette and

Arvanitis because of their focus on measuring national policy effects directly. Another particularly interesting presentation for me

was that made by Francis Narin linking the effects ofbasic research

in subsequent patenting behavior because this could represent a promising technique for assessing the impact ofmany ofthe more

basic-research-oriented activities (within the Framework Program

there are many such activities). Overall, however, it is difficult not to agree with the opinion expressed here and also at our own May

workshop in Brussels that there are a number of dimensions to the measurement problem and a need particularly to establish links between the micro and other more aggregate levels.

3. What are the multiplepurposes andrationalesfor suchpartnershipsfrom a publicpolicyperspective andfrom the perspective of

private industry?

As regards the rationale for intervention, perhaps I can put my

contribution to the market failure argument as follows. Under tradi- tional economic growth theory, because of the assumption of di- minishiag returns, the actions of investors are assumed to only

haveprivate significance and there is no role for the public authorities in the capital formation process. However, in the absence of

diminishing returns, the actions of the private investor lead to under investment. This problem of under investment does not imply

any inadequacy on the part of the private investor - indeed, at the

level ofhis private decision, the assumption ofdiminishing returns may well hold. Another way of putting the argument is by considering that the development of ideas or know-how is a costly process, but once something is known, it can be repeated ad infinitum

at no extra cost. The existence of development costs, however,

implies that ideas or know-how are not in free supply and therefore,

need to be recompensated. Thus the new insight offered by endogenous growth theorists is that there is a spillover from the actions ofprivate investors taken in aggregate in terms ofnew knowledge

which generates additional returns.

This insight applies particularly to R&D investment activity

because ofits public goods character but also to any activity which

results in increased dissemination of knowledge (it is for this rea- son that new growth theorists additionally emphasize the importance oftraining and research networks). The dichotomy between

the private return to investors and the actual return imply market

failure and the case for intervention to boost RTD activity rests on

correcting this failure. From a policy standpoint, the inference is that RTD policy, which has fraditionally been viewed mainly as an

instrument ofmicroeconomic poHcy is now viewed as an economic

policy tool which also has macroeconomic significance with a clear

role to play in addressing problems of economic growth.

The notion of partnerships also corresponds to the needs of

both parties involved. From the viewpoint ofthe public sector, the

rationale is that it is a means of improving the economic efficiency

ofits research activities via expUcit links to the commercial sector. In

addition, it is a means ofguaranteeing the necessary dissemination

ofthe spillovers arising from the investment. From the perspective

of the private sector, it appears that there are a number of advantages to the arrangement. First, it is a means ofaccess to technologies via the other partners in the consortium. Second, there is evi- dence to show that European firms consider that this means of

research fimding is an important mechanism for reducing the risk involved in undertaking an R&D project. However, I think the most

interesting aspect ofthis question however, relates to the question

ofcapital market imperfections. The advantage ofpublic fimding is that it enables the firm to undertake research which it would other- wise not undertake, research ofa more risky character but therefore

likely to generate the sort ofhigh returns that we have seen in some

studies ofR&D impact in Europe. This type ofinvestment, which is not only of benefit to the firm (even under diminishing returns) is

critical to the achievement of societal goals.

Lastly, it is probably worth emphasizing that among the most

important economic effects ofresearch investment is the leaming

effect generated via the process of participation (network effects could be seen as atypical example ofthis). Several European studies have emphasized the importance of this factor in improving

long-term economic growth potential. One ofthe difficulties, however, with such long-term effects is that their measurement can be

extremely problematical.

4. Consider how these rationales and thefoci oftechnologypolicy

initiatives vary among countries: related to economic development, policypriorities, strengths/weaknesses ofthe private sector.

This question is a particularly interesting one from a European

perspective and offers me the opportunity to make the case for having a technology policy based at the European level. I think

the question is perhaps best addressed in any case by comparing technology policy in the US and the EU taken as a whole

156
OPENING REMARKSAND PANEL SESSIONS

because there is a market size dimension to this issue. Although

the larger member states ofthe EU still wield a significant degree

of independence in economic policy setting, the increasing globalization of economic activity means that the scope for mdependent action is diminishing. For the smaller member states, this scope for autonomy has never really been a consideration.

. Perhaps this is the reason why it is so difficult to demonstrate the

effects of R&D policy at individual country level (although Nor- way is not a member-state, the results of Tor Klette's study seem

instructive in this regard). That this interdependence is no bad

thing is recognized by the countries themselves through the

introduction of important institutional changes, the establish- ment ofthe Single Market being the most important ofthese. The

European focus of different aspects ofeconomic policy, notably

monetary policy, follows on logically from the Single Market

project.

The research position and its increasing economic focus

could also be taken to reflect this reality. Science being international in character, there was already a strong rationale for a European effort and the achievement of the Internal Market has

underpinned the establishment of stronger links between eco- nomics and RTD objectives and in particular, to strengthen the economic focus ofthe Framework Program. In this sense, a corre- spondence can be traced between the theoretical emphasis on

the importance of technology transfer and spillovers and the creation of the appropriate institutions at international level to foster these objectives. In summary, the economic rationale for a European research program could be said to be that it underpins the single market

and by encouraging maximum co-operation eliminate duplicative

effort at national level is the most efficient response to underlying economic realities. This market size aspect of technology

policy design has never been an issue to the same extent for the US and the relative success of its policy could be a reflection of

the importance of this structural feature.

157
Philippe Laredo

Centre de Sociologie de I'lnnovation

Ecole des Mines de Paris

1) Why are we having this conference and how can one benefit from it?

The title ofthe conference deals with 'economic evaluation' and 'technological change.' At the same time, a session dealing

with patents is labeled 'private innovation' and public research'. These terminological variations are revealers of the uncertain space in which policy evaluation develops. What is at stake,

what is put forward when promoting such policy initiatives deals with innovative capabilities of firms in a given country.

Both economics and sociology join to underline a set of

lessons about iimovation - and it is so rare a case that policy

analysts should really look carefully at them! Let me mention some very briefly since, to my point of view, they represent the core of arguments developed in cases evaluated (at least those I know of). * A first set of lessons imderlines the role of 'lead' users and

the importance of going from 'sub-contracting' to 'co-develop- ment' with suppliers. Both push towards industry alliances (which

de facto have witnessed and are witnessing an exponential

growth), but these are 'vertical' alliances (i.e., excluding any di- rect competitors)... and should not concern govenmients attentive to competition rules. The rationale for intervening in such

fields has to be found elsewhere: in the public benefits that could

derive from the new products, in answering a public good need. New drugs for orphan diseases, new tissues for cardiovascular

therapies, new energy options are some examples which do not only belong to the sphere of defense mdustries. They should

not be underestimated... and I must say, this is an issue which

has not been addressed yet in satisfactory terms.

* A second set of lessons deals with 'radical innovations'

(which following Apple and Microsoft are very fashionable ones

today). It highlights the decisive role of early choices (the so- called 'small events') in their trajectory; it shows the very long

cycles of most to go from their first niche market to generalized use (which makes them recognized as radical innovations that have transformed the world). Thus all the fashionable and politically very attractive about enlarging the number of those 'inventors' who get a chance for a first test : we have gone here to the simple notion of access to funds to this of helping them build

their very first 'niche', their first network of interested partners,

ATP being clearly positioned along this line. And as time (as well

as criteria) are not there to assess 'generalization', the core ofthe

effort goes along the definition ofwhat enters in this category of

'potential radical innovations': ATP seems to have been quite

innovative in this respect.

* A third set of lessons, following Polanyi's work on tacit dimensions on knowledge, insists, especially in so-called 'frontier science' or 'science in the making' on the need for direct connections for firms to be able to master internally the new

knowledge under production, to turn in into specific assets. Thus

the need for public-private relations and the boom in industryuniversity relations (at least in Europe where more and more

companies pay for PHD students, enter into lasting contracts with public research units...). To better grasp the fi-ame under which such partnerships

develop, this phenomenon needs to be linked with two other ones.

a) Many authors have highlighted the role of 'non technological' dimensions of innovations processes, locating the ability of

successful firms less in their technology portfolio than in their organization of the innovation process and in their so-called 'design capabilities' (I have used the terminology 'block build- ing model of innovation' to highlight the difference between the

'internal mastering of a technology' and the ability to mobilize

the relevant technological option in a given development). Thus

sharing a given technology with others might not be so frightful since the core of the competitive advantage lies in the ways it is mobilized. The danger is then not to share it with others, but to be barred from its access in due time.

b) At the same time iimovation analysts have highlighted the multiplication of technologies any product integrates and for each technology the explosion of potential new options. These

two simultaneous changes (in 'reality' or, may be more, in our understanding of innovation processes) explain why companies

enter more and more into partnerships while being competitors

and why in such partnership the 'public research' type component is often central. This promotes a different analysis of innovation dynamics where the notion of 'reservoir of knowledge'

remains central provided one accepts a small change: each eco- nomic actor has to build and shape its own reservoir. The idea of

doing this alone, however large the firm might be, has now gone,

less for financial reasons than for competence ones. Thus part- nerships have become or are becoming central to the shaping of
the firms' reservoirs ofknowledge, (To my point ofview the risk and cost dimensions are not central at this stage of 'internal mastering oftechnologies' ; they play a major role at later devel- opment stages .. with both the increases in mergers and in alli- ances specific to one product line)

May they be termed 'indirect' effects (Cohendet), 'socio- economic' effects (see Georghiou, this conference), 'structural' effects (see Laredo, 1995) or spillovers (as overwhelmingly used

in the US), these situations have been at the core of assess- ments, audits or evaluations. Still, there is a long way to go from

the micro level of case studies (now quite well documented) to the 'meso' level ofthe policy initiative or to the 'macro' level of a country / 'national system of innovation' (this latter dimension

pushing the issue of comparisons between policy initiatives and

ofthe national frame or 'system' for evaluation).

This long development was necessary for my own understanding of what the conference aims at, and of the differences

that can be witnessed between the American and European evaluation scenes. The former focuses more on producing figures

(what return for the public investment) while the latter insists more on images (what changes in the innovation landscape).

They might well be two comers of the same story, but they still remain to articulate...

2) Which methodologies appear most promising for current usage and which ones needfurther development?

It is very difficult to answer beforehand to this second question. Personally I would rephrase the question : there is no case

I know of which is satisfactorily described by only one method.

Thus the issue is the mix of methods which enable to have both

the 'figures' and the 'image', the numbers and the landscape.

Thus a central issue: which evaluation process for choosing the

relevant methods of investigation, those adapted to the mitiative under review or, as is more often the case, to each part of a given

policy initiative.

2) What are the multiple purposes and rationales for such

partnerships?

From a public policy perspective andfrom perspective ofpri- vate industry?

I needed to answer this question to address the first question posed. So may be it should be useful to start with this question. I have addressed the rationale from the 'innovation pro- moter' standpoint (which is not always equivalent to a private

firm, see Etzkowitz and his notion of research groups as quasifirms.From a public policy perspective, two elements are cenfral

a) 'Public goods' or 'collective issues' such as environment or

health drive a large part ofour economies. The choices are most

of the times built through a triangular relation between public

promoters (government but also states, counties and cities), pri- vate operators which will eventually be delegated operations

('utilities') and public sources of knowledge built by public authorities to advise them, to anticipate new problems, to open new

options for choice (new energy options being a typical case of

such approaches) b) fostering the adequate research infrastruc- ture as government intervene for transport infrastructures, for organizing telecoms, etc: there is now a new challenge for 'generalizing' the capacity of economic actors to access new technologies in the making (cf my third 'lesson')

The third element - high risk ventures - is the US way to the old fashioned French 'grands programs': in France we believed

in military type organized efforts along major indusfrial issues (with it must be said afterwards quite significant successes besides few exemplary failures); in the US there seems to be a belief

in the genius inventors - and that not enough make their way

ahead, so that public action should help giving a chance to 'claim- ing to be radical innovators'. One can easily understand that it faces difficulties for justification.

I leave aside all the 'market failure' type arguments - for those 'poor' small SME's which we must socially assist, for those

lost public researchers who want to create their own companies,

etc. We all face these politically fashionable and sensitive problems. They are out of the present debate.
Five Concluding Remarks from the Presentations and Discussions

Philippe Laredo

It is a difficult but challenging task to draw some conclusions from ten presentations, two panel sessions and two days

of extensive discussions. Being external to the field ofeconometric studies which were the main focus of this conference, I shall focus on the conditions for development and limits of such methods, as put forward by the different speakers and the audience.

My comments will be organized around five points: semantic issues, the importance ofcontext, the take-up ofpolicy choices

and implementation within models, the balance between model

development and data quest, and finally, the focus of research on evaluation (methods vs process).

Semantic issues as revealers of different frames of analysis.

The keyword in this conference forjustifying public intervention

in innovation was without doubt "spillovers". It is symbolic to see its overwhelming presence in all but one titles of presentations by U.S. speakers while it was completely absent from Euro- pean ones where the main wording referred to socio-economic,

indirect or structural effects. Internalizing "externalities" (an ex- pression rarely used by both "sides") takes thus complementary

routes on both sides of the Atlantic Ocean. Spillovers account

for an approach where overflows are both barriers to private

investment and reasons for public involvement. Public intervention is traditionally there to design more effective frames (e.g. fostering appropriability) and/or to compensate for leakages. But

if overflowing is taken as an infrinsic dimension of market dynamics (Gallon, 1998), organizing public intervention to foster "socio-economic" effects is to be considered as an objective for public intervention and a "win win" situation where both given

private actors (e.g. a company dedicated to gene therapy) and

wider subsets of the society (e.g. patients with cystic fibrosis) will benefit from public support to the former. How to identify

such areas? Which criteria and support mechanisms to design?

These questions have been central both to ATP and EU pro- grams. ATP has developed extensive sets of criteria for qualifying these situations while European Programs have insisted on

consortia and private-public collaborations as a marker of such

situations. This joins the point made by Jaffe, Singh, and Fogarty

in their presentation: spillovers are localized, they do not constitute "pools" but are linked to existing "networks" (see our own

approach for EU programs, Laredo 1998). Semantic differences,

as well as attempts to bridge them, act thus as revealers of differ- ent focuses in policy making.

Context Matters:

A striking second result dealt with the context in which tech- nological change and innovation happen. Analyzing patents,

and citations of scientific papers, Narin insisted upon local effects, showing that this phenomenon does not only apply be- tween nations but within nations, highlighting in patents the strong geographical correspondence (similar states) between

inventors and the scientific references made. Also using patents, Jaffe, Fogarty et al, went one step further, developing a model linking the potential for spillovers to fiams' system position. But "innovation networks" do not only encompass those

actors dedicated to technological change may they be from university, national labs or companies. Involvement is far more het- erogeneous. Gompers and Lemer, for mstance, told us that with- out taking into account the 1979 change by the Department of

Labor in the way pension funds account for risk assessment

(from project to portfolio risk assessment), Venture Capital would

not have developed the way it has, hampering the explosion of new high tech companies which build up the core ofthe fransfor- mations observed this last decade. How to take context into consideration in econometric studies is therefore a central challenge.

Policy Matters:

However central, this is not enough a challenge, were we told by

other speakers. The choices and forms ofpublic intervention are

also to be accounted for. To explain the positive effect ofpublic

support on firms efficiency in Israel as opposed to results obtained in Norway or in the U.S., Griliches and Regev arguments

revolved around the framing and implementation ofpoUcies: "gov- ernment has been more lucky in selecting firms", "the policy was

less planned (than in Norway), more bottom-up, with more opened

windows", "they were able to give the grants to good people".

Such explanations, as well as arguments developed by Klette about "institutional inertia" and "complexity of coordination",

raised a new set of issues. What difference does it make in the long run for public intervention when support is focused on

"potential Microsoft in their early stages" or when it is used for "raising the technological capability of a given set of actors"? when "picking up wiimers" is opposed to "caring more for those
OPENING REMARKSAND PANEL SESSIONS

which have difficulty to absorb new technologies (generally

SMEs)? Or when different forms of support are used (tax incentive vs direct support, grants vs reimbursable loans in case of

success, etc)? Can such issues be left out from the models?

The balance between model development and data quest.

Such a question was directly related to the following one: What

is the adequate balance between further model refmement and

the access to relevant data? How much can evaluation studies

rely upon available datasets? The impressive collecting effort made by ATP (as presented by Ruegg) is to be related to the investment made by CHI for being in a position to follow the

relations between patenting and scientific activities or by

Carnegie Mellon for Cohen to study industry appropriability strat- egies. The quest for ad-hoc tailored data "at project level (flmded

or not)" or "through direct surveys of firms" was voiced more

than once in these two days. The conference ended with the very strong plea by Jaffe advocating for controlled experiments

as in clinical research and as witnessed for social policies: would

firms be the sole entities to escape this well established evaluation approach?

Return to semantics: evaluation methods vs evaluation process. Such a question resonated with previous presentations by

European speakers- Georghiou on socio-economic effects and

additionality, Dreher on lessons to be drawn from the German

evaluation experience. The focus was there less on methods and more on the embedment of evaluation m a larger policy frame. Evaluations are not seen as one-off events but as periodic, feeding into "strategic managemenf. What is then at stake is the evaluation process through which present-day issues are identi- fied, relevant approaches selected and interaction with

policymakers and stakeholders at large organized (for a description of such articulations in Europe, see the book edited by Callon

& al, 1 997). Dreher advocated for giving to evaluations a "moderating task creating transparency in a multi-actor network" and

thus "serving as support for enlarged debates in wider arenas than solely administration and directly interested parties".

Henderson spoke of evaluations as a means for "structuring

conversations". These positions underline the fact that evaluations do not exist in a vacuum, that they are part of the policymaking process and widely differ from control exercises. It is not

their primary objective to distribute good and bad marks. Looking into past action serves to learn about future action. As highlighted for innovations, this learning process is cenfral for policy

development. Policy making can no longer be seen as a sequential or linear but as an interactive, whirling process where evaluations' role is to help in the closure of each loop, feeding into the debate for defming the next loop. Following Leeuw & al (1994)

using Argyris approach, this learning process covers the whole

of the policy frame and simultaneously addresses objectives

pursued and implementations structures (Rip, 1987). Evaluation

methods require thus to be related to the stage of development

of the policy under consideration, to the frame in which evaluation takes place and the nature of questions presently raised. It was not then surprising to witness within the audience a plea for enlarging the "toolbox ofrobust methods" and for a greater mix

of qualitative and quantitative methods in evaluation processes.

Callon M., P. Laredo & P. Mustar (eds). "The Strategic Management of Research and Technology, The evaluation of

Programs, Paris." Economica International (distributed by

Brookings Institution Press), 1997.

Callon M. "An Essay on Framing and Overflowing: Economic

Externalities Revisited by Sociology," in M. Callon (ed). The

Laws ofthe Market, Oxford: Blackwell, 1998.

Laredo P. "The Networks Promoted by the EU Framework

Programme and the Questions they Raise about its Formulation

and Implementation," Research Policy, vol. 27, 589-598, 1998.

Leeuw F, R.C. Rist and R.C. Sonnichsen. Can governments

learn? Comparative Perspectives in Evaluation and Organizational Learning. New Brunswick: Transaction Publishers, 1994.

Rip A. & A.J. Nerderhof "Between Dirigism and Laissez-Faire: Implementing the Science Policy Priority for Biotechnology in the Netherlands." Research Policy, vol. 15 253-268, 1986.

161

About the Advanced Technology Program

The Advanced Technology Program (ATP) is a partnership between government and private industry to conduct high-risk research to develop enabling technologies that promise significant commercial payoffs and widespread benefits for the economy.

ATP provides a mechanism for industry to extend its technological reach and push the envelope beyond what it otherwise would

attempt. Promising future technologies are the domain of ATP:

• Enabling technologies that are essential to the development of future new and substantially improved projects, processes, and services across diverse application areas;

• Technologies for which there are challenging technical issues standing in the way of success; Technologies whose development often involves complex "systems" problems requiring a collaborative effort by

multiple organizations;

• Technologies which will go undeveloped and/or proceed too slowly to be competitive in global markets without ATP.

ATP funds technical research, but it does not fund product development. That is the domain of the company partners. ATP is industry driven, and that keeps it grounded in real-world needs. For-profit companies conceive, propose, co-fund,

and execute all of the projects cost-shared by ATP. Smaller companies working on single-firm projects pay a minimum of all the indirect costs associated with the project. Large, "Fortune-500" companies participating as a single firm pay at least 60

percent of total project costs. Joint ventures pay at least half of total project costs. Single-firm projects can last up to three

years; joint ventures can last as long as five years. Companies of all sizes participate in ATP-funded projects. To date, more

than half of ATP awards have gone to individual small businesses or to joint ventures led by a small business. Each project has specific goals, funding allocations, and completion dates established at the outet. Projects are monitored and can be

terminated for cause before completion. All projects are selected in rigorous competitions which use peer-review to identify those that score highest against technical and economic criteria.

Contact ATP for more information:

• On the World Wide Web: http://www.atp.nist.gov

• By e-mail: atp@nist.gov

• By phone: 1 -800- ATP-FUND (1-800-287-3863)

• By writing: Advanced Technology Program

National Institute of Standards and Technology

100 Bureau Drive, Stop 4701

Gaithersburg, MD 20899-4701

About the Editor

Dr. Richard N. Spivack has been with the Advanced Technology Program (ATP) since 1994 where he is a member

of the Economic Assessment Office. Dr. Spivack divides his time between undertaking research into the ATP's long-term

economic impacts upon the U.S. economy and serving as both the business chair and project manager in the Infomration

Infrastructure for Healthcare focused program. Dr. Spivack received his Ph.D. in Economics from the University of

Connecticut.
1901-2001

S >

si

O CL,