9/30/2020 https://web.archive.org/web/20090822160544/http://www.nist.gov/director/prog-ofc/rept-961.htm

https://web.archive.org/web/20090822160544/http://www.nist.gov/director/prog-ofc/rept-961.htm 1/23

Economic Impact Assessments:

Guidelines for Conducting and Interpreting Assessment Studies

by

Albert N. Link

Professor of Economics

University of North Carolina at Greensboro

May 1996

Acknowledgments

I. Introduction to Economic Impact Assessements

A. Purpose of the Guidelines

B. What is an Economic Impact Assessment?

C. Why Conduct an Economic Impact Assessment?

C.1. Public Accountability and Documentation of Value

C.2. Enhance Research Management

D. Outline of the Guidelines

II. How to Conduct an Economic Impact Assessment

A. Understanding the Stakeholders for the Assessment Study

B. Identify the Population of Candidate Research Projects

B.1. Historical Projects

B.2. On-Going Projects

B.3. Planned Projects

C. Developing Project Selection Criteria

C.1. Feasibility

C.2. Representativeness

C.3. Describability

D. Select a Candidate Project

E. Prepare an Historical Overview of the Project

E.1. Chronology of the Project

E.2. Technical Outputs from the Project

E.3. Current and Expected Users

F. Formulate An Economic Impact Assessment Strategy

F.1. Identify Elements of Cost

F.1.a. Direct Costs
9/30/2020 https://web.archive.org/web/20090822160544/http://www.nist.gov/director/prog-ofc/rept-961.htm

https://web.archive.org/web/20090822160544/http://www.nist.gov/director/prog-ofc/rept-961.htm 2/23

F.1.b. Indirect Costs

F.2. Identify Areas of Benefits

G. Select an Economic Impact Assessment Method

G.1. Guidelines for Selecting an Assessment Method

G.2. Alternative Assessment Methods

G.2.a. Benefit-Cost Analysis

G.2.b. Internal Rate of Return Analysis

H. Conducting the Economic Assessment

H.1. Cost Considerations

H.2. Benefit Considerations

H.3. Data Collection Process

I. Presentation and Interpretation of the Assessment Findings

Appendices:

A. A Numerical Illustration of Alternative Assessment Methods

Bibliography

Acknowledgments

Extremely useful comments on earlier versions of these guidelines were received from Elaine Bunten-Mines,

Greg Tassey, Bob Hebner, Harold Marshall, Bob Chapman, Francois Martzloff, George Mulholland, and Joe

Carpenter, all from the National Institute of Standards and Technology. Also, David Leech of TASC, Dave

Roessner of the Georgia Institute of Technology, and John Scott of Dartmouth College offered excellent

suggestions throughout the project.

I. Introduction to Economic Impact Assessments

A. Purpose of the Guidelines

The purpose of these guidelines is to familiarize program and project managers at the National Institute of

Standards and Technology (NIST) with the motivation for and mechanics of an economic impact assessment. It

is intended that this overview information will assist managers:

understand the importance of economic impact assessments,

recognize economic impacts as they occur over the life of a research project so that relevant information

can be documented in real time,

work with independent consultants when formal assessments are conducted, and

interpret the findings from a formal impact assessment for both internal planning and external

communication with industry clients, industry partners, and policy makers.

B. What is an Economic Impact Assessment?

The purpose of an economic impact assessment of a NIST research project is to evaluate, both qualitatively and

quantitatively, the benefits to industry that are associated with the research project and to compare those

benefits, in a systematic manner, to the costs of conducting the research project.[1]
9/30/2020 https://web.archive.org/web/20090822160544/http://www.nist.gov/director/prog-ofc/rept-961.htm

https://web.archive.org/web/20090822160544/http://www.nist.gov/director/prog-ofc/rept-961.htm 3/23

An economic impact assessment is different from other evaluation efforts that take place at NIST. For example,

laboratory directors regularly conduct or contribute to program evaluations. The purpose of these evaluations is

to determine how well the portfolio of research projects within an industry- or technology-focused programmatic

area aligns with the objectives of the program or laboratory; to understand how effectively the program is being

managed; and to assess progress toward program and broader NIST objectives. Thus, a broad range of

evaluation-based metrics are considered, including quantity and quality of technical work, intensity of

interactions with constituents, level of satisfaction among constituent groups, variety and quantity of technical

output, and some general indication of industry impacts.

In contrast, economic impact assessments focus on the changes in financial and strategic variables within

industrial organizations whose activities are directly affected by NIST research and associated services.[2]

Impact assessments are therefore narrower in scope than program evaluations and, in fact, contribute to overall

program evaluation. Economic impact assessments are generally conducted on completed or on-going projects

that have been funded as a result of some prior project selection process. Such assessments are not intended to

identify new research areas or to replace standard research project selection exercises, but they frequently

contribute to program planning.

C. Why Conduct an Economic Impact Assessment?

There are at least two reasons for the laboratories at NIST to conduct economic impact assessments:

to ensure accountability and document value, and

to enhance overall management effectiveness.

C.1. Public Accountability and Documentation of Value

Research at NIST is funded with public resources, and therefore NIST, like any other government agency is

accountable for the efficiency with which these resources are used.[3] The recent history of policy initiatives

demonstrates the increasing concern about accountability with respect to the efficient use of public resources.

The issue of accountability underlies the Competition in Contracting Act of 1984 (P.L. 98-369), the Chief

Financial Officers Act of 1990 (P.L. 101-576), and most recently the Government Performance and Results Act

of 1993 (P.L. 103-62).[4]

Also, project managers will understandably become advocates for their own research. Watching results on a

daily basis and witnessing technical accomplishments understandably leads these managers, and the researchers

working under them, to the intuitive conclusion that the project is valuable. Regardless of the veracity of this

conclusion, such perceptions may not be easily or convincingly communicated to others. Moreover, in-house

criteria for assessing effectiveness are often different from those of policy makers who have, so to speak, the

final say on a project's effectiveness.

Thus, when policy analysts or budget analysts ask, "But how do you know this project is effective?" project

managers often find themselves relying on "activity" indicators, listing technical accomplishments, or simply

telling anecdotal success stories.[5] Although each of these categories of information has some utility, none

completely or accurately captures the scope and magnitude of economic impacts the bottom-line metric for

policy makers.

It is possible through a systematic economic impact assessment to document value and thereby provide a clear,

more precise response to the question of accountability. Future NIST budgets will increasingly depend on the

demonstration of economic impacts.

C.2. Enhance Research Management
9/30/2020 https://web.archive.org/web/20090822160544/http://www.nist.gov/director/prog-ofc/rept-961.htm

https://web.archive.org/web/20090822160544/http://www.nist.gov/director/prog-ofc/rept-961.htm 4/23

The results of economic impact assessments provide information that can be used to improve both overall

managerial efficiency and strategic planning. The findings from such assessments can identify the research

outcomes that are most useful to the program's constituents (e.g., industry), as well as define any barriers that

might exist which inhibit fuller access to these research results. With such information, management will

increase the credibility of its forecast impacts, which is not only critical to strategic planning but also

fundamental to the budget process.

At the same time, the information gained from an assessment provides insights into the types of companies,

stages in the technology life cycle (e.g., commercialization, fast growth, maturity), or states of economic activity

(e.g., R&D;, production, marketing) that benefit the most from NIST research. Such information can be an

important contributor to strategic planning.

D. Outline of the Guidelines

The following section outlines, in a step-by-step fashion, the cognitive and investigatory steps to follow when

conducting an economic impact assessment. The text is written from the perspective of a consultant, and the

issues that concern a consultant therefore motivate much of the format and content. The reason for this approach

is to help NIST technical managers better understand their requirements when participating in an assessment,

and to help select a competent consultant.

This mode of presentation is not intended to suggest that economic impact assessments are conducted in a

cookbook-like fashion. In fact, no template exists that applies to all project assessments. On the contrary,

economic impact assessment is partly art and partly science. The art of assessment is in the creative tailoring of

available techniques and methods to the specific project at hand and to the available data. While there are many

standard conventions in assessment, such studies are seldom routine. The practice of economic impact

assessment involves a series of judgments in the application of assessment methods and the creation of research

designs that are only partly standardized. Creative techniques are frequently required in the actual collection of

data in order to attain acceptable response rates. The science of assessment is focused in two critical areas: the

design of the survey instrument and the various techniques and methods developed for analyzing data and

presenting results.

There is an appendix to the guidelines that numerically illustrates two alternative assessment methods.

II. How to Conduct an Economic Impact Assessment

A. Understanding the Stakeholders for the Assessment Study

The first question that a consultant will ask is, "Why is the assessment study being conducted?" The purpose of

this question is not to attempt to bias the results of the assessment in any way, or to purposely focus it to

highlight a program manager, project manager, or researcher. Rather, it is important to have an up-front

understanding of the purpose or need for the study in order to provide an initial indication of what types of

impacts were intended when the project was formulated and what types of information are important to the

audience(s) for the final report.

One group of stakeholders is the project's managers. Objectives in this case will include a better understanding

of how the project being studied aligns with the broader goals of a program or how the types and magnitudes of

the resulting economic impacts compare with other projects. They may find this information useful for the

general management of the project and the broader set of projects in which they are involved. Additionally, they

may have been asked by others to conduct the study as part of a higher level review, which implies emphasis on

the project's contribution to broader program objectives. Other groups of stakeholders include policy makers in

the executive branch who oversee technology infrastructure policy such as the several management levels above

NIST in the Department of Commerce and in the White House, with the latter including the Office of

Management and Budget (OMB), Office of Science and Technology Policy (OSTP), Council of Economic
9/30/2020 https://web.archive.org/web/20090822160544/http://www.nist.gov/director/prog-ofc/rept-961.htm

https://web.archive.org/web/20090822160544/http://www.nist.gov/director/prog-ofc/rept-961.htm 5/23

Advisors (CEA), and the National Economic Council (NEC) and policy makers in Congress who oversee the

budget process: both the authorization of roles and the subsequent appropriation of funds.

Identifying stakeholders often helps the consultant understand who is likely to read the final report. It is often the

case that a project researcher who reads an assessment study will look primarily at the technical details in the

report and will make a judgment about its validity and usefulness on that basis; a project manager may be

looking primarily for information to help allocate resources more effectively; a laboratory director may look

primarily for bottom-line information that can be used to summarize to others the net value of the projects to its

consumers of the research (e.g., industry); the NIST Director may look primarily for summary statistics, and

anecdotes to substantiate them, so that generalizations can be made and substantiated to administrators in the

Department of Commerce or in Congress; and policy makers outside of NIST may look primarily for a well

documented and readable account of how resources are being allocated and what economic benefits are being

realized. Obviously, the technical detail emphasized in the final assessment report will vary depending on the

primary audience.

The above statements are generalizations designed to emphasize that different individuals will read an

assessment report and will look for different information. For this reason, the consultant will require a clear

understanding of who the stakeholders of the study are and how they might use the results of the study.

In addition to understanding the purpose and need for the assessment study, it is imperative for the consultant

and the stakeholders to reach an up-front agreement on the level of resources to be devoted to and the time frame

for the conduct of the study. Part of the cost, of course, is the payment to the consultant. However, in-house

resources (mostly staff time) must also be expensed to assist with the assessment. The consultant will rely on

those involved in the project under study to participate in the assessment process at several levels and at specific

times during the conduct of the study. For purposes of discussion, the in-house individual(s) will be referred to

throughout the guidelines as the project liaison.

The project liaison will be asked to:

document the historical costs associated with conducting the project, and

provide specific background information on the history and scope of the project, the project's technical

output, the users of the project's output (e.g., particular industries or industry groups), and key individuals

or groups of individuals that can be contacted and perhaps surveyed regarding how the project's outputs

are used.

The technical outputs from a research project are discussed in detail in Section E.2 below. These outputs must be

well-defined and bounded, and directly related to the NIST research. Defining technical outputs too broadly,

such as in terms of basic or fundamental knowledge, and failing to link them to economic impacts, will seriously

inhibit the usefulness of an assessment. Basic or fundamental knowledge generated from NIST projects often

results in specific and identifiable technical outputs, which in turn can be shown to have been embodied in

databases, test methods, calibration services, production process models, standards, and the like. These NISTgenerated technical outputs have economic impacts outside of NIST.

Also, as discussed in Section F.2 below, identifying the users of the project's technical output and individuals

who can act as "gatekeepers" to sources of economic impact information is imperative for the assessment to be

successfully completed. For the consultant to be able to interact with such identified individuals in a meaningful

way, the project liaison will, in all likelihood, have to "open doors" by making initial introductions. These

activities are perhaps the most critical aspect of involvement by the project liaison, and they may be time

consuming.

As one example, when assessing the economic impacts associated with the Electronics and Electrical

Engineering Laboratory's power and energy calibration services, the project liaison identified by name the key

individuals to contact in each company in the affected industries.[6] Had this not been the case, significant

additional effort would have been expensed in locating such individuals within each standard meter and
9/30/2020 https://web.archive.org/web/20090822160544/http://www.nist.gov/director/prog-ofc/rept-961.htm

https://web.archive.org/web/20090822160544/http://www.nist.gov/director/prog-ofc/rept-961.htm 6/23

watthour revenue meter manufacturing company to solicit their participation; in fact, some of these individuals

might never have been identified.

Regarding the time frame of the assessment study, it is commonplace for project managers to underestimate the

up-front effort that they and the project liaison must devote to the study. It is often more realistic to set a

completion date for the final assessment after this initial phase has been completed. Past experiences with the

evaluation of NIST projects leads to an estimate of one work week of effort from the project liaison throughout

this initial phase. Given that the project liaison is key to other research projects, it is often the case in practice

that this work-week of effort becomes spread over a several month period. In such situations, the consultant may

have to acquire the necessary data directly, which adds to the project's cost and delays completion. To ensure the

availability of the project liason's time, higher-level management must remain visibly committed to the project's

objectives and timely completion.

B. Identify the Population of Candidate Research Projects

While the discussion in the previous section presumes that a research project has been identified, that

identification process entails some important considerations. Having some idea of the level of effort that a

project manager and project liaison will have to devote to the assessment is relevant for the selection of a

candidate project.

The population of "assessable" research projects can be broadly grouped into three categories:

historical (completed) projects,

on-going projects, and

planned projects.

Historical projects are those that have been completed and therefore for which the economic outputs and costs

are at least theoretically obtainable. On-going projects, as the name implies, are active and thus there will be

future outputs and costs that are unknown, but perhaps can be estimated through extrapolation or other

forecasting techniques. Finally, there are planned projects, which may or may not have an estimated budget but

certainly have no documentable results. An economic impact assessment can be done on any one of these three

types of projects, but there are both advantages and disadvantages with each that should be considered.

B.1. Historical Projects

The advantage of selecting an historical project to study is that all outputs and costs are theoretically known and

identifiable. Because the project is completed, users of the outputs from the project will likely be able to discuss

their importance.

A disadvantage of selecting an historical project is that some institutional memory and documentation may have

been lost. Moreover, the project may no longer be representative of the type of research project being undertaken

in a program (in the event that someone tries to extrapolate the findings from one project assessment to other

program projects).

B.2. On-Going Projects

On-going projects can either be long lived or recently begun. Studying an on-going project can mitigate the

potential disadvantage of an historical project, namely the loss of some impact data, but it has a potential

disadvantage that future outputs and costs are unknown. However, depending on the nature of the project, these

future outputs and costs can often be estimated with some degree of reliability. In some cases, an "on-going"

project may be an extension of a long-lived project, which means that data collection can suffer for the same

reasons found in historical projects. In these situations, it may be possible to divide a long-lived project into

subprojects with bounded lifetimes.
9/30/2020 https://web.archive.org/web/20090822160544/http://www.nist.gov/director/prog-ofc/rept-961.htm

https://web.archive.org/web/20090822160544/http://www.nist.gov/director/prog-ofc/rept-961.htm 7/23

Ideally, an economic assessment is designed and implemented at the same time as the research project, and

impact data are collected throughout the life of the project. As real-time data collection is the most efficient

approach, this is the preferred strategy. However, availability of funds and priorities assigned to completed

research projects often preclude this approach. Moreover, NIST laboratory research projects are numerous and

typically last at least several years, which makes the a priori selection of projects to be studied difficult. That is,

given the relatively large number of projects in a programmatic area, all of which may last, say, five to seven

years, a priori selection for real-time data collection is problematic at best. To partially address this problem, one

of the purposes of these guidelines is to increase the NIST manager's sensitivity to the need to recognize and

note sources of impact data that arise during the conduct of a research project. Such a commitment will

significantly improve the quality of impact studies and increase the number that can be undertaken.

B.3. Planned Projects

Planned projects fall into the category of strategic planning and therefore should be viewed somewhat differently

from retrospective or on-going projects. In fact, several completed projects should be assessed in a program area

before attempting to forecast the impacts of planned projects, as the former can provide an empirical basis for

choosing values for variables in the latter forecasting exercise.[7]

C. Developing Project Selection Criteria

The criteria that a consultant will emphasize when selecting a candidate project are not independent of the

purpose of the assessment. There are at least three relevant criteria:

feasibility,

representativeness, and

describability.

C.1. Feasibility

This criterion is perhaps the most important to consider. First, as implied in Section B above, relevant

information about the project must be available. This in turn requires that the population of consumers of the

outputs from the project must not only be identifiable, but also accessible in a timely manner.

Second, the scope of the user population has to be considered. Some NIST projects affect a large number of

industries. This can significantly increase the resources required to complete an assessment. Moreover, some

database development or basic standards work can affect multiple industries, and do so indirectly through yet

additional industries or secondary standards sources. Such infrastructure research can have large aggregate

economic impacts, but are also very difficult to assess. For one thing, several distinctively different user

populations have to be defined and surveyed. Moreover, the value added by secondary and tertiary user groups

has to be netted out in order to obtain estimates of NIST impacts.

It is also helpful if the candidate NIST project has an expected impact that is reasonably well-defined with

respect to industry's own investments that are typically occurring simultaneously with NIST research. This is

relatively easy to accomplish when the infrastructure contribution is targeted at a single stage (e.g., production)

in a single industry (e.g., optical fibers). It is more difficult when multiple groups are involved, some of whom

are in the public sector (e.g., users of power and energy calibration services) where non-investment impacts such

as "administrative efficiencies" are found.

C.2. Representativeness

If the results from the economic impact assessment are to be used to generalize to other projects within a

program, then the candidate project should be representative of the projects that are currently underway or

expected to be conducted within that program.[8] Dimensions of representativeness include:
9/30/2020 https://web.archive.org/web/20090822160544/http://www.nist.gov/director/prog-ofc/rept-961.htm

https://web.archive.org/web/20090822160544/http://www.nist.gov/director/prog-ofc/rept-961.htm 8/23

type and scope of the project's target technology infrastructure (e.g., measurement data, test methods,

calibration services, basic standards setting),[9]

project status (historical versus on-going),

project size and budget, and

industrial groups that use or are expected to use the outputs of the project.

C.3. Describability

Given a subset of feasible and representative projects, consideration should be given to the audience for the

assessment report. If it is expected that the report will have a high degree of visibility because of the Director's

agenda, or because of the political atmosphere in Congress or in other government agencies, a project's

describability should be considered. In other words, is the candidate project one that can be described to a wide

audience of non-technical individuals in a way that will be easily understood and for which the importance of

such research will easily be recognized?

For example, a broad multidisciplinary audience should more easily understand an optical fiber infrastructure

project than one dealing with, say electromigration.[10] The physical representation of a fiber and associated

measurement problems such as core diameter, cocentricity, and rate of signal loss are generally easier to

visualize and understand than the concept of measuring the rate at which metallic atoms migrate from a metal

conductor into a surrounding insulator substrate on a semiconductor chip.

D. Select a Candidate Project

Typically, the above criteria will receive different weights depending on a number of factors, such as the number

and type of impact studies already completed and the diversity of projects within an organizational unit. Again,

depending on the use of the economic impact assessment, various individuals at different administrative levels

may be involved in the final selection of a project for study. Once the candidate project is selected, the overall

selection process should be documented, and this documentation should be reported or summarized in the

assessment report. It is generally advisable to have the consultant participate in writing this.

E. Prepare an Historical Overview of the Project

At this stage, the consultant will work closely with the project liaison to prepare an historical overview of the

project. This overview, along with documentation of the project selection process, will become an important part

of the final assessment report.

E.1. Chronology of the Project

This overview should contain background information on the history of the project, including the:

motivation for beginning the project,

targeted industries and expected impacts, and

historical resources devoted to support the project.

E.2. Technical Outputs from the Project

Describing the technical outputs from a research project is often an easy process for a scientist or engineer, as

would be expected given that such an individual is closely involved in the project and is trained to view

technical knowledge per se as the direct and meaningful output. However, the consultant, who is likely not to be

a scientist or engineer, will emphasize that an easily understood description of technical outputs is needed for

general audiences of economic impact assessment, and in some cases, for the conduct of the actual assessment.

Such specific categories of technical outputs might include:
9/30/2020 https://web.archive.org/web/20090822160544/http://www.nist.gov/director/prog-ofc/rept-961.htm

https://web.archive.org/web/20090822160544/http://www.nist.gov/director/prog-ofc/rept-961.htm 9/23

measurement data,

calibration services,

test methods, and

basic standards setting.

Most non-technical readers of an assessment report will be able to identify with such broad categories of

research output in a meaningful way.

However, the actual impact assessment will usually require a focus which is at least one level lower in terms of

subcategories of technical output. Both the government researcher and the industry users of the research project's

results should be comfortable with this level of detail and complexity and must work cooperatively with the

consultant to match technical outputs with economic impacts and benefits (as described in Section F.2 below).

Once the match between research output and economic impact is achieved, the consultant will aggregate the

results to a more general level in the final report.

For example, when assessing the economic impacts of NIST-supported standards by the Electronics and

Electrical Engineering Laboratory for the optical fiber industry, it was not sufficient to define the setting of

standards as the relevant technical output. Rather, each of the twenty-two EIA fiberoptic test procedures

influenced by NIST between 1981 and 1992 (the year in which the study was conducted) was matched with a

performance characteristic of optical fiber.[11] And, industry experts were asked to allocate total realized

economic benefits among these standards. Similarly, when evaluating the economic impacts of NIST research by

the Electronics and Electrical Engineering Laboratory in electromigration on the U.S. semiconductor industry,

direct cost-saving benefits to domestic producers of semiconductors as well as indirect benefits to producers of

testing systems were explored.[12]

E.3. Current and Expected Users

The historical overview of the project should also include a taxonomy of the categories of users of the project's

outputs and the names of accessible individuals within each category. Such categories of current and expected

users should include:

categories of industrial users, including those that are international as well as domestic

user groups within NIST (that is the results from one NIST project may be a fundamental input to other

projects within NIST), and

public-sector users, including those that are domestic as well as international.

And, in addition, the project liaison should help the consultant identify co-producers of project's output. For

example, it may be the case that other organizations outside of NIST (e.g., other federal laboratories) are

engaged in similar research. This is important information when the consultant begins to quantify the benefits

associated with NIST's research.

These three groups represent first-level users, although industrial users are of particular importance. Consumers

of goods and services provided by these users can also benefit from the project's output, as discussed below in

Section F.2.

It is advisable that this historical overview be distributed to appropriate individuals within NIST for comments.

Soliciting internal input at this stage not only has the advantage of assuring a priori accuracy in terms of

documentable information, but also provides the opportunity for others to offer observations or constructive

comments on the relevancy of the categories of users of the project's output, both within NIST and externally.

This is especially important if the project is on-going, and prospective uses of the project's results are what will

be documented by the consultant.

F. Formulate An Economic Impact Assessment Strategy
9/30/2020 https://web.archive.org/web/20090822160544/http://www.nist.gov/director/prog-ofc/rept-961.htm

https://web.archive.org/web/20090822160544/http://www.nist.gov/director/prog-ofc/rept-961.htm 10/23

At this stage, the consultant will begin to identify areas of economic benefits and associated elements of costs as

part of the formulation of an economic impact assessment strategy. This stage is critical in the assessment

process because the decisions made by the consultant at this point, which will have been greatly influenced by

the background information provided by the project liaison, will determine the research methodology used and

the scope of qualitative and quantitative information collected.

F.1. Identify Elements of Cost

The tangible (i.e., measurable) costs considered in an economic impact assessment include direct costs and

indirect costs. Generally, direct costs are those incurred by NIST and indirect costs are those incurred by

industry to implement the research outputs. In other words, direct costs are public costs and indirect costs are

private costs. It is advisable for all costs to be accounted for on a cash basis. That is, relevant to an impact

assessment are costs actually incurred, as opposed to costs encumbered in the current time period but not spent

until a later time period. F.1.a. Direct Costs

The elements of cost that are of concern to the consultant, and hopefully have been collected by the project

liaison as part of the historical overview, include all NIST resources devoted to the research project being

studied, by year. These annual costs include R&D;, equipment, facilities, maintenance, and fully-burdened labor.

[13] It is the case that some research projects use other agency (OA) resources. These are relevant costs and

need to be included in the assessment, but it is useful to separate NIST costs from OA costs.

One class of outputs from some research projects are services for which fees are charged. Depending on the

nature of these services, the consultant may or may not include the NIST expenses to provide the for-fee service

as an element of the project's direct costs. If not considered, the consultant is implicitly assuming that the price

paid by the industrial users for this service fully represents the benefits they receive (which may or may not be

the case). Or if considered, then the consultant will include the services paid for in the calculation of industrial

benefits. In either case, NIST's expenses to establish the facility in order to provide the for-fee services must be

considered. Often, a portion of collected fees is redistributed back to the project or to other NIST research

programs, thus this reallocation is included in next generation research expenditures. Accordingly, this

reallocation portion should be accounted for and subtracted from NIST's costs.

Presumably, these direct cost elements have been collected by the project liaison. This was, for example, the

situation when assessing the power and energy calibration project within the Electronics and Electrical

Engineering Laboratory and the conformance test program for the Structured Query Language (SQL) database

language within the Computer Systems Laboratory. Regarding the power and energy calibration project, its

annual operating budget in 1994 was approximately $230,000, which included about $150,000 in fees received

from calibrating electric power and energy measurement instruments for industry. Thus, only $80,000 was

considered to be the relevant annual NIST cost (against which to compare annual benefits).[14] Similarly, a

portion of the cost data collected when evaluating standards for SQL test suites was offset by annual revenues

from the sale of these test suites.[15]

F.1.b. Indirect Costs

There are two important categories of indirect costs: so-called "push" costs and "pull" costs.

In some research projects costs are incurred when transferring (or "pushing") the outputs from the research

project to the industrial sector. For example, dissemination costs may be incurred in producing and distributing

technical findings (e.g., at conferences), or in operating a more formal technology transfer program through a

consortium, standard committee, or other mechanism. When applicable, these cost estimates will be needed by

the consultant.

It may also be the case that users of the project's outputs incur costs to identify, acquire, and implement (or "pull

in") the outputs from the project. When applicable, these costs estimates will have to be netted from the benefit

estimates to the users. For example, when assessing the Manufacturing Engineering Laboratory's research in
9/30/2020 https://web.archive.org/web/20090822160544/http://www.nist.gov/director/prog-ofc/rept-961.htm

https://web.archive.org/web/20090822160544/http://www.nist.gov/director/prog-ofc/rept-961.htm 11/23

Real-Time Control Systems Architecture, it was learned that the users of the technology devoted substantial

resources over a six-year period to adapting the architecture for their specific purposes.[16] Only then were

economic benefits realized.

F.2. Identify Areas of Benefits

Having identified in the historical overview the current or expected technical outputs from the research project

and the users of these outputs, the consultant will begin to estimate the economic impact that these technical

outputs are likely to have on industry. To begin, the consultant will focus on broad-based benefits affecting many

segments of industry. It is important to have an understanding of these benefit areas prior to gathering data so

that the consultant is fully informed. Such general, yet quantifiable, categories of economic impact that the

consultant will consider include:[17]

productivity,

quality,

time-to-market,

market share, and

transaction costs.

The technical outputs produced through NIST's research have the potential to create economic benefits by:

increasing industrial productivity as measured by an increase in production yield or by a decrease in

research or production costs.

For example, when studying the Computer Engineering Laboratory's role in establishing Integrated

Service Digital Network (ISDN) conformance testing methodologies and standards-based

implementation agreements, it was learned that NIST activities allowed users of ISDN

communication technology to realize a 20 percent annual increase in productivity through reduced

communication costs.[18]

increasing the quality of industrial products through enhanced product performance and reliability, or

reduced attribute variability.

For example, when studying the Physics Laboratory's operation and maintenance of the Facility for

Automated Spectral Calibration, also known as the FASCAL laboratory, qualitative information was

collected from manufacturers of measurement equipment, lighting equipment, and photographic

equipment documenting that, in the absence of NIST's spectral irradiance standards, equipment

customers would be forced to accept greater uncertainty in products.[19]

reducing the time needed to get industrial products to market by shortening the R&D; process.

For example, when studying the Electronics and Electrical Engineering Laboratory's research

program in electromigration characterization, domestic producers of semiconductors reported time

saved in R&D; and improved R&D; efficiency from implementing the results of NIST's

electromigration research, and they valued these benefits to industry at (in 1991) about $4.1 million.

[20]

increasing market share, especially in terms of the world market, as a result of increased productivity,

increased quality, or reduced time-to-market.

For example, when studying the Manufacturing Engineering Laboratory's feasibility demonstrations

of software error compensation techniques, coordinate measuring machine (CMM) manufacturers

reported a reduced time to market of between two and five years for CMMs. This time-to-market

savings translated into increased manufacturing productivity valued at approximately $79 million

over the 1985-1995 period.[21]
9/30/2020 https://web.archive.org/web/20090822160544/http://www.nist.gov/director/prog-ofc/rept-961.htm

https://web.archive.org/web/20090822160544/http://www.nist.gov/director/prog-ofc/rept-961.htm 12/23

reducing transaction costs between companies and their customers through product acceptance test

methods or practices.

For example, when studying the Computer Systems Laboratory's role in establishing conformance

testing programs for the SQL database language, SQL producers reported an average of 2.5 workyears per year reduction in effort to resolve procurement disputes with customers.[22]

Because the project liaison will assist the consultant to "open doors" with users of the technical research outputs,

the project liaison must have an understanding of the breadth of economic benefits that the consultant will be

quantifying. The project liaison must learn to think about NIST's research in such economic terms in order to

help communicate the study's results.

Benefit categories can be thought of in terms of levels. For example, the benefit categories in the examples

above are referred to as "first-level" benefits, meaning that they result from the direct industrial use of the

project's output. Secondary and tertiary levels of benefits are equally as important to understand, and quantify if

possible. Second-level benefits are those received by the users of the goods and services provided by the firstlevel users of the technological infrastructure, and so on to the third-level. Most assessment studies attempt to

characterize as many levels as possible, although for quantitative purposes the consultant will set boundaries on

the benefit areas considered; that is, in practice they rarely go beyond the first level partly because of the cost of

enlarging the study, but more important, because of the difficulty frequently encountered in isolating the

project's partial contribution to the economy beyond the first level of impact.[23]

G. Select an Economic Impact Assessment Method

The consultant will select an assessment method to use in the study prior to formulating a methodology for

collecting data. It is important that the project liaison understand the consultant's thinking about alternative

assessment methods because the project liaison may be in a position at a future point in time of having to answer

questions from third parties about the study.

G.1. Guidelines for Selecting an Assessment Method

The consultant will select an assessment method that meets a number of accepted evaluation guidelines.

Specifically, the consultant will take into account a number of criteria, including:

appropriateness,

replicability,

ends-focused, and

comparability.

The evaluation method selected should be appropriate, meaning that it is not only accepted within the evaluation

community but also suitable to meet the objectives, time period and nature of the project, and resource

constraints. It should be replicable, meaning that at a future point in time the assessment can be updated by

another consultant so that the findings from the original study are comparable to those from the updated study. It

should be ends-focused, meaning that the method clearly distinguishes between the resulting impact of the

research project's output on the users of the research and the means by which NIST generated that output. And

the assessment method should be based on measurement metrics that facilitate comparisons across research

projects, including research projects in other organizations.

G.2. Alternative Assessment Methods

The two most common economic impact assessment methods are benefit-cost analysis and internal rate of return

analysis. As discussed below, these methods are related to each other. In some instances, the consultant will
9/30/2020 https://web.archive.org/web/20090822160544/http://www.nist.gov/director/prog-ofc/rept-961.htm

https://web.archive.org/web/20090822160544/http://www.nist.gov/director/prog-ofc/rept-961.htm 13/23

perform both. Appendix A.1 to these guidelines numerically illustrates each method in detail; however, an

overview of the economic interpretation of each method is provided below. G.2.a. Benefit-Cost Analysis

As the name implies, benefit-cost analysis is a method of comparing the benefits from a project to the costs

incurred to conduct the project (i.e., the costs to achieve the benefits). For an economic impact assessment, the

benefits are calculated as the dollar value of benefits received by the users of the research project's outputs. The

costs are calculated as the sum of direct and indirect costs, as discussed above.[24]

Because benefits and costs occur at different points in time, with costs often occurring before any benefits are

realized, benefits should be compared to costs in present value terms.[25]

The end result from a benefit-cost analysis is a unitless numerical value of the benefit-to-cost ratio. If the ratio is

greater than one, then the present value of benefits is greater than the present value of costs. The consultant will

urge the project liaison not to simply be content with a ratio greater than one. The ratio itself is often useful to

external stakeholders, but an understanding of the benefit areas that can potentially be served is fundamental to

enhancing overall research management. G.2.b. Internal Rate of Return Analysis

The internal rate of return (IRR) is a generally accepted metric used in evaluation studies. By definition, it is the

discount rate that corresponds to a benefit-to-cost ratio of unity.[26]

Other names that have been used in the literature for the discount rate are the "social" or "spillover" rate of

return.[27] As such, this measure is frequently used as an index of the returns to society from an investment

made with public funds. The higher the estimated internal rate of return, the greater the estimated net benefits to

society.

H. Conducting the Economic Assessment

After taking into account all cost and benefit measurement considerations, the consultant will gather the relevant

economic data, analyze them, and summarize the findings in a final report.

H.1 Cost Considerations

On the costs side, the consultant will scrutinize the direct costs provided by the project liaison. This scrutiny is

not intended to cast doubt on the due-diligence of the project liaison, but rather to ensure to the satisfaction of

the consultant that all NIST costs have been included, and that they are defined and collected according to the

same criteria used in other impact assessments. Often, if the project liaison has relied on a budget office for the

cost figures, the budget office may not have, for example, taken into account the costs associated with generating

fundamental principles that feed directly into the candidate project, or they may not have burdened labor costs to

account fully for overhead expenses, or they may not have realized that the project incurred "push" costs to

transfer the project's outputs to industry. The consultant may also contact several representatives in selected

industries to ascertain these costs. The consultant is not the expert on NIST's budget reporting process, but the

consultant is an expert on knowing the categories of direct costs to include in an assessment study.

H.2. Benefit Considerations

On the benefit side, the consultant will ensure that all potential areas of economic benefits have been identified

in the project overview. The project liaison may assist in this step and involve others in the relevant group. The

consultant may request a number of meetings with all those associated with the candidate project in order to

ensure that these benefit areas are well understood. The consultant will next ask for the names of a few

representative individuals in the relevant industries to interview by telephone. The purpose of these

informational interviews is four-fold:

to validate the benefit areas identified by NIST,

to determine if there are other first-level benefit areas that need to be considered as part of the assessment,
9/30/2020 https://web.archive.org/web/20090822160544/http://www.nist.gov/director/prog-ofc/rept-961.htm

https://web.archive.org/web/20090822160544/http://www.nist.gov/director/prog-ofc/rept-961.htm 14/23

to identify potential second-level benefits that are realized by second-level users, and

to identify individuals willing to participate in a pre-test of the survey instrument.

On the basis of this information, the consultant will develop a survey instrument to document both qualitatively

and quantitatively the economic benefits associated with the project's outputs. The purpose of such an instrument

should not vary significantly across assessment studies, but the format and manner in which questions are asked

will vary across projects.[28]

H.3. Data Collection Process

The consultant will rely upon the expertise of the project liaison to identify by name and address the population

of industrial users of the project's outputs. The population may be derived from NIST contact files, conference or

workshop participation lists, membership roles in trade associations or professional societies, etc.

This is a critical stage in the assessment study and one that may be time consuming for the project liaison. Given

this population, the consultant, with assistance from the project liaison, will identify a representative sample of

individuals to contact. The consultant will describe the purpose of the study and urge industry to complete the

survey instrument, but the project liaison may also be called on to encourage timely participation. This

involvement with the survey instrument and with industry will help NIST managers become aware of what

constitutes an economic impact. Thus, NIST managers will have a better concept of how to engage in real-time

data collection for other studies.

The survey instrument will likely vary across projects. Depending on the technical outputs defined for the

project, the survey questions may focus on:

transaction cost savings (i.e., reduced cost to buyers and sellers interacting in the market owing to the

information gained from standards).

For example, when studying spectral irradiance standards, equipment manufacturers were asked the

following questions:[29]

Approximately, how many disputes occur per year with customers regarding the

accuracy of your measurement equipment?

In your opinion, is this number less than it would be in the absence of NIST's spectral

irradiance standards?

If YES, based on your experience in selling products that are not traceable to a national

standard, approximately what would be the number of such disputes per year in the

absence of the FASCAL laboratory?

Approximately, how many work days does it take to resolve such a dispute?

Approximately, what is the cost to your company of a fully-burdened work year?

R&D; efficiencies (i.e., materials characterizations, measurement methods, science and engineering data,

etc.

For example, when studying electromigration characterization, semiconductor manufacturers were

asked:[30]

What has been the approximate cost savings (time saved, improved R&D; efficiency,

production cost reduction, etc.) to your company from implementing the results of

NIST's research program in electromigration?
9/30/2020 https://web.archive.org/web/20090822160544/http://www.nist.gov/director/prog-ofc/rept-961.htm

https://web.archive.org/web/20090822160544/http://www.nist.gov/director/prog-ofc/rept-961.htm 15/23

In addition to these cost savings, has NIST's overall research program helped your

company compete more effectively by influencing R&D; objectives, say, by stimulating

new product or process ideas? Please explain.

production cost savings (i.e., reduced manufacturing costs due to test methods or the public provision of a

fundamental technology).

For example, when studying power and energy calibration services, domestic manufacturers of

standard meters were asked to respond to the following open-ended question:[31]

I understand from NIST that between 1960 and 1986 uncertainty for the watthour meter

was 0.05%, and then it dropped to 0.005% after 1986. Please describe and quantify the

benefits that your company and customers have received from this lowering of

uncertainty.

The survey instrument may be administered by mail or by telephone, or both. For example, when assessing the

economic benefits associated with NIST's support of the Integrated Services Digital Network (ISDN), personal

interviews were first conducted at an annual conference in order to obtain general background information and

to identify participants in the study; then a mail survey was administered; and finally, follow-up telephone

interviews were conducted.[32]

I. Presentation and Interpretation of the Assessment Findings

The consultant will prepare a draft report that will be circulated within the relevant NIST units for comments. It

is important that the report is clearly written so that a lay-person has a "general" idea of the nature of the project

studied and is "comfortable" that the benefit categories considered encompass the potential breadth of the

project's impact.

Quite often a consultant is unable to quantify all benefit areas, but will generally be able to assess them

qualitatively. Often, such a presentation lends credibility to the overall analysis because most lay-people would

be skeptical of a report that purported to quantify all benefit dimensions. If quantitative information is used, the

consultant will state that the findings from the study are lower-bound estimates because several categories of

benefits have been excluded from the quantitative conclusions. For example, when evaluating several Advanced

Technology Program-sponsored projects that were mid-way through their research project, qualitative benefit

information was the most appropriate type to gather and report.[33]

The consultant's report will emphasize assessment methods and the associated findings. Such quantitative

findings should be interpreted with care. If the consultant emphasizes a benefit-to-cost ratio, that number, in

isolation from comparable project assessments, will only tell the reader that the measured benefits from the

project are greater or less that the associated costs. If greater, it will tell the reader how much greater.

If the consultant emphasizes an internal rate of return analysis, it is important to realize that this percentage

return is not the same as a return on investment estimate. The internal rate of return is compared to a minimum

acceptable rate of return. If the estimated internal rate of return exceeds this minimum acceptable level, then ex

post it was a worthwhile investment. This interpretation issue is discussed in Appendix A. 

Appendix A

A Numerical Illustration of Alternative Assessment Methods

1. Introduction
9/30/2020 https://web.archive.org/web/20090822160544/http://www.nist.gov/director/prog-ofc/rept-961.htm

https://web.archive.org/web/20090822160544/http://www.nist.gov/director/prog-ofc/rept-961.htm 16/23

The purpose of this appendix is to illustrate, using a set of hypothetical data, the calculations of the two metrics

commonly used in economic impact assessments. Specifically, the following discussion compares and contrasts

the relationship between a benefit-to-cost ratio and an internal rate of return measure.[34] Because the emphasis

of this Appendix is on calculations rather than measurements, it is assumed that:

the cost estimates presented account fully for all NIST costs including relevant overhead charges, and

all benefit estimates presented account fully for benefits realized within other divisions within NIST and

for benefits received by all relevant industry users of the research results, and all benefits are net of

industry costs to utilize the NIST research results.

Consider the data presented in Table A.1 at the end of the Appendix. The project being evaluated began in 1980

and was terminated at the end of 1993. In its first year, $400,000 in capital equipment and research time was

allocated to begin the project. Annual costs fell thereafter owing to a decreased need for new capital equipment

and an increase in revenues from user fees (e.g., charges for calibration services).[35] At least some companies

in the affected industry(ies) reported, through the data collection instrument(s) used by the consultant, that

$100,000 in benefits were realized in the first year of the project. Reported benefits increased through 1991, and

then decreased through 1994. Although the NIST research project is assumed to have been discontinued in 1993,

some companies reported benefits in 1994.

2. Benefit-Cost Analysis

The following formula for the calculation of a benefit-to-cost ratio was shown and discussed in the text material

above:

where the subscript t indexes the time frame over which benefits have been realized and the time over which

costs have been incurred. In this hypothetical case, n=14 and m=13. For benefits Bt, t ranges from 0

(corresponding to 1980) to 14 (corresponding to 1994). For costs Ct, t ranges from 0 (corresponding to 1980) to

13 (corresponding to 1993).

The selection of the appropriate discount rate r to use in the calculation of a benefit-to-cost ratio is a disputed

conceptual issue.[36] Some argue that the discount rate should approximate the return that could be earned from

alternative investments had the research project being evaluated not been undertaken.[37] Others argue that the

discount rate should approximate the cost to NIST (or to the government in a more general context) if it had to

borrow the resources that were devoted to the project being evaluated. In practice, the consultant doing the

evaluation will select a specific discount rate and justify it.

For the illustrative purpose of this Appendix, the discount rate used in this example (and justified by certain

literature as a reasonable approximation of the cost of borrowing funds, including administrative costs) is the

long-term rate on government bonds plus two percentage points. In this example, a 9 percent discount rate is

used (r=0.09).[38]

The benefit-to-cost ratio formula discounts all costs and benefits back to 1980. Stated alternatively, the formula

above is the ratio of the discounted present value of benefits and costs. Using the data in Table A.1, the present

value of costs, referenced to 1980, is $1,972,550 and the present value of benefits is $8,890,650. Thus, B/C =

4.51. This ratio value should be interpreted to mean that the benefits realized by industry over the period 1980-

1994 are 4.51 times greater than the costs incurred by NIST to generate them.

This metric is correctly viewed as a descriptive piece of information. By itself, it cannot be interpreted to mean

that this project was successful or unsuccessful because such an interpretation implies that some predetermined

value of B/C is optimal. Likewise, one can not infer from this value that the project's funding should have

continued beyond 1993. Those are managerial decisions that are correctly determined in light of the laboratory's
9/30/2020 https://web.archive.org/web/20090822160544/http://www.nist.gov/director/prog-ofc/rept-961.htm

https://web.archive.org/web/20090822160544/http://www.nist.gov/director/prog-ofc/rept-961.htm 17/23

mission, the results from other projects undertaken concurrently, and the prospects for alternative projects that

can be pursued in the future.

An alternative approach, although infrequently used by evaluators, for the calculation of a benefit-to-cost ratio is

to inflate (rather than discount) historic costs and benefits to the present by a price index, such as the Consumer

Price Index (CPI). Values for the CPI are also shown in Table A.1. The relevant index to divide by each year's

costs and benefits is determined by dividing each value of the CPI by 148.2, thus creating an index whose value

is 1.0 for 1994. The ratio of the sum of the so-called inflation-adjusted or real benefits to real costs is 5.25. This

ratio should be larger than the discounted one because the discount rate used to reference values back to 1980 is

larger in value than the rate of inflation used to reference values forward to 1994. Obviously, the usefulness of

this approach is that it avoids the issue of selecting a discount rate, but by so doing it ignores the fact that NIST

resources do have alternative productive uses.

One could interpret these two benefit-to-cost ratios to mean:

adjusting for the cost to NIST to acquire the resources needed for this project, each $100 invested by NIST

yields $451 in benefits to industry, or

adjusting for inflation, each $100 invested by NIST yields $525 in benefits to industry.

3. Internal Rate of Return Analysis

The following formula for the calculation of an internal rate of return was shown and discussed in the text

material above:

NPV = [(B0-C0)/(1+i)0] + [(Bn-Cn)/(1+i)n] = 0

where (Bt-Ct) represents net benefits. In 1980 for t=0, net benefits are less than zero, but they are greater than

zero in all future years. In 1994, when n=14, net benefits equal total benefits because costs are zero in that year.

By definition, the internal rate of return is the value of the discount rate i that equates the net present value

(NPV) of the project being evaluated to zero. Based on the data in Table A.1, the calculated value of i for which

NPV=0 is 1.31, implying an internal rate of return for this project of 131 percent. Before offering an economic

interpretation for this value, it should be noted that when NPV=0, B/C=1. NPV can be rewritten as:

where, as previously noted, r is the discount rate used to reference future benefits and costs to present value.

When NPV=0, then it follows that:

or that the present value of benefits equals the present value of costs, or B/C=1.

An internal rate of return estimate of 131 percent is interpreted to mean that 1.31 is the value of the discount rate

that equates the present value of benefits to the present value of costs. As stated in the text, economists and

policy makers generally use internal rate of return measures, for on-going or completed public-sector research

projects, to estimate what is referred to as the social rate of return. As such, one can infer from the 131 percent

value calculated from the hypothetical data in Table A.1 that if 131 percent is above NIST's hurdle rate or

generally accepted expected rate of return then the project was worthwhile.[39]

4. Interpretative Issues

As just stated, the internal rate of return of 131 percent should be compared to NIST's generally accepted rate of

return, and if greater, then this project was worthwhile. An alternative way of thinking about the internal rate of
9/30/2020 https://web.archive.org/web/20090822160544/http://www.nist.gov/director/prog-ofc/rept-961.htm

https://web.archive.org/web/20090822160544/http://www.nist.gov/director/prog-ofc/rept-961.htm 18/23

return is in terms of a market discount rate. An important behavioral assumption in economics is that the market

values future returns from an investment activity by discounting them using a return on an investment of an

equivalent risk class. The internal rate of return calculation above can be thought of as providing the discount

rate that would result in the market's value of the returns from the research project just equaling the research

project's cost, and so the project would break even (i.e., B/C=1). Hence, if the actual discount rate that is

appropriate for this research project (i.e., NIST's hurdle rate) is less than 131 percent, then the claim is that the

project's returns could be sold in the market for more than the project's costs (i.e., B/C>1).

Some readers might attempt to translate an internal rate of return into an annual yield similar to that earned on,

say, a bank deposit. Such a direct comparison is, however, incorrect. The return earned on a bank deposit is a

compounded rate of return. One invests, say $100, and then earns interest on that $100 each year plus interest on

the interest. That is not the case in an R&D; project, in general, or in the hypothetical example shown in Table

A.1. Referring to Table A.1, the $400 allocated in 1980 is not reoccurring and the benefits derived from that

$400 or from subsequent expenditures are not compounded over time.

Under alternative sets of assumptions, one can calculate implied rates of return from the data in Table A.1, but

these rates of return are not the same as the internal rate of return. For example, if all costs are referenced to

1980 using a discount rate of 7.37 percent, total investments in this hypothetical project are $2,098.[40] If all

benefits are referenced to 1994 using the same rate, total benefits from this hypothetical project are $26,929.

Thus, one could think of this project as one where an initial investment of $2,098 generated, after 14 years,

benefits to industry equalling $26,929. Thus, over all 14 years, the total rate of return was 1,184 percent, and the

average annual rate of return was 85 percent.

Alternatively, one could calculate the annual compounded rate of return that corresponds to an initial investment

of $2,098 in 1980 growing to $26,929 by the end of 1994.[41] Such a compounded rate of return, x, equals 20

percent based on the following relationship:

$2,098 (1+x)14=$26,929

One should not expect these two implied rates of return to be identical. Each was calculated under different

assumptions. And, if the assumptions change, both calculated rates will change.

While both of these annual percentages have a commonplace interpretation, the assumptions underlying them do

not conform to the realities of the data in Table A.1, or to the pattern of investments in research at NIST and the

trend in realized benefits by industry. The internal rate of return is an accepted evaluation metric and should be

interpreted, in comparison to NIST's hurdle rate, to conclude that a project was worthwhile or not. 

Table A.1

Hypothetical Data on Costs and Benefits

Year Costs ($K) Benefits ($K)

1980 $400 $100

1981 $250 $600

1982 $250 $600

1983 $250 $700

1984 $250 $950

1985 $250 $950

1986 $225 $950

1987 $225 $1,500
9/30/2020 https://web.archive.org/web/20090822160544/http://www.nist.gov/director/prog-ofc/rept-961.htm

https://web.archive.org/web/20090822160544/http://www.nist.gov/director/prog-ofc/rept-961.htm 19/23

1988 $150 $1,500

1989 $150 $1,500

1990 $150 $2,200

1991 $150 $2,200

1992 $125 $1,500

1993 $100 $1,500

1994 $0 $900

Bibliography

Leyden, D.P. and A.N. Link, Government's Role in Innovation, Norwell, MA., Kluwer Academic Publishers,

1992.

Link, A.N., "An Economic Evaluation of the Spectral Irradiance Standard," prepared for the Physics Laboratory

at NIST, November 1995a.

Link, A.N., "An Evaluation of the Economic Impacts Associated with the NIST Power and Energy Calibration

Services," prepared for the Electronics and Electrical Engineering Laboratory at NIST, January 1995b.

Link, A.N., "Economic Impact on the U.S. Semiconductor Industry of NIST Research in Electromigration,"

prepared for the Electronics and Electrical Engineering Laboratory at NIST, January 1992a.

Link, A.N., "Economic Impacts of NIST-Supported Standards for the U.S. Optical Fiber Industry: 1981-

Present," prepared for the Electronics and Electrical Engineering Laboratory at NIST, February 1992b.

Link, A.N., "Economic Study of the Advanced Manufacturing Technology for Low-Cost Flat Panel Displays

Joint Venture After Three Years," prepared for the Advanced Technology Program at NIST, August 1994a.

Link, A.N., "Economic Study of the Joint Venture Project on Short-Wavelength Sources for Optical Recording

After Three Years of a Five-Year Research Program," prepared for the Advanced Technology Program at NIST,

March 1994b.

Link, A.N., "Economic Study of the Printed Wiring Board Joint Venture After Two Years," prepared for the

Advanced Technology Program at NIST, April 1993a.

Link, A.N., Evaluating Public Sector Research and Development, New York: Praeger Publishing Company,

forthcoming 1996.

Link, A.N., "Measuring the Economic Impact of the Advanced Technology Program: A Planning Study,"

prepared for the Advanced Technology Program at NIST, May 1992c.

National Institute of Standards and Technology (NIST), Setting Priorities and Measuring Results at the National

Institute of Standards and Technology, U.S. Department of Commerce, January 31, 1994.

Ruegg, R. and H. Marshall, Building Economics: Theory and Practice, New York, Van Norstrand Reinhold,

1990.

TASC, "Software Error Compensation Research in NIST's Manufacturing Engineering Laboratory: An

Evaluation of First-Order Economic Impacts," prepared for the Manufacturing Engineering Laboratory at NIST,

April 1996.
9/30/2020 https://web.archive.org/web/20090822160544/http://www.nist.gov/director/prog-ofc/rept-961.htm

https://web.archive.org/web/20090822160544/http://www.nist.gov/director/prog-ofc/rept-961.htm 20/23

TASC, "An Economic Evaluation of NIST's Research in Real-Time Control System (RCS) Architecture: A Case

Study Approach," prepared for the Manufacturing Engineering Laboratory at NIST, April 1995a.

TASC, "Economic Assessment of the NIST Conformance Test Program for Database Language SQL," prepared

for the Computer Engineering Laboratory at NIST, September 1995b.

TASC, "NIST's Support of the Integrated Services Digital Network (ISDN)," prepared for the Computer

Engineering Laboratory at NIST, November 1994.

Task Force on Alternative Futures for the Development of Energy National Laboratories, "Alternative Futures

for the Department of Energy National Laboratories,: Energy Advisory Board, February 1995.

Tassey, G., Technology Infrastructure and Competitive Position, Norwell, MA, Kluwer Academic Publishers,

1992.

Tassey, G. "Technology and Economic Growth: Implications for Federal Policy," NIST Planning Report 95-3,

October 1995b.

Tassey, G., "The Rates of Return to Private and Public Investment in Technology," NIST mimeo, March 1995a.

Technical Notes

[1] Industry is not the only sector that benefits from NIST's research. Economic impacts accrue to consumers

and the public at large; however, potential users of research outputs within industrial sectors are of primary

importance to NIST.

[2] The specific financial and strategic variables to focus on are a matter of judgment, as discussed later

under the broader topic of setting boundaries on an economic impact assessment.

[3] As stated in Setting Priorities and Measuring Results at the National Institute of Standards and

Technology, "... the need to measure returns on taxpayer investments in federal technology programs is of

paramount importance ..." (p. 10).

[4] Public accountability was also emphasized in the 1993 National Performance Review report. Therein, the

importance of measuring results rather than inputs was reinforced. In this context, the Task Force on Alternative

Futures for the Department of Energy National Laboratories reported (in the so-called Galvin Report) that "...

there is a perception that the U.S. government is spending significant resources on the development of new

technologies but that American industry is not reaping the rewards of that investment" (p. 46).

[5] Managers are not the only individuals or groups of individuals that would ask such a question. The Office

of Management and Budget (OMB) and the General Accounting Office (GAO) conduct evaluations of

government agencies like NIST. Their inquiries across agencies within the government are similar to the

program evaluations across laboratories within NIST. NIST can more effectively document its overall

performance when it has a portfolio of project assessments to illustrate the effectiveness of its laboratories'

research programs. Also, it was explicitly noted as one of three motivations for the enactment of the Government

Performance and Results Act of 1993 that "Federal managers are seriously disadvantaged in their efforts to

improve program efficiency and effectiveness, because of insufficient articulation of program goals and

inadequate information on program performance."

[6] See Link (1995b).

[7] The metrics used for project selection are different from those used for project assessment.
9/30/2020 https://web.archive.org/web/20090822160544/http://www.nist.gov/director/prog-ofc/rept-961.htm

https://web.archive.org/web/20090822160544/http://www.nist.gov/director/prog-ofc/rept-961.htm 21/23

[8] Of course, the representativeness criterion takes as given that the assessment is feasible. Because not all

projects are feasible, selection bias may be introduced at this point. The consultant will address this issue with

the project liaison.

[9] For example. improving the precision of clinical cholesterol measurements may be more representative of

other chemistry projects which also involve some sort of test method and the production of a standard reference

material (SRM) than, say, a project for the development of an equation of state for an alternative refrigerant.

[10] See Link (1992a, 1992b).

[11] See Link (1992b).

[12] See Link (1992a).

[13] It may be the case that the research project being studied relies on fundamental science developed

elsewhere in NIST. A portion of the cost to transfer that knowledge to the project being studied will be

considered by the consultant.

[14] See Link (1995b).

[15] See TASC (1995b).

[16] See TASC (1995a).

[17] These are not independent categories of benefits.

[18] See TASC (1994).

[19] See Link (1995a).

[20] See Link (1992a).

[21] See TASC (1996).

[22] See TASC (1995b).

[23] This is a weakness of many of the economic impact studies conducted at NIST to date. As such the

evaluation metrics associated with these studies are lower-bound estimates of the true social benefits associated

with NIST's research.

[24] It should be noted, as discussed in Link (forthcoming 1996), that some academics take exception with

using a benefit-to-cost ratio to compare public costs with private benefits. However, from a project manager's

perspective, their charge is to in fact do just that; namely, to identify the private sector benefits associated with

their publicly-funded research.

[25] Thus, the ratio of benefits (B) to costs (C) that is used in a benefit-cost analysis is:
9/30/2020 https://web.archive.org/web/20090822160544/http://www.nist.gov/director/prog-ofc/rept-961.htm

https://web.archive.org/web/20090822160544/http://www.nist.gov/director/prog-ofc/rept-961.htm 22/23

where the subscript t indexes the time frame over which costs have been or will be incurred and the time over

which benefits have been or will be realized. Bt represents all measurable benefits in period t, and Ct represents

all measurable costs in period t. It is often the case that C&subt;>B&subt; in the early years of a project, and

then Bt>Ct in the latter years. If in a future year, t=m for mm=0 and Bm>0. The variable r represents the discount

rate used to reference future values in present value terms. The consultant will calculate the appropriate discount

rate in terms of the returns possible from alternative uses of the project's resources (so-called opportunity costs).

[26] Stated alternatively, the IRR is the rate i that equates the net present value (NPV) of a project to zero:

NPV = [(B0-C0)/(1+i)0] + ... + [(Bn-Cn)/(1+i)n] = 0

where (Bt-Ct) represents net benefits. If, as is often the case in the early years of a project, Ct>Bt, then net

benefits are negative in that year. Net benefits will be positive when Bt>Ct.

[27] See Tassey (1995a, 1995b).

[28] In fact, there may be more than one survey instrument, each tailored to a particular group of industrial

users. The program/project manager should be aware that approval may be needed from the Office of

Management and Budget (OMB) prior to conducting the survey.

[29] See Link (1996).

[30] See Link (1992a).

[31] See Link (1995b).

[32] See TASC (1994).

[33] See Link (1993a, 1994a, 1994b). The consultant will make an aggressive effort to quantify all cost

elements associated with the project. The consultant will, however, be conservative when quantifying benefits.

While potentially frustrating to the project liaison (who may feel that he/she is a stakeholder for the evaluation

study), this tendency to downward bias the evaluation results adds not only accuracy but also credibility to the

final assessment.

[34] It must be emphasized that the data herein are hypothetical. A reader should not assume that the growth

pattern underlying either the cost or benefit estimates is necessarily representative of other NIST research

projects.

[35] Thus, the estimates in Table A.1 are net costs.

[36] See Link (forthcoming 1996) for a discussion of this point and for additional references on the issue.

[37] Economists refer to this concept as opportunity cost.

[38] NIST may consider establishing its own criteria for calculating a discount rate to use in impact

assessments in order to ensure some comparability across studies.

[39] There are a number of additional problems associated with an internal rate of return measure when it is

used for research project selection based on forecasted benefit data. Because the types of projects that are
9/30/2020 https://web.archive.org/web/20090822160544/http://www.nist.gov/director/prog-ofc/rept-961.htm

https://web.archive.org/web/20090822160544/http://www.nist.gov/director/prog-ofc/rept-961.htm 23/23

expected to be evaluated at NIST are completed or on-going, as opposed to being proposed, these issues are not

relevant here.

[40] This discount rate equals the average yield on 30-year Treasury bonds in 1994, as reported in the

Economic Report of the President, 1995. The year 1994 was selected because it is the last year for which

benefits are measured. However, there is no methodological reason for the selection of the terminal year or for

the use of the 30-year Treasury bond yield. This illustration is intended only to illustrate alternative

interpretations for the data in Table A.1 as opposed to the implementation of an alternative evaluation metric.

[41] This calculated rate of return in similar to the overall rate of return suggested by Ruegg and Marshall

(1990).