June 2006

Toward a Standard Benefit-Cost

Methodology for Publicly Funded

Science and Technology Programs

NISTIR 7319

National Institute of Standards and Technology • Technology Administration • U.S. Department of Commerce

Jeanne Powell

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
About ATP’s Economic Assessment Office 

The Advanced Technology Program (ATP) is a partnership between government and private 

industry to conduct high-risk research to develop enabling technologies that promise significant 

commercial payoffs and widespread benefits for the economy. 

Since the inception of ATP in 1990, ATP’s Economic Assessment Office (EAO) has performed 

rigorous and multifaceted evaluations to assess the impact of the program and estimate the returns 

to the taxpayer. To evaluate whether the program is meeting its stated objectives, EAO employs 

statistical analyses and other methodological approaches to measure program effectiveness in 

terms of: 

• Inputs (program funding and staffing necessary to carry out the ATP mission) 

• Outputs (research outputs from ATP supported projects) 

• Outcomes (innovation in products, processes, and services from ATP supported projects) 

• Impacts (long term impacts on U.S. industry, society, and economy) 

Key features of ATP’s evaluation program include: 

• Business Reporting System, a unique online survey of ATP project participants, that gathers 

regular data on indicators of business progress and future economic impact of ATP projects. 

• Special Surveys, including the Survey of Applicants and the Survey of Joint Ventures. 

• Status Reports, mini case studies that assess ATP projects on several years after project 

completion, and rate projects on a scale of zero to four stars to represent a range of project 

outcomes. 

• Benefit-cost analysis studies, which identify and quantify the private, public, and social 

returns and benefits from ATP projects 

• Economic and policy studies that assess the role and impact of the program in the U.S. 

innovation system 

EAO measures against ATP’s mission. The findings from ATP surveys and reports 

demonstrate that ATP is meeting its mission: 

• Nine out of 10 organizations indicate that ATP funding accelerated their R&D cycle. 

• The existence of a “Halo Effect.” As revealed by EAO surveys, shows that an ATP award 

establishes or enhances the expected value in the eyes of potential investors. 

• ATP stresses the importance of partnerships and collaborations in its projects. About 85 

percent of project participants had collaborated with others in research on their ATP projects. 

Contact ATP’s Economic Assessment Office for more information: 

• On the Internet: www.atp.nist.gov/eao/eao_main.htm 

• By e-mail: atp-eao@nist.gov 

• By phone: 301-975-8978, Stephanie Shipp, Director, Economic Assessment Office, 

Advanced Technology Program 

• By writing: Economic Assessment Office, Advanced Technology Program, National Institute 

of Standards and Technology, 100 Bureau Drive, Stop 4710, Gaithersburg, MD 20899-4710 
Toward a Standard 

Benefit-Cost Methodology

for Publicly Funded Science

and Technology Programs

Jeanne Powell

Economic Assessment Office

Advanced Technology Program

June 2006

U.S. DEPARTMENT OF COMMERCE

Carlos. M. Gutierrez, Secretary

TECHNOLOGY ADMINISTRATION

Robert Cresanti, Under Secretary of Commerce for Technology

NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY

William Jeffrey, Director

ADVANCED TECHNOLOGY PROGRAM

Marc G. Stanley, Director

NISTIR 7319
ii Toward a Standard Benefit-Cost Methodology

The Economic Assessment Office of the Advanced Technology Program

(ATP) seeks to develop a standard methodology for undertaking benefitcost studies of science and technology projects for purposes of quantifying

federal program impacts. A key objective is to facilitate comparability and

aggregation among benefit-cost studies of individual projects. This report

discusses similarities and differences among the ATP’s benefit-cost studies

performed to date. The emphasis is on identifying methodological steps

that can be taken to facilitate consistency and comparability across studies

and aggregation of results of studies performed at different times. Such

aggregation is needed to enable analysis across a portfolio of projects

funded by a given program over time. This report draws on ATP’s

experience in funding risky, industry-led advanced technology projects and

in conducting and publishing benefit-cost studies of nearly 30 projects. It

helps extend the role of the National Institute of Standards and Technology

in international, economics-based standards development by helping create

a comprehensive standard benefit-cost methodology for the science and

technology community.

Keywords: Advanced Technology Program; benefit-cost analysis;

discounted cash flows; program evaluation; science & technology

programs; research & development; research impacts; prospective analysis;

retrospective analysis; social return on investment; public return on

investment.

Abstract
The author would like to thank ATP’s many contractors for their

inspiration and cooperation in developing economic studies of the

program’s impact. Their dedication to rigorous fact finding and fruitful

benefit-cost analysis, despite the uncertainties and long time horizons faced

in applying this tool to early-stage R&D projects, leaves a strong and

continuing legacy. The author would like to acknowledge Lorel Wisniewski

for asking for a comparison of ATP-funded studies that inspired this study

and would also like to thank Stephanie Shipp and Connie Chang for their

encouragement in pursuing it. Thanks are also due to Brian Belanger,

Stephanie Shipp, and Lorel Wisniewski for their helpful review comments

and to Harold Marshall, Building and Fire Research Laboratory, for his

NIST ERB review.

Acknowledgments

Acknowledgments iii
iv Toward a Standard Benefit-Cost Methodology

The Advanced Technology Program (ATP), part of the National Institute of

Standards and Technology (NIST), provides funding on a cost-shared basis

for industry-led technology development projects that would most likely

not be undertaken by the private sector alone. ATP uses benefit-cost

studies as a major evaluation tool for quantifying long-term outcomes and

impacts of its more successful projects. These studies generate metrics that

directly quantify how well ATP is doing against its mission of accelerating

the development of risky technologies that have broad-based economic

benefit to the nation. 

ATP has conducted and published benefit-cost studies of nearly 30 projects

to date. ATP aggregates the measures of net national benefit reported by

these studies to present a minimum estimate of portfolio earnings relative

to the total costs of the program to date. 

Individually, the benefit-cost studies performed to date are consistent with

economic models for measuring social and private returns on public and

private investment in research and development (R&D). ATP worked

directly with the leaders in the field of innovation impact measurement and

growth economics to adapt public finance and business models to ATPfunded, industry-led projects. Contractors with expertise in modeling

societal economic benefits applied these models in in-depth case studies of

ATP projects; the case studies are based on substantial interviews of funded

companies, their customers, and industry experts and on other primary

data collection activities. All studies are consistent with Office of

Management and Budget (OMB) Circular-A94 recommendations for the

use of benefit-cost analysis in general; the underlying premise of cash-flow

analysis; the use of net present value (NPV) as a key metric of program

outcomes; and the Circular’s specific requirements concerning features of

the analysis such as discount rate, handling of inflation, and sensitivity

Executive Summary
analysis. ATP staff had considerable expertise in applying benefit-cost

analysis in their own research before working at ATP and sought out

contractors with similar expertise to perform studies for ATP. NIST

economists are leaders in the international standards-setting community.

Several ATP economics staff were participants in these NIST activities prior

to joining ATP.

Except for studies that cover multiple technology development projects,

benefit-cost studies applied to federally funded technology development

projects tend to provide a “one-shot-deal” analysis. They are funded at

different times in the history of the projects studied and in different years.

Not all compute the same sets of metrics. As a result, impact metrics

computed in different studies, particularly net present value metrics, are

not strictly comparable, and aggregation across studies done at different

times presents methodological challenges. 

This report seeks to bridge that gap. It culminates a series of efforts to

make both existing and future benefit-cost studies of ATP projects more

useful to the evaluation process. Previous efforts have included an analysis

of studies conducted to date, a workshop of public and private researchers,

and a presentation to the American Evaluation Association. 

Analysis of the published studies of approximately 30 projects revealed a

great many similarities:

• All address ATP’s mission. They aim to show, and succeed in showing,

that ATP enables technology development that benefits industry and end

users and generates broad-based benefits to the U.S. economy. 

• The studies follow the economic paradigms that emphasize the market

failures in early-stage technology development, the substantial gap

between social returns on technology investment (public plus private)

and private returns alone, and the need to determine instances in which

market failures will prevent this gap from being filled without

government assistance.

• Substantial “bottom-up” interviews are conducted with ATP-funded

company representatives and customers, as well as other market

research, to estimate the benefits of ATP-funded technologies (compared

with existing defender technology as a minimum counterfactual to ATP

funding) and market demand. In general, benefits are estimated on the

basis of a single product unit sold and are extended to an estimated

market. Studies assess the effect of ATP funding on the scope and timing

of the projects. 

• Cash-flow analysis techniques are implemented consistent with public

finance literature and good practice in both public and private

investment analysis. 

Executive Summary v
• The studies compute the same basic metrics: net present value, benefitto-cost ratio, and internal rate of return. 

• Considerable uncertainties exist about the impacts being measured.

Further comparison with the guidelines for benefit-cost analysis provided

by OMB Circular A-94 revealed that the published studies were consistent

with those requirements.

• The studies emphasize the net present value of outcomes.

• They consistently use the OMB-mandated rate of 7% (real rate) in

discounting benefits and costs to a common base year.

• They consistently estimate benefits and costs in constant dollars,

generally using the dollar value at the time the study was conducted.

• They address uncertainties and include some sensitivity analysis.

OMB Circular A-94, however, does not provide a complete framework for

using these studies for evaluating R&D project outcomes and impacts for

purposes of retrospective program evaluation of actual program outcomes.

It appears to be targeted more to prospective analysis of new federal

programs or projects. 

The analysis identified four sources of inconsistency in past ATP studies

that impede the use of study results, particularly in aggregation. 

• Timing of studies relative to ATP funding and the technology life cycle

and the treatment of uncertainties about projected future outcomes.

• Identification of the specific counterfactual to ATP funding and the

related issue of attribution to ATP in cases of multiple funding sources.

• Which metrics to use: Social returns (that is, public and private benefits

on combined public-private investments) and/or public return on ATP

investment?

• Different base years and different constant dollar years: Often base year

and constant dollar year differ in a given study and across studies. 

These areas of inconsistency were discussed with workshop practitioners

and explored further for the American Evaluation Association’s Annual

Conference (Toronto, October 2005) and for this report in the context of

three broad criteria:

• ATP’s evaluation objective of measuring program impacts against the

program’s mission while also considering tools appropriate to the

evaluation of public-private science and technology programs more

broadly.

vi Toward a Standard Benefit-Cost Methodology
Executive Summary vii

• Quality relative to analytical models established in the economic and

public finance literature and accuracy as provided by established cashflow analysis procedures used by the business community. 

• Effects on consistency and comparability across studies and the ability to

aggregate results of different studies.

This analysis yields the following conclusions and recommendations:

• Study timing. Highly prospective studies of project outcomes,

particularly if performed before technical risks and uncertainties have

been overcome and business risks significantly mitigated, may not

generate credible or useful estimates of program impacts even if they

meet high standards of economic modeling and rigor. Probability

distributions of long-term advanced technology project outcomes are

extremely difficult to estimate. Given the uncertainties about these

outcomes, at a minimum some combination of retrospective and

prospective (ex ante) analysis is appropriate as long as the analysis

includes direct evidence of actual commercialized products or processes

that incorporate the program-funded technology. 

• Counterfactual and attribution. ATP-funded studies have used various

mechanisms to model incremental benefits of an ATP-funded technology

relative to the counterfactual situation. Mechanisms include acceleration

of benefits and increased likelihood of achieving benefits. Sometimes

achieving any quantifiable economic benefits from a project required

funding from multiple external sources, each of which was indispensable.

A conservative approach is to allocate benefits in some equitable way

among funding sources. Identification and program attribution of

benefits require the following: (1) matching program-funded projects to

direct project outcomes to the extent practical, by tracing product

outcomes backward from existing company products to their R&D

project origins and forward from the ATP-funded projects through the

product development stages to identify the major intersections; and (2)

considering appropriate attribution of all or partial benefits to ATP. 

• Which metrics: Social return on investment and/or public return on

public (ATP) investment? ATP has focused on these two sets of

measures. Unfortunately, some studies produced the social return metric,

which compares combined public and private benefits of ATP-funded

technology with combined public and private investments; others

produced just the public return on ATP’s investment. (Variants of these

two sets of measures of return appear in the evaluation literature, but

ATP has focused on these two basic themes.) Both sets of measures have

utility, and both have advantages and disadvantages. Combined publicprivate benefit-cost analysis is logical for cost-shared technology
viii Toward a Standard Benefit-Cost Methodology

development projects and helps identify substantial gaps between full

societal benefits and private benefits that justify federal funding;

however, data collection is difficult. The public return on ATP’s

investment provides a clearer measure of programmatic impact against

its mission of generating broad-based benefit to the nation. The best

strategy would appear to be to compute both sets of measures, with

attention to keeping the project definitions and traceable benefits

consistent for both and to developing practical estimation techniques for

computing social return metrics that alleviate data problems. 

• Adjusting net present values for timing differences among studies and

projects. Studies performed at different times use different base years and

constant dollar years, although ATP’s cluster and focused program

studies all use a common constant dollar year for a given study and

always use the year of first cash flows as the base year of analysis for all

projects in the study. Nevertheless, other base year choices might be

equally or more meaningful for such studies, for example, the year the

portfolio analysis is performed. 

In cases in which multiple projects are analyzed in a single study, or in

which results of multiple studies are considered together to compute a

minimum estimate of portfolio performance, adjustment to constant

dollars across the studies will be necessary. For comparability with other

studies consideration should also be given to using a common base year for

discounting before aggregating across studies. This consideration should

include proper interpretation of the discounting effects on metrics that

result from different base year choices and an assessment of which

approach makes sense for the evaluation objective. 

The adjustments are a straightforward process and can be made after

studies have been completed using different constant dollar years and

different base years for discounting. 

ATP’s efforts to document and investigate methodological similarities and

differences across its benefit-cost studies have demonstrated the utility and

commonality of the studies, as well as the complexities beyond pure

estimation issues. The challenges do not appear insurmountable. Analysis

of key methodological differences among ATP’s benefit-cost case studies of

nearly 30 projects has helped the program assess the significance of these

differences and pose solutions. The goal is to circulate this report to users

of benefit-cost results and to benefit-cost analysis practitioners. Feedback is

sought and anticipated with the hope that a living, working document will

support a more formal, uniform standard-setting process for benefit-cost

analysis in the future. 
About the Economic Assessment Office . . . . . . . . . . . . .Inside Front Cover

Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .ii

Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .iii

Executive Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .iv

I. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1

II. Rationale and History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5

III. Similarities and Differences in ATP’s Studies . . . . . . . . . . . . . . . . . .13

IV. Study Timing and Uncertainties about Future Outcomes . . . . . . . .17

V. Identifying the Counterfactual and Attribution of Benefits . . . . . . .21

VI. Which Metrics: Social Return on Investment and/or Public Return 

on Public (ATP) Investment? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .27

VII. Adjusting for Timing Differences across Studies and Projects: 

Constant Dollars and Base Year for Discounting . . . . . . . . . . . . . .33

VIII. Conclusions and Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . .43

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .45

About the Author . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Inside Back Cover

About the Advanced Technology Program . . . . . . . . . . .Inside Back Cover

List of Tables

1. ATP’s Benefit-Cost Studies to Date . . . . . . . . . . . . . . . . . . . . . . . . . .10

2. Framework of Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16

3. Study Timing and Uncertainties about Future Outcomes: 

Contrasting Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .18

4. Identifying the Counterfactual and Attribution of Benefits—

Contrasting Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .23

Contents ix

Contents
x Toward a Standard Benefit-Cost Methodology

5. Which Metrics? Contrasting Projects in Component-Based 

Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .28

6. Which Metrics? Other Contrasting Studies . . . . . . . . . . . . . . . . . . . .29

7. Adjusting for Timing Differences across Studies and Projects: 

Constant $ and Base Year (Example 1: Ranking Does 

Not Change) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .36

8. Adjusting for Timing Differences across Studies and Projects: 

Constant $ and Base Year (Example 2: Ranking Changes) . . . . . . . .36

List of Figures

1. Adjusting to Constant 2005 Dollars . . . . . . . . . . . . . . . . . . . . . . . . .37

2. Adjusting Base Year to 2005 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .38
Origins

Program evaluation informs program and policy decision making

concerning how well a program is working in meeting its goals and

objectives. For government programs, these goals and objectives include

social and economic outcomes as well as the operations of the program or

its immediate outputs. Under procedures to ensure independence, program

evaluation may be conducted by independent experts external to the

program or by internal program managers. The subset of program

evaluation comprising impact evaluation, as compared with ongoing

project monitoring and reporting, assesses the net effect of a program by

examining outcomes and comparing them with what would have happened

in the absence of the program. Benefit-cost analysis directly compares a

program’s outputs and outcomes with the costs (resources expended) to

produce them (Office of Management and Budget [OMB] 2004). 

OMB Circular A-94 recommends benefit-cost analysis as the technique to

use in a formal economic analysis of government programs or projects

(OMB 1992). It states that “social net benefits, and not the benefits and

costs to the Federal Government, should be the basis for evaluating

government programs or policies that have effects on private citizens or

other levels of government.” These measurements offer the intuitive

simplicity of tools employed routinely in business management and

finance. However, the OMB mandate is for public programs to apply a

more complex public finance model that extends the analysis to the

benefits to society broadly, not just direct recipients of public funds and

not to the funding agency itself. Economists have long recognized that

there is often a significant divergence between private returns (i.e., the

returns to the innovator) and society-wide benefits. The extended public

finance models enable systematic quantification of societal returns and the

divergence. 

Introduction 1

I.

Introduction
Purpose

The Economic Assessment Office (EAO) of the Advanced Technology

Program (ATP) seeks to develop a standard methodology for undertaking

evaluation benefit-cost studies of science and technology (S&T) projects it

has funded for purposes of quantifying the program’s outcomes and

impacts. ATP seeks to go beyond treating each evaluation case study as a

“one-shot deal” and to develop approaches that facilitate comparison and

aggregation across different studies. ATP has conducted numerous benefitcost studies in the past that are consistent individually with OMB Circular

A-94 and standard published methodologies. Some studies have analyzed a

number of related projects. Although adequate to identify whether

individual projects or small groups of projects analyzed together are costeffective, the net present value (net benefit) impact metrics computed in

different studies are not strictly comparable, and aggregation across studies

is problematic. 

Differences in basic assumptions, timing, and specific metrics used do not

adversely affect the value of any individual study, but they hinder the

maximum use of the studies by ATP in examining its overall portfolio. In

this report, ATP draws on a large body of past benefit-cost studies, the

experience and knowledge gained from a workshop with a number of

practitioners in June 2004, many hours of discussion with ATP’s

contractors, in-house analysis, and a presentation to the American

Evaluation Association in October 2005 to (1) document consistencies and

inconsistencies and sources of incomparability across studies and (2) frame

an approach for extending the use of benefit-cost analysis from its typical

use in case studies of individual “successful” projects to the evaluation of a

portfolio of federal S&T projects. 

ATP Background

ATP is a government-industry partnership whose mission is to accelerate

commercial development of risky, innovative technologies with potential

for broad economic benefit to the nation. Its mechanism is to cost share

with private industry the funding of high-risk research and development

(R&D) with broad commercial and societal benefits. Through a

competitive selection process, ATP chooses projects by applying evaluation

criteria: 

• Scientific and technological merit (50%) 

• Potential for broad-based economic benefits (50%) 

2 Toward a Standard Benefit-Cost Methodology
Selection emphasizes projects that would otherwise not be undertaken

because the risks are too high or because the benefits would not accrue to

private investors. 

Since its inception in 1990, ATP has announced 768 awards to single

companies and joint ventures and has awarded approximately $2.3 billion;

industry has provided approximately $2.1 billion as its cost share. 

The program anticipates that the major benefits accrue through the

creation of new products and processes embodying ATP-funded

technologies and their successful commercialization. ATP’s EAO seeks to

measure outputs, outcomes, and impacts of these technology development

projects through a comprehensive portfolio of survey and statistical

techniques, case studies, and special issue studies. Generally, case studies,

which include high-level overview studies of all completed projects and indepth benefit-cost studies of selected high-impact projects, are performed

by independent contractors. 

Methodology

ATP’s methodology for this report involves the following:

1. Pursue the program evaluation objective: the evaluation of ATP

program impacts against ATP’s mission of accelerating the commercial

development of risky technologies with the potential for broad

economic benefit to the nation.

2. Examine the economics literature for the theoretical rationale and

measurement history of social and private rates of return on R&D

investment.

3. Consider the OMB mandate for benefit-cost analysis, including specific

requirements and general implications for evaluation of public-private

S&T projects.

4. Draw on experience and knowledge gained from an ATP workshop

that convened private sector and National Institute of Standards and

Technology (NIST) practitioners in June 2004.

5. Develop a general methodology from knowledge and experience

derived from these external sources.

6. Analyze ATP studies conducted to date for similarities and differences

and consistency with this model.

7. Identify key sources of the differences, examine their effects, and

consider approaches to addressing inconsistencies.

Introduction 3
8. Introduce the initial version of the methodology at a presentation to

the R&D Technical Interest Group at the American Evaluation

Association in October 2005.

9. Recommend next steps. 

In large measure, this report is the text supporting the presentation the

author made to the American Evaluation Association at its “Crossing

Borders, Crossing Boundaries” Conference on October 26–29, 2005.

4 Toward a Standard Benefit-Cost Methodology
Economic Literature

Benefit-cost methodologies applied to S&T projects derive from Edwin

Mansfield’s work in the 1970s (Mansfield et al. 1977), which is aptly

captured in the recent Journal of Technology Transfer issue

commemorating Dr. Mansfield (Scherer 2005). Mansfield adapted his

private sector analysis specifically to ATP-funded projects in a report for

ATP (Mansfield 1996). His public finance approach is an extension of

traditional business cash-flow analysis that applies basic concepts in public

finance to measure benefits to industry users of technology products and

end users as well as the innovators. 

Return on investment in developing new technology is thus expanded. It

includes consumer surplus in the form of cost reductions and other societal

benefits experienced by technology users as well as innovators’ profits from

technology products and cost-reducing new processes measured relative to

a hypothetical, counterfactual situation in which the new technology does

not exist. 

Mansfield’s work provides a foundation for public sector–funded S&T case

studies following two themes: 

1. His evidence that social rates of return on industry-led projects

(combined public and private returns) exceed private rates of return

indicates the potential for market failure in the form of

underinvestment in R&D relative to the societal optimal level and thus

possible justification for federal subsidies to R&D. His pioneering

work in the 1970s and subsequently in two studies sponsored by the

National Science Foundation (NSF) showed private rates of return

averaging 25%–36% and social rates of return averaging 50%–70%.

(The lower rates were reported in the 1977 study.) Several other

Rationale and History 5

II.

Rationale and History
studies, for example, by Scherer and Piekarz, confirmed and extended

these ideas and results and discussed some of the policy issues (Teece

2005). The macroeconomic studies examining returns to R&D, for

example, by Griliches and Nadiri using broad statistical databases,

showed similar results, although differences across industries were

substantial (Bernstein and Nadiri 1988; Nadiri 1993). 

Further comparison of private rates of return to such projects with

industry and company hurdle rates provides a better indication of

whether the private sector is likely to undertake such investments in

the absence of federal funding. If company hurdle rates (defined here

as the rate of return required to undertake an investment with a given

level of risk) are high relative to opportunity cost private rates of

return on projects of comparable risk, companies either will not make

the investments at all or they will invest only at a minimal exploratory

level, and benefits become increasingly longer term and more

problematic without federal funding. If hurdle rates are comparable

with private rates of return, companies are more likely to undertake a

significant investment without federal funding, and federal funding

may be displacing private funding (Jaffe 1996, 1998). 

Quantification of the difference between social and private rates of

return, called the “spillover gap,” and evidence that private rates are

below industry or company hurdle rates become program

measurement objectives. The theoretical basis of analysis is that a

spillover gap justifies public funding and, therefore, quantification of it

is a primary program evaluation objective. R&D will be

underinvested, and projects merit consideration for public funding to

the extent that private rates are below social rates of return and

therefore R&D is unlikely to be funded at a socially optimal level. The

case is even clearer if private rates of return are below the investment

hurdle rate for risky R&D and, therefore, the investment most likely

would not be made at all without federal funding (Jaffe 1996, 1998). 

2. Mansfield establishes a case study method supported by customized

data collection as an appropriate methodological tool of economic

analysis. Scherer quotes Mansfield’s statement: “If you want to know

something, ask the people who know” (Scherer 2005, p. 5). Mansfield

described the need to interview personally not just the “doers,” but

also the beneficiaries among customers and users (ATP 1996). Data

often do not exist for large statistical analysis of impacts but can be

created at a smaller level. Teece described his instructions to his

students “to collect data in the field” (Teece 2005, p. 18) and quotes

Mansfield’s introduction to his two volumes of collected papers: “in

general, my approach has been to try to get a reasonably solid

empirical footing before attempting to model complex phenomena

about which very little is known; to keep the theoretical apparatus as

6 Toward a Standard Benefit-Cost Methodology
simple, transparent and robust as possible; to collect data directly from

firms (and other economic units) carefully tailored to shed light on the

problem at hand (rather than to try to adapt readily available generalpurpose data, which is often hazardous), and to check the results as

thoroughly as possible with technologists, executives, government

officials and others who are close to whatever phenomenon is being

studied” (Mansfield 1995, p. ix). The advantage is that questions and

resulting data are targeted to what you really need to know.

OMB Circular A-94

OMB’s mandate that federal programs use benefit-cost analysis or costeffectiveness analysis goes back several decades before Circular A-94 was

revised in 1992. Most of the Circular A-94 presentation is geared to a

prospective analysis of an entire federal program and provides little

guidance for evaluating project and program outcomes and impacts, in

general, or basic and applied S&T programs, in particular. It predates

government-wide mandates for program performance metrics and is more

comprehensive and technically challenging than could easily be imposed or

implemented by all agencies. Nevertheless, Circular A-94 is a methodology

tool for program evaluation efforts stimulated by the Government

Performance and Results Act (GPRA) (1993). Its guidelines are consistent

with good practice for both retrospective and prospective analyses, at the

program or project level, and for any program area. 

OMB Circular A-94 establishes the following guidelines:

Outcome Measures

1. Use net present value as the standard criterion for deciding whether a

government program can be justified on economic principles.

2. When an analysis of competing alternatives is done, consider in

addition the present value of benefits relative to a given amount of

cost, or vice versa, the cost relative to a given amount of benefits, all

measured in present value terms. 

3. Be explicit about underlying assumptions.

Net Benefits Measurement

1. Use the net benefits to society, not to the federal program, as the basis

for evaluating government programs. 

2. Identify incremental benefits and costs relative to the situation in

which the government program does not operate. Record displaced

activities as a cost.

3. Include effects on U.S. citizens, and not on others.

4. Use consumer surplus and “willingness to pay” as measures of value

beyond what is captured in market prices.

Rationale and History 7
5. Assume full employment of resources. Avoid multiplier-based estimates

of effects of government spending on the economy. 

Treatment of Inflation

1. Use real or constant dollar values to perform analyses.

2. Use the administration’s GDP deflator if an inflation adjustment is

needed.

Use of Discount Rate 

1. Report net present value and other outcomes using a real discount rate

of 7% in performing constant-dollar benefit-cost analyses of proposed

public investments and regulations. This rate is intended to

approximate a marginal pretax rate of return on an average investment

in the private sector.

Treatment of Uncertainty

Typically, estimates of benefits and costs are uncertain. 

1. Identify sources of uncertainty, provide expected value estimates, and

test sensitivity of estimates to important sources of uncertainty. 

2. Where possible, derive a probability distribution for benefits, costs,

and net benefits.

The Circular A-94 guidance is applicable and appropriate either to

prospective program-level analyses or to retrospective and prospective

project-level analyses. It provides minimum requirements for benefit-cost

analysis of an individual project; however, it does not address problems of

aggregation of results for projects and studies performed at different times,

and it does not provide guidance specific to S&T programs. This report

seeks to fill in some of those gaps.

NIST and ATP

NIST’s economists have been actively engaged for nearly 40 years in cash

flow–based benefit-cost analysis in microeconomic case studies of R&D

projects. NIST’s Building and Fire Research Laboratory (BFRL) economists

are leaders in the international standards community. Their work in

building and energy economics includes development and application of

life cycle cost models for energy conservation investments and development

and implementation of training programs. NIST’s BFRL economists’ work

in modeling and building economics standards has resulted in a series of

American Society for Testing and Materials (ASTM) standards for 

building economics (ASTM 2004). And their work for NIST and other

agencies has sometimes entailed program evaluation. Several members of

8 Toward a Standard Benefit-Cost Methodology
the ATP economics staff were participants in these BFRL activities before

joining ATP. 

Benefit-cost case studies became a cornerstone of the evaluation of longterm outcomes of NIST’s laboratory programs in the 1980s and in

planning studies for new programs. More than 30 separate studies have

been commissioned by NIST’s Program Office. For more information see

the NIST Web site: http://www.nist.gov/director/planning/

impact_assessment.htm.

ATP’s founding legislation required the evaluation of program outcomes

before that practice was common or required by GPRA. ATP naturally

drew on the BFRL and NIST experience. In addition, ATP drew on Albert

Link’s growing involvement in program evaluation, including his benefitcost studies of outcomes of a number of NIST laboratory projects for the

NIST Program Office during the 1980s. Albert Link helped ATP draft a

preliminary plan for program evaluation. In its early years, ATP funded

Link to undertake several early benefit-cost studies that focused on

research cost savings to industry that result from avoiding redundant

research through federal sponsorship and the cost sharing of technology

development by industry consortia. ATP engaged Ed Mansfield and others

to develop appropriate methodologies for assessing longer-term economic

impacts using benefit-cost (among other) techniques. Since that time, more

than a dozen benefit-cost studies have been published. Several published

studies analyze groups of related ATP-funded projects. 

Twelve of these studies, covering 28 ATP projects, are listed in Table 1.

The cost of conducting these studies ranged from $25,000 to $357,000

(not adjusted for inflation). More than half of the studies focused on one

ATP project; the rest covered from two to eight ATP projects. All are

available on ATP’s Web site: http://www.atp.nist.gov/eao/eao_pubs.htm.

Other Agencies

Although a regular tool of public finance, benefit-cost analysis for

evaluating program impacts of federal S&T programs has not been

common outside NIST and ATP. As the Government Performance and

Results Act (GPRA)1 percolated through agencies in the early 1990s,

information sharing increased, and interagency evaluation networks and

workshops evolved with ATP participation. ATP staff engaged in training

Rationale and History 9

1. Enacted in 1993, the GPRA provides for the establishment of strategic planning and

performance measurement in the federal government.
programs and participated in a variety of evaluation forums. The OMB

Performance Assessment Rating Tool (PART) was born in 2002.2

Throughout their interactions with other agencies, ATP staff members

encountered few examples of benefit-cost analysis outside NIST. In the

R&D-S&T arena, agencies had been relying largely on anecdotal

information and peer review. Data difficulties and outcome uncertainties

abound for all programs.

Nevertheless, benefit-cost analysis offers considerable promise for bridging

economic analysis to traditional financial analysis tools and metrics that

10 Toward a Standard Benefit-Cost Methodology

Table 1. ATP’s Benefit-Cost Studies to Date

Study

Short Title (# of projects) Number Contractor Pub Date Cost-$

Photonics cluster (2) GCR 05-879 Delta Research 2005 228,000

Composites cluster (2) GCR 04-863 Delta Research 2004 200,000

2mm project—retrospective (1) GCR 03-856 MIT 2004 357,000

HDTV joint venture (1) GCR 03-859 RTI International 2004 75,000

A-Si Detectors for digital 

mammography (1) GCR 03-844 Delta Research 2003 136,000

Component-based software (8) GCR 02-834 RTI International 2002 154,000

Closed-cycle refrigeration (1) GCR 01-819 Delta Research 2001 77,000

Digital data storage (2) GCR 00-790 RFF 2000 138,000

Flow-control machining in NIST-Building 

auto industry (1) NISTIR 6373 & Fire Research 1999 90,000

Tissue engineering (7) GCR 97-737 RTI International 1998 122,000

2mm—Early assessment (1) GCR 97-709 CONSAD 1997 25,000

Printed wiring board (1) GCR 97-722 Albert N. Link 1997 22,500

2. In 2002 the Director of the Office of Management and Budget (OMB) announced

development of the PART for formally evaluating the effectiveness of federal programs.

He described the PART’s purposes as follows: “The PART is a systematic method of

assessing the performance of program activities across the Federal Government. The

PART is a diagnostic tool; the main objective is to improve program performance. The

PART assessments help link performance to budget decisions and provide a basis for

making recommendations to improve results.”
agency stakeholders understand. And it provides an approach to directly

measure program impact on the economy. ATP’s authorizing legislation

requires economic evaluation and such metrics and thus gave ATP a head

start. GPRA and PART requirements now facing all U.S.-government

funded programs are somewhat broader but have the same goal: to

demonstrate outcomes and impacts affecting society broadly, beyond the

immediate program participants. 

ATP-Sponsored Workshop on Selected Methodological

Issues in Benefit-Cost Analysis

ATP’s Economic Assessment Office conducted a one-day workshop in June

2004 to address a small number of issues that had become apparent in the

course of sponsoring benefit-cost studies. Evaluation practitioners who had

conducted studies for ATP or who had expressed interest in benefit-cost

analysis were invited to the workshop at NIST. NIST experts were invited

also. The workshop had 15 participants, with varying experience in these

types of studies. Through structured questioning, workshop participants

addressed a number of issues arising in the course of ATP’s studies. 

Topic 1: Measures of Performance

• Social (public plus private) returns or public returns

• Advantages and disadvantages of each approach 

• Approaches to sensitivity analysis

• Comparison/aggregation across multiple project cases

Topic 2: Attribution to ATP

• What is the appropriate counterfactual? Is it different for different types

of measures?

• What effect do multiple funding sources have?

• What effect does joint venture participation have?

Topic 3: When is the best time to study project performance?

• After technology hurdles have been overcome? After commercialization

has begun? Later?

• What are best approaches to dealing with uncertainties about project

outcomes in conducting prospective and/or partially prospective

analyses?

Topic 4: Bridging the gap from microeconomic case study to

macroeconomic effects

• What are the basic requirements for implementing macro models

successfully?

• Is it feasible to convert existing micro case studies to the macro level?

Rationale and History 11
• Is it feasible to aggregate existing case studies to generate input to a

macro model?

The discussions cited the overarching role of OMB Circular A-94 and

highlighted similarities, consistency, and good practice in existing studies.

The discussions also demonstrated the need to pursue approaches to

thorny issues in which experienced practitioners either were in some

disagreement about the best practice or had not considered the issue.

Practitioners clearly were focused on providing quality studies that

followed established principles and good practice; however, they disagreed

somewhat on what metrics should be presented among economically

appropriate choices. They focused on delivering a high-quality product for

their own specific contracted work. They had thought little about the way

federal programs should present results of different studies performed at

different times in project lives by different contractors. 

12 Toward a Standard Benefit-Cost Methodology
ATP has used aggregates of results of benefit-cost studies in comparing

benefits from the limited number of projects for which such studies have

been done with the entire costs of the Advanced Technology Program to

date. These comparisons have often showed that estimated benefits exceed

program costs to date by more than an order of magnitude. However, the

results of this portfolio analysis were known to be very rough

approximates of portfolio minimum performance, given that the studies

were performed at different times and at different stages of project life

cycles, a mix of retrospective and totally prospective benefits were

quantified, and somewhat different metrics were reported. In addition,

assumptions used in highly prospective studies sometimes proved too

optimistic over time. 

In recognition of these difficulties, ATP economists seek to develop a more

comprehensive, integrated approach to benefit-cost studies that will

support aggregation of results and comparability. With the passage of time,

the possibilities for more retrospective analysis improve. 

As a foundation for such an integrated effort, ATP has documented the

similarities and differences across its benefit-cost studies that appear to

affect the ability to combine results of individual studies to describe overall

impacts of the program to date. 

Similarities 

ATP’s benefit-cost studies have much in common: 

• All address ATP’s mission. They aim to show, and succeed in showing,

that ATP enables technology development that benefits industry and end

users and generates broad-based benefits to the U.S. economy. 

Similarities and Differences in ATP’s Studies 13

III.

Similarities and Differences 

in ATP’s Studies 
14 Toward a Standard Benefit-Cost Methodology

• The studies follow the analytical paradigms of Mansfield and Jaffe,

which emphasize (1) the large size of public benefits relative to private

benefits that can often be expected from enabling new technologies

(Mansfield et al. 1977), (2) the role for federal funding in addressing

market failures in early-stage technology development (Jaffe 1996), (3)

the substantial gap between social returns on technology investment

(public plus private) and private returns alone (Mansfield et al. 1977;

Jaffe 1996), and (4) the need to determine where federal funding is

needed for these benefits to occur (Jaffe 1996, 1998). 

• Substantial “bottom-up” interviews with ATP-funded company

representatives and customers are involved, as well as other market

research, to estimate benefits of ATP-funded technologies (relative to

existing or anticipated defender technology) and market demand. In

general, benefits are estimated on the basis of a single product unit sold

and extended to an estimated market. In instances in which the project

would most likely have been funded at some level without ATP, the

studies assess the effect of ATP funding on scope and timing of the

projects compared with a hypothetical counterfactual situation without

ATP funding. 

• The studies implement cash-flow analysis techniques consistent with

public finance literature and good practice in both public and private

investment analysis. 

• The same basic metrics are computed: net present value, benefit-to-cost

ratio, and internal rate of return. 

• Considerable uncertainties exist about impacts being measured.

• The studies are consistent with OMB Circular A-94 in computing the net

present value of outcomes as the key metric of cost-effectiveness.

■ They consistently use the OMB-mandated rate of 7% (real rate) in discounting benefits and costs to a common base year.

■ They consistently estimate benefits and costs in constant dollars, generally using the dollar value at the time the study was conducted.

■ They address uncertainties and include some sensitivity analysis.

• The studies follow similar approaches to estimating period of economic

benefits. None assume more than 10 years; all consider competing

developments and assess the economic life of the technology.

• ATP projects a priori believed to be economically successful and

therefore worth the investment in the study are selected for these studies,

although some studies cover a broader cluster or portfolio of projects. 

• Study results are not assumed to be representative of the outcomes for

the portfolio of ATP projects and are not presented as such. Instead they

are presented as impacts of some of ATP’s more successful projects. In

aggregation, the estimated impact of these projects alone is assumed to

be no better than a minimum of impact for the entire portfolio. ATP uses
other evaluation tools, including high-level overview case studies and a

composite performance rating system, to examine all projects—

successful and unsuccessful. The quantitative benefit-cost studies

examine only some of the more successful projects for which advance

screening suggested substantial measurable economic and social impact. 

Differences 

Comparability of studies, potential for aggregating results across studies,

and actual utility in program evaluation are affected by a number of

factors. From hindsight, ATP economists are learning how to compensate

for some nonoptimal conditions and plan to address some key sources of

difference in ongoing efforts such as this methodological study and further

case studies. Differences can be summarized as follows: 

• Timing of studies and uncertainties about projected future outcomes

• Which metrics? Social returns (that is, public and private benefits on

combined public-private investments) and/or public return on ATP

investment

• Identification of specific counterfactual to ATP funding

• Attribution to ATP given multiple funding sources

• Different base years and different constant dollar years (base year and

constant dollar year often differ in a given study and across studies) 

Framework of Analysis

The rest of this report examines these differences as evidenced in our

portfolio of published studies. We draw on results of the workshop and the

more comprehensive analysis of differences across our portfolio to start

laying a foundation for a standard approach to benefit-cost analysis for

publicly funded R&D projects. Three criteria are applied in considering an

appropriate standard approach: 

• ATP’s evaluation objective of measuring program impacts against the

program’s mission while also more broadly considering tools appropriate

to the evaluation of public-private science and technology programs

• Quality relative to analytical models established in the economic and

public finance literature and accuracy as provided by established cashflow analysis procedures used by the business community 

• Effects on consistency and comparability across studies and the ability to

aggregate results of different studies

The matrix shown in Table 2 illustrates this framework. 

Similarities and Differences in ATP’s Studies 15
The analysis is limited to studies that used standard cash flow–based

financial analysis techniques envisioned in OMB Circular A-94, although

many of the issues addressed apply similarly to other quantitative case

study approaches. 

16 Toward a Standard Benefit-Cost Methodology

Table 2. Framework of Analysis

Evaluation Quality Comparability 

Objective and Accuracy and Consistency

Timing of study and 

uncertainties about 

benefit outcomes

Which metrics:

Social and/or public?

Role of ATP: Identifying 

the counterfactual and 

appropriate attribution 

to ATP

Adjusting for 

time differences:

Base year

Constant dollars
What Are the Issues?

Technology development benefits as measured by outcomes and impacts

are highly uncertain and dependent events. At the time that ATP funds a

given technology development project, the risks of technical failure (or of

meeting only limited technical objectives) are very high. By the end of the

period of ATP funding, technical hurdles are reduced; however, even for

projects deemed successful technically, generally, significant R&D is still

needed before the technology has significant economic value. Market,

financial, and business risks remain high even for projects achieving a high

level of technical success. Product development phases are often more

costly than early-stage R&D financing such as ATP funds. In some

technology areas, the time to full commercialization after ATP funding

ends can be a decade or more. 

Nevertheless, ATP initiated benefit-cost studies early on in the brief 15-year

history of the program—partly because the economics staff had significant

expertise in these methodologies, but also because of pressure from

stakeholders for quantitative program impacts. At the time the earliest

studies were conducted, no ATP projects had fully matured. ATP had no

experience with the range of outcomes that should be anticipated, and the

empirical data available to independent researchers for assessing

technology risks and adoption were extremely limited. No one had a good

sense of the true probability distributions.

The more recent studies were conducted somewhat later in the technology

life cycle. Technology risks had been reduced, and projects studied had

made significant progress toward commercialization. All of the studies

conducted to date were either entirely or partially prospective analyses.

The recent ones have had a retrospective component, and/or barriers to

substantial economic impact were considered small. ATP attempts to use

Study Timing and Uncertainties about Future Outcomes 17

IV.

Study Timing and Uncertainties

about Future Outcomes
results from all of the studies to construct a minimum of the net benefits of

the program. 

Contrasting Examples

Table 3 illustrates this variability in timing by contrasting two large studies

that examined a number of different ATP projects funded in the areas of

tissue engineering and component-based software. The two studies were

performed by the same independent research organization.

18 Toward a Standard Benefit-Cost Methodology

Table 3. Study Timing and Uncertainties about Future Outcomes:

Contrasting Examples

Timing Relative

Study to ATP Funding Technology Status Treatment of Uncertainty

Tissue engineering Early in ATP for Technologies not Barriers to meeting technical 

(7 cases) 3 cases; soon after developed and/or not and economic goals not 

ATP $ ended for ready for assessed. Sensitivity analysis 

4 cases commercialization performed, but showed results 

(Biotech projects have not very sensitive to input 

very long timeline) variables tested.

To be conservative, only one 

application considered per case.

Component-based Soon after ATP Technologies complete; Selection emphasized projects 

software $ ended in commercialization with revenues to date and 

(8 cases) (IT projects have short near-term prospects.

timelines) Benefit analysis quantified only 

products actually on the market 

and assumed short product lives.

The tissue engineering study was done in the mid-1990s. Performance

metrics estimated in the study showed huge impacts; however, the study

was conducted very early in the project life cycle of every project covered.

Of the seven projects analyzed (all of those ATP had funded in the area of

tissue engineering), four had completed their ATP funding period and had

significant technical accomplishment; three had just recently started. All

faced major continuing technical hurdles, including additional R&D and

clinical trials regulated by the Food and Drug Administration. The study

included some probabilistic assessment and sensitivity analysis; however, it

did not consider the full breadth of possible outcomes or provide a realistic
weighting for them. Estimates were based on company interviews and

other investigation. 

The methodological emphasis was on state-of-the-art modeling of patient

benefits and an illustration of the patient-benefit model to ATP’s tissue

engineering portfolio at the time. The treatment of uncertainty about

technical and business outcomes needed for these benefits to be realized

was more rudimentary, although it drew on approaches in the economics

literature. 

The treatment failed to take adequate account of the very early stage, risky

nature of these projects. Outright technical or business failure was not

envisioned as an outcome. All the projects were anticipated to yield

substantial medical benefits. ATP was expected to accelerate the technology

life cycle and to improve probabilities of technical success incrementally by

increasing the level of R&D funding. Company-provided assessments of

progress toward demonstrating technical feasibility (adjusted by progress

to date) were used as a proxy for probability of technical success. The

schedule for technology adoption adapted an established diffusion model

that probably was not adequate for addressing the long-term, high-cost

regulatory barriers to adoption of this type of technology. 

From hindsight, ATP has learned that typically biotechnologies face a

particularly long timeline to commercialization, with ATP funding often

coming at an early point in the R&D cycle. A recent survey of the seven

projects in the tissue engineering study confirmed that the study was

overoptimistic. None of the projects had achieved the level of benefits

expected by that time. Three technologies now deemed the most successful

continue to make technical progress and have entered at least the early

stages of commercialization, but benefits are evolving much more slowly

than projected for all of them. Three technologies failed to develop; one is

in transition to a new company, with its future as yet uncertain. 

The component-based software study was performed about five years later.

The eight projects analyzed in depth received ATP funding in

approximately the same time frame as the tissue engineering projects.

Project selection for in-depth study emphasized projects with revenues to

date and near-term prospects. All those selected had reached the

commercialization stage when the study was performed. Estimated benefits

from the eight projects reflected a mix of retrospective and prospective

analysis, but with uncertainties greatly reduced for both compared with

analyses of early-stage R&D projects. 

Theoretically, prospective benefit-cost analysis can model risk through

probability distributions and various estimation tools. But pure

Study Timing and Uncertainties about Future Outcomes 19
20 Toward a Standard Benefit-Cost Methodology

uncertainty, with outcome distributions largely unknown and in instances

in which little empirical data for comparable experience is available, is a

different matter. From hindsight, ATP and possibly others have more

empirical experience at this point to impose on analyses such as the tissue

engineering study; however, the uncertainties are likely so great for studies

performed very early in the technology life cycle that results of highly

prospective benefit-cost analysis will probably not be very useful for

program evaluation purposes. At best, predicted values should most likely

cover a very broad range. An accurate probability distribution is likely to

generate a low expected value if generated at an early stage. 

Modeling and data estimation of prospective project impacts, including

uncertainty, can be useful exercises in themselves; however, once such

impacts have been estimated, there is temptation to use them for program

evaluation purposes without adequate consideration of the original

purpose of the exercise or the credibility and/or likely accuracy of the

estimates. 

As a practical matter, evaluation often cannot wait for fully retrospective

results to be captured. Stakeholders need timely results of project and

program progress. For R&D projects with a long time horizon, the trail to

economic benefits often grows increasingly complex over time—researchers

move on, companies disappear, and ATP-funded technology may resurface

elsewhere. Program evaluators need to plunge in at some interim point.

Summary

In summary, highly prospective studies performed before technical risks

and uncertainties have been overcome and business risks mitigated may not

generate performance metrics that provide a credible or useful estimate of

program impacts, even if they meet high standards of economic modeling

and rigor. Results are likely not comparable to those obtained from

retrospective analysis or even prospective analysis performed after technical

success has been demonstrated but adoption is still in doubt. However,

some combination of prospective and retrospective analysis may be

reasonable if uncertainties have been mitigated through time,

accomplishment, and substantive interviews with participating companies,

their customers, and market experts. 
What Are the Issues?

The issues are: What is the project being analyzed? And what is the

counterfactual to the project that enables definition of project impact?

Both questions entail considerable definitional and measurement

challenges. 

For a benefit-cost study, the project analyzed spans the life cycle from

R&D through long-term outcomes and economic impact. Investment costs

and benefits included in the analysis should reflect the increment over, and

separable from, the counterfactual, or hypothetical situation that would

have existed without the project being analyzed. Typically, the counterfactual

situation is assumed to be the continuation of profits and products from a

defender technology. The counterfactual describes the “losses” in the

process of shifting to a newer technology from the older defender

technology. The analysis seeks to capture the benefits of the newer

technology less the losses from displacing the older one. In a dynamic

environment, the counterfactual may be evolving in parallel with the

technology being analyzed, and the defender technologies may be new

technologies competing with it. 

In practice, data to support analysis of actual outcomes relative to a

counterfactual, as a controlled experiment, probably do not exist. Analysts

use different approaches, or a mix of approaches, to define the project of

study, from investment costs to long-term outcomes and economic impacts,

and compare the project with a hypothetical alternative situation that

reflects the researcher’s best judgment as to what would have occurred in

the absence of the project of analysis. 

In S&T federal program evaluation, “the project” is the R&D that “the

program” funded. In practice, the economic outcomes, or “benefits,” often

Identifying the Counterfactual and Attribution of Benefits 21

V.

Identifying the Counterfactual 

and Attribution of Benefits
are the result of many investments of varying criticality. For example,

public and private benefits from the sale and use of a product embodying

specific new technology can derive from multiple public and private

investments. And benefits may evolve from a stream of product generations

that use increasing or decreasing amounts of the new technology resulting

from the project. 

The project could potentially be defined differently depending on whether

one seeks to emphasize the return on investment from the point of view of

a single government funding agency or the return on all investments that

lead to a particular stream of benefits. Obviously, when both social returns

(returns to all beneficiaries, including innovators, on all investments) and

public returns (in ATP studies, public return is computed as returns,

excluding those to innovators who received ATP funds, on ATP’s

investment) are presented together, the project analyzed for public return

and social return purposes must be the same project, and benefits need to

be traceable to the project. A mix is often needed: ATP-funded technology

is traced forward into the marketplace and products in the product

pipeline or in the marketplace are traced backward through the technology

development process to see whether ATP played a role. In instances in

which the stream of benefits clearly depended on ATP among other

external investment sources, some method of benefit allocation may be

needed. A social return analysis will need to include industry’s full costs for

R&D, including early-stage R&D and product development costs. 

Contrasting Examples

In practice, researchers first try to identify product outcomes with a direct

connection to ATP-funded technology. They then conduct interviews to

attempt to construct a counterfactual situation and elicit the ATP effect.

Some companies have been working in the ATP area before applying to

ATP. Some of them indicate that they would pursue similar goals without

ATP but at a slower pace and with a poorer chance of success. Others

indicate they could have done little or nothing without ATP and/or other

government funding. The latter may be true for large companies pursuing

new directions with ATP funding or for small start-up companies.

Cash flow-based benefit-cost studies have used a variety of mechanisms to

model economic effects of an ATP project relative to a hypothetical

situation in which ATP funding was absent. In general, this occurs in two

stages: The researcher identifies the economic benefits from products and

processes related to the ATP project and then, through a counterfactual

analysis, establishes the portion of these benefits that are attributable to

ATP. Table 4 summarizes many of the approaches used to define the

benefits attributable to ATP.

22 Toward a Standard Benefit-Cost Methodology
The tissue engineering study defined project benefits in regard to

acceleration of benefits and increased probability of technical success

relative to a hypothetical project without ATP. The return on the ATP

investment was the resulting incremental effect of acceleration of R&D and

greater technical success. Benefits of the general technical capabilities (in

regard to improved patient outcomes anticipated when the technology was

put to use) were estimated independently of these effects. However, the

amount of benefit attributed to ATP was the portion of benefits estimated

to be accelerated with ATP funding, the portion attributable to the bigger

research project enabled by ATP funding and collaboration opportunities,

or both.

The component-based software study took a very different approach. By

focusing on ATP projects that were already in commercialization,

researchers could identify market product outcomes that were related to

ATP funding. The study estimated all public and private benefits of these

products and compared them with all costs to bring the products to

market, from R&D through product development. The resulting metrics

showing public plus private return on combined sources of investment did

not attempt to quantify ATP’s direct contribution. Most of the projects

analyzed, however, were undertaken by tiny companies with little access to

Identifying the Counterfactual and Attribution of Benefits 23

Table 4. Identifying the Counterfactual and Attribution of Benefits—

Contrasting Studies

Study Approach

Tissue engineering ATP project benefits are defined in regard to acceleration of 

benefits and increased probability of technical success relative to a 

hypothetical project without ATP. Public plus private benefits are 

compared with public plus private investments to compute social 

return on social investment.

Component-based software Identified “project” that resulted in identifiable product sales and 

compared public/private benefits with all costs associated with that 

project to compute social returns

A-Si Detectors for digital Company interviews indicated “no project without ATP”; public 

mammography and composites benefits from product sales are compared with ATP investment to 

compute public return on ATP investment.

Photonics technologies 50% attribution to ATP where two different federal programs 

providing similar levels of funding were deemed jointly responsible 

for realization of any benefits. Public return on ATP investment is 

emphasized; social rates of return are also estimated.
24 Toward a Standard Benefit-Cost Methodology

private or other public financing. The products assessed were their first

and only products. ATP funding was obviously critical. The study

discussed the need for ATP funding to achieve these benefits and the effects

of ATP on acceleration as qualitative effects. 

The amorphous-silicon detectors for digital mammography study and the

composites cluster case studies all concluded from in-depth company

interviews that the research project funded by ATP would not have been

undertaken without ATP. These studies identified specific products or

product or cost improvements and related user benefits resulting from the

ATP project directly. Benefits to industry customers and end users of these

products or improvements were attributed fully to ATP based on a

counterfactual analysis in computing the return on ATP’s investment. For

example, for the amorphous-silicon detector project, the counterfactual

was the existing, higher-cost digital mammography system. Benefits

attributable to the ATP project were limited to those deriving from the

ATP-funded manufacturing process that reduced costs of production and

delivery of digital mammography systems and thereby enabled more

patients to benefit from digital mammography than occurred with the

earlier system. 

In the study of a cluster of photonics projects, it was determined that ATP

and NSF provided similar levels of funding to one of the projects.

Company officials indicated a strong belief that neither funding would

have occurred without the other and that the two agencies were jointly

responsible for realizing any economic benefits from the company’s first

products. The study attributed to ATP 50% of estimated benefits from

these impending products in computing the return on ATP’s investment. 

Note that computing social returns on investment (public plus private) on

combined sources of investment does not require as strong an assumption

about “causation” as does public return on ATP’s investment. The social

return set of metrics compares all benefits from a specific product/group

with all investments required to generate those benefits. The estimation

challenge is less one of ATP attribution than one of identifying

product(s)/economic outcomes to target in the analysis and obtaining all

relevant data for those products compared with the counterfactual

products. 

Obviously, when both social returns and public returns are presented

together, “the project” analyzed for public return and social return

purposes must be the same project, and benefits need to be traceable to

“the project.” If benefits to the project are traceable back to ATP among

other projects or sources of investment, for purposes of computing the

returns on the ATP investment, benefits may need to be allocated to
different funding sources based on acceleration or other effects due to ATP

funding or a pro rata share of investment costs provided by ATP. 

Summary of Counterfactual and Attribution 

of Benefits Issues

Studies use a variety of mechanisms to model incremental benefits of an

ATP-funded technology relative to a counterfactual situation in which ATP

funding did not exist. Acceleration of benefits, increased likelihood of

achieving benefits, and allocation in accordance with the amount or

importance of individual funding sources (given multiple sources) have all

been used. 

The attribution question increases in complexity over the technology life

cycle. Studies performed later in the commercialization phase encounter a

complex weave of threads from and to ATP. The complexity may increase

with the size of the company ATP funded (or its successor) if technology

products are numerous, or past R&D investments were numerous, or both.

The ATP lineage is clearer for most studies performed earlier; however,

those studies face much more uncertainty about ultimate product benefits

and unit sales. 

Isolating the benefits of the ATP project can be equally or more difficult in

retrospective analysis of fully commercialized technologies than analysis of

R&D outcomes at an earlier stage. Ultimately, ATP-funded technology may

be embodied in multiple products of a company, each of which resulted

from a number of different public and/or private funding sources and

different R&D projects. A mix of “tracing ATP-funded technology

forward” into the marketplace and tracing products in the product pipeline

or in the marketplace backward through the technology development

process to see whether ATP played a role is often needed to (1) define the

ATP project over its life cycle and (2) establish its impact relative to a

hypothetical situation in which ATP funding was not available. 

Identifying the Counterfactual and Attribution of Benefits 25

What Are the Issues?

Some studies and researchers emphasize return on ATP investment metrics;

others emphasize social return on investment metrics. The social return

metrics include both public and private benefits of a given project

compared with all investment costs incurred in the R&D and product

development phases. The public return on ATP investment metrics focus on

the public benefits attributable to ATP relative to ATP’s investment.

(Return on ATP investment metrics can be defined to include both public

and private benefits; however, most of the studies that compute these

measures have focused on public returns, in keeping with ATP’s mission.)

The social return metrics and public return on ATP investment metrics

measure different things, and there is no consistent relationship between

them.

Social Return on Investment and/or Public Return on Public (ATP) Investment? 27

VI.

Which Metrics: Social Return on

Investment and/or Public Return

on Public (ATP) Investment?

Return on Investment Measures

• Public return on ATP investment: Return to nation attributable to

ATP (defined in most ATP studies to exclude benefits to ATP project

participants) on ATP investment 

• Social return: Return to ATP project participants and nation on total

investment from all sources

• Private return: Return to ATP project participants on total

investments from all sources
Contrasting Examples

Tables 5 and 6 illustrate the variation across just the more recent studies.

The metrics for each of the eight component-based software case studies

are presented in Table 5. The metrics for the comparison studies are shown

in Table 6. The component-based software study and digital video study

computed social return metrics. The amorphous silicon detector for digital

mammography study and the composites and photonics cluster study cases,

performed by a different contract researcher, computed public return on

ATP investment metrics. In addition to public return on ATP investment

metrics, the photonics case studies estimated social rates of return, but not

the other social return metrics. 

Both sets of metrics have utility and merit. Social return metrics are rooted

in the literature on the economics of innovation. Public return on program

investment metrics have stronger roots in public finance and program

evaluation. Tassey (2003, p. 19) suggests that “when the objective is to

estimate the impact of the government role, two calculations should be

made; one for the total impact (i.e., social return) and one for the

contribution of the government subsidy (i.e., return on government

investment).”

Work stemming from Edwin Mansfield in the late 1970s consistently finds

evidence that social rates of return on R&D investments exceed private

rates of return and private hurdle rates. The “spillover gap,” or difference

28 Toward a Standard Benefit-Cost Methodology

Table 5. Which Metrics? Contrasting Projects in 

Component-Based Software 

Component-based

Social Return Metrics Public Return Metrics

Software NPV ($M) B:C IRR (%) NPV ($M) B:C IRR (%)

Aesthetic Solutions –1.2 .37 NC

Commerce One JV 789 39 363

Extempo –1.22 .63 NC

Intermetrics 29.6 9.6 103 Not computed for this study

Real-Time Innovations 2.06 1.8 31

Sci Comp 21 7.6 51

Tom Sawyer 52 18 136

Xerox PARC 1.2 1.2 13

Note: IRR = internal rate of return; NC = not computed; NPV = net present value.
between the social and private rates of return, is indicative of a market

failure in funding R&D. This body of work underlies the case for

government support of R&D and national innovation policy. R&D

projects that are cost shared between public and private sources, such as

ATP projects, further imply that a relevant benefit-cost comparison

involves both public and private benefits and investments. 

The component-based software and digital video studies illustrate the

difference between what is being measured in the social return metrics and

what is measured in public return metrics and the variation in the effect of

those differences. Although neither study presents separate public and

private return metrics, the component-based software study provides

measures of producer surplus (benefits captured by innovator firms) and

consumer surplus (benefits captured by customers), which can be

considered proxies for private benefits and public benefits, respectively. For

the component-based software study, across all eight project cases, the

consumer surplus was more than twice the producer surplus; however,

there was a huge variation among projects. Surprisingly, the estimated

producer surplus exceeded the consumer surplus for many of the smallcompany projects; however, the consumer surplus vastly exceeded the

producer surplus for the successful Commerce One Joint Venture and for

the Xerox PARC project. In the digital video study, the private benefits to

U.S. companies were estimated to be very, very small compared with the

Social Return on Investment and/or Public Return on Public (ATP) Investment? 29

Table 6. Which Metrics? Other Contrasting Studies 

Other Contrasting

Social Return Metrics Public Return Metrics

Studies NPV ($M) B:C IRR (%) NPV ($M) B:C IRR (%)

Digital video

(midpoint) 165.9 4.24 28.6 Not computed for this study

A-Si Detector

(base case) NC NC NC 219 125 69

Composites

(base cases)

Applied Sciences NC NC NC 552 221 57

Lincoln Comp. NC NC NC 510 187 58

Photonics

(base cases)

X-Ray Optics NC NC 43 184 75 49

Ion Optics NC NC 51 143 174 75

Note: IRR = internal rate of return; NC = not computed; NPV = net present value.
public benefits. A public return on ATP investment analysis alone would

fail to capture this variation among the separate private benefit and public

benefit components of social benefits. And a public return on ATP

investment analysis would further require a more in-depth assessment of

the degree to which the consumer surplus, or public benefits, are

attributable to ATP than was provided in the social return analysis. (See

the prior section, “Identifying the Counterfactual and Attribution of

Benefits,” for further discussion of this issue.)

Of course, the cost side, as well as the benefits side of the analysis, is

different for the social return metrics compared with the public return on

ATP investment metrics. Full social costs span the earlier stages of R&D,

such as ATP funds, product development, and potentially production scaleup. Of course, this life cycle of costs affects computed rates of return. In

addition, it greatly complicates the computation. The analysis requires

considering a vast amount of historical data. The historical stream of

investment cost data is difficult to come by. Private profit data, on a

product basis, for estimating the private portion of benefits may not exist

or may be highly confidential. 

Researchers work hard to build a positive relationship with company

representatives, including CEOs and presidents of small companies. In

response, companies have been cooperative with these ATP studies, even

though they generally require a full day of on-site interviews plus a number

of phone calls and/or preparation of responses to written questions.

However, the atmosphere changes when researchers probe explicitly for

profit and cost of sales data. Across the board, researchers have settled for

less in this area in return for forthcoming responses in other areas. 

The social return metrics that include investments from multiple sources of

funding of R&D may tend to foster a broader definition of the R&D

project being studied than is most useful for program evaluation. One

source of funding can overlap another one, and one product generation or

R&D project outcome may overlap another. As a result, a narrow R&D

project nucleated with specific federal funding can become buried among a

chain of related R&D projects, with outcomes partially in common. If the

project is interpreted broadly, the federal program’s effect on outcomes—

that is, its additionality effect—becomes invisible in the resulting metric.

Unless the project evaluated is traceable directly and more or less entirely

to a single program that enabled the project, the social return metrics

provide a general picture of the return on investment from multiple sources

but may say little if anything about programmatic impact from a program

evaluation perspective. 

30 Toward a Standard Benefit-Cost Methodology
This potential muddiness concerning project definition with the social

return metrics does not appear to have been a major problem with ATP’s

component-based software and digital video studies. The two large joint

ventures included in these studies—the Sarnoff-led joint venture covered by

the digital video study and the Commerce One Joint Venture in the

component-based software study—were uniquely ATP. It is unlikely that

the resulting products would have come to be in a similar time frame

without ATP. Five of the seven remaining projects covered in the

component-based software study were start-up or otherwise very small

companies that had received little funding besides ATP and whose only

products were closely derived from their ATP project. Although several of

these analyses included investment costs considerably beyond ATP’s

funding and company cost share on the ATP project, most of the

additional funding directly supported commercialization of the ATP-funded

technology. Therefore, the resulting social return metrics can be interpreted

as “ATP project impact.” The social return metric for the photonics project

in which ATP and NSF provided similar levels of funding in a similar

period of time would need to be interpreted more broadly.

Summary of Return on Investment Measures

The best strategy would appear to be to compute both social return metrics

and metrics that show the impact of the specific federal program. 

Metrics that focus specifically on the incremental effect of the specific

federal program (for ATP, the public return on ATP’s investment) have

obvious utility for program evaluation. They also involve substantially less

data. The challenge is to clearly define the funded project’s outcomes

compared with a hypothetical counterfactual in a way that pinpoints

differences in outcomes attributable to the federal program being

evaluated. (See the prior section, “Identifying the Counterfactual and

Attribution of Benefits.”)

Social return metrics capture the return to all investors, which is

particularly important for projects that are cost shared with industry.

Industry proposes projects for federal cost share from which it anticipates

a future return on its investment in the form of profits from resulting

products and processes. Industry’s return is of concern to the federal

program because it provides the incentive to propose. Differences in the

private component and public component of the social return further assist

in assessing the potential market failure that justifies federal funding. 

Social Return on Investment and/or Public Return on Public (ATP) Investment? 31
Given the confidentiality of some of the data needed for computing social

return metrics, public industry data may need to supplement company

data. Relatively simple estimation techniques for social rates of return that

draw on public data may be more practical than comprehensive analysis of

actual private investment and profit data. Such simplified techniques may

actually lead to more consistent methodologies and more comparable

social return metrics across projects. However, little work has been done to

develop practical social return methodologies, apply them illustratively,

and test them for reasonable accuracy. 

32 Toward a Standard Benefit-Cost Methodology
What Are the Issues?

Individual studies follow established good practice in adjusting for the

varying value of the dollar in regard to both purchasing power and

opportunity cost during the span of years covered by the study; however,

projects studied start and end at different times. Studies are performed at

different times and span different time periods. They do not reflect

differences in purchasing power and opportunity costs across the series of

studies. 

Constant Dollars

Studies following the practice prescribed by OMB Circular A94 of

estimating all cash flows in constant dollars use constant dollars of different

years. For ease in estimating, benefits and costs are often estimated at their

value in the year the study is conducted; for example, if the study is done

in 1995, it is likely that all cash flows will be estimated in dollars of 1995

value. Some researchers follow other appropriate practices; for example,

some express results in constant dollars of more than one year, such as the

year the study is conducted and the year the project started. 

Net present value study results expressed in 2005 dollars do not have the

same purchasing power and thus are not equivalent or additive to net

present value study results expressed in 2004 dollars or any other year’s

dollar purchasing power. They can, however, be adjusted to purchasing

power equivalents with a straightforward procedure. 

Discount Rate

When using constant dollars, the fundamental principle of cash flow–based

benefit-cost analysis is that all costs and benefits be adjusted for the

opportunity cost of capital over time before being aggregated across time:

a dollar invested or earned sooner is worth more than a dollar invested or

Adjusting for Timing Differences across Studies and Projects 33

VII.

Adjusting for Timing Differences

across Studies and Projects:

Constant Dollars and Base Year

for Discounting
earned later in time because of the potential for earnings at some rate

during the intervening period. 

Cash flows that occur at different times during the history of a project are

“discounted” to a common point in time to adjust for this difference in

earnings opportunity. OMB Circular A-94 simplified the rate issue by

stipulating that a rate of 7% (real earnings rate, above and beyond

inflation) be assumed in adjusting benefits and costs of most federal

programs and projects to a common point in time. ATP has used the 7%

real rate for all of its cash flow-based benefit-cost studies listed in Table 1

except the tissue engineering study. 

Base Year for Discounting

The most common practice, and that used by financial spreadsheet

software such as Excel, is to discount cash flows incurred in a project back

to “Time Zero,” or one time period ahead of the first cash flow. For

example, if the first cash flow—typically the first investment—occurred in

1996, and other cash flows are entered as annual amounts, Excel discounts

all cash flows back to one year before that, or 1995, in computing the net

present value. The discounting process typically assumes that cash flows

occur at the end of the year. The beginning of the project is the beginning

of the first year, which is equivalent to the end of the previous year (Zero).

This practice of using Time Zero as the base year for discounting evolved

from the use of discounted cash-flow analysis in evaluating the outcomes

of alternative prospective investments under consideration that year. The

study was being conducted at approximately Time Zero, dollar estimates

were expressed in dollar values in Time Zero, and all future cash flows

were discounted back to Time Zero. In this section, we interpret the start

of the project as Time Zero. 

Magnitudes of net present value metrics computed using the same

discounting base year (and in the same constant dollars and with the same

discount rate) can be compared with each other. Selecting the project with

the highest net present value would generate the maximum total net benefit

from a single project. If multiple projects are feasible and independent of

each other, their net present value metrics can be aggregated. 

For program evaluation purposes, both prospective and retrospective

analyses are done at different times in the program’s history. Although,

typically, studies of an individual R&D project use one time period ahead

of the start of the first cash flows (i.e., Time Zero) as the base year for

discounting (as Excel does), the projects analyzed in different studies most

likely started in different years. The interpretation of net present value

34 Toward a Standard Benefit-Cost Methodology
results of different studies looked at collectively becomes somewhat

problematic. However, various alternatives for adjusting net present value

results to a common base year have interpretation problems as well.

Should project metrics be adjusted to look at the projects from the

perspective of a common point in time, for example, from the time of the

comparative or aggregate study, or the start of ATP, or the start of the first

project among a group? Or for project evaluation purposes, is it adequate

to let the base year vary as long as all studies report metrics in the same

constant dollar basis and the same discount rate? To build awareness, this

report attempts to illuminate those questions. Answers will depend on

specific program needs to compare or aggregate net present value results

from different studies. 

It is important to note that although the net present value metric will

change with a change in base year and/or constant dollar year, the internal

rate of return and benefit-to-cost ratio will not change. They remain

constant regardless of the choice of base year or constant dollar year.

Furthermore, they are better measures than the net present value for

assessing the efficiency of a project in regard to output per dollar of input

unless investment costs are the same for all projects. However, they cannot

be used to assess the total dollar value of benefits generated by a single

project or project aggregate. The net present value metric provides that

information, and the net present value metric requires adjustment for

different base years and different constant dollar years when results are

aggregated. 

Examples of Effects of Constant Dollar 

and Base Year Differences 

A comparison of reported net present values with net present values

adjusted to a common constant dollar year and base year provides

examples of the effects of these differences. Because one purpose of looking

at the results of multiple studies would be to identify the relatively more

and less successful projects among a group of basically successful ones, we

also look to see whether the order of projects changes from highest to

lowest net present value.

Tables 7 and 8 illustrate the effects of constant dollar and base-year

differences across a number of ATP’s more recent studies. Reported net

present values are compared with net present values adjusted to constant

2005 dollars and then further adjusted to a common base year of 2005.

Adjusting for Timing Differences across Studies and Projects 35
36 Toward a Standard Benefit-Cost Methodology

Table 7. Adjusting for Timing Differences across Studies and Projects:

Constant $ and Base Year (Example 1: Ranking Does Not Change)

Component- Adj. NPV—

Based Base Year Adj. NPV— 2005 as 

Software Reported Constant for Constant Base Year 

8 Cases NPV ($M) Rank $ Year Discounting 2005$ ($M) ($M) Rank

Aesthetic Solutions –1.2 7 2000 1994 –1.3 –2.8 7

Commerce One JV 789 1 2000 1996 880 1,619 1

Extempo –1.22 8 2000 1994 1.36 –2.86 8

Intermetrics 29.6 3 2000 1997 33.0 56.7 3

Real-Time Innovations 2.06 5 2000 1995 2.30 4.5 5

SciComp 21 4 2000 1994 23 49.3 4

Tom Sawyer 52 2 2000 1995 58 114 2

Xerox PARC 1.2 6 2000 1994 1.3 2.8 6

Table 8. Adjusting for Timing Differences across Studies and Projects:

Constant $ and Base Year (Example 2: Ranking Changes)

Adj. NPV—

Other Cash- Base Year Adj. NPV— 2005 as 

Flow-Based Reported Constant for Constant Base Year 

Cases NPV ($M) Rank $ Year Discounting 2005$ ($M) ($M) Rank

Digital Video 165.9 5 2002 1994 177.7 374 5

A-Si Detector 219 3 2002 1997 235 403 4

Composites:

Applied Sciences 552 1 2003 1996 579 1,065 1

Lincoln Composites 510 2 2003 1997 535 920 2

Photonics:

X-Ray Optics 184 4 2004 1993 188 424 3

Ion Optics 143 6 2004 1999 146 219 6
Adjusting for Timing Differences across Studies and Projects 37

Table 7 examines the eight case study net present values reported in the

component-based software study. All of these studies used a constant

dollar year of 2000, but base years for discounting varied from 1994 to

1997. These variations were not enough to cause the relative order of the

net present values across projects to change with the adjustment to a

constant base year and constant dollar year. 

Table 8 examines six cases that used a narrow spectrum of constant dollar

years because the studies were all done fairly recently; however, the

spectrum of base years used in discounting was broader, ranging from

1993 to 1999. The constant dollar adjustment alone was not enough to

change the order of net present values. However, the additional effect of

the base year adjustment was enough to cause the order of net present

values across the six cases to change along with the significant change in

magnitudes. The X-Ray Optics project moved from fourth position to third

position, and the A-Si Detector project moved from third to fourth. The

effects were greatest for projects in which the difference between the

common base year and original base year for discounting were greatest;

however, the revised ranking will now hold whatever common base year is

used. 

Figure 1 shows the formula for adjusting a reported net present value to

constant 2005 dollars. The constant dollar adjustments were based on

historical implicit price deflators for GDP reported for Q2 2005 in the

September 2005 issue of Survey of Current Business (BEA 2005) (Tables

C.1 and 1.1.9). Figure 2 shows the future value formula for adjusting

reported net present values to a common 2005 base year. 

Figure 1. Adjusting to Constant 2005 Dollars 

NPV in constant 2005 $ = 

1 +

Q2 2005 IPDGDP – IPDGDPstudy

× Reported NPV

IPDGDPstudy

Where: IPDGDP is Implicit Price Deflator for GDP for Q2 2005 and 

IPDGDPstudy is IPDGDP for constant $ year used in study

Source: BEA 2005, Tables C.1 and 1.1.9
Illustrating the adjustments shown in figures 1 and 2 for the Digital Video

case results shown in Table 8:

• Q2 2005 IPDGDP (Implicit Price Deflator for second quarter of 2005

published in September 2005 Survey of Current Business [BEA 2005],

Table C.1) = 111.584

• IPDGDPstudy (Implicit Price Deflator for constant dollar year used 

in study published in September 2005 Survey of Current Business 

[BEA 2005], Table 1.1.9) = 104.187

• Applying the formula in Figure 1 to the reported NPV:

The adjusted NPV in constant 2005 dollars = 

• Applying the formula in figure 2 to adjust this NPV in constant 

2005 dollars to 2005 as the base year for discounting = 

$177.7 million × 1.07(2005 – 1994=11) = $374 million.

Separate analyses, not shown, verified both empirically and mathematically

that once all case study net present values are adjusted to a common base

year, the base year can be changed again without further changing the

order of the values. Magnitudes change, of course, as adjustments for

opportunity costs move the base year of analysis backward or forward

over time; however, the relative ordering does not change once net present

values for all projects have been adjusted to a common base year. 

There is little debate about the need, in theory, for constant dollar

adjustments in comparing or aggregating net present value results from

studies based on different constant dollar years. However, this adjustment

might be considered relatively small and insignificant compared with other

sources of measurement error in these studies. 

38 Toward a Standard Benefit-Cost Methodology

Figure 2. Adjusting Base Year to 2005 

Adjusted NPV for Base Year 2005 = NPV in constant 2005 $ × 1.07n

Where: n = 2005 minus base year used in study

1+





 



 × (111.584 –104.187)

104.187 $165.9 million=$177.7 million
The issue of the base year for discounting is more complex and

controversial. The assumptions behind the discounting process are

confusing to evaluators and sometimes even to others who use return on

investment metrics tools. A decision on what practice to follow requires

discussion and an understanding of the meaning of the discounting process

for individual projects and for an aggregate of projects.

Fundamentally, the purpose of the discounting process is to force

comparison of the project(s) under study to a hypothetical alternative

investment opportunity with a guaranteed 7% real rate of return. For a

single project, the net present value represents the magnitude of the

difference in outcome (excess over or under), in constant dollars, for the

project compared with a 7% federal bond taken out at the start of the

project and cashed in at the end of the project study period. When multiple

projects that started at different times are being analyzed together, the

comparison with the alternative investment with a 7% return becomes

complicated. The common base year adjustment is a mechanism for

appearing to simplify the issue but it creates other interpretation issues. 

The common base year and common constant dollar adjustments force a

consistent base year and dollar reference for a group of projects examined

collectively and compared with a 7% alternative project. However, these

adjustments imply assumptions about earnings and costs of capital for

individual projects prior to or beyond their individual study periods. For

example, consider two projects that started in 1996 and 1998 and whose

study periods ran until 2006 and 2008, respectively. To analyze the two

projects together in the same study, a common practice would be to use a

1996 base year and to discount all benefits and investments from both

projects to the beginning of the earliest project. This implies that a $1,000

investment at the start of the second project in 1998 actually cost less than

$1,000. It cost only $873 because of an assumed earnings potential at 7%

during the two-year period before the investment occurred. (Benefits

likewise would be discounted back to 1996 rather than 1998 and would

thus be valued at fewer dollars.) 

An alternative approach would be to use 2008 as the common year for

discounting. With this approach, the assumption would be that all benefits

(earnings) from all projects are reinvested until 2008 at a 7% rate. Use of

an interim base year, for example, 2005 or the year the study is

undertaken, assumes reinvestment of benefits at 7% per year through that

year. Tables 7 and 8 use this last approach of adjusting reported net

present values to the year the study was undertaken, an interim point

among project cash flows. Initially, each project’s cash flows had been

discounted to their present values as of the start of the project and

reported on that basis. For this study, reported present values were first

Adjusting for Timing Differences across Studies and Projects 39
adjusted to constant 2005 dollars and then further adjusted to a common

“base” year 2005 by applying future value factors for 7% and the number

of years from the start of the project through 2005. Assuming cash flows

are stated in 2005 constant dollars, this process is equivalent to the

alternative approach of adjusting individual cash flows from each project

to 2005 by discounting post-2005 cash flows to 2005 and computing

future values of pre-2005 cash flows and adding them all together with

2005 cash flows. The approach used in this study in Tables 7 and 8 is

more straightforward and can be implemented directly from net present

values computed by Excel and those typically reported. 

The following are a few considerations in the decision as to whether to

adjust for differences in discounting base years for purposes of comparing

or aggregating results. 

• Projects are funded from different annual budgets and budget decision

environments, which argues for the “floating” base year across studies as

long as the base year is consistently the start of the projects evaluated.

The benefit-cost calculation is then aligned with the budgetary economic

events. Aggregation of results of studies with different base years poses

some interpretation questions; however, this approach requires no

assumption about alternative investment opportunities before and after

the time of actual cash flows from the projects analyzed.

• Using “the time of the study valuation” for handling both constant

dollars (purchasing power) and discounted value (opportunity cost of

capital) has the advantage of ease of interpretation, particularly for

retrospective analysis that involves aggregation across studies for

program evaluation purposes. The reader understands the value of a

dollar at the current time, which often approximates the time of the

study. Using the “time of the study” as the base year assumes that all

cash-flow benefits spun out of a project or put into a project are invested

at the discount rate from the time they occur until the time of the study.

• Alternatively, using the start of ATP (in 1990) as the base year, or the

start of the first project in an aggregate group being studied, has been

considered meaningful. ATP’s photonics and composites cluster studies

and the component-based software study used this latter approach for

analyzing a group of projects. Cash flows of all projects studied are

discounted back to the start of the earliest project. As a result, in the

aggregate analysis, benefits from later projects are downsized compared

with earlier benefits from earlier projects. For example, consider the

photonics cluster study, in which the X-Ray Optics individual case study

analysis used a 1993 base year for discounting (first cash flows in 1994),

the Ion Optics individual case study used 1999 as the base year for

40 Toward a Standard Benefit-Cost Methodology
discounting (first cash flows in 2000), and the aggregate cluster analysis

used 1993 as the base year. The individual Ion Optics case study showed

the project to have a net present value of $143 million; however, within

the aggregate cluster analysis, the Ion Optics project contributed only

$95 million as a result of discounting benefits and costs back to 1993 as

the base year rather than to 1999.

• The adjustment to restate project outcomes relative to a common

vantage point base year, such as “time of the study” or “start of ATP,”

or any other point in time, can be done from previously reported

metrics. The process then can be repeated over time for different base

years and different constant dollar years.

Again, results of using a different base year have different magnitudes and

different interpretations. Using the project start as the base year means that

all later positive and negative cash flows are adjusted (discounted)

downward at 7% compounded annual rate to compute the net financial

advantage relative to the alternative safe investment made at the project

start and held until the end of the project. By using some later year as the

base year, for example, the year the study is being undertaken, all cash

flows are adjusted upward for the equivalent of 7% annual compounded

earnings on those cash flows (or lost earnings in the case of negative cash

flows) until that ultimate base year. When multiple projects are examined

together and cash flows are adjusted to a common “project start” or other

common base year of analysis, the 7% compounding or discounting effect

is extended beyond the project periods of at least some projects in the

aggregate.

Using the same constant dollar year and base year for discounting, as was

done in Tables 7 and 8, has the intuitive appeal that resulting dollar

amount metrics can be interpreted more easily. If they are the same as the

year the study is performed, dollar results have even more intuitive

meaning in that they can be compared with what that amount of money

will buy at that time. This concept parallels the original concept of using

the project start as the base year for evaluating prospective investment

alternatives under consideration that year. 

Summary of Analysis of Net Present Value Adjustments

Evaluators and other users of the net present value (or net benefits) metrics

resulting from cash flow-based benefit-cost studies need to recognize the

effects of different constant dollar years and different base years on proper

interpretation and use of these results. For purposes of presenting results of

a single project case study, the traditional approach that uses the beginning

of the project funding year as the base year may be adequate as long as all

Adjusting for Timing Differences across Studies and Projects 41
use the same constant dollar year and thus net present values reflect

purchasing power equivalents. However, when results of different studies

are presented collectively, the interpretation is fuzzy. 

In instances in which multiple projects are analyzed in a single study or in

which results of multiple studies are considered together to compose a

minimum estimate of portfolio performance, adjustment to constant

dollars and a common base year for all projects will most likely be

necessary. 

Adjustment to a common base year and constant dollar year after the

study of an individual project is completed is relatively straightforward and

may be a better solution than trying to impose a standard “fixed” base

year and constant dollar year for all studies. Many practitioners are

committed to the traditional concepts of base year as the “start of the

project.” The traditional approach provides a consistent starting point for

observing the traditional metrics before considering what adjustments

might be needed for comparability or aggregation with other studies.

Adjustments, particularly of the base year used in computing net present

value, will require programmatic consideration of what basis makes most

sense to the evaluation objective. Adjusting all study results to 1990, for

example, would evaluate net benefits whenever they occurred back to the

start of the ATP as if the pool of funds for them existed then and could

have been invested in the interim. It does not recognize that most of the

projects studied were funded out of later budgets. The cluster and portfolio

analyses ATP has performed use the year the first project started as the

base year. All other cash flows were discounted back to that year. This

process is equivalent to computing the net present value for each project

separately using that project’s start as the base year and then discounting

the resulting set of net present values back to the start of the first project.

Adjusting all net present value results forward to the time a collective

assessment is being performed, as is done in Tables 7 and 8, reflects the

assumption that the net present value reported as of the base year of each

study is reinvested at the 7% discount rate until the time of the study.

Equivalently, the assumption is that net cash flows generated in any year

from any project are reinvested at 7% until the year the study is

performed. That assumption may be a more useful than assuming that all

funds for all projects were available at the beginning of ATP or the start of

the first of several projects studied. 

42 Toward a Standard Benefit-Cost Methodology
ATP’s efforts during the past two years to document and investigate the

significance that methodological differences have across its benefit-cost

studies have demonstrated both the utility and commonality of the studies,

but also the complexities beyond pure estimation difficulties. The

challenges do not appear insurmountable. As for other program evaluation

tools, communication and training requirements are significant. However,

the basic underpinnings of cash-flow benefit-cost techniques are well

established in accounting, finance, and business, and they are adapted to

capture societal benefits in public finance and economics literature. 

Practitioners have created a body of work that is consistent with good

practice as spelled out in the business, public finance, and economics

literature. The challenges are in adapting “one-at-a-time case studies” or

studies of small groups of projects to a more comprehensive portfolio

analysis. The cost and time of doing high-quality studies precludes doing

them for all projects funded by a program, never mind doing them in a

similar time frame; however, program evaluators and users of study results

will need to consider results of different studies at a given time to be able

to aggregate and/or compare them. Analyses of key differences across

ATP’s considerable body of such studies to date have helped us to assess

the significance of the differences and to pose solutions. 

In general, the programs being evaluated, such as ATP, will not be

undertaking such studies themselves but rather will be contracting them to

consultants with experience in this type of work. It will be necessary for

the program sponsors of this work to communicate the issues to their hired

consultants. ATP’s experience indicates that the number of consultants with

substantial experience in benefit-cost analysis of publicly funded R&D is

relatively small; furthermore, even the experienced consultants in this area

focus on the study at hand, not how that study will be aggregated with

Conclusions and Future Work 43

VIII.

Conclusions and Future Work
studies done by other researchers or with their own earlier studies.

Different experts have somewhat different opinions about solutions to

problem areas in which multiple approaches have merit, but inconsistency

in approach impedes the interpretation and use of results. 

The immediate goal is to circulate this report to users of benefit-cost results

and to benefit-cost analysis practitioners in NIST as part of the formal

review process and subsequently to ATP contractors and others. Feedback

is sought and anticipated with the hope that a living, working document

will support a more formal, uniform, standard-setting process for benefitcost analysis in the future. 

44 Toward a Standard Benefit-Cost Methodology
Advanced Technology Program (ATP). 1996. Meeting with Edwin

Mansfield at University of Pennsylvania in June 1996. Jeanne Powell

and Dr. Stephen Weber (Building and Fire Research Laboratory,

National Institute of Standards and Technology) met with Dr.

Mansfield to discuss specific research issues concerning application of

Dr. Mansfield’s recent report for ATP, Estimating social and private

returns from innovations based on the Advanced Technology

Program: Problems and opportunities. 

American Society for Testing and Materials (ASTM) International

(Subcommittee E06.81 Chair Harold E. Marshall, Building and Fire

Research Laboratory, National Institute of Standards and

Technology). 2004. ASTM Standards on Building Economics, 5th ed.

West Conshohocken, PA: ASTM International.

Bernstein, Jeffrey I., and M. Ishaq Nadiri. 1988. Interindustry R&D

spillovers, rates of return, and production in high-tech industries.

AEA Papers and Proceedings 78 (May): 433.

Bureau of Economic Analysis (BEA). 2005. Survey of Current Business 85

(September 2005): D4 (Table 1.1.9 Implicit price deflators to Gross

Domestic Product), D48 (Table C.1 GDP and other major NIPA

Aggregates). http://www.bea.gov/bea/pub/0905cont.htm. 

GPRA. 1993. Government Performance and Results Act of 1993. Office of

Management and Budget, Executive Office of the President.

http://www.whitehouse.gov/omb/mgmt-gpra/gplaw2m.html

Jaffe, Adam B. 1996. Economic analysis of research spillovers: Implications

for the Advanced Technology Program. NIST GCR 97-708.

Gaithersburg, MD: National Institute of Standards and Technology.

References 45

References
———. 1998. The importance of “spillovers” in the policy mission of the

Advanced Technology Program. Journal of Technology Transfer 23

(summer 1998): 11–19.

Mansfield, Edwin. 1995. Innovation, technology and the economy: The

selected essays of Edwin Mansfield, Vol. I. Brookfield, United

Kingdom: Edward Elgar.

———. 1996. Estimating social and private returns from innovations

based on the Advanced Technology Program: Problems and

opportunities. NIST GCR 99-780. Gaithersburg, MD: National

Institute of Standards and Technology.

Mansfield, Edwin, John Rapoport, Anthony Romeo, Samuel Wagner, and

George Beardsley. 1977. Social and private rates of return from

industrial innovations. Quarterly Journal of Economics 91 (May

1977): 221–40. 

Nadiri, M. Ishaq. 1993. The contribution of research and development and

economic growth. NBER working paper. Cambridge, Massachusetts:

National Bureau of Economic Research.

Office of Management and Budget (OMB). 2004. Office of Management

and Budget Examiner Training on Program Evaluation by the Federal

Evaluators Group, October 6.

———. 1992. Office of Management and Budget Circular A-94.

Guidelines and discount rates for benefit-cost analysis of federal

programs. http://www.whitehouse.gov/omb/circulars/a094/a094.pdf.

Scherer, F. M. 2005. Edwin Mansfield: An appreciation. Journal of

Technology Transfer 30 (January 2005): 3–9.

Tassey, Gregory. 2003. Methods for assessing the economic impacts of

government R&D. Planning report 03-1, p. 19. Gaithersburg, MD:

National Institute of Standards and Technology.

http://www.nist.gov/public_affairs/budget.htm.

Teece, David J. 2005. Technology and technology transfer: Mansfieldian

inspirations and subsequent developments. Journal of Technology

Transfer 30 (January 2005): 17–33.

46 Toward a Standard Benefit-Cost Methodology
ABOUT THE ADVANCED TECHNOLOGY PROGRAM

The Advanced Technology Program (ATP) is a partnership between government and private industry

to conduct high-risk research to develop enabling technologies that promise significant commercial

payoffs and widespread benefits for the economy. ATP provides a mechanism for industry to extend

its technological reach and push the envelope beyond what it otherwise would attempt. 

Promising future technologies are the domain of ATP:

• Enabling or platform technologies essential to development of future new products, processes, or

services across diverse application areas

• Technologies where challenging technical issues stand in the way of success

• Technologies that involve complex “systems” problems requiring a collaborative effort by

multiple organizations

• Technologies that will remain undeveloped, or proceed too slowly to be competitive in global

markets, in the absence of ATP support

ATP funds technical research, but does not fund product development—that is the responsibility of

the company participants. ATP is industry driven, and is grounded in real-world needs. Company

participants conceive, propose, co-fund, and execute all of the projects cost-shared by ATP. Most

projects also include participation by universities and other nonprofit organizations. 

Each project has specific goals, funding allocations, and completion dates established at the outset.

All projects are selected in rigorous competitions that use peer review to identify those that score

highest on technical and economic criteria. Single-company projects can have duration up to three

years; joint venture projects involving two or more companies can have duration up to five years.

Small firms on single-company projects cover at least all indirect costs associated with the project.

Large firms on single-company projects cover at least 60 percent of total project costs. Participants

in joint venture projects cover at least half of total project costs. Companies of all sizes participate

in ATP-funded projects. To date, nearly two out of three ATP project awards have gone to

individual small businesses or to joint ventures led by a small business.

Contact ATP for more information:

• On the Internet: www.atp.nist.gov

• By e-mail: atp@nist.gov

• By phone: 1-800-ATP-FUND (1-800-287-3863)

• By writing: Advanced Technology Program, National Institute of Standards and Technology, 

100 Bureau Drive, Stop 4701, Gaithersburg, MD 20899-4701

ABOUT THE AUTHOR

Jeanne Powell is Senior Economist in the Economic Assessment Office, ATP, where she manages

contractor evaluation studies, develops methodologies for applying benefit-cost analysis to ATPfunded industry R&D projects, and advises on performance measurement issues. She developed

ATP’s Business Reporting System and has published a number of reports of ATP progress, including

Different Timelines for Different Technologies (2002), Development, Commercialization, and

Diffusion of Enabling Technologies: Progress Report (2000), Business Planning and Progress of

Small Firms Engaged in Technology Development through the Advanced Technology Program

(1999), and Development, Commercialization, and Diffusion of Enabling Technologies: Progress

Report for Projects Funded 1993–1995 (1997). Her previous experience includes serving small

businesses as a private Certified Public Accountant (CPA), performing corporate audits for regional

CPA firms, performing several stints as a federal government economist, and Professor of

Economics, Business Statistics, and Accounting with Montgomery College.