Guide for Conducting Benefit-Cost 

Evaluation of Realized Impacts of 

Public R&D Programs 

Revised working draft, August 2011 

Prepared by: 

Rosalie Ruegg, TIA Consulting, Inc. and Gretchen B. Jordan, Sandia 

National Laboratories 

Prepared for: 

U.S. Department of Energy, Office of Energy Efficiency and Renewable 

Energy 

Guide for Conducting Benefit-Cost Evaluation of Realized 

Impacts of Public R&D Programs 

Authors and Other Contributors 

Co-authors: 

Rosalie Ruegg 

Managing Director 

TIA Consulting, Inc. 

Gretchen B. Jordan 

Project Technical Manager 

Principal Member of Technical Staff 

Sandia National Laboratories 

DOE Direction, Oversight, and Review: 

Jeff Dowd 

DOE Project Manager 

Office of Energy Efficiency and Renewable Energy 

Reviewers and Contributors: 

Members of the Expert Review Panel (first edition draft Guide): 

 Irwin Feller, Panel Chair — Director, Institute for Policy Research and 

Evaluation, and Professor Emeritus of Economics, Pennsylvania State University 

 Elena Besedin — Deputy Project Director, Environment & Resources Division, 

Abt Associates, Inc. 

 Diana Hicks — Professor and Chair of the School of Public Policy, Georgia 

Institute of Technology; previously the Senior Policy Analyst at CHI Research, 

Inc., and a faculty member of SPRU, University of Sussex in the United 

Kingdom. 

 Jon P. Nelson — Economics Consultant, and Professor Emeritus of Economics, 

Pennsylvania State University 

 Richard G. Newell —Gendell Associate Professor of Energy and Environmental 

Economics at the Nicholas School of the Environment, Duke University. 
 Jeanne Powell — Economics Consultant, and retired Senior Economist, 

Advanced Technology Program, National Institute of Standards and Technology, 

U.S. Department of Commerce 

 Dave Roessner — Evaluation Consultant; Associate Director, Science and 

Technology Policy Program, SRI International; and Professor Emeritus, Public 

Policy, and former Director, Technology Policy and Assessment Center, Georgia 

Institute of Technology 

Commissioned Evaluators for the 2010 Benefit-Cost Studies: 

 Albert Link — Professor of Economics, University of North Carolina at 

Greensboro 

 Michael Gallaher — Director of Technology, Energy, and the Environment 

Program, RTI International 

 Alan O’Connor — Research Economist, RTI International 

 Thomas Pelsoci — President, Delta Research Company 

 
Table of Contents 

Authors and Other Contributors.......................................................................................... 2

Table of Contents ................................................................................................................ 4

Part I. Background, Benefit-Cost Overview and Special Features .................................... 5

I.1 Background ............................................................................................................... 5

I.2 Overview of Traditional Economic Benefit-Cost Analysis ...................................... 9

I.3 Special Features of the EERE Benefit-Cost Approach .......................................... 18

Part II. Step-by-Step Guide .............................................................................................. 25

Overview of a Step-by-Step Approach ......................................................................... 25

1. Begin an Evaluation ................................................................................................ 27

2. Estimate Energy and Economic Benefits ................................................................. 33

3. Estimate Environmental Benefits ............................................................................ 46

4. Estimate Energy Security Benefits .......................................................................... 54

5. Estimate Knowledge Benefits .................................................................................. 57

6. Calculate Measures of Economic Performance and Summarize Other Effects ....... 65

7. Perform Sensitivity Analysis ................................................................................... 70

8. Report Results .......................................................................................................... 74

References ......................................................................................................................... 80

Attachment 1 ..................................................................................................................... 83

Attachment 2 ..................................................................................................................... 85

Attachment 3 ..................................................................................................................... 87

 

5

Guide for Conducting Benefit-Cost 

Evaluation of Realized Impacts of Public 

R&D Programs 

Part I. Background, Benefit-Cost Overview and Special Features 

I.1 Background 

This document provides guidance for evaluators who conduct impact assessments to 

determine the “realized” economic benefits and costs, energy, environmental benefits, 

and other impacts of the Office of Energy Efficiency and Renewable Energy’s (EERE) 

R&D programs. The focus of this Guide is on realized outcomes or impacts of R&D 

programs actually experienced by American citizens, industry, and others. Retrospective 

evaluations may be contrasted to prospective evaluations that reflect expected or potential 

outcomes only if assumptions hold. 

The retrospective approach described in this Guide is based on realized results only and 

the extent they can be attributed to the efforts of an R&D program. While it has been 

prepared specifically to guide retrospective benefit-cost analysis of EERE R&D 

Programs, this report may be used for similar analysis of other public R&D 

organizations. 

Directives and guidance from the executive and legislative branches set impact 

evaluation expectations for EERE and other Federal programs. These are listed in 

Attachment 1. 

EERE retrospective benefit-cost evaluations must be performed by independent 

professional evaluators guided by the procedures outlined in this peer-reviewed Guide.1

Evaluators are also expected to be guided by the Guiding Principles for Evaluators

provided by the American Evaluation Association and listed in Attachment 2. Although 

a degree of customization will be necessary based on the selected technologies under 

examination and the available data, the goal is to ensure that a basic consistency in 

method, approach, and convention is applied and maintained across such studies. 2 

 

1 An earlier draft of this Guide was peer reviewed by external experts in November 2009. See 

Acknowledgements for list of members of expert panel who reviewed the earlier version of this Guide.

2 Commissioned evaluators are expected to follow the Guide in performing EERE retrospective benefitcost studies unless specifically exempted from its coverage. They are encouraged to read the Guide at the 

outset of proposing or planning an R&D impact evaluation study.
6

The impacts assessments covered in this Guide are intended to address the following 

questions of interest to managers of the Department of Energy (DOE), Congress, the 

general public, and other stakeholders: 

1. To what extent has EERE produced 

energy and economic benefits relative to 

the next best alternative? 

2. To what extent has EERE achieved 

environmental benefits, and enhanced 

energy security? 

3. To what extent has EERE cultivated a 

knowledge-base in the research 

community that has impacted innovations 

in today’s markets? 

4. Would today’s commercialized 

technologies likely have happened at the 

same time, and with the same scope and 

scale, without EERE’s efforts? 

5. To what extent do benefits attributable to 

EERE involvement exceed EERE 

expenditures? Was the public investment 

worth it? 

The assessment approach outlined in this 

Guide produces impact results findings for the 

metrics defined in Table I-1. In addition to 

energy and economic impacts, the approach 

quantifies air emissions reduction, environmental 

health benefits (e.g., averted mortality and 

morbidity and other health effects, and dollars of 

health cost avoidance), certain energy security 

benefits, and knowledge creation and diffusion as 

reflected by patents and publications. It 

addresses attribution of benefits through the use 

of a counterfactual model which seeks to 

compare outcomes with what would likely have 

happened in the absence of the R&D program. 

 

The impact results quantified from the analysis provide a conservative estimate for three 

reasons: 

 

1) The approach strives to be fully retrospective in its coverage and thereby avoids 

reliance on forecasted data having a higher degree of uncertainty; nevertheless, 

benefits in most cases are expected to continue past the cut-off year of the analysis. 

2) The approach takes into account a portfolio rather than a single project; however, it 

includes the benefits of only a few technologies developed by a program or 

Directives for Evaluation of 

Federal Programs 

Impact evaluation questions for R&D 

programs are motivated by the desire of 

program managers to efficiently and 

effectively manage their R&D 

portfolios to make the best use of public 

investments provided by the American 

people. Consistent with this aim are a 

host of past and recent Government 

directives for impact evaluation of 

Federal Programs. Over the past several 

years there have been multiple 

directives from the executive and 

legislative branches that set program 

evaluation expectations for federal 

programs, as listed in Attachment 1. 

A federal energy R&D program that has 

determined, through systematic 

retrospective evaluation, its net benefits 

is better positioned to communicate its 

value to its agency leadership, 

Congress, stakeholders, and the public 

than one who lacks documented 

evidence. Systematic retrospective 

evaluation also informs program 

managers about possible ways to 

improve their programs and to position 

them for the future by revealing 

strengths and weaknesses in past 

performance. 
7

subprogram while taking into account total program costs to be weighed against the 

partial benefits. 

3) Not all benefits assessed are valued in monetary terms (e.g., the effects of Greenhouse 

Gases and energy security are expressed in tons of CO2 and equivalent barrels of 

imported oil avoided, respectively.3) 

Thus, the approach is empirical-based, and the results are more conservative than studies 

that (a) include forecasted effects, (b) take a project-approach rather than a portfolioapproach, (c) consider only project costs, and (d) use non-verified approaches to benefits 

estimation. 

As such, the results determined from the analysis described in this Guide provide a first 

order, lower bound estimate of the energy, economic, environmental, security, and 

knowledge benefits a program has contributed to the nation. This determination of 

conservative impact results is to be fully described and documented in all final study 

reports. 

The economic benefit-cost method used in evaluation studies supported by this Guide is 

designed to be applied to a portfolio (or cluster) of technologies, such as an entire 

program or subprogram. It should be noted, however, that in some cases the individual 

technologies selected within a cluster study may be appropriately treated as a group (e.g., 

a group of closely related infrastructure technologies), and in other cases each selected 

technology within a cluster will be more appropriately treated as an individual case study. 

This means that there will be some differences in the details of analyses across studies, as 

each study seeks to provide the most credible analysis possible for the types of 

technologies evaluated and the data available. This makes it possible for each individual 

study to stand on its own in terms of the validity of its analysis, and is to be preferred 

over applying an overly simplified, crude rule-of-thumb, one-size-fits-all (and less 

accurate) approach to benefit-cost studies. 

The method presented in this Guide builds on the R&D impact assessment approach used 

by the National Institute of Standards and Technology (NIST),4

 and improves on the 

approach employed by the National Research Council (NRC) in their 2001 study 

“Energy Research at DOE: Was It worth It?” An overview of how the approach offered 

in this Guide makes changes to the earlier NRC approach, as well as a more detailed 

comparison of the two, is provided in Attachment 3. 

 

3

 The review panel did not think that the existing approaches to valuation of these environmental and 

security effects had sufficient levels of confidence for inclusion without unduly increasing the level of 

uncertainty in overall results. Analysis of these effects in dollar terms could be performed as a separate 

analysis and reported as supplemental results. 

4 NIST, through its former Advanced Technology Program (ATP), pioneered in cluster benefit-cost studies 

to assess portfolios of projects rather than single projects. See, for example, Thomas Pelsoci (2005, 2007), 

and O'Connor, Rowe, Gallaher, et al. (2007). These ATP cluster studies, however, unlike the current EERE 

approach, considered only economic benefits and not the other categories of benefits included in the EERE 

benefit-cost cluster approach. 
8

Table I-1. Impact evaluation metrics covered in this Guide 

Outcomes Units 

Economic Performance Metrics 

 Gross economic benefits Millions or billions of dollars

 Net economic benefits (NB) (undiscounted)

1 Millions or billions of dollars 

 Net present value (NPV)

2

 at 3% and 7% discount rates Millions or billions of dollars 

 Internal rate of return (IRR) on public investment 4 Percent 

 Benefit-to-cost ratio (BCR) at 3% and 7% 5 Ratio 

Energy Benefits 

 Energy saved Trillion Btu

 Renewable capacity Mega watts (MW)

Environmental Benefits 

Air Emissions Reduction

 Avoided carbon dioxide emissions (CO2) Million metric tons of CO2 (MMTC) 

 Avoided sulfur dioxide emissions (SO2) Tons 

 Avoided nitrogen oxide (NOx) Tons 

 Avoided particulate matter emissions (PM) Tons 

Health Cost Avoidance Tons 

 Reduced morbidity (e.g., avoided respiratory symptoms, 

chronic bronchitis, nonfatal heart attacks) and mortality 

Mortality & morbidity rates 

 Health costs avoided due to reduced air emissions Millions or billions of dollars 

Energy Security Benefits 

 Displaced petroleum consumption

 Natural gas displacement

 Effect on energy infrastructure

Billions of gallons of gasoline equivalent (GGE); millions 

of cubic feet of natural gas;; barrels of imported oil 

equivalent (BOE) 

Description

Knowledge Benefits 

 Patents, publications, and other knowledge outputs and 

outcomes

Types and numbers of outputs, citation rates, linkages to 

EERE-sponsored R&D, Citation Index values, 

identification of notable patents, knowledge spillovers 

shown by linkages to other technologies and industries

Acceleration Effect (as appropriate) Years the research achievement has advanced due to 

EERE R&D efforts 

Notes: 

1 Net Benefits equal benefits minus costs, with no discounting applied to the cash flow. 2

 Net present value (NPV) equals the present value of the investment’s net positive cash flow, minus the present value of the initial 

investment. A positive NPV means that benefits exceed cost by more than enough to cover all costs including the required rate of 

return expressed by inclusion of the discount rate in the calculations. 3

 OMB issued Circular A94 (1992) and Circular A-4 (2003) that provide directives on discount rates for federal benefit-cost analysis. 4

 Internal rate of return (IRR) is a percentage yield on an investment, found as the solution value interest rate that equates benefits and 

costs, resulting in a Net Present Value (NPV) of zero. The IRR is compared against the investor's minimum acceptable rate of return 

(also known as the hurdle rate) to ascertain the economic attractiveness of the investment. If the IRR equals or exceeds the hurdle 

rate, the investment is economic; if it is less than the hurdle rate, it is uneconomic. 5

 Equals the present value benefits divided by present value investment costs. A ratio greater than one means that benefits exceed 

costs.
9

As of the end of 2010, the initial draft of this Guide had been used successfully in the 

conduct of four EERE benefit-cost cluster studies begun in 2009, and published in 2010 

and 2011. Feedback and “lessons learned” compiled from the first four benefit-cost 

cluster studies have been used to produce this edition of the Guide. 

It is expected that, as methodological advances emerge, including those resulting from 

the experiences of evaluators engaged in other DOE-commissioned evaluation studies, 

the Guide will be updated to reflect the state-of-the-art in the conduct of impact 

evaluations for R&D programs. In addition to fostering best practices in impact 

evaluation, another objective of the Guide is to ensure that basic consistency in approach 

across studies is maintained. 

Part I, Sections 2 and 3 of this Guide provide background on the benefit-cost method and 

highlight special features of this extended cluster approach. Then, in Part II, Sections 1 

through 8, the Guide provides detailed step-by-step instructions to independent evaluators 

who are contracted to perform retrospective benefit-cost cluster studies using this 

methodology. Examples from the four completed studies illustrate the approach. 

I.2 Overview of Traditional Economic Benefit-Cost Analysis 

This section provides a general description of traditional benefit-cost method, as used to 

estimate the economic performance metrics in Table I-1. It is provided both to show how 

economic benefits and related performance metrics are derived, and to provide a point of 

departure for adding to the traditional benefit-cost approach the expanded features 

described in section I.3 of this Guide. 

Dollar Benefits versus Dollar Costs 

Traditional benefit-cost analysis weighs the monetary value of economic benefits of an 

investment against its costs to determine if it was (or, in the case of prospective analysis, 

is expected to be) economically worthwhile. The dollar amounts are tracked as cash 

flows over time in order to adjust them for the time value of money—both for changes in 

the purchasing power of the dollar (due to inflation or deflation) and for the real 

opportunity cost of capital. 

The cash-flow model of Figure I-1 illustrates economic benefit and cost cash flows over 

time. The down-pointing arrows depict costs. The upward-pointing arrows depict 

benefits, which would include energy savings in the analysis of energy efficiency and 

renewable energy programs. Investment costs (i.e., program cluster costs) are identified 

as such; operating, maintenance, and repair costs in each year are typically netted out 

against that year’s benefits. 
10

Cash-Flow Time Adjustments 

As noted above, cash flows occurring at different times must be adjusted both for 

changes in the purchasing power of the dollar and for the real opportunity of capital. It is 

possible to make the adjustments for timing differences in cash flows in either of the 

following two ways: (1) First a price deflator index can be applied to the "current" 

(actual) dollar amounts occurring at different times to adjust them to "constant dollars" 

whereby each dollar has equivalent purchasing power as of a stated base year, and then 

an appropriate discount formula (or derived discount factors) can be applied to the 

constant dollar amounts, based on an interest rate, or "discount rate," that reflects the 

"real" return to capital apart from changes in dollar purchasing power. (2) Both timing 

adjustments can be done at once by applying to the "current (actual) dollar" cash flows 

(without first adjusting them to constant dollars) an appropriate discount formula (or 

derived discount factors) based on a discount rate that reflects the "nominal" (market) 

return to capital. 

Figure I-1. Economic benefit and cost cash flows 

Source: TIA Consulting, Inc.

Initial Investment Costs

Benefits

Other Costs

time

The result of applying either of these procedures is to express the whole of a stream of 

cash flows over time as either a lump-sum equivalent amount at a stated point in time, or 

as an equivalent uniform annual (or monthly) amount, depending on the discount 

formulas used. The procedure of converting a stream of cash flows over time to a timeequivalent amount at another time is often called "discounting cash flows" and the 

interest rate used in the discounting calculations is called the "discount rate."5

 

 

5

 This referenced use of "discounting" and "discount rate" is specifically within the context of capital 

investment analysis, as distinct from use of similar terms with different meanings in other contexts. 
11

OMB Directives on Discounting and Discount rates 

The approach for Federal benefit-cost analysis is subject to White House Office of 

Management and Budget (OMB) directives.6

 OMB Circular A-94, issued in 1992, 

directs the use of a 7% real discount rate for Federal benefit-cost analysis.7

 A more recent 

guidance is provided by OMB Circular A-4, issued in 2003, which pertains to benefitcost analysis used as a tool for regulatory analysis. As Circular A-4 notes, Circular A-94 

states that a real discount rate of 7% should be used in benefit-cost analysis—as an 

estimate of the average before-tax rate of return to private capital in the U.S. economy. 

This rate is an approximation of the opportunity cost of capital. Circular A-4 further 

notes that OMB found in a subsequent analysis that the average rate of return to capital 

remained near 7%. It also points out that Circular A-94 recommends using other 

discount rates to show the sensitivity of the estimates to the discount rate assumption, and 

notes that the average real rate of return on long-term government debt has averaged 

about 3%. It directs the use of both a 3% and a 7% real discount rate for a benefit-cost 

analysis conducted for regulatory purposes. 

Economic Performance Measures 

Use of multiple economic performance measures best meets the preferences of different 

audiences and help to broaden communication. There are multiple, closely related 

measures that are widely used to express the economic performance of an evaluated 

investment. Figure I-2 below summarizes three economic performance measures widely 

used in benefit-cost studies. These three measures are used to assess if, and to what 

extent, benefits attributed to a designated project, program, sub-program, or other 

portfolio exceeded the public investment. 

(1) Net Present Value Benefits (NPV): Total present value benefits minus total present 

value costs. 

A positive NPV means that benefits exceed cost by more than enough to cover all costs 

including the required rate of return expressed by inclusion of the discount rate in the 

calculations. The larger the NPV, the greater the extent that benefits exceed costs, and 

the more worthwhile is a project, other things being equal. 

(Net Benefits (NB) may be shown as undiscounted (i.e., assuming a 0% required rate of 

return or 0% discount rate), emphasizing the effect of discounting using positive rates.) 

The undiscounted results is designated NB in Table I-1. 

 

6

 OMB Circulars are available at www.whitehouse.gov/omb/circulars. 

7

 OMB Circular No. A-94, Oct. 29, 1992. “Guidelines and Discount Rates for Benefit-Cost Analysis of 

Federal Programs.” The 1992 issue of Circular No. A-94 states that a 7% real discount rate should be used 

for benefit-cost analysis. In December 2008, an updated release of OMB Circular No. A-94 provided 

Appendix C that provided revised discount rates for Cost-Effectiveness, Lease Purchase, and Related 

Analyses. However, Appendix C and the cover letter accompanying the release stated that the circular 

applies to cost-effectiveness and lease-buy decisions, rather than benefit-cost studies. Thus, Appendix C to 

OMB Circular No. A-94 is not considered applicable to the benefit-cost approach given in this Guide. 
12

(2) Benefit-to-Cost Ratio ( B/C or BCR): Present value benefits (less non-investment 

costs8

) divided by present value investment costs. 

A ratio greater than one means that benefits exceed costs. A ratio of 10, for example, 

means that 10 dollars are generated in benefits on average for every one dollar of costs 

incurred and taking into account the required rate of return expressed by inclusion of the 

discount rate in the calculations. 

(3) Internal rate of return (IRR): The IRR is a percentage yield found as the solution 

value interest rate that, when used in the appropriate discounting formulas, will equate 

benefits and costs, resulting in an NPV of zero. The yield is useful for comparing against 

a Minimum Acceptable Rate of Return (MARR) or "hurdle rate," which for Federal 

analyses is the discount rate, as well as against yields on other investments. If the 

computed IRR exceeds the MARR or hurdle rate the investment is deemed economically 

worthwhile. Other things being equal, the higher the IRR the more economically 

worthwhile the investment.9 

Figure I-2. Economic Performance Measures 

 

8

 The ratio is sensitive to the placement of costs. Investment costs are to be placed in the denominator; 

operating and maintenance costs are to be subtracted from benefits in the numerator. Note that moving 

these costs to the denominator will change the value of the ratio. 

9

 When either the B/C ratio or the IRR are used to design or size projects, these measures must be applied 

incrementally, because the mutually exclusive alternative with the highest B/C or IRR computed on total 

benefits and costs is not necessarily the one that yields the highest net benefits. However, choosing among 

competing designs and sizes is not the point of these benefit-cost impact studies, and the need for 

incremental or marginal analysis is not expected to arise. 

So urc e: TIA Consulting, Inc.

• Net Present Value Benefits (NPV): time-adjusted benefits 

minus costs

NPV = SBPV – (SCPV + SIPV)

w here SBPV = sum of present value benefits; SCPV = sum of present value 

non-investment cost; and SI PV = pres ent value investment cost

• Benefit-to-Co st Ratio: time-ad jus ted benefits (net of 

time-adjusted non-investment co sts) divided by 

time-adjusted inves tment cost

B/C = (SBPV - SCPV) / SIPV

• Internal Rate of Re turn (IRR ): the solution interest 

rate (i) that equates the values of the streams of 

benefits and costs over time 

SB(i) = (SC(i) + SI(i))
13

Unlike the previous two measures, the discount rate is not used directly in the IRR 

calculation. Rather, the IRR is solved for by substituting an interest rate with unknown 

value in place of the discount rate in discounting formulas and solving for the rate for 

which time-adjusted benefits equal costs, i.e., for which NPV is zero. 10 

Inclusion of Other Effects 

There may be economic effects that are not feasibly captured in dollar terms. If so, these 

are omitted in the economic performance measures. However, it has been commonly 

recognized in benefit-cost analysis that if other effects are potentially important to 

decision making, they should not be ignored, and, at a minimum, should be treated 

qualitatively. 

Yet, because the attention of benefit-cost studies is ultimately on the economic 

performance measures, omitted effects are often given less attention than they deserve. 

Thus, evaluators have in past studies attempted to express various important effects (not 

usually considered economic) in monetary terms, and to include the results in computing 

NPV, BCR, and IRR. For example, the value of a statistical life based on willingness-topay was used by Ruegg and Fuller in 1984 in a benefit-cost study of fire-suppression 

technology, and by Butry, Brown, and Fuller in a related 2007 study. Studies of highway 

safety, consumer product safety, and medical treatments have variously included 

estimated values of life and injury or imputed such values, or have used a cost-utility 

analysis and quality-adjusted life years to avoid placing financial values on life. 

Evaluators have also in past studies assigned monetary values to a variety of intangible 

effects such as environment, view, and business reputation, depending on the topic of 

major importance. However, these approaches have been largely piecemeal, nonsystematic, and controversial. 

This Guide seeks to include the treatment of important, difficult-to measure, effects using 

a systematic, non-controversial approach, as explained below in section I.3, Special 

Features of the EERE Benefit-Cost Approach. 

Federal versus Private Company Benefit-Cost Analysis 

Major differences between a benefit-cost analysis performed for a company and one 

performed for the Federal government are perspective/scope and treatment of taxes. 

Regarding perspective/scope, a company typically counts as benefits and costs the cash 

inflows and outflows it directly realizes as a result of its investment (i.e., the "private rate 

of return"). In contrast, the Federal government typically counts all benefits and costs to 

the nation resulting from its action, regardless of who experiences them (i.e., the "social 

rate of return" which encompasses both private returns and spillover returns). Figure I-3 

depicts in simplistic terms how private return and market and knowledge spillovers 

combine to produce social return. 

 

10 There are computer algorithms available for solving for the IRR. It can also be solved manually by a 

series of iterations, in which trial values of i are used until a solution value is found. 
14

Thus, Firm 1 invests in R&D, and realizes higher profits, due, say, to now having better 

products and lower production costs. But the situation is dynamic, and other firms may 

gain some of the resulting knowledge ("knowledge spillovers"), and they may compete 

with Firm 1 in its markets driving down prices and allowing consumers to benefit from 

better products at lower prices ("market spillovers"). In addition, some of these 

knowledge-acquiring firms may use it to produce other kinds of better or lower cost 

products in other markets. Some of these other benefits are captured by the producing 

companies; in competitive markets, some will "spill over" to consumers. Thus there is a 

private return to Firm 1, but there are also effects from Firm 1's actions that "spillover" to 

others—both to other firms and to consumers. The overall effect is the "social return." 

Figure I-3. Private Return and Spillovers Combine to Produce Social Return 

Source: Jaffe (1996), as discussed and modified by Ruegg and Feller (2003) 

If a Federal government investment contributes to development of an improved 

technology, a benefit-cost analysis to assess if the Federal investment was worthwhile 

would take into account the net effect across all establishments and people in the nation 

attributed to the Federal investment. Thus, a Federal benefit-cost analysis, with its focus 

on social returns, typically has a much broader perspective and scope of coverage than a 

private-company analysis with its focus only on its own returns. An analysis of social 

returns is therefore typically much more complex and difficult to perform than a privatecompany analysis. 

With regard to treatment of taxation effects, a company analysis and a Federal analysis 

also differ, however, this difference tends not to be as major as the differences of 
15

perspective and scope. A company typically wants to know its after-tax bottom-line 

return. In contrast, Federal government benefit-cost analysis is typically performed on a 

before-tax basis. A before-tax estimation is done because government is the recipient of 

Federal taxes collected, and other forms of taxes are typically not separately assessed in 

Federal benefit-cost analysis--unless, of course, a study is specifically aimed at 

assessment of the effect of specific taxes or tax incentives. 

It should also be noted that it is often not necessary for a Federal benefit-cost study to 

estimate fully the social benefits and costs associated with a new technology to achieve 

its evaluation purpose. For example, to estimate if a public R&D investment in a given 

technology area has been worthwhile, the required analysis is of the public returns, 

compared to the public R& D investment cost, rather than total social benefits of having 

the technology versus its total social costs. To estimate the return on public investment in 

a specific set of wind energy technologies, for instance, it is only necessary to compute 

the change in social benefits from wind energy technologies attributed to the public 

investment. That is, the computation does not require that total social benefits from 

having all wind energy technology be computed and compared against the total social 

costs—a much larger task. 

Use of the Mansfield’s Model as a Unifying Framework for the 

Valuation of Economic Benefits 

A model developed and applied by Professors Griliches and Mansfield, and since applied 

by others, has proven a useful framework for estimating social and private returns from 

investments in new technology.11 Mansfield applied the model to assess social benefits 

of private-sector industrial innovations, finding the estimated social rate of return for a 

group of selected industry innovations to be substantially higher than the private rate. He 

concluded that there may be a substantial "spillover gap" between private and social rates 

of return, whereby social rates of return exceed private returns. An implication is that 

private R&D investment decisions, which do not take into account spillover effects, will 

tend to result in less investment in R&D than is optimal from the standpoint of society at 

large. 

The “Griliches/Mansfield Model” has subsequently been applied to the analysis of 

Federal R&D investments. The model, which is well known to economists/practitioners 

of social benefit-cost analyses, is expected to serve as a theoretical anchor and unifying 

framework for the valuation of economic benefits in the EERE retrospective benefit-cost 

studies addressed by this Guide. 

The simplified representation in Figure I-4 of the Griliches/Mansfield model serves to 

illustrate the valuation approach. 

 

11 Griliches (1958), E. Mansfield, J. Rapoport, A. Romeo, S. Wagner, and G. Beardsley (1977); Foster 

Associates, (1978); Nathan Associates (1978), , "Net Rates of Return on Innovations," Report to the 

National Science Foundation, July 1978. 
16

Suppose that an innovation results in a new product used by firms that reduces costs of 

the industry using the innovation. The social benefits from the innovation can be 

measured by the profits of the innovator plus the benefits to consumers of the goods 

produced by the industry using the cost-reducing innovation. To the extent that the 

innovation is adopted (or adapted) in other applications, a similar approach could be 

taken in each application and the total social benefits (less costs) aggregated. Professor 

Mansfield acknowledged that the calculations are not this simple, but indicated that the 

basic model conveys the spirit of the analysis.12 

Figure I-4 Griliches/Mansfield Model of Social Benefits from an Innovation 

that Reduces the Cost of Producing a Good Sold 

Source: Edwin Mansfield, Estimating Social and Private Returns from Innovations Based on the 

Advanced Technology, 1996. 

With reference to Figure I-4, DD' depicts a demand curve for the goods using the costreducing innovation. The horizontal supply curve labeled S1 reflects the pre-innovation 

supply of the goods, and P1 indicates the pre-innovation price paid by consumers. The 

horizontal supply curve labeled S2 reflects the post-innovation shift downward of the 

supply curve due to decreases in production costs, and P2 indicates the new price that 

consumers will pay. The top-hatched area indicates the gain in consumer surplus, due to 

 

12 Professor Mansfield was engaged by the Advanced Technology Program to extend his model to apply in 

benefit-cost studies of ATP-funded innovations. This brief description of his model is from a preliminary 

planning report for that effort. Edwin Mansfield, Estimating Social and Private Returns from Innovations 

Based on the Advanced Technology Program: Problems and Opportunities, National Institute of Standards 

and Technology, GCR 99-780, January 1996 (Available on-line at www.atp.nist.gov/eao/gcr99-

780/contents.htm). 
17

the innovation. It is the excess of what consumers would have been willing to have paid 

for the new quantity versus what they actually had to pay, summed over all purchases. 

How far downward the supply curve will shift depends, of course, on the effect of the 

innovation, the pricing policy of the innovator, and the competitive structure of the 

industry sector. If the industry sector is characterized by little competition, the innovator 

may be able to hold the product prices relatively unchanged, such that the supply curve 

shifts little or none. However, if the industry sector using the innovation is competitive, 

it is expected that the innovator will lower the price for its new product as others enter 

with competing products. 

The social benefits from the innovation can be measured by the sum of the two crosshatched areas in Figure I-4. The top cross-hatched area is the consumer surplus due to 

the lower price (P2 rather than Pl) resulting from the use of the innovation. In addition, 

there is a resource saving, and a corresponding increase in output elsewhere in the 

economy, due to the fact that the resource costs of producing the good using the 

innovation are less than P2 Q2. Instead, they are P2 Q2 minus the profits of the 

innovator from the innovation (r), the latter being merely a transfer from the producers of 

the good using the innovation to the innovator. Thus, besides the consumer surplus 

arising from the price reduction, there is a resource saving amounting to the profits of the 

innovator. For example, suppose the innovator reaps a $100 million profit from its 

innovation. This means that P2 Q2 is an over-estimate of the value of the resources used 

by the industry, in the amount of $100 million; the amount the industry pays the 

innovator in profits. Recall that this payment to the innovator is not in exchange for 

resources; rather, it is a transfer of profit to the innovator. 

Two adjustments are needed in the estimate corresponding to the lower shaded area in 

Figure I-4. First, if the innovation replaces another product, the resource saving cited 

above does not equal the profits of the innovator. Instead it equals these profits less those 

that would have been made (by the innovator and/or other firms) if the innovation had not 

occurred and the displaced product had been used instead. Second, if other firms imitate 

the innovator and begin selling the innovation to the industry that uses it, their profits 

from the sale of the innovation must be added to those of the innovator to get a full 

measure of the extent of the resource saving due to the innovation. 

Using this model, an estimate can be made of the social benefit in each period from the 

investment in a given innovation. For each innovation, the top shaded area in Figure I-4 

equals, 

 (P1 - P2) Q2 (1 - 1/2 Kn) (1) 

where K = (P1 - P2)/P2, and n is the price elasticity of demand (in absolute value) of the 

product of the industry using the innovation. 

To estimate P1 - P2, Mansfield’s approach was to obtain as much information as possible 

on the size of the unit cost reduction due to the innovation. To obtain a reasonably 
18

reliable estimate of (P1 - P2), Mansfield conducted interviews with executives of the 

innovating firm, executives of a sample of firms using the innovation, and reviewed 

reports and studies made by these firms for internal purposes. And with the estimate of 

(P1 - P2), it was then possible to compute K. Q2 was generally available from published 

records. Rough estimates of n were obtained from published studies and from the firms. 

Since K was generally very small, the results were generally not very sensitive to errors 

in n. 

Indeed, Mansfield concluded that the expression in equation (1) could be approximated 

well in most cases by (P1 - P2) Q2, which is the total savings to consumers due to the 

lower price if they buy Q2 units of the product of the industry using the innovation. This 

latter point has been helpful in the practical application of Mansfield’s model. 

To use the Griliches/Mansfield model for estimating net benefits from a public-sector 

innovation, the approach is to estimate social benefits as the stream of consumer and 

producer surplus resulting from an innovation. The counterfactual case is assumed to be 

the technology and associated demand and supply situation that existed just prior to the 

innovation and that would have existed without the innovation. The social cost is the 

combined public and private R&D and related costs over time incurred for the purpose of 

innovation. Public benefit is the part of social benefits attributed to the public 

investment, plus any reductions in realized total social costs compared with 

counterfactual total social costs. Public investment cost is the cost of the public program. 

Note that the Mansfield approach includes market spillover effects which occur as others 

in the same industry as the innovator, within competitive markets, use the innovator’s 

knowledge to imitate the innovation and drive down prices to consumers. Not included 

in the simplistic depiction of Figure I-4 are effects that occur as firms outside the 

innovator’s industry draw from the same knowledge base to produce other goods and 

services in other industries. Also not included are non-economic effects, such as 

environmental, energy security effects, and the more general effects of an enhanced 

knowledge base on the capacity of organizations to innovate in other areas. These later 

effects are addressed explicitly by the EERE approach. 

I.3 Special Features of the EERE Benefit-Cost Approach 

This section briefly describes the special features of the EERE benefit-cost approach. 

They are: 

 Use of a cluster approach 

 Extension of the benefits evaluation to account systematically for multiple 

categories of benefits – energy, labor, and other resource effects, environment, 

energy security, and knowledge diffusion 

 Improved characterization of the next-best alternative 

 Detailed analysis of attribution of benefits 
19

 Focus only on retrospective, empirically-based benefits and costs 

 Exclusion of treatment of employment and regional effects 

 Qualitative/quantitative treatment of international effects only if important to the 

assessment of public benefits 

 Use of sensitivity analysis 

Each of these features is briefly described below. 

Cluster Approach 

EERE's approach extends the analysis to "clusters" or portfolios of projects, with a cluster 

consisting of an entire program or subprogram or other grouping of similar technologies 

or related projects. Selected individual technologies within a cluster are evaluated in 

detail; remaining technologies within the cluster are treated qualitatively. Benefits of the 

selected technologies may be compared against their own costs, but, more importantly, 

they are compared against the investment cost of the entire cluster, as illustrated in Figure 

I-5. The resulting economic performance measures of benefits against cluster costs 

provide a minimum rate of return on EERE's investment in the program cluster – a lower 

bound estimate.13 

The EERE cluster approach tends to be cost-effective because it enables the evaluation to 

be focused on a relatively few projects in a larger R&D portfolio (project or subprogram 

portfolio), while enabling broader conclusions to be drawn about an entire program or a 

subprogram. Furthermore, the approach works well for high-risk R&D programs where a 

few projects tend to be the big winners, but investment in an array of projects is 

necessary to find the few that will turn out to be highly successful. 

From the standpoint of a program manager, cluster analysis offers the possibility of 

presenting a strong evaluative position. The strong position is that of being able to 

demonstrate that benefits from only a few elements in a larger cluster more than offset all 

program cluster investment costs, while additional elements in the cluster also hold 

promise of additional benefits or, at a minimum, offer little threat of offsetting effects. 

 

13 As noted earlier, the use of benefit-cost analysis applied to technology clusters was pioneered by the U.S. 

Advanced Technology Program (ATP) in the early 2000s. 
20

Figure I-5. Illustration of Benefits versus Costs 

in EERE Benefit-Cluster Approach 

Multiple Categories of Benefits 

 

The EERE approach extends the comprehensiveness of the benefit-cost analysis by 

credibly accounting for multiple categories of benefits. EERE assessment of benefits 

includes the following four categories, pushing coverage substantially beyond traditional 

benefit-cost studies: 

 Energy and Economic Benefits — taking into account all affected resources in the 

economy, including energy, labor, and other resource effects estimated in dollars, and 

following the general approach of the Griliches/Mansfield model. Economic benefits 

are increases in the value of goods and services in the economy. Technological 

advancement is one way to increase economic benefits. This occurs by improving the 

performance of existing goods and services and/or reducing their costs, and by 

developing novel goods and services that provide desired new capabilities and 

experiences with economic value. 

Energy effects are part of economic benefits, but they are also treated separately 

because energy is the focus of the EERE programs. Energy effects are assessed in 

terms of energy saved, or MW capacity generated by renewable in lieu of fossil-based 

generation. 

 Environmental Benefits — greenhouse gases, and public health effects from 

reductions in air emissions including mortality and morbidity effects and dollars of 

health cost avoidance calculated using EPA's COBRA Model; plus any notable other 

(cluster = entire program, sub-program, or portfolio)

Quantitative benefits

of selected elements

of a research/technology

“cluster”

versus

Investment costs of the 

selected elements for 

detailed study

Investment costs of 

entire cluster

Qualitative effects of other elements

in the cluster

Select technologies for detailed analysis

Source: Rosalie Ruegg, TIA Consulting, Inc.
21

environmental effects (e.g., water discharges, land resource use, and solid waste 

generation). 

 Energy Security Benefits — expressed as imported barrels of oil equivalent units 

avoided for displaced fossil fuels and also billions of gallons of gasoline equivalent 

(GGE) and millions of cubic feet of natural gas;14 and includes qualitative 

identification of notable effects on the security of energy infrastructure. 

 Knowledge Benefits — creation and dissemination effects of knowledge outputs of 

the cluster as indicated by chiefly by patent and publication counts and citation 

analysis, together with treatment of other knowledge creation and dissemination 

effects as feasible. 

Improved Characterization of the Next-Best Alternative 

The benefits of a new or improved technology are assessed in comparison with the next 

best alternative, i.e., the best choice that could be made in lieu of choosing the new 

technology. The next-best alternative is also often called the "defender technology." The 

performance of the defender technology provides a baseline against which to take the 

performance differences afforded by the new or improved technology. If the defender 

technology would likely have been improved over time if not displaced by the new or 

improved technology, estimation of the baseline may require dynamic modeling. 

Incorrect identification of the defender technology can result in substantial errors in 

benefit estimation. To support sound selection of the defender technology, Part II of the 

Guide provides examples of conditions that will influence its determination. 

Detailed Attribution of Benefits 

Assigning attribution for assessed benefits is a particularly key step in the EERE 

approach. The observation of positive effects does not necessarily mean that the EERE 

R&D cluster investment was responsible for generating the benefits. Other potentially 

causal factors must be taken into account and eliminated as rival explanations for benefits 

generation. 

In contrast to the EERE attention to attribution, benefit-cost analyses often simply 

assume that all observed benefits are attributed to the subject investment. Alternatively, 

they use a simple rule-of-thumb. Rather than rely on a simple rule-of-thumb approach to 

attribution, the evaluation approach presented by this Guide promotes a detailed, case-bycase approach to assessing attribution. 

To focus attention on the detailed attribution of benefits at the various stages of 

technology development, the matrix in Figure I-6 is provided to guide the evaluator in the 

comprehensive assessment of attribution. The tabular framework helps map attribution to 

 

14 Monetary value was not applied to barrels of oil equivalent units because the methodology is considered 

to require further development. 
22

Figure I-6. Detailed Assessment of Benefits Attribution 

A Matrix for Assessing Attribution by 

Technology Stage 

Categories of Information 

Needed for Additionality

Assessment

Technology Timeline (Stage of Research, Development, and Commercialization)

Preliminary & 

detailed 

investigation

Develop 

components

Develop 

system

Validate/ 

demonstrate

CommercializeMarket 

Adoption 

History of the technology

What DOE Did

What Others Did (Rival 

Explanations—Private 

Sector and Other 

Nations)

What Others Did (Rival 

Explanations –US & 

State Government) 

The DOE Effect 

Description of DOE Influence

And its strength 

Basis of evidence of influence

the technology timeline to show when each identified effect is estimated to have 

occurred, to identify and eliminate rival explanations, and to indicate the range of 

attribution to the R&D program. 

Supporting evidence of attribution is provided by the publication and patent analysis that 

is done to assess knowledge benefits. The knowledge benefits assessment demonstrates 

quantitatively and qualitatively the linkages between the publicly funded R&D and 

downstream commercial activities. 

Only Retrospective, Empirically Based Benefits and Costs are 

Included 

The measures for each category of benefits are to be derived fully from retrospective 

analysis, rather than life-cycle or prospective analysis. Only the benefits achieved, and 

costs incurred, are taken into account; no projections are included. The impact results 

quantified are thus also fully retrospective. 

Quantification of monetary value of some externalities (e.g., CO2, energy security) are 

excluded because current approaches to estimating for damage costs do not at this time 

support an acceptable level of confidence. 
23

Exclusion of Employment Effects and Regional and International 

Effects 

Employment Effects 

Beyond the inclusion of labor costs as a resource that may figure in economic benefitcost analysis, the Guide includes no specific requirement for the treatment of 

employment effects, such as estimate of numbers of jobs created or retained, or salary 

effects. 

There are several reasons for this decision to omit the treatment of employment effects. 

One reason for the omission is the fact that many of the past investments treated in 

retrospective evaluations were done under conditions approaching full employment—

they were not planned or implemented with a specific goal of achieving employment. 

Under conditions of national full employment, the assumption was that jobs added by one 

investment would be offset by transfers from jobs elsewhere, rather than resulting in net 

job gains. 

More recently, with unemployment a pressing issue and energy projects part of the 

Federal economic stimulus package, employment effects of energy projects have gained 

attention. With employment goals for energy investments, the explicit treatment of 

employment effects in terms of number of jobs created or retained and salary effects may 

be added to future retrospective benefit-cost studies of energy projects. If this happens, it 

will be noted in the release of an Addendum to this Guide. 

Regional Effects 

Aside from specific regional development efforts taken at the Federal level, regional 

shifts in economic activity are generally not separately included in Federal evaluations. 

Rather, Federal evaluations typically are focused on national economic effects, and 

regional shifts are assumed to have a neutral national effect. Thus, regional effects are 

not called out for separate treatment in the EERE benefit-cost studies. 

International Effects 

There is little precedence for treating international effects in Federal benefit-cost studies. 

Also, there is the more immediate concern that requiring their treatment would exceed the 

resource and time constraints of project funding levels. 

In the case of certain technologies, however, a treatment of international effects will be 

needed to understand national impacts. For example, it is difficult to understand returns 

to the U.S. public investment in NiMH batteries without having at least some 

understanding of the development and interrelationships of R&D, innovation, battery 

commercialization, and markets for battery applications in different parts of the world. 

Thus, evaluators of EERE benefit-cost studies are encouraged to provide an overview of 

international effects, as well as a more detailed assessment of international developments 
24

when they are critical to estimating and understanding national benefits in a given 

technology area. 

Use of Sensitivity Analysis 

Even for a retrospective, empirically based evaluation, there will be uncertainties about 

assumptions, data, and, therefore, results. The EERE approach emphasizes the 

identification of areas of uncertainty and the use of sensitivity analysis to reveal how 

changes in uncertain values of critical inputs will affect overall estimates of benefits. 15 

Examples of input variables that might be subjected to sensitivity analysis are the 

quantity of energy saved or the degree of attribution of benefits to the program cluster. 

 

15 Sensitivity analysis is an approach widely used in economic impact studies to acknowledge that there are 

uncertainties, and to test the effect of changing one or more key input values for which there is uncertainty. When the 

probabilities that input values differ from "best-guess" estimates are known, risk assessment can be used--instead of 

sensitivity analysis--to assess the risk exposure inherent in an investment decision. Risk assessment techniques include 

expected value analysis, decision analysis, simulation analysis, and other techniques described by Ruegg (1996). Risk 

assessment techniques may also be used in the EERE approach, but are not required. For more on sensitivity analysis, 

see Saltelli, Chan, and Scott (2000) and Marshall (1996). 
25

Part II. Step-by-Step Guide 

Overview of a Step-by-Step Approach 

 

An overview of the step-by-step flow diagram of the EERE approach to retrospective 

benefit-cost cluster analysis is given in Figure II-1. 

It should be emphasized that depicting the evaluation approach as a series of sequential 

steps does not mean that the process is either linear or formulaic. Conducting a 

successful study requires an evaluator experienced in both the art and science of benefitcost analysis, able to capture complex and multiple direct and indirect effects of a public 

investment using creativity and a variety of techniques within the organization of a stepwise approach, and aware of the dynamic nature of the process. 

Part II of the Guide follows the logic of the diagram in Figure II-1. Key process steps are 

indicated in the text by the symbol "►". Examples from the first-round of 2010 studies 

are selectively presented to illustrate key steps and are indicated in the text by the symbol 

◙. 
26

Figure II-1. Major Steps in the Evaluation Process 
27

1. Begin an Evaluation 

► Identify EERE programs for study 

A starting point for benefit-cost evaluation is determining which EERE program to 

evaluate. This likely may be done internally by the EERE staff with the subsequent issue 

of an EERE Request for Proposal (RFP), or it may be done jointly by a contracted 

evaluator and EERE staff. The identification of a program cluster follows this initial step. 

In general, an EERE program that could benefit from participating in an impact 

evaluation study conducted according to this Guide would be -- 

 One for which return on investment is currently not known; or 

 One that was previously evaluated in the 2001 NRC study but, 

o had shown no significant realized economic return on R&D investment, in 

part because accrued market value had not been established for its 

commercialized technologies during the period of the NRC study, or 

o had successfully commercialized technologies that were excluded from a 

previous benefit-cost study, or 

 One previously evaluated but requires an update (e.g., 2 to 5 years later), or 

 One that had performed benefit-cost calculations using an alternative approach not 

consistent with the EERE approach presented in this Guide. 

► Define a "program cluster" for benefit-cost evaluation 

Evaluators are expected to work collaboratively with Program staff to define a program 

cluster for benefit-cost analysis. In the event that they are responding to an EERE 

Request for Proposal (RFP), the EERE Program staff will typically have identified the 

program cluster for study in advance. 
28

The program cluster identified for study may be defined as: 

1) An entire program’s portfolio of related projects/activities, 

2) A comprehensive collection of related projects/activities with a shared objective 

that forms a subprogram or category within a larger program, or 

3) A set of related projects/activities that cut across several programs but share a 

common feature that makes it reasonable 

and desirable to group them. 

Thus, the program cluster may comprise an 

entire EERE Program, a Sub-program, or some 

other grouping or portfolio of projects or 

technologies that share common attributes. 

The step of “defining a program cluster” is to be 

implemented in close coordination with 

“selecting specific technologies within the 

cluster” and “characterizing the cluster” – 

described below. This is because often 

preliminary explorations will need to be done to 

assess the feasibility of a given program cluster 

for study. Preliminary explorations would entail 

taking note of the market uptake of candidate 

technologies, preliminarily considering their 

likely impacts, and assessing data availability and other feasibility conditions for studying 

the program cluster. 

► Select specific technologies within the program cluster for detailed 

analysis 

After the program cluster has been identified, the next step is to select a few technologies 

within the cluster for detailed benefit-cost analyses (again in collaboration with EERE 

Program staff or as specified by the RFP). The technologies selected for detailed 

study could be, for example: 

 A whole system (e.g., an advanced wind turbine or a geothermal plant); or 

 A component of a larger system (e.g., blades for a wind turbine, or high 

temperature cement for a geothermal well application); or 

 Infrastructure technology that enables a commercial advancement (e.g., new 

modeling capability, test results, or new designs); or 

 A new or improved process (e.g., turbulence modeling for wind turbine inflow, or 

improved, faster deposition methods for solar PV thin films). 

◙ In a 2010 study for DOE’s 

Advanced Combustion Engine R&D 

(ACE R&D) the program cluster was 

taken as the ACE research effort. 

This included R&D for Laser 

Diagnostics and Optical Engine

Technologies, Combustion Modeling, 

Combustion and Emission Control, 

and Solid State Energy Conversion. 

The ACE R&D Sub-Program is one 

of eight sub-programs within a larger 

DOE Vehicle Technologies Program. 

In the ACE R&D benefits-cost study, 

the program cluster was the entire 

R&D portfolio comprising R&D in 

the ACE subprogram.
29

Because benefits of the few technologies in a cluster selected for detailed study will be 

compared against total EERE cluster costs, it makes sense—and, in fact, it is a feature of 

the approach—that those selected should be 

among the most successful in having achieved 

technical and commercial results prior to the 

study cut-off year16 for the analysis. Further, it 

should be feasible to assess the benefits of 

those technologies selected. 

Near term technologies not yet in commercial 

production are excluded from selection for 

detailed analysis. This exclusion extends to 

technologies that are in final development and 

demonstration stages at the time of selection, 

but which are not expected to have entered the 

market under currently projected economic, 

regulatory, and tax conditions until after the 

study cut-off year. This criterion is consistent 

with retrospective analysis which uses 

empirical data rather than projections. Also 

excluded from selection are technologies 

whose benefits cannot be feasibly measured. 

The example in Table II-1 lists clusters and 

technologies selected for detailed analysis 

within each cluster, from recently completed 

2010 studies. 

► Characterize the program cluster, the selected technologies for 

detailed analysis, and other technologies in the program cluster 

For context, provide an overview of the program cluster to be studied, and of the 

technologies within the cluster selected for detailed analysis. Describe the objectives to 

be achieved by the program cluster, activities undertaken in support of these objectives, 

and how the technologies selected for detailed analysis have affected resource use. 

Explain why the program cluster and the technologies for detailed analysis were selected. 

While the quantitative analysis centers on those several technologies selected for detailed 

analysis, it is important also to provide at a minimum a qualitative review of other 

elements in the program cluster. The aim is to give a sense of the likely overall impact 

on program cluster net benefits of these other elements within the cluster. Is there reason 

to believe that these other elements will not offset any positive effects of the selected 

technologies? Is there reason to believe they will provide additional benefits? 

 

16 For each of the four benefit-cost studies completed in 2010, the study cut-off year was end of 2008. The 

cut-off year will vary, depending on the year a study is initiated and availability of data for the analysis. 

◙ In the previous example, a subset of 

technology/ research areas from within the 

ACE R&D cluster was selected for 

examination in the benefit-cost 

assessment. The specific technologies 

within the cluster selected for detailed 

analysis were 

 Laser diagnostic and optical engine 

technologies focused on heavy-duty 

diesel engines 

 Combustion modeling –in particular 

KIVA modeling that simulates the 

fluid dynamics of combustion

processes in internal combustion

engines 

Combustion and Emission Control and 

Solid State Energy Conversion were not 

included in the quantitative benefits 

analysis for the ACE R&D benefit-cost 

study, although the total cluster cost was 

accounted for in the analysis. 
30

Table II-1. Program Clusters and Selected Technologies for Detailed Analysis 

within Each Cluster, 2010 EERE Benefit-Cost Studies 

Four Program Clusters 

(2010 Studies) 

Technologies within Each Cluster Selected for Detailed 

Analysis (2010 Studies) 

Geothermal Technologies 

Program 

 Polycrystalline diamond compact (PDC) drill bits 

 Binary cycle power plant technology 

 TOUGH series of reservoir models 

 High-temperature geothermal well cements 

Wind Energy Program  Wind turbulence models 

 The unsteady aerodynamic experiment to acquire accurate 

aerodynamic and structural measurements 

 Turbine blade material characterization and analytical 

modeling work 

 Wind turbine component demonstration programs 

Photovoltaic (PV) Energy 

Subprogram of the Solar 

Energy Technology 

Program 

 Flat-Plate Solar Array project 

 PV Manufacturing Technology Project 

 Thin-Film PV Partnerships 

Advanced Combustion 

Engine R&D Subprogram 

of the Vehicle 

Technologies Program 

 Laser diagnostic and optical engine technologies focused on 

heavy-duty diesel engines 

 Combustion modeling --in particular KIVA modeling that 

simulates the fluid dynamics of combustion processes in 

internal combustion engines 

► Identify EERE costs of the program cluster and, if possible, of the 

specific technologies selected for detailed analysis 

Cluster costs are the cost of the program or subprogram or other portfolio chosen for 

study. Thus, costs of all projects within the cluster are to be included; nothing within the 

cluster scope is to be left out of cluster costs. 

Total program and subprogram costs should be available for past years from the program. 

However, costs broken out for parts of subprograms or for individual technologies within 

a cluster may be less readily available. Because total cluster costs are essential to 

computation of the desired economic performance metrics, obtaining them is imperative. 

As soon as the cluster is defined, program staff should take steps to provide cost data for 

the relevant cluster to the evaluators, and the evaluators should request the cost data from 

the program. 

This is a critical step in the entire process. It requires consistency of program budgeting 

practices over time or the regrouping of subprograms by those knowledgeable of past 

practices and adjustments to budget categories to provide the required cluster cost data. 
31

It is program’s responsibility to provide the necessary cluster data to the evaluators. It is 

evaluator’s responsibility to bring this 

requirement to the attention of program staff. 

Cluster costs should be presented year-byyear over the entire period of the program's 

investment. Costs can be expressed in actual 

dollars at the outset, but are to be converted 

to yearly inflation-adjusted constant dollars 

for the designated year.17

At a minimum, evaluators are asked to 

assemble the program investment costs data 

for the entire program and obtain data for the 

program cluster (or subprogram), as well. 

For example, the entire DOE Wind Energy 

Program investment ($1.7 billion, $2008 

inflation adjusted), as well as the R&D costs 

of the selected infrastructure technologies 

($1.2 billion) was assembled for the Wind 

benefit-cost study. For the Wind Energy 

Program example, the time series of costs of 

the program cluster versus the total program 

cost over the historical period 1976 to 2008 

are given in Table II-2. 

Economic performance measures are 

calculated and reported based on both the 

entire program and the cluster costs. If the 

costs of the individual technologies selected for study are separately identified, the 

evaluators are instructed to also calculate net economic benefits based on the investment 

relevant to the selected technologies examined. At a minimum, they are asked to report 

calculated results for the total cluster investment. 

 

17 See addendum pertaining to key dates and other updated information applicable to current studies. 

◙ In a 2010 Wind Energy R&D BenefitCost study, the cluster and selected 

technologies were characterized as 

infrastructure technologies – 

•Wind turbulence models 

•The unsteady aerodynamic 

experiment to acquire accurate 

aerodynamic and structural 

measurements 

•Turbine blade material 

characterization and analytical 

modeling work 

•Wind turbine component 

demonstration programs. 

In the benefit-cost study, the selected group 

of infrastructure technologies were treated 

as a group of infrastructure technologies, 

rather than separately. 

Public investment in these infrastructure 

technologies represented only a portion of 

the total public investment of the Wind 

Energy Program – i.e., 70% of its total 

historical budget. 
32

◙ Table II-2. Investment Costs in the Wind Energy Program (the Benefit-Cost Study 

Cluster) and Investment Costs for Selected Technologies within the Cluster 

Year 

Investments in 

Selected 

Infrastructure 

Technologies 

Nominal 

(Thousand 

Dollars) 

Total Wind 

Energy Program 

Investments 

Nominal 

(Thousand 

Dollars) 

Inflation Adjusted 

Investments 

in Selected 

Technologies 

2008 Dollars 

(Thousand Dollars) 

Inflation 

Adjusted Total 

Wind Energy 

Program 

Investments 

2008 Dollars 

(Thousand 

Dollars) 

(1) (2) (3) (4) 

1976 14,403 44,027 

1977 20,500 58,910 

1978 34,470 35,300 92,560 94,788 

1979 58,155 59,555 144,166 147,636 

1980 56,254 60,555 127,801 137,572 

1981 76,087 77,500 158,050 160,985 

1982 37,700 38,400 73,807 75,178 

1983 31,290 31,390 58,928 59,116 

1984 26,367 26,367 47,860 47,860 

1985 28,155 28,355 49,603 49,955 

1986 12,536 24,786 21,608 42,723 

1987 11,930 16,606 19,983 27,816 

1988 8,064 8,464 13,059 13,707 

1989 8,260 8,760 12,890 13,670 

1990 8,498 8,687 12,768 13,052 

1991 10,836 11,034 15,724 16,011 

1992 21,082 21,282 29,883 30,167 

1993 5,500 23,841 7,628 33,063 

1994 9,334 29,151 12,678 39,593 

1995 11,784 34,309 15,679 45,648 

1996 16,830 31,420 21,974 41,023 

1997 20,540 28,646 26,353 36,752 

1998 17,301 32,128 21,949 40,759 

1999 20,861 34,076 26,082 42,604 

2000 17,219 31,734 21,072 38,835 

2001 19,902 39,132 23,817 46,830 

2002 21,731 38,211 25,592 44,999 

2003 26,282 41,640 30,299 48,005 

2004 26,188 39,803 29,358 44,621 

2005 24,053 40,631 26,093 44,078 

2006 17,276 38,333 18,150 40,273 

2007 29,839 48,659 30,476 49,698 

2008 22,643 49,034 22,643 49,034 

Totals 

Investments 736,967 1,072,692 1,238,531 1,718,989 

Source 1: Pelsoci (2010). 
33

2. Estimate Energy and Economic Benefits

 

► Define the "next-best alternative" (or "defender technology") that 

would have been used in lieu of each selected technology (or group 

of selected technologies), and estimate the change in resource use in 

comparison with the alternative 

The merits of the selected new technology are judged against the next best alternative, 

i.e., the best choice that could have been made in lieu of choosing the new technology. 

For a retrospective benefit-cost analysis, the next best alternative is defined by looking 

back to the time the investment decision was made for the new technology. That is, the 

next best technology at the time of the investment decision is not necessarily today’s next 

best alternative. It is the choice not taken, and therefore it represents a counterfactual 

comparison. 

In defining the next best, or defender technology, it is necessary to determine whether to 

use static or dynamic modeling. Based on what is known about the defender technology, 

it should be possible to determine whether it is more appropriate to model it as remaining 

constant or changing in performance over the period of comparison. 

There is an element of judgment in the selection of the next best alternative—even for 

retrospective studies—but there are also determining factors to consider. One of these 

factors is whether the choice was constrained or unconstrained. For example, if use of a 

renewable energy system were required (say by a State requirement), investment in a 

renewable system would be compared against other renewable energy systems. Another 

factor is whether the technology was new to the world or an improvement over an 

existing system. If it was new to the world, it might be compared against the best 

alternative conventional technology. If the technology was an improvement over an 

existing system, it would likely be compared against the then best existing earlier system 
34

model. Table II-3 gives examples of the next-best alternatives identified for the four 

2010 EERE Benefit-Cost Studies. The first column lists the study and technologies 

evaluated in detail, and the second column describes the next-best alternative. 

◙ Table II-3. Examples of Next-Best Alternatives Identified 

in 2010 Benefit-Cost Studies 

Study 

(1) 

Next-Best Alternative 

(2) 

Advanced Combustion Engine 

(ACE) R&D Subprogram--

laser and optical diagnostics 

The state-of-the-art in diesel engine design and related 

brake thermal efficiency (BTE) that existed prior to 1995. 

Geothermal R&D Program -- with 4 distinct technology cases 

 1. Polycrystalline Diamond 

Compact (PDC) Drill Bits 

Existing roller-bit drill technology. 

 2. Binary cycle power plant For reservoir temperatures in the range of 150 to 190° C, 

flash cycle technology; and for temperatures in the range 

below 150° C, a coal power plant. 

 3. TOUGH series & related 

reservoir models 

"Lumped parameter" models used before capabilities for 

detailed computer simulation of reservoirs were developed. 

4. High-temperature 

geothermal well cements 

Existing Portland cements. 

Solar Photovoltaics R&D 

Subprogram--cSi modules and 

thin-film modules 

Existing inferior crystalline silicon PV modules that would 

have been continued during the delayed introduction of the 

improved PV modules technology. 

Wind Energy R&D Program--

group of infrastructure 

technologies 

Fewer, smaller, less reliable, less cost-competitive wind 

turbines with lower energy capture that would have been 

used during the delayed introduction of turbines based on 

the improved infrastructure technologies 

It is a requirement that the benefit-cost study describe in the final report how the study 

has defined the next best alternative for each technology or group of technologies 

selected for detailed analysis, and what was the rationale. 

► Estimate the changes in resource use in comparison with the nextbest alternative, expressing the results in physical units 

Perform year-by-year comparisons of the selected technology or group of technologies 

against the appropriate next best alternative to derive the differences in each type of 

affected resource stated in physical units. For example, what are the differences in 

quantities of energy consumption by type? What are the effects on labor requirements? 

What are the differences in purchase, installation, and maintenance costs? What are the 

differences in life expectancies? 
35

These effects may differ among the selected technologies or groups of technologies and, 

if so, will require separate, individual comparisons with each next-best alternative. 

It is at this stage that incremental energy savings and/or MW capacity is derived from 

differences in the performance characteristics of the selected technology and its next-best 

alternative. There are a variety of ways resource savings and associated energy/ fuel 

changes can be calculated, and the specific approach depends on the selected technology 

and available data. The calculation approach and results should be well documented in 

the study report. 

The following are two examples for illustrative purposes -- 

(1) Wind Energy Benefit-Cost Study Example, Pelsoci (2010), where billions of 

kWh of electricity otherwise supplied by fossil fuel (calculated by type of 

fuel) were displaced by wind energy. (See Example A below) 

(2) Advanced Combustion Engine (ACE) R&D Benefit-Cost Study Example, 

Link (2011), where billions of gallons of diesel fuel were saved. (See Example 

B and Figure II-2) 

◙ Example A: In the wind energy study, it was determined there was a six year 

acceleration effect, as determined from interviews with experts. Six was the mean 

value of the distribution of responses. This acceleration led to wind energy supplying 

139.8 billion kWh that displaced electricity otherwise supplied by a fossil-fuel 

generation mix (where the mix is determined from the weighed fuel use of energy 

generation from fossil fuel mix of the States with the largest share of wind energy 

power): 

 

 61.5 billion kWh (i.e., 44% of total from wind) supplied by coal-fired 

generation 

 76.9 billion kWh (i.e., 55% of total from wind) supplied by natural gas-fired 

generation 

 1.4 billion kWh (i.e., 1% of total from wind) supplied by petroleum-fired 

generation. 

This information provided the energy benefits estimate (e.g., power generation from 

wind) and also produced the inputs to the subsequent environmental emissions analysis 

for that benefit-cost study. 
36

Figure II-2. Trend in BTE over Time with Counterfactual 

Source: Link (2010). 

There may be multiple potential approaches to estimating energy benefits in benefit-cost 

studies. It is important to fully document the estimation approach used and the rationale. 

► Determine "additionality" — benefits attributable to EERE's 

investment 

The focus of the study is on estimating the return on EERE’s investment, i.e., the “return 

on public investment.” This entails delineating the part of the benefits from using the 

selected technologies in the cluster — in lieu of the next best alternative — that are 

attributable to the cluster investment. This step, often referred to by the program 

evaluation community as "assessing additionality," takes into account the possibility that 

to some extent development of the selected technologies may have occurred without the 

cluster investment. 

This step to assess additionality—also counterfactual in nature—is an essential step in 

evaluating returns to the designated EERE cluster investment. It considers what likely 

would have happened without the EERE cluster investment. Only if the selected Brake Thermal Efficiency 

◙ Example B: In the ACE R&D study, a statistical approach was adopted for the 

calculation of the fuel savings associated with miles per gallon (MPG) fuel economy 

improvements, where MPG improvements were linked to a 4.5% DOE-supported R&D 

improvement in Brake Thermal Efficiency (BTE) – shown in Figure II-2 (expertderived counterfactual BTE). The change in MPG was statistically estimates (∂MPG / 

∂BTE = 0.153) and the reduction in MPG absent the ACE R&D research was 

calculated. The calculated reduction in MPG was translated to reduced fuel 

consumption of 17.6 billion gallons of diesel fuel saved over the period 1995 to 2007.
37

technologies would not have been developed and applied at all without the EERE cluster 

investment would the entire impact of the technology as computed against the next best 

alternative be appropriately attributed to the EERE cluster investment. In contrast, if it 

appears likely that the selected technology development and application would have 

happened in the same way, in the same timeframe without EERE as with it, then none of 

the benefits of the subject technologies can legitimately be attributed to the EERE cluster. 

These two cases represent the extremes, however, and the answer is more often found 

between these end points. 

The assessments of additionality and next-best alternative are sometimes combined. 

However, the EERE approach calls for explicit treatment of each. Keep in mind that the 

purpose of the next-best-alternative component of the analysis is to determine the benefits 

of having the selected technologies versus not having them. In contrast, the additionality 

component of the analysis is to determine what share of the benefits of having the 

selected technologies is attributable to the R&D Program. 

Table II-4 lists ways that Federal technology investments can cause change to occur. Ask 

what would have been different without the EERE cluster investment. Is it logical to 

expect that it gave rise to any of the effects listed in Table II-4? Are there other types of 

effects, not listed, that may have occurred? Identify and describe the expected 

additionality effects of the selected technology or group of technologies. 

Table II-4. Ways Additionality Can Occur 

Types of Effects Specific Examples of types of additionality 

Acceleration of 

technology development 

& commercialization 

 Expanding R&D funding to achieve technical goals faster 

 Developing supporting research models, data, and designs that 

speed R&D 

 Lowering technical risk and removing other barriers to progress 

 Stimulating market awareness and receptivity 

Improvement of 

technology performance 

characteristics 

 Broadening the scope of R&D effort with larger goals 

 Increasing scale of R&D effort to take on more difficult and 

rewarding technical challenges 

Reduction in the cost of a 

technology or the costs 

associated with its use 

 Fostering of collaborative R&D to avoid investment redundancy 

 Provision of specialized facilities and services needed by an 

industry to make technical advances 

Increase in the size of 

the market and amount 

of use of the technology 

 Reduce barriers to market adoption, e.g., through 

demonstrations, information, training, and standards and 

certification activities 

 Increase access of U.S. firms to global markets 

After identifying how the EERE’s investment appears to have influenced change, 

estimate the extent of the additionality. Document and discuss the evidence that the 

identified additionality effects have occurred, using the tabular framework provided in 

Figure I-6 (See Part I). Designed to assist in organizing and reporting the analysis, the 

use of this framework will help to map attribution to the technology timeline to show 

when each identified effect is estimated to have occurred, to identify and eliminate rival 

explanations, and to indicate the range of attribution to the R&D program in terms of a 
38

percentage share of benefits generated by use of the selected technology or group of 

technologies. 

The objective of the analysis of additionality is to assess and describe the degree of 

influence of the EERE in causing the estimated benefits of the selected technologies in 

comparison with their next best alternative, and to express the degree of influence in 

terms of a percentage share or preferably as a range of percentage shares (or, if 

appropriate, in terms of an acceleration effect, which may be expressed as a range of 

years). The following are suggested terms and associated percentage shares for 

representing different levels of influence: 

 Overwhelming Influence (80-100%) 

 Dominant (60-80%) 

 Very Important (40-60%) 

 Influential (20-40%) 

 None to Minimal (0-20%) 

Table II-5 gives an example for the Geothermal Program’s PDC drill bit technology 

attribution analysis as summarized in the attribution matrix framework. The matrix 

serves as a framework and summary of more detailed attribution analysis findings 

documented in each study report. 

The following conditions are expected to be met in attributing effects to the EERE R&D 

Program in the assessment of additionality:

 The postulated effect must make sense in terms of the R&D Program’s logical theory 

(or logic model18) that relates program mission, strategy, activities, and outputs to 

longer term outcomes. 

 If the Program has made a difference, there should be a corresponding time-order 

change whereby the Program’s actions were taken prior to the observed changes and 

in the direction predicted with a sufficient lead-time to allow the changes to occur. 

This time order change should be documented, as illustrated in the examples below 

taken from the recently completed Solar energy benefit-cost study (O'Connor, et al, 

2010). 

 If there are potential rival explanations of the estimated benefits, these rival 

explanations need to be controlled for or separated from the effect of the selected 

technologies, such that it is the effect of the selected technologies that is identified in 

the additionality assessment and not other causes. Eliminating rival explanations is 

important because otherwise the benefits claimed for the selected technologies within 

the cluster could be due to other factors. There are two kinds of rival explanations 

pertaining to “what others did” (as noted in the attribution matrix framework Figure I6 in Part I: (a) what other programs, private sector, and other nations did, and (b) 

what Federal and State Governments did. 

 

18 See McLaughlin and Jordan (2010). 
39

◙ Table II-5. An example of a completed attribution matrix 

Categories of 

Information 

Needed for 

Additionality 

Assessment 

Technology Timeline (Stage of Research, Development, and Commercialization)

Preliminary 

& Detailed 

Investigation 

Develop 

Components 

Develop 

System Validate/Demonstrate Commercialize Market 

Adoption 

What DOE 

support of 

SNL and 

others did 

 Study 

applicability 

of PDC drill 

bits to 

geothermal 

fields 

 Worked on 

improving 

performance 

of drill bits 

 Financed 

contracts 

and R&D 

efforts with 

GE 

 Conducted 

research on 

drill 

mechanics 

and 

hydraulics 

 Developed 

STRATAPA

X and 

PDCWEAR, 

which helped 

place cutters 

on the drill bit

 Sponsored wear and 

friction tests 

 Helped establish best 

practices 

 Held workshops, 

sponsored 

publications and 

presentations 

 DOE efforts 

helped 

commercializ

e PDC bits 

 DOE 

scientists and 

engineers 

contracted 

with 

consortium of 

drill bit 

manufacturers 

to continue 

improving the 

performance 

of PDC drill 

bits 

What others 

did (rival 

explanations) 

 GE 

developed 

PDC in 1955 

and first 

tested in the 

field 1973 

 GE worked 

on DOE 

contracts 

 GE used 

STRATAPA

X to position 

cutters on 

drill bits 

 Industry used 

PDCWEAR 

to create antiwhirl drill bits

 

Driving/ 

restraining 

policies/ 

government 

forces (rival 

explanations) 

 USGS study 

showed 

availability 

of 

geothermal 

fields 

around 

United 

States 

 Oil crisis, 

U.S. 

government 

studied 

energy 

sources 

alternative 

to fossil 

fuels 

 Oil crisis, 

U.S. 

government 

studied 

alternative 

energy 

sources to 

fossil fuels 

(including 

geothermal) 

  Demand for 

oil went up, 

creating a 

demand for 

offshore 

drilling 

 PDC bits 

became 

enabling 

technology 

for horizontal 

drilling 

widely used 

in offshore 

drilling 

 Federal and 

State Tax 

Credits 

Source: Gallaher, et al. (2010 ) 
40

Table II-5 continued. An example of a completed attribution matrix 

Categories of 

Information 

Needed for 

Additionality 

Assessment 

Technology Timeline (Stage of Research, Development, and Commercialization)

Preliminary & 

Detailed 

Investigation 

Develop 

Components 

Develop 

System Validate/Demonstrate Commercialize Market Adoption 

Description 

of DOE 

influence 

 Very 

Important 

(50%)

 DOE efforts 

helped 

consider 

applications 

of costly PDC 

drill bit 

technology 

 Very 

Important 

(50%)

 DOE 

supported 

the 

technology 

at the time 

when it 

seemed too 

costly and 

unreliable 

 Dominant 

(70%)

 Developed 

analytical 

tools that 

helped 

advance the 

application of 

the 

technology 

 Greatly 

improves 

bonding of 

cutters to drill 

bit 

 Dominant (70%)

 DOE efforts helped 

show that it is 

possible to overcome 

the short-comings of 

PDC drill bit 

technology with 

engineering and 

research 

 Influential 

(25%)

 DOE’s efforts 

helped deliver 

PDC bits 

right before 

there was an 

increase in 

demand for a 

similar 

technology, 

which helped 

the adoption 

of PDC bits 

 Influential 

(25%)

 DOE’s 

expertise 

remained 

available for 

the industry 

to use in their 

own R&D 

efforts 

Basis of 

evidence for 

influence 

 Interviews 

with experts 

 Articles 

 Studies 

 Interviews 

with experts 

 Articles 

 Studies 

 Interviews 

with experts 

 Articles 

 Studies 

 Interviews with 

experts 

 Articles 

 Studies 

 Interviews 

with experts 

 Articles 

 Studies 

 Interviews 

with experts 

 Articles 

 Studies 

The DOE 

effect

 Accelerated 

technology 

entry 

 Improved 

performance 

 Improved 

performance 

 Changed 

costs 

 Improved 

performance 

 Improved 

performance 

Source: Gallaher, et al. (2010 ) 
41

“What other programs, private sector, and other nations” did could include fuel prices; 

activities of other entities, including domestic and foreign companies and universities 

(who did not cost share with DOE), and foreign research laboratories. “What Federal and 

State governments did” would include requirements influencing investments in subject 

technologies). Examples of rival explanation factors related to Federal and State 

government actions: 

o Tax credits (e.g., Production Tax Credit (PTC), Investment Tax Credit 

(ITC)) and other technology market-supporting policies (such as lowinterest loans for renewable technologies) that reduce the effective price of 

the subject technology. 

o Increases in fuel or other prices, which raise the cost of the next best 

alternative disproportionately. 

o Renewable Portfolio Standard (RPS) 

 Discuss sources and degree of uncertainty regarding the percentage or range of 

percentages used to describe the degree of influence of the EERE R&D Program. 

 Findings regarding additionality may differ within a given cluster study for each 

selected technology or group of technologies. Thus, the process of assessing 

additionality is expected to be repeated for each selected technology or technology 

group within a cluster. Attribution shares may range from 0% to 100%. 

 

As part of the extensive and important effort to determine additionality, a study should 

employ multiple lines of data and carefully document the basis of evidence of influence. 

Multiple data could include, for example, 

 Public record, patent citations 

 Interviews with third parties 

◙ The period of analysis for the 2010 Solar benefits-cost study was 1975 to 2008. In that 

study, some of the time order change reported was: 

 Module prices were reduced by a factor of 15, and efficiencies for modules in 

commercial production increased from about 5% to 10%. 

 Reliability improvements sparked by testing at DOE's Flat-Plate Solar Array Project 

(FSA) allowed companies to offer at least 10-year warranties on modules and 

subsequently reached 20 years, whereas before FSA, warranties were nonexistent in 

the PV industry. 

 During the period of DOE's Photovoltaic Manufacturing Technology (PVMaT) 

Project, direct costs of module manufacturing fell from $6.00/Wp (Watts peak) in 

1992 to $2.92/Wp in 2005 (2008$). 

 During the same period, MW capacity increased 18.5 times to 251 MW). 
42

 Interviews with partners 

 Interview with DOE or DOE lab staff 

 R&D cost shares 

 Other data source 

Furthermore, any efforts to utilize interviews to assist with construction of counterfactual 

scenarios or determine attribution should seek to obtain necessary data from both DOE 

partners and third parties who are non-DOE partners. In addition, to the extent possible, 

a distribution of responses/ judgments from expert interviews should be documented and 

the mean value chosen as the point estimate further applied in the study for the purpose 

needed. The distribution of expert responses/ judgments should be documented and 

described in the study report. 

After the additionality analysis has been completed, the attribution effect is applied to the 

energy and other resource savings found relative to the next-best-alternatives. This may 

entail taking a percentage share of the pre-attribution savings, or applying the effects of 

an acceleration to the pre-attribution savings. 

►Provide a separate treatment of energy effects, including fuel types 

and quantities 

While all resources are important to estimating economic benefits, there is a special data 

requirement for energy effects. Differences in the quantity of energy by type associated 

with technology selection not only figure into economic benefits, but also drive two of 

the other categories of benefits—environmental and energy security benefits. (Note that 

the quantity changes in fossil fuel consumption by type are carried forward to Steps 3 and 

4.) Thus it is required that all energy effects by type and physical quantity be broken out 

for separate treatment. 

►Estimate resulting year-by-year dollars of energy, labor, and other 

resource savings for each selected technology or group of 

technologies 

Compile additional data needed to compute year-by-year economic resource savings, 

such as prices and labor rates. 

Estimate year-by-year economic benefits for each selected technology or group of 

technologies, taking into account the combined effect of the next-best alternative 

comparison and additionality assessments. Include the effects of all affected resources—

including energy, labor, and capital. 

If the estimates are in actual (current) dollars, convert yearly current dollars to yearly 

constant dollars as of the stated year--as indicated by the Guide or Addendum to it--that 

EERE has selected for which $1.00=1.00. The Guide obtains the needed Gross Domestic 
43

Product (GDP) Implicit Price Deflators from U.S. Department of Commerce's Bureau of 

Economic Analysis (BEA), at http://www.bea.gov/national/index.htm. These price 

deflators are routinely updated by BEA, and to prevent variations among the benefit-cost 

studies, caused merely by using different price deflator series, this Guide prescribes the 

series of price deflators to be used by all studies conducted within a funding cycle. 

Show the calculated year-by-year economic benefits in tabular form for each separately 

treated technology or group of technologies within a cluster, as well as for each type of 

economic benefit. Show the sum of constant dollar cash flows as undiscounted economic 

benefits. An illustration is provided in Table II-6 from the geothermal benefit-cost study 

(Gallaher, et al, 2010). The table shows the estimation of productivity gains attributed to 

the DOE Geothermal Program due to its contributions to development of reservoir 

models. 

Repeat the above tabular treatment for each separately treated technology or group of 

technologies within the cluster. For example, the referenced geothermal study provided 

separate analyses for four distinct technologies--one from drilling, one from reservoir 

development, one from well preparation, and one from plant technologies. 

Provide an overall summation table that brings together the year-by-year constant dollar 

economic benefits for each technology or group of technologies within a cluster. 

Show the overall undiscounted sum and also the overall PV results for economic effects, 

based on discount rates of 3% and 7%. 
44

◙ Table II-6. Example from the Geothermal Benefit-Cost Study (2010) of the 

Estimation of Productivity Benefits from Use of Reservoir Models 

(1) 

Year 

 

(2) 

Price 

per 

MWh 

of 

Electric

ity 

($2008) 

(3) 

Geothermal 

Electricity 

Generated 

(MWh) 

(4) 

Electricity 

Generated by 

Geothermal 

After TOUGH 

Model for 

Reservoir 

Simulation 

Became Widely 

Used (MWh) 

(5) 

Share of 

Geothermal 

Power 

Generated Due 

to Reservoir 

Modeling--

based on 10% 

share (MWh) 

(6) 

Economic 

Benefits from 

Reservoir 

Modeling (prior 

to attribution 

analysis) 

(thousands of 

$2008) 

(7) 

Economic 

Benefits from 

Reservoir 

Modeling 

(prior to 

deduction of 

capital costs) 

Attributed to 

DOE 

(thousands of 

$2008) 

1979 na 3,888,968 na na 0 $0 

1980 $106.8 5,073,079 1,184,111 118,411 $12,646 $2,909 

1981 $114.2 5,686,163 1,797,195 179,720 $20,524 $4,721 

1982 $119.4 4,842,865 953,897 95,390 $11,390 $2,620 

1983 $118.6 6,075,101 2,186,133 218,613 $25,928 $5,963 

1984 $113.4 7,740,504 3,851,536 385,154 $43,676 $10,046 

1985 $113.5 9,325,230 5,436,262 543,626 $61,702 $14,191 

1986 $111.0 10,307,954 6,418,986 641,899 $71,251 $16,388 

1987 $106.7 10,775,461 6,886,493 688,649 $73,479 $16,900 

1988 $102.8 10,300,079 6,411,111 641,111 $65,906 $15,158 

1989 $100.7 14,593,443 10,704,475 1,070,448 $107,794 $24,793 

1990 $98.7 15,434,271 11,545,303 1,154,530 $113,952 $26,209 

1991 $97.9 15,966,444 12,077,476 1,207,748 $118,238 $27,195 

1992 $96.7 16,137,962 12,248,994 1,224,899 $118,448 $27,243 

1993 $96.1 16,788,565 12,899,597 1,289,960 $123,965 $28,512 

1994 $93.9 15,535,453 11,646,485 1,164,649 $109,360 $25,153 

1995 $91.7 13,378,258 9,489,290 948,929 $87,017 $20,014 

1996 $89.6 14,328,684 10,439,716 1,043,972 $93,540 $21,514 

1997 $87.9 14,726,102 10,837,134 1,083,713 $95,258 $21,909 

1998 $85.5 14,773,918 10,884,950 1,088,495 $93,066 $21,405 

1999 $83.0 14,827,013 10,938,045 1,093,805 $90,786 $20,881 

2000 $83.3 14,093,158 10,204,190 1,020,419 $85,001 $19,550 

2001 $87.2 13,740,501 9,851,533 985,153 $85,905 $19,758 

2002 $84.8 14,491,310 10,602,342 1,060,234 $89,908 $20,679 

2003 $85.8 14,424,231 10,535,263 1,053,526 $90,393 $20,790 

2004 $85.3 14,810,975 10,922,007 1,092,201 $93,165 $21,428 

2005 $88.3 14,691,745 10,802,777 1,080,278 $95,389 $21,939 

2006 $93.5 14,568,029 10,679,061 1,067,906 $99,849 $22,965 

2007 $93.3 14,637,213 10,748,245 1,074,825 $100,281 $23,065 

2008 $98.2 14,859,238 10,970,270 1,097,027 $107,728 $24,777 

Total NB, 

Undiscounted $548,675 

Notes: Cols 2 and 3 sourced from EIA by Gallaher, et al. (2010); Col 4= from Col 3, energy in each year after 1979, 

minus energy generated in 1979; Col 5=Col 4 energy amount x 10%; Col 6= price in Col 2 times the energy amounts in 

Col 5; and Col 7= DOE attribution rates for TOUGH (80%) and for other reservoir model development (20%). 

Increased capital costs due to use of the reservoir models offset some of the benefits. These are not included in Table 

II-6 for greater ease in exposition, but they are included in the benefit-cost study by Gallaher, et al. (2010). 

 
45

►Treat qualitatively any economic effects of the selected 

technologies not captured by the quantitative economic assessment, 

as well as any economic effects realized from the remaining part of 

the cluster 

As noted previously, not all economic effects can be quantified. Some can be quantified 

but with great difficulty or with controversial results, while others can be quantified, but 

not necessarily in dollars. Economic effects not quantified in dollars are omitted from the 

economic return measures. Steps should be taken to include these effects as part of the 

overall findings, such that a judgment can be applied if there are conflicts between the 

results of the quantified economic performance measures and the non-quantified or nonmonetized economic effects. 

When there are known non-quantified or non-monetized effects omitted from the 

economic performance measures, OMB Circular A-4 recommends performing a 

"threshold" analysis to evaluate their significance. Threshold or "break-even" analysis 

addresses the question, "How small or how large could the value of the non-quantified or 

non-monetized effects be before it would offset the economic value from the quantified 

benefits?” 

Providing explicit treatment of energy, environmental, security, and knowledge benefits 

within the EERE benefit-cost framework is expected substantially to reduce this problem 

of important effects being omitted from the analysis. Nevertheless, there may be 

important effects that are not captured in any of the four categories of benefits that should 

be considered. 

Furthermore, the study approach is designed to capture only the benefits of selected 

technologies quantitatively. It is important, therefore, to provide a qualitative treatment 

of remaining elements—at minimum providing evidence that there is nothing in the 

cluster that will likely offset the dollar benefits of the selected technologies, or reduce 

confidence in the results. 
46

3. Estimate Environmental Benefits 

►Bring forward from Section II-2 estimated year-by-year changes in 

fossil energy expressed in physical units, after the next-best 

alternative and attribution analyses; and estimate corresponding 

amounts of air emissions. 

The estimate of environmental benefits began in the previous section with the 

determination of changes in fossil-fuel consumption (i.e., coal, natural gas, and 

petroleum), reflective of the next-best alternative and attribution analyses. The year-byyear changes expressed in physical units by type of fuel are brought forward, and the 

corresponding amounts of air emissions are estimated for each fuel type. 

Estimating emissions avoided may be assisted by the use of emission factors. An 

emissions factor is a representative value of the quantity of a given pollutant released to 

the atmosphere in association with using a designated fuel in a given activity. Emission 

factors are typically expressed as pounds per MWh of electricity produced for different 

generating sources, including natural gas, coal, oil, nuclear energy, municipal solid waste, 

hydropower, and a variety of renewable energy sources. 

Examples of emission factors, provided by the U.S. Environmental Protection Agency 

(EPA) are in Table II-7. They are estimates of average emission rates from various fossil 

fuels used to generate electricity. 
47

Table II-7. Emission Factors, Based on Average Emissions Rates 

from Fossil Fuels Used to Generate Electricity 

Fossil Fuel 

Type 

CO2 SO2 NOx

 Coal 2,249 lbs/1,000 kWh 13 lbs/1,000 kWh 6 lbs/1,000 kWh 

Natural Gas 1,135 lbs/1,000 kWh 0.1 lbs/1,000 kWh 1.7 lbs/1,000 kWh 

Petroleum 1,672 lbs/1,000 kWh 12 lbs/1,000 kWh 4 lbs/1,000 kWh 

Source: EPA-issued emission factors for coal, natural gas, and oil, respectively, were found within the text at 

http://www.epa.gov/cleanenergy/energy-and-you/affect/coal.html; http://www.epa.gov/cleanrgy/energy-andyou/affect/natural-gas.html; and http://www.epa.gov/cleanenergy/energy-and-you/affect/oil.html. 

Note the use of the following abbreviations: carbon dioxide (CO2); sulfur dioxide (SO2), nitrogen oxide (NOX),, 

http://www.epa.gov/cleanenergy/energy-and-you/affect/oil.html 

Software and tools for accessing information on EPA’s air emission factors can be found 

at http://epa.gov/ttn/chief/software. EPA’s eGRID comprehensive inventory of 

environmental attributes of electric power systems provides percentage use of multiple 

energy sources to generate electricity, aggregated data by state, by company, by electric 

grid district, and for the nation. CO2 emission rates can also be found at

http://www.epa.gov/cleanenergy/documents/egridzips/eGRID2010V1_1_year07_GHGO

utputrates.pdf. 

It is recognized that the science of assessing the impacts of various types of emissions 

and of multi-pollutant exposures is still under development, and new and improved 

models of assessment and prediction are emerging. Evaluators are encouraged to obtain 

and use the best available data and approaches for estimating changes in greenhouse gas 

(GHG) and air emissions. 

►Apply EPA's Co-Benefits Risk Assessment (COBRA) Model to 

estimate public health benefits from reduced air emissions in terms 

of mortality, morbidity, and year-by-year dollars of health care 

avoided 

EPA's Co-Benefits Risk Assessment (COBRA) model has been adopted by the EERE 

benefit-cost approach to provide first-order estimates of health effects and the economic 

value of health costs avoided resulting from changes in air emissions. The COBRA 

model has been used by EPA for regulatory analysis, such as for the Regulatory Impact 

Analysis of the Clean Air Interstate Rule (U.S. EPA, 2005).19 Use of the COBRA model 

 

19 For a detailed discussion of studies used for health impact functions and unit values, see U.S. EPA 

(Environmental Protection Agency). (2005, March). Regulatory impact analysis for the final Clean Air 

Interstate Rule. EPA-452/R-05-002. Research Triangle Park, NC: Office of Air Quality Planning and 

Standards; Emission, Monitoring, and Analysis Division and Clean Air Markets Division. Available at 

http://www.epa.gov/cair/pdfs/finaltech08.pdf.
48

enables the health impact functions and the unit economic values used in our benefit-cost 

studies to be consistent with prior EPA analyses. 

The COBRA Model is available on request from EPA, in the form of downloadable 

software and a user manual. (U.S. EPA User’s manual for the Co-Benefits Risk 

Assessment (COBRA) screening model. Developed by Abt Associates Inc., June 2006.) 

The model enables selection of air pollutants by type. The changes in pollutants by type 

are entered either as a percentage reduction (or increase) or as number of tons reduction 

(or increase). If the technology being evaluated is concentrated in several states, specific 

states can be selected within COBRA. If the technology’s application is more 

widespread, the option of using “all states” can be selected. The model displays results in 

terms of change in the number of annual cases of respiratory deaths, illnesses by type, 

and associated costs. 

At the core of the COBRA model is a source-receptor (S-R) matrix that translates 

changes in emissions to changes in particulate matter (PM) concentrations. The changes 

in ambient PM concentrations are then linked to changes in mortality risk and changes in 

health incidents that lead to health care costs and/or lost workdays. Figure II-3 provides 

an overview of the modeling steps. 

Figure II-3. COBRA Model Overview 

Source: EPA (2006) 

In addition to entering changes in emissions, the user identifies the economic sector in 

which the emissions are reduced. The specified economic sector drives the underlying 

spatial distribution of emissions and the characteristics of the affected human population. 

The model then calculates the incidence of human health effects using a range of built-in 

health impact functions and estimated baseline incidence rates for each health endpoint. 

Table II-8 shows the different health endpoints that are provided by the model. 
49

Table II-8. Health Endpoints Included in COBRA 

Health Effect Description/Units 

Mortality Number of deaths 

Chronic bronchitis Cases of chronic bronchitis 

Nonfatal heart attacks Number of nonfatal heart attacks 

Respiratory hospital 

admissions 

Number of cardiopulmonary-, asthma-, or pneumonia-related 

hospitalizations 

Cardiovascular related 

hospital admissions 

Number of cardiovascular-related hospitalizations 

Acute bronchitis Cases of acute bronchitis 

Upper respiratory 

symptoms 

Episodes of upper respiratory symptoms (runny or stuffy nose; wet cough; 

and burning, aching, or red eyes) 

Lower respiratory 

symptoms 

Episodes of lower respiratory symptoms: cough, chest pain, phlegm, or 

wheeze 

Asthma emergency 

room visits 

Number of asthma-related emergency room visits 

Minor restricted 

activity days 

Number of minor restricted activity days (days on which activity is reduced 

but not severely restricted; missing work or being confined to bed is too 

severe to be MRAD). 

Work days lost Number of work days lost due to illness 

Source: COBRA User Manual 

COBRA health effects are modeled individually based on epidemiological studies and 

functional forms, as described in the COBRA user’s manual. For instance, 

 Mortality risk estimates in COBRA are from an epidemiological study of the 

American Cancer Society cohort conducted by Pope et al. (2002). COBRA 

includes different mortality risk estimates for both adults and infants. Reductions 

in the risk of premature mortality are monetized using value of statistical life 

(VSL) estimates. Because of the high monetary value associated with prolonging 

life, mortality risk reduction is consistently the largest health endpoint valued in 

the study. 

 Chronic bronchitis reflects results of a study by Abbey et al. (1995), that found 

statistically significant relationships between PM2.5 and PM10 and chronic 

bronchitis. 

 Nonfatal heart attacks are linked by Peters et al. (2001) to PM exposure. 

 Hospital admissions include respiratory and cardiovascular, and are based on 

Sheppard et al. (1999) findings of asthma hospital admissions associated with 

PM, carbon monoxide (CO), and ozone, and findings of Moolgavkar (2000 and 

2003) and Ito (2003) regarding a relationship between hospital admissions and 

PM. 
50

 Acute bronchitis is modeled based on findings of a study by Dockery et al. 

(1996), that found episodes to be related to sulfates, particulate acidity, and, to a 

lesser extent, PM. 

 Upper respiratory symptoms are modeled on finding of Pope et al. (2002) of a 

relationship between PM and the incidence of a range of minor symptoms. 

 Lower respiratory symptoms are modeled on findings of Schwarz and Neas 

(2000) focused primarily on children’s exposure to pollution from parental 

smoking and gas stoves. 

 Asthma related emergency room visits are primarily associated with children 

under the age of 18, and modeled on findings of a study by Norris et al. (1999) 

that found significant associations between asthma ER visits and PM and CO. 

 Minor restricted activity days (MRAD) in COBRA are based on research by 

Ostro and Rothschild (1989) using a national sample of the adult working 

population, aged 18 to 65, in metropolitan areas. 

 Work loss days are modeled on findings of a study by Ostro (1987) which found 

that 2-week average PM levels were significantly linked to work loss days. 

COBRA translates the health effects into changes in monetary impacts using per-unit 

monetary values described in the COBRA user’s manual. Estimation of the monetary unit 

values vary by the type of health effect. 

COBRA is expected to provide a conservative estimate of the health benefits of reducing 

air emissions for two reasons: (1) COBRA does not include the effects of many pollutants 

that may negatively affect health, and (2) COBRA does not fully capture the economic 

value of health effects of those pollutants that are included in the model. For instance, 

estimation of hospital admissions in dollars is based on cost of illness (COI) units that 

include the hospital costs and lost wages of the individual but do not capture the social 

(personal) value of pain and suffering.

 

Evaluators are to show in tables, first, the detailed mortality and morbidity data and 

associated costs avoidance for a selected year; and, second, the year-by-year heath care 

cost savings, and the undiscounted total. The first table is illustrated by the example in 

Table II-9 produced by application of the COBRA model in the ACE Benefit-Cost Study 

by Link (2010). 
51

◙ Table II-9. Illustration of Health Cost Calculations Results 

from the COBRA Model, Year 2000 

(1) 

Category of Health Benefit 

(2) 

Incidence 

(3) 

Monetary Value of Health Impacts 

(millions $2008) 

Mortality 531 $3,373.2 

Infant Mortality 1 $9.1 

Chronic Bronchitis 357 $158.2 

Non-fatal Heart Attacks 836 $91.9 

Respiratory Hospital Admissions 125 $1.7 

Cardio-vascular Related Hospital Admissions 258 $7.2 

Acute Bronchitis 883 $0.38 

Upper Respiratory Symptoms 7,899 $0.24 

Lower Respiratory Symptoms 10,473 $0.20 

Asthma Emergency Room Visits 466 $0.17 

Minor Restricted Activity Days 438,832 $26.8 

Work Loss Days 74,012 $6.0 

Total $3,675.1 

Source: COBRA model results from Link (2010). 

The example continues in Table II-10, with year-by-year fuel savings, associated air 

pollutants, and monetary value of health impacts for the ACE Benefit-Cost Study (Link, 

2010). 

It should be noted that showing both estimates of changes in health incidents and 

associated health care costs entails double counting. However, these are kept separate in 

the study. The year-by-year health care cost savings are carried forward, together with 

economic benefits, to Step 6, and both are reflected in the bottom-line economic 

performance measures. The health incident data are used to explain what underlies the 

health care cost savings. 

Using COBRA to estimate health benefits in dollar terms from reduced air pollution is 

deemed by experts in the field to provide sufficiently credible monetary estimates to 

warrant the approach of carrying the estimates forward and combining them with 

economic benefits to compute overall measures of economic performance (e.g., net 

present value benefits, benefit-to-cost ratios, and internal rate of return) in Step 6. 
52

◙ Table II-10. Health Benefits from Reduced Environmental Emissions from the ACE 

R&D Sub-Program’s Research on Laser and Optical Diagnostics and 

Combustion Modeling (rounded), year-by-year

(1) 

Year 

(2) 

Reduced Fuel 

Consumption 

with ACE R&D 

Sub-Program’s 

Technologies 

(million gallons) 

(3) 

PM 

(g/hp-hr) per 

EPA 

Regulations 

(4) 

NOx 

(g/hp-hr) per 

EPA 

Regulations 

(5) 

SOx 

(ppm) per EPA 

Regulations 

(6) 

Monetary Value of 

Health Impacts 

(millions $2008) 

1995 1,017 0.1 5.0 500 $2,597.8 

1996 1,040 0.1 5.0 500 $2,681.1 

1997 1,005 0.1 5.0 500 $2,615.8 

1998 1,069 0.1 4.0 500 $2,435.4 

1999 1,426 0.1 4.0 500 $3,278.1 

2000 1,545 0.1 4.0 500 $3,675.1 

2001 1,508 0.1 4.0 500 $3,623.5 

2002 1,469 0.1 2.5 500 $2,735.7 

2003 1,205 0.1 2.5 500 $2,263.4 

2004 1,228 0.1 2.5 500 $2,327.9 

2005 1,609 0.1 2.5 500 $3,078.0 

2006 1,698 0.1 2.5 500 $3,279.0 

2007 1,733 0.01 1.2 500 $1,114.0 

Total 17,552 $35,704.8 

Source: COBRA model; Link (2010). 

 

►Determine GHG reduction impacts 

The primary GHG to be accounted for in the benefit-cost analysis—and the only one 

required—is carbon dioxide (CO2), provided in physical units. It is not monetized in 

current studies subject to this Guide. 

The value of GHG is not expressed in monetary terms because the current value for 

social cost of carbon (SCC) is considered too uncertain. The range of confidence is too 

wide to provide useful quantification as an economic metric. 
53

The current SCC value reported in the United States results from a 2009 effort by an 

interagency team of U. S. government specialists to estimate it. The result was a range of 

values from $5 to $65 per metric ton of carbon dioxide, with a “central value” of $21.20

When further improvements in the estimation of the SCC are made, and the range of 

estimate narrowed, GHG could be included in the benefit-cost studies as an additional 

metric valued in dollars, to be combined with the economic and health care cost savings 

and used to compute economic performance metrics. At this time, the advice of the 

review panel is to report GHG emissions in physical units and avoid increasing the 

degree of uncertainty of the economic performance measures. 

►Describe other notable environmental effects 

If there are other notable environmental effects other than air emissions—e.g., changes in 

water consumption, water discharges, land resource use, and solid waste generation—

provide a qualitative treatment. If quantitative data are available, provide them together 

with a commentary description and explanation. 

For indications of how the use of different energy generating sources may differentially 

affect the water and land environment, evaluators may wish to consult information 

compiled by the EPA. By generating source--such as natural gas, coal, oil, municipal 

solid waste, biomass, land-fill gas, nuclear energy, hydropower, wind, geothermal, and 

solar--effects on water, water discharges, land resource use, and solid waste generation 

are discussed.21

 

20 An economic model, “integrated assessment model” (IAM), was used by the U.S. team to estimate the 

value of the SCC. The model incorporates knowledge from a number of fields of study, such as 

engineering, technology, behavior, and climate science, and uses mathematical formulas to simulate the 

relationships between economic activity and measures to control emissions and the desired environmental 

outcomes. Higher SCC numbers have been estimated by others, such as the UK’s range of $41 – $124 per 

ton of CO2 with a central value of $83. The correct valuation remains controversial. For more 

information, see Interagency Working Group on Social Costs of Carbon (2010), and Bell and Callan 

(2011).

21 As a starting point, one can find such information on-line at www.epa.gov/cleanenergy/energy-andyou/affect/index.html, under the heading "How does electricity affect the environment?" 
54

4. Estimate Energy Security Benefits

Energy security benefits result from reducing disruptions in energy supply. They also 

result from reducing threats to the nation’s energy infrastructure. 

 

The energy security benefits included in the EERE approach take into account the 

following components: 

 Quantitative estimates in physical units of reductions in barrels of oil equivalent 

(BOE) units of displaced imported fossil fuel as a proxy for a reduced dependence on 

imported energy supplies. 

 Qualitative treatment of altered threats to the energy infrastructure. 

►Express the estimated net reduction in fossil energy in terms of 

imported barrels of oil equivalent (BOE) 

This estimation of reduction in BOE is driven by net reductions in fossil energy use in 

physical units, as calculated in Step 2. The estimation is accomplished in the following 

steps: 

(1) Bring forward from Section 2 any estimated net reductions in fossil energy use 

expressed in physical units by type of energy. These fossil fuel estimates should 

be after the next-best alternative and attribution analyses. 

(2) Translate reductions in fossil energy use into barrels of oil equivalent units, taking 

into account the energy source or mix of sources, and aggregating barrels across 

sources. Multiply by the percentage of U.S BOE demand that is imported, to 

reflect the share of BOE reduction associated with U.S. imported supply of oil. 
55

Currently this share is approximately 50%.22 Rather than apply year-by-year 

percentages, evaluators can use the share in the cut-off year for the benefit-cost 

study. 

Barrels of oil equivalent is utilized as a recognizable unit of energy that serves as an 

indicator of national energy security benefits from reducing the quantity of imported oil. 

A barrel of oil equivalent (BOE) is based on the approximate energy released by burning 

a barrel of crude oil, where a barrel is defined as containing 42 gallons (approx. 159 

liters) of crude. In terms of BTUs, one BOE is defined as 5.8 million BTU or 1,700 

kilowatt hours or 1.7 MWh. 

Table II-11 shows BOEs of common fossil fuels, including coal, natural gas, gasoline, 

and diesel fuel. It, or a similar conversion table, may be used to convert the quantity 

reductions in each type of fossil fuel to its BOE. Then, the imported share is taken as 

described above, and the results are summed for total BOE and displayed in the study 

report in tabular format. 

Table II-11. Energy Conversion Table for Estimating BOE 

British 

Thermal 

Units 

(Btu) 

Cubic Feet 

Natural 

Gas 

(CF) 

Short Tons 

Bituminous 

Coal 

(T) 

Gallons of 

Gasoline 

(Gal) 

Gallons 

of Diesel 

(Gal) 

Barrels of Oil 

Equivalent 

(BOE) 

5.8 

Million 

5,642 0.29 46.77 41.73 1.0 

Source: Based on statements of BTU contents of common energy units provided by EIA. 

Note: The heat content of crude oil varies among countries from about 5.6 MBtu per barrel to about 6.3 

MBtu. Thus 5.8 MBtu is a nominal conversion factor widely used in the United States. Similarly the heat 

content of coal varies by type and region. 

Energy security results should also be expressed in units, billions of gallons of gasoline 

equivalent (GGE) and millions of cubic feet of natural gas, for the appropriate fuel. 

►Describe notable effects of the cluster technologies on the nation's 

energy infrastructure 

In some cases, the program cluster evaluated may have implications for the security of 

the nation’s energy infrastructure. For example, use of a distributed renewable energy 

source may reduce the vulnerability of central power plants to disruptions. If so, provide 

a qualitative description of the nature of these effects and provide supporting evidence. 

The monetary valuation of energy security benefits from reductions of BOEs is not 

included in the assessment. It is recognized that associations among changes in energy 

 

22 According to EIA, 60% of U.S. crude oil was imported in 2005, and this share had dropped to 51% in 

2009 and to 49% in 2010. 
56

efficiency, energy supply, energy prices, and security impacts involve many assumptions, 

with causal relationships far more uncertain than for those entailed in estimating 

economic, environmental, and knowledge benefits. 

Although attempts have been made by others to value in dollars the energy security 

benefits of achieving a given reduction in oil consumption, a satisfactory existing 

approach has not been identified for calculating the marginal change in the nation’s 

energy security benefits as a result of specified percentage reduction in equivalent oil 

consumption. 

Thus, attempts at monetary valuation of energy security benefits would be subject to far 

greater margins of error than for the retrospective monetary estimates of economic 

benefits and health care cost savings from avoided air emissions. The introduction of 

greater uncertainty in the monetary valuations would compromise the bottom-line 

benefit-cost results. For this reason, and based on recommendations of the Expert Panel 

which reviewed the Benefit-Cost Evaluation Methodology set forth in this Guide, the 

decision is to avoid monetary estimates of energy security benefits at this time—until an 

improved valuation approach is developed. Until then, reductions in physical units of 

imported BOEs, together with qualitative treatments of infrastructural effects will serve 

as an indicator of energy security benefits. 
57

 5. Estimate Knowledge Benefits 

 

Two motivations drive the estimation of knowledge benefits in these benefit-cost studies: 

(1) Additions to the nation's scientific and technical knowledge bases from R&D have a 

value far beyond the impacts that can be captured in monetary terms. To include 

knowledge benefits as an explicit component of R&D benefits is to recognize the larger 

and enduring impact of R&D on innovation and economic growth. 

(2) One of the more challenging aspects of benefit-cost analysis of public R&D 

programs is adequately assessing attribution. By tracing from specific knowledge outputs 

of these evaluated programs to downstream innovations and commercial developments, 

the influence of the R&D is documented. This analysis, therefore, is expected to 

contribute to multiple lines of evidence in assessing attribution in Step 2. 

►Identify modes of knowledge benefits from the cluster investment 

Knowledge outputs of R&D programs are typically embodied in papers, patents, 

presentations, computer-based algorithms and other computational models and tools, 

prototypes and demonstration systems, test results, and standards; in humans—including 

students, professors, administrators, and researchers; and eventually in goods and 

services. Knowledge benefits may result from both explicit and tacit knowledge outputs. 

An R&D Program may contribute to the development of knowledge networks as EERE 

program researchers interact with and fund those in companies, universities, and other 

organizations in the United States and abroad. 

As the Griliches/Mansfield’s Model (See Part I, Section 2) shows, private returns and 

market spillovers in a targeted industry may result as knowledge drives innovation in 
58

competitive markets. Thus, the benefits of a Program’s knowledge outputs are partially 

captured in a benefit-cost study's estimates of economic, environmental, and security 

benefits. 

The realized and potential benefits of knowledge, however, are likely to extend well 

beyond those captured in a target market. As knowledge flows into, and yields benefits 

in other markets, the spillover gap—i.e., the excess of social returns over private 

returns—continues to expand. 

The separation of knowledge benefits that have already been captured in the monetary 

valuation of other categories of benefits from those not yet captured is complex and is not 

attempted in this benefit-cost approach. Rather, identifying contributions of the cluster 

investment to the scientific and technical knowledge bases both within and outside the 

targeted industry area of application is an analysis goal. As noted above, the result is a 

more comprehensive and quantitative assessment of knowledge benefits than was 

provided by previous benefit-cost studies.23 

The pathways from R&D knowledge outputs to commercial application of these outputs 

are typically long and complex. Use of a historical tracing framework employing 

interviews with program managers, researchers, and company managers; web searches; 

document and database review; bibliometric analysis; and other techniques is helpful in 

identifying and documenting these pathways. There may be multiple decades of DOE 

R&D in a given program cluster area. A variety of approaches will help to avoid missing 

critical connections. The objective is to assess the extent to which the knowledge was 

both created and used to influence downstream applications, and by whom. 

To provide objectively derived, quantitative evidence of linkages that can be developed 

in a relatively non-intrusive manner, patent citation analyses has proven particularly 

useful. Patents have a central role in the innovation system, are consider closer to 

application than publications, and, as noted by Jaffe and Trajtenberg (2005), have been 

used extensively in the study of technological change. Patents are in the public domain 

and search engines can find them. 

Publication analysis has also proven useful in assessing the knowledge benefits of 

program cluster investment. Analysis of coauthoring by program cluster researchers with 

those in other organizations may show linkages through collaborative research; patent-toscientific paper citation analysis shows early influence of laboratory research on 

innovation; and publication-to-publication citation analysis shows pathways of 

knowledge flow through publications. 

 

23 Past benefit-cost studies have typically not separately treated knowledge benefits. An exception was the 

2001 NRC benefit-cost study. However, a limitation of the 2001 NRC study was that it defined knowledge 

benefits as a “catch-all” for situations in which technology development had either failed, was still under 

development with no commercialization in sight, or had succeeded but was expected not to be adopted 

because economic and policy conditions were expected never to become favorable. Hence, the EERE 

approach to knowledge benefits represents a much more comprehensive treatment. 
59

►Construct databases needed for analyzing knowledge creation and 

dissemination 

To perform patent analysis, evaluators will need to construct databases of patents derived 

from the cluster investment. To do this, it is necessary to search Program files, the Office 

of Scientific and Technical Information (OSTI) database, and databases of patent offices, 

such as the U.S. Patent and Trademark Office (PTO), the European Patent Office (EPO), 

and the World Intellectual Property Organization (WIPO), using patent filters. It has 

been found that companies whose R&D has received public funding do not always 

acknowledged the public interest when filing resulting patents, and this omission may 

require searches of DOE annual reports, matching of identified patents with DOE-funded 

company research, and verification with DOE experts. When subprograms are assessed, 

special attention may be needed to identify the set of patents associated with a highly 

focused R&D effort. 

To avoid double counting of patents, evaluators will need to construct "patent families" 

which contain all patents based on an original patent. To perform comparisons of 

patenting intensity resulting from program cluster investment with that of other 

organizations, and to trace from commercial users back to earlier R&D sources, 

evaluators will need to identify relevant organizations who are innovators in the 

technology area of focus and their relevant patent portfolios. To provide country 

comparisons, relevant patent databases will need to be constructed by country of first 

issue. To identify highly significant patents, evaluators will need to construct and use 

citation indices to adjust for technology area and year of issue. 

►Trace pathways of knowledge flows and compare knowledge 

effects attributed to EERE with those of other organizations 

Two main approaches are used to trace patent linkages between program cluster R&D 

and downstream developments. One approach—forward patent tracing—takes a broad 

look at downstream linkages. Its purpose is to determine the influence of program cluster 

patents have had on the development of downstream technologies in all areas. The other 

approach—backward patent tracing—focuses specifically on linkages from downstream 

commercial developments in the targeted industry back to patents attributable to the 

program cluster R&D. These analyses are performed both at the organization level and at 

the individual patent level. 

To test the strength of linkages of program cluster attributed patents to commercial 

applications, perform a backward citation analysis, starting with patents of the leading 

innovators in the field and see to what extent they link back to patents funded by the 

Program cluster, as compared with the extent of linkages to other organizations. This 

approach is illustrated by Figure II-4, which is drawn from the solar PV knowledge 

benefits study by Ruegg and Thomas (2011), performed in support of the solar PV 

benefit-cost study by O'Connor, et al. (2010). 
60

◙ Figure II-4. Percentage of Solar Energy Patent Families of Top U.S. Solar PV 

Producers Linked to Earlier DOE-Attributed Solar PV

 

Patents

Source: Ruegg and Thomas, Solar PV (2011). 

To determine linkages through patents attributed to the program cluster to not only the 

targeted industry but beyond to other industries, perform a forward citation analysis, 

starting with program-attributed patents and identifying subsequent patents that link back 

to these earlier program-attributed patents. An illustration of forward tracing at the 

organizational level is provided by Figure II-5, which is also drawn from the solar PV 

knowledge benefits study by Ruegg and Thomas (2011), in support of the solar PV 

benefit-cost study by O'Connor, et al. (2010). It highlights influence of the EERE's solar 

PV research beyond U.S. leading PV producers to leading companies in the 

semiconductor industry. 
61

◙ Figure II-5. Organizations from All Industry Areas with the Largest Number of 

Patent Families from All Technologies Linked to Earlier DOE-Attributed PV Patents

 Source: Ruegg and Thomas, Solar PV (2011).

►Identify notable knowledge outputs and innovations attributed to or 

linked to the program cluster 

A concept useful in tracing knowledge flows is that highly cited patents (i.e., patents 

cited by many later patents) tend to contain technological information of particular 

importance. A patent that forms the basis for many new innovations tends to be cited 

frequently by later patents.24 

 

24 This does not mean that every highly cited patent is important, or every infrequently cited patent is unimportant, but, 

research studies have shown a correlation between the rate of citations of a patent and its technological importance. 
62

An example of notable (i.e., highly cited) PV innovations of companies linked back to 

EERE-attributed PV patents is given in Table II-12, drawn from the Ruegg and Thomas 

knowledge benefits study performed in support of the 2010 Solar PV Benefit-Cost Study 

by O'Connor, et al, 2010. The Citation Index adjusts for the type of technology and for 

the age of the patent, such that, for example, the Index value of 4.52 in the table's first 

row means that this patent (#4419533) has been cited approximately 4.5 times more often 

than would be expected of a patent of its age, within its technology area. 

◙ Table II-12. Highly Cited Solar Energy Patents of Top U.S. PV Producers Linked to 

Earlier DOE-Attributed PV Patents

Patenta Issue 

Date 

# Cites 

Received 

Citation 

Index 

Assignee Title 

4419533 1983 47 4.52 ECD Photovoltaic device having incident 

radiation directing means for total 

internal reflection 

5164019 1992 51 4.08 SunPower Monolithic series-connected solar 

cells having improved cell isolation 

and method of making same 

6534703 2003 12 3.12 SunPower Multi-position photovoltaic 

assembly 

6111189 2000 19 2.93 BP Photovoltaic module framing 

system with integral electrical 

raceways 

6353042 2002 12 2.92 Evergreen 

Solar 

UV-light stabilization additive 

package for solar cell module and 

laminated glass applications 

6570084 2003 11 2.86 SunPower Pressure equalizing photovoltaic 

assembly and method 

4514583 1985 31 2.82 ECD Substrate for photovoltaic devices 

4419530 1983 28 2.69 ECD Solar cell and method for producing 

same 

5746839 1998 30 2.64 SunPower Lightweight, self-ballasting 

photovoltaic roofing assembly 

Source: Ruegg and Thomas, Solar PV (2011). 

Patent-to-publication citation analysis can be used to identify when a subsequent 

technological development has drawn more directly on a scientific base. Thus an 

extended feature of the patent analysis is to assess program-cluster papers and 

publications cited by patents as prior art. An example is provided by Table II-13, drawn 

from the knowledge benefits study by Ruegg and Thomas (2011).

Publication-to-publication citation analysis can also indicate notable technologies, and 

show pathways of knowledge dissemination. 
63

◙ Table II-13. EERE Geothermal Paper/Publications Linked to the largest Number of 

Patent Families through Two Generations of Citations

# 

Linked 

Patents 

DOE Papers/Publications 

203 "Interfaces and Mechanical Behaviors of Fiber-Reinforced Calcium Phosphate 

Cement Compositions," by T. Sugama, et al., prepared for the Geothermal Division 

U.S. Department of Energy; Department of Applied Science (June 1992) 

197 "Microsphere-Filled Lightweight Calcium Phosphate Cements," by Sugama, T., et 

al., U.S. Department of Energy, Washington, D.C. under contract No. DE-AC02-

76CH00016 (December 1992) 

197 "Hot Alkali Carbonation of Sodium Metaphosphate Fly Ash/Calcium Aluminate 

Blend Hydrothermal Cements," by T. Sugama, Cement and Concrete Research 

Journal, vol. 26, No. 11, pp. 1661-1672 (1996) 

192 "Calcium Phosphate Cements Prepared by Acid-Base Reaction," by Sugama, T. et 

al., Journal of the American Ceramic Society, vol. 75, No. 8, p. 2076-2087 (August 

1992) 

185 "Carbonation of Hydrothermally Treated Phosphate-Bonded Calcium Aluminate 

Cements," by T. Sugama, et al., U.S. Department of Energy, Washington, D.C. 

under contract No. DE-AC02-76CH00016 (Undated) 

108 "Use of Single-Cutter Data in the Analysis of PDC Bit Designs: Part 1--

Development of a PDC Cutting Force Model," by Glowka, D.A., JPT, pp. 797-799, 

844-849 (August 1989) 

105 "Use of Single-Cutter Data in the Analysis of PDC Bit Designs: Part II--

Development and Use of PDCWEAR Computer Code," by Glowka, D.A., JPT, pp. 

850-859 (August 1989) 

101 "Acoustical Properties of Drill Strings," by Drumheller, D., The Journal of the 

Acoustical Society of America, No. 3, New York, pp. 1048-1064 (March 1989) 

56 "The Propagation of Sound Waves in Drill Strings," by Drumheller, D., et al., The 

Journal of the Acoustical Society of America, No. 4, pp. 2116-2125 (April 1995) 

37 "Acoustical Properties of Drill Strings," by Drumheller D, Sandia National 

Laboratories, SAND88 0502 (August 1988) 

32 Sourcebook on the Production of Electricity from Geothermal Energy, Kestin, J., 

editor, Publication No. DOE/RA/4051, Chap. 4, p. 536 (1980) 

Source: Ruegg and Thomas, Geothermal (2011). 
64

►Analyze knowledge dissemination through collaborative research 

Linkages from the R&D of program clusters are also assessed and demonstrated through 

the analysis of publication coauthoring, and the identification of networks that have 

formed among individuals, organizations, associations, and other groups involved in an 

R&D effort and the use of R&D outputs. 
65

6. Calculate Measures of Economic Performance and 

Summarize Other Effects 

►Combine year-by-year dollar benefits from economic resources and 

health cost effects and compute economic performance measures, 

following conventions stated in the Guide. 

Bring forward the year-by-year series of economic benefits (undiscounted) from Step 2 

and the year-by-year series of health care benefits (undiscounted) from Step 3. Show the 

year-by-year series separately and also combined. Table II-14 gives an example from a 

recent benefit-cost study. 

Calculate the economic performance measures for the two data series separately and 

combined, using the formulas given in Part I, Figure I-2. 

Spreadsheet functions or similar computational tools can be used to calculate the 

economic performance measures, provided their use satisfies the set of conventions listed 

in Table II-15. (An EERE goal is that consistency be followed across the set of EERE 

benefit-cost studies.) 

An example of the bottom-line economic performance measures is given in Table II-16 A 

and B, from the ACE R&D Benefit-Cost Study by Link (2010). 
66

◙ Table II-14. Economic Benefits of Reduced Fuel Consumption, Monetary 

Value of Health Impacts, and Total economic benefits from the 

2010 ACE R&D benefit-cost study 

Year Dollar Value of 

Reduced Fuel 

Consumption 

(millions $2008) 

Monetary Value 

of Health Impacts 

(millions $2008) 

Total Economic 

Benefits (millions 

$2008) 

1995 $1,502.0 $2,597.8 $4,099.8 

1996 $1,683.7 $2,681.1 $4,364.8 

1997 $1,547.3 $2,615.8 $4,163.1 

1998 $1,410.7 $2,435.4 $3,846.1 

1999 $1,996.8 $3,278.1 $5,274.9 

2000 $2,817.2 $3,675.1 $6,492.3 

2001 $2,526.5 $3,623.5 $6,150.0 

2002 $2,283.6 $2,735.7 $5,019.3 

2003 $2,097.7 $2,263.4 $4,361.1 

2004 $2,491.7 $2,327.9 $4,819.6 

2005 $4,189.2 $3,078.0 $7,267.2 

2006 $4,834.5 $3,279.0 $8,113.5 

2007 $5,115.4 $1,114.0 $6,229.4 

Total $34,496.4 $35,704.8 $70,201.1 

 Source: Link, 2010 
67

Table II-15. Summary of Conventions for Computing 

Economic Performance Measures 

 Discount Rate: 7% and a 3% real discount rates are to be used per OMB Circulars 

A-94 and A-4 pertaining to benefit-cost analysis, respectively, and the sensitivity of 

economic performance results will be displayed for 0% (undiscounted case), 3%, and 

7% discount rates. 

 Base Year (time to which all cash amounts are converted in a present value analysis): 

The base year to be used marks the onset of cash flow, which for the cluster 

economic performance measures is the year in which the investment in the 

technology cluster began. 

 Cash-Flow Modeling Conventions: Cash flows are to be expressed annually and 

can be modeled as though they occur at the end of the year. A common practice in 

cash flow analysis is to model investment costs as occurring at the beginning of each 

year and benefits net of operating and maintenance costs as occurring at the end of 

each year. However, built-in formulations of popular spreadsheet program tend not 

to make this distinction automatically and require that the Evaluator to adjust the 

designated timing of investment cash flows, which is simple to do. Use of either 

cash-flow modeling convention is acceptable for these benefit-cost studies—that is, 

modeling all cash flows at the end of the year in which they occur, or, alternatively, 

modeling investment costs at the beginning of the year in which they occur and 

benefits net of operating and maintenance costs at the end of the year in which they 

occur. Alternatively a mid-year convention can be used. However, the study should 

indicate explicitly the cash-flow modeling convention is has used. 

 Constant Dollars: All cash flows are to be converted to constant dollars as of the 

designated year for which $1.00 = 1.00 as found in the list of year-by-year GDP price 

deflator indices issued in the most recent Addendum to this Guide. (For the four 

EERE Benefit-Cost studies published in 2010, all current dollars were converted to 

constant 2008 dollars.) 

 Present Values: All constant dollar cash flows are to be adjusted to equivalent 

present value amounts as of the base year (i.e., as of the year which marks the onset 

of cash flow). 

 Economic Performance Measures: The three measures —NB, B/C, and IRR—in 

Figure I-2 and discussed in the section following are to be used to estimate the return 

on EERE's program investment in a defined technology cluster. 
68

◙ Table II-16A. ACE R&D Sub-Program and CRF Costs and Economic Benefits 

Associated with the ACE R&D Sub-Program’s Research in Laser and Optical 

Diagnostics and Combustion Modeling 

(1) 

Year 

(2) 

Costs: ACE R&D 

Sub-Program 

(millions $2008) 

(3) 

Costs: Combustion 

Research Facility 

(millions $2008) 

(4) 

Total ACE R&D Costs 

(millions $2008) 

(5) 

Total Economic Benefits 

(millions $2008) 

1986 $27.402 $5.602 $33.004 – 

1987 $29.005 $5.930 $34.935 – 

1988 $27.785 $5.680 $33.465 – 

1989 $26.525 $5.423 $31.948 – 

1990 $25.929 $5.588 $31.517 – 

1991 $22.869 $6.240 $29.109 – 

1992 $23.611 $6.223 $29.834 – 

1993 $20.550 $6.073 $26.623 – 

1994 $17.587 $5.665 $23.252 – 

1995 $13.890 $5.549 $19.439 $4,099.8 

1996 $21.574 $6.154 $27.728 $4,364.8 

1997 $24.714 $6.743 $31.457 $4,163.1 

1998 $23.239 $6.547 $29.786 $3,846.1 

1999 $46.230 $6.281 $52.511 $5,274.9 

2000 $57.211 $5.796 $63.007 $6,492.3 

2001 $62.475 $6.538 $69.013 $6,150.0 

2002 $55.538 $6.332 $61.870 $5,019.3 

2003 $63.714 $6.842 $70.556 $4,361.1 

2004 $59.119 $6.605 $65.724 $4,819.6 

2005 $52.593 $6.983 $59.576 $7,267.2 

2006 $42.649 $6.567 $49.216 $8,113.5 

2007 $49.379 $7.811 $57.190 $6,229.4 

Total $793.59 $137.17 $930.76 $70,201.1 
69

◙ Table II-16B. Example of Evaluation Performance Metrics 

Calculated from the Combined Economic and Health Care Benefits and the 

ACE R&D Subprogram Costs

Metric 7% Discount Rate 3% Discount 

Rate 

Internal Rate of 

Return 

Present Value of Net Benefits 

(billions $2008) 

$23.1 $42.6 

Benefit-to-Cost Ratio 53 to 1 66 to 1 

Internal Rate of Return 63% 

 Source: Link (2010). 

►Characterize the extent to which the EERE cluster investment has 

been worthwhile from the standpoint of monetized benefits and costs 

and in terms of the other effects considered 

The tests for using the various economic performance metrics to determine if an 

investment has been worthwhile are noted in Part I, section 2, ("Economic Performance 

Measures"). These simple tests are that the NPV is positive; that the BCR is greater than 

one; and that the IRR is greater than the required rate of return (as indicated by the OMBspecified discount rate). The extent to which an investment has been economically 

worthwhile is signaled by the size of these measures—the larger the measure, the more 

economically worthwhile the investment has been up to the evaluation cut-off year, 

others factors being equal. However, it is to be remembered that these economic metrics 

are based on partial benefits. Their implications for how worthwhile an investment has 

been are conservative and need to be conditioned by the other impacts which were not 

included in the metrics. 
70

7. Perform Sensitivity Analysis 

Previously acknowledged is that multiple discount rates are required by the Guide to 

calculate the economic performance measures: discount rates of 0%, 3%, and 7% (See 

Part 2, Step 2). In effect, use of the multiple rates will show sensitivity of the results to 

the discounts rate. 

Some other input variables may have been expressed using multiple values, such as a 

range of possible values. Other input variables may have higher degree of uncertainty 

surrounding them. Uncertainties in the evaluation need to be explicitly acknowledged 

and treated. 

►Identify and discuss areas of major uncertainty in the analyses, 

such as the following: 

 When a range of implied or explicit values were obtained for an input variable; 

 When an alternative estimation approach could have been justified for use, and its use 

is expected to have produced different results than the approach taken; and 

 When a given input variable or assumption is expected to have a large effect on 

outcome, and there is uncertainty about it. 

►Select areas for which sensitivity analysis will be performed, and, 

for each, substitute alternative values for uncertain input variable and 

calculate results 

 To test sensitivity of results to a range of input values, use the low and high ends of 

the range to generate a range of outcomes. (See Example A below, Table II-17) 
71

 To test the sensitivity of results to a different estimation approach for a key input 

variable, go back to the stage of the analysis in which the key input first appears and 

recalculate it using the alternative approach; then feed the results through all 

subsequent affected calculations. (See Example B below, including Tables II-18 and 

II-19) 

◙ Sensitivity Analysis--Example A: Table II-17. Sensitivity to Variation in 

Acceleration Effect and Discount Rate

Measure 12-year Acceleration 

Effect

10-year Acceleration 

Effect

15-year Acceleration 

Effect

Net benefits (billion 2008$) $15.03 $10.68 $22.17 

Internal rate of return 17% 14% 20% 

NPV @ 7% (billion 2008$; base year = 

1975) $1.46 $0.86 $2.39 

Benefit-to-cost ratio @ 7% 1.83 1.49 2.37 

NPV @ 3% (billion 2008$; base year = 

1975) $5.72 $3.99 $8.53 

Benefit-to-cost ratio @ 3% 3.24 2.56 4.35 

 Source: O'Connor, et al. (2010) 
72

◙Sensitivity Analysis--Example B: Test the Sensitivity 

of Results to a Different Estimation Approach 

The test is for sensitivity to the way the energy savings were calculated. Initially they 

were calculated based on a statistical relationship found between brake thermal efficiency 

(BTE), a measure of fuel efficiency, and resulting miles per gallon (MPG), based on the 

period 1970-2007, and this relationship was assumed to apply to the period 1995-2007. 

In the initial analysis, this statistical relationship resulted in 17.6 billion gallons of diesel 

fuel oil saved from 1995 through 2007, with an IRR of 63%. 

The alternative approach assumed that new heavy-duty diesel trucks would each year 

consume a proportionate amount of fuel each year and that proportion would remain 

constant over time. The sensitivity analysis showed that using the alternative method of 

calculating energy savings resulted in 15.1 billion gallons of diesel fuel oil saved, and an 

IRR of 50%. Results on energy savings and on the bottom-line economic performance 

measures from using the alternative approach to estimating energy savings are shown in 

Tables II-18 and II-19.

Table II-18. Sensitivity Testing of Reduced Fuel Consumption from the 

ACE Subprogram R&D Using an Alternative Method (Link, 2010) 

Year Reduced Fuel 

Consumption 

with 

ACE R&D 

SubProgram’s 

Technologies 

(million 

gallons) 

Average 

Retail Price 

Diesel Fuel 

(per gallon) 

Dollar Value of 

Reduced Fuel 

Consumption 

(millions $) 

GDP Implicit 

Price 

Deflator 

(2008=100) 

Dollar Value of 

Reduced Fuel 

Consumption 

(millions $2008) 

Dollar Value 

of Health 

Impacts 

(millions 

$2008) 

1995 169.2 $1.11 $187.85 75.160 $249.94 $432.2 

1996 318.5 $1.24 $394.89 76.591 $515.59 $821.1 

1997 475.1 $1.20 $570.06 77.943 $731.38 $1,236.6 

1998 651.0 $1.04 $677.03 78.824 $858.91 $1,483.1 

1999 874.4 $1.12 $979.28 79.983 $1,224.37 $2,010.1 

2000 1,067.6 $1.49 $1,590.67 81.715 $1,946.61 $2,539.5 

2001 1,197.0 $1.40 $1,675.86 83.561 $2,005.55 $2,876.2 

2002 1,314.9 $1.32 $1,735.61 84.915 $2,043.94 $2,448.7 

2003 1,437.8 $1.51 $2,171.05 86.742 $2,502.88 $2,700.7 

2004 1,595.4 $1.81 $2,887.74 89.203 $3,237.27 $3,024.4 

2005 1,809.2 $2.40 $4,342.18 92.180 $4,710.54 $3,461.0 

2006 2,038.1 $2.71 $5,523.29 95.183 $5,802.81 $3,935.8 

2007 2,171.8 $2.89 $6,276.54 97.908 $6,410.65 $1,396.1 

Total 15,119.9 $32,240.44 $28,365.5 
73

◙ Example B, continued. 

Table II-19 Evaluation Metrics for ACE Subprogram 

Re-Calculated to Show Sensitivity to the Alternative Assumption for Calculating 

Energy Savings in Table II-18 

 

Metric 7% Discount Rate 3% Discount Rate Internal Rate of 

Return 

Present Value of Net Benefits 

(billions $2008) 

$17.8 $35.0 

Benefit-to-Cost Ratio 41 to 1 54 to 1 

Internal Rate of Return 50% 

 Source: From Link (2010) 

Note that the sensitivity testing in Example B does not say which estimation approach is 

better for estimating diesel fuel cost savings; it does show the results based on either 

calculation approach are quite strong. 
74

8. Report Results 

Goals to be achieved by Step 8 are that evaluators: 

 Produce benefit-cost reports, and supporting study outputs, that will serve as 

effective communication tools with diverse program stakeholders about the study, 

its findings, and its implications. 

 Maintain best practices and a level of consistency with other benefit-cost 

studies/reports that have similar purpose—while reflecting the unique 

requirements and data availability issues that evaluators typically encounter 

across studies. 

 Produce reports whose input data, assumptions, and calculations are sufficiently 

transparent that they can be replicated and verified by others.

 Present data that can be added to a meta data infrastructure for EERE's benefitcost studies which will allow for multiple, discrete packages of study data that are 

linked to allow aggregate calculations and that inform program data collection 

plans and activities. 

► Ensure the study report provides essential content characteristics 

by following the “Contents Checklist” found in Table II-20 
75

Table II-20. Contents Checklist 

Use this column 

to rate each 

content element 

as: 

_Found 

_Not Found 

_Adequate 

_Inadequate 

Content Topics Essential Characteristics 

 Statement of Study Design, 

Objectives, Approach, Nature 

of Findings; and Impacts 

Included and Excluded 

 Benefit-cost framework 

 Cluster approach 

 Statement of evaluation objectives 

 Retrospective analysis with cut-off 

year and no life-cycle benefits 

counted 

 Conceptual models based in theory 

used 

 Focus on public returns only 

 All findings are evidence-based 

 Conservative, lower-bound 

estimates of findings, with reasons 

why 

 Clear delineation of all categories 

of impacts included--those with 

monetary valuation, those with 

other quantitative valuation, and 

those with qualitative valuation 

 Identification of effects excluded 

from the evaluation 

 Description of Program Cluster 

Overall and Selected 

Technologies for Detailed 

Treatment 

 How and why program cluster was 

selected 

 Overall description of cluster 

 Description of selected 

technologies for detailed treatment, 

and rationale for selection 

 Method of treatment for each 

technology (or group) separately 

assessed 

 Discussion of other elements in the 

cluster (not included for detailed 

treatment) and their likely impact 

on economic returns 

 Cost: year-by-year total cluster 

costs and year-by-year costs of the 

selected technologies 
76

 Next-best Alternative  Designation for each selected 

technology (or group) individually 

assessed 

 Description of resulting baseline 

used in estimating differential 

effects of each technology (or 

group) 

 Explicit treatment of next-best 

alternative, separate from 

attribution assessment 

 Attribution Assessment  Assessment of context in which the 

program operated and external 

influences that may constitute rival 

explanations of outcomes 

 Fully documented attribution 

matrix for each technology (or 

group) individually treated 

 Timeline of relevant developments 

 Treatment of rival explanations of 

outcome 

 Clear representation of attribution 

level, such as by % share of 

differential effect for each 

technology (or group) attributable 

to the public program cluster 

 Explicit treatment of attribution, 

separate from next-best alternative 

assessment 

 Data Quality, Collection Tools, 

Uncertainties, and Exposition 

 Use of valid protocols and 

procedures in data collection 

 Statement of critical assumptions 

 Inclusion of all data used in the 

analysis 

 Identification of data sources 

 Identification of uncertainties, and 

data distributions where relevant 

 Inclusion of interview and survey 

tools 

 List of interviewees 

 Estimation of Each of 4 

Category of Benefits--(1) 

Energy & Other Economic 

Benefits, (2) Environmental 

Benefits, (3) Energy Security 

Benefits, and (4) Knowledge 

 Systematic and transparent 

analyses, with all steps 

documented, and approach/results 

replicable 

 Credible treatment of (and 

explanation thereof) of both 

demand and supply sides of the 

analysis, with fully documented 

assumptions 
77

 Separate treatment of: 

Energy Effects, including types of 

energy and physical quantities 

 

 Calculation of Economic 

Performance Measures 

 Measures include NB 

(undiscounted), NPV based on 3% 

and 7% discount rates, BCR based 

on 3% and 7% discount rates, and 

IRR 

 These measures are separately 

calculated for: 

(1) Energy & Other Economic 

benefits 

(2) Combined Economic and 

Health Cost Avoidance 

 Sensitivity Analysis  Performed to highlight effects of 

uncertain or controversial variables, 

assumptions, and estimation 

methods 

 Overall Conclusions  Summary of evidence-based 

findings 

 Implications of findings 

 Indication that all stated evaluation 

objectives have been achieved 

 Identification of study limitations 

►Ensure the study report provides essential format characteristics 

by following the "Format Checklist" found in Table II-21 

Table II-21. Format Checklist 

Use this 

column to 

rate each 

report 

component as: 

_Found 

_Not Found 

_Adequate 

_Inadequate

Report Components Description 

 

 Title Page  Title 

 Date 

 Prepared by... 

 DOE cover design 
78

 Preface (if desired by 

DOE) 

DOE prepared, e.g., description of mission, 

objectives, programs, rationale for public 

investment, and purposes of retrospective impact 

evaluation 

 Acknowledgements Contributors and reviewers 

 Notice DOE prepared 

 Executive Summary  Written for audience of diversion 

backgrounds, designed to communicate 

quickly and concisely the most important 

findings, conclusions, and implications 

 Specific inclusion of overall results 

summation table including the metrics 

included in Table I-1 of this Guide 

 Table of Contents and 

lists of Tables and Figures

 3-levels of headings for TOC 

 Electronically keyed to report sections to 

facilitate easy movement of the reader 

through the report. 

 Main Body of the Report  All elements of essential contents as 

outlined in Table II-20 

 Separate Sections on each of the 4 

categories of benefit : 

o Energy & Other Economic 

Benefits 

o Environmental Benefits (including 

GHG & Health Care Cost 

Savings) 

o Energy Security Benefits 

o Knowledge Benefits 

 References listing of references cited in the report; not a 

general reading list 

 Appendices/Attachments Supporting information that can be moved out of 

the main body of the report for improved 

readability, but that is strongly germane to the 

presentation and likely desired by certain readers 

 Index, List of Terms, List 

of Abbreviations 

Discretionary 
79

►Ensure the study report provides essential presentation 

characteristics according to the "Presentation Checklist" found in 

Table II-22 

Table II-22. Presentation Checklist 

Use this column to rate the report's presentation 

according to each essential characteristic as: 

_Adequate 

_Inadequate 

Essential Presentation 

Characteristics 

 Concise, clear, transparent exposition 

 Rigor demonstrated in data collection, 

analyses, and interpretation 

 Document is accurate and reliable; free 

of errors of fact or logic 

 Findings are objectively derived, 

testable, and reproducible 

 Study has internal and external validity 

 Study has credibility among 

stakeholders 

►Use these Content, Format, and Presentation Checklists 

throughout the study 

The checklists are intended for use by evaluators and by project managers on an on-going 

basis to keep the report development on track. They may also be used by report 

reviewers. 
80

References 

Bell, Ruth Greenspan, and Dianne Callan (2011) "More than Meets the Eye: The Social 

Cost of Carbon in U.S. Climate Policy, in Plain English, Policy Brief" (World 

Resources Institute). 

Butry, D. T.; Brown, M. H., Fuller, S. K. (2007) Benefit-Cost Analysis of Residential Fire 

Sprinkler Systems, NISTIR 7451 (Gaithersburg, MD, National Institute of 

Standards and Technology). 

Gallaher, Michael, A. Rogozhin, and J. Petrusa (2010) Retrospective Benefit-Cost 

Evaluation of U.S. DOE Geothermal Technologies Program Investments: Impacts 

of a Cluster of Energy Technologies (Washington DC: U.S. Department of 

Energy. 

Griliches, Z. (1958). "Research Costs and Social Returns: Hybrid Corn and Related 

Innovations," Journal of Political Economy, 66: 419-431. 

Jaffe, A. (1996) Economic Analysis of Research Spillovers: Implications for the 

Advanced Technology Program. NIST GCR 97-708 (Gaithersburg, MD: National 

Institute of Standards and Technology) Available at 

http://www.atp.nist.gov/eao/gcr708.htm. 

Jaffe, A. and Trajtenberg (2005). Patents, Citations, and Innovations: A Window on the 

Knowledge Economy. MIT Press (Originally published in 2002; reissued in 2005). 

Interagency Working Group on Social Costs of Carbon (2010) "Social Cost of Carbon for 

Regulatory Impact Analysis Under Executive Order 12866." In Final Rule 

Technical Support Document (TSD): Energy Efficiency Program for Commercial 

and Industrial Equipment: Small Electric Motors. Washington, DC: U.S. 

Department of Energy, Appendix 15A. Available at 

http://www1.eere.energy.gov/buildings/appliance_standards/commercial/pdfs/sma

llmotors_tsd/sem_finalrule_appendix15a.pdf. 

Link, A.N. (2010) An Economic Evaluation of the U.S. Department of Energy's 

Investments in Vehicle Combustion Engine R&D. (Washington, D.C.: U.S. 

Department of Energy). 

Mansfield, E., J. Rapoport, A. Romeo, S. Wagner, and G. Beardsley (1977) "Social and 

Private Rate of Return from Industrial Innovation," Quarterly Journal of 

Economics, 91: 221-240. 

Mansfield, E. (1996). Estimating Social and Private Returns from Innovations Based on 

the Advanced Technology Program: Problems and Opportunities. NIST GCR 
81

99-780 (Gaithersburg, Maryland: National Institute of Standards and 

Technology). 

Marshall, Harold (1996) "Sensitivity Analysis", in Dorf, R. (editor) The Engineering 

Handbook (CRC Press, pp. 1962-1967). 

McLaughlin, John A. and Jordan, Gretchen B. (2010, 3rd edition), "Logic Models," in 

Wholey, Hatry, and Newcomer (editors), Handbook of Practical Program 

Evaluation (Jossey-Bass). 

National Research Council, Committee on Benefits of DOE R&D on Energy Efficiency 

and Fossil Energy (2001) Energy Research at DOE: Was it worth it? Energy 

Efficiency and Fossil Energy Research 1978 to 2000 (Washington, DC: The 

National Academies Press, 2001. Available on-line at 

http://www.nap.edu/openbook.php?isbn=0309074487. 

 

O'Connor, Rowe, Gallaher, Sevinsky, and Wood (2007) Economic Impact of ATP's 

Contributions to DNA Diagnostics Technologies, GCR 06-898 (Gaithersburg: 

MD, National Institute of Standards and Technology). Available at 

http://www.atp.nist.gov/eao/gcr06-898.pdf. 

O'Connor, Alan, R. Loomis, and F. Braun (2010) Retrospective Benefit-Cost Evaluation 

of DOE Investment in Photovoltaic Energy Systems, DOE/EE-0349 (Washington 

DC: U.S. Department of Energy. 

Pelsoci, Thomas M. (2005) Photonics Technologies: Applications in Petroleum Refining, 

Building Controls, Emergency Medicine, and Industrial Materials Analysis;, 

Economic Analysis of a Cluster of ATP-Funded Projects, NIST GCR 05-879 

(Gaithersburg, MD: National Institute of Standards and Technology). Available at 

http://www.atp.nist.gov/eao/gcr05-879/contents.htm. 

Pelsoci, Thomas M. (2010) Retrospective Benefit-Cost Evaluation of U.S. DOE Wind 

Energy R&D Program: Impact of Selected Energy Technology Investments, U.S. 

Department of Energy, DOE/EE-0348 (Washington, D.C.: U.S. Department of 

Energy.) 

Ruegg, Rosalie (1996) "Risk Assessment," in Dorf, R. (editor) The Engineering 

Handbook (CRC Press, pp. 1953-1961). 

Ruegg, Rosalie and Irwin Feller (2003) A Toolkit for Evaluating Public R&D Investment 

Models, Methods, and Findings from ATP's First Decade (Gaithersburg, MD: 

National Institute of Standards and Technology). Available at 

http://www.atp.nist.gov/eao/gcr03-857/contents.htm. 

Ruegg, R., and P. Thomas (2011) Linkages of DOE’s Geothermal R&D to Commercial 

Power Generation, DOE/EE-0382 (Washington, D.C.: U.S. Department of 

Energy). 
82

Ruegg, R., and P. Thomas (2011) Linkages of DOE’s Solar Photovoltaic R&D to 

Commercial Renewable Power from Solar Energy, DOE/EE-0403 (Washington, 

DC: U.S. Department of Energy). 

Ruegg, R., and P. Thomas (2009) Linkages from DOE’s Wind Energy Program R&D to 

Commercial Renewable Power Generation (Washington, DC: U.S. Department of 

Energy). 

Saltelli, A., K. Chan, and E.M. Scott, editors. Sensitivity Analysis (Wiley, 2000). 

U.S. Environmental Protection Agency (EPA). (2006). User’s manual for the Co-Benefits 

Risk Assessment (COBRA) screening model. Developed by Abt Associates Inc. 

U.S. Office of Management and Budget. (1992, 2003, 2008) OMB Circulars No. A-94, 

and A-4. Available at www.whitehouse.gov/omb/circulars. 
83

Attachment 1 

Drivers for Public Accountability 

A number of directives and guidance memorandum from the Executive and Congress 

branches set impact evaluation expectations for EERE programs. 

Executive Orders and OMB Memorandum: 

 OMB Memorandum for the Heads of Executive Departments and Agencies, July 29, 

2010 (Memo M-10-32) and Oct. 7, 2009 (Memo M-10-01) -- increased emphasis on 

program evaluation in Federal Agencies. 

http://www.whitehouse.gov/sites/default/files/omb/assets/memoranda_2010/m10-01.pdf 

 OMB Budget Action Request (Memo 10-49), July 29, 2010 – mandatory agency 

program evaluation inventory. http://www.whitehouse.gov/omb/memoranda_default 

 OMB and White House Office of S&T Policy Memorandum for the Heads of Executive 

Departments and Agencies on Science and Technology Priorities for the FY 2011 

Budget, August 2009; calls for R&D agencies to conduct evaluations and strengthen 

capacity. http://www.whitehouse.gov/briefing-room/presidential-actions/presidentialmemoranda 

 OMB Performance Rating Assessment Tool (PART), 2003-2008; set expectations for 

periodic systematic evaluations to be used to demonstrate results.

http://www.whitehouse.gov/omb/memoranda_m03-06/ 

 ARRA unprecedented requirements for transparency & accountability, 2009. 

http://www.recovery.gov/About/Documents/InitialRecoveryActImplementingGuidance_

Feb18.pdf

 Executive Order 13450: Improving Government Program Performance, November 2007; 

agencies shall spend taxpayers’ dollars efficiently & effectively.

http://www.whitehouse.gov/sites/default/files/omb/assets/performance_pdfs/eo13450.pdf

Congress: 

 GPRA Modernization Act of 2010 – each agency shall make available on its public 

website an update on its performance. Agency strategic plans must include "...a 

description of the program evaluations used in establishing or revising general goals 

and objectives," and Agency performance reporting has to " include the summary 
84

findings of those program evaluations completed during the period covered by the 

update." http://www.gpo.gov/fdsys/pkg/BILLS-111hr2142enr/pdf/BILLS111hr2142enr.pdf 

 House Committee Reports HEWD, 2008/2009/2010, calls for reporting on return on 

investment.

http://www.google.com/url?sa=t&source=web&cd=7&ved=0CEgQFjAG&url=htt

p%3A%2F%2Fscience.energy.gov%2F~%2Fmedia%2Fbudget%2Fpdf%2Fsccongressional-appropriations%2FFy-2012%2FHouse-bill%2FHEWD-FY12-

Committee-Report---

Final_SC_Only.pdf&rct=j&q=house%20committee%20reports%20hewd&ei=tLJ

nTr3VIcj50gGHipDKCw&usg=AFQjCNHDI8M3zhyIR3C7rQZrcdAbjDDd5A&

cad=rja

 Department of Energy Organization Act of 1977 (42 USC 5815(b)) – grants 

administrative authority for agencies to conduct program evaluations. 

http://frwebgate.access.gpo.gov/cgibin/getdoc.cgi?dbname=browse_usc&docid=Cite:+42USC5815 
85

Attachment 2 

Guiding Principles for Evaluators 

The following is a summary of highlights of the American Evaluation Association (AEA) 

guiding principles that bear on expected conduct of evaluators in performing the EERE 

benefit-cost studies: 

1. Systematic Inquiry: Evaluators conduct systematic, data-based inquiries. 

This principle requires that evaluators adhere to the highest technical standards 

appropriate to the methods they use; that they explore with the client the strengths 

and weaknesses of various evaluation questions and approaches; and that they 

communicate their methods and approaches accurately and in sufficient detail to 

allow others to understand, interpret and critique the results. 

2. Competence: Evaluators provide competent performance to stakeholders 

This principle requires that the evaluation team has the education, abilities, skills, 

and experience to carry out the proposed evaluation tasks. 

3. Integrity/Honesty: Evaluators display honesty and integrity in their own 

behavior, and attempt to ensure the honesty and integrity of the entire evaluation 

process. 

This principle requires evaluators to avoid conflict of interest and the appearance 

of a conflict; that evaluators not misrepresent their procedures, data, or findings; 

and that they should attempt to prevent or correct misuse of their work by others. 

4. Respect for People: Evaluators respect the security, dignity and self-worth of 

respondents, program participants, clients, and other evaluation stakeholders. 

This principle requires that evaluators seek a comprehensive understanding of the 

important contextual elements of the evaluation; that they obtain informed 

consent from those participating and inform participants of limits of 

confidentiality; and that evaluators should conduct the evaluation and 

communicate its results in a way that avoids unnecessarily negatively affecting 

the interests of stakeholders while not compromising the integrity of the 

evaluation findings. 

5. Responsibilities for General and Public Welfare: Evaluators articulate and take 

into account the diversity of general and public interests and values that may be 

related to the evaluation. 
86

This principle requires that evaluators should consider not only the immediate 

outcomes but also broader assumptions, implications and potential side effects; 

that they should present results clearly and simply so that clients and other 

stakeholders can easily understand the evaluation process and results; and 

evaluators have obligations that encompass the public interest. 

In summary, Principal Investigators are asked to support their benefit-cost analysis by 

collecting as many lines of evidence from independent sources as possible within 

practical constraints of data availability, time, and resources, and to use transparency in 

discussions of data collection, calculations, and analysis. 
87

Attachment 3 

Comparison of the EERE Approach with the 2001 NRC Approach 

Summary: 

For those who wish to know how the EERE approach modified an earlier NRC approach, 

a brief overview is provided here. 

Figure A3-1 summarizes a comparison of features of the EERE and 2001 NRC 

approaches.. The upper left shows the original 2001 NRC framework, and the lower 

right shows the EERE modified framework as the yellow highlighted portion of the table, 

emphasizing EERE’s greater focus on retrospective benefits and costs than the NRC 

approach. 

Figure A3-1. Modified NRC Framework25 Only Retrospective Benefits are Included 

Table A3-1 summarizes the comparison of 2001 NRC Approach to EREE modifications. 

In the first column are main features of the approach of the 2001 NRC Study; in the 

 

25 Source: Benefits Workshop, 2002 

Modified Framework

Original NRC Framework

Degree of uncertainty

1/ Includes energy benefits in the calculation

U ltim ate outcom es

Knowledge

Security

Environmental

Economic1

Option

Value

Expected

(Prospective)

Realized

(Retrospective)

Knowledge

Security

Environmental

Economic1

Option

Value

Expected

(Prospective)

Realized

(Retrospective)

Source: EERE Benefits Analysis Workshop, 2004

Source: Energy Research at DOE: Was It Worth It?, NRC, 2001

Security

Environmental

Economic

Options Knowledge

Value

Realized

Security

Environmental

Economic

Options Knowledge

Value

Realized
88

second are main features of the EERE approach. Features are aligned to allow the reader 

to see the modifications. 

Table A3-1: Comparison of 2001 NRC Approach to EERE Modifications 

NRC Study, 2001 EERE Modifications

SCOPE 

• Selected technology mix, winners 

& losers 

• Detailed benefit-cost analysis of several 

technologies where economic and other 

benefits from a program cluster is 

compared against the entire program or 

cluster investment cost 

• Technologies selected from the 

early 1970’s to 2000 period 

• Technologies selected from the early 

1970’s to 2008 period 

BENEFITS INCLUDED 

• Benefits resulting from all 

capital stock installed through 

the present plus 5 years (in that 

case 2005) calculated over the 

entire future life-cycle of all 

these installations 

• Economic benefits resulting from all 

capital stock installed up to the cutoff 

year. Future life-cycle benefits are 

excluded. 

• It is to be determined if the NRC 

study included projected impacts 

for technologies not yet 

commercialized 

• ‘Realized’ outcomes counted only for 

technologies that are already in the 

market, as indicated above 

• Knowledge treated as qualitative 

catch-all for situations of 

technology failure in development 

or/and in deployment, plus 

descriptive listing of what were 

considered major technical 

accomplishments 

• Assessment of knowledge creation and 

dissemination will not be limited to 

cases of failure. Rather identifying 

knowledge creation and dissemination 

will encompass both successful and 

unsuccessful technologies, within and 

outside the target industries. Historical 

tracing will identify paths and extent of 

knowledge flow, as well as recipients of

the knowledge 

• Environmental benefits - NOx, 

SO2, and Carbon. Proxy values for 

the mitigation/ damage costs 

• Avoided NOx, SO2, PM, CO2

equivalents. Avoided adverse health 

incidences associated with air 

emissions. Health care costs valued in 

dollars using EPA CPBRA model. 

Dollar value of CO2 is excluded. 
89

NRC Study, 2001 EERE Modifications

• Security benefits – Oil and LPG 

(Q); Electricity reliability (Y/N); 

Valued using $3-20/barrel based 

on the probability and potential 

impact of oil disruptions; no 

valuation of infrastructure threat 

avoidance beyond yes/no/don’t 

know. 

• Security benefits for oil and natural gas 

in physical units and BOE equivalent. 

Qualitative treatment of energy 

benefits. 

• Work productivity, exports 

addressed qualitatively 

• Value of increase in work productivity 

and exports addressed qualitatively 

• Macroeconomic effects (e.g., job 

creation) not considered; Regional 

shifts not considered; Rebound 

effect not considered 

• Job impacts excluded. Regional shifts 

not considered; Rebound effect not 

considered 

• Options value addressed 

qualitatively 

• Options value not applicable because 

only commercialized technologies are 

selected 

RULES ABOUT CALCULATIONS, ATTRIBUTION 

• Next best technology is 

conventional technology 

• Next best technology could be 

conventional, best available, or earlier 

generation of subject technology - 

determined on case by case basis 

• 5 year rule-of-thumb to apportion 

credit for impact to Govt. vs. 

private R&D. 5 year rule assumes 

anything the public sector does 

would have been done by the 

private sector anyway without the 

Govt. within 5 years. 

• Additionality analyzed on a case-bycase basis; not using a rule of thumb 

It is to be clarified whether 

benefits/costs for lifetime of 

installations were cut off at 2005, or 

whether as stated on p. 88 of the 

2001 report, benefits were calculated 

for the entire lifetime of 

installations—including lifetimes of 

all the installations up to 2000 and 

also all the installation up to 2005—

i.e., a 5 year cut-off appears to have 

applied to the installations 

included—but not to the assumed 

• Prospective benefits are excluded 
90

NRC Study, 2001 EERE Modifications

lifetimes of those installations. 

However, there is disagreement 

about the approach that needs to be 

clarified. 

• No distinction in attribution of 

Govt. R&D vs. other factors 

driving market success of 

innovation 

• Addresses various aspects of attribution 

(Govt. R&D vs. private sector; other 

market drivers—e.g., Production Tax 

Credits, etc.); ; use of an attribution 

matrix framework 

• Partitioning attribution—NRC 

study not able to apply a 

satisfactory approach 

• Use of the concept of “additionality”, 

which describes what the gov’t. R&D 

added that would not have occurred 

otherwise. Other qualitative, logical 

arguments will be provided in support 

of additionality findings. 

• Levels of influence of R&D vs. 

standards/deployment activities 

not attempted 

• Qualitative discussion will address the 

levels of influence of the R&D vs. 

standards/deployment activities 

• No consideration of international 

effects 

• Flows of technologies between 

countries and benefits of this will be 

recognized, as well as benefits of 

developing technologies within U.S. 

• No discounting • Discounting, using the current OMB 

guidance for public benefit-cost 

analysis 

• Deflators -- all values in constant 

1999 dollars, adjusted using GDP 

deflators 

• All values will be appropriately 

adjusted to constant dollars as of the 

end of study’s cutoff year, taking into 

account the discounting approach used. 

(Because the discount rate is a real rate, 

GDP price deflators are applied prior to 

discounting.)