Evaluating Realized Impacts of 

DOE/EERE R&D Programs

Standard Impact Evaluation Method

August 2014

Prepared by

Rosalie Ruegg, TIA Consulting Inc.

Alan C. O’Connor, RTI International

Ross J. Loomis, RTI International
i 

Acknowledgement 

The report was prepared in fulfillment of a contract with the Lawrence Berkeley National 

Laboratory (LBNL) to commission independent economists and evaluators to develop a peerreviewed methodological guidance for impact evaluations and benefit-cost analyses of R&D 

programs. The guidance was prepared for the Department of Energy’s (DOE) Office of Energy 

Efficiency and Renewable Energy (EERE). 

The authors of this Guide are Rosalie Ruegg, Managing Director, TIA Consulting, Inc.; Alan C. 

O'Connor, Senior Economist, RTI International; and Ross J. Loomis, Senior Economist, RTI 

International. Yaw Agyeman, LBNL, was also a contributor to Sections I.1 and II.B.1, II.B.2, 

II.B.3, and II.B.4 of the Guide. Project Managers for this Guide include Jeff Dowd, DOE/ EERE 

and Yaw Agyeman, LBNL. 

The Guide has benefited from critical review by both external and DOE internal reviewers. The 

authors wish to thank the following reviewers for their useful comments and suggestions on 

earlier drafts of this Guide. 

Internal reviewers: 

x Fredric Beck, on behalf of the EERE Wind and Water Power Technologies Office 

x Joe Cresko, of the EERE Advanced Manufacturing Office 

x Rick Farmer, Fred Joseck, and Tien Nguyen of the EERE Fuel Cell Technologies Office 

x Zia Haq, EERE Bioenergy Technologies Office 

x Thomas Jenkin, Senior Energy Analyst, NREL 

x Robert Margolis, NREL, on behalf of the EERE Solar Energy Technologies Office 

x Jack Mayernik, ORISE Fellow, on behalf of the Building Technologies Office 

x Ed Vine, Staff Scientist, LBNL 

x Jacob Ward, EERE Vehicle Technologies Office 

External reviewers: 

x Irwin Feller, Principal, Feller Research Associates, LLC 

x Ken Keating, Consultant 

x Cheryl Oros, President, Oros Consulting Inc. 

Gratitude is also extended to the external reviewers who reviewed the previous 2009 version of 

this Guide, including Irwin Feller (Feller Research Associates, LLC), Elena Besedin (Abt 

Associates, Inc.), Diana Hicks (School of Public Policy, Georgia), Jon P. Nelson (Pennsylvania 

State University), Richard G. Newell (Duke University), Jeanne Powell (Consultant), and Dave 

Roessner (Georgia Institute of Technology).

This document replaces the “Guide for Conducting Benefit-Cost Evaluation of Realized Impacts 

of Public R&D Programs,” working drafts of 2009 and 2011, prepared by Rosalie Ruegg and 

Gretchen Jordan (360 Innovation, LLC and formally of Sandia National Laboratory). 
ii 

Notice

This document was prepared as an account of work sponsored by an agency of the United States 

government. Neither the United States government nor any agency thereof, nor any of their 

employees, makes any warranty, express or implied, or assumes any legal liability or 

responsibility for the accuracy, completeness, usefulness, of any information, apparatus, product, 

or process disclosed, or represents that its use would not infringe privately owned rights. 

Reference herein to any specific commercial product, process, or service by trade name, 

trademark, manufacturer, or otherwise does not necessarily constitute or imply its endorsement, 

recommendation, or favoring by the United States government or any agency thereof. The views 

and opinions of authors expressed herein do not necessarily state or reflect those of the United 

States government or any agency thereof. 

This report is being disseminated by the U.S. Department of Energy. As such, this document was 

prepared in compliance with Section 515 of the Treasury and General Government 

Appropriations Act for Fiscal Year 2001 (Public Law 106-554) and information quality 

guidelines issued by the Department of Energy. [Further, this report could be “influential 

scientific information” as that term is defined in the Office of Management and Budget's 

Information Quality Bulletin for Peer Review (Bulletin). This report has been peer reviewed 

pursuant to section II.2 of the Bulletin.] 

 
iii 

Table of Contents

Acknowledgement ........................................................................................................................... i

Notice .............................................................................................................................................. ii

PART I. Introduction, Background, and Overview ........................................................................ 1

I.1 Introduction ........................................................................................................................... 1

I.2 Key Concepts and Terminology ............................................................................................ 8

I.2.1 Retrospective Evaluation versus Prospective Analysis ................................................. 8

I.2.2 Formative versus Summative Evaluation ...................................................................... 8

I.2.3 Portfolio Approach to Impact Evaluation ...................................................................... 8

I.2.4 Multiple Categories of Impacts ................................................................................... 10

I.2.5 Social Return on EERE Program Investments ............................................................ 12

I.2.6 Conservative, Lower-Bound Estimates ....................................................................... 13

I.2.7 Theoretical Foundation for Evaluation of Social Returns ........................................... 13

I.3 Deciding When to Evaluate the Return on Investment ....................................................... 17

I.4 Overview of the 14-Step Evaluation Approach ................................................................... 20

I.5 Roles and Responsibilities of Independent Evaluators and of EERE Evaluation 

 Project Managers ................................................................................................................ 20

PART II. Conduct an Evaluation Step-by-Step ............................................................................ 23

II.1 Begin an Evaluation ............................................................................................................... 24

1.A Define a portfolio for evaluation ........................................................................................ 24

1.B Assess evaluability of the portfolio .................................................................................... 25

1.C Select specific technologies within the portfolio for in-depth analysis ............................. 25

1.D Characterize the portfolio and its components, including the selected technologies, 

 in the context of a logic model ........................................................................................... 28

1.E Compile investment costs and other portfolio data ............................................................ 28

1.F Describe qualitatively the elements of the portfolio not selected for detailed analysis ...... 30

II.2 Identify Next-Best Alternative and Assess Additionality ...................................................... 32

2.A Identify next-best alternative for each technology selected for detailed analysis .............. 32

2.A.1 Develop an in-depth understanding of the selected technologies ............................ 32

2.A.2 Characterize the selected technologies as infratechnologies, technology 

 platforms, and/or product technologies .................................................................... 34

2.A.3 Define the next-best alternative ................................................................................ 35

2.B Assess additionality using the most rigorous evaluation design that is feasible ................ 38

2.B.1 Experimental study design ........................................................................................ 39

2.B.2 Quasi-experimental study design .............................................................................. 39

2.B.3 Nonexperimental study design, including the counterfactual approach ................... 38

2.B.4 Order of preference in selecting study design .......................................................... 40

2.B.5 Using a counterfactual approach to assess additionality .......................................... 40

2.B.6 Improve the reliability of the additionality assessment ............................................ 43

II.3 Estimate Energy and Other Resource Impacts ....................................................................... 48

3.A Estimate energy and other resource impacts year-by-year ................................................ 48

3.A.1 Assemble data ........................................................................................................... 48
iv 

3.A.2 Take into account whether EERE’s effect is primarily through R&D or if there 

 are also market adoption strategies .......................................................................... 49

3.A.3 Take into account import and export effects as needed ........................................... 50

3.A.4 Estimate energy savings or capacity added in physical units by energy type, 

 taking into account the next-best alternative and additionality ................................ 50

3.A.5 Estimate other resource impacts in physical units .................................................... 52

3.A.6 Estimate resulting year-by-year dollars of savings for all resource categories ........ 52

3.B Calculate intermediate effects, and treat qualitatively any resource effects not captured 

 by the quantitative assessment ........................................................................................... 53

3.B.1 Analyze intermediate results metrics ........................................................................ 53

3.B.2 Treat qualitatively any resource effects of the technologies selected for detailed 

 analysis not quantitatively captured ......................................................................... 53

II.4 Estimate Environmental Impacts ........................................................................................... 57

4.A Estimate changes in air pollution emissions ...................................................................... 57

4.B Estimate avoided adverse health events and environmental health benefits using 

COBRA ................................................................................................................................ 59

4.C Estimate the dollar value of avoided GHG emissions using the social cost of carbon ...... 64

4.D Describe other notable environmental effects .................................................................... 67

II.5 Estimate Energy Security Impacts ......................................................................................... 71

5.A Bring forward from Step 3 the estimated net reduction in oil consumption in 

physical units ....................................................................................................................... 71

5.B Report in physical units the reductions in barrels of oil consumed and imported ............. 72

5.C Describe notable effects of the portfolio technologies on the nation's energy 

infrastructure ........................................................................................................................ 72

II.6 Estimate Knowledge Benefits ................................................................................................ 75

6.A Construct database of portfolio patents and publications attributed to EERE, 

plus other databases needed for comparisons ...................................................................... 75

6.B Conduct patent citation analyses ........................................................................................ 78

6.C Identify notable knowledge outputs and innovations attributed to or linked to 

the EERE portfolio ............................................................................................................... 81

6.D Apply additional techniques to assess knowledge benefits as budget permits .................. 81

6.E Consider if and how knowledge-benefit findings inform the attribution analysis 

of Step 2. .............................................................................................................................. 83

6.F Provide summary metrics for knowledge benefits ............................................................. 84

II.7 Calculate Measures of Economic Performance ..................................................................... 85

7.A Compute economic performance measures for each level of analysis .............................. 85

7.B Calculate economic performance measures for level two of a three-level analysis ........... 94

7.C Calculate economic performance measures for level three of a three-level analysis ......... 94

7.D Prepare a graph showing year-by-year benefits by category and for the total, together 

with portfolio investment cost ............................................................................................. 96

7.E Characterize how worthwhile the EERE portfolio investment has been based on 

monetized benefits and investment costs for each level of analysis considered .................. 97

7.F Perform a threshold or break-even analysis if sizable negative impacts have been 

revealed by any of the supporting qualitative analyses ....................................................... 97

II.8 Account for Remaining Effective Useful Life (EUL) of Purchased & Installed Systems 

in an Extended Analysis....................................................................................................... 99
v 

8.A Document EUL of technologies selected for detailed analysis .......................................... 99

8.B Estimate remaining EUL .................................................................................................. 100

8.C Estimate benefits for each year of remaining EUL .......................................................... 100

8.D Calculate economic performance measures for level 3, inclusive of remaining 

EUL effects ........................................................................................................................ 102

8.E Report the results over the remaining EUL in a separate chapter .................................... 102

II.9 Perform Sensitivity Analysis ................................................................................................ 104

9.A Identify areas of major uncertainty in the analyses .......................................................... 105

9.B Recalculate results using alternative values for uncertain input variables, alternative 

assumptions, or alternative calculation approaches ........................................................... 105

9.C Show sensitivity results in a separate chapter of the report ............................................. 107

II.10 Assess Technical and Market Context; Discuss Success Factors ...................................... 108

10.A Characterize the portfolio's broader technological and market context ......................... 108

10.B Assess EERE's program strategies ................................................................................. 113

10.C Provide actionable recommendations where possible .................................................... 113

II.11 Report Results .................................................................................................................... 116

11.A Meet the stated standards of the report for content, format, presentation, 

transparency, and credibility ......................................................................................... 116

11.B Present summary results in a uniform tabular format .................................................... 117

11.C Apply checklists ............................................................................................................. 118

II.12 Document All Input Data, Assumptions, Calculations, and Results .................................. 126

12.A Facilitate data quality, transparency, and reproducibility of results by improving 

documentation .................................................................................................................... 126

12.B Document data sources, assumptions, and calculations ................................................. 126

12.C Make it possible for a third party with a reasonable level of effort to replicate 

results ................................................................................................................................. 129

II.13 Use Study Findings to Inform EERE Decision Making and for Effective 

Communications ................................................................................................................ 130

13.A Develop a strategy for using study findings in communications with stakeholders ...... 130

13.B Apply relevant study findings to inform EERE decisions ............................................. 130

II.14 Implement Routine Tracking and Compilation of Data to Support Future Study 

Updates .............................................................................................................................. 131

14.A Implement routine data collection in support of evaluation .......................................... 131

14.B Maintain records of information that could be used to support future evaluations ........ 132

14.C Consider additional data collection ................................................................................ 133

References ................................................................................................................................... 137

Attachment 1: Drivers for Public Accountability ....................................................................... 142

Attachment 2: Technology Readiness Levels ............................................................................. 144

 
vi 

List of Tables 

Table I.1 EERE Impact Evaluation Metrics Covered in this Guide ............................................... 4

Table I.2 Overview of Factors to Consider in Deciding Whether to Launch a 

Retrospective R&D ROI Impact Evaluation of an EERE Portfolio ................................. 19

Table I.3 Roles of EERE Staff and of Evaluators throughout the Process ................................... 22

Table II.1Portfolios and Selected Technologies for Detailed Analysis (2013 and 2010 EERE 

Benefit-Cost Studies) ........................................................................................................ 27

Table II.2-1 Technology Characterizations and Next-Best Alternatives in 

EERE Benefit-Cost Analyses ........................................................................................... 37

Table II.2-2 Illustrative Ways that Additionality May Occur ....................................................... 44

Table II.2-3 Example Table to Aid with Identifying Rival Explanatory Factors ........................ 47

Table II.3-1 Example from the Geothermal Benefit-Cost Study of the Estimation of 

Productivity Benefits from Use of Reservoir Models ....................................................... 54

Table II.3-2 Example from the Geothermal Benefit-Cost Study Showing the Estimation 

of Drill Rig Rental Savings from Use of PDC Drill Bit Technology ............................... 56

Table II.4-1 100 Year Global Warming Potentials of Greenhouse Gases Used for National 

Inventory Reporting .......................................................................................................... 58

Table II.4-2 Health Endpoints Included in COBRA ..................................................................... 62

Table II.4-3 Calculating Appropriate Year Values from COBRA 2017 Results ......................... 63

Table II.4-4 Illustration of Health Benefit Calculations Results from the COBRA Model, 

Year 2012, Using a 7% Discount Rate ............................................................................. 64

Table II.4-5 Pump-to-Wheel Time Series of Environmental Health Benefits, 1999–2022 

(Thousands of 2012$ Using 7% Discount Rate)............................................................... 65

Table II.5-1 Percentage of U.S. Electricity Production from Petroleum Products and 

Percentage of Net Oil Imports by Year, 1976-2012 ......................................................... 73

Table II.5-2 Calculating Avoided Imported Oil ........................................................................... 74

Table II.6-1 EERE Geothermal Portfolio Paper/Publications Linked to the largest Number 

of Patent Families through Two Generations of Citations ................................................ 80
vii 

Table II.6-2 Highly Cited Solar Energy Patents of Top U.S. Solar Photovoltaic Producers 

Linked to Earlier EERE-Attributed Solar Photovoltaic Patents ....................................... 82

Table II.6-3 Example Knowledge Outputs and Outcomes: Summary Metrics ........................... 84

Table II.7-1 Calculating Economic Performance Measures ......................................................... 87

Table II.7-2 Summary of Standardized Conventions for Computing Constant Dollar Cash 

Flows, Discounting, and Computing Economic Performance Measures ......................... 88

Table II.7-3 Illustration of Year-by-Year and Total Investment Costs for the Vehicle 

Technologies Office R&D Investments in Energy Storage Technology, 1992-2012 ...... 89

Table II.7-4 Illustration of Year-by-Year and Total Attributable Energy Savings, Vehicle 

Technologies Office R&D Investments in Energy Storage Technology, 1992-2012 ...... 90

Table II.7-5 Illustration of Separate Reporting of Economic Performance Measures for 

Individual Technologies Selected for Detailed Analysis .................................................. 92

Table II.7-6 Combining Year-by-Year Energy and Other Resource Benefits .............................. 95

Table II.7-7 Example of Economic Performance Metrics Calculated from the Combined 

Energy and Other Resource Benefits and Health Benefits Compared with Portfolio 

Investment Costs ............................................................................................................... 96

Table II.8-1 Illustration of Benefits for the Retrospective Evaluation (1992-2012) and the 

Extended Remaining EUL Analysis (1992-2022) .......................................................... 101

Table II.8-2 Illustration of Economic Performance Measures for a Retrospective Evaluation 

(1992-2012) and Remaining EUL Evaluation (1992-2022) ........................................... 103

Table II.9-1 Sensitivity of Environmental Health Benefits to Displaced Fuel Mix of a 

Geothermal Power Plant ................................................................................................. 105

Table II.9-2 Sensitivity of Results to a Range of Expert Opinion on EERE's Acceleration 

of Flat-Plate Solar Array (FSA) Photovoltaic Technologies ......................................... 106

Table II.9-3 Parts A and B. Sensitivity of Results to Using a Different Estimation Approach 

for Reduced Fuel Consumption ...................................................................................... 106

Table II.10-1 Illustration of Presenting Notable Technology Outcomes for each EERE 

Initiative against a Timeline of U.S. Photovoltaic Industry Achievements .................... 111

Table II.10-2 Illustration of Summary Accomplishments under FSA's Process Development 

Area ................................................................................................................................. 112
viii 

Table II.11-1 Uniform Tabular Format for Reporting Summary Evaluation Results from 

EERE Portfolio Investment ............................................................................................. 118

Table II.11-2 Contents Checklist ................................................................................................ 121

Table II.11-3 Format Checklist ................................................................................................... 124

Table II.11-4 Presentation Checklist ........................................................................................... 125

Table II.12-1 Illustration of Documenting Data and Calculation Steps by the Use of 

Table Notes ..................................................................................................................... 127

Table II.14-1 Data to Support R&D Portfolio Impact Evaluations ............................................ 135

 
ix 

List of Figures 

Figure I.1 High Level Diagram of EERE Logic ............................................................................. 3

Figure I.2 Illustration of EERE's Portfolio Approach ................................................................... 11

Figure I.3 Mansfield Model of Social Benefits from an Innovation that Reduces the Cost of 

Producing a Good Sold ..................................................................................................... 18

Figure I.4 When to Conduct a Retrospective Impact Evaluation of an Emerging Technology ... 18

Figure I.5 14 Step Evaluation Process .......................................................................................... 21

Figure II.1 Simplified Portfolio Logic Model Showing Basic Elements ...................................... 30

Figure II.2 Example of Timeline of Rival Explanatory Factors, Showing Wind Market 

Timeline and Capacity Additions for the State of Illinois During the Period Covered 

by the Wind Powering America Initiative (1999-2010) ................................................... 46

Figure II.3-1 Comparing Actual and Counterfactual BTE ........................................................... 51

Figure II.3-2 Actual and Counterfactual Solar Photovoltaic Module Production Cost ................ 55

Figure II.4-1 COBRA Model Overview ....................................................................................... 61

Figure II.4-2 Social Cost of Carbon by Scenario and Year (2007$) ............................................ 66

Figure II.6-1 Counts of Energy Storage Patent Families by Priority Year ................................... 77

Figure II.6-2 Percentage of Solar Energy Patent Families of Top U.S. Solar Photovoltaic 

Producers Linked to Earlier EERE-Attributed Solar Photovoltaic Patents ..................... 78

Figure II.6-3 Organizations from All Industry Areas with the Largest Number of Patent 

Families Linked to Earlier EERE-Attributed Solar Photovoltaic Patents ........................ 79

Figure II.6-4 A Network of Organizations Facilitates Combustion Knowledge Creation 

and Flow............................................................................................................................ 83

Figure II.7 Illustration of Elements that Make up a Level Three Analysis .................................. 96

Figure II.10 Market Context Illustration: Number of Electric-Drive Vehicles Sold in the 

United States by Battery Technology, 1999-2012 .......................................................... 112
1 

PART I. Introduction, Background, and Overview 

I.1 Introduction 

This document (referred to as the “Guide”) 

provides guidance for evaluators who 

conduct impact assessments of research and 

development (R&D) programs for the U.S. 

Department of Energy’s (DOE) Office of 

Energy Efficiency and Renewable Energy 

(EERE). It is also targeted at EERE program 

staff responsible for initiating and managing 

commissioned impact studies. The Guide 

specifies how to estimate economic benefits 

and costs, energy saved and installed or 

generated, environmental impacts, energy 

security impacts, and knowledge impacts of 

R&D investments in advanced energy 

technologies. 

 

The Guide helps EERE satisfy major 

directives for evaluation issued by the U.S. 

Office of Management and Budget (OMB) 

and by the U.S. Congress (see Attachment 

1). The impact evaluation method covered in 

this Guide is intended to address the 

following questions of interest to EERE and 

its stakeholders: 

1. What has been EERE’s economic return 

on investment (ROI)1

 in energy R&D? 

2. To what extent has EERE's investment 

produced energy and environmental 

benefits and enhanced energy security? 

3. To what extent has EERE added to 

knowledge that has impacted 

innovations in today’s markets?

4. Would today’s commercialized 

technologies likely have happened at the 

 

1 In this Guide, the term "Return on Investment" (ROI) is 

used informally in a broad sense to refer to returns resulting 

from EERE investment. This is in keeping with popular 

usage by program directors. 

same time, and with the same scope and 

scale, without EERE’s efforts?

5. Has the public investment been worth it? 

6. Which R&D strategies, research 

environment, and other factors led to a 

robust return on public investment and 

which did not, and what lessons learned 

can be applied to future R&D 

investments? 

The logic of how EERE achieves its mission 

is illustrated in Figure I.1. The high-level 

diagram of EERE logic in the figure shows 

how EERE's resources, activities, and 

Why Perform Impact Evaluation of 

Federal R&D Programs? 

Impact evaluation of R&D programs is motivated by 

the desire of program directors to manage their R&D 

portfolios strategically, efficiently, and effectively 

so as to make the best use of public investments by 

the American people. Systematic retrospective 

evaluation informs directors about possible ways to 

improve their programs and to position programs for 

the future by revealing strengths and weaknesses in 

past performance. It also informs stakeholders of 

returns from the investment. 

Consistent with this aim are a host of Government 

directives for evaluation of Federal programs. These 

directives emphasize accountability, transparency, 

strengthened capacity, and use of results for 

evidence-based decision making. 

Additionally, a federal energy R&D program that 

has determined its net impacts and return on 

investment is better positioned to communicate its 

value to its agency leadership, Congress, 

stakeholders, and the public than one that lacks 

documented credible evidence
2 

outputs lead to interim and ultimate 

outcomes/impacts. From the resources 

provided by taxpayer dollars EERE assesses 

costs to society of the current energy system 

and develops programs to achieve cost 

reductions and other benefits. These 

activities create research knowledge and 

engineering solutions using public-private 

partnerships, and validate and accelerate 

development of advanced 

technologies/practices. This in turn results in 

commercialized technologies/ products, 

diffusion of knowledge, accelerated market 

development and uptake of new 

technologies. This leads to reduced energy 

use, cleaner energy supply, reduced air 

emissions, reduced oil imports, growth in 

market share of advanced energy 

technologies, and growth in the economy. 

Many of the outcome/impacts shown in 

Figure I-1 are measured by applying the 

methods in this Guide. Table I.1 lists the 

specific impact metrics that result from 

applying this Guide. 

In addition to economic performance 

metrics, the Guide provides measures of 

changes in energy used and installed or 

generated, reductions in air emissions and 

related health mortality and morbidity, 

energy security, and knowledge creation and 

diffusion. All outcomes/impacts reported in 

Table I.1 are based on the share that can be 

reasonably attributed to the EERE 

investment rather than to other causes.2

This new edition of the Guide—revised and 

extended from a previous 2011 version—

reflects experience and insights gained 

 

2 While policy makers and program directors may be 

interested in additional impacts such as distributional 

effects, the focus of this Guide is on assessing the impacts 

that together determine benefits and costs and return on 

investment. This focus reflects intensifying federal 

budgetary pressures and is responsive to federal directives 

on evaluation.

through application of methods from 

previous editions in five studies completed 

from 2010 through 2013. In addition, 

meetings were held in 2012 and 2013 to 

solicit feedback from evaluators who 

conducted previous EERE benefit-cost 

analyses and workshops were held with 

EERE program analysts to get their ideas for 

changes. 

Improvements in this edition of the Guide 

include the following: 

 

x Increased standardization of approach to 

enhance consistency among studies;3

x Expansion of impact metrics, including 

monetization of reduced greenhouse gas 

(GHG) emissions; 

x Improved documentation of data, 

assumptions, and computations; 

x Increased rigor in assessing the share of 

impacts resulting from EERE 

investment; 

x Additional guidance for deciding when 

to do a R&D ROI impact assessment; 

and

x Formative analysis (see Section 1.2.2) to 

provide insights into factors that led to 

(or not led to) ROI that may inform 

future R&D decisions for similar 

investments.

 

3 Increasing standardization of approach is intended to 

increase consistency among studies. It is not intended to 

inhibit innovation or impede improvements in approach, or 

to cause EERE to become out of step with OMB if it 

modifies its directives. To avoid such undesirable side 

effects, the Guide recognizes the creative role of the 

evaluator in confronting unique conditions of a study and in 

designing and implementing a suitable evaluative analysis. 

Further, EERE will continue to solicit feedback from 

evaluators and program directors on ways to further 

improve the Guide, and EERE will update the Guide as 

necessary to maintain consistency with executive and 

congressional directives. 
3 

Note: The outcomes in blue lettering in italics can be calculated using the standard method outlined in this Guide. 

Figure I.1 High Level Diagram of EERE Logic 

• FY13: $1.8 

Billion

• FY14: $2.8 

Billion

• FY15: $2.1, 

(Requested)

Resources Ultimate 

Outcomes

Activities Interim 

Outcomes

• Commercialized 

technologies/ products, processes

• Knowledge 

diffusion

• Accelerated 

development and market uptake of clean energy technologies

• Private 

leveraged dollars

• Determine 

impact of current energy system

• Evaluate costs 

to society

• Identify 

opportunities to reduce costs

• Design and 

implement programs to realize opportunities, for 

- Renewable 

Energy

- Energy 

Efficiency

- Sustainable 

Transportation

• Reduced 

energy use

• More clean 

energy supply

• Growth in 

market share of clean energy technologies

• Reduced GHGs• Reduced oil 

imports

• Economic ROI 

supporting domestic economic growth and opportunities

• More innovative 

domestic manufacturing

• Created 

knowledge

• Engineering 

solutions yielding greater performance, reliability, cost structure for new technologies, processes, manufacturing

• Established 

public-private partnerships

• Validated new 

technologies, processes under real world conditions

• Accelerated 

development of next generation technologies

Outputs
4 

Table I.1 EERE Impact Evaluation Metrics Covered in this Guide 

Outcomes/Impacts Units Definitions 

Overall Economic Performance Resulting from EERE's Investments 

Portfolio Investment Cost 

(undiscounted) 

Dollars - 

undiscounted 

DOE/EERE total research portfolio investment (the portfolio budget 

over the performance period covered in the evaluation). 

Gross economic benefits 

(GB)

Dollars - 

undiscounted

A simple summation of all impacts measured in dollars (undiscounted) 

generated by the successful market adoption of EERE-supported 

technologies, products, or processes. 

Net economic benefits 

(NB) 

Dollars -

undiscounted 

Gross economic benefits (undiscounted) less EERE investment costs 

(undiscounted). 

Net present value (NPV) 

at 3% and 7% discount 

rates 

Dollars - discounted

Present value (PV) of an investment's positive cash flow, less any 

negative cash flow, and less the present value of the total EERE 

investment cost, where discounting is performed at both 3% and 7% 

real discount rates. 

Benefit-to-cost ratio 

(BCR) at 3% and 7% Ratio 

A ratio formed by setting the numerator as the present value 

summation of monetized benefits resulting from EERE's investments, 

and the denominator as the present value of EERE investment costs, 

where discounting is performed at both 3% and 7% real discount rates.

Social return on public 

investment (measured 

using the Internal rate of 

return (IRR) 

Percent 

The percentage yield on an investment found as the solution interest 

rate that, when used in appropriate discount formulas, equates social 

benefits resulting from EERE's investments with EERE's investment 

costs, resulting in a Net present value (NPV) of zero. The IRR can be 

compared with the OMB-specified real discount rate, currently set at 

7% for benefit-cost analysis. 

Monetary Value of Energy and Other Resource Impacts and of Environmental Impacts Resulting from EERE's 

Investments

Monetary value of energy 

and other resource impacts 

Dollars – 

discounted 

Year-by-year and total monetary value of energy saved or installed or 

generated, and of labor and other resource impacts, undiscounted and 

discounted at both 3% and 7% real discount rates. 

Monetary value of 

greenhouse gas emissions 

reduction 

Dollars – 

discounted 

Year-by-year and total monetary value of avoided cost of carbon 

dioxide (and equivalents) emissions, using the social cost of carbon 

(3% discount rate values). 

Monetary value of avoided 

adverse health incidence 

due to reduced air 

emissions 

Dollars – 

discounted 

Year-by-year and total monetary value of avoided adverse health 

events due to air emissions, discounted, at both 3% and 7% real 

discount rates. 

Energy and Other Resource Impacts, in Physical Units, Resulting from EERE's Investments 

Energy saved Btu, type of fuel 

saved 

The quantified amount of energy savings resulting from an EERE 

energy-efficient technology, product, or process. 

Installed renewable 

capacity MW The quantified amount of renewable energy capacity increase attributable to an EERE investment in renewable energy. 

Renewable Generation kWh The quantifiable amount of increased renewable energy generated 

attributable to an EERE investment in renewable energy. 

Non-fossil fuel volume 

produced Gallons 

The quantifiable amount of non-fossil fuel (e.g., bio-diesel and ethanol 

(e.g., E85) for transportation) produced by an EERE-funded process or 

technology. 

(Continued) 
5 

Other Resource Impacts 

(e.g., Changes in land 

resource use, labor, or 

materials) 

Relevant units Descriptions of changes in land resource use, labor, or materials 

Environmental Impacts in Physical Units Resulting from EERE's Investments

Avoided Air Emissions 

Avoided greenhouse gas 

emissions in CO2e 

(carbon dioxide emissions 

(CO2) or equivalents – 

e.g., methane (CH4), and 

nitrous oxide (N2O)) 

Metric tons 

(MMTCO2e) 

Year-by-year and total amount of avoided greenhouse gas emissions 

from advancements in clean energy technologies enabled by EERE 

investment. 

Avoided sulfur dioxide 

emissions (SO2) Short tons 

Year-by-year and total amount of avoided sulfur dioxide emissions 

accrued from advancements in energy technologies enabled by EERE 

investment. 

Avoided nitrogen oxides 

(NOx) Short tons 

Year-by-year and total amount of avoided nitrogen oxide emissions 

accrued from advancements in clean energy technologies enabled by 

EERE investment. 

Avoided particulate 

matter emissions ≤ 2.5 

micrometers (PM2.5) 

Short tons 

Year-by-year and total amount of avoided PM2.5 emissions accrued 

from advancements in energy technologies enabled by EERE 

investment. 

Additional emissions, 

including carbon 

monoxide (CO), volatile 

organic compounds 

(VOC), and ammonia 

(NH3) 

Short tons 

Year-by-year and total amount of these additional avoided emissions 

accrued from advancements in energy technologies enabled by EERE 

investment. 

Other Types of Environmental Emissions

Changes in water 

consumptions and 

discharges; and solid 

waste generation 

Relevant units Descriptions of changes in water or waste generation 

Health Impacts in Physical Units

Reduced morbidity (e.g., 

avoided respiratory 

symptoms, chronic 

bronchitis, nonfatal heart 

attacks) and mortality 

Number of 

incidents, episodes, 

cases 

Year-by-year and total avoided adverse health incidence due to 

reduction in air emissions resulting from advances in energy 

efficiency and renewable energy technologies sponsored by EERE. 

Energy Security Impacts Resulting from EERE's Investments 

Displaced petroleum 

consumption 

Imported oil 

measured in gallons 

of gasoline 

equivalent, or 

barrels of oil 

Year-by-year and total imported oil displacement associated with 

EERE-attributed technologies, products, or processes. 

Reduced vulnerability of 

U.S. energy infrastructure Qualitative 

Descriptions of changes in potential vulnerability of the energy 

infrastructure to damage or other disruptions. 

Knowledge Created and Disseminated Resulting from EERE's Investments 

EERE-attributed patents 

issued Number of patents Identification of EERE-attributed patents by date, patent number, 

assignee, and title. 

(Continued) 
6 

Patent citations 

Citation rates - 

forward and 

backward linkages, 

citation index 

values*, and 

identification of 

notable patents 

Frequency with which other organizations reference earlier EEREattributed patents indicating in the case of backward linkages to major 

innovators in the target industry their potential influence on 

commercialization; in the case of forward linkages their potential 

influence both inside and outside the target industry; and in the case of 

high citation rates the identification of particularly influential patents. 

DOE patent citation rank 

among organizations in the 

field 

Rank DOE rank among organizations in terms of having its patents in the 

target industry cited by later patents of others in the field 

Optional Knowledge Measures 

Knowledge spillovers 

Linkages to other 

technologies, 

industries, and 

organizations 

outside the targeted 

areas 

The frequency with which EERE-attributed patents influence 

technology areas and organizations beyond the original area or industry 

of focus. 

EERE-attributed 

publications 

Number of 

publications 

Identification of EERE related publications by date, sponsoring 

organization, author, and title 

EERE publications most 

cited by other 

publications 

Listing EERE publications most cited by other publications 

Publication citations by 

patents 

Citation rates of 

publications by 

patents 

The frequency with which patents reference EERE publications, 

indicating those publications that are particularly important to 

innovation. 

Technology Acceleration (as applicable) Resulting from EERE's Investments 

Technology acceleration 

Number of years 

that technology 

development and 

commercialization 

are accelerated. 

The extent to which an EERE investment facilitates the accelerated 

achievement of a technical advance and its commercialization. 

Note: Several measures within and across categories of impact are different facets of the same impact, such as mortality and 

morbidity incidence and the economic value of avoided incidence, and energy saved in physical units and energy saved in 

dollars. These overlapping values occur in the process of providing intermediate values that are of interest particularly to EERE 

program directors. Double counting would result from summing across all individual impacts (if they were expressed in 

commensurate terms). However, there is no double counting in the economic performance measures. 

* The Citation Index adjusts for the type of technology and for the age of the patent, such that, for example, the Index value of

5.0 means that a patent has been cited approximately 5 times more often than would be expected of a patent of its age, within its 

technology area. 

The method presented in this Guide builds 

not only on the preceding edition of the 

Guide, but also on previous efforts by 

others. It adopts a portfolio assessment 

approach pioneered by the Advanced 

Technology Program (ATP) of the National 

Institute of Standards and Technology 

(NIST) and significantly improves the 

approach employed by the National 

Research Council (NRC) in the 2001 study, 

Energy Research at DOE: Was It worth It? 

It builds on work of the U.S. Environmental 

Protection Agency (EPA) to extend the 

evaluation to encompass environmental 

impacts. It draws on earlier technology 

characterizations and approaches to research 

evaluation performed under the direction of 

NIST, along with the growing body of 
7 

evaluation literature in science and 

technology.4

Special features of the EERE impact 

evaluation methodology presented in this 

Guide include: 

x Criteria for deciding when to perform 

R&D impact evaluation using the 

Guide; 

x A portfolio approach that offers a costeffective means of estimating a lowerbound return on a relatively large group 

of R&D investments; 

x Characterization of technologies by 

type and related characterization of the 

next-best alternative; 

x Five economic performance measures: 

undiscounted benefits, undiscounted net 

benefits, net present value (NPV), 

benefit-to-cost-ratio (BCR), and 

internal rate of return (IRR) on 

investment, where discounting is 

performed for multiple rates; 

x Economic performance measures 

computed at three levels of analysis that 

reflect differences in degrees of 

certainty in results. This includes, first, 

dollar value of resource impacts (most 

certain); second, dollar value of 

resource impacts plus value of 

environmental health impacts; and, 

third, all of the above monetary values 

plus dollar value of GHG impacts (least 

certain); 

x Multiple categories of impacts –

energy, labor, and other resource 

impacts, environmental impacts, energy 

security impacts, and knowledge 

creation and diffusion; 

 

4 Particularly helpful were earlier guides and handbooks on 

benefit-cost analysis and other aspects of evaluation of 

public science and technology programs, such as those by 

Tassey (2003), Ruegg and Feller (2003), Powell (2006), 

Link and Scott (2011a), Link and Scott (2011b), and Link 

and Vornotas (2013 ed.)

x Monetary and physical valuations of 

energy and other resource impacts; 

x Monetary and physical valuations of 

environmental impacts including 

reductions in CO2 and other air 

emissions, and health-related changes; 

x Energy security impacts in physical and 

in qualitative terms; 

x Knowledge creation and diffusion 

impacts; 

x A primary focus on retrospective, 

empirically based benefits and costs; 

x Extension of benefits to show impact of 

taking account of remaining effective 

useful life (EUL)5

 of an installed 

systems; 

x Provision of conservative, lower-bound 

estimates subjected to sensitivity 

analysis; and 

x Analysis to identify R&D strategy, 

research environment, and other factors 

conducive (or not conducive) to 

realizing a robust ROI. 

The Guide is organized in two main parts: 

PART I introduces the Guide in Section I.1. 

Section I.2 explains underlying concepts and 

terminology. Section I.3 explains when it 

makes sense to launch an impact evaluation 

study. Section I.4 gives an overview of the 

14 steps for conducting an R&D ROI 

evaluation. Section I.5 highlights the roles 

and responsibilities of EERE staff and of 

independent evaluators, respectively, in 

conducting evaluations using the Guide. 

PART II leads EERE program staff and 

evaluators step-by-step through the process 

of conducting an evaluation of the social 

return on EERE's R&D investments. It 

 

5 Effective useful life (EUL) is the period over which an 

asset, such as plant, equipment, and systems and 

components, with normal maintenance and repair, can be 

expected to continue to be usable for the intended purpose.
8 

emphasizes the use of common criteria and 

consistent approaches across studies and 

provides examples drawn from previous 

studies. 

I.2 Key Concepts and 

Terminology 

I.2.1 Retrospective Evaluation versus 

Prospective Analysis 

This Guide primarily has a retrospective 

focus. The Guide's retrospective approach to 

evaluation means that it estimates impacts 

based on what has actually happened. This 

approach may be contrasted to a prospective 

evaluation approach that estimates what 

potentially may happen, but only if stated 

assumptions hold. 

Although the analysis defined in this Guide 

has a retrospective focus, it also includes a 

separate, extended analysis that takes into 

account the remaining Effective Useful Life 

(EUL) of products/processes put in place 

during the retrospective study period. The 

rationale for taking account of remaining 

EUL is that the public policy decision and 

resulting program investment have been 

made, and the customer has already made 

the decision to adopt the subject energyrelated product/process, thus removing 

much uncertainty. However, to preserve the 

retrospective focus, the remaining EUL 

analysis results are reported separately. 

I.2.2 Formative versus Summative 

Evaluation 

Formative evaluation examines the program 

management process and looks for ways to 

improve it; thus, it is usually emphasized 

relatively early during an investment when 

there is still time to make adjustments.6

 In 

 

6 EERE has also supported development of an evaluation 

framework that guides retrospective evaluations in the 

contrast, summative evaluation estimates 

outcomes and impacts and assesses the 

overall worth of an investment. 

Because of the Guide's focus on 

retrospective evaluation, it features 

summative evaluation, (i.e., measuring the 

realized return on EERE's investment). 

However, it also features elements of 

formative evaluation—not in the 

conventional sense of real-time or near-term 

process evaluation, but rather in the sense of 

providing an assessment of EERE's program 

strategies and identifying which among 

those examined have proven particularly 

successful in contributing to return on 

investment. Along with this formative 

assessment, the evaluator is asked to provide 

actionable recommendations where possible. 

The objective is to provide feedback that 

informs future similar investments. 

I.2.3 Portfolio Approach to Impact 

Evaluation 

The impact evaluation approach supported 

by this Guide is designed for application to a 

portfolio of projects or technologies rather 

than to a single project or single technology. 

A portfolio may be defined as an entire 

program, a subprogram, a group of 

programs, or other grouping of projects or 

technologies or activities.7

 

 

early-to-intermediate term of an EERE investment. It 

features progress metrics and formative analysis, 

particularly of supply chain development, but also includes 

measurement of intermediate outcomes where they can be 

measured. See Jordan, Mote, Ruegg, Choi, and BeckerDippmann (2014).

7

 Users of previous editions of the Guide are alerted to a 

change in terms. Previously, "cluster" was used to 

designate the larger portfolio under study. However, this 

usage proved confusing to users. The point of confusion 

was whether “cluster” referred to the portfolio (which it 

did) or to the subset of projects selected from the portfolio 

for detailed analysis (which it did not). To avoid this 

confusion, “portfolio” replaces “cluster” in this 2014 

edition of the Guide. 
9 

The portfolio approach calls for first 

defining the portfolio and then selecting 

individual projects/technologies from the 

defined portfolio for detailed evaluation of 

their positive and negative impacts. The 

impacts of the remaining 

projects/technologies within the portfolio 

that are not selected for detailed analysis are 

treated qualitatively; they are not ignored. 

Investment cost is defined as the total EERE 

investment cost for the entire portfolio. 

Thus, investment costs include both the 

projects/technologies that are evaluated in 

detail and those that are not. 

The portfolio approach offers a costeffective alternative to the need to evaluate 

the return on R&D investments. It does this 

by focusing the quantitative analysis on a 

few projects/ technologies within a larger 

portfolio, while allowing broader 

conclusions to be drawn about the entire 

portfolio. In this way, it saves in evaluation 

costs by reducing the number of projects 

that must be evaluated in detail in order to 

draw conclusions about the portfolio overall. 

The ability to focus on only a few projects 

relies on the fact that a relatively small share 

of projects in many high-risk R&D 

portfolios account for most of the benefits.8

 

 

8 Policy makers and program directors funding high-risk 

R&D, like venture capitalists, generally have to choose an 

array of projects in order to find the few that will turn out 

to be highly successful. When a few projects account for 

most of the returns, a randomly drawn sample of projects 

will not provide a fair test of whether the portfolio 

investment has been worthwhile because the sample, unless 

extremely large, will tend to contain all or a high 

proportion of less commercially successful projects and 

miss the successful projects. In the case that the evaluator 

confronts a portfolio comprised of projects all with modest 

gains, drawing a relatively large random sample of projects 

for detailed evaluation is a reasonable selection approach. 

In this case, the economies of the portfolio approach 

described here will likely be reduced, but other aspects of 

the approach should still apply. On the other hand, if the 

evaluator confronts a portfolio comprised mostly of 

commercial successes, either a random selection process or 

a focused selection could be used to select projects for 

When it makes sense to hand-select the 

projects/technologies for detailed analysis 

rather than to draw them randomly, the 

selection should focus on projects that have 

experienced both technical and commercial 

success. This project selection criterion is in 

keeping with the objective of an efficient 

approach to determining if a portfolio 

investment has been economically 

worthwhile based on a lower-bound 

estimate. 

It should be noted that if program directors 

wish to apply other criteria for selecting 

projects for detailed assessment (such as to 

examine a range of performers or projects 

that have failed) they should recognize that 

use of these other selection criteria may 

serve a different evaluation objective than 

that served by the Guide.9

 

Impacts10 of the group of selected 

projects/technologies are compared against 

the total portfolio investment cost. The 

result is a lower-bound estimate of return, 

because not all of the portfolio benefits are 

 

detailed analysis. Based on experience, however, it is 

expected that the more typical distribution of R&D projects 

will be one where a few projects account for a 

disproportionate share of portfolio benefits. 

9 If a program director wishes to extend the benefit-cost 

analysis to include detailed assessment of a limited number 

of projects that have been commercial failures―such as to 

better inform the formative part of the analysis or to assess 

in more detail the effects of "failed projects"―a fair test of 

portfolio performance requires that the commercial failures 

be in addition to the projects/ technologies selected because 

of their commercial success. If the program director wishes 

to look only at commercially failed projects perhaps he/she 

actually wishes to conduct a different type of study than 

that covered by this Guide, (e.g., a "failure analysis" rather 

than a "benefit-cost analysis” or “impact analysis"). 10 This edition of the Guide emphasizes use of the term 

"impacts" in place of "benefits" to provide a more neutral 

term that does not presuppose a positive result. However, 

once positive and negative impacts have been combined 

and are being compared against investment costs to 

compute economic performance measures, the terms 

"benefits and costs" are used. 
10 

quantified, while all of the costs are 

quantified. 

Therefore, the question of whether the 

portfolio investment has been worthwhile 

rests on evidence provided by a lower-bound 

estimate of economic returns. The unit of 

analysis remains the portfolio, despite the 

use of a selection of projects/ technologies 

on which to measure benefits of the 

portfolio.11 If the projects/ technologies 

selected for detailed analysis have generated 

sufficient economic benefits to cover or 

exceed total portfolio investment cost and, 

in addition, there is no evidence that the 

remaining projects/ technologies have 

produced negative impacts offsetting those 

benefits, this finding can be taken as 

evidence that the portfolio investment has 

been economically worthwhile. 

It is possible to expand the selection of 

projects/technologies for detailed analysis in 

subsequent evaluations. Likewise, 

subsequent evaluations may allow additional 

time for market development for currently 

immature technologies. Thus, the lowerbound estimate of return could be raised by 

subsequent evaluations of a given portfolio. 

Figure I.2 illustrates the elements of the 

Guide's portfolio approach. The bracketed 

green rectangles together represent the 

portfolio under study and its impacts. The 

darker green rectangle represents the 

individual technologies selected from the 

portfolio for detailed impact assessment. 

The lighter green rectangle represents other 

technologies in the portfolio that are not 

 

11 This portfolio evaluation approach with its lower-bounds 

estimate may be contrasted with a "cherry picking" 

approach to evaluation that selects only successful projects 

and compares their benefits against their individual project 

investment costs and uses the results to represent the costeffectiveness of the entire portfolio. That is not the 

approach used here. 

selected for detailed study but rather are 

treated qualitatively. Qualitative analysis of 

this group of remaining technologies in the 

portfolio may provide evidence that they 

have added to overall portfolio benefits. 

Conversely, the qualitative analysis may 

indicate if they have had potentially negative 

impacts that may offset positive impacts of 

those selected for detailed analysis. In 

Figure I.2 the orange box to the right 

represents the portfolio's entire investment 

cost. The portfolio cost is inclusive of costs 

of the individual technologies selected for 

detailed study that may be broken out for an 

additional comparison. By comparing the 

dollar value of benefits for the selected 

technologies against entire portfolio cost a 

lower bound estimate--a minimum rate of 

return--is found for EERE's investment in 

the portfolio. 

I.2.4 Multiple Categories of Impacts 

The current approach provides a more 

comprehensive impact treatment than 

previous EERE approaches by accounting 

for the following categories of impacts: 

x Energy and Other Resource Impacts 

resulting from changes in energy, labor, 

capital, and other economic resources. 

x Energy Impacts are called out for 

separate treatment because energy is the 

focus of the EERE programs. Energy is 

treated in terms of the physical quantity 

of energy saved or installed/ generated 

as a result of the technology evaluated. 

x Environmental Impacts driven by 

changes in air emissions―including 

CO2, NOx, SO2, PM2.5, and other air 

emissions―resulting from changes in 

use of fossil fuels due to the subject 

technologies. Greenhouse gas reductions 

are valued in physical units and also in 

dollars by applying Social Cost of 

Carbon (SCC) estimates. Changes in 

other air emissions in turn drive changes 
11 

in public health mortality and morbidity. 

Health effects are calculated using the 

Environmental Protection Agency’s

(EPA's) Co-Benefits Risk Assessment 

(COBRA) model. Avoided mortality and 

morbidity are stated in terms of incident 

rates, and the value of these avoided 

adverse health events is stated in dollars. 

If significant, other environmental 

impacts, such as water discharges, land 

resource use, and solid waste generation, 

are treated qualitatively. 

x Energy Security Impacts are treated in 

part quantitatively and in part 

qualitatively, but not in dollars. 

Displaced petroleum consumption is 

measured both in terms of gallons of 

gasoline equivalent and barrels of 

imported oil. Notable changes in the 

security of energy infrastructure (such as 

the utility grid and storage and delivery 

systems for natural gas, petroleum, and 

other energy sources) are treated 

qualitatively. The evaluation approach 

does not monetize any energy security 

benefit at this time because the valuation 

methods are not settled. 

 

Figure I.2 Illustration of EERE's Portfolio Approach 

x Knowledge Impacts focus on the 

creation and dissemination of knowledge 

outputs of the entire portfolio investment 

as indicated by patent and publication 

counts and citations. Knowledge impacts 

are not expressed in dollar terms, except 

in so far as knowledge advances underlie 

the energy, other resource, and 

environmental benefits that are 

expressed in dollars. Because knowledge 

provides a foundation for additional 

innovation, it is separately reported as a 

spillover impact in nonmonetary terms, 

but nonetheless quantitatively. 

By comparing combined benefits valued in 

dollars against portfolio investment costs, a 

financial return on EERE R&D investment 

is obtained. Additional positive impacts are 

measured but not in dollars. Because they 

are incommensurable with the dollar 

valuation, they are not part of the calculated 

Note: “portfolio” is entire program, subprogram, or other grouping to be studied

Source: Rosalie Ruegg, TIA Consulting

Technologies selected 

from the portfolio for 

detailed quantitative 

assessment of impacts 

Portfolio Impacts 

versus 

 Portfolio Cost 

Remaining part of the 

portfolio whose impacts 

are qualitatively assessed 

Entire 

portfolio 

costs 

Investment cost 

of the selected 

technologies 
12 

economic performance metrics. However, 

impacts―whether measured in dollars, in 

physical units, or described 

qualitatively―reflect, in a broad sense, 

evidence of the larger return on R&D 

investment. 

In this Guide, the term "Return on 

Investment" (ROI), as noted previously, is 

used informally in a broad sense to refer to 

the social returns from EERE investments 

inclusive of economic returns measured in 

dollars as well as other impacts measured in 

other terms. It is recognized, of course, that 

only impacts measured in dollars are 

actually incorporated in percentage rate-ofreturn measures, but conceptually this 

broader use of ROI serves as a reminder of 

total impacts. 

I.2.5 Social Return on EERE Program 

Investments 

An objective of the impact evaluation 

performed according to the Guide is to 

provide an estimate of the social return on 

EERE's investment. To explain the meaning 

of this term, it is useful to distinguish 

between an evaluation performed for a 

company and one performed for a public 

program. A major difference is perspective 

and breadth of coverage.12 In simplistic 

 

12 Another essential difference in a federal impact 

evaluation and that of a private company is the treatment of 

taxes. For a private company, it is important to include tax 

effects on company cash flows (i.e., adjust for taxable and 

tax-deductible items), taking into account tax credits, 

depreciation, tax deductions of interest payment and other 

business expenses, and related effects. In contrast, an 

impact evaluation of a federal investment program that is 

not directly using the tax system as a policy mechanism, 

such as that covered by this Guide, can generally ignore tax 

effects in its impact analyses. Moreover, there is a growing 

"public values" literature that argues for additional impacts 

to be taken into account in public impact evaluation, such 

as effects on income distribution, employment effects, and 

balance of payments―effects that traditionally have been 

omitted from benefit-cost analysis. The Guide expands 

coverage of benefits to include environmental, energy 

security, and knowledge effects, in addition to energy and 

terms, private Company X takes into 

account its own cash inflows and outflows 

in calculating the return on its investment. 

Impacts of its actions on others, such as on 

Company B who loses revenue in the face of 

increased competition from X, or Company 

C who gains revenue by learning from 

Company X, are not included by Company 

X in the calculations of return on its 

investment. 

In contrast, a public program typically takes 

into account impacts resulting from its 

actions within a broad sphere of interest, 

including, but extending beyond, private 

returns. That is, all resulting gains and losses 

of significance to the nation, regardless of 

who experiences them, are relevant to 

computing the social return on a federal 

program investment. Thus, a federal impact 

evaluation typically has a much broader 

perspective and scope of coverage than does 

a private-company analysis. For this reason, 

it is typically much more complex and 

challenging to perform than a private 

company analysis. 

"Social return" is a term used to signify the 

broad inclusion of impacts to society from 

an investment―inclusive of both private 

returns and spillover returns―which 

together comprise social returns. Here, the 

qualified term "social return on public 

investment" or, more specifically, "social 

return on EERE's investment" is used to

indicate inclusiveness of all social benefits 

on the impact side resulting from EERE's 

activities. This benefits perspective is 

contrasted with an investment-side focus on 

 

other resource effects, but does not extend the coverage to 

income distribution effects, employment effects, or balance 

of payment effects. Regarding employment effects, it 

should be recalled that many of the investments covered 

retrospectively took place largely during periods that were 

characterized by full employment, such that they would be 

small in any case.
13 

EERE's expenditures on the portfolio under 

evaluation. That is, investment costs refer to 

DOE/EERE's investment13 rather than total 

societal investment costs, which would 

include investment not only by DOE/EERE 

but also by all other parties. The social rate 

of return on EERE's investment is computed 

using the Internal Rate of Return (IRR)

method. The IRR is supplemented by other 

measures of economic performance 

described in Table I.1. 

The quantitative economic performance 

measures, which are based on impacts 

measured in constant dollars, are presented 

in a three-level analysis. Each level adds 

additional benefits to be compared against 

EERE's portfolio investment costs: 

1. Level one compares energy and other 

resource savings against total portfolio 

investment costs.

2. Level two compares the combination of 

energy and other resource savings and 

the economic value of avoided adverse 

health events against total portfolio 

investment costs.

3. Level three compares the combination of 

energy and other resource savings, value 

of avoided adverse health events, and 

value of reduced GHGs against total 

portfolio investment costs. 

I.2.6 Conservative, Lower-Bound 

Estimates 

The resulting measures of economic 

performance are expected to provide a 

conservative, lower-bound estimate of actual 

returns for three reasons: 

 

13 In most evaluations performed under the Guide, 

investment costs are exclusively EERE's. However, in at 

least one case DOE Office of Science costs also contributed 

to an EERE technology portfolio that was evaluated. In 

such case, all DOE investment costs are taken into account. 

To emphasize this point, DOE/EERE is used here instead 

of EERE alone.

1. The retrospective approach avoids 

reliance on forecasted data with higher 

degrees of uncertainty; nevertheless, 

benefits in most cases are expected to 

continue past the cut-off year of the 

analysis. When, in a separate, secondary 

analysis, remaining EUL impacts 

resulting from investments already made 

are included, it is expected that the 

estimates will remain conservative for 

the additional two reasons given below. 

2. The portfolio approach includes all 

investment cost for the portfolio, but 

quantitative benefits are based on only a 

subset of technologies drawn from the 

portfolio for detailed analysis. 

3. Not all impacts of the technologies 

selected for detailed analysis are valued 

in monetary terms. Impacts of avoiding 

environmental effects from land and 

water pollution, energy security impacts, 

and knowledge spillover impacts are 

among the effects that are only 

expressed in nonmonetary units. 

I.2.7 Theoretical Foundation for 

Evaluation of Social Returns

This section treats in brief the theory 

developed largely by Zvi Griliches and 

Edwin Mansfield for analyzing private 

returns and spillovers, upon which the 

Guide's approach to evaluation of social 

returns is derived. Social returns encompass 

private returns plus consumer surplus (i.e., 

market spillovers), plus other spillover 

effects (e.g., knowledge spillovers and 

network spillovers) that may apply.14

Early analysis applications by Griliches and 

Mansfield proved useful for conceptualizing 

and estimating social and private returns on 

 

14 Mansfield's model, which is described in this section, 

takes into account private returns and market spillovers 

(called "consumer surplus" within the context of the 

model). 
14 

investment in new technologies.15 Griliches 

applied his approach to measuring social 

returns on investments to develop hybrid 

corn. Building on Griliches' work, Mansfield 

applied his model to assess the social 

benefits of private-sector industrial 

innovations, and found that the estimated 

social rate of return for a group of selected 

industry innovations were generally higher 

than the private rate of return. Like 

Griliches, he concluded that there may be a 

substantial "spillover gap" between private 

and social rates of return on innovation. A 

public policy implication is that private 

investment decisions, which do not take into 

account spillover effects, will tend to result 

in less investment in R&D and innovation 

than is optimal from the standpoint of 

society at large.16 

The seminal work by Griliches and 

Mansfield has long informed the analysis of 

federal R&D investments. Their work, 

which is well known to 

economists/practitioners of social benefitcost and impact evaluations, continues to 

provide a theoretical anchor and unifying 

framework for such evaluation studies.17 

The simplified representation in Figure I.3 

of Mansfield's model serves to illustrate the 

basic concepts. Suppose that an innovation 

reduces costs of the industry using the 

innovation. The social benefits from the 

 

15 See Z. Griliches (1958), E. Mansfield, J. Rapoport, A. 

Romeo, S. Wagner, and G. Beardsley (1977); Foster 

Associates, (1978); and Nathan Associates (1978) to learn 

more about the model's development and early use. 

16 This is not to say that the private rate of investment is 

lower than it should be based on the private risk adjusted 

expected rate of return relative to the private marginal cost 

of capital. Rather, the presence of large spillover effects not 

captured by private investors and, hence, not factored into 

their investment decisions may argue for supplementary 

public investment in cases where the gap between social 

benefits and private benefits is particularly large due to 

large spillover benefits. 

17 Z. Griliches (1958), and E. Mansfield, et al. (1977). 

innovation can be measured by the profits of 

the innovator plus the benefits to consumers 

of the goods produced by the industry using 

the cost-reducing innovation. To the extent 

that the innovation is adopted (or adapted) in 

other applications, a similar approach could 

be taken in each application and the total 

social benefits (less costs) aggregated. 

Mansfield acknowledged that the 

calculations are not this simple, but 

indicated that the basic model conveys the 

spirit of the analysis.18 

With reference to Figure I.3, DD' depicts a 

demand curve for good(s) using the costreducing innovation. The horizontal supply 

curve labeled S1 reflects the pre-innovation 

supply of the goods, and P1 indicates the 

pre-innovation price paid by consumers. The 

horizontal supply curve labeled S2 reflects 

the post-innovation shift downward of the 

supply curve due to decreases in production 

costs, and P2 indicates the new price that 

consumers will pay. The top-hatched area 

indicates the gain in consumer surplus, due 

to the innovation. It is the excess of what 

consumers would have been willing to have 

paid for the new quantity versus what they 

actually had to pay, summed over all 

purchases. 

How far downward the supply curve will 

shift depends, of course, on the effect of the 

innovation, the pricing policy of the 

innovator, and the competitive structure of 

the industry sector. If the industry sector is 

characterized by little competition, the 

innovator may be able to hold the product 

prices relatively unchanged, such that the 

supply curve shifts little or none. However, 

if the industry sector using the innovation is 

 

18 Professor Mansfield was engaged by the Advanced 

Technology Program to extend his model to apply in 

benefit-cost studies of ATP-funded innovations. This brief 

description of his model is from a planning report for that 

effort. See Mansfield (1996). 
15 

competitive, it is expected that the innovator 

will lower the price for its new product as 

others enter with competing products. 

The social benefits from the innovation can 

be measured by the sum of the two crosshatched areas in Figure I.3. The top crosshatched area is the consumer surplus due to 

the lower price (P2 rather than Pl) resulting 

from the use of the innovation. In addition, 

there is a resource saving and a 

corresponding increase in output elsewhere 

in the economy due to the fact that the 

resource costs of producing the good using 

the innovation are less than P2 Q2. Instead, 

they are P2 Q2 minus the profits of the 

innovator from the innovation (r), the latter 

being merely a transfer from the producers 

of the good using the innovation to the 

innovator. Thus, besides the consumer 

surplus arising from the price reduction, 

there is a resource saving amounting to the 

profits of the innovator. For example, 

suppose the innovator reaps a $100 million 

profit from its innovation. This means that 

P2 Q2 is an over-estimate of the value of the 

resources used by the industry, in the 

amount of $100 million; the amount the 

industry pays the innovator in profits. Recall 

that this payment to the innovator is not in 

exchange for resources; rather, it is a 

transfer of profit to the innovator. 

Two adjustments are needed in the estimate 

corresponding to the lower shaded area in 

Figure I.3. First, if the innovation replaces 

another product, the resource savings cited 

above does not equal the profits of the 

innovator. 

Instead, it equals these profits less those that 

would have been made (by the innovator 

and/or other firms) if the innovation had not 

occurred and the displaced product had been 

used instead. Second, if other firms imitate 

the innovator and begin selling the 

innovation to the industry that uses it, their 

profits from the sale of the innovation must 

be added to those of the innovator to get a 

full measure of the extent of the resource 

saving due to the innovation. 

Using this model, an estimate can be made 

of the social benefit in each period from the 

investment in a given innovation. For each 

innovation, the top shaded area in Figure I.3 

equals: 

(P1 - P2) Q2 (1 - 1/2 Kn) 

where K = (P1 - P2)/P2, and n is the price 

elasticity of demand (in absolute value) of 

the product in the industry using the 

innovation. 

To estimate P1 - P2, Mansfield’s approach 

was to obtain as much information as 

possible on the size of the unit cost 

reduction due to the innovation. To obtain a 

reasonably reliable estimate of (P1 - P2), 

Mansfield conducted interviews with 

executives of the innovating firm, executives 

of a sample of firms using the innovation, 

and reviewed reports and studies made by 

these firms for internal purposes. And with 

the estimate of (P1 - P2), it was then 

possible to compute K. Q2 was generally 

available from published records. Rough 

estimates of n were obtained from published 

studies and from the firms. Since K was 

generally very small, the results were 

generally not very sensitive to errors in n. 

Indeed, Mansfield concluded that the 

expression in equation (1) could be 

approximated well in most cases by (P1 - 

P2) Q2, which is the total savings to 

consumers due to the lower price if they buy 

Q2 units of the product of the industry using 

the innovation. This latter simplification has 
16 

been helpful in the practical application of 

Mansfield’s model. Despite simplification in 

the model, there is a substantial challenge in 

compiling the data necessary for the 

analysis. 

Not included in the depiction of Figure I.3 

are knowledge spillovers that occur as firms 

outside the innovator’s industry draw from 

the same knowledge base to produce other 

goods and services in other industries. Also 

not included in Figure I.3 are non-economic 

impacts, such as environmental and energy 

security impacts, and the more general 

impacts of an enhanced knowledge base on 

the innovative capacity of organizations. 

In addition, the Mansfield approach 

incorporates supply side shifts only, thus 

ignoring possible dynamic effects that may 

also affect demand schedules. In fact, while 

EERE emphasizes R&D, which is expected 

to have its impact primarily on supply 

schedules, EERE also conducts deployment 

activities that are aimed at influencing 

demand schedules―a condition not 

addressed by the Mansfield model. 

Areas not covered by the Mansfield 

approach have necessitated departure from 

strict adherence to the Mansfield model, 

primarily to account for demand-side effects 

and additional types of benefits. At the same 

time, most evaluation studies to date have 

focused on supply-side effects from R&D, 

and have conformed to the model in terms of 

resource-driven economic impacts. Despite 

the lack of perfect alignment of EERE 

evaluation requirements and the Mansfield 

model, the model continues to be considered 

foundational to the Guide's 

approach―conditioned by an understanding

that modifications/extension can be made to 

reflect actual circumstances that do not 

conform strictly to the model's limiting 

assumptions. 

Figure I.3 Mansfield Model of Social Benefits from an Innovation that 

Reduces the Cost of Producing a Good Sold 

 P2 - 

Q2 

D' 

Cost per unit of output of 

industry using the 

innovation 

P1 

P2 

S1 

S2 

D 

0 Output of industry using the 

innovation 

Source: Edwin Mansfield (1996) 
17 

I.3 Deciding When to Evaluate 

the Return on Investment 

Impact evaluations are conducted to learn 

about and improve programs, determine 

whether they are effective, and communicate 

the value to stakeholders. These intended 

uses of impact results condition when to 

conduct an evaluation. This section focuses 

on some of the key considerations for when 

to evaluate EERE’s return on investment. 

Not every possible portfolio grouping within 

an R&D program need be recommended for 

a ROI impact evaluation. Before launching a 

retrospective evaluation using the Guide, the 

program director should consider the factors 

described below that influence when it 

makes sense. The first consideration is to 

determine if a portfolio area―and elements 

within it―have had sufficient time to 

progress to a state of readiness for a 

retrospective impact evaluation. Figure I.4 

shows when it is too early to evaluate 

retrospectively the economic return on an 

emerging technology, and a point beyond 

which such an evaluation may be 

appropriate. 

As the diagram shows, during the stages in a 

technology/product/process life cycle (i.e., 

when a technology is at Technology 

Readiness Levels (TRL) 1 through 9, or 

early in post-TRL 9)19 (see Attachment 2), it 

has not had sufficient time to generate 

benefits. At an early stage, a technology 

may be found not yet to have been 

economically worthwhile, although it may 

be at a future time. Findings from an 

 

19 TRL is a measure of technical readiness originally 

developed by NASA in the 1980s, and endorsed in 2001 by 

the Government Accountability Office (GAO) for use in 

new major federal programs to identify the maturity of 

emerging technologies. Based on a 1-9 scale, TRL 1 

indicates a newly invented technology on which analysis 

and testing has not yet been performed, and TRL 9 

indicates that the system has been proven and is ready for 

commercialization.

economic evaluation of a technology 

conducted prematurely may possibly not 

show returns because the assessment was 

performed too early, before the market could 

develop. After a technology has been 

commercialized and benefits have had time 

to accrue, then the return on investment can 

be evaluated. Stage of technology 

development/market penetration is, 

therefore, a make or break consideration in 

deciding whether to launch a R&D impact 

evaluation.20 After determining if the stage 

of technology development/market 

penetration is conducive to undertaking a 

R&D impact evaluation, there are additional 

factors to consider. Table I.2 lists factors 

that will influence whether to proceed with 

an evaluation using the Guide. 

Closely related to the above are 

considerations on the level of rigor 

necessary. Since R&D ROI impact 

evaluations typically are performed to meet 

one or more of the conditions such as those 

listed below, it is important that they be 

robust and defensible. 

 A program/subprogram/ portfolio/ 

initiative is receiving a high level of 

external attention; 

 

20 There are other evaluations that may be appropriate to 

conduct in the early TRL stages (e.g., pre-TRL 9), such as 

that covered by EERE's Framework for Evaluating R&D 

Impacts and Supply Chain Dynamics Early in a 

Product/Technology Life Cycle (Jordan, et al., 2014). For 

an account of additional evaluation methods that are 

applied at different stages and for a variety of purposes, see 

Ruegg and Jordan (2007).
18 

 

Figure I.4 When to Conduct a Retrospective Impact Evaluation of an 

Emerging Technology 

 A program/ subprogram/ portfolio/ 

initiative is a White House/ Congress/ 

DOE Secretary/ EERE Assistant 

Secretary priority; 

 The portfolio area is a major budgetary 

priority or involves a large public 

investment; 

 The portfolio investment addresses a 

particularly pressing and important 

problem; 

 The portfolio investment is of critical 

path importance to achieving DOE and 

EERE strategic goals; and 

 There is interest in adopting a similar 

major investment strategy elsewhere, 

and a need to first assess its achieved 

impact. 
19 

Table I.2 Overview of Factors to Consider in Deciding Whether to Launch a Retrospective 

R&D ROI Impact Evaluation of an EERE Portfolio 

Decision Criteria Supportive of Launch of 

a R&D ROI Evaluation 

Not Supportive of 

Launch 

Technologies that make up the portfolio are still at 

an early stage of market penetration

No Go, even if other 

conditions are 

favorable, because 

long-term impacts 

cannot yet be 

measured.

Technologies that make up the portfolio are 

sufficiently past the early market stage as to have 

achieved market growth

Go if one or more of the 

other conditions are 

favorable 

x For the portfolio, there is direct evidence, 

or a perception, of substantial market 

penetration in at least some areas. 

X 

x For the portfolio, sufficient time has 

passed for market effects to be observed 

(e.g., at least 3 years) in at least some 

areas, but not too far in the past to make 

data collection infeasible. 

X 

x The portfolio includes one or more 

technologies with realized benefits, 

though the ROI is unknown. 

X 

x Selected technologies from the portfolio, 

or the portfolio overall, were previously 

evaluated but had not shown notable 

realized economic ROI, in part due to 

insufficient passing of time. 

X 

x Selected technologies from the portfolio, 

or the portfolio overall, were previously 

evaluated more than 5 years ago and an 

updated assessment is required. 

X 

x Congress directs a study be done for the 

portfolio. 

X 

x The portfolio of interest has recently 

received a retrospective impact 

evaluation. 

X 

x The portfolio of interest largely overlaps 

previous portfolios that have received a 

retrospective impact evaluation. 

X 

x The portfolio area has lower priority than 

other areas based on stakeholder interest. 

X 
20 

I.4 Overview of the 14-Step 

Evaluation Approach 

An overview of the step-by-step process of 

conducting an R&D ROI impact evaluation 

for EERE is given in Figure I.5. Depicting 

the evaluation approach as a series of 

sequential steps in Figure I.5, as well as in 

Part II of the Guide, is intended to foster a 

common approach across studies, increasing 

the amount of "science" that goes into each 

EERE impact study. Studies done according 

to the Guide are not intended as "free-form" 

stand-alone entities; rather they comprise a 

series of related studies that together 

measure EERE's impact over time. 

As discussed in the next section, most of the 

steps listed in Figure I.5 are the 

responsibility of the evaluator. However, 

EERE staff have responsibilities throughout, 

particularly in Steps 1, 13, and 14. 

I.5 Roles and Responsibilities of 

Independent Evaluators and of 

EERE Evaluation Project 

Managers 

Both the evaluation project manager and the 

independent evaluator have critical roles and 

responsibilities in applying the standards set 

in this Guide to R&D evaluation. EERE 

program directors have, first, to decide if 

they will recommend that a given R&D 

portfolio be evaluated. If the decision is to 

proceed, the evaluation project manager has 

to obtain an evaluator using a competitive 

solicitation process. Then he/she will be 

expected to collaborate with the evaluator to 

verify the definition of the portfolio, select 

projects/technologies from the portfolio, and 

assist the evaluator to obtain portfolio 

investment costs, project contacts (e.g., 

principal investigators, technology 

developers), and other data. He/she will be 

responsible for ensuring peer review of the 

draft study plan and organizing others―both 

independent external experts and internal 

DOE experts―to participate in the review 

The involvement of the evaluation project 

manager is expected to increase again as the 

study nears completion, and the draft report 

is peer reviewed, the final report is 

published, and its results are used by the 

program office. 

Independent evaluators are chiefly 

responsible for successfully conducting an 

EERE evaluation study in compliance with

the standards and specifications set forth in 

the Guide. Early steps toward that success 

include working with EERE staff to scope

and plan the study and to make revisions and 

adjustments per external reviews of the plan 

and approach. The final steps are to provide 

the report and data documentation, and to 

assist with interpreting and communicating 

the findings 

Table I.3 highlights the respective roles of 

EERE staff and evaluators through the 14-

step process shown in Figure I.5. 

 
21 

Figure I.5 14 Step Evaluation Process 

Step 1 

•Begin an evaluation by defining the portfolio, preparing a logic model, and selecting elements for 

in-depth analysis 

Step 2 •Identify next-best alternative and determine how to assess additionality 

Step 3 •Estimate energy and economic impacts 

Step 4 •Estimate environmental impacts 

Step 5 •Estimate security impacts 

Step 6 •Estimate knowledge benefits 

Step 7 •Calculate economic performance metrics 

Step 8 •Calculate remaining Effective Useful Life (EUL) effects

Step 9 •Perform sensitivity analysis 

Step 10 •Provide a technological and context assessment 

Step 11 •Report results 

Step 12 •Document all input data, assumptions, calculations, and results

Step 13 •Use study findings to inform decision making and for effective communications 

Step 14 •Implement routine tracking and compilation of data to support future study updates 
22 

Table I.3 Roles of EERE Staff and of Evaluators throughout the Process 

Evaluation 

Steps EERE Staff Role Evaluator Role 

Before the evaluation begins

x Support the continuous tracking of technologies and 

compilation of data. 

x Prior to beginning the evaluation, verify that an 

evaluation of a R&D portfolio’s ROI is needed and 

timely. 

After deciding to perform an evaluation

 Step 1 x Work with evaluator to finalize selection of the 

portfolio and technologies within the portfolio for indepth analysis. 

x Connect evaluators with program informational 

sources, including cost data. 

x Participate as an internal reviewer and arrange for and 

oversee external and internal review of the proposed 

evaluation plan. 

x Work collaboratively with program staff to define 

portfolio and projects/technologies for detailed 

evaluation. 

x Get the analysis underway, following instructions 

in Part II, Step 1. 

Steps 2-11 x Answer evaluator’s questions as needed. x Estimate energy, environmental, energy security, 

and knowledge impacts. 

x Calculate economic performance metrics. 

x Calculate remaining Expected Useful Life (EUL) 

effects. 

x Perform sensitivity analysis. 

x Provide a formative analysis. 

Steps 12 x Participate in the internal review of draft report and 

data documentation. 

x Oversee the external and internal review of the draft 

report. 

x Submit draft report and data documentation for 

internal and external review. 

x Revise report and resubmit final draft publication 

report, accompanied by data documentation. 

x As necessary, work with a copy editor to prepare 

report for publication. 

Step 13 x Schedule a program office briefing on study approach 

and findings. 

x Work with evaluator to prepare a 1-2 page Executive 

Brief that provides an overview of the findings. 

x Use study findings to inform decision making, as 

necessary. 

x Communicate findings to stakeholders inside and 

outside EERE. 

x Prepare briefing slides and present seminar for 

EERE staff on study approach and findings. 

x Work with EERE staff to prepare a 1-2 page 

Executive Brief on findings. 

x Present study approach and findings in 

conferences and publications as desired (not 

required). 

Step 14 x Implement routine tracking and recording of data 

within the program to support future analyses. 

x Answer questions about the details of the study 

data as needed for EERE data collection. 
23 

PART II. Conduct an Evaluation Step-by-Step

Notation used in Part II 

Each of the 14 steps to follow in conducting 

an evaluation is introduced with the 

following arrow symbol, on which is 

superimposed the step number in the 

rectangle, and a listing of multiple sub-steps 

within the body of the arrow. The sub-steps 

are lettered sequentially in the text. 

Examples from past retrospective evaluation 

studies are presented for illustrative 

purposes throughout Part II, and are 

indicated in the text by the following 

symbol: ◙

Specifications of calculation formulations 

are indicated in the text by the symbol: 

Boxed text on a shaded field is used 

variously to highlight key points and to 

present examples. 

For Background in 

Conducting PART II, see 

PART I 

PART I introduces the Guide, explaining the 

approach taken and underlying concepts 

employed in PART II. It summarizes roles 

and responsibility in carrying out the 14 

steps. Once the material in PART I is 

understood, it is expected that the evaluator 

will work primarily in PART II, returning to 

PART I only as needed for clarification of 

the approach and underlying concepts. 

 

• Sub-step A 

• Sub-step B 

• Sub-step C 

STEP 

X 

o s

: 
24 

II.1 Begin an Evaluation 

1. Begin an Evaluation

 

Sub-section 1.B was prepared by Dr. Yaw 

Agyeman of Lawrence Berkeley National 

Laboratory 

1.A Define a portfolio for 

evaluation 

A starting point for the evaluation is 

defining an EERE portfolio for evaluation. 

The portfolio may be defined as: 

1) An entire EERE program or group of 

programs that share common attributes; 

2) A subprogram or group of subprograms 

within a larger program that share 

common attributes; or 

3) A grouping of projects/activities that cut 

across several programs but share a 

common feature that makes it reasonable 

and desirable to group them. 

EERE program staff may have defined at 

least a tentative portfolio for impact 

evaluation prior to engaging an evaluator. 

The evaluator, once engaged, may work 

with EERE staff to refine the portfolio 

definition. Whether defined in advance by 

EERE staff, or collaboratively with the 

evaluator, the following criteria are 

recommended for deciding on the portfolio: 

 The grouping makes logical sense; for 

example, it may represent a portion of 

one of EERE's three major technology 

portfolio groups: renewable energy, 

energy efficiency, or transportation; 

 The area of EERE investment has had 

sufficient time for commercialization or 

other implementation to have occurred; 

 The portfolio as defined may help to 

balance the representation of EERE's 

diverse R&D groupings with those that 

have already received evaluation; 

 The portfolio contains technologies that: 

(a) were not included in past EERE 

studies; (b) were included but had not at 

the time shown the degree of market 

penetration that has since occurred; or 

(c) were previously evaluated using an 

alternative approach not consistent with 

the approach presented in this Guide; 

and 

 The portfolio as defined represents an 

area of EERE investment that has 

emerged as a priority with stakeholders. 

 Define a portfolio for evaluation. 

 Assess evaluability of the portfolio. 

 Select technologies from the portfolio for in-depth analysis. 

 Characterize the portfolio and the selected technologies in the context 

of a logic model. 

 Compile investment cost and other data. 

 Describe qualitatively the elements of the portfolio not selected for 

detailed analysis.

1. Begin an 

Evaluation
25 

The evaluator should explain in the final 

report why the portfolio is defined as it is.21 

1.B Assess evaluability of the 

portfolio 

It is important that a determination is made 

as to whether the defined portfolio is 

evaluable, that is, whether it is ready to be 

evaluated and conducive to evaluation. 

Depending on the scope, importance, and 

the evaluation budget, the evaluability 

assessment might be informal, or it may be 

more extensive. An informal evaluability 

assessment might be conducted as part of 

the background research for the study to be 

done. A formal evaluability assessment 

would be more extensive. In the first 

instance, the evaluability assessment must 

ensure that the evaluation focus is within the 

R&D scope. 

The clear articulation of goals is the first 

step in an evaluability assessment. It is not 

uncommon for the evaluator to discover, 

upon arrival, that the program or portfolio to 

be evaluated lacks clear goals, and that this 

lack of clearly articulated goals, along with a 

lack of clarity of investment pathways, 

means that an evaluability assessment is 

necessary. Just as important is the 

acquisition of relevant data. The evaluator 

must determine, through initial consultation 

with program staff and stakeholders, that the 

requisite information that would be needed 

for the evaluation can be acquired. 

In addition, the outcome of interest must be 

clear and flow logically from the investment 

under investigation. Moreover, it must be 

 

21 The Program Office will make the initial decision on 

portfolio selection. In step 1, the evaluator may discuss the 

selection with program staff, and they may jointly agree to 

keep or to modify the selection. An evaluability assessment 

may also influence the final selection decision. 

measured only when an impact could 

plausibly have occurred. 

1.C Select specific technologies 

within the portfolio for in-depth 

analysis 

From the portfolio, the evaluator and EERE 

staff will identify a selection of technologies 

for in-depth impact analysis. The 

highlighted box provides an example of 

specific technologies selected from within a 

subprogram of the Vehicles Technologies 

Office (VTO). If the technologies contained 

in the portfolio appear fairly evenly 

distributed in terms of the extent of their 

commercial development, the selection may 

be made by a random draw. However, if 

some of the technologies have shown more 

extensive commercial development than 

others (as often occurs in R&D programs), 

selecting those with more extensive 

commercialization offers evaluation 

efficiencies. 22 

Technologies selected for detailed analysis 

may include applications that lie outside the 

area targeted by the EERE program. At the 

same time, the study should report that the 

reported achievement was not within the 

program's focus. The second highlighted 

box in this section gives an example for the 

Geothermal Technologies Office (GTO), 

where the Advanced Polycrystalline 

Diamond Compact (PDC) drill bit was a 

technology selected for an in-depth analysis, 

and the study found a strong positive 

spillover impact on the oil and gas industry. 

Table II.1 lists portfolios and technologies 

selected for detailed analysis in evaluation 

studies completed in 2010 and 2013. In 

 

22 See PART 1.2.3 for a discussion of how the distribution 

of project returns affects the selection process and, in turn, 

the efficiency of conducting the evaluation. 
26 

those five studies, the portfolios were 

defined along program or sub-program lines. 

Technologies selected for detailed analysis 

might include any of the following types: 

x Technologies embodied in a component, 

product, or a system; 

x Technologies embodied in a new or 

improved process (e.g., improved 

deposition method for solar photovoltaic 

thin films); 

x Technology platforms that have multiple 

and varied applications; and 

x Infratechnologies that are used to 

improve performance or reduce cost but 

are not directly present in the products or 

processes they influence (e.g., new 

research tools, modeling capability, data 

bases, or test results). 

The highlighted box provides an example of 

an infratechnology ― "new research tools" 

― selected for in-depth focus in a past 

EERE R&D evaluation. 

Laser and optical diagnostics and 

combustion modeling applied to heavy-duty 

diesel engines were selected in part because 

they are research areas with measurable 

milestones and outcomes that are directly 

associated with the ACE R&D 

subprogram’s research. 

 

◙ Example of New Research Tools 

Selected for Detailed Analysis:

An EERE evaluation study completed in 2008 

included a portfolio of combustion technologies, 

designated the Advanced Combustion Engine 

R&D (ACE R&D) portfolio. The specific 

technologies from the portfolio selected for 

detailed analysis were new research tools: 

x Laser diagnostic and optical engine 

technologies focused on heavy-duty diesel 

engines, and 

x Combustion modeling –in particular KIVA

modeling that simulates the fluid dynamics 

of combustion processes in internal 

combustion engines. 

Not selected for detailed study, but included in 

the ACE R&D portfolio, were Combustion and 

Emission Control and Solid State Energy 

Conversion. 

◙ Example of Selecting a Technology 

whose Main Application was Outside the 

Program's Focus: 

Advanced Polycrystalline Diamond Compact (PDC) 

drill bits were selected for detailed study in an 

evaluation that defined EERE's Geothermal 

Technology Office (GTO) investments as the 

portfolio. The PDC drill bits were designed better to 

withstand the high temperatures and hard rock 

encountered in drilling geothermal wells. 

An early adopter of these advanced drill bits was the 

petroleum industry, where the advanced drill bits 

also offered substantial advantages over existing drill 

bits. Over the timeframe of the analysis, many more 

oil and gas wells were drilled than geothermal wells, 

making the benefits of the new drill bit in the oil and 

gas application area many times larger than in 

geothermal applications. 

The application of the technology in the oil and gas 

industry in no way lessened the value of its benefits 

in geothermal applications. Rather, it increased total 

benefits and the social return on the GTO's 

investment in drill bits, which is a desirable 

outcome. Thus, it is acceptable to draw technologies 

for detailed analysis that have had large applications 

outside of EERE's initial intended focus. The study 

should in this case identify the application area as 

having a spillover benefit that goes beyond the 

geothermal industry targeted by the GTO 

investment. 
27 

 

◙ Table II.1 Portfolios and Selected Technologies for Detailed Analysis (2013 and 

2010 EERE Benefit-Cost Studies)*

Five Portfolios Technologies within Each Portfolio Selected for Detailed Analysis 

Energy Storage Technologies 

for Hybrid and Electric 

Passenger Vehicles and 

Light-duty Trucks (in the 

Vehicle Technologies Office) 

x Lithium-ion (Li-ion) battery technologies for electric and hybrid 

cars and trucks 

x Nickel Metal Hydride (NiMH) battery technologies for electric 

and hybrid cars and trucks 

Geothermal Technologies 

Office 

x Polycrystalline diamond compact (PDC) drill bits 

x Binary cycle power plant technology 

x TOUGH series of reservoir models 

x High-temperature geothermal well cements 

Wind Energy Program x Wind turbulence models 

x The unsteady aerodynamic experiment to acquire accurate 

aerodynamic and structural measurements 

x Turbine blade material characterization and analytical modeling 

work 

x Wind turbine component demonstration programs 

Solar Photovoltaic (solar PV) 

Energy Subprogram of the 

Solar Energy Technologies 

Office

x Crystalline silicon PV module technologies, Thin Film PV module 

technologies, Manufacturing technologies, Technology 

infrastructure for measurement, characterization, and reliability 

Advanced Combustion 

Engine R&D Subprogram of 

the Vehicle Technologies 

Office

x Laser diagnostic and optical engine technologies focused on 

heavy-duty diesel engines 

x Combustion modeling―in particular KIVA modeling that 

simulates the fluid dynamics of combustion processes in internal 

combustion engines 

* Note: In all of these examples, each portfolio is within a given program. However, a portfolio could alternatively be 

defined as crossing program lines or representative of the entire EERE. 
28 

1.D Characterize the portfolio 

and its components, including 

the selected technologies, in 

the context of a logic model 

Using a logic model,23 characterize the 

portfolio and the selected technologies. The 

logic model provides a model of how the 

elements of the portfolio were conceived, 

planned, expected to operate, and what they 

were intended to deliver. It is expected that 

the evaluator will review program and other 

documents and interview program staff to 

obtain the background information needed 

to develop this characterization of the 

portfolio. 

Describe portfolio goals, objectives and 

strategies, and activities undertaken in 

support of goal achievement. The activities 

are here considered synonymous with the 

EERE interventions intended to bring about 

the desired change or impact. 

The logic model representation of the 

portfolio should contain the basic elements 

shown in Figure II-1. The basic logic model 

elements are inputs or resources, major 

activities, outputs, intermediate outcomes, 

and ultimate outcomes and impacts. The 

intended audience is often also included, as 

shown. The logic model prepared for the 

portfolio should flesh out these basic 

elements in sufficient detail to depict the 

portfolio as defined for the study. It can be 

presented using a linear depiction, as shown. 

The logic model should be useful in 

explaining, presenting, and understanding 

the overall portfolio. But it also should 

 

23 There is extensive literature on logic models and their 

development. A logic model development guide, for 

example, is provided on-line by the W.K. Kellogg 

Foundation (see www.wkkf.org). Guidance is also provided 

by McLaughlin and Jordan (2010, 2nd ed.), and examples 

of logic models for public R&D programs can be found in 

Jordan, et al (2014).

provide sufficient detail that each of the 

technologies selected for detailed analysis 

can be located within the portfolio’s major 

activities. The logic model is a starting point 

and a tool for planning the evaluation―both 

the quantitative analysis of the selected 

technologies and the qualitative assessment 

of the parts of the portfolio not selected for 

detailed analysis. 

1.E Compile investment costs 

and other portfolio data 

Portfolio investment costs are defined as 

yearly expenditures incurred by DOE in 

implementing the portfolio's activities over 

the period covered. The data should include 

the entire portfolio cost; they should omit 

nothing within portfolio scope.24 

 

In the case of the five portfolios evaluated 

through 2013, the issue of other DOE 

investment costs (outside of EERE) has 

arisen only once: The development of 

EERE’s Advanced Combustion Engine 

(ACE) R&D's portfolio of technologies 

relied on a special facility, the DOE Office 

of Science's Combustion Research Facility 

(CFR), in addition to the EERE's ACE 

R&D's investment. Portfolio investment 

costs in this case include total ACE R&D 

investment costs plus the CRF research 

budget. 

Economic return is computed by comparing 

the benefits of the selected technologies 

against total portfolio investment cost. 

Nonetheless, the evaluator is encouraged to 

also obtain investment costs for the 

individual technologies selected for detailed 

analysis if these costs are available. 

Individual investment costs for a past 

 

24 Directly related expenditures by other parts of DOE are 

expected to be included, but related expenditures by other 

federal agencies or by other organizations are not to be 

included.
29 

evaluation are illustrated in the text box. The 

purpose is to know the share of total 

portfolio costs that are comprised by costs of 

the selected technologies. Knowing the 

share gives a sense of what portion of the 

portfolio has been analyzed in detail. See 

highlighted box. 

Total program and subprogram costs should 

be available for past years from the relevant 

EERE program. However, costs for 

individual technologies within a portfolio 

may be less readily available. 

Because total portfolio costs are essential to 

computation of the desired economic 

performance metrics, obtaining them is 

imperative; the cost data for individual 

technologies is desirable, but not essential 

for the evaluation. 

◙ Table II.2 EERE Geothermal 

Program Cost, 1976-2008 

(thousands $2008)

Year 

Program 

Expenses 

1976 $92,819 

1977 $130,899 

1978 $288,654 

1979 $367,328 

1980 $316,935 

1981 $324,090 

1982 $142,327 

1983 $93,412 

1984 $41,677 

1985 $57,793 

1986 $45,668 

1987 $34,891 

1988 $35,623 

1989 $30,829 

1990 $26,832 

1991 $43,236 

1992 $38,143 

1993 $31,619 

1994 $31,251 

1995 $50,360 

1996 $38,384 

1997 $37,373 

1998 $36,402 

1999 $35,194 

2000 $28,554 

2001 $32,205 

2002 $32,149 

2003 $30,550 

2004 $29,348 

2005 $27,414 

2006 $24,478 

2007 $5,107 

2008 $19,307 

Undiscounted Total $2,600,851 

Source: Gallaher et al. (2010). 

◙ Portfolio Investment Costs and Selected 

Technology Investment Costs

In the 2010 evaluation of EERE's solar photovoltaic 

energy R&D, the study defined the portfolio investment 

cost as the total EERE expenditure on Photovoltaic 

Energy Systems between 1975 and 2008. The total was 

$3,710 million in 2008 dollars. This is the amount that 

was compared against benefits to compute return on 

EERE's investment. 

The portfolio included the following three initiatives for 

which technologies research was selected for detailed 

analyses: (1) the Flat-Plate Solar Array Project (FSA) that 

had an investment cost of $535 million; (2) the 

Photovoltaic Manufacturing Project (PVMaT) that had an 

investment cost of $200.7 million; and the Thin-Film PV 

Partnerships Program (TFP) that had an investment cost 

of $294.3 million. 

The sum of the investment cost for the three initiatives 

examined was $1,030 million, while the cost for the entire 

solar PV R&D portfolio was $3,710 million. 

Thus, the combined investment cost of the selected 

technologies amounted to 28% of portfolio investment 

cost.
30 

 

Inputs 

.

.

.

.

Portfolio 

Strategies/ 

Activities

.

.

.

Portfolio 

Outputs 

Intended 

Audiences 

Participants

Users

Intermediate 

Outcomes 

Ultimate 

Impacts

◙ Figure II.1 Simplified Portfolio Logic Model Showing Basic Elements 

EERE program staff have responsibility for 

obtaining and providing portfolio costs to 

the evaluator. As soon as the portfolio is 

defined, the evaluator should request the 

cost data from the program, and program 

staff should take immediate steps to provide 

the investment cost data for the relevant 

portfolio, and for the selected technologies if 

available. 

Portfolio investment costs should be 

presented year-by-year over the entire 

period of EERE's investment in the 

portfolio. There should be clear designation 

of how the dollar values are expressed, i.e., 

in actual (current) dollars or in constant 

dollars, and, if the latter, it should be made 

clear in what base-year dollars they are 

expressed and what was the source for the 

conversion from current to constant dollars. 

It is preferred that the evaluator receives the 

expenditure data from EERE in current 

dollars and convert to constant dollars as 

part of the evaluation (as treated in Step 7). 

Table II.2, shown in the highlighted box, 

provides an example of year-by-year 

portfolio investment cost data that have been 

converted to constant dollars. The data are 

from the 2010 impact evaluation report on 

geothermal technologies. In this case, the 

portfolio was the entire GTO program. The 

year-by-year investment costs are carried 

forward into Step 7, and used to calculate 

economic performance measures. 

Additional data that the evaluator should 

seek to compile while performing this initial 

step are data that describe the portfolio over 

time in terms of its activities, participants, 

technical achievements, barriers overcome, 

outputs, and outcomes. A search, for 

instance, may turn up earlier studies that 

may show how key metrics, such as cost and 

performance, have changed over time. 

1.F Describe qualitatively the 

elements of the portfolio not 

selected for detailed analysis 

While the quantitative analysis centers on 

the technologies selected for detailed 

analysis, it is important also to provide a 

qualitative review of other elements in the 

program portfolio. What other research 

areas/ technologies does the portfolio 

contain? What is the status of these other 

elements? To what extent were there 

technical failures? To what extent are 

commercialized technologies still in an early 

market stage? How are they related to the 

technologies selected for detailed analysis? 
31 

The aim is to give a sense of the likely 

overall impact of the portfolio once the 

detailed quantitative analysis of the selected 

technologies is completed. Is there reason to 

believe that the other elements in the 

portfolio at a minimum will not offset 

positive impacts of the selected 

technologies? Is there reason to believe they 

have provided additional benefits? Might 

they provide additional benefits in the 

future? What type of benefits? What share 

of the total portfolio cost do they represent? 

The qualitative assessment of elements of 

the portfolio that are not selected for 

quantitative evaluation should be 

documented in the study report. 

 
32 

II.2 Identify Next-Best Alternative and Assess 

Additionality 

 

Sub-sections 2.B.1, 2.B.2, 2.B.3, and 2.B.4 

were prepared by Dr. Yaw Agyeman of 

Lawrence Berkeley National Laboratory 

Step 2 guides the estimation of energy and 

other resource effects resulting from end use 

of the technologies selected for detailed 

analysis. This step proceeds through two 

sub-steps (2.A and 2.B), with emphasis on 

identifying the next-best alternative for 

comparison and treating additionality in the 

analysis (i.e., the extent to which the 

estimated impacts have resulted from the 

EERE investment). 

2.A Identify next-best alternative 

for each technology selected for 

detailed analysis 

There are 3 tasks within this sub-step, called 

out by headers. The first is to develop an indepth understanding of the selected 

technologies that was begun in Step 1. The 

second is to characterize the selected 

technologies as infratechnologies, 

technology platforms, product technologies, 

or enabling product technologies. The third 

is to define the next-best alternative.

2.A.1 Develop an in-depth 

understanding of the selected 

technologies 

The objective of Step 2.A is to identify what 

alternative each of the selected technologies 

must be compared against in order to 

determine its resource effects. To do this, it 

is necessary to have a solid understanding of 

the technologies selected for detailed 

analysis. 

Continue the compilation of secondary data 

that was begun in Step 1, now focusing on 

the selected technologies for detailed 

analysis. Review technology assessments 

from the peer-reviewed literature. Look for 

accounts of events, breakthroughs, and 

explanatory factors in journal publications, 

reports, and presentations, taking note of 

mentions of EERE and/or DOE’s programs, 

investment, and technical support. In 

addition to researching trends for the period 

after EERE's investment began, consult 

 Identify next-best alternative for each technology selected for 

detailed analysis. 

 Assess additionality using the most rigorous evaluation design that 

is feasible. 

2. Identify 

Next-Best 

Alternative 

and Assess 

Additionality
33 

contemporary resources that document the 

state of the technology prior to that period. 

This "before-DOE's investment" 

investigation is necessary in order to acquire 

insight into the state of the art, challenges, 

and stated needs that led to EERE's 

involvement, and to establish the time 

period of analysis. To assess technical and 

market trends look for information on the 

relevant EERE programmatic initiative(s), 

technical and market trends, and funded 

entities for each selected technology or 

technology family in the subject portfolio. 

See also the results of Step 6 on knowledge 

benefits if that step is conducted 

concurrently or in advance of this step, to 

learn what inventions have resulted from the 

EERE funding and who has used them. 

Resources relevant for finding secondary 

data include: 

x Peer-reviewed scientific, engineering, 

and economic literature; 

x Business and industry literature and 

reports prepared by professional and 

industry associations; 

x Policy briefs, technical reports, and 

annual reports prepared by DOE offices 

and national laboratories; 

x Business and industry press releases; 

x Financial and market analysis reports; 

and 

x Economic and technical histories in the 

area of investigation. 

Assemble a timeline of EERE and nonEERE initiatives and technological 

milestones and achievements. The timeline 

should contain the following: 

x Time frames of EERE and non-EERE 

programmatic initiatives; 

x Periods of performance for firms 

working on EERE-funded and co-funded 

initiatives; 

x Targeted achievements and actual 

outcomes for firms receiving EERE 

support and, if possible, firms not 

receiving EERE support; 

x Technological and market milestones for 

technologies and technology families 

included in the detailed analysis; and 

x U.S. and non-U.S. policies or other 

exogenous shocks. 

Conduct initial unstructured interviews with 

a subset of knowledgeable individuals from 

government, national laboratories, academia, 

industry consortia, and firms. Candidate 

interviewees must have industry, academic, 

and/or management experience germane to 

the selected technology families. Examples 

include institute directors, program leaders, 

research directors, chief technology officers, 

and senior scientists working in the field. 

Discussion topics for initial interviews 

should include, but not necessarily be 

limited to: 

x State of the selected technologies and 

their rates of progress prior to EERE 

investment;

x Significance of EERE investment and 

support in introducing the new or 

improved emerging technologies or 

technology families or in revitalizing 

maturing technologies;

x Main technical challenges/barriers to be 

overcome;

x EERE goals, strategies, and targets;

x Technical performance characteristics 

that can be used to assess technological 

progress across the technology families 

(e.g., reliability, production cost, energy 

density, watt rating); 

x Technical contributions of EERE and 

national laboratory staff relative to 

academic and industry researchers’ 

contributions; 

x Infratechnology, technology platform, 

product technology characteristics of the 

selected technologies; 
34 

x Areas of application of the 

technologies,value chain characteristics 

and competitive landscape; 

x Alternative funders to EERE for 

technology development; 

x Demand-side initiatives and their 

relationship to the technology 

development initiative along dimensions 

related to technological progress and 

timing; and

x Opinions about next-best alternatives to 

the technologies selected for detailed 

analysis from the portfolio.

2.A.2 Characterize the selected 

technologies as infratechnologies, 

technology platforms, and/or product 

technologies (including enabling 

product technologies) 

As part of the in-depth characterization of 

the selected technologies, define and 

characterize the economic role of each 

technology according to a typology that 

characterizes a given technology by its 

infratechnology, technology platform, and 

product technology content. Correctly 

characterizing the technologies being 

evaluated is essential for next-best 

alternative selection and also for the 

quantification of additionality in sub-step 

2.B. A given selection of technologies, or 

even a given technology, may contain more 

than one type of technology characteristic, 

which may necessitate separate treatment of 

technologies by type in the next-best 

alternative analysis. 

Technology typology: 

Infratechnologies are technologies that 

influence the efficiency of R&D, 

production, and marketing of other 

technologies. Infratechnologies often 

represent advances in and understanding of 

scientific and technical phenomena, such as 

improvements in measurement and testing, 

concepts, tools, and techniques. Examples of 

infratechnologies are building efficiency 

standards, appliance standards, test methods, 

modeling capabilities, scientific databases, 

equipment calibration procedures, standard 

reference materials, and research tools and 

user facilities. 

Benefits of infratechnology may be 

measured, for example, in terms of gains in 

efficiency in applications to which they are 

applied and the resulting acceleration of the 

introduction of more efficient or productive 

products and services. The vehicle 

combustion study prepared by Link (2010) 

assessed the economic benefits associated 

with infratechnology. That study analyzed 

the use of laser and optical diagnostics and 

combustion modeling (a suite of 

infratechnologies) to improve combustion in 

heavy-duty diesel truck engines by 

advancing understanding and control of the 

in-engine combustion process. Application 

of infratechnology in R&D supports the 

development of technology platforms. 

Technology Platforms are precompetitive 

prototypes and concepts that serve as proofof-concepts and demonstrations of the 

commercial viability of new end-use 

products, processes, or services. They have 

alternatively been termed "generic 

technologies." Technology platforms may 

support the creation of multiple products and 

processes. Technology platforms are often a 

precursor to product technologies. 

There are numerous EERE-supported 

technology platforms that serve as the basis 

for multiple downstream products. One 

example is novel wind turbine blade 

designs, which improved the efficiency of 

wind turbines relative to preexisting 

turbines, and were adopted by multiple 

producers for a variety of commercialized 

wind turbines. 
35 

Product Technologies are technological 

innovations directly embodied in and closely 

identified with a specific new or improved 

product, process, or service.25 An example 

from the first round of EERE benefit-cost 

evaluation studies is geothermal hightemperature cement innovation embodied in 

the product, ThermaLock cement (Gallaher 

et al, 2010). 

Product technologies can be enabling in the 

sense of making possible the wider use of 

another technology. Examples of enabling 

product technologies are advanced batteries 

for powering electric cars, and essential for 

their expanded use; and binary cycle 

geothermal plant technology which 

expanded the geographical regions over 

which geothermal plants can be located to 

include lower temperature applications. 

Table II.2-1 presents technologies analyzed 

in previous EERE benefit-cost studies and 

their characterization. 

2.A.3 Define the next-best alternative 

Next, the evaluator identifies the next-best 

alternative that would have been used or 

would have occurred as an alternative 

scenario in lieu of each technology or 

technology family grouping being analyzed 

quantitatively. The merits of the technology 

being analyzed are judged against those of 

the best alternative, i.e., the best technology 

alternative that would otherwise have been 

used. The term “next-best alternative” is 

synonymous with the term “defender 

technology.” 

 

25 Note, too, that some product technologies are so 

ubiquitous that they become defacto standards (having 

quasi-public good content) despite no formal designation as 

such (e.g., Microsoft Windows).

For a retrospective benefit-cost evaluation 

such as that covered by this Guide, the nextbest alternative is defined by looking back to 

the time the investment decision was made 

for the new technology. Note that the nextbest technology at the time of the investment 

decision is not necessarily today’s next-best 

alternative. Also note that the analysis must 

account for the next-best alternative’s 

technological progress over the period of the 

evaluation. In other words, do not assume 

that the next-best alternative technology’s 

technical performance characteristics 

necessarily remained fixed. 

In the case of a comparison against an 

alternative counterfactual scenario, the nextbest alternative technology is the technology 

as it would have existed in the 

counterfactual scenario. For example, the 

next-best alternative for use in the solar 

photovoltaic evaluation prepared for EERE 

(O’Connor et al., 2010), was the state of 

solar photovoltaic modules in the absence of 

DOE support. By assessing counterfactual 

progress through in-depth survey procedures 

and technical assessments, O’Connor et al. 

constructed a dynamic counterfactual 

scenario against which actual progress was 

compared to quantify economic benefits. 

Many technologies (e.g., solar photovoltaic 

modules and wind turbines) existed prior to 

DOE involvement, such that DOE played a 

role in advancing their development, rather 

than in creating them. 

The next-best alterative analysis should 

consider not only domestically sourced 

alternatives but also internationally available 

alternatives. For example, in the case of the 

Photovoltaic Energy Systems benefit-cost 

analysis (O’Connor et al, 2010), 

contemporary programs funded by other 

national governments were reviewed for a 
36 

next-best alternative but none was found.26

This international look is important because 

of the global nature of the technologies that 

EERE supports. 

Conditions of use―as well as technology 

type―can affect the choice of next-best 

alternative. An illustration is provided by the 

evaluation of geothermal binary-cycle plant 

technology. In the evaluation of GTO R&D, 

binary-cycle plants were considered in two 

applications: (1) Application of binarycycle plant technology to take advantage of 

lower temperature reservoir resources, in 

which case the technology enabled the use 

of geothermal plants where they otherwise 

could not effectively be used; and (2) 

Application of binary-cycle technology 

where higher temperature resources existed 

and flash geothermal plant technology was a 

viable alternative. In the former application 

the technology is geothermal enabling; 

without the binary-cycle technology, 

geothermal could not be used and the nextbest alternative is a conventional power 

plant. In the latter application the binarycycle technology functions as an improved 

product technology offering productivity 

gains over the next-best geothermal plan 

alternative, a flash geothermal plant. Thus, 

the application area will, in some cases, 

affect the appropriate specification of nextbest alternative, and therefore should be 

taken into account by the evaluator. 

 

26 In 1974, following the 1973 oil price shocks, Japan 

created an energy R&D program called the Sunshine 

Project, which was intended to support solar photovoltaics, 

coal gasification and liquefaction, geothermal, and 

hydrogen technologies. The project was organized by the 

Agency of Industrial Science and Technology within the 

Ministry of International Trade and Industry. That 

program’s work was reviewed to see if technologies could 

have emerged and entered the U.S. market from it instead 

of the DOE-sponsored program. It is important to review 

such efforts to ensure that the counterfactual scenario is 

appropriately constructed.

Energy and other impacts of the batteries 

derive from their use in hybrid and electric 

vehicles. Therefore, a dynamic 

counterfactual scenario was constructed of 

market adoption of hybrid and electric 

vehicles, given the best available battery 

technology that would likely have been 

available in absence of EERE's battery 

R&D. 

In the case of a simple product technology, 

the identification of the next-best alternative 

tends to be more straightforward than for the 

other technology types. Geothermal hightemperature cement provides an example. 

The next-best alternative at the beginning of 

the study period and continuing throughout 

the study period was an alternative cement, 

identified as Portland Cement. 

It is a requirement that the study describe in 

the final report how the study has defined 

the next-best alternative for each technology 

or group of technologies selected for 

detailed analysis, and the rationale. Table 

II.2-1 gives examples of the next-best 

alternatives identified by recent EERE 

benefit-cost studies. 

To facilitate understanding, it may also be 

helpful to consider how not to define the 

next-best alternative. For this purpose, the 

text box gives an example in which the 

choice of next-best alternative is not 

appropriately aligned with the technology 

under evaluation. The example provided in 

the highlighted text box is of inappropriate 

alignment for a hypothetical wave 

technology. 

 
37 

◙ Table II.2-1 Technology Characterizations and Next-Best Alternatives in 

EERE Benefit-Cost Analyses 

Study Technology Focus Technology 

Characterization 

Next-Best Alternative 

Advanced Combustion 

Engine (ACE) R&D 

Subprogram

Laser and optical diagnostics Infratechnology The state-of-the-art in diesel engine 

design and related brake thermal 

efficiency (BTE) that existed prior to 

1995. 

Vehicle Technology 

Office Energy Storage 

Subprogram

Li-ion and NiMH energy storage 

technologies

Enabling product technology Lower market adoption of electric 

and hybrid versions of passenger 

vehicles with inferior battery 

technologies

Geothermal Technology 

Office R&D

Polycrystalline Diamond Compact 

(PDC) Drill Bits

Product technology Roller-cone drill bit technology that 

existed prior to introduction of the 

PDC drill bit and continued in use 

over the study period as market 

penetration of the PDC drill bit

occurred. 

Binary cycle power plant

x for reservoir temperatures in the 

range of 150 to 190°C 

x for reservoir temperatures below 

150°C 

Product technology Geothermal flash plant. 

Enabling product technology Conventional power plant.

TOUGH reservoir model series 

and 

related reservoir models

Infratechnology 

and technology platform

"Lumped parameter" models used 

before capabilities for detailed 

computer simulation of reservoirs 

were developed. 

High-temperature geothermal well 

cements 

Product technology Existing Portland cements.

Solar Photovoltaic Energy 

Systems Subprogram 

cSi modules and thin-film modules Technology platforms and 

product technologies 

Inferior solar photovoltaic modules 

due to slower progress without 

EERE investment. Standards and measurement science 

and infrastructure 

Infratechnology

Thus, appropriate next-best comparison can 

vary depending on whether the technology is 

a product technology (enabling or not), a 

technology platform, or an infratechnology. 

It can also vary according to conditions of 

use. These variations have been 

demonstrated in the examples given above. 

They point to the need for evaluators to be 

careful and diligent in specifying the nextbest alternative for comparison. The choice 

is critical because the basis of the 

comparison is an important determinant in 

the estimation of impacts. 

Do not proceed with economic analysis until 

the technology characterization, the 

specification of the next-best alternative, and 

the impact categories align in an internally 

consistent narrative, and there is no 
38 

disconnect or contradiction with the results 

of the desk analysis. 

2.B Assess additionality using 

the most rigorous evaluation 

design that is feasible 

An evaluation objective is to find the social 

return on EERE’s investment. This requires 

going beyond estimating benefits of the 

subject technology in comparison with its 

next-best alternative. It is also necessary to 

assess to what extent the estimated benefits 

have resulted from EERE’s investment 

rather than from other sources. 

There are three types of study design: 

experimental, quasi-experimental, and 

nonexperimental. The following subsections 

identify essential differences among these 

designs and discuss their strengths and 

weaknesses. 

This sub-step is divided into three tasks, 

identified by sub-heading. The first is to 

choose the study design. The second is to 

model how additionality occurs. The third is 

to strengthen the counterfactual approach, 

when counterfactual evaluation methods are 

used. 

Hypothetical Example of Incorrect Next-Best Alternative 

In 1980 the first commercial wave energy system was introduced to the market. Although the system had poor 

reliability and efficiency, there was a sufficient value proposition for some applications and for some consumers. A 

small number of systems produced by multiple firms were installed between 1980 and 1990. 

In 1990, a Federal agency launched a series of initiatives aiming to improve the efficiency, quality, and reliability of 

these energy systems. The agency-sponsored initiatives included test beds, standards, production technology 

development, new blade designs (introduced commercially in 1995), and other technologies. 

Experts stated that the wave energy system was accelerated by more than 10 years, and, indeed, there were impressive 

gains along all the technical and economic performance characteristics of interest. The rate of growth in system 

installation increased 10% annually between 1995 and 2010.

In conducting the evaluation and in specifying the next-best alternative in terms of a counterfactual scenario, the analyst 

delayed the introduction of wave energy systems by 10 years, citing the acceleration claims of interviewed experts. All 

of the technologies sponsored by DOE were necessary for successful commercial systems, the analyst argued, and, 

without all of them system deployment would have been delayed by 10 years. Over the 10-year delay, the next-best 

alternative was defined as traditional fossil fuel systems. 

Although the analyst correctly characterized the technologies in economics terms, he incorrectly established the 

counterfactual scenario and therefore the next-best technology alternative. In actuality, introduction of wave energy 

systems was not accelerated by 10 years, rather progress in wave technology was. Systems were available prior to the 

Federal program, and, while growth and progress would have been comparatively anemic without technological 

advance, there would have been some progress. The appropriate next-best alternative to improved wave technology 

development would have meant reviewing performance and economic characteristics without the improvements. The 

time series of benefits associated with advances in wave energy systems should have been estimated using a 

comparison of actual wave energy systems’ performance characteristics in a given year with what they would have been 

for the counterfactual case of performance characteristics absent the advances.
39 

2.B.1 Experimental study design 

An experimental study design is the most 

robust of the three evaluation study designs, 

because it compares groups that are identical 

except for having received or not received 

the intervention in question. Thus, any 

measured differences in outcome between 

the two groups can reasonably be attributed 

to the intervention. 

Prior to the intervention, the study 

population is randomly assigned into an 

intervention group and a control group. 

After the intervention, pre-post observations 

from each group are compared. It is a design 

widely used in medical and laboratory 

research. 

Experimental design is generally not 

practical or feasible for use in retrospective 

evaluation of research, development, and 

technology programs, where companies selfselect to participate or not; participants are 

known and may differ in a number of ways, 

such as size and age; populations tend to be 

small; and events occur in the field rather 

than in controlled environments. 

2.B.2 Quasi-experimental study 

design 

This design is similar to an experimental 

design, with the exception that participants 

are not randomly assigned into intervention 

and control groups, and a comparison group 

is constructed as a proxy control. Quasiexperimental research designs compare 

observations for those in the intervention 

group and those in the proxy control group 

to assess whether there are significant 

differences in outcome for those who 

received the subject program intervention 

versus those who did not. The more the 

intervention group and the proxy control 

group are alike in all ways except receiving 

or not receiving the subject program 

intervention, the more likely that observed 

difference in outcomes has resulted from the 

subject program intervention. If the 

comparison group is poorly constructed and 

validity threats are unsatisfactorily treated, a 

quasi-experimental design may not provide 

a robust assessment of additionality. 

2.B.3 Nonexperimental study design, 

including the counterfactual 

approach 

A nonexperimental study design lacks a 

proxy control group comprised of those who 

did not receive the program intervention. 

Thus, nonexperimental study designs do not 

provide as strong a basis for establishing 

causality as does experimental or quasiexperimental designs. Yet, in many 

circumstances, they are the only available 

design. 

When neither the construction of a true 

control group (experimental design) nor a 

proxy control group (quasi-experimental 

design) is feasible, the evaluator may use a 

counterfactual approach to increase the 

robustness of the approach for assessing 

additionality. This may entail interviews of 

those who received the intervention (i.e., 

participants) to assess what they would have 

done in absence of the intervention, and in 

effect use the results as a "pseudo 

comparison group"― i.e., the participants in 

actuality versus the participants acting in 

counterfactual mode. 

Alternatively, the evaluator may use 

interviews of independent experts (i.e., 

nonparticipants) and secondary data to 

establish a counterfactual case. 

As another alternative, the evaluator may 

employ both of the above―interview of 

participants and interview of independent 

experts―in conducting a nonexperimental 

design using a counterfactual approach.
40 

The use of a counterfactual scenario, while 

not as strong a basis for establishing 

additionality as an experimental or quasiexperimental design, is nonetheless better 

than a nonexperimental design without the 

use of a counterfactual.

2.B.4 Order of preference in selecting 

study design 

For evaluations covered by this Guide, the 

EERE standard for design selection is: 

x Chose experimental design― the gold 

standard for evaluation research ― when 

possible. If it is not possible to use 

experimental design, then the evaluator 

must consider using a quasiexperimental design. 

x If data and evaluation conditions exist on 

which a proxy control group27 can be 

established, then use a quasiexperimental design.28

x If it is not possible to use quasiexperimental design, then the evaluator 

must use nonexperimental design 

including the counterfactual evaluation 

approach. 

x A nonexperimental study design with 

counterfactual method can be 

 

27 While it is generally not possible outside an experimental 

design to draw a true control group―for which members of 

the group differ from the participant group solely in terms 

of not receiving the subject intervention, there are 

techniques that can be employed to control for at least part 

of the differences and bring a proxy control group closer to 

a true control group. 

28 An example of when a quasi-experimental design could 

be performed in the R&D evaluation context, is when it is 

feasible to identify firms who receive DOE R&D funding 

awards as well as firms who applied but were rejected 

because funding was not available for them (i.e., they were 

nearly as qualified in their proposed research to be awarded 

funds, but just missed the cutoff). Those just missing the 

award cutoff could comprise a comparison group to enable 

a quasi-experimental design to proceed as the design 

method. Still, unless the firms were similar to each other in 

other attributes, such as size, age, innovation propensity, 

and other factors that might cause difference in outcomes, 

steps would be needed to control for these differences.

strengthened through the use of mixed 

methods.29 For instance, a counterfactual 

design combined with use of additional 

methods is preferred to use of a 

counterfactual design alone. Mixed 

methods could take the form of using 

different data collection and analysis 

methods with the same population, or 

using different sources of data across 

different sub-populations within the 

study population, patent citation 

analysis, historical tracing analysis, and 

network analyses. For example, an 

assessment of the impacts of the Vehicle 

Technologies Office’s investments in 

energy storage technologies used patent 

citation analysis to provide evidence of 

documented linkages between patents 

resulting from EERE-sponsored R&D 

and advanced battery patents of leading 

research organizations. In a different 

assessment, such as a building 

technology assessment, one might be 

able to use different types of data 

(evaluation, measurement and 

verification; billings; surveys; focus 

groups) in the R&D study. 

x Nonexperimental designs that do not 

include the counterfactual evaluation 

method are unacceptable to EERE. 

2.B.5 Using a counterfactual 

approach to assess additionality 

When a nonexperimental, counterfactual 

approach is used to assess additionality, the 

evaluator should provide a robust analysis. 

The following are suggested ways for 

providing a robust counterfactual analysis 

when (a) interviewing/surveying participants 

about what they would have done had there 

been no EERE investment; and (b) 

interviewing/surveying nonparticipant 

 

29 Mixed methods are complementary data/analysis that 

produces additional lines of evidence to strengthen the 

results/findings. 
41 

experts about what they think would 

otherwise have happened: 

(a) Providing a robust additionality 

analysis when interviewing/ 

surveying participants about what 

they would have done had there 

been no EERE investment: 

Who and how many participants? 

x If participants received equivalent EERE 

support, draw a random sample for 

interview. 

x If the population of participants is large 

and heterogeneous in terms of the EERE 

support they received or have other 

characteristics, then in keeping with the 

portfolio approach, segment the 

population in subgroups based on 

distinguishing characteristics. Draw 

interview samples from each subgroup. 

Justify the sample design. 

x If the population of participants is small 

and heterogeneous, oversample on those 

who received the greatest amount of 

EERE support, or if there are too few 

participants to support a random sample 

then include 100% of the participants in 

the interview/survey. Justify the sample 

design. 

What to ask? 

x Design questions are to be free of bias, 

to have the same meaning for all those 

interviewed, and to be carefully crafted 

to get at the desired piece of information. 

If an interview guide is used it must be 

at least semi-structured. 

x Establish when the participant first 

became aware of EERE support 

programs. 

x Establish baseline technical and financial 

performance metrics relevant to the 

evaluation. Review technology progress 

over time with participant. 

x Relate EERE program to participant’s 

technical and performance metrics. Ask 

what the participant would have done 

had they not participated in the EERE 

programs. 

x Ask participants to rate the influence the 

EERE investment from the support 

program had on their R&D efforts. 

Establish what they would they have 

done differently, and how those actions 

would have affected their performance 

metrics. 

How to interpret results? 

x Use the interview/survey results to 

develop a counterfactual scenario. 

x Compare the actual behavior of 

participants with the counterfactual 

scenario to estimate additionality, taking 

account of variability in responses. 

x If the population of participants is 

heterogeneous, weight responses by 

market size or whatever measure is to be 

used to aggregate responses and/or scale 

to industry-level results. 

(b) Providing a robust additionality 

analysis when asking nonparticipant 

experts what they think would 

otherwise have happened: 

Who and how many experts? 

x Interview or survey EERE-funded 

principal investigators, EERE and other 

DOE programmatic and technical staff, 

executives with industry and trade 

associations, U.S. and international 

academic researchers, and other experts 

with significant germane industry, 

academic, and/or management 

experience. 

x Although there is no specified number of 

experts or participants that must be 

interviewed/surveyed, evaluators must 

report descriptive statistics for the 

populations, for those 
42 

interviewed/surveyed, and for impact 

data, including mean, median, standard 

deviation, and the number of 

respondents. 

x Increase the number of experts, such that 

the results are not dependent on the 

judgments of a few experts, especially 

when their views are widely divergent. 

Provide mean, median, and standard 

deviation, and confidence intervals for 

key impact result parameters. 

x Ensure that the sample of interviewed 

and/or surveyed experts includes nonDOE-funded experts. 

x Ensure that experts are qualified through 

professional experience to respond to 

questions about early DOE or EERE 

initiatives before asking them to provide 

assessments of pre- and/or post-DOE 

engagement. 

x Indicate how experts were selected and 

what part of the industry value chain 

they represent. 

x Consider how likely it is that results 

would be significantly different simply if 

different experts had been chosen. 

What to ask? 

x Provide experts advance materials 

including the timeline (in Step 2.A) and 

an initial list of rival explanatory factors 

(see 2.B.3) assembled earlier for their 

review prior to the interview time. 

x Ask the experts to comment on or offer 

any corrections to the timeline as part of 

the process of discussing rival 

explanatory factors and before they 

provide judgments. 

x Assemble question topics carefully to 

query on firm-specific and industry-level 

technical progress, with an emphasis on 

the technical and/or cost performance 

characteristics. 

x Begin by reviewing observed technical 

progress with the experts, and then 

determine what they think progress 

would have been in the absence of the 

R&D enabled by EERE funding and the 

technical support and infrastructure 

provided by EERE. Determine 

technological progress not only for the 

technology of interest but also the nextbest alternative. 

x Query changes in market behavior, 

resource efficiency, resource 

productivity, rate of adoption, market 

penetration, and acceleration effects, 

depending on the expert's area of 

specialty. 

x Capture impact data not only as point 

estimates but also as ranges for 

sensitivity analysis. 

Confirm at interview or survey close that 

judgments offered by the experts on 

counterfactual behavior reflect solely the 

effect resulting from subject EERE 

intervention. 

x Be sure the expert gives careful 

consideration to rival explanations 

factors when making final judgments, 

e.g., the normal patterns of technology 

maturation, business cycle impacts, 

changes in the costs of the factors of 

production due to factors other than the 

EERE intervention, investments by other 

private sector entities, other DOE and 

non-DOE actions by public entities in 

the U.S. and abroad, and demand-side 

policies such as tax credits. 

x If it is not possible for the expert to 

isolate the EERE-only effect, then 

apportion benefits using the share of 

EERE, other sources of public funding, 

and private funding for the outcomes. 

How to interpret results from either 

participant or nonparticipant experts? 

x If economic models are populated using 

firm-level responses as opposed to the 

industry-level responses, ensure that 

appropriate data for weighting the 
43 

response (e.g., production volume, 

installed capacity) are collected. These 

must be applied to each individual 

response when scaling to industry-level 

impacts. 

x Aggregate responses such that firm- or 

individual-specific responses are 

appropriately masked to protect any 

company proprietary data. 

x Fully document the number and 

distribution of responses, including 

providing the frequency and the standard 

deviation of the mean. 

x Prepare to test the sensitivity of analysis 

results to uncertainty arising from 

variation in responses by using a 95% or 

90% confidence interval, and specify the 

interval used. 

x Compare results to findings from the 

literature review, market and technical 

trends assessment, and timeline prepared 

earlier in Step 2, as well as with the 

R&D knowledge linkage analysis (Step 

6). 

2.B.6 Improve the reliability of the 

additionality assessment30 

When using a non-experimental study 

design or often when using a quasiexperimental design, additional effort will 

be needed to improve the reliability of the 

additionality assessment. One approach is to 

identify the mechanism by which 

additionality is expected to have occurred. 

This could be based on information obtained 

from the logic model, records on how 

resources were used and what activities were 

undertaken, as well as from interviews with 

experts. Table II.2-2 illustrates potential 

ways that additionality may occur. For 

 

30 As explained in sections 2.B.1 and 2.B.2, the assessment 

of additionality is internal to an experimental or robust 

quasi-experimental study design, and the use of these 

supporting assessment techniques may not be required 

when the study design is sufficiently strong.

instance, a program investment may 

accelerate technology development and 

commercialization, leading to an earlier 

accrual of benefits than would have resulted 

otherwise. Alternatively, EERE investment 

may provide incentive to private companies 

to broaden the scope and scale of their 

research, such that spillover effects are 

increased. Another mechanism is that EERE 

investment may foster collaborative research 

or provide special facilities shared by 

multiple companies, thereby eliminating 

duplicative research and lowering overall 

R&D costs for a given level of benefits. The 

EERE investment may also enlarge markets 

by increasing customer awareness and 

acceptance of emerging technologies, such 

as through demonstrations, standards, and 

purchase incentives. 

Knowing how the intervention has likely 

altered the outcome over what it otherwise 

would have been is helpful in computing the 

quantitative effects of additionality. If the 

effect is acceleration, for example, a lag 

effect may be used to show a slower rate of 

progress in benefit accrual without the 

intervention. An expansion of research 

scope and scale may be analyzed as a larger 

stream of benefits than would have 

otherwise occurred. Cost-reducing actions, 

such as fostering collaboration and 

providing special shared facilities may be 

analyzed as an increase in the investment 

cost that otherwise would have been 

required to provide the same level of 

technical performance. An expanded market 

may be analyzed as an increase in the use of 

the technology. 

The point is not to provide here an 

exhaustive treatment of how additionality of 

various interventions are treated 

quantitatively, but rather to note that it may 

be helpful for the evaluator to translate the 

“how” into a computational procedure that 
44 

can be used to estimate the share of benefits 

that resulted from the intervention. 

In addition, when using a non-experimental 

study design or often when using a quasiexperimental design, always strengthen the 

assessment of additionality by explicitly 

identifying and systematically eliminating 

rival explanations for observed impacts. 

Identifying and eliminating rival explanatory 

factors as the cause for an observed effect is 

a key element in establishing additionality 

when the evaluation study design lacks 

robust control or comparison groups to 

accomplish this goal. Examples of rival 

explanatory factors are energy tax incentives 

such as the production tax credit (PTC), 

interventions from other entities such as a 

utility rebate program, and sociocultural 

factors. 

Table II.2-2 Illustrative Ways that Additionality May Occur

Additionality Effects Examples 

Accelerating technology development & 

commercialization

x Expanding R&D funding to speed achievement of technical 

goals and commercialization. 

Broadening the scope of private-sector R&D x Increasing the scope and scale of private-sector R&D to 

overcome more difficult technical challenges with greater 

spillovers and social return. 

Reducing redundancy in pre-competitive private-sector 

research, which reduces the costs of technology 

advancement

x Fostering collaborative R&D to develop pre-competitive 

technologies that serve an entire industry, to avoid investment 

redundancy 

Enabling development of new research infratechnologies 

such as testing facilities, databases, research tools, 

standards, and models that will make an entire industry 

more efficient and which individual companies are 

unlikely to undertake

x Providing specialized facilities, services, and research tools 

needed by an entire industry to make technical advances. 

Increasing the size of the market and use of the 

technology

x Reducing barriers to market adoption, e.g., through 

demonstrations, information, training, and standards and 

certification activities 

x Increasing access of U.S. firms to global markets. 

In randomized control trials (RCTs) the use 

of randomization theoretically eliminates the 

influence of potential rival explanatory 

factors, known and unknown, across the 

treatment and control groups. The lack of 

randomization in quasi-experimental and 

nonexperimental designs imposes the need 

for careful treatment of potential 

confounders, rival explanatory factors.31 

 

31 Rival explanatory factors and confounders are used 

interchangeably in this guide. A rival explanatory factor 

that is not identified and its effects accounted for will 

“confound” the findings, which is to say that the 

intervention might be wrongly viewed to have caused the 

effect due simply to the fact that it operated at the same 

time as the true causative factor, or was correlated to it and 

to the outcome in some other way.
45 

The evaluator must explicitly identify and 

list the major rival explanatory factors. Once 

identified, the following steps are 

recommended: 

1. Develop an initial timeline for the rival 

explanatory factors 

2. Engage industry experts and study 

participants to clarify and refine the 

accuracy and completeness of the 

explanatory factors timeline 

3. Use the refined timeline to support the 

subsequent analysis. 

Figure II.2 presents an illustrative example 

of what such a timeline of explanatory 

factors might look like.32 Developing the 

initial timeline for rival explanatory factors 

provides the evaluator an explicit view of 

the major, known factors that could 

confound findings of effects, as well as the 

occurrence of these factors relative to the 

outcome. This explicit exposition is 

important to do early in the evaluation 

process because, during analysis, one of the 

considerations for determining possible 

confounding is whether there has been a 

plausible time lag between the major 

potential causes and the effect. Every major 

factor, including the EERE intervention, 

must be subjected to this basic 

determination. 

Data collection instruments designed for the 

study should explicitly inquire about the 

major rival explanatory factors to ensure that 

the study participants base their responses 

on the same underlying assumptions. 

Some rival explanatory factors may not lend 

themselves easily to a timeline. For 

example, a sociocultural factor might be a 

factor of interest in terms of its effects on 

the outcome, but would not be an easy fit on 

 

32 Figure II-2 is derived from an actual EERE evaluation 

study, but presented here for illustrative purposes only.

a timeline. Rather, it would provide a 

context that might interact with, or perhaps 

supersede, any effect from the intervention. 

Finally, the evaluator must be attuned to 

potential unknown confounders. These, 

however, may only be treatable through the 

use of statistical approaches (e.g., propensity 

score matching) that create statistical, 

pseudo comparison groups/factors for the 

analysis. 

When using a nonexperimental 

counterfactual approach, the identification 

of rival explanatory factors and their 

treatment in the analysis is critically 

important.33 Table II.2-3 is an example aid 

for use in soliciting opinion from 

nonparticipant experts about what 

DOE/EERE contributed as compared with 

other organizations. Walking these experts 

through Table II.2-3 or a similar device may 

assist them in preparing their responses to 

interview or survey questions. 

 

33 When a quasi-experimental design has a weak 

comparison group, it is recommended that, the major rival 

factors be explicitly treated.
46 

Figure II.2 Example of Timeline of Rival Explanatory Factors, Showing Wind Market Timeline and Capac

State of Illinois During the Period Covered by the Wind Powering America Initiative (1999

Source: Navigant Consulting, Inc. (2013). 
47 

Table II.2-3 Example Table to Aid with Identifying Rival Explanatory Factors

 Modified from Ruegg and Jordan, 2011.

 

 

Categories of Information 

Needed for Additionality

(Attribution) Assessment

Technology to Early Market Timeline Æ

Integrating, 

accelerating 

activities

Develop 

components, 

systems

Validate/ 

demonstrate, 

Commercialize

Manufacture, 

Supply

(in U.S.)

Early Market 

Adoption 

Capabilities 

for 

Continuing 

Growth

History of the technology/ 

market

What DOE Did

What Others Did (Rival 

Explanations—Private 

Sector and Other 

Nations)

What Others Did (Rival 

Explanations –US & 

State Government) 

The DOE Effect 

Description of DOE Influence

And its strength 

Basis of evidence of influence
48 

II.3 Estimate Energy and Other Resource Impacts 

 

Step 3 guides the estimation of energy and 

other resource impacts resulting from end 

use of the technologies selected for detailed 

analysis. This step proceeds through two 

sub-steps (3.A and 3.B), with emphasis on 

determining how the selected technologies 

affect the consumption of resources—

energy, labor, capital, and materials— and 

how to estimate impacts that have resulted 

from the EERE investment. 

3.A Estimate energy and other 

resource impacts year-by-year 

This sub-step comprises six tasks: 

1. Assemble data and develop a taxonomy. 

2. Take into account whether EERE’s 

effect is primarily through R&D or if it 

also entails market promotion strategies. 

3. Take into account import and export 

effects as needed. 

4. Estimate energy savings or capacity 

added in physical units by energy type, 

taking into account the next-best 

technology and additionality. 

5. Estimate other resource impacts in 

physical units, taking into account the 

next-best technology and additionality. 

6. Estimate the resulting year-by-year 

dollars of savings for all resource 

categories, showing, where a breakout is 

possible, results before additionality is 

taken into account and after.

34 

3.A.1 Assemble data 

Apart from data collected during interviews, 

time series data used in the economic 

analysis should come from U.S. government 

statistical sources, where available, such as 

the Energy Information Administration, 

Bureau of Labor Statistics, Bureau of 

Economic Analysis, or the U.S. Census 

 

34 When the next-best alternative is defined as the same 

technology in a scenario without the EERE investment, 

generally there will be no benefits absent EERE to report 

(e.g., the solar photovoltaic evaluation, O’Connor et al., 

2010, and the advanced combustion evaluation, Link, 

2008). In contrast, when the next-best alternative is defined 

separately from the EERE intervention, generally there will 

be benefits without EERE and a share of those benefits that 

are considered the result of the EERE investment (e.g., the 

geothermal evaluation, Gallaher et al., 2010).

 Estimate energy and other resource impacts year-by-year. 

 Calculate intermediate effects, and treat qualitatively any 

resource effects not captured by the quantitative assessment. 

3. Estimate 

Energy and 

Other 

Resource 

Impacts
49 

Bureau. If required data are unavailable 

from federal surveys, data from 

nongovernmental organizations, such as 

trade associations, professional associations, 

universities, and private entities may be used 

provided their use has been previously 

accepted, as evidenced by use in quality 

peer-reviewed publications and EERE 

assessments. For example, materials prices 

or technology-specific technical and 

economic performance characteristics may 

not be captured by federal surveys and only 

be available from such organizations. 

The exception is data that comes from 

interviews (unpublished), such as, for 

example, a time series of profit data 

constructed based on profit margins. 

It may be useful to also relate the physical 

change metric or the economic value of the 

metric to metrics that are commonly 

reviewed by EERE analysts. For example, 

although an analysis may explore production 

cost per watt, it would be helpful to EERE 

renewable energy technology offices to 

discuss the implications of this metric for 

Levelized Cost of Energy (LCOE). The 

evaluator should review the metrics used by 

the technology office and discuss how the 

metrics used by the analysis relate to the 

metrics used by EERE staff. 

The use of rules of thumb (e.g., 5-year 

assumed acceleration) is unacceptable. 

Rules of thumb are often simplifying 

assumptions, which can introduce 

calculation errors. For example, a 2001 

study used a fixed 5-year acceleration 

assumption for all technologies it examined. 

However, recent counterfactual-based 

studies of EERE technologies have shown 

empirically-derived accelerations of 2, 6, 

and 12 years, depending on the technology. 

Using a fixed rule of thumb in a R&D 

impact analysis risks adding unacceptable 

uncertainty to the accuracy of calculated 

returns and other impacts. 

3.A.2 Take into account whether 

EERE’s effect is primarily through 

R&D or if there are also market 

adoption strategies 

There is a strong symbiotic relationship 

between technology R&D programs and 

demand side policies. Indeed, these are 

complementary tools that policymakers use 

to catalyze technological and market 

solutions deemed to be in society’s best 

interest. Whereas R&D initiatives spur 

technological progress, demand side policies 

stimulate technology adoption. It is possible 

for a technology’s progress (e.g., efficiency, 

cost) to be accelerated by some number of 

years but for there to be negligible changes 

in the demand for that product. However, a 

tax credit, for example, could stimulate 

demand for the product. Yet, the tax credit 

may not have been enacted if technological 

progress had not been promising. 

Most EERE R&D investments have their 

impacts mainly on the supply side—

reducing production costs and increasing 

performance. The objective of the EERE 

evaluation in this case is to quantify the 

returns to R&D for each product unit 

installed or in operation. Demand side 

policies should be left unchanged. 

Therefore, collect impact data based on 

changes in cost/price or quantity from 

experts and program participants. Without 

having specific information on relevant 

elasticities, interviewed company 

participants and industry experts should not 

be expected to reason through simultaneous 

changes in quantity and price/cost. If annual 

quantity demanded is to be left unchanged, 

interviewees should be instructed that when 

providing impact estimates in terms of cost 

reductions, cost reductions from returns to 
50 

scale remain as observed. In this case, 

interviewees should be asked to provide 

estimates of cost changes influenced only by 

R&D. 

However, in the case of an EERE R&D and 

deployment program, EERE is directly 

supporting both technology development 

and market adoption. In these cases, not 

only can EERE claim economic benefits for 

technological progress, but also for market 

facilitation activities that serve to increase 

the number of installations or units in use. 

The evaluator is not only quantifying the 

returns to technological progress (e.g., 

changes in efficiency, productivity) but the 

entire returns associated with the combined 

R&D and deployment investments. In this 

case, estimates likely will be needed both for 

changes in costs and performance, but also 

in quantity. 

3.A.3 Take into account import and 

export effects as needed 

The following items apply to treating 

imports and exports in EERE evaluations: 

x For U.S. exports, estimation of producer 

surplus is acceptable; however the 

consumer surplus must be ignored as it 

accrues to a non-U.S. stakeholder. 

x For U.S. imports, estimation of producer 

surplus is acceptable for U.S. firms’ 

overseas production. Such imports can 

be treated in the same manner as 

domestically-produced, domesticallyconsumed installations. 

x For U.S. imports of foreign firm 

production, where there is sufficient 

evidence of EERE influence, estimation 

of consumer surplus is acceptable but the 

estimation of producer surplus is not. 

Thus, for exports, the capture of rents from 

non-U.S. parties and the cost savings 

associated with any production changes are 

calculable as benefits. Changes in 

environmental emissions, energy security, or 

other benefits are not calculable as they 

accrue to non-U.S. parties. Imports from 

non-U.S. firms can be included in 

environmental and energy security benefits 

estimation if there is a scenario under which 

non-U.S. firms’ product offerings were 

accelerated in direct response to competitive 

pressures from U.S. firms, where results are 

adjusted to reflect the extent of acceleration 

resulting from EERE investment. U.S. 

subsidiaries of international firms can be 

treated as domestic firms in this analysis if 

they received EERE funding. 

3.A.4 Estimate energy savings or 

capacity added in physical units by 

energy type, taking into account the 

next-best alternative and additionality 

It is at this stage that incremental energy 

savings and/or MW capacity is derived from 

differences in the performance 

characteristics of the selected technology 

and its next-best alternative. There are a 

variety of ways resource savings and 

associated energy/fuel changes can be 

calculated, and the specific approach 

depends on the selected technology and 

available data. The calculation approach and 

results should be fully documented in the 

study report. 

The following example illustrates the 

calculation of fuel savings in the Advanced 

Combustion Engine (ACE) R&D BenefitCost Study, where billions of gallons of 

diesel fuel were saved. (See the background 

in the highlighted box below and then 

Figure II.3-1.) 
51 

Source: Link (2010). 

◙ Figure II.3-1 Comparing Actual and Counterfactual BTE

While all resources are important to 

estimating economic benefits, there is a 

special data requirement for energy effects. 

Differences in the quantity of energy by type 

associated with technology selection not 

only figure into economic benefits, but also 

drive two of the other categories of 

impacts—environmental and energy security 

impacts. (Note that the quantity changes in 

fossil fuel consumption by type estimated 

here are carried forward to Steps 4 and 5.) 

Thus, all energy effects by type and physical 

quantity should be broken out for separate 

treatment. 

The best metric(s) for quantifying resource 

effects is that which is closest to the goal of 

the technology. For the solar photovoltaic 

evaluation, for example, it was change in 

production cost per watt. For the combustion 

efficiency evaluation, it was change in break 

thermal efficiency. These metrics best 

captured the objective of the R&D for these 

two particular technology assessments. 

An illustration of estimating an energy 

impact is provided in Table II.3-1 from the 

geothermal benefit-cost study (Gallaher et 

al., 2010). The table shows the estimation of 

energy resulting from productivity gains 

attributed to the Geothermal Technologies 

Office due to its contributions to 

development of reservoir models. Benefits 

of the new reservoir models in comparison 

with the next-best alternative are given, as 

◙ In the ACE R&D study, a statistical approach was adopted for the calculation of the 

fuel savings associated with miles per gallon (MPG) fuel economy improvements, 

where MPG improvements were linked to a 4.5% DOE-supported R&D improvement 

in Brake Thermal Efficiency (BTE) – shown in Figure II-2-1 (expert-derived 

counterfactual BTE). The change in MPG was statistically estimated (∂MPG / ∂BTE = 

0.153) and the reduction in MPG absent the ACE R&D research was calculated. The 

calculated reduction in MPG was translated to reduced fuel consumption of 17.6 billion 

gallons of diesel fuel saved over the period 1995 to 2007. Brake Thermal Efficiency 
52 

well as that part of these benefits estimated 

to result from the EERE investment. 

An illustration of resource savings from 

using PDC Drill Bits is illustrated in Figure 

II.3-2 (Gallaher et al. 2010). A main 

advantage of PDC drill bits over the 

conventional roller cone bits is that they 

allow higher rates of penetration, reducing 

the time that expensive drill rigs must be 

rented. Oil well footage drilled, percentage 

of well footage drilled with PDC bits, and 

average savings per foot drilled were inputs 

to the calculated savings shown in Figure 

II.3-2. 

Finally, data appearing in report tables may 

be independently rounded to reflect the 

desired number of significant digits; 

however, rounded values may not be used in 

any intermediate calculations, because it 

becomes difficult to replicate results 

electronically. 

3.A.5 Estimate other resource 

impacts in physical units 

Also compile changes in other resources in 

physical units. Use the collected data to 

perform year-by-year comparisons of the 

selected technology or group of technologies 

against the appropriate next-best alternative 

to derive the differences in the other types of 

affected resources stated in physical units. 

For example, what are the effects on labor 

requirements? What are the differences in 

purchase, installation, and maintenance 

costs? What are the differences in 

reliability? These effects, like energy, may 

differ among the selected technologies or 

groups of technologies and, if so, will 

require separate, individual comparisons of 

each technology or group of technologies 

with its respective next-best alternative. 

3.A.6 Estimate resulting year-by-year 

dollars of savings for all resource 

categories 

Compile additional data needed to compute 

year-by-year economic resource savings, 

such as prices and labor rates. 

Estimate year-by-year economic benefits for 

each selected technology or group of 

technologies. Include the effects of all 

affected resources—including energy, labor, 

and capital. If the estimates are in actual 

(current) dollars, convert from current to 

constant terms. The conversion from current 

to constant dollars is done using Gross 

Domestic Product (GDP) Implicit Price 

Deflators from U.S. Department of 

Commerce's Bureau of Economic Analysis 

(BEA). These price deflators are routinely 

updated by BEA, and to prevent variations 

among the benefit-cost studies merely from 

using different price deflator series, this 

Guide prescribes the use of the same series 

of price deflators to be used by all studies 

conducted within a funding cycle. See also 

Step 7. 

Show valued in dollars the calculated yearby-year resource benefits in tabular form for 

each separately treated technology or group 

of technologies within a portfolio, as well as 

for each type of economic benefit. Show the 

sum of constant dollar cash flows as 

undiscounted economic benefits. Where 

there is a clear separation of the estimation 

of benefits prior to and after the 

additionality analysis, report the results 

separately. 

Repeat the above tabular treatment for each 

separately treated technology or group of 

technologies within the portfolio. For 

example, the referenced geothermal study 

provided separate analyses for four distinct 

technologies―one from drilling, one from
53 

reservoir development, one from well 

preparation, and one from plant 

technologies. 

Provide an overall summation table of 

resource impacts that brings together the 

year-by-year constant dollar cash flows of 

energy impacts from 3.A.4 and other 

resource impacts, such as those for labor 

cash, for each technology or group of 

technologies within a portfolio. 

3.B Calculate intermediate 

effects, and treat qualitatively 

any resource effects not 

captured by the quantitative 

assessment 

This sub-step completes Step 3. It entails 

two tasks. The first is to record intermediate 

effects. The second is to assess whether 

there are any resource effects that were not 

captured by the quantitative assessment, 

and, if so, treat them qualitatively. 

3.B.1 Analyze intermediate results 

metrics 

Various stakeholders have an interest in 

several of the intermediate metrics of 

benefit-cost studies, in addition to the final 

outcome metrics. Intermediate metrics are 

those that can be derived as part of the 

analytics of calculating the final outcome 

type metrics. 

Economic analyses can be performed using 

software packages or programming 

languages that best suit the evaluator and 

evaluation; however, data, calculations, and 

results must be available in spreadsheet 

format in order to provide one of the 

required deliverables. Also, the analysis 

must be fully replicable from tables, figures, 

formulae, and text presented in the report, 

and readers will expect to be able to follow 

all calculations. 

For example, in the solar photovoltaic 

analysis (O’Connor et al., 2010), figures 

were drawn that showed how technology 

development forced down the production 

cost per watt, all else held equal. Figure II.3-

3 below shows the acceleration effect of 

EERE’s Solar photovoltaic R&D. In the 

absence of EERE partnership funding, 

technical expertise, and technology 

infrastructure, industry progress would have 

proceeded at a slower pace. The acceleration 

effect due to EERE R&D is estimated to be 

12 years, on average. As shown in Figure 

II.3-3, solar photovoltaic module production 

cost per watt, as a result of EERE R&D, was 

$8.16 in 1988 instead of reaching that cost 

in 2000 had there been no EERE R&D. 

3.B.2 Treat qualitatively any resource 

effects of the technologies selected 

for detailed analysis not 

quantitatively captured 

As noted in Section I.2.3, not all resource 

effects can be quantified either in monetary 

terms or in physical units. If there are 

resource effects for the selected technologies 

that cannot feasibly be quantified in 

monetary or physical units, describe those 

effects qualitatively. 

 
54 

◙ Table II.3-1 Example from the Geothermal Benefit-Cost Study of the Estimation of 

Productivity Benefits from Use of Reservoir Models

(1) 

Year 

 

(2) 

Price per 

MWh of 

Electricity 

($2008) 

(3) 

Geothermal 

Electricity 

Generated 

(MWh) 

(4) 

Electricity 

Generated by 

Geothermal 

After TOUGH 

Model for 

Reservoir 

Simulation 

Became Widely 

Used (MWh) 

(5) 

Share of 

Geothermal 

Power 

Generated Due 

to Reservoir 

Modeling―ba

sed on 10% 

share (MWh) 

(6) 

Economic 

Benefits from 

Reservoir 

Modeling 

(prior to 

attribution* 

analysis) 

(thousands of 

$2008)

(7) 

Economic 

Benefits from 

Reservoir 

Modeling (prior 

to deduction of 

capital costs) 

Attributed to 

DOE (thousands 

of $2008)

1979 na 3,888,968 na na 0 $0 

1980 $106.8 5,073,079 1,184,111 118,411 $12,646 $2,909 

1981 $114.2 5,686,163 1,797,195 179,720 $20,524 $4,721 

1982 $119.4 4,842,865 953,897 95,390 $11,390 $2,620 

1983 $118.6 6,075,101 2,186,133 218,613 $25,928 $5,963 

1984 $113.4 7,740,504 3,851,536 385,154 $43,676 $10,046 

1985 $113.5 9,325,230 5,436,262 543,626 $61,702 $14,191 

1986 $111.0 10,307,954 6,418,986 641,899 $71,251 $16,388 

1987 $106.7 10,775,461 6,886,493 688,649 $73,479 $16,900 

1988 $102.8 10,300,079 6,411,111 641,111 $65,906 $15,158 

1989 $100.7 14,593,443 10,704,475 1,070,448 $107,794 $24,793 

1990 $98.7 15,434,271 11,545,303 1,154,530 $113,952 $26,209 

1991 $97.9 15,966,444 12,077,476 1,207,748 $118,238 $27,195 

1992 $96.7 16,137,962 12,248,994 1,224,899 $118,448 $27,243 

1993 $96.1 16,788,565 12,899,597 1,289,960 $123,965 $28,512 

1994 

1995 $91.7 13,378,258 9,489,290 948,929 $87,017 $20,014 

1996 $89.6 14,328,684 10,439,716 1,043,972 $93,540 $21,514 

1997 $87.9 14, 726,102 10,837,134 1,083,713 $95,258 $21,909 

1998 $85.5 10,884,950 1,088,495 $93,066 $21,405 

1999 $83.0 10,938,045 1,093,805 $90,786 $20,881 

(Continued) 
55 

2000 $83.3 10,204,190 1,020,419 $85,001 $19,550 

2001 $87.2 9,851,533 985,153 $85,905 $19,758 

2002 $84.8 14,491,310 10,602,342 1,060,234 $89,908 $20,679 

2003 $85.8 14,424,231 10,535,263 1,053,526 $90,393 $20,790 

2004 $85.3 14,810,975 10,922,007 1,092,201 $93,165 $21,428 

2005 $88.3 14,691,745 10,802,777 1,080,278 $95,389 $21,939 

2006 $93.5 14,568,029 10,679,061 1,067,906 $99,849 $22,965 

2007 $93.3 14,637,213 10,748,245 1,074,825 $100,281 $23,065 

2008 $98.2 14,859,238 10,970,270 1,097,027 $107,728 $24,777 

Total NB, 

Undiscounted $548,675 

Gallaher et al, 2010.

Notes: Cols 2 and 3 sourced from Energy Information Administration (EIA) by Gallaher et al. (2010); Col 4= from Col 3, energy 

in each year after 1979, minus energy generated in 1979; Col 5=Col 4 energy amount x 10%; Col 6= price in Col 2 times the 

energy amounts in Col 5; and Col 7= DOE attribution rates for TOUGH (80%) and for other reservoir model development (20%). 

Increased capital costs due to use of the reservoir models offset some of the benefits. These are not included in the Table for

greater ease in exposition, but they are included in the benefit-cost study by Gallaher et al. (2010). 

Resource benefit cash flows are shown both before and after the estimation of additionality (which is called "attribution" in the 

table). 

◙ Figure II.3-2 Actual and Counterfactual Solar Photovoltaic Module Production Cost
56 

◙ Table II.3-2 Example from the Geothermal Benefit-Cost Study Showing the Estimation 

of Drill Rig Rental Savings from Use of PDC Drill Bit Technology

Year

Crude Oil Well Footage 

Drilled with PDC Bits 

Savings Based on Wells’ Footage 

(Thousand $2008) 

1982 5,450,440 $467,911 

1983 7,584,720 $651,136 

1984 12,028,720 $1,032,646 

1985 12,787,400 $1,097,777 

1986 8,148,536 $699,538 

1987 8,013,192 $687,919 

1988 7,896,244 $677,880 

1989 6,495,548 $557,632 

1990 9,152,820 $785,755 

1991 9,663,764 $829,618 

1992 8,156,540 $700,226 

1993 8,300,148 $712,554 

1994 7,115,144 $610,823 

1995 7,815,280 $670,929 

1996 9,066,535 $778,347 

1997 12,005,000 $1,030,610 

1998 7,997,250 $686,551 

1999 4,444,750 $381,574 

2000 7,958,000 $683,181 

2001 11,795,225 $1,012,601 

2002 10,504,000 $901,751 

2003 14,823,000 $1,272,530 

2004 16,867,000 $1,448,004 

2005 23,271,600 $1,997,829 

2006 33,625,200 $2,886,668 

2007 44,910,000 $3,855,450 

2008 48,893,400 $4,197,418 

Total $31,314,860 

Gallaher et al., 2010. 

 
57 

II.4 Estimate Environmental Impacts

 

This section describes how to estimate 

environmental impacts, particularly changes 

in air pollution emissions, changes in the 

incidence of adverse health events (and their 

economic value), and the economic value of 

changes in GHG emissions. It also explains 

how to account for other notable 

environmental benefits. 

4.A Estimate changes in air 

pollution emissions 

The estimation of environmental benefits 

likely began in Step 3 with the 

determination of changes in energy 

consumption by fuel type (i.e., coal, natural 

gas, oil). Bring forward to Step 4 any yearby-year changes in energy expressed in 

physical units, and estimate the 

corresponding air emission changes. 

Estimate changes in GHGs, including 

carbon dioxide (CO2), methane, and nitrous 

oxide, for example. Estimate emissions 

changes by GHG and convert to CO2

equivalents using the factors listed in Table 

II.4-1. Report changes in GHG emissions in 

terms of CO2 equivalents based on 

individual GHGs’ 100 year global warming 

potential, as estimated by the 

Intergovernmental Panel on Climate Change 

(IPCC) (1995). 

Also estimate emission changes for other air 

pollutants, including particulate matter 

(PM2.5), carbon monoxide, sulfur dioxide 

(SO2), nitrogen oxides (NOx), and volatile 

organic compounds (VOCs), as appropriate. 

Refer to emissions changes associated with 

reduced fuel combustion estimated in Step 3 

as “avoided emissions” and those associated 

with increased fuel combustion as 

“increased emissions.” This section assumes 

that evaluations will normally deal with 

reduced fuel consumption and, hereafter, 

will use the term “avoided emissions.” 

 Estimate changes in air pollution emissions. 

 Estimate avoided adverse health events and environmental 

health benefits using COBRA. 

 Estimate the value of changes in GHG emissions using the 

social cost of carbon. 

 Describe other notable environmental effects. 

4. Estimate 

Environmental 

Impacts
58 

Table II.4-1 100 Year Global Warming Potentials of Greenhouse Gases Used for National 

Inventory Reporting 

Name Chemical 

Formula

100 Year Global Warming Potential

(CO2 equivalent Conversion Factor)

Carbon dioxide CO2 1

Methane CH4 21

Nitrous oxide N2O 310

Sulfur hexafluoride SF6 23,900 

HFC-23 CHF3 11,700 

HFC-32 CH2F2 650 

HFC-125 CHF2CF3 2,800 

HFC-134a CH2FCF3 1,300 

HFC-143a CH3CF3 3,800 

HFC-152a CH3CHF2 140 

HFC-227ea CF3CHFCF3 2,900 

HFC-236fa CF3CH2CF3 6,300 

PFC-14 CF4 6,500 

PFC-116 C2F6 9,200 

Perfluoropropane C3F8 7,000 

Octafluorocyclobutane c-C4F8 8,700 

Perfluorobutane C4F10 7,000 

Perfluoropentane C5F12 7,500 

Perfluorohexane C6F14 7,400 

Note: Global warming potentials are consistent with international reporting guidelines. Greenhouse gases that are short-lived 

and vary regionally such as black carbon and water vapor, are not included in national inventories and need not be treated in 

the evaluation. Source: Intergovernmental Panel on Climate Change (IPCC) (1995); Second Assessment Report. 

If changes in energy generation estimated in 

Step 3 are not directly associated with a fuel 

source, the evaluator must make appropriate 

assumptions for what fuel sources are 

affected. A simplifying assumption for 

technologies that displace electricity 

consumption would be to assume 

displacement of the average fuel source per 

MWh based on data from the Energy 

Information Administration (EIA). If statelevel results are available, the estimates of 

fuel sources displaced should be refined 

using the energy mix by state. See, for 

example, the estimate of avoided fuel 

consumption for electricity generation in the 
59 

solar photovoltaic evaluation study prepared 

by O’Connor et al. (2010; Chapter 6). 

For renewable energy topics, more detailed 

analyses may be needed to capture the 

differential displacement of peaking and 

base units depending on the expected timing 

of energy generation or savings. For 

example, geothermal would be considered 

base load, but solar is akin to peaking units 

because of the timing of these resources. 

Estimating changes in emissions from 

changes in fuel consumption requires the use 

of emission factors.35 Where the relevant 

emissions factors are technology-specific, 

such as an all-electric vehicle or a 

conventional gasoline vehicle, use the 

appropriate technology-specific emissions 

factors. Stationary point and area source 

emissions factors are available using the 

EPA WebFIRE.36 Mobile source emissions 

factors are available from the EPA Office of 

Transportation and Air Quality.37 

If the relevant emissions factor is for a 

sector, such as for coal-fired electricity 

generation, use EPA’s National Emissions 

Inventory (NEI). NEI data in 2008 provides 

county-level emissions estimates at a 

detailed sector level.38 These total emissions 

must be converted to a per unit emissions 

factor using relevant usage data, such as 

electricity generated based on EIA 

 

35 An emissions factor is a representative value of the 

quantity of a given pollutant released to the atmosphere in 

association with using a designated fuel in a given activity. 

Emission factors are typically expressed as pounds of 

emissions per MWh of electricity produced from different 

fuel types including natural gas, coal, oil, nuclear energy, 

municipal solid waste, hydropower, and a variety of 

renewable energy sources. 36 The Factor Information Retrieval System is available 

online at http://www.epa.gov/ttn/chief/webfire/index.html. 37 Available at http://www.epa.gov/otaq/models.htm . 38 Available at 

http://www.epa.gov/ttn/chief/net/2008inventory.html.

estimates. In addition to the detailed data in 

2008, national emissions estimates by Tier 1 

sector are available from 1970 to 2012 and 

national emissions by Tier 3 sector from 

1970 to 1998.39 For GHGs, evaluators 

should use emissions factors used by EPA’s 

Center for Corporate Climate Leadership.40

Use linear interpolation to estimate data for 

any missing years. 

4.B Estimate avoided adverse 

health events and 

environmental health benefits 

using COBRA

EPA's Co-Benefits Risk Assessment 

(COBRA) model has been adopted by the 

Guide to provide first-order estimates of 

avoided adverse health events and their 

economic value, termed environmental 

health benefits, resulting from avoided air 

emissions. COBRA is a user friendly tool 

designed to be methodologically consistent 

with EPA’s Environmental Benefits 

Mapping and Analysis Program (BenMAP), 

which is a detailed set of modeling tools 

used by EPA for regulatory analysis, such as 

for the Regulatory Impact Analysis of the 

Final Mercury and Air Toxics Standards 

(U.S. EPA, 2011).41 Use of COBRA enables 

the health impact functions and the unit 

economic values used in EERE benefit-cost 

studies to be consistent with EPA analyses. 

 

39 Available at 

http://www.epa.gov/ttn/chief/trends/index.html. 40 Available at 

http://www.epa.gov/climateleadership/inventory/ghgemissions.html.

41 For a detailed discussion of studies used for health 

impact functions and unit values, see EPA (2011, 

December). Regulatory Impact Analysis for the Final 

Mercury and Air Toxics Standards. EPA-452/R-11-011. 

Research Triangle Park, NC: Office of Air Quality 

Planning and Standards; Health and Environmental Impacts 

Division. Available at 

http://www.epa.gov/ttnecas1/regdata/RIAs/matsriafinal.pdf
60 

COBRA version 2.442, released in 2012, 

estimates the health impact of air quality 

changes for 2017 based on projected 

changes in air quality, national income, and 

population growth. At the core of the 

COBRA model is a county-level sourcereceptor (S-R) matrix that translates changes 

in emissions to changes in PM2.5

concentrations. The changes in ambient 

PM2.5 concentrations are then linked to 

changes in mortality risk and changes in 

health incidents that lead to healthcare costs 

and/or lost workdays. Figure II.4-1 provides 

an overview of the modeling steps. With 

some minor adjustments, COBRA can 

provide retrospective estimates, although 

those estimates will be conservative because 

the COBRA will assume that regulations in 

place as of 2017 will also be in place in 

earlier years.43 As EPA makes 

improvements to their modeling of air 

quality health impacts in BenMAP, these 

changes are expected to be reflected in 

future versions of COBRA. 

COBRA supports selection of air pollutants 

by type. The annual changes in emissions by 

type are entered either as a percentage 

reduction (or increase) or as number of tons 

reduction (or increase). If the technology 

being evaluated is concentrated in several 

states, specific states can be selected within 

COBRA. If the technology’s application is 

 

42 The COBRA Model is available on request from EPA, in 

the form of downloadable software and a user manual 

(EPA User’s manual for the Co-Benefits Risk Assessment 

(COBRA) screening model. Version: 2.4 October 2012).

43 COBRA is expected to provide a conservative estimate 

of the health benefits of reducing emissions for the 

following reasons: COBRA does not include the effects of 

many pollutants that may negatively affect health, and 

COBRA does not fully capture the economic value of 

health events of those pollutants that are included in the 

model. For instance, estimation of hospital admissions in 

dollars is based on cost of illness (COI) units that include 

the hospital costs and lost wages of the individual but do 

not capture the social (personal) value of pain and 

suffering.

more widespread, the option of using 

“nationwide” can be selected. The model 

displays results in terms of change in the 

number of annual cases of respiratory 

deaths, illnesses by type, and associated 

costs by region for the year 2017. 

In addition to entering changes in emissions, 

the user identifies the sector in which the 

emissions are reduced. For example, the 

sector is light-duty gas vehicles in the case 

of the evaluation study of NiMH and Li-ion 

Energy Storage by Link et al. (2013). The 

specified sector drives the underlying spatial 

distribution of emissions and the 

characteristics of the affected human 

population. The model then calculates the 

incidence of human health effects using a 

range of built-in health impact functions and 

estimated baseline incidence rates for each 

health endpoint. Table II.4-2 shows the 

different health endpoints that are included 

in the COBRA model. 

Two of the health endpoints are valued 

based on present value calculations that 

require the specification of a discount rate. 

Incidents of premature mortality are 

assumed to occur over 20 years, with 30% 

of incidents of premature mortality assumed 

to occur in the analysis year, 50% spread 

evenly between years 2 and 5, and the 

remaining 20% between years 6 and 20. 

Additionally, nonfatal heart attacks are 

assumed to incur costs for five years 

following the event. For this reason, the 

value estimates for these health endpoints 

vary based on the discount rate used. The 

evaluator should include results with both a 

7% and 3% discount rate consistent with the 

other cost and benefit estimates. COBRA 

must be run for each discount rate for each 

year. 

COBRA translates the health effects into 

changes in monetary impacts using per-unit 
61 

monetary values in 2010 dollars (2010$) 

described in the COBRA user’s manual. 

(evaluators need to convert from the 2010$ 

values to the appropriate dollar year using 

the BEA GDP deflator, consistent with the 

rest of the analysis.) 

Because COBRA is calibrated to a 2017 

baseline, two adjustments must be made to 

COBRA results after the simulation runs—

population adjustment and income 

adjustment. 

First, the ratio of retrospective population 

estimates to COBRA’s 2017 projected 

population should be used to scale health 

outcomes. This scaling will adjust for 

overall magnitude changes; however, it will 

not adjust for how demographic shifts 

influenced the estimated health impacts. For 

nationwide analyses, evaluators should use 

intercensal estimates of the resident U.S. 

population up to 2010. After 2010, 

evaluators should use estimates of the 

resident U.S. population as of July 1st for 

each year until intercensal estimates are 

available following the 2020 census. 

Source: U.S. EPA. 2012c. User’s manual for the Co-Benefits Risk Assessment (COBRA) screening model. Version: 2.4.

October 2012. 

Figure II.4-1 COBRA Model Overview 

Where evaluations consider remaining 

effective useful life (EUL) impacts44 of 

retrospectively adopted technology in future 

years, populations should be projected 

forward based on the percentage change in 

U.S. population projections in the most 

recent Annual Energy Outlook (AEO). 

Using the percentage change from the AEO 

is necessary to ensure that all calculations 

and adjustments within COBRA remain 

internally consistent and without unintended 

discontinuities. 

 

44 EUL is treated in Step 8. 

The 2017 population estimate for the 

continental U.S. in BenMAP is 315,761,800. 

To compare to the U.S. Census population 

estimates, this value first needs to be 

adjusted to reflect a projected population 

estimate including Hawaii and Alaska. For 

example, using the fraction of the U.S. 

population in Hawaii and Alaska in the 2010 

Census, the corresponding total U.S. 

resident population based on the 2017 

BenMAP value would be 317,893,682. 
62 

Table II.4-2 Health Endpoints Included in COBRA 

Health Effect Description/Units 

Mortality Number of adult deaths 

Infant Mortality Number of infant deaths 

Nonfatal heart attacks Number of nonfatal heart attacks 

Respiratory hospital admissions Number of cardiopulmonary-, asthma-, or pneumonia-related 

hospitalizations 

Cardiovascular related hospital 

admissions 

Number of cardiovascular-related hospitalizations 

Acute bronchitis Cases of acute bronchitis 

Upper respiratory symptoms Episodes of upper respiratory symptoms (runny or stuffy nose; wet cough; 

and burning, aching, or red eyes) 

Lower respiratory symptoms Episodes of lower respiratory symptoms: cough, chest pain, phlegm, or 

wheeze 

Asthma emergency room visits Number of asthma-related emergency room visits 

Minor restricted activity days Number of minor restricted activity days (days on which activity is reduced 

but not severely restricted; missing work or being confined to bed is too 

severe to be MRAD). 

Work days lost Number of work days lost due to illness 

Asthma exacerbations Number of episodes with cough, shortness of breath, wheeze, and upper 

respiratory symptoms in asthmatic children

Source: U.S. EPA. 2012c. User’s manual for the Co-Benefits Risk Assessment (COBRA) screening model. Version: 2.4. October 

2012. 

Second, several valuation endpoints in 

COBRA, including the value placed on 

premature mortality, are adjusted over time 

based on projected changes in income and 

elasticity estimates for how willingness-topay for health outcomes change with 

income. For the retrospective part of the 

analyses, evaluators should estimate the 

adjustment factor based on observed per 

capita real GDP changes over time. The 

ratio of the retrospective adjustment factors 

to COBRA’s 2017 adjustment factor can be 

used to scale the valuation of health 

outcomes. Where evaluations consider 

remaining EUL impacts of retrospectively 

adopted technology in future years, per 

capita real GDP should be projected forward 

for the future years based on the percentage 

change in per capita real GDP projections in 

the most recent AEO. 

Table II.4.3 shows the formulas used to 

adjust 2017 COBRA results to the 

appropriate year. 

The changes in adverse health events and 

the equivalent monetary value are derived 

using the formulations in Table II.4-3. 

COBRA produces both a low and a high 

result to represent a range of estimates for 

premature mortality and nonfatal heart 

attacks based on mortality literature. Carry 

forward the mean of the low and high 

estimates to Step 7 to calculate economic 

performance measures. 

.
63 

Table II.4-3 Calculating Appropriate Year Values from COBRA 2017 Results 

Measure Formulation 

Health Incidents An adjustment to the avoided health incidents estimated in 2017 to account for changing 

population.

ܪ௜ǡ௬ ൌܪ௜ǡଶ଴ଵ଻ ൈ ܲܣ݌݋݆݀௬

where ܪ௜ǡ௬= number of avoided health incidents of type i in year y. 

ܪ௜ǡଶ଴ଵ଻ = number of avoided health incidents of type i reported by COBRA for the year 2017. 

ܲܣ݌݋݆݀௬= population adjustment factor for year y. 

Valued Health 

Incidents

An adjustment to the values of avoided health incidents estimated in 2017 to account for price 

changes and changing income.

ܶݐ݋݈ܸ݈ܽܽ௬ ൌ  ሺܦ݂݈݁ܽܣݎ݋ݐ݆݀ ൈ ܲܣ݌݋݆݀௬ሻሺݐݎ݋ܯܸ݈ܽଶ଴ଵ଻ ൈ ܣݐݎ݋ܯ݆݀௬ ൅ ܯ݅݊ݎ݋ܸ݈ܽଶ଴ଵ଻

ൈ ܯ݅݊ܣݎ݋݆݀௬ ൅ ܱݐ݄݁ݎܸ݈ܽଶ଴ଵ଻ሻ

where ܶݐ݋݈ܸ݈ܽܽ௬= value of avoided health impacts in year y. 

ܦ݂݈݁ܽܣݎ݋ݐ݆݀ = price deflator to adjust values from $2010. 

ܲܣ݌݋݆݀௬= population adjustment factor for year y. 

ݐݎ݋ܯܸ݈ܽଶ଴ଵ଻= value of avoided mortality reported by COBRA for the year 2017. 

ܣݐݎ݋ܯ݆݀௬= income adjustment factor for mortality impacts in year y. 

ܯ݅݊ݎ݋ܸ݈ܽଶ଴ଵ଻= value of avoided minor health impacts reported by COBRA for the year 

2017.

ܯ݅݊ܣݎ݋݆݀௬= income adjustment factor for minor health impacts in year y. 

ܱݐ݄݁ݎܸ݈ܽଶ଴ଵ଻= value of other health impacts reported by COBRA for the year 2017. 

Evaluators should show in tables, first, the 

detailed year-by-year mortality and 

morbidity data and associated monetary 

values, and the totals over the analysis 

period, using the 7% discount rate; and, 

second, the year-by-year environmental 

health benefits based on the 7% discount 

rate, footnoted to indicate that each year’s 

value has discounting internal to the 

COBRA model. The first table is illustrated 

by the example in Table II.4-4 produced by 

application of the COBRA model in the 

NiMH and Li-ion Energy Storage 

Evaluation by Link et al. (2013). A separate 

set of tables could be provided for the 3% 

discount rate, or additional columns could 

be added to the tables to provide 7% and 3% 

results side by side (Table II.4-5) 

The year-by-year environmental health 

benefits at both the 3% and 7% discount rate 

are carried forward to Step 7, and used in the 

bottom-line economic performance 

measures. Using COBRA to estimate health 

benefits in dollar terms provides sufficiently 

credible monetary estimates to warrant the 

approach of carrying the estimates forward 

and combining them with energy and other 

resource benefits in a Level 2 analysis, as 

well as with social cost of carbon estimates 

in a Level 3 analysis of overall measures of 

economic performance (e.g., net present 

value benefits, benefit-to-cost ratios, and 

internal rate of return) in Step 7. 

When calculating performance measures in 

Step 7, the corresponding COBRA results 

should be used when estimating net present 

values and benefit-to-cost ratios. Use the 7% 

COBRA results when calculating 

.
64 

◙ Table II.4-4 Illustration of Health Benefit Calculations Results from the COBRA Model, 

Year 2012, Using a 7% Discount Rate 

Health Effect Avoided Incidence Benefit (thousands of 2012$) 

Adult mortality 2.3 – 6.0 $17,348 - $44,589 

Infant mortality 0.0a $43 

Nonfatal heart attacks 0.3 – 2.6 $34 - $316 

Resp. hospital admissions 0.9 $26 

Cardio-vascular (CDV) hospital admissions 0.8 $34 

Acute bronchitis 3.8 $2 

Upper respiratory symptoms 68.4 $5 

Lower respiratory symptoms 47.9 $2 

Asthma ER visits 1.6 $1 

Minor restricted activity days (MRAD) 2,114.6 $147 

Work loss days 357.3 $56 

Asthma exacerbations 71.9 $8 

Total health costs (2012$) $17,706 to $45,229 

Source: COBRA model results from Link et al. (2013). 

performance measures with the 7% discount 

rate, and use the 3% COBRA results when 

calculating performance measures with the 

3% discount rate. This is necessary because 

COBRA generates values using internal 

user-specified discount rates. 

Evaluators should be aware of how COBRA 

may affect the calculation of the IRR 

performance measure for Level 2 and 

higher. When estimating the IRR, evaluators 

should use the time series of values 

discounted at 7% with the understanding 

that there may be some slight bias to the 

overall internal rate of return. COBRA 

results are exogenous. Therefore, when 

calculating the IRR results are not being 

iterated as the solution rate is sought. Since 

the values remain generated at a 7% 

discount rate, the internal rate of return on 

the stream of total net benefits will be 

slightly biased. 

4.C Estimate the dollar value of 

avoided GHG emissions using 

the social cost of carbon 

To estimate the dollar value associated with 

avoiding a metric ton of GHG emissions, an 

intergovernmental working group has 

developed estimates of the social cost of 

carbon (SCC) (Interagency Working Group 

[IWG] on Social Cost of Carbon, 2013). The 

values are based on results from three 

Integrated Assessment Models (IAMs) of 

how GHGs released in 2010, 2020, 2030, 

2040 and 2050 will influence global 

economic output over a multi-century 

timescale. See Figure II.4-2. 
65 

◙ Table II.4-5 Pump-to-Wheel Time Series of Environmental Health Benefits, 1999–2022 

(Thousands of 2012$ Using 7% Discount Rate) 

Year 

(1) 

Miles Driven (in 

thousands) 

(2) 

U.S. Resident Populationa

(in thousands) 

(3) 

Environmental Health 

Benefits (low)b 

(4) 

Environmental Health 

Benefits (high)b 

1999 97 279,040 $0 $0 

2000 58,895 282,162 $68 $174 

2001 195,901 284,969 $229 $584 

2002 449,485 287,625 $531 $1,357 

2003 788,913 290,108 $946 $2,416 

2004 1,393,754 292,805 $1,703 $4,351 

2005 2,877,430 295,517 $3,578 $9,140 

2006 4,632,866 298,380 $5,548 $14,172 

2007 6,918,298 301,231 $8,133 $20,776 

2008 8,736,302 304,094 $10,169 $25,975 

2009 10,178,950 306,772 $11,684 $29,844 

2010 11,337,092 309,326 $13,140 $33,563 

2011 12,395,177 311,588 $14,757 $37,695 

2012 14,273,723 313,914 $17,706 $45,229 

2013 13,689,024 316,969 $17,183 $43,894 

2014 12,945,702 320,051 $16,492 $42,129 

2015 11,883,232 323,156 $15,349 $39,209 

2016 10,418,077 326,283 $13,634 $34,830 

2017 8,742,915 329,427 $11,766 $30,057 

2018 6,806,933 332,586 $9,475 $24,206 

2019 5,213,999 335,758 $7,552 $19,293 

2020 3,833,351 338,942 $5,831 $14,896 

2021 2,662,274 342,133 $4,289 $10,958 

2022 1,368,481 345,328 $2,271 $5,802 

1999–2012 $88,192 $225,277 

1999–2022 $192,034 $490,552 

a

 Source: 1999 through 2012 estimates from U.S. Census Bureau (2004, 2011, 2013). 2013 through 2022 based on percentage 

changes in population projected in the 2013 Annual Energy Outlook (U.S. Energy Information Administration, 2012). b

 Values of future health impact have been discounted at 7% in each year. 
66 

Multiplying avoided GHG emissions (in 

CO2 equivalents) by the SCC provides an 

estimate of the economic value of the 

avoided GHG emissions. 

SCC estimates were developed specifically 

to support regulatory impact analysis. EERE 

evaluations are not regulatory impact 

analyses, however using SCC values permits 

inclusion of technologies’ GHG avoidance 

in economic performance measures. 

Working from information and data 

provided by the IWG, the co-authors of this 

Guide estimated the SCC for each year 

between 1976 and 2010, solely to support 

EERE benefit-cost evaluations. 

There are four SCC values, but for EERE 

purposes, the 3% average value is the central 

value that should be reported in the 

evaluation report summary tables. The 5% 

average, the 2.5% average, and the 3% 95th 

percentile case should be used to create a 

range of estimated values. 

Figure II.4-2 Social Cost of Carbon by Scenario and Year (2007$) 

Note: 1976 to 2010 values were calculated by RTI International’s Ross Loomis, in consultation with the National Center for 

Environmental Economics at the Environmental Protection Agency. Historical SCC values were generated by RTI specifically for use in 

EERE retrospective economic impact analyses of EERE-supported energy technology R&D, based on the SCC estimates released in 

2013. Methods and results were reviewed by economists with the NCEE to ensure consistency with the IWG results. 2010 through 2050 

are from the Interagency Working Group on Social Cost of Carbon (2013). 

5.0%

Average 

3.0%

Average 

2.5%

Average 

3.0% 

95th Percentile 

$0

$50

$100

$150

$200

$250

5.0%

Average

3.0%

Average

2.5%

Average

3.0%

95th Percentile
67 

Due to uncertainty surrounding the 

appropriate intergenerational discount rate 

for impacts of GHGs, the first three SCC 

values—at 5%, 3%, and 2.5%—reflect 

different intergenerational discount rates for 

future cost and benefits.45 A large 

uncertainty beyond the choice of discount 

rate is the actual damage that may occur in 

any given year due to higher GHG 

emissions. The fourth value is intended to 

represent some of the uncertainty associated 

with both temperature change and its impact 

on the environment on a multi-century scale. 

This 95th percentile of damages discounted 

at 3% is intended to reflect the potential for 

less likely but higher damages associated 

with GHGs. It may still be conservative 

given the limited ability of models to 

estimate damages or properly account for 

the potential consequences of feedback loop. 

Table II.4-6 presents the SCCs by year to be 

used in retrospective analyses. IWG did not 

estimate SCCs prior to 2010. EPA’s 

National Center for Environmental 

Economics recommended estimating 

previous years using a backcasting approach 

by applying the growth rate between 2010 

and 2020 to prior years.46 This methodology 

was recommended because the SCC is not 

expected to have increased linearly prior to 

2010 and the resulting SCC estimates are 

consistent with alternative nonlinear 

functional forms that fit the 2010 to 2050 

estimated SCCs, such as quadratic or loglinear relationships. 

 

45 The central intergenerational discount rate used to 

estimate the SCC, 3%, was chosen by the IWG to reflect 

the impacts of GHGs consistent with losses of future 

consumption. The high intergenerational discount rate, 5%, 

was chosen to reflect a post-tax 7% discount rate. The low 

value of 2.5% was chosen as a certainty-equivalent rate to 

reflect that interest rates are uncertain over time 

(Interagency Working Group on Social Cost of Carbon, 

2010). 

46 U.S. Environmental Protection Agency, personal 

communication, August 13, 2013.

Given the uncertainty associated with SCC 

values, the dollar valuation of GHGs is 

included only in the Level 3 analysis of Step 

7. 

For the 7% benefit cost analysis results, use 

the 3% average SCC value and discount 

these values at 3%, and include a footnote 

that the use of 3% instead of 7% is to 

account for intergenerational effects, per the 

IWG and to be consistent with DOE 

conservation standards. This footnote must 

accompany the BCR and the NPV. For the 

3% results, use the 3% average SCC value. 

Evaluators need to convert from the 2007$ 

SCC values to the appropriate dollar year 

using the BEA GDP deflator, consistent 

with the rest of the analysis. 

4.D Describe other notable 

environmental effects 

If there are notable environmental effects 

other than emissions—e.g., changes in water 

consumption, water discharges, land 

resource use, and solid waste generation—

provide a qualitative treatment of those 

effects. If quantitative data are available, 

provide them along with a description and 

explanation. 

The EPA provides information by 

generating source (e.g., natural gas, coal, oil, 

municipal solid waste, biomass, land-fill 

gas, nuclear energy, hydropower, wind, 

geothermal, and solar) on the differential 

effects the generating sources have on water, 

water discharges, land resource use, and 

solid waste generation.47

 

47 As a starting point, one can find such information on-line 

at www.epa.gov/cleanenergy/energy-andyou/affect/index.html, under the heading "How does 

electricity affect the environment?" 
68 

Measure Formulation 

Value of Avoided 

Greenhouse Gas Emissions Multiply the social cost of carbon by avoided greenhouse gas 

emissions and sum over all years while discounting to the initial year 

in the analysis.

ܩܪܩܸ݈ܽ௦ ൌ ෍ ܵܥܥ௦ǡ௬ܥܱଶ݁ݍ௬Ȁሺͳ ൅ ݀௦ሻ௬ ே

௬ୀଵ

where ܩܪܩܸ݈ܽ௦= value of avoided greenhouse gas emissions for 

social cost of carbon scenario s. 

ܵܥܥ௦ǡ௬ = social cost of carbon in year y and scenario s. 

ܥܱଶ݁ݍ௬= avoided greenhouse gas emissions in year y reported in 

CO2eq. 

݀௦=discount rate used in scenarios. 

Table II.4-6 Social Cost of Carbon by Scenario and Year, 

per Metric Ton of CO2 equivalent (2007$) 

Year 5.0% Average 3.0% Average 2.5% Average 3.0% 95th Percentile 

1976 7 13 25 27 

1977 8 13 26 28 

1978 8 13 26 29 

1979 8 14 27 30 

1980 8 14 28 31 

1981 8 15 28 32 

1982 8 15 29 33 

1983 8 15 29 34 

1984 8 16 30 36 

1985 8 16 31 37 

1986 8 17 31 38 

1987 8 17 32 40 

1988 8 18 33 41 

1989 9 18 33 43 

1990 9 19 34 44 

1991 9 19 35 46 

1992 9 20 36 47 

1993 9 20 36 49 

1994 9 21 37 51 

1995 9 21 38 53 

1996 9 22 39 55 

1997 9 23 40 57 

(Continued) 
69 

1998 10 23 40 59 

1999 10 24 41 61 

2000 10 25 42 63 

2001 10 25 43 65 

2002 10 26 44 68 

2003 10 27 45 70 

2004 10 28 46 73 

2005 10 28 47 75 

2006 10 29 48 78 

2007 11 30 49 81 

2008 11 31 50 84 

2009 11 32 51 87 

2010 11 33 52 90 

2011 11 34 54 94 

2012 11 35 55 98 

2013 11 36 56 102 

2014 11 37 57 106 

2015 12 38 58 109 

2016 12 39 60 113 

2017 12 40 61 117 

2018 12 41 62 121 

2019 12 42 63 125 

2020 12 43 65 129 

2021 13 44 66 132 

2022 13 45 67 135 

2023 13 46 68 138 

2024 14 47 69 141 

2025 14 48 70 144 

2026 15 49 71 147 

2027 15 49 72 150 

2028 15 50 73 153 

2029 16 51 74 156 

2030 16 52 76 159 

2031 17 53 77 163 

2032 17 54 78 166 

2033 18 55 79 169 

2034 18 56 80 172 

2035 19 57 81 176 

2036 19 58 82 179 

2037 20 59 84 182 

2038 20 60 85 185 

2039 21 61 86 189 

(Continued) 
70 

Note: 1976 to 2010 values were calculated by RTI International’s Ross Loomis, in consultation with the National Center for 

Environmental Economics at the Environmental Protection Agency. These values may be used for nonregulatory purposes only. 2010 

through 2050 are from the Interagency Working Group on Social Cost of Carbon (2013).

2040 21 62 87 192 

2041 22 63 88 195 

2042 22 64 89 198 

2043 23 65 90 201 

2044 23 65 91 204 

2045 24 66 92 206 

2046 24 67 94 209 

2047 25 68 95 212 

2048 25 69 96 215 

2049 26 70 97 218 

2050 27 71 98 221 
71 

II.5 Estimate Energy Security Impacts 

The energy security impacts take into 

account the following components: 

 Reduction in oil consumed as fuel, and 

the proportion of that consumption that 

would have been imported. 

 Qualitative treatment of altered threats 

to the energy infrastructure. 

Report energy security benefits from 

reductions in oil consumption and imports in 

physical units only; do not estimate the 

monetary value using an energy security 

premium. Do not estimate barrels of oil 

equivalents for reduction in the consumption 

of nonpetroleum forms of fossil fuel. 

5.A Bring forward from Step 3 

the estimated net reduction in 

oil consumption in physical 

units 

To estimate the reduction in oil 

consumption, the evaluator must estimate 

what portion of the net energy savings from 

Step 3 came from oil products or to what 

extent generation of renewable energy 

displaces generation from oil products. 

Given that Steps 3 and 4 analyzed the 

energy and environmental impacts, it is 

likely that avoided oil consumption effects 

have already been calculated. If so, bring the 

data forward to use in Step 5 calculations. If 

these effects were not calculated in Steps 3 

or 4, do so now. 

If the net reductions in oil consumption are 

associated with avoided conventional 

electricity generation, the evaluator must use 

EIA data available from the Annual Energy 

Review (U.S. Energy Information 

Administration, 2012a) to determine the 

fraction of avoided conventional electricity 

that would have been generated from oil 

products. Future years’ data will be 

available from the EIA when those Annual 

Energy Reviews are released. For an 

example of these data current as of 2013, see 

Column 1 of Table II.5-1. 

If changes in electricity use and generation 

developed in Step 3 were generated at the 

state-level, the analysis should use the state-

 Bring forward from Step 3 the estimated net reduction in oil 

consumption in physical units. 

 Report in physical units the reductions in barrels of oil 

consumed and imported. 

 Describe notable effects of the portfolio technologies on the 

nation’s energy infrastructure. 

5. Estimate 

Energy 

Security 

Impacts
72 

level fractions. Otherwise, the national 

fraction must be used. State-level net 

electricity generation by fuel type and year 

is available from 1990 (U.S. Energy 

Information Administration. 2012b), and 

previous years can be generated using 

detailed data.48

Next, the evaluator must convert from 

MWhs of avoided electricity generation 

from oil products to barrels of oil using a 

conversion factor of 1.84 MWh/barrel (EIA, 

2013b). 

To estimate avoided oil importation, apply 

EIA data on the proportion of internationally 

sourced fuel to the avoided oil consumption 

estimates. The assumption is that each 

marginal barrel of oil consumed is sourced 

proportionate to total consumption. This 

assumption must be emphasized in the 

evaluation report at the point the estimation 

is made. Column 2 in Table II.5-1 illustrates 

the percentage of internationally sourced oil 

from 1976 to 2011. Data for subsequent 

years become available when the EIA 

releases its Annual Energy Reviews. 

5.B Report in physical units the 

reductions in barrels of oil 

consumed and imported 

Report the energy security benefits from 

reductions in oil consumption and imports in 

physical units (gallons and barrels, 

respectively) only; do not estimate the 

monetary value using an energy security 

premium. Do not estimate barrels of oil 

equivalents for reduction in the consumption 

of other forms of fossil fuel.49

 

48 Available online at 

http://www.eia.gov/electricity/data/eia923/. 49 While the Guide does not provide for the monetary 

valuation of energy security effects, some have provided 

partial estimation. Monetized energy security benefits 

result from the concept that oil prices are above what they 

Avoided oil imports are derived using the 

formulation in Table II.5-2. 

5.C Describe notable effects of 

the portfolio technologies on 

the nation's energy 

infrastructure 

In some cases, the technology portfolio 

evaluated may have implications for the 

security of the nation’s energy 

infrastructure. For example, use of a 

distributed renewable energy source may 

reduce the vulnerability of central power 

plants to disruptions; and improved backup 

electric generation and energy storage in 

vulnerable locations can fortify against 

natural, accidental, and deliberately caused 

disruptions. If the portfolio technologies 

have the potential to affect security of the 

nation's energy infrastructure, provide a 

qualitative description of the nature of these 

effects and provide supporting evidence. 

 

would be under competitive market conditions due to 

market power exerted by the Organization of the Petroleum 

Exporting Countries. Prices above their competitive market 

levels due to market power result in losses of potential 

GDP as well as transfers of wealth from oil imports. In 

addition, price shocks from disruptions in oil supply result 

in short term dislocational losses of GDP (Greene and 

Leiby, 2006). Monetized energy security benefits of 

reduced or avoided oil consumption are associated with 

reducing market failures from existing market power, not 

market failures due to externalities (National Research 

Council, 2010). All else held equal, by reducing oil demand 

in the U.S. through technology programs, the global oil 

price is reduced and the U.S. economy is less vulnerable to 

shocks in oil prices. The reduction in imported oil price, 

referred to as the monopsony component, represents a 

transfer of wealth between the U.S. and oil exporting 

countries, not a loss of potential economic output. As such, 

the monopsony component should not be included in the 

analysis.
73 

Table II.5-1 Percentage of U.S. Electricity Production from Petroleum Products and 

Percentage of Net Oil Imports by Year, 1976-2012 

Year Electricity Generated from Petroleum Products 

as a Percentage of Net Electricity Generation(1) 

Net Oil Imports as a Percentage of 

Total Oil Supplied(2) 

1976 15.7% 40.6% 

1977 16.9% 46.5% 

1978 16.5% 42.5% 

1979 13.5% 43.1% 

1980 10.8% 37.3% 

1981 9.0% 33.6% 

1982 6.5% 28.1% 

1983 6.3% 28.3% 

1984 5.0% 30.0% 

1985 4.1% 27.3% 

1986 5.5% 33.4% 

1987 4.6% 35.5% 

1988 5.5% 38.1% 

1989 5.6% 41.6% 

1990 4.1% 42.2% 

1991 3.8% 39.6% 

1992 3.1% 40.7% 

1993 3.5% 44.2% 

1994 3.2% 45.5% 

1995 2.1% 44.5% 

1996 2.3% 46.4% 

1997 2.6% 49.2% 

1998 3.5% 51.6% 

1999 3.2% 50.8% 

2000 2.9% 52.9% 

2001 3.3% 55.5% 

2002 2.4% 53.4% 

2003 3.1% 56.1% 

2004 3.0% 58.4% 

2005 3.0% 60.3% 

2006 1.5% 59.9% 

(Continued) 
74 

2007 1.5% 58.2% 

2008 1.1% 57.0% 

2009 0.9% 51.5% 

2010 0.9% 49.2% 

2011 0.7% 44.9% 

2012 0.6% 39.9% 

Sources: U.S. Energy Information Administration. 2012a. “Annual Energy Review 2011.” http://www.eia.gov/totalenergy/data/annual/pdf/aer.pdf; 

U.S. Energy Information Administration. 2013a. “August 2013 Monthly Energy Review.” http://www.eia.gov/totalenergy/data/monthly/pdf/mer.pdf 

Table II.5-2 Calculating Avoided Imported Oil 

Avoided Oil 

Imports

Multiply the estimated barrels of avoided petroleum products by the percentage of net 

oil imports of total oil supplied for each year

௬݌݉ܫݐܰ݁ ൈ ௬݀݋ݎܲݐܲ݁ ൌ ௬ݐݎ݋݌݉ܫܱ݈݅

where ܱ݈݅ܫ݉ݐݎ݋݌௬= estimated avoided imported barrels of oil in year y. 

ܲ݁ݐܲ݋ݎ݀௬ = estimated avoided barrels of petroleum products in year y. 

ܰ݁ܫݐ݉݌௬= net oil imports as a percentage of total oil supplied in year y. 

 
75 

II.6 Estimate Knowledge Benefits 

Step 6 guides the estimation of knowledge 

benefits resulting from EERE R&D 

expenditures in support of a given portfolio. 

This step proceeds through five sub-steps 

(6.A to 6.E), with emphasis on knowledge 

creation and dissemination signaled by 

outputs of patents and publications—

particularly patents.50

6.A Construct database of 

portfolio patents and 

publications attributed to EERE, 

plus other databases needed for 

comparisons51

To perform patent and publication analyses, 

evaluators will need databases of patents and 

publications that derive from EERE's 

 

50 Patents have a central role in the innovation system, are 

generally considered closer to application than 

publications, and, as noted by Jaffe and Trajtenberg (2005), 

have been used extensively in the study of technological 

change. Tracing patent citations of scientific papers is a 

method of finding early influences of laboratory research 

on innovation.

51 An overview of patent analysis is provided by Ruegg and 

Thomas (2013), and of bibliometrics by Hicks and Melkers 

(2013).

portfolio investment. The evaluator should 

determine if EERE program directors have 

provided for a careful compilation of these 

databases, thus relieving the evaluator of the 

task. Previously completed evaluations have 

required an additional effort on the part of 

the evaluator to construct the required 

databases of patents and publications. 

Data sources include EERE Annual Progress 

Reports of R&D programs, which generally 

list patents and publications associated with 

its sponsored research. Data sources also 

include EERE program and DOE laboratory 

databases. Patent and publication records are 

centrally maintained by the DOE Office of 

Scientific and Technical Information 

(OSTI). Beyond DOE/EERE databases, 

patent records are maintained by the U.S. 

Patent and Trademark Office (PTO), the 

European Patent Office (EPO), the World 

Intellectual Property Organization (WIPO), 

and other country patent offices. 

Organizations whose R&D has received 

public funding, such as funds from EERE, 

are obligated to acknowledge the public 

interest when filing for patents. The 

information enables identification from 

 Construct databases of portfolio patents and publications 

attributed to EERE, plus other databases needed for 

comparisons. 

 Conduct patent citation analyses. 

 Identify notable patents attributed to or linked to EERE. 

 Apply additional techniques to assess knowledge benefits as 

budget permits.

6. Estimate 

Knowledge 

Benefits
76 

these larger databases of those patents that 

were assisted by DOE. The Web of Science 

contains extensive bibliometric records.52

That said, difficulties may arise in obtaining 

reliable databases for evaluation of EERE 

portfolios. In the case of patents, 

complexities that may be encountered 

include the following: 

1) EERE funds relevant R&D across many 

organizations, and most patents resulting 

from EERE funding are not assigned to 

EERE. 

2) Some assignees omit acknowledging the 

public interest. Even if the public 

interest is noted, a given patent may not 

be identifiable as resulting from the 

EERE portfolio of interest. 

3) EERE may have funded part, but not all 

of the R&D underlying the patent, 

clouding the need for acknowledging 

public interest.

4) There may be a lag between database 

entry of patents or publications and the 

evaluation study, such that some may 

not yet be entered into a given database, 

including the OSTI database.

5) The portfolio may deal with a subset of a 

research area that causes it to be difficult 

to identify the subset of relevant patents 

from the titles that should be included in 

the evaluation data set. 

For all of these reasons, past evaluation 

studies have found it necessary to use a 

variety of approaches to construct the 

relevant EERE database needed for the 

evaluation. 

Once the set of relevant patents have been 

identified, it is necessary to avoid double 

 

52 For EERE portfolios evaluated to date, a large share of 

publications have comprised technical reports and 

conference papers and presentations, and a smaller share, 

journal articles. The composition of publications affects the 

most suitable search engine to use.

counting that can arise for two reasons: (1) 

organizations often file for protection of 

their inventions across multiple patent 

systems, such that the same patent exists 

with different designations; and (2) patent 

filers may apply for a series of patents all 

based on the same underlying invention. To 

avoid double counting, evaluators will need 

to construct "patent families" which group 

patents around their original patent. Thus, 

there are fewer patent families typically than 

there are patents. 

It is helpful to show the trend in output of 

relevant EERE-attributed patent families and 

publications, and, if feasible, to compare 

these with overall outputs. For example, 

Figure II.6-1 (parts a & b) shows the output 

of patent families attributed to EERE and 

overall. The illustration is from a recent 

evaluation of EERE's portfolio of energy 

storage technologies (including li-ion, 

NiMH, and ultracapacitors) for hybrid and 

electric cars. 

In addition to compiling a database 

containing patents and publications 

attributed to the EERE portfolio, additional 

databases will be needed. To perform 

comparisons and to trace from commercial 

innovations back to earlier EERE R&D 

investments, evaluators will need to identify 

relevant organizations that are leading 

innovators/commercializers in the 

technology area of focus and construct a 

database of their patent portfolios in the area 

of interest. To provide country comparisons, 

the evaluator will need to construct relevant 

patent databases by country of first issue. To 

identify highly significant patents, 

evaluators will need to use citation indices 

to adjust for technology area and year of 

issue. Examples and discussion of how such 

additional databases are constructed are 

found in Ruegg and Thomas, 2013. In 

addition, previously conducted analyses of 

 
77 

◙ Figure II.6-1 Counts of Energy Storage Patent Families by Priority Year 

 

 

0

500

1000

1500

2000

2500

1978-82 1983-87 1988-92 1993-97 1998-2002 2003-2007 2008-2012

Number of Patent Families 

Priority Year 

0

10

20

30

40

1988-92 1993-97 1998-2002 2003-2007 2008-2012

Number of Patent Families 

Priority Year 

Part b: Overall Patent Families 

Note: "priority year" refers to the date of filing of the original patent around which a family of related patents is based. Energy 

storage patents include li-ion, NiMH, and ultracapacitors. 

Source: Table by Ruegg and Thomas, appearing in Link et al. (2014). 

Part a: Patent Families Attributed to EERE's Vehicle Technologies Office (VTO)
78 

knowledge benefits in support of past 

evaluation studies show how various 

techniques were carried out, such as 

specifying filters for identifying patent 

sets.53 

6.B Conduct patent citation 

analyses 

Patent citation analysis can be used to 

compare knowledge influences of EERE 

with those of other organizations. Backward 

patent citation analysis in particular is useful 

for assessing how the patents of the leading 

company innovators in the targeted industry 

link back to the EERE-attributed set. 

 

53 See Ruegg and Thomas, 2011.

Backward tracing begins with a downstream 

technology/product/company and traces 

upstream to find earlier sources of influence. 

This approach is illustrated by Figure II.6-2, 

which is taken from a solar photovoltaic 

knowledge benefits study by Ruegg and 

Thomas (2011), performed in support of the 

solar photovoltaic benefit-cost study by 

O'Connor et al. (2010). 

To determine linkages of the program 

portfolio not only to the targeted industry 

but beyond to other industries, forward 

citation analysis is used. This approach 

starts with EERE portfolio-attributed patents 

and identifies all subsequent patents that link 

back to these earlier EERE portfolioattributed patents.

0% 20% 40% 60% 80% 100%

Schott

Solar World

First Solar

Evergreen Solar

BP Solar

ECD

SunPower

Global Solar

Percentage of Patent Families 

Source: Ruegg and Thomas, Solar Photovoltaic (2011) 

◙ Figure II.6-2 Percentage of Solar Energy Patent Families of Top U.S. Solar 

Photovoltaic Producers Linked to Earlier EERE-Attributed Solar Photovoltaic Patents 
79 

An illustration of forward tracing at the 

organizational-level is provided by Figure 

II.6-3, which is also taken from the solar 

photovoltaic knowledge benefits study by 

Ruegg and Thomas (2011). It shows the 

influence of EERE's investment in solar 

photovoltaic research to extend beyond the 

U.S. leading solar photovoltaic producers to 

leading companies in the semiconductor 

industry. 

Patent-to-publication citation analysis can be 

used to identify when a subsequent 

technological invention has drawn directly 

on a scientific paper. Thus an extended 

feature of the patent citation analysis is to 

assess portfolio papers and publications that 

are cited by patents as prior art. An example 

is provided by Table II.6-1, drawn from a

knowledge benefits study by Ruegg and 

Thomas (2011) for use in the related benefitcost evaluation of EERE's Geothermal 

Technologies Office by Gallaher et al. 

(2010). The table lists papers and 

publications on thermal cement technology 

cited by thermal cement patents. 

0 100 200 300 400 500 600 700

Panasonic

NEC

Sanyo

Samsung

Motorola

Texas Instruments

Hitachi

Advanced Micro Devices

Toshiba

General Electric

Sharp

IBM

Applied Materials

Canon

Semiconductor Energy Lab

Micron Technology

Number of Linked Families 

Source: Ruegg and Thomas, Solar photovoltaic (2011).

◙ Figure II.6-3 Organizations from All Industry Areas with the Largest Number of Patent 

Families Linked to Earlier EERE-Attributed Solar Photovoltaic Patents 
80 

◙ Table II.6-1 EERE Geothermal Portfolio Paper/Publications Linked to the largest 

Number of Patent Families through Two Generations of Citations 

# Linked 

Patents EERE Portfolio Papers/Publications 

203 "Interfaces and Mechanical Behaviors of Fiber-Reinforced Calcium Phosphate Cement 

Compositions," by T. Sugama et al., prepared for the Geothermal Division U.S. Department of 

Energy; Department of Applied Science (June 1992)

197 "Microsphere-Filled Lightweight Calcium Phosphate Cements," by Sugama et al., U.S. Department 

of Energy, Washington, D.C. under contract No. DE-AC02-76CH00016 (December 1992) 

197 "Hot Alkali Carbonation of Sodium Metaphosphate Fly Ash/Calcium Aluminate Blend 

Hydrothermal Cements," by T. Sugama, Cement and Concrete Research Journal, vol. 26, No. 11, 

pp. 1661-1672 (1996) 

192 "Calcium Phosphate Cements Prepared by Acid-Base Reaction," by Sugama et al., Journal of the 

American Ceramic Society, vol. 75, No. 8, p. 2076-2087 (August 1992) 

185 "Carbonation of Hydrothermally Treated Phosphate-Bonded Calcium Aluminate Cements," by T. 

Sugama et al., U.S. Department of Energy, Washington, D.C. under contract No. DE-AC02-

76CH00016 (Undated) 

108 "Use of Single-Cutter Data in the Analysis of PDC Bit Designs: Part 1―Development of a PDC 

Cutting Force Model," by Glowka, D.A., JPT, pp. 797-799, 844-849 (August 1989) 

105 "Use of Single-Cutter Data in the Analysis of PDC Bit Designs: Part II―Development and Use of 

PDCWEAR Computer Code," by Glowka, D.A., JPT, pp. 850-859 (August 1989) 

101 "Acoustical Properties of Drill Strings," by Drumheller, D., The Journal of the Acoustical Society of 

America, No. 3, New York, pp. 1048-1064 (March 1989) 

56 "The Propagation of Sound Waves in Drill Strings," by Drumheller et al., The Journal of the 

Acoustical Society of America, No. 4, pp. 2116-2125 (April 1995) 

37 "Acoustical Properties of Drill Strings," by Drumheller D, Sandia National Laboratories, SAND88 

0502 (August 1988)

32 Sourcebook on the Production of Electricity from Geothermal Energy, Kestin, J., editor, 

Publication No. DOE/RA/4051, Chap. 4, p. 536 (1980)

Source: Ruegg and Thomas, Geothermal (2011). 
81 

6.C Identify notable knowledge 

outputs and innovations 

attributed to or linked to the 

EERE portfolio 

A concept useful in tracing knowledge flows 

is that highly cited patents (i.e., patents cited 

by many later patents) tend to contain 

technological information of particular 

importance. A patent that forms the basis for 

many new innovations is generally cited 

frequently by later patents.54 

Examples of notable (i.e., highly cited) solar 

photovoltaic patents of leading companies 

that are linked back to EERE-attributed solar 

photovoltaic patents are given in Table II.6-

2. The table is drawn from the Ruegg and 

Thomas knowledge benefit study performed 

in support of the 2010 solar photovoltaic 

benefit-cost study. The Citation Index 

adjusts for the type of technology and for the 

age of the patent, such that, for example, the 

Index value of 4.52 in the table's first row 

means that the patent (#4419533) has been 

cited approximately 4.5 times more often 

than would be expected of a patent of its 

age, within its technology area. 

6.D Apply additional techniques 

to assess knowledge benefits 

as budget permits 

Other techniques may be used to extend the 

assessment of knowledge benefits from 

patent and publication analyses. Among 

these techniques are licensing analysis, 

publication co-authoring analysis, and 

network analysis. The potential use of 

additional methods of assessing knowledge 

 

54 This does not mean that every highly cited patent is 

important, or every infrequently cited patent is 

unimportant, but, research studies have shown a correlation 

between the rate of citations of a patent and its 

technological importance. See Bretzman and Mogee 

(2002). 

benefits should be discussed with EERE 

staff while formulating the evaluation plan. 

Interviews with EERE program staff may 

indicate an important role of intellectual 

property licensing as a knowledge transfer 

mechanism. In this case an analysis of 

licensing is needed. The analysis of EERE's 

investment in energy storage, for example, 

showed notable development of NiMH 

battery technology by ECD Ovonic, 

supported by EERE funding. ECD Ovonic 

subsequently licensed its NiMH patents to 

many prominent worldwide companies who 

developed NiMH batteries for hybrid and 

electric vehicles. Investigation of patent 

infringement cases may also shed light on 

knowledge creation and dissemination. 

Royalty payments linked to licensing can 

also provide an indication of the use and 

importance of the new knowledge. 

As another example, the formation of 

collaborative R&D networks may be an 

important tacit means of knowledge 

dissemination. The evaluation study may 

find it worthwhile to construct diagrams 

showing collaborative relationships such as 

that shown in Figure II.6-4, taken from a 

study by Ruegg and Thomas (2010) in 

support of an evaluation study by Link 

(2010). It depicts networking arising from 

EERE's investment in the EERE/ACE R&D 

in advanced combustion engines. 

Alternatively, it may be feasible to apply 

social network analysis and develop 

networks and related metrics to assess how 

relationships change after the EERE 

investment is made.
82 

◙ Table II.6-2 Highly Cited Solar Energy Patents of Top U.S. Solar Photovoltaic 

Producers Linked to Earlier EERE-Attributed Solar Photovoltaic Patents 

Patenta Issue 

Date

# Cites 

Received

Citation 

Index

Assignee Title

4419533 1983 47 4.52 ECD Photovoltaic device having incident 

radiation directing means for total 

internal reflection

5164019 1992 51 4.08 SunPower Monolithic series-connected solar cells 

having improved cell isolation and 

method of making same

6534703 2003 12 3.12 SunPower Multi-position photovoltaic assembly

6111189 2000 19 2.93 BP Photovoltaic module framing system 

with integral electrical raceways

6353042 2002 12 2.92 Evergreen 

Solar

UV-light stabilization additive package 

for solar cell module and laminated glass 

applications

6570084 2003 11 2.86 SunPower Pressure equalizing photovoltaic 

assembly and method

4514583 1985 31 2.82 ECD Substrate for photovoltaic devices

4419530 1983 28 2.69 ECD Solar cell and method for producing 

same

5746839 1998 30 2.64 SunPower Lightweight, self-ballasting photovoltaic 

roofing assembly

Source: Ruegg and Thomas, Solar photovoltaic (2011). 

 
83 

Source: Ruegg and Thomas (2010), as used in the related benefit-cost study by Link (2010) 

◙ Figure II.6-4 A Network of Organizations Facilitates Combustion Knowledge Creation 

and Flow 

6.E Consider if and how 

knowledge-benefit findings 

inform the attribution analysis 

of Step 2 

The results of the backward tracing citation 

analysis shows the extent to which patents 

of downstream company innovators and 

commercial product developers in the field 

are linked to patents attributed to the EERE 

portfolio, and, thereby, may support 

attribution. See, for example, Figure II.6-1 

that links patents of top U.S. producers of 

solar photovoltaic modules to earlier EEREattributed patents via citations. Details of 

highly cited EERE-attributed patents (not 

shown here, but provided in the report), and 

highly cited patents of others that connect 

back to EERE-attributed patents, as 

identified in Figure II.6-4, can help to show 

areas of specific EERE knowledge that were 

particularly influential. Similarly, licensing 

and royalty payments can show evidence of 

linkages of EERE's R&D to downstream 

innovators and developers. 

While documentation of such linkages is 

supportive of there being cause and effect 

relationships between the EERE's portfolio 

investment and estimated impacts, it is not 

proof of causation. Also, while suggesting 

areas and strength of influence, citation 

linkages do not provide acceleration 

measures or indicate the share of benefits 
84 

attributable. Additional analysis generally 

will be needed for estimation of attribution. 

6.F Provide summary metrics 

for knowledge benefits 

Prepare a table of summary metrics for 

knowledge benefits, using the format shown 

in Table II.6-3. The illustrative data in the 

table are drawn from the knowledge benefits 

study by Ruegg and Thomas (2010), 

performed in support of the benefit-cost 

study of vehicle combustion engine 

technology by Link (2010). 

The knowledge benefit studies conducted in 

support of the evaluation studies conducted 

to date have all included patent analysis, but, 

depending on the budget provided, have 

variously included other approaches, such as 

publication analysis, licensing assessment, 

historical tracing, including expert 

interviews, and network descriptions. 

Therefore, not all measures are available for 

every study. The knowledge benefits 

assessment in support of the combustion 

benefit-cost study, for example, included 

only patent analysis and a simple network 

description. 

The standard measures in Table II.6-3 

comprise a section of the report's summary 

metrics. The optional measures are included 

in the knowledge benefits section of the 

report but not in the report's summary 

metrics. 

◙ Table II.6-3 Example Knowledge Outputs and Outcomes: Summary Metrics 

Knowledge Outputs (partial, covers 

period mid-1970s-2008) Units Combustion Study Results 

Standard Measures 

DOE-attributed patents Number of patent 

families

109 patent families (i.e., groups of patents based on the same 

invention), containing 127 patents filed with the U.S. patents 

(USPTO), 14 with the European Patent Office (EPO), and 25 patent 

with the World Intellectual Property Organization (WIPO)

(Represents less than 1% of the total number of combustion patent 

families of the 10 leading companies.) 

EERE rank among organizations in terms 

of having its patents in the target industry 

cited by later patents of others in the field

Rank Second ranked

Organization with rank higher than 

EERE 

Organization(s) ranked 

higher

Nissan was first ranked

Spillover areas

Highest Rated IPC 

Grouping after 

targeted area

Detection and analysis of materials

Examples of high-impact EERE-attributed 

patents ― subject areas 

Patent Citation Index 

(CI) Value 

Homogeneous Charge Compression Ignition (HCCI) (CI 8.51); Mixed 

mode fuel injector (CI 6.05); and Ion mobility spectrometer (CI 5.41) 

Note: Completed using results from Ruegg and Thomas (2010) Patent Study for Vehicle Combustion Engine R&D.
85 

II.7 Calculate Measures of Economic 

Performance 

Economic performance measures are 

computed based on monetized benefits and 

costs in six sub-steps. They are calculated 

for three levels of analysis (see highlighted 

textbox.) 

7.A Compute economic 

performance measures for 

level- one analysis 

Economic performance measures are 

calculated by adding benefits by category 

successively in three levels of analysis and 

comparing benefits against portfolio 

investment costs. 

The first level of analysis—Level 1—

follows the traditional practice of comparing 

energy and other resource benefits valued in 

dollars against investment cost. The next 

two levels of analysis introduce 

environmental spillover impacts: 

Level 2 incorporates an environmental 

spillover effect for health benefits from 

reduced air emissions. Level 3 incorporates 

an additional environmental spillover for 

intergenerational benefits from changes in 

GHG emissions. Portfolio investment costs 

remain unchanged for each level. 

Three Levels of Analysis 

 Level 1: computes economic returns by 

comparing energy and other resource benefits 

valued in dollars against portfolio investment 

cost. 

 Level 2: adds to Level 1, a second stream of 

benefits―environmental health benefits 

valued in dollars―and compares combined 

benefits against portfolio investment cost. 

 Level 3: adds to Level 2 benefits, a third 

stream of benefits―intergenerational 

environmental benefits associated with 

changes in GHG emissions valued in dollars 

― and compares combined benefits against 

portfolio investment cost. 

 Compute economic performance measures for level-one analysis. 

 Compute economic performance measures for level-two analysis. 

 Compute economic performance measures for level-three analysis. 

 Prepare a graph showing cumulative benefits by category, versus portfolio 

investment cost. 

 Characterize how worthwhile the EERE portfolio investment has been, taking 

into account each level of impacts considered. 

 Perform a threshold or break-even analysis if negative impacts have been 

revealed by any of the supporting qualitative analyses.

7. Calculate 

Measures of 

Economic 

Performance
86 

As environmental spillovers are introduced 

into the analysis, it is assumed that 

uncertainty increases and the known 

precision decreases, particularly with the 

addition of intergenerational benefits from 

changes in GHG emissions. The three levels 

of analysis helps to reflect the increasing 

uncertainty and decrease in precision as the 

two types of environmental spillovers are 

added in turn. EPA's environmental health 

modeling tools have advanced rapidly and 

are widely used and accepted, and as such 

the environmental spillovers computed using 

COBRA are sufficiently robust to include in 

the Level 2 analysis that adds indirect 

environmental health impacts to direct 

resource impacts. For the calculation of 

intergenerational impacts from changes in 

carbon emissions, the results are considered 

more uncertain, and are thus presented in a 

Level 3 analysis. 

To perform the Level 1 analysis, bring 

forward from Step 1 the total year-by-year 

cash flows for investment costs. Second, 

bring forward from Step 3 the total year-byyear cash flows estimated for energy and 

other resource impacts, ensuring that all 

amounts are expressed in constant dollars as 

of the last year of the study period. 

Use the total summary year-by-year benefit 

and cost cash flows from Steps 1 and 3 to 

calculate each of the five economic 

performance metrics for level 1. Ensure that 

calculations conform to the descriptions in 

Table I-1, and the formulae in Table II.7-1. 

Ensure that the standardized conventions 

specified in Table II.7-2 are followed in 

adjusting current dollars to constant dollars, 

in discounting, and in calculating the 

economic performance metrics.55

 

55 Note that these requirements are intended to promote a 

more standardized approach, and to limit variation among 

studies that result only from following different 

conventions. 

(Spreadsheet and other computer programs 

can be used, provided these requirements are 

met.) 

(1) Sum year-by-year constant dollar 

undiscounted investment costs and yearby-year constant dollar undiscounted 

benefits based on energy and other 

resource impacts. The latter provides a 

measure of undiscounted Gross Benefits 

(GB). 

(2) Calculate the present value (PV) of the 

year-by-year constant dollar investment 

costs (PV investment) for discount rates 

7% and 3%. Calculate the PV of the 

year-by-year constant dollar benefits 

based on energy and other resource 

impacts (PV-benefits) by discounting the 

year-by-year amounts using first a 7% 

rate and then a 3% rate. These PVs will 

be used as the denominator and 

numerators, respectively, of the BCRs 

calculated in item (4) below. 

Table II.7-3, draws from the 2013 EERE 

evaluation report by Link et al. (2013) to 

illustrate year-by-year and total investment 

costs, undiscounted and discounted, for the 

Vehicle Technologies Investment in Energy 

Storage Technologies for Hybrid and 

Electric Cars and Trucks. Table II.7-4 shows 

the year-by-year and total attributable 

energy savings from the same evaluation 

report. 

(3) Subtract year-by-year undiscounted 

investment costs from year-by year 

undiscounted benefits based on energy 

and other resource impacts, and sum the 

results to provide a measure of total 

undiscounted Net Benefits (NB). 

(4) Divide PV-economic benefits by PVinvestment costs, first using the PVs 

based on 7% and then on 3%, to provide 

measures of Benefit-to-Cost Ratios 

(BCRs).

 
87 

(5) Use the year-by-year undiscounted Net 

Benefits in constant dollars (not the 

total) to calculate the Internal Rate of 

Return (IRR).

Table II.7-1 Calculating Economic Performance Measures 

Measure Formulation 

Total Investment 

Cost ( I ) 

A simple summation of year-by-year constant dollar portfolio investment costs over the evaluation 

study period, i.e., 

¦ 

 

N

y

I Iy

1

, where Iy is an undiscounted constant dollar portfolio investment cost in year y and 

N is the number of years in the study period. 

Gross Benefits 

(GB)

A simple summation of total benefits measured in dollars. This computation excludes both the EERE 

investment cost ( I ) and the required rate of return, as indicated by using an implied 0% discount rate; 

hence, it is only a partial measure of economic performance, i.e.,

 GB = , 1

¦ 

N

y

By where By is an undiscounted constant dollar benefit in year y (computed net of 

noninvestment costs in that year), and N is the number of years in the study period. 

PV - investment Present value of investment costs, i.e., 

 PV-investment = ¦



 



1

0

/(1 )

N

y

Iy d y

, where Iy is an undiscounted constant dollar portfolio 

investment cost in year y , N is the number of years in the study period, d is the discount rate, and 

investment costs are assumed to occur at the beginning of each year. 

PV- benefits Present value of benefits, i.e., 

 PV-benefits = ¦ 



N

y

By d

1

/(1 ) y

, where By is an undiscounted constant dollar benefit in year y

(computed net of noninvestment costs in that year), N is the number of years in the study period, d is 

the discount rate, and benefits are assumed to occur at the end of each year. 

Net Present Value 

(NPV)

Net present value of benefits less Investment costs, i.e., 

 NPV = {¦ 



N

y

By d

1

/(1 ) y

} - {¦



 



1

0

/(1 )

N

y

Iy d y

}, where all terms are as previously defined. 

Benefit-to-Cost 

Ratio (BCR) 

Ratio of PV-benefits to PV-investment, i.e., 

 BCR= {¦ 



N

y

By d

1

/(1 ) y

} / {¦



 



1

0

/(1 )

N

y

Iy d y

}, where all terms are as previously defined. 

Internal Rate of 

Return (IRR)

The real interest rate solution value (i) for which PV-benefits = PV-investment, NPV = 0 and BCR = 

1, when inserted in the following equality:

 

 {¦ 



N

y

By i

1

/(1 ) y

} - {¦



 



1

0

/(1 )

N

y

Iy i y

} = 0, where all terms are previously defined. 
88 

Table II.7-2 Summary of Standardized Conventions for Computing Constant Dollar Cash 

Flows, Discounting, and Computing Economic Performance Measures 

Standardized 

Conventions Instructions 

Discount Rates Use 7% and 3% real discount rates for discounting all constant dollar cash flows per 

OMB Circulars A-94 and A-4, respectively. The 7% discount rate is the primary rate for 

this evaluation, with the 3% rate presented for informational purposes. For 

intergenerational discounting with the SCC in Level 3, a 3% discount rate should be used 

in combination with both the 7% and 3% intragenerational Level 2 values. 

Constant Dollars Convert all cash flows to constant dollars as of the last year of the evaluation study 

period, and do this prior to discounting; i.e., if the study is conducted primarily in 2013, 

all current dollar cash flows would be adjusted to constant 2012 dollars prior to 

computing economic performance measures. Make the conversion using annual Gross 

Domestic Product (GDP) Implicit Price Deflators from U.S. Department of Commerce's 

Bureau of Economic Analysis (BEA), available at 

http://www.bea.gov/national/index.htm. The price deflators are routinely updated by 

BEA. Check with EERE program staff to determine if it is specifying a given BEA series 

to use; if not, use the most recently available annual series from BEA at the time of the 

study.

Study Period Define the study period as beginning upon the onset of investment costs, and ending on 

the last day of the year near the time the study is conducted and for which data are 

available; i.e., if the study is conducted primarily in 2013, and if investment cost for the 

portfolio began in 1986, and data are available through 2012, the study period would be 

set as the beginning of 1986 through the end of 2012.

Cash-Flow Modeling Model investment costs as though they occur at the beginning of the year in which they 

occur, and model benefits as though they occur at the end of the year in which they occur. 

56 

Time for Expressing 

Present Value Amounts 

(Base Year)

Express all present value amounts as of the beginning of the year in which the onset of 

cash flow begins; i.e., if first cash flows occur in 1986, the study period begins in 1986, 

and all amounts will be discounted back to the beginning of that year, i.e., in this 

illustration, all present values will be stated as a lump-sum equivalent amount occurring 

as of the beginning of 1986.57 

Use of Five Different 

Performance Measures Compute and report each of the five measures defined in Table II-7-1. 

 

56 It is recognized that cash-flow modeling conventions vary and include beginning of year, end of year, beginning of year for 

investment cost and end of year for other cash flows, middle of year, and continuous through-out the year models. These are 

merely conventions rather than true representations of actual cash flows. Cash-flow modeling conventions are considered 

appropriate for evaluations; more exact cash flow representation is used for financial transactions. To promote consistency, 

evaluators must follow the cash flow modeling convention specified in the table.

57 Because evaluation studies subject to the Guide will be conducted at different times and will address different portfolios, it is 

considered infeasible to require consistency of study period, year in which constant dollars are stated, and base year for stating 

present values. Instead, these variations can be adjusted to a common basis across studies at a later time if needed by EERE. 
89 

◙ Table II.7-3 Illustration of Year-by-Year and Total Investment Costs for the Vehicle 

Technologies Office R&D Investments in Energy Storage Technology, 1992-2012 

Year 

(1) 

VTO R&D Investments 

(thousands of 2012$, 

rounded) 

(2) 

VTO R&D Investments 

Discounted at 7% to 

1/1/1992 (thousands of 

2012$) 

(3) 

VTO R&D Investments 

Discounted at 3% to 

1/1/1992 (thousands of 

2012$) 

1992 39,783 39,783 39,783 

1993 45,557 42,577 44,230 

1994 51,699 45,156 48,731 

1995 39,200 31,999 35,874 

1996 37,145 28,338 33,003 

1997 34,763 24,786 29,987 

1998 37,397 24,919 31,319 

1999 30,272 18,852 24,614 

2000 30,474 17,736 24,056 

2001 35,234 19,165 27,004 

2002 33,375 16,966 24,834 

2003 30,051 14,277 21,709 

2004 26,987 11,983 18,928 

2005 29,832 12,379 20,314 

2006 31,444 12,195 20,788 

2007 48,238 17,484 30,962 

2008 52,128 17,658 32,484 

2009 77,347 24,486 46,796 

2010 82,035 24,271 48,187 

2011 84,778 23,442 48,348 

2012 93,034 24,042 51,511 

Totals 970,773* 492,491** 703,463*** 

*Total undiscounted Investment Costs; **Total PV-investment @7%; ***Total PV-investment @ 3%. 

Source: Link et al. (2014). 
90 

◙ Table II.7-4 Illustration of Year-by-Year and Total Attributable Energy Savings, Vehicle 

Technologies Office R&D Investments in Energy Storage Technology, 1992-2012 

Year 

(1) 

Attributable Fuel 

Savings (thousands of 

2012$) 

(2) 

Attributable Fuel 

Savings Discounted at 

7% to 1/1/1992 

(thousands of 2012$) 

(3) 

Attributable Fuel Savings 

Discounted at 3% to 

1/1/1992 (thousands of 

2012$) 

1999 2 1 2 

2000 1,611 876 1,235 

2001 5,027 2,555 3,741 

2002 10,773 5,118 7,783 

2003 21,419 9,510 15,023 

2004 43,500 18,051 29,621 

2005 106,175 41,176 70,194 

2006 187,326 67,896 120,237 

2007 295,234 100,006 183,980 

2008 423,230 133,984 256,061 

2009 356,448 105,460 209,376 

2010 461,639 127,647 263,266 

2011 627,142 162,065 347,233 

2012 736,723 177,928 396,025 

Undiscounted total 3,276,249* 

Discounted totals 952,275** 1,903,777*** 

*Gross Benefits (GB); **Present value-benefits at 7%; ***Present value-benefits at 3%. 

Source: Link et al. (2014). 

Recall, as explained in Step 2, that the 

impact evaluation study may examine 

technologies that are characterized as 

infratechnology, technology platform, or 

product/process technologies. When the 

technologies selected from a portfolio for 

detailed analysis differ in their 

characterization and how they generate 

benefits, they are generally analyzed 

individually instead of as a group. 

When technologies in a portfolio are 

analyzed individually, and their benefits are 

calculated separately, it may be desirable 

also to calculate their economic performance 

measures individually―in addition to the 

overall performance measures for the 
91 

portfolio―provided that the individual 

investment costs can be obtained. The 

evaluator should work with EERE program 

staff in this case to determine (1) if the 

individual investment costs can be broken 

out of portfolio investment cost, and (2) if 

there is interest of EERE program staff in 

knowing the economic performance of the 

individual technologies that together make 

up return on the total portfolio. 

An example of the separate reporting of 

economic performance measures for 

individual technologies within the portfolio 

is provided by the evaluation of geothermal 

technologies in the report by Gallaher et al. 

(2010). Table II.7-5, drawn from that report, 

shows individual returns for each of the four 

technologies selected for detailed analysis: 

PDC drill bit technology, binary-cycle plant 

technology, TOUGH reservoir computer 

models, and high-temperature cement―each 

quite distinctive, affecting energy and other 

resources in unique ways, and requiring 

different data sets and calculations to 

estimate benefits. (Not shown in Table II.7-

5, but provided by the report, are the 

economic performance measures for the 

geothermal portfolio overall.) 
92 

◙ Table II.7-5 Illustration of Separate Reporting of Economic Performance Measures for Individual Technologies Selected 

for Detailed Analysis (These are in addition to those provided for the Total Geothermal Portfolio) 

PDC Binary TOUGH Cement 

Year Total Benefits Program Expenses Net Benefits Total Benefits Program Expenses Net Benefits Total Benefits Program Expenses Net Benefits Total Benefits Program Expenses Net Benefits 

1976 $0 $2,081 -$2,081 $0 $2,036 -$2,036 $0 $797 -$797 $0 $142 -$142 1977 $0 $2,081 -$2,081 $0 $2,036 -$2,036 $0 $797 -$797 $0 $142 -$142 1978 $0 $2,081 -$2,081 $0 $2,036 -$2,036 $0 $797 -$797 $0 $142 -$142 1979 $0 $2,081 -$2,081 $0 $2,036 -$2,036 $0 $797 -$797 $0 $142 -$142 1980 $0 $2,081 -$2,081 $0 $2,036 -$2,036 -$3,158 $797 -$3,955 $0 $142 -$142 1981 $0 $2,081 -$2,081 $0 $2,036 -$2,036 $7,041 $797 $6,244 $0 $142 -$142 1982 $319,823 $2,081 $317,742 $0 $2,036 -$2,036 $3,495 $797 $2,698 $0 $142 -$142 1983 $400,600 $2,081 $398,519 $0 $2,036 -$2,036 -$6,065 $797 -$6,862 $0 $142 -$142 1984 $630,196 $2,081 $628,115 $2,114 $2,036 $78 $10,446 $797 $9,649 $0 $142 -$142 1985 $662,306 $2,081 $660,225 $2,047 $2,036 $11 $6,455 $797 $5,658 $0 $142 -$142 1986 $436,938 $2,081 $434,857 $9,822 $2,036 $7,786 $21,896 $797 $21,099 $0 $142 -$142 1987 $443,817 $1,844 $441,973 $14,398 $1,639 $12,759 $21,946 $797 $21,149 $0 $142 -$142 1988 $474,236 $1,683 $472,553 $3,750 $1,335 $2,415 $15,404 $797 $14,607 $0 $142 -$142 1989 $465,778 $1,601 $464,177 $8,238 $1,015 $7,223 $28,656 $797 $27,859 $0 $142 -$142 1990 $619,142 $1,490 $617,652 $15,229 $1,025 $14,204 $38,331 $797 $37,534 $0 $142 -$142 1991 $610,952 $1,586 $609,366 $7,337 $2,277 $5,060 $44,331 $797 $43,534 $0 $142 -$142 1992 $548,896 $1,575 $547,321 $15,926 $3,542 $12,384 $42,944 $159 $42,785 $0 $142 -$142 1993 $655,894 $1,455 $654,439 $26,175 $3,745 $22,430 $44,169 $159 $44,010 $0 $142 -$142 1994 $640,427 $1,335 $639,092 $6,700 $3,770 $2,930 $42,213 $159 $42,054 $0 $142 -$142 1995 $618,012 $1,341 $616,671 -$12,363 $2,890 -$15,253 $34,046 $159 $33,887 $0 $142 -$142 1996 $731,344 $1,474 $729,870 -$5,514 $1,844 -$7,358 $35,623 $159 $35,464 $0 $142 -$142 1997 $922,792 $1,558 $921,234 $3,362 $1,925 $1,437 $38,239 $159 $38,080 $0 $142 -$142 1998 $778,133 $1,817 $776,316 $978 $1,787 -$809 $37,958 $159 $37,799 $0 $142 -$142 1999 $564,958 $1,979 $562,979 -$391 $1,616 -$2,007 $37,664 $159 $37,505 $41 $142 -$101 2000 $862,685 $2,045 $860,640 -$2,399 $1,452 -$3,851 $33,581 $159 $33,422 -$123 $142 -$265 2001 $1,454,710 $2,188 $1,452,522 -$2,202 $1,203 -$3,405 $35,559 $159 $35,400 $1 $150 -$149 2002 $1,454,567 $2,188 $1,452,379 $2,226 $1,040 $1,186 $37,844 $159 $37,685 $165 $150 $15 2003 $2,025,308 $2,188 $2,023,120 $2,489 $971 $1,518 $37,994 $159 $37,835 $291 $150 $141 2004 $2,497,349 $2,188 $2,495,161 $1,721 $953 $768 $39,428 $159 $39,269 $472 $120 $352 2005 $3,429,701 $2,188 $3,427,513 $6,471 $972 $5,499 $38,802 $159 $38,643 $1,540 $130 $1,410 

(Continued) 
93 

PDC Binary TOUGH Cement 

Year Total Benefits Program Expenses Net Benefits Total Benefits Program Expenses Net Benefits Total Benefits Program Expenses Net Benefits Total Benefits Program Expenses Net Benefits 

2006 $4,631,182 $2,188 $4,628,994 $4,882 $972 $3,910 $38,797 $159 $38,638 $1,666 $150 $1,516 2007 $5,589,616 $2,188 $5,587,428 $2,677 $972 $1,705 $40,735 $159 $40,576 $1,811 $142 $1,669 2008 $6,081,340 $2,188 $6,079,152 $10,533 $972 $9,561 $41,817 $159 $41,658 $1,955 $142 $1,813 

Undiscounted Total 

$38,550,70

2 $63,178 

$38,487,52

4 $124,206 $60,313 $63,893 $846,193 $15,455 $830,738 $7,820 $4,684 $3,136 

PVa at 

7% $7,813,212 $26,461 $7,786,751 $42,848 $26,819 $16,029 $219,445 $8,619 $210,826 $1,013 $1,938 -$925 

PVa at 

3% 

$18,514,20

1 $41,015 

$18,473,18

6 $76,269 $40,701 $35,568 $457,957 $11,655 $446,302 $3,199 $3,037 $162 

BCR at 7% 295.3 1.6 25.5 0.5 BCR at 3% 451.4 1.9 39.3 1.1 IRR 139% 16% 48% NA 

a PV Base year is 1976. 

Source: Gallaher et al. (2010). 
94 

7.B Calculate economic 

performance measures for level 

two of a three-level analysis

The level 2 analysis is accomplished by 

adding the environmental health benefits to 

Level 1 benefits and comparing combined 

benefits against total portfolio investment 

cost.58

Bring forward from Step 4 the year-by-year 

constant dollar cash flows for environmental 

health benefits, using the midpoint between 

the low and high estimates for each year, for 

each discount rate. Combine these benefits 

with energy and other resource benefits. Pair 

the combined year-by-year cash flows with 

the portfolio investment costs to calculate 

the five economic performance metrics 

described in Table II.7.1, while following 

the conventions specified in Table II.7-2 

Table II.7-6 is an example drawn from the 

impact evaluation of engine combustion 

technology by Link (2010). 

Table II.7-7 uses the combined benefits data 

from the last column of Table II.7-6 and the 

total portfolio investment costs to compute 

additional NPV, BCR, and IRR measures. 

 

58 As noted in Step 4, when calculating economic 

performance measures, the corresponding COBRA results 

should be used when estimating net present values and 

benefit-to-cost ratios. Use the 7% COBRA results when 

calculating performance measures with the 7% discount 

rate, and use the 3% COBRA results when calculating 

performance measures with the 3% discount rate. This is 

necessary because COBRA generates values using internal 

user-specified discount rates. Evaluators should be aware of 

how COBRA may affect the generation of the IRR for 

Level 2 economic performance measures and higher. When 

estimating the internal rate of return, evaluators should use 

the time series of values discounted at 7% with the 

understanding that there may be some slight bias to the 

overall internal rate of return. COBRA results are 

exogenous: when calculating the IRR results are not being 

iterated as the solution rate is sought. Since the values 

remain generated at a 7% discount rate, the internal rate of 

return on the stream of total net benefits will be slightly 

biased. 

The table is drawn from the same impact 

evaluation of engine combustion technology 

by Link (2010). 

7.C Calculate economic 

performance measures for level 

three of a three-level analysis 

Bring forward from Step 4 the year-by-year 

constant dollar cash flows for GHG 

reduction and combine them with the other 

benefits data from Level 2. Compare the 

combined year-by-year cash flow impacts 

with the total portfolio investment costs to 

calculate the five economic performance 

metrics defined in Table II.7.1, following 

the conventions specified in Table II.7-2. 
95 

◙ Table II.7-6 Combining Year-by-Year Energy and Other Resource Benefits* 

and Health Benefits 

Year 

Dollar Value of Economic 

Benefits* (Reduced Fuel 

Consumption) (millions 

$2008) 

Economic Value of 

Avoided Adverse Health 

Incidence (millions $2008) 

Combined Benefits 

(millions $2008) 

1995 $1,502.0 $2,597.8 $4,099.8

1996 $1,683.7 $2,681.1 $4,364.8

1997 $1,547.3 $2,615.8 $4,163.1

1998 $1,410.7 $2,435.4 $3,846.1

1999 $1,996.8 $3,278.1 $5,274.9

2000 $2,817.2 $3,675.1 $6,492.3

2001 $2,526.5 $3,623.5 $6,150.0

2002 $2,283.6 $2,735.7 $5,019.3

2003 $2,097.7 $2,263.4 $4,361.1

2004 $2,491.7 $2,327.9 $4,819.6

2005 $4,189.2 $3,078.0 $7,267.2

2006 $4,834.5 $3,279.0 $8,113.5

2007 $5,115.4 $1,114.0 $6,229.4

Level II 

Gross 

Benefits 

$34,496.4 $35,704.8 $70,201.1

Source: Link (2010). 

Note: the Link study did not use the "Level I and II" terminology. The economic value of avoided adverse health 

incidence was generated using a 7% social discount rate. 

*At the time of the referenced study, energy and other resource impacts were called “economic benefits”. 
96 

◙ Table II.7-7 Example of Economic Performance Metrics Calculated from the 

Combined Energy and Other Resource Benefits* and Health Benefits Compared with 

Portfolio Investment Costs

Metric 7% Discount 

Rate 

3% Discount 

Rate

Internal Rate of 

Return 

Present Value of Net Benefits 

(billions $2008)

$23.1 $42.6

Benefit-to-Cost Ratio 53 to 1 66 to 1

Internal Rate of Return 63% 

Source: Link (2010). 

* At the time of the referenced study, energy and other resource impacts were called “economic benefits.”

7.D Prepare a graph showing 

year-by-year benefits by 

category and for the total, 

together with portfolio 

investment cost 

This is a new requirement in the Guide. 

Figure II.7 illustrates such a graph using 

hypothetical data to show all the elements 

that go into a Level Three analysis: 

monetary value of energy and other resource 

benefits, monetary value of avoided adverse 

health incidents , monetary value of GHG 

reductions, the combined economic benefits, 

and portfolio investment cost. Each of these 

cash-flow series is expected to be available 

for most evaluation studies, although it is 

possible that air emissions will not be 

affected by some technologies resulting in 

the need for only a Level 1 analysis. 

Based on hypothetical data for purposes of illustration only. 

Figure II.7 Illustration of Elements that Make up a Level Three Analysis 

0

50

100

150

200

250

300

350

400

450

1990 1992 1994 1996 1998 2000 2002 2004 2006 2008 2010 2012

THOUSANDS 

YEAR 

Combined Benefits

Portfolio Investment Cost (thousand

$2012)

Energy/Other Resource Benefits

(thousand $2012)

Healthcare Cost Reductions (thousand

$2012)

Monetary Value of CO2 Reductions

(thousand $2012)

Portfolio Investment Cost 

(thousand $2012) 

Monetary Value of Energy/Other 

Resources (thousand $2012) 

Monetary Value of Avoided Adverse 

Health Incidents (thousand $2012) 

Monetary Value of CO2 Reductions 

(thousand $2012) 

Combined Economic Benefits (Level 3) 
97 

7.E Characterize how 

worthwhile the EERE portfolio 

investment has been based on 

monetized benefits and 

investment costs for each level 

of analysis considered 

Discuss what the economic performance 

measures mean in terms of how worthwhile 

the EERE investment has been 

economically. The test that an investment 

has been economically worthwhile is that 

the NPV is positive, or that the BCR is 

greater than one, or that the IRR is greater 

than the required rate of return (as indicated 

by the OMB-specified discount rate). The 

extent to which an investment has been 

economically worthwhile is signaled by the 

magnitude of these measures—the larger the 

measure, the more economically 

worthwhile the investment has been, others 

factors being equal. However, it is important 

to keep in mind that estimates for these 

economic performance metrics are 

conservative; they are based on partial 

benefits because not all benefits have been 

expressed in monetary terms. Benefits not 

expressed in dollars also contribute to how 

worthwhile an EERE portfolio investment 

has been. 

7.F Perform a threshold or 

break-even analysis if sizable 

negative impacts have been 

revealed by any of the 

supporting qualitative analyses 

The Guide recognizes that it may not be 

possible to measure all relevant impacts 

quantitatively―either in monetary or 

physical units. In cases where quantitative 

measurement either could not be done or 

was not required, the evaluator is instructed 

to provide qualitative treatments. As a result, 

there may be qualitative results for 

economic, environmental, and energy 

security impacts for the technologies 

selected for detailed analysis, as well as 

qualitative results for the part of the 

portfolio not selected for detailed analysis. It 

is expected that in most cases these qualified 

results will be either positive on net, adding 

to the returns from EERE's investment, and 

providing additional evidence that the 

economic performance measures represent 

conservative, lower-bound estimates or, at 

worse, neutral. 

It is also possible, however, that there could 

be negative impacts that have been captured 

qualitatively. In this case, the evaluator 

should conduct a threshold or breakeven 

analysis, per the recommendation of OMB 

Circular A-4, to address the question, "How 

large could the combined value of negative 

nonquantified effects be before it would 

offset the NPV?" 59 In fact, this threshold or 

 

59 Recall that the investment costs of "failed projects" are 

already taken into account. To offset NPV would require 

that there be projects in the portfolio that not only did not 

contribute on net to portfolio benefits, but actually reduced 

them.

How Worthwhile? 

 

The economic performance measures 

indicate if, and to what extent, an 

investment has been economically 

worthwhile based on benefits express 

in dollars. 

Conditions that must be met to show 

that a federal investment has realized 

or exceeded the required minimum 

acceptable rate of return on 

investment are the following: 

NPV ≥ 0, when d=7% 

BCR ≥ 1, when d=7%

IRR ≥ 7%
98 

breakeven value will be equal to the NPV of 

the impacts measured in dollars. Although 

the breakeven value appears self-evident, the 

breakeven analysis is recommended because 

it serves to focus attention on all impacts 

and explicitly considers the significance of 

impacts not included in the economic 

performance measures. 
99 

II.8 Account for Remaining Effective Useful Life 

(EUL) of Purchased & Installed Systems in an 

Extended Analysis 

The inclusion of benefits to reflect 

remaining effective useful life (EUL) of 

purchased and installed systems represents 

an extension of the retrospective analysis,in 

that they are future benefits not yet realized 

but are reasonably assured to accrue. 

Effective useful life is the period over which 

an asset, such as plant, equipment, and 

systems and components, with normal 

maintenance and repair, can be expected to 

continue to be usable for the intended 

purpose. 

An argument for the inclusion of these 

future benefits is that total portfolio 

investment costs include the R&D costs of 

systems that have only recently been 

purchased and whose benefits are expected 

to continue for years beyond the end of the 

retrospective analysis period. Further, the 

purchase decision has already been made, 

removing much uncertainty from the 

occurance of these future benefits. Thus, 

taking into account the remaining EUL 

benefits of those technologies selected for 

detailed analysis may represent a more 

balanced treatment of benefits and costs. 

8.A Document EUL of 

technologies selected for 

detailed analysis 

To take into account remaining EUL in an 

evaluation study, a starting point is to 

estimate the EUL of the major physical 

assets that represents the technologies 

selected for detailed analysis. 60 

 

60 Because coverage of technologies by this Guide is quite 

broad (ranging from product technologies to 

infratechnologies, and from energy generating plants to 

vehicle batteries), the physical assets for which EUL may 

be needed are too varied to be named.

 Document EUL: of technologies selected for detailed analysis. 

 Estimate remaining EULs. 

 Estimate benefits for each year of remaining EUL. 

 Calculate economic performance measures at level 3, inclusive of 

remaining EUL effects. 

 Report results over the remaining EUL in a separate chapter. 

8. Account for 

Remaining EUL 

of Purchased 

and Installed 

Systems in an 

Extended 

Analysis
100 

Information sources for EUL include the 

following: 

x Product warranties;

x Supplier product documents;

x Records on past experience;

x Simulations of EUL;

x Interview of experts; and

x Literature review of related EUL studies. 

If indications of the reliability of EUL 

estimates are available, this information 

should be collected for the purpose of 

performing sensitivity analysis. For 

example, records of past experience or 

reliability studies may express remaining 

life as a range or otherwise reveal data 

uncertainties.61

8.B Estimate remaining EUL 

With EUL data in hand, it is possible to 

estimate the remaining EUL needed for the 

extended analysis for those units or systems 

installed or in operation by the close of the 

retrospective evaluation period. 

By comparing the EUL with the time of 

purchase/installation and the cut-off time for 

the retrospective evaluation study, the 

remaining time over which the asset is 

expected to continue yielding benefits can 

be estimated. For example, for a system 

installed in 2010, with a 10-year EUL and a 

retrospective study period cut-off of 2012, 

the remaining EUL is eight years, i.e., two 

 

61 For example, an EUL study sponsored by the California 

Public Utilities Commission (CPUC) of 

retrocommissioning (RCx) found high uncertainty for 

existing EUL data for situations under assessment, and 

concluded that a larger investigation was needed to provide 

reliable data. (Report of ACEEE Summer Study on Energy 

Efficiency in Buildings, 2010, John Roberts and Bing Tso, 

SBW Consulting, Inc., "Do Savings from 

Retrocommissioning Last? Results from an Effective 

Useful Life Study." 

years have already been taken into account 

by the retrospective evaluation. 

8.C Estimate benefits for each 

year of remaining EUL 

For systems with remaining EUL, estimate 

the benefits for each year remaining beyond 

the retrospective study cut-off date. Take 

into account energy and other resources, 

environmental, and energy security benefits. 

Knowledge benefits need not be estimated 

because additions to the knowledge base 

relate to EERE's R&D investment and that 

does not change in relation to remaining 

EULs. For systems with extremely long 

remaining EULs, the evaluator may choose 

to impose a constraint on the time extension 

for assessing EUL effects. 

To estimate the value of remaining EUL 

benefits, it is necessary to re-examine 

assumptions made for the retrospective 

evaluation and to determine if they apply to 

the future years over which benefits are to 

be estimated. 

These considerations include if the next-best 

alternative identified in the Step 2 analysis 

will continue to apply and if other 

assumptions underlying the evaluation will 

continue to hold. If the next-best alternative 

is not expected to change and other 

estimating relationships are expected to 

hold, the same estimating algorithms that 

were used for the retrospective analysis can 

be used to extend the analysis.62 If the nextbest alternative is expected to change during 

the extended period, then it will be 

necessary to return to the beginning of Step 

2 and repeat Step 3 (to calculate energy and 

other resource impacts over the extended 

period). Likewise, it will be necessary to 

repeat Steps 4 (environmental benefits) and 

 

62 Projected input data for future years, such as prices will 

be required. 
101 

5 (energy security effects) to calculate these 

impacts over the extended period. If 

underlying assumptions are expected to 

change, then the formulae for estimating 

impacts going forward should reflect that 

change. 

If there are reasons to modify any of the 

underlying factors in the estimation, the 

evaluator should document the details of the 

modification(s) in the report chapter on 

EUL. 

Show in tabular form the year-by-year series 

of remaining EUL benefits for each benefit 

category ―i.e., for energy and other 

resources, environmental, and energy 

security benefits (but not for knowledge 

benefits). Also show totals in physical units 

for each benefit category over the period of 

remaining EUL. An example of the latter, 

i.e., total benefits in physical units for each 

benefits category, is provided in Table II.8-

1. The table shows a side-by-side 

comparison of totals for the retrospective 

evaluation and for the extended analysis 

including remaining EUL, with all measures 

in physical units. The example is drawn 

from the evaluation of EERE's energy 

storage investment by Link et al. (2013). 

The retrospective evaluation stops at the end 

of year 2012. The extended analysis stops at 

the end of year 2022. 

◙ Table II.8-1 Illustration of Benefits for the Retrospective Evaluation (1992-2012) and the 

Extended Remaining EUL Analysis (1992-2022) 

Retrospective 

Analysis through 

2012 

Life-Cycle Analysis*

through 2022 Unit of Measure 

Energy and Energy Security Benefits

Avoided petroleum consumption 54,199,182 111,870,462 Gallons of gasoline 

equivalent 

Avoided foreign petroleum consumption 27,226,445 47,833,782 Barrels of oil 

Emissions Benefits

Avoided GHG emissions (CO2eq) 6,989,237 14,461,042 Metric tons 

Avoided volatile organic compounds emissions (VOCs) 3,928 7,926 Short tons 

Avoided nitrogen oxides (NOx) 1,217 2,324 Short tons 

Avoided particulate matter emissions (PM2.5) 2 16 Short tons 

Avoided sulfur dioxide emissions (SO2) 128 265 Short tons 

Avoided ammonia emissions (NH3) 643 1,329 Short tons 

Environmental Health Benefits

Incidence

Avoided mortalitya 20.04 42.39 Deaths 

Avoided infant mortalitya 0.02 0.05 Deaths 

Avoided nonfatal heart attacks 6.96 14.73 Attacks 

Avoided resp. hospital admissions. 4.34 9.17 Admissions 

Avoided CDV hospital admissions 4.05 8.57 Admissions 

Avoided acute bronchitis 18.24 38.57 Cases 

Avoided upper respiratory symptoms 331.47 701.04 Episodes 

(Continued) 
102 

Avoided lower respiratory symptoms 232.31 491.31 Episodes 

Avoided asthma ER visits 7.79 16.48 Visits 

Avoided MRAD 10,265.31 21,710.22 Incidences 

Avoided work loss days 1,734.48 3,668.28 Days 

Avoided asthma exacerbations 348.78 737.64 Episodes 

8.D Calculate economic 

performance measures for level 

3, inclusive of remaining EUL 

effects 

Using guidance provided in Step 7, calculate 

economic performance measures for a Level 

three analysis, taking into account remaining 

EUL effects. Hold portfolio investment cost 

unchanged because it already reflects the 

investment cost of the systems whose 

remaining EULs are taken into account. 

Table II.8-2, also drawn from the evaluation 

study of EERE's energy storage investments, 

shows the side-by-side comparisons of 

economic performance measures for the 

retrospective analysis and the EUL-extended 

analysis. Because the study illustrated was 

performed prior to the requirement in this 

Guide for the monetary valuation of 

environmental benefits from reduction in 

GHGs, the illustration is based on combined 

energy and other resource benefits and 

environmental health benefits only. 

Evaluations performed under this 2014 

edition of the Guide are to include extended 

combined total monetary benefits, including 

benefits from reduction in GHGs where 

applicable. 

8.E Report the results over the 

remaining EUL in a separate 

chapter

In keeping with the principal focus on 

retrospective evaluation, report the extended 

EUL analysis and results in a separate 

chapter of the evaluation study report. 

Include documentation of: estimated EULs; 

uncertainties associated with these values; 

remaining EULs; the study period used in 

the extended analysis; and any changes in 

the next-best alternative or other underlying 

assumptions. Also include tables of benefit 

estimates year-by-year by category in 

physical and dollar units over the future 

period; investment cost used for the 

extended analysis (to confirm that 

investment cost has not been changed); and 

economic performance measures for the 

EUL-extended analysis in comparison with 

those of the original retrospective-only 

analysis. 

If information is available on uncertainties 

of EULs, perform sensitivity analysis for the 

extended analysis results. Provide details of 

the analysis. 

Describe the effects of including remaining 

EULs in the portfolio evaluation. Include in 

the report's Executive Summary a table that 

includes both retrospective and EUL results. 

 

Source: Link, et al. 2014.

*The labeling of this table column reflects the fact that the term "Life-Cycle Analysis" was used in the previous edition of the Guide to 

describe remaining EUL effects. 
103 

◙ Table II.8-2 Illustration of Economic Performance Measures for a Retrospective 

Evaluation (1992-2012) and Remaining EUL Evaluation (1992-2022) 

Measures based on combined energy and 

environmental health benefits

Retrospective 

Analysis 

through 2012 

Life-Cycle 

Analysis*

throu

gh 2022

Unit of 

Measure

Net present value @ 7% [Base year = 1992] $506 $1,294 Million, 2012$ 

Net present value @ 3% [Base year = 1992] $1,303 $3,334 Million, 2012$ 

Benefit-to-cost ratio @7% 2.03 3.63 

Uncertainty bound around BCR@7% 1.73—2.32 3.08—4.18 

Benefit-to-cost ratio @3% 2.85 5.74 

Internal rate of return 14.3% 17.7% 

Portfolio R&D Investments 

Present value @ 7% [Base year = 1992] $492 $492 Million, 2012$ 

Present value @ 3% [Base year = 1992] $703 $703 Million, 2012$ 

Economic (incl. Energy) Benefits 

Present value @ 7% [Base year = 1992] $952 $1,706 Million, 2012$ 

Present value @ 3% [Base year = 1992] $1,904 $3,836 Million, 2012$ 

Source: Link et al. 2014. 

*The labeling of this table column reflects the fact that the term "Life-Cycle Analysis" was used in the previous edition of the 

Guide to describe remaining EUL effects. 
104 

II.9 Perform Sensitivity Analysis 

Sensitivity analysis is a technique for 

determining how the evaluation outcome 

would change if uncertain inputs, 

assumptions, or modeling approaches were 

changed. It allows the evaluator to show 

uncertainty in the outcome by expressing 

results as a range of potential values rather 

than as a point estimate. 

 

One case of sensitivity analysis built into the 

evaluation is the use of multiple alternative 

discount rates. The discount rate has a large 

effect on the economic performance 

measures NPV and BCR by incorporating a 

required real opportunity cost of capital into 

their calculations, and on IRR by setting a 

Minimum Acceptable Rate of Return against 

which the IRR must be compared. 

The 7% real discount rate for EERE 

evaluations is the principal discount rate and 

is set by OMB for agencies to use for 

benefit-cost analysis aimed at determining if 

a given investment is worthwhile.63 OMB 

recommends using other discount rates to 

 

63 OMB Circular A-94 (OMB 1992). 

show the sensitivity of the estimates to the 

discount rate assumption. Noting that the 

average real rate of return on long-term 

government debt has averaged about 3%, 

OMB directs the use of 3% real discount 

rate for benefit-cost analysis of regulatory 

programs.64 

Although evaluations covered by this Guide 

are not for regulatory purpose, the Guide has 

adopted both discount rates. Seven percent 

is the primary rate and 3% is used for 

purposes of sensitivity analysis. In addition, 

by reporting undiscounted NB, the Guide in 

effect also uses a 0% discount rate. Thus, by 

basing results on discount rates ranging from 

0% to 7%, the analysis shows the sensitivity 

of results to the discount rate.65 

 

64 OMB Circular A-4 (OMB 2003). 65 Per OMB Circular A-94, the test to determine if an 

investment has been economically worthwhile requires that 

at least a 7% annual compound return on investment be 

realized.

 Identify areas of major uncertainty in the analysis. 

 Recalculate results using alternative values for uncertain input 

variables, alternative assumptions, or alternative calculation 

approaches. 

 Show sensitivity results in a separate chapter of the report. 

9. Perform 

Sensitivity 

Analysis
105 

9.A Identify areas of major 

uncertainty in the analyses 

The following situations also warrant the use 

of sensitivity analysis: 

x When a range of implied or explicit 

values were obtained for an input 

variable; 

x When a given input variable or 

assumption is expected to have a large 

effect on outcome, and there is 

uncertainty about its value; and 

x When an alternative estimation approach 

could have been justified for use, and its 

use may have produced different results 

than the approach taken. 

9.B Recalculate results using 

alternative values for uncertain 

input variables, alternative 

assumptions, or alternative 

calculation approaches 

Apply the following principles for the 

scenarios listed below. To test for sensitivity 

of results resulting from: 

x A range of input values, use the low and 

high ends of the range to generate a 

range of outcomes. Table II.9-1, shows 

an example of the results of sensitivity 

analysis of environmental health benefits 

to two estimates of the displaced fuel 

mix by a geothermal plant. 

x The distribution of responses from 

experts, use the upper and lower bounds 

of the distribution, or a statistical 

measure of variation. The example in 

Table II.9-2 shows sensitivity of solar 

photovoltaic results to a range of expert 

opinions on EERE's acceleration of 

technology advancement. 

x Different estimation approaches, go to 

the stage of the analysis in which the 

estimation approach is used, substitute 

the alternative approach, and recalculate 

results. The example in Table II.9-3 

shows sensitivity of results to using a 

different, but equally valid, approach to 

calculate the change in diesel fuel 

consumption from an improvement in 

truck brake thermal efficiency. 

◙ Table II.9-1 Sensitivity of Environmental Health Benefits to Displaced Fuel Mix of a 

Geothermal Power Plant 

Displaced Generation 

60% Coal, 39% NG, Percentage Reduction 

1% Oil 50% Coal, 50% NG 

PM (short tons) 22,193 18,780 15.4% 

SO2 (short tons) 9,614 7,992 16.9% 

NOX (short tons) 5,037 4,227 16.0% 

GHG (thousand tCO2e) 6,585 6,268 4.8% 

Monetized health benefit 

(PVa

 at 7%, thousands $2008) $126,644 $107,501 15.1% a PV base year is 1976. 

Source: Gallaher et al. (2010). 

Note: The study obtained the applicable fuel mix for each state in which geothermal power was assumed likely to offset 

conventional power production and averaged the value of the mix across those states: 60% coal, 39% natural gas, and 1% oil. The

primary analysis used this mix to calculate the change in air emissions and related environmental health benefits (col. 2). The

sensitivity analysis used an alternative mix: 50% coal and 50% natural gas (col. 3). The percentage reductions in air emissions

and related health benefits from using the alternative fuel mix are given in col. 4. Estimated health benefits would have been 

15.1% lower if the alternative fuel mix had been used. 
106 

◙ Table II.9-2 Sensitivity of Results to a Range of Expert Opinion on EERE's 

Acceleration of Flat-Plate Solar Array (FSA) Photovoltaic Technologies 

Measure 

Results 

(12-year acceleration) 

Under 10-Year FSA 

Acceleration Effect 

Under 15-Year FSA 

Acceleration Effect 

Total benefits (million 2008$) $18,734.8 $14,389.8 $25,875.7 

Total costs (million 2008$) $3,707.9 $3,707.9 $3,707.9 

Net benefits (million 2008$) $15,026.8 $10,681.8 $22,167.7 

Internal rate of return 17% 14% 20% 

NPV at 7% (million 2008$; base 

year = 1975) 

$1,458.9 $858.8 $2,394.6 

Benefit-to-cost ratio at 7% 1.83 1.49 2.37 

NPV at 3% (million 2008$; base 

year = 1975) 

$5,724.7 $3,987.2 $8,531.5 

Benefit-to-cost ratio at 3% 3.24 2.56 4.35 

Source: O'Connor et al. (2010) 

Note: The study approach was to ascertain from experts when the photovoltaic technology advances would have been made in 

the absence of the EERE's solar photovoltaic investment. The range of opinion was 10-years to 15-years; the principal analysis 

used an average of 12-years. Sensitivity analysis shows the range of results when the low and high bounds of opinion are used. 

◙ Table II.9-3 Parts A and B. Sensitivity of Results to Using a Different Estimation 

Approach for Reduced Fuel Consumption 

Part A. Results from Estimation Approach Used in Principal Analysis: 

Metric 7% Discount Rate 3% Discount Rate Internal Rate of Return 

Present Value of Net Benefits

(billions $2008)

$23.1 $42.6 

Benefit-to-Cost Ratio 53 to 1 66 to 1 

Internal Rate of Return 63% 

Source: Link (2010). 

Part B. Results from Using an Alternative Estimation Approach in the Sensitivity Analysis: 

Metric 7% Discount Rate 3% Discount Rate Internal Rate of Return 

Present Value of Net Benefits

(billions $2008)

$17.8 $35.0 

Benefit-to-Cost Ratio 41 to 1 54 to 1 

Internal Rate of Return 50% 

Source Link (2010). 
107 

In this latter example shown in Table II.9-3, 

two estimation approaches are used to 

estimate diesel fuel cost savings from an 

improvement in brake thermal efficiency 

(BTE)―a driver of benefits in the analysis 

of EERE's investment in advanced 

combustion engine efficiency. Note that the 

sensitivity testing does not show which of 

the two estimation approaches is better for 

estimating diesel fuel cost savings. 

According to the referenced evaluation 

report, either estimation approach could 

reasonably be used in the analysis. The 

sensitivity analysis, however, plays a helpful 

role by demonstrating that the outcome is 

quite strong regardless of which of the two 

calculation approaches is used. 

9.C Show sensitivity results in a 

separate chapter of the report 

In a separate chapter of the evaluation 

report, discuss uncertainties, identify areas 

selected for sensitivity analysis, and explain 

why and how these were selected. Show 

how the sensitivity analysis is performed. 

Explain how the results are affected by the 

use of alternative values or approaches, and 

discuss the implications. Present summary 

results of the sensitivity analysis in the 

evaluation report's executive summary. 

 
108 

II.10 Assess Technical and Market Context; 

Discuss Success Factors

In conducting economic impact analyses of 

EERE R&D portfolios, evaluators gain 

insight into the effectiveness of programs 

and investment initiatives and how they 

relate to technology development and 

market challenges. These insights are 

particularly valuable to EERE technical 

managers, program directors, and policy 

analysts because they illuminate possible 

success factors that could be replicated in 

other R&D initiatives. Similarly, they may 

help avoid repeating R&D investment 

shortcomings. 

10.A Characterize the portfolio's 

broader technological and 

market context 

Describe the beginning- and end-of-study 

technical conditions. Draw on the initial 

portfolio analysis from Steps 1-

3―particularly the logic model analysis, the 

review of program records and other 

documents, and the interviews with experts 

to describe the technical conditions that 

characterized the beginning-of-study R&D 

environment for the subject technologies. 

Draw also on the results of the 

evaluation―particularly Steps 2-7―to 

describe the technical conditions that 

characterized the end-of-study R&D 

environment for the subject technologies. 

Address the following questions as 

applicable: 

 What technical barriers were the subject 

EERE portfolio investment and the 

specific technologies drawn from the 

portfolio intended to overcome? What 

were the key technical objectives? 

 To what extent were these technical 

objectives met? 

 Were there other technical barriers 

standing in the way of achieving boarder 

technical goals even if those targeted by 

the evaluated portfolio investment were 

overcome? 

Characterize the market environment faced 

by commercial products and processes 

incorporating the EERE-funded innovations. 

Address the following questions as 

applicable: 

 Characterize the portfolio’s broader technological and market context. 

 Assess EERE’s program strategies. 

 Provide actionable recommendations where possible. 

10. Assess 

Technical and 

Market 

Context; 

Discuss Success 

Factors
109 

x Has the EERE investment improved an 

existing product or advanced 

development of a new product? 

x What are the implications of the type of 

innovations for market entry? 

x What is the state of the industry and 

market into which the resulting products 

or processes advanced by the EERE 

R&D investment are launched? 

x What is the competitive situation faced 

by U.S. producers? 

x Are there market barriers that may 

inhibit the diffusion of the technology 

into domestic and global markets even if 

the EERE investment is successful 

technically? 

x Is there an EERE strategy that addresses 

market barriers? 

x How large is the potential national and 

global market and who holds existing 

market share? 

x What progress have commercialized 

products and processes incorporating the 

EERE-funded innovations made in 

penetrating new and existing markets? 

x What is the market outlook? 

x Are the major application areas those 

that were originally targeted by EERE? 

What are the most promising application 

areas, and how does past experience 

inform future EERE strategy and 

objectives? 

An example of providing technological and 

market context for an EERE portfolio 

investment is provided by the evaluation of 

solar photovoltaic investment. The 

evaluation report explains how each 

initiative was a response to specific 

technical barriers and technology needs 

existing at the time the initiative was 

launched. 

The solar photovoltaic evaluation study was 

able to obtain an industry study from the 

Massachusetts Institute of Technology's 

Energy Laboratory (Linden, L., et al, 1977) 

that identified primary market failures that 

◙ Study Context: Illustration of 

Providing an Account of Market 

Failures Existing at the Outset of 

EERE's Solar Photovoltaic 

Investment

x Energy prices that do not account for 

deleterious environmental or human health 

impacts associated with fossil fuel 

consumption and combustion. 

x Production uncertainties concerning prices, 

availability, quality, reliability, production 

volumes, and the ready supply of renewable 

and fossil fuel technology alternatives. 

x Technological uncertainties, particularly 

with respect to development costs, time, and 

R&D performance. 

x Interdependencies of production and 

technology development, which are the 

confluence of uncertainties, indivisibilities, 

and externalities that impede market 

function through asymmetries in information 

and poor convergence of expectations. 

x Indivisibilities, and inability to appropriate 

returns for technology development, such 

that, despite solar photovoltaics being in the 

national interest, the costs of developing and 

maturing the technology may preclude 

private-sector innovation if returns from 

innovation cannot be appropriated as profits 

within a suitable time horizon. 

x Imperfections in financial markets 

attributable to the chasm between internal 

sources of funding and the risk-reward 

profile that influences private equity 

financing. 

Non-competitive market structures that may 

inhibit new, competing sector development
110 

were inhibiting the development of 

terrestrial photovoltaics at the outset of the 

EERE photovoltaic investment. (See listing 

in the text box.) 

The EERE solar photovoltaic evaluation 

report provides a historical timeline context 

for EERE's contributions by describing how 

each initiative built on earlier advances, and 

by reporting yearly progress. Table II.10-1, 

drawn from the EERE solar photovoltaic 

evaluation report, shows notable technology 

outcomes of the EERE photovoltaic 

initiatives against a timeline of U.S. solar 

photovoltaic industry progress from 1976 

(the time of the initial EERE investment) to 

2008 (the end of the study period). Progress 

is shown in terms of photovoltaic module 

production, production cost per watt, and 

years of reliability. 

The solar photovoltaic evaluation report also 

provides accounts of the achievement of 

technical objectives by the EERE initiatives. 

For example, Table II.10-2 summarizes a 

sample of technical accomplishments under 

the Flat-plate Solar Array (FSA) 

development effort. 

An illustration of providing market context 

is provided by the evaluation study of EERE 

investment in energy storage technologies 

for hybrid and electric cars and light trucks 

(Link et al., 2013). As the study points out in 

its evaluation of NiMH and Li-ion batteries, 

all electric and hybrid cars and trucks on the 

road today use either of these battery 

technologies. The study provides 

background market data to help provide 

context for understanding the significance of 

the technology and where it stands in terms 

of market penetration. Among the extensive 

background market data the study provides 

is that given in Figure II.10. The data are 

relevant because of the market 

interdependence of battery technology and 

electric-drive vehicles. Demand for the 

battery technology examined is dependent 

on sales of electric-drive vehicles. The 

availability of batteries with improved 

performance and lower cost in turn increases 

the performance and reduces the cost of 

electric-drive vehicles, and, thereby, 

increases demand for electric-drive vehicles 

and, in turn, demand for batteries. 

This is only one example of the various 

market context measures that the evaluator 

can assemble to understand and explain the 

market context for an EERE research 

investment over its period of performance. 
111 

◙ Table II.10-1 Illustration of Presenting Notable Technology Outcomes for each EERE Initiative 

against a Timeline of U.S. Photovoltaic Industry Achievements, 1976-2008

Year 

Module Production (MW) Production 

Cost 

($/W)

Reliability 

(Years) Notable Technology Outcomes c-Si Thin Films Total 

1974 0.19 0.00 0.19 $114.44 2 

1975 0.37 0.00 0.37 $83.86 2 

1976 0.80 0.00 0.80 $53.28 2 Flat-Plate Solar Array Project 

x Block Purchases I-V 

x EVA for encapsulants 

x UCC silicon refining process 

x Silicon ingot growth 

x Silicon ribbon growth 

x Automated module assembly 

x Design and test methods for durability, 

performance, and safety 

x Laboratory cells reaching 22% efficiency 

x 10-year module warranties 

1977 1.22 0.00 1.22 $37.60 2 

1978 1.65 0.00 1.65 $25.64 2 

1979 2.07 0.00 2.07 $23.93 2 

1980 2.50 0.00 2.50 $22.22 2 

1981 4.46 0.00 4.46 $19.65 2 

1982 5.05 0.00 5.05 $17.09 5 

1983 5.63 0.00 5.63 $14.53 5 

1984 6.22 0.05 6.27 $11.96 5 

1985 7.30 0.50 7.80 $9.40 10 

1986 6.40 0.85 7.25 $8.99 10 

1987 7.45 1.40 8.85 $8.58 10 

1988 9.70 1.85 11.55 $8.16 10 

1989 12.95 1.45 14.40 $7.75 10 

1990 13.78 1.37 15.15 $7.34 20 

1991 16.48 1.00 17.48 $6.93 20 Thin-Film PV Partnerships 

x National teams 

x Basic research in a-Si, CdTe, and CIS 

x a-Si modules (ECD/Uni-Solar) 

x CdTe modules (First Solar [Solar Cells Inc.]) 

x CIS/CIGS modules (Global Solar) 

PV Manufacturing Technology Project

x Wire saw technology adoption for silicon ingot 

wafering 

x Automated cell and module assembly 

processes 

x In-line diagnostics and monitoring 

x High-efficiency c-Si cells 

x Cost reductions from $6.93 per watt in 1991 to 

$1.92 per watt in 2008 

x 25-year module warranties 

x Funded AstroPower (GE), BP Solar (Solarex), 

Evergreen, First Solar, Global Solar, SCHOTT 

Solar, SolarWorld USA (Arco/Siemens/ 

Shell), SunPower, Uni-Solar 

1992 16.95 1.65 18.60 $6.00 20 

1993 20.91 1.53 22.44 $5.69 20 

1994 24.31 1.95 26.26 $4.84 20 

1995 33.30 1.66 34.96 $4.53 20 

1996 37.35 2.46 39.81 $3.93 20 

1997 48.00 3.10 51.10 $3.77 25 

1998 48.10 5.80 53.90 $3.71 25 

1999 53.80 7.00 60.80 $3.45 25 

2000 66.00 9.00 75.00 $2.96 25 

2001 86.70 13.80 100.50 $3.00 25 

2002 109.40 18.20 127.60 $2.85 25 

2003 86.82 15.80 102.62 $2.91 25 

2004 115.20 23.50 138.70 $2.80 25 

2005 133.60 44.50 178.10 $2.96 25 

2006 175.30 92.50 267.80 $2.67 25 

2007 189.20 263.00 452.20 $2.11 25 

2008 379.90 642.70 1,022.60 $1.92 25 

Source: O’Conner et al. 2010. Informational sources: O’Conner et al. 2010; Christensen (1985); PV News (Maycock, 1986–2004; PV News, 

2005–2009); EIA and IEA (EIA, 2008; IEA, 2009); Friedman et al., 2005; Green (2005).
112 

◙ Table II.10-2 Illustration of Summary Accomplishments under FSA's Process 

Development Area (drawn from the evaluation of EERE's Solar Portfolio)

Surface Preparation Metallization

Technological and economic feasibility studies of 

automated surface preparation

Test patterns for process development and monitoring 

tools

Industry-standard texturizing process 

Spin drying

Silicon nitride as a multipurpose cell coating 

Thick-film screenable cost-effective processes using Ag, 

AgAI, Cu, and MOD AgBi

Reliable plating systems using Pd and Ni followed by 

solder build-up by immersion or Cu plating to provide 

conductivity

MOD films for low-temperature contact systems 

Generic fabrication systems for MOD films

Junction Formation Module Fabrication

Large-area, large-volume gaseous diffusion processes 

Spin-on, spray-on, and meniscus coating processes 

Simultaneous front and back junction-forming processes 

using liquid dopant and RTP

NMA ion implementation of front and back junctions 

Fully automated interconnect soldering equipment

Fully automated ultrasonic bonding equipment

Source: Gallagher et al. (1986), as referenced by O'Connor et al. (2010). 

Source: Link et al. (2013). 

◙ Figure II.10 Market Context Illustration: Number of Electric-Drive 

Vehicles Sold in the United States by Battery Technology, 1999-2012 

 -

 50,000

 100,000

 150,000

 200,000

 250,000

 300,000

 350,000

 400,000

 450,000

 500,000

1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012

Li-ion PHV/EV

Li-ion HEV

NiMH HEV Light-duty Trucks

NiMH HEV Cars
113 

10.B Assess EERE's program 

strategies66

Consider what strategies the program has 

used, and which of the program strategies 

assessed have been more, or less, successful 

in achieving stated goals. Identify and 

discuss any problems or remaining barriers 

that the study has uncovered. This is to be 

done in the spirit of sharing "lessons 

learned," with the objective of strengthening 

future EERE program strategies and their 

implementation. The evaluator should 

address the following questions: 

x What strategies did the program pursue 

in achieving its portfolio objectives? 

x Was there an observable difference in 

the effectiveness of strategies? 

x What strategies appear to have worked 

well? 

x What strategies appear to have worked 

less well? 

x What barriers were encountered? 

x Which barriers were overcome, and 

which were not? 

x If possible, explain what accounted for 

different levels of success in achieving 

positive economic returns? 

x What lessons can be learned from the 

R&D effort evaluated that can be 

translated into actionable 

 

66 This formative analysis represents a new assignment in 

this edition of the Guide. In adding this task, EERE 

recognizes that in-depth formative evaluation is normally 

performed during the course of a program's 

implementation, and not as part of a long-term 

retrospective benefit-cost (or impact) study, such as that 

covered by this Guide. It is also recognized that conducting 

formative evaluation requires different experience and a 

different skill-set than conducting summative (or impact) 

evaluation requires. Moreover, impact evaluators may not 

necessarily be experts in R&D management or market 

analysis―expertise that may be needed to address some of 

the questions presented. The recommendation is for the 

evaluator to obtain market expertise as needed to conduct 

the evaluation as set forth by the Guide, and to share 

lessons learned during the course of conducting the 

evaluation study in a "best effort" mode. 

recommendations for future similar 

EERE R&D investments? 

This assessment may draw on interview 

data, it may present anecdotal evidence, and 

in other ways it may go beyond the 

presentation of formal analysis and 

numerical results. EERE program staff, 

industry participants and nonparticipants, 

experts in other organizations such as 

associations, and others may be asked their 

opinions about what worked well and what 

did not. They can be asked what they would 

have changed if they had it to do over again. 

Where there has been limited uptake of the 

technology by the market, potential adopters 

can be asked why they did not, and what 

they think it would have taken (or would 

take) to make the technology commercially 

successful. Were the remaining obstacles 

mainly technical problems? Were there 

prerequisite developments that did not 

happen as expected? Did competing 

technologies win out? Were there other 

barriers? 

In short, the evaluator is expected to conduct 

investigative inquiries throughout the 

evaluation study to learn more about the 

technical and market research environment 

in which the EERE investment was made, 

the strategies pursued by EERE to achieve 

its goals, outcomes and impacts, and 

possible reasons why the subject 

technologies had or had not achieved 

economic success. 

10.C Provide actionable 

recommendations where 

possible 

There are a number of factors that could 

help explain why the technologies in the 

portfolio examined have or have not 

achieved technical and economic success. 

Based on the preceding assessment and 

resulting insights and lessons learned, the 
114 

evaluator is asked to list and discuss 

actionable recommendations. The objective 

is to provide feedback that can improve 

future similar program investments. 

Where possible, provide recommendations, 

using the following questions listed by 

category for organizing recommendations. 

x R&D goals, scope, strategies, and 

program implementation plan: 

o Were long-term goals of the R&D 

effort clearly defined and realistic? 

o Were goals in line with the broader 

research environment and national 

priorities? 

o Was the scope neither too narrow 

nor too broad? 

o Did the R&D effort include 

practical strategies designed to 

achieve targeted technical and 

market goals? 

o Was there adequate planning for 

program implementation? 

o To what extent was there 

successful execution of program 

strategies and activities? 

x Resources/inputs: 

o Was funding sufficient to achieve 

goals? 

o Was the EERE funding provided in 

a timely way? 

o Were participants in the R&D 

effort successful in attracting thirdparty funding? 

o Was there sufficient time for 

achieving both technical and 

commercial goals? 

o Were challenges encountered in 

resource availability or timing? 

o Were any funding gaps identified 

and filled/ not filled? 

x Collaboration: 

o Did it seem there was the right mix 

of research teams, private 

companies, private and 

government laboratories, 

universities, research associations, 

other parts of DOE, and customers 

involved in the R&D effort? 

o Was the research environment 

conducive to excellent work and 

goal achievement? 

o Were there any key elements 

missing? 

o If the collaboration was missing 

key elements, were remedies 

applied? 

x Alignment of technology and markets 

characteristics: 

o Were technical challenges 

impeding progress adequately 

defined? 

o Were technical goals achieved? 

o Were technology achievements 

incremental or radical? 

o Was the marketing approach 

aligned with technology 

characteristics? 

o Were there prerequisite or 

supporting technology elements, 

and, if so, were they 

adequately addressed? 

o Was the technology sufficiently 

superior to its alternatives over an 

extended period to give it a clear 

market advantage? 

o Does the technology have potential 

for use in widespread and/or 

multiple applications? 

o Was a path to market envisioned 

that was suitable for characteristics 

of the technology and the 

innovators? 

o Was market demand present, 

accessible, and growing, or not? 
115 

o Did the R&D effort include 

deployment support? Was it 

effective? 

o Was speed to market critical to 

success? 

o Did the R&D effort accelerate 

technology development and 

commercialization? 

o Has a domestic industry emerged? 

o Has a robust domestic supply chain 

emerged? 

o Is production moving offshore? 

o What additional technical and 

market bottlenecks or barriers were 

encountered and were they 

successfully overcome? 

x Agility of the program and its 

partners to respond to change: 

o Did the program modify its course 

of action in response to a specific 

need? 

o Did program partners modify their 

courses of action as needed? 

o Were there any external factors 

that threatened to derail program 

goals and strategies, and, if so, 

what was the program’s response?
116 

II.11 Report Results 

11.A Meet the stated standards 

of the report for content, format, 

presentation, transparency, and 

credibility 

The evaluator is expected to deliver a highquality study report that meets the following 

standards: 

 Effective tool for communicating with 

diverse program stakeholders to convey 

the study's method, findings, and 

implications; 

 Highly consistent with other benefitcost study reports performed according 

to the Guide and free of differences that 

are not necessitated by (1) changes in 

the Guide, (2) unique study 

requirements, and (3) data availability 

issues; 

 Transparent in use of input data, 

assumptions, and calculations, allowing 

replication and verification of benefits 

and economic performance measures; 

 Documents all data, assumptions, and 

calculations, and provides complete 

references to data and other sources. 

Presents summative evaluation contents 

including economic returns for up to 

three levels of analysis as needed; 

additional impacts including those 

reported using physical measures and 

qualitative descriptions; a separately 

reported extended analysis based on 

remaining system EUL; and sensitivity 

analyses to reflect uncertainties; and 

 Presents formative evaluation contents 

consisting of evaluator's lessons learned 

and recommendations, derived from the 

evaluation and enhanced by conducting 

investigative inquiries to learn more 

about the environment in which the 

EERE investment was made, the 

strategies pursued by EERE to achieve 

its goals, and possible reasons why 

subject technologies in the portfolio had 

achieved economic success. 

A feature of the body of the report and its 

executive summary is the inclusion of 

economic returns for up to three levels of 

analysis, as described in Step 7, and relisted 

below: 

Level 1: economic returns computed by 

comparing the monetary value of energy and 

other resource benefits against portfolio 

investment cost. 

11. Report 

Results

 Meet the stated standards of the report for content, format, 

presentation, transparency, and credibility. 

 Present summary results in a uniform tabular format. 

 Apply checklist. 

11. Report 

Results
117 

Level 2: economic returns computed by 

comparing combined monetary value of 

energy and other resource impacts and 

environmental health benefits against 

portfolio investment cost. 

Level 3: economic returns computed by 

comparing combined monetary value of 

energy and other resource impacts, 

environmental health benefits, and 

environmental benefits from reduction in 

GHGs against portfolio investment cost. 

While three levels of analysis may not be 

needed for every evaluation study and 

report, evaluators should ensure that the 

appropriate level or levels have been 

provided. The number of levels needed 

depends on the benefits categories impacted 

by a given portfolio. Monetized benefits 

come from energy and other resource 

benefits, environmental health benefits, and 

environmental benefits from reduction in 

GHGs. The other two benefits 

categories―energy security benefits and 

knowledge benefits—are not valued in 

dollars, and, hence, do not affect the 

computation of economic returns. It is 

possible that a given technology will not 

generate environmental effects, such that 

only a Level 1 analysis will be needed.67

The report must be explicit in laying out the 

following contents concerning the 

evaluator’s actual implementation of study 

method and data collection: 

 

67 For example, the evaluation of a portfolio of geothermal 

technologies included PDC drill bits, for which the effect 

was measured in terms of non- energy costs. (Higher rates 

of penetration and reduced rental time for drill rigs.) Hence, 

for that technology evaluation, there were no environmental 

impacts, and a Level I analysis sufficed for that part of the 

portfolio evaluation. However, the overall geothermal R&D 

portfolio contained other technologies that did generate 

environmental impacts. 

x Research questions 

o What are the specific researchable 

questions for the evaluation? 

x Research design 

o What specific design is being used to 

answer the research questions and 

why is that the best design? Would it 

be sufficient for answering the 

questions of the evaluation? 

x Study population 

o Who will participate in the study? 

How are they selected? What 

sampling approaches are to be used? 

x Data collection 

o How would the data be collected 

(e.g., through survey, in-depth 

interviews, direct measurement and 

verification, focus groups, etc.)? 

o What instrument would be used for 

data collection? 

o Has the instrument been tested for 

validity and reliability? For example, 

is a measuring instrument capturing 

what it is supposed to capture? Or if a 

survey is being used, are the questions 

valid, properly phrased, properly 

sequenced, with appropriate skip 

patterns, etc.? 

x Analysis 

o As described above, what levels of 

analyses are being done for the study? 

Provide a systematic presentation of 

the levels of analysis. 

x Limitations 

o What are the design, validity threats, 

execution, and analysis issues that 

could not be overcome? Be 

transparent about the limitations of 

the study and provide appropriate 

caveats to the presentation of results. 

11.B Present summary results 

in a uniform tabular format 

Evaluators are asked to provide summary 

results in a uniform tabular format for easy 

accessing of impacts by readers. The tabular 
118 

format presents summary results for overall 

economic performance, as well as a number 

of impacts stated in physical units rather 

than valued in dollars, and some described 

in words. The table, as shown presents the 

economic performance measures for up to 

three levels of analysis. It delineates 

between the purely retrospective benefits 

and the extension of benefits to include 

remaining EULs of retrospectively 

purchased/installed systems. A uniform 

tabular format is shown in Table II.11-1. At 

the discretion of the evaluator, the material 

in Table II.11-1 may be presented as a series 

of smaller tables. 

11.C Apply checklists 

Before submitting the draft report to EERE, 

the evaluator should cross-check the report 

against Tables II.10-2 through II.10-5, and 

rectify any deficiencies and missing parts. 

Table II.11-2 provides a contents checklist; 

Table II.11-3, a format checklist; Table 

II.11-4, a presentation checklist; and Table 

II.11-5, a process checklist. 

The checklists are for use by: evaluators and 

DOE project managers on an on-going basis 

to keep the report development on track. 

Upon receiving the submitted draft report, 

the DOE evaluation project manager should 

compare the report against the four tabular 

checklists to verify that key features are 

present. This should be done before the draft 

report is sent to internal and external 

reviewers. 

Table II.11-1 Uniform Tabular Format for Reporting Summary Evaluation Results from 

EERE Portfolio Investment 

Metric 

Unit of 

Measure 

Retrospective Analysis through Year X 

Extended Analysis 

through Year Y (to 

reflect remaining 

EUL in the highest 

level of analysis) 

Level 1 

Energy 

and other 

resource 

benefits 

Level 2 

Combined 

Level 1 & 

environmental 

health 

benefits 

Level 3 

Combined 

Level 2 & 

environmental GHG 

reductions 

Economic Impacts 

Overall Economic Performance Resulting from EERE's Investments 

Portfolio Investment Cost 

(undiscounted) 

Million, 

constant $ 

 

Gross Benefits (undiscounted) Million, 

constant $ 

 

Net Benefits (undiscounted) Million, 

constant $ 

 

Net present value @ 7% Million, 

constant $ 

 

(Continued) 
119 

Net present value @ 3% Million, 

constant $ 

 

Benefit-to-cost ratio @7% Ratio 

Benefit-to-cost ratio @3% Ratio 

Internal rate of return % 

Monetary Value of Energy and Other Resource Impacts, and of Environmental Impacts Resulting from EERE's 

Investments 

Monetary value of energy and other 

resource impacts 

Million, 

constant $ 

Monetary value of greenhouse gas 

emissions reduction 

Million, 

constant $ 

Monetary value of avoided adverse 

health incidence due to air 

emissions 

Million, 

constant $ 

Energy and Other Resource Impacts, in Physical Units, Resulting from EERE’s Investments 

 

Energy Impacts (saved or installed/ 

generated) 

Btu, type of fuel 

aved, MW, kWh, 

gallons 

Other Resource Impacts (e.g., 

Changes in land resource use) 

Relevant units 

Environmental Impacts in Physical Units Resulting from EERE’s Investments 

Avoided Air Emissions 

Avoided greenhouse gas emissions 

in CO2e (carbon dioxide emissions 

(CO2) or equivalents – e.g., 

methane (CH4), and nitrous oxide 

(N2O)) 

Metric tons 

(MMTCO2e) 

Avoided nitrogen oxides (NOx) Short tons 

Avoided particulate matter 

emissions (PM2.5) 

Short tons 

Avoided sulfur dioxide emissions 

(SO2) 

Short tons 

Carbon monoxide (CO) Short tons 

(Continued) 
120 

Volatile organic compounds (VOC) Short tons 

Ammonia (NH3) Short tons 

Changes in water consumptions 

and discharges; and solid waste 

generation 

Relevant units 

Health Impacts in Physical Units 

Avoided mortality Adult Deaths 

Avoided infant mortality Infant Deaths 

Avoided nonfatal heart attacks Attacks 

Avoided resp. hospital admissions. Admissions 

Avoided CDV hospital admissions Admissions 

Avoided acute bronchitis Cases 

Avoided upper respiratory 

symptoms Episodes 

Avoided lower respiratory 

symptoms Episodes 

Avoided asthma ER visits Visits 

Avoided MRAD Incidences 

Avoided work loss days Days 

Avoided asthma exacerbations Episodes 

Energy Security Impacts Resulting from EERE’s Investments 

Avoided petroleum consumption 

Gallons of 

gasoline 

equivalent 

Avoided foreign petroleum 

consumption 

Barrels of 

imported oil 

Reduced vulnerability of U.S. 

energy infrastructure Qualitative 

Knowledge Created and Disseminated Resulting from EERE's Investments

EERE-attributed patents issued Number of 

patents 

Patent citations Citation rates, 

Citation Index 

(CI) Value 

(Continued) 
121 

DOE patent citation rank among 

organizations in the field Rank 

Optional Knowledge Measures 

Knowledge spillovers 

Linkages to other 

technologies, 

industries, and 

organizations 

outside the 

targeted areas 

EERE-attributed publications Number of 

publications 

EERE publications most cited by 

other publications 

Listing 

Publication citations by patents Citation rates of 

publications by 

patents 

Technology Acceleration (as applicable) Resulting from EERE’s Investments 

Acceleration effect Number of years

Table II.11-2 Contents Checklist 

Content topic Check mark, 

if completed 

 Benefit-cost framework □

 Portfolio approach □

 Definition of portfolio and explanation of why it was selected □

 Description of EERE role and rationale, goals, strategies, and activities in the 

context of a logic model for the portfolio 

□

 Identification of individual technologies selected from the portfolio for in-depth 

analysis and the reason for their selection 

□

 Characterization of the technologies selected for in-depth analysis and of the 

remainder of the portfolio as well 

□

 Characterization of the technological and market contexts within which EERE's 

investments are made 

□

(Continued) 
122 

 Timeline of relevant developments □

 Portfolio investment cost table showing year-by-year expenditures in current 

and constant dollars 

□

 Four categories of benefits presented (as applicable), energy and other resource 

impacts, environmental health benefits, GHG reduction benefits, energy 

security benefits, and knowledge benefits 

□

 Impact measures stated in constant dollars, physical units, or qualitatively, as 

required by Steps 1-6 of the Guide 

□

 Up to 3 levels of retrospective analysis (Level 1, Level 2, Level 3) □

 Separate extended evaluation taking into account remaining EUL benefits in 

each level of analysis 

□

 Measures of social return on EERE investment, including NB, NPV, BCR, and 

IRR 

□

 Discounting using rates of 0%, 3%, and 7% in calculating PVs, NPVs and 

BCRs and in comparing against IRR, with 7% results designated as primary for 

testing economic performance 

□

 Appropriate designation of next-best alternative for comparing the selected 

technologies depending on whether they are product technologies, technology 

platforms, or infratechnologies 

□

 Most robust feasible approach used to assess additionality, with first preference 

to experimental and quasi-experimental designs, and second preference to 

nonexperimental design approaches using a counterfactual approach 

□

 In a non-RCT research design is employed, explicitly identify the major rival 

explanatory factors to be addressed in the analysis 

□

 Strengthening of nonexperimental approaches by avoiding biases in expert 

selection, by briefing experts on context and plausible rival explanations of 

outcomes, by using a Delphi-type approach to move closer to consensus 

opinion, by having experts complete a documented attribution matrix, and by 

using a systematic interview or survey approach to gather data from participants 

□

 Summary of evidence-based findings □

 Identification of uncertainties and sensitivity analysis □

(Continued) 
123 

 Identification of study limitations □

 Adherence to all standards and conventions for consistency given in the Guide □

 Qualitative treatment of parts of the portfolio not selected for detailed analysis □

 Use of valid protocols and procedures in data collection □

 Systematic and transparent analyses, with all data, assumptions, and 

calculations documented and presented, and results replicable 

□

 Explanation of why findings are conservative, lower-bound estimates □

 Inclusion of interview and survey tools in an appendix □

 Formative results and rationales □

 Lay out the following contents concerning the evaluator’s actual 

implementation of study method and data collection 

o Research questions 

o Research design 

o Study population 

o Data collection 

o Analysis 

o Limitations 

□
124 

Table II.11-3 Format Checklist 

Element Required Characteristics Check mark 

(if completed) 

Title Page x Title 

x Date 

x Prepared by 

□

□

□

Preface (if desired by 

DOE)

x DOE prepared description of mission, objectives, 

programs, rationale for public investment, and 

purposes of retrospective impact evaluation 

□

Acknowledgements x Contributors and reviewers, with separate listings of 

internal and external reviewers 

□

Notice x DOE prepared notice □

Executive Summary x Written for audiences of diversion backgrounds 

x Designed to communicate quickly and concisely the 

most important findings, and implications 

x Overall results summation 

x Specific inclusion of Uniform Tabular Format for 

Reporting Summary Results 

□

□

□

□

Table of Contents and 

lists of Tables and 

Figures

x 3-levels of headings for Table of Contents 

x Headings electronically keyed to report sections to 

facilitate easy movement by the reader through the 

report. 

□

□

Main Body of the 

Report x All elements of essential contents as outlined in Table 

II.11-2Overview of analysis method and data 

collection, describing actual implementation of study 

method and data collection 

x Separate sections on each of the 4 categories of impact 

that apply to a given evaluation study: 

o Energy and other resource impacts 

o Environmental health impact s and impact s from 

reductions in GHG emissions 

o Energy security impact s 

o Knowledge impact s 

x Separate section on extended analysis to take into 

account remaining EUL 

x Separate section on sensitivity analysis 

□

□

□

□

□

References x List of all references cited in the report (not a general 

reading list), with clear and concise documentation that 

allow locating all references 

□

(Continued) 
125 

Appendices/Attachments x Supporting information strongly germane to the 

presentation, but that can be moved out of the main 

body of the report for improved readability 

□

List of Terms, List of 

Abbreviations x After the list of figures in the front of the report □

Index x Discretionary □

Table II.11-4 Presentation Checklist 

Essential Characteristics Check mark 

(if completed) 

x Concise, clear, transparent exposition □

x Rigor demonstrated in data collection, analyses, and interpretation □

x Presentation accurate and reliable; free of errors of fact or logic □

x Findings objectively derived, testable, and reproducible □

x Credibility of study among stakeholders □

Table II.11-5 Process Checklist 

Peer review of study Check mark 

(if completed) 

x Evaluation plan, with emphasis on Steps 2 and 3, is peer reviewed internally 

by EERE staff and externally by at least three independent experts □

x Evaluator responds to major comments on evaluation plan, and prepares a 

summary of responses to provide to reviewers □

x Full draft report is peer reviewed internally by EERE staff and externally by at 

least three independent experts □

x Evaluator responds to major comments on draft report, and prepares a 

summary of responses to provide to reviewers □

 
126 

II.12 Document All Input Data, Assumptions, 

Calculations, and Results

12.A Facilitate data quality, 

transparency, and 

reproducibility of results by 

improving documentation 

The evaluator is contractually responsible 

for a study’s data quality, documentation, 

transparency, and the reproducibility of 

results. To allow an effective review, the 

data quality, documentation, transparency, 

and reproducibility requirements apply to 

both the final draft report and to the final 

report. In support of achieving this standard, 

the evaluator is required to deliver not just 

the report, but also data and analysis tables 

in spreadsheet, suitable for reviewer use in 

verifying that the results conform to 

standards and can be easily replicated. Thus, 

the final deliverables from the contractor 

include: 

 Evaluation study report, and 

 Data and analysis tables in spreadsheet. 

To emphasize the importance of this 

requirement, reports that do not comply will 

be returned to the contractor for additional 

work. If there are issues of proprietorship 

regarding data contained within the report or 

contractor spreadsheets, special 

arrangements will be made to protect data 

while enabling the review and checking of 

results. 

12.B Document data sources, 

assumptions, and calculations 

The evaluator should make the analysis 

process sufficiently transparent that those 

who read the report can understand the data 

sources, assumptions, and calculations. 

Table II.12-1, drawn from the evaluation 

report on EERE's investment in advanced 

engine combustion, illustrates that the 

simple use of table notes to document data 

and explain calculations. 

 Facilitate data quality, transparency, and reproducibility of results 

by improving documentation. 

 Document data sources, assumptions, and calculations. 

 Make it possible for a third party with a reasonable level of effort to 

replicate result. 

12. Document 

All Input Data, 

Assumptions, 

Calculations, 

and Results
127 

◙ Table II.12-1 Illustration of Documenting Data and Calculation 

Steps by the Use of Table Notes

(1) 

Year 

(2) 

VTP Budget 

(3) 

ACE R&D 

SubProgram 

Budget 

(4) 

CRF 

Budget* 

(5) 

GDP Implicit 

Price 

Deflator 

(2005=100) 

(6) 

GDP Implicit 

Price 

Deflator 

(2008=100) 

(7) 

InflationAdjusted ACE 

R&D SubProgram 

Budget

($2008)

(8) 

InflationAdjusted 

CRF Budget 

($2008) 

1976 $12.540 35.489 32.714 

1977 $28.425 37.751 34.799 

1978 $63.798** 40.400 37.241 

1979 $99.170 43.761 40.339 

1980 $110.500 47.751 44.017 

1981 $105.050 52.225 48.141 

1982 $58.944 55.412 51.079 

1983 $53.856 57.603 53.099 

1984 $64.900 59.766 55.093 

1985 $61.772 61.576 56.761 

1986 $57.457 $15.897*** $3.250 62.937 58.016 $27.402 $5.602 

1987 $55.393 $17.316*** $3.540 64.764 59.700 $29.005 $5.930 

1988 $51.360 $17.157*** $3.508* 66.988 61.750 $27.785 $5.680 

1989 $54.330 $16.998*** $3.475 69.518 64.082 $26.525 $5.423 

1990 $68.394 $17.257 $3.719 72.201 66.555 $25.929 $5.588 

1991 $83.564 $15.760 $4.300 74.760 68.914 $22.869 $6.240 

1992 $109.282 $16.657 $4.390 76.533 70.548 $23.611 $6.223 

1993 $138.632 $14.818 $4.379 78.224 72.107 $20.550 $6.073 

1994 $177.249 $12.949 $4.171 79.872 73.626 $17.587 $5.665 

1995 $191.065 $10.440 $4.171 81.536 75.160 $13.890 $5.549 

1996 $174.288 $16.524 $4.714* 83.088 76.591 $21.574 $6.154 

1997 $172.457 $19.263 $5.256 84.555 77.943 $24.714 $6.743 

1998 $189.972 $18.318 $5.161 85.511 78.824 $23.239 $6.547 

1999 $198.665 $36.976 $5.024 86.768 79.983 $46.230 $6.281 

2000 $228.756 $46.750 $4.736 88.647 81.715 $57.211 $5.796 

2001 $251.462 $52.205 $5.463 90.650 83.561 $62.475 $6.538 

2002 $181.352 $47.160 $5.377 92.118 84.915 $55.538 $6.332 

2003 $174.171 $55.267 $5.935 94.100 86.742 $63.714 $6.842 

2004 $172.395 $52.736 $5.892 96.770 89.203 $59.119 $6.605 

2005 $161.326 $48.480 $6.437 100 92.180 $52.593 $6.983 

(Continued) 
128 

2006 $178.351 $40.594 $6.251 103.257 95.183 $42.649 $6.567 

2007 $183.580 $48.346 $7.648 106.214 97.908 $49.379 $7.811 

2008 $208.359 $43.443 $6.755 108.483 100 $43.443 $6.755 

Notes: 

When data are not available for a particular year/program, the cell is blank. 

* Two years of CRF construction began in 1978, with early years of operation beginning in 1980 through 1985. The complete 

funding information for those years is unknown. 

** denotes values that were constructed as the average for the juxtaposed years. 

** * denotes values that were constructed on the basis of the average ratio of the CRF budget to the ACE R&D sub-program 

budget for all available years. 

Column (4) represents DOE Office of Science funding for cross-cutting research programs that are related to combustion and that

are within the CRF. 

Column (6) = Column (5) / (108.483 / 100). 

Column (7) = Column (3) / (Column (6) / 100). 

Column (8) = Column (4) / (Column (6) / 100). 

Year 2008 shown to benchmark the GDP deflator in Column (6). 

Sources: 

Nominal budget data in Columns (2) – (4) provided by EERE. 

GDP Implicit Price Deflator (2005=100) from U.S. DoC (2009). 

Another example from a previous EERE 

retrospective impact evaluation—this one 

for EERE solar photovoltaic portfolio—

shows the use of supporting evidence to 

back up a study's assumption. The study 

makes the case that companies substantially 

increased photovoltaic module warranties 

over the evaluation period as module 

performance improved. Warranty extension 

is illustrated in Figure II.12-1. Also, note 

that, as in the preceding example, the data 

sources are given, and detailed references 

are included elsewhere in the report for use 

in locating the source documents. 

◙ Figure II.12-1 Illustration of the Use of Supporting Timelines to Support 

Underlying Assumptions (drawn by O'Connor et al., 2010, Fig. 5-1)
129 

12.C Make it possible for a third 

party with a reasonable level of 

effort to replicate results 

In addition to the published final study 

report, evaluators are expected to provide 

EERE tables containing input data and 

calculations electronically in spreadsheet. 

These data tables should be suitable for 

review by reviewers to estimate all benefits 

(including intermediate impacts) and to 

calculate economic performance measures. 

Data input into the spreadsheet from an 

external source should be labeled as such 

and the external source should be clearly 

documented. 

Also, the analysis must be fully replicable 

from tables, figures, formulae, and text 

presented in the report, and readers will 

expect to be able to follow all calculations. 

 
130 

II.13 Use Study Findings to Inform EERE 

Decision Making and for Effective 

Communications 

12. Us 

Stud Findings to Inform Decision Making for Effective Communications 

13.A Develop a strategy for 

using study findings in 

communications with 

stakeholders 

In collaboration with evaluators, EERE 

program staff is encouraged to develop and 

implement a plan for using its studies to 

improve communications with stakeholders. 

The objective is to get the most from the 

impact evaluations. Related activities may 

include the following: 

 Prepare a one-to-two page "highlights" 

handout (e.g., an Executive Brief) that 

provides an overview of the study 

findings, suitable for distributing both 

inside and outside DOE. 

 Hold a seminar with the Program Office 

and other EERE staff to allow evaluators 

to present the study's impact results firsthand, and also to discuss the 

implications of the formative analysis. 

 Prepare briefing points for EERE 

Management for use in presentations. 

13.B Apply relevant study 

findings to inform EERE 

decisions 

EERE program staff should consider how a 

given retrospective impact study informs 

EERE decisions, referring both to the 

summative results and the additional 

formative analysis provided by Step 10. The 

staff is encouraged to look beyond the 

evaluator's deductions to see if other 

inferences or insights that inform future 

investments can be drawn from a study, 

based on their first-hand perspectives. 

 

 Develop a strategy for using study findings in communications with 

stakeholders. 

 Apply relevant study findings to inform EERE decisions. 

13. Use Study 

Findings to 

Inform Decision 

Making and for 

Effective 

Communications
131 

II.14 Implement Routine Tracking and 

Compilation of Data to Support Future Study 

Updates3

Data compilation by EERE programs on a 

routine basis is a cost-effective way to 

support a program's monitoring and 

evaluation effort. Without this support, 

evaluators must later attempt to recreate 

information that would have been more 

straight-forward and more reliable to collect 

by the EERE in real time. Thus, the lack of 

routine data collection adds to the time and 

cost of generating performance metrics and 

evaluation impact results, while increasing 

the uncertainty of results. The following are 

recommendations for EERE for (1) a 

minimum level of EERE program data 

collection, and (2) extended data collection. 

14.A Implement routine data 

collection in support of 

evaluation 

EERE program staff are expected to collect 

certain data via the grantee reporting forms. 

These may include data amenable to routine 

collection via Funding Opportunity 

Announcement (FOA) reporting 

requirements. Data amenable to routine 

collection should be collected even before 

an evaluation is initiated, so it will be 

available when needed. For certain data (see 

Table II.14-1) this may already be part of 

the routine collection supported by EERE’s 

project data collection systems. In addition 

to routine data collection by EERE staff, it is 

expected that the evaluator will also be 

responsible collecting other data needed for 

a given evaluation study. The evaluator may 

collect primary data via surveys and 

interviews, and secondary data from other 

databases, published reports, or by other 

collection mechanisms. 

After an evaluation is concluded, EERE 

program staff should consult with the 

evaluator to determine if any new data, not 

previously identified (see below) is 

recommended for routine collection by 

EERE for future evaluations of the portfolio 

or program. 

 Implement routine data collection in support of evaluation. 

 Maintain records of information that could be used to support future 

evaluations. 

 Consider additional data collection 

14. Implement 

Routine 

Tracking and 

Compilation of 

Data to Support 

Future Study 

Updates
132 

14.B Maintain records of 

information that could be used 

to support future evaluations 

EERE programs and subprograms at a 

minimum should maintain the following 

records of information in a format readily 

available for internal data reporting and for 

transfer to outside evaluators who are 

engaged to conduct impact studies:68

x Annual expenditures on main program 

activities and major technology 

categories. These data will be helpful in 

defining portfolios for evaluation and are 

necessary for calculating ROI metrics. 

The data should be available for 

provision to evaluators near the outset of 

a study. 

x Records of EERE-funded program 

partnerships, including identities of the 

funded companies and other 

participating organizations by name, 

address, and contact information; years 

of funding and yearly funding amount; 

main partnership objectives and 

accomplishments; and list of outputs 

resulting from EERE's support, such as 

publications, patents filed and granted, 

patents licensed, products under 

development including research and 

production prototypes, sales and sales 

revenue (if available), and jobs created 

or retained as a result of EERE's support; 

achievement of larger technical and 

market goals; and identification of issues 

and outlook. 

x Records of EERE-funded university 

programs that identify the funded 

universities by name, branch, address, 

and contact information; give the yearly 

funding amount and identify main 

 

68 Some of the recommended data collection will be done 

as part of EERE FOA reporting requirements starting in 

2015. 

objectives of funded programs; and list 

outputs including publications, patents 

filed and granted, patents licensed, and 

spin-out companies. 

x Records of in-house EERE research 

outputs, including publications, patents 

filed and granted, patents licensed, and 

other relevant outputs, including those of 

DOE laboratories managed by other 

organizations. 

x Cost and use records for special DOE 

facilities closely related to major EERE 

technology development efforts. 

x Data on other relevant DOE 

collaborative activities with other 

government agencies, industrygovernment consortia, industry 

associations, and other organizations. 

In addition to providing needed data for 

evaluators, these data would allow the 

EERE programs consistently to provide 

progress metrics and monitor early 

outcomes. 

Another area that may merit routine data 

compilation is the management of research 

grant applications, review, and award 

administration. Application and award 

ranking information for grantees and nongrantees might make it possible to perform a 

R&D impact evaluation that uses a quasiexperimental research design ― by 

establishing a comparison group. One 

example of when a quasi-experimental 

design could be performed in the R&D 

evaluation context, is when it is feasible to 

identify firms who receive DOE R&D 

funding awards as well as firms who applied 

but were rejected because funding was not 

available for them (i.e., they were nearly as 

qualified in their proposed research as those 

who were awarded funds, but just missed the 

cutoff). Those just missing the award cutoff 

could comprise a comparison group to 

enable a quasi-experimental design to 
133 

proceed as the design method, following the 

principles delineated in Section 2B.1-4. 

Patents and publications attributed to EERE 

programs could also be routinely collected 

in EERE annual project progress report or 

national laboratory reports. It should be 

noted that all EERE-attributed publications 

and patents at the time of an evaluation 

study may not have yet been entered into 

existing agency-wide databases, such as the 

Office of Science and Technology 

Information (OSTI) database. Furthermore, 

it may be difficult to determine which 

publications and patents in a database are 

attributable to a specific EERE portfolio 

without special notation. In addition, some 

patents may be missed in searches of 

national and international patent databases 

because organizations funded by EERE do 

not always give credit in the patent section 

entitled “Government Interest,” or, if they 

do, it may be given at a level that is not 

identifiable as having been funded by a 

specific EERE program. When these data 

are compiled at the program or subprogram 

level it is easier to link outputs to funding by 

specific EERE programs and portfolios 

selected for evaluation―particularly data for 

companies and other organizations which 

may have filed patents or published 

following direct interaction with EERE. 

14.C Consider additional data 

collection 

The following types of data are also 

recommended in support of evaluation: 

Baseline Documentation: Impact 

evaluations generally require baselines 

against which to assess change. However, as 

indicated in Step 2, defining a baseline is 

often more complex than simply 

establishing performance data just prior to 

the EERE investment. The next-best 

alternative may change over time, such that 

dynamic modeling is needed. Moreover, the 

alternative may best be represented as a 

counterfactual scenario. 

Retrospective impact evaluations are 

generally conducted some years after a 

program activity begins. Evaluators are 

often trying to reconstruct a baseline or 

alternative scenario some years after the 

fact. Program efforts to document conditions 

just prior to or near the start of an EERE's 

program will be helpful to later impact 

evaluations. 

Early and Intermediate Outcomes Data: 

These data generally come from those 

outside the program and often with a 

considerable lag after program funding. 

Unless funding arrangements include a 

requirement that funded companies agree to 

participate in routine surveys and interviews 

or other reporting arrangements, it may 

prove difficult to obtain post-project early 

and intermediate outcomes data. Thus, it is 

recommended, if feasible and allowable by 

DOE and OMB, to include routine data 

reporting requirements in funding 

partnerships with others. The program may 

arrange for web-based or other survey 

approach to assess post-project 

developments, or it may interview firms to 

obtain post-project data. This effort would 

focus on an on-going collection of output 

data from funded firms, such as data on 

patents granted, formation of spin-out 

companies, jobs created and retained, and 

product development and sales. 

Compilation of, or Linkages to, Relevant 

Industry and Market Data: Although 

specific data needed to conduct impact 

studies will vary by technology and study 

approach, the need for industry and market 

data is highly likely. A program may 

compile industry or market data, or obtain 

data from other sources, such as by linking 
134 

to other databases of industry associations, 

or collecting relevant market studies 

provided commercially. 

In-house data collection is invaluable to 

evaluation, but is difficult or impossible to 

anticipate all data that will be required to 

conduct a retrospective impact evaluation, 

such as is the subject of this Guide. 

Knowing the key variables for calculating 

retrospectively, for example, the fuel 

savings attributable to a new laser research 

tool to be used in advanced combustion 

engine technology would likely have been 

difficult at the outset of the R&D. It took ex 

post interviews of an experienced evaluator 

working with EERE experts to figure out a 

measurement strategy, followed by 

econometric modeling to work from 

available data to the data needed for impact 

measurement. Further, it was not known in 

advance that this research tool would 

become part of an EERE portfolio slated for 

evaluation, and it was not known that it 

would be selected for detailed analysis. This 

observation is made simply to recognize that 

there generally are limits to in-house data 

collection in support of impact evaluations. 

Table II.14-1 is provided in support of 

EERE's on-going effort to develop an 

effective data collection infrastructure. 
135 

Table II.14-1 Data to Support R&D Portfolio Impact Evaluations 

Cost data

x Total yearly program budget 

x Subprogram level budget 

x Key activity budget 

x Breakdown by specific technology areas (e.g., for VTP energy storage R&D study the breakdown might be by 

NiMH, Li-ion research, other) 

Sponsored-partner data

x Sponsored performing organization, amount of funding, dates of funding 

x Organization lead's name 

x Organization location 

x Organization lead's contact information 

x FOA number 

x Partner cost share amount 

x Partner cost share type (cash, in-kind) 

x Type of funding award (e.g., Cooperative agreement, contract, interagency, grant, other) 

Project data 

x Project title 

x Statement of project objectives 

x Project description 

x Project location 

x Project's performing organization 

x Project Officer name and contact information 

x DOE HQ project technical manager 

x DOE HQ project technical manager contact information 

x Total project funding by year 

x Cumulative project spending over project life up to current year 

x Project start date 

x Project actual completion date 

x Project planned completion date 

x Project milestone titles 

x Project milestones' completion status by title 

x Project cancelation/ termination date and cause 

x Project's technology focus area(s) (e.g., for vehicle energy storage areas might be Li-ion, NiMH) 

x Project's primary research area focus (e.g., for VTP energy storage, is funded R&D focused on Batteries – High 

Energy, improved power, Improved Life and Abuse Tolerance, Ultracapacitors, materials, or New Tools and 

Techniques) 

x RDDD phase 

x Technology Readiness Level (TRL) status 

(Continued) 
136 

Relational data 

x Role in the value chain (e.g., supplier, manufacturer, system integrator, distributor, service provider, end user, R&D 

organization, financier, multiple roles, other) 

x Addition of any new suppliers as part of a DOE grant? 

Intellectual property data 

x Title of patent awarded for EERE-attributed research in a specified technology area 

x Patent number 

x Patent date 

x Patent assignee 

x Licensing agreement title 

x Licensing agreement date 

x Licensing agreement assignees 

x Title of publications of EERE-attributed research in a specified technology area 

x Publication date 

x Publication authors 

Technical achievements data 

x Current status of technology development 

x Technology development goals/ planned technology development 

x Other research accomplishments 

x Technology development milestones 

x Technology durability/ reliability 

x Technology efficiency 

x Current Next-Best Alternative (NBA) technology 

x NBA technology development historical milestones 

x NBA technology cost 

x NBA technology durability/ reliability 

x NBA technology efficiency 

Self-reported outside awards, recognition, and achievements (reported by funded partners and others) 

* NOTE: Further analysis of data is required to assess EERE's contribution

x Name of commercialized technology 

x Year technology commercialized 

x Technology developer or co-developers 

x Other commercialized technology partners 

x Commercialized technology seller 

x Promulgated efficiency standards 

x Growth in Market Share 

x Sales data 

x Energy saved 

x Renewable Capacity Increase (e.g., MWs) 

x Licensing/Royalty Revenue 
137 

References 

American Evaluation Association, 2014. Guiding Principles for Evaluators. Accessed June 3, 

2014: http://www.eval.org/p/cm/ld/fid=51

Argonne National Laboratory (ANL). (2012). GREET life-cycle model. Accessed August 14, 

2014: http://greet.es.anl.gov/

Dockery, D. W., Cunningham, J., Damokosh, A. I., Neas, L. M., Spengler, J. D., Koutrakis, P., 

Speizer, F. E. (1996). “Health effects of acid aerosols on North American Children—

Respiratory symptoms.” Environmental Health Perspectives, 104(5), 500–505. 

Gallagher, G., Alexander, P, & Burger, D. (1986). Volume V: Process Development. Flat-Plate 

Solar Array Project Final Report. Project managed by the Jet Propulsion Laboratory for 

the U.S. Department of Energy. JPL Publication 86-31. Pasadena, CA: 

Gallaher, M.P, Link, A.N., and O’Connor, A. C. (2012). Public Investments in Energy 

Technology. London, Edward Elgar. 

Gallaher, M., Rogozhin, Alex, and Petrusa, J. (2010). Retrospective Benefit-Cost Evaluation of 

U.S. DOE Geothermal Technologies R&D Program Investment: Impact of a Cluster of 

Energy Technologies. U.S. Department of Energy, Energy Efficiency and Renewable 

Energy. 

Greene, David L., Leiby, Paul N. (2006). The Oil Security Metrics Model: A Tool for Evaluating 

the Prospective Oil Security Benefits of DOE’s Energy Efficiency and Renewable Energy 

R&D Programs. ORNL/TM-2006/505. Accessed August 14, 2014: 

http://cta.ornl.gov/cta/Publications/Reports/ORNL_TM_2006_505.pdf

Griliches, Z. (1957). “Hybrid Corn: An Exploration in the Economics of Technological Change,” 

 Econometrica, 25(4), 501-522. 

Griliches, Z. (1958). “Research Cost and Social Returns: Hybrid Corn and Related Innovations,” 

 Journal of Political Economy, LXVI (5), 419-431. 

Hicks, Diana, and Melkers, Julia. (2013). "Bibliometrics as a Tool for Research Evaluation." 

Handbook on the Theory and Practice of Program Evaluation, edited by Albert N. Link 

and Nicholas S. Vonortas. Edward Elgar Press. 

Interagency Working Group on Social Cost of Carbon, United States Government. (2010). 

Technical Support Document: Social Cost of Carbon for Regulatory Impact Analysis 

Under Executive Order 12866. 

Interagency Working Group on Social Cost of Carbon, United States Government. (2013). 

Technical Support Document: Technical Update of the Social Cost of Carbon for 

Regulatory Impact Analysis Under Executive Order 12866. 

Jordan, Gretchen, Mote, Jonathon, Ruegg, Rosalie, Choi, Thomas, and Becker-Dippmann, 

Angela. (2014). A Framework for Evaluating R&D Impacts and Supply Chain Dynamics 

Early in a Product/Technology Life Cycle. U.S. Department of Energy. 
138 

W.K. Kellogg Foundation. (2006). W.K. Kellogg Foundation Logic Model Development Guide.

Accessed December 2013: wkkf.org. 

Krewski, D., Jerrett, M., Burnett, R. T., Ma, R., Hughes, E., Shi, Y., Turner, M. C., Tempalski, 

B. (2009). “Extended follow-up and spatial analysis of the American Cancer Society 

study linking particulate air pollution and mortality,” Research Report of the Health 

Effects Institute, 140: 5–114; discussion 115-136. 

Laden, F., Schwartz, J., Speizer, F. E., & Dockery, D. W. (2006). “Reduction in fine particulate 

air pollution and mortality: Extended follow-up of the Harvard Six Cities Study,” 

American Journal of Respiratory and Critical Care Medicine, 173(6), 667–672. 

Leiby, Paul N. (2008). Estimating the Energy Security Benefits of Reduced U.S. Oil Imports.

ORNL/TM-2007/028. Accessed August 14, 2014: 

http://cfpub.epa.gov/si/si_public_file_download.cfm?p_download_id=504469. 

Linden, L., Bottaro, D., Moskowitz, J., & Ocasio, W. (1977). The Solar Photovoltaics Industry: 

The Status and Evolution of the Technology and the Institutions. MIT Energy Laboratory 

Report, MIT-EL-77-021. Prepared for the U.S. Department of Energy. 

Link, Albert N. (2009). An Economic Evaluation of Advanced Combustion Engine Technologies 

within the U.S. Department of Energy. U.S. Department of Energy, Office of Energy 

Efficiency and Renewable Energy. 

Link, A., Scott, T., O’Connor, A., Casey, S., Loomis, R., Davis, J. (2014). Retrospective BenefitCost Evaluation of U.S. DOE Investment in Energy Storage Technologies for Hybrid and 

Electric Cars and Trucks. Washington, D.C.: U.S. Department of Energy, Office of 

Energy Efficiency and Renewable Energy. 

Link, A. and J. Scott. (2011a). Public Goods, Public Gains. Oxford: Oxford University Press. 

Link, A. and J. Scott. (2011b). The Theory and Practice of Public Sector R&D Economic Impact 

Analysis. Planning Report 11-1. Gaithersburg, MD: National Institute of Standards & 

Technology. 

Link, A. and N. Vornatos (eds). (2013). Handbook on the Theory and Practice of Program 

Evaluation. Northampton, MA: Edward Elgar. 

Mansfield, Edwin. (1996). Estimating Social and Private Returns from Innovations Based on the 

Advanced Technology Program: Problems and Opportunities. National Institute of 

Standards and Technology, GCR 99-780. Accessed December 2013: 

www.atp.nist.gov/eao/gcr99-780/contents.htm

McLaughlin, John A. and Gretchen B. Jordan. (2010, 2nd ed.). "Using Logic Models." 

Handbook of Practical Program Evaluation, ed. by Joseph S. Wholey, Harry P. Hatry, 

and Kathryn E. Newcomer. John Wiley & Sons. 
139 

Moolgavkar, S. H. (2000). “Air pollution and hospital admissions for chronic obstructive 

pulmonary disease in three metropolitan areas in the United States,” Inhalation 

Toxicology, 12 (Supplement 4), 75–90. 

Moolgavkar, S. H. (2003). “Air pollution and daily deaths and hospital admissions in Los 

Angeles and Cook Counties,” In Revised analyses of time-series studies of air pollution 

and health (pp. 183–198). Boston, MA: Health Effects Institute. 

National Research Council (2010). Hidden Costs of Energy: Unpriced Consequences of Energy 

Production and Use. Washington, DC: The National Academies Press. 

National Research Council (2001). Energy Research at DOE: Was It worth It? Washington, DC: 

The National Academies Press. 

Navigant Consulting, Inc. (2013). Impact and Process Evaluation of the U.S. Department of 

Energy's Wind Powering America Initiative. 

O'Connor, A., Loomis, R., & Braun, F. (2010). Retrospective Benefit-Cost Evaluation of DOE 

Investment in Photovoltaic Energy Systems. Washington, D.C.: U.S. Department of 

Energy, Office of Energy Efficiency and Renewable Energy. 

OMB (2003) Circular A-4. Accessed August 14, 2014: 

http://www.whitehouse.gov/omb/circulars_a004_a-4 

OMB (1992) Circular A-94, Revised. Accessed August 14, 2014: 

http://www.whitehouse.gov/omb/circulars_a094 

Ostro, B. D., & Rothschild, S. (1989). “Air pollution and acute respiratory morbidity—An 

observational study of multiple pollutants.” Environmental Research, 50(2), 238–247. 

Ostro, B. D. (1987). “Air pollution and morbidity revisited: A specification test.” Journal of 

Environmental Economics and Management, 14, 87–98. 

Ostro, B., Lipsett, M., Mann, J., Braxton-Owens, H., & White, M. (2001). “Air pollution and 

exacerbation of asthma in African-American children in Los Angeles.” Epidemiology, 

12(2), 200–208. 

Peters, A., Dockery, D. W., Muller, J. E., & Mittleman, M. A. (2001). “Increased particulate air 

pollution and the triggering of myocardial infarction.” Circulation, 103(23), 2810–2815. 

Pope, C. A., Burnett, R. T., Thun, M. J., Calle, E. E., Krewski, D., Ito, K., & Thurston, G. D. 

(2002). "Lung cancer, cardiopulmonary mortality, and long-term exposure to fine 

particulate air pollution." JAMA, 287(9), 1132–1141. 

Powell, J. (2006). Toward a Standard Benefit-Cost Methodology for Publicly Funded Science 

and Technology Programs. NSTIR 7319. Gaithersburg, MD: National Institute of 

Standards & Technology Advanced Technology Program. 

Ruegg, Rosalie & Jordan, Gretchen. (2011). Guide for Conducting Benefit-Cost Evaluation of 

Realized Impacts of Public R&D Programs (Revised Working Draft). U.S. Department of 

Energy, Office of Energy Efficiency and Renewable Energy. 
140 

Ruegg, Rosalie & Jordan, Gretchen. (2007). Overview of Evaluation Methods for R&D 

Programs; A Directory of Evaluation Methods Relevant to Technology Development 

Programs. Washington, D.C.: U.S. Department of Energy, Office of Energy Efficiency 

and Renewable Energy. 

Ruegg, Rosalie & Feller, Irwin. (2003). A Toolkit for Evaluating Public R&D Investment. NIST 

GCR 02-842. Gaithersburg, MD: National Institute of Standards and Technology. 

Accessed August 14, 2014: http://www.atp.nist.gov/eao/eao_pubs.htm. 

Ruegg, Rosalie & Thomas, Patrick. (2013). "Patent Analysis." Handbook on the Theory and 

Practice of Program Evaluation, edited by Albert N. Link and Nicholas S. Vonortas. 

Edward Elgar Press. 

Ruegg, Rosalie & Thomas, Patrick. 2011. Linkages from DOE's Solar Photovoltaic R&D to 

Commercial Power from Solar Energy. Washington, DC: U.S. Department of Energy 

Efficiency and Renewable Energy. 

Schwartz, J., & Neas, L. M. (2000). “Fine particles are more strongly associated than coarse 

particles with acute respiratory health effects in schoolchildren.” Epidemiology, 11(1), 6–

10. 

Sheppard, L. (2003). “Ambient air pollution and nonelderly asthma hospital admissions in 

Seattle, Washington, 1987-1994.” Revised analyses of time-series studies of air pollution 

and health (pp. 227–230). Boston, MA: Health Effects Institute. 

Sheppard, L., Levy, D., Norris, G., Larson, T. V., & Koenig, J. Q. (1999). “Effects of ambient air 

pollution on nonelderly asthma hospital admissions in Seattle, Washington, 1987-1994.” 

Epidemiology, 10(1), 23–30. 

Tassey, G. (2003). Methods for Assessing the Economic Impacts of Government R&D. Planning 

Report 03-1. Gaithersburg, MD: National Institute of Standards & Technology. 

Tassey, G. (1997). The Economics of R&D Policy. Greenwood Publishing Group. 

U.S. Census Bureau. (2013). “Table 1. Monthly Population Estimates for the United States: April 

1, 2010 to June 1, 2013 (NA-EST2012-01-613).” July 2013. 

U.S. Census Bureau. (2011). “Table 1. Intercensal Estimates of the Resident Population by Sex 

and Age for the United States: April 1, 2000 to July 1, 2010 (US-EST00INT-01).” 

September 2011. 

U.S. Census Bureau. (2004). “1990-2000 Monthly Intercensal Estimates of the United States 

Resident Population by Age and Sex for Internet Display.” August 13, 2004. 

U.S. Census Bureau. (2000). “Historical National Population Estimates: July 1, 1900 to July 1, 

1999.” June 28, 2000. 

U.S. Department of Commerce, Bureau of Economic Analysis. (2013). “Table 7.1. Selected Per 

Capita Product and Income Series in Current and Chained Dollars.” June 26, 2013. 
141 

U.S. Energy Information Administration. (2013a.) “August 2013 Monthly Energy Review.” 

Accessed August 14, 2014: http://www.eia.gov/totalenergy/data/monthly/pdf/mer.pdf

U.S. Energy Information Administration. (2013b). “Frequently Asked Questions: How much 

coal, natural gas, or petroleum is used to generate a kilowatt-hour of electricity?” 

Accessed August 14, 2014: http://www.eia.gov/tools/faqs/faq.cfm?id=667&t=2 

U.S. Energy Information Administration. (2012a). Annual Energy Review 2011. Accessed 

August 14, 2014: http://www.eia.gov/totalenergy/data/annual/pdf/aer.pdf

U.S. Energy Information Administration. (2012b). “Net Generation by State by Type of Producer 

by Energy Source (EIA-906, EIA-920, and EIA-923). 1990-2011.” Accessed August 14, 

2014: http://www.eia.gov/electricity/data/state/

U.S. Environmental Protection Agency. (2012a) BenMAP: Environmental Benefits Mapping and 

Analysis Program. User’s Manual. October 2012. Accessed August 2014: 

http://www.epa.gov/airquality/benmap/models/BenMAPManualOct2012.pdf

U.S. Environmental Protection Agency. (2012b. Greenhouse gas emissions. Accessed August 

2014: http://www.epa.gov/climatechange/ghgemissions/gases.html# 

U.S. Environmental Protection Agency. (2012c). User’s Manual for the Co-Benefits Risk 

Assessment (COBRA) Screening Model. Version: 2.4. October, 2012. 

U.S. Environmental Protection Agency. (2012d). What are the six common air pollutants. 

Accessed August 2014: http://www.epa.gov/air/urbanair/ 

U.S. Environmental Protection Agency. (2011). Regulatory Impact Analysis for the Final 

Mercury and Air Toxics Standards. EPA-452/R-11-011. Research Triangle Park, NC: 

Office of Air Quality Planning and Standards; Health and Environmental Impacts 

Division. Available at http://www.epa.gov/ttnecas1/regdata/RIAs/matsriafinal.pdf 
142 

Attachment 1

Drivers for Public Accountability

A number of directives and guidance memorandum from the Executive and Congressional 

branches set impact evaluation expectations for EERE programs. 

Executive Orders and OMB Memorandum: 

x OMB Memorandum to the Heads of Executive Departments and Agencies, July 26, 2013 (Memo 

M-13-17) ― next steps in the evidence and innovation agenda.

http://www.whitehouse.gov/sites/default/files/omb/memoranda/2013/m-13-17.pdf 

x OMB Memorandum to the Heads of Executive Departments and Agencies, May 18, 2012 

(Memo M-12-14) ― use of evidence and evaluation in the 2014 budget.

http://www.whitehouse.gov/sites/default/files/memoranda/2012/m-12-14.pdf 

x OMB Memorandum for the Heads of Executive Departments and Agencies, July 29, 2010 

(Memo M-10-32) and Oct. 7, 2009 (Memo M-10-01) ― increased emphasis on program 

evaluation in Federal Agencies. 

http://www.whitehouse.gov/sites/default/files/memoranda_2010/m10-01.pdf

x OMB Budget Action Request (Memo 10-49), July 29, 2010 – mandatory agency program 

evaluation inventory. http://www.whitehouse.gov/omb/memoranda_default

x OMB and White House Office of S&T Policy Memorandum for the Heads of Executive 

Departments and Agencies on Science and Technology Priorities for the FY 2011 Budget, 

August 2009; calls for R&D agencies to conduct evaluations and strengthen capacity.

http://www.whitehouse.gov/briefing-room/presidential-actions/presidential-memoranda

x OMB Performance Rating Assessment Tool (PART), 2003-2008; set expectations for periodic 

systematic evaluations to demonstrate results.

http://www.whitehouse.gov/omb/memoranda_m03-06/

x ARRA unprecedented requirements for transparency & accountability, 2009. 

http://www.recovery.gov/Pages/default.aspx
143 

x Executive Order 13450: Improving Government Program Performance, November 2007; 

agencies shall spend taxpayers’ dollars efficiently & effectively.

http://www.whitehouse.gov/sites/default/files/omb/assets/...pdfs/eo13450.pdf

Congress: 

x GPRA Modernization Act of 2010 – each agency shall make available on its public website 

an update on its performance. Agency strategic plans must include "...a description of the 

program evaluations used in establishing or revising general goals and objectives," and 

Agency performance reporting has to " include the summary findings of those program 

evaluations completed during the period covered by the update."

http://www.gpo.gov/fdsys/pkg/BILLS-111hr2142enr/pdf/BILLS-111hr2142enr.pdf

x House Committee Reports HEWD, 2008/2009/2010, calls for reporting on return on 

investment. 

http://science.energy.gov/~/media/budget/pdf/sc-congressional-appropriations/Fy-2012/Housebill/HEWD-FY12-Committee-Report---Final_SC_Only.pdf 

x Government Accountability Office. January 2012. Designing Evaluation; 2012 Revision. GAO12-208G. 

http://gaonet.gov/assets/590/588146.pdf 

x Government Accountability Office. November 2009. Program Evaluation: A Variety of 

Rigorous Methods Can Help Identify Effective Interventions. GAO-10-30. 

http://www.gao.gov/new.items/d1030.pdf 

x Congressional Research Service, CRS Report for Congress, March 7, 2006. Congress and 

Program Evaluation: An Overview of Randomized Controlled Trials (RCTs) and Related 

Issues. http://assets.opencrs.com/repts/RL33301_20060307.pdf 

x Department of Energy Organization Act of 1977 (42 USC 5815(b)) – grants administrative 

authority for agencies to conduct program evaluations. http://frwebgate.access.gpo.gov/cgibin/getdoc.cgi?dbname=browse_usc&docid=Cite:+42USC5815
144 

Attachment 2

Technology Readiness Levels 

 Technology Readiness Level Definition (US DOE EERE)

TRL 1 Basic Research: Initial scientific research begins. Focus is on fundamental 

understanding of a material or process. 

TRL 2 Applied Research: Initial practical applications are identified. Potential of material 

or process to satisfy a technology need is confirmed. 

TRL 3 Critical Function or Proof of Concept Established: Applied research continues 

and early stage development begins. Includes studies and initial laboratory 

measurements to validate analytical predictions of separate elements of the 

technology. 

TRL 4 Lab Testing/Validation of Alpha Prototype Component/Process: Design, 

development and lab testing of technological components are performed. Results 

provide evidence that applicable component/process performance targets may be 

attainable based on projected or modeled systems. 

TRL 5 Laboratory Testing of Integrated/Semi-Integrated: Component and/or process 

validation in relevant environment (beta prototype component level). 

TRL 6 Prototype System Verified: System/process prototype demonstration in an 

operational environment (beta prototype system level). 

TRL 7 Integrated Pilot System Demonstrated: System/process prototype demonstration 

in an operational environment (integrated pilot system level). 

TRL 8 System Incorporated in Commercial Design: Actual system/process completed 

and qualified through test and demonstration (pre-commercial demonstration). 

TRL 9 System Proven and Ready for Full Commercial Deployment: Actual system 

proven through successful operations in operating environment, and ready for full 

commercial deployment. 

)

OFFICE OF STRATEGIC PROGRAMS

DOE/EE-1117 • August 2014

Printed with a renewable-source ink on paper containing at least 

50% wastepaper, including 10% post-consumer waste.