# **Conference on**

# **Estimating the Benefits of Government-Sponsored R&D**[^1]

**March 4 and 5, 2002**

**SUMMARY OF DISCUSSION IN WORKSHOP D:**

**SECURITY, ECONOMIC, AND ENVIRONMENTAL BENEFITS**[^2]

## 

## Paul Leiby, Oak Ridge National Laboratory[^3]

**May 1, 2002**

## 

# 

# Workshop D considered the topics of energy security, environment and the economy. For these topics, the Workshop sought to:

## Define the major *types of benefits* that can result from R&D;

## Identify *methods* to estimate these benefits; and

## Identify *data sources*.

The suggested point of departure for this discussion was a
methodological framework that was represented by a matrix of benefits
categories. This framework was described in the conference draft white
paper, based largely on the framework developed by the National Research
Council (NRC) committee. The major points made in the workshop are
summarized below in bullet form, organized by general topic.

**Overall Summary of Perspectives on R&D Benefits Estimation**

The following perspectives appeared to have emerged among many of the
workshop participants:

## It was recognized that we are going through a cultural change whether we like it or not. This applies to the U.S. Department of Energy (DOE), other U.S. government agencies, and the government-sponsored R&D community. There is a need for a structured framework and methods for R&D benefits assessment.

## The NRC-derived matrix of benefits categories provided in the white paper, and our efforts at this conference, may be helpful. 

## The matrix held up well over the course of the workshop.

## It will take time to work out many of the details of a structured approach for estimating the benefits of R&D programs.

## There was a widespread expression of hope that this process for developing an R&D benefits assessment framework, and this dialog, would continue.

## 

# **Discussion on the Suggested Methodological Framework**

# Some workshop participants had questions about the matrix:

-   Concern was voiced that the boxes in the matrix are too limiting,
    due to the complexity and interdependencies of energy systems and
    the effects of one component on another

-   Some asked whether we are "stovepiping" energy security from other
    interrelated categories

-   Some asked whether we should amend the row categories (economic,
    environment, and energy security).

Others had less of a problem with the suggested matrix. They liked the
fact that it breaks the problem up for manageability:

-   Boxes are useful as overall organizing structure; indicate
    categories

-   A report that uses such a framework would have discussions and
    explanations of interdependencies and other special concerns.

-   Do we want to define the boxes (and their components) even if we
    know the data are unavailable, or should those components be \"off
    the table?\" The basic response was that it is useful to define, and
    at least to identify, the data that are required

-   Matrix approach was generally favored for the following reasons:

## Said to provide a helpful menu

## Avoids too mechanical an approach

## Expected that no R&D program likely to have benefit components in only one cell

## Leaves weights (i.e., relative importance of different types of benefits) to those who must make decisions about program focus

## Compatible with subjective evaluation of categories

Workshop participants discussed whether different types of benefits are
adequately addressed in the framework. There seemed to be a sense among
many conferees that the framework was flexible enough to admit most or
possibly even all types of benefits, but questions were raised by some
of the workshop participants about whether the framework adequately
accounted for the following three factors: Equity, Risk and Timing.

## Equity

### Equity, or distributional issues were not explicitly indicated in the proposed white paper framework. These factors refer to which groups, regions or sectors receive the benefits (and bear the costs) of the R&D programs.

## Risk 

### There is a need to capture the risk-related implications of R&D programs. Risk has many attributes, including the risk of an R&D project\'s or technology\'s success, and the potential effects of an R&D project on environmental risk (e.g. from possible climate change or exposure to toxic chemicals) or energy security risk. Also, there is a need to consider the interactive effects of two or more R&D programs on risk and their uncertain returns on investment, that is, to consider *portfolio* effects. This is a significant challenge.

### The approach of the modified NRC matrix in the draft white paper was viewed by some as a pragmatic simplification of those risk factors, putting them largely into the Option category.

### The approach was said to "take a continuum (of factors) and divide it into meaningful slices."

## Timing

### The matrix is static, in that it does not explicitly address timing issues, such as the timing of R&D benefits and costs. It was not clear whether the matrix was expected to summarize the discounted (net present value) of benefits in each category, or whether a series of matrices were to be used as snapshots at various points in time. However timing is important in R&D decision-making, given the inherent delay between research and payoff, and given the fact that critical uncertainties that bear on R&D benefits will unfold over time.

# 

# **Framework for Energy Security Benefits of Energy R&D**

# 

# Following the general discussion on the methodological framework, Workshop D focused on energy security issues, as charged. This naturally led to a call for a definition of \"security.\"

**Defining Energy Security**

## It was suggested that the definition involves both:

### Catastrophic events, short-term spike interruptions

### Long-term changes in energy system or markets (\"world\" or structural changes).

## An alternative definition was proposed:

### Energy Security is "Energy available when and where needed, at a predictable price."

### The definition has the virtue that it does not focus on the cause of any disruption.

D. Greene made a presentation on oil security, which was defined as a
market failure due to monopoly power, as distinct from a conventional
"externality."

-   Monopoly power, as exercised by OPEC, is more precisely an imperfect
    monopolistic cartel of the von Stackelberg type.

-   The result is unreliable supply of energy, and a lack of short-run
    substitutes.

# It was noted that the white paper provided to the workshop suggests the following security definition, with three components:

### oil or fuel security

### infrastructure security

### power reliability.

The overall preference among many workshop participants appeared to be
for a broad definition of energy security that admitted disruptions from
a diverse set of causes.

# The workshop group listed and defined some major *types of security benefits*:

## Reduced net import costs

## Lower macroeconomic disruption costs

## Lower environmental costs

## Health and safety

## Lower liability costs

## Lower insurance costs

## Greater predictability of price

## Costs of protection (military, guarding etc.)

######### Indicators of Energy Security, and Energy Security Benefits

# In order to measure energy security benefits, workshop participants discussed what measures or indicators of security might be used. Indicators of energy security seemed to fall into three broad categories:

## *Supply and demand conditions*. These are mostly levels of flows or stocks, and patterns of energy or financial flows, which are thought to be correlated with energy security.

## *Measures of system flexibility and sensitivity*. These refer to the ability of the system to adjust to changes or shocks, or to accommodate those shocks with lower cost.

## *Measures of supply reliability/volatility*. These are measures of the actual frequency or incidence of shocks, sharp fluctuations, supply disruptions, and outages.

Each of these categories of measures was elaborated in the surrounding
discussion. A large number of specific examples of measures were
proposed. Below are listings of the measures proposed, without
elaboration.

# Indicators of Energy Security Benefits 1: [Supply and Demand Conditions]{.underline}

## Energy import levels (in terms of both quantity and \$)

## Diversity of delivery channels

## Diversity of Supply sources

## Degree of monopoly/cartel power (measured by OPEC market share)

## \$ sent to potentially unfriendly nations

## Insurance rates/costs.

# 

# Indicators of Energy Security Benefits 2: [System Flexibility and Sensitivity Measures]{.underline}

## Price elasticity of world demand

## Price elasticity of U.S. demand (for various fuels)

## Substitutability: dual-fuel electric generating capacity

## Fuel Stocks (especially oil and gas)

## Macroeconomic sensitivity to energy shocks (measurable econometrically?).

# 

# Indicators of Energy Security Benefits 3: [Reliability/Volatility Measures]{.underline}

## Interruption frequency and duration (outages, supply disruptions)

## Other measures of grid reliability (power quality, brownouts, etc.)

## Capacity to meet peak demands for electric power

## Investor confidence, PE (price-earnings)ratios of companies

## Price volatility measures for various fuels

## Transmission/transportation reliability

## Survey responses of infrastructure owners and users.

## 

# **Reduction of Security Benefits to Three Simple Metrics/Components**

The workshop group realized that excessive detail and specificity in the
proposed measures of security benefits would not necessarily be helpful
for evaluating the full and broad array of R&D programs supported by the
government. So there was general enthusiasm for the idea of
consolidating these measures and for reducing the benefits categories to
a small number of broad categories suitable for inclusion in the
evaluation framework. At the same time, the fuller list of possible
categories and measures mentioned above may be helpful to remind
evaluators of possible areas for consideration. The three simple
security benefits categories proposed were: Prevention Benefits,
Management Benefits, and Recover Benefits.

-   *Prevention Benefits* (e.g. these measures to reduce disruption
    probability, and to prevent security breaches or system failures)

-   *Management Benefits* (to reduce costs of events)

-   *Recovery/Fix Benefits* (to promote recovery and restoration after
    disasters, and to increase the speed of response and recovery.
    Included here might be measures to change the overall system, in
    order to address and repair the fundamental problem that led to the
    security risk.)

# Upon reflection, it was noted that this approach for characterizing security issues seems similar to that used by FEMA (Federal Emergency Management Agency) for natural disasters. Therefore it might be possible to leverage the FEMA methodology and apply it to energy security.

For the three broad categories above, specific examples of activities
that promote security were discussed:

# Prevention Activities: (prospective or ex ante)

### Probability assessment/risk analysis

### Backstop resource development 

### Substitution

### Intelligence gathering

### Decentralization

### Identify key vulnerabilities (risk assessment)

### Hardening/guarding

# Security Management Activities: (responsive, after an event)

### Event management and damage reduction

### Preparedness activity

### Flexibility measures

### Stockpiling

# Recovery Activities (Fixing)

### Response and reconstitution measures

### Standardization and stockpiling of vulnerable commodities

#### e.g. standardized transformers.

# 

# At this point, on the second day of the conference and after completing three of Workshop D's six sessions, the discussion moved on to the other two categories of possible R&D benefits:

## Economic benefits

## Environmental benefits.

######## Estimating Overall Economic Benefits of Energy R&D

# In this session, workshop participants discussed ways to estimate the overall effect of R&D programs on the economy, through both direct and indirect channels. Direct observation of these economic effects is difficult.

# 

# **The RFF Cost/Expenditure Index**

#  

# One possible approach for assessing economic benefits of R&D that produces a new technology was described by M. Macauley of Resources for the Future. This index is essentially a ratio of expected consumer costs, with and without the new technology. This approach is conceptually grounded in the economic literature (see Bresnehan, AER 1986).

Features of the Cost/Expenditure Function Index Approach:

### It relies on assumed adoption rates for the new technology.

### Uses a model to estimate electricity generation costs and externality costs,

### Uses Monte Carlo simulation with a variety of probability distributions

### It is simple, allows incorporation of uncertainty, and the value of "externalities" such as environmental effects.

# 

# **Aggregate Estimation of R&D Macroeconomic Benefits**

#  Discussion turned to the estimation of economic benefits in the most aggregate sense -- through macroeconomic measures such as changes in industrial sector output of particular economic sectors. I. Nadiri presented an example of a methodology for estimating the benefits of industry-sponsored R&D in the manufacturing sector. The presentation and ensuing discussion raised important issues. Some of these issues are listed below.

# Important Issues Associated with the Estimation of Economic Benefits of R&D:

### There may be some degree of "crowding out" of private R&D, or substitution of public for private R&D. This raises the question: "Is a government R&D dollar the same as a private R&D dollar?

### Spillover of government R&D to other sectors and activities is important and measurable. Nadiri found large spillover benefits from government sponsored R&D in the manufacturing sector.

### There is a tradeoff between public funded R&D, and R&D tax credits. This should be considered. Even among tax incentives, there is a distinction between the full economic effects of R&D tax credits versus R&D expensing.

# 

# Conclusions on the macro effects of R&D in [manufacturing,]{.underline} based on the work of Nadiri:

## This work considered the total effect of *all* manufacturing R&D (not individual projects or programs) and found:

### Government R&D has *high return and is very cost-effective*

### It has a measurable, positive spillover effect

### Rates of return (including social) are very impressive

## Unfortunately, the current approach described by Nadiri does *not* appear to be applicable to individual R&D projects.

# 

# Suggestion on General Economic Benefits Evaluation for [Energy R&D]{.underline}:

## There may be substantial value to performing a comparable aggregate analysis of total [energy]{.underline} sector R&D

## While such an aggregate analysis will not directly evaluate specific R&D projects or programs, it could provide:

### An overall "tent" for individual project analysis;

### A calibration and bounds of the broad macroeconomic effects.

# 

**Various Types of Economic Benefits**

# In the ensuing discussions, the workshop group demonstrated that it could identify many categories of potential benefit, some of which are clearly in the "economic" category. For the record, the list included:

## Reduction of costs

## Accelerate introduction

## Private risk reduction

## Consumer savings

## Reduced cost of production

## Royalties, taxes

## Advice for policymakers

## Increased supply

## Increased student pool

## Community economic

## Reduced risk of conflict

## Regulatory streamlining

## Government revenue

## Technological capacity

## Economic stability

## Increased consumer choice

## Economic stability

## Productivity, jobs

## Information/knowledge

## Product quality

## Diversity of supply & delivery

## Enhanced supply or capacity for domestic fuels/energy

## Lower world energy prices (reducing net import cost)

## 

# **Economic Benefits: Methods and Data Sources**

The workshop group was asked to identify methods and data sources for
the estimation of R&D economic benefits. While little time was available
for discussing data sources, the general estimation methods offered and
considered were the following:

## Index-based

## Econometric

## Case-study

## Technical and economic models

## Direct measurement

## Detailed tracking of R&D outcomes 

### Adding markers on work and tracking their movement through the economy.

## Sampling techniques (statistical monitoring)

# **Estimating Environmental Benefits of R&D**

# **Environmental Value of R&D Information: A Bayesian Approach**

## The approach is to add information, often narrowing the probability distribution on a key variable

## Increase confidence/reduce environmental risk

## Increase chance of a good decision

-   Key point is that the value of information depends on how it is used
    in regulatory/policy environment

## Lessons about properly using information, and properly targeting R&D.

# 

# **Environmental Benefits in Economy and Society**

## Call for broader view of well being, and environmental and human factors:

### Four types of capital

#### conventional economic capital

#### human capital

#### natural capital

#### social capital (institution, rules, and norms)

### Note production also emits waste

### Also recognize non-market contributions to welfare.

Aggregating to national accounting level:

## Index of Sustainable Economic Welfare (ISEW). This index of welfare is like GNP, but more inclusive. In particular, it adds or subtracts the following factors:

### + non defensive less defensive public expenditure

### + human and environmental capital formation

### - environmental degradation

## Big: estimates of ecosystems services benefit exceeding GNP

# 

# How to Account for Sustainability?

## Inherent issue is dynamics

### need dynamic modeling approach

## To avoid double counting (e.g. resources have value in and of themselves as well as value as inputs).

### Need general equilibrium approach with (global) model

# Issues in Measuring Environmental Benefit:

## Keep evaluation approach simple

## Focus first on direct costs/benefits

## Avoid double counting

### distinguish market and non-market (external) components

## Distinction between net and gross effects

## Establishing reference point matters

## Timing and dynamics critical 

### (e.g. for sustainability)

# **Environmental Measures and Evaluation Tools**

## Indices useful and appealing

### Index of Sustainable Economic Welfare (ISEW)

### Can we construct an index of these many environmental factors?

### A relative cost index, such as that proposed by RFF

## *Intensities* may be appealing measure. Intensities, such as emissions per unit of output, or energy per unit output, are clearly defined, and have recently gained some attention in the discussion of how to gauge progress in climate policy. But, it was noted, absolute levels matter as well, and intensities may be incomplete as measures of environmental benefit.

####### Other established tools for estimating environmental benefits. The discussion recognized the existence of a wide range of established economic tools for valuing environmental benefits, but did not discuss them at any length. Included were

### Hedonic methods

### Contingent valuation methods

### Direct measures of market value, such as loss in productivity, health costs, and recreation expenditure losses.

# 

# **Summary Discussions**

Some significant points made during final, summary discussions in the
workshop included the following:

## Concerns were raised about the need for consistency and objectivity in R&D project selection.

## These goals were felt to be promoted by "transparency."

## Developing a structure such as this matrix could be big first step.

# **Other Overall Perspectives Expressed**

## Many good ideas were offered over the course of the conference, now we need to bring them together.

## In devising an assessment framework, there will inevitably be a tension between representing important details and keeping it simple.

## It was hoped that "triage" could be performed on the material coming out of the conference, to preserve the most vital ideas.

## Many expressed the view that the focus now should be on a process for this community to come out with report.

## This dialogue on the assessment process should continue (this was said many times).

## Many said that we should not lose sight of the importance of peer review as a component of evaluation method. 

# **Colorful Phrases and Insights**

####### 

####### Some of the viewpoints offered during the summary discussion came in the form of colorful phrases, which have the merit of being pithy and memorable.

##  "Be unafraid of the imperfect." Despite the many challenges and inevitable limitations, it is necessary, and worthwhile, to press on with devising a methodology.

## "Avoid reinventing the wheel." Many pieces of this process, particularly in the areas of energy security benefits assessment, environmental benefits assessment, and economy-wide benefits assessment, already exist in some form.

## "To talk about the bull is not the same thing as being in the bullring." This is true of both talking about an assessment framework as opposed to actually building it, and true of talking about the assessment process as opposed to actually doing it.

[^1]: Organized by Oak Ridge National Laboratory and sponsored by the
    Office of Energy Efficiency and Renewable Energy, Office of Fossil
    Energy, Office Nuclear Energy, Science and Technology, and Office of
    Science of the U.S. Department of Energy. Information about the
    conference is available on the conference web site,
    [www.esd.ornl.gov/benefits_conference](http://www.esd.ornl.gov/benefits_conference);
    in the white paper distributed prior to the conference, \"Ideas on a
    Framework and Methods for Estimating the Benefits of
    Government-Sponsored Energy R&D;\" and in the report summarizing the
    conference proceedings, \"Synthesis of Conference Discussions.\"

[^2]: This document is believed to be a reasonably accurate summary of
    discussions in Workshop D of the conference on \"Estimating the
    Benefits of Government-Sponsored Energy R&D;\" but the accuracy is
    not guaranteed by the workshop rapporteur, Oak Ridge National
    Laboratory, UT-Battelle LLC, or the U.S. Department of Energy.
    Furthermore, the opinions expressed by those at the conference are
    their own and therefore nothing in the reporting of the discussions
    in Workshop D or of the conference proceedings should be construed
    as government policy.

[^3]: Oak Ridge National Laboratory is managed by UT-Battelle, LLC for
    the U.S. Department of Energy under contract DE-AC05-00OR22725.