**IDEAS ON A FRAMEWORK AND METHODS FOR ESTIMATING THE BENEFITS OF
GOVERNMENT-SPONSORED ENERGY R&D**

**Russell Lee, Oak Ridge National Laboratory\***

**With contributions from:**

**James L. Wolf, Independent Consultant**

**Mary Beth Zimmerman, Office Energy Efficiency and Renewable Energy,
DOE**

**Jay Braitsch, Office of Fossil Energy, DOE**

**Trevor L. Cook, Office of Nuclear Energy, Science and Technology,
DOE**

**Robert Vallario, Office of Science, DOE**

**Jeanne Powell, Advanced Technology Program, NIST**

**Robert Pearce, Office of Aerospace Technology, NASA**

**Robert C. Ricci, Volpe Transportation Center, DOT**

February 27, 2002

Prepared as pre-conference reading material for the conference on

\"Estimating the Benefits of Government-Sponsored Energy R&D,\"

March 4 and 5, 2002, Arlington, Virginia, USA.

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

\* Oak Ridge National Laboratory is managed by UT-Battelle, LLC for the
U.S. Department of Energy under Contract No. DE-AC05-00OR22725. The
views expressed in various sections of this paper are solely those of
their respective authors; and do not necessarily represent those of Oak
Ridge National Laboratory, UT-Battelle, the U.S. Department of Energy,
the National Institute of Standards and Technology, the National
Aeronautical and Space Administration, or the U.S. Department of
Transportation.

**TABLE OF CONTENTS**

Page

SUMMARY

1.  PURPOSE

2\. CONTEXT AND MOTIVATION FOR THE CONFERENCE

3\. METHODS USED BY DOE ENERGY RESOURCES AND SCIENCE OFFICES TO ESTIMATE
OR ASSESS THE BENEFITS OF THEIR R&D PROGRAMS

1.  OVERVIEW OF METHODS USED BY THE OFFICE OF ENERGY EFFICIENCY AND
    RENEWABLE ENERGY FOR GPRA

2.  MEASURING FUTURE BENEFITS FOR FOSSIL ENERGY RESEARCH AND DEVELOPMENT

3.2.1. Fossil Fuel Conversion

3.2.2. Fossil Energy Resources

3.2.3. Future Directions

3.  Overview of Methods Used by the Office of Science

4\. METHODS USED BY OTHER FEDERAL AGENCIES TO MEASURE THE
OUTCOME-RELATED PERFORMANCE OF THEIR PROGRAMS

4.1. MEASURING THE BENEFITS OF THE ADVANCED TECHNOLOGY PROGRAM

4.1.1. ATP\'s Mission and Operations

2.  ATP\'s Multi-Component Assessment Program

3.  ATP\'s Evaluation Results

4.2. FEDERAL RAILROAD ADMINISTRATION\'S (FRA\'S) R&D PROJECT EVALUATION
AND INVESTMENT ANALYSIS

5\. NATIONAL RESEARCH COUNCIL'S METHODOLOGY FOR DEFINING THE BENEFITS OF
ENERGY R&D

5.1. METHODOLOGICAL FRAMEWORK

5.2. DERIVATION OF COLUMNS FOR THE BENEFITS MATRIX

**TABLE OF CONTENTS (Cont'd)**

Page

5.3. MAJOR ISSUES CONSIDERED -- ALL NOT NECESSARILY RESOLVED

5.3.1. Establishing a Baseline for Realized or Projected Benefits

5.3.2. Effect of Government Program

5.3.3. Nature of Benefits

6\. BUILDING ON THE NATIONAL RESEARCH COUNCIL'S STUDY -- IDEAS ON
FRAMEWORK AND METHODS FOR ESTIMATING THE BENEFITS OF GOVERNMENT R&D

6.1. PROSPECTIVE ANALYSIS OF BENEFITS

6.1.1. Definition of Expected Prospective Benefits

6.1.2. Projected Baseline

6.1.3. Government Impact

6.2. OPTION VALUE

6.2.1. Option Concept as Described in the NRC Study

6.2.2. Applying Real Options Theory to R&D Investments

6.3. KNOWLEDGE VALUE

6.3.1. Assessment of Knowledge Benefits

6.3.2. Spillovers

6.4. SECURITY, ECONOMIC, AND ENVIRONMENTAL BENEFITS

6.4.1. Security Benefits

6.4.2. Economic Benefits

6.4.3. Environmental Benefits

7\. CONSIDERATIONS IN IMPLEMENTING A FRAMEWORK AND METHODS

7.1. DATA AND DATA MANAGEMENT CONSIDERATIONS

7.2. PROCESS CONSIDERATIONS

7.3. ASSESSING A PORTFOLIO OF R&D PROGRAMS AS OPPOSED TO JUST INDIVIDUAL
PROJECTS

**TABLE OF CONTENTS (Cont'd)**

Page

7.4. ASSESSING MULTI-YEAR PROJECTS OR PROGRAMS AS OPPOSED TO AN
INDIVIDUAL YEAR'S BUDGET REQUEST FOR A PROJECT

8\. CONFERENCE FORMAT

8.1. PLENARY SESSIONS

8.2. WORKSHOP A: PROSPECTIVE BENEFITS -- CROSS-CUTTING ISSUES

1.  Developing a Baseline, Absent a Technology Developed in Part

by a Government Program

8.2.2. Impact of the Government Program

8.3. WORKSHOP B: OPTION VALUE

8.4. WORKSHOP C: KNOWLEDGE VALUE

8.5. WORKSHOP D: SECURITY, ECONOMIC, AND ENVIRONMENTAL BENEFITS

**IDEAS ON A FRAMEWORK AND ASSOCIATED METHODS FOR ESTIMATING THE
BENEFITS OF GOVERNMENT-SPONSORED ENERGY R&D**

**SUMMARY**

This white paper presents a starting point for discussions at the
conference on \"Estimating the Benefits of Government-Sponsored Energy
R&D.\" The specific objectives of this conference are to identify major
streams of thought on:

(a) a useful methodological framework for identifying the benefits of
    government-sponsored energy R&D; and

(b) practical approaches for developing improvements to current methods
    of estimating the benefits of energy resource and science R&D, which
    might be used to enhance the performance-based management of these
    programs under the Government Performance and Results Act of 1993
    (GPRA).

The contents of this paper should *not* be interpreted as
recommendations to any government agency. Rather, the ideas are intended
to stimulate discussion at the conference.

**Background and Motivation for the Conference**

The motivation for this conference is the priority that the energy
resources and science offices in the U.S. Department of Energy (DOE) are
giving to measuring and assessing the performance of their R&D programs.
Under GPRA, DOE and other federal agencies are required to report
annually on their programs\' plans and performance. Furthermore,
President George W. Bush and the U.S. Congress are placing great
emphasis on performance-based assessments in the funding of federal
programs.

Estimates of program benefits are one part of the performance
requirements established by GPRA and implemented throughout the federal
government. GPRA addresses both expected performance and retrospective
evaluation at all levels of government activity -- project, program,
office, agency---and in all time frames---monthly and quarterly
execution, annual and multi-year milestones, final outputs, and
resulting market impacts or benefits. This conference focuses on ways of
improving our ability to estimate the ultimate benefits that result from
the outputs of energy R&D programs. The conference does not directly
address other GPRA requirements.

Estimating the benefits of R&D programs is challenging because of the
very long time frames required for basic and applied research, the
process of introducing the resulting product into the market, and the
lifetime of investments made in the product. In some cases, over half a
decade might be required to examine the full impacts of a line of
research. In addition, the results of a research portfolio are not
simply the sums of the results of its individual parts. Because benefits
are often the result of multiple areas of effort, this conference
focuses on benefits at the program level and above, rather than on
performance related to individual research projects.

The level of benefits depends both on the success of the research effort
itself (outputs) and on changes in the various "external factors" --
market or policy conditions -- that affect the market penetration and
market shares of these new technologies. Another challenge is the
implementation of a methodological framework in a sufficiently
transparent way that the relative contribution of the R&D, and of the
external assumptions, on projected benefits is clear, and that the
different reasons why projected benefits are (or are not) realized can
be documented.

A government agency\'s research portfolio can affect literally thousands
of products throughout the economy over several decades. It is essential
to carefully consider the value of this research, both retrospectively
and prospectively. It is also important to undertake such studies of the
benefits of R&D \-- for the purposes of GPRA and program planning \-- in
ways that are practical and cost-effective.

**Benefits Framework Developed by National Research Council Study**

There are many ways of addressing the stated objectives of the
conference. However, the recent study done by the National Research
Council\'s (NRC) Committee on Benefits of DOE R&D on Energy Efficiency
and Fossil Energy provides an important context for this conference.[^1]
The NRC study developed and implemented a methodological framework to
estimate the retrospective benefits of individual energy efficiency and
fossil energy R&D programs in DOE.

Thus, the framework that we suggest as a starting point for discussions
at the conference is *adapted* from the one developed in the NRC study.
The NRC committee used a matrix with three rows and three columns to
represent the benefits framework it developed. Three basic types of
impacts are listed along one dimension of the matrix (i.e., the rows):
economic, environmental, and security. The other dimension of the matrix
(i.e., the columns) reflects the degree of commercialization, certainty,
and specific use of the benefits. Along this dimension, the benefits are
classified as being realized, options, or knowledge. Each of these
categories of benefits is described below.

-   Economic Benefits: Measured by the change in the market value of
    goods and services that are produced under "normal" economic
    conditions resulting from the introduction of a technology stemming
    from DOE research. The benefit is measured net of all public and
    private costs, and can typically be reflected in changes in the
    level of goods and services produced or in their market prices. The
    NRC considered the program costs \-- the costs borne by DOE and
    private industry in conducting the R&D \-- as well as any
    incremental costs of the technology borne by the end-user or
    consumer.

```{=html}
<!-- -->
```
-   Environmental Benefits: Based on changes in the quality of the
    environment because of DOE research. The benefit is typically not
    measured directly by changes in market prices, but rather by some
    measure of the value individuals in society are willing to place on
    changes in the quality of the environment and improved public
    health. In cases where the research reduces the costs of complying
    with environmental regulations, the benefit takes the form of lower
    compliance costs rather than improved environmental conditions.

-   Security Benefits: Measured by changes in the probability or
    severity of abnormal energy-related events that would adversely
    impact the overall economy, public safety, or the environment.
    Traditionally, this was primarily a concern about volatile oil
    markets. Recently, there is increased concern about the reliability
    and security of the energy supply infrastructure.

The NRC recognized that R&D might lead to benefits even when a
technology developed by that R&D does not enter the market immediately
or to a significant degree at all. This lack of commercial use might be
due to changes in the forecasted economic or policy conditions, or to
technical shortcomings. To account for this uncertainty, and to reflect
the different degrees of technology development in a retrospective
assessment, the NRC established three categories:

-   Realized Benefits: These economic, environmental or security
    benefits have been realized by the current time, or are almost
    certain to be realized in the near future, as a result of the
    research effort leading to the development of
    commercially-attractive technologies. Often, these results are an
    acceleration of technologies that would have otherwise been
    commercially available at a later date. In other cases, they might
    represent new technologies or changes in product attributes that the
    private sector would not have produced on its own. The NRC included
    all lifecycle benefits of the units of the evaluated technologies
    that were projected to be installed by the year 2005.

-   Option Benefits: This category covered technologies that are fully
    developed but for which existing economic or policy conditions are
    not likely to be favorable for commercialization. To be considered
    an option by the NRC committee, the technologies needed to be
    favorable for commercialization under some credible or plausible
    circumstances. For example, carbon sequestration technologies are
    options whose value might be realized if new regulations or trading
    of carbon emission permits take effect.

-   Knowledge Benefits: R&D, whether successful or not, typically
    produces knowledge. The generation of scientific knowledge is a key
    part of DOE's mission. The NRC considered knowledge benefits to be
    scientific knowledge and useful technological concepts that have not
    yet been incorporated into commercialized results from the R&D
    program, but that hold promise for future use or that are useful in
    unintended applications. These benefits are in addition to those
    accounted for in the other areas of realized and option benefits.
    Knowledge benefits tend to be the earliest benefits from the R&D
    process.

Benefits Framework Adapted from the NRC Study

As we turn from the *retrospective* context of the NRC study, to the
*prospective* one needed for program planning and GRPA, we suggest that
the NRC benefits framework could be adapted or expanded in the following
ways:

-   The concept of realized benefits in the retrospective context could
    be augmented with a projection of *expected prospective benefits*
    under the most likely scenario. This change would effectively add
    one column to the benefits matrix to represent these expected
    prospective benefits.

-   The definition of the baseline condition, against which a new
    technology or new science is \"added\" as a result of R&D, could be
    expanded from a strictly retrospective context to include the
    prospective situation.

-   The impact of a government R&D program, on technologies developed in
    part by the private sector, could be estimated prospectively.

-   The definition of options could be expanded to include not only the
    retrospective definition (that options are technologies that are
    fully developed but for which existing economic or policy conditions
    are not likely to be favorable for commercialization), but also the
    prospective context of R&D investments under uncertainty.

-   Retrospective indicators of the knowledge generated by past R&D
    could somehow be expanded to consider the potential knowledge from
    ongoing and planned science projects.

-   Methods for estimating some of the security, economic, and
    environmental impacts of R&D programs, which the NRC committee did
    not have an opportunity to develop, could be re-analyzed, especially
    in light of the terrorist events of \"September 11.\"

-   Data and data management requirements for prospective analyses could
    be assessed, including whether the requisite data are available and
    affordable. Also, consideration could be given to whether the
    methods can be implemented without having to rely on unverifiable
    assumptions, which would limit the credibility of the resulting
    estimates.

-   Within a GPRA and program management context, greater attention
    could be given to developing methods for program planning and
    decision making that *use* the estimates of the benefits of R&D
    programs, a point made by the NRC study as well.

This white paper expands on each of points above to provide a basis for
discussions at the conference. Each point is discussed further in the
rest of this summary.

**Expanded Benefits Framework to Accommodate the Need for Prospective
Analysis**

Pursuant to its charge, the NRC study developed its framework to assess
retrospective benefits on a case-study basis. In order to adapt the
framework to GPRA purposes, we suggest a framework that builds on the
NRC\'s 3 x 3 matrix. GPRA requires that performance statements be
tracked from pre-program planning through post-program evaluation. As a
result, the benefits framework is extended to address *prospective*
benefits in a way that allows the benefit estimates to be tracked over
time. This requires the addition of a fourth column in the matrix. The
need to undertake prospective analysis also requires consideration of
the ways in which benefits develop as R&D matures, and considerations of
the differences between retrospective and prospective information. In
the following figure, we have modified the NRC matrix by adding the
\"prospective\" column. It was not part of the NRC framework because
that study considered only retrospective benefits.

Modification of National Research Council Committee\'s Framework

For Identifying Benefits of Energy R&D

+-------------+-------------+-------------+-------------+-------------+
|             | Realized    | Expected    | Option      | Knowledge   |
|             |             | Prospective |             |             |
|             | Re          |             |             |             |
|             | trospective |             |             |             |
+-------------+-------------+-------------+-------------+-------------+
| Economic    |             |             |             |             |
+-------------+-------------+-------------+-------------+-------------+
| En          |             |             |             |             |
| vironmental |             |             |             |             |
+-------------+-------------+-------------+-------------+-------------+
| Security    |             |             |             |             |
+-------------+-------------+-------------+-------------+-------------+

Once a framework is established, practical means are required for
estimating the benefits represented by each cell. Populating each of
these twelve cells with estimates is a daunting task, especially because
many of the cells represent cutting-edge areas of valuation theory, are
areas of controversy in the literature, or require data that are either
unavailable or extremely costly to collect and maintain. We suggest a
framework in which benefits can be described in ways as closely related
to the research efforts as possible (e.g., physical units of energy
produced or saved), with augmentation, where possible, with transparent
valuation into dollars. We also recognize that benefits that are not
directly measurable even in physical units must sometimes be represented
by proxy or index information.

**Expected Prospective Benefits**

We suggest that the expected prospective benefits could be defined as
those expected from future deployment of a technology developed as a
result of R&D, under the projected baseline set of future market and
policy conditions, compared to the expected conditions under the same
projected baseline, without the technology. Thus, estimation of expected
prospective benefits requires a baseline characterization of future
energy markets without the government research, and an estimation of how
baseline markets will react to the new or accelerated technology,
including its expected market penetration. These expected prospective
benefits could be economic, environmental or security-enhancing; and
they could be calculated over some pre-determined time horizon, such as
the next twenty years. Longer time frames might be required to capture
the full benefits of basic research or technologies requiring
fundamental changes in energy infrastructure.

This definition of expected prospective benefits is somewhat analogous
to that used by the NRC committee for retrospective benefits in that the
benefits of the technology are net of those of the next-best
alternative. In the prospective case, the next-best alternative is
reflected in the reference baseline, which includes technical
improvement. The question of defining the projected baseline is
addressed in Section 6.1.2.

The projected market penetration of a new technology is a key parameter
that greatly affects the magnitude of its expected prospective benefits.
In this regard, we suggest that it could be preferable to characterize
the attributes of the technology and to use a model to project how well
the technology will do in the market, than to simply assume a certain
level of market penetration after some number of years.

We further suggest that the model used for this purpose be calibrated,
to extent possible, to the same one used for the projected baseline.
Then, the Reference Case run of that model would be the projected
baseline, *without* the technology in question. A run of the same model
augmented with a simple representation of the technical performance and
supply (curve) of the technology in question, could be the case *with*
the R&D program. Where a different model is better suited to analyzing
the specific technology than the one used for the baseline, then the
other model can be calibrated to the baseline. The difference between
the economic, environmental and security conditions in these two runs of
the model would be the expected prospective benefits of the technology.

We suggest that the results of these calculations, which provide
projections of annual net benefits, could be reported in both of the
following ways:

-   Total net benefits, in five-year intervals over the next twenty
    years (for example), in real dollars, with no time discounting,
    *and*

-   Net present value, in which the estimated future benefits and costs
    over the operating life of equipment deployed are discounted.

**Projected Baseline**

Defining a baseline is an exercise in identifying the next-best
alternative. Expected improvements in technology, that can occur
regardless of whether a DOE program exists, must be accounted for.

For the prospective case, we suggest that the \"reference case\" of a
DOE forecasting model such as the National Energy Modeling System (NEMS)
could be considered for defining the projected baseline. If this model
is inadequate for this purpose, then we suggest the use of alternative
models that have better descriptions of the technology or markets under
consideration. Alternative models that could be used include MARKAL and
consumer choice models that identify the marginal reference-case choices
of technologies (i.e., the next-best alternative). If such models are
used, then we suggest that they could be calibrated to the more
important assumptions in the NEMS reference case to the extent possible.

In the retrospective case, the issue is one of defining the
counterfactual past condition. In either case, engineering judgement
might be required to \"back out\" the technology in question, which
might implicitly already be included in the reference case assumptions.

**Impact of the Government**

The 5-Year Rule adopted by the NRC study as a \"conservative\" means of
valuing benefits (NRC, p. 18) assumes that the private sector would lag
DOE by five years, with the same level of R&D and the same outcomes, if
DOE had not itself invested. We suggest that this rule could be
generalized and that the impact of a government R&D program could be
represented as a change in the timing of the benefits of the technology
developed from that program. We further suggest that the specific time
lag could be determined on a program by program basis. Just as, for
example, supply curves that describe the quantity of energy services
available at different prices are different for different technologies,
government R&D programs will reduce the time it takes for technologies
to reach market by different amounts of time. For prospective estimates
of benefits, the technology improvements in the reference case
implicitly define the time frame for commercialization without the
government R&D program.

For technologies that are developed to the extent that they are fairly
close to commercialization, such many of those funded in the past by the
National Institute of Standards and Technology\'s (NIST\'s) Advanced
Technology Program (ATP), the median lag was found to be about 3 years.
The lag ranges from about 1 year to an infinite number of years (i.e.,
the firm would never have carried out the R&D without support from ATP).

The magnitude of the lag will affect the magnitude of the calculated
benefits associated with public R&D programs, and it would be
informative to compare the relative values of potential R&D efforts when
different rates of accelerated commercialization are assumed. Thus, we
suggest that lags could be assessed on a program-by-program basis. If
the technologies under development are on the verge of
commercialization, then we suggest that a three- to five-year lag could
be appropriate. For new programs and R&D initiatives in their early
stages, the lag might be ten years or more. Many programs would have
infinite lags, i.e., the private sector would never undertake the R&D.

Note that whatever lag is used, it represents the impact or contribution
of the government program in prospective estimates of benefits. Then
there would be no need to reduce the benefits beyond this time-lag
adjustment. Alternative approaches are discussed in the paper.

**Option Benefits**

The state of the future projected in the \"reference case\" is (in
theory) the most likely, but certainly not the only possible future. We
suggest expanding the NRC study\'s concept of option technologies, which
was defined in a retrospective context, to include the prospective
situation.

In general, options provide discretionary choices to deal with
deviations from planned scenarios. We suggest that a real options
approach could be used for plausible energy scenarios other than the
reference case scenario used for estimating the expected prospective
benefits.

In a [prospective]{.underline} sense, there could be option value to R&D
on technologies that are not being developed *primarily* to enter the
market under the most likely conditions, but that would provide
economically viable solutions under alternative plausible conditions.
These plausible futures are generally of energy, environmental, security
or other policy concern. Physical or intellectual assets (not necessary
fully developed technologies as defined in the NRC study) that might be
deployed in the future (not just under improbable conditions) may also
contain significant option value. In this sense, options provide
insurance in the face of market uncertainties, yet retain the ability to
capture the upside benefits should improbably scenarios be realized.

In a [retrospective]{.underline} sense, technologies that are already
developed, but that are unlikely to be commercialized under current or
anticipated market conditions, may yet contain option value. This value
is derived from the uncertainty surrounding future market conditions.
These technologies might have been developed to enter the market but did
not do so because of changes in conditions \-- the technologies remain
available for the market in the future should conditions change to make
them commercially viable. For example, carbon sequestration technologies
could be options whose value might be realized if new regulations or
trading of carbon emission permits take effect.

Often, a research line is pursued because it has limited, but important,
expected benefits through niche market applications, but would be
expected to have much wider market applications under some alternative
futures, e.g., an oil disruption. While option values can persist after
the research is completed (often referred to as backstop or shelf
technologies), the value of the option will change as we move further
into the future and more certain about whether the technology
application is economically viable.

R&D can be considered as investments in options that provide
opportunities to realize benefits, in the event of alternative future
events. In addition to the option to commercialize, R&D investments
contain a host of other investment timing options. At each stage, as new
information becomes available about the probabilities of different
outcomes, choices can be made regarding continuation of the research,
abandoning the research, mothballing the research, etc.

To calculate option values, input data are typically required for:

-   Amount of the R&D investment, for which the corresponding value of
    the option is to be calculated, as well as some estimate of the
    technology outcomes that result from the R&D;

-   Degree of uncertainty about the costs to provide energy services
    using the technology in question, as a function of R&D;

-   Degree of uncertainty about the costs to provide energy services
    using competing technologies, i.e., degree of uncertainty of the
    projected baseline prices; and

-   Degree of technical uncertainty (risk) about the cost of bringing
    the technology to market.

Developing these input data is a real challenge, especially when done
for the first time. The uncertainties can be expressed \"continuously\"
using probability density functions, or \"discretely\" using a few
scenarios. Much of the real options literature uses continuous
representations of uncertainty over the range of relevant future market
factors. On the other hand, scenario analysis is frequently more
intuitively appealing. It is a well-established approach to planning
that allows individuals to assign their own views about the
probabilities associated with various scenarios.

**Knowledge Benefits**

We suggest that bibliometric methods could be used to develop indicators
of the value of the knowledge associated with research programs. For
retrospective analysis, we suggest that indicators could be developed
that reflect the linkages between science and innovation, such as:

-   Current Impact Index. This index considers how frequently other
    patents are cited in patents granted in a current year.

-   Science linkage. This method counts the number of times a patent
    cites scientific papers or similar research publications. This
    measure indicates how strongly a patent has relied on fundamental
    scientific research.

-   Technology-cycle time. This index is the median age of patents cited
    on the front page of a U.S. patent document. This measure indicates
    the speed of innovation in a company or industry.

Similar analysis could be done with R&D 100 Awards.

For prospective analysis, research might first be done on whether
results from retrospective studies could be used to develop simple
models that describe the linkages between the outputs of science
programs and their ultimate outcomes. Then, statistical relationships
might be developed for these linkages, which might in turn be used for
prospective assessments.

We suggest that peer-review methods could be adapted and used as the
primary method for assessing the prospective benefits of scientific
programs. An example might be the use of Delphi processes to develop
assessments of the prospective knowledge-related benefits of various
science programs. GPRA-like projections are generally much different
from typical technical peer reviews, though, so that peer reviews for
the purpose of GPRA-related requirements would need to be crafted to
address these needs.

**Security Benefits**

We suggest that security benefits related to oil use and imports, the
vulnerability of energy infrastructure and systems, and the reliability
of electric power supply could be included among the possible benefits
of energy R&D. This is a broader definition than what the NRC study
addressed. In particular, we suggest that the definition of security
benefits could be broadened to include those that reduce oil dependence
and its costs, not only those associated with abnormal events; and those
that improve electric power reliability, including that under \"normal\"
operations.

*Oil Security*: Oil security measures are well developed in the
literature. They can be expressed in barrels of reduced oil consumption
or reductions in oil intensity (oil used per dollar of GDP). The energy
security literature has also estimated values in dollars per barrel
reduction in oil consumption.

*Infrastructure and Systems:* We suggest that benefits of energy
infrastructure and system security could be estimated using
probabilistic risk analyses to estimate the impacts of the technologies
on the following:

\(a\) Reduced cost of outage or disruption to users,

(b) Reduced cost of providing a given, desirable level of security
    (e.g., cost of security personnel and equipment),

(c) Reduced impact on an industry and on the economy region- or
    nationwide (e.g., estimated using economic input-output models).

*Electric Power Reliability*: We suggest that the benefits of electric
power reliability could be valued by taking into account:

(a) \"Real-time energy value\" -- replacing the measures of energy
    savings and their financial value (Btu\'s and Btu's times *average*
    price, respectively), which are currently used, with a measure of
    financial savings that accounts for energy savings (and increases)
    and the *different* electricity rates at different times of the day
    and year;

(b) "Reduced outage costs" -- the financial value of reliability as
    measured by the estimated reduction in customers' outage costs; and

(c) Reduced costs of providing required level of reliability, including
    reduced costs of providing alternative means of reliability.

**Economic Benefits**

We suggest that, in addition to the economic benefits associated with
energy savings, other economic benefits could be included:

a.  Consumer benefits other than energy savings, such as increased
    productivity, which could be estimated using engineering analysis,
    survey, and experimental economic methods; and

b.  Macroeconomic benefits that will result from spillover effects, and
    which could be estimated using econometric methods.

The NRC study did not address these benefits in the same detail as it
did the economic benefits of energy savings.

**Environmental Benefits**

We suggest that environmental benefits could be reported based on
physical units of reduction, and valued using \$/ton values for criteria
pollutants and for carbon dioxide, as developed in the extensive
literature on these subjects. This literature advocates using estimates
of the damages from the pollutants as measures of their (deleterious)
value, as opposed to the costs of controlling these emissions.
Sensitivity analyses could be done to account for the significant range
of estimates of these values, particularly since the values are
sensitive to the size and geographic distribution of the population
exposed to pollutants.

We also suggest that a separate index could be used for ecosystem
impacts. This index could be an estimate of the value of increased
\"services\" provided by ecological systems. Values expressed in
\$/hectares per year might be used for open-ocean, coastal, forest,
grass/rangeland, wetland, lakes/rivers, desert, tundra, and ice/rock
biomes. Technology impacts might then be subjectively evaluated, and
expressed in terms of this measure.

**Data Considerations**

Regardless of which framework and methods are identified, they are
useful only if they can be implemented using publicly available and
affordable data. To the extent possible, data should be compiled from
field-verified results. For retrospective analysis, statistical studies
can provide useful information on technical performance.

We suggest that where subjective estimates of key input parameters are
required, that they could be periodically reviewed through a systematic
process with independent experts.

We further suggest that it could be unnecessary, and sometimes it cannot
be justified, to develop quantitative estimates of all of the benefits.
Sometimes, the quality and imprecision of input data might not justify
detailed numerical results \-- order of magnitude estimates might very
well be better in these cases. In other cases, the very nature of the
benefits in terms of their social or ecological impacts might inherently
preclude their being quantitatively estimated. In such situations, these
benefits might be just as, or even more, important than other types of
benefits that can be quantified. If this were the case, then GPRA
reporting and program planning would have to somehow take this into
account.

**Portfolios of Programs**

We suggest that R&D programs could be evaluated in the context of the
overall portfolio of options they provide, not simply in terms of their
expected individual benefits. We suggest that financial portfolio
methods could be used to help construct R&D portfolios among different
types of R&D that have different timeframes, different risks, different
*types* of risk, and different types of possible benefits.

Some R&D programs would be expected to achieve economic benefits; others
would be targeted more to achieving environmental benefits; still others
would have security benefits. Likewise, some programs should likely
focus on attaining expected prospective benefits; others should focus
more on attaining option value, given the great uncertainty in all R&D;
and still other programs, especially the science programs, will have as
their primary mission the development of new knowledge.

Yet, in all types of programs, \"losers\" should be expected in these
portfolios and decision rules could be developed to help decide which
programs to continue and which to terminate.

IDEAS ON A FRAMEWORK AND METHODS FOR ESTIMATING THE BENEFITS OF
GOVERNMENT-SPONSORED ENERGY R&D

**1. PURPOSE**

This white paper provides a common basis and point of departure for
discussions at the conference on \"Estimating the Benefits of
Government-Sponsored Energy R&D.\" The ideas presented as possible
suggestions should not be interpreted as recommendations or preferred
approaches for any government agency. Rather, the ideas are offered as a
starting point for discussions.

The purpose of this conference is to synthesize insights and information
from conference participants, which the energy resource and science
offices of the U.S. Department of Energy (DOE) might use to improve
their methods for estimating the benefits of their R&D programs.[^2]

The motivation for this conference is the priority that these DOE
offices are giving to measuring and assessing the performance of their
R&D programs. Under the Government Performance and Results Act of 1993
(GPRA), DOE and other federal agencies are required to report annually
on their programs\' plans and performance. Furthermore, President George
W. Bush and the U.S. Congress are placing great emphasis on
performance-based assessments and funding of federal programs.

The specific objectives of this conference are to identify major streams
of thought on:

(a) a useful methodological framework for identifying the benefits of
    government-sponsored energy R&D; and

(b) practical approaches for developing improvements to current methods
    of estimating the benefits of energy resource and science R&D, which
    might be used to enhance the performance-based management of these
    programs under GPRA.

The next section of this paper discusses more the motivation for the
conference. Section 3 summarizes methods which some of the energy
resources and science offices in DOE use to estimate or assess the
benefits of their R&D programs. Section 4 presents methods used by some
of the other, non-DOE departments in the Federal government. Section 5
summarizes the framework and concepts that were developed in an
important study done by the National Research Council (NRC).[^3] Section
6 offers suggestions that build on the NRC framework to address issues
beyond the scope of that study, but that are important for assessing
prospectively the benefits of R&D and for enhancing performance-based
management of these programs. Section 7 discusses important issues
related to implementation of such methods. Section 8 describes the
objectives and agenda of each session in the conference.

The NRC framework and the suggestions that build on it are certainly not
the only useful set of methods. However, given the context for this
conference and the progress made by the NRC committee in its study, we
have decided to \"pick up where they left off.\"

**2. CONTEXT AND MOTIVATION FOR THE CONFERENCE**

An important part of the U.S. Department of Energy\'s (DOE's) mission is
to foster a secure and reliable energy system that is environmentally
and economically sustainable. Several offices in DOE play key roles in
carrying out this mission. The Office of Energy Efficiency and Renewable
Energy (EERE) conducts research to develop and deploy clean and
efficient energy technologies. The Office of Fossil Fuel (FE) undertakes
research and development to promote the efficient and environmentally
sound production and use of fossil fuels. The Office of Nuclear Energy,
Science and Technology (NE) advances the application of nuclear
technology by investing in new or innovative opportunities for its
expanded use. The Office of Science (SC) complements the "energy
resources business line" of these other offices by advancing basic
research and the foundations of science. All of these offices assess the
benefits of their programs to assist their planning and budget
development, to meet the requirements of the Government Performance and
Results Act (GPRA), and to respond to requests for information from the
White House, Congress, and others.

These estimates are important means of assessing both the potential
future benefits from public R&D, and the performance and results of past
research efforts. Under the Government Performance and Results Act of
1993 (GPRA), federal agencies are required to report annually on their
plans and performance. In his "management agenda," President George W.
Bush emphasized that the federal government needs to measure the
effectiveness of its R&D investments and he chose \"energy resources\"
as the first area to apply new R&D selection criteria, including their
contributions to public benefits.[^4] The National Energy Policy
identifies a number of such potential benefits, including energy
security and environmental improvements.[^5]

Improved methods of estimating the value of energy R&D can increase the
effectiveness of future investments in it. A recent National Research
Council (NRC) study, cited in Section 1, developed an initial framework
for evaluating the benefits of DOE's past energy efficiency and fossil
energy R&D programs. The study\'s implementation of this framework
included conventional methods used in programmatic and economic
analysis, as well as cutting-edge methods in need of further assessment
and development.

This paper adapts the framework that the NRC study developed to define a
starting point for discussions at the conference.

**3. METHODS USED BY DOE ENERGY RESOURCES AND SCIENCE OFFICES TO
ESTIMATE OR ASSESS THE BENEFITS OF THEIR R&D PROGRAMS**

**3.1 OVERVIEW OF METHODS USED BY THE OFFICE OF ENERGY EFFICIENCY AND
RENEWABLE ENERGY FOR GPRA**[^6]

Each year, the Office of Energy Efficiency and Renewable Energy (EERE)
estimates future benefits of its portfolio of research and deployment
programs. Metrics currently estimated include economic, environmental,
and security elements, and are listed in the table below. The types of
benefits estimated are not complete -- for example, reliability-related
security benefits are not currently addressed.

The following chart, reproduced from EERE's FY 2003 Congressional Budget
request, is a summary of the results of EERE's *FY 2003 GPRA Benefits
Reports*. It is an example of the sort of performance data that GPRA
measurements can produce for decision makers to better evaluate the
results of funding EERE's program and activities.

![](media/image1.wmf)EERE initiated the effort to develop these metrics
prior to GPRA and the methodology has evolved significantly since then.
The annual effort is a multi-stage process, initiated with an annual
"call" and guidance identifying key budget, energy, economic, and other
assumptions to be used in developing program- and sector-level inputs.
Each program uses this guidance to develop estimates of likely
commercialization dates and initial market penetration rates of the
technologies under development. Program assumptions are reviewed on a
rotating basis by AD Little to help ensure consistency and improve
projections. Interactions among technologies developed (e.g., when two
technologies under development compete for the same market segment) are
addressed by integrating technology results across sectors and the
economy as a whole.

The reference case for EIA's most recent Annual Energy Outlook (AEO) is
used as the baseline for analysis, although it is modified to "back out"
any program results already included in the AEO. This approach provides
forecasted baseline energy prices and market sizes, as well as
incorporating underlying (non-program) improvements in energy technology
performance. To help ensure that analysis occurs on a marginal, rather
than average basis, EIA\'s National Energy Modeling System (NEMS) is run
at lower energy demand levels to determine which fuel sources might be
most effected as efficiency and renewable technologies make their way
into the market.

**3.2 Measuring Future Benefits for Fossil Energy Research and
Development**[^7]

Most past Fossil Energy (FE) R&D benefits estimation has focused on two
broad program categories:

1.  Fossil Fuel Conversion -- conversion of coal and gas to electricity
    and other fuels, and

2.  Fossil Energy Resources -- domestic oil and gas supply.

The nature of these programs and their benefits are very different,
which has led to rather different approaches for calculating their
future benefits.

**3.2.1 Fossil Fuel Conversion**

FE has been estimating future benefits for many years. The benefits
resulting from deployment of advanced fossil generation and
environmental technologies (including CO~2~ sequestration) are
quantified in terms of dollars saved due to deployment of less expensive
technologies:

1.  Increased GDP due to the use of lower cost technology;

2.  New direct and indirect domestic jobs;

3.  Reduced emissions of SO~x~, NO~x~, and carbon dioxide; and

4.  Dollars of exports.

These benefits estimates are based on the the Energy Information Agency
(EIA) forecast of electricity demand growth through 2020, and
extrapolation of this forecast beyond 2020 to estimate the demand for
new gas-fired and coal-fired electric generators in the U.S. through
2050. In addition, several studies (by WEFA, Resource Dynamics Corp,
Parsons, and others) are also used which estimate the export of clean
coal technologies, the use of advanced emissions control technologies,
and the associated macroeconomic effects. In this analysis, no effort is
made to parse the credit for the benefits between industry and
government.

With respect to estimating the benefits of deploying advanced fossil
generating technologies in the U.S., the methodology assumes that all
new fossil plants built after 2015 are high-efficiency Vision 21 plants
with sequestration of carbon emissions. The existing inventory of fossil
plants is assumed to retire on a business-as-usual schedule. Generation
from new Vision 21 plants is assumed to be 10% less expensive than an
equivalent amount of generation from new conventional fossil plants.
Carbon emissions reductions are estimated by comparing total emissions
from the electric generation sector through 2050 with base case carbon
emissions assuming new conventional gas and coal plants deployed without
sequestration.

**3.2.2 Fossil Energy Resources**

Estimates of benefits of advanced technology for oil and gas production
depend heavily on the nature of the existing resource. Extensive efforts
have been undertaken to measure benefits using economic and engineering
models. FE developed the first oil model in 1984 in conjunction with the
National Petroleum Council. Model development included input from
experts from industry, universities, government, and private nonprofit
organizations. Over time, this model has evolved into an integrated
suite of databases and analytical computer models, all striving to
emulate the actual complexities of producing domestic oil. This
engineering and economic model was joined by another model to mimic
domestic gas production. The models contain actual data from the
Nation's oil and gas wells and fields.

Within appropriate bounds set by other databases, like those from EIA,
the two engineering based models analyze the amount of oil and gas on a
field by field basis to calculate how much was, is, or will be produced
over a specified time interval. This is the oil and gas world
[without]{.underline} FE R&D.

FE R&D includes a portfolio of oil and natural gas projects. For each
project FE develops key parameters such as the probability of success,
likelihood of the technology penetrating into the marketplace, and the
size of the applicable oil or gas resource; and analyzes the potential
impacts. Results for individual projects are then fed into the oil and
gas models to estimate impacts [with]{.underline} FE R&D. The difference
between these two simulations provides estimates of the benefit of FE
R&D.

The models use a consistent methodology, and the results are
periodically peer reviewed and projects validated. The primary benefit
measured is the additional oil and gas resulting from FE R&D. This
approach has also allowed FE to estimate economic benefits and new
direct and indirect jobs, as well as certain environmental benefits,
like emission and effluent impacts.

Currently, staff at the National Energy Technology Laboratory are
working to integrate separate models into a single one to permit more
accurate and efficient benefit computations.

**3.2.3 Future Directions**

Today's standard decision-making tool is Net Present Value (NPV ). NPV
calculates the value of a project by predicting future payout, adjusting
for the risk, and subtracting the investment cost. NPV does not account
for the ability of government and industry leaders to react to new
circumstances. FE has begun to explore the use of Real Options to better
quantify the value of its programs. Real Options is an investment
valuation tool that accounts for and credits flexibility. Given a highly
unpredictable future, the Real Options approach assesses the value of
spending money now so that in the future there is better information on
which to exercise the option to deploy, discard, or continue to develop
a technology.

### 3.3 Overview of Methods Used by the Office of Science[^8]

The challenge of evaluating science programs has been the subject of
widespread attention for many years, including initial recognition
within the Government Performance and Results Act, and more recently by
significant focus of the National Academy of Science's Committee on
Science, Engineering, and Public Policy (COSEPUP), as well as the White
House Office of Management and Budget's present efforts to develop
separate evaluation criteria for basic research. Problems arise because
of the inherent difficulties in modeling the non-linear innovation and
knowledge diffusion process \-\-- knowledge that is the primary goal of
the research and the principal output of science. Consequently, the
Office of Science draws from a broad array of methods, and is
researching and developing many more, to evaluate its basic research
programs.

Across the various groups exploring this issue, some elements appear to
emerge as common themes.

-   Evaluation is more successful around the process and outputs of
    science (sometimes referred to as process-outcomes)

-   Expert review forms the cornerstone for evaluating the process and
    outputs of science, and the more useful dimensions of analysis
    include:

    -   Quality of the science

    -   Relevance of science

    -   Leadership and/or Performance of the science

-   Evaluating the outcomes (and estimating the benefits) of science,
    while a desirable goal, cannot completely and effectively account
    for the complexities of knowledge diffusion and the broad range of
    interactions and factors that contribute to innovation.

-   Techniques that explore the downstream impacts are often incomplete,
    and provide only focused insights into the benefits through various
    indicators.

Listed below in Table 1 are evaluation techniques that are currently
employed within the Office of Science (SC) to evaluate, either
retrospectively, prospectively, or both, the science programs.
Collectively, these techniques address the basic research and the
infrastructure stewardship role of SC, the latter consisting of the vast
array of scientific user facilities and premier tools of science, the
human capital, and the critical support and sponsorship for core science
capabilities that SC contributes to the nation's public science
enterprise.

Beyond these more traditional tools, the Office of Science has been
sponsoring research and investing staff resources in a variety of
promising areas intended to help illuminate the pathways of knowledge
diffusion and, ultimately, improve the evaluation of science outcomes.
Although no individual technique appears a panacea for the
aforementioned difficulties, collectively they seem to offer promise,
providing a powerful, complementary perspective to the continued
mainstay of expert review.

Additionally, the categorical/topical structure of one item, Case Study
Enhancements, further reveals methodological areas that SC is exploring,
both through continued research, and through pilot studies based on SC's
recently completed "Science Manager's Resource Guide to Case Studies".
Methodologies include:

> •Classic, Interview- and Records-Based Method
>
> •Expert Review Method
>
> •Historical Tracing Method
>
> •Bibliometrics and Citation Methods
>
> •Content Analysis Methods (Co-Word, Data Tomography, etc.)
>
> •Sociometric/Social Network Methods
>
> •User/Participant Survey Method
>
> •Benefit-Cost (Cost-Benefit) Methods
>
> •Statistical/Econometric Methods
>
> •Key Indicator Method

**4. METHODS USED BY OTHER FEDERAL AGENCIES TO MEASURE THE
OUTCOME-RELATED PERFORMANCE OF THEIR PROGRAMS**

**4.1 Measuring the Benefits of the Advanced Technology Program**[^9]

**4.1.1 ATP's Mission and Operations**

The ATP partners with industry to accelerate the development of
innovative technologies for broad national economic benefit. The
program's focus is on co-funding collaborative, multi-disciplinary
technologies and enabling technology platforms that appear likely to be
commercialized, with private sector funding, once the high technical
risks are reduced. Industry-led projects are selected for funding in
rigorous competitions on the basis of technical and economic merit.
Since 1990, ATP has co-funded 581 projects, with 1,250 participants and
another 1,200 subcontractors. More than 60 percent of the projects are
led by small businesses. More 160 different universities and more than
20 national laboratories participate. ATP has committed over \$1.8
billion of funding to these projects, and industry has committed another
\$1.75 billion. Although ATP competitions are open to all technologies,
to date 35 projects, with \$30 million in ATP funding, are directed at
solutions to the nation's energy challenges.

**4.1.2 ATP's Multi-Component Assessment Program**

Evaluation has been a central part of ATP operations from the beginning,
as a management tool to provide feedback to project selection and
program operations and to meet requests from external sources for ATP
program results.

The ATP has developed a multi-component evaluation strategy to provide
measures of progress and performance matched to the stage of project
evolution:

-   for the short-term, from the time of project selection and over the
    course of the ATP-funding period;

-   for the mid-term, as commercial applications are pursued, early
    products reach the market, and dissemination of knowledge created in
    the R&D projects occurs; and

-   for the longer-term, as more fully-developed technologies diffuse
    across multiple products and industries, with related net impacts on
    formation of new industries, job creation, and U.S. economic growth.

Current approaches to evaluation include:

1.  statistical profiling of applicants, projects, participants, and
    technologies

2.  progress tracking of all projects and participants (through a
    business reporting system and other surveys)

3.  status reports of all completed projects, with a performance rating
    against ATP's mission

4.  detailed microeconomic case studies of selected projects, including
    the following metrics: public rate of return (return attributable to
    ATP), private and social rates of return, net present value, and
    benefit-cost ratio

5.  econometric and other statistical studies of innovation,
    productivity, and portfolio impacts, including use of control groups

6.  macroeconomic analysis extending firm and industry level impacts to
    impacts on GDP and national employment

7.  development and testing of new assessment models and techniques.

The ATP's Business Reporting System, a unique internal database created
in 1993, tracks progress during and after the performance of each
project. This system first identifies the project's goals and expected
commercial advantage, its strategies for commercialization, and
collaborative activities and experiences of its members, with emphasis
on comparing what ATP funding makes possible with what would likely
occur in the absence of ATP funding. Each project is evaluated in terms
of the effect of ATP on the project's timing, scale, scope, risk level,
ability to do long-term R&D, and the ability to attract private
investment dollars. After the project is finished, the ATP's Economic
Assessment Office follows up with regular surveys and studies of
progress in commercialization and knowledge dissemination. For select
projects and programs, detailed benefit-cost analyses are performed.

**4.1.3 ATP's Evaluation Results**

Recently completed mini case studies of ATP's first 50 completed
projects provide an evaluation of the performance of each project using
ATP's Composite Performance Rating System. Each project is scored on a
set of measures of knowledge creation and dissemination and progress
toward commercial goals.

ATP's Composite Performance Rating System[^10]

+-----------------------------------+-----------------------------------+
| **Knowledge Creation and          | **Commercialization Progress      |
| Dissemination Measures**          | Measures**                        |
|                                   |                                   |
| Technical awards                  | New product/process in market or  |
|                                   | expected soon                     |
| Collaborations                    |                                   |
|                                   | Attraction of capital             |
| Patent filings                    |                                   |
|                                   | Employment gains                  |
| Publications and presentations    |                                   |
|                                   | Business awards                   |
| New product/process in market or  |                                   |
| expected soon                     | Outlook                           |
+-----------------------------------+-----------------------------------+

The results from all the measures are used to construct a composite
performance score to indicate the overall project effectiveness against
ATP's mission (measured 2-3 years after the end of ATP funding). The
result is a four-star system of ratings, with scores ranging from zero
to four stars. The results of this analysis for the first 50 completed
ATP projects find that sixteen percent of the projects are top-rated in
terms of overall project performance, with four stars. Twenty-four
percent are in the bottom group of zero or one stars. Sixty percent make
up the middle group.

Not all ATP projects are fully successful. The program's emphasis on
funding high-risk, technology development that the private sector is
unwilling and unable to fund alone dictates that most projects will fail
to accomplish all their goals. Some projects are stopped before
completion of the funding period. Others fail to meet all their
technical goals, or encounter business difficulties before the
technologies are commercialized.

Evaluation studies to date suggest ATP is delivering results: 1)
Estimated net benefits (including projections) from just a few projects
exceed the total cost of ATP to date; 2) a vast proportion of these
benefits extends beyond the organizations who received ATP funding, in
keeping with the program's emphasis on generating significant spillover
effects; and 3) there is substantial evidence that ATP has made a
difference in the ability of the nation to realize these benefits.

The ATP evaluation studies are available on the website at:

2.  **Federal Railroad Administration's (FRA's) R&D Project Evaluation
    and Investment Analysis**[^11]

At the urging of the Transportation Research Board (TRB) Committee for
Review of the FRA R&D Program, FRA has developed a structured process to
identify safety research areas and select specific safety R&D projects
for funding. The approach consists of five logical steps that were
applied to the FRA's safety R&D program. Subsequently, as new
information becomes available about sources of hazards, the logical
steps may be followed for specific types of safety hazards to add to the
list of potential safety R&D projects.

**Step 1: Review of Rail Industry Historical and Potential Harm**

The first step in the FRA safety R&D project development and selection
process is a review of recent rail industry harm data and an assessment
of causes of potential safety hazards. In this context, harm is
considered to be the aggregate cost of fatalities, injuries and property
damage due to rail accidents. Historical hazard data is compiled in FRA
rail accident databases and accident investigation reports. Potential
for future safety hazards can be understood by reviewing rail industry
operating trends with expert knowledge of how railroad accidents occur.

The four relevant databases which hold historical rail incident data are
the FRA\'s Rail Accident/Incident Reporting System (RAIRS), Highway-Rail
Grade Crossing Accident/Incident Database, Railroad Injury and Illness
Summary Database, and the Research and Special Projects
Administration\'s (RSPA\'s) Hazardous Materials Incident Database. The
information in these databases is very detailed in terms of accident
causes. However, these databases, typically, do not address specific
contributing causes that result in railroad accidents or incidents.

Detailed accident reports from the National Transportation Safety Board
(NTSB) and the FRA are the most important source of information,
compiled by experts, about accident circumstances that contribute to
hazards. Since accident databases and accident reports can only reflect
historical accident causes and circumstances, railroad industry
operational trends must also be taken into account. In this way, an
understanding of causes of hazards that are not reflected in the
historical databases can be developed.

**Step 2: Conduct Failure Analysis**

For a given cause of an accident, or factor contributing to hazard,
fault-tree logic is applied to identify specific items to be addressed
by countermeasures. These specific items represent points along the
accident chain-of-events at which the accident, or subsequent harm, or
both, could have been prevented. Thus, the fault tree approach permits
identification of multiple causes of accidents, enriching the
information gleaned from the accident databases, which typically assign
a single, primary accident cause. Countermeasures are proposed with the
goal of breaking the accident or hazard chain-of-events at the
identified points.

These countermeasures are advanced with an understanding of current
regulatory and industry practices for the relevant area of rail
operations.

Types of possible countermeasures include:

-   New or revised regulations

-   Industry standards and best practices

-   Equipment and infrastructure improvements

-   Enforcement

-   Education.

**Step 3: Survey Government and Industry Countermeasures and R&D
Requirements**

Once specific countermeasures are identified, FRA R&D will review
current and potential industry and government countermeasures to
identify and assess areas of technological opportunity for R&D. That is,
FRA R&D will identify countermeasures that would be enabled by R&D. For
example, a potential operating rule may need research into the train
speed regimes at which a particular type of train control system affords
safe operation.

**Step 4: Develop and Rate Individual Projects**

For each countermeasure that may be aided by R&D, one or more R&D
project summaries are developed to describe projects that provide
information to enable the countermeasures. The project summaries are
structured descriptions of projects that will be used to compare and
select projects during R&D program development. Project summaries
address expected outputs and outcomes, project costs and durations, as
well as implementation issues for project results. Based on the project
summaries, projects are then rated according to objective criteria for
expected contribution to safety and likelihood of success. For a given
program area, these project ratings are plotted in two dimensions
(likelihood of success versus contribution to safety) to provide a
high-level comparison tool for the project selection process.

**Step 5: Select Projects and Assign to Program Areas**

The last step in the FRA safety R&D program development process entails
selecting projects for each program area based on the two-dimensional
plots and project summaries. The goal is to select the best research
opportunities available to obtain the best return on investment possible
from the FRA R&D budget. That is, the most highly rated projects,
regardless of program area, are selected to develop budget request
estimates. Once the budget has been finalized, the projects are
revisited, and funding levels and schedules are adjusted appropriately.
The FRA R&D budget request, for each program area, becomes the sum of
the funding required for each of the selected projects in the program
area.

This process, augmented by Congressional directives as well as by
requirements to carry out research in direct support of rulemakings
mandated by law, was used by FRA in the preparation of its FY 2002 and
FY2003 budget requests and its soon to be released *Five-Year RD&D
Plan*. The FRA anticipates that it will continue to use it in the future
in the development of its R&D program.

**5. NATIONAL RESEARCH COUNCIL\'S METHODOLOGY FOR DEFINING THE BENEFITS
OF ENERGY R&D**[^12]

There are many ways of addressing the stated objectives of the
conference. However, the recent study done by the National Research
Council (NRC) Committee on Benefits of DOE R&D on Energy Efficiency and
Fossil Energy provides an important context for the conference. In
legislation approving the FY 2000 energy R&D budget for DOE's energy
efficiency and fossil energy programs, Congress directed the NRC to
conduct an evaluation of the benefits that have accrued to the nation
since 1978 from these programs. This charge was *a retrospective* look
at the benefits that had been realized to date. The NRC did not consider
any prospective benefits from the programs.

To undertake its task, the NRC developed and implemented a
methodological framework to estimate the retrospective benefits of
individual energy efficiency and fossil energy R&D programs. Thus, the
framework we suggest as a starting point for discussions at the
conference is adapted from the one developed in the NRC study. In this
section of the paper, we review this NRC methodology. In Section 6, we
turn to a discussion of \"strawperson\" ideas for building on and
extending NRC framework because of the need to consider R&D programs
prospectively, for the purposes of GPRA and program planning.

**5.1 Methodological framework**

To undertake this task, the NRC first had to develop a methodology or
framework to assess what were the benefits realized. The NRC recognized
that there are many different types of benefits. The NRC believed that
the benefits considered foremost must meet DOE's stated mission - which
the NRC culled by reviewing statures and public mission statements
issued by DOE. The NRC viewed the public benefits as:[^13]

-   Economic: Measured by the change in the market value of goods and
    services that are produced under "normal" economic conditions
    resulting from the introduction of a technology stemming from DOE
    research. The benefit is measured net of all public and private
    costs, and can typically be reflected in market prices of goods or
    services in the economy. The NRC considered the costs for the
    program - the costs borne by DOE and private industry in conducting
    the R&D -, as well as the costs of the technology borne by the
    end-user or consumer so that the net benefits were assessed. In
    essence, an economic benefit is reducing the cost of energy
    services.

```{=html}
<!-- -->
```
-   Environmental: Based on changes in the quality of the environment
    because of DOE research. The benefit is typically not measured
    directly by changes in market prices, but rather by some measure of
    the value society is willing to place on changes in the quality of
    the environment.

-   Security: Measured by changes in the probability or severity of
    abnormal energy- related events that would adversely impact the
    overall economy, public health and safety, or the environment.
    Traditionally, this was an oil focused concern, but recently has
    expanded to include the reliability and security of the energy
    supply infrastructure- both electric and gas.

The NRC recognized that R&D might lead to benefits even when a
technology developed by that R&D does not enter the market immediately
or to a significant degree. This lack of commercial deployment might due
to changes in the forecasted economic or policy conditions, or to
technological barriers. To account for this uncertainly, and to reflect
the different degrees of technology development in the snapshot in time
the NRC was evaluating the benefits, the NRC established three
categories:

-   Realized Benefits: These benefits have been realized or are almost
    certain to be realized in the near future - the technology is
    developed and economic and policy conditions are favorable for
    commercialization. The NRC included all lifecycle benefits of the
    units of the evaluated technologies that were projected to be
    installed by the year 2005.

-   Option Benefits: This category covered technologies that are fully
    developed but for which existing economic or policy conditions are
    not likely to be favorable for commercialization. To be considered
    an option by the NRC, the technologies needed to be favorable for
    commercialization under some credible or plausible circumstances.

-   Knowledge Benefits: R&D, whether successful or not, typically
    produces knowledge benefits. This scientific knowledge produced by
    R&D is a key component of DOE's mission. The NRC considered as
    knowledge benefits scientific knowledge and useful technological
    concepts that have not yet been incorporated into commercialized
    results from the R&D program but hold promise for future use or are
    useful in unintended applications. These hold value over and above
    that accounted for in the other areas of realized and option
    benefits.

The following table reflects the interplay of the stages of technology
development and economic and policy conditions as the NRC conducted its
retrospective review of benefits.

+-------------------+----------------+----------------+----------------+
| *Technology*      | *Technology    | *Technology    | *Technology    |
|                   | Developed*     | Development in | Development    |
| *Development*     |                | Progress*      | Failed*        |
|                   |                |                |                |
| ###               |                |                |                |
| ##### *Economic/* |                |                |                |
|                   |                |                |                |
| *Policy           |                |                |                |
| Conditions*       |                |                |                |
+-------------------+----------------+----------------+----------------+
| *Will be          | Realized       | Knowledge      | Knowledge      |
| favorable for     | benefits       | benefits       | benefits       |
| c                 |                |                |                |
| ommercialization* |                |                |                |
+-------------------+----------------+----------------+----------------+
| *Might become     | Options        | Knowledge      | Knowledge      |
| favorable for     | benefits       | benefits       |                |
| c                 |                |                | Benefits       |
| ommercialization* |                |                |                |
+-------------------+----------------+----------------+----------------+
| *Will not become  | Knowledge      | Knowledge      | Knowledge      |
| favorable for     | benefits       | benefits       | benefits       |
| c                 |                |                |                |
| ommercialization* |                |                |                |
+-------------------+----------------+----------------+----------------+

**5.2 Derivation of columns for the benefits matrix**

This assessment of the types of benefits led the NRC to construct the
following matrix to evaluate the R&D programs conducted by DOE. It
reflects the interplay between stages of technology development,
economic and policy conditions, and economic, environmental and security
benefits.

Matrix for Assessing Benefits and Costs

  --------------- ----------------- ------------------ ------------------
                  Realized Benefits Options Benefits   Knowledge Benefits
                  and Costs         and Costs          and Costs

  Economic                                             
  Benefits and                                         
  Costs                                                

  Environmental                                        
  Benefits and                                         
  Costs                                                

  Security                                             
  Benefits and                                         
  Costs                                                
  --------------- ----------------- ------------------ ------------------

Since the NRC study was only retrospective and charged with evaluating
benefits achieved to date, only "realized" benefits and costs were
evaluated \-- there was no column for "prospective" benefits or costs.
If DOE uses this framework for program planning and budgeting looking
toward the future, a column of prospective benefits and costs should be
added to the matrix.

To do its evaluation, the NRC conducted case studies of 22 fossil energy
and 17 energy efficiency programs. The case studies were evaluated both
quantitatively and qualitatively to fill out the matrix for each
program. The full NRC report is "Energy Research At DOE -- Was it Worth
it? Energy Efficiency and Fossil Energy Research 1978 to 2000." One of
the NRC's recommendations was that DOE should convene a workshop to
refine and address issues the NRC identified in using the framework so
DOE could begin the implementation process.

**5.3 Major Issues Considered -- All not necessarily resolved**

**5.3.1 Establishing a Baseline for Realized or Projected Benefits**

Looking both retrospectively as well as prospectively in assessing an
R&D program, the technology under evaluation must always be compared
with the "next best alternative." Technologies change and improve over
time, and it was difficult for the NRC to retrospectively assess over
different time periods what the next best alternative was.

Looking prospectively, this is an even more difficult challenge. One
needs to project what the next best alternative technology to the R&D
program under consideration is projected to be over a long period of
time - perhaps decades. This requires judgment and rigor, but the NRC
did not tackle this issue since it was only charged with a retrospective
evaluation.

Indeed, in its charge to evaluate realized benefits to date, the NRC was
extremely limited in considering the projected future installations of
any technology. The NRC imposed this restriction due to uncertainties
about both technology development and future economic and policy
conditions.

The NRC adopted a "2005 year rule" -- it allowed the lifecycle benefits
of all the technologies that were projected to be installed by 2005 to
be considered in its evaluation of "realized benefits", but none after
that date. The benefits for a technology installed in for example 2003,
with a 20 year life, would get its life-cycle benefit assessment in the
realized benefit column. Any technology installed in 2006 or after got
none.

Looking prospectively, a baseline must be established to estimate
prospective benefits. This baseline is a projection based on the "next
best technological alternative." Then a projection based on the
development of the new technology must be developed at a modeled
projection rate. Finally, the effect of the government program must be
considered and the differences in benefit curves somehow allocated if it
is not found that "but for" DOE this technology never would be
developed.

This conference needs to extend the NRCs work and develop methodologies
that DOE can use to address these issues, recognizing both data
constraints and cost considerations in DOE's planning activities. The
primary reason for extending the NRC framework is that the scope of its
study was limited to being a retrospective analysis, whereas, for the
purposes of GPRA and program planning, prospective estimates are needed
on the benefits of R&D programs.

**5.3.2 Effect of Government Program**

To evaluate the benefits of the government R&D program, one must
evaluate what the effect of the program was or is projected to be. The
total impact or benefit of a technology is not necessarily the effect of
the government program of R&D for that technology.

For example, if the government provided 10% of the funding to develop a
technology that has an estimated \$100 million of economic benefits to
date - should the government expenditure get credit for all \$100
million of benefits? In the vast majority of cases, it is highly
unlikely that "but for" the DOE R&D program, the technology never would
have been developed. The benefits attributable to a technology and the
benefits attributable to the government program that helped develop that
technology need to be separately considered.

The NRC developed a rule of thumb for considering the effect of the
government R&D program. It assumed the DOE R&D program advanced the
introduction of new technology into the market by 5 years. It adopted
this rule in all the case studies examined for consistency. In each
case, it did find a significant DOE role. In essence, the NRC compared
two traditional "S" shaped" market penetration curves - one starting 5
years earlier than the other due to the effect of the DOE program. The
curves were the actual penetration observed in the market and a similar
shaped one starting 5 years later. The benefits of the program were the
difference in the sum of benefits under the benefit curves due to the
technology.

Questions abounded about whether this was a reasonable assumption, if it
should vary by case study, or if there were other ways to assess the
impact of the government program. What factors, such as the percentage
of government funding, stage of technology development or nature of the
industry, are appropriate to consider if the effects should vary on a
case by case basis? How are these vetted for reasonableness? The issue
on assessing the effect of the government program needs more
consideration and relates to the question of baseline discussed
previously in Section 5.3.1.

**5.3.3 Nature of Benefits**

In evaluating benefits, the NRC found that most of the focus of DOE has
been on economic benefits. Several points need to be made, however,
about all categories of benefits and the issues that arose in the NRC's
examination.

In evaluating economic benefits, The NRC did not believe it appropriate
to consider any macroeconomic stimulation to the economy or the creation
of jobs as an economic benefit from any R&D initiative. Similarly, any
potential trade benefits were not deemed appropriate to include.

On environmental benefits, the NRC considered the saving in costs from
achieving any given standard of emission control as an economic benefit.
Environmental benefits result only if there is a net improvement in the
quality of the environment.

The NRC did calculate the tonnage of pollutants that the technology
avoided. The NRC looked at the wide range of values that have been done
assessing environmental damage for various pollutants, but declined to
adopt any one of them to attribute an economic value to the pollution
prevented.

Security benefits -- both oil and energy infrastructure reliability and
safety- were difficult for the NRC to address. The security benefits
were considered as reductions in the probability or severity of an
energy disruption. Both quantitatively, and even qualitatively, these
were hard to evaluate for the R&D programs examined.

The barrels of oil saved or produced by DOE sponsored technologies could
be calculated in most instances, but the linkage to security and how to
evaluate its significance was difficult. This linkage to security and
assessment in the area of energy infrastructure was even more difficult.
How significant was the benefit achieved by the program? What were the
criteria for how should it be measured? These issues need more
resolution.

Option benefits were precisely defined as technologies for which
development work was done and successful, but for which economic or
policy conditions were not present to enable it to currently enter the
market to a significant degree. To be considered as an option benefit,
the technology has to be able to enter the market under situations that
have "credible" possibilities of occurring. It may be viewed as a form
of prudently purchased insurance.

The NRC found that several of the DOE programs had developed options,
but this was a retrospective judgment. When undertaken, these programs
were trying to develop technologies that were projected to enter the
market. The NRC did not attempt to value these option benefits either
quantitatively or qualitatively.

Especially if it is deemed appropriate for DOE to develop options as a
goal in its program development and part of its portfolio, then a better
a priori assessment of option values is needed. A process for defining
plausible or credible circumstances needed to make the option viable
also needs to be developed.

The NRC considered knowledge benefits as scientific knowledge and useful
technological concepts resulting from R&D that have not yet been
incorporated into commercial results of the program but hold promise for
future use or unintended applications. This category was very broad and
is almost an "everything else" category of benefit. More precision is
needed with criteria to evaluate how significant the knowledge gained is
and how it should be categorized.

**6. BUILDING ON THE NATIONAL RESEARCH COUNCIL\'S STUDY \-- IDEAS ON
FRAMEWORK AND METHODS FOR ESTIMATING THE BENEFITS OF GOVERNMENT R&D**

The National Research Council (NRC) report offers a point of departure
for efforts to improve methods for estimating the benefits of
government-sponsored energy R&D. With this conference, we build on that
study, addressing several of the areas that the NRC committee
identified, which were discussed in Section 5.3:

(a) prospective analysis of benefits,

(b) option valuation,

(c) knowledge valuation, and

(d) security, economic, and environmental benefits.

Each of these four areas is the subject of one of the workshops in the
conference. Workshop A focuses on the prospective-benefits column in the
benefits matrix discussed in Section 6.1. Workshop B focuses on the
option column. Workshop C focuses on the knowledge column. And Workshop
D addresses the rows of the matrix. The format of the conference, which
is described more fully in Section 8, is geared to having participants
offer their suggestions for building on the NRC study in these four
areas. All four workshops also have a session on the additional topic of
how measures of benefits might be used in GPRA-related program planning,
evaluation, and decision-making.

To give examples of how conference participants might consider building
on the NRC study, we offer some ideas in each of the four areas.

**6.1 PROSPECTIVE ANALYSIS OF BENEFITS**

**6.1.1 Definition of Expected Prospective Benefits**

By *prospective* analysis, we mean one that looks into the future. In
contrast, the scope defined for the NRC study was limited to a
*retrospective* analysis of the benefits of past R&D programs. As
described in the previous section, the degree of certainty and time
frame of these benefits were classified as being \"realized,\" including
the foreseeable future to the year 2005, \"option\" benefits, and
\"knowledge\" benefits. Prospective benefits were not of primary
concern, given the stated scope of the NRC\'s study. However, they are
of the utmost concern in program planning and decision making.

Before proceeding to consider the question of prospective benefits, we
suggest that the NRC committee\'s definition of \"realized benefits\"
could be modified. The NRC definition includes benefits that have
already occurred (i.e., this \"scenario\" has a probability of occurring
of 1.0), *plus* those from a \"best guess\" forecast of the likely
market penetration and sales of the technology in question, up to the
year 2005. Option and knowledge benefits are considered to be *in
addition* to these realized benefits.

The definition of realized benefits that we suggest differs from the NRC
definition. In the NRC study, the benefits between now and the year 2005
(or, more generally, five years from now, for example) would be part of
the realized benefits. Whereas, we suggest that these benefits could be
part of the prospective benefits, which would extend from the current
year to at least the year 2020, or whatever time horizon is required.
The definition we suggest above for realized benefits would include only
the benefits from deployments of the technology, developed at least in
part by the R&D program, up to the current time.

Consistent with the NRC study, we suggest that the benefits of the
technology in question be net of its R&D costs, *as well as* the
benefits of the next-best alternative. That is, a next-best alternative
\-- perhaps a hypothetical one \-- would be identified and its economic,
environmental and security benefits would be subtracted from those of
the technology in question.

In a prospective context, *all* benefits are prospective, including
option and knowledge benefits. But for the purpose of this discussion,
we generally use the term \"expected prospective benefits\" to refer to
those under the most likely scenario. To account for these prospective
benefits, we suggest adding a column to the NRC benefits matrix. This
idea is illustrated in the following figure.

Modification of National Research Council Committee\'s Framework

For Identifying the Benefits of Energy R&D

+-------------+-------------+-------------+-------------+-------------+
|             | Realized    | Expected    | Option      | Knowledge   |
|             |             | Prospective |             |             |
|             | Re          |             |             |             |
|             | trospective |             |             |             |
+-------------+-------------+-------------+-------------+-------------+
| Economic    |             |             |             |             |
+-------------+-------------+-------------+-------------+-------------+
| En          |             |             |             |             |
| vironmental |             |             |             |             |
+-------------+-------------+-------------+-------------+-------------+
| Security    |             |             |             |             |
+-------------+-------------+-------------+-------------+-------------+

We suggest that the expected prospective benefits could be defined as
those expected from future deployment of a technology developed as a
result of R&D, under the projected baseline set of market and policy
conditions, compared to the expected conditions under the same projected
baseline, without the technology. Thus, estimation of expected
prospective benefits requires a baseline characterization of future
energy markets without the government research, and an estimation of how
baseline markets will react to the new or accelerated technology,
including its expected market penetration.

We suggest that conference participants consider several alternative
ways of defining and estimating the expected prospective benefits:

(a) Benefits in the reference case (sometimes referred to as the \"most
    likely scenario\") of an energy-market forecasting model, that would
    be augmented by the technology under consideration,

(b) Mathematical mean or expected value of the benefits, as calculated
    using an energy-market forecasting model, again with each scenario
    being augmented by the technology under consideration,

(c) Benefits based on extrapolation of technology deployment, in cases
    where there is already limited deployment,

(d) Benefits based on subjective assessments of the likely market
    penetration of the technology,

(e) Subsumed in the option value, which includes all prospective
    benefits (except knowledge benefits).

\(a\) [Expected Prospective Benefits are Those in the Projected
Baseline, but *With* the DOE R&D Program Under Consideration, Versus
Those in the Projected Baseline *Without* the Program]{.underline}

In this first possible definition, the expected prospective benefits are
the economic, environmental, and security benefits that result from the
future deployment of the technology in question, under the projected
baseline set of conditions, versus the situation under the same
projected baseline, without the technology.

This definition is analogous to that used by the NRC committee for
retrospective benefits in that the benefits of the technology in
question are net of those of the next-best alternative. In the
prospective case, the next-best alternative is reflected in the
projected baseline, which includes technical improvement, without the
technology in question. The question of defining the projected baseline
is addressed in Section 6.1.2.

The benefits of a technology depend considerably on its technical
performance, particularly those aspects that affect its economic,
environmental, and security impacts. The actual performance of a new
technology will generally differ, at least to some extent, from its
design or target level. We suggest here that the projected performance
of the technology could set as the most likely actual performance,
rather than the design or target level.

The projected market penetration of a new technology is another key
input parameter that greatly affects the magnitude of its expected
prospective benefits. In this regard, we suggest that it could be
preferable to characterize the attributes of the technology and to use a
model to project how well it will do in the market, than to simply
assume a certain level of market penetration after some number of years.

The model used for this purpose would be the same one used for the
projected baseline. Then, the Reference Case run of that model would be
the projected baseline, *without* the technology in question. A run of
the same model, augmented with a simple representation of the technical
performance and supply (curve) of the technology in question, would be
the case *with* the R&D program. The difference between the economic,
environmental and security conditions in these two runs would be the
expected prospective benefits of the technology.

We suggest that the results of these calculations, which are projections
of the annual net benefits of the technology, could be reported in the
following ways (i.e., both ways):

-   Total net benefits in five-year intervals, in \"real\" dollars, but
    without discounting, *and*

-   Net present value, in which the estimated future benefits and costs
    over the operating life of equipment deployed are discounted.

The first way of reporting the results is more intuitively informative
for most people.[^14] The second way of reporting the results provides a
single value for the present value of the stream of net benefits.

We also suggest that DOE offices could consider attaching a probability
to these expected prospective benefits occurring. This probability would
be the probability of the projected baseline occurring (which could be
set DOE-wide) multiplied by the probability of technical success under
the projected baseline conditions (which would be estimated on a
program-by-program basis following a uniform set of guidance). These two
probabilities reflect market risk and technology-specific risk,
respectively. It is obviously difficult to speculate on these
probabilities because there is such great \"uncertainty about these
uncertainties.\" However, if done, these estimates could be used to
improve R&D portfolio management. Portfolio management involves risk
management.

The option benefits, incidentally, would be those associated with
scenarios other than the reference case. Methods for estimating the
option benefits would still need to be determined (refer to Section
6.2).

\(b\) [Expected Prospective Benefits are the Mean Value]{.underline}

With this alternative definition, the expected prospective benefits are
those that correspond to the mean value in a mathematical sense. A range
of scenarios is considered for the market penetration and use of the
technology. This range might be represented by a probability
distribution(s) of values for key parameters that affect the
forecast.[^15] Alternatively, the range might be represented by a
limited number of alternative scenarios.

With probability distributions, a Monte Carlo analysis is a common
approach for calculating expected outcomes. With a limited number of
scenarios, each can be assigned a probability value, with the values
adding to one. In either case, the expected present value of the net
benefits can be calculated.

The option benefits would in this case be either *zero* (because *all*
possible scenarios including the very unlikely ones are taken into
account in calculating the mean value), or treated as in the NRC study
(if the sum of the scenario probabilities is less than one).

\(c\) [Benefits Based on Extrapolation of Technology
Deployment]{.underline}

In cases where there is already limited deployment, an approach that
could be used is to use data on the market penetration of the technology
thus far. These data could be used to estimate, or example, the
parameters of a logistic curve commonly used to predict the diffusion of
new technologies.

The calibrated equation for market penetration could then be used to
predict the future market penetration of the technology. The approach is
relatively straightforward but has a major disadvantage that it cannot
be used unless sufficient data exist to estimate a statistical model (at
least thirty years of annual data \-- in which case the technology is
hardly a new technology in need of government R&D).

\(d\) [Benefits Based on Subjective Assessment of Likely Market
Penetration]{.underline}

Another approach is to obtain predictions of the likely market
penetration of the technology over time. For example, predictions could
be obtained on the ultimate market penetration of the technology and the
time expected to reach that level. Then, interpolation can be done,
assuming a specific type of equation, such as a logistic curve.

This approach is relatively simple to implement, but has the
disadvantage that it is difficult to obtain unbiased subjective
predictions of market penetration levels. For example, R&D program
managers likely know the potential of the technologies they are
developing as well as anyone, but they are also likely to be advocates
of these technologies. The predictions are also unlikely to be
consistent with the macroeconomic, policy and regulatory assumptions
used in DOE\'s other models that are used to help develop its forecasts
and policy positions.

\(e\) [Expected Prospective Benefits Are Subsumed in the Option
Value]{.underline}

In this case, the definition of an option, which we suggest, differs
from the NRC definition. With the definition we suggest that there is
considerable uncertainty about future energy markets and technologies;
and that R&D could lead to an option(s), that is not necessarily a fully
developed technology, that has value. The NRC definition is narrower and
states that options are fully developed technologies that could be
deployed, but for existing economic or policy conditions are not likely
to be favorable for their commercialization.

Thus, in the definition we suggest, the reference case itself might be
improbable. This uncertainty could be about: whether the R&D will lead
to some commercialized product; the market penetration not only of this
product but also of the whole category of similar products; and the many
macroeconomic, policy, and regulatory possibilities.

With this approach, there is no expected prospective benefit, only
option benefits (in addition to any realized benefits and knowledge
benefits). The definition of option benefits is generalized to include
all possible scenarios, not just the unlikely ones.

**6.1.2 Projected Baseline**

To estimate the prospective benefits of a government program, we suggest
two key steps. The first step is to calculate the benefits of the
technology associated with that program. The second step is to calculate
the portion of these benefits that can be attributed to the government
program.

The baseline is the case without the government R&D that would help
develop the technology in question. The baseline is the condition of the
energy market, environmental impacts, and energy security that would
exist, absent this technology.

Each of the following alternatives seeks to establish the baseline set
of conditions, to which the benefits of the technology under
consideration would be viewed as an incremental addition. These benefits
could be thought of as the difference between the state of the economic,
environmental and security conditions with the technology, versus that
without the technology, but with some level of \"autonomous\" technical
improvements.

For the retrospective case, the NRC study took the next-best alternative
as the baseline. For the prospective case, we suggest that at least
three alternative approaches be considered:

(a) Use the National Energy Modeling System (NEMS), or other DOE
    model\'s, reference case;

(b) Use the base case of another model that focuses more on the type of
    technology and/or part of the energy market under consideration; or

(c) Use a hypothetical next-best alternative, without resorting to using
    a forecasting model.

Each of these alternatives is discussed in turn.

\(a\) [NEMS Reference Case as the Projected Baseline]{.underline}

The reference case forecast of the Energy Information Administration\'s
(EIA\'s) NEMS model could be used as the projected baseline. This
approach has the advantage that EIA is independent of the other offices
such as EERE, FE, NE, and SC. Thus, there is an element of
\"neutrality\" in EIA\'s projections. The NEMS model is a comprehensive,
large-scale model of the supply, transportation, and use of energy in
the U.S.

NEMS produces forecasts of the market penetration, use, and energy
consumption of alternative technologies across all major supply and
end-use sectors. The specific values used for NEMS\' input parameters
reflect implicit and in some cases explicit assumptions about
macroeconomic conditions, as well as regulatory and policy changes*.
NEMS also assumes technical change (i.e., improvement) in energy
technologies.* As such, this projected baseline implicitly includes the
next-best alternative. Thus, when using NEMS, users should first be
aware of these assumptions.

In particular, users should review and evaluate assumptions made about
the technological changes in the energy sector, and especially about
technologies similar to or even encompassing the technology under
consideration. If this technology is a very small component of the
market, then it might be reasonable to assume that the technology would
have little impact on NEMS\' assumptions about technical change and the
resulting prices, market penetrations, and other baseline conditions,
regardless of whether the technology was commercialized and deployed. In
this case, the reference case forecast can be used as given for the
projected baseline. However, if the technology is one that could have a
major impact on the market, then it is important to adjust the inputs
for the NEMS reference case NEMS to *not* *include* the technology, but
rather the next-best alternative.

\(b\) [Reference Case of A Model Other Than NEMS]{.underline}

An alternative to using NEMS is to use another model(s) that might have
more detail about, or is in some other way better suited to assessing,
the technology under consideration. An advantage of this approach is
that the alternative model might very well have a more accurate
representation of the technology in question, as well as of the market
and other conditions important to this technology. The alternative
model(s) could, for example, include MARKAL and consumer choice
functions that depend not only the relative prices of alternative
technologies but on other attributes which are important to consumers
and which affect their purchase decisions. Data from both retrospective
and experiment studies could be used to identify and estimate the
parameters of these models.

The use of such a model might be a more \"formal\" way of implementing
the third alternative approach below. A major disadvantage of this
approach, on the other hand, is that there would be no uniformity among
baselines used to assess alternative technologies

This approach has both advantages and disadvantages. For example, there
is no uniformity among energy sectors and technologies in the baseline
conditions. On the other hand, greater attention might be given to the
specific technology and market under consideration.

If this approach were to be used, we suggest that the key macroeconomic
and other assumptions could be as consistent as possible with those in
the NEMS reference case. Users might want to \"calibrate\" this other
model against NEMS. However, clearly, there will be differences between
the two models.

\(c\) [Hypothetical, Next-Best Alternative]{.underline}

This approach is analogous to the one used in the NRC\'s retrospective
study. A hypothetical, next-best technology is posited to develop, in
the absence of any government R&D. The energy efficiency, emissions,
security-related attributes, and other important impacts of
technological development of the next-best alternative must be
specified; and the technology in question then compared to this
alternative.

The next-best alternative could be defined as a hypothetical composite
of several alternatives that are assumed to have significant shares of
the market in which the technology competes. The attributes of the
next-best alternative would then be a weighted-average of the
alternatives\' energy efficiencies and other attributes. Perhaps the
simplest method of characterizing the next-best alternative this way is
to assume a straight-line average percentage improvement in technical
performance for the energy sector.

The advantage of this its approach is its simplicity. The disadvantage
is that it might be rather ad hoc and inconsistent with other DOE
projections, including their underlying macroeconomic, regulatory and
policy assumptions.

**6.1.3 Government Impact**

Given some estimate of the benefits of an energy technology, relative to
the baseline, a separate but related issue is that of estimating the
relative contribution of the government R&D program to the total
benefits of the technology.

The benefits of a technology are not solely attributable to DOE. In most
cases, to bring a technology to market, DOE, industry, and other
organizations contribute to the outcome. The NRC study found no reliable
way to quantify the DOE contribution in most cases, and it regards that
doing so remains a major challenge for future research (NRC 2001). Thus,
the NRC study did not apportion shares of benefits between DOE and
industry. Hence, the net benefits in the NRC report should be
interpreted as the benefits *associated* with the respective technology
programs rather than the benefits that can be completely attributed to
these programs.

We use Figure 1 to discuss this issue. In this hypothetical case, DOE
R&D takes place early in the technology development process. Some basic
scientific research might be involved. Then industry, either through a
cooperative research and development agreement (CRADA) with DOE or
independently, invests to develop a product using the technology. If
product development is successful and the market appears promising,
industry then begins commercialization of the product. Finally,
production begins. The firm must invest in capital and pay for labor and
other costs of production. In return, the firm has revenues from sales
of the product.

![](media/image2.jpeg){width="6.496527777777778in"
height="6.307638888888889in"}

Figure 1. Benefits and Costs of a Research, Technology Development and
Product Cycle

Some of the private benefits belong to the firm, reflecting revenues
from the sale of the product. Much of the rest of the benefits accrue to
consumers, who in this case save energy. There are also spillover
benefits to other firms who improve their products by learning about the
technology in question. Finally, there are other, societal benefits such
as improvements in environmental quality. To attain these various
benefits, investments are required from both DOE and the firm. A
government program might lead to the eventual commercialization of a
technology, or accelerate its introduction into market; but the private
sector also plays a critical role. The conundrum is deciding how to
attribute the appropriate share of these benefits to each party.

To provide a starting point for discussion at the conference, we offer
the following alternative ways of characterizing the impact of the
government program on a technology\'s overall benefits:

(a) No attempt is made to estimate the impact of the government,
    separate from that of the private sector and other parties.

(b) All societal benefits including the consumer benefits, plus the
    spillover benefits.

(c) Ratio of DOE to private sector investment.

(d) Typical equity held by initial investors.

(e) Case-by-case survey.

(f) The 5-Year Rule.

Each of these is discussed in turn.

\(a\) [No Apportionment of Benefits]{.underline}

In this approach, the position taken is that it is either too difficult
or impossible to separate non-additive contributions among many
participants responsible for developing a technology. Thus, the benefits
of a new technology would be considered as being *associated* with (but
not solely the benefits of) the DOE R&D program that led to its eventual
deployment.

\(b\) [Credit DOE with the Societal Benefits Including the Consumer
Benefits, Plus the Spillover Benefits]{.underline}

The reasoning behind this option lies in the position that the role of
the public sector is only to provide societal and spillover benefits.
Since a private firm generally does not invest to obtain a return on any
form of societal benefit, DOE might be apportioned all of the societal
benefits. This would include the consumers\' surplus (the difference
between what consumers were willing to pay, and what they actually paid,
for a product or service), plus the spillover effects to other firms
(which would need to be estimated), but none of the firm's own benefits.

\(c\) [Ratio of DOE to Private Sector Investment]{.underline}.

In this alternative approach, DOE's share would be calculated by
multiplying the present value of all of the net benefits, by the
fraction of its R&D spending to the firms' costs (i.e., the present
value of each). In this case, this fraction might be fairly small in
spite of the fact that DOE's up-front spending will carry greater weight
compared to the firm's discounted costs. All of the firm's production
costs would be included in calculating this ratio because these
expenditures are necessary for the benefits of the product to be
realized.

\(d\) [Typical Equity Held by Initial Investors]{.underline}

The relative value of DOE\'s contribution to a technology might be
estimated by what its equity might be if it were a private partner in a
new start-up company selling a product that uses that technology. This
share could be estimated empirically -- for example, the ratio of the
equity held by the initial partners in technology companies, to the
equity held by investors who purchase shares in the company later
through initial public offerings. This calculation would take into
account the relative size of the investment made by each party, compared
to their equity stakes in the company.

\(e\) [Case-by-Case Survey of Qualitative Contribution]{.underline}

This approach uses a qualitative assessment and classifies DOE's
contribution to the eventual successful commercialization of the
technology as being either "Dominant," "Influential," Minimal," or
"Absent." This assessment could be done on a case by case basis for each
major program that is evaluated. Representatives from the firms that
eventually designed and produced the product were asked to make an
assessment on this four-point scale. Such assessments can be made for
retrospective studies but appear to be problematic for prospective
studies because the commercialization of the technology has not taken
place, and the technology itself not yet developed.

\(f\) [Generalization of The 5-Year Rule to a Program-by-Program
Basis]{.underline}

The 5-Year Rule adopted by the NRC study assumes that the private sector
would lag DOE by five years, with the same level of R&D and the same
outcomes, if DOE had not itself invested. We suggest that this rule
could be generalized and that the impact of a government R&D program
could be represented as a change in the timing of the benefits of the
technology developed from that program. We further suggest that the
specific time lag could be determined on a program by program basis.
Just as, for example, supply curves that describe the quantity of energy
services available at different prices are different for different
technologies, government R&D programs will reduce the time it takes for
technologies to reach market to different extents.

The 5-Year Rule was used in the NRC study to estimate the impacts of the
government R&D program. Thus, rather apportion benefits to DOE according
to the recipient of the benefit, as in approach (b), the 5-Year Rule
calculated DOE\'s *incremental* impact as being the difference between
the stream of net benefits associated with the technology in question,
and that same stream shifted 5 years later. Figure 2 illustrates this
situation.

![](media/image3.jpeg){width="6.4944444444444445in"
height="4.884027777777778in"}

Figure 2. Illustration of the 5-Year Rule in Which the Private Sector
would have Invested in Technology R&D as DOE did, Only Five Years Later.
The 5-Year Rule can be used to Estimate the Contribution of DOE\'s R&D
Program to the Benefits of the Technology Under Consideration

To consider the NRC committee\'s approach in more detail, we can review
Laidlaw's (1997) study, one of the few on the effect of government R&D
on product development cycles. Laidlaw (1997) studied the acceleration
of technology development as a result of funding by the Advanced
Technology Program (ATP).[^16] She interviewed representatives from 28
companies that had been funded by ATP in 1991. None of the projects
involved an energy efficiency technology, but all involved advanced
technology development. Laidlaw (1997, Table 6) found that the companies
viewed ATP's involvement to have cut their R&D cycle times in about
half, which was the median and by far the most common response. The
median estimate of the R&D cycle was six years so that the median
reduction in the cycle was estimated to be three years. Thus, the time
to bring the product to market and the time when benefits commence, from
the sales of the product, was presumably reduced by three years.

This estimate of a three-year lag might be a reasonable approximation
for DOE R&D programs that are at phases of their R&D cycle similar to
those of the ATP projects, specifically those that need to have the gap
between research labs and the marketplace bridged. For most DOE
programs, however, we suggest it likely that the lag is likely to be at
least the five years suggested by the NRC study.

Two reasons for this suggestion are that:

-   ATP's projects are *all* with companies developing specific
    technologies that, for the most part, are well along in their R&D
    cycles. DOE's projects, on the other hand, generally involve
    technologies that are at earlier stages of their R&D cycle. ATP\'s
    projects are, by definition, involve specific technologies that
    particular companies have decided are worth investing in R&D. Many
    of DOE\'s programs, on the other hand, do not involve any
    private-sector participation at all. There are several possible
    reasons for the lack of private-sector involvement in an R&D
    initiative: its long time horizon, the risk in developing new
    technologies, uncertainty about market conditions, and the private
    gains being insufficient compared to the sum of the public and
    private benefits.

-   The ATP technologies under development might have benefited from
    previous government research, ranging from direct support from other
    federal agencies to spillover knowledge from other
    government-supported projects. When this is the case, then is time
    interval would be added to the lag identified in Laidlaw\'s study.

Furthermore, in five of the twenty-eight ATP projects, or about 18%, the
companies responded that the applied research would not even have been
done, and the products would presumably not have been brought to market,
without the ATP program.[^17] This latter situation would likely be
expected to be more prevalent with DOE-related technologies.

Thus, we suggest that the 5-Year Rule, which the NRC study adopted,
could be used for certain types of DOE programs that involve
technologies in the later stages of development. In some cases, for
technologies that are on the verge of commercialization, a three-year
lag might be more likely. In yet other cases, where research is being
initiated or still at the early stages, we suggest that the lag is
likely to much longer than five years \-- possibly ten years or more.
Finally, for a large portion of the programs, the lag is likely to be
infinite, i.e., the private sector would never have undertaken the R&D.

Note that if method (e) above is used (a qualitative assessment of the
relative contribution of the government program to the success of the
technology in question), in conjunction with the 5-year rule, then this
might be a double \"discounting\" of the government impact. If so, only
one method should be used, not both.

**6.2 OPTION VALUE**

In this section, we discuss building on the option concept as introduced
in the NRC report (refer to Section 5.1). Options provide discretionary
choices to deal with deviations from planned scenarios.

In a [prospective]{.underline} sense, there could be option value to R&D
on technologies that are not being developed *primarily* to enter the
market under the most likely conditions, but that would provide
economically viable solutions under alternative plausible conditions.
These plausible futures are generally of energy, environmental, security
or other policy concern. Physical or intellectual assets (not necessary
fully developed technologies as defined in the NRC study) that might be
deployed in the future (not just under improbable conditions) may also
contain significant option value. In this sense, options provide
insurance in the face of market uncertainties, yet retain the ability to
capture the upside benefits should improbably scenarios be realized.

In a [retrospective]{.underline} sense, technologies that are already
developed, but that are unlikely to be commercialized under current or
anticipated market conditions, may yet contain option value. This value
is derived from the uncertainty surrounding future market conditions.
These technologies might have been developed to enter the market but did
not do so because of changes in conditions \-- the technologies remain
available for the market in the future should conditions change to make
them commercially viable. For example, carbon sequestration technologies
could be options whose value might be realized if new regulations or
trading of carbon emission permits take effect.

Often, a research line is pursued because it has limited, but important,
expected benefits through niche market applications, but would be
expected to have much wider market applications under some alternative
futures, e.g., an oil disruption. While option values can persist after
the research is completed (often referred to as backstop or shelf
technologies), the value of the option will change as we move further
into the future and more certain about whether the technology
application is economically viable.

R&D can be considered as investments in options that provide
opportunities to realize benefits, in the event of alternative future
events. In addition to the option to commercialize, R&D investments
contain a host of other investment timing options. At each stage, as new
information becomes available about the probabilities of different
outcomes, choices can be made regarding continuation of the research,
abandoning the research, mothballing the research, etc.

This discussion implies that there are at least four alternative, but
related, approaches to considering option value:

a.  The expected prospective benefits calculation is retained. If
    feasible, the calculated value could also be weighted by the
    probability of the reference case occurring, and real options
    methods could be used to calculate an option value for all other
    scenarios. If feasible, the calculated value could be weighted by
    one minus the probability of the reference case occurring; \[this is
    consistent with alternative (a) for defining expected prospective
    benefits in Section 6.1.1\].

b.  Expected net present value calculation for prospective benefits,
    using a Monte Carlo consideration of the probabilities of various
    scenarios occurring; the benefits of the technology are considered
    under a range of scenarios, weighted by the probabilities of their
    occurring; in this case, there would be no separate option benefit
    or option column in the benefits matrix; the uncertainty in the
    projections would be reflected in the \"confidence bands\" that
    arise from Monte Carlo sensitivity analyses; \[this is consistent
    with alternative (b) for defining expected prospective benefits in
    Section 6.1.1\];

c.  Numerical valuation based on application of methods developed for
    \"real options;\" the option valuation encompasses all scenarios so
    that there is no expected prospective benefits column in the
    benefits matrix; \[this is consistent with alternative (c) for
    defining expected prospective benefits in Section 6.1.1\]; or

d.  Qualitative assessment of the option value of technologies that are
    fully developed, that are unlikely to be commercialized under the
    most likely scenario, but that might enter the market under
    reasonably plausible economic or policy conditions; the prospective
    benefits column in the benefits framework is retained and these
    benefits are calculated for the technology under the reference case
    scenario.

Alternative (b) is a straightforward extension of the traditional
discounted cash flow analysis currently used in, for example, NRC\'s
calculation of the retrospective realized benefits, to account for the
probabilities of alternative future scenarios being realized.
Alternative (a) is a hybrid of (b) and (c). Below, we describe (d) and
(c) in Sections 6.2.1 and 6.2.2, respectively.

1.  **The Option Concept as Described in the NRC Study**

The NRC study defined options as technologies that are fully developed,
that are unlikely to be commercialized under the most likely scenario,
but that might enter the market under reasonably plausible economic or
policy conditions. If these conditions are undesirable from a social and
financial standpoint, then the options can be considered as a form of
insurance. This description evokes the concept of a "backstop
technology" that could provide some "insurance" because it would be in
plentiful supply if energy prices rise to the level at which it can be
used.

Technologies might have an option benefit prospectively if they are
intentionally being developed, not to enter the market under forecasted
conditions, but for the purpose of having an option available if
conditions plausibly change. It is unlikely, however, that an R&D
program would be funded if it were projected to \"fail\" under the most
likely scenario.

Another benefit of options is that they might reduce or at least limit
the prices of competing technologies. This effect is clear-cut with
backstop technologies. They can be included in a supply curve that
describes the total quantity (e.g., of an energy service) that is
available as a function of price.

Technologies may also have option values retrospectively, as defined in
the NRC study. In this case, these technologies were originally
developed to enter the market but did not because of changes in
conditions. The technologies are available for the market in the future
should conditions change to make them commercially viable. Although the
NRC study considered the option value of R&D to be possibly significant
in some programs, it did not apply a quantitative method to estimate
this value.

**6.2.2 Applying Real Options Theory to R&D Investments**

R&D can be considered as investments in options that provide
opportunities to realize benefits, in the event of alternative future
events. The real option approach stems from the finance literature on
options, though real options are non-financial and have important
differences from financial options such as stock options (Boer 2002):

-   Real options often depend on the situation and have limited
    liquidity;

-   Real options frequently do not have a fixed \"strike price\" at
    which they can be \"exercised\" or deployed;

-   Real options frequently do not have an expiration date, though their
    value can deteriorate over time; and

-   Transaction costs involved in real options are different in nature
    from those for financial options.

Valuation methods that have been developed extensively in the financial
options literature may be adapted to value R&D investment alternatives
under uncertainty. Schimelpfennig (1995) and Davis and Owens (2001)
provide application of this theory to renewable energy. Schimelpfennig
(1995) noted that option value represents the value of flexibility to
use or discard new technologies. Davis and Owens (2001) use real option
pricing techniques to estimate the value of renewable electric
technologies in the face of uncertain fossil-fuel prices. Faiz, Hamm,
and Mathews have applied real options methods to valuing high-risk
energy alternatives and/or to R&D.

The value of the option, or premium, is essentially equivalent to the
contingency funds that would have to be kept on hand to address an
eventuality, discounted by the risk of it actually occurring, taking
into account all plausible outcomes. An option can provide insurance
against an adverse outcome while leaving open the possibility of
capturing the upside economic, environmental and/or security benefits.

Investment to create options need not be directed only at insuring
against *adverse* outcomes, or at only the most unlikely scenarios.
According to its proponents, the option approach is more suitable than
the traditional discounted cash flow approach when there is considerable
uncertainty about the future.

To calculate the value of the R&D option, one must set values for:

-   Amount of the R&D investment, for which the corresponding value of
    the option is to be calculated, as well as some estimate of the
    technology outcomes that result from the R&D;

-   Degree of uncertainty about the costs to provide energy services
    using the technology in question, as a function of R&D;

-   Degree of uncertainty about the costs to provide energy services
    using competing technologies, i.e., degree of uncertainty of the
    projected baseline prices; and

-   Degree of technical uncertainty (risk) about the cost of bringing
    the technology to market.

Then, computational algorithms can be used to derive numerical results.

Developing these input data is a real challenge, especially when done
for the first time. The uncertainties can be expressed \"continuously\"
using probability density functions, or \"discretely\" using a few
scenarios. Much of the real options literature uses continuous
representations of uncertainty over the range of relevant future market
factors. On the other hand, scenario analysis is frequently more
intuitively appealing. It is a well-established approach to planning
that allows individuals to assign their own views about the
probabilities associated with various scenarios.

If the real option approach were to be adopted, then it could used in
place of, or at least complement, the traditional discounted cash flow
approach used to estimate expected prospective benefits, which was
described in Section 6.1.1(a). The scenarios implicitly considered in
option valuation could encompass the reference case used to calculate
the expected prospective benefits. In this case, the expected
prospective benefits and option benefits would be combined into one
column called option benefits.

Alternatively, we suggest that the real option approach could be used
for all scenarios except for the expected prospective benefits. This
approach is likely more appealing for energy R&D planning because the
concept of a reference case is a familiar and broadly accepted approach.

**6.3 KNOWLEDGE VALUE**

R&D produces scientific knowledge. It might be either specific
information or general insight about a field of study. Knowledge
benefits tend to be the earliest benefits of the R&D process. In fact,
the main mission of science programs is the \"creation\" of new
knowledge. Even technology-oriented R&D programs that fail might
contribute to the state of knowledge about the field in terms of what
does not work.

Knowledge benefits are comprised of useful or potentially useful
scientific knowledge and technology, that have resulted from R&D
initiatives, and that are not reflected in realized or option benefits.
In a prospective context, the goal of a science program is primarily to
produce knowledge. In a retrospective context, whether intended or not,
a program might lead to results that have commercial value and to
economic, environmental, or energy security benefits.

1.  **Assessment of Knowledge Benefits**

In this discussion, we note some common methods that are used as
indicators of the benefits of knowledge. Since science programs have no
immediate commercial objective, the metrics for measuring the knowledge
created by these programs should be different from those developed for
technology or applied research programs.

The eventual, longer-term outcomes of science are exceedingly difficult
to project. Thus, the literature on science performance measures has
focused more on methods for measuring outputs of science programs:

-   Accomplishment of project or program milestones

-   Peer review of programs\' contribution to knowledge

-   Bibliometric methods.

[Bibliometrics]{.underline}

The first two categories are rather straightforward and we focus on
*bibliometric* methods. Bibliometrics refers to measures of scientific
and technical outputs from science, their publications in scientific and
technical journals, and the indexing of citations by other scientists in
these journals. Bibliometric methods count the number of publications
and other outputs of knowledge transfer or use:

-   Publications

-   Citations

-   Patents

-   Patent citations.

For example, Narin (2001) used data from the U.S. Patent System that
contains information on patents granted to both U.S. and foreign
inventors. The number of patents indicates the technological output. The
citation of patents is a measure of the quality of the patents and their
impact. Additional indicators imply some of the benefit of this
knowledge:

-   Current Impact Index. This index provides a normalized measure of
    the impact that an earlier set of patents is having on technology
    appearing in the current year \-- essentially a patent citation
    measure. The method considers how frequently other patents are cited
    in patents granted in a current year.

-   Science linkage. This method counts the number of times a patent
    cites scientific papers or similar research publications. This
    measure indicates how strongly a patent has relied on fundamental
    scientific research.

-   Technology-cycle time. This index is the median age of patents cited
    on the front page of a U.S. patent document. This measure indicates
    the speed of innovation in a company or industry. Typical cycle
    times might be about ten years.

Major advantages of these indicators are that they are conceptually
straightforward to explain, and they have been implemented and used.
Limitations of these methods are that: published articles are only one
measure of scientific output (technical reports might be important as
well); citations measure influence, not necessarily quality; and authors
who inordinately cite themselves and \"friends\" bias any citation
indicator.

[Process-Outcome Metrics]{.underline}

Process-outcome metrics are based on models of the stages of an
innovation process (Geisler ). Such models typically have four
categories of outputs:

(1) immediate or direct outputs such as publications and patents, which
    are usually the output of scientists and researchers, and their
    organizations;

(2) intermediate outputs such as new products, materials or methods that
    are usually the output of companies that have developed these
    products from the technology or scientific knowledge of either their
    own scientists or others;

(3) pre-ultimate output, or intermediate outcome, which are products and
    services deployed by society or an economy (e.g., widespread
    commercialization and deployment of a technology); and

(4) ultimate outcome, which indicates quality of life \-- such as
    economic, environmental, and security benefits.

To assess the contribution of knowledge to these ultimate outcomes, it
might be possible to construct models of the process linkages beginning
with the direct outputs and ending with the ultimate outcomes. Such
models would also serve the purpose of describing the relationships
between the near-term milestones of R&D programs to their ultimate,
longer-term goals.

[Prospective Analysis]{.underline}

The challenge of prospective analysis is that the future is of course
unknown. The bibliometric methods discussed above are generally limited
to retrospective analysis. It would be interesting to assess
statistically whether certain characteristics of research and the
knowledge it creates tend to lead to better bibliometric results.
Furthermore, it would be interesting to assess whether it would be
possible to identify general attributes of scientific research that can
be linked to current-impact, science-linkage, and technology cycle-time
indicators that might be used in conjunction with process-outcome models
of the science, innovation, commercialization, and deployment process.

For the time being, however, it would appear that prospective analysis
must rely on expert panels\' identification of high-priority areas of
research, and peer-level assessments of research proposals. Peer-review
methods be adapted and used as the primary method for assessing the
prospective benefits of scientific programs. An example might be the use
of Delphi types of processes to develop assessments of the prospective
knowledge-related benefits of various science programs. GPRA-like
projections are generally much different from typical technical peer
reviews, though, so that peer reviews for the purpose of GPRA-related
requirements would need to crafted to address these types of needs.

**6.3.2 Spillovers**

The organization creating the knowledge might retain intellectual
property rights to it, but knowledge can spillover to other
organizations, firms and to the market at large. Jaffe (1996) identifies
the following types of spillovers --

-   Knowledge spillover. Sometimes, knowledge created by one
    organization can be used by another without any compensation, or
    with compensation less than the value of the knowledge. Economic
    exploitation of knowledge requires the sale or productive use of a
    new commercial product or process that somehow embodies some of that
    knowledge (Jaffe 1996, p.5).

-   Market spillover. These spillovers result when the operation or
    activity of one firm in the market causes some of the benefit to
    flow to other market participants, other than the innovating
    organization.

-   Network spillovers. These result when the economic value of a new
    technology depends greatly on the development of a set of related
    technologies. An example is certain types of alternative-fuel
    vehicles whose success in the market will depend greatly on there
    being a geographically distributed re-fueling infrastructure.

The econometric findings that R&D leads to macroeconomic growth on a
large geographic scale likely reflect these sorts of spillover
phenomena, certainly the market and network spillovers (see also,
Section 6.4.2). Such spillovers might be a significant portion of the
overall benefits of R&D.

Prospective estimates of spillover effects would need to rely on
statistical and econometric findings of retrospective studies.

**6.4 SECURITY, ECONOMIC, AND ENVIRONMENTAL BENEFITS**

**6.4.1 Security Benefits**

Energy security in the past has typically focused on vulnerability to
oil imports in a cartel market. Since September 11, 2001, the
vulnerability of energy infrastructure and systems to catastrophic
disruptions has become a greater concern than previously considered. In
addition to oil security and energy infrastructure security, energy
security includes electric power reliability.

In this white paper, energy security refers to the adequate and reliable
delivery of energy products and services with reasonably stable prices
in a competitive market. This definition is more general than the one
used by the NRC committee. The NRC study\'s definition of security
benefits refers to changes in the probability or severity of abnormal
energy- related events that would adversely impact the overall economy,
public health and safety, or the environment. The definition suggested
here includes these changes, but also includes oil dependence and the
costs of maintaining electric power reliability under \"normal\"
conditions.

[Security of Energy Infrastructure and Systems]{.underline}

There has been much more concern since \"9/11\" about the risks of major
damage and disruption to domestic energy infrastructure and systems.
Among the more prominent public concerns are nuclear power plants, the
electric power grid, natural gas pipelines, oil pipelines and
refineries, and large hydropower dams.

Different types of energy technologies and systems can affect the
vulnerability of the country\'s energy supply to catastrophic
disruptions. For example, small-scale distributed generation reduces
reliance on, and thus the risks of disruptions to, the electric power
grid; nuclear technologies that inherently preclude the possibility of a
melt-down would make facilities using these technologies less
vulnerable; and sensors and other instrumentation that monitor
facilities for intrusion would reduce the risk of attack.

We suggest that the benefits of energy R&D, that contributes to reducing
these sorts of risks, might be estimated using probabilistic risk
analyses of the impacts of the technologies on the following:

\(a\) Reduced cost of outage or disruption to users,

(d) Reduced cost of providing a given, desirable level of security
    (e.g., cost of security personnel and equipment),

(e) Reduced impact on an industry and on the economy region- or
    nationwide (e.g., estimated using economic input-output models).

[Oil Security]{.underline}

Oil security usually refers to our (i.e., the United States\')
dependence on imported oil. There are three types of costs in situations
in which an oil cartel is able to raise prices above competitive market
prices. These costs are (Greene and Tishchishyma 2000, Greene and Leiby
1993):

a.  The economy\'s ability to produce is reduced because a key factor of
    production, oil, is more expensive than what the price would be in a
    competitive market,

b.  sudden disruptions to oil prices increase unemployment and reduce
    economic output, and

c.  oil producing nations are extracting an "economic rent" from oil
    consuming regions by influencing prices to a far greater degree than
    could be done in a competitive market.

This literature has estimated a range of values, expressed in dollars
per barrel of displaced oil, for the oil-security related benefits of
technologies that reduce oil consumption.

Energy security, in the context of imported oil, is primarily a
transportation-sector issue. Seventy-one percent of the petroleum
consumed in the U.S. is used in transportation (EIA 1998, as cited in
NRC 2001). If none of the transportation technologies have been
commercially used, then a retrospective analysis of these programs would
find that no oil has been displaced and thus no economic benefits
associated with these technologies.

A major assumption for a prospective analysis, then, is the market
penetration of new technologies, from which the associated reduction in
oil consumption can be calculated. Systematic methods should be
developed for making these assumptions.

In addition to the market-related costs of imported oil in a
cartel-dominated market, there are the military costs of ensuring
reliable supply of crude oil from politically volatile regions. Some
would argue that these costs would not have to be incurred if there were
a competitive global oil market.

[Reliability Benefits: Power-System Technologies]{.underline}

Several R&D programs in the power technologies area are striving to
improve energy efficiency and reliability in the areas of
superconductivity, energy storage (e.g., utility battery storage,
superconducting magnetic energy storage), distributed generation, and
transmission reliability. The benefits of electric power reliability,
although widely recognized, have thus far been difficult to quantify,
certainly in the rigorous manner demanded by the NRC committee's
methodological framework.

Electric power system reliability is reflected in the frequency,
duration, and magnitude of adverse effects on electric supply.
Reliability issues arise from \"normal\" as well as catastrophic (e.g.,
terrorist) events. The benefits of reliability are reflected in the
damages that are *avoided* by having a reliable system, and in the
higher rates which customers are willing to pay for more reliable power
(e.g., non-interruptible-power contracts). Through additional analysis,
it might be possible to estimate quantitatively various components of
the benefits of electric power reliability:

(a) \"Real-time energy value\" -- replace the measures of energy savings
    and their financial value (Btu\'s and Btu's times *average* price,
    respectively), which are currently used, with a measure of financial
    savings that accounts for energy savings (and increases) and the
    *different* electricity rates at different times of the day and
    year;

(b) "Reduced outage costs" -- the financial value of reliability as
    measured by the estimated reduction in customers' outage costs; and

(c) "Reduced costs of providing reliability" -- a quantitatively more
    important economic value of reliability than the previous one, is
    measured by the reduced costs of providing reliability (i.e.,
    reduced generation costs that would in turn reduce industrial and
    commercial consumers\' costs of doing business, as well as
    residential customers' utility bills).

**6.4.2 Economic Benefits**

The NRC committee defined economic net benefits to be based on changes
in the total market value of goods and services that can be produced in
the U.S. economy under normal market conditions, absent energy
disruptions or other energy shocks (NRC 2001). As noted by the
committee, the net benefits must be estimated relative to the benefits
that could have been attained with the next-best alternative, and by
subtracting the incremental costs relative to the alternative.

There might be other important economic benefits, in addition to the
energy-savings benefits that were the focus of the NRC study. In this
discussion, we identify two: consumer benefits other than energy savings
and macroeconomic benefits.

[Consumer Benefits Other than Energy Savings]{.underline}

New technologies, such as in the area of commercial lighting systems,
might provide benefits other than energy savings. For example, the
quality of light they provide might enhance worker productivity. More
analysis is needed in this area, and more generally in developing and
applying alternative ways of estimating quantitatively consumer benefits
(Jones and Bjornstad 2001). In this context, Austin and Macauley (2000)
have estimated future consumer benefits from government-funded
innovation by estimating the "consumers surplus" from innovation.

[Macroeconomic Benefits]{.underline}

Several econometric studies have identified statistically significant
relationships between R&D and economic growth. For example, Martin et
al. (1997) have developed a framework for estimating the national
economic benefits of government funding of medical technology
development. Lev used an econometric model with operating income as the
output measure of the payoff, and investments in physical capital and
R&D as the inputs. He found significant correlation between a chemical
company\'s R&D intensity and its return on R&D.

**6.4.3 Environmental Benefits**

Environmental net benefits are based, in the NRC report, on changes in
the quality of the environment that have occurred, will occur, or may
occur (with some non-zero probability) as a result of the technology
(NRC 2001). Changes in environmental quality are often measured using
some indicator such as the net change in emissions of a pollutant. These
benefits are typically not directly measurable by market prices but by
some measure of the value individuals are willing to place on these
changes.

[Environmental Impacts with Market Value]{.underline}

Markets for trading some types of emissions are being introduced,
however, and these markets provide direct evidence of the prices paid to
obtain rights to emit these pollutants. However, these prices are good
indicators of the \"value\" of these pollutants only if the marginal
cost of controlling the pollutants equals the marginal benefit of
reducing their damages.

Some environmental damages, such as reduced crop yields as a result of
exposure to ozone, have a direct economic impact as well in terms of the
value of the crop loss.

[Non-Market Environmental Impacts]{.underline}

For non-market environmental impacts, studies in the environmental
economics literature have developed methods to estimate quantitatively
the damages associated with several pollutants. Even though markets
generally do not exist for environmental quality, per se, estimates of
environmental damages can be expressed in economic terms using the
willingness-to-pay paradigm (as applied by Ottinger et al. 1990,
ORNL/RFF 1994, EC 1996-2001, Pearce et al. 1996, Tol 1999). These
studies allow us to estimate some of the environmental benefits of
various technology options. The values are not universally accepted but
they can and have been used.

[Ecosystem Impacts]{.underline}

Economists and environmental scientists still frequently disagree on how
to quantify other types of environmental impacts, particularly those to
ecosystems. These impacts are generally site-specific so that it is
difficult to gauge the benefits of R&D programs on reducing these sorts
of impacts. On the other hand, it might be possible to generalize that
certain types of technologies have more significant impacts on
ecosystems than other technologies. To the extent that these sorts of
impacts are valued (though not necessarily in monetary terms), some
systematic method would be useful for gauging their relative importance,
if only in subjective qualitative terms. Thus, one might propose a
qualitative scale that reflects the severity of different types of
ecosystem impacts.

Such a measure might be used in conjunction with estimates of the value
of the services provided by ecological systems. For example, values
expressed in \$/hectares per year might be used for open-ocean, coastal,
forest, grass/rangeland, wetland, lakes/rivers, desert, tundra, and
ice/rock biomes.[^18] Technology impacts might then be subjectively
evaluated, and expressed in terms of this measure. Other thinking on the
value of natural systems considers the current economic paradigm to be
flawed in terms of its valuing of ecosystems.[^19]

**7. CONSIDERATIONS IN IMPLEMENTING A FRAMEWORK AND METHODS**

Regardless of which framework and methods are identified, there are
several important considerations that should be kept in mind as these
methods are identified, discussed, and assessed.

1.  **Data CONSidERATIONS**

One of the most important considerations is the availability of data.
The data required for the calculations should be available and should
not require massive effort to collect, solely for the purpose of
estimating the benefits of R&D programs.

Another data consideration is that baseline forecasts and other
assumptions should, if possible, be consistent with \"official\" DOE
forecasts, usually from EIA. This data source would provide a reasonably
consistent set of baseline assumptions. If EIA sources of information do
not exist or are inadequate for the purpose at hand, then the
assumptions and projections of other models or baselines used should be
checked against the corresponding parameters in the EIA models.

Inevitably, subjective assessments will have to be made. For example,
the market penetration projections are key to estimating their possible
benefits. If these projections must be made through subjective
assessments, then they could be made, or at least reviewed, by a peer
group whose members have an arms-length relationship with DOE, to the
extent possible. Program managers are knowledgeable about the
technologies being researched by their programs. But they are obviously
biased and their projections might be questioned. Reviews of program
managers/ projections by an independent party are a good check on the
reasonableness, if not likely accuracy, of the projections. An even
better approach might be to use expert panels whose members do not have
a conflict of interest.

**7.2 PROCESS CONSIDERATIONS**

There are several aspects to be considered in estimating benefits and
other performance measures that might improve their cost-effectiveness:

\(a\) Output-outcome process. Shorter-term, direct outputs of an R&D
program are usually of greater immediate interest to R&D program
managers because they can relate to, and control, these outputs much
more than their ultimate outcomes. As discussed in Section 6.3.1,
descriptions (or what might be termed conceptual models) of the possible
process by which outputs might become outcomes could provide roadmaps to
help design R&D programs better.

\(b\) If requests for information from R&D programs are such that they
can provide that information in a consistent, uniform way, then this is
likely to be more cost-effective than multiple requests of different
variations of similar information. Also, computer software that not only
has an easy-to-use interface but that interpolates input data, would
again reduce the burden on program managers (e.g., software that
interpolates annual market penetration based on a projection of the
ultimate long-term market penetration and the time to attain that
level).

\(c\) If the measures and the ways in which they are calculated are
transparent and easy to communicate, even though the details of the
mathematics might not, then the measures are likely to be better
understood and received by their users.

\(d\) It is unnecessary, and sometimes it cannot be justified, to
develop quantitative estimates of all of the benefits. Sometimes, the
quality and imprecision of input data might not justify detailed
numerical results \-- order of magnitude estimates might very well be
better in these cases. In other cases, the very nature of the benefits
in terms of their social or ecological impacts might inherently preclude
their being quantitatively estimated. In such situations, these benefits
might be just as, or even more, important than other types of benefits
that can be quantified.

**7.3 ASSESSING A Portfolio of r&d programs as Opposed to Just
individual Projects**

In conducting its retrospective review, the NRC was evaluating
individual projects - or in some cases - programs. Looking both
retrospectively as well as prospectively, the NRC recognized that the
portfolio concept was very important to properly evaluate the benefits
from an overall research program. The portfolio concept has many
attributes- ranging from the types of research that should be conducted-
from basic to applied - to the time frames expected for the research
results, and other variables.

The NRC recognized that a portfolio of many projects would have both
failures as well as unanticipated successes. The NRC did not formally
assess the overall appropriateness of the DOE portfolio, as this was not
within the scope of the task assigned.

The NRC matrix for the individual programs did not evaluate some of the
features of a balanced portfolio, such as the time frame for projects.
The issue of how individual project benefit assessments should be used
in constructing a portfolio benefit assessment needs to be addressed.

**7.4 ASSESSING Multi-Year Projects or Programs as Opposed to An
Individual Year\'s Budget Request for a Project**

The NRC evaluation was retrospective, so it evaluated many years worth
of research that were part of ongoing programs. Using this framework
prospectively presents more challenge. Most programs are multi- year,
and an individual year's budget request may include out- year
projections of costs, benefits and milestones.

The question if one should assess - and if so how -- any benefits of one
year of a program remains. Does achieving interim milestones for a
multi-year project account for a benefit apart from program management?
Milestones for project management and off- ramps need to be established
for multi-year programs, but how these interplay with benefit assessment
remains an issue to be considered.

Finally, there is the issue of how to make decisions to propose
initiating a new program or to terminate an ongoing program. Should
there be a threshold or payback consideration, in addition to any net
present or option valuation? And for ongoing projects, how should sunk
costs, benefits that depend on previous findings, and successful
completion (or failure in attaining) program milestones be taken into
account in decisions to continue their funding?

**8. CONFERENCE FORMAT**

This conference is about framework and methods, not policy positions,
nor advocacy of particular technologies or science, nor viewpoints for
or against government R&D. The conference will consist of plenary
sessions and workshops. The plenary sessions will set the stage for the
in-depth discussions that will take place in the workshops, and provide
a forum for exchanging insights from these workshops. The workshops will
identify methodological framework(s) for estimating the benefits of
government-sponsored R&D, and the methods that could be used for
estimating the individual types of benefits.

The emphasis of the discussions in the conference workshops will be on
identifying the more promising and practical methods, particularly those
that can be easily communicated.

Also, the role of the conference, and of this and other draft reports
related to the conference, is to identify and to provide suggestions
about these methods. It is not to provide recommendations or advice to
DOE.

To provide some focus to the discussions, rather than a potpourri of
ideas, we make clear the objectives of each session in the conference.

**8.1 PLENARY SESSIONS**

The plenary sessions will convey information about the objectives of the
conference and the perspectives of both the U.S. Department of Energy
and congressional staff members. The plenary sessions will also provide
some common background information about how some DOE offices, as well
as some other Federal agencies, currently estimate and/or consider the
benefits of their programs in the context of GPRA and/or program
planning.

In addition to providing foundation- and background-types of
information, the plenary sessions will enable workshop participants to
exchange the insights, developed in the four workshops, among all
conference participants.

**Objectives of Plenary** Sessions

+-------------------------+--------------------------------------------+
| **Session**             | **Objectives**                             |
+-------------------------+--------------------------------------------+
| 1\. Conference Opening, | 1.  Present DOE and Congressional views on |
| Overview, Background,   |     conference objectives.                 |
| and Objectives          |                                            |
+-------------------------+--------------------------------------------+
| 2\. Methods Currently   | 2.1 Provide summaries/overviews of methods |
| Used by DOE Offices to  | used by energy resources and science       |
| Estimate the Benefits   | offices in DOE.                            |
| of Their R&D Programs   |                                            |
+-------------------------+--------------------------------------------+
| 3\. Federal             | 3.1 Present information on what other      |
| Perspectives on         | Federal agencies do, as well as OMB\'s     |
| Performance Measurement | perspectives.                              |
+-------------------------+--------------------------------------------+
| 4\. Establishing a      | 1.  Summarize the NAS/NRC benefits         |
| Starting Point for      |     framework.                             |
| Discussion of a         |                                            |
| Methodological          | 2.  Present suggestions that build on the  |
| Framework for Defining  |     NAS/NRC framework.                     |
| R&D Benefits            |                                            |
+-------------------------+--------------------------------------------+
| 5\. Structure, Goals,   | 1.  Describe the structure of the          |
| and Priorities of the   |     workshops.                             |
| Workshops               |                                            |
+-------------------------+--------------------------------------------+
| 6\. Summaries and       | 1.  Summarize workshops\' discussions.     |
| Feedback on the         |                                            |
| Workshops\' Discussions |                                            |
| of a Benefits Framework |                                            |
+-------------------------+--------------------------------------------+
| 7\. Review the First    | 1.  Summarize workshops\' discussions from |
| Day\'s Workshops and    |     Day 1 and provide overview for Day     |
| Review of the Second    |     2\'s sessions.                         |
| Day of Workshops        |                                            |
+-------------------------+--------------------------------------------+
| 8\. Workshop Summaries  | 1.  Summarize workshops\' discussions,     |
| and Discussion          |     particularly on Day 2.                 |
+-------------------------+--------------------------------------------+
| 9\. Next Steps and      | 1.  State next steps and how conference    |
| Closing Remarks         |     results might be used.                 |
+-------------------------+--------------------------------------------+

**8.2 Workshop A: Prospective Benefits \-- Cross-Cutting Issues**

Workshop A will address issues that cut across the problems of
estimating the prospective security, economic, and environmental
benefits of energy technologies associated with government R&D programs.
The key question is how to estimate the *\"delta\"* or *incremental*
contribution of a government-sponsored R&D whose goal is to develop
particular energy technologies. To address this question, two important
issues demand special attention:

8.21 Developing a Baseline, Absent a Technology Developed in Part by a
Government Program

To estimate the prospective benefits of a government program, a baseline
must be established. The baseline is the case without the government
effort that affected the development of the technology. It is the energy
market, environmental impacts, and energy security that would exist,
absent the energy technology that is to be developed through
government-sponsored R&D. The task in the first part of Workshop A is to
identify methodologies for developing both retrospective and prospective
baselines for the market adoption of different energy technologies under
the most likely scenario, absent the technology under consideration.
Several questions are germane to this issue. How are plausible scenarios
developed for technological progress (including that of existing
technologies), economic conditions, and regulatory and policy changes;
and how would these affect the projected baseline absent a government
R&D program? Should alternative baselines be used to reflect uncertainty
about future conditions? How does one consider the potential effects of
other government-sponsored R&D programs that are being conducted when
evaluating one related program? What are the different issues and
potential approaches for considering these questions prospectively as
contrasted with retrospectively?

8.2.2 Impact of the Government Program

Given some estimate of the benefits of an energy technology, relative to
the baseline, a separate but related issue is that of estimating the
relative contribution of the government R&D program to the total
benefits of the technology. That is, what is the impact of the
government relative to the overall beneficial outcome as a result of the
technology? Typically, a government program might lead to the eventual
commercialization of a technology, or accelerate its introduction into
market; but the private sector also plays a critical role. Looking
prospectively, this session in Workshop A will focus on identifying a
method to estimate the effect of a government program -- what is its
impact on the innovation, development and design of a technology, and
how is the market penetration of the technology be affected? Would the
technology never be developed absent a government program? Does the
government program just accelerate the development of the technology,
and if so by how much? Does the program affect its projected market
penetration -- by how much? Also, how does the definition of the
baseline condition affect how the impact of the government program
should be calculated? Does the program have other beneficial effects and
how are they weighed? How does one relate future streams of benefits to
past, present, and future R&D investment? Should one measure the
government\'s impact on the basis of individual R&D programs or across a
portfolio?

**Objectives of Workshop A Sessions on**

Prospective Benefits \-- Crosscutting Issues

+--------------------------+-------------------------------------------+
| **Session**              | **Objectives**                            |
+--------------------------+-------------------------------------------+
| A1. Framework for        | A1.1 Discuss methodological framework for |
| defining the benefits of | identifying benefits of energy R&D        |
| R&D programs, and its    | programs.                                 |
| use for GPRA and R&D     |                                           |
| planning and evaluation  | A1.2 Identify how various \"benefits\"    |
|                          | fall within this framework.               |
|                          |                                           |
|                          | A1.3 Identify benefits inadequately       |
|                          | addressed by the framework.               |
+--------------------------+-------------------------------------------+
| A2. Defining and         | A2.1 Define the meaning of                |
| calculating the          | \"retrospective baseline\" and            |
| projected baseline on    | \"projected baseline.\"                   |
| which new technologies   |                                           |
| are to be introduced     | A2.2 Identify methods for developing      |
|                          | retrospective and prospective baselines,  |
|                          | absent the new technology.                |
+--------------------------+-------------------------------------------+
| A3. Continued discussion | A3.1 Identify data sources.               |
| of projected baseline    |                                           |
|                          | A3.2 Identify how to define *alternative* |
|                          | scenarios.                                |
|                          |                                           |
|                          | A3.3 Identify issues for additional       |
|                          | consideration.                            |
+--------------------------+-------------------------------------------+
| A4. The effect of a      | A4.1 Identify a method(s) to estimate or  |
| government\'s R&D        | systematically describe the impact of a   |
| program on technology    | government R&D program on technology      |
| deployment and use       | deployment and use.                       |
+--------------------------+-------------------------------------------+
| A5. Continued discussion | A5.1 Identify data sources.               |
| of the effect of a       |                                           |
| government\'s R&D        | A5.2 Identify issues for additional       |
| program on technology    | consideration.                            |
| deployment and use       |                                           |
+--------------------------+-------------------------------------------+
| A6. Using estimates of   | A6.1 Suggest how to use estimates of      |
| benefits in R&D program  | benefits and other performance measures   |
| planning and evaluation  | in the planning and evaluation of R&D     |
|                          | programs.                                 |
+--------------------------+-------------------------------------------+

**8.3 Workshop B: Option Value**

Options provide discretionary choices to deal with deviations from
planned scenarios.

In a [prospective]{.underline} sense, option value helps justify R&D in
technologies that are being developed *intentionally* not to enter the
market under the most likely conditions, but that would provide
economically viable solutions under alternative plausible conditions.
Physical or intellectual assets (not necessary fully developed
technologies as defined in the NRC study) that might be deployed in the
future (not just under what are now considered to be improbable
conditions) may also contain significant option value. In this sense,
options provide insurance in the face of market uncertainties, yet
retain the ability to capture the upside benefits should improbably
scenarios be realized.

In a [retrospective]{.underline} sense, technologies that are already
developed, but that are unlikely to be commercialized under current or
anticipated market conditions, may yet contain option value. This value
is derived from the uncertainty surrounding future market conditions.
These technologies were developed to enter the market but did not do so
because of changes in conditions \-- the technologies remain available
for the market in the future should conditions change to make them
commercially viable.

R&D can be considered as investments in options that provide
opportunities to realize benefits, in the event of alternative future
events. In addition to the option to commercialize, R&D investments
contain a host of other investment timing options. At each stage, as new
information becomes available about the probabilities of different
outcomes, choices can be made regarding continuation of the research,
abandoning the research, mothballing the research, etc. Consequently,
R&D programs tend to have a compound (or nested) options feature and,
thus, call for algorithms to compute numerical estimates for option
values. Developing the probability data required as input to these
models remains problematic in many cases, however, and is a continuing
barrier to the use of option methods.

Workshop B will define more precisely the concept of options and will
identify methodologies to evaluate the value of their benefits.

Objectives of Workshop B Sessions on

Option Valuation

+--------------------------+-------------------------------------------+
| **Session**              | **Objectives**                            |
+--------------------------+-------------------------------------------+
| B1. Framework for        | B1.1 Discuss methodological framework for |
| defining the benefits of | identifying benefits of energy R&D        |
| R&D programs, and its    | programs.                                 |
| use for GPRA and R&D     |                                           |
| planning and evaluation  | B1.2 Identify how various \"benefits\"    |
|                          | fall within this framework.               |
|                          |                                           |
|                          | B1.3 Identify benefits inadequately       |
|                          | addressed by the framework.               |
+--------------------------+-------------------------------------------+
| B2. Definition of        | B2.1 Define \"option\" in the context of  |
| options and the          | R&D programs.                             |
| calculation of option    |                                           |
| value                    | B2.2 Describe how the definition relates  |
|                          | to retrospective and prospective          |
|                          | benefits.                                 |
+--------------------------+-------------------------------------------+
| B3. Continued discussion | B3.1 Identify a method(s) for calculating |
| of option value          | the value of an option in the context of  |
|                          | an R&D program.                           |
|                          |                                           |
|                          | B3.2 Identify data sources.               |
|                          |                                           |
|                          | B3.3 Identify issues for additional       |
|                          | consideration.                            |
+--------------------------+-------------------------------------------+
| B4. The baseline         | B4.1 Define what is meant by the          |
| scenario for             | baseline, in the context of option        |
| option-value             | valuation.                                |
| calculations             |                                           |
|                          | B4.2 Identify methods for developing      |
|                          | retrospective and prospective baselines,  |
|                          | in the context of option valuation.       |
|                          |                                           |
|                          | B4.3 Identify data sources.               |
|                          |                                           |
|                          | B4.4 Identify how to define alternative   |
|                          | scenarios.                                |
|                          |                                           |
|                          | B4.5 Identify issues for additional       |
|                          | consideration.                            |
+--------------------------+-------------------------------------------+
| B5. Effect of a          | B5.1 Identify a method(s) to estimate or  |
| government\'s R&D        | systematically describe the impact of a   |
| programs on option value | government R&D program on option value.   |
|                          |                                           |
|                          | B5.2 Identify data sources.               |
|                          |                                           |
|                          | B5.3 Identify issues for additional       |
|                          | consideration.                            |
+--------------------------+-------------------------------------------+
| B6. Using estimates of   | B6.1 Suggest how to use estimates of      |
| benefits in R&D program  | benefits and other performance measures   |
| planning and evaluation  | in the planning and evaluation of R&D     |
|                          | programs.                                 |
+--------------------------+-------------------------------------------+

**8.4 Workshop C: Knowledge Value**

R&D produces scientific knowledge. It might be either specific
information or general insight about a field of study. The very nature
of science programs is that they have no immediate commercial objective.
Even technology-oriented R&D programs that fail might contribute to the
state of knowledge about the field in terms of what does not work.
Workshop C will focus on how to evaluate the benefits of scientific
knowledge -- what indicia could be used and how could one assess them?
The primary task of Workshop C is to identify methodologies to assess
knowledge benefits both prospectively and retrospectively. In a
prospective context, the goal of a science program is primarily to
produce knowledge. In a retrospective context, whether intended or not,
a program might lead to results that have commercial value and to
economic, environmental, or energy security benefits. Workshop C will
also address the relationship between knowledge and technology
innovation.

**Objectives of Workshop C Sessions on**

Knowledge Valuation

+--------------------------+-------------------------------------------+
| **Session**              | **Objectives**                            |
+--------------------------+-------------------------------------------+
| C1. Framework for        | C1.1 Discuss methodological framework for |
| defining the benefits of | identifying benefits of energy R&D        |
| R&D programs, and its    | programs.                                 |
| use for GPRA and R&D     |                                           |
| planning and evaluation  | C1.2 Identify how various \"benefits\"    |
|                          | fall within this framework.               |
|                          |                                           |
|                          | C1.3 Identify benefits inadequately       |
|                          | addressed by the framework.               |
+--------------------------+-------------------------------------------+
| C2. Definition of        | C2.1 Describe the value or benefits of    |
| knowledge value and      | knowledge.                                |
| indicators               |                                           |
+--------------------------+-------------------------------------------+
| C3. Continued discussion | C3.1 Identify indicators for the benefits |
| of knowledge value       | of knowledge and a method(s) for          |
|                          | calculating them.                         |
|                          |                                           |
|                          | C3.2 Identify data sources.               |
|                          |                                           |
|                          | C3.3 Identify issues for additional       |
|                          | consideration.                            |
+--------------------------+-------------------------------------------+
| C4. The baseline         | C4.1 Define what is meant by the          |
| scenario for             | baseline, in the context of knowledge     |
| knowledge-value          | valuation.                                |
| calculations             |                                           |
|                          | C4.2 Identify methods for developing      |
|                          | retrospective and prospective baselines,  |
|                          | in the context of knowledge valuation.    |
|                          |                                           |
|                          | C4.3 Identify data sources.               |
|                          |                                           |
|                          | C4.4 Identify how to define alternative   |
|                          | scenarios.                                |
|                          |                                           |
|                          | C4.5 Identify issues for additional       |
|                          | consideration.                            |
+--------------------------+-------------------------------------------+
| C5. Effect of a          | C5.1 Identify a method(s) to estimate or  |
| government\'s R&D        | systematically describe the impact of a   |
| programs on knowledge    | government R&D program on knowledge       |
| value                    | value.                                    |
|                          |                                           |
|                          | C5.2 Identify data sources.               |
|                          |                                           |
|                          | C5.3 Identify issues for additional       |
|                          | consideration.                            |
+--------------------------+-------------------------------------------+
| C6. Using estimates of   | C6.1 Suggest how to use estimates of      |
| benefits in R&D program  | benefits and other performance measures   |
| planning and evaluation  | in the planning and evaluation of R&D     |
|                          | programs.                                 |
+--------------------------+-------------------------------------------+

**8.5 Workshop D: Security, Economic, and Environmental Benefits**

Energy security in the past has typically focused on vulnerability to
oil imports in a cartel market. Since September 11, 2001, the
vulnerability of energy infrastructure to catastrophic disruptions has
become a greater concern. In general, energy security refers to the
adequate and reliable delivery of energy products and services with
reasonably stable prices in a competitive market. In addition to oil
security and energy infrastructure security, energy security includes
electric power reliability. Workshop D will focus on identifying the
nature of energy security, how it can be measured and valued, and how
improvements in it can be determined and evaluated. Workshop D will also
address economic and environmental benefits. These latter topics have
already been the subject of extensive study (e.g., global climate
change). Sessions in Workshop D will identify the economic- and
environmental-benefit literatures and, possibly, the range of values
which could be used. These sessions will also identify other economic
and environmental considerations, such as the economy-wide impacts of
R&D, and suggest methods for estimating their relative value.

**Objectives of Workshop D Sessions on**

Security, Economic, and Environmental Benefits

+--------------------------+-------------------------------------------+
| **Session**              | **Objectives**                            |
+--------------------------+-------------------------------------------+
| D1. Framework for        | D1.1 Discuss methodological framework for |
| defining the benefits of | identifying benefits of energy R&D        |
| R&D programs, and its    | programs.                                 |
| use for GPRA and R&D     |                                           |
| planning and evaluation  | D1.2 Identify how various \"benefits\"    |
|                          | fall within this framework.               |
|                          |                                           |
|                          | D1.3 Identify benefits inadequately       |
|                          | addressed by the framework.               |
+--------------------------+-------------------------------------------+
| D2. Security benefits of | D2.1 Define the major types of security   |
| R&D on energy            | benefits that can ultimately result from  |
| technologies and systems | energy R&D.                               |
+--------------------------+-------------------------------------------+
| D3. Continued discussion | D3.1 Identify methods to estimate these   |
| of security benefits     | benefits.                                 |
|                          |                                           |
|                          | D3.2 Identify data sources.               |
+--------------------------+-------------------------------------------+
| D4. Economic benefits of | D4.1 Define the major types of economic   |
| R&D on energy            | benefits that can ultimately result from  |
| technologies and systems | energy R&D.                               |
|                          |                                           |
|                          | D4.2 Identify methods to estimate these   |
|                          | benefits.                                 |
|                          |                                           |
|                          | D4.3 Identify data sources.               |
+--------------------------+-------------------------------------------+
| D5. Environmental        | D5.1 Define the major types of            |
| benefits of R&D on       | environmental benefits that can           |
| energy technologies and  | ultimately result from energy R&D.        |
| systems                  |                                           |
|                          | D5.2 Identify methods to estimate these   |
|                          | benefits.                                 |
|                          |                                           |
|                          | D5.3 Identify data sources.               |
+--------------------------+-------------------------------------------+
| D6. Using estimates of   | D6.1 Suggest how to use estimates of      |
| benefits in R&D program  | benefits and other performance measures   |
| planning and evaluation  | in the planning and evaluation of R&D     |
|                          | programs.                                 |
+--------------------------+-------------------------------------------+

[^1]: National Research Council's Committee on Benefits of DOE R&D on
    Energy Efficiency and Fossil Energy, *Energy Research at DOE: Was It
    Worth It?*, Washington, DC: National Academy Press, July 2001. The
    report was requested by the Appropriations Committee of the U.S.
    House of Representatives.

[^2]: Conference facilitators and rapporteurs will identify convergence,
    as well as divergence, of ideas on methods that might be useful to
    these DOE offices. The purpose of the conference and of the
    subsequent draft report, to be written following the conference, is
    to identify and to suggest methods to these DOE offices, but not to
    provide recommendations or to act in any advisory capacity.

[^3]: National Research Council's Committee on Benefits of DOE R&D on
    Energy Efficiency and Fossil Energy, *Energy Research at DOE: Was It
    Worth It?*, Washington, DC: National Academy Press, July 2001. The
    report was requested by the Appropriations Committee of the U.S.
    House of Representatives.

[^4]: Executive Office of the President, Office of Management and
    Budget, *The President's Management Agenda, Fiscal Year 2002*,
    August 2001.

[^5]: ^?^ National Energy Policy Development Group, *National Energy
    Policy*, Washington, DC, May 2001.

[^6]: This section was written by Mary Beth Zimmerman, Office of Energy
    Efficiency and Renewable Energy, U.S. Department of Energy.

[^7]: This section was written by Jay Braitsch, Office of Fossil Energy,
    U.S. Department of Energy.

[^8]: This section was authored by Robert Vallario of the Office of
    Science, U.S. Department of Energy.

[^9]: This section was written by Jeanne Powell, Senior Economist,
    Economic Assessment Office, Advanced Technology Program.

[^10]: The scoring system was developed by Rosalie Ruegg, TIA
    Consulting.

[^11]: Chapter 4, Five Year Strategic Plan for Railroad Research,
    Development, and Demonstrations, FRA, soon to be released. Provided
    by Robert C. Ricci of the Volpe Center.

[^12]: James L. Wolf authored Section 5 and Sections 7.4 and 7.5.

[^13]: Note that these benefits are those from energy technologies.
    Other R&D programs and technologies might have other types of
    impacts, e.g., public safety, mobility, and reduced (as well as
    economic, environmental, and security) congestion in the
    transportation sector.

[^14]: We suggest that tabulations that report the benefits in five-year
    intervals have a footnote stating that the present value of benefits
    in distant years is considerably less than those in the near-term,
    other things being equal.

[^15]: In developing estimates of such distributions, the burden on
    respondents should be kept to a minimum. For example, the functional
    form of the probability distribution can be assumed *a priori* and
    the specific distribution obtained through one or two simple,
    intuitive questions.

[^16]: ATP is part of the National Institute of Standards and Technology
    (NIST) in the U.S. Department of Commerce. ATP aims to bridge the
    gap between research labs and the marketplace by co-funding projects
    to help bring technologies closer to commercialization. Companies
    conceive, propose, co-fund and execute the projects. ATP focuses on
    only those projects in which there is significant potential for
    national or societal gain, in addition to private gain to the
    company.

[^17]: These five cases were obviously not included in calculating the
    average lag-time above.

[^18]: Robert Constanza, Ralph d\'Arge, Rudolf de Groot, Stephen Farber,
    Monica Grasso, Bruce Hannon, Karin Limburg, Shahid Naeem, Robert V.
    O\'Neill, Jose Paruelo, Robert G. Raskin, Paul Sutton, and Marjan
    van den Belt (1997) \"The Value of the World\'s Ecosystem Services
    and Natural Capital,\" *Nature*, vol. 387, 15 May 1997, pp. 253-260.

[^19]: Charles Hall, Dietmar Lindeberger, Reiner Kummel, Timm Kroeger,
    and Wolfgang Eichhorn (2001) \"The Need to Reintegrate the Natural
    Sciences with Economics,\" *BioScience*, vol. 51, no. 8, pp.
    663-673.