Conference on

## Estimating the Benefits of 

## Government-Sponsored Energy R&D[^1]

**March 4 and 5, 2002**

**Arlington, Virginia**

### SYNTHESIS of CONFERENCE DiscussionS[^2]

**May 1, 2002**

Conference on

## Estimating the Benefits of Government-Sponsored Energy R&D

**March 4 and 5, 2002**

**Arlington, Virginia**

### Synthesis of CONFERENCE DiscussionS

**SUMMARY**

A conference was held on March 4 and 5, 2002 to gain insights that the
energy resource and science offices of the U.S. Department of Energy
(DOE) might use to improve their methods for estimating the benefits of
their R&D programs. The conference participants consisted of 151 experts
in R&D program assessment, planning, and management; and in the
specialized methods used in these activities. They were from federal and
state government, industry, academia, and private research
organizations.

Offices in DOE are giving significant priority to measuring and
assessing the performance of their R&D programs. Under the Government
Performance and Results Act of 1993 (GPRA), DOE and other federal
agencies are required to report annually on their programs\' plans and
performance. Furthermore, President George W. Bush and the U.S. Congress
are placing great emphasis on performance-based assessments in the
planning of federal programs.[^3]

The Appropriations Committee of the U.S. House of Representatives had
also requested that the National Research Council (NRC) conduct a
retrospective study of the benefits of some of DOE\'s energy efficiency
and fossil energy programs.[^4] As part of its study, the NRC committee
developed a methodological framework for estimating the benefits of
these programs. The March 4 and 5 \"Benefits Conference\" built on the
results of this study and considered ways of adapting and refining the
NRC framework for possible use by DOE offices to address their GPRA and
other performance management needs.

***Benefits Framework***

The NRC committee developed a framework for retrospective analysis that
is represented by a matrix with three rows and three columns. The rows
reflect DOE\'s energy-resources programs\' strategic objectives, which
are the ultimate impacts of their energy R&D: economic, environmental,
and security benefits. The columns reflect the degree of uncertainty
about the possible commercialization of the technologies that result
from the R&D: realized, option, and knowledge benefits.

The NRC framework appeared to hold up well to conference participants\'
questions and critiques. Three modifications emerged as recurring themes
among many of the conference participants:

-   A new column to account for the projected *prospective* benefits of
    R&D programs under the most likely scenario of events (i.e., the
    base case).[^5] This modification would assist these DOE offices to
    integrate their assessments of the prospective and retrospective
    benefits of their programs, for the purposes of program planning and
    evaluation.

-   A change in the way in which knowledge benefits are considered, so
    that they are represented by a row in the matrix rather than by a
    column. This change reflects the fact that knowledge is a core
    mission of the science programs in DOE, as well as of some of the
    energy resource programs. Also, many participants that addressed the
    issue in the conference workshop thought that various types of
    knowledge are enablers of innovation, which contributes to all other
    types of benefits generated by applied research and technology
    development programs.

-   A refined interpretation of the columns of the matrix, to represent
    scenarios with varying degrees of certainty \-- the past (the
    realized benefits that apply only to retrospective assessments); the
    projected base case of future conditions; and other \"option
    cases,\" which are scenarios used to implement the notion of option
    value. Many participants thought that more discussion is necessary
    to determine how these columns would be applied to the basic
    research programs.

The latter idea of option cases (i.e., alternative, less probable
scenarios) reflects the suggestion favored by many of the participants
in the workshop on option value. Their idea was to introduce the idea of
\"optionality\" into the R&D planning process. An option approach was
favored because the outcomes of R&D are inherently uncertain, as are
future economic, geopolitical, regulatory, and policy conditions.
Several conference participants noted that planning for less-probable
scenarios is an integral aspect of DOE\'s mission.

Figure S1 represents a synthesis of conference participants\'
suggestions about the methodological framework. This figure was actually
*not* constructed during the conference, but appears to represent a
reasonable synthesis of many of the salient ideas about the framework
that emerged during the discussions.

The top part of the figure casts the framework within the broader
context of a program-planning or evaluation scorecard, or of possible
R&D investment criteria. Measures of benefits generally fall under the
\"Performance\" category of measures or criteria. But they could also be
reflected in the other categories, as represented by the blue-shaded
areas. These areas could be criteria such as the \"public benefit\"
aspects of a program, or the \"commercial timeframe\" of the technology
being developed.

> R&D Investment or Scorecard Criteria:

Relevance/Role Quality/Merit Performance

+------------------+-----------------+---------------+-----------------+
|                  | **Past**        | **Future**    |                 |
+------------------+-----------------+---------------+-----------------+
|                  | **Realized**    | **Projected** | **Other         |
|                  |                 |               | Option**        |
|                  |                 | **Base Case** |                 |
|                  |                 |               | **Cases**       |
+------------------+-----------------+---------------+-----------------+
| **Economic**     |                 |               |                 |
+------------------+-----------------+---------------+-----------------+
| *                |                 |               |                 |
| *Environmental** |                 |               |                 |
+------------------+-----------------+---------------+-----------------+
| **Security**     |                 |               |                 |
+------------------+-----------------+---------------+-----------------+

  ------------------ ----------------- ---------------- -----------------
  **Knowledge**                                         

  **Knowledge &                                         
  Ideas**                                               

  **Research Tools**                                    

  **Human Capital**                                     

  **Communities of                                      
  Practice**                                            

  **Transitions &                                       
  Spin-offs**                                           
  ------------------ ----------------- ---------------- -----------------

**Figure S1. Framework for Estimating the Benefits of Energy R&D**

**(Blue-Shaded Areas Represent Benefits-Related Metrics)**

The middle part of the figure is an adaptation of the NRC matrix in
which knowledge is no longer represented by a column of the matrix, but
by a row. Many participants in the workshop on knowledge benefits
suggested that knowledge is also a third dimension of the other cells of
the matrix because knowledge is a contributor to outcomes for technology
R&D programs.

The bottom part of the figure expands the knowledge benefits definition
to be \"knowledge-based capacity,\" which includes five categories of
benefits. These would pertain principally to the science programs.
Economic, environmental and security benefits can similarly be expanded
with categories of benefits. Many conference participants suggested that
such a framework, or something similar to it, could be adopted and
implemented DOE-wide, or at least among the offices involved in the
conference.

Many conference participants considered the projected baseline
conditions, and the market penetration of the new technology that would
be the result of the R&D, as two of the more important factors that
affect estimates of a program\'s benefits. The projected baseline
establishes the next-best alternative and the conditions upon which the
new technology is to be introduced. Many conference participants
suggested that the Reference Case, used in the Energy Information
Administration\'s (EIA\'s) Annual Energy Outlook (AEO), could be a
starting point for defining both the projected baseline conditions and
the market penetration of the new technology. EIA\'s Reference Case is
generated using the National Energy Modeling System (NEMS), which is a
large-scale integrated model of energy markets and technologies.

Many conference participants suggested that NEMS is limited in its
representation of specific types of technologies. To address this
limitation, several conference participants suggested the use of models
that can more easily characterize technologies, such as MARKAL, which
would be calibrated to the NEMS AEO Reference Case. However, other
conference participants considered that any large-scale model is
inherently limited in making accurate long-term projections. They argued
that information gathered from direct face-to-face discussions with
industry, stakeholders, and other knowledgeable parties could be more
reliable than the projections of any model.

The private sector, and possibly other organizations, generally
contributes significantly to the success of a new technology, in
addition to DOE\'s contribution. An important concern among many
conference participants was that the impact of the government on the
overall success of a technology should be properly considered.
Conference participants identified different ways of estimating the
impact of the government. One approach discussed was a generalization of
the so-called 5-year rule defined by the NRC study, which assumes that
the impact of the government is to accelerate commercialization of
technologies by five years. Many conference participants favored a
generalization of the rule to account for the fact that the
government\'s acceleration of the commercialization process is expected
to vary, depending on the type of technology as well as on many other
factors. Another approach supported by several participants was to base
the estimate on direct discussions with companies involved in the
technology. Many conference participants thought that the prospective
impact of the government on commercialization could be estimated on a
case-by-case basis but that such assessments should be transparent,
clear, and peer reviewed.

***Estimating the Various Benefits***

Conference participants offered many different measures for security,
economic, environmental, and knowledge benefits. There appeared to be
agreement among many conference participants that it might be
appropriate to have more than one measure for any given type of benefit.

In addition to the economic value of the reduction in energy consumption
and costs, now used by several DOE offices, conference participants
noted other measures and methods for estimating economic benefits:

-   A cost index based on expected consumer costs, with and without the
    new technology,

-   Econometric analysis to estimate spill over and macroeconomic
    impacts of sector-wide R&D investment, and

-   Case studies (which can be used for estimating other types of
    benefits as well).

In addition to the estimates of reduced emissions of criteria pollutants
and carbon dioxide, now used by several DOE offices, conference
participants noted other measures and methods for estimating
environmental benefits:

-   An Index of Sustainable Economic Welfare, based in part on impacts
    on ecosystem services (which have been estimated on a \$ per hectare
    basis for different types of ecosystems), and

-   Economic approaches for valuing environmental benefits, such as
    hedonic methods (econometric methods that identify relationships
    between attributes such as property value and environmental
    quality), contingent valuation (economic experiments to estimate
    individuals\' valuing of environmental damages avoided), and direct
    measures of market value (such as lost productivity or crop yield).

In addition to the estimates of reduced oil consumption, which some DOE
offices now use, conference participants suggested many indicators of
energy security benefit:

-   Supply and demand conditions such as energy import levels, diversity
    of energy supply, and insurance rates and costs for protecting
    energy infrastructure;

-   System flexibility and sensitivity, such as price elasticity of U.S.
    demand for various fuels, substitutability, fuel stocks, and
    macroeconomic sensitivity to energy shocks; and

-   Reliability/volatility measures such as the frequency and duration
    of interruptions (e.g, power outages), grid reliability (e.g., power
    quality), capacity to meet peak energy demands, transmission
    congestion costs, and surveys of public confidence about energy
    security.

Knowledge-based capacity benefits were identified as including
generation of new or improved ideas, research tools, human capital, and
communities of practice. These benefits lead to enhanced agility and
capacity for research and increased opportunities to transition or apply
knowledge, which in turn lead to benefits in the market place. Suggested
indicators and methods included:

-   Expert judgement on the quality, relevance, progress and prospects
    of the research.

-   Analysis of publications, citations and patents (bibliometric
    methods) that consider, for example, patents that cite papers and
    patent portfolios, to identify the networks and linkages between
    science and technology innovation \-- insights that can be applied
    prospectively as well as retrospectively;

-   Technical milestones attained, in cases where these can be
    projected;

-   Number of graduate students trained;

-   Connectivity of communities of practice; and

-   Intellectual property generated.

In general, many participants discussed that some of these approaches
are not yet well defined for GPRA- or performance-based applications, or
are costly to implement; and they suggested a phased, multi-year
approach to add metrics to improve the performance management of DOE\'s
programs.

***Program Planning and Decision Making***

For program planning and decision making, many conference participants
thought it important to view individual programs as part of a broader
energy R&D portfolio, and to regard estimates of benefits as but one
criterion in R&D investment decisions. Other criteria could include the
R&D\'s time frame, project cost, and technical risk. Many conference
participants liked the use of visual representations of portfolio
analyses. These graphics depict the interplay and trade-off among
decision criteria.

The importance of portfolio management was highlighted by several
conferees. The workshop on option value discussed the need to cut across
DOE stovepipes to assess the benefits of R&D portfolios. Decisions to
terminate a program were regarded by many as being as important to
consider as those to initiate or continue one.

A few conference participants cautioned against relying too much on
complicated frameworks and methodologies. They reminded conferees that
there is no substitute for \"common sense\" in R&D program planning and
evaluation.

Certain themes tended to surface frequently throughout the conference.
Many participants appeared to agree on the need to have a uniform
framework that would be tailored to capture programmatic differences but
used across all relevant DOE offices, as well as a consistent set of
assumptions and projected baseline. They thought the methods should be
transparent, and that methods and assumptions should be peer reviewed.
Many participants also agreed on the need to continually improve the
methods for estimating the various types of benefits and to do pilot
testing of these methods on selected programs before implementing them
office-wide.

Conference on

## Estimating the Benefits of Government-Sponsored Energy R&D

**March 4 and 5, 2002**

**Arlington, Virginia**

### Synthesis of CONFERENCE DiscussionS

# 

A conference on \"Estimating the Benefits of Government-Sponsored Energy
R&D\" was held on March 4 and 5, 2002. Its purpose was to gain insights
about methods, which the energy resource and science offices of the U.S.
Department of Energy (DOE) might use to improve their assessment of the
benefits of their R&D programs for the purposes of program planning and
measuring their performance. The conference was organized by Oak Ridge
National Laboratory (ORNL), and was sponsored by DOE\'s offices of
Energy Efficiency and Renewable Energy (EERE), Fossil Energy (FE),
Nuclear Energy, Science and Technology (NE), and Science (SC). There
were 151 participants including: senior DOE management; DOE R&D program
managers; R&D program managers from other federal agencies; staff of the
Office of Management and Budget (OMB); staff from the General Accounting
Office (GAO); Congressional staff; stakeholders; researchers and
analysts from universities and national laboratories; and consultants
and contractors.

The motivation for the conference stemmed from a widespread desire among
Congress, the Executive Office of the President, GAO, and DOE to
undertake a thorough performance-based approach to managing, planning
and evaluating federal R&D investments. All of these parties are placing
great emphasis on improving the implementation of the Government
Performance and Results Act of 1993 (GPRA) which requires federal
agencies to report annually on their programs\' plans and performance.

Figure 1 illustrates relationships between the inputs to an R&D program
(i.e., the resources devoted to it) and its ultimate, end outcomes.
Various research-planning scorecards, R&D investment criteria, GPRA data
calls, and other metrics have been used as performance-based measures of
R&D success. The more recently proposed metrics have placed greater
emphasis on assessing the out*comes* of the programs\' performance \--
their longer-term impacts and benefits \-- in addition to their
out*puts* such as program deliverables and milestones. Thus, regardless
of the specific scoring system or investment criteria used to evaluate
programs, one would expect that their estimated benefits \-- both
retrospective and prospective \-- will be a key performance measure.

**Figure 1. Logical Flow of R&D Inputs, Activities, Milestones, Outputs
and Outcomes**

A recent National Research Council (NRC) report, which was requested by
Congress, assessed the performance of many of DOE\'s energy efficiency
and fossil energy technology programs by estimating their retrospective
benefits.[^6] The methodological framework developed by the NRC study
was the basis for a strawperson framework that provided a starting point
for discussions at the conference.[^7]

The conference consisted of plenary sessions and four workshops. The
plenary sessions presented the objectives of the conference, challenges
to improve the methods used by DOE offices, and summaries of various
methods that they and other federal agencies currently use. Each
workshop focused on a key methodological issue(s) related to the
structure and refinement of the NRC framework and to the methods for
estimating the various types of benefits within the refined
framework.[^8] The primary focus in all of these discussions was on a
consistent framework for measuring the benefits of R&D, and on methods
for estimating them, which the DOE offices could begin to implement.

Many different methods were mentioned among conference participants for
the various types of benefits, within the overall methodological
framework. Given the limited time available at the conference, its
participants purposefully focused on the \"forest and not the trees.\"
Thus, conference participants generally did not offer detailed
descriptions of specific methodologies. Instead, participants focused
more on the general methodological framework and on its key concepts,
and referred to the *general types* of methods that could be used to
estimate the various types of benefits.

***Setting the Stage: Current Practices and Challenges***

The Assistant Secretary of the Office of Fossil Energy in DOE and the
Chief of Staff of the U.S. House of Representatives Subcommittee on
Appropriations for Interior and Related Agencies challenged conference
participants to address some of their key needs:

-   identifying the benefits of the R&D carried out by various DOE
    programs,

-   setting criteria for estimating the prospective benefits of these
    programs,

-   establishing objective methods that can be applied across all
    programs, and

-   having practical ways of improving the programs and for
    communicating their benefits.

DOE senior staff described their current approaches for estimating the
benefits of their programs. In one presentation, the EERE staff member
discussed that EERE has begun to implement many of the methods developed
for the NRC study. She noted the types of benefits within NRC\'s
framework, which EERE is currently estimating; the types of benefits
which it is planning to address; and those for which little study has
been done thus far. She also noted that EERE is planning to integrate,
over the next several years, its estimating of retrospective and
prospective benefits. The FE staff member reviewed the two different R&D
areas in FE: fossil fuel conversion and domestic supply. Different
evaluation methods and models are used for each. NE staff discussed the
diverse R&D areas within that office: nuclear power, medical diagnostics
and therapeutics, space, defense, and industrial applications.
Allocation of resources to these areas is based on priorities to
maintain current streams of benefits, expand programs and their
benefits, or abandon them. The SC staff member pointed out the inherent
difficulty in characterizing the value of new knowledge. He reviewed
methods SC currently uses to assess its programs such as peer review and
cost-schedule milestones. He noted that although scientific progress is
often serendipitous, methods for linking science to energy portfolios
are indicators of the value of science to applications that benefits
society.

Staff from other federal agencies, specifically the Advanced Technology
Program (ATP) in the National Institute of Standards and Technology
(NIST), and the Volpe Center in the U.S. Department of Transportation
(DOT), described some of the methods they currently use:

-   estimates of outputs (e.g., number of R&D partners), intermediate
    outcomes (e.g., acceleration of commercial activity), and impacts or
    ultimate outcomes (e.g., economic benefits).

-   business reporting systems, supplemented by telephone surveys,

-   case studies of the return on investment and the net present value
    of programs,

-   macroeconomic analyses,

-   composite performance rating of project performance, based on
    knowledge creation and dissemination, commercialization, and
    technology diffusion,

-   fault-tree analysis to identify causes of damaging events
    (accidents, in the case of DOT) and the potential benefits of
    technology R&D to address them, and

-   assessments by experts.

A senior OMB manager discussed the priority given by the current
administration to improving accountability in government. He discussed
the need to be able to communicate the benefits of R&D clearly and
simply, and to be able to discriminate among different programs. He made
the point that OMB is skeptical of using knowledge benefits as the sole
rationale for *applied* research. He emphasized the need to have better
estimates of the government\'s contribution to the overall benefits of
the technologies associated with an R&D program, and reiterated that
performance-based assessments of R&D programs are one of the
administration\'s top management objectives.

The conference\'s keynote speakers highlighted, respectively:

-   NE\'s systematic approach for doing the \"right R&D\": understanding
    the big picture and the needs, determining the technology goals,
    deciding how to reach these goals, deciding who performs the work,
    and continuously reassessing scope, performance and quality.
    \[William D. Magwood, IV, Director of the Office of Nuclear Energy,
    Science and Technology, DOE\];

-   EERE\'s commitment to improving its methods for estimating the
    benefits of its R&D, and the importance of results-oriented GPRA
    evaluations of their programs to ensure that EERE is delivering the
    best possible improvements in energy efficiency and renewable energy
    to the country\'s taxpayers. \[David K. Garman, Assistant Secretary
    of the Office of Energy Efficiency and Renewable Energy, DOE\]; and

-   The great importance of performance-based assessment and the GPRA
    process to this administration, the usefulness of measuring
    *intermediate* outcomes in the long-term R&D process, and the
    recognition that the final outcomes of R&D programs might be too
    distant to effectively defend their value. \[Joseph S. Wholey,
    Professor at the University of Southern California and, recently,
    senior advisor at GAO\].

Members of the NRC Committee that authored the aforementioned report
described the origins and motivation for the study. They recalled that
some in Congress questioned the magnitude of the benefits of government
R&D. The Chair of the Committee pointed to key challenges in taking the
next steps from their study: the need to account for uncertainty in
prospective analysis; the need to have consistent, transparent, and
tractable methods; and the need for rigorous rules for things that are
fuzzy (such as the impact of the government on the success of a
technology). The Chairman of the Committee\'s Subgroup on the Benefits
Framework then outlined the framework and its rationale. The framework
is described more fully in the NRC report, and is summarized in the
white paper that was distributed prior to the conference.

The Conference Chairman then offered some thoughts on the priorities,
challenges, and \"strawman\" ideas for building on the NRC framework so
that DOE offices could improve their methods for estimating the benefits
of their R&D programs for the purposes of GPRA reporting, program
planning, evaluation and budgeting. These methodological challenges were
the basis of the conference\'s four workshops that focused,
respectively, on:

-   cross-cutting issues in estimating prospective benefits;

-   option value;

-   the benefits of knowledge; and

-   security, economic, and environmental benefits.

This report is a synthesis of the discussions in the four workshops.
Consensus or recommendations were not sought from conference
participants. The remaining sections of this report synthesize
conference participants\' views on:

-   a general methodological framework, adapted from that developed by
    the NRC committee,

-   measuring and estimating retrospective benefits,

-   measuring and estimating prospective benefits,

-   option value,

-   economic benefits,

-   environmental benefits,

-   security benefits

-   knowledge benefits,

-   data sources, and

-   use of estimated benefits in program planning and evaluation.

***Framework for Estimating the Benefits of R&D Programs***

The starting point for discussions in the conference\'s workshops was
described in the conference white paper as the NRC committee\'s
methodological framework for estimating R&D benefits, with the addition
of a category for projected prospective benefits. The reason for adding
this category was that GPRA analysis, planning, and budgeting of R&D
programs all require prospective assessments of the value or return on
the R&D investment; whereas, due to the limited scope of the study, the
NRC matrix was developed solely for the purpose of a retrospective
estimate of benefits. That is, the NRC study was about the past success
of programs, whereas program planning and budgeting must also estimate
the future success toward which DOE\'s R&D programs are striving.

The white paper offered some \"strawman\" approaches and suggested that
a matrix with three rows and four columns could be used to represent the
methodological framework. Following the NRC framework, the white paper
suggested a matrix that had three basic types of impacts listed along
one dimension of the matrix (i.e., the rows): economic, environmental,
and security. These impacts reflect the core objectives, or ultimate
outcomes, toward which the DOE R&D enterprise is striving. The other
dimension of the matrix (i.e., the columns) reflects the degree of
certainty about the specific commercial benefits of the R&D: realized,
prospective base case, options, and knowledge.

Realized benefits are those that have already occurred as a result of
past R&D. Prospective base-case benefits are estimates of future
benefits under a base case, or most likely, scenario of future
conditions. Option benefits are the value of an asset (e.g., a
technology) that provides the ability to retain choices that can be
implemented in the future, when there is uncertainty about future
conditions. Knowledge benefits are those associated with the immediate
insights, ideas and research tools gained from scientific studies; and
the effects of the scientific process on human capital and scientific
and technical communities of practice.

With such a diverse and expert group of conference participants, it was
not surprising that conference participants\' discussions about adapting
the NRC framework to consider benefits prospectively, as well as
retrospectively, resulted in a rather thorough scrutiny and critique of
the framework. Some conference participants thought it was too complex.
They favored using fewer measures of a program\'s benefits and success,
and more \"common sense\" in making such assessments. Other conference
participants thought the framework was overly simplified and restrictive
about certain types of benefits such as security and knowledge, and
argued for a more complex and comprehensive framework.

In the end, however, many conference participants appeared to find the
framework to be generally acceptable and useful as part of a larger
framework for R&D planning and evaluation \-- albeit with suggestions
for its expansion or refinement. Table 1 summarizes the main
suggestions.

**Table 1. Major Issues Raised and Needs for Refinement in the Benefits
Matrix**

+--------------------+-------------------------------------------------+
| **Aspect of the    | **Nature of the Caveat or Suggested             |
| Framework**        | Refinement**                                    |
+--------------------+-------------------------------------------------+
| Overall framework  | -   Defining carefully the interaction and      |
|                    |     relationships among the different benefits  |
|                    |     categories to ensure that they are          |
|                    |     well-defined and non-overlapping            |
|                    |                                                 |
|                    | -   Considering refinements to the matrix such  |
|                    |     that the *columns represent scenarios* that |
|                    |     reflect the temporal dimension as well as   |
|                    |     varying degrees of certainty about the      |
|                    |     outcomes \[i.e., the scenarios would be:    |
|                    |     the past, the projected base case, and      |
|                    |     other alternative futures (option cases)\]; |
|                    |     and *the rows represent DOE strategic       |
|                    |     objectives* \[i.e., energy security,        |
|                    |     economic, environmental, and knowledge      |
|                    |     benefits\]                                  |
|                    |                                                 |
|                    | -   Developing better ways of attributing the   |
|                    |     impacts of the government R&D program       |
|                    |     within the overall success of a technology  |
+--------------------+-------------------------------------------------+
| Economic impacts   | -   Identifying and estimating important        |
|                    |     second-order benefits, such as improved     |
|                    |     productivity, spill over effects, and       |
|                    |     macroeconomic benefits                      |
|                    |                                                 |
|                    | -   Considering equity and the distribution of  |
|                    |     impacts among different parts of society    |
+--------------------+-------------------------------------------------+
| Environmental      | -   Expanding the nature of the environmental   |
| impacts            |     impacts that are considered to include      |
|                    |     ecosystem services                          |
+--------------------+-------------------------------------------------+
| Security impacts   | -   Recognizing and estimating the many         |
|                    |     different types, and the interdependence    |
|                    |     among, security impacts and benefits        |
+--------------------+-------------------------------------------------+
| Prospective        | -   Defining precisely what is meant by the     |
| benefits           |     base case, business-as-usual case, or       |
|                    |     reference case                              |
|                    |                                                 |
|                    | -   Developing methods to define a common set   |
|                    |     of baseline assumptions, and to calibrate   |
|                    |     analyses and models to this baseline        |
+--------------------+-------------------------------------------------+
| Option value       | -   Recognizing that real options valuation is  |
|                    |     a well-established investment-management    |
|                    |     approach that estimates the benefits of R&D |
|                    |     by taking *advantage* of the uncertainty    |
|                    |     and risks about both the ultimate technical |
|                    |     performance of the technology and future    |
|                    |     economic and regulatory conditions (perhaps |
|                    |     initially implementing this concept with a  |
|                    |     scenarios approach, as a simple,            |
|                    |     alternative means of characterizing the     |
|                    |     value of R&D under uncertain futures)       |
+--------------------+-------------------------------------------------+
| Knowledge value    | -   Modifying the matrix so that knowledge,     |
|                    |     which is a core mission of the science      |
|                    |     programs, is a row of the matrix, and also  |
|                    |     a third dimension in all other benefits,    |
|                    |     rather than a column of the matrix          |
+--------------------+-------------------------------------------------+
| Use of the         | -   Addressing the variability in project       |
| framework          |     timeframes and the timing of their future   |
|                    |     benefits and costs                          |
|                    |                                                 |
|                    | -   Developing practical cost-effective ways of |
|                    |     implementing and putting into practice the  |
|                    |     framework and the estimation methods        |
|                    |                                                 |
|                    | -   Developing an office-wide, or possibly      |
|                    |     business-line, approach to assessing and    |
|                    |     planning the *portfolio* of R&D programs    |
|                    |     using estimates of their benefits and other |
|                    |     key performance measures                    |
+--------------------+-------------------------------------------------+

One of the major modifications of the NRC framework, suggested by many
conference participants, was to represent knowledge benefits more
meaningfully from the perspective of those engaged in planning and
assessing basic research programs. Many of the participants in the
workshop on knowledge benefits thought that the creation of
\"knowledge,\" being a core objective of DOE, should be a row in the
matrix. Furthermore, these conference participants viewed knowledge as a
third dimension of the matrix, underlying all of the benefits of applied
research and technology programs. Their idea reflects their point that
knowledge interfaces with, and is an input to, all other benefits.
Others at the conference, on the other hand, favored retaining the NRC
matrix for use with applied research programs, with knowledge as a
column; and using a separate matrix for the science programs. Many
participants in the workshop on knowledge benefits also generally
accepted the idea of knowledge as a \"thin\" column in the matrix for
the applied research programs, that is, as an enabling but not primary
category for the analysis of their benefits.

Many conference participants favored adding a column to the matrix to
represent projected prospective base-case benefits. A few participants
did not relate to the idea of having such a prospective-benefits column,
however; and some favored using two different matrices for retrospective
versus prospective assessments.

Several participants in the workshop on option value questioned the need
to have a prospective benefits column, or for that matter a knowledge
column in the matrix. For these participants, real options valuation is
a complementary alternative to the traditional type of net present value
analysis used to calculate prospective benefits \-- but is not additive
to these benefits. These participants generally deferred, however, to
others in the workshop concerned about the amount of information needed
for a comprehensive option valuation. The latter group of participants
also cautioned that the real options paradigm, although promising, is so
different from the current thinking of most people that it would have to
be introduced *gradually* into R&D management and benefits-estimation
practice.

Figure 2 represents a synthesis of many of the various viewpoints,
shaped into an integrated framework. Here, the \"End Outcomes\" box in
Figure 1 is placed into the broader context of criteria for assessing
whether to invest in R&D and is also expanded into its basic categories.
Figure 2 was actually *not* constructed during the conference, but
appears to represent a reasonable synthesis of many of the salient ideas
about the framework that emerged during the discussions.

The top part of the figure represents the fact that \"benefits\" can be
viewed as one set of many possible performance measures. In the figure,
the blue-shaded areas represent benefits-related measures. R&D
evaluation or investment \"scorecards\" also typically include criteria
such as \"relevance\" and \"quality.\" Although measures of benefits
generally fall under the \"performance\" category of criteria, they
could also be in the other two categories, as represented by the
blue-shaded areas under those categories. Examples could be the
\"*public* benefit\" aspects of a program, or the \"commercial
timeframe\" of the technology being developed.

The middle part of Figure 2 is a \"drilling down\" of the benefits set
for applied R&D programs, to the next level of detail. The figure
reflects modifications to the NRC framework, which many conference
participants appeared to favor:

-   A new column to account for the projected *prospective* benefits of
    R&D programs under the base case, so that DOE offices can integrate
    their assessments of the prospective and retrospective benefits of
    their programs and can have both types of information for the
    purposes of program planning and evaluation.

```{=html}
<!-- -->
```
-   Knowledge benefits represented as a row rather than as column in the
    matrix, to reflect the fact that knowledge is a core mission of most
    of the science programs, as well as of some of the energy resource
    programs. Also, many participants in the workshop on knowledge value
    thought that the knowledge row should be in a third dimension of the
    other rows, to reflect their view that knowledge is an enabler of
    innovation that contributes to all other types of benefits.

-   The columns of the matrix now representing scenarios of varying
    degrees of certainty \-- the past (the realized benefits that apply
    only to retrospective assessments); the \"most-likely\" future,
    which is the projected base case; and other option cases.

The latter idea of option cases (i.e., less-probable scenarios) reflects
the suggestion favored by many of the participants in the workshop on
option value. They suggested generalizing the concept of options as
defined in the NRC report by introducing the idea of \"optionality\"
into the R&D planning and evaluation process to account for the
considerable uncertainty in the planning environment. Their idea was to
use scenario analysis as an initial, simplified approach to options
valuation.

R&D Investment or Scorecard Criteria:

Relevance/Role Quality/Merit Performance

+------------------+-----------------+---------------+-----------------+
|                  | **Past**        | **Future**    |                 |
+------------------+-----------------+---------------+-----------------+
|                  | **Realized**    | **Projected** | **Other         |
|                  |                 |               | Option**        |
|                  |                 | **Base Case** |                 |
|                  |                 |               | **Cases**       |
+------------------+-----------------+---------------+-----------------+
| **Economic**     |                 |               |                 |
+------------------+-----------------+---------------+-----------------+
| *                |                 |               |                 |
| *Environmental** |                 |               |                 |
+------------------+-----------------+---------------+-----------------+
| **Security**     |                 |               |                 |
+------------------+-----------------+---------------+-----------------+

  ------------------ ----------------- ---------------- -----------------
  **Knowledge**                                         

  **Knowledge &                                         
  Ideas**                                               

  **Research Tools**                                    

  **Human Capital**                                     

  **Communities of                                      
  Practice**                                            

  **Transitions &                                       
  Spin-offs**                                           
  ------------------ ----------------- ---------------- -----------------

**Figure 2. Framework for Estimating the Benefits of Energy R&D**

**(Blue-Shaded Areas Represent Benefits-Related Metrics)**

The bottom part of Figure 2 is a \"drilling down\" to obtain the next
level of detail about the knowledge benefits that result from basic
research programs and from the process through which applied research
programs lead to commercialized technologies. This \"sub-framework\"
reflects many of the suggestions that emerged from the workshop on
knowledge value.

The suggested refinements of the matrix that emerged from the conference
would allow the applied and basic research programs to be viewed under a
single framework. Perhaps more importantly, the refined framework
conveys the integrated nature of knowledge as a foundation for all other
types of benefits.

Many of the other suggestions during the conference emphasized the
multi-faceted nature of the individual categories of benefits (e.g.,
security is not a simple number) and the challenges and opportunities to
estimate them. Some conference participants noted that just as the
middle and lower parts of Figure 2 represent a drilling down of parts of
the upper section of Figure 2, each cell of the framework can be drilled
down to access information about the specific nature of the benefits in
that cell.

Figure 3 illustrates the evolution of the benefits framework, beginning
with the one developed in the NRC study, to the one suggested in the
white paper distributed prior to the conference, to the one that
appeared to emerge from the conference\'s discussions.

A frequent theme of comments at the conference was that R&D programs and
their estimated benefits should be considered collectively in deciding
on an overall R&D portfolio, where benefits are one of several
investment criteria. Thus, in the near term, DOE offices could use this
framework and their existing analysis tools to estimate the more
important types of benefits for each of their major programs. To ensure
comparability among estimates, DOE offices could use the same framework,
measures of each type of benefit, projected baseline conditions (to the
extent possible), and other common input assumptions. Several conference
participants suggested that, in the longer term (i.e., over several
years), these DOE offices could improve the analysis tools they use for
the more important categories of benefits that pertain to the more
important programs of the department.

However, several participants cautioned against overreaching by trying
to incorporate too complex a framework when the data to support it are
limited in scope, detail or precision. Some participants suggested that
common-sense rankings and peer reviews are a more sensible route to take
than complicated modeling.

Many conference participants emphasized that there are many different
types of benefits and, thus, different alternative measures of these
benefits. Many alternatives were mentioned, but conference participants
frequently suggested the need for simple measures and methods, where
possible; and for DOE offices to use the same non-overlapping measures
and indicators, and consistent units or reference points. Conference
participants did not resolve, however, how many types of measures, and
the level of specificity, DOE might use.

In the next sections of this report, we synthesize participants\'
suggestions for each category or type of benefit, following the
framework of the benefits matrix that appeared to emerge from the
conference.

NRC Matrix:

  ------------------------ --------------- ------------ ------------------
                           Realized        Option       Knowledge

  Economic                                              

  Environmental                                         

  Security                                              
  ------------------------ --------------- ------------ ------------------

Strawperson Matrix in White Paper:

+-----------------+----------+---------------+---------+-------------+
|                 | Realized | Expected      | Option  | Knowledge   |
|                 |          |               |         |             |
|                 |          | Prospective   |         |             |
+-----------------+----------+---------------+---------+-------------+
| Economic        |          |               |         |             |
+-----------------+----------+---------------+---------+-------------+
| Environmental   |          |               |         |             |
+-----------------+----------+---------------+---------+-------------+
| Security        |          |               |         |             |
+-----------------+----------+---------------+---------+-------------+

Framework that Emerged from the Conference:

  --------------------- ------------- ----------------- ------------------
                        Past          Future            

                        Realized      Projected Base    Other Option Cases
                                      Case              

  Economic                                              

  Environmental                                         

  Security                                              

  Knowledge                                             
  --------------------- ------------- ----------------- ------------------

**Figure 3. Refinement of the NRC Benefits Framework**

***Retrospective Benefits***

**Definition of Retrospective Benefits**

Retrospective benefits are those that are the result of past R&D. They
include benefits that have already been realized, plus the value of any
continuing benefits under the projected base case as well as those under
alternative future scenarios (i.e., option benefits).

Retrospective benefits could be measured in the following areas \--

-   Energy savings, reduced emissions, and reduced oil consumption
    associated with deployment and use of a technology whose development
    was partly the result of past DOE-sponsored R&D.

```{=html}
<!-- -->
```
-   Other types of economic, environmental and security benefits (refer
    to the discussions of these benefits in later parts of this report).

-   Knowledge-related benefits (see later discussion).

**Methods for Estimating Retrospective Benefits**

Basic methods to estimate retrospective benefits could account for the
following:

-   Energy and cost savings, and/or knowledge compared to those
    estimated for a counterfactual, next-best alternative. For many
    participants in the workshop on knowledge, the counterfactual
    alternative for \"knowledge\" is \"none\" when the advance is a
    breakthrough.

-   Benefits and costs that occurred in the past or that are expected to
    occur with a high degree of certainty even if the R&D program were
    immediately terminated. The NRC committee credited the life cycle
    benefits of technologies that it projected to be deployed up to the
    year 2005 because benefits of prior R&D could extend beyond the
    2000-2001 timeframe of the NRC study.

-   Advances in knowledge-based capacity as measured primarily by peer
    review and citation and patent analysis.

Retrospective benefits are important for reporting on the past
performance of programs, as required by GPRA. In fact, some conference
participants thought that the conference should also have devoted
greater attention to retrospective benefits than to prospective
benefits.

Several aspects of the calculation of retrospective benefits that
deserve some consideration, but which were not discussed at the
conference are:

-   The high cost of acquiring data on the past market penetration and
    use of technologies developed from DOE programs. For retrospective
    analysis, the NRC approach requires data on the historical, annual
    use of energy technologies \-- data that DOE does not compile.

-   (Inverse) discounting of past benefits and costs \-- should the
    discount rate be applied inversely to past years\' benefits and
    costs?

-   Should the projected deployment of a technology up to the year 2005
    (or over whatever near-term time horizon) be counted as realized
    benefits even though they occur in the future? This approach
    implicitly assumes that if the technology is already being used,
    then sales would continue over the next five years, even if the R&D
    program were completed or terminated. If that is the case, then
    these benefits would count as a *past* success, but would not count
    toward its *prospective* benefits.

***Prospective Benefits***

**Definition of Prospective Benefits**

Prospective benefits are the result of ongoing R&D or contemplated
future R&D that is not yet funded. They include the benefits under the
projected base case scenario, as well as option-related benefits (refer
to later discussion about option benefits). Prospective benefits could
be measured analogously to retrospective benefits:

-   Economic, environmental, and security benefits, net of any
    additional cost of the new technology, compared to the next-best
    alternative(s) from future deployment of the technology. Note that
    these benefits would include benefits that result from future
    outputs of ongoing research but *not* the benefits that might accrue
    in the future as a result of *past* R&D outputs.

-   As in the case of retrospective benefits, the benefits (and costs)
    are relative to the next-best alternative. For prospective analysis,
    this next-best alternative is reflected in the projected baseline
    conditions. One of the key issues debated by conference participants
    was the definition and estimation of this baseline (as discussed
    next).

-   The prospective benefits of knowledge are measured by peer review as
    part of competitive and merit-based selection of portfolios and
    projects.

**Methods for Estimating Prospective Benefits**

The key issue in estimating prospective benefits is that, unlike
retrospective benefits, they pertain to R&D investments that are ongoing
or being *contemplated*, rather than already incurred. Past conditions
(technical, economic, regulatory, and policy) are not directly relevant
to prospective benefits because they do not include benefits that have
been realized in the past. Realized benefits pertain only to
retrospective benefits. But *future* conditions pertain to *both*
retrospective and prospective benefits.

Many conference participants (excluding those in the knowledge workshop
which did not discuss this topic) considered prospective benefits
associated with the base case as those associated with the most-likely
scenario (although participants disagreed as to whether \"most-likely\"
is an apt term for this scenario). This base case is sometimes referred
to as the \"business-as-usual\" or \"reference case\" scenario. Many
participants in the workshop on option value initially questioned the
reasons for singling out a prospective base case, noting that it is
simply one of any number of scenarios that could be realized in the
future (their discussions are summarized under the Option Value
section). The response of other participants was that DOE, including the
Energy Information Administration (EIA), have traditionally devoted
considerable attention to a base case in their forecasts and analyses.
It was also noted that they have never attached any probabilities to
this case.

Three important considerations in estimating prospective benefits were
discussed:

-   Determining the projected baseline conditions, on which new
    technologies are to be introduced,

-   Projecting the market penetration and performance of a new
    technology, and

-   Estimating the effect of the government\'s R&D program on technology
    development and use.

These issues took up most of the discussion in the workshop on
cross-cutting issues in estimating prospective benefits, and are
discussed below.[^9]

[Projected Baseline]{.underline}

The need to establish a projected baseline when estimating the
prospective benefits of R&D is analogous to the need to define a
counterfactual condition for retrospective estimates of the benefits of
past programs. The workshop on prospective benefits had a technology
focus and there was extensive discussion about the proper method for
estimating the baseline. Many workshop participants described the
baseline as the best "guesstimate" of what is likely to happen, without
the technology under consideration (but *with* technological advance in
other sectors).

Many workshop participants agreed that the projected baseline would:

-   be an evolutionary one, with some progress in energy efficiency,

-   include the most likely changes in the market and technology,

-   include expectations about fuel prices,

-   be based on *existing* laws and regulations.

There was considerable discussion about the merits and limitations of
various, possibly complementary, methods for estimating the projected
baseline such as:

-   large-scale integrated models of energy markets,

-   specialized models developed for the technology and sector under
    consideration,

-   extrapolation of current trends, or

-   interviews, polls, and case study of the specific R&D or technology
    under review.

Many workshop participants deemed EIA\'s large-scale integrated National
Energy Modeling System (NEMS), and the assumptions underlying the Annual
Energy Outlook\'s (AEO\'s) reference base, to be the best starting point
for the baseline. They suggested that a good approach would one starting
with NEMS, and then using other models such as MARKAL, as needed, if
NEMS were not suitable for the specific technologies under
consideration.

It was discussed that different approaches might need to be used for
different programs -- there was not a \"one size fits all\" solution.
But an important point made was that if different models are used for
different R&D programs, then the models should be calibrated to the
projected baseline conditions (e.g., as generated by NEMS) to assure
consistency and comparability in results. Their respective assumptions
should likewise be made as consistent as possible with those in the
projected baseline (e.g., NEMS AEO Reference Case).

Several conference participants pointed out that the NEMS Reference Case
includes, in a very general way, the effects of R&D programs underway at
DOE. Thus, one should be careful to net out the assumed technical change
in that technology sector when trying to assess the benefits of a
specific technology against the EIA baseline, if that technology sector
was already in the baseline.

Although many workshop participants favored the use of models for
projecting a baseline and for estimating the effects of a government R&D
program, several workshop participants questioned whether models are
useful for these purposes. The concerns voiced against the use of models
included the suggestion that the effects of an individual, relatively
small R&D program are not reflected in large integrated or macro-models.
Another concern was about how the models deal with ongoing R&D project
interactions. Concerns were also expressed that the baseline definition
was dealing with only individual projects and not the portfolio; and
that the models were less useful for the longer term than for projects
focused on the near-term -- there was simply too much uncertainty in the
longer term.

Many conference participants regarded that, whichever method is used to
project the baseline, the market under consideration for this baseline
should be the one relevant to the technology under consideration.
Baselines could be developed with interviews of independent,
non-involved experts in the field of the R&D project, competitors, NGOs
(non-government organizations), and others. This approach could be used
as an alternative to the use of mathematical modeling techniques or as a
complement to them. In peer-review assessments of proposed R&D projects,
there is an implicit baseline considered when rendering an expert
judgement.

Many workshop participants also expressed the need to make all the
assumptions and defined scenarios transparent and clear, whichever
method is used. Data limitations were raised as a limitation for all of
these approaches, as was the great cost of too much or too refined an
analysis.

[Market Penetration and Technical Performance]{.underline}

EIA\'s NEMS model, with the assumptions underlying the Annual Energy
Outlook\'s (AEO\'s) Reference Case, was favored by many workshop
participants for projecting market penetrations of new technologies. The
suggested advantages of this approach are that NEMS is an EIA model and
is thus \"neutral\" in terms of it not being associated with any one
energy resource office in DOE. Also, the model accounts for interaction
among competing new technologies.

As in the case of estimating a projected baseline, MARKAL or other
models might be better suited for estimating market penetration of
individual technologies because they might have greater detail about
these technologies or be easier to modify to incorporate them. If such
models are used to predict market penetrations, then several conference
participants thought that they should be calibrated to the EIA NEMS AEO
Reference Case.

As in the discussion on methods to use to estimate the projected
baseline, workshop participants generally held one of two alternative
positions on the more appropriate method to use: either integrated
models, or person-to-person interactions that obtain information from
those directly engaged in that line of business. The concerns raised
about models, which were previously summarized, are pertinent to this
discussion as well.

Two other approaches are commonly used to estimate market penetration:
experts\' judgements of the market penetration of the prospective
technology (e.g., defining logistic curves for the diffusion and market
adoption of the technology); and extrapolation of current trends, if the
technology has already begun to penetrate the market. Neither of these
approaches was extensively discussed during the workshop. But a general,
widespread sentiment expressed during the conference about the need for
peer review, especially of subjectively determined projections, would
pertain to these approaches.

[Impact of the Government]{.underline}

Conference participants discussed how DOE funding could impact the path
of an energy technology or practice. The impacts could be categorized
as:

-   the characteristics, scale and function of the technology (e.g. its
    emissions profile),

-   acceleration of its commercialization and date of introduction in
    the market,

-   how quickly the technology gains market share after it is initially
    introduced, and

-   the \"final\" level of market penetration, i.e., the maximum
    deployment or use of the technology.

The NRC committee focused on the acceleration impact of
government-sponsored technology R&D and defined the \"5-year rule.\"
This approximation assumes that the private sector would have initiated
the same R&D at the same level of investment as the government, only 5
years later \-- so that the effect of the government program was
\"only\" to accelerate commercialization by 5 years. Several
participants in the workshop on prospective benefits noted that the
effect could be different from the 5-year acceleration assumed by the
NRC, mentioning the broader array of possible impacts listed above.

The view of many workshop participants was that the impact of the
government varies on a case-by-case basis, and that an across-the-board
"5-year rule" provides some benefits (such as easy comparisons among R&D
programs), but is probably incorrect. Many workshop participants
stressed the importance of other factors influencing market penetration.
They listed several factors to consider when evaluating both the impact
of an government R&D program as well as the ultimate market penetration
of the technology:

-   The economics of the technology/product -- both in terms of its
    scale (e.g., a \$50 technology or product versus a \$500 million
    technology or product) and its comparison to the next-best
    alternative.

-   The percentage of government funding in the total R&D. Workshop
    participants cautioned that focusing on this ratio might result in
    an incentive to increase the amount of government funding to raise
    the benefits attributable to the government. This situation could be
    counter to other objectives, such as increasing private investment.

-   The product development cycle \-- how long and what is involved in
    developing the product.

-   The nature of the industry -- fragmentation; whether highly
    investing in R&D or little investment; nature of the competition
    (both foreign and domestic); etc.

-   The nature of the deployment and installed base of the product which
    the new technology is seeking to replace.

-   Customers\' non-economic attributes that affect the demand for the
    product.

-   Whether the market that the product was entering is a competitive or
    highly regulated one.

-   The likelihood and reasons for adoption by "early adopters" of
    technology.

-   The nature of the market failure which justified a government role.

-   The compatibility of the product with other products -- does it
    enhance any other products?

-   The riskiness of the research.

-   The timing of when the benefits are projected to be realized.

-   The precise actions of the government. Where is it sponsoring
    research \-- in private industries, universities, or national
    laboratories?

-   Broad market conditions, policies, and regulations.

-   The interaction of R&D with other government policies -- tax credits
    for deployment, market transformation programs, etc.

-   The stage of product development. If the technology is ten or more
    years from development and deployment, then some of these
    considerations were viewed as being much less important -- e.g. the
    reasons for adoption by early adopters.

Some workshop participants noted that private industry is often
extensively involved in technology innovation and development, and
provides funding in most cases -- from the R&D itself to the
commercialization of the product. The workshop thus considered how to
allocate some portion of the overall benefits of a technology to the
government program itself, taking into account the previous list of
factors. Since there are 100\'s if not 1000\'s of projects in the
relevant DOE offices, most workshop participants thought that this
analysis should be done on a program (and not on a project) basis.

Two generic cases were considered: 1) where the government involvement
was necessary but not sufficient to develop the technology, and 2) where
the government had an important effect but perhaps was not necessary for
the development of the technology. In the first case, many workshop
participants thought the government should get more credit for the
technology development. Workshop participants stressed the need to
clearly and precisely identify what the government and the private
sector each was doing at different project stages, in order to allocate
benefits between them.

The NRC committee, as part of its study, had tried to categorize the
government's role in the most important energy technology developments
of the past 20 years. For a retrospective assessment, the NRC committee
categorized the government's impact as either dominant, influential,
minimal or absent.

Looking prospectively, a similar *qualitative* assessment that considers
the factors listed previously could be made, and the prospective
significance of the government program could be either:

-   Dominant,

-   Very important,

-   Influential, or

-   Minimal.

Several workshop participants cautioned that independent panels should
be used to make this qualitative assessment, and that not all the
projects should get the same rating. Making distinctions among programs
is important, they thought, despite the reluctance of program managers
to make them.

Several other approaches were seen as methods to allocate benefits
*quantitatively*:

1)  Based on percentage of government funding. The cautions noted
    previously about this approach were expressed again.

2)  Using "traditional" market penetration models and comparing the
    effects with and without a government program.

3)  The 5-year rule, which assumes that the impact of the government is
    to accelerate commercialization of technologies by five years.

4)  A modification of the 5-year rule based on the time for deployment
    of the technology. If the time were 0 to 10 years, then the
    acceleration or government impact would be set between 4 to 5 years.
    If the time for deployment were 10 years or more, then the
    acceleration effect would be greater, say 7 to 10 years. This
    difference recognized that the longer the time horizon for
    deployment, the less the private sector would be interested in
    investing in R&D and developing the project.

5)  A generalization of the 5-year and time-for-deployment rules \-- to
    estimate the acceleration of commercialization on a case by case
    basis. The nature of R&D, and of the prospective technologies they
    are seeking to develop, vary considerably \-- from compact
    fluorescent lighting, to biomass conversion, to fusion energy, to
    fourth-generation nuclear power technologies, to carbon
    sequestration \-- that the uncertainties and incentives for private
    sector investment in research likewise vary considerably. Thus, a
    variable-year rule for the acceleration of commercialization could
    be used, with the number of years estimated on a program-specific
    basis.

6)  Methods that were useful in previous evaluations of projects.
    Whatever methods were successful for \"backcasting\" could be used
    to allocate benefits prospectively.

7)  Government payback \-- return on investment to the government. This
    approach would estimate returns achieved by the project and the
    government would be involved in ways different from the usual grant
    or contract approach. The involvement could be in the form of loans
    or even equity investments where the government would be receiving a
    return on its investment, in addition to the public benefits that
    are presumably achieved by the technology. It was noted that NYSERDA
    (New York State Energy Research and Development Authority) is making
    such investments.

8)  Random sampling of projects to obtain an idea of the importance of
    the government to the development and commercialization of the
    technology. This approach would involve field interviews and expert
    advice as well. The process could focus on only "big" projects if
    deemed appropriate.

9)  If a qualitative rating is used, as previously described, then
    different percentages could be assigned to each qualitative rating
    to allocate numerical benefits to the government program. The
    percentages assigned should be the same across all programs for the
    same qualitative rating.

10) Peer review -- forming independent panels with objective criteria to
    judge importance. The four qualitative ratings discussed above could
    be used in this process.

For science programs, attribution of impacts to government programs is
similarly difficult because of the long and diffuse path from
government-sponsored activities, and the application of knowledge and
knowledge capacity generated by those activities to the end outcomes.
Expert judgment and trends in funding by sector are two indicators of
government\'s contribution. Some participants in the workshop on
knowledge benefits noted that the industry trend is to depend more on
the federal government for basic research, as well as on outsourcing and
collaboration. When the contributions and products of research are not
separable, however, several participants in the workshop on option value
considered it impossible to calculate precisely the impact of the
government program. One participant in that workshop suggested a
production function approach to disentangle the approximate public and
private sector contribution to the marginal increment of technology
success.

***Option Value***

**Definition of Option**

Many of the participants in the workshop on option value agreed with the
tenor of the following definition, though there was not agreement on
precise wording:

> A real option (as opposed to financial options on a company\'s stock,
> for example) is an asset (such as a technology) that allows a decision
> maker to learn and to make choices to utilize the asset at future
> points in time when there is uncertainty about future conditions.

Some workshop participants favored the definition developed in the NRC
study \-- that options are technologies that are fully developed but for
which existing economic or policy conditions are not favorable for
commercialization. The extrapolation of this latter definition to
prospective analysis would mean that the technology is developed
intentionally *not* to enter the market if it turns out to be as
currently projected. According to this definition, the technology would
serve as \"insurance\" against unexpected developments in the future.
Many of the real-options specialists in the workshop regarded this
definition to be restrictive and that providing insurance is only one of
the possible benefits of real options.

At least one conference participant pointed out that the motivation for
many of DOE\'s programs is to provide technology options to address
uncertain conditions in the future, such as oil-price shocks. Thus, from
this perspective, providing option benefits is an integral part of
DOE\'s mission.

Many of the real-options practitioners in the workshop initially thought
that there should be no \"prospective base case benefits\" column in the
matrix because the projected base case is simply one of many possible
outcomes that might be realized in the future. These conference
participants regarded the base case as having some probability of
occurring, just as other possible future conditions have various
probabilities of occurring.

However, it was pointed out that such a view is not consistent with
current, well-ingrained program planning practices at DOE, which use a
base case. Also, most people\'s limited familiarity with the real-option
concept might lead them to discount any option value that is calculated.
Thus, several workshop participants suggested a *gradual*, rather than
comprehensive, introduction of the real-option concept into the methods
with DOE uses to estimate its R&D benefits, as discussed below.

The measures or indicators of option value are the same as those
discussed previously for prospective benefits (e.g., reduced energy
costs measured in dollars). However, the methods for estimating option
value differ from the discounted cash flow approach usually used for
estimating prospective benefits of technology R&D programs.

**Methods for Estimating Option Value**

Several participants in the workshop on option value worked to outline a
generic 4-step methodology for option valuation of technology R&D
programs and identified approaches for conducting each step.

1.  [Describe the uncertainty]{.underline}

    a.  Use integrated energy-model scenarios and assign a probability
        to each scenario.

    b.  Identify the source(s) underlying the uncertainty (technical
        performance, energy prices, environmental regulations, policies,
        market conditions).

```{=html}
<!-- -->
```
2.  [Calculate outcomes]{.underline}

    a.  Use the results of the scenarios developed in step 1.

    b.  Develop a forecast using the underlying uncertainty
        characterizations.

    c.  Use a simplified modeling approach.

3.  [Determine the decision actions]{.underline}

    a.  Consider the following decision possibilities: (1) continue, (2)
        abandon, (3) expand, or (4) hold.

4.  [Do the expected-value calculation]{.underline}

    a.  Carry out a standard net present value (NPV) calculation.

    b.  Decide between a risk-neutral and a risk-adjusted approach.
        \[Workshop participants had an animated discussion about this
        technical issue.\]

Several workshop participants pointed out that the type of data required
could be a significant deterrent to its full implementation. Given the
desire among many workshop participants for a *gradual* introduction of
this paradigm into DOE\'s planning process, at least one of the
participants suggested that the following approach be considered in the
near-term:

-   Consider a scenario analysis and base it on common scenarios used
    DOE-wide.

```{=html}
<!-- -->
```
-   Create scenarios that are:

```{=html}
<!-- -->
```
-   applicable to many and diverse DOE programs

-   widely accepted as unbiased

-   capture a wide range of future conditions

-   as few in number as possible.

```{=html}
<!-- -->
```
-   Use the NEMS AEO reference case (i.e., the prospective base case)
    and others that vary across the economic, environmental, and
    security dimensions.

```{=html}
<!-- -->
```
-   To illustrate option value, calculate benefits under one or two
    scenarios in addition to the base case scenario.

-   Note that the values based on a few alternative scenarios illustrate
    \"optionality;\" they do not provide an estimate of option value. A
    future improvement in this approach would be to assign probabilities
    to the scenarios. This would allow a crude expected option value
    calculation to be performed.

-   DOE should support investigation of the probabilities (volatility)
    and correlation among key energy market drivers such as GDP, fuel
    prices, and environmental regulations. Probabilities of dramatic
    individual events, such as a nuclear moratorium or fusion
    breakthrough, should also be examined. If the range of probability
    estimates were to become generally accepted in the future, then
    these could form the basis for an options analysis that would be
    preferable to the simple optionality approach suggested here.

***Economic Benefits***

**Measures of Economic Benefits**

Economic benefits are those primarily associated with a reduction in the
costs of providing energy services and in the quantity of energy
\"consumed\" to provide energy services. Economic benefits also include
spillover and macroeconomic impacts on the economy due to investments in
R&D. Spillover occurs from the use of the results of the R&D in
applications other than in the technology(s) primarily being developed.
The macroeconomic impact result from reduced cost to, or greater
productivity in, other sectors of the economy, and the associated
multiplier effects.

Participants in the workshop that discussed economic benefits offered
the following measures of economic benefits:

-   \$ value of net reduction in energy consumption, net of any
    additional costs

-   reduced \$ cost in providing a unit of energy service

-   \$ value of spillover effects, or

-   macroeconomic impacts on the overall economy (including reduced
    energy consumption and reduced costs of energy), or

-   \$ value of increase in consumer surplus, which is the amount
    consumers would be willing to pay minus what they actually pay for
    energy.

The first three economic measures are additive. Both of the last two
measures, listed separately above, overlap with the first three
measures. The various measures represent different approaches to
estimating economic benefits.

**Methods for Estimating Economic Benefits**

Some DOE offices estimate economic benefits by:

-   using engineering analysis to estimate the expected reduction in
    energy consumption for each new technology (i.e., its technical
    performance),

-   projecting the annual market penetration and use of the technology,

-   multiplying the per-unit energy consumption times the use of the
    technology, and then

-   comparing the estimated energy consumption to the estimates of
    energy consumption from the technology(s), which the technology
    improved or replaced.

The workshop that had a session on economic benefits discussed in broad
terms other types of methods that could be used to estimate economic
benefits:

-   Use of a cost index to estimate the change in cost to energy
    consumers

-   Econometric analysis to estimate spillover and related macroeconomic
    impacts

-   Case studies

-   Technical and economic models

-   Detailed tracking (and extrapolation) of R&D outcomes

-   Sampling techniques (statistical monitoring).

The cost index referred to above is essentially a ratio of expected
consumer costs, with and without the new technology. It relies on
assumed, or exogenously determined, adoption rates for the new
technology. A model is used to estimate electricity generation costs and
the associated external costs (e.g., from public health effects due to
exposure to emitted pollutants). Monte Carlo simulation is used to
incorporate uncertainty into the analysis.

The econometric approach has been applied to estimate the benefits of
industry-sponsored R&D in the manufacturing sector. The study, which was
reported in the workshop, found that there were large spillover benefits
of government R&D to other sectors and activities \-- and that these
were important and measurable. The study also found that there was some
substitution of public R&D for private R&D. These econometric methods
have thus far been applied to a whole sector (i.e., the manufacturing
sector) rather than to individual R&D programs.

***Environmental Benefits***

**Measures of Environmental Benefits**

Environmental benefits are those associated with a reduction in damages
to public health, or to the environment, from exposure to pollutants.
The NRC study used the following measures:

-   physical units (i.e., tons) of reduced emissions, or

-   \$ value of reduced pollutant emissions, for which the economics
    literature has estimated average \$/ton values (for criteria
    pollutants and carbon dioxide).

Other measures suggested at the conference were:

-   environmental value of R&D, based on the value of information (which
    depends on the regulatory and policy situation)

-   changes in an index of sustainable welfare.

Intensity measures were also offered (though it was recognized that
these measures do not reflect the overall magnitude of the impacts or
benefits):

-   pollutant emissions per unit of output (e.g., reduction in tons/\$
    output)

-   energy use per unit of output (e.g., reduction in Btu\'s per \$
    output).

**Methods for Estimating Environmental Benefits**

Some DOE offices estimate environmental benefits by:

-   using engineering analysis to estimate the expected reduction in
    emissions of criteria pollutants and carbon dioxide for each new
    technology (e.g., in tons/MWh, tons per vehicle-miles traveled, tons
    per Btu, etc.),

-   projecting the annual market penetration and use of the technology,

-   multiplying the emissions rates of the technology times the use of
    the technology, and then

-   comparing the estimated emissions to the estimates of emissions from
    the technology(s), which the technology replaced.

Several workshop participants thought that, given the complexity of
environmental impacts, indices are useful and appealing because they are
relatively simple. One of the workshop panelists suggested an Index of
Sustainable Economic Welfare (ISEW), defined as the sum of non-defense
GNP plus human environmental capital formation, minus environmental
degradation (the latter could be measured in terms of reduction in
ecosystem services). \[The relationships between prospective new
technologies and the impacts of their deployment on changes in ISEW
would need to be estimated.\]

Several workshop participants also recognized a wide range of economic
tools for valuing environmental benefits:

-   Hedonic methods that identify relationships between environmental
    quality and attributes such as property value,

-   Contingent valuation which uses economic experiments to estimate
    individuals\' willingness to pay to avoid specific types of damages
    to the environment or to human health

-   Direct measures of market value such as loss in productivity or
    yields of agricultural crops, medical and health costs, and
    reductions in expenditures on recreation.

***Security Benefits***

**Measures of Security Benefits**

Participants in one of the four workshops discussed security benefits,
both as they have been viewed traditionally (i.e., reduced oil
consumption in a market in which a cartel has monopoly power), and as
those associated with vulnerabilities of energy infrastructure and
systems to terrorist and other disruptive events.

One presentation offered a definition for \"energy security\" to be
\"energy being available when needed at a predictable price.\"
\[Presumably, this definition applies to near-term availability and
prices, and not to the ability to predict energy prices many years into
the future.\]

Workshop participants listed several major types and measures of
security benefits, and grouped them under three categories (suggested
units of measurement are in parentheses):

a\) Prevention or reduced probability of disruption, security breach, or
system failure, and their associated costs \--

-   Reduced net oil import costs (measured in \$)

-   Reduced expected damages (\$) and injuries (pubic health and safety)

-   Lower environmental costs from possible terrorist activities that
    might damage the natural environment (reduced environmental impact)

-   Lower liability costs (\$)

-   Lower insurance costs (\$)

-   More predictable energy prices, which reduces the damages (accuracy
    of predicted prices)

-   Reduced costs for military presence and activities in the Middle
    East (\$)

-   Reduced costs of protecting domestic energy infrastructure\] (\$).

b)  Management of response, given that an event has occurred \--

-   Lower macroeconomic disruption costs when there are price spikes due
    to major unanticipated disruptions in energy supply (\$)

-   Lower cost of emergency management (\$).

c)  Recovery and repair \--

-   Increased speed of response and recovery (engineering estimates of
    the time it would take to restore energy services)

-   Reduced cost of recovery and repair, after a disaster occurs (\$).

**Methods for Estimating Security Benefits**

Several offices in DOE currently estimate the number of barrels of
reduced oil consumption, as a result of substitution by other energy
sources such as renewable energy (e.g, ethanol from biomass as a
substitute for gasoline from crude oil). The estimates are based on the
projected market penetration of these alternative sources, and on their
technical efficiencies relative to conventional technologies.

One of the presentations in this workshop summarized a synthesis of the
literature on oil security and its benefits. In this literature,
estimates of the benefits of oil security are expressed in units of
\$/barrel. This parameter could be applied to estimates of the barrels
of reduced oil consumption to calculate energy security benefits in
economic terms.

A few workshop participants noted that a complex simulation model would
be very useful to assess the impacts of disruptions, and thus the
benefits of R&D that mitigates them, in a highly integrated system of
electric power, telecommunication, transportation, emergency service,
and other government service infrastructures.

However, without such models, some workshop participants suggested that
some of the following indicators of energy security benefits could be
used:

[Supply and Demand Conditions]{.underline}

-   Energy import levels (both in quantity and dollars)

-   Diversity of delivery channels

-   Diversity of supply sources

-   Degree of monopoly/cartel power (measured by OPEC market share)

-   Funds sent to potentially unfriendly nations

-   Insurance rates/costs.

[System Flexibility and Sensitivity Measures]{.underline}

-   Price elasticity of world demand for various fuels

-   Price elasticity of U.S. demand for various fuels

-   Substitutability: dual-fuel electric generating capacity

-   Fuel stocks (oil and gas)

-   Macroeconomic sensitivity to energy shocks (perhaps measurable
    econometrically).

[Reliability/Volatility Measures]{.underline}

-   Electric power interruption frequency and duration (outages, supply
    disruptions)

-   Other measures of power grid reliability (power quality, brownouts,
    etc.)

-   Capacity to meet peak electricity demands

-   Investor confidence, PE (price-earnings) ratios

-   Price volatility measures for various fuels

-   Transmission/transportation reliability indicators

-   Survey responses of infrastructure owners and users.

***Knowledge Benefits***

**Measures of Knowledge Benefits**

One of the presentations to the workshop on knowledge benefits suggested
that science is a cumulative cascading process that involves the
generation and transmission of knowledge. The perspective of many of the
workshop participants was that government funds science for two reasons:
knowledge creation and knowledge as a foundation for application (i.e.,
developing competencies such as human capital). Thus, although the
participants in the workshop noted that knowledge has many meanings,
many participants regarded knowledge as both an ultimate outcome as well
as an \"enabler\" of other benefits \-- an intermediate outcome or even
an output in the context of applied R&D programs.

The group also viewed knowledge benefits within the broader criteria
that COSEPUP (the Committee on Science, Engineering, and Public Policy,
National Academy of Sciences, National Academy of Engineering, and
Institute of Medicine) recommended for GPRA assessment, and in the
context of OMB\'s recommended criteria for investment in basic research:

-   Quality (of the research)

-   Relevance

-   Performance (in terms of both management and results).

Given the conference\'s focus on benefits as being outcome measure of
performance, we concentrate on synthesizing the workshop group\'s
discussions about knowledge within the performance-outcomes part of the
framework (i.e., the blue and blue-striped portion in Figure 2).

The group discussed adapting a generic model for the knowledge-related
outputs and outcomes of research, first developed for the Basic Energy
Sciences \"research value mapping\" project in 1994, and recalled by one
of the workshop speakers. Several in the group developed the following
depiction of knowledge outputs, outcomes and benefits, given in Figure
4.

**Figure 4. Development of Knowledge Outputs and Outcomes**

Participants in workshops, other than the one focusing on knowledge
benefits, did not have an opportunity to discuss knowledge benefits
extensively. However, some participants in these other workshops noted,
as did some participants in the knowledge-benefits workshop, that
knowledge and ideas, research tools, human capital, communities of
practice and spin-offs could result from applied R&D, just as they could
from basic research (though they did not use these exact terms).

Some participants in one of these other workshops suggested that a
measure of the effectiveness of R&D is reduction in uncertainty, which
can also be considered acquisition of knowledge. This idea reflects the
suggestion of the knowledge-workshop participants that knowledge
benefits underpin the benefits of the applied research and technology
programs; that is, they are a third dimension. Some participants in the
knowledge-benefits workshop indicated that they would like OMB to place
a higher value on knowledge-based capacity considerations when
evaluating the prospective benefits of R&D programs. They recognized,
however, the need to address the challenge of \"setting the bar higher\"
in defining such benefits.

**Methods for Estimating Knowledge Benefits**

The participants in the workshop on knowledge benefits suggested the
following indicators for benefits of knowledge, listed in Table 2.

**Table 2. Examples of Indicators of Knowledge Benefits**

  ----------------------------------- -----------------------------------
  **Type of Knowledge Benefit**       **Example of Indicator**

  Scientific merit                    Quality and relevance(prospects and
                                      accomplishments) as judged by peers

  New ideas                           Research questions answered.
                                      Program milestones attained

  Research tools                      Operability of scientific
                                      facilities. Expert reviews of the
                                      construction of research
                                      facilities.

  Human capital                       Number of graduate students trained

  Communities of practice             Connectivity index or measure

  Transitions and spin-offs           Removed barriers, intellectual
                                      property
  ----------------------------------- -----------------------------------

One speaker in the workshop on knowledge benefits presented the many
ways in which bibliometric techniques can be used in the assessment of
research. Bibliometrics could be used prospectively for human capital
issues and to trace networks. It could also be used as an indicator of
vitality, and of where one might need to make investments in R&D. By
investigating papers that cite other papers, the organization can assess
the knowledge incoming and outgoing from an organization. By considering
the percentages of top-cited papers, one has an indicator of their
quality and a value distribution across a portfolio. In addition, by
considering patents that cite papers and patent portfolios, one can do a
network analysis of the \"innovation process.\" Tracking individual
researchers\' names through the patent system would also be valuable,
for example to show the need for expertise.

Similarly, other kinds of prospective analysis can lead to theories of
how inputs lead to outputs, that is, theories about what goes on inside
the \"black box\" process of managing and doing R&D. Many workshop
participants felt that it is important to describe this process. That
information would help understand the use of management as a decision
tool. Black-box answers \"show what managers do that makes programs
work.\" These studies could also use a \"logic model\" to attribute or
link program activities to knowledge value communities, or communities
of practice, which were viewed by many workshop participants as being
the sort of outcomes that are important.

Another workshop speaker spoke from 30 years experience managing
research at Dupont and participating in the Industrial Research
Institute's Research on Research Committee. He also spoke of knowledge
as a third dimension in all R&D investment decisions and of the
importance of managing the black box. He pointed out that within the
non-linear innovation system that includes inputs, processes and
outcomes, different stakeholders are interested in different metrics. He
listed different types of benefits estimating techniques: net present
value, rules of thumb, database of assessments, studies by independent
analysts, value of intellectual property and orphan patents, and
financial analysts' estimates of value.

This speaker also addressed the question of what to do about assessing
the benefits of basic research and pointed to ideas on radical
innovation, the knowledge-drivers-of-the-future diagram, and strategy
tables. According to some workshop participants, a possible benchmark
for DOE to use when thinking about risk is a study on the success rate
of new products that showed that it took 3,000 new ideas to get 300
submitted ideas, and eventually end up with one new commercial success.
Thus, these conference participants stressed that an organization needs
a steady pipeline of R&D.

The nature of research is experimentation. Therefore, some conference
participants noted that finding out what does or doesn\'t work has
value. One participant suggested that knowledge benefit is gained from
what is learned from a \"failed\" program that reduces the uncertainty
about what would be a \"successful\" tack of R&D; and suggested that the
order of magnitude of this benefit could be quantified. This notion of
reducing uncertainty and risk by investment is central to the real
option methods discussed in that workshop. Thus, it might be worth
investigating whether some of the quantitative tools discussed in that
workshop could be relevant in this context as well.

***Data Sources***

DOE participants at the conference emphasized that data availability
must be a major factor in considering the alternative measures to use
and the methods for estimating them. Conference participants identified
various general sources of data:

-   Publicly available data such as those from EIA and the Department of
    Commerce,

-   Data from private sources which are reliable, relatively
    inexpensive, and available on a continuing long-term basis,

-   Polls and surveys, both formal and informal, to assess consumer
    confidence in regard to security issues, baseline conditions (e.g,
    what would happen with and without the technology), and the impacts
    of the government program (e.g., in a prospective sense, is impact
    of the program likely to be dominant, very important, influential,
    or minimal),

-   Engineering and expert judgement (e.g., key technical performance
    parameters, peer review of basic research programs), and

-   Outputs or predictions of other models (e.g, from EIA or Department
    of Commerce).

When it is necessary to use experts\' subjective judgements to peer
review the quality of basic research or to estimate key parameters, such
as the market penetration of a technology, many conference participants
stressed the importance of impartiality. It was noted, however, that is
difficult to have complete impartiality and objectivity in such reviews
because the people who are most knowledgeable about a technology are
involved directly in its development or are working under contract to
DOE in a related project(s).

***Use of Estimated Benefits in Program Planning and Evaluation***

Discussion in one of the workshops raised the question of \"what it is
that OMB wants to know.\" Retrospective assessment answers the question
\"was it worth it\" \-- as in the NRC study. Some participants in that
workshop suggested that OMB would like the R&D programs to use criteria,
such as estimated benefits, as a planning tool, not only as a
retrospective assessment tool. DOE programs could use these criteria to
make decisions before their proposed budget goes to OMB, using the
criteria to differentiate among different kinds of programs and their
prospective benefits.

Several conference participants highlighted the importance of portfolio
management. The workshop on option value discussed the need to cut
across DOE stovepipes to assess the benefits of R&D portfolios.
Decisions to terminate a program were regarded by many as being as
important to consider as those to initiate or to continue one. Many
conference participants also thought it important to view individual
programs as part of a broader energy R&D portfolio, and to regard
estimates of benefits as but one criterion to consider in R&D investment
decisions. Other criteria mentioned in the workshop were the R&D\'s time
frame, project cost, and technical risk. Workshop participants liked the
use of visual representations of portfolio analyses that depict the
interplay and trade-off among decision criteria.

One speaker showed a portfolio tool, the familiarity matrix, developed
at MIT\'s Sloan School. The matrix considers the interdependence between
the newness of a technology with the newness of the market to the firm.
An organization can use it to help manage risk. Pursuing a new
technology in a new market would be in the "suicide square\" of the MIT
familiarity matrix, for example. Several conferees suggested that it is
important to remember that order of magnitude estimates are sufficient
at first.

Some conference participants emphasized the importance of setting goals
for the programs, quantitative to the extent possible, in terms of their
various benefits (e.g., reduced carbon). Some R&D programs would
presumably lead to certain types of benefits, but not to others; and
some programs would in retrospect be more successful than others,
depending on what happens in terms of future economic, geopolitical,
regulatory and policy developments. By their comments, some conference
participants appeared to view the portfolio approach as one of
*efficient diversification* of R&D options to account for:

-   national needs and priorities for foundational, basic research

-   synergistic as well as the counter-productive aspects of jointly
    developing various technologies, and synergies among basic and
    applied research efforts

-   competing and supportive aspects of various technologies in the
    market, and

-   uncertainty about future conditions, as well as about the
    effectiveness of the R&D and the technical performance of the
    technology and its competitors.

Several participants pointed out the importance of evaluating past
performance \-- meeting interim milestones that were set \-- as part of
the planning process. Some conference participants noted that the
\"alternative format,\" as provided for under GPRA, could be used
fruitfully to report on intermediate outcomes, which R&D programs could
more directly affect than ultimate, end outcomes such as the various
benefits measures discussed in this conference. The waiver an agency
receives to use the alternative format allows the use of qualitative
statements of performance objectives. The National Science Foundation
uses this format, for example.

According to several conference participants, DOE should assure that the
tools used for program decision making recognize option value. Some
conference participants noted that options-thinking leads to a different
method of planning, so that in addition to being a tool for estimating
benefits, options consideration is also a more general management tool.

Scoring systems that use various scales can include optionality
considerations and can be applied to many programs. The scales could
cover the following criteria:

-   Potential impact of the technology

-   Uncertainty and potential for learning

-   Flexibility in developing and implementing the technology.

For example, the flexibility scale might be built around the following
concepts:

-   High flexibility

```{=html}
<!-- -->
```
-   Technology creates value in many different scenarios

-   Much of the research is applicable to many other technologies

-   Research investment is *not* lumpy, that is investment can be made
    in stages. For example a project that requires \$10 million/year
    over 5 years is more flexible than a project that requires \$50
    million of immediate investment.

```{=html}
<!-- -->
```
-   Low flexibility

```{=html}
<!-- -->
```
-   Technology creates value in only a single scenario

-   Research is specialized and contributes to a single technology

-   One large lump sum investment is required.

Application of this type of scoring tool would require a peer review to
assure a relevant scoring process.

Many participants agreed on the need to have a uniform framework that
would be used across all relevant DOE offices, and a consistent set of
assumptions and projected baseline. Many participants also appeared to
agree on the need to continually improve the methods for estimating the
various types of benefits and to do pilot testing of these methods on
selected programs before implementing them office-wide.

Finally, some conference participants argued for more common sense in
R&D program planning and evaluation. They regarded the use of
complicated frameworks and methodologies as overreaching their current
capabilities to accurately predict the benefits of R&D.

In any event, it was recognized that \"to talk about the bull is not the
same thing as being in the bullring.\" This is true both of talking
about a benefits framework as opposed to actually building and
implementing one, and true of talking about the R&D planning process as
opposed to actually doing it.

[^1]: Organized by Oak Ridge National Laboratory and sponsored by the
    Office of Energy Efficiency and Renewable Energy, Office of Fossil
    Energy, Office Nuclear Energy, Science and Technology, and Office of
    Science of the U.S. Department of Energy. Information about the
    conference is available on the conference web site,
    [www.esd.ornl.gov/benefits_conference](http://www.esd.ornl.gov/benefits_conference);
    in the white paper distributed prior to the conference, \"Ideas on a
    Framework and Methods for Estimating the Benefits of
    Government-Sponsored Energy R&D;\" and in the report summarizing the
    conference proceedings, \"Synthesis of Conference Discussions.\"

[^2]: This document is believed to be a reasonably accurate summary of
    discussions at the conference on \"Estimating the Benefits of
    Government-Sponsored Energy R&D,\" but the accuracy is not
    guaranteed by Oak Ridge National Laboratory (ORNL), UT-Battelle LLC,
    or the U.S. Department of Energy. Furthermore, the opinions
    expressed by those at the conference are their own and therefore
    nothing in the reporting of the conference proceedings should be
    construed as government policy. In particular, the views do not
    necessarily represent those of ORNL, UT-Battelle LLC, which operates
    ORNL for the U.S. Department of Energy, or the U.S. Department of
    Energy.

[^3]: Executive Office of the President, Office of Management and
    Budget, *The President's Management Agenda, Fiscal Year 2002*,
    August 2001. National Energy Policy Development Group, *National
    Energy Policy*, Washington, DC, May 2001.

[^4]: ^?^ National Research Council's Committee on Benefits of DOE R&D
    on Energy Efficiency and Fossil Energy, *Energy Research at DOE: Was
    It Worth It?*, Washington, DC: National Academy Press, July 2001..

[^5]: This idea was one of the \"strawperson\" suggestions made in the
    white paper distributed prior to the conference: Lee, R., Wolf,
    J.L., Zimmerman, M.B., Braitsch, J., Vallario, R., Powell, J., and
    R.C. Ricci, \"Ideas on a Framework and Methods for Estimating the
    Benefits of Government-Sponsored Energy R&D,\" Oak Ridge, Tennessee:
    Oak Ridge National Laboratory.

[^6]: National Research Council's Committee on Benefits of DOE R&D on
    Energy Efficiency and Fossil Energy, *Energy Research at DOE: Was It
    Worth It?*, Washington, DC: National Academy Press, July 2001.

[^7]: Lee, R., Wolf, J.L., Zimmerman, M.B., Braitsch, J., Vallario, R.,
    Powell, J., and R.C. Ricci, \"Ideas on a Framework and Methods for
    Estimating the Benefits of Government-Sponsored Energy R&D,\" Oak
    Ridge, Tennessee: Oak Ridge National Laboratory, February 27, 2002.
    This \"white paper\" is available on the conference web site:
    www.esd.ornl.gov/benefits_conference.

[^8]: ^?^ The workshop rapporteurs\' summaries of each of the four
    workshops are available on the conference web site:
    www.esd.ornl.gov/benefits_conference.

[^9]: The idea of a baseline did not seem relevant to many of those in
    the workshop on knowledge benefits, which had a basic science,
    rather than technology, orientation.