Conference on

## Estimating the Benefits of Government-Sponsored Energy R&D[^1]

**March 4 and 5, 2002**

### Summary of Discussion in Workshop C:

###  KNOWLEDGE VALUE[^2]

# Gretchen Jordan, Sandia National Laboratories[^3]

**May 1, 2002**

# The primary task of Workshop C was to identify methodologies to assess knowledge benefits both prospectively and retrospectively. In a prospective context, the goal of basic research programs is to produce knowledge. In a retrospective context, basic research leads to results that have commercial value and to economic, environmental, energy security or other benefits. Discussion in Workshop C also touched on the relationship between knowledge and technology programs.

# 

# The task of the National Research Council (NRC) study was retrospective analysis of the benefits of technology development programs. From this perspective, knowledge benefits are "economic, environmental, or security net benefits that flow from technology for which R&D has not been completed or that will not be completed." The workshop on knowledge value allowed a group of experts to address knowledge value in more depth, and to consider the perspective of basic research, rather than technology, programs 

# 

# Summary of Commonly Held Views Within the Knowledge Benefits Workshop

Workshop C handled the key questions in depth but did not address all of
questions initially proposed. There appeared to be some agreement within
the group on several ideas and suggestions. However, the open questions
and differences in opinion on the scope and use of a DOE framework for
consistent assessment of knowledge benefits of R&D suggest that Workshop
C\'s ideas will serve as information for further discussion among DOE
management and perhaps members of the 2000 NRC study.

Commonly held views among many of the participants from Workshop C were
that:

-   "Knowledge" is not in the right place in the NRC matrix, according
    to many participants in the workshop. They suggested both that
    DOE-generated "knowledge\" should be a new row, and that it should
    be a consideration in each of the three other areas of benefit. Many
    participants considered it essential for explaining the benefits of
    science programs, and therefore, necessary if DOE desires a single,
    seamless process combining science and technology programs.

-   Estimating the benefits of DOE "knowledge" requires detail on what
    those benefits are. "Knowledge-based Capacity" is a possible name
    for the row that represents all aspects of knowledge benefits. This
    detail is included in a sub-framework that assesses several benefits
    of "knowledge" against four criteria suggested by COSEPUP (the
    Committee on Science, Engineering, and Public Policy, National
    Academy of Sciences, National Academy of Engineering, and Institute
    of Medicine) and the Office of Management and Budget (OMB). See
    Figure 1.

    -   The benefits of knowledge can be summarized as knowledge-based
        capacity which includes: new ideas, new research tools, enhanced
        human capital, stronger communities of practice, and transitions
        and opportunities for transition to applications.

    -   The criteria against which knowledge-based capacity can be
        assessed are quality, relevance or strategic fit, performance,
        and international R&D leadership. Performance could be
        interpreted as outcomes or "benefits", or more broadly.

```{=html}
<!-- -->
```
-   Technology programs would not feature "knowledge-based capacity"
    prospectively in discussion of what to fund, but they do need a way
    to prospectively view and justify funding core competence and
    capacity. More often, they might evaluate knowledge-based capacity
    benefits retrospectively.

-   Science and the applied research programs can use the proposed
    sub-framework prospectively or retrospectively.

-   Links between the rest of the framework (the top nine cells of the
    remaining Conference draft matrix, following the removal of the
    knowledge column) and the proposed bottom knowledge row are crucial.

    -   Knowledge populates all the cells. Thus it could be visualized
        as a third dimension in the matrix or as a thin fourth column in
        the first three rows. In other words, the participants did not
        reject the NRC notion that the technology programs produce
        knowledge along the way. Instead, many participants wanted to
        highlight that knowledge was also a legitimate and important DOE
        goal in its own right, in other programs.

    -   Stakeholders and downstream potential users need to be included
        in the judgment of strategic fit; that is, in defining links
        between science and technology and application.

Many workshop participants agreed that knowledge could also remain as a
thin column in the upper part of the matrix, that is, an enabling but
not primary category for analysis of benefits of technology programs,
using the suggested definition and elements of knowledge-based capacity.
Knowledge would also be a row showing benefits of basic research, or
separate matrices could be used for basic research and technology
development programs.

Other areas of discussion were that:

-   DOE, OMB and Congress need to make explicit how they intend to use
    the information on benefits. Prospective and retrospective
    assessments have different needs and uses, yet the criteria need to
    be reconciled.

-   The group did not have time to address other questions of options,
    baseline, and attribution to government activities in depth. However
    there was agreement among many that: R&D does provide options; the
    role of the government in fundamental science is growing as industry
    moves away from longer term research; and R&D evaluators do not talk
    in terms of a baseline for knowledge, although everyone
    distinguishes between incremental and breakthrough accomplishments
    which implies different baselines.

# Critique of the Proposed Framework for Defining the Benefits of R&D Programs, and its Use for GPRA and R&D Planning and Evaluation

#  

The focus of the initial session was to critique the proposed framework
for defining the benefits of R&D programs, and its use for GPRA and R&D
planning and evaluation. Is the framework clear and consistent on how
\"knowledge\" fits into it? If not, what clarification is needed or how
should it be changed? Many workshop participants felt strongly that the
framework did not adequately capture the benefits of knowledge creation
or knowledge contributions to technology programs and to the three areas
of benefits \-- the economy, the environment, and energy security. After
considerable discussion throughout the two days, many in the group
agreed that knowledge and capabilities should be added as a row, that
is, as an area of benefit. Knowledge also needs to be considered at
least a retrospective benefit in the other three rows.

The following summarizes other concerns raised by many of the workshop
participants about the proposed framework.

The proposed framework and retrospective benefits assessment are related
to the reporting of outcomes, required by the Government Performance and
Results Act (GPRA). The more recent President's Management Agenda and
R&D Investment Criteria are prospective. The framework is also not
sufficiently broad to cover the recommendations for the basic research
response to GPRA made by COSEPUP. These recommended measures of success
are Quality, Relevance and International Leadership. Draft R&D
investment criteria for basic research that have been proposed by OMB
use these criteria as well, but include International Leadership under
Quality and add a criterion of performance or results. One OMB staff
present suggested that performance also included good management, thus
was not restricted to benefits assessment. It is important to more
clearly define and rationalize these values and requirements to avoid
conflicting incentives and to avoid unnecessary expenditure of
resources.

For many workshop participants, some of the specifics about filling in
the matrix are likely to be harmful for basic research. For example, the
proposed five-year time frame for benefits will truncate the R&D process
and push researchers away from more long term, risky research. There was
also concern expressed about the description of failures. The group was
reminded that no one saw John Nash's (A Beautiful Mind) Nobel
Prize-winning work as relevant until 25 years after he published it.

Concern was expressed about using only the "most likely" scenario. A
better approach may be to have several scenarios to use in planning and
assessment.

The matrix as proposed does not provide sufficient information for
planning, particularly at the portfolio level. It seems to emphasize the
trees and not the forest. The matrix is geared toward downstream impact.
It does not recognize the whole R&D process/system of innovation, and
the rich role of government R&D in that process.  The issue is how to
manage the uncertainty of basic and applied research; thus the process,
such as use of peer review, becomes important. The matrix does not
include important portfolio questions of timing, risk, and who is
impacted.  The matrix has no critical needs-information such as the U.S.
competitive position in an area.

One purpose of assessment is to make a case before Congress and to
communicate with the outside world. But it is not just a budget game.
There is also the need for assessment to provide information about the
organization, management, and incentives. Managers need prospective
information to make investment decisions, particularly in the case of
long term, lumpy research. Managers need to understand their programs in
terms of the total logic, including knowledge gaps for applied research.
To be a learning organization we must go beyond the current proposed
framework. OMB also looks more broadly than impacts and wants programs
to present the big picture, especially for knowledge benefits. Perhaps
the "Planning Process" would be better than a "Prospective benefits"
column.

Finally, the relationship between retrospective and prospective measures
and assessment is critical. How does one link them?

##### "Knowledge" Has Many Meanings

In his presentation to the group, Irwin Feller suggested that science is
a cumulative, cascading, process \-- the generation and transmission of
knowledge. "Knowledge" has multiple meanings.

MacHlup, in his book *Knowledge, Its Creation, Distribution and Economic
Significance* lists 33 different questions that relate to different
kinds of knowledge such as knowing how and knowing what. The fact that
knowledge has multiple meanings means that it can be measured in
multiple ways and measures may have different meanings.

**Knowledge As a Row Instead of a Column**

> After much discussion, many of the participants in the workshop held
> the view that knowledge-based capacity should not be treated as only
> one of the range of benefits in the rows of the NRC matrix, but should
> also be treated as an area of benefit in its own right (a row). The
> DOE Office of Science and other fundamental and applied research
> programs in DOE have as their goal to advance knowledge, knowledge
> that is foundational to DOE missions and national needs. Knowledge is
> a DOE goal just as economic competitiveness, environmental quality,
> and national security, including energy security, are. Furthermore the
> technology programs represented in the group wanted those funding
> their programs to recognize the value of building and maintaining
> capabilities and they saw the knowledge benefits as a way to measure
> benefits that are not in the NRC matrix such as student competitions
> and technology diffusion efforts.  Much of the group also saw a
> "knowledge row\" as responding to OMB and GAO requests for providing
> key information related to investment criteria, planning, and
> performance reporting. More detail on this discussion follows.
>
> If knowledge is a column, many participants in the workshop would not
> know what to do with it. Many conference participants wanted to tie
> knowledge back to the expected payoffs that link it to benefits.
> Knowledge is also an area of benefit. Advances in knowledge underpin
> all advances in many aspects of our quality of life. Part of this area
> of benefit is the development of the capabilities of the research
> community.
>
> Members of the group turned to the OMB budget examiners present to
> explain what OMB wants to know. According to them, OMB wants to know
> if R&D programs make the progress they expected, what happened that
> the program didn't expect, and why it is important.

Many voiced the perspective that we should not split out basic and
applied research and technology programs \-- that we would lose more
than we gain. Part of this is perception. Almost all voiced the idea
that how the DOE visualizes R&D benefits is important. The visualization
of R&D benefits, as stakeholders see it, gets transferred into the
system of measurement and management.

At times in the discussion, a few participants expressed the view that
two separate frameworks should be used, with knowledge as a column for
the applied research programs, and as a row for the science programs.
The concerns expressed were that the more the NRC benefits matrix was
changed to apply to pure research, the trickier it was, and that it
might be better to use a different matrix for technology programs than
for science programs. The argument is that if we try to integrate the
evaluation of the science and technology programs we create a framework
that does not work well for either of them. If we accept their
fundamental differences, we could create two frameworks, each optimized
for the types of decisions and assessments that need to be made for the
two different types of offices. This reasoning was that:

-   The energy resource offices would probably not consider the
    generation of knowledge as part of their core mission and would
    probably not find it useful to have knowledge as a row in the
    matrix.

-   Discussions about viewing knowledge as a row talked about evaluating
    the quality of research management. Although this is an appropriate
    measure of the performance of a program, some conferees felt that it
    was not a measure of the benefit of the R&D program.

The concept of implementing two separate frameworks would require
further discussion, however, because the group did not discuss what to
do if the energy resources offices and science offices R&D benefits
could not be viewed together. Workshop participants did not define
knowledge or what would be measured if it remained a column for the
technology programs. 

##### Knowledge Permeates All the Framework -- the Third Dimension

The group grappled with how to convey in the framework and in
benefit-assessments the fact that knowledge permeates all the cells of
the matrix, that is, it is a third dimension. The government funds
science for two reasons, knowledge creation and knowledge as foundation
for application. Depending on the reason, there are different views of
what is an outcome. Knowledge can be an outcome or an enabler. Placement
will depend on the time scale for benefits, the level of uncertainty,
and the concrete nature vs. the breadth of the research. Knowledge
creation is an outcome for those programs that have advancing knowledge
as a specific goal of current program activities.

Much of the knowledge sought under DOE programs is focused on solving
problems related to the other three rows. Knowledge is a contributor to
an outcome for technology programs as well as a part of planning and
analysis for all R&D programs. The example was given of basic research
on mid-efficiency furnaces that resulted in a vent design that saved a
great deal of energy and money.

Identifying knowledge in a separate column does not display the
dependencies well. Parry Norling pointed out to the group that industry
sees knowledge as a third dimension of the matrix, underlying all its
R&D activities. There is valuation of knowledge at the time of sale and
mergers, and many donate IP (intellectual property) to universities and
value that carefully as a credit on their books. But the group did not
come up with suggestions on how to include knowledge as a third
dimension, other than one proposal to explore displaying knowledge as a
diagonal on the matrix.

There was also concern that double-counting would have to be addressed.
Benefits should be additive. Some will be embedded in products or
mission needs, but other knowledge advances are more generic and will be
appropriately described in the knowledge row.

##### The Benefits of Knowledge -- A Proposed Sub-Framework

In answering the question "What are the benefits of knowledge and how do
(or don\'t) they fit into the framework", many in the group wanted to
adopt a sub-framework for the assessment of knowledge benefits, where
knowledge benefits were broadly defined.

The outputs of research are more than \"simple knowledge.\" Other
outputs are educating and training people (referred to as human
capital), and networks, knowledge infrastructure and capacity. These
allow a rapid response to changes in circumstances and the ability to
handle tougher problems. The group was pleased to have Dave Roessner
organize its thoughts by describing a generic logic model for research
investment, that is, a description of the outputs and outcomes or
benefits of research (refer to figure below). This model was developed
for the BES "research value mapping" project in 1994. Workshop
participants modified the wording slightly as they discussed these
ideas.

This definition of knowledge benefits captures the three major
categories of outcomes used by the NSF: People, Tools, and Ideas. Many
workshop participants thought that it articulated the process well. It
is the way industry now views the R&D process, that is, bringing
together R&D information and business opportunities. And it was
suggested that with this scheme it is possible to ascertain whether you
have a good or bad project. And one could consider the value of a
program as all of these things, an in an ecosystem. Depending on what
profile of these contributions those who fund R&D want, they will manage
to build and maintain activities among the benefits on this list. Not
everyone agreed that this scheme helped make resource allocation
decisions, however.

The most popular idea among many participants was to add columns with
the criteria COSEPUP recommended for GPRA assessment and OMB
recommendations for prospective investment criteria, modifying some
words slightly. (Precise definitions and overlap were not discussed.)

-   Quality of the research

-   Relevance of the research, or strategic fit as industry uses the
    term

-   Performance (defined both as results or more broadly)

-   International scientific leadership

Together the knowledge row and the proposed sub-matrix respond to OMB
and GAO requests with regard to investment criteria and performance
reporting.  The OMB has indicated that the applied-research investment
criteria are moving more toward the basic-research criteria, in part
because applied research cannot meet the DOE pilot criteria which are
better suited to technology development programs.  Joe Wholey of GAO,
speaking to the larger conference, reminded participants that a good
response to GPRA includes intermediate outcomes, not just impacts. 

##### The Sub-Framework Tested on a Hypothetical Technology Program

As a test of the knowledge sub-framework the group was challenged to
populate the matrix cells with measures for prospective assessment.
Representatives of technology programs indicated that they do not
consider research activities exclusive of what they are trying to
accomplish in a market sense. Thus they would not justify their programs
to the Administration or Congress on the basis of knowledge benefit.
Intermediate steps are not discussed at this level except as milestones.
There may be spin-offs but those will not justify a program either.
Knowledge benefits can be documented retrospectively. OMB
representatives indicated this was a reasonable approach for programs
well down the applied path.

Even if knowledge is not embodied in a successful technology, knowledge
can be used in future efforts and save money and time. This would fit
under transition activities, for example, the achievement of technical
performance such as cost per therm. Some of the technology programs
activities fit in the cells of the proposed sub-framework. One office
has a student competition to build alternative fuel vehicles. This
contributes to human capital. It also enhances communities of practice,
as do cooperative research agreements. Other programs fund testing
facilities, which fit under research tools and generation of new
knowledge. Web site dissemination of technical results fits under
transitions as well. This type of program, as well as diffusion programs
such as Clean Cities do not fit in the NRC framework, but could fit in
the knowledge sub-framework.

The sub-framework helps identify what programs should move from basic to
applied, and assists with asking for funding for the new programs. The
sub-framework also points out knowledge barriers where programs could
ask for assistance from basic programs. And more and more private
industry considers that there is justification in funding knowledge
capabilities or capacity, so can they keep a steady stream in the R&D
pipeline and so they can do R&D fast. The knowledge row is about
developing competencies (communities of practice, human capital,
transition opportunities). The DOE programs are also concerned about
having a critical mass required to make progress. This is not captured
well in NRC framework, except perhaps under the option benefit. Some
workshop participants from technology programs felt that if
capacity/critical mass is seen as valuable and programs were allowed to
use as that argument as budget justification, that would be a good
outcome of this workshop.

Many in the group shared their perspectives that the technology programs
would probably not complete most of the cells in the sub-framework
because they would only include significant parts of their programs and
what is suggested above are small portions of their work. Thus the sub-
framework would probably not be a great help for applied programs, and
benefits would not justify the applied programs allocating significant
amounts of funding to \"fill out\" these cells. But if programs had
resources, they would \"pick up\" these knowledge benefits
retrospectively.

##### Testing the Sub-Framework For A Hypothetical Science Program: Prospective Benefits

The experienced R&D evaluators assured the group that filling in this
sub-framework retrospectively is easy to do. Thus the challenge to the
group was to fill in the cells with prospective questions and measures.
Although more thought is needed, the group was able to provide examples
of questions or measures for each cell. These examples are shown in
Figure 2. Workshop participants observed where there was overlap and
items could be combined, but felt it was important and useful to keep
them separate. Keeping them separate gives needed emphasis and retains
important detail which the group did not want to lose. Many workshop
participants saw this sub-framework as being useful and as the
intersection of knowledge and prospective benefits.

##### How Is This Prospective and Retrospective Assessment Going To Be Used?

A senior DOE program planner and evaluator asked what it is that OMB
wants to know. Retrospective assessment answers the question "was it
worth it", but can only be answered on completed projects. The programs
need to use the criteria as a planning tool, not only as an assessment
tool. Programs need to use the criteria to make decisions before the
budget gets to OMB. They also need the criteria to differentiate between
different kinds of programs. Is it possible that one set of information
can do this? Some have thousands of stakeholders, while others have
three. Some have lower barriers than others. Some have to be solved in
stages. An annual budget snapshot on the current criteria does not
reflect those differences, in part because they assess the merits
project by project and not at the portfolio level. Projects that are
"2s" and terminated may affect the ability to do projects that are "4s".
It is also important to assess a program based on its original purpose
and that supporting legislation. It is not appropriate to impose today's
view on something designed and implemented under conditions of
uncertainty.

OMB staff responded that the intent of R&D investment criteria is to
identify and communicate the data that is useful to OMB. They see it 90
percent as a planning tool. However, they agreed that OMB could do a
better job of communicating how these criteria are to be used. Also OMB
is redrafting the criteria for applied R&D programs. Retrospective
assessments show OMB how well specified and managed programs are, and
their relevance and fit. The criteria are used along with other
information such as 5-year plans that help put portfolios in perspective
and provide understanding of the relationships between projects.

Dave Roessner's presentation provided a good summary of measures/methods
and uses of prospective and retrospective assessment in planning and
evaluation, respectively. This is highlighted in Table 1.

To summarize, the focus on prospective or retrospective analysis and
decisions leads to different kinds of studies. Usually only prospective
analysis leads to developing theories of how to get from inputs to
outputs because retrospective-benefit studies typically ignore the
"Black box", the process of managing and doing R&D. Many workshop
participants felt that we need to describe what's in the black box. That
forces the use of management as a decision tool. Black box answers "show
you how the manager is valuable." What did managers do that made
programs work.

**Table 1.** **Effective Benefit Measures and Methods -- Prospective and
Retrospective**

+-----------------------------------+-----------------------------------+
| ###                               | **Program evaluation/             |
| ### Program planning & management | justification**                   |
|                                   |                                   |
| **(Prospective)**                 | **(Retrospective**)               |
+-----------------------------------+-----------------------------------+
| ####### Criteria for Effective Us |                                   |
| e of Benefit Measures and Methods |                                   |
+-----------------------------------+-----------------------------------+
| "Rigorous" - not necessarily      | Credible                          |
| quantitative                      |                                   |
+-----------------------------------+-----------------------------------+
| Detailed                          | Defensible                        |
+-----------------------------------+-----------------------------------+
| Formal                            | Intuitive                         |
+-----------------------------------+-----------------------------------+
| Quantitative                      | Transparent                       |
+-----------------------------------+-----------------------------------+
| Use of rigorous methods and       | Evidence of use of\               |
| measures                          | "Rigorous" methods & measures     |
+-----------------------------------+-----------------------------------+
| ###### Benefit Measures & Me      |                                   |
| thods That Exhibit Those Criteria |                                   |
+-----------------------------------+-----------------------------------+
| Inside the black box              | Impacts/inputs                    |
|                                   |                                   |
| Inside the organization           | Summative                         |
|                                   |                                   |
| Process/formative                 | Focus on what benefits as opposed |
|                                   | to how                            |
| Formal logic models of activity   |                                   |
|                                   | Case studies/anecdotes            |
| Portfolio anal, balancing risk,   |                                   |
| long short term, types of impacts |                                   |
| (e.g. human capital)              |                                   |
+-----------------------------------+-----------------------------------+
| ###### How Benefits Are Realized  |                                   |
+-----------------------------------+-----------------------------------+
| Portfolio analysis                | Peer assessments                  |
|                                   |                                   |
| Institutional, Organizational,    | Client satisfaction               |
| Managerial variable               |                                   |
|                                   | Nuggets                           |
|                                   |                                   |
|                                   | Additionality/counterfactual      |
|                                   |                                   |
|                                   | Cost, value, impact variables     |
+-----------------------------------+-----------------------------------+

Concern was expressed that the group hadn't spent much time on what are
estimated outcomes. But several pointed out that we have done so. The
logic model of knowledge benefits can be used to attribute program
activities to knowledge value communities, or communities of practice,
for example, and these are the types of outcomes that are important. And
peer review with competitive and merit-based selection of research
projects assumes prospective review of quality, strategic fit, and to
the extent possible, anticipated opportunities for application.

## Bibliometric and Industry Measures of Knowledge

Diana Hicks presented the many ways bibliometric techniques can be used
in the assessment of research. Bibliometrics could be used prospectively
for human capital issues and to trace networks. It is also possible to
use bibliometric analysis as an indicator of vitality and where one
might need to make investments. For science, by investigating papers
that cite other papers, the organization can assess knowledge incoming
and outgoing from an organization. By considering the percentage of
top-cited papers, one has an indicator of quality and a value
distribution across a portfolio. For technology, by considering patents
that cite papers and patent portfolios, one can do network analysis.
Tracking people through the patent system would be very valuable, for
example to show a need for expertise.

Parry Norling spoke from 30 years experience managing research at Dupont
and participation with the Industrial Research Institute's Research on
Research Committee. He also spoke of managing the black box. He pointed
out that within the non-linear innovation system that includes inputs,
processes and outcomes, different stakeholders are interested in
different metrics. Norling listed different types of benefits estimating
techniques: net present value, rules of thumb, database of assessments,
studies by independent analysts, value of IP and orphan patents, and
financial analysts' estimates of value. References were provided to
several project-scoring mechanisms that might be helpful to DOE.

Norling also addressed the question of what to do about assessing the
benefits of basic research and pointed to ideas on radical innovation,
knowledge drivers of the future diagram, and strategy tables. A possible
benchmark for DOE to use when thinking about risk is a study on the
success rate of new products that showed that it took 3000 new ideas to
get 300 submitted ideas, and eventually end up with one new commercial
success. Thus an organization needs a steady pipeline of R&D.

Parry also showed a portfolio tool, the familiarity matrix, developed by
MIT Sloan School. The matrix considers the interdependence between the
newness of a technology with the newness of the market to the firm. An
organization can use it to manage risk. Pursuing a new technology in a
new market is "suicide square,\" for example. There are many tools, and
many estimating techniques. It is important to remember that an order of
magnitude estimate is sufficient at first. It is important to verify or
rule out assumptions and establish value, and build these reviews into a
stage gate process.

To relate back to the matrix, Dupont's approach would fit into the
scheme where R&D is a capacity or row, and a third dimension to all R&D
activities. Business judgments don't really fit in the matrix, except as
management makes decisions on how much risk they want to take.

##### Role of Government in R&D Benefits

For science programs, attribution of impacts to government programs is
particularly difficult because of the long and diffuse path from
government activities to the application of knowledge and knowledge
capacity generated by those activities. Expert judgment and trends in
funding by sector are two indicators of contribution. The industry trend
is more dependence on federal basic research. They are doing more
outsourcing of basic research and collaboration, being a smart buyer,
because they can't do it all and time frames are short in many cases.
This trend suggested to the group that in the process of shaping
programs, DOE might include a broad group of stakeholders, including
industry, financial institutions, and large customer groups.

**Figure 1. A Framework For Valuation of Knowledge Benefits**

![](media/image1.wmf){width="6.781944444444444in"
height="4.893055555555556in"}

**Figure 2. Example Prospective Questions and Measures for A Basic
Research Program**

![](media/image2.wmf){width="6.5375in" height="4.794444444444444in"}

[^1]: Organized by Oak Ridge National Laboratory and sponsored by the
    Office of Energy Efficiency and Renewable Energy, Office of Fossil
    Energy, Office Nuclear Energy, Science and Technology, and Office of
    Science of the U.S. Department of Energy. Information about the
    conference is available on the conference web site,
    [www.esd.ornl.gov/benefits_conference](http://www.esd.ornl.gov/benefits_conference);
    in the white paper distributed prior to the conference, \"Ideas on a
    Framework and Methods for Estimating the Benefits of
    Government-Sponsored Energy R&D;\" and in the report summarizing the
    conference proceedings, \"Synthesis of Conference Discussions.\"

[^2]: This document is believed to be a reasonably accurate summary of
    discussions in Workshop C of the conference on \"Estimating the
    Benefits of Government-Sponsored Energy R&D;\" but the accuracy is
    not guaranteed by the workshop rapporteur, Oak Ridge National
    Laboratory, UT-Battelle LLC, Sandia National Laboratories, Lockheed
    Martin Corporation, or the U.S. Department of Energy. Furthermore,
    the opinions expressed by those at the conference are their own and
    therefore nothing in the reporting of the discussions in Workshop C
    or of the conference proceedings should be construed as government
    policy.

[^3]: Sandia National Laboratories is managed by Lockheed Martin
    Corporation for the U.S. Department of Energy.