Conference on

## Estimating the Benefits of Government-Sponsored Energy R&D[^1]

**March 4 and 5, 2002**

### Summary of Discussion in Workshop B:

###  Option Value[^2]

# Brandon Owens, National Renewable Energy Laboratory[^3] 

**May 1, 2002**

**Framework for Defining the Benefits of R&D Programs, and its Use for
GPRA and R&D Planning and Evaluation**

# Session Summary

-   Many in the group considered the basic National Research Council
    (NRC) benefits framework to be acceptable, given the complexity of
    the task.

-   However, there was concern among some participants about the
    inter-dependence and interaction between cells. The concern is that
    the inter-dependence can lead to a fuzzy set of numbers that can be
    easily manipulated.

-   There was an additional concern that the column introduced into the
    NRC matrix does not mean "expected" prospective benefits. The white
    paper seemed to have meant the "most-likely" prospective benefits.
    If only the most likely scenario is considered, then the expected
    value is not computed. A suggestion was made to label the column
    something like the "Business-As-Usual" value.

-   A point was made that both economic and uneconomic technologies have
    option value. This point speaks to the issue of retrospective and
    prospective option value -- technologies have both.

-   Many workshop participants noted that the framework only addresses
    the first-level benefits, and not any multiplier effects. The key is
    that when proceeding with the first-level benefits matrix, we make
    sure to "keep the door open" to consider second- and third-level
    effects.

-   The white paper is unclear about the definition of an option. There
    was some concern that option benefits can be calculated for all of
    the columns.

Specific Comments Made by Individual Workshop Participants About the
Modified NRC Framework for Defining the Benefits of R&D Programs, and
its Use for GPRA and R&D Planning and Evaluation

> \"The knowledge area allows for the programs to easily claim success.
> This is just a caution that needs to be considered.\"
>
> \"Looking at the matrix, I see a lot of segregation. All of the cells
> in the matrix are related, so how do we appropriately capture the
> inter-play between the items? From a portfolio perspective, it's not
> the individual pieces that are important, but the way that they
> interact.\"
>
> \"When you have criteria, they need to be independent, not
> inter-dependent. What we have here is a matrix with highly
> inter-dependent criteria. So they are not really individual criteria
> at all. One has to be very careful in making sure the criteria we
> select. We cannot get crisp results; we can end up with a fuzzy set of
> numbers that can be made to say anything you want. We need lower-level
> criteria that are independent of each other.\"
>
> \"The three rows are the three mission statements of DOE. Their
> mission is economics, environment, and security. So this matrix is a
> way of communicating what DOE is doing in each of these areas. That
> was the rationale behind the matrix.\"
>
> \"\'How are scenarios reflected in this matrix?\' We never solved that
> problem. Many of us believe that if a technology is ready to be
> commercialized but doesn't have the right economic attributes it has
> option value. But we never figured out a way to compute the option
> value.\"
>
> \"We need to be aware that we need to consider the option and
> knowledge benefits of a single program or technology. NRC didn't have
> to address this issue because they looked only at retrospective
> benefits.\"
>
> \"It seems to me that the white paper doesn't mean \'expected\'
> prospective benefits. The white paper seems to have meant the
> \'most-likely\' prospective benefits. Because they looked at the most
> likely scenario, they didn't compute the expected value. We need to
> call that column the \'Business-As-Usual\' value.\"
>
> \"From a framework perspective, there is a huge distinction between
> expected prospective and options benefits. We need to figure out how
> to compute options benefits for renewables.\"
>
> \"With respect to photovoltaic (PV) technologies, we need to figure
> out exactly how to calculate option values. Right now, PV technologies
> are supported because of some vague notion of \'optionality\' but we
> are not properly represented in the expected value analysis.\"
>
> \"Option value can also be used to assign a value to knowledge
> value.\"
>
> \"\'What\'s an option?\' Its any discretionary ability that one gets
> to react in the future in response to some uncertainty. The discretion
> is important, the flexibility is important, this is for the future.
> What's important is the resolution of uncertainty over time. As more
> information arrives we can make a more informed decision.\"
>
> \"Options have both a positive connotation and an insurance
> connotation. Options are about opportunities. Something that may not
> have a value today, but it provides an opportunity in the future.\"
>
> \"It's not true that only uneconomic technologies contain option
> value. Even currently economic technologies have option value. This
> speaks to the issue of retrospective and prospective value --
> technologies have both.\"
>
> \"In the conference white paper somewhere it says that the way to
> calculate the expected prospective value is to calculated the
> \'expected\' value of the technology. This is the net present value
> (NPV) of the technology. The options value is the real options value
> (ROV) of the technology. \"
>
> \"Remove sequestration from the example on page vi in the options
> white paper.\"
>
> \"Why do we have four columns -- seems like there is option value
> associated with each column. I don't understand the distinction --
> isn't option value embedded in expected prospective value?\"
>
> \"With regard to the three rows: economic value is very quantitative
> (Quantitative), environmental impacts are often less clear so we
> proceed with qualitative considerations (Qualitative), for security we
> look at the hedge value of our options (Hedge Value). This may be a
> way we can deal with these items. \"
>
> \"The framework only addresses the first-level benefits, cascading
> benefits are not included. This matrix may constrain us from looking
> at multiplier effects.\"
>
> \"Multiplier effects were purposely left out of the NRC matrix because
> we believe that the investment funds would have resulted in the same
> multiplier benefits if they were spent on an alternative use.\"
>
> \"One large private sector company has looked closely at the
> interaction between programs and benefits, and there are established
> ways to do this. The key is that when we proceed with the first-level
> benefits matrix, we need to make sure that we keep the door open to
> consider second- and third-level effects. We need to make sure that we
> build the funding into the first-level analysis to do this. We have to
> enable future benefits.\"

## Definition of Option and the Calculation of Option Value

**Summary of Discussion on Definition of Option**

-   The definition in the white paper is not really a definition because
    "option" is never defined -- the white paper paragraph simply
    categorized types of technologies. One definition generally accepted
    by many workshop participants is that an option is anything that
    provides "the discretionary ability to react in the future in
    response to some uncertainty." Workshop participants did not all
    agree on this definition, however, and struggled from a terminology
    point of view.

-   There is terminology in the white paper definition that is
    restrictive. For example, there was wording in the white paper that
    restricted the technological sub-set to those that are developed or
    being developed to intentionally not enter the marketplace. Workshop
    participants strove to remove restrictive words and add expansive
    words.

-   There needs to be consideration of energy technologies that are
    currently commercially viable but that have not significantly
    entered the market. A concept or definition is needed to allow
    option value for commercial technologies that are being improved
    through continued R&D.

-   Words like "most likely" and "anticipated" should be replaced with
    the \"EIA Base Case\" if that is what we are really talking about.

**Summary of a Four-Step Methodology**

Several participants in the workshop outlined a 4-step methodology and
identified approaches to conducting each step.

1.  [Describe the uncertainty]{.underline}.

    a.  Use integrated energy-model scenarios and assign probabilities
        to scenarios.

    b.  Identify the underlying uncertainty (technological performance,
        energy prices, environmental regulations, policies, changing
        competitive environment).

```{=html}
<!-- -->
```
2.  [Calculate outcomes]{.underline}.

    a.  Use the results of the scenarios developed in step 1

    b.  Develop a forecast using the underlying uncertainty
        characterizations

    c.  Use a simplified modeling approach

3.  [Determine the decision actions]{.underline}.

    a.  Consider the following decision possibilities: (1) continue, (2)
        abandon, (3) expand and (4) hold.

4.  [Do the expected value calculation]{.underline}.

    a.  Carry out a standard NPV calculation, must decide between
        risk-neutral and risk-adjusted approach. Workshop participants
        had an animated discussion around this technical issue.

**Specific Comments Made by Individual Workshop Participants About
Defining Options and Calculating Their Value**

> \"Please replace technologies with [sciences]{.underline} and
> technologies.\"
>
> \"We need to operationalize the definition so we can put it into
> practice.\"
>
> \"I like the definition on page 47.\"
>
> \"There is no need to restrict technologies to technologies are being
> intentionally not to enter the market. Need to remove the word
> "intentionally" from page 47.\"
>
> \"What if the option value is negative? In many cases, the option
> values will be negative because the BAU scenario does not contain
> uncertainty.\"
>
> \"The definition of the "most-likely" could be gamed to paint a
> certain scenario that is politically loaded.\"
>
> \"Yes but using the EIA Base Case makes it harder to game.\"
>
> \"There needs to be a third paragraph in the definition because there
> are energy technologies that are currently commercial viable but are
> currently being developed. Need something to allow option value for
> commercial technologies that are being improved through continued R&D.
> Big issue -- this is an option value that is not considered in the
> current definition. \"
>
> \"In the prospective statement we could say that technologies are
> being developed OR improved.\"
>
> \"Sometimes you actually know something is not equally viable, but you
> just want to test the market. The notion of testing the market is not
> included in the definition.\"
>
> \"Problem with last sentence in first paragraph: Discard it.\"
>
> \"No, just remove the word \'scenarios.\'\"
>
> \"Yet, but government is the lowest-cost solution provider in the
> marketplace. The ability of the government to lower risk by creating
> options. We need to include this.\"
>
> \"No, that's already there.\"
>
> \"The most likely conditions should be replaced with EIA baseline.
> Also, this is not a definition because \'option\' is never defined.\"
>
> \"Options thinking injects value into the economy. We need to
> incorporate this into the definition of options. Programs need to be
> designed to capture optionality.\"
>
> \"One approach for capturing optionality is as follows:\
> Base the core options analysis on common scenarios

-   A set of alternative scenarios should be created for the evaluation
    of technologies. Criteria for the set include.

```{=html}
<!-- -->
```
-   Are applicable to many and diverse DOE programs

-   Widely accepted as unbiased

-   Capture a wide range of future conditions

-   As few scenarios as possible.

```{=html}
<!-- -->
```
-   The scenarios should include the AEO reference case and others that
    vary across the economic, environmental, and security dimensions.

-   To illustrate option value, value could be calculated under one or
    two scenarios in addition to the baseline scenario (probably the AEO
    reference case by EIA).

-   The values based on one or two alternatives illustrate optionality;
    they do not provide an estimate of options value. A future
    improvement in this approach would be to assign probabilities to the
    scenarios. This would allow a crude expected option value
    calculation to be performed.

-   DOE should support investigation of the probabilities (volatilities)
    and correlations among key energy market drivers such as GDP, fuel
    costs, and environmental regulations. Probabilities of dramatic
    individual events, such as a nuclear moratorium or fusion
    breakthrough, should also be examined. If in the future these were
    generally accepted, these could form the basis for a still more
    sophisticated options analysis.\"

> \"The above approach would provide an extremely rough indicator of
> option value. It is not an estimate of options value. For some subset
> of programs or projects, a more sophisticated real options valuation
> that actually estimates options value should be encouraged as a useful
> alternative valuation approach. Characteristics of a good real options
> valuation include:

-   A very full exploration of uncertainty

-   Creative consideration of many types of options for responding to
    the uncertainties

-   Careful treatment of adjustments for risk with a consideration of
    market information about public valuations of risk

-   Good models for outcomes under a wide variety of scenarios or
    unfolding of uncertainties

-   Calculation of expected value using appropriate uncertainty modeling
    tools (Monte Carlo simulation, stochastic dynamic models, decision
    trees, etc.)

-   Peer review.

> These analyses would help with the valuation and strategic planning of
> important programs or projects with significant options value and
> would begin to build experience with options analysis within DOE.\"
>
> \"With regard to the retrospective definition, the only thing that
> changes is the universe of technologies.\"
>
> \"Again, I think the words \'most likely\' and \'anticipated\' should
> be changed to EIA Base Case.\"
>
> \"The issue of either commercialized or not is a problem. Renewables
> fall into the second category. There needs to be a better distinction
> between whether a technology falls into the prospective or
> retrospective category.\"
>
> \"The National Renewable Energy Laboratory/Colorado School of Mines
> real options analysis is a perfect example of how to do the analysis.
> Just need to apply to individual technologies and then optimize across
> the portfolio. This could easily be done with the existing model.\"
>
> \"No cookie cutter recipe that we can give to DOE. Every time we look
> at one of these things we have to do a custom made analysis. Each time
> you do this you have a different set of data.\"
>
> \"If we were to do this on a project basis -- and we have several
> hundred projects, then the cost of that project would be prohibitive.
> We need to reduce this to make it most cost-effective.\"
>
> \"You've got to consider tangible and intangible benefits. Then you've
> got to translate those into dollars. Tangibles are \'Will it increase
> my revenue? Reduce costs?\' Intangible \'Will it improve goodwill? How
> will stakeholders react?\'\"
>
> \"One step of unfolding the uncertainty, probabilities assigned to
> different outcomes (this is the same as defining the price process and
> determining the drift and volatility), next step is to identify the
> flexibility that is available to us (menu of discretionary choices),
> once you have those two things the rest is just going through the
> process of performing the expected value calculation.\"
>
> \"How about a balanced scorecard approach? The balance scorecard
> criteria depend on the objective of the organization. \"
>
> \"Can we develop a methodology at the program level instead of a
> project level?\"
>
> \"Granularity is important. Need to determine the most efficient unit
> of analysis. We need to group things is manageable chunks.\"
>
> \"The renewable R&D options model is very complex. The model presented
> by Mobil is very simple. We need to end up somewhere in between.
> Suggestion made to compare and contrast the models.\"
>
> \"Don't confuse performance valuation and option value. Option value
> is part of performance value.\"
>
> \"Any real options methodology has to involve: (1) describing
> uncertainty (drift, volatility), (2) calculating outcomes (revenue or
> cost savings), (3) determining the decision points or options, and (4)
> all other factors needed to make the expected value calculation.
> Let\'s go through the 4 steps.\"
>
> 1\. [Describe the uncertainty]{.underline}
>
> \"Before you describe the uncertainty you have to identify which
> uncertainties the R&D are subject to.\"
>
> Suggestion 1: Have EIA develop several scenarios and assign
> probabilities.
>
> \"The scenarios will determine the outcome. We used to find that if
> you have a big model driving the scenario then you'd get no
> divergence. \"
>
> \"What is the relationship between real options parameters and
> scenarios? What if multiple things change, then we have to estimate
> multiple uncertainties.\"
>
> \"Need to look at factors in NEMS that are fixed\"
>
> Suggestion 2: Characterize the underlying uncertainty.
>
> \"List of underlying uncertainty: technology, energy prices,
> environmental, policies, changing competitive environment.\"
>
> \"Need to use market data.\"
>
> \"Can't build things into this that you don't understand very well.
> The problem with NPV is that it doesn't predict market penetration as
> well as you like. If you incorporate uncertainty you're still not
> going to predict market penetration very well. The model will not be
> any stronger than our ability to predict what will happen in the
> market place.\"
>
> 2\. [Calculate outcomes]{.underline}
>
> Suggestion 1: Use the same set of models and assign probabilities.
>
> Suggestion 2: Develop a price process which characterizes uncertainty
> and develop a drift and volatility parameter.
>
> Suggestion 3: Use a simplified model (like John Wise's model from
> Mobil).
>
> \"Start out with on program and see if the three different results
> provide you with the same answer.\"
>
> \"The three approaches have a different ramp-up approach. Scenarios
> are probably most accessible because some DOE offices are already
> doing them. I like the idea of picking one approach and then explore
> different alternative methodologies.\"
>
> \"Look in the literature to see when options values are higher. This
> generally occurs when uncertainty is very high and lead-times are very
> long. You can use these project characteristics as filters. Pilot
> projects with high uncertainties and long lead-times.\"
>
> \"Also, anything that is currently uneconomic on an NPV basis is a
> good candidate for inclusion in the pilot program.\"
>
> \"Need to get consensus from the professional community to determine
> what these probabilities are.\"
>
> 3\. [Determine "option" points or decision actions]{.underline}
>
> Suggestion 1: Use (1) continue, (2) abandon, (3) expand and (4) hold.
>
> \"Must consider compound options because this takes into account the
> learning that occurs. Not sure how relevant these are. These decisions
> are made at a lower level.\"
>
> 4\. [Perform expected value calculation]{.underline}
>
> Suggestion 1: One needs to determine which approach is being used:
> \'Risk Neutral\': Payoffs times the probabilities discounted at a
> risk-free rate, or \'Risk Adjusted\': Payoffs times the probabilities
> discounted at a risk-free rate TIMES a risk adjustment factor.
>
> \"Advantage of risk neutral (approach) is that you ignore risk. The
> most difficult part of this is calculating the risk.\"

**The Baseline Value for Option-Value Calculations**

# Summary of Discussion on Baseline for Option-Value Calculations

-   There is no "options calculation" baseline. Several workshop
    participants suggested starting from the "expected prospective"
    benefits baseline which is the Annual Energy Outlook (AEO) Reference
    Case, which give the expected value of R&D project. A "No R&D" case
    would then need to be defined, according to workshop participants.
    This approach was confirmed by a former EIA staff member familiar
    with the AEO calculations.

-   The key is using the AEO Reference Case as the baseline scenario.
    Several workshop participants noted that we have to be careful to
    back out technology improvements associated with public-sector R&D
    funding.

-   In terms of the options analysis, some workshop participants
    suggested that we could develop alternative cases by perform the
    options-value calculation twice: once with positive public-sector
    R&D, once with negative public-sector R&D.

-   Another approach suggested is to identify a set of scenarios for
    examination. Whether or not DOE would assign a probability to these
    scenarios in order to arrive at an expected option value was left
    undecided by the group.

**Specific Comments Made by Individual Workshop Participants About the
Baseline for Option-Value Calculations**

> \"One of the roles of government is to inject \'optionality\' into the
> economy, so it is important to measure how we measure this. Need to
> develop a richer view of investment in order to capture all of the
> potential scenarios. At least within DOE we have to understand the
> value of and distribution of options.\"
>
> \"There seems to be a disconnect between what industry and academics
> are saying about how real optionsare used and what DOE is saying it
> needs for reporting benefits to OMB. Real options appear to be a
> portfolio management activity (or a way of viewing the portfolio); but
> DOE is saying that it wants to measure R&D, it does not want to manage
> its portfolio. We need to document this for DOE -- if you want real
> options to work for DOE, you have to look at the big picture. It\'s
> not just a number and a value -- its about creativity. We have to
> structure this discussion around (DOE's immediate needs), but it is
> critical that we look at the big picture, otherwise I feel that we are
> shortchanging this entire discussion.\"
>
> \"Remember that DOE deals in a political environment -- these are
> ultimately political decisions. For example, we cannot really explore
> scenarios that are against the Administration's policies (i.e. climate
> change). Interactions with Congress and the Administration are front
> and center for DOE everyday. That affects everything.\"
>
> \"What options is about is looking for value. The idea is to manage a
> portfolio in a way that seeks out value, or maximizes the potential
> value. Unfortunately, what DOE does is just the opposite. For example,
> they get fixated on saving energy to the neglect of other
> technological benefits that may add value.\"
>
> \"Can we not define a spin-off as an option? Yes, we can limit our
> discussion of the target market or primary technology, or we can
> attempt to model alternative markets and spin-off applications as
> options.\"
>
> \"There is no "options calculation" baseline. We start from the
> \'expected prospective\' benefits baseline which is the AEO Reference
> Case which gives us the expected value of R&D project. We need a \'No
> R&D\' case.\"
>
> \"The key is using the AEO Reference Case as the baseline scenario, we
> have to be careful to back out technology improvements associated with
> public-sector R&D funding.\"
>
> \"Baseline doesn't make sense in the options world. The \'baseline\'
> is simply one point in the options scenario.\"
>
> \"Yes, but if we are moving from traditional analysis to real options
> analysis, it is useful to identify the baseline, which is a starting
> point.\"
>
> \"There is no need inherent in real options analysis to identify a
> baseline. It\'s just that the department has a history of using a
> baseline. This is a way of business for DOE. This would be a departure
> to dismiss the notion of a baseline. \"
>
> \"The thinking of the conference organizers didn't understand options
> valuation, because this idea of baseline just doesn't work.\"
>
> \"We need a \'No R&D\' NEMS base case. One DOE participant claims that
> the EIA Base Case does not include R&D effects. However, this doesn't
> appear consistent with EERE's QM/GPRA practice.\"
>
> \"How do we go about defining specific scenarios, other than the
> baseline?\"
>
> \"We perform the options value calculation twice. Once with positive
> public sector R&D, once with negative public sector R&D.\"
>
> \"Wait, we can define scenarios -- but calculating the value of a
> particular scenario occurring is the difficulty.\"
>
> \"The framework is more valuable if we are able to come up with
> scenarios, not probabilities. Because no one will believe what EERE's
> probabilities are.\"
>
> \"Maybe we need to develop a common set of scenarios.\"
>
> \"It is not politically feasible to come up with a set of
> probabilities, or worse yet -- to bury a set of probabilities into an
> analysis.\"
>
> \"Four measures of a good scenario: (1) consistent across the
> technologies, (2) need to be accepted as unbiased, (3) have to capture
> a wide range of pictures of the world, (4) want as few as possible.\"
>
> \"Why do scenarios -- aren't we characterizing the underlying
> uncertainty.\"
>
> \"The managers are more comfortable to talk in terms of scenarios. So
> that's why we are doing scenarios.\"
>
> \"Some uncertainties, you don't have to get into this discussion at
> all because there is market data.\"
>
> \"Yes, but DOE may not want to use that data. What is most politically
> expedient is to develop scenarios without assigning probabilities. No
> one will believe our probabilities. We should leave it to others to
> assign a probability to the scenarios.\"
>
> \"No. You need to identify the key uncertain drivers and characterize
> these drivers. Characterizing the uncertainty is what you'll have to
> do anyway so scenarios don't make sense.\"
>
> \"No. Scenario analysis is a very well defined analytic approach, so
> we want to rely on this accepted approach. Need to wait until the
> literature catches up before we starting applying probabilities. We
> could possibly apply probabilities if we were able to perform a
> meta-analysis of the existing literature. We simply don't know what
> the probabilities are. We don't want to overstate what we know about
> the world.\"
>
> \"You can't just develop scenarios without assigning probabilities.
> That doesn't tell you what the option value is.\"
>
> \"Another approach is to identify a set of scenarios for examination.
> Whether or not we'd assign a probability to these scenarios in order
> to arrive at an expected option value was left undecided by the
> group.\"

## Effect of the Government\'s R&D Program on Option Value

# Summary of Workshop Discussion on the Effect of the Government's R&D Program on Option Value

There was no consensus on this issue because this is extremely difficult
to grapple with. Highlights of some of the key ideas in this session
were:

-   If R&D is what we call "joint" meaning the outcome cannot happen
    without both pieces being there, then it is impossible to calculate
    without separating the individual contributions of each piece. The
    key is whether or not the research is separable.

-   One participant suggested the use of a production function approach
    to disentangle the public and private sector contributions to
    technology success. However, it was pointed out that this would give
    us only the marginal contribution of R&D, not the total contribution
    so a suggestion was made to re-word the question to one that we can
    answer.

-   For the NRC study group interviewed people in the private sector on
    what they perceived to be their contribution relative to the
    government's contribution. Based on these interview NRC developed
    the admittedly crude 5-year rule. So one participant suggested that
    the thing to do is to sit down with private-sector R&D participants
    and get their opinion.

-   One participant noted that the reason that you do R&D is to reduce
    the uncertainty of a controllable process. The reason a private
    company doesn't take up R&D is because there is too much
    uncertainty. One measure of the effectiveness of R&D is reduction in
    uncertainty which can also be considered knowledge acquisition so
    the group saw some strong linkages with the knowledge group in this
    area.

-   The government's role in the technology development cycle is to
    ensure scientific, technical and economic feasibility. If these
    feasibilities were not demonstrated then the chances of this
    technology being picked up by the private sector would be quite
    small. Therefore government should get full credit. Government takes
    the uncertainty out of the technology.

# Specific Comments Made by Individual Workshop Participants About the Effect of the Government's R&D Program on Option Value

> \"The Reference Case in the National Energy Modeling System (NEMS) is
> supposed to capture continuing trends in R&D and technology
> development in general. However, different portions of NEMS have
> different structures. Some are very detailed -- for example the
> residential demand module is very technology-specific. For those areas
> of the model that are very detailed different experts are consulted to
> get a handle on current and expected future technologies. For all of
> the sectors there is also a "high-technology" case that represents
> accelerated technology development. *[R&D is included in the Reference
> Case]{.underline}*. No uncertainty or stochastic process is included
> in the analysis. No probabilities are attached to the NEMS results."
>
> \"A theoretical point is that if R&D is what we call a "joint"
> (activity) meaning the outcome cannot happen without both pieces being
> there, then it is impossible to calculate (the impact of the
> government) without separating the individual contributions of each
> piece. Consider a bicycle**,** no good without wheels. What value does
> the wheel contribute? You cannot tell. The key is whether or not the
> research is separable.\"
>
> \"I disagree. We (can) consider R&D as products of production
> functions. You can separate out the marginal contribution of different
> factors of production. Can 't we do that here.\"
>
> \"Yes, we can separate out the marginal contribution of R&D. Perhaps
> we should re-word the question to one that we can answer.\"
>
> \"If you use the NRC 5-year rule, then you've already solved this
> problem.\"
>
> \"When you have a change in product quality we try to look at the
> attributes of the product. But it gets fuzzy because you have to
> attribute some share of the improvement in the product quality to each
> participant.\"
>
> \"For the NRC study we sat down with people in the private sector on
> what they perceived to be their contribution relative to the
> government's contribution. It varied significantly. Some thought that
> the government's contribution was large and some thought it was small.
> Based on that we developed the 5-year rule, which is a very subjective
> thing. The error-bar in the calculation is probably 50 percent. So the
> thing to do is to sit down with R&D participants and get their
> opinion.\"
>
> \"Direct interviewing is ok, but some of the bigger companies don't
> want to disclose this information. It opens the door to regulation.
> Plus there may be some biases. You need to complement the direct
> interview method with some other analytics.\"
>
> \"We also must consider the enabling technology effect.\"
>
> \"The reason that you do R&D is to reduce the uncertainty of a
> controllable process. The reason a private company doesn't take up R&D
> is because there is too much uncertainty. One measure of the
> effectiveness of R&D is reduction in uncertainty which can also be
> considered knowledge acquisition so (the workshop participants) saw
> some strong linkages with the knowledge group in this area.\"
>
> \"We've got to consider the role that the government plays as a
> catalyst. Retrospectively you might say that you can divide the
> funding 50-50. Prospectively, without government funding some projects
> would not occur so you have to give full credit to the government.\"
>
> \"Government should get all of the credit. Just look at marginal
> technologies. The government's role in the technology development
> cycle is to ensure scientific, technical and economic feasibility. If
> these feasibilities were not demonstrated then the chances of this
> technology being picked up by the private sector would be quite small.
> Therefore government should get full credit. We take the uncertainty
> out of the technology.\"
>
> \"Reducing uncertainty is about creating knowledge. Acceleration is
> another R&D contribution (can be captured with time-value-of-money).
> Also, superior products may be developed through government R&D. All
> of these are element of the R&D contribution. The next question is how
> to measure this.\"
>
> \"There are two different models of behavior that are implied here.
> One participant is saying that the key to R&D is to reduce
> uncertainty, others may say that the key is to improve uncertainty.
> This makes a difference in terms of a benefits discussion.\"
>
> \"In a sense we are talking about two different kinds of decisions. At
> the margin, DOE divides up their budget. what they'd like to do is
> spend the marginal dollar that provides the biggest bang for the buck.
> Afterwards, that's a different question.\"
>
> \"Is the government contribution to R&D unknown or unknowable? Some
> are judgments. In the science side it is knowable. Once you've proved
> scientific feasibility these things are knowable, it\'s just very
> difficult to discover the value.\"
>
> \"We have to be careful to avoid doing something because it is
> difficult to get a handle on it\".
>
> \"Is there reason to believe that the private sector is investing in
> the same areas as the government. The government may be looking for
> options (e.g. \'insurance\') while the private sector may be looking
> at expected benefits (e.g. IRR).\"
>
> \"That distinction may work well in some cases but not others because
> government invests in a wide variety of projects, not just those that
> create options.\"
>
> \"Your point is that the government invests in projects that provide
> the ability to react in a highly unlikely event that provides a large
> magnitude impact.\"
>
> \"Don't underestimate the role of the private sector. Also, there is
> more public money than just federal. There is state-level R&D,
> etc...\"
>
> \"If you are teasing out the value of federal R&D, then you'll want to
> do it in both the options and the expected benefits calculation.\"
>
> \"There are no hard and fast rules, you've got to drill down to find
> the answer.\"
>
> \"Problem is that we just don't have a model to do this right now. We
> could apply a simple production model (Cobb-Douglas) to do this.\"
>
> \"It\'s a complex issue but there are models that allow us to get a
> handle on the issue.\"

**Using Estimates of Benefits in R&D Program Planning and Evaluation**

## Session Summary of Discussion on Using Estimates of Benefits in R&D Program Planning and Evaluation

-   A three-step process was suggested: (1) develop a well-defined set
    of scenarios that can be used across DOE programs, (2) encourage the
    use of the alternative- valuation clause of GPRA to work on
    incorporating real options analysis, (3) develop a set of screening
    scales that include consideration of optionality.

-   Options thinking leads to a different method of planning. So in
    addition to being a "GPRA" type tool, options consideration is also
    a management tool.

-   R&D options need to be managed or else they become very costly. Need
    to take a holistic view, the way you manage your options is the
    essence.

-   If we can't stop programs, or expand programs when we need to then
    we still need to put this in the model and the option values will be
    reduced.

Specific Comments Made by Individual Workshop Participants About Using
Estimates of Benefits in R&D Program Planning and Evaluation

> \"One of the interesting messages of this session is that we began by
> trying to identify some of the advantages of the program that weren't
> being captured, but consideration of \'optionality\' leads to a
> different method of planning. So in addition to being a \'GPRA\' type
> tool, options consideration is also a management tool.\"
>
> \"Three (suggestions are): (1) develop a well-defined set of scenarios
> (including AEO Reference Case), it would be recognized that you could
> select from 3 of these scenarios to calculate program benefits. Also
> need to point toward a time when you'd assign probabilities to these
> scenarios and look at the uncertainty of the fundamental drivers of
> these scenarios. (2) Encourage the use of the alternative-valuation
> clause of GPRA. This will open the door to real "real options." Get
> this idea of options in the culture at DOE. (3) Develop a set of
> screening scales that include consideration of optionality. These are
> simple enough to apply that the managers of a program could get
> together in an exercise where all of the programs are scaled.\"
>
> \"If tools are used for quickly screening individual projects, DOE
> should assure that the tools recognize option value. Scales that
> include consideration of optionality and can be applied quickly to
> many projects can be developed. Scales should cover three areas:

-   Potential impact of the technology

-   Uncertainty and potential for learning

-   Flexibility in developing and implementing the technology

> For example, the flexibility scale might be built around the following
> concepts:

-   High flexibility

```{=html}
<!-- -->
```
-   Technology creates value in many scenarios

-   Much of the research is applicable to many other technologies

-   Research investment is **not** lumpy, that is investment can be
    staged. For example a project that requires \$10 million/year over 5
    years is more flexible than a project that requires \$50 million of
    immediate investment.

```{=html}
<!-- -->
```
-   Low flexibility

```{=html}
<!-- -->
```
-   Technology creates value in a single scenario only

-   Specialized research that contributes to a single technology

-   One large lump sum investment.

> Application of the scoring tool would require a peer review process to
> assure a relevant scoring process.\"
>
> \"Approved scenarios need to be defined differently for different
> programs in DOE. They need to be program-specific scenarios as opposed
> to DOE-wide scenarios.\"
>
> \"Need to consider DOE's guiding principles and look at things from a
> portfolio perspective.\"
>
> \"R&D options need to be managed or else they become very costly. So I
> would caution DOE on the hype associated with option value. Need to
> take a holistic view, the way you manage your options is the
> essence.\"
>
> \"If we can't stop programs, or expand programs when we need to then
> we still need to put this in the model and the option values will be
> reduced.\"
>
> \"Next steps:

-   Need to train, educate and communicate to the various parties (all
    the way up the management ladder) the fundamentals of option values.

-   Begin moving forward in the development of a well-defined set of
    scenarios to begin calculating option benefits.

-   Start a pilot project to apply real options analysis to a specific
    R&D program.\"

[^1]: Organized by Oak Ridge National Laboratory and sponsored by the
    Office of Energy Efficiency and Renewable Energy, Office of Fossil
    Energy, Office Nuclear Energy, Science and Technology, and Office of
    Science of the U.S. Department of Energy. Information about the
    conference is available on the conference web site,
    [www.esd.ornl.gov/benefits_conference](http://www.esd.ornl.gov/benefits_conference);
    in the white paper distributed prior to the conference, \"Ideas on a
    Framework and Methods for Estimating the Benefits of
    Government-Sponsored Energy R&D;\" and in the report summarizing the
    conference proceedings, \"Synthesis of Conference Discussions.\"

[^2]: This document is believed to be a reasonably accurate summary of
    discussions in Workshop B of the conference on \"Estimating the
    Benefits of Government-Sponsored Energy R&D;\" but the accuracy is
    not guaranteed by the workshop rapporteur, RDI Consulting, Oak Ridge
    National Laboratory, UT-Battelle LLC, National Renewable Energy
    Laboratory, Midwest Research Institute, Battelle, Bechtel, or the
    U.S. Department of Energy. Furthermore, the opinions expressed by
    those at the conference are their own and therefore nothing in the
    reporting of the discussions in Workshop B or of the conference
    proceedings should be construed as government policy.

[^3]: Brandon Owens was on the staff of the National Renewable Energy
    Laboratory at the time of the conference, and is now with RDI
    Consulting. The National Renewable Energy Laboratory is managed by
    Midwest Research Institute, Battelle and Bechtel for the U.S.
    Department of Energy.