# Presentation: 319653

## Estimating Program Benefits

- NREL Analysis Brown Bag
- October 9, 2003
- Mary Beth Zimmerman
- Office of Energy Efficiency & Renewable Energy
- U.S. Department of Energy

## EERE’s been at this a while

*1993 Managing for Results Initiative*.
- Quanity metrics, GPRA benefits
- Always the same goal: a consistent set of methodologies, assumptions, and metrics
- Annual guidance covering baseline, methodology & economic data
- AD Little, Inc. reviews of assumptions
- Lots of help from our friends
- 1996 GAO report
- National Academies of Science

## Why benefits analysis?

- Portfolio management
- Select a mix of R&D, deployment that achieves desired results
- Be agile in keeping portfolio aligned with evolving needs
- Congress wants to buy benefits, not technologies or activities
- Public health, national defense, a clean environment
- It’s required
- 1992 Government Performance & Results Act (GPRA)
- 2001 President’s Management Agenda (PMA)

## Who wants what?

- GPRA
- Strategic plans, annual performance plans
- Missions, outputs,outcomes, goals, verification
- Accountability reports
- PMA
- Performance Assessment Rating Tool
- R&D Investment Criteria
- Decision points, end points, off-ramps, goals
- Performance, outcome information in budget, on web
- CFO
- 5-year budget plans
- Budget text

## GPRA Definitions

- Outcome measure  [Is it important?]
- Output measure  [Is it well focused?]
- Performance goal  [When are you done?]
- Performance indicator  [How’s it going?]
- Program activity  [Is it tied to the budget]
- Program evaluation  [How do you know?]

## R&D Criteria

| NUMBER | CRITERIA |
| --- | --- |
| Part 1: | Federal Role |
| 1.a | Is it a Presidential priority? |
| 1.b | Does it address market barriers? |
| 1.c | Does it obtain public benefits? |
| 1.d | Is it the best policy option for the job at hand? |
| Part II:   | Management |
| 2.a | Well designed, integrated with other research, technically feasible? |
| 2.b | Was/is industry involved in developing the research plan? |
| 2.c | Is/will industry provide cost-sharing? |
| 2.d | Does the R&D plan have performance indicators? |
| 2.e | Does the plan have “off ramps” (decision points where might be terminated)? |
| 2.f | Does/will the effort utilize competitive, merit-based awards?		 |
| 2.g | What is the timeframe for commercialization? (not scored) |
| 2.h | What R&D phase is addressed, e.g., applied, demonstration? (not scored) |
| 2.i | What is the overall level of risk? |
| Part III: | Performance |
| 3.a | What economic, environmental, or security benefits are expected? |
| 3.b | How is effectiveness measured? (not yet defined, basically administration) |

## Benefits result from realizing program goals

- Program Performance Goals
*Wind energy @ 3cents/kWh*
- EERE Strategic Goals
*Efficiency, renewable capacity, production *
- Resulting Benefits
*Economic, Environmental, Security/reliability*

## The story is simple . . .

- If you give me some $$,

- I’ll do some stuff with it,

- making some progress each year

- until I produce a product

- which will provide the benefits you want

**Notes:**
By the way, it’s not a co-incident that you see some corporate slogans in this.  “Just Do It” and “Make Progress Every Day.”  After all, the folks who develop the corporate way of doing business learned about integrated budget-and-performance, delivery-based management in school, too.  

## The jargon

- If you give me some $$,

- I’ll do some stuff with it,

- making some progress each year

- until I produce a product

- which will provide the benefits you want

- Budget, Resources, inputs

- Key activities

- Targets, milestones

- Outputs, program goals, end points

- Outcomes, impacts,
- Benefits

- The GPRA
- “logic chain”

- Program
- performance

- Portfolio benefits

**Notes:**
By the way, it’s not a co-incident that you see some corporate slogans in this.  “Just Do It” and “Make Progress Every Day.”  After all, the folks who develop the corporate way of doing business learned about integrated budget-and-performance, delivery-based management in school, too.  

## Basic Performance Management Elements

- Inputs
- Budget
- Expenditures by others

- Activities
- Key
- Project

- Milestones
- Annual
- Critical Path
- Graduation

- Outcomes
- Interim
- Final

- Outputs
- Interim
- Final

- _PROGRAM PERFORMANCE:_
- What we are held accountable for

- _BUDGET:_
- Available resources

- _BENEFITS:_
- How Programs Address
- Energy Goals,
- Needs_ _

## Assessing benefits of achieving program goals

- Inputs
- Budget
- Expenditures by others

- Activities
- Key
- Projects

- Milestones
- Annual
- Decision points

- Outcomes
- Strategic goals
- Benefits

- Outputs
- Program performance goals

**Strategic Planning**

**Evaluation**

**Performance Planning**

**Annual Operating Planning**

## What Product Does It Support?

- Inputs
- Budget
- Expenditures by others

- Activities
- Key
- Project

- Milestones
- Annual
- Critical Path
- Graduation

- Outcomes
- Interim
- Final

- Outputs
- Interim
- Final

**Planning:**
- Budget
- Benefits Rprt.
**Evaluation**
- Benefits Rprt.

**Planning:**
- Budget
- AOP
**Evaluation:**
- Monthly revs

**Planning:**
- Budget
- APP
- R&D Scorecard
- PART
- Joule
**Evaluation:**
- Monthly revs
- Quarterly revs
- Account Rept.
- PART

**Planning:**
- Benefits Rept.
- Budget
- Strategic Plan
- R&D Scorecard
- PART
**Evaluation:**
- Benefits Rept.
- PART

**Planning:**
- Budget
- APP
- R&D Scorecard
- Joule
- PART
**Evaluation:**
- Account Rept.
- PART

## EERE Logic Model

- Source: John Mortenson

## Applied R&D timeline

- EERE R&D:
- 3 to 30 years

- 5 to 30 years
- for appreciable
- stock turnover

- Frequent gap between federal
- R&D and commercialization

## Start with Program (Output) Goals

- Usually for each major area of research
- E.g., CSP, PV, Solar Thermal
- Exclude exploratory work to establish a pathway
- Two elements
- An ***end point*** or date when the effort will be completed
- An ***output***, or level of accomplishment when done

## Trendable performance measures

- C/kWh

- Conceptual

## Add External Factors

- Outputs:
- What the program produces

- Outcomes
- What it means for the public

- External
- factors

**R&D output:**
*Technologies capable of producing solar power @ 7 cents/kWh*
**Deployment output:**
*Technical assistance supporting the installation of __ MW of solar power*

- Electricity prices
- Air quality needs
- Other DG techs

- Systems purchased
- Energy savings
- Lowe emissions
- Improved reliability

## And a Few More Steps

**R&D output**

**Resulting benefits**

- Technology
- Performance
- & Cost

**External factors**

- Market,
- policy
- conditions

- Emissions
- reductions

- Energy cost
- savings

- Net economic
- benefits

**Notes:**
Our challenge in estimating the benefits of EERE R&D programs starts with the fact that R&D develops knowledge and technologies.  

These are the program outputs.  

The translation of these R&D results into benefits depends on a large number of external factors.  

A large part of our job is understanding these factors, both on a market-by-market basis and for the energy economy overall.   

## EERE Reorganization

- 	Streamlined (re)organization facilitates performance management
**TD:  Clearly defined programs provide basis for measuring program performance.**
**BA:  Integrated planning & budget formulation provide basis for linking budget decisions to corporate results.**

- Geo

- EERE

- Solar

- Wind

- TD

- BA

- Hydrogen

- Vehicles

- Ind

- Bldgs

- Wx

- DEER

- Bio

- FEMP

- Planning
- Budget &
- Analysis

- Program
- Execution

- Information &
- Management
- Systems

## EERE’s Reorganized Approach

- Programs provide output goals as part of their multi-year performance plans.
- Market-specific information is developed and assessed for importance to bottom-line.
- Market-specific information is basis for energy-economic modeling of benefit estimates.
- Where model cannot develop benefit estimates, off-line results are adjusted to reflect economic interactions.

## What would have happened without us?
Baseline technology improvement
Baseline market evolution
How will markets respond to EERE technologies
Individual markets
Energy economy
How do deployment efforts change market receptivity

- Baseline technology improvement
- Baseline market evolution
- How will markets respond to EERE technologies
- Individual markets
- Energy economy
- How do deployment efforts change market receptivity

- Need to Understand Future with & W/out EERE Programs

## The Ground Rules

- Benefit estimates based on stated program output goals achievable with current funding levels
- Goals are assumed to be successful
- Benefits = difference between how energy would be used without us & with us.
- For the budget, benefits are counted for program activities from here on out only
- One baseline for everyone
- Energy prices, demographics, GPD, industrial output
- Demands for energy services
- Improvements in competing technologies
- Policies don’t change

## Empirical Foundation

- Not good enough to just assert lock-out, valley of death -- some products do succeed without us!
- Need to explain--not explain away--observed behavior because this is the key to changing markets.
- If we don’t account for price effects, market interactions, we’ll over- or under-estimate the value of our efforts.

## Technology Characteristics

- Cost
- EERE technology vs. future baseline technology
- Performance
- Energy efficiency
- Energy production efficiencies
- Non-energy performance (e.g., acceleration, quality of light)

## Electric resistance heat 
Electric heat pump 
New & improved heat pump 
Advanced electric heat pump
Gas heat pump 
Ground source heat pump
Oil burner to space heat
Improved oil burner to space heat 
New oil burner to space heat 
LPG burner to space heat 
Kerosene burner to space heat

- Electric heat pump
- New & improved heat pump
- Advanced electric heat pump
- Gas heat pump
- Ground source heat pump
- Oil burner to space heat
- Improved oil burner to space heat
- New oil burner to space heat
- LPG burner to space heat
- Kerosene burner to space heat

- Gas burner to space heat
- Improved gas burner to space heat
- New gas burner to space heat
- Solar heat - electric backup
- Solar heat - oil burner backup
- Solar heat - gas burner backup
- District heat exchanger - space heat
- Fireplaces & woodstoves
- Building shell conservation (3 levels)
- New building shell conservation (3 levels)

**Residential Sector Space Heating Technologies**

## MSW - mass burning-electricity
Biomass combine cycle (ATS/IGCC)
Biomass direct fired electric
Geothermal electric, liquid
Geothermal flashed steam electric
Geothermal binary cycle
Geothermal central heating plant
Hydroelectric
Hydroelectric pumped storage
Solar central thermal electric

- Biomass combine cycle (ATS/IGCC)
- Biomass direct fired electric
- Geothermal electric, liquid
- Geothermal flashed steam electric
- Geothermal binary cycle
- Geothermal central heating plant
- Hydroelectric
- Hydroelectric pumped storage
- Solar central thermal electric

- Central photovoltaic
- Photovoltaic - residential
- Solar central heating plant
- Solar thermal dish sterling
- Ocean thermal electric
- Wind central electric - class 5-7
- Wind central electric - class 4
- Wind central electric - class 3
- Local wind electric generator

**Renewable Electric Generation Technologies**

## Market characteristics

- Private sector R&D investment rates
- Market fragmentation/integration
- Consumer preferences
- Price responsiveness, non-price attributes
- Differences among customer classes
- Niche markets
- Regional variations
- Price, availability

## Regional variation

- Source:   Energy Information Administration, May 2000.

**Deregulated States**

**Notes:**
No 2 markets are alike and the distinct market, policy, and technology factors found in each market – electricity, buildings, fuels – will help shape whether and how new technologies emerge.

Energy-economy models can capture some, but not all, of the critical differences in the structure of their sub-modules and in the parameters specified. 

Market-level information is usually needed to augment or inform the economy-wide models. 

EERE maintains a number of such efforts, that address some of the factors beyond simple-cost effectiveness that can shape a markets future:  
     Is market regulated?
What technology characteristics do consumers prefer?
Do resources or markets vary by region?
Are there niche markets available?
What are the resource markets likes? 
How prepared is the industry to gear-up for production?
Is new infrastructure required?

## Why Focus on Markets?

- That’s where EERE technologies got bought and put into use
- These are the realities we are up against
- Competition from improved fossil technologies
- Consumer discount or hurdle rates
- Market frictions
- The more we understand our markets, better we can tailor our technologies, deployment effort for success
- Is a value to technical potential, addressed later

## R&D, Deployment modeled differently

- R&D modeled as different TC, or earlier start date for techs already in the model

- Deployment modeled as accelerated market penetration or lower consumer hurdle rate

## Deployment impacts

- How much do we wish to increase market share?
- What deployment methods best address the market barriers observed?
- How many potential consumers must be reached to generate the needed additional benefits?
- Are there problems with degredation or pesistance of performance?
- Can the market be transformed so that deployment efforts are not required to sustain?

## Relating Trees to Forest

- Specific markets
- What do suppliers supply, & why?
- What do consumers buy, & why?
- Nitche markets
- Regulatory, policy factors
- Regional, income, or other market variations

- Energy-economy
- Supply & demand curves
- “Take-back” effect
- Upstream impacts
- Cross-fuel interactions
- New energy pathways

## Why Use Models

- Keep track of interactions
- Feed-back effects
- Down-stream effects
- Competition, synergies among our technologies
- Get the big picture
- Price reductions
- Balance of growth, fall in natural gas
- Back-out what would have happened without us
- Figure out what’s important
- Confusing results sometimes lead to insights
- “What if” runs can help us understand which technologies, policies are having the biggest impact

## Models Summarize Information

- Rely on empirical information about markets to specify these conditions
- Can be modified to reflect new types of energy, energy services, changes in consumer behavior, policy, etc. ***IFF*** the necessary information is available.

## EERE Models

- NEMS
- Developed by EIA
- Through 2025
- Rich in market detail, can be limited in technology detail
- Regional
- Consumer hurdle rates

- MARKAL
- Developed by Brookhaven
- Through 2050 (our verision)
- Used in many countries; IEA & EIA developing international versions
- Rich in technology detail, can be limited in market detail
- Optimization

## Market interactions

## Integrated Modeling of Technologies in MARKAL  

**Integrated Modeling of Technologies in MARKAL****  **

- Fuel
- Cell

- Hydrogen

- Ethanol

**Notes:**
MARKAL is an Integrated Energy Technology system
For each energy service demand, the model identifies all available technologies, which will compete to meet the demand on the basis of technology cost, efficiency and fuel costs.
All energy services can be linked to technologies by fuel type, fuel supply, and sources of primary energy supply.

## The big picture 

## NAS/NRC Benefits/Costs Framework

|  | Realized 
Benefits and Costs | Options 
Benefits and Costs | Knowledge 
Benefits and Costs |
| --- | --- | --- | --- |
| Economic Benefits and Costs |  |  |  |
| Environmental Benefits and Costs |  |  |  |
| Security 
Benefits and Costs |  |  |  |

## NAS Benefits frameworkhelps us tell more of our benefits story

|  | Realized
Retrospective | Expected
Prospective | Option |
| --- | --- | --- | --- |
| Economic |  |  |  |
| Environ-mental |  |  |  |
| Security/
Reliability |  |  |  |
| Knowledge |  |  |  |

## Modeling Market Uncertainty 

- Scenarios “easiest” next step
- High/low fuel prices
- Environmental constraints
- Consistent with other DOE offices
- Value of technology on the shelf
- Technical Potential defines the extent of the options developed
- Analytical techniques for ascribing value to the shelf
- “Real options” approaches to portfolio management use this information about uncertainty to shape R&D, deployment strategies

## Market uncertainties

- EIA coal price projections

**Notes:**
We all think of oil markets as subject to unexpected price swings.  

But even in markets viewed as relatively stable, uncertainties can be very large within the timeframes of R&D investments.

This is an example of how EIA estimates of future coal prices changed over a period of just 8 years, during a time of relatively stable energy markets.

Three events conspired to change the coal price picture:
	1.	Policy factor:  CAA & western production shift to lower-cost coal
	2.	Technology factor:  Long wall mining lowered price
	3.	Market factor:  Competition with natural gas in electricity markets dampened prices

This spread in expectations is important to our portfolios because it shapes how and when various competing energy technologies might enter the market--and what the research goals ought to be in order to meet those needs.

Reliance on BAU forecasts only will miss these elements of structuring energy R&D portfolios.

## Where we are: Options

- Taking uncertainty into account changes R&D choices
- Positions firm for greater range of future markets
- More likely to invest in higher risk, longer term R&D
- Better manage stage-gating of R&D, more likely to cancel poorer performing projects earlier
- A product doesn’t have to enter the market to provide value
- Lots of approaches to uncertainty
- Some don’t seem quite ready for prime time
- Some would be tedious, data-intensive to apply to our portfolio
- We can stage-gate our development of the options column
- Scenarios give good first picture of how portfolio meets possible future needs.
- Risk assessment tools give a more realistic expectation of research results.

## Where we are: Security

- There are standard approaches to evaluating security that we can use:
- Probability, severity, duration, recovery
- Important to identify which of these we address--and which we do not.
- Valuable to be in-sync with these general approaches.
- Getting a handle on energy infrastructure will be key to assessing security, reliability benefits.
- Agent-based approaches may be necessary to assessing security & reliability

## What we’re working on now

- Streamline benefits analysis within new organization
- Use Logic Chain to improve transparency
- Extend timeframe looking forward
- Add retrospective component
- Develop metrics for all NAS rows
- Economic, environment, security
- Develop 2-3 scenarios for option column
- Select and adopt a risk assessment approach
- Removing EERE impacts from baseline
- Improved reflection of market features
- Regional variation
- Infrastructure
- Consumer preferences