NREL is a national laboratory of the U.S. Department of Energy

Office of Energy Efficiency & Renewable Energy

Operated by the Alliance for Sustainable Energy, LLC

This report is available at no cost from the National Renewable Energy

Laboratory (NREL) at www.nrel.gov/publications.

Contract No. DE-AC36-08GO28308

The Value of Improved ShortTerm Wind Power Forecasting

B.-M. Hodge and A. Florita

National Renewable Energy Laboratory

J. Sharp

Sharply Focused, LLC

M. Margulis and D. Mcreavy

Lockheed Martin

Technical Report

NREL/TP-5D00-63175

February 2015
NREL is a national laboratory of the U.S. Department of Energy

Office of Energy Efficiency & Renewable Energy

Operated by the Alliance for Sustainable Energy, LLC

This report is available at no cost from the National Renewable Energy

Laboratory (NREL) at www.nrel.gov/publications.

Contract No. DE-AC36-08GO28308

National Renewable Energy Laboratory

15013 Denver West Parkway

Golden, CO 80401

303-275-3000 • www.nrel.gov

The Value of Improved ShortTerm Wind Power Forecasting

B.-M. Hodge and A. Florita

National Renewable Energy Laboratory

J. Sharp

Sharply Focused, LLC

M. Margulis and D. Mcreavy

Lockheed Martin

Prepared under Task Nos. WTCU.1000 and WE11.0706

Technical Report

NREL/TP-5D00-63175

February 2015
NOTICE

This report was prepared as an account of work sponsored by an agency of the United States government. 

Neither the United States government nor any agency thereof, nor any of their employees, makes any warranty, 

express or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of 

any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately 

owned rights. Reference herein to any specific commercial product, process, or service by trade name, 

trademark, manufacturer, or otherwise does not necessarily constitute or imply its endorsement, recommendation, 

or favoring by the United States government or any agency thereof. The views and opinions of authors 

expressed herein do not necessarily state or reflect those of the United States government or any agency thereof.

This report is available at no cost from the National Renewable Energy

Laboratory (NREL) at www.nrel.gov/publications.

Available electronically at http://www.osti.gov/scitech

Available for a processing fee to U.S. Department of Energy

and its contractors, in paper, from:

U.S. Department of Energy

Office of Scientific and Technical Information

P.O. Box 62

Oak Ridge, TN 37831-0062

phone: 865.576.8401

fax: 865.576.5728

email: mailto:reports@adonis.osti.gov

Available for sale to the public, in paper, from:

U.S. Department of Commerce

National Technical Information Service

5285 Port Royal Road

Springfield, VA 22161

phone: 800.553.6847

fax: 703.605.6900

email: orders@ntis.fedworld.gov

online ordering: http://www.ntis.gov/help/ordermethods.aspx

Cover Photos: (left to right) photo by Pat Corkery, NREL 16416, photo from SunEdison, NREL 17423, photo by Pat Corkery, NREL 

16560, photo by Dennis Schroeder, NREL 17613, photo by Dean Armstrong, NREL 17436, photo by Pat Corkery, NREL 17721.

NREL prints on paper that contains recycled content.
iii

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Executive Summary

This report summarizes an assessment of improved short-term wind power forecasting in the 

California Independent System Operator (CAISO) market and provides a quantification of its 

potential value. Performed for the Lockheed Martin Corporation, the study was accomplished in 

a technology-agnostic fashion to estimate savings from regulation and flex reserves, as well as 

production savings, to provide insight into their product within the context of current and future 

CAISO markets. A simulation approach was required with a design of experiment to capture 

feasible operating points, and state-of-the-art modeling and Western Interconnection (western 

United States) data were used to estimate realistic value streams. Two major scenarios were 

considered: (1) a low wind scenario (SC4) with 8% wind penetration and (2) a high wind 

scenario (SC3) with 25% wind penetration. The design of experiments consisted of short-term 

(sub-hourly) wind power forecasting improvements of 0%, 10%, 25%, and 50% above the 

current state of the art. Results show that cost savings from flex reserves are estimated to range 

from $1.27 to $17.1 million. Cost savings from regulation reserves are estimated to range from 

$0.917 to $12.7 million. Production cost savings are estimated to range from $2.87 to $116 

million. Total cost savings from improved short-term wind power forecasting are estimated to 

range from $5.05 to $146 million. The study results led to three main points for consideration: 

(1) Economics are changed by commitments, and reliability is changed by the dispatch, so shortterm forecasting inherently fails to address the lion’s share of improvement opportunities; (2) 

The results from the study are a strong function of gas prices, which are currently low but have

an uncertain future; and (3) From the low penetration scenario, results show that cost savings are 

likely “in the noise.” What follows from the study’s outcomes is the recommendation that within 

the current CAISO market an investment in short-term wind power forecasting technologies 

would be risky; however, within the next decade or two, wind penetration levels will be 

significant enough to warrant investment in short-term wind power forecasting technologies. In 

the near term, these technologies might be better suited for reliability issues and later evolve into 

more dispatch-centric devices as wind penetration levels increase. 
iv

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Table of Contents

Introduction ................................................................................................................................................. 1

CAISO Market Design ........................................................................................................................... 2

Unit Commitment and Economic Dispatch............................................................................................ 3

CAISO Within the Western Interconnection.......................................................................................... 3

Reserve Requirements ............................................................................................................................... 7

Flexible Reserve Cost Savings............................................................................................................. 14

Regulation Reserve Cost Savings......................................................................................................... 16

Results, Impacts, and Features of Interest............................................................................................. 18

Generation by Type .............................................................................................................................. 18

Generation by Type—Impact from CAISO Improvements ................................................................. 19

Renewable Penetration ......................................................................................................................... 20

Renewable Penetration—Impact from CAISO Improvements ............................................................ 21

Curtailment........................................................................................................................................... 21

Capacity Started by Type ..................................................................................................................... 22

Generation per Start—Impact from CAISO Improvements................................................................. 23

Generation Emissions—Impact from CAISO Improvements.............................................................. 24

Cost Implications ...................................................................................................................................... 25

Production Costs................................................................................................................................... 25

Cycling Costs ....................................................................................................................................... 25

CAISO Savings .................................................................................................................................... 26

CAISO Production Cost Savings.......................................................................................................... 27

CAISO Total Cost Savings................................................................................................................... 27

Summary and Recommendations ........................................................................................................... 29

References................................................................................................................................................. 30

Appendices................................................................................................................................................ 31

Appendix A: Distribution of Forecasting Errors in CFE...................................................................... 31

Appendix B: Plots of Errors Versus Wind Power for Reserve Analysis in CFE ................................. 33

Appendix C: Distribution of Forecasting Errors in IID........................................................................ 35

Appendix D: Plots of Errors Versus Wind Power for Reserve Analysis in IID................................... 37

Appendix E: Distribution of Forecasting Errors in PGEVLY.............................................................. 39

Appendix F: Plots of Errors Versus Wind Power for Reserve Analysis in PGEVLY ......................... 41

Appendix G: Distribution of Forecasting Errors in SCE...................................................................... 43

Appendix H: Plots of Errors Versus Wind Power for Reserves Analysis in SCE ............................... 45

Appendix I: Distribution of Forecasting Errors in SDGE .................................................................... 47

Appendix J: Plots of Errors Versus Wind Power for Reserve Analysis in SDGE ............................... 49

Appendix K: Distribution of Forecasting Errors in SMUD ................................................................. 51

Appendix L: Plots of Errors Versus Wind Power for Reserve Analysis in SMUD ............................. 53
v

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

List of Figures

Figure 1. Spin and non-spin reserve defined by confidence intervals .......................................................... 7

Figure 2. Calculation of one error observation using the persistence method .............................................. 8

Figure 3. Reduction in the error observation with an improved forecast...................................................... 8

Figure 4. Error distributions using the persistence method and a forecast with 50% uniform improvement 

on low wind scenario (SC4) data ............................................................................................. 9

Figure 5. Error distributions using the persistence method and a forecast with 50% uniform improvement 

on high wind scenario (SC3) data .......................................................................................... 10

Figure 6. Error distributions as a function of wind power in the low wind scenario (SC4) ....................... 11

Figure 7. Error distributions as a function of wind power in the high wind scenario (SC3) ...................... 12

Figure 8. Comparing error distributions for persistence and forecasts as a function of wind power in the 

low wind scenario (SC4)........................................................................................................ 13

Figure 9. Comparing error distributions for persistence and forecasts as a function of wind power in the 

high wind scenario (SC3)....................................................................................................... 14

Figure 10. Generation by type observed in January and July for the low wind scenario (SC4)................. 18

Figure 11. Generation by type observed in January and July for the high wind scenario (SC3)................ 18

Figure 12. Impact in generation by type because of wind power forecasting improvements: average 

reduction in fossil fuel–based generation [%]........................................................................ 19

Figure 13. Renewable penetration observed in January and July for the low wind scenario (SC4)........... 20

Figure 14. Renewable penetration observed in January and July for the high wind scenario (SC3).......... 20

Figure 15. Impact on renewable penetration due to wind power forecasting improvements: average 

increase in renewable generation [%] .................................................................................... 21

Figure 16. Curtailment observed in January and July for the low wind scenario (SC4)............................. 21

Figure 17. Curtailment observed in January and July for the high wind scenario (SC3) ........................... 22

Figure 18. Capacity started by type observed in January and July for the low wind scenario (SC4)......... 22

Figure 19. Capacity started by type observed in January and July for the high wind scenario (SC3)........ 23

Figure 20. Impact on generation per start as a result of wind power forecasting improvements: average 

reduction in generation per start [%]...................................................................................... 23

Figure 21. Impact on generation emissions because of of wind power forecasting improvements: average 

reduction in emissions [%]..................................................................................................... 24

Figure 22. Lower- and upper-bound production costs [billions of dollars] for the low wind [SC4] and high 

wind [SC3] scenarios............................................................................................................. 25

Figure 23. Lower- and upper-bound cycling costs [billions of dollars] for the low wind [SC4] and high 

wind [SC3] scenarios............................................................................................................. 26

Figure 24. CAISO savings over state-of-the-art forecasting method using improvement assumptions..... 26

Figure A-1. Distribution of forecasting errors for the low wind (SC4) and high wind (SC3) scenarios: 0% 

improvement over state-of-the-art forecasting in CFE .......................................................... 31

Figure A-2. Distribution of forecasting errors for the low wind (SC4) and high wind (SC3) scenarios: 

10% improvement over state-of-the-art forecasting in CFE .................................................. 31

Figure A-3. Distribution of forecasting errors for the low wind (SC4) and high wind (SC3) scenarios: 

25% improvement over state-of-the art forecasting in CFE .................................................. 31

Figure A-4. Distribution of forecasting errors for the low wind (SC4) and high wind (SC3) scenarios: 

50% improvement over state of the art forecasting in CFE................................................... 32

Figure B-1. Forecasting errors as a function of wind power for the low wind (SC4) and high wind (SC3) 

scenarios: 0% improvement over state-of-the-art forecasting in CFE ................................... 33

Figure B-2. Forecasting errors as a function of wind power for the low wind (SC4) and high wind (SC3) 

scenarios: 10% improvement over state-of-the-art forecasting in CFE ................................. 33

Figure B-3. Forecasting errors as a function of wind power for the low wind (SC4) and high wind (SC3) 

scenarios: 25% improvement over state-of-the-art forecasting in CFE ................................. 34
vi

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Figure B-4. Forecasting errors as a function of wind power for the low wind (SC4) and high wind (SC3) 

scenarios: 50% improvement over state-of-the-art forecasting in CFE ................................. 34

Figure C-1. Distribution of forecasting errors for the low wind (SC4) and high wind (SC3) scenarios: 0% 

improvement over state-of-the-art forecasting in IID ............................................................ 35

Figure C-2. Distribution of forecasting errors for the low wind (SC4) and high wind (SC3) scenarios: 10% 

improvement over state-of-the-art forecasting in IID ............................................................ 35

Figure C-3. Distribution of forecasting errors for the low wind (SC4) and high wind (SC3) scenarios: 25% 

improvement over state-of-the-art forecasting in IID ............................................................ 35

Figure C-4. Distribution of forecasting errors for the low wind (SC4) and high wind (SC3) scenarios: 50% 

improvement over state-of-the-art forecasting in IID ............................................................ 36

Figure D-1. Forecasting errors as a function of wind power for the low wind (SC4) and high wind (SC3) 

scenarios: 0% improvement over state-of-the-art forecasting in IID..................................... 37

Figure D-2. Forecasting errors as a function of wind power for the low wind (SC4) and high wind (SC3) 

scenarios: 10% improvement over state-of-the-art forecasting in IID................................... 37

Figure D-3. Forecasting errors as a function of wind power for the low wind (SC4) and high wind (SC3) 

scenarios: 25% improvement over state-of-the-art forecasting in IID................................... 38

Figure D-4. Forecasting errors as a function of wind power for the low wind (SC4) and high wind (SC3) 

scenarios: 50% improvement over state-of-the-art forecasting in IID................................... 38

Figure E-1. Distribution of forecasting errors for the low wind (SC4) and high wind (SC3) scenarios: 0% 

improvement over state-of-the-art forecasting in PGEVLY.................................................. 39

Figure E-2. Distribution of forecasting errors for the low wind (SC4) and high wind (SC3) scenarios: 10% 

improvement over state-of-the-art forecasting in PGEVLY.................................................. 39

Figure E-3. Distribution of forecasting errors for the low wind (SC4) and high wind (SC3) scenarios: 25% 

improvement over state-of-the-art forecasting in PGEVLY.................................................. 39

Figure E-4. Distribution of forecasting errors for the low wind (SC4) and high wind (SC3) scenarios: 50% 

improvement over state-of-the-art forecasting in PGEVLY.................................................. 40

Figure F-1. Forecasting errors as a function of wind power for the low wind (SC4) and high wind (SC3) 

scenarios: 0% improvement over state-of-the-art forecasting in PGEVLY........................... 41

Figure F-2. Forecasting errors as a function of wind power for the low wind (SC4) and high wind (SC3) 

scenarios: 10% improvement over state-of-the-art forecasting in PGEVLY......................... 41

Figure F-3. Forecasting errors as a function of wind power for the low wind (SC4) and high wind (SC3) 

scenarios: 25% improvement over state-of-the-art forecasting in PGEVLY......................... 42

Figure F-4. Forecasting errors as a function of wind power for the low wind (SC4) and high wind (SC3) 

scenarios: 50% improvement over state-of-the-art forecasting in PGEVLY......................... 42

Figure G-1. Distribution of forecasting errors for the low wind (SC4) and high wind (SC3) scenarios: 0% 

improvement over state-of-the-art forecasting in SCE .......................................................... 43

Figure G-2. Distribution of forecasting errors for the low wind (SC4) and high wind (SC3) scenarios: 

10% improvement over state-of-the-art forecasting in SCE .................................................. 43

Figure G-3. Distribution of forecasting errors for the low wind (SC4) and high wind (SC3) scenarios: 

25% improvement over state-of-the-art forecasting in SCE .................................................. 43

Figure G-4. Distribution of forecasting errors for the low wind (SC4) and high wind (SC3) scenarios: 

50% improvement over state-of-the-art forecasting in SCE .................................................. 44

Figure H-1. Forecasting errors as a function of wind power for the low wind (SC4) and high wind (SC3) 

scenarios: 50% improvement over state-of-the-art forecasting in SCE ................................. 45

Figure H-2. Forecasting errors as a function of wind power for the low wind (SC4) and high wind (SC3) 

scenarios: 10% improvement over state-of-the-art forecasting in SCE ................................. 45

Figure H-3. Forecasting errors as a function of wind power for the low wind (SC4) and high wind (SC3) 

scenarios: 25% improvement over state-of-the-art forecasting in SCE ................................. 46

Figure H-4. Forecasting errors as a function of wind power for the low wind (SC4) and high wind (SC3) 

scenarios: 50% improvement over state-of-the-art forecasting in SCE ................................. 46
vii

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Figure I-1. Distribution of errors for the low wind (SC4) and high wind (SC3) scenarios: 0% 

improvement over state-of-the-art forecasting in SDGE ....................................................... 47

Figure I-2. Distribution of forecasting errors for the low wind (SC4) and high wind (SC3) scenarios: 10% 

improvement over state-of-the-art forecasting in SDGE ....................................................... 47

Figure I-3. Distribution of forecasting errors for the low wind (SC4) and high wind (SC3) scenarios: 25% 

improvement over state-of-the-art forecasting in SDGE ....................................................... 47

Figure I-4. Distribution of forecasting errors for the low wind (SC4) and high wind (SC3) scenarios: 50% 

improvement over state-of-the-art forecasting in SDGE ....................................................... 48

Figure J-1. Forecasting errors as a function of wind power for the low wind (SC4) and high wind (SC3) 

scenarios: 0% improvement over state-of-the-art forecasting in SDGE ................................ 49

Figure J-2. Forecasting errors as a function of wind power for the low wind (SC4) and high wind (SC3) 

scenarios: 10% improvement over state-of-the-art forecasting in SDGE .............................. 49

Figure J-3. Forecasting errors as a function of wind power for the low wind (SC4) and high wind (SC3) 

scenarios: 25% improvement over state-of-the-art forecasting in SDGE .............................. 50

Figure J-4. Forecasting errors as a function of wind power for the low wind (SC4) and high wind (SC3) 

scenarios: 50% improvement over state-of-the-art forecasting in SDGE .............................. 50

Figure K-1. Distribution of forecasting errors for the low wind (SC4) and high wind (SC3) scenarios: 0% 

improvement over state-of-the-art forecasting in SUMD ...................................................... 51

Figure K-2. Distribution of forecasting errors for the low wind (SC4) and high wind (SC3) scenarios: 

10% improvement over state-of-the-art forecasting in SMUD.............................................. 51

Figure K-3. Distribution of forecasting errors for the low wind (SC4) and high wind (SC3) scenarios: 

25% improvement over state-of-the-art forecasting in SMUD.............................................. 51

Figure K-4. Distribution of forecasting errors for the low wind (SC4) and high wind (SC3) scenarios: 

50% improvement over state-of-the-art forecasting in SMUD.............................................. 52

Figure L-1. Forecasting errors as a function of wind power for the low wind (SC4) and high wind (SC3) 

scenarios: 0% improvement over state-of-the-art forecasting in SMUD............................... 53

Figure L-2. Forecasting errors as a function of wind power for the low wind (SC4) and high wind (SC3) 

scenarios: 10% improvement over state-of-the-art forecasting in SMUD............................. 53

Figure L-3. Forecasting errors as a function of wind power for the low wind (SC4) and high wind (SC3) 

scenarios: 25% improvement over state-of-the-art forecasting in SMUD............................. 53

Figure L-4. Forecasting errors as a function of wind power for the low wind (SC4) and high wind (SC3) 

scenarios: 50% improvement over state-of-the-art forecasting in SMUD............................. 54

List of Tables

Table 1. Scenarios for the Study ................................................................................................................... 1

Table 2. Utilities and Mean Loads (MW)..................................................................................................... 4

Table 3. Low Wind (TEPPC) Scenario......................................................................................................... 5

Table 4. High Wind (Future) Scenario.......................................................................................................... 6

Table 5. Numerical Comparison of Persistence Method and Forecasts...................................................... 15

Table 6. Cost Savings from Flex Reserve................................................................................................... 16

Table 7. Cost Savings from Regulation Reserve ........................................................................................ 17

Table 8. Cost Savings from Production Savings as a Result of Improved Wind Power Forecasting......... 27

Table 9. Total Cost Savings from Flex and Regulation Reserves, as well as Production Savings, From 

Improved Wind Power Forecasting ....................................................................................... 28
1

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Introduction

A modeling and simulation study was conducted to quantify the potential value of Lockheed 

Martin’s product in a technologic-agnostic fashion. The study relied on data-driven analyses of 

short-term wind power forecasting in the California Independent System Operator (CAISO) 

market. It is important to note that CAISO in this study includes some additional small California 

balancing authority areas, which were considered as part of the larger CAISO system because of 

their frequent and strong interactions. Within a design-of-experiments approach, short-term (less 

than one-hour) improvements were considered in wind power forecasting to evaluate the gridwide impact of such improvements and to estimate savings in terms of both costs and emissions. 

Unit commitment and economic dispatch models were simulated using the PLEXOS tool as part 

of a scenario-based approach. Both current and future scenarios of the CAISO market were 

considered as shown in Table 1 and further detailed below. The design of experiments and 

analysis were conducted within the context of two primary scenarios: the current and bestestimate future CAISO market with corresponding wind penetration levels. To ensure that 

system-wide impacts were captured and realistic dispatch patterns were observed in response to 

variable and uncertain wind power production, the complete Western Interconnection was

modeled and simulated with a focus on the CAISO market.

Table 1. Scenarios for the Study

Scenario Current 

Forecasts

Perfect 

Forecasts

Uniform Forecast 

Improvements

(40-Minute Interval)

TEPPCa Case – Low 

Wind

(8% Wind, SC4)

WWSIS-2 40-minute 

interval

10%, 25%, 50%

Future Case – High 

Wind

(25% Wind, SC3)

WWSIS-2 40-minute 

interval

10%, 25%, 50%

a Transmission Expansion Planning Policy Committee of the Western Electricity Coordinating Council (WECC)

This study was based on the Western Wind and Solar Integration Study Phase 2 (WWSIS-2) [1].

However, significant extensions were applied to this Western Interconnection model to capture 

current CAISO market realities and a best-estimate future CAISO market according to evolving 

policy. The forecasting improvements shown in the table were assumed to occur only in the 

CAISO market, and it was not possible to decouple this analysis from the rest of the western 

United States, so the remaining markets utilized the numerical weather prediction forecasts from 

WWSIS-2. Within the CAISO market, assumptions on market design and wind forecasting 

implementation were extracted from policy documents [2-5]. As implemented in the PLEXOS 

tool, the following sections give a brief overview of the CAISO market design, assumptions 

required to solve the unit commitment and economic dispatch problem, and the importance of 

the CAISO market relative to the Western Interconnection as a whole.
2

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

CAISO Market Design

Toward the goals established for Task 1b by Lockheed Martin and the project team, the 

assessment of short-term wind power forecasting improvements within the CAISO market

required four time intervals for modeling the unit commitment and economic dispatch process. 

The following time intervals were used in the PLEXOS simulations:

• Day-ahead

• 4-hour-ahead

o Representative of rolling reliability unit commitment

• 15-minute market

o Where forecasting improvements were made

o Forecasts were of 40-minute persistence to represent the CAISO market 

• 5-minute “real-time” dispatch

o No forecasts; actual values were used 

o Deviations were handled by regulation reserve

A previous portion of this study (Task 1a) focused on forecasting improvements in the first two 

time intervals and quantified their value in terms of the reduced reserves that must be held over 

these longer, sequential time frames. As discussed in this report, Task 1b required the first two 

time intervals, but the forecasting improvements were made only in the 15-minute market. In 

addition to the unit commitment and economic dispatch within the PLEXOS tool, this required a 

completely new analysis of reserves, as detailed in a later section. The cost and emissions 

savings from improved forecasts were separately evaluated in terms of regulation and flexibility 

reserves, as well as production savings. Benefits from forecasting improvements are attributed to 

the CAISO market generically, and this work does not seek to provide guidelines and 

recommendations on how those benefits can be extracted. 

The two scenarios shown in Table 1, and the design of experiments contained within them, 

provided an adequate study of the value of improved short-term wind power forecasting through 

the examination of feasible ranges of forecast improvements via the technology according to 

Lockheed Martin staff. Nevertheless, the forecasting improvements could be realized in a 

technology-agnostic fashion because of their numerical nature. The improvements were aligned 

with the timescale of interest for the technology and examined as such, in a uniform convention 

at all points in time, without the consideration of ramp forecasting magnitude or timing 

improvements. That is, all analyses herein were a function of the validity of the improvements 

examined in a numerical setting, and no physical modeling of the forecasting process was 

considered, only the impact of such numerical realizations of such forecasts. 

Generating the numerical, uniform forecasting improvement was accomplished by examining the 

forecast error and decreasing this error by different percentages according to the design of 

experiments. This required scripting programming routines to query the WWSIS-2 database, 

locate the specific bus of interest in the CAISO region, and perform the forecasting improvement

calculations. The “actuals” are the realizations of sampled events at 5-minute intervals, and the 
3

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

forecasts are also the realizations of sampled events at 5-minute intervals but inherently 

erroneous because of imperfect knowledge of the physical processes. The original forecasts came 

from numerical weather prediction routines as detailed in the WWSIS-2 final report, and the 

forecasts utilized in this study were improved according to Table 1.

Unit Commitment and Economic Dispatch

PLEXOS is an electricity market modeling and power system simulation tool. The modeling 

utilized in this study made use of its deterministic linear programming methods to minimize an 

objective function. In this case, the objective function was the expected cost of electricity 

dispatch while numerous constraints were observed—for example, the operational, part-load 

performance of generating plants and their availability, fuel costs, renewable power availability, 

transmission constraints, etc.

This study made use of the CAISO market established in the previous section for a full year’s 

time. During this year-long period, the PLEXOS model accounted for every period of energy 

trading and maintained chronological order throughout the optimization horizon. Optimization 

decisions were based on each generator’s start-ups and shutdowns while observing the 

parameters governing individual plants’ operating characteristics, such as minimum generation 

and ramping up/down times. Tracking all of these technical details requires the mathematical 

sophistication of mixed-integer programming to solve not only binary decisions—i.e., whether 

the generator is on or off—but the fractional operating points to ensure that supply meets 

demand. Such an approach allows for a realistic model of the actual, physical operation of 

generators in a power market.

Because this study uses the WWSIS-2 as a basis, numerous assumptions were required to 

adequately model the Western Interconnection. For brevity, the reader is referred to the WWSIS2 final report [1] for all assumptions, in which all methodologies and analyses were overseen by 

a technical review committee for credibility and realism. Only global, fundamental assumptions 

that were carried over into this study are stated below:

• An average gas price of $4.60/MMBtu was used.

• Significant balancing authority area cooperation was inherent to the analysis.

• Least-cost economic dispatch and transmission usage with bilateral transactions were not 

explicitly modeled.

Because the primary concern of this study was the improvement in short-term wind power 

forecasting in CAISO, variability could be expected not only in the wind as a power source 

during seasonal, daily, and hourly time frames, but of particular interest were sub-hourly 

variations and how reductions in these errors could lead to costs and emissions savings. Only 

through modeling the Western Interconnection and PLEXOS simulations or similar mixedinteger programming could this be quantified. This was the approach undertaken in this study.

CAISO Within the Western Interconnection 

The CAISO market was chosen as a test bed for quantifying the value of short-term wind power 

forecasts for numerous reasons: (1) It has a high-quality resource for wind power; (2) It has the 

distinction of being the most developed market in the Western Interconnection; (3) California 
4

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

has one of the highest renewable energy portfolio standards in the United States; (4) California is 

expected to add significant amounts of new wind power capacity by 2020; and (5) Vast amounts

of data are available for studying the CAISO system. 

From the WWSIS-2 study, a data set of simulated wind power for a large number of plants is 

available with time-synchronized load data. The data are for the accurate value assessment of 

wind power forecasting in power system operations, and, further, creating one for new regions is 

a costly and time-consuming process [8].

The CAISO control area consists of three primary utility companies: Pacific Gas & Electric, San 

Diego Gas & Electric, and Southern California Edison. Because it is likely that wind power 

generated outside the CAISO control area is used within the CAISO control area, load and wind 

power from surrounding utility areas were also included in this study. Table 2 shows the utilities 

and their mean loads based on the WWSIS-2.

Table 2. Utilities and Mean Loads (MW)

Utility Utility Name

Mean 

Load 

(MW)

CFE Comisión Federal de Electricidad 1,965

IID Imperial Irrigation District 523

LDWP

Los Angeles Department of Water and 

Power 3,358

PG&E_BAY Pacific Gas & Electric (Bay Area) 5,258

PG&E_VLY Pacific Gas & Electric (Central Valley) 7,529

SCE Southern California Edison 13,082

SDGE San Diego Gas & Electric 2,643

SMUD Sacramento Municipal Utility District 2,117

Total 36,474

Details on the Western Interconnection are provided in Table 3 and Table 4 for the low wind 

(WECC TEPPC) and high wind (Future) scenarios. These tables illustrate the diversity of wind 

power generation and its spatial distribution. Of course, the temporal variability in the wind 

resource will determine its instantaneous (renewable) penetration level and the extent to which 

the forecasting improvements influence the dispatch of generation.
5

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Table 3. Low Wind (TEPPC) Scenario

Rooftop PV Utility-Scale PV CSP Wind Total

State Capacity

(MW)

CF Capacity 

(MW)

CF Capacity 

(MW)

CF Capacity 

(MW)

CF Capacity 

(MW)

CF

Arizona 1,171 22% 472 43% 3,681 30% 5,324 30%

California 3,545 25% 3,221 44% 7,299 30% 14,065 32%

Colorado 1,342 20% 169 37% 3,256 29% 4,767 27%

Idaho 523 27% 523 27%

Montana 838 34% 838 34%

Nevada 304 22% 334 42% 150 25% 788 31%

New Mexico 140 27% 156 39% 494 28% 790 30%

Oregon 4,903 26% 4,903 26%

South Dakota

Texas

Utah 571 20% 323 31% 894 24%

Washington 4,652 27% 4,652 27%

Wyoming 1,784 42% 1,784 42%

Total 7,074 23% 4,352 43% 27,900 29% 39,326 30%
6

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Table 4. High Wind (Future) Scenario

Rooftop PV Utility-Scale PV CSP Wind Total

State Capacity

(MW)

CF Capacity 

(MW)

CF Capacity 

(MW)

CF Capacity 

(MW)

CF Capacity 

(MW)

CF

Arizona 1,975 19% 2,330 25% 3,303 43% 4,941 30% 12,548 31%

California 4,875 18% 5,372 25% 2,469 45% 11,109 30% 23,824 28%

Colorado 1,059 18% 1,128 22% 169 37% 6,226 35% 8,581 31%

Idaho 3 15% 2 16% 1,333 29% 1,338 29%

Montana 22 15% 34 17% 6,658 36% 6,714 36%

Nevada 398 19% 344 22% 439 42% 3,270 31% 4,452 30%

New Mexico 172 20% 209 27% 156 39% 4,784 38% 5,321 37%

Oregon 91 14% 101 22% 5,473 26% 5,665 26%

South Dakota 4 17% 6 19% 2,640 36% 2,650 36%

Texas 76 20% 122 27% 198 24%

Utah 361 17% 489 21% 1,343 32% 2,193 27%

Washington 371 13% 492 20% 5,882 27% 6,745 26%

Wyoming 9 18% 18 21% 10,184 43% 10,211 43%

Total 9,417 18% 10,647 24% 6,536 43% 63,840 34% 90,439 32%
7

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Reserve Requirements

Although Task 1a focused on the anticipated cost savings from improved short-term wind power

forecasting, it did so for the 1-hour and 4-hour-ahead markets in CAISO. Representatives from 

Lockheed Martin determined that their product was more applicable to the sub-hourly time frame 

and thus a new reserve requirement analysis had to be conducted. Even in the sub-hourly CAISO 

market, the reduction in the amount of reserves that must be carried to accommodate the 

uncertainty of wind power output was anticipated to be one of the largest cost savings. An 

advanced reserve calculation algorithm was thus applied to estimate the reserve reductions that 

various wind power forecasting improvements would allow; this methodology was originally 

developed during the WWSIS-2 project. Improved forecasting (on average) reduces the amount 

of reserves that must be held, and the various types of flexibility reserves are defined by: 

• Spin reserve = 70% confidence interval

• Non-spin reserve = 95% confidence interval

and illustrated in Figure 1. 

Figure 1. Spin and non-spin reserve defined by confidence intervals

Base case reserves are calculated using a persistence method. Both the spin and non-spin 

reserves are a function of the distribution of forecast errors. During a given forecast interval Δt, 

one error observation is power measured at t + Δt minus power measured at t, as shown in Figure 

2, adapted from [6].

SPIN NONSPIN

NONSPIN
8

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Figure 2. Calculation of one error observation using the persistence method

Using the persistence approach, the forecast at ti is Pi-1 and error is (Pi – Pi-1). The two types of 

reserves can be calculated for different time intervals, or requirements such as percent of time 

needed to meet the unanticipated change in power. Figure 3 illustrates how an improved forecast 

can potentially reduce the error, which in turn would reduce the amount of reserves required. In 

this case, the forecast at ti replaces Pi-1 in the error equation. For an overall reduction in reserves 

to be realized, the forecast must be consistently closer to Pi than to Pi-1 for most time periods.

Figure 3. Reduction in the error observation with an improved forecast

An overall evaluation of each forecast in the design of experiments was obtained by calculating 

errors using the persistence method for the complete year under investigation and then using the 

Error

Error
9

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

forecast method for the same year. Figure 4 illustrates the persistence distribution for the low 

wind scenario (SC4) compared to the distribution using the sub-hourly forecast with 50% 

uniform improvement. Figure 5 does the same for the high wind scenario (SC3). The distribution

did not appear to follow normality in either of these cases. Appendices provided include all 

distribution of errors for each region of interest and all design-of-experiment variables 

considered.

Figure 4. Error distributions using the persistence method and a forecast with 50% uniform 

improvement on low wind scenario (SC4) data
10

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Figure 5. Error distributions using the persistence method and a forecast with 50% uniform 

improvement on high wind scenario (SC3) data

In both cases, there appeared to be a narrower distribution when the sub-hourly forecasts with 

50% uniform improvement were used to determine errors. A differentiation needed be made 

between the positive and negative errors, i.e., an “over-forecast” or “under-forecast” for the type 

of reserve being evaluated. This was accomplished by using a confidence interval for the specific 

type of reserve and extracting the lower percentile of the distribution (under-forecast) and an

upper percentile of the distribution (over-forecast). Over-forecast minus under-forecast gives the 

confidence interval range. A percent improvement could then be estimated by comparing the 

confidence interval range determined using the forecast to the confidence interval range 

determined using the persistence method.
11

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Also of interest in evaluating the reserves is that on average the error varied with turbine power, 

as shown in Figure 6 and Figure 7 with the 70th percentile range (under-forecast = 15th

percentile, over-forecast = 85th percentile) and 99th percentile range (under-forecast = 0.5th

percentile, over-forecast = 99.5th percentile) for ramps calculated using the persistence method 

on low wind (SC4) and high wind (SC3) data, respectively.

Figure 6. Error distributions as a function of wind power in the low wind scenario (SC4)
12

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Figure 7. Error distributions as a function of wind power in the high wind scenario (SC3)

As observed in the figures, the distribution in error values was narrower at low power conditions, 

wider at midrange power conditions, and narrower again at high power conditions. In general, 

this pattern was observed for almost all wind turbine sites. The figures also demonstrate the 

sizeable difference in a moderate range (70th percentile) versus the range needed to achieve high 

certainty (99th percentile).
13

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Comparing the errors among persistence and forecast allows one to understand how the accuracy 

of the forecasts changes as a function of wind power, as shown in Figure 8 and Figure 9 for the 

low wind and high wind scenarios, respectively. Again, as shown in the figures, the distribution 

in ramp values was narrower at low power conditions, wider at midrange power conditions, and 

narrower again at high power conditions. Appendices provided include all error (ramp) versus 

wind power plots for each region of interest and all design-of-experiment variables considered. 

Figure 8. Comparing error distributions for persistence and forecasts as a function of wind power 

in the low wind scenario (SC4)
14

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Figure 9. Comparing error distributions for persistence and forecasts as a function of wind power 

in the high wind scenario (SC3)

Flexible Reserve Cost Savings

To evaluate flexible reserve, the 70% confidence interval was utilized: an under-forecasting error 

was determined by evaluating the 15th percentile of the error values, and an over-forecasting 

error was determined by evaluating the 85th percentile of the error values. The sub-hourly time 

interval was used to determine flexible reserve. A visual comparison of the persistence method 

and forecasts is shown in Figure 8 and Figure 9, and Table 5 shows a numerical comparison. 
15

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Table 5. Numerical Comparison of Persistence Method and Forecasts

Persistence Forecast

Wind 

Scenario

Improvement 

Description

UpRampDownRampMean 

CI diff.

UpRampDownRampMeanCI 

diff.

Percent 

Improvement

Low

10% uniform 

improvement 225 -177 402 203 -159 362 10%

High

10% uniform 

improvement 296 -246 542 268 -220 488 10%

Low

25% uniform 

improvement 225 -177 402 169 -133 302 25%

High

25% uniform 

improvement 296 -246 542 223 -183 406 25%

Low

50% uniform 

improvement 225 -177 402 113 -88 201 50%

High

50% uniform 

improvement 296 -246 542 149 -122 271 50%

The holding of flexibility reserve is determined by economics, and the 70% and 95% confidence 

intervals were based upon economic realities of power system operations. The Eastern Wind 

Integration and Transmission Study [7] reserve methodology has been accepted in the wind 

integration community as a reasonable approach to determining reserve levels necessary under 

high wind power penetration scenarios. A similar methodology was adopted in the WWSIS-2

and used in this study. Carrying spin reserve is expensive, because this requires keeping 

generating units online throughout the 8,760 hours of a year, even if they are rarely used. As 

such, the methodology used utilizes spin reserve to cover only the 70% confidence interval of 

variability at this timescale. These are the levels of reserve that will be utilized more often, and 

thus are cost-effective to be held at all times. Non-spin reserve is held for more rare events, up to 

the 95% confidence interval. This type of reserve is slower to act (taking up to 10 minutes) but is 

much more cost-effective. This combination of reserve types helps to cover the vast majority of 

the additional variability seen from increased wind power penetration while doing so in a costeffective manner. The 70% and 95% figures approximately represent the confidence intervals 
16

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

that would be covered by one and two standard deviations of variability, if the distributions were 

to follow a normal distribution.

Using the methodology described above and the equations detailed below, the cost savings from 

flex reserve were calculated for the design of experiments. As shown in Table 6, the costs 

savings trend with forecasting improvements, and the technology could be helpful for cost 

savings on the order of millions of dollars in terms of flexible reserve. 

C

flex 

= H

year

x (cspinx R

flex

+ cnon-spin x R

95%)

where 

C

flex 

= total annual cost of flexible reserve

H

year

= hours in one year 

c

spin = cost in $/MWh of spin reserve

R

flex = MW range for flexible reserve determined at the 70% confidence interval

c

non-spin = cost in $/MWh of non-spin reserve

R

95% = MW range outside of the 70% confidence interval

but within the 95% confidence interval.

Table 6. Cost Savings from Flex Reserve

Wind 

Scenario Improvement Description

Persistence 

Method Cost 

Per Year

Forecast 

Method Cost 

Per Year

Annual 

Savings

Low 10% uniform improvement $11,900,000 $10,600,000 $1,270,000 

High 10% uniform improvement $34,600,000 $30,900,000 $3,690,000 

Low 25% uniform improvement $11,900,000 $8,900,000 $2,940,000 

High 25% uniform improvement $34,600,000 $26,000,000 $8,530,000 

Low 50% uniform improvement $11,900,000 $5,950,000 $5,900,000 

High 50% uniform improvement $34,600,000 $17,400,000 $17,100,000 

Regulation Reserve Cost Savings

For regulation reserve, the forecast error for a 10-minute moving window was calculated and a 

95% confidence interval was selected—more stringent than the flexible reserve. This leads to 

increased certainty that a large deviation in energy demand can be accommodated. The 95% 
17

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

confidence interval was chosen with consideration given to the North American Electric 

Reliability Corporation (NERC) area control performance standards (CPS). The CPS2 standard 

states that the 10-minute average area control error has to remain within a proscribed range 90% 

of the time. Because regulation reserve acts at this timescale, being able to cover 95% of the 

required regulation service provides assurance that the utility will fulfill their CPS2 obligations. 

Using the methodology described above and the equations detailed below, the cost savings from 

regulation reserve were calculated for the design of experiments. As shown in Table 7, the costs 

savings were slightly less than those for flexible reserve and trend with forecasting 

improvements.

C

reg 

= H

year

x (c

up

x R

up

+ c

down 

x R

down)

where

C

reg 

= total annual cost of regulation reserves

H

year

= hours in one year 

c

up

= cost in $/MWh of up-ramp reserves

R

up

= MW range for regulation up-ramp reserves

c

down 

= cost in $/MWh of down-ramp reserves

R

down = MW range for regulation down-ramp reserves.

Table 7. Cost Savings from Regulation Reserve

Wind 

Scenario Improvement Description

Persistence 

Method Cost 

Per Year

Forecast 

Method Cost 

Per Year

Annual 

Savings

Low 10% uniform improvement $8,660,000 $7,750,000 $917,000 

High 10% uniform improvement $25,400,000 $22,900,000 $2,460,000 

Low 25% uniform improvement $8,660,000 $6,490,000 $2,170,000 

High 25% uniform improvement $25,400,000 $19,100,000 $6,310,000 

Low 50% uniform improvement $8,660,000 $4,330,000 $4,330,000 

High 50% uniform improvement $25,400,000 $12,700,000 $12,700,000 
18

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Results, Impacts, and Features of Interest

The set of results from the PLEXOS production simulations included a tremendous amount of 

detail on all aspects of the power system model, including generator commitment and dispatch, 

production costs, emissions, and transmission path flows for each 5-minute time step. Not 

included were fixed capital costs and power purchase agreements, which are not within the scope 

of this work. The power flow modeled in PLEXOS is an optimal direct current power flow that 

respects transmission constraints while using power transfer distribution factors. The following 

sections detail high-level outcomes that dictate the cost and emissions savings observed from the 

improved wind power forecasting. 

Generation by Type

The generation by type will vary on a monthly basis based on the load, the availability of 

renewable resources, and the fossil fuel–based generation that must make up the difference. Of 

course, the dispatch stack is a function of price sensitivities, and slight trade-offs will occur 

between all of these factors throughout a year. Figure 10 and Figure 11 show the generation by 

type for all generators used in the Western Interconnection study for the months of January and 

July. These months were chosen to illustrate the extremes in differences throughout a year: on 

average, the month of January corresponds to maximum wind availability and minimum solar 

available, and the month of June is the opposite. As shown in the figures, most changes in 

production proportions were on the order of a few percentage points; however, even these slight 

differences constitute enormous value potentials because of the proportions of the Western 

Interconnection. Following sections further examine why and analyze details.

Figure 10. Generation by type observed in January and July for the low wind scenario (SC4)

Figure 11. Generation by type observed in January and July for the high wind scenario (SC3).
19

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Generation by Type—Impact from CAISO Improvements

Improved wind power forecasting in the CAISO led to an increased penetration of wind power 

because the cost of utilizing this additional amount of energy is practically zero and generators 

on the margin will be significantly more expensive. Any reduction in forecasting error is a 

reduction in uncertainty, and more certainty leads to increased reliance on renewable power. 

Figure 12 shows the average reduction in fossil fuel–based generation as a function of improved 

wind power forecasting for both low (SC4) and high (SC3) wind power scenarios. 

Figure 12. Impact in generation by type because of wind power forecasting improvements: 

average reduction in fossil fuel–based generation [%]

A few points should be noted when observing the figure and considering the relative changes in 

generation by type for the various levels of improved wind power forecasting within each of the 

scenarios: 

• An average gas price of $4.60/MMBtu was used in accordance with best estimates 

detailed in the WWSIS-2 methodology.

• Increasing levels of wind and solar mostly displaced gas generation, while enhanced 

forecasting also reduced coal ramping.

• Day-ahead unit commitment was not impacted by short-term forecasting improvements, 

so savings were mostly from combustion turbine dispatch reductions.

These observations about the impact of short-term forecasting can be observed in the figure, and 

most changes were seen in fast-acting gas generators while minor generation displacement was 

felt elsewhere. The results are indicative of intuition, but their relative cost and emissions impact 

are assessed further with more focused analysis in later sections.

0

2

4

6

8

10

12

14

16

18

Coal Gas CC Gas CT

Average Reduction in 

Fossil Fuel Based Generation [%]

LowWind 10%

LowWind 25%

LowWind 50%

HiWind 10%

HiWind 25%

HiWind 50%
20

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Renewable Penetration

With more accurate short-term wind power forecasting, the relative penetration of wind 

generation can be enhanced. In addition, for high levels of wind penetration, there will be 

reductions in the amount of curtailment required. Figure 13 and Figure 14 illustrate the daily 

generation of renewable power for the months of January and July. Again, these months were 

chosen to represent extremes in differences through a year (on average) and are indicative of the 

results obtained for the production cost simulation for improved wind power forecasting in both 

the low and high wind power scenarios.

Figure 13. Renewable penetration observed in January and July for the low wind scenario (SC4)

Figure 14. Renewable penetration observed in January and July for the high wind scenario (SC3)

As shown in the figures, only slight curtailment occurred during the windy month of January for 

the high wind power scenario. On average, the wind penetration was slightly enhanced, but this 

was mostly in the noise for the low wind power scenario. In all cases, solar power was not 

curtailed. This was because of the relatively low penetration of solar in both scenarios. 

The figures and analysis showcase only slight increases in the penetration of renewables because 

the short-term wind power forecasts are sub-hourly and most fossil fuel based generation is 

already committed and the enhanced accuracy of the forecasts only slightly reduces the amount 

of fossil fuel based generation. As the forecasting accuracy improves on longer time horizons, 

one would expect the level of renewable penetration would increase more significantly because 

fossil fuel based generation would not be committed in the first place.
21

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Renewable Penetration—Impact from CAISO Improvements

With respect to the CAISO market, the impacts of enhanced wind power forecasting led to 

average increases in the percentage of energy served by renewable generation on the order of 1% 

or less. Figure 15 shows an approximately exponential decay of increased renewable generation 

as short-term wind power forecasting moved away (right to left) from the maximum 

improvement considered and approached the current state of the art.

Figure 15. Impact on renewable penetration due to wind power forecasting improvements: 

average increase in renewable generation [%]

Curtailment

Curtailment was minimal for both the low and high wind penetration scenarios. However, it was 

observed when the other generators reached their minimum generation levels. Figure 16 and 

Figure 17 show the percentage of curtailment for the representative (extreme) months of January 

and July. As shown in both scenarios, the curtailment was mostly in the noise. Nevertheless, 

these curtailments are “lost-cost savings” that could be had with wind power forecasting 

improvements beyond the extent considered in this study.

Figure 16. Curtailment observed in January and July for the low wind scenario (SC4)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Wind

Average Increase in Renewable 

Generation [%]

LowWind 10%

LowWind 25%

LowWind 50%

HiWind 10%

HiWind 25%

HiWind 50%
22

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Figure 17. Curtailment observed in January and July for the high wind scenario (SC3)

Capacity Started by Type

The capacity started by type indicates the trade-offs between the improvements in wind power 

forecasting and the increased gas usage that comes with less-accurate forecasting. Figure 18 and 

Figure 19 show the capacity started by type for the months of January and July. The frequency of

the gas generator starts was enormous compared to the coal units, but that does not mean that the 

gas-unit fluctuations do not affect the commitment of the coal unit, however slight that 

dependence might be. 

Figure 18. Capacity started by type observed in January and July for the low wind scenario (SC4)
23

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Figure 19. Capacity started by type observed in January and July for the high wind scenario (SC3)

Generation per Start—Impact from CAISO Improvements

The average reduction in generation per start as a result of improved forecasting for both the low 

and high wind power scenarios is shown in Figure 20. As expected, the trade-offs between 

forecasting accuracy and combustion turbine modulation are evident. For the most aggressive 

assumption about forecasting improvement (i.e., 50% in the high wind scenario), an 8% 

reduction in generation per start for the combustion turbines was observed. For the other units 

(e.g., gas combined cycle and coal), the reductions were likely in the noise of the simulations.

Figure 20. Impact on generation per start as a result of wind power forecasting improvements: 

average reduction in generation per start [%]

0

1

2

3

4

5

6

7

8

9

Coal Gas CC Gas CT

Average Reduction in Generation per Start [%]

LowWind 10%

LowWind 25%

LowWind 50%

HiWind 10%

HiWind 25%

HiWind 50%
24

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

A couple of points can be gleaned from the analysis when coupled with other results:

• Uniform forecasting improvements slightly reduce the time and the capacity that 

combustion turbine and combined-cycle units must run to meet load.

• Savings were observed in the reduced ramping and cycling of coal units, which had a 

slight effect on CO2, NOX, and SO2 emissions.

The emissions saved in the CAISO are discussed in the following section.

Generation Emissions—Impact from CAISO Improvements

The impact of generation emissions as a result of wind power forecasting improvements is shown 

in Figure 21. The average reduction was on the order of a couple percentage points or less for CO2, 

NOX, and SO2 emissions. Although not explicitly addressed in this study, these savings could be very 

important to future markets in which the monitoring of emissions is crucial to operations (e.g., 

carbon markets).

Figure 21. Impact on generation emissions because of of wind power forecasting improvements: 

average reduction in emissions [%]

0

0.5

1

1.5

2

2.5

CO2 NOx SO2

Average Reduction in Emissions [%]

LowWind 10%

LowWind 25%

LowWind 50%

HiWind 10%

HiWind 25%

HiWind 50%
25

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Cost Implications

All of the analyses shown thus far have cost implications. The goal of this study was to quantify 

the value of improved short-term forecasting such that Lockheed Martin can assess the potential 

market value of their technology. The following sections detail the production cost savings, 

cycling costs, and implications of these savings in the CAISO market. 

Production Costs

Figure 22 shows the lower and upper bound on estimated production costs, in terms of billions of 

U.S. dollars, for the low and high wind scenarios. The results are classified according to various 

costs: fuel; start-up fuel; and start-up, ramping, and noncyclical variable operations and 

maintenance (VOM). The dominant production cost savings come from fuel requirements.

Figure 22. Lower- and upper-bound production costs [billions of dollars] for the low wind [SC4] 

and high wind [SC3] scenarios

Cycling Costs

Inaccurate wind power forecasts can lead to cycling in thermal units because of the increased 

variability and uncertainty. As one might imagine, this leads to excessive wear and tear on the 

units, and operators try to avoid such cycling. Figure 23 presents the lower and upper bound on 

cycling costs in billions of U.S. dollars for the low and high wind scenarios. Start-up VOM 

dominates the cycling cost, followed by start-up fuel and ramping VOM.

0

2

4

6

8

10

12

14

16

18

Low Wind (Lower

Bound)

Low Wind (Upper

Bound)

High Wind (Lower

Bound)

High Wind (Upper

Bound)

Production cost [billions of $]

Fuel NonCyclic VOM

Start-up VOM Start-up Fuel

Ramping VOM
26

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Figure 23. Lower- and upper-bound cycling costs [billions of dollars] for the low wind [SC4] and 

high wind [SC3] scenarios

CAISO Savings

The implication of production costs and cycling within the CAISO market are expressed in 

Figure 24. Savings of state-of-the-art wind power forecasting range from approximately nil to 

almost 5%. This can be a sizeable amount of money as quantified in the following sections. 

Figure 24. CAISO savings over state-of-the-art forecasting method using improvement 

assumptions

0

100

200

300

400

500

600

700

800

900

Low Wind (Lower

Bound)

Low Wind (Upper

Bound)

High Wind (Lower

Bound)

High Wind (Upper

Bound)

Cycling cost [millions of $]

Start-up

VOM

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

Wind Scenario

Savings over SOA Forecasting [%]

LowWind 10%

LowWind 25%

LowWind 50%

HiWind 10%

HiWind 25%

HiWind 50%
27

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

CAISO Production Cost Savings

The annual cost savings from production because of improved wind power forecasting are shown 

in Table 8. They range from slightly less than $2 million to more than $100 million. Although 

these savings are within CAISO for a one-year period, savings of the same order of magnitude 

could likely be obtained in other markets. However, it is important to note that the exact amount 

of savings in other markets would vary strongly with other factors, such as the transmission 

congestion, the generation portfolio, and the specific market procedures in place.

Table 8. Cost Savings from Production Savings as a Result of Improved Wind Power Forecasting

Wind Scenario Improvement Description SOA Forecasting Improved Forecasting Annual Savings

Low 10% uniform improvement 3,590,000,000 $3,590,000,000 $2,870,000

High 10% uniform improvement 2,540,000,000 $2,520,000,000 $18,980,000

Low 25% uniform improvement 3,590,000,000 $3,580,000,000 $9,690,000

High 25% uniform improvement 2,540,000,000 $2,490,000,000 $48,100,000

Low 50% uniform improvement 3,590,000,000 $3,560,000,000 $24,400,000

High 50% uniform improvement 2,540,000,000 $2,420,000,000 $116,000,000

CAISO Total Cost Savings

The total cost savings from improved short-term wind power forecasting in the CAISO market 

can be expressed with the simple summation below:

Total Cost Savings = Flex Reserve + Regulation Reserve + Production

Combining Table 6 and Table 7 with Table 8 leads to Table 9, the total cost savings from flex 

reserve, regulation reserve, and production cost savings from improved wind power forecasting.
28

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Table 9. Total Cost Savings from Flex and Regulation Reserves, as well as Production Savings, 

From Improved Wind Power Forecasting

Wind Scenario Improvement 

Description Annual Savings

Low

10% uniform 

improvement $5,050,000

High

10% uniform 

improvement $25,100,000

Low

25% uniform 

improvement $14,800,000

High

25% uniform 

improvement $62,900,000

Low

50% uniform 

improvement $34,700,000

High

50% uniform 

improvement $146,000,000
29

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Summary and Recommendations

This study assessed the value of improved short-term wind power forecasting in the California 

Independent System Operator (CAISO) market and provided an estimation of its potential systemwide value. Performed for the Lockheed Martin Corporation, the study was accomplished in a 

technology-agnostic fashion to estimate savings from regulation and flex reserves, as well as 

production savings, to provide insight into the potential of their product within the context of 

current and future CAISO markets. A simulation approach was required with a design of 

experiments to capture feasible operating points, and state-of-the-art modeling and Western 

Interconnection data (for the western United States) was used to produce realistic value estimates.

Although this study did not make a business case for investing in short-term wind power 

forecasting technologies, it does aid such a decision-making process with the quantification of its 

potential value. Further, to summarize the study and provide some recommendations:

• Short-term, sub-hourly forecast improvements have relatively small impacts on systemwide costs:

o Improvements in persistence may be “too little, too late” because of only slight 

deviations from commitment schedules.

o Although the WWSIS-2 model is well-vetted, savings at low penetrations of wind 

power are likely “in the noise” because of assumptions—e.g., no bilateral 

(purchase power) agreements.

o Forecasting improvements replace chiefly gas in quick-start units, therefore 

savings are a function of gas prices; gas prices are currently low but have an 

uncertain future.

• Results follow the adage “Economics are changed by commitments; reliability is changed 

by the dispatch.”

• Additional implications:

o The first study showed the dominance of ramp prediction on reserve prices, which 

the technology could be quite helpful for predicting and is currently used by 

system operators for “situational awareness.”

o The savings from the technology are a strong function of gas prices, which have 

an uncertain future as a result of energy policy.

o FESTIV simulations could quantify the reliability benefits. This NREL-created 

model goes down to the four-second timescale to quantify the benefits of 

changing operational policies on control performance standard (CPS) scores. 

These results show that although there may not be an economic case for extensive capital 

investments in improving ultra-short-term wind power forecasting at the moment, the need for 

this service will increase significantly with increased wind power penetration rates. The results 

shown follow the assumption that the last chance to improve the forecast will occur 40 minutes 

before real time in the market. With increased penetration of renewable generation, we expect to 

see corresponding changes in power system operations. This could also include shorter “gateclosing” times, which would allow even shorter-term forecasts to play a role. Moving closer to 

the operating time would presumably allow even greater increases in forecasting accuracy over 

the persistence model, and thus perhaps greater cost savings.
30

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

References

1. Lew, D.; Brinkman, G.; Ibanez, E.; Florita, A.; Heaney, M.; Hodge, B.-M.; Hummon, M.; 

Stark, G.; King, J.; Lefton, S.A.; Kumar, N.; Agan, D.; Jordan, G.; Venkataraman, S. The 

Western Wind and Solar Integration Study Phase 2. NREL/TP-5500-55888. Golden, CO: 

National Renewable Energy Laboratory, 2013. Accessed March 2014: 

http://www.nrel.gov/docs/fy13osti/55588.pdf.

2. GE. Western Wind and Solar Integration Study. Golden, CO: National Renewable Energy 

Laboratory, 2010. Accessed March 2014: http://www.nrel.gov/docs/fy10osti/47781.pdf. 

3. CAISO Control Area. 2012. Accessed March 2014: 

http://currentenergy.lbl.gov/ca/calInfo.php#caiso.

4. CAISO. CAISO Open Access Same-time Information System (OASIS). 2012. Accessed 

March 2014: http://www.oasis.caiso.com.

5. Wind Powering America Third Quarter 2012 Market Report. Golden, CO: National 

Renewable Energy Laboratory, 2012.

6. Lew, D.; Brinkman, G.; Ibanez, E. WWSIS2 Technical Review Committee June Meeting.

2012.

7. EnerNex. Eastern Wind Integration and Transmission Study. Golden, CO: National 

Renewable Energy Laboratory, 2010. Accessed March 2014: 

http://www.nrel.gov/docs/fy11osti/47078.pdf. 

8. Potter, C.; Lew, D.; McCaa, J.; Cheng, S.; Eichelberger, S.; Grimit, E. Creating the Dataset 

for the Western Wind and Solar Integration Study, Wind Engineering, Vol. 32, No. 4, 2008, 

pp 325-338
31

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Appendices

Appendix A: Distribution of Forecasting Errors in CFE

Figure A-1. Distribution of forecasting errors for the low wind (SC4) and high wind (SC3) 

scenarios: 0% improvement over state-of-the-art forecasting in CFE

Figure A-2. Distribution of forecasting errors for the low wind (SC4) and high wind (SC3) 

scenarios: 10% improvement over state-of-the-art forecasting in CFE

Figure A-3. Distribution of forecasting errors for the low wind (SC4) and high wind (SC3) 

scenarios: 25% improvement over state-of-the art forecasting in CFE
32

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Figure A-4. Distribution of forecasting errors for the low wind (SC4) and high wind (SC3) 

scenarios: 50% improvement over state of the art forecasting in CFE
33

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Appendix B: Plots of Errors Versus Wind Power for Reserve Analysis 

in CFE

Figure B-1. Forecasting errors as a function of wind power for the low wind (SC4) and high wind 

(SC3) scenarios: 0% improvement over state-of-the-art forecasting in CFE

Figure B-2. Forecasting errors as a function of wind power for the low wind (SC4) and high wind 

(SC3) scenarios: 10% improvement over state-of-the-art forecasting in CFE
34

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Figure B-3. Forecasting errors as a function of wind power for the low wind (SC4) and high wind 

(SC3) scenarios: 25% improvement over state-of-the-art forecasting in CFE

Figure B-4. Forecasting errors as a function of wind power for the low wind (SC4) and high wind 

(SC3) scenarios: 50% improvement over state-of-the-art forecasting in CFE
35

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Appendix C: Distribution of Forecasting Errors in IID

Figure C-1. Distribution of forecasting errors for the low wind (SC4) and high wind (SC3) 

scenarios: 0% improvement over state-of-the-art forecasting in IID

Figure C-2. Distribution of forecasting errors for the low wind (SC4) and high wind (SC3) 

scenarios: 10% improvement over state-of-the-art forecasting in IID

Figure C-3. Distribution of forecasting errors for the low wind (SC4) and high wind (SC3) 

scenarios: 25% improvement over state-of-the-art forecasting in IID
36

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Figure C-4. Distribution of forecasting errors for the low wind (SC4) and high wind (SC3) 

scenarios: 50% improvement over state-of-the-art forecasting in IID
37

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Appendix D: Plots of Errors Versus Wind Power for Reserve Analysis 

in IID

Figure D-1. Forecasting errors as a function of wind power for the low wind (SC4) and high wind 

(SC3) scenarios: 0% improvement over state-of-the-art forecasting in IID

Figure D-2. Forecasting errors as a function of wind power for the low wind (SC4) and high wind 

(SC3) scenarios: 10% improvement over state-of-the-art forecasting in IID
38

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Figure D-3. Forecasting errors as a function of wind power for the low wind (SC4) and high wind 

(SC3) scenarios: 25% improvement over state-of-the-art forecasting in IID

Figure D-4. Forecasting errors as a function of wind power for the low wind (SC4) and high wind 

(SC3) scenarios: 50% improvement over state-of-the-art forecasting in IID
39

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Appendix E: Distribution of Forecasting Errors in PGEVLY

Figure E-1. Distribution of forecasting errors for the low wind (SC4) and high wind (SC3) 

scenarios: 0% improvement over state-of-the-art forecasting in PGEVLY

Figure E-2. Distribution of forecasting errors for the low wind (SC4) and high wind (SC3) 

scenarios: 10% improvement over state-of-the-art forecasting in PGEVLY

Figure E-3. Distribution of forecasting errors for the low wind (SC4) and high wind (SC3) 

scenarios: 25% improvement over state-of-the-art forecasting in PGEVLY
40

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Figure E-4. Distribution of forecasting errors for the low wind (SC4) and high wind (SC3) 

scenarios: 50% improvement over state-of-the-art forecasting in PGEVLY
41

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Appendix F: Plots of Errors Versus Wind Power for Reserve Analysis 

in PGEVLY

Figure F-1. Forecasting errors as a function of wind power for the low wind (SC4) and high wind 

(SC3) scenarios: 0% improvement over state-of-the-art forecasting in PGEVLY

Figure F-2. Forecasting errors as a function of wind power for the low wind (SC4) and high wind 

(SC3) scenarios: 10% improvement over state-of-the-art forecasting in PGEVLY
42

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Figure F-3. Forecasting errors as a function of wind power for the low wind (SC4) and high wind 

(SC3) scenarios: 25% improvement over state-of-the-art forecasting in PGEVLY

Figure F-4. Forecasting errors as a function of wind power for the low wind (SC4) and high wind 

(SC3) scenarios: 50% improvement over state-of-the-art forecasting in PGEVLY
43

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Appendix G: Distribution of Forecasting Errors in SCE

Figure G-1. Distribution of forecasting errors for the low wind (SC4) and high wind (SC3) 

scenarios: 0% improvement over state-of-the-art forecasting in SCE

Figure G-2. Distribution of forecasting errors for the low wind (SC4) and high wind (SC3) 

scenarios: 10% improvement over state-of-the-art forecasting in SCE

Figure G-3. Distribution of forecasting errors for the low wind (SC4) and high wind (SC3) 

scenarios: 25% improvement over state-of-the-art forecasting in SCE
44

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Figure G-4. Distribution of forecasting errors for the low wind (SC4) and high wind (SC3) 

scenarios: 50% improvement over state-of-the-art forecasting in SCE
45

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Appendix H: Plots of Errors Versus Wind Power for Reserves 

Analysis in SCE

Figure H-1. Forecasting errors as a function of wind power for the low wind (SC4) and high wind 

(SC3) scenarios: 50% improvement over state-of-the-art forecasting in SCE

Figure H-2. Forecasting errors as a function of wind power for the low wind (SC4) and high wind 

(SC3) scenarios: 10% improvement over state-of-the-art forecasting in SCE
46

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Figure H-3. Forecasting errors as a function of wind power for the low wind (SC4) and high wind 

(SC3) scenarios: 25% improvement over state-of-the-art forecasting in SCE

Figure H-4. Forecasting errors as a function of wind power for the low wind (SC4) and high wind 

(SC3) scenarios: 50% improvement over state-of-the-art forecasting in SCE
47

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Appendix I: Distribution of Forecasting Errors in SDGE

Figure I-1. Distribution of errors for the low wind (SC4) and high wind (SC3) scenarios: 0% 

improvement over state-of-the-art forecasting in SDGE

Figure I-2. Distribution of forecasting errors for the low wind (SC4) and high wind (SC3) scenarios: 

10% improvement over state-of-the-art forecasting in SDGE

Figure I-3. Distribution of forecasting errors for the low wind (SC4) and high wind (SC3) scenarios: 

25% improvement over state-of-the-art forecasting in SDGE
48

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Figure I-4. Distribution of forecasting errors for the low wind (SC4) and high wind (SC3) scenarios: 

50% improvement over state-of-the-art forecasting in SDGE
49

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Appendix J: Plots of Errors Versus Wind Power for Reserve Analysis 

in SDGE

Figure J-1. Forecasting errors as a function of wind power for the low wind (SC4) and high wind 

(SC3) scenarios: 0% improvement over state-of-the-art forecasting in SDGE

Figure J-2. Forecasting errors as a function of wind power for the low wind (SC4) and high wind 

(SC3) scenarios: 10% improvement over state-of-the-art forecasting in SDGE
50

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Figure J-3. Forecasting errors as a function of wind power for the low wind (SC4) and high wind 

(SC3) scenarios: 25% improvement over state-of-the-art forecasting in SDGE

Figure J-4. Forecasting errors as a function of wind power for the low wind (SC4) and high wind 

(SC3) scenarios: 50% improvement over state-of-the-art forecasting in SDGE
51

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Appendix K: Distribution of Forecasting Errors in SMUD

Figure K-1. Distribution of forecasting errors for the low wind (SC4) and high wind (SC3) 

scenarios: 0% improvement over state-of-the-art forecasting in SUMD

Figure K-2. Distribution of forecasting errors for the low wind (SC4) and high wind (SC3) 

scenarios: 10% improvement over state-of-the-art forecasting in SMUD

Figure K-3. Distribution of forecasting errors for the low wind (SC4) and high wind (SC3) 

scenarios: 25% improvement over state-of-the-art forecasting in SMUD
52

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Figure K-4. Distribution of forecasting errors for the low wind (SC4) and high wind (SC3) 

scenarios: 50% improvement over state-of-the-art forecasting in SMUD
53

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Appendix L: Plots of Errors Versus Wind Power for Reserve Analysis 

in SMUD

Figure L-1. Forecasting errors as a function of wind power for the low wind (SC4) and high wind 

(SC3) scenarios: 0% improvement over state-of-the-art forecasting in SMUD

Figure L-2. Forecasting errors as a function of wind power for the low wind (SC4) and high wind 

(SC3) scenarios: 10% improvement over state-of-the-art forecasting in SMUD

Figure L-3. Forecasting errors as a function of wind power for the low wind (SC4) and high wind 

(SC3) scenarios: 25% improvement over state-of-the-art forecasting in SMUD
54

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Figure L-4. Forecasting errors as a function of wind power for the low wind (SC4) and high wind 

(SC3) scenarios: 50% improvement over state-of-the-art forecasting in SMUD