NREL is a national laboratory of the U.S. Department of Energy, Office of Energy 

Efficiency & Renewable Energy, operated by the Alliance for Sustainable Energy, LLC.

Contract No. DE-AC36-08GO28308

Economic Evaluation of ShortTerm Wind Power Forecasts in 

ERCOT: Preliminary Results

Preprint

K. Orwig, B.-M. Hodge, G. Brinkman, E. Ela, 

and M. Milligan

National Renewable Energy Laboratory

V. Banunarayanan and S. Nasir

ICF International

J. Freedman

AWS Truepower

To be presented at the 11th Annual International Workshop on 

Large-Scale Integration of Wind Power into Power Systems as Well 

as on Transmission Networks for Offshore Wind Power Plants

Conference

Lisbon, Portugal

November 13–15, 2012

Conference Paper

NREL/CP-5500-56257

September 2012
NOTICE

The submitted manuscript has been offered by an employee of the Alliance for Sustainable Energy, LLC 

(Alliance), a contractor of the US Government under Contract No. DE-AC36-08GO28308. Accordingly, the US 

Government and Alliance retain a nonexclusive royalty-free license to publish or reproduce the published form of 

this contribution, or allow others to do so, for US Government purposes.

This report was prepared as an account of work sponsored by an agency of the United States government. 

Neither the United States government nor any agency thereof, nor any of their employees, makes any warranty, 

express or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of 

any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately 

owned rights. Reference herein to any specific commercial product, process, or service by trade name, 

trademark, manufacturer, or otherwise does not necessarily constitute or imply its endorsement, recommendation, 

or favoring by the United States government or any agency thereof. The views and opinions of authors 

expressed herein do not necessarily state or reflect those of the United States government or any agency thereof.

Available electronically at http://www.osti.gov/bridge

Available for a processing fee to U.S. Department of Energy

and its contractors, in paper, from:

U.S. Department of Energy

Office of Scientific and Technical Information

P.O. Box 62

Oak Ridge, TN 37831-0062

phone: 865.576.8401

fax: 865.576.5728

email: mailto:reports@adonis.osti.gov

Available for sale to the public, in paper, from:

U.S. Department of Commerce

National Technical Information Service

5285 Port Royal Road

Springfield, VA 22161

phone: 800.553.6847

fax: 703.605.6900

email: orders@ntis.fedworld.gov

online ordering: http://www.ntis.gov/help/ordermethods.aspx

Cover Photos: (left to right) PIX 16416, PIX 17423, PIX 16560, PIX 17613, PIX 17436, PIX 17721

Printed on paper containing at least 50% wastepaper, including 10% post consumer waste.
1

Economic Evaluation of Short-Term Wind Power 

Forecasts in ERCOT

Preliminary Results

Kirsten D. Orwig

Bri-Mathias Hodge

Greg Brinkman

Erik Ela

Michael Milligan

National Renewable Energy Laboratory

Golden, CO USA

Venkat Banunarayanan

Saleh Nasir

ICF International

Fairfax, VA USA

Jeff Freedman

AWS Truepower

Albany, NY USA

Abstract—A number of wind energy integration studies have 

investigated the monetary value of using day-ahead wind power 

forecasts for grid operation decisions. Historically, these studies 

have shown that large cost savings could be gained by grid 

operators implementing the forecasts in their system operations. 

To date, none of these studies have investigated the value of 

shorter term (0- to 6-h ahead) wind power forecasts. In 2010, the 

Department of Energy and the National Oceanic and 

Atmospheric Administration partnered to form the Wind 

Forecasting Improvement Project (WFIP) to fund improvements 

in short-term wind forecasts and determine the economic value 

of these improvements to grid operators. In this work, we

discuss the preliminary results of the economic benefit analysis

portion of the WFIP for the Electric Reliability Council of 

Texas. The improvements seen in the wind forecasts are 

examined and the economic results of a production cost model 

simulation are analyzed.

Keywords-wind power forecasting; numerical weather 

prediction; economic value; grid operations

I. INTRODUCTION

Wind power forecasting is recognized as a key technology 

needed to aid in the integration of increasing penetrations of

variable and uncertain wind power. Previous wind integration 

studies have shown the economic and reliability benefits of 

wind power forecasting for grid operators but have primarily 

focused on day-ahead forecasts needed for the unit 

commitment process [1–4]. The Wind Forecasting 

Improvement Project (WFIP) was formed as a partnership 

between the United States Department of Energy, the National 

Oceanic and Atmospheric Administration, private industry, 

and academia. It is unique in that it focuses on the potential 

value provided by improvements in very short-term wind 

power forecasts, defined here as between 0 and 6 h ahead.

Within the meteorological community, it is commonly 

believed that increasing the number and quality of 

meteorological measurements assimilated into numerical 

weather prediction (NWP) models will have a positive impact 

on the quality of wind power forecasts derived from them. The 

WFIP was designed to test this assumption. Additional 

information on WFIP can be found in [5]. Fig. 1 shows the 

project region.

Figure 1. Southern study region and instrumenation.

II. ANALYSIS PROCESS

A preliminary analysis to understand the impact of 

improving accuracy of short-term wind power forecasts was 

performed using production simulation software. The tool 

used was GE-MAPS (MAPS) [6], which is a securityconstrained unit commitment and dispatch model simulating

the operation of a locational marginal price (LMP)–based 

power market. MAPS simulates the hour-by-hour operation of 

the power market, committing and dispatching generating 

units to meet hourly system load with the objective of 

minimizing overall system production cost subject to 

generating unit constraints (such as minimum up-/downtimes, minimum and maximum capacity levels, etc.), 

transmission constraints (including DC-based power flow and 

contingency constraints), and other system constraints (such as 

operating reserves). System production cost is defined as the 

sum of fuel costs, variable operation and maintenance costs, 
2

start-up costs, and emission costs for all generators that are 

chosen to provide energy. Wind generators in MAPS were

modeled using hourly wind shapes and assumed to have a zero 

variable cost.

The key direct benefits because of improved accuracy of 

short-term forecasts are reduced system production costs 

because of more efficient commitment of medium-start 

generators (e.g., with start-up times less than 6 h) and reduced 

curtailment of wind generation. Other potential benefits 

include reduction in energy imbalance cost because of possible 

decrease in penalties paid by wind generators that deviate 

from forecast (applicable in markets that allow participation in

the day-ahead and real-time market by wind units with energy 

imbalance fulfillment requirements), decreases in total 

wholesale electricity price paid by demand (otherwise known 

in this paper as load payment costs/cost to serve load), and 

possible reductions in NOx emissions and unit start-ups,

depending on specific changes in unit commitment and 

dispatch decisions in response to changes in wind forecasts.

Using MAPS, the system benefits mentioned above were

estimated for the impact of an improved 6-h-ahead wind 

forecast for an historical six-month period, from October 2011 

to March 2012. This analysis was performed for the Electric 

Reliability Council of Texas (ERCOT). First, MAPS was run 

using day-ahead wind forecasts. Then, MAPS was run twice—

once with the old 6-h-ahead wind forecasts and then with the 

new and more accurate 6-h-ahead wind forecasts. In both of 

these runs, the commitment patterns of generating units with 

start-up times greater than 6 h were fixed. In ERCOT, these 

units were the nuclear units and primarily large coal and large 

steam, oil, and gas units. The generating units that were 

allowed to change their unit commitment status in MAPS were 

those with minimum down times of less than or equal to 6 h 

because these were the units that can respond, if needed (by 

turning on or off), to accommodate the new 6-h wind forecast. 

Those units whose commitment status was allowed to change 

to the 6-h forecasts were primarily combined-cycle units, 

combustion turbines, and some small steam units. Once the 

commitment was revised to consider the impact of the 6-h 

forecasts, unit dispatch was performed in MAPS utilizing the 

actual output of wind generators.

Another benefit from improved short-term wind power 

forecasts is the cost savings from the reduced need to carry 

operating reserves. In 2008, ERCOT completed a study 

evaluating the impacts that high wind penetrations would have 

on operating reserve requirements [7]. Requirements for 

various reserve categories in ERCOT are now based on the 

wind power penetration and its associated impact to wind 

power forecast errors [8]. In WFIP, we focus on the impact to 

non-spinning reserve service, which is the most impacted by 

short-term forecast errors of wind.

III. SHORT-TERM FORECAST ACCURACY IMPROVEMENT

Before looking at the economic benefits from the 

production cost simulation process, it is instructive to 

understand the forecast improvements for the six-month 

period and their variations in space (between various wind 

generators) and time (during a day/week). It is also useful to 

look at a brief statistical analysis of improved wind forecasts 

to understand error distributions.

Fig. 2 shows a sample of the improvements in forecast for

various wind generators (as a percent of their installed 

capacity) and the net improvement if the output from those 

wind generators were aggregated.

Figure 2. Benefits of aggregation.

A few observations can be made from Fig. 2. First, at any 

point in time there are generally a few wind generators with

outputs that are over-forecast and a few wind generators with

outputs that are under-forecast. Therefore, although individual 

wind generators can have relatively high forecast errors, the 

aggregate forecast error can be quite low. This is shown in 

Fig. 1, which is a visual comparison of the variations in error 

for each of the sites versus the variation of the aggregate 

curve. The lower overall error in the aggregate wind forecasts 

was because of the net sum of the under-forecast and overforecast errors for individual wind generators. It is important 

to note that in the production simulation process, to create the 

net hourly load at each bus the outputs of wind generators 

(either forecast or actual) are subtracted from load at the bus at 

which the wind generator is interconnected. Hence, barring 

transmission constraints between individual wind generators, 

the system simulations (and system operations, in reality) will 

see the error of the aggregated curve shown in Fig. 2 and 

therefore could see correspondingly less forecast errors than 

those for individual wind generators. Fig. 3 and Table 1 and 

Table 2 show a summary statistical description of the overall 

improvements in short-term forecasts at a system level by

aggregating the forecasts for all wind generators.

Figure 3. Distribution of 6-h forecast errors (old and new) for the 6-month 

period.
3

TABLE I. STATISTICAL SUMMARY OF OLD AND NEW FORECASTS 

(ERROR AS % OF CAPACITY)

Percent Error (% of Total Wind Capacity)

Mean Absolute Error (%) Standard Deviation (%)

Month New Old ImprovementNew Old Improvement

October 6.8 8.6 -1.7 6.3 7.1 -0.7

November 6.6 7.4 -0.8 5.6 6.3 -0.7

December 8.6 9.1 -0.6 7.4 7.4 0.0

January 7.8 8.6 -0.8 6.5 6.9 -0.4

February 9.7 10.1 -0.4 8.8 9.3 -0.5

March 8.1 8.9 -0.8 6.8 6.7 0.0

All 7.9 8.8 -0.8 7.0 7.4 -0.3

TABLE II. STATISTICAL SUMMARY OF OLD AND NEW FORECASTS 

(MW ERROR)

MW Error (Based on 8,132 MW of Wind Capacity)

Mean Absolute Error (MW) Standard Deviation (MW)

Month New Old ImprovementNew Old Improve

-ment

October 556 697 -141 514 574 -59

November 539 601 -62 459 512 -53

December 696 741 -45 600 598 2

January 635 699 -65 531 564 -33

February 789 824 -35 718 756 -38

March 661 723 -61 549 548 1

All 645 714 -68 572 599 -26

As shown in Fig. 3, the new forecast was much sharper 

than the old—reducing the frequency of larger deviations from 

the actual, thus increasing the overall accuracy of the forecast. 

The accompanying Table 1 and Table 2 show the 

improvements in accuracy from a percentage and absolute 

MW basis using mean absolute error (MAE) and standard 

deviation as metrics. Table 1 shows that an average sustained

improvement of about 0.8% was achieved in the accuracy of 

the 6-h-ahead forecasts, translating into 68 MW of reduction 

in average MAE, as shown in Table 2. Although these 

numbers may seem relatively small, they have an impact on

system production costs, as shown below. Further, the 

reduction in MAE will also result in potential reductions in the 

amount of operating reserves that need to be carried by the 

system and the reductions in cost.

Figs. 4 through 7 show the hourly variation of the new and 

old 6-h forecast errors for three different days to illustrate the 

significant changes in behavior of the forecast errors. Fig. 4 

shows a day in March in which the new forecast was better 

than the old forecast (lesser magnitude of deviations from 

actual output) during all hours of the day. Fig. 5 shows a 

sample day in which the old forecast was more accurate than 

the new forecast during all hours of the day, and Fig. 6 depicts

a sample day in which the old forecast was better than the new 

during peak hours of the day. Finally, Fig. 7 shows the 

variations in forecast errors (shown as deviations from actual 

output) during the week of March 22, 2011, as an illustration 

of the temporal variations of the forecast errors over the 

commitment horizon of MAPS.

Figure 4. New forecast closer to actual than old forecast.

Figure 5. Old forecast closer to actual than new forecast.

Figure 6. Old forecast closer to actual than new forecast—primarily during 

peak hours.

Figure 7. Typical weekly variation in forecast error.
4

The above figures show that the forecast accuracy 

improvements varied significantly during a day and during

multiple days. There were periods during which the old 

forecast was better than the new forecast, as shown in Fig. 7

during peak hours. This implies that the corresponding 

benefits because of improved accuracy of forecasting will also 

vary, with some hours showing a decrease in system 

production cost and other hours showing an increase as the 

impact of change in accuracy of the wind forecasts.

IV. ERCOT MARKET SIMULATION RESULTS

This section presents the results of the production 

simulation analysis to estimate the benefits of short-term 

forecast accuracy improvement. The simulation was 

performed for six months from the period of October 2011 to 

March 2012. The ERCOT power system was calibrated for 

actual system capacity, including fuel prices, peak load, 

installed capacity, and generating unit and transmission line 

outages. The key input assumptions are shown below in 

Tables III through V.

TABLE III. SYSTEM PEAK DEMAND (BASED ON HISTORICAL DATA)

USED IN STUDY

Month Peak Demand (MW)

October 49,328

November 41,375

December 50,049

January 46,914

February 42,093

March 42,607

TABLE IV. GAS PRICES ($/MMBTU) USED IN STUDY (BASED ON 

HISTORICAL DATA)

Year Month Houston South North West

2011 October 3.45 3.45 3.42 3.40

2011 November 3.21 3.18 3.14 3.19

2011 December 3.15 3.11 3.09 3.15

2012 January 2.65 2.62 2.61 2.62

2012 February 2.44 2.46 2.45 2.45

2012 March 2.10 2.07 2.09 2.08

TABLE V. COAL PRICES ($/MMBTU) USED IN STUDY (BASED ON

HISTORICAL DATA)

Month Merchant Regulated Average

October 1.91 1.97 1.91

November 1.97 2.07 1.99

December 1.85 1.96 1.88

January 2.67 2.04 2.55

February 1.87 2.01 1.83

March 1.77 1.96 1.68

The impacts on production costs, load payments, wind 

curtailment, and energy imbalance costs paid by wind 

generators are shown below in Table VI.

In the total six-month period, there was a measurable 

sustained reduction in system production cost of 

approximately $1 million. The decrease in production cost was

a direct result from a more efficient unit commitment of 

resources with start-up times less than 6 h. Also, it can be 

observed that the total price paid by system loads, as 

determined by the resulting LMP, decreased significantly 

because of more efficient operation of conventional units, 

resulting from improved forecast accuracy. This result is 

noteworthy because it indicates a reduction in electricity prices 

that result from improved forecast accuracy. The reduction in 

wind curtailment is a function of the level of wind penetration 

and system conditions such as transmission sufficiency. In 

ERCOT, there is a relatively high penetration of wind;

therefore, wind energy is curtailed during hours in which 

actual wind generation deviates significantly from forecast or 

because of insufficient transmission or conventional unit 

minimum up-/down-time constraints. The wind energy 

curtailed was reduced, as shown in Table VI, because the 

accuracy of the forecast was higher, resulting in more efficient 

usage of maximum wind power with given generation and 

transmission constraints. It can also be noted from Table VI 

that the benefits in load payments were significantly higher in 

the January-to-March period compared to the October-toDecember period. This was because there were considerably 

higher wind over-forecast errors in the new forecast than in the 

old forecast during the month of December, which contributed 

to increase in energy prices (because the energy expected to 

come from wind had to be supplied from conventional 

generation) and, hence, to load payments in December that 

“netted out” some of the benefits realized during the months of 

October and November.

The relatively low amount of benefits in system production 

cost for six months (~ 0.05% of total system production cost) 

was because of the modest improvement in forecast accuracy 

(at ~ 1%) and almost directly correlated to natural gas prices 

because natural gas–burning generators set the price in 

ERCOT most of the time. To illustrate this point, and to better 

understand the variation in results to natural gas prices, a 

sensitivity case was run in which the natural gas prices were 

doubled (from an average of ~ 2.8 $/MMBtu in the base case 

to ~ 6 $/MMBtu). Table VII shows the results of this 

sensitivity.

TABLE VI. SUMMARY OF BENEFITS FOR BASE CASE

Benefit ($ Savings)

Parameter Oct.–Dec.

2011

Jan.–Mar.

2012

Total

Production Cost ($) 833,355 137,706 971,061

Load Payment ($) 3,898,864 28,316,608 32,215,472

Reduction in Wind 

Curtailment (MWh)

10,407 24,400 34,807

Energy Imbalance Costs Paid 

by Wind Generators ($)

1,562,626 845,930 2,408,556

TABLE VII. SUMMARY OF BENEFITS FOR GAS SENSITIVITY CASE

Benefit ($ Savings)

Parameter Oct.–Dec.

2011

Jan.– Mar.

2012

Total

Production Cost ($) 1,173,509 1,119,428 2,292,937

Load Payment ($) 1,535,411 21,823,012 23,358,423

Reduction in Wind 

Curtailment (MWh)

17,197 21,103 38,301

Energy Imbalance Costs Paid 

by Wind Generators ($)

693,983 295,842 989,825
5

A comparison of Tables VI and VII shows that a doubling 

in natural gas price doubled the system production cost 

reductions for the new, more accurate short-term wind 

forecast. This illustrates the highly dependent nature of 

benefits on system conditions such as natural gas prices in the 

case of ERCOT. The comparison between the two cases also 

revealed that the savings in load payments and the energy 

imbalance costs actually decreased with increased gas prices. 

Although there are benefits to improved forecast accuracy and 

corresponding savings, the doubling of gas prices results in the 

benefits coming primarily from savings because of energy 

displacement between lower efficiency and higher efficiency 

combined-cycle units rather than from energy displacement 

between coal and natural gas units in the base case, where a 

smaller gap between coal and gas prices made the fuel 

switching more economical.

V. RESERVE REQUIREMENT REDUCTIONS

Although the benefits currently shown in the economic 

evaluation were modest, other benefits are expected to accrue 

as well. For example, because the forecasts are expected to 

improve in the 0- to 6-h timeframe, the amount of operating 

reserves necessary to compensate for wind power forecasting 

errors is expected to decrease. This study will focus on the 

non-spinning reserve service requirements (NSRS). The NSRS 

requirements are currently calculated by ERCOT with direct 

impact on the net load, and therefore wind power, forecast 

uncertainty. The calculation assumed in this study follows that 

of (1).

NSRS = Max{Pmax–500,Min{2000, Φ95 –500–Regup}} (1)

where Pmax is the largest unit in the ERCOT system, Φ95 is the 

95th percentile of 4-h-ahead net load forecast errors from the 

previous month, and Regup is the average regulation uprequirement for the period.

This value is calculated for 4-h blocks for each month. The 

floor of Pmax—500—is used only for peak hours of 0700 

through 2200. The average net load uncertainty will also be 

calculated for each of these blocks to see if the net load had 

been on average over-forecasted during those periods. If so, 

the average over-forecasted value will be added to the NSRS 

requirement.

Because the 95th percentile value is the main driver for the 

NSRS requirements, the value may be different than the other 

benefits seen thus far. This analysis will be part of the final 

results, where the change in NSRS requirements will be part 

of the production simulation analysis. This will allow the 

study team to quantify benefits from efficient commitment of 

resources as well as those from reduced NSRS requirements to 

understand the details of benefits coming from improved 

short-term wind power forecasts.

VI. CONCLUSION

Preliminary analysis on estimating the economic systemlevel benefits for improved short-term forecast accuracy on the 

ERCOT system showed sustained savings in system 

production cost and load payments, reductions in wind energy 

curtailed and in energy imbalance payments from wind 

generators. The analysis was performed for a historical time 

period of six months, from October 2011 to March 2012. 

However, the amount of savings in production cost was

relatively small when compared to total system production 

cost. Although this is to be expected for an aggregate average 

forecast accuracy improvement of approximately 1% MAE 

combined with low natural gas prices, note that this savings 

was sustained over the six-month period and is expected to 

continue into the future. Also, as natural gas prices increase 

from the historic current lows of around 2 to 3 $/MMBtu, the 

benefits will only increase, as shown in the sensitivity case. 

Future work will include repeating this analysis for one year of 

improved forecast data as it becomes available. It will be 

further augmented by considering benefits because of

reductions in operating reserves from increased short-term 

forecast accuracy and exploring the benefit of improved 

forecast accuracy on system reliability.

ACKNOWLEDGMENTS

The authors would like to thank the U.S. Department of 

Energy and the National Oceanic and Atmospheric 

Administration for the sponsorship and support of this project.

This work was supported by the U.S. Department of 

Energy under Contract No. DE-AC36-08-GO28308 with the 

National Renewable Energy Laboratory.

REFERENCES

[1] GE Energy, Western Wind and Solar Integration Study, 

NREL/SR-550-47434, Golden, CO: National Renewable 

Energy Laboratory, 2010.

[2] D. Lew, M. Milligan, G. Jordan, and R. Piwko, "The 

value of wind power forecasting," 91st AMS Annual Meeting, 2nd Conference on Weather, Climate, and the 

New Energy Economy Proceedings, Washington, D.C., Jan. 26, 2011.

[3] E. Ela, M. Milligan, B. Parsons, D. Lew, and D. Corbus, 

“The evolution of wind power integration studies: Past, 

present, and future,” IEEE PES General Meeting Proceedings, Calgary, CA, July 26–30, 2009.

[4] M. Milligan et al., “Operational analysis and methods for 

wind integration studies,” IEEE Trans. Sustainable 

Energy, DOI: 10.1109/TSTE.2011.2160881, 2012.

[5] K. Orwig et al., "Enhanced short-term wind power 

forecasting and value to grid operations: The wind 

forecasting improvement project (WFIP)," 11th 

International Workshop on the Large-Scale Integration of Wind Power into Power Systems Proceedings, Lisbon, 

Portugal, Nov. 13–15, 2012.

[6] GE Energy, GE MAPS Software for Informed Economic 

Decisions Fact Sheet, 2011, www.ge- energy.com/content/multimedia/files/downloads/EC_Do wnload_WilliamsS_Concorda MAPS Engine Fact Sheet 

GEA19669-dvz.pdf.

[7] GE Energy, Analysis of Wind Generation Impact on ERCOT Ancillary Services Requirements, Prepared for 

the Electric Reliability Council of Texas, March 2008.

[8] ERCOT Methodologies for Determining Ancillary Service Requirements, April 2012.