NREL is a national laboratory of the U.S. Department of Energy, Office of Energy Efficiency and Renewable Energy, operated by the Alliance for Sustainable Energy, LLC.

Metrics for Evaluating the Accuracy 

of Solar Power Forecasting

Jie Zhang, Bri-Mathias Hodge, Anthony Florita

National Renewable Energy Laboratory

Siyuan Lu, Hendrik F. Hamann

IBM TJ Watson Research Center

Venkat Banunarayanan

U.S. Department of Energy

October 21–22, 2013

3rd International Workshop on 

Integration of Solar Power 

into Power Systems

London, UK

NREL/PR-5D00-60465
2

Solar Power Forecasting

 Forecasting solar energy generation is a challenging task due to

the variety of solar power systems and weather regimes

encountered.

 Solar power generation presents different challenges for

transmission and distribution networks:

 On the transmission side: solar power takes the form of centralized solar

power plants, a non-dispatchable component of the generation pool.

 On the distribution side: solar power is generated by a large number of

distributed panels installed on building rooftops, which changes the load

profile without providing visibility to the system operator.

 Forecast inaccuracies of solar power generation can result in

substantial economic losses and power system reliability issues.
3

Research Motivation and Objectives

• Develop a suite of generally applicable, value-based metrics for solar

forecasting for a comprehensive set of scenarios.

• Analyze the sensitivity of the proposed metrics to different types of solar

forecasting improvements.

 Solar forecasting accuracy evaluation does not have a standard metric, because

different forecasting stakeholders use different metrics.

 Establishing a standard set of metrics for assessing solar forecasting is:

 Critical to evaluating the success of a solar forecasting method.

 Useful for guiding the decision making of power system operators with a high

penetration of solar power.

Motivation

Research Objectives
4

Solar Power Data Summary

• The data was obtained from the Western Wind and Solar Integration Study

Phase 2 (WWSIS-2).

• The 60-minute solar power plant output for 2006 is used as the actual data.

• Day-ahead (DA) solar forecasts are taken from the WWSIS-2 solar forecasts

based on numerical weather prediction model simulations.

• One-hour-ahead (1HA) forecasts were synthesized using a persistence of

cloudiness approach.

Western 

Interconnection

Colorado Denver Single Plant
5

Proposed Metrics for Solar Forecasting

 Statistical Metrics

 Pearson’s correlation coefficient

 (Normalized) root mean square error (RMSE)

 Maximum absolute error (MaxAE), mean absolute error (MAE), mean

absolute percentage error (MAPE), mean bias error (MBE)

 Standard deviation, skewness, kurtosis

 Skewness: a positive (or negative) skewness of the forecast errors leads to a

forecasting bias

 Kurtosis: a measure of the magnitude of the peak of the error distribution and

the thickness of the tails

 Kolmogorov–Smirnov test Integral (KSI)

 KS test is to determine if two data sets differ significantly

 The KSI is the integration of differences between the cumulative distribution

function (CDF) for the two sets

 OVER

 The OVER is also the integrated differences between the CDFs, but only for

those in which a critical value Vc is exceeded
6

Proposed Metrics for Solar Forecasting

 Variability Estimations

 Different time and geographic scales

 Distributions of forecast errors: estimated using the kernel density estimation

(KDE) method

 Uncertainty Quantification

 Standard deviation

 Rényi entropy: a larger Rényi entropy indicates a high uncertainty in forecasting

 Ramping Metrics

 Heat maps: allow the operator to simultaneously see the timing and magnitude

of forecast errors

 Swinging door algorithm: extracts ramp periods in a series of power signals,

by identifying the start and end points of each ramp

 Economic Metrics

 Flexibility reserves are (i) primarily determined by net load forecast error

characteristics and (ii) represented by using hourly time steps and 95th confidence

intervals of solar power forecast errors
7

Distributions at Different Geographic Locations

One 

Plant Denver

Colorado Western

Interconnection
8

Metrics Values Estimated by Using an Entire Year of Data

Metrics

One Plant Denver Colorado Western 

Interconnection

Dayahead1-houraheadDayahead1-hourahead Day-ahead 1-houraheadDayahead1-hourahead

Correlation 

Coefficient 0.65 0.76 0.87 0.94 0.91 0.96 0.990 0.995

RMSE (MW) 22.07 17.12 438.25 284.36 624.19 378.65 2711.31 1488.28

NRMSE 0.22 0.17 0.13 0.08 0.10 0.06 0.04 0.02

MaxAE (MW) 84.10 74.33 2260.94 1304.73 3380.28 1735.24 17977.53 16127.32

MAE (MW) 14.81 11.34 286.65 191.17 413.11 256.69 1973.90 1064.52

MAPE 0.15 0.11 0.08 0.06 0.07 0.04 0.03 0.02

MBE (MW) 4.27 2.19 131.82 31.64 172.54 43.32 1497.29 132.13

KSIPer (%) 216.73 104.42 184.30 52.84 143.38 48.28 132.92 47.76

OVERPer (%) 136.36 28.16 94.43 0.77 54.65 0.37 41.43 0

Standard dev. 

(MW) 21.65 39.57 418.00 282.62 599.94 376.20 2260.09 1482.44

Skewness -0.19 0.08 0.20 -0.20 0.18 -0.21 0.62 -0.23

Kurtosis 2.04 2.40 3.79 2.52 3.35 2.47 3.76 4.82

95th Percentile 

(MW) 50.59 39.57 990.66 637.45 1394.85 838.27 5652.60 3079.32

Capacity (MW) 100 100 3463 3463 6088 6088 64495 64495
9

Rényi Entropy at Multiple Spatial and Temporal Scales

Cases

One Solar Power Plant Denver Region Colorado Region Western 

Interconnection

Day-ahead 1-hourahead Day-ahead 1-hourahead Day-ahead 1-hourahead Day-ahead 1-hourahead

Year 4.83 4.64 4.24 4.63 4.33 4.73 4.47 4.01

January 4.71 5.06 5.18 5.06 5.46 4.79 5.24 5.11

July 4.64 4.74 4.25 4.87 5.02 5.09 4.75 4.86

14:00 5.07 5.00 4.83 4.99 5.13 5.27 4.97 5.72

10:00 –

16:00 4.95 4.73 4.60 4.79 4.82 4.94 4.90 4.45

• The length of the forecasting period affects the uncertainty in the

forecasting.

• The uncertainty in the forecasting of the whole year data is less than

that in any of the other cases (January, July, 14:00, and 10:00 to 16:00).
10

Single Plant Denver Western Interconnection

1HA

DA

Ramping: Heat Maps of Mean Forecast Errors
11

Solar Forecasting Improvement 

 Uniform Forecasting Improvement

 The uniform forecasting improvement is accomplished by examining

the forecast error then essentially decreasing this error at each time

period by a set percentage.

 Ramp Forecasting Improvements

 The ramp extraction algorithm performs a piecewise linear

approximation to the original signal (actuals and day-ahead forecasts).

 Only ramps that exceed a threshold value (e.g., 20%) of the maximum

plant capacity are modified in the improved forecasts.

 Threshold Value: percentage of the maximum wind power

plant capacity

 Range: 10% – 30%
12

Overall Structure of Metrics Analysis

Design of Experiments (DoE)

Generate improved solar forecasting data 

and calculate performance metrics

Evaluate the performance of response surfaces

Build response surfaces 

using support vector regression method

Sensitivity analysis for metrics

 Design of Experiments (DoE)

 Sobol’s Quasi-random 

Sequence Generator

 Sobol sequences use a base of

two to form finer uniform

partitions of the unit interval

and reorder the coordinates in

each dimension.

 Surrogate Models

 Support Vector Regression

 Sensitivity Analysis

 Extended Fourier Amplitude

Sensitivity Test
13

Surrogate Modeling Methods

• Surrogate modeling is concerned with the construction of

approximation models to estimate the system performance and to

develop relationships between specific system inputs and

outputs.

• Support Vector Regression (SVR)

 A regression method to construct smooth, nonlinear, regression

approximations by formulating the surrogate model construction problem

as a quadratic programming problem
14

Surrogate Models of Multiple Metrics

NRMSE KSIPer Skewness

Kurtosis MaxAE Renyi Entropy
15

Sensitivity Analysis

• The Extended Fourier Amplitude Sensitivity Test

 It is a variance-based sensitivity analysis method.

 The sensitivity value is defined based on conditional

variances that indicate the individual or joint effects of the

uncertain inputs on the output.

 Main effect index: measures the contribution of Xi alone to the

uncertainty (variance) in Y

 Total effect index: gives the total variance in Y caused by Xi

and its interactions with any of the other input variables
16

Sensitivity Analysis of Metrics to Forecasting Improvements

Metrics

Uniform 

Improvement

Ramp 

Improvement

Ramp 

Threshold

Main effect Total effect Main 

effect Total effect Main effect

Total 

effect

R2 0.913 0.974 0.009 0.043 0.001 0.065

RMSE 0.883 0.958 0.016 0.060 0.000 0.088

NRMSE 0.883 0.958 0.016 0.060 0.000 0.088

MaxAE 0.899 0.977 0.011 0.063 0.002 0.057

MAE 0.896 0.955 0.016 0.056 0.003 0.077

MAPE 0.896 0.955 0.016 0.056 0.003 0.077

MBE 0.844 0.923 0.032 0.090 0.002 0.112

KSIPer 0.843 0.921 0.033 0.092 0.003 0.111

OVERPer 0.872 0.942 0.014 0.069 0.016 0.097

Standard Dev. 0.897 0.969 0.012 0.052 0.001 0.075

Skewness 0.305 0.686 0.126 0.473 0.161 0.343

Kurtosis 0.394 0.858 0.041 0.169 0.081 0.207

95th Percentile 0.886 0.965 0.013 0.060 0.002 0.082

Rényi Entropy 0.120 0.624 0.071 0.599 0.260 0.463
17

Concluding Remarks

 This paper proposed a suite of metrics for evaluating the performance of solar

power forecasting.

 The performance of the proposed metrics was evaluated using the actual and forecast

solar power data from WWSIS-2.

 The distribution of forecast errors indicates that relative forecast errors are smaller for

a large geographic area.

 The results showed that the developed metrics can successfully evaluate the

quality of a solar forecast.

 More over- than under-forecasting events were observed for the 1-hour-ahead solar

forecasting; whereas the day-ahead forecasting tended to be under-forecast in summer.

 A sensitivity analysis methodology was developed based on a design of

experiments and response surfaces approach:

 All proposed metrics were sensitive to solar forecasts with uniform forecasting

improvements.

 The metrics of skewness, kurtosis, and Rényi entropy were also sensitive to solar

forecasts with ramp forecasting improvements and ramp forecasting threshold.
18

• This research was supported by the U.S. Department of Energy

under Contract No. DE-AC36-08-GO28308 with the National

Renewable Energy Laboratory.

Acknowledgement

• Determine baseline values and achievable target values for the

metrics for independent system operators and utilities.

• Develop a suite of probabilistic solar forecasting metrics.

Future Work

Future Work
19

Questions?