# Presentation: 747558

## Common verification methods for ensemble forecasts

- Tom Hamill
*NOAA Earth System Research Lab, *
*Physical Sciences Division, Boulder, CO*
- tom.hamill@noaa.gov

- NOAA Earth System
- Research Laboratory

## What constitutes a “good” ensemble forecast?

- Here, the observed is outside of the range of the ensemble,
- which was sampled from the pdf shown.  Is this a sign of
- a poor ensemble forecast?

## Rank 1 of 21

- Rank 14 of 21

- Rank 5 of 21

- Rank 3 of 21

## One way of evaluating ensembles: “rank histograms” or “Talagrand diagrams”

- Happens when
- observed is
- indistinguishable
- from any other
- member of the
- ensemble. Ensemble
- is “reliable”

- Happens when
- observed too
- commonly is
- lower than the
- ensemble members.

- Happens when
- there are either
- some low and some
- high biases, or when
- the ensemble doesn’t
- spread out enough.

- We need lots of samples from many situations to evaluate the characteristics of the ensemble.

- ref: Hamill, *MWR*, March 2001

## Rank histograms of Z500, T850, T2m(from 1998 reforecast version of NCEP GFS)

- Solid lines indicate ranks after bias correction. Rank histograms are particularly
- U-shaped for T2M, which is probably the most relevant of the three plotted here.

## Rank histograms tell us about reliability - but what else is important?

- “Sharpness”
- measures the
- specificity of
- the probabilistic
- forecast.  Given
- two reliable forecast
- systems, the one
- producing the
- sharper forecasts
- is preferable.
- But: don’t want
- sharp if not reliable.
- Implies unrealistic
- confidence.

## “Spread-skill” relationships are important, too.

- ensemble-mean
- error from a sample
- of this pdf on avg.
- should be low.
- ensemble-mean
- error should be
- moderate on avg.
- ensemble-mean
- error should be
- large on avg.

- Small-spread ensemble forecasts should have less
- ensemble-mean error than large-spread forecasts.

## Spread-skill for 1990’s NCEP GFS

- At a given grid point,
- spread *S *is assumed
- to be a random
- variable with a
- lognormal distribution
- where *S**m* is the mean
- spread and **
- is its standard deviation.
- As ** increases, there
- is a wider range of
- spreads in the sample.
- One would expect
- then the possibility for
- a larger spread-skill
- correlation.

## Spread-skill and precipitation forecasts.

- True spread-skill relationships
- harder to diagnose if forecast
- PDF is non-normally distributed,
- as they are typically for
- precipitation forecasts.
- Commonly, spread is no longer
- independent of the mean value;
- it’s larger when the amount is
- larger.
- Hence, you get an apparent
- spread-skill relationship, but
- this may reflect variations in the
- mean forecast rather than
- real spread-skill.

## Reliability Diagrams

## Reliability Diagrams

- Curve tells you what
- the observed frequency
- was each time you
- forecast a given probability.
- This curve ought to lie
- along  *y = x*  line. Here this
- shows the ensemble-forecast
- system over-forecasts the
- probability of light rain.

- Ref: Wilks text, *Statistical Methods in the Atmospheric Sciences*

## Reliability Diagrams

- Inset histogram tells
- you how frequently
- each probability was
- issued.
- Perfectly sharp:
- frequency of usage
- populates only
- 0% and 100%.

- Ref: Wilks text, *Statistical Methods in the Atmospheric Sciences*

## Reliability Diagrams

*BSS* = Brier Skill Score

*BS*(•) measures the
- Brier Score, which you
- can think of as the
- squared error of a
- probabilistic forecast.
- Perfect: *BSS* = 1.0
- Climatology: *BSS* = 0.0

- Ref: Wilks text, *Statistical Methods in the Atmospheric Sciences*

## Brier Score

- Define an event, e.g., precip > 2.5 mm.
- Let      be the forecast probability for the ith forecast case.
- Let      be the observed probability (1 or 0).  Then

- (So the Brier score is the averaged squared error of
- the probabilistic forecast)

## Reliability after post-processing

- Statistical correction
- of forecasts using
- a long, stable set of
- prior forecasts from
- the same model
- (like in MOS).

- Ref: Hamill et al., MWR, Nov 2006

## Cumulative Distribution Function (CDF)

*F**f*(x) = Pr {X ≤ x}
- where X is the random variable, x is some specified threshold.

## Continuous Ranked Probability Score

-  Let           be the forecast probability CDF for the *i*th forecast case.
-  Let           be the observed probability CDF (Heaviside function).

## Continuous Ranked Probability Score

-  Let           be the forecast probability CDF for the *i*th forecast case.
-  Let           be the observed probability CDF (Heaviside function).

- (squared)

## Continuous Ranked Probability Skill Score (CRPSS)

- Like the Brier score, it’s common to convert this to
- a skill score by normalizing by the skill of climatology

## Relative Operating Characteristic (ROC)

## Relative Operating Characteristic (ROC)

## Method of Calculation of ROC:Parts 1 and 2

|  | Y | N |
| --- | --- | --- |
| Y | 0 | 0 |
| N | 0 | 1 |

- 55         56        57         58        59         60         61        62         63        64         65        66

- T

- F

- F

- F

- F

- F

- F

- Obs ≥ T?

- Fcst ≥ T?

- (1) Build contingency tables for each sorted ensemble member

- Obs

|  | Y | N |
| --- | --- | --- |
| Y | 0 | 0 |
| N | 0 | 1 |

- Obs ≥ T?

- Fcst ≥ T?

|  | Y | N |
| --- | --- | --- |
| Y | 0 | 0 |
| N | 0 | 1 |

- Obs ≥ T?

- Fcst ≥ T?

|  | Y | N |
| --- | --- | --- |
| Y | 0 | 1 |
| N | 0 | 0 |

- Obs ≥ T?

- Fcst ≥ T?

|  | Y | N |
| --- | --- | --- |
| Y | 0 | 1 |
| N | 0 | 0 |

- Obs ≥ T?

- Fcst ≥ T?

|  | Y | N |
| --- | --- | --- |
| Y | 0 | 1 |
| N | 0 | 0 |

- Obs ≥ T?

- Fcst ≥ T?

- (2) Repeat the process for other locations, dates, building
- up contingency tables for sorted members.

## Method of Calculation of ROC:Part 3

|  | Y | N |
| --- | --- | --- |
| Y | 4020 | 561 |
| N | 2707 | 72712 |

- Obs ≥ T?

- Fcst ≥ T?

- (3)  Get hit rate and false alarm rate for each from contingency table
- for each sorted ensemble member.

- Obs ≥ T?

- Fcst ≥ T?

- Obs ≥ T?

- Fcst ≥ T?

- Obs ≥ T?

- Fcst ≥ T?

- Obs ≥ T?

- Fcst ≥ T?

- Obs ≥ T?

- Fcst ≥ T?

|  | Y | N |
| --- | --- | --- |
| Y | H | F |
| N | M | C |

- Obs ≥ T?

- Fcst ≥ T?

- HR = H / (H+M)      FAR = F / (F+C)

- Sorted
- Member 1

- Sorted
- Member 2

- Sorted
- Member 3

- Sorted
- Member 4

- Sorted
- Member 5

- Sorted
- Member 6

|  | Y | N |
| --- | --- | --- |
| Y | 1106 | 3 |
| N | 5651 | 73270 |

|  | Y | N |
| --- | --- | --- |
| Y | 4692 | 1270 |
| N | 2035 | 72003 |

|  | Y | N |
| --- | --- | --- |
| Y | 3097 | 176 |
| N | 3630 | 73097 |

|  | Y | N |
| --- | --- | --- |
| Y | 5297 | 2655 |
| N | 1430 | 70618 |

|  | Y | N |
| --- | --- | --- |
| Y | 6603 | 44895 |
| N | 124 | 28378 |

- HR = 0.163
- FAR = 0.000

- HR = 0.504
- FAR = 0.002

- HR = 0.597
- FAR = 0.007

- HR = 0.697
- FAR = 0.017

- HR = 0.787
- FAR = 0.036

- HR = 0.981
- FAR = 0.612

## Method of Calculation of ROC:Part 3

- HR = 0.163
- FAR = 0.000

- HR = 0.504
- FAR = 0.002

- HR = 0.597
- FAR = 0.007

- HR = 0.697
- FAR = 0.017

- HR = 0.787
- FAR = 0.036

- HR = 0.981
- FAR = 0.612

- HR = [0.000, 0.163, 0.504, 0.597, 0.697, 0.787, 0.981, 1.000]
- FAR = [0.000, 0.000, 0.002, 0.007, 0.017, 0.036, 0.612, 1.000]

- (4) Plot hit rate
- vs. false alarm
- rate

## Economic Value Diagrams

- These diagrams
- tell you the
- potential economic
- value of your
- ensemble forecast
- system applied to
- a particular forecast
- aspect.  Perfect
- forecast has value
- of 1.0, climatology
- has value of 1.0.
- Value differs with
- user’s cost/loss
- ratio.

- Motivated by search for a metric that relates ensemble forecast
- performance to things that customers will actually care about.

## Economic Value: calculation method

- Assumes decision maker
- alters actions based on
- weather forecast info.
*C* = Cost of protection
*L = L**p**+L**u** = *total cost of
-   a loss, where ...
*L**p* = Loss that can be
-   protected against
*L**u* = Loss that can’t be
-   protected against.
*N* = No cost

## Economic value, continued

- Suppose we have the contingency
- table of forecast outcomes, [*h, m, f, c*].
- Then we can calculate the expected
- value of the expenses from a forecast,
- from climatology, from a perfect forecast.

- Note that
- value will vary
- with *C, L**p**, L**u;*
*Different users*
*with different *
*protection costs*
*may experience*
*a different value*
*from the forecast*
*system.*

## From ROC to Economic Value

- Value is now seen to be related to FAR and HR, the
- components of the ROC curve.  A (HR, FAR)
- point on the ROC curve will thus map to a
- value curve (as a function of C/L)

## The red curve is

- from the ROC
- data for the member
- defining the 90th
- percentile of the
- ensemble distribution.
- Green curve is for
- the 10th percentile.
- Overall economic
- value is the maximum
- (use whatever member
- for decision threshold
- that provides the
- best economic value).

## Forecast skill often overestimated!

- - Suppose you have a sample of forecasts from two islands,
-    and each island has different climatology.
- - Weather forecasts impossible on both islands.
-  Simulate “forecast” with an ensemble of draws from climatology
-  Island 1: F ~ N(,1).          Island 2: F ~ N(-,1)
-  Calculate ROCSS, BSS, ETS in normal way.  Expect no skill.

- As climatology of the two islands begins
- to differ, then “skill” increases though
- samples drawn from climatology.
- These scores falsely attribute differences
- in samples’ climatologies to skill of the forecast.
- Samples must have the same climatological
- event frequency to avoid this.

## Useful References

**Good overall references** for forecast verification:
- (1): Wilks, D.S., 2006:  *Statistical Methods in the Atmospheric Sciences (2nd Ed).*  Academic Press, 627 pp.
- (2) Beth Ebert’s forecast verification web page, _[http://tinyurl.com/y97c74](http://tinyurl.com/y97c74)_
**Rank histograms**:  Hamill, T. M., 2001: Interpretation of rank histograms for verifying ensemble forecasts.  *Mon. Wea. Rev*., **129**, 550-560.
**Spread-skill relationships**:  Whitaker, J.S., and A. F. Loughe, 1998: The relationship between ensemble spread and ensemble mean skill.  *Mon. Wea. Rev*., **126**, 3292-3302.
**Brier score, continuous ranked probability score, reliability diagrams**: Wilks text again.
**Relative operating characteristic**:  Harvey, L. O., Jr, and others, 1992:  The application of signal detection theory to weather forecasting behavior.  *Mon. Wea. Rev*., **120**, 863-883.
**Economic value diagrams**:
- (1)Richardson, D. S., 2000:  Skill and relative economic value of the ECMWF ensemble prediction system.  *Quart. J. Royal Meteor. Soc*., **126**, 649-667.
- (2) Zhu, Y, and others, 2002:  The economic value of ensemble-based weather forecasts.  *Bull. Amer. Meteor. Soc*., **83**, 73-83.
**Overforecasting skill**:  Hamill, T. M., and J. Juras, 2006:  Measuring forecast skill: is it real skill or is it the varying climatology?  *Quart. J. Royal Meteor. Soc.*, Jan 2007 issue. _[http://tinyurl.com/kxtct](http://tinyurl.com/kxtct)_