# Presentation: 925925

## Measuring forecast skill: is it real skill or  is it the varying climatology?

- Tom Hamill
*NOAA / ESRL / PSD, Boulder, Colorado *_[tom.hamill@noaa.gov](mailto:tom.hamill@noaa.gov)_*; www.cdc.noaa.gov/people/tom.hamill*
- Josip Juras
*University of Zagreb, Croatia*

## Hypothesis

- If climatological event probability varies among samples, then many verification metrics will credit a forecast with extra skill it doesn’t deserve - the extra skill comes from the variations in the climatology.

## Consider Equitable

- Threat Scores...

## Consider Equitable

- Threat Scores...
- ETS location-dependent,
- related to climatological
- probability.

## Consider Equitable

- Threat Scores...
- ETS location-dependent,
- related to climatological
- probability.
- (2) Average of ETS at
- individual grid points = 0.28

## Consider Equitable

- Threat Scores...
- ETS location-dependent,
- related to climatological
- probability.
- (2) Average of ETS at
- individual grid points = 0.28
- (3) ETS after data lumped into
- one big table = 0.42

## Considering three metrics...

- (1) Brier Skill Score (1=perfect, 0=reference)

- (2) Relative Operating
- Characteristic

- (3) Equitable Threat Score

- (each will show this tendency to have scores vary depending on how they’re calculated)

## Equitable Threat Score:standard method of calculation

- Assume we have a deterministic forecast
- All *m* samples, at all locations and times,
- populate one contingency table
- 			   Event forecast?
- 				YES	             NO
- 			 -------------------------------------------------
- 		YES	|	*a* 	|	*b* 	|
-        Event		-------------------------------------------------
-      Observed?	 NO	|	*c* 	|	*d* 	 |
- 			 -------------------------------------------------

-   where

## Equitable Threat Score:alternative method of calculation

- Consider the possibility of different regions with different
- climates. Assume *n**c* contingency tables, each
- associated with samples with a distinct climatological
- event frequency.  *n*s(*k*) out of the *m* samples were
- used to populate the *k*th table.  ETS calculated separately
- for each contingency table, and alternative, weighted-
- average ETS is calculated as

- _[BSS?](slide22.xml)_

- _[ROC?](slide26.xml)_

## ETS calculated two ways

## Example of unexpected skill: two islands, zero meteorologists

- Imagine a planet with a global ocean and two isolated
- islands.  Weather forecasting other than climatology for
- each island is impossible.
- Island 1:  Forecast, observed uncorrelated, ~ *N *(+, 1)
- Island 2:  Forecast, observed uncorrelated, ~ *N *(–, 1)
- 			0 ≤  ≤ 5
-                        Event:  Observed > 0

## Two islands

- As  increases...

- Island 1

- Island 2

- But still, each island’s forecast is no better than
- a random draw from its climatology.  Expect no skill.

## Skill with conventional methods of calculation

- Reference climatology implicitly becomes
*N*(+,1) + *N*(–,1)       not      *N*(+,1) OR *N*(–,1)

## The new reference climatology

## Are standard methods wrong?

**Assertion**: we’ve just re-defined climatology, they’re the correct scores with reference to that climatology.
**Response**: You *can* calculate them this way, but you shouldn’t.
- You will draw improper inferences due to “lurking variable” - i.e., the varying climatology should be a predictor.
- Discerning real skill or skill difference gets tougher

- “*One method that is sometimes used is to combine all the *
*data into a single 2x2 table ... this procedure is legitimate *
*only if the probability ****p**** of an occurrence (on the null *
*hypothesis) can be assumed to be the same in all the *
*individual 2x2 tables.  Consequently, if ****p**** obviously *
*varies from table to table, or we suspect that it may vary, *
*this procedure should not be used.”*
*	**W. G. Cochran, 1954, discussing ANOVA tests*

## Related problem when means are the same but climatological variances differ

**Event**:  v > 2.0
**Island 1**: f ~ N(0,1),   v ~ N(0,1),   Corr (f,v) = 0.0
**Island 2**: f ~ N(0,),   v ~ N(0,), 1 ≤ ≤ 3,   Corr (f,v) = 0.9
**Expectation**: positive skill over two islands, but not a function of 

## Scores vary with 

- _[more](slide28.xml)_

## the island with the greater climatological uncertainty of the 

- observed event ends up dominating the calculations.

- _[more](slide28.xml)_

## Solutions ?

-    (1) Analyze events where climatological probabilities are the same at all locations, e.g., terciles.

## Solutions, continued

- (2) Use sample-weighted averages

- ROC:

## Conclusions

- Many conventional verification metrics like BSS, RPSS, threat scores, ROC, potential economic value, etc. can be overestimated if climatology varies among samples.
- results in false inferences: think there’s skill where there’s none.
- complicates evaluation of model improvements; Model A better than Model B, but doesn’t appear quite so since both inflated in skill.
**Fixes**:
- Consider events where climatology doesn’t vary such as the exceedance of a quantile of the climatological distribution
- Combine after calculating for distinct climatologies.
**Please**: Document your method for calculating a score!

**Acknowledgements**:  *Matt Briggs*, Dan Wilks, Craig Bishop, Beth Ebert,
- Steve Mullen, Simon Mason, Bob Glahn, Neill Bowler, Ken Mylne, Bill Gallus,
- Frederic Atger, Francois LaLaurette, Zoltan Toth, Jeff Whitaker.

## Brier Skill Scores from raw ensembles

- Event is whether the observed weather will be above threshold T.
- Let** ****X**e(*j*) = [ *X*1(*j*),  ...  , *X**n*(*j*) ] be *n*-member ensemble forecast of the
- relevant scalar variable for the *j** *th of *m* samples (taken over many
- case days and / or locations).  Ensemble sorted from lowest to highest.
- Convert sorted ensemble to an *n*-member binary forecast
**I**e(*j*) = [ *I**1*(*j*),  ... , *I**n* (*j*) ].       { = 1 if *X*i(*j*) >T,  = 0 if *X*i(*j*) ≤ T  }
- Observed weather also converted to binary, denoted by *I**o*(*j*)*.*

- forecast probability

- Brier score of the forecast

- _[back](slide9.xml)_

## Brier Skill Score, continued

*BSS* = 1.0 – *BS**f* / *BS**c*      (def. of Brier Skill Score)

**Standard method of calculation**:
- Single climatological probability
- calculated over all *m* samples

- climatological event probability

- Brier Score of climatology

## Brier Skill Score, continued

- Suppose *m* samples split up into *n**c* subsets, each with a distinct climatological event frequency.
- Let *p*c(*k*) be the climatological probability in the *k*th of the *n**c* subsets, with *n*s(*k*) samples in this subset.
- Let **r***k *= [*r*(1) , ... , *r*(*n*s(*k*))] be the associated set of sample indices out of the *m* samples.

- Brier score of climatology calculated separately for each subset.

- The overall Brier score of climatology is the sum of scores for each subset.

**Alternative 1**: Multiple climatological probabilities
- calculated for different regions, then summed.

## Brier Skill Score, continued

**Alternative 2**: Final BSS is sample-weighted
- average of BSS for each subset.

- calculate forecast Brier
- score separately for
- each distinct region, as
- was done for climatology

*n**s*(*k*) */ m* is the weight
- applied to that region’s
- BSS

## Relative Operating Characteristic:standard method of calculation

- Populate 2x2 contingency tables, separate one for each sorted ensemble member.   The contingency table for the *i*th sorted ensemble member is
- 		    Event forecast by *i*th member?
- 			           YES	             NO
- 			-------------------------------------------------------
- 		YES	|	*a**i* 	|	*b**i*        	|
- Event			-------------------------------------------------------
- Observed?	NO	|	*c**i* 	|	*d**i* 	|
- 			-------------------------------------------------------
-                                                                   (* **a**i* + *b**i* + *c**i* + *d**i* = 1)

- (hit rate)

- (false alarm rate)

- ROC is a plot of hit rate (y) vs. false alarm rate (x).  Commonly
- summarized by “area under curve” (AUC), 1.0 for perfect forecast,
- 0.5 for climatology.

- _[back](slide9.xml)_

## Relative Operating Characteristic: alternative method of calculation

- As with the BSS, suppose samples can be partitioned into *n**c* subsets,
- each associated with a distinct climatological event frequency. Using the
*n*s(*k*) samples, the hit rates and false alarm rates for the *kth* climatology

- Then calculate sample-weighted average hit rates and false alarm rates:

## Island 1,    = 0.0232

|  | F = yes | F = no |
| --- | --- | --- |
| O = yes | 0.0171 | 0.0117 |
| O = no | 0.0108 | 0.9603 |

|  | F = yes | F = no |
| --- | --- | --- |
| O = yes | 0.0004 | 0.0228 |
| O = no | 0.0223 | 0.954 |

- Island 1,    = 0.0232
- ETS = -0.0022, HR = 0.0172

- Island 2,  = 1,      = 0.0288
- ETS = 0.4195, HR = 0.5937

|  | F = yes | F = no |
| --- | --- | --- |
| O = yes | 0.2022 | 0.0578 |
| O = no | 0.0597 | 0.6802 |

- Island 2,  = 3,      = 0.26
- ETS = 0.5327, HR = 0.778

- ETS(combined table) = 0.193, HR = 0.336

- ETS(combined table) = 0.499, HR = 0.715

- _[back](slide17.xml)_