# Presentation: 190593

## Measuring forecast skill: is it real skill or  is it the varying climatology?

- Tom Hamill
*NOAA Earth System Research Lab, Boulder, Colorado *_[tom.hamill@noaa.gov](mailto:tom.hamill@noaa.gov)_*; www.cdc.noaa.gov/people/tom.hamill*
- Josip Juras
*University of Zagreb, Croatia*

## Hypothesis

- If climatological event probability varies among samples, then many verification metrics will credit a forecast with extra skill it doesn’t deserve - the extra skill comes from the variations in the climatology.

## Example: Brier Skill Score

- Brier Score:  Mean-squared error of probabilistic forecasts.

- Brier Skill Score: Skill relative to some reference, like climatology.
- 1.0 = perfect forecast, 0.0 = skill of reference.

## Overestimating skill:  example

- _5-mm threshold_

**Location A**: *P**f* = 0.05, *P**clim* = 0.05, Obs = 0

**Location B**: *P**f* = 0.05, *P**clim* = 0.25, Obs = 0

**Locations A and B**:

## Overestimating skill:  example

- _5-mm threshold_

**Location A**: *P**f* = 0.05, *P**clim* = 0.05, Obs = 0

**Location B**: *P**f* = 0.05, *P**clim* = 0.25, Obs = 0

**Locations A and B**:

- why not
- 0.48?

- for more detail, see Hamill and Juras, QJRMS, Oct 2006 (c)

## Another example of unexpected skill: two islands, zero meteorologists

- Imagine a planet with a global ocean and two isolated
- islands.  Weather forecasting other than climatology for
- each island is impossible.
- Island 1:  Forecast, observed uncorrelated, ~ *N *(+, 1)
- Island 2:  Forecast, observed uncorrelated, ~ *N *(–, 1)
- 			0 ≤  ≤ 5
-                        Event:  Observed > 0
- Forecasts: random ensemble draws from climatology

## Two islands

- As  increases...

- Island 1

- Island 2

- But still, each island’s forecast is no better than
- a random draw from its climatology.  Expect no skill.

## Consider three metrics...

-  Brier Skill Score
-  Relative Operating Characteristic
-  Equitable Threat Score

- (each will show this tendency to have scores vary depending on how they’re calculated)

## Relative Operating Characteristic:standard method of calculation

- Populate 2x2 contingency tables, separate one for each sorted ensemble member.   The contingency table for the *i*th sorted ensemble member is
- 		    Event forecast by *i*th member?
- 			           YES	             NO
- 			-------------------------------------------------------
- 		YES	|	*a**i* 	|	*b**i*        	|
- Event			-------------------------------------------------------
- Observed?	NO	|	*c**i* 	|	*d**i* 	|
- 			-------------------------------------------------------
-                                                                   (* **a**i* + *b**i* + *c**i* + *d**i* = 1)

- (hit rate)

- (false alarm rate)

- ROC is a plot of hit rate (y) vs. false alarm rate (x).  Commonly
- summarized by “area under curve” (AUC), 1.0 for perfect forecast,
- 0.5 for climatology.

## Relative Operating Characteristic (ROC) skill score

## Equitable Threat Score:standard method of calculation

- Assume we have a deterministic forecast
- 			   Event forecast?
- 				YES	             NO
- 			 -------------------------------------------------
- 		YES	|	*a* 	|	*b* 	|
-        Event		-------------------------------------------------
-      Observed?	 NO	|	*c* 	|	*d* 	 |
- 			 -------------------------------------------------

-   where

## Two islands

- As  increases...

- Island 1

- Island 2

- But still, each island’s forecast is no better than
- a random draw from its climatology.  Expect no skill.

## Skill with conventional methods of calculation

- Reference climatology implicitly becomes
*N*(+,1) + *N*(–,1)       not      *N*(+,1) OR *N*(–,1)

## The new implicit reference climatology

## Related problem when means are the same but climatological variances differ

**Event**:  v > 2.0
**Island 1**: f ~ N(0,1),   v ~ N(0,1),   Corr (f,v) = 0.0
**Island 2**: f ~ N(0,),   v ~ N(0,), 1 ≤ ≤ 3,   Corr (f,v) = 0.9
**Expectation**: positive skill over two islands, but not a function of 

## the island with the greater climatological uncertainty of the 

- observed event ends up dominating the calculations.

- _[more](slide27.xml)_

## Are standard methods wrong?

**Assertion**: we’ve just re-defined climatology, they’re the correct scores with reference to that climatology.
**Response**: You *can* calculate them this way, but you shouldn’t.
- You will draw improper inferences due to “lurking variable” - i.e., the varying climatology should be a predictor.
- Discerning real skill or skill difference gets tougher

- “*One method that is sometimes used is to combine all the *
*data into a single 2x2 table ... this procedure is legitimate *
*only if the probability ****p**** of an occurrence (on the null *
*hypothesis) can be assumed to be the same in all the *
*individual 2x2 tables.  Consequently, if ****p**** obviously *
*varies from table to table, or we suspect that it may vary, *
*this procedure should not be used.”*
*	**W. G. Cochran, 1954, discussing ANOVA tests*

## Solutions ?

-    (1) Analyze events where climatological probabilities are the same at all locations, e.g., terciles.

## Solutions, continued

- (2) Calculate metrics separately for different points with different climatologies.  Form overall number using sample-weighted averages

*ROC*:

## Real-world examples: (1) Why so little skill for so much reliability?

- These reliability diagrams formed from locations with different
- climatologies.  Day-5 usage distribution not much different from
- climatological usage distribution (solid lines).

## Degenerate case:

- Skill might
- appropriately
- be 0.0 if all
- samples with
- 0.0 probability
- are drawn from
- climatology with
- 0.0 probability,
- and all samples
- with 1.0 are
- drawn from
- climatology with
- 1.0 probability.

## (2) Consider Equitable

- Threat Scores...

## (2) Consider Equitable

- Threat Scores...
- ETS location-dependent,
- related to climatological
- probability.

## (2) Consider Equitable

- Threat Scores...
- ETS location-dependent,
- related to climatological
- probability.
- (2) Average of ETS at
- individual grid points = 0.28

## (2) Consider Equitable

- Threat Scores...
- ETS location-dependent,
- related to climatological
- probability.
- (2) Average of ETS at
- individual grid points = 0.28
- (3) ETS after data lumped into
- one big table = 0.42

## Equitable Threat Score:alternative method of calculation

- Consider the possibility of different regions with different
- climates. Assume *n**c* contingency tables, each
- associated with samples with a distinct climatological
- event frequency.  *n*s(*k*) out of the *m* samples were
- used to populate the *k*th table.  ETS calculated separately
- for each contingency table, and alternative, weighted-
- average ETS is calculated as

## ETS calculated two ways

## Conclusions

- Many conventional verification metrics like BSS, RPSS, threat scores, ROC, potential economic value, etc. can be overestimated if climatology varies among samples.
- results in false inferences: think there’s skill where there’s none.
- complicates evaluation of model improvements; Model A better than Model B, but doesn’t appear quite so since both inflated in skill.
**Fixes**:
- Consider events where climatology doesn’t vary such as the exceedance of a quantile of the climatological distribution
- Combine after calculating for distinct climatologies.
**Please**: Document your method for calculating a score!

**Acknowledgements**:  Matt Briggs, Dan Wilks, Craig Bishop, Beth Ebert,
- Steve Mullen, Simon Mason, Bob Glahn, Neill Bowler, Ken Mylne, Bill Gallus,
- Frederic Atger, Francois LaLaurette, Zoltan Toth, Jeff Whitaker.