# Presentation: 917417

## Ensemble Verification

- Yuejian Zhu
- Environmental Modeling Center
- NOAA/NWS/NCEP
- Acknowledgements:
- Zoltan Toth*** EMC***

## Outlines

- Climatological Data
- RMS and Spread
- Mean Error and Absolute Error
- Histogram and Outlier
- RPS and RPSS
- CRPS and CRPSS
- BSS (Resolution and Reliability)
- ROC (Hit Rate and False Alarm Rate)
- Economic Value (cost-loss analysis)

## Climatological Data

- NCEP/NCAR 40 years (1958-1997) reanalysis
- Monthly Sampling
- For example: 40*30=1200
- 10 equally-a-likely, based on sampling
- Projected to verify date
- All forecast skills will base on 10 equally-a-likely climatological bins.

## Due to model imperfection ?

- One day advantage

## Winter 0607 NAEFS Statistics

- _Winter 0607 NAEFS Statistics_

- _BIAS_

## Prob. Evaluation (simple measurement)

- 1. Talagrand Distribution (histogram distribution):
-       Sorting forecast in order, to check where the analysis is falling
-       ***Reliability measurement, system bias detected***.
-       positive/negative biased for forecasting model,
-       example of these forecasts --> cold bias,
-       assume analysis is bias-free (perfect). Common -"U" sharp

_***avg distribution***_

## Slide 7

## Prob. Evaluation (simple measurement)

- 1. Talagrand distribution (continue).
-         . Outlier evolution by different leading time
-         .. Adding up two outliers subtract the average.
-         ... Ideal forecasts will have zero outliers.

_***Due to inability of ensemble to capture model related errors?***_

## Prob. Evaluation (simple measurement)

-  Outlier --> diagnostic
-         forecasts .vs. next forecasts ( f+24hrs valid at same time)
-         assume forecasting model is perfect, f+24.
-         perfect forecast system will expect the outliers are zero.

_***Detecting model initial uncertainty?***_

## Prob. Evaluation (multi-categories)

***Based on climatological equally likely bins ( for example. 5 bins )***
***For verifying multi-category probability forecasts.***
***       ******measure both reliability and resolution.***
- 1. Ranked (ordered) probability score ( RPS) and RPSS
-     ***RPSS=( RPSf - RPSc )/( 1 - RPSc )***

## What is THORPEX’s goal for 10 years ?

- _What is THORPEX’s goal for 10 years ?_

## Continuous Rank Probability Score 

- 0%

- 100%

- 50%

- p07

- p09

- p08

- p06

- p03

- p02

- p01

- p04

- p05

- p10

- _Obs (truth)_

- _Continuous Rank Probability Score__ _

- X

- Xo

- Heaviside Function H

- Order of 10 ensemble members (p01, p02,...,p10)

## CRPSS for winter 0607

- _CRPSS for winter 0607_

- _CRPS for winter 0607_

## Prob. Evaluation (multi-categories)

- 2. Brier Score(BS, non-ranked), Brier Skill Score(BSS).
-     from two categories to multi-categories/probabilistic
-     ----measure both reliability and resolution

_***Brier Skill Score***_

_***Skill line (ref. is climatology)***_

## Prob. Evaluation (multi-categories)

- 3. Decomposition of Brier Score:
-     consider _***sub-sample***_ and _***overall-sample***_
-     reliability, resolution and uncertainty.
-     for reliability: 0 is perfectly reliable
-     for resolution: 0 is no resolution ( = climatology )
-     when resolution = reliability  no skill
-     example of global ensemble:

_***resolution***_

_*No skill beyond this point*_

_***reliability***_

## Prob. Evaluation (multi-categories)

- 4. Reliability and possible calibration ( remove bias ):
-     For period precipitation evaluation

_*Calibrated forecast*_

_*Raw forecast*_

_*Skill line*_

_*Resolution line*_
_*Climatological prob.*_

## Prob. Evaluation (multi-categories)

- 4. Reliability and possible probabilistic calibration:
-    *** ******re-label fcst prob by obs frequency associated with fcst***

_***calibrated***_

_***Un-calibrated***_

## Prob. Evaluation (cost-loss analysis)

- Based on hit rate (HR) and false alarm (FA) rate.
- 1. Relative Operating Characteristics (ROC) area -*** ******Appl. of signal detection theory for measuring discrimination between two alternative outcome.***
-     ***ROCarea = Intergrated area * 2  *** ( 0-1 normality )

_*Relative Operating Characteristics*_

- --------------------------        |  o\f   |  y(f)  |  n(f)  |
- --------------------------
- | y(o)  |   h    |   m   |
- --------------------------
- | n(o)  |   f     |   c    |
- --------------------------

- _h/(h+m)_

- _f/(h+f)_

## Relative Operating Characteristics area (ROC area)

- _Relative Operating Characteristics area (ROC area)_

_**f(noise)**_

_**f(signal)**_

- False alarm rate

- 0

- 1

- 1

- Decision threshold

- Hit rate

- _Near perfect forecast_

- _No skill forecast_

- _Real forecast_

## Slide 20

## Prob. Evaluation (cost-loss analysis)

- 2. Economic Value (EV) of forecasts.
-     ***Given a particular forecast, a user either does or does not take  action***

_*Deterministic forecast*_

_*Ensemble forecast*_

_*Highest value (1:10)*_

_*Value line*_

## Prob. Evaluation (cost-loss analysis)

- Based on hit rate (HR) and false alarm (FA) analysis
- .. Economic Value (EV) of forecasts

_*Ensemble forecast*_

_*Deterministic forecast*_

_*Average 2-day advantage*_

## Decision Theory Example

- Critical Event: sfc winds > 50kt
- Cost (of protecting):   $150K
- Loss (if damage ):  $1M

*Hit*
*False*
*Alarm*

***Miss***
*Correct*
*Rejection*

**YES          NO**

**YES **
**NO**

- _Forecast?_

- _Observed?_

_**Decision Theory Example**_

- $150K

**$1000K**

- $150K

- $0K