# Presentation: 550261

## Verifying NCEP NAEFS ensemble and probabilistic forecasts

- Dr. Zoltan Toth and Dr. Bill Bua
- Session 4: 10:30 a.m. 10 October 2007

## Outline

- The need for evaluation and verification
- What attributes do we verify in ensemble forecasts?
- Probability distributions and their reliability and resolution
- Verification concepts
- Tools for verification (concentration on probability)
- Reliability, resolution, and combined verification

## Forecast Evaluation 

**Statistical approach**
- Evaluates sample set of forecasts and not a single forecast
- Interest in comparing *forecast systems*
- Forecasts generated by same procedure
- Size of sample set affects how fine stratification is possible
- Size of observational record limits sample size, even for what are called hindcasts

## Forecast Evaluation (con’d)

**Types**
- Forecast statistics
- Depends only on forecast properties
- Verification
- Comparison of forecast to proxy for “truth” in statistical sense
- Depends on both natural and forecast systems
- Nature represented by “proxy”
- Observations (including observational error)
- Numerical analysis (including analysis error)

## Forecast Verification

- Types
- Measures of quality
- Meteorology issues (main focus here)
- Measures of utility
- Multidisciplinary
- Social & economic issues, beyond environmental sciences
- Socio-economic value of forecasts is ultimate measure
- Approximate measures can be constructed
- Quality vs. utility
- Improved quality
- Generally permits enhanced utility (assumption)
- How to improve utility if quality is fixed?
- Providers make all information that can be made available known
- E.g., offer probabilistic or other information on forecast uncertainty
- Engage in education, training
- Users identify forecast aspects important to them
*Can providers selectively improve certain aspects of forecasts?*
- E.g, improve precipitation forecasts without improving circulation forecasts?

## Evaluating Quality of Forecast Systems

- Compare information from *forecast systems*
- Value added by
- New methods
- Subsequent steps in end-to-end forecast process (eg., manual changes)
- Critical for monitoring and improving operational forecast systems
- Attributes of forecast systems
- Usually defined separately for each forecast format
- Need to compare forecasts
- From any system and of any type / format
- Single, ensemble, categorical, probabilistic, etc.
- Supports systematic evaluation of end-to-end (provider-user) forecast process
- Statistical post-processing as integral part of system

## Outline

- The need for evaluation and verification
- What attributes do we verify in ensemble forecasts?
- Probability distributions and their reliability and resolution
- Verification concepts
- Tools for verification (concentration on probability)
- Reliability, resolution, and combined verification

- Questions?

## But First, A Word About Standard Verification Measures

- Anomaly correlation scores from summer 2006-07 for 500 hgts
- GFS=T382L64 operational
- CTL=T126L28 ensemble low-resolution control
- ENS=ensemble mean

- Ens mean beats T382 GFS past day 2

- traditional skill/no skill AC threshold
- AC = 0.60
- Gain about 1 day of skill

## But First, A Word About Standard Verification Measures

- Anomaly correlation scores from winter 2006 for 500 hgts
- GFS=T382L64 operational
- CTL=T126L28 ensemble low-resolution control
- ENS=ensemble mean

- Ens mean beats T382 GFS past day 3

- traditional skill/no skill AC threshold
- AC = 0.60
- Gain of a bit less than 1 day of skill

## What are the attributes of  ensemble forecasts?

- Probability distributions!
- Two different attributes to evaluate for verification of probability distributions
- Reliability
- How much forecast looks like “nature” as represented by analysis or observations
- Resolution
- Ability of the forecast to distinguish between different forecast outcomes in the future
- Examples
- A forecast of an extreme event at a particular forecast lead time will likely lead to the occurrence of that extreme event at that lead time

## Statistical Reliability – Time Aggregation

**BACKGROUND:**
- Consider particular forecast class – *F**a*
- Consider frequency distribution of obs that follow forecasts  *O**a *
**DEFINITION:**
***The forecast system is perfectly reliable if ...***
- Forecast *F**a* has the exact same *form* as *O**a*, for all forecast classes, and
- Forecast system is *statistically consistent* with observations (all frequencies Fa’s equal frequencies for *O**a* ’s
**MEASURES OF RELIABILITY:**
- Based on different ways of comparing *F**a* and *O**a*

**EXAMPLES:**
*             **CONTROL FCST*

*ENSEMBLE*

- below
- normal

- normal

- above
- normal

## Statistical Resolution – Time Evolution

**BACKGROUND****:**
- Assume observed events are classified into finite number of classes, like *O**a*
**DEFINITION****:**
*The forecast system has perfect resolution if*
- All observed classes (*O**a**, O**b*,...) are preceded by
- Distinctly different forecasts (*F**a**, F**b*,...), such that...
- The forecast system “resolves” the problem
**MEASURES OF RESOLUTION:**
- Based on degree of separation of obs that follow diff forecast classes
- Measured by difference between fdo’s & climate distribution
- Measures differ by how differences between distributions are quantified

*FORECASTS*

*OBSERVATIONS*

**EXAMPLES**

## Characteristics of Reliability and Resolution

**Reliability****  **
- Related to *form of forecast*, not forecast content
- Reproduces nature when resolution is perfect, forecast looks like nature
*Not related to time sequence of forecast/observed systems*
- How to improve?
- Make model more realistic (also expected to improve resolution)
- Statistical bias correction: Can be statistically imposed at one time level
- If both natural & forecast systems are stationary in time and ...
- If there is a large enough set of observed-forecast pairs
- Link with verification through replacement of forecast with  corresponding observational forecast distribution
**Resolution**** **
- Related to *inherent predictive value* of forecast system
- Not related to form of forecasts
- Statistical consistency at one time level (reliability) is irrelevant
- How to improve?
- Enhanced knowledge about time sequence of events
- More realistic numerical model should help
- May also improve reliability

## Outline

- The need for evaluation and verification
- What attributes do we verify in ensemble forecasts?
- Probability distributions and their reliability and resolution
- Verification concepts
- Tools for verification (concentration on probability)
- Reliability, resolution, and combined verification

- Questions?

## Verification Concepts: Categorical and Probabilistic Forecasts

*Categorical* forecasts
- One out of a set of events either will or will not occur.
- Prob of the event is either 0% or 100%.
- Can be multiple categories (e.g. precipitation categories of 0-T, 0.01-0.1”, 0.1-0.25”, etc.)
- Only two possible outcomes  dichotomous
- E.g. Probability of exceedance forecasts
- Verification of categorical forecasts  "hit rates" and "false alarm rates”
*Probabilistic* forecasts  ***likelihood*** of a weather event or events
- Express degree of uncertainty
- Variable can take on discrete or continuous values
- Continuous variables must be restated in discrete ranges for probabilities to be calculated.
- Probabilities always range from 0-100%.
- Can treat probabilistic forecasts as categorical forecasts by choosing a threshold and using it for defining an event.
- E.g. Probability over range of heat index values converts to Heat index threshold of 105°, exceeding value is a “yes” event

## Verification Concepts: Skill

- Comparison to some reference or baseline
- Climatology
- Persistence
- Static NWP model score
- General Formula

**Skill score =**** **** **_**forecast accuracy **__**─**__** reference accuracy**_
**                    ****perfect accuracy ****─**** reference accuracy**

## Operational Probabilistic/Ensemble Forecast Verification

- Requirements
- Use same general dimensionless probabilistic measures for verifying any event against either observations or analysis
- Measures used at NCEP
- Probabilistic forecast measures – ensemble interpreted probabilistically
- Reliability
- Component of Brier Skill Score (BSS), Rank Probability Skill Score (RPSS), Continuous RPSS (CRPSS)
- Attributes and Talagrand diagrams
- Resolution
- Component of BSS, RPSS, CRPSS
- Relative Operating Characteristics (ROC), attributes diagram, potential economic value
- Special ensemble verification procedures
- Designed to assess performance of finite set of forecasts
- Most likely member statistics, Perturbation vs. Error Correlation Analysis (PECA)

## Forecast Performance Measures

**  ****MEASURES OF RELIABILITY:**** **
*DESCRIPTION: S*tatistically compares ***any sample of forecasts with sample of corresponding observations***
*GOAL: *To assess similarity of samples (e.g., whether 1st and 2nd moments match)
*EXAMPLES:*
- Reliability component of
- Brier Score
- Ranked Probability Score
- Analysis Rank Histogram
- Spread vs. Ens. Mean error
- Etc.

**COMMON CHARACTERISTIC:**	Function of both forecast and observed values

## Forecast Performance Measures

**MEASURES OF RESOLUTION:**
*DESCRIPTION:  *Compares the ***distribution of observations that follows different classes of forecasts with the climate distribution (as reference)****** ***
*GOAL:  *To assess how well the observations are separated when grouped by different classes of preceding fcsts
*EXAMPLES:*
- Resolution component of
- Brier Score
- Ranked Probability Score
- Information content
- Relative Operational Characteristics (ROC)
- Relative Economic Value
- Etc.

**COMBINED (REL+RES) MEASURES:** Brier, Ranked Probab. Scores, rmse, PAC, etc

## Reliability > resolution

**Reliability > resolution**

## EXAMPLE – PROBABILISTIC FORECASTS 

**EXAMPLE – PROBABILISTIC FORECASTS **
**RELIABILITY:**
- Forecast probabilities for given event match observed frequencies of that event (with given prob. fcst)
**RESOLUTION:**
- Many forecasts fall into classes corresponding to high or low observed frequency of given event
- (Occurrence and non-occurrence of event is *well resolved* by fcst system)

## Probabilistic Forecast Performance Measures

**Probabilistic Forecast Performance Measures**

- Assess *reliability and resolution*
- Example: Brier Skill Score (BSS)
-  Combines reliability and resolution

- Total of *n* pairs of cases
*N**k* cases with *f**k* probability

- Observation *o**i*
- Forecast prob *f**i*

- reliability

- resolution

- uncertainty

## BRIER SKILL SCORE (BSS)

- 	**METHOD:**
- Compares pdf against analysis
- Resolution (random error)
- Reliability (systematic error)
- 	**EVALUATION**
- BSS		Higher better
- Resolution	Higher better
- Reliability	Lower better
**	****RESULTS**
- Resolution dominates initially
- Reliability becomes important later
**ECMWF** best throughout
- Good analysis/model?
**NCEP** good days 1-2
- Good initial perturbations?
- No model perturb. hurts later?
**CANADIAN** good days 8-10
- Model diversity helps?

- May-June-July 2002 average Brier skill score for the EC-EPS (grey lines with full circles), the MSC-EPS (black lines with open circles) and the NCEP-EPS (black lines with crosses). Bottom: resolution (dotted) and reliability(solid) contributions to the Brier skill score. Values refer to the 500 hPa geopotential height over the northern hemisphere latitudinal band 20o-80oN, and have been computed considering 10 equally-climatologically-likely intervals (from Buizza, Houtekamer, Toth et al, 2004)

- COMBINED MEASURE OF RELIABILITY AND RESOLUTION

- Resolution

- Reliability

## BRIER SKILL SCORE

**BRIER SKILL SCORE**

- COMBINED MEASURE OF RELIABILITY AND RESOLUTION

## RANKED PROBABILITY SCORE

**RANKED PROBABILITY SCORE**

- COMBINED MEASURE OF RELIABILITY AND RESOLUTION

## ANALYSIS RANK HISTOGRAM   (TALAGRAND DIAGRAM)

**ANALYSIS RANK HISTOGRAM   (TALAGRAND DIAGRAM)**

- MEASURE OF RELIABILITY

## ENSEMBLE MEAN ERROR VS. ENSEMBLE SPREAD

- Statistical consistency between the ensemble and the verifying analysis means that the verifying analysis should be statistically indistinguishable from the ensemble members =>
- Ensemble mean error (distance between ens. mean and analysis) should be equal to ensemble spread (distance between ensemble mean and ensemble members)

- MEASURE OF RELIABILITY

- In case of a *statistically consistent ensemble*, ens. spread = ens. mean error, and they are both a MEASURE OF RESOLUTION. In the presence of bias, both rms error and PAC will be a combined measure of reliability and resolution

## INFORMATION CONTENT

**INFORMATION CONTENT**

- MEASURE OF RESOLUTION

## RELATIVE OPERATING CHARACTERISTICS

**RELATIVE OPERATING CHARACTERISTICS**

- MEASURE OF RESOLUTION

## ECONOMIC VALUE OF FORECASTS

**ECONOMIC VALUE OF FORECASTS**

- MEASURE OF RESOLUTION

## PERTURBATION VS. ERROR CORRELATION ANALYSIS (PECA)

*MULTIVARIATE COMBINED MEASURE OF*
*RELIABILITY & RESOLUTION*

**METHOD****:** Correlate ens perts and error in control fcst for
- Individual members
- Optimal combination of members
- Each ensemble
- Various areas, all lead time
**EVALUATION:** Large corr indicates ens captures error in control forecast
- Caveat – errors defined by analysis
**RESULTS:**
**Canadian** best on large scales
- Benefit of model diversity?
**ECMWF** gains most from combinations
- Benefit of orthogonalization?
**NCEP** best on small scale, short term
- Benefit of breeding (best estimate initial error)?
- PECA increases with lead time
- Lyapunov convergence
- Nonlinear saturation
- Higher values on small scales

## Outline

- The need for evaluation and verification
- What attributes do we verify in ensemble forecasts?
- Probability distributions and their reliability and resolution
- Verification concepts
- Tools for verification (concentration on probability)
- Reliability, resolution, and combined verification

- Questions?

## Summary and Conclusions

- The need for evaluation and verification
- Statistical approach is used for probabilistic forecasts
- The bigger the sample size, the better
- Choice of what to use as “truth” is important
- Forecast attributes to verify
- Reliability:  do probabilities of forecast reproduce those of “nature”
- Not related to matching order of individual events
- Resolution:  ability of forecasts to reproduce matched events
- Related directly to matching order of individual events

## Summary and Conclusions

- Verification concepts
- Categorical versus probabilistic forecasts
- Probabilistic forecasts can be converted to categorial forecasts
- The concept of skill
- Verification tools
- The reliability/attributes diagram
- Reliability
- Brier Score/Skill Score, ranked probability skill score, rank histogram (Talagrand diagram), ensemble mean error versus ensemble spread,
- Resolution
- Information content, relative operating characteristics, economic value, perturbation versus error correlation analysis

## Extra Slides

## Verification Concepts: Hit Rates, False Alarm Rates, and Contingency Tables

- Hit = event or non-event is forecast correctly ("yes/yes" and "no/no“ cells)
- False alarm = event is forecast, but not observed
*Hit rate* :
- Ratio of all correct forecasts ("yes/yes" and "no/no") to **all **forecasts
*False alarm rate*:
- Ratio of false alarms to the total % of times an event was** **forecast
*Probability of Detection (POD)*
- Ratio of correct “yes” forecasts to “yes” events.
*Threat Score*
- Ratio of correct “yes” forecasts to

**Hit rate =(0.40+0.35)/1.00 = 0.75**
**POD = 0.35/0.50 = 0.70**

**False alarm rate = 0.10/0.45 = 0.22**

## Verification Concepts: Categorical and Probabilistic Forecasts

*Categorical* forecasts
- One out of a set of events either will or will not occur.
- Prob of the event is either 0% or 100%.
- Can be multiple categories (e.g. precipitation categories of 0-T, 0.01-0.1”, 0.1-0.25”, etc.)
- Only two possible outcomes  dichotomous
- E.g. Probability of exceedance forecasts
- Verification of categorical forecasts  "hit rates" and "false alarm rates”
*Probabilistic* forecasts  ***likelihood*** of a weather event or events
- Express degree of uncertainty
- Variable can take on discrete or continuous values
- Continuous variables must be restated in discrete ranges for probabilities to be calculated.
- Probabilities always range from 0-100%.

## Verification for Categorical Forecasts

- Brier Score (BS)
- Applied to categorical forecasts
- Value of 1 assigned to category for forecast and verification
- Subtract verification from forecast (zero is a hit), square, and divide by number of forecast verification pairs
- Zero is perfect, one is worst score possible

## Verification of Categorical Forecasts

- Brier Skill Score (BSS)
- Compare Brier Score for forecast to Brier Score for benchmark (e.g. climatology)

**            ****Brier Skill score =**** **** **_**BS(forecast) **__**─**__** BS(climatology)**_
**                                           ****0 ****─**** BS(climatology)**

## Relative Operating Characteristics Diagram (ROC)

- Compares hit rate and false alarm rate
- Diagonal
- Hit rate=false alarm rate  no skill
- Increasing distance from diagonal means increasing skill

## How well fcst probs map into (i.e. match) obs freqs is called reliability
Diagrams showing such a mapping are called reliability diagrams
Graphic shows days 1, 4, 7, and 10 forecast probabilities of 0.5” 24-hr accum. precip. vs. obs frequency.
Additional lines are:
Diagonal solid line is perfect reliability
Climatological occurrence  (horizontal line, no resolution)
Skill/no-skill threshold (dashed line 1⁄2 the distance between perfect reliability and climatology)

- Diagrams showing such a mapping are called reliability diagrams
- Graphic shows days 1, 4, 7, and 10 forecast probabilities of 0.5” 24-hr accum. precip. vs. obs frequency.
- Additional lines are:
- Diagonal solid line is perfect reliability
- Climatological occurrence  (horizontal line, no resolution)
- Skill/no-skill threshold (dashed line 1⁄2 the distance between perfect reliability and climatology)

- Probabilistic Verification Tools: Reliability Diagram

**Notes:**
[start with title only]
The second concept we will discuss is called “reliability”.  Reliability [bring in 1st bullet] is the measure of how well forecast probabilities map into, or match, observed frequencies over a verification period.  

[bring in 2nd bullet]
Diagrams comparing forecast probabilities versus observed frequencies are called, appropriately enough, reliability diagrams, and .  
[bring in sub-bullet] The diagram [bring in graphic] was produced by the Hydrological Prediction Center (HPC), and shows the forecast probability on the horizontal axis, versus the observed frequency on the vertical axis, of 0.5” accumulated precipitation in 24-hours at [highlight each line] 1, 4, 7, and 10 days over the continental U.S.

[bring in 3rd bullet]
There are three additional lines on the diagram.  
[bring in 1st sub-bullet]
The solid diagonal [highlight solid diagonal line] represents perfect reliability, where all forecast probabilities match identical observed frequencies. 
[bring in 2nd sub-bullet]
The long-short dashed horizontal line represents the climatological probability of the event in question, 24-hour precipitation amounts exceeding 0.5” over the CONUS.  That probability here is a shade over 0.1.  Remember that varying forecast probabilities all mapping into the climatological observed frequency represents a forecast with no resolution, so this also represents a “no-resolution” line.
[bring in 3rd sub-bullet]
Finally, the dashed line between the climatology and perfect resolution represents a skill/no-skill threshold, and lies half-way between perfect reliability and climatology.  For more details, you can go to the module and review the verification section on the Brier Score.

## Rank Histogram (Talagrand Diagram)

- Answers the question: do we cover all possible outcomes?
- Rank all forecast values from lowest to highest to create n+1 bins (n is number of ens members)
- Place verification in proper bin
- Perfect ensemble will have same % in each bin, but usually do not.
- U-shape:  not enough spread or “underdispersive”
- Humped: too much spread or “overdispersive”
- L-shape (too many in bottom bins):  high bias
- J-shaped (too many in top bins): low bias

- U

- Hump

- L

- J

**Notes:**
[start with title only]
Another way to examine the probability distribution is to look at how frequently the observed value falls within or outside the range or “envelope” of ensemble forecasts. We do that with what is called a Rank Histogram or Talagrand Diagram.  This is typically done over a large domain for a sensible verification period, such as over the Northern Hemisphere for a 30-day or seasonal period.
[bring in 1st bullet] 
The Rank Histogram answers the question: does our ensemble forecast system cover all the forecast outcomes with its members?  How is it constructed?  Part of the answer is in the name.  We first [bring in 1st sub-bullet] rank the forecast variables from lowest to highest, and use the values to create n+1 bins, where n is the number of ensemble members. The upper and lower bins are open-ended.
[bring in 2nd sub-bullet]
Then, each observed value is placed into the proper bin.  
[bring in 3rd sub-bullet]
Theoretically, over the long run, an equal number of observations should appear in each bin, including the open-ended top and bottom bins.  This is rarely seen in rank histograms, however.
More typically, we see one of 4 shapes in the histogram:
[bring in 4th sub-bullet]
For a [bring in U graphic] U-shaped histogram, where more members than theoretically expected fall in the low and high value bins with fewer than expected in the middle value bins, this indicates insufficient ensemble spread.  This is representative of what is called an “underdispersive” ensemble.
[bring in 5th sub-bullet]
For a [bring in Hump graphic] hump-shaped histogram, where more members than theoretically expected fall into the middle value bins and fewer than expected fall into the high and low value bins,  this is indicative of too much ensemble spread.  Ensemble prediction systems with this type of error are called “overdispersive”.
[bring in 6th bullet]
For L- [bring in L graphic] shaped graphics, where more observations than expected fall into low value bins, and less values than expected fall into high value bins, the ensemble prediction system is said to have a high bias.
[bring in 7th bullet]
For J- [bring in J graphic] shaped graphics, where less observations than expected fall into low value bins respectively, and more values than expected fall into high value bins, the ensemble prediction system is said to have a low bias.

## Rank Histogram (Talagrand Diagram)

**Rank Histogram (Talagrand Diagram)**

- U

**Hump**

- L

- J

- Black is verification distribution, red the ensemble distributions associated with Talagrand diagrams
- Each non-flat Talagrand diagram is usually associated with deficiencies in the model underlying the EPS

**Values**

**Values**

**Probability Distributions**

- underdispersive

- overdispersive

- Values

- high bias

- Values

- low bias

**Notes:**
[start with title only]
Another way to examine the probability distribution is to look at how frequently the observed value falls within or outside the range or “envelope” of ensemble forecasts. We do that with what is called a Rank Histogram or Talagrand Diagram.  This is typically done over a large domain for a sensible verification period, such as over the Northern Hemisphere for a 30-day or seasonal period.
[bring in 1st bullet] 
The Rank Histogram answers the question: does our ensemble forecast system cover all the forecast outcomes with its members?  How is it constructed?  Part of the answer is in the name.  We first [bring in 1st sub-bullet] rank the forecast variables from lowest to highest, and use the values to create n+1 bins, where n is the number of ensemble members. The upper and lower bins are open-ended.
[bring in 2nd sub-bullet]
Then, each observed value is placed into the proper bin.  
[bring in 3rd sub-bullet]
Theoretically, over the long run, an equal number of observations should appear in each bin, including the open-ended top and bottom bins.  This is rarely seen in rank histograms, however.
More typically, we see one of 4 shapes in the histogram:
[bring in 4th sub-bullet]
For a [bring in U graphic] U-shaped histogram, where more members than theoretically expected fall in the low and high value bins with fewer than expected in the middle value bins, this indicates insufficient ensemble spread.  This is representative of what is called an “underdispersive” ensemble.
[bring in 5th sub-bullet]
For a [bring in Hump graphic] hump-shaped histogram, where more members than theoretically expected fall into the middle value bins and fewer than expected fall into the high and low value bins,  this is indicative of too much ensemble spread.  Ensemble prediction systems with this type of error are called “overdispersive”.
[bring in 6th bullet]
For L- [bring in L graphic] shaped graphics, where more observations than expected fall into low value bins, and less values than expected fall into high value bins, the ensemble prediction system is said to have a high bias.
[bring in 7th bullet]
For J- [bring in J graphic] shaped graphics, where less observations than expected fall into low value bins respectively, and more values than expected fall into high value bins, the ensemble prediction system is said to have a low bias.