# Presentation: 745388

## Introduction to probabilistic ensemble forecast products

**Introduction to probabilistic ensemble forecast products**

- Session 3, 9 October 2007, 9:00 a.m.
- Bill Bua, UCAR/COMET
- and
- Rich Grumm, NOAA/NWS

- Major acknowledgment:
- Canadian Met. Center for materials
- adapted for this lecture

**Notes:**
The next section discusses how we get good information from an ensemble prediction system.  Think about trying to visualize having to look at graphic showing all the data from 10, 15, or 50 ensemble member forecasts valid at the same time.  You’d have a hard time figuring out what the forecast even was, let alone assessing its quality.  We’ll now talk about some forecast tools unique to ensemble prediction systems, that distill the huge amounts of data in the ensemble prediction system into a useful form.

## Outline

- Taking advantage of ensemble forecasts ... but keeping in mind the limitations
- Products revealing details of the PDF
- Plan view
- Mean and spread
- Spaghetti diagrams
- Probability of exceedance
- Dominant forecast
- Thumbnails
- Clusters
- Grid point diagrams
- Plume diagram
- Box and whisker

## Outline

- Predictability measures
- Plan View
- Relative measure of predictability
- Normalized spread (versus spread)
- Number of outliers
- Confidence diagram
- Grid point diagrams
- Confidence plume

## Taking advantage of ensemble prediction

- Value of EPS resides in probabilistic and pdf forecast:
- Can provide estimates of probabilities as measure of forecast uncertainty
- Can provide estimates of pdf
- Spread-skill relationship can be used to assess uncertainties, confidence and predictability

## Keep in mind...

- Limitations:
- Sampling initial condition uncertainties and model uncertainties not perfect.
- Members are model integrations:
- With their own deficiencies / biases
- In multi-model EPS, some members may be more skilful than others
- Optimally, mbrs. should have resolution similar to deterministic model
- Each member subject to model jumpiness
- Each individual member has little value *per se* compared to the deterministic model:
- Lower quality and skill
- Lower resolution
- Initialized off perturbed initial conditions
- Perturbed boundary conditions and physics

## Keep in mind ...

- Limitations:
- It is pointless to try to choose the member of the day:
- Very difficult task particularly when the number of members is large.
- Preferable in this case to go with the higher resolution deterministic model.
- Membership may not be large enough to correctly depict tails of the distribution:
- Compromise between membership / resolution / available computer resources.
- The size of verification sample must be a function of number of members.
- Likely to have problems with low probability (rare) events – potentially high impact events.
- Not always calibrated and reliable:
- Requires a large verification sample for statistical post-processing – reforecast.

## Keep in mind...

- Limitations:
- The ensemble mean is a poor forecast when the distribution departs from a bell curve – though it can provide valuable information on forecast:
- It is likely to verify better than any individual member.
- And even better than the deterministic model.
- But this is partly because it is smoothed out – smaller variance.
- Provides wrong information in case of clustering or multi-modal distributions.
- Ensemble mean and median work better as forecasts when the distribution is near normal.
- Most EPS are generally under-dispersive – ensemble of members do not always include the truth.

**There is uncertainty on the estimate of uncertainty.**

## Why do we need to summarizeensemble data?

- EPS presents a challenge in post-processing
- Graphic shows a 500-hPa height forecast at 6 decameter intervals
- Each ensemble member has a different color
- Which member is which, and where are the features?
- This information must be simplified to be made useful in the forecast process.

**Notes:**
[start with title only]

All model data is postprocessed to some degree; regridding to different resolutions, deriving products such as precipitation type and stability parameters, and so on. [bring in 1st bullet] 
Ensemble forecast data presents an additional problem in developing useful forecast products; the sheer magnitude of data involved.

In this graphic [have graphic appear], we can see the problem quite clearly.  What is being shown [bring in 2nd bullet] is the 500-hPa heights in decameters, contoured at 10-dm intervals, for all 11 of the 00Z 19 November 2001 ensemble members in the EPS, for verification time 12Z 22 November 2001. 
[bring in 1st sub-bullet] Each member is colored differently, with the ensemble control in black.  [bring in 2nd sub-bullet] Even with the color coding, it’s difficult to assess the degree of certainty in the forecast features, or even which features exist!  

[bring in 3rd bullet]
We need to find a way to convey this and other information in such as way that it can be useful in the forecast process.  The rest of this section discusses how ensemble information is simplified to meet forecaster needs.  Each ensemble product used for this purpose has advantages and disadvantages; these will be covered in turn.

## Ensemble Mean and “Spread”

- Ensemble mean value for model variable *x* is given by
- Standard deviation or “spread” is given by

**Notes:**
[start with title]
We’ve already shown the ensemble mean for one contour in the last spaghetti diagram.  To review, [bring in 1st bullet] the ensemble mean is the average value [bring up meaneq.gif graphic] for the model output of interest for all ensemble members. 

[bring in 2nd bullet]
Standard deviation or “spread” is the square root of the average mean-squared-difference [flash up stdev.gif graphic] between the ensemble members and the ensemble mean. 

## Week 2 probabilistic mean temperature anomaly

- Produced globally and for N America
- Added Greenwich meridian (GM) and Equator (EQ) to graphic, outline for Africa in light blue dash
- Uses terciles from a 30-yr climatology
- Normal is 33% probability for each category
- NAEFS distribution
- Bias-corrected
- Greater than 50% for any one is shaded, *including* near normal

**GM**

**EQ**

- http://www.cpc.ncep.noaa.gov/products/predictions/short_range/NAEFS/

- T1   T2 thresholds

- 1st tercile

- 3rd tercile

## Upper and Lower Terciles

- Based on actual bias-adjusted NAEFS ensemble distribution
- For NAEFS’s 38 members:
- Lower tercile value interpolated at 12.67
- Weighted average of 1/3 ranked member 12, 2/3 ranked member 13
- Upper tercile at 25.33

-  T1        T2

- Missing information? Bimodal? Range?

- 1st tercile

- 3rd tercile

## The Ensemble Mean and Spread Diagram

- Ensemble mean and spread plot for same data as “crazy spaghetti” diagram
- Characteristics of mean
- The ensemble mean performs better on average than operational model on which it is based.  Why is this so?
- Because predictable features remain intact, less predictable features are smoothed out
- Characteristics of spread
- Allows assessment of uncertainty, since more spread means more uncertainty

**1**

**2**

**3**

**4**

**Notes:**
[start with heading/title only]
[bring in 1st bullet and 1st sub-bullet] In performance scores like the anomaly correlation and the root-mean-square error, the ensemble mean forecast, on average, performs better than operational forecast of even considerably higher resolution. Why is that?Before, we noted that the ensemble mean contour was smoother than the contour for the individual ensemble members. This is [bring in 2nd sub-bullet] because predictable features remain intact in the mean, while less predictable features tend to be smoothed out. 

[bring in 2nd bullet and associated sub-bullet]
The spread is useful in that it indicates regions where the atmosphere is less predictable since there is less agreement in the ensemble members.  

The graphic shown here [have mean and spread graphic come up] uses the same data as that old unreadable graphic, but rather than showing the contours for all the ensemble members, it shows the ensemble mean, contoured at 60-meter intervals; and spread, in meters, with values indicated by the shading.  Note the high values of spread (orange or ‘warmer’ colors) over [bring in circle #1] the west coast of the US, [bring in circle #2] the central plains, [bring in circle #3] the mid-Atlantic coast, and [bring in circle #4] near and east of Labrador.

## Advantages of ensemble mean and spread

- Conceptually easy to understand
- Compact presentation
- Calculation of ensemble mean removes unpredictable features
- Ensemble mean gives best forecast ***in general*** after about 84 hours
- Spread lets us know how much uncertainty there is in the forecast
- Low spread has been shown to indicate high probability that the ensemble mean is a good forecast

**Notes:**
[start with title]
Like all ensemble products, the goal of ensemble mean and spread plots is to present [bring in 1st bullet] the most information possible in a coherent, understandable way.  By showing the ensemble mean and spread on the same diagram, we have compactly presented the ensemble data with the following additional advantages:

 [bring in 2nd bullet] The features in the flow that are unpredictable are smoothed out.  As a result, the ensemble mean [bring in associated sub-bullet] generally gives the best forecast after about 84 hours.
 [bring in 3rd bullet] The ensemble spread gives us a picture of how much uncertainty there is in the forecast; [bring in associated sub-bullet] low spread indicates that there is a higher than usual probability that the ensemble mean is a good forecast.

## Disadvantages of ensemble mean and spread

- Mean/spread work when forecasts are normally distributed
- Chaotic behavior and clustering around multiple solutions
- Ensemble mean will not represent the best solution
- Ensemble spread not meaningful in the traditional sense (i.e. 67% between ± 1σ won’t be valid in such cases)
- Interpretation of ensemble mean and spread requires information on ensemble distribution*** shape***
- Spaghetti diagrams help to infer the shape of a distribution about the ensemble mean
- Size of spread with respect to ensemble mean features helps infer the spatial uncertainty of the feature

**Notes:**
[start with title/heading]
Disadvantages accrue in using the ensemble mean and spread, however, when [bring in 1st bullet] the ensemble members are not normally distributed about the ensemble mean. Because the atmosphere [bring in 1st sub-bullet] is chaotic, there is a tendency for clustering around different forecast outcomes, particularly when the sensitivity to initial conditions is higher than usual.  If we do have clustering around, say, two solutions, [bring in 1st sub-sub-bullet] the ensemble mean will not be reflecting the most likely verification, but a highly unlikely solution between the two more likely clusters. 
[bring in 2nd sub-bullet] Additionally, the spread will not be representative because its calculation assumes that the data are normally distributed, though it will likely be higher than usual. What can we use to help us interpret the ensemble mean and spread? [bring in associated sub-sub-bullet] “Spaghetti” diagrams, the plotting of one or a few contours from all of the ensemble members, can help greatly, as we’ll now show.

## “Spaghetti” Plots

- Plots at most two or three contours for ease of readability (here 552 and 582 dm lines)
- Each member a different color
- Also ensemble mean contours in bold black
- Gives qualitative picture of distribution of 500-hPa heights near chosen contour
- Graphic to right plots same data as prior mean/spread plot
- Take a look at actual distribution at 44°S, 0°E (red closed circle)
- Bimodal distribution
- Spaghetti diagram indicates 4 members below the mean, rest at or above the mean

- c00	p01	p02	p03	p04
- p05	p06	p07	p08	p09
- p10	p11	p12	p13	p14
**ens mean**

**Notes:**
[start with title only]
One of the most simple yet informative diagrams that can be produced from ensemble prediction system forecasts is the so-called “spaghetti” diagram. It’s a simplification of the graphic we saw previously with all those ensemble member contours.  An example can be seen here [have graphic appear].  Believe it or not, this is a spaghetti diagram from the same ensemble run, for the same verification time as the previous unreadable graphic!

Spaghetti diagrams plot at most a few contours for ease of readability.  Here [bring in 1st bullet] we’ve only plotted one contour line, the 564 decameter height contour at 500-hPa, and as before, each ensemble member has its own unique contour color. [bring in sub-bullet] In this particular graphic, the ensemble mean contour for the 564-dm height is also shown.  We’ll be talking about the ensemble mean shortly. 

[bring in 2nd bullet]
From the position of each contour, we can get a qualitative idea of how the 500-hPa height from this ensemble run are distributed.  

[bring in 3rd bullet]
Off of the east coast of the U.S., for example, we can see that three ensemble members are indicating a deep trough [highlight maroon, yellow, and orange contours], four others show a moderate trough [highlight purple, light blue, yellow green, and red contours], and the final four contours show a weak trough [highlight medium blue, dark blue, green, and dark yellow contours] well offshore. The additional contour often found in spaghetti plots, is the ensemble mean contour. Here, this contour for the 564 decameter line is plotted with black and bold. Note that this contour is smoother than the others overall; a definition for the ensemble mean is and the reason for its relative smoothness compared to individual ensemble members will be discussed next.

## Advantages/Limitations of Spaghetti Diagrams

- Advantages
- All ensemble members are shown
- We can get an idea of the probability distribution for the forecast for the contours we display
- Compact presentation of information
- Disadvantages
- Still do not have a full picture of the forecast probability distribution
- Requires additional information for good interpretation of products

**Notes:**
Advantages [bring in first bullet] of the spaghetti diagram include the compact presentation of ensemble information [bring in 1st sub-bullet], and the showing of ALL ensemble members [bring in 2nd sub-bullet], rather than a summary.  As a result of seeing contours for all the ensemble members, the forecaster also gets an idea of [bring in 3rd sub-bullet] what the probability distribution for the variable of interest is, at least in the areas where the contours appear. 
There is a disadvantage [bring in 2nd bullet] to using spaghetti plots, however. We still do not have a complete picture [bring in 1st sub-bullet under second bullet] of the forecast probability distribution in other areas away from where the contours are plotted.  Additional information [bring in 2nd sub-bullet] is required to interpret the spaghetti diagrams well; we’ll see a good example of that with the use of the ensemble mean and spread maps below.

## Using Spaghetti and Mean/Spread Diagrams Together

- Use mean/spread diagrams to choose spaghetti contour containing the most information (i.e., crossing areas of maximum uncertainty)
- Based on ensemble mean/spread graphic, choose spaghetti diagram of 552 and 582-dm 500-hPa height contour
- Take a look at actual distribution at 44°S, 0°E (red closed circle)
- Spaghetti diagram indicates 4 members below the mean (three seem very close together), rest at or above the mean
- Bimodal distribution
- Combined use of mean/spread and spaghetti helps understand forecast distribution

- Histogram: Categories in 10
- equally likely climo bins

- M

**Notes:**
[start with title]
[bring in 1st bullet] We can use the mean and spread diagram [have mean/spread graphic, but no circles or numbers, appear] to decide which spaghetti diagram contour will give us the most information in the uncertain areas.  Those ensemble mean contours crossing high uncertainty regions will have spaghetti diagrams containing the most information on the high spread in those regions.

Here, we have relatively large uncertainty in regions 1, 2, and 3 [bring in circles and numbers in the first diagram in sequence as they’re said]. Regions 2 and 3 indicate an ensemble mean trough, while region 1 shows some ridging with a very strong height gradient. We also note that the 564 decameter 500-hPa ensemble mean height contour [highlight the 5640 meter height contour, and bring in 1st bullet] passes through these high-uncertainty regions. Now, we can go to a 5640-meter spaghetti diagram over the same general area at the same valid time. [bring in 2nd bullet, and bring in spaghetti graphic without circles and numbers]
In region 2 [have region 2 circle and number appear in the spaghetti plot and bring in 1st sub-bullet], we see that six ensemble members [highlight red, yellow, yellow-green, green, med-blue, and dark blue members] show deeper troughs than are indicated by the ensemble mean, three show a slower, weaker trough [highlight two purples and yellow-orange members], and two show almost no feature at all [blue-green and orange members highlighted].   In region 3 [have region 3 circle and number appear in the spaghetti plot and bring in 2nd sub-bullet], we also have a strength and location issue for a short-wave trough. Five members show a deeper trough than the ensemble average [highlight maroon, orange, purple, yellow, and light blue/teal members] in various locations both up- and downwind of the ensemble mean trough axis, three members are faster and about the same depth as the ensemble mean [highlight red, yellow-green and dark blue members], and three more are faster and shallower with their short-wave [highlight medium blue, green, and yellow-orange members].  Also, note that some members have a lead short-wave trough downwind of the depicted one.

Finally [bring in 3rd sub-bullet], we have a somewhat different situation in region 1 [have circle and 1 appear]. We’re not dealing with a trough position, but rather the location of a height gradient.We can see that three members are significantly further north of the ensemble mean 5640 meter height contour [highlight purple, orange, and teal members], while two or three [highlight green, dark blue, and red] are significantly further south. The others are tightly clustered around the ensemble mean.

[bring in 3rd bullet] 
From this illustration, we can see that the contours that pass through high spread regions will give you a good idea of the nature of the uncertainty. These different solutions have a significant impact on the expected weather over different regions of the CONUS.  Examining other fields will help sharpen up the forecast uncertainty in these areas. 

## Probability of Exceedance

-  Helps determine the probability of extreme events
-  Gives probability of exceeding meaningful threshold
-  Calculation represents count of what % of ensemble members exceed the threshold of interest
-  Adjustment may be made for systematic model bias
-  Example here is for probability of exceeding 20 kt winds at 10-m

**Notes:**
[start with title only]
[bring in 1st bullet]
Forecasters are paid to predict extreme weather events and other events involving the exceedance of a threshold. 
[bring in 2nd bullet] 
Ensemble products indicating the probability for exceeding a forecast value help with this task. The probability of exceedance [bring in 1st sub-bullet] is calculated by taking a count of the number of ensemble members that exceed the chosen threshold, and then dividing by the total number of members in the ensemble. [bring in 2nd sub-bullet] Sometimes, an adjustment is made to the probability distribution based on verifications over a period of time before figuring out the probability of exceeding a critical threshold. We’ll talk about that later. 
[bring in 3rd bullet]
For example, the graphic below on the right shows the 0000 UTC on 19 November 2001 probability of 24-hour precipitation amounts exceeding 0.5" for the period ending 1200 UTC on 22 November 2001. The related spaghetti diagram for 24-hour accumulated precipitation exceeding 0.5" for each ensemble member is shown on the left. 

## From NCEP Global Ensemble Web Page

**http://****www.emc.ncep.noaa.gov/gmb/yzhu/html_pqpf/pqpf_region.html**

## Advantages and Limitations of Probability of Exceedance Graphics

- Advantages
- Focuses on critical threshold values
- Not sensitive to outliers
- Compact display
- Disadvantages
- Only gives a slice of the probability distribution for the forecast, rather than the full distribution and as a result:
- Does not directly show degree of uncertainty in ensemble forecast
- May hide less likely solutions that, if they were to occur, would have major impact on the public

**Notes:**
[bring in title, 1st bullet]
The advantages of the probability of exceedance diagrams are [bring in 1st sub-bullet] the compact display and [bring in 2nd sub-bullet] the focus on critical threshold values of interest to the forecaster.

[bring in 2nd bullet]
Disadvantages include: 
[bring in 1st sub-bullet]
The probability of exceedance only gives a slice of the probability distribution for an ensemble forecast, rather than the full distribution.  Because of this, [bring in 1st sub-sub-bullet] we do not see the degree of uncertainty in the ensemble forecast, and [bring in 2nd sub-sub-bullet] less likely solutions may be hidden that, if they were to occur, would have a major impact on the public.

## Prob of Exceedance Related Diagrams:  Prob of precipitation type

- Pr {precipitation type} = fraction of total ensemble members with diagnosed precipitation type
- Types in ensemble include
- Rain
- Snow
- Freezing rain
- Sleet/ice pellets

## Most Likely or Dominant Event Diagram

- Used to show what is most often predicted by the ensemble forecast
- A common extratropical example
- Precipitation type (snow, sleet, freezing rain, rain)
- Good in combination with precipitation amount probability of exceedance, for winter weather warnings

**Notes:**
[bring in title]
A somewhat related product to the probability of exceedance diagram is the most likely or dominant event diagram. We count items predicting events of concern and then produce a graphic [bring out 1st bullet] showing those most frequently predicted in the ensemble run.  The most common example of this is the Dominant Precipitation Type [bring out 2nd bullet and then 1st sub-bullet] graphic, which can be used in combination with probability of exceedance products for precipitation amount for winter weather warnings.

This graphic [bring in graphic] is an example from the NCEP Short-Range Ensemble Prediction system from the 12 UTC 12 November 2002 forecast valid on 00 UTC 15 November 2002.  Precipitation type is indicated by shading, based on legend in lower left side of graphic.  Note that sleet and snow were combined in this graphic into a “snow” precipitation type for purposes of counting.
“Dominant precipitation type” in this case means:
The one that occurs most frequently in all areas where precipitation occurs in at least one ensemble member, as diagnosed by the NCEP precipitation type algorithm.
If there is a tie among two or more ensemble members, a mix of dominant precipitation types is indicated.
In this case, rain is indicated in areas shaded in green [highlight green], snow in areas shaded in blue [highlight blue], freezing rain in areas shown in red [highlight red], purple for snow and freezing rain equally likely [highlight purple well-north of Montana and Washington state], and orange for rain and freezing rain equally likely [highlight orange well-north of Montana].

## Advantages and Limitations of Most Likely Forecast Graphics

- Advantages
- Compact display
- Gives most likely ensemble forecast outcome
- Disadvantages
- May hide other forecast outcomes almost as likely
- Most likely" forecast outcome may not be majority (50% or more) of ensemble members
- E.g. 5 ensemble members predict snow, 4 freezing rain, 3 rain, and 3 sleet as precipitation type in a 15-member EPS run
- Snow will be indicated as the dominant precipitation type, even though in only 33% of the cases

**Notes:**
[bring in title alone]
The advantages [bring in 1st bullet] of most likely forecast graphics include the, as per usual, [bring in 1st sub-bullet] compact display. Additionally, it succinctly [bring in 2nd sub-bullet] gives the most likely ensemble forecast outcome over the domain of interest in one map. 
[bring in 2nd bullet] 
The disadvantages include [bring in 1st sub-bullet] the potential hiding of other forecast outcomes that are almost as likely, and that [bring in 2nd sub-bullet] the “most likely" forecast outcome may not be in the majority of ensemble members. [bring in 1st sub-sub-bullet and associated sub-sub-sub-bullet] For example, if five ensemble members predict snow, four freezing rain, three rain, and three sleet as precipitation type in a 15-member EPS run, snow will be indicated as the most likely precipitation type even though it occurs in only 33% of the cases. 

## Application of Joint Distributions

- Probability of multiple events taking place concurrently
- Examples:
- Combination of severe weather indices such as helicity, CAPE and CIN
- Cold season precipitation types and precipitation amount (for warning thresholds)
- Probability of repeated rainfall amounts on consecutive days at the same location

**Question:  Can you think of other examples relevant to your own country?**

## Examples of Joint Distribution Graphics

- Top graphic at right combines precipitation type with precipitation amount for ensemble member 2 (PERT 02)
- For full ensemble ...
- P{prec>1mm *and* snow} = number of ensemble members with both criteria met
- Graphic at lower right is for same valid time as upper right, for probability distribution

- 6-mm
- and
- snow

## Advantages and Limitations of Joint Probability Graphics

- Advantages
- High impact events are result of multiple events or thresholds reached simultaneously, for example:
- High precipitable water, high CAPE, favorable K Index  heavy precipitation
- High temperature and high dewpoints  heat related illness and death
- Cannot get accurate gage of risk via probabilities for each event or reached threshold individually
- Limitations
- Sample size in ensemble may be too small to capture high impact event, especially if EPS tends to have spread that is too small (underdispersive)
- Resolution of model used for EPS
- Inadequate to capture localized events?
- Uncorrected (or inadequately corrected) *systematic* model biases and errors

## Thumbnail Graphics

- A small graphic of each ensemble member at a forecast valid time.
- Helpful to see how each ensemble member is evolving
- Tempting to choose “member of the day”
- Defeats the purpose of probabilistic forecasting (each member is theoretically equally likely)
- Do you really know which member is most likely to verify?
- The initial condition perturbations are not identified with any particular member
- Each is independent of the others by construction
- DON’T DO IT!

## 1st 16 Thumbnails from 00z23Jun07 Ensemble Run vt 00z27Jun07

-  PMSL black
-  2-m T red
-  6-h prec shaded

## Statistical Clustering as Applied to Ensemble Data

- Choose a variable
- Typically 500-hPa height (ECMWF)
- Choose a distance measure
- Anomaly correlation, RMS difference from ensemble mean or climatology are possibilities
- Choose forecast period over which to define clusters
- Does each valid time defines its own clusters, or are clusters defined at one specific valid time?
- Determine the best way to cluster by distance
- Number of groups allowed can be based on total ensemble spread up to max of 6 (ECMWF over Europe)
- Maximum allowable within-group “distance”
- Minimum allowable between group “distance”

## What We’ve Covered So Far...

- Taking advantage of ensemble forecasts ... but keeping in mind the limitations
- Products revealing details of the PDF
- Plan view
- Mean and spread
- Spaghetti diagrams
- Probability of exceedance
- Dominant forecast
- Thumbnails
- Clusters

- Any QUESTIONS?

## Plume Diagrams

- Spaghetti diagram for grid boxes
- Typically used for near-surface variables
- Can plot multiple variables in same graphic
- Use nearest neighbor grid box for data
- Use raw data to avoid smoothing from interpolation
- Keep in mind representativeness problems in areas of complex terrain
- Ensemble elev here is 1389-m; Jo’burg obs taken at 1700+ m, Pretoria at 1444-m (close to model)

**10-m winds**

_**Questions**_**:  Where is the largest uncertainty in the near-surface forecast? Do you see more than one preferred forecast cluster?  Where? **

- Legend:

- p01
- p02
- p03
- p04

- p05
- p06
- p07
- p08
- p09

- p10
- p11
- p12
- p13
- p14

- p15
- p16
- p17
- p18
- p19

- p20
- ctl

## Advantages and Limitations of Plume Diagrams

- Advantages
- At a grid point, see full distribution of EPS
- Can “eyeball” multiple preferred forecast outcomes
- Outliers are easy to see
- Limitations
- Model (and ensemble) limitations affecting the forecast, such as
- Representativeness
- Especially in areas of rugged terrain
- What does the climatology favor that the underlying model cannot capture?
- Processes that are not modeled adequately, for example ...
- Grid scale precipitation processes
- Convection and convective schemes
- Sub-grid scale variability in vegetation and soils
- Sample from EPS is too small, EPS spread is too small
- Uncorrected (or inadequately corrected) *systematic* model biases and errors

## Box and Whisker

- Time series of variable at a grid point/gridbox
- Whiskers represent the extremes
- Box represents middle two quartiles
- Dot within the box is the median
- Mean and spread for grid points
- Diagram gives middle and middle quartiles (similar measures to mean and standard deviation)

## Box and Whisker Example

- Same ens run, same location as plume diagram
- Note 12 and 18UTC 26 June 2007 in both diagrams
- How do the forecast clusters show up during the 27th of June in the box and whisker diagram?
- How does the median value for the distribution each time help with determining this?

-   00z         00z         00z          00z         00z         00z         00z
- 23 Jun    24Jun     25 Jun     26 Jun    27 Jun    28 Jun    29Jun

## Advantages and Limitations of Box and Whisker Diagrams

- Advantages
- Compact visualization
- Quantifies the center, width, and range of the distribution
- Goes well with spaghetti!
- Limitations
- May miss some subtleties in the probability distribution because of coarseness of representation (0/25/50/75/100)
- Model (and ensemble) limitations affecting the forecast
- Sample from EPS and EPS spread may be too small to pick up possibility of extreme events
- Uncorrected (or inadequately corrected) *systematic* model biases and errors will give an incorrect probability distribution.

## Flow Regime Dependent Graphics

- Predictability is flow dependent!
- Forecast for control/perturbation pair starting well within Regime B not sensitive to perturbation
- Forecast for control/perturbation pair starting in green “Sensitive Region” strongly affected by small perturbation

**  ****Regime **
**       ****A**

**o**

**x**

**x**

**Regime **
**B**

**Sensitive**
**Region**

**o**

**x**

**x**

**Notes:**
[start with title only] 
We can show that for a simple forecast model like the Lorenz forecast system, that [bring in 1st bullet] chaotic systems have differing degrees of predictability depending on the flow.  Let’s review the Lorenz graphic again [bring in graphic]. 

If the initial conditions and a small perturbation both lie within 1% of each other at the starting point shown at the “o” above and [bring in 1st sub-bullet] well embedded in Regime “B” [highlight “o”], and the forecast is run out far enough to get to the other side of Regime B [trace trajectories in blue and red, then place “x” at end of each], the perturbation doesn’t much matter; this is a predictable flow regime.  [bring in 2nd sub-bullet] However, if we start a run within the [bring up sensitive region] sensitive region and apply the same sized small perturbation, the result is quite different.  The control in blue continues in regime B through the forecast [trace blue trajectory to “x”], while the perturbed trajectory [trace red trajectory to “x”] moves over into Regime B and stays there.

## Normalized Spread

- How does spread compare to past ensemble forecasts (30 day mean spread) at each forecast lead time?
- Normalized spread = where *s**forecast** *is the current forecast spread at any gridpoint and
*s* is the mean value of spread over the last 30 days for that gridpoint.
- Normalized spread < 1:  more predictable than average
- Normalized spread > 1:  less predictable than average

## Confidence Measures: Normalized Spread

- Left: hi-res GFS control and raw ensemble spread (m)
- Right: ens mean and spread, normalized by mean value for last 30 days
- Normalized spread of ~ 1  about as predictable as recent average
- Norm spread < 1  more predictable than recent average
- Norm spread > 1  less predictable than recent average

## Confidence Measures: Relative Measure of “Predictability” (RMOP)

- RMOP objectively measures “predictability” of the atmosphere
- Categorizes ensemble mean and members using long term climatology
- Based on categorical ensemble mean verification over past 30 days
- Assumptions:
- Predictability of atmospheric flow is directly related to number of ensemble members in the same climatological category as ensemble mean

## Relative Measure of Predictability (RMOP)

- Calculation of RMOP
- Place ensemble mean and individual members into bins
- Count number of members in same bin as mean
- Keep statistics over 30 days
- At each grid point and for each forecast hour
- Count number of members in same bin as mean in each forecast over 30 days, do probability distribution of RMOP during the period
- Companion statistic:  How often did the forecast with a given RMOP verify (“verification” = valid time analysis in same climatological “bin” as the ensemble mean)
- In **current** forecast, how many members are with the mean?
- Compare to 30 day probability distribution to determine the RMOP
- Will also know verification percentage for that RMOP based on long-term performance of ensembles (over more than 5 years!)

## Hypothetical 500-hPa Distribution

**Mean value for distribution A**

**Mean value for distribution B**

**Hypothetical 500-hPa Distribution**

## RMOP Example: 90h fcst 500-hPa height Antsiranana, Madagascar, vt 06z14Sep07

|  |  |  | l | lllll | llllllllllMean | lllll |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  |  |  |  |  |  |  |  |  |

-    580.1   581.3  582.2  583.0  583.7   584.4  585.2  586.1  587.4

- 10%

- 10%

- 10%

- 10%

- 10%

- 10%

- 10%

- 10%

- 10%

- 10%

**584.10**
**584.06**
**584.01**
**583.82**
**583.65**
**583.39**
**583.63**
**584.89**
**584.67**
**584.40**
**583.28**

**  **_**Break points**_** for climatological 500-hPa height bins for 06z 14Sep07 for grid box nearest Antsiranana (10% likelihood of occurrence in each bin), based on mean of 583.7 dm and std dev of 28.4 dm.**
**  ****Here, we have 10 members in the same bin as the ensemble mean for the grid point nearest Antsiranana**

**583.82**
**583.16**
**584.54**
**584.49**
**583.45**
**584.53**
**583.74**
**584.33**
**584.32**
**583.97**

**Forecasts by ensemble members (dm)**

**values depend on day of year**

## RMOP Example (con’d)

- Say over the history of the ensemble forecasts, we have the following *hypothetical *cumulative distribution of members in the same bin as the ensemble mean (120 ensemble forecast cycles) for 90 hour forecasts at the Antsiranana grid box:

**0    1    2    3    4    5    6    7    8    9   10   11   12   13  14   15   16  **

**1.0**
**0.9**
**0.8**
**0.7**
**0.6**
**0.5**
**0.4**
**0.3**
**0.2**
**0.1**
**0.0**

## RMOP Example (con’d)

- Predictability is one thing .... verification is quite another though .... similar forecasts have only verified 38% of the time

**0    1    2    3    4    5    6    7    8    9   10   11   12   13  14   15   16  **

**1.0**
**0.9**
**0.8**
**0.7**
**0.6**
**0.5**
**0.4**
**0.3**
**0.2**
**0.1**
**0.0**

**current ensemble forecast**

**Percentage **
**Verification **

**number of members with ensemble mean**

## RMOP Example (con’d)

- So...today’s forecast flow would be as (or more) predictable than 65% of the forecasts over the past 30 days.

**0    1    2    3    4    5    6    7    8    9   10   11   12   13  14   15   16  **

**1.0**
**0.9**
**0.8**
**0.7**
**0.6**
**0.5**
**0.4**
**0.3**
**0.2**
**0.1**
**0.0**

**current ensemble forecast**

**Frequency over**
**last 30 days**

**number of members with ensemble mean**

## 24-hour RMOP for Africa

- Large predictable area extending from equatorial Atlantic southeastward to S Africa
- Trough about 45S and near GM
- 30% or greater predictability  greater than 50% prob for verification

**95**
**92**
**82**
**78**
**72**
**63**
**56**
**48**
**33**
**17**

## 120-hr RMOP Example from Africa

- By 120 hours, verification rates for most predictable areas about 50-75%
- Trough off Angola and zonal flow southeast of South Africa is most predictable

**74**
**49**
**36**
**28**
**24**
**21**
**18**
**16**
**14**
**9**

## 168 hour RMOP for Africa

- Predictable features are troughs southeast and southwest of South Africa
- >90% more predictable than previous flows = nearly 60% prob of verifying.

**59**
**33**
**23**
**18**
**16**
**15**
**13**
**12**
**10**
**8**

## 240-hr RMOP Example for Africa

- At 240 hours, more features in the ensemble mean flow...
- But not much more chance of verification (18-19% for top two tiers)

**18**
**19**
**15**
**13**
**12**
**11**
**11**
**10**
**10**
**8**

## 15 day RMOP for southern Africa from 00z 19 Sep 2007 ens run valid time 00z 4 Oct 2007
More or less featureless
Near climo (10%) verification throughout the range of predictability

- More or less featureless
- Near climo (10%) verification throughout the range of predictability

- 360-hour RMOP Example for Africa

**7**
**11**
**10**
**10**
**10**
**10**
**10**
**10**
**9**
**9**

**Notes:**
[start with title]
An example of a predictable regime, at least in the long-wave pattern, can be seen in the [bring in graphic and 1st bullet] following 192-hr NCEP ensemble forecast for 500-hPa heights from 00Z 13 September 2004, valid at 00Z 20 September 2004.  Even as far out as 8 days, [bring in 1st sub-bullet] the ensemble mean forecast indicates a ridge [bring in white dashed circle] over southeastern Canada and the eastern continental United States, while a trough [bring in blue dashed circle] is forecast over the intermountain western U.S. In the relative measure of predictability, the last 30 days of ensemble forecasts are used to determine both the degree of predictability and the probability of verification. [bring in 1st sub-sub-bullet] At the axes of this ridge-trough regime, the level of predictability at 8 days is [highlight portion of color bar indicating level of predictability, orange and red, and the areas within the circles with this level of predictability] about 80-90%, but at such a long lead time the probability of verification at these levels of predictability [highlight the blue numbers in the color bar and the associated colors] is only 19-45%.

We won’t show these in the interests of time, [bring in 2nd sub-sub-bullet] but the interim forecasts for 500-hPa heights show a similar pattern of predictability in both areas, with the probability of verification increasing as forecast lead time decreases.  

[bring in 3rd sub-sub-bullet] 
Finally, at 24 hours [bring in 24-hour graphic, leaving the dashed circles in place], the long-wave ridge and trough axes, while shifted somewhat to the west, still exist with a high level of predictability; of course, at 24 hours the verification percentages [highlight orange/red in 24-hour graphic] are 91-95% rather than the 8-day probabilities of 19-45%, for the same 80-90% levels of predictability.

## Predictability and Probability of Verification as a Function of Forecast Hour

- Varies with atmospheric flow type and season
- Higher numbers generally in cool season than in warm season, lowest in transition seasons
- Near climo.  10% prob of verification for each bin
- As expected, lower predictability  faster drop in verification rate
- As forecast lead time increases, curve changes shape from convex to concave, then to straight line at climo prob of verification at longest leads

## Experimental Plan View Confidence Map

- Confidence based on comparison to size of spread from long-term ensemble ?reforecast? by ?Climate Diagnostics Center? at Earth Systems Research Lab (ESRL)
- Values are size above or below normal spread, below normal spread = above normal confidence (green and blue shades)

## 2-m Temperature confidence plume

- Time series of spread (= 1 standard deviation)
- Average spread for variable for 45 days centered on forecast valid time (blue)
- Climatological standard deviation from NCEP reanalysis
- Current ensemble forecast spread in black with closed circles at each verification time
- Green shading = above average confidence
- Red shading = below average confidence (no significant areas in this graphic)

**Pretoria, South Africa**

- http://moe.met.fsu.edu/confidence/

## Advanced Applications: What we’ve covered so far...

- Confidence measures
- Plan view maps:
- Normalized spread
- Relative measure of “predictability”
- Confidence maps
- Grid point/grid box diagrams:
- Confidence plumes

**Any QUESTIONS??**

## Summary and Conclusions

- EPS greatest strength is in the probability distributions it gives the forecaster
- There are caveats in that EPS’s are not perfect, but we can make adjustments through verification and application of results to some of the imperfections
- Shape of distribution determines the best derived statistics for the forecast
- E.g. Ensemble mean not a good forecast with multiple clusters of forecasts

## Summary and Conclusions (con’d)

- Plan view maps help give a “big picture” of the amount of uncertainty in the forecast, and the distribution of possible forecast outcomes
- Use ensemble mean and spread and spaghetti diagrams together to get sense of probability distribution for location and strength of features
- Probability of exceedance diagrams use critical thresholds to give a probability at that level
- Some plan view maps aren’t customized to the thresholds preferred in a particular region
- Most likely event maps can be used for categorical forecasts that break down by event type (severe weather, precipitation type)
- Joint probability maps combine multiple indices or conditions that potentially can result in significant weather events
- Thumbnail graphics give a picture of each ensemble member
- Can be useful, but don’t be tempted to choose a “member of the day”!
- Objective clustering of similar forecasts can reveal aspects of probability distributions that may not be obvious from visual inspection of other plan view maps

## Summary and Conclusions (con’d)

- Point diagrams help localize the forecast
- Plume diagrams for single or multiple variables can help assess severity of events (though joint probability diagrams at a point would do a better job)
- Helps visualize the evolution of local weather as a function of uncertainty of feature timing and strength
- Box and whisker diagrams quantify the distribution at a point for each forecast lead time by quartile, median, extreme maximum and minimum values
- Useful with plume diagrams in a similar manner as with mean/spread with spaghetti plots

## Summary and Conclusions

- Plan view *quantified *confidence maps
- Normalized spread
- Based on average spread for previous 30 day period
- RMOP
- Based on number of members in category of ensemble mean being a measure of predictability of atmospheric flow
- Compare to past 30 days to get degree of predictability (%)
- Verification value based on many years of ensemble forecasts
- Experimental confidence maps similar to normalized spread, but ...
- Compares size of ensemble spread to size of spread over 45 days, rather than normalizing current ensemble spread (10-m winds and 2-m temps available so far for N America only)
- Point confidence diagrams (experimental)
- Confidence plumes give sense of local uncertainty
- Used currently for 10-m winds and 2-m temperatures (available for Pretoria)