# Presentation: 940346

## An evaluation of the probabilistic information in multi-model ensembles 

**David Unger**
**Huug van den Dool , Malaquias Pena, Peitao Peng, and Suranjana Saha **
- 30th Annual Climate Prediction and Diagnostics Workshop
- October 25, 2005

## Objective

**Produce a Probability Distribution Function (PDF) from the ensembles**.
_***Challenge***_
**Calibration**
**Account for skill**
**Retain information from ensembles**
**    ****(Or not if no skill)   **

## Schematic illustration

- Temperature

**Notes:**
Ensembles are point forecasts
The ensemble mean doesn’t care about the distribution

## σ

**σ**

**Notes:**
Classic regression uses the ensemble mean (regardless of whether you derive the equation from the individual members or not.  
  The ensemble mean ignores the fact that this “Near normal” forecast is arrived at by averaging warm and cold realizations

## Schematic example

- Temperature (F)

**σ****z**

**Notes:**
A Kernel method tries to use the information in an ensemble.  A Kernel Density Filter is a histogram smoothing technique that places a small error distribution around each member.   This, however is not properly calibrated.  This talk will present a method to calibrate the ensemble 

## Kernel vs. Mean

**Notes:**
Ultimately, we will have to choose whether to pay attention to individual members or not.

## Regression 

**Step 1.  Standardization**

**Step 3.  Make the forecast**

**Step 2.  Skill Adjustment**

**^**

**^**

**^**

**Notes:**
Regression can be thought of as a calibration process.  The forecast is standardized, then adjusted to account for the skill. Then restored using means and sigmas from the obs.

## σe

- σ**e**

- σe= σc√ 1-Rm2

- _______     _

**Notes:**
If we assume Normally distributed variables, and a linear fit, the expected error distribution is normal with an standard deviation specified by the correlation and the climo.   This tells us what error distribution will be appropriate. Note that the forecast has to be “damped” toward climatology for this to be good.

## Analysis of Ensemble Variance

**V = Variance**
-  **Total            = Explained  + Unexplained**
** ****Variance          Variance       Variance**
**              ****σ****c****2****   =   V****e    ****+    V****u**
**               ****σ****c****2****   = ****σ*****Fm*****2**** ****+    ****σ****e****2**
**               ****σ****c****2****   = ****σ*****Fm*****2**** ****+    (E****2****  +  ****σ****z****2****)**

- ^

- ^

**Notes:**
The trouble is that regression ignores the ensemble set.  It just looks at the ensemble mean.  
  We can look at the ensemble set by using principles from ANOVA.  The error variance is partitioned into two components, one “predicted” by the ensemble spread, and a residual

## Analysis of Ensemble Variance (Continued)

-    **With help of some relationships commonly used in linear regression:**
**<E****2**** > = **** (R****m****2 ****- R****i****2****) ****σ****c****2**** **
**σ****z****2**** = **** (1-2R****m****2 ****+ R****i****2****) ****σ****c****2**** **
**R****z****2**** = **** 2R****m****2 ****- R****i****2                        ****R****z****≤**** 1.**

**^**

**Notes:**
Analysis of variance and some theory commonly used in regression can provide appropriate parameters to apply to a Kernel density filter.   In order to be properly calibrated, the forecasts and ensemble spread must be damped, and the residual error variance (sigmaz) must be properly specified. 

## Ensemble Calibration 

**Step 1.  Standardization**

**Step 3.  Skill Adjustment**
**     ****Z****i ****  =  R****z ****Z****i ****’**** **

**Step 2.  Ensemble Spread Adjustment**
**  ****Z****i****’  =  K(Z****i****  - Z****m****)  + Z****m**

**^**

**^**

**^**

**Step 4.  Make the Forecast**

**Notes:**
Here is the ensemble calibration procedure.

## Schematic example

- Temperature (F)

**σ****z**

## Rz=.97, Rfm=.94, Ri=.90

**Notes:**
If there is high skill, the distribution follows individual ensembles closely.

## Rz=.93, Rfm=.87, Rf=.30

**Notes:**
As skill lowers the forecast becomes damped toward climo, and the ensemble members themselves become damped toward the ensemble mean.  The resulting distribution becomes more normal

## Rz=.85, Rfm=.67, Ri=.41

## Rz=.62, Rfm=.46, Rf=.20

## Weighting

**Notes:**
Multi-model ensembles add the issue of weighting

## Wgts: 50% Ens. 1, 17%Ens 2, 3, 4

**Notes:**
Here 50% weighting is given to model 17 percent is given to each of the others 

## Real time system

- Time series estimates of Statistics.
- Exponential filter
- FT+1 = (1-α)FT  +  α fT+1
- Initial guess provided from 1956-1981 CA statistics

**Notes:**
In a real time system we compute statistics with an adaptive approach.  Statistics are calculated with and adaptive smoother that is somewhat like a running mean, of 1/a cases.  In a running mean, we take 1/a of the next year and subtract 1/a of the obs 10 years ago.   Here we add 1/a of the next year but subtract .9*the average of the last 10 years.

## Continuous Ranked Probability Score

**Notes:**
Verification gets confusing if you use many scores.  Here we will use the Continuous ranked probability score.  Lower is better.  The CRPS is equivalent to the Absolute error for a very certain forecast.  You can “buy” some insurance against bad forecasts at the cost of increasing (worsening) your score when your forecast is good.  Overall the system is proper. 

## Some Results

***Nino 3.4 SSTs***
-    **Operational system**
- 15 ***CFS*** ,12 ***CA,   ***1 ***CCA*** , 1  ***MKV***
**Demeter Data**
- 9 ***CFS***, 12 ***CA***, 1 ***CCA***, 1 ***MKV***
- 9 ***UKM***, 9 ***MFR***, 9 ***MPI***,
- 9 ***ECM***, 9  ***ING***, 9 ***LOD***, 9 ***CER***,

**Notes:**
We make operational forecasts each month 
We will also show some results from a greatly expanded ensemble set provided by the DEMETER data

## Nino 3.4 SSTs

**Notes:**
Here is a box and whisker plot of our recent SST forecast.  This plot is derived from the calibrated ensemble method.

## Nino 3.4 SST 5-month lead by initial time 1982-2001

**Notes:**
Here is the skill, CON-OP is the operational method, CON-86 is for all 86 ensembles from DEMETER models, ENS86b is a bias corrected ensemble mean forecast, and ENS86 is the ensemble mean without bias correction.  
This slide and the next are from the adaptive filters passed through the data twice, so they are not fully independent.  
Bias correction is necessary.
The Demeter data improves the forecast in spring, not in fall.
Not much difference between CON and ENS-b

## CRPSS – Nino 3.4 SSTs 

_***CRPSS – Nino 3.4 SSTs***_
- All Initial times   1982-2001

**Notes:**
Here is the results for Lead times 1-month,- 5 months.  
The CON has a slight edge.  

## CRPSS – Nino 3.4 SSTs 

_***CRPSS – Nino 3.4 SSTs***_
- All Initial times   1990-2001 (Independent)

**Notes:**
Here is independent data tests, calibrated on the first half, applied (with time series adaptation) to the second half

## CRPSS – Nino 3.4 SSTs  

_***CRPSS – Nino 3.4 SSTs***_
**5-month Lead, All Initial times   1990-2001 (Independent data)**

**Notes:**
What about ensemble size.  Do more members help?
 Left most is 9 members from one of the models, then another is added, and another etc, making progressively larger ensembles.  
No clear trends.  No clear winner between ENS and CON, 
However, there is a clear relationship between skill of the model and skill of the ensemble.  Right- difference between the consolidation of the best 3 and worst 3 models 
(27 members total).   In fact the ups and down’s of the other part of the graph are caused by the introduction of models that are more or less skillful than the average to that point. 

## Reliability Nino 3.4 SST (1990-2001)

**Notes:**
SST’s are fairly reliable.  They have a cool bias (Independent data), and most notably miss the high end of the SST values.  Models miss the big warm events.  
  The consolidation tends to be slightly over confident (is predicting 2% confidence that is exceeded 20% of the time.  Worse than the ens-86b

## U.S. Temperature and Precipitation Consolidation

- 15 CFS
- 1 CCA
- 1 SMLR
-    Trends are removed from models
-    Statistics and distribution are computed
-    Trend added to end result.

**Notes:**
Now lets look at the US T and P seasonal forecasts.

## Trend Problem

**Should a skill mask be applied? How much?**
**    ****- This technique requires a quantitative estimate of the trend.**
**Component models sometimes “learn” trends, making bias correction difficult. – Doubles trends.**
**Errors in estimating high frequency model forecasts**

**Notes:**
Note  Trend problem

## U.S. T and P consolidation

**Skill Mask on Trends**

**No Skill Mask on Trends**

**Notes:**
Left – Trends are dampened according to their skill.  Too weak 
Right – Trends are included according to bias correction. Too strong, unusable Precip
How do we decide – arbitrary decisions?   
Different problem than consolidation, really, it is a problem of what is the true trend.  

## Ensm

| .060      | .091 |
| --- | --- |
| .053 | .063 |

| .101 | .160 |
| --- | --- |
| .074 | .105 |

| .228 | .372 |
| --- | --- |
| .173 | .259 |

| .010 | .067 |
| --- | --- |
| .000 | .000 |

**Ensm**

**CRPS**

**RPS**

**Cons**

**CRPS and RPS-3 (BNA)  Skill Scores:         Temperature   **

| .020 | .040 |
| --- | --- |
| .010 | -.025 |

| .026 | .047 |
| --- | --- |
| .051 | .045 |

| .057 | .070 |
| --- | --- |
| .039 | .045 |

| .016 | -.007 |
| --- | --- |
| .052 | .029 |

| 190 | .253 |
| --- | --- |
| .138 | .190 |

| .033 | .048 |
| --- | --- |
| .030 |  .037 |

| -.001 | -.013 |
| --- | --- |
|  .039 |  .019 |

| .084 |  .121 |
| --- | --- |
| .043 |  .074 |

| .086 | .110 |
| --- | --- |
| .044 | .050 |

| High |
| --- |
| Moderate |
| Low |
| None |

**Skill**

**.15**

**.07**

**.02**

- 1-Month Lead, All initial times

**Notes:**
Here are the skills.  The CRPS measures the whole PDF including tails.  The RPS measures just the traditional Long-Lead Below, Near Above categories.
  All initial times, 1-month lead.
The CONS has the edge over ENSM.  The forecast is more accurate in the center of the distribution (BNA) than on the tails.  Outliesrs lower skill. 
Skill mostly in west.
RPS and CRPS values look low because forecasts are close to climo.

## Reliability U. S. Temperatures(1995-2003)

**Notes:**
Reliability shows that the improvement over ENS is a bias correction. The only difference is that the trend is added to individual models (skill weighted) vs the ensemble mean. Somewhat mysterious.  

## Ensm

| .018      | .018 |
| --- | --- |
| .017 | .018 |

| .005 | .002 |
| --- | --- |
| .005 | .002 |

| .078 | .059 |
| --- | --- |
| .076 | .059 |

| .055 | .064 |
| --- | --- |
| .055 | .064 |

**Ensm**

**CRPS**

**RPS**

**Cons**

**CRPS and RPS-3 (BNA)  Skill Scores:         Precipitation **

| .001 | .002 |
| --- | --- |
| .002 | .002 |

| .005 | .005 |
| --- | --- |
| .005 | .007 |

| -.004 | -.009 |
| --- | --- |
| -.004 | -.009 |

| -.006 | -.004 |
| --- | --- |
| -.006 | .004 |

| .006 | .011 |
| --- | --- |
| .006 | .009 |

| .030 | .037 |
| --- | --- |
| .030 |  .037 |

| .013 | .007 |
| --- | --- |
|  .012 |  .007 |

| .016 |  .031 |
| --- | --- |
| .016 |  .029 |

| .014 | .020 |
| --- | --- |
| .013 | .020 |

| High |
| --- |
| Moderate |
| Low |
| None |

**Skill**

**.10**

**.05**

**.01**

- 1-Month Lead, All initial times

**Notes:**
Precipitation skill:  ENSO looks to be the only game in town.  California is a disappointment.

## Reliability U.S. Precipitation(1995-2003)

**Notes:**
Forecasts are so close to climo that everyone is reliable.  The reliability diagrams on continuous PDF’s are less useful than ones that follow the anomalies, Like the more traditional ones on BNA cateories (not shown here because there just wasn’t enough data.

## Conclusions

**Calibrated ensemble and ensemble means score very closely (by CRPS) **
**    ****Calibrated ensembles seem to have a slight edge.**
**No penalty for including many ensembles ****(but not much benefit either)**
**Considerable penalty for including less skillful ensembles – Weighting is critical.**
**Probabilistic predictions are reliable ****(when looked at in terms of a continuous PDF)**

**Notes:**
Conclusion:  From a score standpoint there is not much difference between CON and ENS.  It’s a matter of faith in ensembles.  My opinion:  Tie goes to the more faithful use of ensembles.  If we run them we should use them.  Cons seems to have a slight edge anyway.

## Conclusions (Continued)

**Calibrated ensembles tend to be slightly overconfident**
**Trends are a major problem – and are outside the realm of consolidation (but they are critically important for seasonal temperature forecasting).**

## 6-10 day Forecasts (based on Analogs)

**Notes:**
Couldn’t resist showing a 6-10 day application.