# Presentation: 899037

## Multi-Model Ensembling:A Comparison of MethodsLisa GoddardInternational Research Institute for Climate ResearchThe Earth Institute of Columbia UniversityPalisades, New York, USA

- 24 March 2005

**Notes:**
This work is more an illustration and comparison of techniques than a quantitative examination of real-time forecast skill.

A manuscript distilling the methods, ideas and results shown here is in preparation.

## Outline

- Illustration of different methodologies for multi-model ensembling (MME)
- 1) Equal weighting (“pooling”) of raw ensembles
- 2) Equal weighting of recalibrated ensembles
- 3) Performance-based weighting of raw ensembles
- 4) Combination of (2) & (3)
- Sensitivity of performance-based MME on number of models
- Sensitivity of performance-based MME on length of training period

## Data

- AGCMs:	*Simulations for 1950-1997*
- 		* CCM3 (NCAR)
- 		* ECHAM4.5 (MPI)
- 		* NSIPP1 (NASA-GSFC)
- Observations:
- 	2m Air Temperature and Precipitationfrom CRU-UEA

**Notes:**
The model data is taken from 3 atmospheric GCMs. The CCM3.6 (Hack et al., 1998; Hurrell et al., 1998; Kiehl et al., 1998) and the ECHAM4.5 (Roeckner et al., 1996) AGCMs are configured at T42 horizontal resolution, with 18 vertical levels. The NSIPP1 AGCM (Bacmeister et al., 2000) is a grid point model with a horizontal resolution of 2x2.5 degrees and 34 levels in the vertical. The NSIPP1 model data was linearly interpolated to the T42 grid for purposes of combining the models.

The observations for temperature and precipitation, against which the model combinations are validated, are from the Climate Research Unit of the University of East Anglia (New et al., 1999; New et al., 2000). The data are provided at 0.5x0.5 degree resolution, and were upscaled to T42 for use in the validation.

Note, all AGCM predictions(/simulations) are cast as probabilistic 3-category forecasts for 3-monthly seasonal values of mean temperature and total rainfall, where the prior probability of any 1 category is 1/3. Because seasonal forecasts are probabilistic, they will be assessed as such. The goal is to produce reliable forecasts with good resolution, where the forecast probabilities mean what they say.

Bacmeister, J., P.J. Pegion, S.D. Schubert, and M.J. Suarez, 2002: Atlas of Seasonal 	Means Simulated by the NSIPP 1 Atmospheric GCM. NASA/TM-2000-104505, Vol. 	17, 190pp.
Hack, J.J., J.T. Kiehl, and J.W. Hurrell, 1998:The hydologic and thermodynamic 	characteristics of the NCAR CCM3. J. Climate, 11, 1179-1206.
Hurrell, J.W., J.J. Hack, B.A. Boville, D.L. Williamson, and J.T. Kiehl, 1998: The dynamical 	simulation of the NCAR Community Climate Model version 3 (CCM3). J. Climate, 	11, 1207-1236.
Kiehl, J.T., J.J. Hack, G.B. Bonan, B.A. Boville, D.L. Williamson, and P.J. Rasch, 1998: The 	National Center for Atmospheric Research Community Climate Model, J. Climate, 	11, 1131-1149.
New, M., M. Hulme, and P.D. Jones, 1999: Representing twentieth-century space-time 	climate variability. Part I: Development of a 1961-90 mean monthly terrestrial 	climatology. J. Climate, 12, 829-856.
New, M., M. Hulme, and P.D. Jones, 2000: Representing twentieth-century space-time 	climate variability. Part II: Development of a 1901-96 monthly grid of terrestrial 	surface climate. J. Climate, 13, 2217-2238.
Roeckner, E., K. Arpe, L. Bengtsson, S. Brinkop, L. Dumenil, M. Esch, E. Kirk, F. Lunkeit, 	M. Ponater, B. Rockel, R. Sausen, U. Schlese, S. Schubert, and M. Windelband, 	1996: Simulation of the present-day climate with the ECHA< model: Impact of 	model physics and resolution. Report No. 93, Max-Planck-Institut fur Meteorologie, 	Hamburg, Germany, 171pp

## Benefit of Combining Models

- Ranked Probability Skill Score (RPSS) for 2m TemperatureJan-Feb-Mar 1950-1995

**Notes:**
The Ranked Probability Skill Score (RPSS) is one probabilistic measure of skill. It measures the squared error between the cumulative forecast and observed probabilities, then weights that score against a reference forecast of climatology. 
Score 1 = always forecasting observed category with 100% probability
	0 = always (or effectively) forecasting climatological probabilities
       <0 = worse than climatology

This figure shows that (as for other measures) each model has regions where it does better/worse than the other models. By simply combining them all into 1 super-ensemble, deficiencies seen in individual models are reduced. 

However, by taking into account which models work where (and perhaps under what conditions) can we improve over the straightforward pooling of models?
 How much and in what ways can more sophisticated multi-model combinations improve over the baseline of pooling?

## Combining: 

- 	(1) POOLED MM ENSEMBLE
- 		Each model weighted equally
- 	(2) BAYESIAN MM ENSEMBLE
- 		Optimal weights determined for AGCMs relative                          to climatology by *maximizing likelihood* measure.

**Refining**:
- 	(1) RAW MODEL PROBABILITIES
- 		Tercile thresholds determined by model history
- 	(2) CORRECTED PROBABILITIES
- 		Using ensemble mean, conditional probabilities
- 		are determined from past model performance

**Notes:**
Two approaches are considered here for treating the model probability distributions and also for combining the probability distributions from the individual models. In each case, there is a more simple and less simple approach.

The more simple approach for the model probabilities is to take them straight from the AGCMs, assuming that the model is generating an appropriate PDF. The more simple approach for the combination is to weight each model equally. These are the baseline methods that we hope to improve upon.
The less simple methods are described next.

## Contingency Table Refinement

_**Refining:**_

|  | BELOW-
NORMAL | NEAR-
NORMAL | ABOVE-
NORMAL |
| --- | --- | --- | --- |
| BELOW-
NORMAL | ηBB | ηBN | ηBA |
| NEAR-
NORMAL | ηNB | ηNN | ηNA |
| ABOVE-
NORMAL | ηAB | ηAN | ηAA |

- MODEL

- OBSERVATIONS

**Based on ****model****history ...**

**Notes:**
Using history of model ensemble means, tabulate conditional frequencies. For example, when the model’s ensemble mean indicated “below-normal”, how frequently was it below-normal, near-normal, and above-normal, and similarly for each category. This is done for each variable, seasonally. 
A perfect deterministic forecast system would result in a purely diagonal matrix of 1s; no skill would yield values of approximately 1/3 in every box.

Then, given the current seasonal forecast [at that location], where the ensemble mean indicates below-normal, the probabilities are taken from that row.

## Bayesian Model Combination

- Climo Fcst“Prior”

- GCM Fcst“Evidence”

*t*=1

*2*

*3*

*4*

*...*

- Combine “prior” and “evidence” to produce weighted “posterior” forecast probabilities, by maximizing the likelihood.

*...*

*...*

_**Combining:**_

**Notes:**
The Bayesian approach to multi-model combination of AGCMs (Rajagopalan et al., 2002) determines the relative weights, pointwise, of climatology and each AGCM that will maximize the likelihood score over time.

The climatology forecast has 33.3% probabilities for each category always. The AGCM probabilities change in each case.

Initially a weighted combination of the AGCMs and climatology is performed, and the likelihood score is calculated. The weights are then adjusted, and the above procedure is repeated, iteratively until the likelihood score has been maximized.

Result is that if the models have little or no skill over a particular region, climatology receives more/all weight, and the resulting forecast for that location is closer to climatology. If one or more models frequently predict the observed category with confidence, then the model(s) receive more/all weight, and the forecast appears closer to the models’ probabilities.

Rajagopalan, B., U. Lall, and S.E. Zebiak, 2002: Categorical climate forecasts through a regularization and optimal combination of multiple GCM ensembles. J. Climate, 15, 1792-1811.

- Jan-Feb-Mar (1950-1995)

**Notes:**
So how do these approaches (more vs. less simple) compare? First an overall answer is provided by use of reliability diagrams.

Figure caption: Reliability diagrams for global precipitation forecasts. Each graph plots the forecast probability versus the observed relative frequency for a given category (above-normal, upper row; below-normal, lower row). Different forecast methods are shown in the colored lines: individual AGCMs (blue lines); Pooled multi-model probabilities (black line); Bayesian multi-model probabilities (green line). In the left column, the probabilities from the models are used as is; in the right column, the probabilities are first refined through the contingency table approach. The inset graphs shows the frequency with which the range of forecast probabilities were assigned.

In this case, either correcting the PDF before combination, or using the Bayesian approach with the uncorrected probabilities dramatically improve over the pooling of the raw AGCM probabilities (which is itself some improvement over the individual AGCMs).

- Jan-Feb-Mar (1950-1995)

**Notes:**
Temperature is done better in that models than rainfall; however, there is still room for improvement. Again, improvement is seen in both the Bayesian combination and the corrected probabilities. The corrected probabilities are almost perfectly reliable across the range of probabilities issued, which indicates excellent resolution of the PDF. The one potential short-coming of the corrected probabilities is that the tails of the distribution are not represented, meaning that these forecasts will be more conservative than those from the Bayesian approach with the raw AGCM PDFs. 

Note that climatology would be an almost perfectly reliable forecast, but contains no information. Need to balance reliability with forecast sharpness (or difference from climatological probabilities).

This figure is for global land in JFM; the results for the tropics are nearly identical in all cases. What about tropical rainfall in JAS – monsoon season – something more models do not do well overall.

- Jul-Aug-Sep (1950-1995)

**Notes:**
See the same sort of result. Improvements leading to much more reliable forecasts from the Bayesian combination and from the corrected probabilities, over the pooling of the raw AGCM probabilities. Again, much more conservative forecasts from the corrected probabilities.

## Ranked Probability Skill Scores

**Temperature**
- Jan-Feb-Mar (1950-1995)

**Notes:**
How do these apparent improvements in overall reliability impact probabilistic skill locally?

Figure caption: Maps of RPSS for temperature during the JFM season 1950-95. The maps on the left are from the pooled combination of the raw AGCM probabilities (top); the Bayesian combination of the raw AGCM probabilities (middle); and the pooled combination of the corrected AGCM probabilities (bottom). On the right, the more sophisticated multi-model approaches are contrasted with the baseline approach of pooling the raw probabilities.

All 3 maps on left look similar, although one may notice stronger negative RPSS values over some areas on the raw-pooled map.  In fact, if we look at the difference between the “new” and the “baseline” approach, one sees that most improvement is realized over the negative areas on the “baseline” map (but not always).

## Ranked Probability Skill Scores

- _Ranked Probability Skill Scores_
**Precipitation**
- Jul-Aug-Sep (1950-1995)

**Notes:**
As seen before, precipitation is not done as well as temperature; there are much fewer positive values on the maps. However, now more difference between the maps is visible. Not only are negative values neutralized, but even areas of positive skill on the “baseline” maps are improved, such as over western Africa, northern South America, and even Indonesia.

## Example:

_*Example*_:
- Probabilistic precipitation forecast

**Notes:**
So what would a forecast look like from these approaches?

Figure caption: Probabilistic global precipitation forecast for JFM 1950. The probability for the dominant category is plotted. Color suggests the tendency of the dominant category: greater likelihood for above-normal rainfall in the green-blue colors, and greater likelihood for below-normal rainfall in the yellow-brown colors. Forecasts are from the pooled combination of the raw AGCM probabilities (top); the Bayesian combination of the raw AGCM probabilities (middle); and the pooled combination of the corrected AGCM probabilities (bottom). 

The raw-pooled version has very confident probabilities everywhere. The raw-Bayesian version shows similar pattern of probability shifts, but the confidence of the probabilities are weaker, particularly in the extra-tropics. The corrected-pooled version shows very conservative probabilities compared to the other 2 maps, although there are a few instances where it shows less conservative values than seen in the raw-Bayesian.
Thus, there are similar patterns overall, but there is a very big difference in the forecast sharpness.

##  Precipitation Forecast – Feb-Mar-Apr 1983

-  **Precipitation Forecast – Feb-Mar-Apr 1983**

-  **Precipitation Forecast – Feb-Mar-Apr 1989**

**Notes:**
Figure caption: Similar to (last figure), but for South America during FMA 1983, influenced by an El Nino (top row), and during FMA 1989, influenced by a La Nina (bottom row).

Same story as global forecast, but have a chance to look a bit more closely at the detail.

## Conclusions

- AGCMs are over-confident in their probability distributions
- Improved reliability of seasonal climate forecasts can be achieved by recalibrating the probability distributions
- 	  Directly refining PDFs of AGCMs, OR
- 	  Combining AGCM PDFs with      climatological PDF

## Influence of Number of ModelsPrecipitation

**Notes:**
From Robertson et al, 2004 Mon Wea Rev.

## Influence of Number of Models2m Air Temperature

**Notes:**
From Robertson et al, 2004 Mon Wea Rev.

## IRI Operational Dynamical Forecast System(as of September 2004)

- 30

-  12

- 30

- 12

- 10

- 24

- 10

- 24

- 10

**        ****FORECAST SST**
**        ****TROP. PACIFIC **
**  ****(multi-models, dynamical**
**          ****and  statistical)**
**    ****TROP. ATL, INDIAN**
**             ****(statistical)**
**     ****EXTRATROPICAL **
**    ****(damped persistence**)

**GLOBAL **
**ATMOSPHERIC**
**MODELS**
**ECPC(Scripps)**
**ECHAM4.5(MPI)**
**      **
**CCM3.x(NCAR)**
**NCEP(MRF9)**
**        ****NSIPP(NASA)**
**COLA2**
**GFDL**

- Forecast
- SST
- Ensembles
- 3/6 Mo. lead

- Persisted
- SST
- Ensembles
- 3 Mo. lead

**POST**
**PROCESSING**
**MULTIMODEL**
**ENSEMBLING**

**PERSISTED **
**GLOBAL**
**SST**
**ANOMALY**

**OCEAN             ATMOSPHERE**

- 30

- 10

## Influence of Number of ModelsPrecipitation – AMJ 1950-98

## Influence of Training Period Length

## Even if training period is ‘short’, performance-based weighting yields better, and more stable, skill over the US.

*Even if training period is ‘short’, performance-based weighting yields better, and more stable, skill over the US.*

## References

- Rajagopalan, B., U. Lall, and S.E. Zebiak, 2002: _Categorical climate forecasts through a regularization and optimal combination of multiple GCM ensembles_. *Mon.Wea.Rev.*, **15**, 1792-1811.
- Robertson, A.W., U. Lall, S. E. Zebiak, and L. Goddard, 2004: _Improved combination of atmospheric GCM ensembles for seasonal prediction_. *Mon. Wea. Rev*., **132**, 2732-2744.
- Goddard, L., S.J. Mason, and A.W. Robertson, 2005. _Multi-model ensembling: Combining and refining_. *CLIVAR Exchanges*, Jan 2005, No 32. (http://www.clivar.org/publications/exchanges/ex32/pdf/Exchanges32%20.pdf)