# Presentation: 627012

## Multi-Model Ensembling for Seasonal-to-Interannual Prediction:From Simple to Complex

**Multi-Model Ensembling for ****Seasonal-to-Interannual Prediction:****From Simple to Complex**

- Lisa Goddard and Simon MasonInternational Research Institute for Climate & SocietyThe Earth Institute of Columbia University

## Benefit of Combining Models

- Benefit of Using Multiple Models
- RPSS for 2m Temperature (JFM 1950-1995)

**Notes:**
The Ranked Probability Skill Score (RPSS) is one probabilistic measure of skill. It measures the squared error between the cumulative forecast and observed probabilities, then weights that score against a reference forecast of climatology. 
Score 1 = always forecasting observed category with 100% probability
	0 = always (or effectively) forecasting climatological probabilities
       <0 = worse than climatology

This figure shows that (as for other measures) each model has regions where it does better/worse than the other models. By simply combining them all into 1 super-ensemble, deficiencies seen in individual models are reduced. 

However, by taking into account which models work where (and perhaps under what conditions) can we improve over the straightforward pooling of models?
 How much and in what ways can more sophisticated multi-model combinations improve over the baseline of pooling?

## Varying Complexity in Building MM

- 	(1) RAW MODEL PROBABILITIES (simple)
-      	      Tercile thresholds determined by model history -- Counting
- 	(2) RECALIBRATED PDF PROBABILITIES (less simple)
- 	- Contingency table recalibration (CT): categorical probabilities
- 	determined by category of ensemble mean
- 	- Uncertainty in forecast PDFs based on ensemble mean MSE

- Varying **Complexity** in Building MM

**Notes:**
Two approaches are considered here for treating the model probability distributions and also for combining the probability distributions from the individual models. In each case, there is a more simple and less simple approach.

The more simple approach for the model probabilities is to take them straight from the AGCMs, assuming that the model is generating an appropriate PDF. The more simple approach for the combination is to weight each model equally. These are the baseline methods that we hope to improve upon.
The less simple methods are described next.

## Varying Complexity in Building MM

- (1) POOLED MM ENSEMBLE (simple)
-      Each model weighted equally
- (2) PERFORMANCE-BASED MM ENSEMBLE (less simple)
- 	- _Bayesian:_ determine optimal weights for AGCMs & climatology 		by *maximizing likelihood*
*	**-* _Multiple linear regression (MLR):_ obtain probabilities from prediction 	   error variance using first few moments of ensemble distributions
- 	- _Canonical Variate (CV):_ maximize discrimination between categories
- 		using first few moments of ensemble distributions

- Varying **Complexity** in Building MM

**Notes:**
Two approaches are considered here for treating the model probability distributions and also for combining the probability distributions from the individual models. In each case, there is a more simple and less simple approach.

The more simple approach for the model probabilities is to take them straight from the AGCMs, assuming that the model is generating an appropriate PDF. The more simple approach for the combination is to weight each model equally. These are the baseline methods that we hope to improve upon.
The less simple methods are described next.

## Bayesian Model Combination

*...*

- Combine “prior” and “evidence” to produce weighted “posterior” forecast probabilities, by maximizing the likelihood.

*...*

*...*

_**Combining**_**:** Based on model performance

**Notes:**
The Bayesian approach to multi-model combination of AGCMs (Rajagopalan et al., 2002) determines the relative weights, pointwise, of climatology and each AGCM that will maximize the likelihood score over time.

The climatology forecast has 33.3% probabilities for each category always. The AGCM probabilities change in each case.

Initially a weighted combination of the AGCMs and climatology is performed, and the likelihood score is calculated. The weights are then adjusted, and the above procedure is repeated, iteratively until the likelihood score has been maximized.

Result is that if the models have little or no skill over a particular region, climatology receives more/all weight, and the resulting forecast for that location is closer to climatology. If one or more models frequently predict the observed category with confidence, then the model(s) receive more/all weight, and the forecast appears closer to the models’ probabilities.

Rajagopalan, B., U. Lall, and S.E. Zebiak, 2002: Categorical climate forecasts through a regularization and optimal combination of multiple GCM ensembles. J. Climate, 15, 1792-1811.

## Canonical Variate Analysis

- The canonical variates are defined to maximize the ratio of the between-category to within-categoryvariance.

***Canonical Variate Analysis***

_**Combining**_**:** Based on obs. relative 			to model

## NINO3.4

## DEMETER Data

**DEMETER Data**

## Equal Weighting

**Equal Weighting**

- Probabilistic forecasts were obtained by counting the number of ensemble members beyond the outer quartiles, and then averaging across the three models.
- The pooled ensemble is thus an equally-weighted combination of predictions uncorrected for model skill (although corrected for model drift).
- Reliability is good for all three categories.

**Notes:**
Reliability diagrams for DEMETER forecasts of NINO3.4 SSTs for all start dates and lead-times. Warm (El Nino) and cold (La Nina) conditions are defined as in the warmest 25% of cases. Note that reliability for forecasts even for neutral conditions is good. Comparison of the frequency histogram for neutral forecasts with those for warm and cold conditions is difficult because of the different prior probabilities.
Note that the quartiles were defined separately for each model, season and lead-time, thus removing the effects of model drift and bias.

## Canonical Variate Analysis

**Notes:**
The canonical variate procedure was applied in different ways. The first distinction is between applying the procedure to the forecasts from all three models together (the “combined” approach), and applying it to each model individually and then equally averaging the results from the three models (the “averaged” approach). The averaged results (thick lines and top frequency histograms) are clearly superior to the combined results (thin lines and bottom histograms), indicating that the best approach is to recalibrate the models based on their historical skill, and to then apply an equal weighting to the models. The second distinction is between the actual predictors used in the canonical variate scheme: predictors included the individual ensemble members sorted so that the first ensemble member always represents the coldest forecast (“members”); the first few principal components of the sorted ensemble members (“PCs” –up to 4 components were considered); and the first few moments of the ensemble members (“moments” – again up to 4 moments were considered). For both the PCs and the moments, results are shown based on only the first PC/moment. In most cases using only the first moment gives the best results, although adding additional moments has little effect. Differences in results between the different predictors are small compared to differences between the averaged and the combined ensembles. An eyeball analysis suggests that the individual ensemble members may give the best results, although based on scoring metrics (which include consideration of the neutral category) the use of the ensemble moments has a slight edge.
Results using canonical variate analysis are somewhat sensitive to the number of canonical variates used. The selection procedure adopted involved rejection of any canonical variates with a canonical correlation that was not significant at the 95% level of confidence. If no canonical correlations were statistically significant the raw probabilities were retained.

## Multiple Linear Regression

**Notes:**
As with the canonical variate analysis, multiple linear was applied in different ways. The first distinction is between applying the procedure to the forecasts from all three models together (the “combined” approach), and applying it to each model individually and then equally averaging the results from the three models (the “averaged” approach). The averaged results (thick lines and top frequency histograms) are clearly superior to the combined results (thin lines and bottom histograms), indicating that the best approach is to recalibrate the models based on their historical skill, and to then apply an equal weighting to the models. The second distinction is between the actual predictors used in the multiple regression scheme: predictors included the individual ensemble members sorted so that the first ensemble member always represents the coldest forecast (“members”); the first few principal components of the sorted ensemble members (“PCs” –up to 4 components were considered); and the first few moments of the ensemble members (“moments” – again up to 4 moments were considered). For both the PCs and the moments, results are shown based on only the first PC/moment. Generally using only the first moment gives the best results. Differences in results between the different predictors are small compared to differences between the averaged and the combined ensembles. The use of the first three ensemble moments gives marginally the best results, which are a slight improvement over the bayesian optimal weighting.
Note that the best reliability is obtained using the individual ensemble members, but that resolution is relatively poor.

## Conclusions I

**Conclusions I**

## Terrestrial Climate

## Data

- AGCMs: *Simulations for 1950-2000*
- 		* CCM3 (NCAR) – 24 runs
- 		* ECHAM4.5 (MPI) – 24 runs	* ECPC (Scripps) – 10 runs
- 		* GFDL AM2p12b – 10 runs
- 		* NCEP-MRF9 (NCEP/QDNR) – 10 runs
- 		* NSIPP1 (NASA-GSFC) – 9 runs
- Observations:
- 	2m Air Temperature and Precipitationfrom CRU-UEA (v2.0)

**Notes:**
The model data is taken from 3 atmospheric GCMs. The CCM3.6 (Hack et al., 1998; Hurrell et al., 1998; Kiehl et al., 1998) and the ECHAM4.5 (Roeckner et al., 1996) AGCMs are configured at T42 horizontal resolution, with 18 vertical levels. The NSIPP1 AGCM (Bacmeister et al., 2000) is a grid point model with a horizontal resolution of 2x2.5 degrees and 34 levels in the vertical. The NSIPP1 model data was linearly interpolated to the T42 grid for purposes of combining the models.

The observations for temperature and precipitation, against which the model combinations are validated, are from the Climate Research Unit of the University of East Anglia (New et al., 1999; New et al., 2000). The data are provided at 0.5x0.5 degree resolution, and were upscaled to T42 for use in the validation.

Note, all AGCM predictions(/simulations) are cast as probabilistic 3-category forecasts for 3-monthly seasonal values of mean temperature and total rainfall, where the prior probability of any 1 category is 1/3. Because seasonal forecasts are probabilistic, they will be assessed as such. The goal is to produce reliable forecasts with good resolution, where the forecast probabilities mean what they say.

Bacmeister, J., P.J. Pegion, S.D. Schubert, and M.J. Suarez, 2002: Atlas of Seasonal 	Means Simulated by the NSIPP 1 Atmospheric GCM. NASA/TM-2000-104505, Vol. 	17, 190pp.
Hack, J.J., J.T. Kiehl, and J.W. Hurrell, 1998:The hydologic and thermodynamic 	characteristics of the NCAR CCM3. J. Climate, 11, 1179-1206.
Hurrell, J.W., J.J. Hack, B.A. Boville, D.L. Williamson, and J.T. Kiehl, 1998: The dynamical 	simulation of the NCAR Community Climate Model version 3 (CCM3). J. Climate, 	11, 1207-1236.
Kiehl, J.T., J.J. Hack, G.B. Bonan, B.A. Boville, D.L. Williamson, and P.J. Rasch, 1998: The 	National Center for Atmospheric Research Community Climate Model, J. Climate, 	11, 1131-1149.
New, M., M. Hulme, and P.D. Jones, 1999: Representing twentieth-century space-time 	climate variability. Part I: Development of a 1961-90 mean monthly terrestrial 	climatology. J. Climate, 12, 829-856.
New, M., M. Hulme, and P.D. Jones, 2000: Representing twentieth-century space-time 	climate variability. Part II: Development of a 1901-96 monthly grid of terrestrial 	surface climate. J. Climate, 13, 2217-2238.
Roeckner, E., K. Arpe, L. Bengtsson, S. Brinkop, L. Dumenil, M. Esch, E. Kirk, F. Lunkeit, 	M. Ponater, B. Rockel, R. Sausen, U. Schlese, S. Schubert, and M. Windelband, 	1996: Simulation of the present-day climate with the ECHA< model: Impact of 	model physics and resolution. Report No. 93, Max-Planck-Institut fur Meteorologie, 	Hamburg, Germany, 171pp

## Effect of Probability TreatmentJFM 2m air temperature over land

**Notes:**
So how do these approaches (more vs. less simple) compare? First an overall answer is provided by use of reliability diagrams.

Figure caption: Reliability diagrams for global precipitation forecasts. Each graph plots the forecast probability versus the observed relative frequency for a given category (above-normal, upper row; below-normal, lower row). Different forecast methods are shown in the colored lines: individual AGCMs (blue lines); Pooled multi-model probabilities (black line); Bayesian multi-model probabilities (green line). In the left column, the probabilities from the models are used as is; in the right column, the probabilities are first refined through the contingency table approach. The inset graphs shows the frequency with which the range of forecast probabilities were assigned.

In this case, either correcting the PDF before combination, or using the Bayesian approach with the uncorrected probabilities dramatically improve over the pooling of the raw AGCM probabilities (which is itself some improvement over the individual AGCMs).

## Effect of Probability Treatment

## Effect of Combination Method

**Notes:**
So how do these approaches (more vs. less simple) compare? First an overall answer is provided by use of reliability diagrams.

Figure caption: Reliability diagrams for global precipitation forecasts. Each graph plots the forecast probability versus the observed relative frequency for a given category (above-normal, upper row; below-normal, lower row). Different forecast methods are shown in the colored lines: individual AGCMs (blue lines); Pooled multi-model probabilities (black line); Bayesian multi-model probabilities (green line). In the left column, the probabilities from the models are used as is; in the right column, the probabilities are first refined through the contingency table approach. The inset graphs shows the frequency with which the range of forecast probabilities were assigned.

In this case, either correcting the PDF before combination, or using the Bayesian approach with the uncorrected probabilities dramatically improve over the pooling of the raw AGCM probabilities (which is itself some improvement over the individual AGCMs).

## Effect of Combination MethodRAW Probabilities

## Effect of Combination MethodPDF Probabilities

## Conclusions II

- Reliability of **N** models pooled together, with uncalibrated PDFs, is better than any individual AGCM.
- Gaussian (PDF) recalibration gives some improvement, but Bayesian recalibration gives the greatest benefit.
- Reliability is typically gained at the expense of resolution.

## ISSUES

-  Number of Models
-  Length of training period
-  When simple is complex enough?

## Effect of # of Models3 vs 6 AGCMS; 45-year training period

- (*Robertson et al., 2004, MWR)*

- Different approaches are more similarwith more models.

## RPSS for 2m TemperatureBayesian MM from Raw Probs. – 6 models, 45-yr training

- _Jan-Feb-Mar_

- _Jul-Aug-Sep_

## RPSS for PrecipitationBayesian & Pooled MM from Raw Probs. – 6 models, 45-yr training

- _Jan-Feb-Mar_

- _Jul-Aug-Sep_

## Reliability Diagrams* several methodsyield similar results overthe United States.

_**Reliability Diagrams**_* several methodsyield similar results overthe United States.
- * MMs are remarkablyreliable over the US,even though the accuracyis not high.

## CONCLUSIONS III

- MM simulations over the US are remarkably reliable, even if their not terribly accurate.
- Simple pooling of the AGCMs, with uncalibrated probabilities, is equivalent to any of our techniques over the U.S.
-  Doesn’t require long history, but largenumber of models (>5?) is desirable.

## GRAND CONCLUSIONS

- Overall, we find that **recalibrating individual models gives better results** than putting models together in complex combination alorithm.
- In comparing different recalibration/combination methods, we find that generally **a gain in reliability is countered with a loss in resolution**.
- More **complicated approaches are not necessarily better**. This needs to investigated for different forecast situations (i.e. variables, region, season).

## Ranked Probability Skill Scores

**Temperature**
- Jan-Feb-Mar (1950-1995)

**Notes:**
How do these apparent improvements in overall reliability impact probabilistic skill locally?

Figure caption: Maps of RPSS for temperature during the JFM season 1950-95. The maps on the left are from the pooled combination of the raw AGCM probabilities (top); the Bayesian combination of the raw AGCM probabilities (middle); and the pooled combination of the corrected AGCM probabilities (bottom). On the right, the more sophisticated multi-model approaches are contrasted with the baseline approach of pooling the raw probabilities.

All 3 maps on left look similar, although one may notice stronger negative RPSS values over some areas on the raw-pooled map.  In fact, if we look at the difference between the “new” and the “baseline” approach, one sees that most improvement is realized over the negative areas on the “baseline” map (but not always).

## Ranked Probability Skill Scores

**Precipitation**
- Jul-Aug-Sep (1950-1999)

- _Comparing treatment of probability_
- - Even with 6 models, have regionsof large negative RPSS Suggests common model errors
-  Recalibration reduces, but does noteliminate, large errors
-  Some improvement of positive skill

- Recal-Raw

**Notes:**
How do these apparent improvements in overall reliability impact probabilistic skill locally?

Figure caption: Maps of RPSS for temperature during the JFM season 1950-95. The maps on the left are from the pooled combination of the raw AGCM probabilities (top); the Bayesian combination of the raw AGCM probabilities (middle); and the pooled combination of the corrected AGCM probabilities (bottom). On the right, the more sophisticated multi-model approaches are contrasted with the baseline approach of pooling the raw probabilities.

All 3 maps on left look similar, although one may notice stronger negative RPSS values over some areas on the raw-pooled map.  In fact, if we look at the difference between the “new” and the “baseline” approach, one sees that most improvement is realized over the negative areas on the “baseline” map (but not always).

## Ranked Probability Skill Scores

**Precipitation**
- Jul-Aug-Sep (1950-1999)

- _Comparing combination methods_
-  Performance-based combination eliminates large errors
-  More improvement of positive skill
-  More cases of negative skill turned topositive skill

**Notes:**
How do these apparent improvements in overall reliability impact probabilistic skill locally?

Figure caption: Maps of RPSS for temperature during the JFM season 1950-95. The maps on the left are from the pooled combination of the raw AGCM probabilities (top); the Bayesian combination of the raw AGCM probabilities (middle); and the pooled combination of the corrected AGCM probabilities (bottom). On the right, the more sophisticated multi-model approaches are contrasted with the baseline approach of pooling the raw probabilities.

All 3 maps on left look similar, although one may notice stronger negative RPSS values over some areas on the raw-pooled map.  In fact, if we look at the difference between the “new” and the “baseline” approach, one sees that most improvement is realized over the negative areas on the “baseline” map (but not always).

## Canonical Variate Analysis

- A number of statistical  techniques involve calculating linear combinations (weighted sums) of variables. The weights are defined to achieve specific objectives:
-   PCA – weighted sums maximize *variance*
-   CCA – weighted sums maximize *correlation*
-   CVA – weighted sums maximize *discrimination*

## Canonical Variate Analysis

**Canonical Variate Analysis**

## Effect of Probability Treatment

- Effect of Probability Treatment

**Notes:**
So how do these approaches (more vs. less simple) compare? First an overall answer is provided by use of reliability diagrams.

Figure caption: Reliability diagrams for global precipitation forecasts. Each graph plots the forecast probability versus the observed relative frequency for a given category (above-normal, upper row; below-normal, lower row). Different forecast methods are shown in the colored lines: individual AGCMs (blue lines); Pooled multi-model probabilities (black line); Bayesian multi-model probabilities (green line). In the left column, the probabilities from the models are used as is; in the right column, the probabilities are first refined through the contingency table approach. The inset graphs shows the frequency with which the range of forecast probabilities were assigned.

In this case, either correcting the PDF before combination, or using the Bayesian approach with the uncorrected probabilities dramatically improve over the pooling of the raw AGCM probabilities (which is itself some improvement over the individual AGCMs).