# Presentation: 817368

## Consolidation of Multi Method Forecasts

**Consolidation of Multi Method Forecasts**

**Application to monthly predictions of Pacific SST**

**NCEP Climate Meeting, April 4, 2007**

**Acknowledgments: Suru Saha retrieved and organized the data, Dave Unger and Peitao Peng provided discussion to the subject **

## DATA

- Forecasting tools: 8 CGCMs, 1 Statistical model
- NCEP CFS: 1981-2006, 15 membs, 9 leads
- DEMETER : 1980-2001, 9 membs, 6 leads
- ECMWF
- MPI
- MF
- UKMO
- INGV
- LODYC
- CERFAX
- CPC’ Constructed Analog (CA) : 1956-2006, 12 membs,12 leads
- This is what all have in common:
- Monthly Forecasts, leads 0 to 5
- Initial months: Feb, May, Aug, Nov
- Length of retrospective forecasts: 21 years, 1981-2001

**FOCUS: TROPICAL PACIFIC SST: 12.5 S TO 12.5 N **

## Consolidation: Making the best single forecast out of a number of forecast inputs.
Objective consolidation necessary as large supply of forecasts are available.
If K is the number of participant forecast systems, ζ, predicting a particular target month with a given lead time, the consolidation is the following linear combination:

- Objective consolidation necessary as large supply of forecasts are available.
- If *K* is the number of participant forecast systems, *ζ*, predicting a particular target month with a given lead time, the consolidation is the following linear combination:

**DEFINITIONS**

- For convenience, systematic errors and observed climatology are removed in *ζ*.
- The regression coefficients (weights), α, are based on past performance of the forecast system.
***o*** is the verifying field (e.g. observation; climatology removed). Suppose there are *N* cases of retrospective forecasts, then one can train a consolidation method by comparing:

## OPTIMIZING WEIGHTS

**OPTIMIZING WEIGHTS**

-  Find weights, *α**i* ,for each forecasting tool, *ζ**i*, that minimizes the (sum of square of) errors *ε**j* in

- Where **Z** is a matrix whose columns are the forecasting tools and rows are the data points in the training period, ***o*** is the column vector containing the verifying field, and ***ε*** is a vector of errors.

-  Least square method (unconstrained regression):

## ILL-POSED MATRIX PROBLEM

| eigenvalues | Nino 3.4 | PNA | NAO |
| --- | --- | --- | --- |
| 1 | 8.4584 | 5.9156 | 3.6889 |
| 2 | 0.1763 | 0.8394 | 1.402 |
| 3 | 0.1516 | 0.7808 | 1.1173 |
| 4 | 0.0707 | 0.42 | 0.8759 |
| 5 | 0.0536 | 0.3488 | 0.6316 |
| 6 | 0.0384 | 0.2874 | 0.5277 |
| 7 | 0.0297 | 0.1919 | 0.3978 |
| 8 | 0.0186 | 0.139 | 0.2462 |
| 9 | 0.0027 | 0.0772 | 0.1126 |

| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 0.482 | 0.2532 | -0.5526 | -0.5615 | 0.0189 | 0.0348 | 0.018 | 0.0381 | 0.0488 |

- Corresponding weights for UR for lead 1, im 1

**ILL-POSED MATRIX PROBLEM**

**too large**

## RIDGE REGRESSION

**RIDGE REGRESSION**

- Constrained to:

- Minimize:

- leads to

- Ridge Regression

- (DelSole, 2007)

- (ad hoc)

- where

- and

-  Van den Dool estimates  such that the weights are small and stable
-  Many more ways to find it
-  Depends on characteristics of covariance matrix *Z**T**Z*

## RIDGE REGRESSION

-  Model weights (*α**i**, i=1..9*) as a function of **λ**  for three ridge consolidation methods.
-  Figure illustrates asymptotic values. Our methods stop at **λ****=0.5**.
-  Unconstrained regression (**λ****=0**) results in a wide range (including negative values) of weights.

## CONSOLIDATION METHODS ASSESED

**CONSOLIDATION METHODS ASSESED**

## CROSS-VALIDATION

- Anomaly Pattern correlation over the tropical Pacific. Average for all leads and initial months. Empty bar: Full (dependent), filled bar: 3-yr out cross-validated.

## GRIDPOINT BY GRIDPOINT PERFORMANCE

## EQUATORIAL PACIFIC

**EQUATORIAL PACIFIC**

## WESTERN TROPICAL PACIFIC

**WESTERN TROPICAL PACIFIC**

**Trust in good models when performed well in a gridpoint. It goes to the opposite direction of the bad models**

## WESTERN TROPICAL PACIFIC

**WESTERN TROPICAL PACIFIC**

**MIXES CLOSEST NEIGHBORING GRIDPOINT**

**Trust in good models when performed well in a 3x3 box of gridpoints. It goes to the opposite direction of the bad models**

## WESTERN TROPICAL PACIFIC

**WESTERN TROPICAL PACIFIC**

**Trust less good models, damps towards climatology as negative weights are set to zero**

**DOUBLE PASS AND MIXES CLOSEST NEIGHBORING GRIDPOINT**

## INCREASING EFFECTIVE SAMPLE 

**INCREASING EFFECTIVE SAMPLE **

**1 ****	****GRIDPOINT BY GRIPOINT**
**3X3 BOXES**
**5X5 BOXES **
**ALL GRIDPOINTS IN THE DOMAIN**
**GRIDPOINTS IN AND OUT DOMAIN**

**Multi-methods average**

**AC**

**Skill of most consolidation methods improve when effective sampling size increases**

**Tropical Pacific SST. AC average for all leads and initial months**

## INCREASING EFFECTIVE SAMPLE 

**INCREASING EFFECTIVE SAMPLE **

**Consistency: Percentage cases (leads and initial months) outperforming MM**

## RELATIVE OPERATIONING CURVES 

-  Assess the ability to anticipate correctly the occurrence or non occurrence that SST anomalies will fall in the upper, middle and lower terciles.
-  Class limits defined by the observed SST during the training period
-  Probability information from the ensemble: counting the fraction of ensemble members that falls into the “above-normal”, “near-normal”, and “below-normal” categories, and interpreting this fraction as the probability that forecasts will fall in such categories.
-  Approach for the optimized weights: each ensemble member forecast is multiplied by normalized weights.

**Lead 3**

**Upper tercile**

**Lower tercile**

## Slide 18

## LOWER TERCILE

## SUMMARY

**All the points below are for the particular case of SST anomalies in the tropical Pacific.****  **
**  ****Forecasts arising from a combination of multiple models of similar skill generally outperform those from individual models ****but not UR after CV-3.**
**  ****Even the simple average of multi-methods shows consistent improvement over individual participant models.**
**  ****Over all and after cross-validation, sophisticated consolidation methods marginally improve over the simple average.**
**  ****Increasing the effective sampling size increases the skill and consistency of consolidation methods.**
**  ****Consolidation methods improve significantly over the multi-methods average in the western Pacific.**
**  ****Probabilistic assessment, as measured by ROC shows some improvement of consolidation methods over MM. Construction of the probability density function of the consolidation requires optimization. **