# Presentation: 160405

## Consolidation methods for SST monthly forecasts for MME

**Consolidation methods for SST monthly forecasts for MME**

**Acknowledgments: Suru Saha retrieved and organized the data, Dave Unger and Peitao Peng provided discussion to the subject **

**3****rd**** CTB Science Advisory Board Meeting, August 28-29, 2007**

## Making the best single forecast out of a number of forecast inputs
Necessary as large supply of forecasts available
Expressed as a linear combination of participant models:

- Necessary as large supply of forecasts available
- Expressed as a linear combination of participant models:

**Consolidation**

**Data:**      *Nine ensemble prediction systems** **(DEMETER+CFS+CA)*
- At least 9 ensemble members per model
- Hindcast length: Twenty-one years (1981-2001)
- Monthly mean forecasts; Leads 0 to 5 months
- Four initial month: Feb, May, Aug, Nov

*   **K*  number of participating models
*   **ζ**   *input forecast at a particular initial month and lead time

**Task:** Finding *K* optimal weights, *α**k* , corresponding to each input model

## Tropical Pacific SST

**Tropical Pacific SST**

-  Region of appreciable skill
-  Multi-Model Ensemble Average (MMA) more skillful than any single ensemble model average
-  Can sophisticated consolidation methods be better?

**Model Ensemble average**

**MMA**

**Pattern Anomaly Correlation.**
**Average over all leads and months.**
**Full data**** **

**Anomaly Correlation gridpointwise of MMA**

**12.5****o****N**

**12.5****o****S**

**82.5****o****W**

**140****o****E**

## Issues

**HINDCAST should be sufficiently long to**
- Remove Systematic Errors
- Carry out weight optimization procedures (consolidation)
- Allow cross-validation assessment
- A CV-1 produce degeneracy; a CV-3yrs out will leave only 18 data points.
**OVERFITTING: **Due to short training dataset compared to the number of input forecast models.
- Rigorous cross-validation procedure to measure realistic skill
**COLLINEARITY: **A large number of participating models may lead to problems when at least one of the models is not independent from the rest.
- Even with plentiful data
- Covariance matrix ill-conditioned:  Regression coefficients not accurately computed
- Regularization methods required

## Ridging Methods

**Ridging Methods**

**RIDGING**

**RIDGING w/ MMA constraint**

**RIDGING w/ **
**weighted mean constraint**

- Climatology

***1/K***

- Optimized weights for the 9 model ensemble averages
- as a function of the ridging amount: λ

- Weights approach to skill (weighted) of corresponding model

## Consolidation Methods Assessed

| Acronym | Method | Characteristics |
| --- | --- | --- |
| MMA | Multi-method ensemble average | Simple average of all the input forecasts |

| UR | Unconstrained regression | Typical multiple linear regression. Assumes no collinearity among models. Weights sometimes negative and too large |
| --- | --- | --- |
| COR | Correlation coefficient | Skill weighted method. Collinearity among models is not considered |
| FRE | Frequency of the best | Gives weights depending on how many times the model has been the best in the training period. |

| RID | Ridging | Considers both skill and collinearity among models |
| --- | --- | --- |
| RI2 | Double pass Ridging | First pass to identify unskillful and/or redundant models. Then, set their corresponding weights to zero and perform a second pass with the reduced set of models.  |
| RIM | Ridging with MMA penalty | Ridging with a penalty term for weights departing from MMA   |
| RIW | Ridging with COR penalty | Ridging with a penalty term for weights departing from COR |

## Pattern Anomaly Correlation

- Average over all leads and initial months

**Consolidation methods**

## INCREASING EFFECTIVE SAMPLE SIZE 

-  Selection-combination (double pass strategy)
-  Objective procedure to remove or set to zero weights of bad or redundant models
-  First pass: Ridging identifies negative weights. Set these to zero
-  Second pass: Ridging is carried out on the models with positive weight
-  Mixing data from neighboring gridpoints
-  Previous studies show a gain in weight stability
-  Reduces flexibility in cases where the raking of the model skill changes from region to region
-  If hindcasts allowed, mixing neighboring lags
-  Mixing information from individual ensemble members
-  Most studies have used the ensemble average for each model
-  Each member is a unique realization of model
-  Constrain weights of ensemble members to be same within each model

**APPROACHES**

## INCREASING EFFECTIVE SAMPLE SIZE 

**INCREASING EFFECTIVE SAMPLE SIZE **

- Gridpoint by gridpoint
- 3x3 box that includes the point of analysis plus the 8 closest gridpoints
- 9x9 box with the 80 closest neighboring gridpoints
- All the grid points in the domain

**Approaches**

**USING ENSEMBLE AVERAGES ONLY**

**USING ALL ENSEMBLE MEMBERS**

**MMA**

## INCREASING EFFECTIVE SAMPLE 

**INCREASING EFFECTIVE SAMPLE **

**Consistency: Percentage number of cases that a sophisticated consolidation method outperforms MM**

**Consolidation method**

**Percentage**

- Only ensemble means and gridpoint by gridpoint

- All ensemble members and closest neighboring gridpoint

- All ensemble members and all gridpoints in the domain

**Sophisticated consolidation methods more frequently outperform MMA for larger effective sample **

## Performance

**Performance**

- Ridging AC improves considerably over western Pacific

**AC for MMA. All initial months included**

**AC: RID minus MMA (%). All initial months included**

## Probabilistic assessment 

-  PDF formation from optimized weights
-  _Method1:_ Weights used as factors that multiply corresponding forecast values.
-  Width of PDF becomes too narrow when weights are small.
-  Optimization to inflate PDF required
-  _Method2:_ Weights used to determine stacks given to corresponding model without changing the value of the forecast
-  Both methods produce similar results with the latter more strightforward
-  Relative Operating Characteristic Curve:
-  To measure the ability of method to predict occurrence or non-occurrence that observation will fall in the “upper”, “middle” or “lower” tercile.
-  Class limits defined by the observed SST during the training period

## PDF formation 

**PDF formation **

**observation**

**3/81**

**11/81**

**67/81**

**0**

**22/81**

**59/81**

**0**

**.057**

**0.942**

**Forecasts**

- Illustration for a particular gridpoint, lead and initial month

***p *****= fraction of ensemble members**

- Observed climate

**Bars’ height depends **
**on the weights**

- Method 1: weights are
- multiplicative corrections to forecasts

- Method 2: weights determine stacks

- Equal Weights Method

**Weights are: 0.1532    0.1946    0.0275    0.1402    0.1014    0.0000    0.0646    0.1393    0.1792**

## Area below ROC curve 

**Area below ROC curve **

- Regions where RID outperforms MMA are shaded

- Lower tercile

- Middle tercile

- Upper tercile

- MMA

- RID minus MMA

## Summary of results

-  Eight consolidation methods assessed under rigorous cross-validation procedure, CV-3RE, to combine up to 81 monthly forecasts of tropical Pacific SST.
-  The simple multi-methods ensemble average (MMA) shows large and consistent skill improvement over individual participant models as measured by AC.
-   When the sampling size is small sophisticated consolidation methods are as skillful as (or, in the case of UR, worse than) MMA
-   Increasing the effective sampling size produces more stable weights and affects positively the skill of sophisticated consolidation
-   In the western tropical Pacific, sophisticated consolidation methods improve significantly over MMA. This is true both using a deterministic (AC) and Probabilistic (ROC) assessment.

## Remarks

-   Hindcast
-  MMA requires long hindcast to remove SE
-  Sophisticated consolidation methods need sufficiently long hindcast not only to remove SE but to optimize weights
-   Cross-validation
-  Rigorous CV-3RE shows artificial skill prevalent in most of the consolidation methods
-   Probability Density Function
-  Efficient treatment is needed to adjust posterior PDF in ridge regression methods.
-  Gaussian Kernel, Bayesian Methods
-  SE of the standard deviation and other higher moments of the PDF: hard to obtain given shortness of the hindcasts
-   Application dependent
-  Results for SST monthly predictions (high skill and large collinearity) may not apply to consolidation of other variables