# Presentation: 924850

## Anthony Barnston, Lisa Goddard, Simon Mason and Andrew Robertson 

International Research Institute
for Climate and Society (IRI) 

- International Research Institute
- for Climate and Society (IRI)

- Consolidation of Predictions of Seasonal Climate by Several Atmospheric General Circulation Models at IRI

##                   2-tiered

- 12

- 30

- 24

- 12

- 24

- 24

- 24

- 10

**        ****FORECAST SST**
**  ****TROP. PACIFIC:     ****THREE scenarios**
**  ****(multi-models, dynamical**
**          ****and  statistical)**
**    ****TROP. ATL, INDIAN**
**         ****(****ONE**** statistical)**
**     ****EXTRATROPICAL **
**    ****(damped persistence**)

**GLOBAL **
**ATMOSPHERIC**
**MODELS**
**   ****ECPC(Scripps)**
**   ****ECHAM4.5(MPI)**
**      ****CCM3.6(NCAR)**
**        ****NCEP(MRF9)**
**        ****NSIPP(NASA)**
**               ****COLA2**
**                ****GFDL**

- Forecast
- SST
- Ensembles
- 3/6 Mo. lead

- Persisted
- SST
- Ensembles
- 3 Mo. lead

_**IRI DYNAMICAL CLIMATE FORECAST SYSTEM**_

**POST**
**PROCESSING**
**MULTIMODEL**
**ENSEMBLING**
**-Bayesian**
**-Caninical**
** ****variate**

**PERSISTED **
**GLOBAL**
**SST**
**ANOMALY**

-                   2-tiered
**    ****OCEAN           ATMOSPHERE**

- 30

##                                           2-tiered

**  ****TROP. PACIFIC:****     ****THREE scenarios:**
**  ****1) CFS prediction     2) LDEO prediction**
**         ****3) Constructed Analog prediction**
**    ****TROP. ATL, and INDIAN oceans**
**        ****CCA, or slowly damped persistence**
**     ****EXTRATROPICAL **
**        ****damped persistence**

**GLOBAL **
**ATMOSPHERIC**
**MODELS**
**   ****ECPC(Scripps)**
**   ****ECHAM4.5(MPI)**
**      ****CCM3.6(NCAR)**
**        ****NCEP(MRF9)**
**        ****NSIPP(NASA)**
**               ****COLA2**
**                ****GFDL**

_**IRI DYNAMICAL CLIMATE FORECAST SYSTEM**_

**PERSISTED **
**GLOBAL**
**SST**
**ANOMALY**

-                                           2-tiered
**              ****OCEAN                                      ATMOSPHERE**

## Six GCM Precip. Forecasts, JAS 2000

## RPSS Skill of Individual Models: JAS 1950-97

## Goals

- To combine the probability forecasts of several
- models, with relative weights based on the
- past performance of the individual models
- To assign appropriate forecast probability
- distribution: e.g. damp overconfident forecasts
- toward climatology

## Probabilities and Uncertainty

- 1/3

- 1/3

- 1/3

- 6/24

- 8/24

- 10/24

- Above Normal

- Near-Normal

- Below Normal

- Climatological
- Probabilities

- GCM
- Probabilities

- Tercile boundaries are identified for the models’ own climatology, by aggregating all years and ensemble members. This corrects overall bias.

- k = tercile number
- t = forecast time
- m = no. ens members

## Bayesian Model Combination

- Combine climatology forecast (“prior”) and an AGCM  forecast, with its evidence of historical skill, to produce weighted (“posterior”) forecast probabilities, by maximizing the historical likelihood score.

## Aim to maximize the

*that were hindcast for the category *
*that was observed.*
- (Could maximize other scores, such as RPSS)
- Prescribed, observed SST used to force AGCMs.
- Such simulations used in absence of ones using
- truly forecasted SST for at least half of AGCMs.

- likelihood score

-       k=tercile category
- t=year number

## PMMkt= weighted linear

**1. Calibration of each model, *****individually*****, against climatology**

**2. Calibration of the weighted model combination against climatol**

-                       k=tercile category (1,2, or 3)
- t=year number
-                j=model number (1 to 7)
-                                 w=weight for climo (c) or for model j

- where wMM uses wj proportional to results of the first step above

- PMMkt= weighted linear
- comb of Pjkt over all j,
- normalized by Σ(wj)

- Optimize
- likelihood
- score

- Optimize
- likelihood
- score

## Algorithm used to maximize the designated score:

- Feasible Sequential Quadratic Programming (FSQP)
- “Nonmonotone line search for minimax problems”

- C   M :        TOTAL NUMBER OF CONSTRAINTS.
- C   ME :       NUMBER OF EQUALITY CONSTRAINTS.
- C   MMAX :     ROW DIMENSION OF A. MMAX MUST BE AT LEAST ONE AND GREATER
- C              THAN M.
- C   N :        NUMBER OF VARIABLES.
- C   NMAX :     ROW DIMENSION OF C. NMAX MUST BE GREATER OR EQUAL TO N.
- C   MNN :      MUST BE EQUAL TO M + N + N.
- C   C(NMAX,NMAX): OBJECTIVE FUNCTION MATRIX WHICH SHOULD BE SYMMETRIC AND
- C              POSITIVE DEFINITE. IF IWAR(1) = 0, C IS SUPPOSED TO BE THE
- C              CHOLESKEY-FACTOR OF ANOTHER MATRIX, I.E. C IS UPPER
- C              TRIANGULAR.
- C   D(NMAX) :  CONTAINS THE CONSTANT VECTOR OF THE OBJECTIVE FUNCTION.
- C   A(MMAX,NMAX): CONTAINS THE DATA MATRIX OF THE LINEAR CONSTRAINTS.
- C   B(MMAX) :  CONTAINS THE CONSTANT DATA OF THE LINEAR CONSTRAINTS.
- C   XL(N),XU(N): CONTAIN THE LOWER AND UPPER BOUNDS FOR THE VARIABLES.
- C   X(N) :     ON RETURN, X CONTAINS THE OPTIMAL SOLUTION VECTOR.
- C   U(MNN) :   ON RETURN, U CONTAINS THE LAGRANGE MULTIPLIERS. THE FIRST
- C              M POSITIONS ARE RESERVED FOR THE MULTIPLIERS OF THE M
- C              LINEAR CONSTRAINTS AND THE SUBSEQUENT ONES FOR THE
- C              MULTIPLIERS OF THE LOWER AND UPPER BOUNDS. ON SUCCESSFUL
- C              TERMINATION, ALL VALUES OF U WITH RESPECT TO INEQUALITIES
- C              AND BOUNDS SHOULD BE GREATER OR EQUAL TO ZERO.

## Circumventing the effects of sampling variability

- Sampling variability appears to be an issue: noisy weight distribution with large number of zero weights and some unity weights
- Bootstrap the optimization, omitting contiguous 6-year blocks of the 48-yr time series
- yields 43 samples of 42 years
- shows the sampling variability of the likelihood over subsets of years
- We average the weights across the samples

## Example

- Six GCMs’ Jul-Aug-Sep precipitation simulations
- Training period: 1950–97
- Ensembles of between 9 and 24 members

## Model Weights – initially, by individual model

## Climatological Weights – Multi-model

## Model Weights – after second (damping) step

## Model Weights – step 2, and Averaged over Subsamples

## X  X  X

- weighting of each grid point is averaged
- with that of its 8 neighbors, using
- binomial weighting.
- X  X  X

## Climatological  Weights 

- Climatological  Weights

## Combination Forecasts of July-Sept Precipitation

- After first stage only

- After second (damping) stage

- After spatial smoothing

- After sampling subperiods

## ReliabilityJAS Precip., 30S-30N

- Above-Normal

- Below-Normal

- Forecast probability

- Forecast probability

- Observed relative Freq.

- Observed relative Freq.

- Bayesian

- Pooled

- (3-model)

- from Goddard et al. 2003

- Individual
- AGCM

## RPSS Precip.

- from Roberson et al. (2004):
*Mon. Wea. Rev*.,
**132**, 2732-2744

## RPSS 

- 2-m Temp.

- from Roberson et al. (2004):
*Mon. Wea. Rev*.,
**132**, 2732-2744

## Conclusions - Bayesian

- The “climatological” (equal-odds) forecast provides a useful prior for combining multiple ensemble forecasts
- Sampling problems become severe when attempting to combine many models from a short training period (“noisy weights”)
- A two-stage process combines the models together according to a pre-assessment of each against climatology
- Smoothing of the weights across data sub-samples and spatially appears beneficial

## IRI’s forecasts use also a second consolidation

- scheme, whose result is averaged with
- the result of the Bayesian scheme.
- 1. Bayesian scheme
- 2. Canonical Variate scheme

## Canonical Variate Analysis (CVA)

- A number of statistical  techniques involve calculating linear combinations (weighted sums) of variables. The weights are defined to achieve specific objectives:
-   PCA – weighted sums maximize *variance*
-   CCA – weighted sums maximize *correlation*
-   CVA – weighted sums maximize *discrimination*

## Canonical Variate Analysis

**Canonical Variate Analysis**

## Canonical Variate Analysis

**Canonical Variate Analysis**

- The canonical variates are defined to maximize the ratio of the *between-category* (separation between the crosses) to the *within-category* (separation of dots from like-colored crosses) variance.

## Conclusion

- IRI presently using a 2-tiered prediction system.
- It is interested in using fully coupled systems also,
- and is looking into incorporating those.
- Within its 2-tiered system it uses 4 SST prediction
- scenarios, and combines the predictions of 7 AGCMs.
- The merging of 7 predictions into a single one uses
- two multi-model ensemble systems: Bayesian and
- canonical variate. These give somewhat differing solutions, and are presently given equal weight.