# Presentation: 653514

## Verification of IRI Forecasts

- 1997 - 2008

- Tony Barnston and Shuhua Li

**Notes:**
Regional Projects, through regional COFs
Sectoral workshops (agric, hydrol, health, coastal management
Visitors to/from regions

## IRI’s Forecast System

- IRI has been using a 2-tiered prediction system to probabilistically predict global temperature
- and precipitation with respect to terciles of the
- historical climatological distribution.
- Within the 2-tiered system IRI uses
- 4 SST prediction scenarios, and combines the ensemble predictions from 7 AGCMs.
- Predictions from the 7 AGCMs are merged into a single one using multi-model ensembling.

##                   2-tiered

- 12

- 30

- 24

- 12

- 24

- 24
- 10

- 24

- 10

**        ****FORECAST SST**
**  ****TROP. PACIFIC: ****THREE**
**  ****(multi-models, from 2 dyn**
**  ****and 1 statistical model)**
**    ****TROP. ATL, INDIAN**
**   ****Similar to trop. Pacific    **
**     ****EXTRATROPICAL **
**    ****(damped persistence**)

**GLOBAL **
**ATMOSPHERIC**
**MODELS**
**   ****ECPC(Scripps)**
**   ****ECHAM4.5(MPI)**
**      ****CCM3.6(NCAR)**
**        ****NCEP(MRF9)**
**        ****NSIPP(NASA)**
**               ****COLA2**
**                ****GFDL**

- Forecast
- SST
- Ensembles
- 3/6 Mo. lead

- Persisted
- SST
- Ensembles
- 3 Mo. lead

_**IRI DYNAMICAL CLIMATE FORECAST SYSTEM**_

**POST**
**PROCESSING**
**MULTIMODEL**
**ENSEMBLING**

**PERSISTED **
**GLOBAL**
**SST**
**ANOMALY**

-                   2-tiered
**    ****OCEAN           ATMOSPHERE**

- 30

**model**
**weighting**

##                                           2-tiered

**  ****TROP. PACIFIC:     ****THREE scenarios:**
**1) Average of predictions of CFS (NCEP), **
**       ****LDEO5 (Lamont) and Constructed**
**       ****Analogue (statistical; NCEP/CPC)**
**2)  same as 1), plus an uncertainty pattern**
**3)  same as 1), minus an uncertainty pattern**
**    ****TROP. ATL, and INDIAN oceans**
**Same as Pacific, without LDEO5 and with**
**     ****statistical (CCA) forecast for Indian Oc.**
**    ****EXTRATROPICAL:**** ****damped persistence **

**MULTIPLE**
**GLOBAL **
**ATMOSPHERIC**
**MODELS**
**   ****ECPC(Scripps)**
**   ****ECHAM4.5(MPI)**
**      ****CCM3.6(NCAR)**
**        ****NCEP(MRF9)**
**        ****NSIPP(NASA)**
**               ****COLA2**
**GFDL AM2p12b**

_**IRI DYNAMICAL CLIMATE FORECAST SYSTEM**_

**PERSISTED **
**GLOBAL**
**SST**
**ANOMALY**

-                                           2-tiered
**              ****OCEAN                                      ATMOSPHERE**

##                          25N                                                           

-                          25N
-  **CFS                           CFS                         CFS                            CFS**
**LDEO                            CA                         CCA                       LDEO**
**  ****CA                                                                 CA                           CA**
-                          25S

- Contributors to SST Predictions for Forcing IRI AGCMs
-                                 damped
-                           persistence
-                                       damped persistence

## History of AGCMs used in IRI Seasonal Climate Forecasts

|          YEAR
AGCM | 98 | 99 | 00 | 01 | 02 | 03 | 04 | 05 | 06 | 07 | 08 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ECHAM | 3.6 | 3.6 | 3.6 | 4.5      | 4.5  | 4.5  | 4.5  | 4.5  | 4.5  | 4.5  | 4.5  |
| CCM3 |  |  |  |  |  |  |  |  |  |  |  |
| NCEP
MRF-9 |  |  |  |  |  |  |  |  |  |  |  |
| NASA |  |  |  |  |  |  |  |  |  |  |  |
| COLA |  |  |  |  |  |  |  |  |  |  |  |
| Scripps |  |  |  |  |  |  |  |  |  |  |  |
| GFDL |  |  |  |  |  |  |  |  |  |  |  |

## Merging of Forecasts of 7 AGCMs

- The merging of 7 predictions into a single one uses two multi-model ensemble systems: Bayesian and canonical variate. These give somewhat differing solutions, and are presently given equal weight.

## IRI’s monthly issued probability forecasts of 

-   15                _Jan-Feb-Mar_

-  _Feb-Mar-Apr_

- _Mar-Apr-May_

- IRI’s monthly issued probability forecasts of
- seasonal global precipitation and temperature

- Forecasts are issued at four lead times. For example:

- Forecast models are run 7 months into future. Observed data
- are available through the end of the previous month (end of
- October in example above). Probabilities are given for the
- three tercile-based categories of the climatological distribution.

## Slide 9

## Ranked Probability Skill Score (RPSS)

- “probability” for the obs is 1 for the observed  category, 0 for the others
- RPSS =  1 - (RPSfcst / RPSclim)
- RPSclim is RPS of forecasts of climatology (33%,33%,33%)

## Slide 11

## Slide 12

## Slide 13

## Slide 14

## Slide 15

## Slide 16

## Slide 17

## Slide 18

## Slide 19

## Slide 20

## Earlier

- Real-time
- Forecast
- Skill
- from Goddard
- et al. 2003,
- BAMS, p1773

## Some other verification measures:

- Likelihood score
- ROC area generalized to all categories
- Temporal correlation (standard)
- Temporal correlation (sample mean not removed)

##                The likelihood score

- The likelihood score is the nth root of the product of the
- probabilities given for the event that was later observed.
- For example, using terciles, suppose 5 forecasts were
- given as follows, and the category in red was observed:
- 45 35 20                        *The likelihood score*
- 33 33 33                        *disregards what prob-*
- 40 33 27                        *abilities were forecast*
- 15 30 55                        *for categories that did*
- 20 40 40                        *not occur.*
- The likelihood score for this example would then be

**= 0.40**

**= **

- This score could then be scaled such that 0.333 would be 0%, and 1 would be
- 100%. A score of 0.40 would translate linearly to (0.40 - 0.333) / (1.00 - 0.333)
- = **10.0%.** A nonlinear translation between 0.333 and 1 might also be selected.

## GROC =        --------------------------------------------------

- GROC =        **--------------------------------------------------**

-                                                                            0   if Plj(k) < Pki(k)

- Where                                                                                                 0.5 if Plj(k) = Pki(k)  (tie)
-                                                                       1   if Plj(k) > Pki(k)

- Generalized ROC area (integrated over categories):

- Each pair of forecasts having differing observations is
- examined to see if the forecast for the higher obs result
- was higher than that for the lower observational result.

-   Number of
- unique forecast
-  pairs having
- differing obs
-      results

## Comparison of verification measures

## Comparison of verification measures

## Comparison of verification measures

## Comparison of verification measures

## Comparison of verification measures

## Slide 30

## Slide 31

## Slide 32

## Slide 33

## Slide 34

## Slide 35

## Slide 36

## Slide 37

## Slide 38

## Slide 39

## Earlier

- Real-time
- Forecast
- Skill
- from Goddard
- et al. 2003,
- BAMS,  p1765

## Comparison of verification measures

## Comparison of verification measures

## Comparison of verification measures

## Comparison of verification measures

## Comparison of verification measures

##            1998-08   2002-08

- |           |           |           |           |           |           |           |           |           |           |
- | 1998  | 1999  | 2000  | 2001  | 2002  | 2003  | 2004  | 2005  | 2006  | 2007  | 2008

**OBS**
- Lead 1
- Lead 4

- | 1998  | 1999  | 2000  | 2001  | 2002  | 2003  | 2004  | 2005  | 2006  | 2007  | 2008
- |           |           |           |           |           |           |           |           |           |           |

-            1998-08   2002-08
- Lead 1  .92            .91
- Lead **4**  **.80            .78**

- Correlations   

- How did we do in predicting ENSO fluctuations?

## Slide 47

## Slide 48

## Slide 49

## Slide 50

## Slide 51

## Slide 52

##       |                |

- absolute value

- r =.56

- r =.25

- r=.35

- r=.43

-       |                |

-                       El Nino
- neutral         or LaNina

## 3 months earlier

- 3-month lead

- r =.74

- r=.66

- r =.63

- r=.65

**La Nina****                                             ****El Nino**

##                    Reliability:

- Degree and nature of relationship
- between forecast probabilities and
- subsequent relative frequency of
- occurrence of corresponding obser-
- vations. Large sample of cases for
- any probability level is needed.
-     20 cases of
- 60% for “above”    what % of time
-                                  did “above” occur?

## OND

- Frequency
- of issuance

- 1997
-   to
- MJJ
- 2008:
- first
- lead
- time

## JFM

- Frequency
- of issuance

- 1998
-   to
- MJJ
- 2008:
- first
- lead
- time

## Conclusions

- Real-time predictive skill of 11 years of IRI seasonal climate forecasts
- ranges from **nonexistent to moderate**, with “modest” being the most
- common skill level. Predictions at such levels are worthwhile and
- economically beneficial over time.
- We do not know what proportion of potential predictability are realized
- in IRI’s skills.
- Precipitation is predicted with generally lower skill than temperature,
- due to its smaller scale, and skill is more seasonally/regionally specific.
**ENSO is by far the largest source of seasonal climate prediction skill**
- for precipitation in most regions.
- For temperature, **climate change generally equals, or overshadows,**
**ENSO **as a cause of anomalies.  But this would change if we defined
- anomalies with respect to a more current (shorter) recent period, or
- or even conservatively incorporated forecast data.