# Presentation: 324675

## A Review of Bureau of Meteorology Outlooks for Australia

**Andrew Watkins**
- National Climate Centre
- Bureau of Meteorology, Melbourne
**31****st**** Climate Diagnostics and Prediction Workshop, **
**Boulder, Colorado, October 23-27 2006**

**Notes:**
results of a verification system for the Bureau of Meteorology’s seasonal forecasting efforts. 

## Introduction

- Australian Outlook Models (Rain/Temp/Nino)
- Validation (1950-99)
- Verification (2000-2006)
- 2005/06 verification
- Summary

**Notes:**
Intro...

## Terminology – this talk

**Validation**
- assessment of skill by scoring cross-validated hindcasts
- essential for assessing new models and expected future performance of current models
**Verification**
- assessment of skill by scoring independent real-time forecasts
- used for assessing how a model has performed
- undertaken for accountability purposes
- limited use for assessing expected future performance due to small sample size

**Notes:**
Script:

Because the terms “validation” and “verification” are used somewhat interchangeably in the meteorological literature, for the purposes of clarity in this talk I am going to make the following distinctions.

By “validation”, I mean assessment of forecast model skill obtained by assessing independent hindcasts, usually under some sort of cross-validation arrangement. The National Climate Centre typically uses the “leave-one-out” cross-validated hindcast technique because auto-correlation within relevant time series is typically small. Model validation is essential for assessing the forecast skill of new models and describing the expected future performance of current models.

By “verification”, I mean assessment of skill obtained by assessing independent forecasts, typically in real-time. This is useful for assessing how well a forecast model has performed in the recent past. It is typically undertaken for accountability purposes, for example, to check that the model is actually performing in a desirable manner, and also to be able to report meaningfully on recent performance. It is tends to be of limited use in seasonal climate forecasting in assessing expected future performance because the sample sizes available for verification are typically small.

## Australian Models – Seasonal Outlook

**Australian Models – Seasonal Outlook**

**Notes:**
Basic set up of our climate forecasting empirical scheme
Still the backbone of forecasting in Australia

Issued monthly.

## Seasonal Outlook Model

**Validation 1950-1999**

- Rainfall

- Tmax

- Tmin

- Percent consistent

- http://www.bom.gov.au/silo/products/verif/

**Notes:**
Script:
So how well has the seasonal outlook  model performed – long term

Cross validated hindcasts 1950-99
Using the relatively simple “percent consistent” here, as displayed on our website to give users a generalised idea of the accuracvy of the model. 
Of course this does not take into account  the fact that we expect out model to be “wrong” a certain percentage of the time when we forecast in probabilities!

Stippled area is the region of statistical significance, but at the 80% conficence level, howevr as we can see by the amount of “red” on the plot, the spatial significance is above 99%.
Overall, rainfall is generally useful, with greatest accuracy in the north east

Tmax has skill over much of the continent, while Tmin likewise.

So in general, the hindcasts suggest useful overall for the model

## Observed Deciles

**Observations 2000-2006**

- Rainfall

- Tmax

- Tmin

**Notes:**
Looking quickly at what has occurred over the post hindcast period – the verification period of you like – Australia has generally been very dry inn the east (see the areas of decile 1 rainfall, for instance: years back to 1900) and on the far western coast.

For temperature, data only goes back to 1950, but clearly a huge area of warmest such 7 year period on record, less so for tmin.

So in general, we’ve had a warm/hot and dry eastern Australia, wet and average to cool north west.
Surprisingly like the Global warming suggestions for Aus....

## Seasonal Outlook Model

**Verification 2000-2006**

- Rainfall

- Tmax

- Tmin

- Percent consistent

## Seasonal Outlook Model

**Verification 2000-2006**

- Rainfall

- Tmax

- Tmin

- Reliability

**Notes:**
Rainfall: model apears relaible 
Combining all Australian grid points, 687 of them, the figure shows the reliability data. For each integer percentage forecast probability for above median rainfall, the observed rate of this outcome is plotted. The desired outcome for this plot is a collection of plotted points close to the diagonal line. Unfortunately no histogram on this plot, but rainfall forecasts are concentrated between about 0.4 and 0.65. Results here are fairly pleasing, but the noisiness is indicative of the small sample size.

Tmax: Model appears underconfident
Tmin: Model overconfident

Reliability is calibration and not skill.
It is an indicator of bias rather than skill – an unreliable model cant have skill, but an unskillfull model can be eliable (ie random forecast can be reliable)
Tells us about the bias in the model.

Reliability data (all Aust. grid points)
77 verified forecasts
1 percent probability bins
noisiness suggests “small” sample size
Model is largely reliable
Evidence that wet conditions are “easier” to forecast

## Seasonal Outlook Model

**2005-06 verification**

- Rainfall

- All Australia, all Outlooks

- LEPS score

**Notes:**
LEPS compares the cumulative distribution function for forecasts with that for observations – rewards more emphatic forecasts, of which our rainfall model is anything but!

Looking at just the past year – on average weak to moderate skill for rainfall and Tmax over Australia. However the rainfall outlooks have been good for southern Australia over recent months. In some contrast, Tmax has displayed relatively low skill this year until recent months. 

The low skill in early 2006 appears to be the result of persistently warm conditions in the south Pacific, which, despite a cool equatorial Pacific, tended to dominate the EOF weights, resulting in a positive SST1 value suggesting tom the model that the Aussie land temps should be warm. In fact they were cool.

{The EOF score coefficients – ie the magnitude of the EOF1 pattern – is multiplied by the observed (normalised) SSTAnom pattern (ie SSTA/standard deviation) to give the EOF weights. The sum of these weights gives the amplitude of the EOF}

Better skill for rainfall over the southern states

Rainfall - Largely indicative of no firm forcing from the oceans during this period, as can be seen by best skill being achieved during mid 2006 and in 2004, when weak drivers were present.
The graph shows Australia-wide averaged scores for each forecast, using the LEPS scoring method. Vertical lines indicate the start of a new year. There has been considerable temporal evolution in the success of the forecasts for all three variables, with the skill being better in the second half of the sequence. Predictability comes from ENSO events!

LEPS Measures the error in probability space as opposed to measurement space, where CDFo() is the cumulative probability density function of the observations, determined from an appropriate climatology. 

positive averages through most of 2002/03 El Niño event
Periods of low skill generally correspond to periods of low forecast signal

## Seasonal Outlook Model

**2005-06 verification**

**Notes:**

Being a little more subjective....
Basic comparisons for “our” autumn and winter seasons for rainfall.
some area of good comparisons (western Australia, southern Australia – particularly in winter,)
Warnings for winter were useful for farming overall, but still did not prevent widescale hardship with little rain over the main grop and pasture growing season (April-October).

## Australian Models - POAMA

- Global coupled model seasonal forecasting system
- Components AGCM (T47L17) + OGCM (0.5x2.0xL25) + OASIS Coupler. OGCM is Australian Community Ocean Model
- Operational since Oct 2002. Daily 9-months forecasts produced
- Operational products issued by the BoM National Climate Centre
- _[www.bom.gov.au/bmrc/ocean/JAFOOS/POAMA](http://www.bom.gov.au/bmrc/ocean/JAFOOS/POAMA)_
_**www.bom.gov.au/climate/coupled_model/poama.shtml**_

**Notes:**
Brief description of the POAMA model
Global coupled model seasonal forecasting system
Components AGCM (T47L17) + OGCM (0.5x2.0xL25) + OASIS Coupler. OGCM is Australian Community Ocean Model
Operational since Oct 2002. Daily 9-months forecasts produced
Operational products issued by the BoM National Climate Centre 

## POAMA (dynamical) Model

**POAMA (dynamical) Model**

**2005-06 verification**

**Notes:**
A cool bias in the early parts for parts of the year due to unforeseen consequences of a change in soil moisture scheme in the model which was used to provide the initial conditions for the POAMA ensemble members.
Basically... 
1) Full EC land surface scheme replaced a simple bucket scheme in the GASP model.
2) GASP model output was used as init conditions
3) the land surface change led to wetter soil ove Sth america,
4) this led to a cooler land
5)this perturbed the Walker circulation
6)increased vigor of trades and hence
7) cool SST
Also... 15m temps in POAMA, skin in Reynolds
also different resultion (hiher res in POAMA)
also different climatologies (POAMA 1987-2001)

However in general the POAMA model did pick up on the warming trend through the year, even it does still analyses a little cool.

The cool bias problem has now been fixed (by fixing the land surface wetness back to climatology in the initial conditions)

Pleasingly the model has continued to pick up MJO events

## Summary

- Validation of Empirical model shows skill above climatology (an unskilled model)
- Verification shows skilful Temp forecasts, but less skilful Rainfall forecasts
- Bias tends to be fairly conservative
- Past year has shown skill in the southern states, and in the east in recent months
- POAMA Dynamical model suffered a cool bias earlier in the year – problem fixed; did predict a warming Pacific

**Notes:**
Summary
Current operational models (rainfall, Tmax) show skill above climatology (an unskilled model)
Models are moderately reliable
to the extent that they are biased, it is “conservative”
sample size is too small to allow regional break-down of estimates of reliability, but Australia-wide results reveal forecasts to be fairly reliable
In regions of lower predictability, the model tends to generate near-climatological probabilities.

## The End

*“**Whatever may be the progress of sciences, never will observers who are trustworthy, and careful of their reputation, venture to foretell the state of the weather.”*
*Fran**ç**ois Arago* in the Annual Report of the Paris (astronomy) observatory, 1846
- (as quoted in *Storm Watchers*, by John D Cox)

## Fawcett R J B, Jones D A and Beard G S. 2005. A verification of publicly issued seasonal forecasts issued by the Australian Bureau of Meteorology: 1998-2003. Australian Meteorological Magazine, 54, 1-13. 

Drosdowsky and Chambers – Journal of Climate 14, 1677-1687 (2001)
validation results for operational SST model (all 4 SST predictors) – 44 year climatology (1950-1993)
verification results for 5 years of independent hindcasts (1994-1998)

Jones – BMRC Research Report No. 70 (1998)
validation results for operational SST model (all 4 SST predictors) – 45 year climatology (1950-1994)
verification results for 3.5 years of independent hindcasts (1995-1998)
http://www.bom.gov.au/climate/ahead/rr70/ 

Validation results for current operational models (rain & temps) on web: http://www.bom.gov.au/silo/products/verif/

- Drosdowsky and Chambers – Journal of Climate **14**, 1677-1687 (2001)
- validation results for operational SST model (all 4 SST predictors) – 44 year climatology (1950-1993)
- verification results for 5 years of independent hindcasts (1994-1998)
- Jones – BMRC Research Report No. 70 (1998)
- validation results for operational SST model (all 4 SST predictors) – 45 year climatology (1950-1994)
- verification results for 3.5 years of independent hindcasts (1995-1998)
- _http://www.bom.gov.au/climate/ahead/rr70/__ _
- Validation results for current operational models (rain & temps) on web: _[http://www.bom.gov.au/silo/products/verif/](http://www.bom.gov.au/silo/products/verif/)_

**References **

**Notes:**
Script:

The basis for this talk is a paper by the authors recently accepted for publication in the Australian Meteorological Magazine. The results presented in this talk though have been updated to the FMA 2004 season. The operational forecast models used by the National Climate Centre are based on work by Drosdowsky and Chambers (2001) and Jones (1998). Both of these are available on the Bureau’s website. These reports present long-period independent hindcast validation results and short-period independent forecast verification results. The validation results season by season and all seasons together for the current operational models are also available on the Bureau’s website.

## Seasonal Outlook Model

**Verification 2000-2006**

- Rainfall

- Tmax

- Tmin

- Brier Score