# Presentation: 821306

## Lorenz 1963 Multi-model Ensemble Test-bed

- Christine Johnson
- NAEFS Teleconference
-  September 2006

## Research Aim

- 	The Met Office is currently developing a real-time multi-model ensemble using the TIGGE data.
- Which methods should be used to calibrate and combine the multi-model ensemble?
- Evaluate the benefits of the multi-model ensemble.
- 	Investigate within an idealized framework using the Lorenz 1963 model.

## Lorenz 1963 multi-model ensemble testbed

- Model 1

- Model 2

- Near =3/2

- Near =/2

- The 1963 Lorenz equations, with a  dimensional scaling so that the fast-timescale is similar to the synoptic timescale of the atmosphere.
- Add ‘seasonally’ dependent biases that are sinusoidal and dependent on , which varies from 0 to 2 in one ‘model-year’ (300 days).
- Generate 8-member, 15-day ensembles every 12 hours.

- Truth

## Ensemble members

- Truth

- Control

- Member

-  The models give different forecasts.
- The true state tends to lie within the ensemble.
- The ensemble spread is representative of the forecast error.

## RMS Errors

- Control

- Mean

- Spread

- Climate

- The deterministic (**control**) forecast error is the same as that for climate, by 15 days.
- The ensemble **mean **is more accurate by the end of the forecast.
- The ensemble **spread** is too small at the end of the forecast.

## Calibration and Combination Data

- Use previous forecasts (in the blue box ) together with observations, to calibrate and combine the new forecast (red).
- Use a 50 day moving calibration window.

## Calibration Method

- Choose the values of 0 and 1 to minimize the mean square error (MSE) over the calibration window (linear regression).
- Choose the value of  so that the ensemble variance is equal to the MSE of the bias-corrected ensemble mean.

## Rank Histograms

- No bias correction or variance inflation

- With bias correction

- With bias correction and variance inflation

- Note: Rank i means that i ensemble members are larger than the verification

- Most ensemble members are smaller than the verification: negative bias.

- Ensemble is under-dispersive, so either all the members are larger or all the members are smaller than the verification.

- There is no bias, and the verification lies within the ensemble.

## Impact of the position of the window

- What is the impact of the position of the calibration window, relative to the forecast?

- The best result is with a CENTRED window.
- It is difficult to correct the errors at LONG LEAD TIMES.

## Bias coefficients

- Bias coefficients show the general seasonal variation.
- Bias coefficients have more variability at longer lead times and therefore are more sensitive to the calibration sample.

## Use of reforecast data

- Following Hamill et al 2004, use 20 years of forecast data with a 50 day window centred on the forecast to be verified.

- Raw data

- With bias correction

- Bias correction gives a reduction in the errors at long lead times, but not as large as for the centred moving window.

## Summary: Calibration

- Bias coefficients exhibit large fluctuations at long lead times due to flow-dependent errors.
- We can only expect realistic moving-window bias correction to correct the errors at short lead times.
- Reforecast data can be used to correct the seasonally varying component at long lead times. A centred moving-window is needed to correct the flow dependent component.

## Combination Method

- Assume that the multi-model pdf is formed by the sum of the pdfs of each single-model (Raftery et al 2005).
- Estimate the model-dependent weights, wk.

## Use of the model-dependent weights wk

- Multi-model ensemble mean:
- Multi-model ensemble probabilities (Stefanova and Krishnamurti 2002):
- Where       and        are the single-model means and probabilities respectively.

## Combination methods summary

- Note: For a fair comparison, the multi-model ensembles has the same number of members as the single model ensembles (8 – member).

## Bayesian model averaging

- In a similar way to a mixture-density problem,  estimate the weights and variances using the expectation-maximization algorithm
- The BMA estimated weights are sensitive to the sample due to ill-conditioning. Hence it is important to add prior information to the log-likelihood function, in the form of a beta-distribution on w (Fraley and Raftery, 2005):

- No prior

- With prior

## The model-dependent weights

- RMS-based

- Multiple-regression

- BMA

- All three methods give **similar estimates** for the model dependent weights.
- The weights show the **‘seasonal’ variation,** despite having been bias-corrected.

## Combination methods results

- Resolution

- Reliability

- The simple combination gives an improvement in RESOLUTION.
- The model-dependent weights gives an improvement in RELIABILITY.

- Single-model with bias correction & variance inflation

- It is difficult to determine the best combination method.

## Impact of the position of the window

- RMS error of Mean

- Reliability

- Realistic window

- Centred window

- The model-dependent weights have more impact in improving the RMS errors and reliability when using a centred window.
- The multiple-regression method gives slightly better results using the ideal centred window.

## Summary: combination

- Model combination gives improvements in resolution
- Model-dependent weights give improvements in reliability.
- All 3 methods give similar estimates for the weights.

## Conclusions and future plans

- 	The results show that model combination gives the most improvements to the ensemble resolution. Calibration and model-dependent weights give improvements in the reliability.
- In the Met Office multi-model ensemble:
- Apply Kalman-filter type bias correction.
- Use the most simple method (skill-based) for estimating model-dependent weights.
- Consider spatial correlations.

## References

- Hamill, T. M., J. S. Whitaker and X. Wei, 2004: Ensemble reforecasting: Improving medium-range forecast skill using retrospective forecasts. *Mon. Weather Rev*., 1**32**, 1434-1447.
- Fraley C. and A. Raftery, 2005: Bayesian regularization for normal mixture estimation and model-based clustering. Technical report 486, Dept. Statistics. *Univ. Washington.*
- Raftery A., T. Gneiting, F. Balabdaoui and M. Polakowski , 2005: Using Bayesian model averaging to calibrate forecast ensembles. *Mon. Weather Rev.,* **133**, 1155-1174.
- Stefanova L. and T. N. Krishnamurti, 2002: Interpretation of seasonal climate forecast using Brier skill score, the Florida state university superensembles and the AMIP I dataset. *J. Climate*, **15**, 537-544.