# Presentation: 379439

## Issues in ensemble weather prediction

- Tom Hamill
- NOAA Earth System Research Lab, Boulder, Colorado, USA
- _[tom.hamill@noaa.gov](mailto:tom.hamill@noaa.gov)_

- NOAA Earth System
- Research Laboratory

## Success story

- ECMWF’s “Lothar” forecast, c/o Tim Palmer.  Many members forecast storm several days hence; deterministic did not.

## Success: limited-area ensemble forecast provides evidence of signal of severe weather two days prior

_***48 hr***_* SREF Forecast Valid 21 UTC 7 April 2006*

- Prob (MLCAPE _>_ 1000 Jkg-1)
- X
- Prob (6 km Shear _>_ 40 kt)
- X
- Prob (0-1 km SRH _>_ 100 m2s-2)
- X
- Prob (MLLCL _<_ 1000 m)
- X
- Prob (3h conv. Pcpn _>_ 0.01 in)
*Shaded Area Prob *_*>*_* 5%*

- Tornadoes related to large-scale patterns of instability and shear, often predictable several days hence.

- c/o David Bright, NOAA SPC, and Jun Du, NCEP/EMC

- Max 40%

## Severe event of April 7, 2006

**First ever** day-2 outlook “high risk” of severe weather issued by NOAA Storm Prediction Center; in past have been cautious
- > 800 total severe reports, 3 killer tornadoes,10 deaths
- Diagnostics from SREF and good past SREF performance aided forecaster confidence

## Still, unresolved problems with ensemble weather predictions

- Verification of ECMWF precipitation reforecasts (Fall, 1981-2000) using 2005 version of ECMWF forecast model, with 15 members.
- Substantial lack of calibration of precipitation forecasts.
- Similar story to be told for surface temperature forecasts; raw ensemble still subject to significant systematic errors.
- The variables we don’t really care much about (Z500) we do pretty well at; the variables we do care about (surface weather) are tougher to fix.

- horizontal
- lines indicate
- distribution
- of climatology

- error bars
- from block
- bootstrap

## Topics

- Limitations of ensemble forecasts & summary of ongoing work
- Sub-optimal sets of initial conditions
- Model errors
- Linkages & where WGNE can help

## Initial condition issues

- Ensemble of initial conditions:
- Should sample distribution of plausible analyses (and subsequent forecasts should provide samples of forecast uncertainty).
- Given limited size ensemble, maximal forecast error variance if initial conditions should project onto analysis-error covariance singular vectors (Ehrendorfer&Tribbia, JAS, 1997)
- Are our methods providing suitable approximations to this?
- Are we perturbing all aspects of the initial condition to which there is initial condition sensitivity?

## Structure of analysis errors

- Estimated from differences between operational analyses and forecasts from NCEP, US Navy, and Canada.  Here, data for 45o N
- Note small analysis errors in mid-troposphere, larger errors near surface and tropopause.

- 0 h

- 12 h

- 24

- 36

- 48

- Ref: Hakim 2005, *MWR*, **133**, 567-578.

- analysis
- spectrum

## Analysis error spectrum

- Analysis errors are
- a larger fraction
- of the climatological
- variance at small scales
- than at large scales.
- Still, there is more total
- error in the large scales
- than in the small scales.

- analysis
- spectrum

- analysis
*error*
- spectrum

- 12h

- 72h

- Ref: Hakim, *MWR*, March 2005

## Singular vectors(selecting maximally growing perturbations)

- Singular vectors(selecting maximally growing perturbations)

- Property of singular vectors
- sized initial time (dashed) and 48 h later (solid).
- Main point: analysis errors
- may be structurally
- different than total-energy
- singular vectors. TESVs
- have large amplitude in
- mid-troposphere, much
- more power at small scales
- than analysis errors, this
- suggests.  Grow less rapidly.
- Note 1: ECMWF’s singular
- vectors are combination of
- initial-time and evolved.
- Note 2: TESVs have produced, thus far, generally
- better forecasts than AECSVs

- from Barkmeijer et al., *QJRMS*, 1999

- Total-energy
- singular vectors

- Analysis-error
- covariance
- singular vectors

- *10

- initial*10

- *10

- *10

- 48-h evolved

## The ensemble Kalman filter: a schematic

- (This schematic
- is a bit of an
- inappropriate
- simplification,
- for EnKF uses
- every member
- to estimate
- background-
- error covariances)

## Will ensemble filters improve the initialization of ensemble forecasts?

- Test of ensemble square-root filter (EnKF; Whitaker and Hamill, MWR, 2002); no perturbed observations
- 64-member, T126L64 NCEP GFS, May-Jun 2007
- All ‘conventional’ data, cloud-track winds, COSMIC GPS, + AMSU radiances
- Compare with 20-member ensembles forecasts with from EnKF perturbations, vs. operational NCEP Ensemble Transform (ET) perturbations re-centered on EnKF ensemble mean.
- ET technique is an update of old “breeding technique” that includes some orthogonalization of perturbations.

## Evolution of T850 Spread,EnKF and ET perturbation methods

- EnKF spreads initially more homogeneous.  Tough here
- to visualize how spreads evolve with time, so (next slide)

- Ref: Whitaker&Hamill presentation at AMS Annual, 2008

## Evolution of T850 spread,EnKF and ET perturbation methods

- ET spreads grow quickly in midlatitudes, EnKF does not. Hints at
- possible dynamical imbalances in EnKF IC’s. ET spread growth from
- leakage of excess in polar regions?

## Perturb the land surface?

- The land state can be thought of as part of the initial condition.
- Why not perturb it?
- Perturbing the soil moisture (here, WRF initialized with 2 different soil moisture
- analyses) increased
- warm-season precipitation forecast spread, modulated the
- details of thunderstorm
- activity.
- Likely to have biggest
- impact in warm season,
- when insolation is large.   Though In winter, perturb snow cover/depth?

- Ref: Sutton et al., MWR, Nov 2006

## Conclusions (initial conditions)

- Each of existing operational methods fall short of optimal in some important ways
- Singular vectors: don’t resemble analysis errors, detrimental especially to short-range forecasts (but they do grow quickly!)
- Ensemble filters: apparently slow growth of spread in ensembles, perhaps due to factors such as the introduction of imbalances through use of “covariance localization.”
- Comparisons of methods between centers un-revealing, as different centers use different models.
- Clean comparisons between methods is something the WMO might facilitate.

## Ensemble model error

- Deal with it directly
- Ameliorate model bias (increase resolution, improve parameterizations, etc.). Won’t discuss.
- Increase spread among ensemble members (stochastic backscatter, multi-model, multi-parameterization, better treatment of lateral boundary conditions).
- Deal with it indirectly (statistical post-processing).

## Spectral Stochastic Backscatter Scheme (SSBS) 

- Spectral Stochastic Backscatter Scheme (SSBS)

- Rationale: A fraction of the dissipated energy is scattered upscale and acts as streamfunction forcing for the resolved-scale flow (originally developed in the context of LES now adapted to NWP) Shutts, 2005; Berner et al., 2008a, Berner et al., 2008b

- Total dissipation rate from numerical dissipation, convection, gravity / mountain wave drag.

- Spectral Markov chain: temporal and spatial correlations prescribed

## SBSS resultstropical U850, T255L40 model

- BSS, Climo + 1.5

- RPSS

- Rank Histogram

- % outliers

- Solid: operational ensemble (OPER);
- Dashed: ensemble with stochastic backscatter (SSBS);
- Dotted: operational configuration with reduced initial perturbations 	(OPER-IPRED).

- Berner et al. 2008, JAS, in press, DOI: 10.1175/2008JAS2677.1

## Combination:multiple models and/or multiple parameterizations

- Combination:multiple models and/or multiple parameterizations

- “One possible approach to operational mesoscale guidance is to produce an ensemble forecast using a combination of different initial conditions* ****and different trigger functions***.”
- Dave Stensrud, 1994, from Stensrud and Fritsch, *MWR*, Sep. 1994 (part III).

- Photo credit: Monty Python and the Holy Grail

## Multi-(whatever) ensembles

- Potential plusses:
- Provide different but still plausible predictions.
- Models may have particular strong/weak aspects. Leverage the strong, discount the weak.
- Implicitly samples analysis uncertainty through assimilation of somewhat different sets of observations, use of different data assimilation techniques.
- Leverage each other’s hard work and CPU cycles.
- Potential minuses
- Models may all be developed under similar set of assumptions (e.g., which terms to neglect in equations of motion).  What if some of these are consistently wrong and forecasts have similar biases?
- Complex task to share data internationally in real time.
- Must be flexible to use whatever is available, given outages / production delays.

## THORPEX Interactive Grand Global Ensemble (TIGGE)

- archive centers and data providers

**Archive Centre**

**Current Data Provider**

**NCAR**

**NCEP**

**CMC**

**UKMO**

**ECMWF**

**MeteoFrance**

**JMA**

**KMA**

**CMA**

**BoM**

**CPTEC**

**IDD/LDM**

**HTTP**

**FTP**

## TIGGE,RPSSof Z500

- TIGGE,RPSSof Z500

- Skill of forecasts against own analyses for 4 different periods using TIGGE data
- Models obviously of different quality.

- Oct-Nov 2007

- Oct-Nov 2007

- Dec 06-Feb 07

- Apr-May 2007

- Jun-Aug 2007

- Ref: Park et al., 2008, ECMWF TM 548

## TIGGE, Z500, ECMWF & UKMO

- Trained with bias correction using last 30 days of forecasts and analyses.
- ECMWF analysis as reference.
- Conclusions:
- Small benefit from multi-model relative to best model.
- + impact of bias correction at short leads, - at long leads. [Reforecast seminar will discuss why]

- UKMO raw

- ECMWF raw

- multi-model raw

- multi-model
- bias-corrected

- Ref: Park et al., 2008, ECMWF TM 548

## T850, two-model

- Trained with bias correction using last 30 days of forecasts and analyses.
- ECMWF analysis as reference.
- Conclusions:
- UK so contaminated by systematic errors that its raw forecasts add no value.
- Multi-model calibrated uniformly beneficial (presumably because of large, ~consistent biases)

- UK raw

- ECMWF raw

- multi-model raw

- multi-model
- bias-corrected

- Ref: Park et al., 2008, ECMWF TM 548

## Dealing with model error indirectly: problems we’d like to correct through calibration

## bias (drizzle 

- bias (drizzle
- over-forecast)

- ensemble members too
- similar to each other.

- Probabilities too smooth;
*downscaling* needed.

## Calibration: basic concept

- Use time series of past (F,O) to help determine pdf of O | today’s F.
- Major questions:
- What calibration scheme to use?
- How much training data?
- Are gains from calibration worth the drawbacks (extra computational expense)?

## Disadvantages to calibration

- Calibration won’t correct the underlying problem.  Prefer to achieve unbiased, reliable forecasts by doing numerical modeling correctly in the first place.
- No one general approach that works best for all applications.
- Corrections may be model-specific; the calibrations for NCEP v 2.0 may not be useful for ECMWF, or even NCEP v 3.0.
- Could constrain model development.  Calibration ideally based on long database of prior forecasts (reforecasts, or hindcasts) from same model.  Upgrading model good for improving raw forecasts, may be bad for skill of post-processed forecasts.
- Users beware: Several calibration techniques that have been recently proposed are conceptually flawed / only work properly in certain circumstances.

## Analog calibration method using reforecasts

- On the left are old forecasts
- similar to today’s ensemble-
- mean forecast.  For feeding
- ensemble streamflow model,
- form an ensemble from
- the accompanying
- analyzed weather on the
- right-hand side.

- Hamill and Whitaker, Nov. 2006 MWR.

## Analog calibration method using reforecasts

- On the left are old forecasts
- similar to today’s ensemble-
- mean forecast.  For feeding
- ensemble streamflow model,
- form an ensemble from
- the accompanying
- analyzed weather on the
- right-hand side.

## Downscaled analog probability forecasts

## Very substantial

- Verified over 25 years of forecasts;
- skill scores use conventional
- method of calculation which may
- overestimate skill
- (Hamill and Juras 2006).

- impact on
- improving forecast
- skill; removes
- state-dependent
- bias, improves
- reliability, permits
- statistical
- downscaling.

## Linkages &where WGNE can help

## “HEPEX”an international, cooperative project toadvance ensemble hydrologic predictions.Needs close linkage with global ensemble prediction & datasharing efforts.

- “HEPEX”an international, cooperative project toadvance ensemble hydrologic predictions.Needs close linkage with global ensemble prediction & datasharing efforts.

## Example: 1-2 day lead hydrologic forecast for a basin in Northern Italy

- Skill of hydrologic forecast tied to the skill of the precipitation/temperature forecasts.  Here, all forecasts missed timing of rainfall event, so subsequent hydrologic forecasts missed event. Reservoir regulation, hydrologic model may have also had effects.

- Source: A meteo-hydrological prediction system based on a multi-model approach for ensemble precipitation forecasting.  Tomasso Diomede et al, ARPA-SIM, Bologna, Italy.

- Hydrologic model forced with multi-model weather ensemble data.

## HEPEX’senvisioned“Ensemble Hydrological Prediction System”

- HEPEX’senvisioned“Ensemble Hydrological Prediction System”

- WMO’s hydrological program
- not well integrated with
- THORPEX, WWRP, WCRP.
- WGNE recommendation helpful.

## Ensemble forecast verification

- Verification plan drafted under THORPEX/TIGGE, but little follow-through.
- Need:
- Observations shared, readily accessible.
- Sensible weather elements, not just Z500.
- Defined standards on routine set of verification metrics.
- Against own analyses?
- Against observations?
- Dealing with different grid spacings, etc.
- Develop & freely disseminate a toolbox of routines, display packages.
- Either centralized production of verification
- WGNE recommendation to boost prominence of cooperation on ensemble verification helpful.