# Presentation: 447826

## A comparison of calibrated T2m probabilistic forecasts from GFS and ECMWF reforecasts

- Tom Hamill and Jeff Whitaker
*NOAA / ESRL / PSD, Boulder, CO*
- Renate Hagedorn
*ECMWF, Reading, England*

- NOAA Earth System
- Research Laboratory

## Bottom-line messages

- Calibrated GFS based on 1998 ensemble more skillful
- than probabilities from raw ECMWF ensemble.
- (2) Substantial improvement of ECMWF ensemble based
- on reforecasts; smaller amount than GFS, but still large.

## Bottom-line messages

- (3) 30-day bias corrections do a good job of correcting
- short-term forecasts.  Somewhat less useful in medium
- range.

## ECMWF’s reforecast data set

**Model**: 2005 version of ECMWF model; T255 resolution.
**Initial Conditions**: 15 members, ERA-40 analysis + singular vectors
**Dates of reforecasts**: 1982-2001, Once-weekly reforecasts from 01 Sep - 01 Dec, 14 total.  So, 20*14 ensemble reforecasts = 280 samples.
**Data** sent to NOAA / ESRL : T2M ensemble over most of North America, excluding Alaska.  Saved on 1-degree lat / lon grid.  Forecasts to 10 days lead.

## ECMWF domain sent to us for reforecast tests

## NOAA’s reforecast data set

**Model**:  T62L28 NCEP GFS, circa 1998
**Initial States**: NCEP-NCAR Reanalysis II plus 7 +/- bred modes.
**Duration**: 15 days runs every day at 00Z from 19781101 to now. (_[http://www.cdc.noaa.gov/people/jeffrey.s.whitaker/refcst/week2](http://www.cdc.noaa.gov/people/jeffrey.s.whitaker/Manuscripts/multimodel.pdf)_*).*
**Data**:  Selected fields (winds, hgt, temp on 5 press levels, precip, t2m, u10m, v10m, pwat, prmsl, rh700, heating).  NCEP/NCAR reanalysis verifying fields included (Web form to download at _[http://www.cdc.noaa.gov/reforecast](http://www.cdc.noaa.gov/people/tom.hamill/skill_overforecast_QJ_v2.pdf)_).  Data saved on 2.5-degree grid.
- Here, use only the subset of data overlapping with ECMWF reforecast data set.

**Notes:**
As promised - details of dataset.

11 months done - dataset will be complete in about a month.

## Observation Locations

- Uses stations from
- NCAR’s DS472.0
- database that have
- more than 96%
- of the yearly records
- available, and overlap
- with the domain that
- ECMWF sent us.

## Calibration Procedure: “NGR”“Non-homogeneous Gaussian Regression”

**Reference**: Gneiting et al., *MWR*, **133**, p. 1098
**Predictors**: ensemble mean and ensemble spread
**Output**: mean, spread of calibrated normal distribution
**Advantage**: leverages possible spread/skill relationship appropriately. Large spread/skill relationship, c ≈ 0.0, *d *≈1.0.  Small, *d *≈ 0.0
**Disadvantage**: iterative method, slow...no reason to bother (relative to using simple linear regression) if there’s little or no spread/skill relationship.

## Training Data forNon-homogeneous Gaussian Regression(all cross validated)

**01 Sep**: *01 Sep*, 08 Sep, 15 Sep
**08 Sep**: 01 Sep, *08 Sep*, 15 Sep, 22 Sep
**15 Sep**: 01 Sep, 08 Sep, *15 Sep*, 22 Sep, 29 Sep
**17 Nov**: 03 Nov, 10 Nov, *17 Nov*, 24 Nov, 01 Dec
**24 Nov**: 10 Nov, 17 Nov, *24 Nov*, 01 Dec
**01 Dec**: 17 Nov, 24 Nov, *01 Dec*
- Use a centered training data set for weeks 3 - 12, uncentered for weeks 1, 2, 13, and 14

## ... but first, rank histograms

- Members randomly perturbed by 1.0K to account for observation error; probably a bit small for GFS on its coarser 2.5o grid, which would make their histograms slightly more uniform. Ref: Hamill, *MWR*, **129**, p. 556.

- GFS

- ECMWF

## Continuous Ranked Probability Score (CRPS) and Skill Score (CRPSS)

- (This conventional way of calculating CRPSS exaggerates skill if some samples have more climatological
- spread than others. Will use a modified version where we
- calculate CRPSS separately for 8
- different categories of climatological spread and then average them.
- See Hamill and Juras, January 2007, *QJRMS, *and Hamill and Whitaker* *(2007)* MWR, *to appear, tinyurl.com/29oy8s )

## ECMWF, raw and post-processed

- 95% confidence interval using paired block bootstrap described in Hamill, *WAF*, 1999, **14**, p. 155

- Small confidence intervals imply **significant improvement at all leads**

## ECMWF, raw and post-processed

- 95% confidence interval using paired block bootstrap described in Hamill, *WAF*, 1999, **14**, p. 155

- Small confidence intervals imply **significant improvement at all leads**

- in this metric, calibrated 4-5 day
- forecasts now as skillful as
- uncalibrated 1-day forecast.

## ECMWF and GFS 

- GFS reforecasts sub-sampled only to the dates of ECMWF reforecasts

## How much from simple bias correction?

- ~ 60 percent of total improvement at short leads, 70 percent at longer leads.

## How much from short ECMWF training data sets?

## How much from short GFS training data sets?

## How much from long GFStraining data set?

- Here GFS reforecasts
- sampled once per
- week are compared
- to those sampled
- once per day (“full”).

## Multi-model reforecasts

- Slight benefit from incorporating information from both
- forecast models, even though GFS much poorer than ECMWF

## Results from GFS, Tmin

- from Wilks and Hamill, 2006,
-  Met. Apps., 13, 243-256.

- Somewhat contradictory results
- for short training data set:
-  bug?
-  difference between Tmin and
-    12Z ?

- largest improvement
- at the stations with
- the highest original
- CRPS.

## CRPSS, Day 1

## CRPSS, Day 4

## CRPSS, Day 7

## Notes

- Same benefit to precipitation calibration, winds, other variables?  Perhaps not, w/o more full reforecast data set.

- (results from full GFS reforecast data set.  See Hamill et al., *BAMS*, January 2007)

- more rare events, like heavy precipitation forecasting,
- tend to benefit more from long reforecast data sets.

## Preliminary Conclusions

- Still substantial benefit to calibrating forecasts, even with a much better model than used in 1st-generation GFS reforecast.
- Old GFS + reforecast calibration » more skill than ECMWF uncalibrated.
- 30-day training does good job of calibration for short-term forecasts (consistent with previous NCEP results).
- For temperature calibration, weekly reforecasts samples ~ as good as daily.
- Still need to test calibration of other variables (precipitation, wind speed, etc..)