# Presentation: 468903

## Reforecasting, an integral component of probabilistic forecasting in the NWS

- Tom Hamill
*NOAA Earth System Research Lab*
*Boulder, CO*
- tom.hamill@noaa.gov

- NOAA Earth System
- Research Laboratory

- Presentation to NFUSE Steering Group, 6 June 2007

## Motivation

-    “*Hydrometeorological services in the United States are an Enterprise effort.  Therefore, effective incorporation of uncertainty information will require a **fundamental and coordinated shift by all sectors of the Enterprise**.  Furthermore, it will take time and perseverance to successfully make this shift.  As the Nation’s public weather service, **NWS has the responsibility to take a leading role in the transition** to widespread, effective incorporation of uncertainty information into hydrometeorological prediction*.”
-  From finding 1 of 2006 NRC report “_Completing the Forecast_”

## Problem with current ensemble forecast systems

- Forecasts may be biased and/or deficient in spread; probabilities are mis-estimated.

- Heavy rain in an
- area where none of
- the ensemble members
- predicted it.

- http://www.spc.noaa.gov/exper/sref/

## NRC vision: NWS to make all products probabilistic. How?

**General option 1**: Work harder at current vision of developing hi-resolution models and ensembles.  Probabilistic products based on these, perhaps QC’ed by humans.

## Models ARE improving dramatically,

- and with them, ensemble forecasts.

- We now have
- models with
- explicit convection
- that produce
- forecasts that look,
- for the first time,
- like radar images
- of precipitation.

## Unfortunately, even the state-of-the art ensemble from ECMWF still produces unreliable forecasts

- Probabilistic forecasts
- from raw ensembles
- are not very reliable,
- due to deficiencies in
- forecast model,
- ensemble methods.
- Users want “sharp”
- and “reliable” forecasts.

## Manually QC new probabilistic products? Tough task.

- Different WFOs have different ideas about what corrections to make, leading to discontinuities. Expect
- probabilistic QC even tougher.
- In most circumstances, it may be a better use of forecasters’ time to
- focus on the shorter-range, more severe-weather problems.

## NRC vision: NWS to make all products probabilistic. How?

**General option 2**: Automated probabilistic forecast products, or “Ensemble MOS”; ensemble forecast system + computer-based statistical post-processing using “*reforecasts”* (past forecasts from same system used operationally).

## NOAA’s reforecast data set

**Model**:  T62L28 NCEP GFS, circa 1998
**Initial Conditions**: NCEP-NCAR Reanalysis II plus 7 +/- bred modes.
**Duration**: 15 days runs every day at 00Z from 19781101 to now. (_[http://www.cdc.noaa.gov/people/jeffrey.s.whitaker/refcst/week2](http://www.cdc.noaa.gov/people/jeffrey.s.whitaker/Manuscripts/multimodel.pdf)_*).*
**Data**:  Selected fields (winds, hgt, temp on 5 press levels, precip, t2m, u10m, v10m, pwat, prmsl, rh700, heating).  NCEP/NCAR reanalysis verifying fields included (Web form to download at _[http://www.cdc.noaa.gov/reforecast](http://www.cdc.noaa.gov/people/tom.hamill/skill_overforecast_QJ_v2.pdf)_).
**Real-time** probabilistic precipitation forecasts: _[http://www.cdc.noaa.gov/reforecast/narr](http://www.cdc.noaa.gov/people/tom.hamill/whitepaper_reforecast.pdf)_

**Notes:**
As promised - details of dataset.

11 months done - dataset will be complete in about a month.

## ECMWF’s reforecast data set

**Model**: 2005 version of ECMWF model; T255 resolution.
**Initial Conditions**: 15 members, ERA-40 analysis + singular vectors
**Dates of reforecasts**: 1982-2001, Once-weekly reforecasts from 01 Sep - 01 Dec, 14 total.  So, 20*14 ensemble reforecasts = 280 samples.
**Data** sent to NOAA / ESRL : T2M, precip. ensemble over most of North America, excluding Alaska.  Saved on 1-degree lat / lon grid.  Forecasts to 10 days lead.

## Reforecasts and T2m calibration

- Calibrated GFS based on 1998 ensemble more skillful
- than probabilities from raw ECMWF ensemble.
- (2) Substantial improvement of ECMWF ensemble based
- on reforecasts; smaller amount than GFS, but still large.

## Good news: for some variables a few prior forecasts are adequate to calibrate. 

- Statistically correcting a
- probabilistic surface temp.
- forecast with 30-day training
- data set and 20-year,
- once-weekly training data set.
- At short leads, there is no
- advantage to a long training
- data set. At longer leads,
- there is an advantage.
- Short-lead temperature
- bias in this 2005 version of
- ECMWF model is relatively
- consistent from one day
- to the next.

## Tougher news: for other problems such as calibrating heavy precipitation, larger training data sets are necessary. 

- Consider training with a short sample in a climatologically dry
- region.  How could you calibrate this latest forecast?

- You’d like
- enough
- training data
- to have some
- similar events
- at a similar
- time of year
- to this one.

## Boost sample size in statistical calibration by compositing statistics over different locations?

- Panels (a) and (b) provide the cumulative density function (CDF) of 1-day forecasts of precipitation for 1 January (CDFs determined from reforecast data and observations in Dec-Jan).  Panel (a) is for a location near Portland, Oregon, and panel (b) is in north central Oregon, east of the Cascades. Panel (c) provides the implied function for a bias correction from the forecast amount to a presumed observed amount.  Note the very different corrections implied at two nearby locations.

- A good idea, if done with care.
- However, even nearby grid points may have different forecast errors.

## Using reforecasts for PQPF calibration: analog technique

- On the left are old forecasts
- similar to today’s ensemble-
- mean forecast.  For making
- probabilistic forecasts,
- form an ensemble from
- the accompanying
- analyzed weather on the
- right-hand side.

## Producing a distribution of observed given forecast using analogs

- On the left are old forecasts
- similar to today’s ensemble-
- mean forecast.  For making
- probabilistic forecasts,
- form an ensemble from
- the accompanying
- analyzed weather on the
- right-hand side.

## Verified over 25 years of forecasts; 

- Verified over 25 years of forecasts;
- skill scores use conventional
- method of calculation which may
- overestimate skill
- (Hamill and Juras 2006, QJRMS, Oct).

## Comparison against NCEP medium-range T126 ensemble, ca. 2002

- the improvement is a little bit
- of increased reliability, a lot
- of increased resolution.

## Nov ‘06

- OR-WA
- floods,
- 3-6 day
- forecast

## Effect of training sample size

- colors of dots indicate which size analog ensemble
- provided the largest amount of skill.

## Real-time products

## Can we do both hi-res model development and reforecasting, or a compromise?

**Alternative 1**: Continue development of high-res. models.  Do reforecasting with inexpensive, low-res. model, so operations are impacted minimally.
- Suppose operational T300, 60-layer, 50-member ensemble forecast system.
- Reforecast T150, 40 layer, 5-member ensemble :
- Operational cost: 120x less
- 120 days of reforecasts for one day of operational forecast, so a 20-year reforecast for the cost of 60 days of operational model forecasts.
- If new reforecast model implemented once, say, every 4 years, minimal impact to operations integrated over time.
**Alternative 2**: Continue development of high-res. models. Do reforecasting offline, on non-operational computer system.
- ~ $700K would buy a computer system that could do a T170L42, 5-member reforecast out to 10 days in ~ 1 year wall time.

## What’s next for reforecasting?

- Growing interest from NWP centers worldwide
- ECMWF exploring once-weekly ensemble reforecasts (with my participation)
- Canadians planning 5-year ensemble reforecasts
- NCEP envisioning 1-member, real-time reforecast for bias correction.
- Possibility that NOAA/ESRL may get money to do a more complete, 2nd-generation reforecast data set.

## Research questions

- Given computational expense of reforecasts, how do we best:
- Limit the number of reforecasts that we need to do (fewer ensemble members, not every day, etc.)
- Can we do things like composite the data across different locations to boost sample size?
- Do we need a new reanalysis every time we do a new reforecast?
- Do the benefits of reforecasts propagate down to users like hydrological forecasters?

## FY 10-14 Reforecast Alternative

- FY07-09: Weather-Climate Connection funding reforecast research (OAR/Climate) zeroed out. Begging/borrowing to keep going.
- FY10: Produce a next-generation reforecast data set (~T170). [ESRL]
- FY11-13: Develop calibration methods for suite of experimental probabilistic weather applications (PQPF, severe, fire wx, etc) , determine reforecast configuration [ESRL, NSSL, EMC, CPC, OHD, MDL].
- FY14: Transition to operations.

## References

- Hamill, T. M., J. S. Whitaker, and X. Wei, 2003: Ensemble re-forecasting: improving medium-range forecast skill using retrospective forecasts. *Mon. Wea. Rev*., **132**, 1434-1447. http://www.cdc.noaa.gov/people/tom.hamill/reforecast_mwr.pdf
- Hamill, T. M., J. S. Whitaker, and S. L. Mullen, 2005: Reforecasts, an important dataset for improving weather predictions.  *Bull. Amer. Meteor. Soc*., **87**, 33-46.
- http://www.cdc.noaa.gov/people/tom.hamill/refcst_bams.pdf
- Whitaker, J. S, F. Vitart, and X. Wei, 2006:   Improving week two forecasts with multi-model re-forecast ensembles. *Mon. Wea. Rev*., **134**, 2279-2284. http://www.cdc.noaa.gov/people/jeffrey.s.whitaker/Manuscripts/multimodel.pdf
- Hamill, T. M., and J. S. Whitaker, 2006: Probabilistic quantitative precipitation forecasts based on reforecast analogs: theory and application. *Mon. Wea. Rev*., in press. http://www.cdc.noaa.gov/people/tom.hamill/reforecast_analog_v2.pdf
- Hamill, T. M., and J. Juras, 2006: Measuring forecast skill: is it real skill or is it the varying climatology? *Quart. J. Royal Meteor. Soc*., in press. http://www.cdc.noaa.gov/people/tom.hamill/skill_overforecast_QJ_v2.pdf
- Wilks, D. S., and T. M. Hamill, 2006: Comparison of ensemble-MOS methods using GFS reforecasts. *Mon. Wea. Rev*., in press. http://www.cdc.noaa.gov/people/tom.hamill/WilksHamill_emos.pdf
- Hamill, T. M. and J. S. Whitaker, 2006: White Paper.  “Producing high-skill probabilistic forecasts using
- reforecasts: implementing the National Research Council vision.”  Available at http://www.cdc.noaa.gov/people/tom.hamill/whitepaper_reforecast.pdf .

## Calibrating Week 2 and 6-10 day probability forecasts

- An example
- of the operational
- 6-10 day temperature
- forecast produced by
- NCEP/CPC.

## Week-2 Temperature Forecasts

- Correction of
- biases estimated
- from full 22 years
- of forecast data

- Correction of biases
- estimated from last
- 45 days of data

- Probabilities
- from raw
- ensemble

**Notes:**
What happens if you just take an ensemble and treat it as a random sample of the forecast PDF - calculate probabilities by counting how may fall in each category.

We used the 23 year dataset of MRF forecasts that I will describe shortly to see how much useful information you can get from this approach.

Describe reliability plots - they give you a way to quickly assess two important aspects of a probability forecast - reliability and sharpness.

Note bias - two tercile forecasts are different.

What is a perfect RPSS (what should we expect) - calculate perfect prog RPSS.

Clearly, something needs to be done to the model output.  Note: this is not how CPC does it - they look only at the ensemble mean and subjectively determine probabilities.

## Calibration using a long data setof observed and forecast anomalies

- Dashed lines: tercile boundaries
- Red points: samples above upper tercile
- Blue points: samples below upper tercile
- Solid bars: probabilities by bin count
- Dotted line: a fitted logistic regression

- Calibration using a long data setof observed and forecast anomalies

- With our reforecasts,
- we have 20+ years
- of data. Let’s use
- old data in a 31-day
- window around the
- date of interest to
- make statistical
- corrections.

**Notes:**
A concrete example to illustrate how we do this:

T850 for a given grid point - scatter plot of all forecasts and verifications over 23*31 cases.
If we just calculate the observed frequency for discrete bins, you get probs given by the solid horiz lines.  If you use logistic regression, you get the smooth dotted curve

Note there may be problems with extrapolation - over-forecast extreme probs

## 6-10

- Day

- Week
- 2

**Notes:**
Revisit the T850 forecast problem - bias correction improves RPSS, but does not make fcsts more reliable.

## Comparison against NCEP / CPC forecasts 

- at 155 stations, 100 days in winter 2001-2002

- MOS-based
**Week 2 **
- forecasts
- using low-res
- T62 model
- more skillful
- than
- operational
- NCEP/CPC
**6-10 day**.
- (NCEP now
- heavily
- utilizes
- reforecasts
- in these
- products)

**Notes:**
Precip skill - some skill at 6-10,  a wee bit at week 2.

Note: for precip terciles are sometimes ill-defined at dry stations.

## Example: floods causing La Chonchita, CA mudslide, 12 Jan 2005

- week-2 forecast

- 6-10 day forecast

## Skill as function of location

## Asymptotic behavior of analog technique

- Q: What happens as correlation(F,O)  0 ?	A: Ensemble of observed analogs becomes random draw from climatology.
- Q: What happens as correlation(F,O)  1 ? A: Ensemble of observed analogs looks just like today’s forecast.  Sharp, skillful forecasts.