# Presentation: 731528

## Importance of Large Training Datasets for MOS/PQPF

- Jeff Whitaker and Tom Hamill
- _[tom.hamill@noaa.gov](mailto:tom.hamill@noaa.gov)_
- _[jeffrey.s.whitaker@noaa.gov](mailto:jeffrey.s.whitaker@noaa.gov)_

**Notes:**
• define ‘reforecasts’, history, acknowledgements.
• review pentad2,week2 tercile prob/CPC stuff, update you on the transition to operations in first half,
• 2nd half talk about new products, extreme temps and downscaled precip.

## Interannual variability of 850T bias

- ?

**Notes:**
• model bias is a well known problem.  This figure illustrates why a large dataset (will describe in a couple of slides) is needed to correct systematic errors.
• spread of yearly estimates larger than signal.  Using the last year can make things worse instead of better.

## CDC MRF Reforecast Data Set

**Definition: **a data set of retrospective numerical forecasts using the same model to generate real-time forecasts
**Model**:  T62L28 MRF, circa 1998 (*http://www.cdc.noaa.gov/people/jeffrey.s.whitaker/refcst* for details).
**Initial States**: NCEP Reanalysis plus 7 +/- bred modes (Toth and Kalnay 1993).
**Duration**: 15 days runs every day at 00Z from 19781101 to now. (*http://www.cdc.noaa.gov/people/jeffrey.s.whitaker/refcst/week2**).*
**Data**:  Selected fields (winds, hgt, temp on 5 press levels, precip, t2m, u10m, v10m, pwat, prmsl, rh700, heating).  NCEP/NCAR reanalysis verifying fields included (Web form to download at *http://www.cdc.noaa.gov/reforecast*).

**Notes:**
As promised - details of dataset.

About 2Tb in all, available for download by all.  BAMS article in preparation.

## Applications

- Predictability studies
- Diagnosis of model error
*Statistical correction of real-time forecasts*
- 6-10 day and week 2 CPC temp and precip tercile probabilities  using logistic regression at stations (Hamill et al, 2004, MWR, p. 1434) - ***Now Operational***
-  daily PQPF using forecast analogs - ***Experimental***

## Analogtechnique:(pioneered by van den Dool, Toth, von Storch, others)

- Analogtechnique:*(pioneered by van den Dool, Toth, von Storch, others)*

- Step 1:
- make today’s
- forecast

- Step 3: extract
- observed weather

- Observed Wx, 3/1/83

- Observed Wx, 2/12/95

- Observed Wx, 1/16/98

- Forecast Analog 3,
- 3/1/83

- Forecast analog 1,
- 2/12/95

- Forecast analog 2,
- 1/16/98

- TODAY’S ENS MEAN
- PRECIP FORECAST_x0003__x0016_

- BMA?

- Step 2: find dates
- of old analogs

**Notes:**
• advantages of analogs 1) preserve spatial covariances, 2) non-parametric, 3) can correct for regime-dependant errors, 4) produces calibrated PDF.
• this technique is different because it uses forecast, not observed analogs.

## Local analogs are patched together

- Initial implementation very simple:
-  *Single forecast field (precip).*
* **L2 norm (rms) using ens. mean fcst.*
* **Analog ensemble members receive equal weight.*
* **75 analog members - NARR.*

**Notes:**
Blue are t62 model grid points, black are NARR anal pts.  In a local region, comprised of 16 T62 model grid points, we find set of forecast analogs.  Corresponding NARR analyses used in the middle (red points).  Shift grid over by one blue grid point, do it again to get adjacent set of red points.  These are patched together.  Norm is forecast precip (could use other fields, like humidity, circulation).  Equal weighting of analogs (Tom has done some work that indicates you can do about 10% better with a more sophisticated weighting).

## Example: 4-6 day analog forecasts, valid 29-31 Dec 1996 

**Notes:**
Analogs might have trouble with extreme, rare, or even record-breaking events.  Here’s new year’s 1997 flood (10+” in 2 days).  

## CONUS skill JFM 1979-2003 (P > 10mm)

**Notes:**
Patched together to form maps for CONUS.  Here’s the 10mm threshold skill map.  Fascinating regional variations in skill.  Note west coast skill, may be counter intuitive since there is a data void offshore (mtns, PNA).  Problems over mountains in west.  May be due in part to problems in precip dataset (unrepresentatitive obs, poor snow).  Note overall BSS is good, forecasts are well calibrated.

## Skill of Analog Forecasts

## Skill of Analog Forecasts

**Notes:**
Green Curve:  NCEP bias correction (Zhu) - construct cum freq dist for forecast
and observed, for each fcst value find observed value that has the same freq as 
Forecast val (last 60 days). Re-label forecast val with corresponding
observed value.  All grid pts are aggregated.

## Skill of Analog Forecasts

***3 days***

## Skill of Analog Forecasts

**Notes:**
• purple vs green curve shows the benefit of regime-dependant error correction provided by analogs.  
(logistic regression, using only ensemble mean at one pt).   The multi-variate approach, using the local pattern, yields about a 1 day increase in lead time.

## Skill of Analog Forecasts

**Notes:**
• purple vs green curve shows the benefit of regime-dependant error correction provided by analogs.  
(logistic regression, using only ensemble mean at one pt).   The multi-variate approach, using the local pattern, yields about a 1 day increase in lead time.

## P > 10mm

- Oper

- Analogs (4 yrs)

***P > 10mm***
***Reliability Diagrams***
***4-day forecasts***

- Analogs (24 years)

- Oper (bias corr)

## Skill of Analog Forecasts

**Notes:**
• purple vs green curve shows the benefit of regime-dependant error correction provided by analogs.  
(logistic regression, using only ensemble mean at one pt).   The multi-variate approach, using the local pattern, yields about a 1 day increase in lead time.

## Skill of Analog Forecasts

**Notes:**
• purple vs green curve shows the benefit of regime-dependant error correction provided by analogs.  
(logistic regression, using only ensemble mean at one pt).   The multi-variate approach, using the local pattern, yields about a 1 day increase in lead time.

## Skill of Analog Forecasts

**Notes:**
• purple vs green curve shows the benefit of regime-dependant error correction provided by analogs.  
(logistic regression, using only ensemble mean at one pt).   The multi-variate approach, using the local pattern, yields about a 1 day increase in lead time.

## Skill of Analog Forecasts

**Notes:**
• purple vs green curve shows the benefit of regime-dependant error correction provided by analogs.  
(logistic regression, using only ensemble mean at one pt).   The multi-variate approach, using the local pattern, yields about a 1 day increase in lead time.

## Skill of Analog Forecasts

**Notes:**
• purple vs green curve shows the benefit of regime-dependant error correction provided by analogs.  
(logistic regression, using only ensemble mean at one pt).   The multi-variate approach, using the local pattern, yields about a 1 day increase in lead time.

## Forecast from last Thursday for this Mon-Wed

## Forecast from last Thursday for this Mon-Wed

## Multi-model ensemble (84 cases over 12 DJFs)

- 850T Week 2 Extreme Terciles (Logistic Regression)

***T159 EC is ***
***better than ***
***T62 MRF***

***Combo is better***
***than either alone***

## Conclusions

- Forecast analogs arguably the best post-processing technique with large training sets.
-  *preserves covariances.*
* **non-parameteric.  Simple!*
* **downscales, corrects for regime-dependent errors*
* **produces 1-3-day lead time improvement in PQPF skill relative to operational ensemble run at twice the resolution, with CDF bias correction.*
- Can get by with 10 year set for frequent events, extreme events benefit most from longer set.
***Long training sets worth the effort!***  Improvements in forecast skill equivalent to a decade of forecast system improvement.

## Noise in yearly bias estimates

**Notes:**
• another view of same thing - note signal to noise decreases with lead time.  Might get away with a year for a 1 day forecast, certainly can’t for week 2.

## Skill of Analog Forecasts

**Notes:**
• purple vs green curve shows the benefit of regime-dependant error correction provided by analogs.  
(logistic regression, using only ensemble mean at one pt).   The multi-variate approach, using the local pattern, yields about a 1 day increase in lead time.

## Skill of Analog Forecasts

**Notes:**
• purple vs green curve shows the benefit of regime-dependant error correction provided by analogs.  
(logistic regression, using only ensemble mean at one pt).   The multi-variate approach, using the local pattern, yields about a 1 day increase in lead time.