# Presentation: 731561

## Downscaling ensembles using forecast analogs

- Jeff Whitaker and Tom Hamill
- _[tom.hamill@noaa.gov](mailto:tom.hamill@noaa.gov)_
- _[jeffrey.s.whitaker@noaa.gov](mailto:jeffrey.s.whitaker@noaa.gov)_

**Notes:**
• we’ve been working on methods for improving prediction skill by using model output statistics.  Dan Gilman quote “fcst by itself is an incomplete product’ - corollary ‘numeric model output by itself is not a forecast’.

## CDC MRF Reforecast Data Set

**Definition: **a data set of retrospective numerical forecasts using the same model to generate real-time forecasts
**Model**:  T62L28 MRF, circa 1998 (*http://www.cdc.noaa.gov/people/jeffrey.s.whitaker/refcst* for details).
**Initial States**: NCEP Reanalysis plus 7 +/- bred modes (Toth and Kalnay 1993).
**Duration**: 15 days runs every day at 00Z from 19781101 to now. (*http://www.cdc.noaa.gov/people/jeffrey.s.whitaker/refcst/week2**).*
**Data**:  Selected fields (winds, hgt, temp on 5 press levels, precip, t2m, u10m, v10m, pwat, prmsl, rh700, heating).  NCEP/NCAR reanalysis verifying fields included (Web form to download at *http://www.cdc.noaa.gov/reforecast*).

**Notes:**
Dataset:  old MRF T62.  
‘Gold mine’ - really unique.

## Applications

- Predictability studies
- Diagnosis of model error
*Statistical correction of real-time forecasts*
- 6-10 day and week 2 CPC temp and precip tercile probabilities  (now operational)
- 	Uses logistic regression at stations (Hamill et al, 2004, MWR, p. 1434)

**Notes:**
Barely scratched the surface - we’ve focused on statistical corrections.  
Using a techique called logistic regression we were able to improve 6-10 and week 2 probabilistic forecast skill relative to the operational CPC product.  We’ve just finished crossing the ‘valley of death’, now these forecasts are generated operationally at NCEP.

## HSS scores

*HSS scores*
- 9/10/03-
- 9/9/04
- Week 2
**Temp**:
- Official: 14.74
- CDC:    16.80
**Precip**:
- Official: 10.27
- CDC:     8.09

**Notes:**
Here’s an example of the product that the forecasters have - tercile probs at station locations.  In terms of HSS, over the last year the temp forecasts are a little more skillful than operational.  Precip less so.  Why are precip forecasts less skillful, at least in the categorical measure?  

## But these forecasts are very coarse resolution...

- Finer-scale detail is desirable, especially for precip.
- How can we take large-scale NWP/GCM output and “downscale” it to provide skillful higher-resolution forecasts?
- How to correct for ‘regime-dependant’ errors?

**Notes:**
• could be that forecasts with T62 just cannot capture the finer scale detail
• logistic regression is univariate - correct every grid point indepenently.  How to correct for regime dependant biases?
• How to create random samples which preserve covariances (for hydro)?

## Analogtechnique:(pioneered by van den Dool, Toth, von Storch, others)

- Analogtechnique:*(pioneered by van den Dool, Toth, von Storch, others)*

- Step 1:
- make today’s
- forecast

- Step 3: extract
- observed weather

- Observed Wx, 3/1/83

- Observed Wx, 2/12/95

- Observed Wx, 1/16/98

- Forecast Analog 3,
- 3/1/83

- Forecast analog 1,
- 2/12/95

- Forecast analog 2,
- 1/16/98

- TODAY’S ENS MEAN
- PRECIP FORECAST_x0016_

- BMA?

- Step 2: find dates
- of old analogs

**Notes:**
• advantages of analogs 1) preserve spatial covariances, 2) non-parametric, 3) can correct for regime-dependant errors, 4) produces calibrated PDF.
• this technique is different because it uses forecast, not observed analogs.
• trick, as Huug pointed out in his paper, is to use local analogs.

## Local analogs are patched together

- Initial implementation very simple:
-  *Single forecast field (precip).*
* **L2 norm (rms) using ens. mean fcst.*
*Analog ensemble members receive equal weight.*
* **50 analog members - NARR.*

**Notes:**
Blue are t62 model grid points, black are NARR anal pts.  In a local region, comprised of 16 T62 model grid points, we find set of forecast analogs.  Corresponding NARR analyses used in the middle (red points).  Shift grid over by one blue grid point, do it again to get adjacent set of red points.  These are patched together.  Norm is forecast precip (could use other fields, like humidity, circulation).  Equal weighting of analogs (Tom has done some work that indicates you can do about 10% better with a more sophisticated weighting).

## Example: 4-6 day analog forecasts, valid 29-31 Dec 1996) 

**Notes:**
Analogs might have trouble with extreme, rare, or even record-breaking events.  Here’s new year’s 1997 flood (10+” in 2 days).  

## Skill of Analog Forecasts

**Notes:**
• skill evaluated over West Coast for JFM 2001-2003. P > 10mm for days 1-7.
• ensemble relative frequency from refcst T62 ensemble. Skill low (peculiar increase from day 1 to 2).

## Skill of Analog Forecasts

**Notes:**
• analog forecasts vastly superior (day 7 more skillful than day 2).

## Skill of Analog Forecasts

***3 days***

**Notes:**
• ensemble relative frequency from operational ensemble (run at twice the resolution, from a better analysis).  
• Analogs much better, a 3 day increase in lead time.

## Skill of Analog Forecasts

**Notes:**
• purple vs green curve shows the benefit of regime-dependant error correction provided by analogs.  
(logistic regression, using only ensemble mean at one pt).   The multi-variate approach, using the local pattern, yields about a 1 day increase in lead time.

## Application - Tercile Forecasts

- Prob of above normal for 2nd *N* days of forecast (*N*=1 to 6).
- All JFMs 1979-20035 (no analogs within +/- 45 days of verifying analysis used).
- NARR precip over entire CONUS.

***day 2***

***days 3-4***

***days 4-6***

***days 5-8***

***days 6-10***

**Notes:**
• We’ve also evaluated the analog forecasts for tercile prob fcsts.  Here we are verifying forecasts for 25 JFMs over the entire CONUS for P of above normal precip.  Skill as a function of lead time is shown in the graph for forecasts for the 2nd N days in the forecast period (N=1 is day 2 fcst, N=2 is days 3-4 etc).  No analogs are used within +/- 45 days of the verifying analysis.  Signficant skill into week 2.  The main thing I’d like to show here is the spatial distribution of the skill ...

## Analog Forecast Skill - Upper Tercile

**Notes:**
Patched together to form maps for CONUS.  Here’s the 10mm threshold skill map.  Fascinating regional variations in skill.  Note west coast skill, may be counter intuitive since there is a data void offshore (mtns, PNA).  Problems over mountains in west.  May be due in part to problems in precip dataset (unrepresentatitive obs, poor snow).  Note overall BSS is good, forecasts are well calibrated.  Explain reliability diagram.

## Analog Forecast Skill - Upper Tercile

**Notes:**
As lead time increases, skill over eastern U.S. declines.  Forecasts remain reliable.

## Analog Forecast Skill - Upper Tercile

**Notes:**
By 2nd 3-day period, skill has declined considerably over EUS, but is still strong on the W. Coast.

## Analog Forecast Skill - Upper Tercile

## Analog Forecast Skill - Upper Tercile

**Notes:**
6-10 day forecasts are really only useful on W. Coast.

## Analog Forecast Skill - Upper Tercile

**Notes:**
Patched together to form maps for CONUS.  Here’s the 10mm threshold skill map.  Fascinating regional variations in skill.  Note west coast skill, may be counter intuitive since there is a data void offshore (mtns, PNA).  Problems over mountains in west.  May be due in part to problems in precip dataset (unrepresentatitive obs, poor snow).  Note overall BSS is good, forecasts are well calibrated.

## Analog Forecast Skill - Upper Tercile

**Notes:**
Week 2 overall BSS is very small, but useful in CA, AZ.

## Analog Size

Analog Search Region (75 analogs)

Finding analogs for each member:  5 analogs per member, skill is degraded (BSS = 0.183).
Forecast variable, analog weighting?

- Analog Search Region (75 analogs)
- Finding analogs for each member:  5 analogs per member, skill is degraded (*BSS = 0.183*).
- Forecast variable, analog weighting?

**Free parameters** (WCoast, 4-6 day upper decile)

| # of Analogs | 25 | 50 | 75 | 100 |
| --- | --- | --- | --- | --- |
| BSS | 0.2050 | 0.2179 | 0.2185 | 0.2168 |

| Grid Points | 4 | 16 | 36 |
| --- | --- | --- | --- |
| BSS | 0.1967 | 0.2185 | 0.2165 |

**Notes:**
Our initial implementation was very simple - we have looked at the sensitivity of some of the parameters.
Analog ens size - insensitive.
Search region - initially chose 16 points.  A larger region does not help, but using only the surrounding 4 points is worse (agrees with logistic regression result).
Using all members hurts rather than helps.
Things we haven’t yet looked at - using other forecast variables to define analogs (humidity, low level winds).
Probs are computed by treating all analogs equally - can determine weights from a regression.  Tom’s results indicate that this may increase skill by about 10%.

## Conclusions

- Forecast analogs (using ensemble mean) hold great promise.
-  *preserves covariances.*
* **non-parameteric.*
* **corrects for regime-dependant errors.*
* **produces 3-day lead time improvement in PQPF skill relative to operational system run at twice the resolution.*

**Notes:**
Conclusions - analog technique I believe is an important advance over univariate regression techniques.
Need a large dataset of reforecasts to find good analogs, especially for rare events.  Applies to all types of forecasts, from short range to SI.