# Presentation: 632614

## Experiments calibrating 3-km ensemble forecasts over the American River Basin

- Tom Hamill
- ESRL/PSD
- _[tom.hamill@noaa.gov](mailto:tom.hamill@noaa.gov)__, x3060_

## Review

- 4 nested reforecasts (3 km); IOPs 1, 4, 10, 12.
- 20 ensemble members (4 members * 5 lags).
- Verification data from Stage IV.

- IOP 1
- 2005113006
- 2005113012
- 2005113018
- 2005120100
- 2005120106
- 2005120112
- 2005120118
- 2005120200
- 2005120206
- 2005120212
- 2005120218

- IOP 4
- 2005122918
- 2005123000
- 2005123006
- 2005123012
- 2005123018
- 2005123100
- 2005123106
- 2005123112
- 2005123118
- 2006010100

- IOP 10
- 2006020112
- 2006020118
- 2006020200
- 2006020206
- 2006020212
- 2006020218
- 2006020300

- IOP 12
- 2006022606
- 2006022612
- 2006022618
- 2006022700
- 2006022706
- 2006022712
- 2006022718
- 2006022800

- treat each IOP
- as separate
- sample rather
- than each initial
- time within the
- IOP

## HMT

- Domain

- terrain data from US
- Navy 5-minute data set.

## IOP 1, Observed and Forecast Mean

- overforecasting
- earlier in IOP,
- better later

## IOP 4, Observed and Forecast Mean

- late with onset.
- again, some
- overforecasting,
- but pretty good.

## IOP 10, Observed and Forecast Mean

- pretty good.  slight dry bias?

## IOP 12, Observed and Forecast Mean

- pretty good.

## Forecast Mean vs. Observed

- Red line is
- conditional mean
- of observed
- given forecast.
- Overforecast bias
- for IOP 1, 4.

## Are errors related to elevation?

- Not obviously so.

## Rank Histograms

- Here, if obs > 0, ensemble members are perturbed
- by ~ *N*(0, 1+obs/10).
- Ranks randomly assigned with ties.
- Overforecast bias evident for high amounts.

## Rank Histograms

- Now, without perturbing ensemble members.
- Not much change.

## Questions

- Can calibration improve upon raw forecasts?
- Can meaningful calibration be done with only 4 cases?
- Previous (Hamill and Colucci 1998) results found most information for calibrating precipitation seemed to be in ensemble mean.  Will it be any different with this ensemble [ i.e., is ensemble spread a useful predictor? ]

## Ensemble mean and spread

- IOP 4 has heavy
- precipitation, others
- much lighter.  Will
- cross-validated
- training data prove
- useful for IOP 4?

## Cross-validated mean and spread for the 4 cases

- Here we plot the training
- data that would be used for
- each IOP, consisting of the
- other 3 cases. The orange
- lines are regression fits for the
- average spread as a function
- of the ensemble mean and
- √(mean), i.e.,
- Avg spread(mean) = b0*mean +
-        b1* √ (mean)

## Ensemble mean andnormalized spread

- Here instead of plotting the
- spread we plot the normalized
- spread, which is the ratio
- of a given spread to the
- regression-predicted avg.
- spread appropriate for that
- ensemble mean amount.
- You can see that normalized
- spread looks much more
- uncorrelated with the
- ensemble  mean than the
- raw spread on the previous
- plot.
- Let’s examine mean and
- normalized spread as
- potential predictors.

## Logistic Regression

- Given predictors x1, ... , xN  (such as the mean and
- normalized spread), find regression coefficients
**0, **1, ...,  **N for the equation

- This generates an S-shaped curve
- (with one predictor)

## 2-parameter logistic regression (√ mean & normalized spread) , 1 mm

- Effect of normalized
- spread varies
- strongly from one
- case to another.
- Suggests normalized
- spread may not be
- important predictor.

## 2-parameter logistic regression, 25 mm

- Similar for 25 mm;
- variability with
- normalized spread

## 2-parameter logistic regression, 25 mm

- Odd that probabilities
- tend to INCREASE
- as spread gets larger
- for large amounts.
- Expect that fit of
- regression coefficients
- is dominated by
- large number of
- samples with near-zero
- precipitation.

## 3-parameter logistic regression, 1 mm

- A large variation of the
- forecast probabilities
- from one IOP to the next,
- but very few samples
- with normalized spread
- greater than 2.5

- Predictors:
- √ens mean
- Normalized spread
- (1)*(2)

- this pattern conforms
- with my expectations

## Mean and normalized spread

## 3-parameter logistic regression, 25 mm

- Not what I expected;
- a low-spread, 50-mm
- mean forecast has
- higher probability than
- a high-spread, 50-mm
- mean forecast.

## 1-parameter logistic regression (ens mean)

- Back to a simple model.
- Even with 1 parameter,
- it’s hard to escape
- conclusion that 4 cases
- are not enough to
- reliably estimate
- probabilities, given
- large case-to-case
- variability.

## Probabilistic precip. forecasts...

## Probabilistic precip. forecasts...

## Probabilistic precip. forecasts...

## Probabilistic precip. forecasts...

## Reliability, 2.5 mm, raw and 1-parameter logistic regression

## Reliability, 10 mm, raw and 1-parameter logistic regression

## Reliability, 25 mm, raw and 1-parameter logistic regression

## Reliability, 50 mm, raw and 1-parameter logistic regression

## Conclusions

- Large variability of regression coefficients, statistical corrections from case to case.
- Pretty good raw ensemble forecasts.
- Small improvement from calibration (good raw forecasts, small training sample, unknown predictors)
- More cases?

## Mean and spread (again)

- Even though
- these are cases
- dominated by
- heavy precip,
- is logistic regression
- dominated by
- trying to get the
- fit correct for the
- many samples with
- little or no precip?