# Presentation: 624427

## Short-Range Ensemble Forecasts of Precipitation Type

- Matt Wandishin		John Cortinas
- Steve Mullen		Mike Baldwin
- University of Arizona      	OU/CIMMS

**Notes:**
This talk extends work by Cortinas (funded by COMET) details of which can be found at www.spc.noaa.gov/exper/ptax/.
That work applied 6 different algorithms to output from the operational Eta and RUC models to determined probabilities for precipitation type.

## The Algorithms

- Ramer (1993)—ice fraction, I, determined by 	phase at generation level and subsequent 	melting or freezing based on *T**w* and *p* as 	hydrometeor falls
- Baldwin (et al. 1994)—compares warm and cold 	layers within column (area between *T**w* and
- 		0 oC)
- Bourgouin (2000)—similar to Baldwin, but 	computes melting and freezing energies from 	tephigram

**Notes:**
The algorithms are all applied as post-processing.  That is, there is no interaction between algorithm and model.  Also, the algorithms only use the thermodynamic data of the column above the given surface station.
Cortinas’s work also included a thickness algorithm.  We have ignored this algorithm because a great many stations do not have 1000mb data and thus the algorithm cannot be applied.

## The Algorithms

- Czys (et al. 1996)—computes ratio of time an ice 	sphere remains in a warm layer to time 	required to completely melt the sphere—uses 	fixed sphere size and bulk properties of warm 	layer
- Cortinas (2001)—similar to Czys but computes 	melting rate of ice sphere at each vertical level

**Notes:**
In Cortinas’s work, the Cortinas algorithm was simplified to only differentiate between rain and snow due to excessive computational time of the full algorithm.  We have reinstated the full algorithm.

## The Ensemble

- Short-range ensemble from NCEP
- 10 members
- 		5 Eta, 5 RSM (Regional Spectral Model)
- Regional bred modes
- 48 km horizontal grid spacing
- 3 hourly output to 48 hours

**Notes:**
The ensemble is the same as is run by NCEP except that we do not have the 5 members using the Kain-Fritsch convective parameterization.
Also, the operational short-range ensemble is run out to 63 hours and we have only taken the first 48 hours.

## The Experiment

- Precipitation-type algorithms applied to ensemble runs from January and February 2002—43 days
- Only conditional statistics computed
- 		Some type of precipitation must be both 	forecast and observed
- Surface temperature must be less than 5 oC
- Time window +/– 0 h
- Unless otherwise noted statistics computed for all forecast hours lumped together

**Notes:**
The sfc temperature cutoff is to rule out “easy” rain forecasts.
Cortinas used a less-restrictive +/- 1 hour time window.  The performance of the ensemble should then be expected to be somewhat worse.

The algorithms were applied to model output at nearly 500 surface stations resulting in nearly 18,000 forecast/observation pairs in which precipitation was both observed and forecast by at least one ensemble member, breaking down as follows:
	rain  4325			freezing rain  498			ice pellets  189			snow  12,815

Statistics used in this talk to evaluate performance are attributes diagrams, ROC curves, and spread.

## Full Ensemble

- 0.663

- -0.033

- -1.069

- 0.629

**Notes:**
The 4 panels for the next group of slides show performance for the 4 precipitation types: rain (RN), freezing rain (ZR), ice pellets (IP), snow (S).

For any unfamiliar with attributes diagrams the colored solid lines plot the forecast probabilities against the frequency of event occurrence for the times on which that forecast probability was issued and so measure the reliability of the forecasts.  The thin line on the 45 degree diagonal is the perfect-skill line: relative frequency = forecast probability.  The vertical and horizontal lines mark the frequency of event occurrence regardless of the forecast, essentially climatology for the period of the experiment.  The shallow diagonal is the no-skill line, where skill is measures against the climatology.  When the reliability curve is above the perfect-skill line, the event is underforecast.  For example, snow occurs 40% of the time a 20% chance of snow is forecasted.  Similarly, when the reliability curve is below the perfect-skill line the event is overforecast.  The Brier Skill Score (the number plotted beneath the ptype label) average squared distance between the reliability curve and the perfect-skill line weighted by the frequency with which each forecast probability is used (dashed line).

For the full ensemble, rain forecasts are excellent and snow forecasts are very good.  Visually the snow forecasts look to be substantially worse, but the strong underforecasts are made for probabilities that are not frequently forecasted and thus do not impact the final score greatly.
The freezing rain and ice pellet forecast are very poor, strongly overforecasted. 
*Note that the overforecasting occurs despite the fact that most of the forecasts are already 0% probabilities—92% of all freezing rain forecast are for non-occurrence.  This is because the event is rarer still, with a climatological frequency of only 1% for freezing rain.
*Note also that no high-confidence forecasts are made for either freezing rain or ice pellets: no forecasts above 50% for ZR and 30% for IP.

## Eta Ensemble

- 0.574

- 0.096

- -0.502

- 0.523

**Notes:**
Performance of 5-member subset using just the Eta-model-based members.

Skill for the RN and S forecasts drops somewhat.  The rain forecasts suffer from an overforecast at moderate and high probabilities and the underforecast of snow is worse.

Note, however, that the ZR and IP forecast, while still bad have improved greatly and that higher probabilities are now employed.

## RSM Ensemble

- 0.581

- 0.122

- -0.281

- 0.604

**Notes:**
Subset using the 5 RSM-based members.

The drop in skill relative to the full ensemble for RN and S forecasts not as great as for Eta subset (snow underforecast not and rain overforecast not as great) and the ZR and IP forecast are better still—the ZR forecasts are even skillful.  Again higher probabilities are used for the ZR and IP forecasts, though still no high confidence forecasts of IP.

## Fixed-model/variable-algorithm--Eta

- 0.556

- 0.025

- -0.541

- 0.522

**Notes:**
5 algorithms/ 1 model: a representative Eta member—forecast resolution of only 20% now possible.

Only slightly worse than the Eta ensemble, though the RN and S look worse for the slightly used moderate probabilities.
IP forecasts of 80% now used.

## Fixed-model/variable-algorithm--RSM

- 0.527

- -0.036

- -0.366

- 0.556

**Notes:**
Same as before but with 1 RSM member

Bigger drop in skill relative to the ensemble subset than was seen for the Eta, but still not bad.

## Fixed-algorithm/variable-model

- 0.593:0.461

- 0.145:-0.095

- 0.007:-7.279

- 0.598-0.484

**Notes:**
All 10 members/ 1 algorithm—Brier Skill Scores for the best and worst performing algorithm plotted

Broad range of skillfulness, but no algorithm is better for all precipitation types.

*Note that Cortinas algorithm strongly overpredicts IP and underpredicts RN, suggesting that the algorithm is overdoing freezing or underdoing melting.  Since the snow is also underpredicted, the overproductive freezing is the more likely culprit.

## Full Ensemble

**Notes:**
ROC diagrams for the full ensemble

ROC measures the discrimination ability of the forecasts.  Whereas the attributes diagram is a forward-looking measure of performance, the ROC
is backward looking: given that an event occurs what was the forecast (POD), given that an event does not occur what is the forecast (POFD)?
A perfect forecast has an area beneath the curve equal to 1, ie, want points toward the upper-left corner of the graph.  An area > 0.7 is generally
considered good (roughly one standard deviated separation between the means of the event and non-event distributions)

RN, S forecasts are outstanding, but the IP and ZR forecast are quite good given their poor skillfulness and reliability.
ROC curves for the ensemble subsets are similar, with performance decreasing somewhat for the smaller ensembles.

## Spread Ratio

**Notes:**
Please read the “Y”s in the equations as union symbols and the “I”s as intersections, equation editors do not port well between systems.

The Spread Ratio measure the area of the union of a set of forecast fields relative to the area of the intersection of those forecasts.  The intersection can be defined as the intersection of all forecasts (dark blue area) or specified fractions of the total number of forecasts (medium blue area).  The SR can be unstable; it blows up when the forecast diverge completely and the area of intersection goes to zero.  Thus, the Correspondence Ratio (CR), which inverts the ratio can be used in its stead.  However, the SR fits better with the concept of spread, namely, the curve goes up as spread increases.

SR/CR is particularly well-suited for measuring the spread of categorical fields where traditional measures, such as RMS, do not work.  The SR/CR can be used as a measure of accuracy, too, by including the observed field, but that aspect is not examined here.

## Full Ensemble

- 100 %

- 80 %

- 50 %

**Notes:**
Spread Ratio for the full ensemble forecasts of RN and S as a function of forecast hour.  Intersections of 100, 80, and 50% of the ensemble members are used.

Ignoring the apparent diurnal cycle, the spread increases with time, but barely so for the 50% line—we generally don’t see complete diverge of ensemble members.

Very little spread or change in spread for the snow forecasts.  Snow forecasts are very confident, matching what was seen in the attributes diagrams, namely that the vast majority forecasts were for forecasts of 0 or 100%.  

- RN-rsmp1

- RN-etan2

- RN-alg1

- RN-alg5

**Notes:**
Spread for RN forecasts from two 1-model/5-algorithm subsets (top) and two 1-algorithm/10-model subsets (bottom).
Note that the vertical scale is larger for the bottom row (0-5 vs. 0-3 for the top).

Much more spread is achieved by holding the algorithm constant and varying the model than vice versa.  That is, changing the vertical profile produces greater differences than changing how a particular profile is handled.

- ZR-rsmn2

- ZR-etac0

- ZR-alg5

- ZR-alg1

- Correspondence Ratio

- Correspondence Ratio

- Correspondence Ratio

- Correspondence Ratio

**Notes:**
Same as previous but for ZR.  Also, switched to Correspondence Ratio since intersection goes to zero.

Noisy plot because of small sample size, but again holding the algorithm constant and changing the models gives more spread than the other way around.

## Conclusions

- Rain and snow forecasted very well
- ZR, IP forecasts have low reliability (overforecast), and low skill but still discriminate well and provide some value
- Skill of ZR, IP forecast actually decreases for the larger ensemble
- Forecast spread is much greater for IP, ZR than for R, SN
- Different models provide more spread than different algorithms

**Notes:**
On the second point, the value of the forecast (from the cost-loss formulation) was not discussed, but the numbers for both the maximum value and the range of users for whom the forecasts provide positive value behave similarly to the ROC scores, though the numbers are not quite as high and the decrease in performance for the smaller ensembles is more substantial.

The 3rd conclusion is rather discouraging and implies that the full ensemble does not do a good job of modelling the evolution of the observed probability density function (pdf).