# Presentation: 000308

## Fuzzy Forecast VerificationGiving credit to close forecasts

- Beth Ebert
- Centre for Australian Weather and Climate Research
-  Melbourne, Australia

- Thanks to Felix Ament, Tatjana Bähler, Nigel Roberts, Barbara Casati, Uli Damrath, Fred Atger, Urs Germann, Mike Kay, Susanne Theis, Daniela Rezacova, Mike Baldwin

- NCEP 29 April 2008

## Outline

- Motivation
- What is fuzzy verification?
- Framework for fuzzy verification
- Demonstration – WRF high resolution rain forecast
- Comparison with entity-based (CRA) approach
- Spatial Verification Methods Intercomparison Project

- NCEP 29 April 2008

## Which rain forecast would you rather use?

- High vs. low resolution

- NCEP 29 April 2008

**Notes:**
Although high resolution forecasters usually look more realistic, and in fact may give more useful information in terms of intensities and even placement of significant weather, they rarely verify better against the observations than low resolution forecasts using standard scores.

## What makes a useful forecast?

- Resembles the observations on the broader scale
- Predicts an event somewhere near where it was observed
- Predicts the event over the same area (i.e., with the same frequency) as observed
- Has a similar distribution of intensities as the observations
- Looks like what a forecaster would have predicted if she'd had knowledge of the observations

- NCEP 29 April 2008

**Notes:**
Many criteria could be used to define a "good" forecast. Some may be more important than others for certain applications.

## "Fuzzy" verification methods

- Don't require an exact match between forecasts and observations

**Hi res forecast**
- RMS ~ 4.7
- POD=0, FAR=1
- TS=0

**Low res forecast**
- RMS ~ 2.7
- POD~1, FAR~0.7
- TS~0.3

- Reduces "double penalty" - event predicted where it did not occur, no event predicted where it did occur
- Rewards "close" forecasts
- Addresses uncertainties
- Unpredictable scales
- Uncertainty in observations

- NCEP 29 April 2008

**Notes:**
Impossibility of exact match of high resolution forecasts with observations means that requiring an exact match to get a good score no longer makes sense.

## Why "fuzzy"?

- observation

- forecast

- NCEP 29 April 2008

## "Fuzzy" verification methods

- Look in a space / time neighborhood around the point of interest
- Evaluate using categorical, continuous, probabilistic 	          scores / methods

- NCEP 29 April 2008

**Notes:**
Fuzzy methods give a neighborhood of points to verify instead of a single matched point. Since they are unconcerned about the precise location of the data inside the neighborhood the fuzzy approach acts like a filter, to filter out scales which are than the size of the neighborhood.

## "Fuzzy" verification methods

- First (?) suggested by H. Brooks at 1998 Mesoscale Verification workshop
- Brooks et al. (1998)
- Zepeda-Arce et al. (2000), Weygandt et al. (2004)
- Atger (2001)
- Damrath (2004)
- Casati et al. (2004)
- Germann and Zawadski (2004)
- Theis et al. (2005)
- Rezacova et al. (2007)
- Roberts and Lean (2007)

- NCEP 29 April 2008

**Notes:**
Many recently developed fuzzy verification methods, all with slightly different approaches and metrics. Each uses a different decision model to specify what makes a good forecast.

## A mini tour of some fuzzy verification methods...

- NCEP 29 April 2008

## Spatial verification 101

|  | Observed |  |
| --- | --- | --- |
|  | yes | no |
| yes | hits | false alarms |
| no | misses | correct negatives |

**Predicted**

*Contingency Table*

- NCEP 29 April 2008

## Upscaling(e.g. Weygandt et al. 2004)

- Average the forecast and observations to successively larger grid resolutions, then verify

**% change in ETS**

- NCEP 29 April 2008

**Notes:**
Upscaling measures the similarity between the forecasts and observations as the scale is increased. These results are for mesoscale model predictions of 6h precipitation over the Great Plains of the US during the IHOP experiment in June 2002.

## Spatial multi-event contingency table(Atger, 2001)

- Experienced forecasters interpret output from a high resolution deterministic forecast in a *probabilistic* way

-  "high probability of some heavy rain near Sydney", ***not***  "62 mm of rain will fall in Sydney"

- The deterministic forecast is mentally "calibrated" according to how "close" the forecast is to the place / time / magnitude of interest.
- Very close  high probability
- Not very close  low probability

- NCEP 29 April 2008

**Notes:**
This fuzzy method quantifies the way forecasters judge high resolution forecasts. 

## Spatial multi-event contingency table(Atger, 2001)

- Measures how well the forecast can
- separate events from non-events
- based on some decision threshold
- Decision thresholds to vary:
- magnitude (ex: 1 mm h-1 to 20 mm h-1)
- distance from point of interest (ex: within 10 km, .... , within 100 km)
- timing (ex: within 1 h, ... , within 12 h)
- anything else that may be important in interpreting the forecast

- Verify using the Relative Operating Characteristic (ROC)

- NCEP 29 April 2008

**Notes:**
In the figure here, decision thresholds based on both the rain threshold and the spatial distance from the point of interest are varied.

## Fractions Skill Score(Roberts and Lean, 2008)

- Verifies forecasts on a variety of spatial scales, with a view to determining which scales have sufficient accuracy.
- Compares the fractional coverage of events in windows surrounding the observations and forecasts

- Fraction of pixels with rain accumulation ≥ 4 mm in 1x1 km windows (top) and in 35x35 km windows (bottom)

- Pobs 35x35

- Pfcst 35x35

- Pobs  1x1

- Pfcst 1x1

- NCEP 29 April 2008

**Notes:**
"The spatial scales we are interested in, and the spatial scales we verify on, should be compatible. We judge forecasts by eye over a variety of scales by saying that they are good at getting a large feature like a front in the right place, but are not quite so accurate for organized thunderstorms and poorer still for individual showers." (N. Roberts NWP report, 2005).

## Fractions Skill Score(Roberts and Lean, 2008)

- Fractions Skill Score(Roberts and Lean, 2008)

- Fractions Brier score
- Fractions skill score

- NCEP 29 April 2008

**Notes:**
A range of scales at which the forecast is both useful and meets minimum accuracy requirements can be determined by plotting the score as a function of spatial scale.

## How we are using it

**10% threshold**

**FSS**

**L(FSS>0.5)**

**Freq. Bias**

- 12.5 mm

- 0.5

- 20 km

- Courtesy of Marion Mittermaier

- NCEP 29 April 2008

## Practically perfect hindcasts (Brooks et al. 1998)

- If the forecaster had all of the observations in advance, what would the  "practically perfect" forecast look like?
- Apply a smoothing function to the observations to get probability contours, choose appropriate yes/no threshold

- 	SPC convective outlook	Practically Perfect hindcast

- NCEP 29 April 2008

**Notes:**
Rare events are, by nature, difficult to predict accurately and difficult to observe accurately (for example, tornadoes). Forecasts often given as watch or warning regions. Standard verification doesn't work well. Most of warning region is false alarm, even when the event occurs in the region. Observations of non-events generally not reported. Persistence or climatology aren't really appropriate reference forecasts for skill scores.

## Practically perfect hindcasts (Brooks et al. 1998)

- How do the forecast and the Practically Perfect hindcast perform when verified against the observations, and how do the two scores compare?

- 	SPC convective outlook	Practically Perfect hindcast
- 	threat score=0.34	threat score=0.48

**Forecast was 3⁄4 of the way to being "practically perfect"**** **

- NCEP 29 April 2008

**Notes:**
Relative performance is quite a useful way to judge a verification score.

## Where is this all leading to?

- NCEP 29 April 2008

**Notes:**
These different approaches all use neighborhoods in some way

## Fuzzy verification framework

- Fuzzy methods use one of two approaches to compare forecasts and observations:

- single observation – neighborhood forecast
- (user-oriented)
- neighborhood observation – neighborhood forecast
- (model-oriented)

- NCEP 29 April 2008

**Notes:**
The single observation – neighborhood forecast is user-oriented (e.g. how good was the forecast at the place where I live?)
The neighborhood observation – neighborhood forecast is model-oriented (e.g. compares models and observations on consistent scales)

## Fuzzy verification framework

- Treatment of forecast data within a window:
- Mean value (upscaling)
- Occurrence of event* somewhere in window
- Frequency of event in window  probability
- Distribution of values within window

- * *Event* defined here as a value exceeding a given threshold, for example, rain exceeding 1 mm/hr

- NCEP 29 April 2008

## Example: Multi-category contingency table

- Compares occurrence of event in forecast with observed occurrence of event

- Hit = at least one forecast event in vicinity of observed event

- NCEP 29 April 2008

**Notes:**
There is more to this method, but the first part of it is shown here to illustrate one possible treatment of observed and forecast data in the window.

## Example: Fractions skill score

- Compares fractional coverage in forecast with fractional coverage in observations

- NCEP 29 April 2008

**Notes:**
There is more to this method, but the first part of it is shown here to illustrate one possible treatment of observed and forecast data in the window.

## Moving windows

- Accumulate scores as windows are moved through the domain

- NCEP 29 April 2008

## Decision models

- *NO-NF = neighborhood observation-neighborhood forecast,
-  SO-NF = single observation-neighborhood forecast

- NCEP 29 April 2008

**Notes:**
The various methods differ in:
(1) their matching strategy (SO-NF and NO-NF)
(2) their decision models for what makes a good forecast
The methods with the red asterisks are the methods demonstrated in this presentation.

## Multi-scale, multi-intensity approach

- Forecast performance depends on the scale and intensity of the event

- Intensity

- Spatial
- scale

- NCEP 29 April 2008

**Notes:**
Using plotting approach similar to that used by Barbara Casati. Red-pink colors indicate good performance, yellows and blues indicate poor performance.

## Case study – 13 May 2005

- 24 h forecast of 1 hr rainfall from WRF/ARW model @ 4 km resolution
- 	(Spring Program 2005 data courtesy of Mike Baldwin)

**ETS****≥0.01in****= 0.17 , ****ETS****≥0.10in****= 0.24**

- Stage II

- WRF ARW

- (0.01 in/hr)

- NCEP 29 April 2008

**Notes:**
Most people would say that this was a very good forecast, yet the ETS was only 0.17, and the maximum grid-scale value of ETS was 0.24 using a rain threshold of 0.10 in.

## Fuzzy verification framework

- NCEP 29 April 2008

**Notes:**
Using the framework all of the fuzzy methods can be run and displayed together. Two of the methods do not use thresholds but rather evaluate all intensities at once. Some things to note:
(1) Different fuzzy verification methods suggest different levels of skill.
(2) The intensities and spatial scales at which the maximum skill is obtained varies by method.
The reason for these differences lie in the way the scores are constructed, and the nature of the decision models for what makes a "useful" forecast. The next several slides examine the results from some of the fuzzy verification methods in more detail.
One other thing: Barbara Casati's (2004) intensity-scale method is included among the fuzzy techniques, but it is unique in two ways: It uses discrete rather than overlapping windows and therefore tends to be a bit noisier, and it *separates* individual scales rather than filtering them. In this sense it is not strictly fuzzy.

## Upscaling

- Decision model – Useful forecast resembles observations when averaged to coarser scales

**ETS**

- NCEP 29 April 2008

**Notes:**
Upscaling generally shows better forecast performance as the scales are averaged to coarser resolution. Not only categorical scores can be used here, but also continuous scores such as RMSE and correlation. Both hits (see POD) and false alarms (see FAR) improve with scale, leading to higher values of ETS.

## Multi-scale contingency table

- Decision model – Useful forecast predicts at least one event close to the observed event

**HK**

- NCEP 29 April 2008

**Notes:**
Plot Hanssen and Kuipers score (also known as True Skill Statistic) rather than ETS because that is related to the area under the ROC curve (the original way multi-scale contingency table results were presented). These results show that the relative skill at predicting at least one event close to the observed event is best for heavier rain rates and larger scales.

## Fractions Skill score

- Decision model – Useful forecast has similar frequency of fcst and obs events within a region

**FSS**

- NCEP 29 April 2008

**Notes:**
The Fractions skill score uses a probabilistic verification approach. 

## Practically Perfect hindcast

- Decision model – Useful forecast resembles one drawn from knowledge of observations

- ETSfcst/ ETSPP

- NCEP 29 April 2008

**Notes:**
The ratio of ETS for the forecast to ETS for the Practically Perfect hindcast is one way to summarize the relative performance of the forecast. However, since ETS can be improved by overforecasting when the rain pattern is offset from the observations (see Baldwin and Kain, WAF 2006) sometimes this ratio becomes greater than 1 at large scales. The solution to this is probably not to report ETS ratios at the large scales.

## Aggregate results for 9 SPC 2005 test cases

- NCEP 29 April 2008

**Notes:**
Like most verification, fuzzy verification is probably most useful when aggregated over a large number of cases – a month, a season, all similar synoptic situations in a dataset, etc. Careful aggregation and stratification of samples are really the keys to getting the most out of diagnostic verification.

## Advantages of fuzzy verification

- Knowing which scales have skill suggests the scales at which the forecast should be presented and trusted
- Can give good results for forecasts that verify poorly using exact-match approach
- Results match with our intuition when decision models are considered
- Suitable for discontinuous / messy fields like precipitation
- Can be used to compare forecasts at different resolutions
- Multiple decision models and metrics
- Direct approach  verification of intensities
- Categorical approach  verification of binary events
- Probabilistic approach  verification of event frequency

- NCEP 29 April 2008

## Weaknesses and limitations

- Less intuitive than object-based methods
- Imperfect scores for perfect forecasts for methods that match neighborhood forecasts to single observations
**Information overload** if all methods invoked at once
- Let appropriate decision model(s) guide the choice of method(s)
- Even for a single method ...
- there are lots of numbers to look at
- evaluation of scales and intensities with best performance depends on metric used (CSI, ETS, HK, etc.). Be sure the metric addresses the question of interest!

- NCEP 29 April 2008

## How do fuzzy verification methods compare with object-oriented verification methods?

- NCEP 29 April 2008

## Entity-based approach for verifying high resolution forecasts

- Define *entities* using a threshold
- Horizontally translate the forecast until a *pattern matching* criterion is met
- Displacement is the vector difference between the original and final locations of the forecast
- Compare properties of whole objects

- NCEP 29 April 2008

**Notes:**
Entity and object-oriented approaches have a lot of appeal because they are so intuitive.

## CRA verification of rain system

- (0.01 in)

- NCEP 29 April 2008

**Notes:**
This nice looking WRF forecast was verified using a CRA threshold of 5 mm, and the matching forecast and observed rain blobs were fairly clear. Only the largest CRA is shown here, although several others were verified.

## Comparison of verification strategies

- _Fuzzy verification_
- Handles messy fields and discrete objects
- Assessment across multiple scales
- Fairly intuitive
- Completely objective
- Can be extended to time domain
- Offers many decision models

- _Entity-based verification_
- Handles discrete objects only
- Holistic assessment on scale of object
- Very intuitive
- Some choice of parameters
- Can be extended to time domain
- Gives location error and total error decomposition

- NCEP 29 April 2008

## Spatial Verification Intercomparison Project

- Begun February 2007
- The main goals of this project are to:
- Obtain an inventory of the methods that are available and their capabilities
- Identify methods that may be useful in operational settings
- Identify methods that might be useful for providing automated feedback into forecasting systems
- Identify methods that are particularly useful for specific applications (e.g., model diagnostics, hydrology, aviation)
- Identify where there may be holes in our capabilities and more work is needed
- Identify areas where future research efforts should focus

- NCEP 29 April 2008

## Spatial Verification Intercomparison Project

- _[http://](http://www.ral.ucar.edu/projects/icp/index.html)__[www.ral.ucar.edu/projects/icp/index.html](http://www.ral.ucar.edu/projects/icp/index.html)_
- Test cases
- Preliminary results
- References
- Code!

- NCEP 29 April 2008

**Notes:**
Still time to join project if interested.

## Spatial Verification Intercomparison Project

**Object-oriented verification methods:**
- Method for Object-based Diagnostic Evaluation (MODE)
- Contiguous Rain Area (CRA)
- Procrustes shape analysis
- Composite approach (Nachamkin)
- Cluster analysis (Marzban)
- Structure Amplitude Location (SAL)
**Neighborhood and filtering methods:**
- Fuzzy (neighborhood) verification methods
- Fractions skill score (FSS)
- Intensity-scale
**Field methods:**
- Bias-corrected ETS (Mesinger)
- Forecast Quality Measure (FQM, Keil and Craig)
- Forecast Quality Index (FQI, Basu)

- NCEP 29 April 2008

## Spatial Verification Intercomparison Project

- What types of information does the method give?
- Is the method able to identify performance at separate scales?
- Does the method provide information on:
- Location errors?
- Intensity errors and distributions?
- Structure errors?
- Can the method identify timing errors?
- Is the method “diagnostic” regarding sources of model error?
- Does this approach have the ability to provide info about false alarms and misses?
- Are there any undesirable behaviors shown by the method?
- Could a forecast be hedged to artificially improve its score using this method?
- Does the method do anything that is counter-intuitive?
- Does the method have selectable parameters?
- Is the method strongly sensitive to selected parameters?

- NCEP 29 April 2008

## Spatial Verification Intercomparison Project

- Specific questions regarding implementation:
- Can the results be easily aggregated across cases?
- Is the method amenable to different user applications (hydrology, aviation, etc.)?
- Could the method be used in operational settings (e.g., computer time issues, data availability)?
- Could the method be used to give automated feedback (e.g., for calibration) into a forecasting system?
- Could the method be used for other types of variables (e.g., wind, turbulence)?
- Is the method sensitive to missing data?
- Is the method sensitive to the size of the domain?

- NCEP 29 April 2008

## Fuzzy verification paper published in latest Meteorological Applications

- Fuzzy verification paper published in latest *Meteorological Applications*
- Paper and code (IDL) available from
- _[http://www.bom.gov.au/bmrc/wefor/staff/eee/beth_ebert.htm](http://www.bom.gov.au/bmrc/wefor/staff/eee/beth_ebert.htm)__ _

- NCEP 29 April 2008

**Notes:**
Web site is my home page in BMRC - can get there from the Bureau of Meteorology's home page.

## References for original fuzzy methods

- Atger, F., 2001: Verification of intense precipitation forecasts from single models and ensemble prediction systems. *Nonlin. Proc. Geophys.*, **8**, 401-417.
- Brooks, H.E., M. Kay and J.A. Hart, 1998: Objective limits on forecasting skill of rare events. *19th Conf. Severe Local Storms, AMS*, 552-555.
- Casati, B., Ross, D.B. Stephenson, 2004: A new intensity-scale approach for the verification of spatial precipitation forecasts, *Met. App*., **11**, 141-154.
- Damrath, U., 2004: Verification against precipitation observations of a high density network – what did we learn? *Intl. Verification Methods Workshop, 15-17 September 2004, Montreal, Canada.*
- Ebert, E.E., 2002: Fuzzy verification: Giving partial credit to erroneous forecasts. *NCAR/FAA Verification Workshop: Making Verification More Meaningful, NCAR, Boulder, Colorado, 30 July - 1 August 2002.*
- Germann, U. and I. Zawadzki, 2004: Scale dependence of the predictability of precipitation from continental radar images. Part II: Probability forecasts. *J. Appl. Meteorol.*, **43**, 74-89.
- Rezacova, D., Z. Sokol and P. Pesice, 2007: A radar-based verification of precipitation forecast for local convective storms. *Atmos. Res.*, **83**, 211-224.
- Roberts, N.M. and H.W. Lean, 2008: Scale-selective verification of rainfall accumulations from high-resolution forecasts of convective events. *Mon. Wea. Rev.*.
- Theis, S.E., A. Hense and U. Damrath, 2005: Probabilistic precipitation forecasts from a deterministic model: a pragmatic approach. *Meteorol. Appl.*, **12**, 257-268.
- Weygandt, S.S., A.F. Loughe, S.G. Benjamin and J.L. Mahoney, 2004: Scale sensitivities in model precipitation skill scores during IHOP. *22nd Conf. Severe Local Storms, Amer. Met. Soc., 4-8 October 2004, Hyannis, MA*.
- Zepeda-Arce, J., E. Foufoula-Georgiou, and K.K. Droegemeier, 2000: Space-time rainfall organization and its role in validating quantitative precipitation forecasts. *J. Geophys. Res.*, **105** (D8), 10,129-10,146.

- NCEP 29 April 2008

**Notes:**
Many of these references are non-standard

## Threshold-dependent results

- NCEP 29 April 2008

**Notes:**
It would be possible to apply scale and intensity dependent verification to entity- and object-based verification. For example, the previous forecast was verified using a range of CRA thresholds, and the relative error components were plotted as a function of this threshold. In principle the CRAs could also be thresholded according to their size and the verification results plotted as intensity-scale quilts similar to the other fuzzy methods.