# Presentation: 144578

## Development of an object-oriented verification technique for QPF

- Michael Baldwin1
- Matthew Wandishin2, S. Lakshmivarahan3
- 1 Cooperative Institute for Mesoscale Meteorological Studies,
- University of Oklahoma
- 2 Institute for Atmospheric Physics, University of Arizona
- 3 School of Computer Science, University of Oklahoma

**Notes:**
This talk includes material from Baldwin’s in progress PhD dissertation and some material branching off from that work.
Please note that the terms object, feature, and event are used interchangeably in this talk.

## Traditional verification

- Compare a collection of matching pairs of forecast and observed values at the same set of points in space/time
- One “score” might end up representing the accuracy of millions of points, thousands of cases, hundreds of meteorological events
- Boiling down that much information into a couple of numbers is not very meaningful

**Notes:**
Note on the 3rd bulleted item: we want a balance between simplicity and throwing out important information.

A workshop was held in Boulder, CO in summer of 2002 under the title Making Verification More Meaningful.  One point of agreement was that it is difficult to beat the image processing ability of the human eye in terms of matching two fields (eg, one forecast and one observed).  How can one get quantitative measures to do the same?

## Forecast #1: smooth

- OBSERVED

- FCST #1: smooth

- FCST #2: detailed

- OBSERVED

**Notes:**
For example, here is an idealized situation in which we have two rainfall forecasts: one highly smoothed and one detailed.  Visually it seems apparent that despite getting some of the details wrong (ie, too many storm cells and incorrect alignment) FCST #2 is the better forecast.  The orientation of FCST #1 is off by about 90 degrees and the area is too large.
However, the measures that are frequently used in forecast verification tell a different story.

## “Measures-oriented” (Brooks and Doswell, 1996) approach to verifying these forecasts

| Verification Measure | Forecast #1 (smooth) | Forecast #2 (detailed) |
| --- | --- | --- |
| Mean absolute error | 0.157 | 0.159 |
| RMS error  | 0.254 | 0.309 |
| Bias | 0.98 | 0.98 |
| Threat score (>0.45) | 0.214 | 0.161 |
| Equitable 
threat score (>0.45) | 0.170 | 0.102 |

**Notes:**
The smooth forecast actually scores as well or better according to each of the measures shown here (the biases of the two forecast fields was constrained to be the same so as to avoid the influence of different biases on other measures such as the ETS).  Pairwise verification punishes detailed forecasts that are shifted to any degree in time or space.  In fact, many of the measures punish these detailed forecasts twice.

## Characterize the forecast and observed fields

- Verify the forecast with a similar approach that a human forecaster would use to **visualize** the forecast/observed fields
- Characterize features, phenomena, events, etc. found in forecast and observed fields by assigning *attributes* to each *object*

**Notes:**
We want to get away from the pairwise constraint.  Visually, we do not compare point-to-point, but identify certain features and compare the rendering of those features in bulk.  Are they the same size, shape, magnitude, in the same location, etc?

## Object-oriented approach to verification

- Decompose fields into sets of *objects* that can be objectively identified and described by *attributes*
- Use image processing and data mining techniques to *locate* and *classify* events
- Produce scores based upon the *similarity/dissimilarity* between forecast and observed objects
- Analyze **joint distribution** of forecast and observed objects
- Similar to Neilley (1993)

**Notes:**
The attributes can and should be looked at individually to determine the specifics of the performance of a given forecast system, but comparing multiply-dimensioned values is not an intuitive process and so the attribute vectors and can be used to produce similarity or dissimilarity scores that can be easily compared.

## Possible scores produced by this approach

- f = (af, bf, cf, ..., xf, yf )
- o = (ao, bo, co, ..., xo, yo)
- score = function( f , o )
- d ( f , o ) = ( f - o )t A ( f - o )
-    Generalized Euclidean distance, measure of *dissimilarity*
- A is a weight matrix, different attributes would probably have different weights
- c ( f , o ) = cov ( f , o )
-    Covariance, measure of *similarity*

- f

- o

- d

**Notes:**
The concept of similarity and dissimilarity measures is well-developed in the field of data mining (aka multivariate or spatial statistics).

## Characterization: How?

- Locate an event
- 	Could use image processing edge detection routines

**Notes:**
By setting the edge detection threshold to different levels one can essentially look at different scales within the field.  For example, using a low threshold we can identify the entire system (eg, an MCS) and then by raising the threshold identify objects within the systems (eg, the individual storm cells).

## Characterization: How?

- Assign attributes
- 	Examples: location, mean, variance, structure

**Event #16**
- x=37.3,
- y=87.8,
- =2.8

**Notes:**
Once objects have been identified they can be described by the various attributes it possesses including but not limited to those listed in the slide.

## Multiscale statistical properties (Harris et al 2001)

- Fourier power spectrum
- Generalized structure function: spatial correlation
- Moment-scale analysis: intermittency of a field, sparseness of sharp intensities
- Looking for “power law”, much like in atmospheric turbulence (–5/3 slope)

- FIG. 3. Isotropic spatial Fourier power spectral density (PSD) for forecast RLW (qr; dotted line) and radar-observed qr (solid line). Comparison of the spectra shows reasonable agreement at scales larger than 15 km. For scales smaller than 15 km, the forecast shows a rapid falloff in variability in comparison with the radar. The estimated spectral slope with fit uncertainty is  = 3.0 ± 0.1

**Notes:**
Due to time constraints, we will focus on Fourier spectra in this talk as one possible attribute used to describe an object or an entire field, but the structure function and moment-scale (or singular) analyses can give a more detailed and less ambiguous picture of the structure of a given field.  
Geophysical fields tend to possess power law relationships.  That is, when plotted on a log-log graph, the Fourier power spectrum will be linear over a certain range of scales.  The slope of this linear region is a measure of the smoothness (and degree of stationarity) of the field.

## Fourier power spectra

- Compare 3h accumulated QPF to radar/gage analyses
- Forecasts were linearly interpolated to same 4km grid as “Stage IV” analysis
- Errico (1985) Fourier analysis code used.  2-d Fourier transform converted to 1-d by annular average
- Fixed grid used for analysis located away from complex terrain of Western U.S.
- Want to focus on features generated by model physics and dynamics, free from influence of orographically forced circulations

**Notes:**
The Fourier power spectra will give evidence as to how well a model can replicate the structure seen in the observed rain fields.  It does not care whether the rain is in the right place but rather whether the model is producing similar features.
The forecasts in the next slide are from one case this spring and are 3 hr accumulations covering the 3-6 hr forecast period.  The analyses were performed over the entire region shown in the plots.

## Example

- log[E(k)]

- log[wavenumber]

- Obs_4

- Eta_12

- Eta_8

- WRF_22

- WRF_10

- KF_22

**Notes:**
Forecasts from 5 models are plotted along with the corresponding observed field.  The power spectra are plotted in the upper left: Observed (cyan), 12 km Eta (blue), 8 km Non-hydrostatic Eta (yellow), 22 km WRF (green), 10 km WRF (purple), and the 22 km Eta-KF (red).
In general, the models all got too excited about the rain in NW Oklahoma and underforecast the line in eastern KS, but on the large scale the models are in general agreement.  However, focusing on the structure, the Eta models are significantly smoother than the KF and WRFs and are nowhere close to capturing the structure in the observed field.
This analysis is reflected in the power spectra.  The 10 km WRF matches the power of the observations down to about 40 km, or about 4 delta_x.  The 22 km WRF and KF match the obs down to about 100 km , or 5 delta_x.  The Eta models begin to fall off around 300 km.

- June 2002 00z runs 12, 24, 36, 48h fcsts

**Notes:**
The same analysis was performed over the month of June, 2002 on 3 hr accumulations for 12, 24, 36, and 48 hr forecasts from 00z runs of the models shown previously (excluding the 8 km Eta).  The performance of the WRFs and KF on the single case was apparently exceptional as the average power spectra do not shadow the observed spectrum as closely.  Still, the fall off is not as sharp as for the Eta, indicating the the rainfall fields from these models, while too smooth at medium to small scales, are more detailed than Eta models.

## Summary

- Developing an “object-oriented” verification approach by characterizing forecasts and observations
- Examining use of spatial structure and variability as potential attributes
- Provides information on realism of forecasts that traditional QPF verification measures do not
- Working with forecasters/users to determine useful attributes for characterizing events

**Notes:**
The statistics examined in this talk would not be immediately useful to forecast users other than in showing what types of fields one can expect to see from the different models.  Yet, they could provide insights to model developers who need to know not just that their model produces incorrect forecast, but in what way the forecasts go wrong.  Also, the object-oriented approach using other attributes could provide great value to forecast users.  In fact, the specific attributes used could be tailored to the specific interests of different user groups.
By looking at the statistics describing the spatial structure of forecast fields, we are examining whether the models are type of rainfall as is observed.  In other words, we are answering the question, “Does the model climatology match the observed climatology?”  If a model cannot even produce the right kind of storms we cannot expect to forecast rainfall in the right place at the right time.