# Presentation: 924015

## Gridded warning verification 

- Harold E. Brooks
- NOAA/National Severe Storms Laboratory
- Norman, Oklahoma
- Harold.Brooks@noaa.gov

## Basic verification

- Match observations and forecasts
- Clearly defined events
- Information to fill in 2x2 table

## 2x2 Table

- a = Correct fore. of events, d = Correct non-event
- b = False alarm, c = Missed event

|  | Ob. Yes | Ob. No | Sum |
| --- | --- | --- | --- |
| Fore.Yes | a | b | a+b |
| Fore. No | c | d | c+d |
| Sum | a+c | b+d | N |

## Scores

- Probability of detection=a/(a+c) (POD)
- Probability of false detection=b/(b+d) (POFD)
- False alarm ratio (rate) = b/(a+b) (FAR)
- Frequency of hits=a/(a+b) (FOH=1-FAR)
- Detection failure ratio=c/(c+d) (DFR)
- F-score=2*POD*(FOH)/(POD+FOH)
- Odds ratio=ad/bc
- Threat score (CSI)=a/(a+b+c)=f(POD, FAR)

## Skill scores (1)

- Equitable threat score
- ETS=(a-CH)/(a+b+c-CH)
- CH=(a+b)(a+c)/n2
- Extreme dependency score
- EDS=2{log([a+c]/N)/log(a/N)}-1
- Doesn’t go to zero as event becomes rare

## Skill scores (2)

- Peirce (Hanssen-Kuipers)
- (ad-bc)/[(a+c)(b+d)]=POD-POFD
- (Correct-CH)/(1-CHclim)
- Doolittle (Heidke)
- (ad-bc)/[(ad-bc)-(1/2)(b+c)]
- (Correct-CH)/(1-CH)
- Clayton
- (ad-bc)/[(a+b)(c+d)]=FOH-DFR

## Current status

- Warnings are issued for counties (or parts)
- Reports are points
- How can we make the 2x2 table?
- Correct forecast-either covers the report or report is within county
- False alarm-counties without reports
- Missed event-report without warning
- Correct no event?

## Quantities

- Probability of detection
- Events within warning areas/total events
- False alarm ratio
- Forecasts without events/total forecasts
- Other quantities can’t be calculated

## Current approach

- Calculate POD based on events
- Calculate FARatio based on areas
- Calculate CSI from POD and FARatio

## Problems

- Inconsistent definition, no information on d
- Provides little information on performance

## A vision

- Consistency between area and event definitions
- Consistency with other forecast products
- Allow for growth

## Gridding the events

- High resolution time/space grid
- O(1-5 km, 5-15 minutes)
- Grid boxes are either 0 or 1 for each location, time for all weather types
- Grid SPC products on the same grid

## Output

- Series of 0,1 values (could be probabilities)
- Outlooks
- Watch status
- Warning
- Reports
- Doswell and Keller (1993) did this for watches on an hourly time scale, MDR block

## 2x2 table

- If you can decide what d is, you can make 2x2 tables
- General thunder?
- Convective outlook?
- Some events may not be in table
- Massively large
- 4x4 km, 15 min-~500,000 locations, 35,000 times

## Issues

- Traditional scores would be really different
- Clean up data collection
- Could draw warning on grid, translate to county definitions
- Time domain
- If warning comes out in middle of time block, what to do?  (Block could be smaller or 0 or 1 for this purpose)

## Watch  Warning  Report  Black Box-County

**Watch  ****Warning  ****Report  ****Black Box-County**

## Current-1 Correct, 1 Missed Event, 1 FA

**Current-1 Correct, 1 Missed Event, 1 FA**
**POD=0.5, FAR=0.5**

## Grid-based-4 Correct, 1 Missed Event, 36 FA, 759 CN

**Grid-based-4 Correct, 1 Missed Event, 36 FA, 759 CN**
**POD=0.8, FAR=0.9, POFD=.0013 (Watch background)**

## Advantages

- Encourages data collection (could be probabilistic events or NCAR/RAP approach)
- Allows for baseline comparisons
- Are warnings better when watches are in effect?
- Stratify by time of day, location, etc.
- More informative scores can be derived