# Presentation: 174882

## Marine Verification at WFO Key West

**Marine Verification at WFO Key West**
**Where to go and how to get there...**
**	****Andy Devanas, SOO  and**
**	****Matt Parke, Verification Focal Point**

## There are three kinds of lies: 

**lies, damned lies, and statistics.**** **
**	**_[Benjamin Disraeli](http://www.quotationspage.com/quotes/Benjamin_Disraeli/)__ (1804 - 1881)_

## X

- X – No Surface Data Available
- 	NO WAVE DATA AT ALL

**USF Buoy**
**Not Operational**

**Marine Area approx. **
**22000 square miles **
**Massachusetts – **
**10555 square miles**

## Slide 4

## Slide 5

## Key West Verification System Needs...

**Need to integrate verification into the forecast process**
**Need to provide meaningful, frequent and timely verification data to forecasters (  e.g. Percent Correct vs. ESS)**
**Need to identify forecast and guidance biases**
**Enhanced discrimination in critical categories**

## How To Get There...

***Use MVerify for rapid feedback to forecasters**
**	****-Local AWIPS application**
**	****-Provides most flexibility in stratification of desired parameters**
**	****	**** ****Minimum forecast period adjustable**
**	****	**** ****Can segregate by forecaster**
***Investigate various statistical measures/distributions which best track       forecast performance**
***Set reasonable goals of forecast performance.**
***Monitor forecast performance.**
**	****-Accountability**
***Use verification statistics in post-mortem evaluations as feedback to forecasters.**

## DRAWBACKS

- *NMVP is not timely.  Must wait until after end of month to access statistics.
- *NVMP does not allow defining of periods.
- Minimum period is one month.
- *Can’t define inter-month periods.
- *NMVP does not provide access to individual forecast-observation pairs.
- 	-Cannot do post-mortem analysis.
- 	-Do not know which matched pairs are producing statistics.
- 	-No Quality control of forecasts and observations.
- *Some measures (ESS) are heavily weighted towards rare climatological events.
- 	-ESS is volatile.
- 	-Critical missing matched pairs can have large impact on ESS.
- BENEFITS
- *Provides relatively easy and quick statistics spanning > 3 year dataset
- *Can stratify data with regards to forecast period, buoy (cman), and season

## Equitable Skill Score reward/penalty Characteristics (from NMVP)- 

- A relatively small reward for correctly forecasting common events
- A large reward for correctly forecasting rare events
- Less penalty is assigned to an incorrect forecast of a rare event than a similar size error of a common event. "Near hits" of rare event receive a modest reward.
- Uses a graduated reward/penalty system, whereby a large forecast error is penalized more than a small forecast error for a given category of events.

**What’s wrong with this picture??**

## MOST CRAFT

|  |
| --- |

| FORECAST |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| OBS |   | < 8 | 8 - 12 | 13 - 17 | 18 - 22 | 23 - 27 | 28 - 32 | > 32 | TOTAL |   |   |
|  | < 8 | 10 | 24 | 8 | 0 | 0 | 0 | 0 | 42 |   |   |
|  | 8 - 12 | 2 | 61 | 66 | 16 | 0 | 0 | 0 | 145 |   |   |
|  | 13 - 17 | 0 | 15 | 114 | 91 | 7 | 0 | 0 | 227 |   |   |
|  | 18 - 22 | 0 | 1 | 32 | 51 | 28 | 0 | 0 | 112 |   |   |
|  | 23 - 27 | 0 | 0 | 1 | 6 | 1 | 0 | 0 | 8 |   |   |
|  | 28 - 32 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |   |   |
|  | > 32 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |   |   |
|  | TOTAL | 12 | 101 | 221 | 164 | 36 | 0 | 0 | 534 |  |  |

- S
- C
- E
- C

- S
- C
- A

- REMAIN
- IN PORT

**ESS Does Not Account For Value of Forecast to Our Customers...**

## Before Matched Sets at NMVP Site

## This Forecast Was Never Made

| CONTINGENCY TABLE  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| FORECAST |  |  |  |  |  |  |  |  |  |  |  |
| OBS |   | < 8 | 8 - 12 | 13 - 17 | 18 - 22 | 23 - 27 | 28 - 32 | > 32 | TOTAL |   |   |
|  | < 8 | 22 | 46 | 7 | 4 | 0 | 0 | 0 | 79 |   |   |
|  | 8 - 12 | 27 | 116 | 59 | 14 | 0 | 0 | 0 | 216 |   |   |
|  | 13 - 17 | 0 | 39 | 134 | 47 | 2 | 1 | 0 | 223 |   |   |
|  | 18 - 22 | 0 | 0 | 13 | 45 | 7 | 0 | 0 | 65 |   |   |
|  | 23 - 27 | 0 | 0 | 0 | 8 | 0 | 0 | 0 | 8 |   |   |
|  | 28 - 32 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |   |   |
|  | > 32 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |   |   |
|  | TOTAL | 49 | 201 | 213 | 118 | 9 | 1 | 0 | 591 |   |   |
|  | Bias | 0.62 | 0.93 | 0.96 | 1.82 | 1.12 | -- | -- |   |   |   |

**This Forecast Was Never Made**
**Where Did This Number Come From?**

**This Will Severely Skew ESS Score. As Well As Other Statistical Measures.**

**EYW Forecasts: January 2004**

##  Percent Correct Definition From NOAA Verification Glossary 

| 
2x2 Contingency Table |  | Event Observed |  |
| --- | --- | --- | --- |
|  |  | YES | NO |
| 
Event Forecast | YES | A | B |
|  | NO |  C | D |

-  **Percent Correct Definition From NOAA Verification Glossary **
**PC= (A+D)/(A+B+C+D). **

## Definition: Percentage of the correct forecasts (hits) from the total number of events. Formula:                      

- Definition: Percentage of the correct forecasts (hits) from the total number of events. Formula:

- Percent Correct Definition From
- Marine Verification Program Glossary

**In Other Words: A/(A+B+C+D)**

## “gaming”

- “gaming”

## Some EYW verification results:

**Our forecast system had a hard time discriminating between light winds (< 8kts) and winds of 8 – 12 kts.**
**Our forecast system is slightly better discriminating in the 8 – 12 kt range, but still tends to overforecast more often than underforecast.**
**At 13 – 17 kts, our system is the least biased.**
**We underforecast (bias > 1) moderate to high wind events.**
**Forecasters tend to “game” light wind forecasts which do not have much value to customers.**

**Some EYW verification results:**

## Gridded Verification at EYW....

**Otherwise known as...**

**What are you *****Kidding***** me??**

## From Murphy 1993...

**“****Since Forecasters’ judgments necessarily contain an element of uncertainty, their forecasts must reflect this uncertainty accurately in order to satisfy the basic maxim of forecasting. In general, then, forecasts** **must be expressed in probabilistic terms.”**

## Melbourne’s ADAS

*Closer to grid verification*
- *Supported by relatively  dense surface data network over land
- *Buoy in Marine Zones

## LAPS analysis highly influenced by single data point.

**LAPS analysis highly influenced by single data point.**
**Winds in this region most likely NE 10-15 knots.**

## Again – note influence of single data point over huge region...

**Again – note influence of single data point over huge region...**

## We do not have ANY guidance to forecast wave height inside the reef at this resolution and detail, and little skill – therefore we do not attempt to forecast this region in our grids and zero out this region – but do include in our CWF.

- Wavewatch III is poor at best outside of the reef – and does not account for gulf stream interactions.

- AND NOT ONE WAVE HEIGHT DATA POINT!!
- So How Do You Verify Wave Height??
- Answer: You don’t.

## Marine Forecast

- 730 PM EST SUN MAR 14 2004
- SMALL CRAFT ADVISORY
**TONIGHT**: EAST WINDS AROUND 20 KNOTS. SEAS OUTSIDE THE REEF 5 TO 7 FEET...EXCEPT HIGHER IN THE GULF STREAM. SEAS INSIDE THE REEF 2 TO 3 FEET. NEARSHORE WATERS CHOPPY TO ROUGH. SCATTERED SHOWERS.
**MONDAY**: EAST WINDS 15 T0 20 KNOTS...DECREASING TO 10 TO 15 KNOTS BY NOON. SEAS OUTSIDE THE REEF 4 TO 7 FEET...EXCEPT HIGHER IN THE GULF STREAM. SEAS INSIDE THE REEF 2 TO 3 FEET. NEARSHORE WATERS CHOPPY. SCATTERED SHOWERS.
- MONDAY NIGHT: SOUTHEAST WINDS NEAR 15 KNOTS. SEAS OUTSIDE THE REEF 3 TO 5 FEET. SEAS INSIDE THE REEF 2 TO 3 FEET. NEARSHORE WATERS A MODERATE CHOP. ISOLATED SHOWERS.

## GRID VERIFICATION “THOUGHTS”...

***You must have relatively dense surface data network to support gridded verification field.***
***Perception that numerical models are the better performer are incorrect and not supported.***
***Need high resolution shallow water wave guidance.***

- Given the current state of model guidance and surface data available for the EYW marine area –
- - It is entirely possible that our forecasters will have to DEGRADE our forecasts to match verification fields...