# Presentation: 540623

## Verification of the Monte Carlo Tropical Cyclone Wind Speed Probabilities:

**A Joint Hurricane Testbed Project Update**
**John A. Knaff and Mark DeMaria**
** *****NOAA - NESDIS - Office of Research and Applications, Fort Collins, CO***
**Chris Lauer**
** *****NOAA - NWS – Tropical Prediction Center, Miami, FL***

## TPC Monte Carlo Wind Probabilities

- Hurricane Gordon

- Hurricane Helene

## Motivation for Verification

- Assess how good the current wind probability algorithm performs
- Assess future improvements to the wind probability algorithm
- Another way to track the performance of the deterministic forecasts

## How good are the Monte Carlo wind probabilities? 

**QUESTIONS**
*How does the average forecast magnitude compare to the average observed magnitude?*
*What is the magnitude of the probability forecast errors?*
*What is the relative skill of the probabilistic forecast over the deterministic forecasts, in terms of predicting whether or not an event* *occurred?*
*How well do the predicted probabilities of an event correspond to their observed frequencies?*
*What is the ability of the forecast to discriminate between events and non-events?*

**STATISTICS THAT ANSWER ...**
*Multiplicative Bias*
*(half) Brier Score*
*Brier Skill Score*
*Reliability Diagrams*
*Relative Operating Characteristics*

## 34-kt Probability Grids

- Hurricane Gordon

- Hurricane Helene

## 34-kt OFCL Forecast

- Hurricane Gordon

- Hurricane Helene

## 34-kt Best Track

- Hurricane Gordon

- Hurricane Helene

## How good are the Monte Carlo wind probabilities? 

**QUESTIONS**
*How does the average forecast magnitude compare to the average observed magnitude?*

**STATISTICS THAT ANSWER ...**
*Multiplicative Bias*

## Results: Multiplicative Biases

- Atlantic, July 18 – October 4, 2006;  105W – 1W, 1N – 60N

**Range:** minus infinity to infinity. **Perfect score:** 1

## How good are the Monte Carlo wind probabilities? 

**QUESTIONS**
*How does the average forecast magnitude compare to the average observed magnitude?*
*What is the magnitude of the probability forecast errors?*
*What is the relative skill of the probabilistic forecast over the deterministic forecasts, in terms of predicting whether or not an event* *occurred?*
*How well do the predicted probabilities of an event correspond to their observed frequencies?*
*What is the ability of the forecast to discriminate between events and non-events?*

**STATISTICS THAT ANSWER ...**
*Multiplicative Bias*
*(half) Brier Score*
*Brier Skill Score*
*Reliability Diagrams*
*Relative Operating Characteristics*

## Results: Brier Skill Score

**Range:** minus infinity to 1, 0 indicates no skill when compared to the reference forecast. **Perfect score:** 1.

- Atlantic, July 18 – October 4, 2006;  105W – 1W, 1N – 60N

## How good are the Monte Carlo wind probabilities? 

**QUESTIONS**
*How does the average forecast magnitude compare to the average observed magnitude?*
*What is the magnitude of the probability forecast errors?*
*What is the relative skill of the probabilistic forecast over the deterministic forecasts, in terms of predicting whether or not an event* *occurred?*
*How well do the predicted probabilities of an event correspond to their observed frequencies?*
*What is the ability of the forecast to discriminate between events and non-events?*

**STATISTICS THAT ANSWER ...**
*Multiplicative Bias*
*(half) Brier Score*
*Brier Skill Score*
*Reliability Diagrams*
*Relative Operating Characteristics*

## Reliability Diagrams

**Observed frequency of events**

**Bins of Forecast Probabilities **

## Results: Reliability Diagrams

- Atlantic, June 17 – October 4, 2006;  105W – 1W, 1N – 60N

## How good are the Monte Carlo wind probabilities? 

**QUESTIONS**
*How does the average forecast magnitude compare to the average observed magnitude?*
*What is the magnitude of the probability forecast errors?*
*What is the relative skill of the probabilistic forecast over the deterministic forecasts, in terms of predicting whether or not an event* *occurred?*
*How well do the predicted probabilities of an event correspond to their observed frequencies?*
*What is the ability of the forecast to discriminate between events and non-events?*

**STATISTICS THAT ANSWER ...**
*Multiplicative Bias*
*(half) Brier Score*
*Brier Skill Score*
*Reliability Diagrams*
*Relative Operating Characteristics*

## Results: ROC Diagram

- 2%

- 5%

- 10%

**No skill**

**skill**

## Results: ROC Skill Score

**Range:** -1 to 1, 0 indicates no skill. **Perfect score:** 1.

## Summary

- Based upon a preliminary verification of Atlantic Monte Carlo Wind Probabilities
- Probabilities have small negative biases
- Probabilities are more skillful than the deterministic (i.e. OFCL) in determining if an event will happen
- Predicted probabilities appear well related to observed frequencies
- The Probabilities are skillful in their ability to discriminate between “events” and “non-events”

## Next

- Complete the verification of the wind probabilities in the Atlantic, East Pacific, Central Pacific, and West Pacific
- Summarize results and identify potential problems with the Monte Carlo software
- Deliver the verification code to TPC

## Future Plans

- Begin our new project to use the model forecast spread [i.e., the Goerss Predicted Consensus Error (GPCE)]  to better estimate track error distributions used for the Monte Carlo integration
- Assist, where possible,  M. Mainelli (TPC) on the understanding the verification of the wind probabilities at the watch/warning break points

## Multiplicative Biases

***(Multiplicative) bias*** -
***Answers the question: ****How does the average forecast magnitude compare to the average observed magnitude?*
**Range:** minus infinity to infinity. **Perfect score:** 1.

## Brier Score

***(half) Brier score*** -
***Answers the question:**** What is the magnitude of the probability forecast errors?*
**Range:** 0 to 1.  **Perfect score:** 0.

## Brier Skill Score

***Brier skill score*** -
***Answers the question:**** What is the relative skill of the probabilistic forecast over that of climatology, in terms of predicting whether or not an event* *occurred?*
**Range:** minus infinity to 1, 0 indicates no skill when compared to the reference forecast. **Perfect score:** 1.

## Statistical Methods:  Relative Operating Characteristics

|  | Forecasts (contingent on the forecast probability) |  |  |
| --- | --- | --- | --- |
| Observation | Warning (W) | No Warning (W’) | Total |
| Event (E) | h | m | E |
| Nonevent (E’) | f | c | E’ |
| Total | w | w’ | N |

- A series of 2x2 contingency tables, which are conditional on a range of forecast probabilities are constructed.
- For instance a warning would be issued if the probability exceeded 1,2,3,4,...100 %

- The results of the contingency tables can be quantified in terms of
- hit rate (***hr***) = *h/(h+m)*  and false-alarm rate (***far***)=*f/(f+c)*
- A plot of far vs. hr can be created and a skill score created from the area under the curve.

## Statistical Methods:  ROC diagram & Skill Score

- ,where* A* is the area under the curve

- Mason and Graham (1999)

**skill**

**No skill**

*ROC Skill Score*