# Presentation: 662257

## New methods for evaluating rainfall forecasts from operational models for landfalling tropical cyclones

**Timothy Marchok**
- NOAA / GFDL
**Robert Rogers**
- NOAA / AOML / HRD
**Robert Tuleya**
- NOAA / NCEP / EMC / SAIC

**This project was funded by the Joint Hurricane Testbed (JHT)**

## Goals

- Develop a set of rainfall validation schemes specifically designed for TCs
- Produce model QPF error statistics for a set of historic U.S. landfalling storms.

## Outline

- Models & storms
- Development of TC QPF validation techniques
- 1998-2004 base sample vs. 2005 season
- Skill indices based on the new techniques
- Examples of applications

## Models included in this study

- GFDL

- Regional
- 1/2o, 1/6o

- (2-nest)
- 42 levels

- 2003 version

## U.S. Landfalling Cases for Model Evaluation:1998-2004 Base Sample

- Matthew

- 40

- Jeanne

- 105

- Ivan

- 110

- Gaston

- 65

- Frances

- 95

- Charley

- 125

- Bonnie

- 45

***2004***

- Kyle

- 35

- Hermine

- 35

- Isabel

- 90

- Isidore

- 55

- Irene

- 70

- Georges

- 90

- Henri

- 30

- Hanna

- 45

- Harvey

- 50

- Frances

- 45

- Grace

- 35

- Fay

- 50

- Gabrielle 60

- Floyd

- 90

- Earl

- 70

- Claudette

- 75

- Edouard

- 35

- Barry

- 60

- Helene

- 65

- Dennis

- 60

- Charley 40

- Lili

- 85

***2003***

***2002***

***2001***

***2000***

***1999***

***1998***

- Bill

- 50

- Bertha

- 35

- Allison

- 45

- Gordon 55

- Bret

- 100

- Bonnie

- 95

- Matthew

- 40

- Jeanne

- 105

- Ivan

- 110

- Gaston

- 65

- Frances

- 95

- Charley

- 125

- Bonnie

- 45

***2004***

- Kyle

- 35

- Hermine

- 35

- Isabel

- 90

- Isidore

- 55

- Irene

- 70

- Georges

- 90

- Henri

- 30

- Hanna

- 45

- Harvey

- 50

- Frances

- 45

- Grace

- 35

- Fay

- 50

- Gabrielle
- 60

- Floyd

- 90

- Earl

- 70

- Claudette

- 75

- Edouard

- 35

- Barry

- 60

- Helene

- 65

- Dennis

- 60

- Charley
- 40

- Lili

- 85

***2003***

***2002***

***2001***

***2000***

***1999***

***1998***

- Bill

- 50

- Bertha

- 35

- Allison

- 45

- Gordon
- 55

- Bret

- 100

- Bonnie

- 95

- U.S. Landfalling Cases for Model Evaluation:1998-2004 Base Sample

## U.S. Landfalling Cases for Model Evaluation:2005 Season

- Arlene
- Cindy
- Dennis
- Katrina (Florida)
- Katrina (Louisiana)

- Ophelia
- Rita
- Tammy
- Wilma

## Outline

- Models & storms
- Development of TC QPF validation techniques
- 1998-2004 base sample vs. 2005 season
- Skill indices based on new techniques
- Examples of applications

## Parameters describing skill of TC QPF forecasts

-  Rainfall volume
-  Extreme amounts
-  Sensitivity to track errors

## Rainfall patterns

- 1998-2004

- 2005

- Equitable threat score comparison

## Rainfall volume

- Stage IV (Observed)

- GFS

- Example:  Tropical Storm Cindy (2005)

## Rainfall volume

- 1998-2004

- 2005

- Comparison of rain volume bias by model

## Rainfall volume:  “Rain flux” and track-relative analyses

- _Rainfall volume_:  “Rain flux” and track-relative analyses

- Observed rain flux PDF for all 1998-2004 storms in selected bands surrounding best track

## Rain volume: Rain flux in select bands

- 0–100 km

- 300-400 km

- GFDL, NAM

- GFS, R-CLIPER

## Extreme amounts: Comparison of top 5% of rain flux

## Sensitivity to track error

- Lili Stage IV

- r increased from 0.36 (unshifted) to 0.85 (shifted)

- Example of grid-shifting of rain field

## Sensitivity to track error

- Impact of grid shift on pattern correlations

- 1998-2004

- 2005

## Outline

- Models & storms
- Development of TC QPF validation techniques
- 1998-2004 base sample vs. 2005 season
- Skill indices based on new techniques
- Examples of applications

## Matrix of TC QPF Skill Indices

|   | Dependency on Track Error |  |   | QPF attribute described |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Index | Dependent | Independent |   | Pattern  | Mean / Volume | Maximum | Impact  ofTrack Error |
| Large-Scale ETS | ● |   |   |  |   |   |   |
| Pattern Correlation | ● |   |   |  |   |   |   |
| Mean Rainfall Error Index |   | ● |   |  |  |   |   |
| Large-ScaleCDF Median Value |   | ● |   |   |  |  |   |
| Track-RelativeCDF Median Value |   | ● |   |   |  |  |   |
| Large-Scale CDF % in95th percentile |   | ● |   |   |   |  |   |
| Track-Relative CDF % in95th percentile |   | ● |   |   |   |  |   |
| Grid-Shifted Pattern Correlation |   | ● |   |   |   |   |  |
| Grid-Shifted ETS |   | ● |   |   |   |   |  |

## Skill Indices: Pattern Matching

- 1998-2004

- 2005

- GFS performs the best in both samples
- All models have skill relative to R-CLIPER

## Skill Indices: Volume

- 1998-2004

- 2005

- R-CLIPER significantly better in 2005 season
- GFS worse in 2005 due to over-forecast bias

## Skill Indices: Extreme Amounts

- 1998-2004

- 2005

- GFDL worse in 2005 due to core region over-forecast bias
- GFS performs best despite lowest resolution

## Skill Indices: Sensitivity to track error

- 1998-2004

- 2005

## Outline

- Models & storms
- Development of TC QPF validation techniques
- 1998-2004 base sample vs. 2005 season
- Skill indices based on new techniques
- Examples of applications

## Evaluating enhancements to R-CLIPER

- Equitable Threat Score

- Rain flux CDF

- Adding topography had more of an impact on QPF skill than adding effect of shear

## Evaluating upgrades to 2006 GFDL model

- Slight reduction in over-prediction of extreme (>10”) core amounts...

- ....but very little change in most metrics, including Rain Volume Bias

## Summary

- Validation techniques address unique TC rainfall attributes:
- Pattern matching
- Rainfall volume
- Extreme amounts
- Sensitivity to track error
- TC QPF skill indices for operational forecast evaluation
- Baseline 1998-2004 stats (GFS most skillful)
- Utility in model evaluation & development

## Slide 27

## Extra slides.....

## Rainfall volume

## Rain volume: GFDL & NAM rain flux in select bands

- 1998 - 2004

- 2005

- 0 – 100 km

- 300-400 km