# Presentation: 915619

## Development and Calibration of Ensemble Based Hazardous Weather Products at the Storm Prediction Center

**David Bright**
**Gregg Grosshans, Jack Kain, Jason Levit, Russ Schneider, Dave Stensrud, Matt Wandishin, Steve Weiss**
*October 11, 2005*
*NCEP Predictability Discussion Group*

*Where Americas Climate and Weather Services Begin*

**Notes:**
Mention western roots to get going.  From Portland, OR ... graduate of OSU; UW; and UA.  NWS employee at Salt Lake City, UT and Tucson, AZ for a number of years, so a pleasure to speak to the Western Gov. Assoc.

## STORM  PREDICTION CENTER 

**MISSION STATEMENT**
-  **The Storm Prediction Center (SPC) exists **
**solely to protect life and property of the American people**
**through the issuance of timely, accurate watch and forecast products **
**dealing with tornadoes, wildfires and other hazardous mesoscale weather phenomena**.

**MISSION STATEMENT**
-  **The Storm Prediction Center (SPC) exists **
**solely to protect life and property of the American people**
**through the issuance of timely, accurate watch and forecast products dealing with hazardous mesoscale weather phenomena**.

## Hail, Wind, Tornadoes
Excessive rainfall
Fire weather 
Winter weather

- Hail, Wind, Tornadoes
- Excessive rainfall
- Fire weather
- Winter weather

**STORM  PREDICTION CENTER**

- HAZARDOUS PHENOMENA

## TORNADO & SEVERE THUNDERSTORM WATCHES
 WATCH STATUS MESSAGE
 CONVECTIVE OUTLOOK 
Day 1; Day 2; Day 3; Days 4-8
 MESOSCALE DISCUSSION
Severe Thunderstorm Potential/Outlook Upgrade
Thunderstorms not expected to become severe
Hazardous Winter Weather
Heavy Rainfall
 FIRE WEATHER OUTLOOK
Day 1; Day 2; Days 3-8
OPERATIONAL FORECASTS ARE BOTH DETERMINISTIC AND PROBABILISTIC

** ****WATCH STATUS MESSAGE**
** ****CONVECTIVE OUTLOOK **
**Day 1; Day 2; Day 3; Days 4-8**
** ****MESOSCALE DISCUSSION**
**Severe Thunderstorm Potential/Outlook Upgrade**
**Thunderstorms not expected to become severe**
**Hazardous Winter Weather**
**Heavy Rainfall**
** ****FIRE WEATHER OUTLOOK**
**Day 1; Day 2; Days 3-8**
**OPERATIONAL FORECASTS ARE BOTH DETERMINISTIC AND PROBABILISTIC**

- SPC Forecast Products

***75% of all SPC products are ***
***valid for ***_***<***_*** 24h period***

## EXPERIMENTAL WATCH PROBABILITIES

|  Tornadoes |  |
| --- | --- |
|  Probability of 2 or more tornadoes | Low (10%) |
|  Probability of 1 or more strong (F2-F4) tornadoes | Low (<5%) |
|  Wind |  |
|  Probability of 10 or more severe wind events | Mod (60%) |
|  Probability of 1 or more wind event > 65 knots | Low (10%) |
|  Hail |  |
|  Probability of 10 or more severe hail events | Low (10%) |
|  Probability of 1 or more hailstones >2 inches	 | Low (<5%) |
|  Combined Severe Hail/Wind |  |
|  Probability of 6 or more combined severe wind/hail events | Mod (60%) |

- Severe Thunderstorm Watch 688 Probability Table

**Notes:**
This shows both the old and the new Watch, they are similar BUT different.  From a service aspect, in the past we constantly got e-mails and letters from people saying that they could not figure out if they were in a watch without looking at a map. Hopefully this will solve that problem, of course it will require that people know what county they are in. Perhaps our TV colleagues, like Tom, will have to translate watches into interstate highway information 

## CONVECTIVE OUTLOOKSOperational through Day 3

**Notes:**
A second common sense rule is that the strength of a building must be considered before using it for shelter do not go into something simply because its there.

## Thunderstorm Outlooks:

**24h General Thunderstorm**

**12h Enhanced Thunderstorm (Tonight)**

**12h Enhanced Thunderstorm (Today)**

- 24h Period (_>_ 10%)

- 12h Periods (_>_ 10%; 40%; 70%)

## Operational emphasis on...
Observational data
Short-term, high-resolution NWP guidance
Specific information predicting hazardous mesoscale phenomena
NWP needs range from the very-short range to medium range
Very short-range: Hourly RUC; 4.5 km WRF-NMM
Short-range: NAM, GFS, SREF
Medium-range: GFS, ECMWF, MREF
Today’s focus: SREF
Overview of the ensemble product suite
Specific ensemble calibrated guidance

- Observational data
- Short-term, high-resolution NWP guidance
- Specific information predicting hazardous mesoscale phenomena
- NWP needs range from the very-short range to medium range
- Very short-range: Hourly RUC; 4.5 km WRF-NMM
- Short-range: NAM, GFS, SREF
- Medium-range: GFS, ECMWF, MREF
- Today’s focus: SREF
- Overview of the ensemble product suite
- Specific ensemble calibrated guidance

**Product Guidance at the SPC**

## Objective: Provide a wide range of ensemble guidance covering all of the SPC program areas

- _Objective_: Provide a wide range of ensemble guidance covering all of the SPC program areas

## Sample of Ensemble Products Available...

**Sample of Ensemble Products Available...**

- http://www.spc.noaa.gov/exper/sref/

**MEAN & SD: 500 mb HGHT**

**MEAN: PMSL, DZ, 10M WIND**

**MEAN: MUCAPE, 0-6 SHR, 0-3 HLCY**

**SPAGHETTI: SFC LOW**

## http://www.spc.noaa.gov/exper/sref/

**PROB: DENDRITIC GROWTH**

*Omega *_*<*_* -3*
*-11 *_*<*_* T *_*<*_* -17*
*RH *_*>*_* 80%*

**PROB: SIG TOR PARAM **_**>**_** 3**

**MEDIAN, UNION, INTERSECTION:**
**SIG TOR PARAM**

**MAX OR MIN: MAX FOSBERG **
**                         ****INDEX**

**STP = *****F***** (mlCAPE, mlLCL, SRH, Shear)**
**Thompson et al. (2003)**

**STP = *****F***** (mlCAPE, mlLCL, SRH, Shear)**
**Thompson et al. (2003)**

**Sample of Ensemble Products Available...**

## F63 SREF POSTAGE STAMP VIEW: PMSL, HURRICANE FRANCES 

**Red = **
**EtaBMJ**
**Yellow= EtaKF**
**Blue = RSM**
**White = OpEta**

**SREF Member **

## Combined Probability 

**Probability surface CAPE >= 1000 J/kg**
**Relatively low  **
**Ensemble mean is        < 1000 J/kg (no gold dashed line)**

**CAPE (J/kg)**
**Green solid= Percent Members >= 1000 J/kg****; ****Shading >= 50%**
**Gold dashed = Ensemble mean (1000 J/kg)**
**F036: Valid 21 UTC 28 May 2003**

**Notes:**
Low CAPE event....not a high probability for more than 1000 joules of cape.

## Probability deep layer shear >= 30 kts
Strong mid level jet through Iowa

**Strong mid level jet through Iowa**

**10 m – 6 km Shear (kts)**
**Green solid= Percent Members >= 30 kts****; ****Shading >= 50%**
**Gold dashed = Ensemble mean (30 kts)**
**F036: Valid 21 UTC 28 May 2003**

*Combined Probability *

**Notes:**
Largely shows the position of mid level jet, which we saw earlier.  Remember, the mean was over IA, with most of the spread over cntrl IL and srn IN.  Here, we see prob of > 30 kts (min svr threshold) is > 50% over this area as well.

## Convection is likely WI/IL/IN
Will the convection become severe?

**Will the convection become severe?**

**3 Hour Convective Precipitation >= 0.01 (in)**
**Green solid= Percent Members >= 0.01 in;**** ****Shading >= 50%**
**Gold dashed = Ensemble mean (0.01 in)**
**F036: Valid 21 UTC 28 May 2003**

*Combined Probability *

## Combined Probability 

**                                                                                           ****F036: Valid 21 UTC 28 May 2003**

*Combined Probability *

**A quick way to determine juxtaposition of key parameters**
**Fosters an ingredients-based approach**
**Not a “true” probability**
**Dependence**
**Different members contribute**

**Notes:**
not a true probability...parameters are not independent.  But, perhaps the lack of internal (to the model) consistency is a good way to get around model biases (if the ensemble size is large enough).

## Combined Probability 

**Severe Reports**
- Red=Tor; Blue=Wind; Green=Hail

**Prob Cape >= 1000    X    Prob Shear >= 30 kts    X    Prob Conv Pcpn >= .01”**
**                                                                                           ****F036: Valid 21 UTC 28 May 2003**

*Combined Probability *

**A quick way to determine juxtaposition of key parameters**
**Fosters an ingredients-based approach**
**Not a “true” probability**
**Dependence**
**Different members contribute**

**Notes:**
Interesting concept...useful of further “statistical” investigation.  I.e., just what do we have here?  It’s not a true probability...parameters are not independent.  But, perhaps the lack of internal (to the model) consistency is a good way to get around model biases (if the ensemble size is large enough).

We’ve been testing different ways on combining the probabilities.  Basically, rescaling the data following the multiplication or internal dependence.  Jury is still out on the best way to go.  Note that the more parameters that are combined, it’s likely that the internally dependent formulations may be higher (this is at first counterintuitive, but is based on the fact that the probs decrease extremely fast when you start multiplying them together).

## Combined Probability

**F15 SREF PROBABILITY**
**TPCP x RH x WIND x TMPF**
**(< .01” x < 10% x  > 30 mph x > 60 F)**

- Ingredients for *extreme* fire weather
- conditions over the Great Basin

*Combined Probability*

## Objective: Develop calibrated probabilistic guidance for CG lightning

*Calibrated Thunderstorm Guidance*

## Combine Lightning Ingredients into a Single Parameter

- Three first-order ingredients (readily available from NWP models):
- Lifting condensation level _>_ -10o C
- Sufficient CAPE in the 0o to -20o C layer
- Equilibrium level temperature _<_ -20o C
***Cloud Physics Thunder Parameter (CPTP)***
-         CPTP = _(-19__o__C – T__el__)(CAPE__-20__ – K)_
-                                         K
- 	where K = 100 Jkg-1 and CAPE-20 is MUCAPE in the
-    0o C to -20o C layer

## Example CPTP: One Member

**18h Eta Forecast Valid 03 UTC 4 June 2003**

- Plan view chart showing where grid point soundings support
- lightning (given a convective updraft)

## SREF Probability CPTP > 1

- 15h Forecast Ending: 00 UTC 01 Sept 2004
**Uncalibrated probability: Solid/Filled; Mean CPTP = 1 (Thick dashed)**

**3 hr valid period: 21 UTC 31 Aug to 00 UTC 01 Sept 2004**

## SREF Probability Precip > .01”

*SREF Probability Precip *_*>*_* .01”*

- 15h Forecast Ending: 00 UTC 01 Sept 2004
**Uncalibrated probability: Solid/Filled; Mean precip = 0.01” (Thick dashed)**

**3 hr valid period: 21 UTC 31 Aug to 00 UTC 01 Sept 2004**

## Joint Probability (Assume Independent)

- 15h Forecast Ending: 00 UTC 01 Sept 2004
**Uncalibrated probability: Solid/Filled**

**P(CPTP **_**>**_** 1) x P(Precip **_**>**_** .01”)**
**3 hr valid period: 21 UTC 31 Aug to 00 UTC 01 Sept 2004**

## Uncalibrated Reliability 

- Perfect Forecast

- No Skill

- Climatology

**P(CPTP > 1) x P(P03I > .01”)**

*Uncalibrated Reliability*** **
**(5 Aug to 5 Nov 2004)**

***Frequency***
***[0%, 5%, ..., 100%]***

## Adjusting Probabilities

- Calibrate ensemble thunderstorm guidance based on the observed frequency of occurrence

## Ensemble Thunder Calibration

- Bin separately P(CPTP _>_ 1) and P(P03M _>_ 0.01”) into 11 bins (0-5%; 5-15%; ...; 85-95%; 95-100%)
- Combine the two binned probabilistic forecasts into one of 121 possible combinations (0%,0%); (0%,10%); ... (100%,100%)
- Use NLDN CG data over the previous 366 days to calculate the frequency of occurrence of CG strikes for each of the 121 binned combinations
- Construct for each grid point using 1/r weighting
- Bin ensemble forecasts as described in steps 1 and 2 and assign the observed CG frequency (step 3) as the calibrated probability of a CG strike
- Calibration is performed for each forecast cycle (09 and 21 UTC) and each forecast hour; domain is entire U.S. on 40 km grid (CG strike within ~12 miles)

## Before Calibration

## Joint Probability (Assumed Independence)

**P(CPTP **_**>**_** 1) x P(Precip **_**>**_** .01”)**
**3 hr valid period: 21 UTC 31 Aug to 00 UTC 01 Sept 2004**

- 15h Forecast Ending: 00 UTC 01 Sept 2004
**Uncorrected probability: Solid/Filled**

## After Calibration

## Calibrated Ensemble Thunder Probability 

*Calibrated Ensemble Thunder Probability *

- 15h Forecast Ending: 00 UTC 01 Sept 2004
**Calibrated probability: Solid/Filled**

**3 hr valid period: 21 UTC 31 Aug to 00 UTC 01 Sept 2004**

## Calibrated Ensemble Thunder Probability 

*Calibrated Ensemble Thunder Probability*

- 15h Forecast Ending: 00 UTC 01 Sept 2004
**Calibrated probability: Solid/Filled; NLDN CG Strikes (Yellow +)**

**3 hr valid period: 21 UTC 31 Aug to 00 UTC 01 Sept 2004**

## Calibrated Reliability 

- No Skill

- Perfect Forecast

- No Skill

*Calibrated Reliability*** **
**(5 Aug to 5 Nov 2004)**

**Calibrated Thunder Probability**

- Climatology

***Frequency***
***[0%, 5%, ..., 100%]***

## 3h probability of > 1 CG lightning strike within ~12 mi

**3h probability of **_**>**_** 1 *****CG lightning strike***** within ~12 mi**
**09Z and 21Z SREF valid at F003 through F063 **
**May 15 – Sept 15 2005**

- Economic Potential Value

- Reliability

## 12h probability of > 1 CG lightning strike within ~12 mi

**09Z SREF valid at F012 through F063 **
**May 15 – Sept 15 2005**

- Economic Potential Value

- Reliability

## Objective: Develop calibrated probabilistic guidance of the occurrence of severe convective weather(Available for 3h, 12h, and 24h periods; calibration not described today)

*Calibrated Severe Thunderstorm Guidance*

## SVR WX ACTIVITY

**24h probability of **_**>**_** 1 *****severe thunderstorm***** within ~25 mi**
- SREF: 2005051109
- Valid 12 UTC May 11, 2005 to 12 UTC May 12, 2005

**SVR WX ACTIVITY**
**12Z 11 May to 12Z 12 May, 2005**
**a= Hail**
**w=Wind**
**t=Tornado**

## 24h probability of > 1 severe thunderstorm within ~25 mi

**Hail **_**>**_** .75”**

**Wind **_**>**_** 50 kts**

**Tornado**

**24h probability of **_**>**_** 1 *****severe thunderstorm***** within ~25 mi**
**21Z SREF valid at F039 through F039 (i.e., Day 1 Outlook)**
**May 15 – Sept 15 2005**

- Economic Potential Value

- Reliability

## Objective: Develop calibrated probabilistic guidance of snow accumulation on road surfaces

- _Objective_: Develop calibrated probabilistic guidance of snow accumulation on road surfaces

*Experimental Calibrated Snow Accumulation Guidance*

## Use frequency of occurrence technique -- similar to the calibrated probability of CG lightning
Produce 8 calibrated joint probability tables 
Take power mean (RMS average) of all 8 tables for the 3h probability of snow accumulating on roads in the grid cell
Calibration period is Oct. 1, 2004 through Apr. 30, 2005
MADIS “road-state” sensor information is truth (SREF is interpolated to MADIS road sensor)

- Produce 8 calibrated joint probability tables
- Take power mean (RMS average) of all 8 tables for the 3h probability of snow accumulating on roads in the grid cell
- Calibration period is Oct. 1, 2004 through Apr. 30, 2005
- MADIS “road-state” sensor information is truth (SREF is interpolated to MADIS road sensor)

*Ensemble Snow Calibration*

## SREF probability predictors
Two precipitation-type algorithms 
Baldwin algorithm in NCEP post. (Pr[Sn, ZR, IP])
Czys algorithm applied in SPC SREF post-processing. 
      (Pr[Sn, ZR, IP])
(2)  Two parameters sensitive to lower tropospheric and ground temperature
Snowmelt parameterization (RSAE)– Evaluates fluxes to determine if 3” of snow melts over a 3h period.  If yes, then parameter is assigned: 273.15 – TG.   (Pr[>1; >2; >4])
Simple algorithm (RSAP) F (Tpbl, TG, Qsfc net rad. flux,) where values > 1 indicate surface cold enough for snow to accumulate. (Pr[>1])

- SREF probability predictors
- Two precipitation-type algorithms
- Baldwin algorithm in NCEP post. (Pr[Sn, ZR, IP])
- Czys algorithm applied in SPC SREF post-processing.
-       (Pr[Sn, ZR, IP])
- (2)  Two parameters sensitive to lower tropospheric and ground temperature
- Snowmelt parameterization (RSAE)– Evaluates fluxes to determine if 3” of snow melts over a 3h period.  If yes, then parameter is assigned: 273.15 – TG.   (Pr[_>_1; _>_2; _>_4])
- Simple algorithm (RSAP) ***F ***(Tpbl, TG, Qsfc net rad. flux,) where values _>_ 1 indicate surface cold enough for snow to accumulate. (Pr[_>_1])

***Goal: Examine the parameter space around the ***
***lower PBL T, ground T, and precip type and calibrate ***
***using road sensor data.***

## Frequency Calibration Tables 

-     _**LAYER**_**          **_**SREF INGREDIENT 1**_**              **_**SREF INGREDIENT 2            **_**                            ****  ****1                    Prob(RSAE > 1)             ****	****Prob(Baldwin Snow, ZR, or IP)****  2                    Prob(RSAE > 2) ****	****	****Prob(Baldwin Snow, ZR, or IP) **
**	****  ****3                    Prob(RSAE > 4) ****	****	****Prob(Baldwin Snow, ZR, or IP) **
**	****  ****4                    Prob(RSAE > 1)             ****	****Prob(Czys Snow, ZR, or IP)****  5                    Prob(RSAE > 2) ****	****	****Prob(Czys Snow, ZR, or IP) **
**	****  ****6                    Prob(RSAE > 4) ****	****	****Prob(Czys Snow, ZR, or IP) ****	**
**	****  ****7                    Prob(RSAP > 1) ****	****	****Prob(Baldwin Snow, ZR, or IP) **
**	****  ****8                    Prob(RSAP > 1)             ****	****Prob(Czys Snow, ZR, or IP)**

## 3h probability of freezing or frozen pcpn (Baldwin algorithm; uncalibrated)

**SREF 32F Isotherm**
**(2 meter air temp)**
**Mean (dash)**
**Union (At least**
**one SREF member at**
**or below 32 F - dots)**
**Intersection (All members **
**at or below 32F- solid)**

**3h probability of freezing or frozen pcpn (Baldwin algorithm; uncalibrated)**

**Example: New England Blizzard (F42: 23 January 2005 03Z)**

**SREF 32F Isotherm**
**(Ground Temp)**
**Mean (dash)**
**Union (At least**
**one SREF member at**
**or below 32 F - dots)**
**Intersection (All members **
**at or below 32F- solid)**

**3h calibrated probability of snow accumulating on roads**

## 3h probability of freezing or frozen pcpn (Baldwin algorithm; uncalibrated)

**SREF 32F Isotherm**
**(2 meter air temp)**
**Mean (dash)**
**Union (dots)**
**Intersection (solid)**

**3h probability of freezing or frozen pcpn (Baldwin algorithm; uncalibrated)**

**Example: Washington, DC Area (F21: 28 February 2005 18Z)**

**SREF 32F Isotherm**
**(Ground Temp)**
**Mean (dash)**
**Union (dots)**
**Intersection (solid)**

**3h calibrated probability of snow accumulating on roads**

## 6h Prob Snow Accum on Roads

**6h Prob Snow Accum on Roads**
**Oct 15, 2005 (F006 v15 UTC)**

**3h Prob Snow Accum on Roads**
**Oct 15, 2005 (F006 v15 UTC)**

## Blind Test

- Calibration period: Oct 1, 2004 through April 30, 2005
- 5  days randomly selected for each month in the sample => 35 days in test
- Test days withheld from the monthly calibration tables (i.e., cross validation used)
- The SREF forecasts were reprocessed for the 35 days and verified against the MADIS surface state observations (F03 – F63)

## Verification

- Reliability Diagram: All 3 h forecasts (F00 – F63);   35 days (Oct 1 – Apr 30)

- Economic Potential Value

- Reliability

## Test Results

- 3 h forecast results (F00 – F63)
- Forecast are reliable
- Brier score is a 21% improvement over sample climatology
- ROC area = .919
- Ave probability where new snow detected: 23%
- Ave probability where new snow not detected: 4%
- Economic value for a wide range of users peaking over 0.7

## Road-Snow: Summary 

- Method appears reliable – although 3h probabilities rarely exceed 50%
- Highlights importance of ground temp predictions from SREF and deterministic models
- Possible improvements:
- Bias correction to 2m and ground temps from SREF*
- Statistical post-processing of 2m and ground temps* prior to road-state calibration
- Addition of asphalt tile to LSM of SREF members

- * *See the next slide for temp correction information*

## F15 SREF 2m Temp

- Under dispersive
- SREF 2m temp
- forecast (F15)
- and cold bias

- F15 cold bias in
- 2m temp removed
- but remains
- under dispersive

- Uniform VOR after
- statistical adjustment
- to SREF

- Raw 2m Temp

- Bias adjusted 2m Temp

- Recalibrated 2m Temp

- Bias adjustment
- and recalibration
- with the addition
- of asphalt-type
- ground temp
- tile in LSM
- might be very
- useful for snow
- accumulation
- from SREF

**F15 SREF 2m Temp**
*Verf Period: ~August, 2005*