# Presentation: 521173

## HIC INNOVATION MEETING

**Part II – Prototype ESPVS**
**October 27-28, 3003**
**Kansas City, MO**
**	****	**
**By**
**Dave Brandon / CBRFC**

## The Current ESP Verification System (ESPVS)

- _Was Developed in the late 90s_

- Software is not modular
- It is hard to maintain
- It is hard to make improvements/changes
- It Lacks Probabilistic Statistical Metrics:
-    Reliability Diagrams
-    Discrimination Diagrams
-    Rank Probability Score
-    Rank Probability Skill Scores
- It does not use the same code (ESPADP)
- that is used in operations when producing
- probabilistic forecasts
- The current version does not work ( very well ).

## Design of a Prototype  (Next)ESPVS

- _RTI_:        Develop the design document and
-                write the code ( in cooperation with
-                OHD and CBRFC & others? ).
-                ( First draft about finished,
-                  some Code written.)
- _U of A_:     Provide the science/math for the
-                 statistical metrics.
-                 ( Currently in progress.)
- _CBRFC_:   Provide test sets, and test and
-                 evaluate prototype code
-                 ( Expect delivery in the new year )

## Prototype Verification System For NWSRFS-ESP

- ( Next ESPVS  -  3 Components )

- _Component 1_
- ESPESG
- ESP
-   _E_nsemble
-    _ __S_eries
-       _G_enerator

- _Component 2_
- ESPADP
- ESP
-   _A_nalysis
-      _D_isplay
-        _P_rogram

- _Component 3_
- ESPVDP
- ESP
-   _V_erification
-     _D_isplay
-       _P_rogram

## n

- 1972

- 1973

- 1971

- MAPS/MATs

- .  .  .  .  .  .  .  .

- Flow Ensembles

- Uses These

- Generates These

- _ESPESG_
- ESP_E_nsemble_S_eries_G_enerator

## Uses These

- Flow Ensembles

- 1971

- n

- 1973

- 1972

- .  .  .  .  .  .  .  .

- Generates These

- Forecasts (e.g.)

- APR-JUL Volume

- APR-JUL Volume

- APR-JUL Volume

- APR-JUL Volume

- _ESPADP_
- ESP_A_nalysis_D_isplay_P_rogram

- Analysis Window

## ESPVDP

- ESP_V_erification_D_isplay_P_rogram

- Uses These

- Forecasts ( e.g.)

- n APR-JUL Volume

- 1971 APR-JUL Volume

- 1972 APR-JUL Volume

- 1973 APR-JUL Volume

- .  .  .  .  .  .  .  .

- Generates These

- Verification Displays

- _ENSEMBLE INFORMATION_
- Talagrand Diagram_ _
- _PROBABILISTIC VERIFCATION_
- Ranked Probability Score RPS
- Ranked Probability Skill Score RPSS
- Discrimination Diagrams
- Reliability Diagram

## FOUR MAIN METRICS TO BE INCLUDED

- (1) Ranked Probability Score RPS
- (2) Ranked Probability Skill Score RPSS
- (3) Discrimination Diagrams
- (4) Reliability Diagram

## Ranked Probability Score (RPS)

- The Ranked Probability Score (RPS) is used to assess
- the overall forecast performance of the probabilistic
- forecasts.
- A perfect forecast would result in a RPS or zero.
- Gives credit for forecasts close to observation...
- Penalizes forecasts further from the observation.
- Looks at the entire distribution ( all traces ).
- Good overall summary statistic at all flow levels.
- It is a multi categorical Brier Score.

## Because the actual RPS

- value is difficult to evaluate
- independently, the use of
- the RPS in the absence
- of reference forecasts is
- limited to forecast comparison
- among different locations.
- It can be used to analyze
- regional consistency, i.e.,
- possible need for recalibration.

**Credit: “Verification on NWS Probabilistic Hydrologic Forecasts” – Franz/Sorooshian – U. of AZ**

## RPS used to compare various basins. (Note RPS here was computed with 100 bins.)

## Ranked Probability Skill Score RPSS

- _Ranked Probability Skill Score RPSS_

- Useful to compare the forecast of interest to a
- reference forecast, e.g.,  climatology.
- Is expressed as a percent improvement, e.g.,
- over the reference forecast  (e.g. how much
- better/worse than climatology).
- Perfect score is 1.0 (100%).
- Negative score indicates forecasts performed
- worse than reference forecast.

## Observed Hydrograph

- Degradation                                                             Improvement

## Discrimination Diagram

- A discrimination diagram displays forecast probability
- distribution(s) conditioned on observation(s).
- For example,  given that the observations
- were high flows,  did the forecasts prior to the
- observations reflect high flows.

## Discrimination seen through CDFs conditioned on observation:

- CDFs are sorted and averaged according to the observed volume. Good discrimination indicated by seperation between conditional CDFs

- Observed                    Forecast

## Discrimination

- Another way to look at discrimination using PDF’s in lieu of CDF’s.
- The more separation between the PDF’s the better the discrimination.

## Reliability Diagram

- A reliability diagram is used to display forecast
- reliability for a particular ‘type’ of forecast
- ( e.g., flows above flood stage, flows in the lower
- quartile ).
- It displays how often an observation occurred given
- a particular forecast probability.
- For a set of forecasts where a forecast probability
- value was given to a particular observation, the
- forecasts are considered perfectly reliable if the
- relative frequency of the observations equals the
- forecast probability.

## Reliability Diagram

- Relative Frequency of Observations

**Forecasts are not assigning enough probability **

**Forecasts are  assigning too much probability **

**Forecasts are perfectly reliable on the diagonal line**

-                      Forecast Probability

## We have developed a:

- _Probabilistic Forecast Verification Primer_
- 72 PowerPoint Slides
- Specific Examples of Metrics
- A few example slides will follow:

- Version 2.1 October 9, 2003

## Reliability Diagram

- _Reliability Diagram_
- _Example_
- Under Forecasting
- if area is above the
- diagonal
- Perfect if on the
- diagonal
- Over Forecasting
- if area is below the
- diagonal

## Discrimination with sample data set

- YEAR	E1      E2      E3     E4
- 1981 	42      74       82     90
- 1982 	65     143     223   227
- 1983 	82     192     295   300
- 1984 	211   397     514   544
- 1985 	142   291     349   356
- 1986 	114   277     351   356
- 1987 	98     170     204   205
- 1988 	69     169     229   236
- 1989 	94     219     267   270
- 1990 	59     175     244   250
- 1991 	108   189     227   228
- 1992 	94     135     156   158

- OBS
- 112
- 206
- 301
- 516
- 348
-  98
- 156
- 245
- 233
- 248
- 227
- 167

- 2. Group forecast ensemble members according to OBS tercile.

- OBS Tercile
- Low
- Middle
- High
- High
- High
- Low
- Low
- Middle
- Middle
- High
- Middle
- Low

- Low OBS Forecasts:
- 42, 74, 82, 90,
- 114, 277, 351, 356,
- 98, 170, 204, 205, 94,
- 135, 156, 158

- Hi OBS Forecasts:
- 82, 192, 295, 300,
- 211, 397, 514, 544,
- 142, 291, 349, 356,
- 59, 175, 244, 250

- Mid OBS Forecasts:
- 65, 143, 223, 227,
- 69, 169, 229, 236,
- 94, 219, 267, 270,
- 108, 189, 227, 228

## RPS with sample data set

- YEAR	E1      E2      E3     E4
- 1981 	42      74       82     90
- 1982 	65     143     223   227
- 1983 	82     192     295   300
- 1984 	211   397     514   544
- 1985 	142   291     349   356
- 1986 	114   277     351   356
- 1987 	98     170     204   205
- 1988 	69     169     229   236
- 1989 	94     219     267   270
- 1990 	59     175     244   250
- 1991 	108   189     227   228
- 1992 	94     135     156   158

- OBS
- 1.12
- 2.06
- 3.01
- 5.16
- 3.48
- 0.98
- 1.56
- 2.45
- 2.33
- 2.48
- 2.27
- 1.67

- 2. Choose number of bins and bin spacing. For Simplicity, let’s choose four bins and set them to be: (100,200,300,400)