# Presentation: 674891

## Categorical Forecast Verification

- Kevin Werner

*RFC Verification Workshop, 08/14/2007*

## Outline

- Deterministic Forecasts
- Contingency tables
- Sample Forecast
- Statistics
- POD, FAR, etc
- ROC Area
- Ensemble Forecasts
- Brier Score
- Ranked Probability Score
- Skill Scores
- Sample Forecast Calculations

## Contingency Tables

- Used to analyze the relationship between 2 categorical variables in a single dataset
- 2x2 contingency table shows categories relative to a single threshold of each variable (e.g. forecast and observed flow)
- Wilkes, 2006 section 7.2.1

|  |  | Event observed |  |   |
| --- | --- | --- | --- | --- |
|  |  | Yes | No | Total |
| Event forecasted | Yes | a | b | a+b |
|  | No | c | d | c+d |
|  | Total | a+c | b+d | a+b+c+d=n |

## Contingency Tables

- 3x3 contingency table uses two thresholds
- Wilkes, 2006
- 	section 7.2.6

|  |  | Event observed |  |  |   |
| --- | --- | --- | --- | --- | --- |
|  |  | Below | Within | Above | Total |
| Event forecasted | Below | A | B | C | A+B+C |
|  | Within | D | E | F | D+E+F |
|  | Above | G | H | I | G+H+I |
|  | Total | A+D+G | B+E+H | C+F+I | n |

## Sample Deterministic Verification Data Set

- YEAR	FCST
- 1981 	82
- 1982 	223
- 1983 	295
- 1984 	514
- 1985 	349
- 1986 	351
- 1987 	204
- 1988 	229
- 1989 	267
- 1990 	304
- 1991 	227
- 1992 	156

- OBS
- 112
- 206
- 301
- 516
- 348
-  98
- 156
- 245
- 233
- 248
- 227
- 167

- Four forecasts of peak flow (cfs) were made on May 14 for a two day window ending May 16 for a number of years.
- “Monitor” stage is 200 cfs
- “Flood” stage is 300 cfs

- For illustrative purposes, a small sample data set of ensemble forecasts was created. Each of the verification techniques will be applied to this dataset.

## Create 2x2 contingency table

- YEAR	FCST
- 1981 	82
- 1982 	223
- 1983 	295
- 1984 	514
- 1985 	349
- 1986 	351
- 1987 	204
- 1988 	229
- 1989 	267
- 1990 	304
- 1991 	227
- 1992 	156

- OBS
- 112
- 206
- 301
- 516
- 348
-  98
- 156
- 245
- 233
- 248
- 227
- 167

- Step 1: Convert forecasts and observed from deterministic to categories based on “flood” flow (300 cfs)

- Step 1: Categorize all forecasts and observations

**FCST CAT**

**OBS CAT**

**Below**
**Below**
**Below**
**Above**
**Above**
**Above**
**Below**
**Below**
**Below**
**Above**
**Below**
**Below**

**Below**
**Below**
**Above**
**Above**
**Above**
**Below**
**Below**
**Below**
**Below**
**Below**
**Below**
**Below**

## Create 2x2 contingency table

- YEAR	FCST
- 1981 	82
- 1982 	223
- 1983 	295
- 1984 	514
- 1985 	349
- 1986 	351
- 1987 	204
- 1988 	229
- 1989 	267
- 1990 	304
- 1991 	227
- 1992 	156

- OBS
- 112
- 206
- 301
- 516
- 348
-  98
- 156
- 245
- 233
- 248
- 227
- 167

**FCST CAT**

**OBS CAT**

**Below**
**Below**
**Below**
**Above**
**Above**
**Above**
**Below**
**Below**
**Below**
**Above**
**Below**
**Below**

**Below**
**Below**
**Above**
**Above**
**Above**
**Below**
**Below**
**Below**
**Below**
**Below**
**Below**
**Below**

- Step 1: Categorize all forecasts and observations

- Step 1: Convert forecasts and observed from deterministic to categories based on “flood” flow (300 cfs)

## Create 2x2 contingency table (con’t)

- Step 2: Count number of forecasts for each of four “contingencies”

|  |  | Event observed |  |   |
| --- | --- | --- | --- | --- |
|  |  | Yes | No | Total |
| Event forecasted | Yes | 2 | 2 | 4 |
|  | No | 1 | 7 | 8 |
|  | Total | 3 | 9 | 12 |

**FCST CAT**
**Below**
**Below**
**Below**
**Above**
**Above**
**Above**
**Below**
**Below**
**Below**
**Above**
**Below**
**Below**

**OBS CAT**
**Below**
**Below**
**Above**
**Above**
**Above**
**Below**
**Below**
**Below**
**Below**
**Below**
**Below**
**Below**

## Create 2x2 contingency table (con’t)

- Step 2: Count number of forecasts for each of four “contingencies”

|  |  | Event observed |  |   |
| --- | --- | --- | --- | --- |
|  |  | Yes | No | Total |
| Event forecasted | Yes | 2 | 2 | 4 |
|  | No | 1 | 7 | 8 |
|  | Total | 3 | 9 | 12 |

**FCST CAT**
**Below**
**Below**
**Below**
**Above**
**Above**
**Above**
**Below**
**Below**
**Below**
**Above**
**Below**
**Below**

**OBS CAT**
**Below**
**Below**
**Above**
**Above**
**Above**
**Below**
**Below**
**Below**
**Below**
**Below**
**Below**
**Below**

## Probability of Detection (POD)

- Sometimes called “hit rate”
- Discrimination - Conditional probability that given an observed value in a category, the forecast value is also in that category
- How often were you not ‘surprised’?
- Wilkes, 2006 section 7.2.2
- POD = a/(a+c)

|  |  | Event observed |  |   |
| --- | --- | --- | --- | --- |
|  |  | Yes | No | Total |
| Event forecasted | Yes | a | b | a+b |
|  | No | c | d | c+d |
|  | Total | a+c | b+d | a+b+c+d=n |

## Probability of Detection (POD)

- For sample data:
- POD = a/(a+c)
-             = 2/3
-             = 67%
- Ref: IVP 8.2 appendix B

|  |  | Event observed |  |   |
| --- | --- | --- | --- | --- |
|  |  | Yes | No | Total |
| Event forecasted | Yes | 2 | 2 | 4 |
|  | No | 1 | 7 | 8 |
|  | Total | 3 | 9 | 12 |

## False Alarm Ratio (FAR)

- Reliability - Conditional probability that given a forecast value in a category, the observed value is not in that category
- How often were you led astray?
- Wilkes, 2006 section 7.2.2
- FAR = b/(a+b)

|  |  | Event observed |  |   |
| --- | --- | --- | --- | --- |
|  |  | Yes | No | Total |
| Event forecasted | Yes | a | b | a+b |
|  | No | c | d | c+d |
|  | Total | a+c | b+d | a+b+c+d=n |

## False Alarm Ratio (FAR)

- For sample data:
- FAR = b/(a+b)
-             = 2/4
-             = 50%
- Ref: IVP 8.2 Documentation Appendix B

|  |  | Event observed |  |   |
| --- | --- | --- | --- | --- |
|  |  | Yes | No | Total |
| Event forecasted | Yes | 2 | 2 | 4 |
|  | No | 1 | 7 | 8 |
|  | Total | 3 | 9 | 12 |

## Other terms based on the 2x2 contingency table in IVP

- Hits = a
- Misses = c
- False Alarms = b
- Critical Success Index (CSI)
- = hits / (a+b+c)
- Measure of Accuracy
- Gilbert Score – accounts for random error
- Would be a measure of bias?

|  |  | Event observed |  |   |
| --- | --- | --- | --- | --- |
|  |  | Yes | No | Total |
| Event forecasted | Yes | a | b | a+b |
|  | No | c | d | c+d |
|  | Total | a+c | b+d | a+b+c+d=n |

## Other terms based on the 2x2 contingency table not in IVP

- Probability of False Detection (PoFD)
- The probability that, given the event did not occur, it was forecast to occur
- Discrimination
-     = b / (b+d)
- Covered discrimination, reliability, and accuracy... is there a measure of bias?
- Bias: (a+b) / (a+c)

|  |  | Event observed |  |   |
| --- | --- | --- | --- | --- |
|  |  | Yes | No | Total |
| Event forecasted | Yes | a | b | a+b |
|  | No | c | d | c+d |
|  | Total | a+c | b+d | a+b+c+d=n |

## Create 3x3 contingency table

- YEAR	FCST
- 1981 	82
- 1982 	223
- 1983 	295
- 1984 	514
- 1985 	349
- 1986 	351
- 1987 	204
- 1988 	229
- 1989 	267
- 1990 	304
- 1991 	227
- 1992 	156

- OBS
- 112
- 206
- 301
- 516
- 348
-  98
- 156
- 245
- 233
- 248
- 227
- 167

- Step 1: Convert forecasts and observed from deterministic to categories based on “flood” stage and “monitor” stage
- “Monitor” stage is 200 cfs
- “Flood” stage is 300 cfs

- Step 1: Categorize forecasts and observations relative to TWO thresholds

**FCST CAT**
**Below**
**Within**
**Within**
**Above**
**Above**
**Above**
**Within**
**Within**
**Within**
**Above**
**Within**
**Below**

**OBS CAT**
**Below**
**Within**
**Above**
**Above**
**Above**
**Below**
**Below**
**Within**
**Within**
**Within**
**Within**
**Below**

## Create 3x3 Contingency Table (con’t)

**OBS CAT**
**Below**
**Within**
**Above**
**Above**
**Above**
**Below**
**Below**
**Within**
**Within**
**Within**
**Within**
**Below**

**FCST CAT**
**Below**
**Within**
**Within**
**Above**
**Above**
**Above**
**Within**
**Within**
**Within**
**Above**
**Within**
**Below**

|  |  | Event observed |  |  |   |
| --- | --- | --- | --- | --- | --- |
|  |  | Below | Within | Above | Total |
| Event forecasted | Below | 2 | 0 | 0 | 2 |
|  | Within | 1 | 4 | 1 | 6 |
|  | Above | 1 | 1 | 2 | 4 |
|  | Total | 4 | 5 | 3 | 12 |

- 	Step 2: Count number of forecasts for each of nine “contingencies”

## 3 Category Metrics in IVP

- Traditional False Alarm Ratio = (D+F)/(D+E+F)
- Hydrologic False Alarm Ratio = D/(D+E+F)
- Underforecast Rate = B/(B+E+H)
- Overforecast Rate = H/(B+E+H)
- Sample, observed = B+E+H
- Sample, forecast = D+E+F

|  |  | Event observed |  |  |   |
| --- | --- | --- | --- | --- | --- |
|  |  | Below | Within | Above | Total |
| Event forecasted | Below | A | B | C | A+B+C |
|  | Within | D | E | F | D+E+F |
|  | Above | G | H | I | G+H+I |
|  | Total | A+D+G | B+E+H | C+F+I | n |

## Relative Operating Characteristic (ROC)

- Measures ability of forecast to discriminate between events and non-events
- Plots POD vs POFD for a set of thresholds
- Area under curve is a useful measure of forecast skill        (1 = perfect; 0.5 = no skill)
- Does NOT account for full information contained in joint distribution of forecasts and observations (unlike reliability or discrimination diagrams)
- No information on bias
- Originally developed in psychology and medicine in the 1970s
- Wilkes, 2006 section 7.4.6

- Probability of False Detection (POFD)

- Probability of Detection (POD)

## ROC Calculation

- Step 1: Choose thresholds for contingency table calculations
- Typically based on increments of probability on forecast space
- For simplicity, for sample data, we choose thresholds of flow value:
**200, 250, and 300 cfs**

- YEAR	FCST
- 1981 	82
- 1982 	223
- 1983 	295
- 1984 	514
- 1985 	349
- 1986 	351
- 1987 	204
- 1988 	229
- 1989 	267
- 1990 	304
- 1991 	227
- 1992 	156

- OBS
- 112
- 206
- 301
- 516
- 348
-  98
- 156
- 245
- 233
- 248
- 227
- 167

## ROC Calculation (con’t)

- Step 2: Create 2x2 contingency tables for each threshold:
-         300 cfs                250 cfs                 200 cfs

|  |  | O? |  |   |
| --- | --- | --- | --- | --- |
|  |  | Y | N |  |
| F? | Y | 2 | 2 | 4 |
|  | N | 1 | 7 | 8 |
|  |  | 3 | 9 | 12 |

|  |  | O? |  |   |
| --- | --- | --- | --- | --- |
|  |  | Y | N |  |
| F? | Y | 3 | 2 | 5 |
|  | N | 1 | 6 | 7 |
|  |  | 4 | 8 | 12 |

|  |  | O? |  |   |
| --- | --- | --- | --- | --- |
|  |  | Y | N |  |
| F? | Y | 8 | 2 | 10 |
|  | N | 0 | 2 | 2 |
|  |  | 8 | 4 | 12 |

## ROC Calculation (con’t)

- Step 3: Compute POD and POFD for each threshold:
-         300 cfs                250 cfs                 200 cfs

|  |  | O? |  |   |
| --- | --- | --- | --- | --- |
|  |  | Y | N |  |
| F? | Y | 2 | 2 | 4 |
|  | N | 1 | 7 | 8 |
|  |  | 3 | 9 | 12 |

|  |  | O? |  |   |
| --- | --- | --- | --- | --- |
|  |  | Y | N |  |
| F? | Y | 3 | 2 | 5 |
|  | N | 1 | 6 | 7 |
|  |  | 4 | 8 | 12 |

|  |  | O? |  |   |
| --- | --- | --- | --- | --- |
|  |  | Y | N |  |
| F? | Y | 8 | 2 | 10 |
|  | N | 0 | 2 | 2 |
|  |  | 8 | 4 | 12 |

- POD = 2/3
- POFD = 2/9

- POD = 3/4
- POFD = 2/8

- POD = 8/8
- POFD = 2/4

## ROC Calculation (con’t)

- Step 4: Plot each pair of POD and POFD
-     (2/3,2/9), (8/8, 2/4), and (3/4, 2/8):

- Probability of False Detection (POFD)

- Probability of Detection (POD)

**0**

**1**

**1**

- No skill line

## ROC Calculation Alternative

|  |  | Event observed |  |
| --- | --- | --- | --- |
|  |  | Yes | No |
| Event forecasted | >80% - 100% | 30 | 5 |
|  | >60% - 80% | 25 | 10 |
|  | >40% - 60% | 20 | 15 |
|  | >20% - 40% | 15 | 20 |
|  | >0% - 20% | 10 | 25 |
|  | 0% | 5 | 30 |
|  | total | 105 | 105 |

|  |  | Event observed |  |
| --- | --- | --- | --- |
|  |  | Yes | No |
|  | >80% | 30 | 5 |
|  | >60% | 55 | 15 |
|  | >40% | 75 | 30 |
|  | >20% | 90 | 50 |
|  | >0% | 100 | 75 |
|  | ≥0% | 105 | 105 |
|  |  |  |  |

|  |  |
| --- | --- |
| POD | POFD |
| 30/105 | 5/105 |
| 55/105 | 15/105 |
| 75/105 | 30/105 |
| 90/105 | 50/105 |
| 100/105 | 75/105 |
| 105/105 | 105/105 |
|  |  |

- Create modified 2x2 contingency table and cumulative contingency table to calculate PODs and POFDs for each threshold.
- Result is the same

## IVP ROC Calculation

- User identifies an observed threshold, and the IVP then generates a user specified (default: 100) number of thresholds for the forecast value in order to build 2x2 contingency tables.  It computes the POD and POFD for each combination of forecast and observed threshold and that becomes a point along the curve.
- Default thresholds are equally spaced in observed/forecast space (e.g. ft or cfs or ...) between lowest and highest observed / forecasted values
- Ref: IVP 8.2 documentation Appendix B

- Credit: Hank Herr

## Probabilistic

**Forecast Evaluation**
**~Categorical Forecasts~**

## Observed Outcome

**Probabilistic Forecast Evaluation: Brier Score (BS)**

**Forecast:**
- “80% chance of rain”

**With this outcome, **
**Which **_**forecast**_** would you say is better?**

***55%***

***45%***

**Forecast:**
- “55% chance of rain”

- Credit: Holly Hartmann

## Observed Outcome

**Forecast:**
- “80% chance of rain”

**With this outcome, **
**Which **_**forecast**_** would you say is better?**

***55%***

***45%***

**Forecast:**
- “55% chance of rain”

_**Good**_

- _Not as Good_

- Credit: Holly Hartmann

**Probabilistic Forecast Evaluation: Brier Score (BS)**

## Observed Outcome

**Forecast:**
- “80% chance of rain”

***55%***

***45%***

**Forecast:**
- “55% chance of rain”

**The “Brier” Score considers:**
***Direction *****of forecast**
***Strength***** of forecast**

- Credit: Holly Hartmann

**Probabilistic Forecast Evaluation: Brier Score (BS)**

## Probabilistic Forecast Evaluation: Brier Score (BS)

- BS is bounded by 0 and 1.
- A forecast with BS=0 is perfect.

**Probabilistic Forecast Evaluation: Brier Score (BS)**

## 100%

- 80%

**If ‘High flows’ are your focus, and ‘High’ actually occurs...**

**100%**

- -0.22

- 0.04

- 30%

**100%**

- -0.702

- 0.49

- (“Climatology”    -   The Observed)2 = Climatology’s Performance

- Credit: Holly Hartmann

**Probabilistic Forecast Evaluation: Brier Score (BS)**

## How Good? Compared to What?

**Skill Score: (****0.50**** – ****0.54****)/(1.00-****0.54****) = -8.6%**
**~worse than guessing~  **

**Skill Score =**

**S****Forecast**** – S****Baseline**
**S****Perfect**** – S****Baseline**

**How Good? Compared to What?**

**What is the appropriate Baseline?**

**= 1 - **

**S****Forecast**
**S****Baseline**

- Credit: Holly Hartmann

## Brier Skill Score = (0.04-0.49)/(0-0.49) = +92% 

- 80%

**If ‘High flows’ are your focus, and ‘High’ actually occurs...**

**100%**

- -0.22

- 0.04

- 30%

**100%**

- -0.702

- 0.49

- (“Climatology”    -   The Observed)2 = Climatology’s Performance

- Brier Skill Score = (0.04-0.49)/(0-0.49) = +92%

- Credit: Holly Hartmann

**Probabilistic Forecast Evaluation: **
**Brier Skill Score (BSS)**

## Brier Skill Score = (0.36-0.49)/(0-0.49) = 27% 

- (Your Forecast    -   The Observed)2 = Your Performance

- 40%

**100%**

- -0.602

- 0.36

- 30%

**100%**

- -0.702

- 0.49

- (Climatology    -   The Observed)2 = Climatology’s Performance

- Brier Skill Score = (0.36-0.49)/(0-0.49) = 27%

**If ‘High flows’ are your focus, and ‘High’ actually occurs...**

- Credit: Holly Hartmann

**Probabilistic Forecast Evaluation: **
**Brier Skill Score (BSS)**

## Sample Verification Data Set

- YEAR	E1      E2      E3     E4
- 1981 	42      74       82     90
- 1982 	65     143     223   227
- 1983 	82     192     295   300
- 1984 	211   397     514   544
- 1985 	142   291     349   356
- 1986 	114   277     351   356
- 1987 	98     170     204   205
- 1988 	69     169     229   236
- 1989 	94     219     267   270
- 1990 	59     175     244   250
- 1991 	108   189     227   228
- 1992 	94     135     156   158

- OBS
- 112
- 206
- 301
- 516
- 348
-  98
- 156
- 245
- 233
- 248
- 227
- 167

- Four forecasts of peak flow (cfs) were made on May 14 for a two week window ending May 29 for a number of years. The four Sample Ensemble Members (E1 – E4) were ranked lowest to highest and correlated with the corresponding observation.

- For illustrative purposes, a small sample data set of ensemble forecasts was created. Each of the verification techniques will be applied to this dataset.

## Brier Score Example

- YEAR	E1      E2      E3     E4
- 1981 	42      74       82     90
- 1982 	65     143     223   227
- 1983 	82     192     295   300
- 1984 	211   397     514   544
- 1985 	142   291     349   356
- 1986 	114   277     351   356
- 1987 	98     170     204   205
- 1988 	69     169     229   236
- 1989 	94     219     267   270
- 1990 	59     175     244   250
- 1991 	108   189     227   228
- 1992 	94     135     156   158

- OBS
- 112
- 206
- 301
- 516
- 348
-  98
- 156
- 245
- 233
- 248
- 227
- 167

- Step 1: Compute probability of flood / no flood based on flood flow of 300 cfs.

## Brier Score Example

- YEAR	E1      E2      E3     E4
- 1981 	42      74       82     90
- 1982 	65     143     223   227
- 1983 	82     192     295   300
- 1984 	211   397     514   544
- 1985 	142   291     349   356
- 1986 	114   277     351   356
- 1987 	98     170     204   205
- 1988 	69     169     229   236
- 1989 	94     219     267   270
- 1990 	59     175     244   250
- 1991 	108   189     227   228
- 1992 	94     135     156   158

- OBS
- 112
- 206
- 301
- 516
- 348
-  98
- 156
- 245
- 233
- 248
- 227
- 167

- Step 1: Compute probability of flood / no flood based on flood flow of 300 cfs.

## Brier Score Example

- YEAR	E1      E2      E3     E4
- 1981 	42      74       82     90
- 1982 	65     143     223   227
- 1983 	82     192     295   300
- 1984 	211   397     514   544
- 1985 	142   291     349   356
- 1986 	114   277     351   356
- 1987 	98     170     204   205
- 1988 	69     169     229   236
- 1989 	94     219     267   270
- 1990 	59     175     244   250
- 1991 	108   189     227   228
- 1992 	94     135     156   158

- OBS
- 112
- 206
- 301
- 516
- 348
-  98
- 156
- 245
- 233
- 248
- 227
- 167

- Step 1: Compute probability of flood / no flood based on flood flow of 300 cfs.

## Brier Score Example

- YEAR
- 1981
- 1982
- 1983
- 1984
- 1985
- 1986
- 1987
- 1988
- 1989
- 1990
- 1991
- 1992

- OBS
- 112
- 206
- 301
- 516
- 348
-  98
- 156
- 245
- 233
- 248
- 227
- 167

- Step 2: Determine whether event (flooding) occurred (1) or not (0).

- Flooding?
- 0
- 0
- 1
- 1
- 1
- 0
- 0
- 0
- 0
- 0
- 0
- 0

## Brier Score Example

- YEAR
- 1981
- 1982
- 1983
- 1984
- 1985
- 1986
- 1987
- 1988
- 1989
- 1990
- 1991
- 1992

- OBS
- 112
- 206
- 301
- 516
- 348
-  98
- 156
- 245
- 233
- 248
- 227
- 167

- Step 2: Determine whether event (flooding) occurred (1) or not (0).

- Flooding?
- 0
- 0
- 1
- 1
- 1
- 0
- 0
- 0
- 0
- 0
- 0
- 0

## Brier Score Example

- YEAR
- 1981
- 1982
- 1983
- 1984
- 1985
- 1986
- 1987
- 1988
- 1989
- 1990
- 1991
- 1992

- Step 3: Calculate Brier Score by year, (y – o)2

- P(flood)
- 0
- 0
- 0.25
- 0.75
- 0.5
- 0.5
- 0
- 0
- 0
- 0
- 0
- 0

- Flooding?
- 0
- 0
- 1
- 1
- 1
- 0
- 0
- 0
- 0
- 0
- 0
- 0

- (y – o)2
- 0
- 0
- 0.56
- 0.06
- 0.25
- 0.25
- 0
- 0
- 0
- 0
- 0
- 0

## Brier Score Example

- YEAR
- 1981
- 1982
- 1983
- 1984
- 1985
- 1986
- 1987
- 1988
- 1989
- 1990
- 1991
- 1992

- Step 3: Calculate Brier Score by year, (y – o)2

- P(flood)
- 0
- 0
- 0.25
- 0.75
- 0.5
- 0.5
- 0
- 0
- 0
- 0
- 0
- 0

- Flooding?
- 0
- 0
- 1
- 1
- 1
- 0
- 0
- 0
- 0
- 0
- 0
- 0

- (y – o)2
- 0
- 0
- 0.56
- 0.06
- 0.25
- 0.25
- 0
- 0
- 0
- 0
- 0
- 0

## Brier Score Example

- YEAR
- 1981
- 1982
- 1983
- 1984
- 1985
- 1986
- 1987
- 1988
- 1989
- 1990
- 1991
- 1992

- Step 4: Calculate average BS = 1/n SUM[(y – o)2]

- P(flood)
- 0
- 0
- 0.25
- 0.75
- 0.5
- 0.5
- 0
- 0
- 0
- 0
- 0
- 0

- Flooding?
- 0
- 0
- 1
- 1
- 1
- 0
- 0
- 0
- 0
- 0
- 0
- 0

- (y – o)2
- 0
- 0
- 0.56
- 0.06
- 0.25
- 0.25
- 0
- 0
- 0
- 0
- 0
- 0

- BS = 1/12 * (0.56+0.06+0.25+0.25+...)
- BS = 0.093

## Probabilistic Forecast Evaluation: Measures

- RPS      = (0.10 - 0)2 + (0.43 - 0)2  + (1 - 1)2  = 0.20
- RPSclim  = (0.30   - 0)2 + (0.70   - 0)2  + (1 - 1)2 = 0.58
- SSrps      = (0.20 - 0.58)/(0 - 0.58) = 0.67

**Brier Score: **2 categories – ‘In’ or ‘Out’
**Ranked Probability Score: **multiple categories – ‘Near Miss’ credit
** ****similar to RMSE, but for cumulative probability**
** ****observation ‘probability’ = 0 or 1 **

***57%***

***33%***

***10%***

- Credit: Holly Hartmann

**Probabilistic Forecast Evaluation: Measures**

## Bad

**Conservative Forecaster**

**Bold Forecaster**

**Not **
**Good**

**Really**
**Good**

- RPSB      = (0.03 - 0)2 + (0.20 - 0)2  + (1 - 1)2  = 0.04
- RPSC     = (0.27 - 0)2 + (0.60 - 0)2  + (1 - 1)2  = 0.43
- RPSclim  = (0.30   - 0)2 + (0.70   - 0)2  + (1 - 1)2 = 0.58
- SSBrps      = (0.04 - 0.58)/(0 - 0.58) = 0.931 = 93%
- SSCrps      = (0.43 - 0.58)/(0 - 0.58) = 0.259 = 26%

- 0.80/0.17/0.03

- 0.40/0.33/0.27

- Credit: Holly Hartmann

**Probabilistic Forecast Evaluation: **
**Ranked Probability Score (RPS)**

## RPS used to compare various basins. (Note RPS here was computed with 100 bins.)

## Colorado River Watersheds: Historical Water Supply Outlooks

**Colorado River Watersheds: Historical Water Supply Outlooks**

***Morrill, Hartmann, and Bales, 2007***

***Baseline = climatology***

***Computed across 5 categories***

## RPS Formulation: Many “Categories”

**RPS Formulation: Many “Categories”**

- Goal: Compare forecast CDF to observed CDF

- Notes:
- Here an empirical distribution is assumed (not necessary).
- Observation is one value, in this case 300.

## RPS Formulation

**RPS Formulation**

## RPS Formulation

- A “normalized RPS” may be defined as
- NRPS = RPS / (# of bins)
- NRPS allows comparison between RPSs calculated with different numbers of bins and is bounded by 0 and 1. Again a score of zero indicates a perfect forecast.

**RPS Formulation**

## RPS Formulation

- Courtesy Kevin Werner, NWS

**RPS Formulation**

## RPS with sample data set

- YEAR	E1      E2      E3     E4
- 1981 	42      74       82     90
- 1982 	65     143     223   227
- 1983 	82     192     295   300
- 1984 	211   397     514   544
- 1985 	142   291     349   356
- 1986 	114   277     351   356
- 1987 	98     170     204   205
- 1988 	69     169     229   236
- 1989 	94     219     267   270
- 1990 	59     175     244   250
- 1991 	108   189     227   228
- 1992 	94     135     156   158

- OBS
- 112
- 206
- 301
- 516
- 348
-  98
- 156
- 245
- 233
- 248
- 227
- 167

- Step 1: Assume an empirical CDF; for example, 1981 and 1984 are shown below. Note the Y values are simply 0/4,1/4,2/4,3/4, and 4/4.

## RPS with sample data set

- YEAR	E1      E2      E3     E4
- 1981 	42      74       82     90
- 1982 	65     143     223   227
- 1983 	82     192     295   300
- 1984 	211   397     514   544
- 1985 	142   291     349   356
- 1986 	114   277     351   356
- 1987 	98     170     204   205
- 1988 	69     169     229   236
- 1989 	94     219     267   270
- 1990 	59     175     244   250
- 1991 	108   189     227   228
- 1992 	94     135     156   158

- OBS
- 1.12
- 2.06
- 3.01
- 5.16
- 3.48
- 0.98
- 1.56
- 2.45
- 2.33
- 2.48
- 2.27
- 1.67

- Step 2: Choose number of bins and bin spacing. For Simplicity, let’s choose four bins and set them to be: (100,200,300,400)

## RPS with sample data set

- YEAR	E1      E2      E3     E4
- 1981 	42      74       82     90
- 1982 	65     143     223   227
- 1983 	82     192     295   300
- 1984 	211   397     514   544
- 1985 	142   291     349   356
- 1986 	114   277     351   356
- 1987 	98     170     204   205
- 1988 	69     169     229   236
- 1989 	94     219     267   270
- 1990 	59     175     244   250
- 1991 	108   189     227   228
- 1992 	94     135     156   158

- OBS
- 112
- 206
- 301
- 516
- 348
-  98
- 156
- 245
- 233
- 248
- 227
- 167

- Step 3: Pick off the probability that the flow will be less than the break point at each bin (i.e. non-exceedence probability). For example, 1984 would be:

- Flow	Probability
-        0
-        0
-        0.25
- 400         0.5

## RPS with sample data set

- YEAR	E1      E2      E3     E4
- 1981 	42      74       82     90
- 1982 	65     143     223   227
- 1983 	82     192     295   300
- 1984 	211   397     514   544
- 1985 	142   291     349   356
- 1986 	114   277     351   356
- 1987 	98     170     204   205
- 1988 	69     169     229   236
- 1989 	94     219     267   270
- 1990 	59     175     244   250
- 1991 	108   189     227   228
- 1992 	94     135     156   158

- OBS
- 112
- 206
- 301
- 516
- 348
-  98
- 156
- 245
- 233
- 248
- 227
- 167

- Step 3 (con’t): Pick off probabilities at each bin for all years. Note that the 300 cfs column is the same as the BS calculation.

- 100 cfs	200 cfs	300 cfs	400 cfs
- 1.0	1.0	1.0	1.0
- 0.25	0.5	1.0	1.0
- 0.25	0.5	1.0	1.0
- 0	0	0.25	0.5
- 0.0	0.25	0.5	1.0
- 0.0	0.25	0.5	1.0
- 0.25	0.5	1.0	1.0
- 0.25	0.5	1.0	1.0
- 0.25	0.25	1.0	1.0
- 0.25	0.5	1.0	1.0
- 0.0	0.5	1.0	1.0
- 0.25	1.0	1.0	1.0

- Probability(forecast < ...)

## RPS with sample data set

- YEAR	E1      E2      E3     E4
- 1981 	42      74       82     90
- 1982 	65     143     223   227
- 1983 	82     192     295   300
- 1984 	211   397     514   544
- 1985 	142   291     349   356
- 1986 	114   277     351   356
- 1987 	98     170     204   205
- 1988 	69     169     229   236
- 1989 	94     219     267   270
- 1990 	59     175     244   250
- 1991 	108   189     227   228
- 1992 	94     135     156   158

- OBS
- 112
- 206
- 301
- 516
- 348
-  98
- 156
- 245
- 233
- 248
- 227
- 167

- Step 3 (con’t): Pick off probabilities at each bin for all years. Note that the 300 cfs column is the same as the BS calculation.

- Probability(forecast < ...)

- 100 cfs	200 cfs	300 cfs	400 cfs
- 1.0	1.0	1.0	1.0
- 0.25	0.5	1.0	1.0
- 0.25	0.5	1.0	1.0
- 0	0	0.25	0.5
- 0.0	0.25	0.5	1.0
- 0.0	0.25	0.5	1.0
- 0.25	0.5	1.0	1.0
- 0.25	0.5	1.0	1.0
- 0.25	0.25	1.0	1.0
- 0.25	0.5	1.0	1.0
- 0.0	0.5	1.0	1.0
- 0.25	1.0	1.0	1.0

## RPS with sample data set

- YEAR
- 1981
- 1982
- 1983
- 1984
- 1985
- 1986
- 1987
- 1988
- 1989
- 1990
- 1991
- 1992

- OBS
- 112
- 206
- 301
- 516
- 348
-  98
- 156
- 245
- 233
- 248
- 227
- 167

- Step 4: Compute Probabilities for observations. Since there is only one observation the probability will be either 0 or 1.

- 100 cfs	200	300 	400
- 1.0	1.0	1.0	1.0
- 0.25	0.5	1.0	1.0
- 0.25	0.5	1.0	1.0
- 0.0	0.0	0.25	0.5
- 0.0	0.25	0.5	1.0
- 0.0	0.25	0.5	1.0
- 0.25	0.5	1.0	1.0
- 0.25	0.5	1.0	1.0
- 0.25	0.25	1.0	1.0
- 0.25	0.5	1.0	1.0
- 0.0	0.5	1.0	1.0
- 0.25	1.0	1.0	1.0

- P(forecast peak < ... )

- P(observed peak < ...)

- 100 cfs	200	300	400
- 0	1	1	1
- 0	0	1	1
- 0	0	0	1
- 0	0	0	0
- 0	0	0	1
- 1	1	1	1
- 0	1	1	1
- 0	0	1	1
- 0	0	1	1
- 0	0	1	1
- 0	0	1	1
- 0	1	1	1

## RPS with sample data set

- YEAR
- 1981
- 1982
- 1983
- 1984
- 1985
- 1986
- 1987
- 1988
- 1989
- 1990
- 1991
- 1992

- OBS
- 112
- 206
- 301
- 516
- 348
-  98
- 156
- 245
- 233
- 248
- 227
- 167

- Step 4: Compute Probabilities for observations. Since there is only one observation the probability will be either 0 or 1.

- 100 cfs	200	300 	400
- 1.0	1.0	1.0	1.0
- 0.25	0.5	1.0	1.0
- 0.25	0.5	1.0	1.0
- 0.0	0.0	0.25	0.5
- 0.0	0.25	0.5	1.0
- 0.0	0.25	0.5	1.0
- 0.25	0.5	1.0	1.0
- 0.25	0.5	1.0	1.0
- 0.25	0.25	1.0	1.0
- 0.25	0.5	1.0	1.0
- 0.0	0.5	1.0	1.0
- 0.25	1.0	1.0	1.0

- P(forecast peak < ... )

- P(observed peak < ...)

- 100 cfs	200	300	400
- 0	1	1	1
- 0	0	1	1
- 0	0	0	1
- 0	0	0	0
- 0	0	0	1
- 1	1	1	1
- 0	1	1	1
- 0	0	1	1
- 0	0	1	1
- 0	0	1	1
- 0	0	1	1
- 0	1	1	1

## RPS with sample data set

- YEAR
- 1983

- Step 5: Compute the RPS. 1983 is done as an example...

- 100	200	300	400
- 0.25	0.5	1.0	1.0

- P(forecast peak < ... )

- P(observed peak < ...)

- 100	200	300	400
- 0.0	0.0	0.0	1.0

- RPS =	[0.25 – 0.0]2 + [0.5 – 0.0]2 +
- 	[1.0 – 0.0]2 + [1.0 - 1.0]2
-         = 1.31

## RPS with sample data set

- YEAR
- 1981
- 1982
- 1983
- 1984
- 1985
- 1986
- 1987
- 1988
- 1989
- 1990
- 1991
- 1992

- RPS
- 1.0
- 0.31
- 1.31
- 0.32
- 0.31
- 1.81
- 0.31
- 0.31
- 0.13
- 0.31
- 0.25
- 0.06

- Step 5 (con’t): RPS computed for all years...

- 100	200	300	400
- 1.0	1.0	1.0	1.0
- 0.25	0.5	1.0	1.0
- 0.25	0.5	1.0	1.0
- 0.0	0.0	0.25	0.5
- 0.0	0.25	0.5	1.0
- 0.0	0.25	0.5	1.0
- 0.25	0.5	1.0	1.0
- 0.25	0.5	1.0	1.0
- 0.25	0.25	1.0	1.0
- 0.25	0.5	1.0	1.0
- 0.0	0.5	1.0	1.0
- 0.25	1.0	1.0	1.0

- P(forecast peak < ... )

- P(observed peak < ...)

- 100	200	300	400
- 0.0	1.0	1.0	1.0
- 0.0	0.0	1.0	1.0
- 0.0	0.0	0.0	1.0
- 0.0	0.0	0.0	0.0
- 0.0	0.0	0.0	1.0
- 1.0	1.0	1.0	1.0
- 0.0	1.0	1.0	1.0
- 0.0	0.0	1.0	1.0
- 0.0	0.0	1.0	1.0
- 0.0	0.0	1.0	1.0
- 0.0	0.0	1.0	1.0
- 0.0	1.0	1.0	1.0

## RPS with sample data set

- YEAR
- 1981
- 1982
- 1983
- 1984
- 1985
- 1986
- 1987
- 1988
- 1989
- 1990
- 1991
- 1992

- RPS
- 1.0
- 0.31
- 1.31
- 0.31
- 0.31
- 1.81
- 0.31
- 0.31
- 0.13
- 0.31
- 0.25
- 0.06

- Step 6: Now we can make statements such as...

- 1986 was worst forecast (RPS was highest)
- 1992 was best forecast (RPS lowest)
- What else can you say about the forecasts?

## Ranked Probability Skill Score (RPSS)

- RPS

- RPS

*f*

*cl*

- =  Rank Probability Score (forecasts)

- =  Rank Probability Score (climatology)

**RPS****f**** and RPS****cl**** must be calculated with the same bins!**

## RPSS with sample data set

- YEAR
- 1981
- 1982
- 1983
- 1984
- 1985
- 1986
- 1987
- 1988
- 1989
- 1990
- 1991
- 1992

- RPSfor
- 1.0
- 0.31
- 1.31
- 0.31
- 0.31
- 1.81
- 0.31
- 0.31
- 0.13
- 0.31
- 0.25
- 0.06

- Step 1: Take the RPS vector calculated in the RPS sample section and call it Forecast RPS or RPSfor.

## RPSS with sample data set

- YEAR
- 1981
- 1982
- 1983
- 1984
- 1985
- 1986
- 1987
- 1988
- 1989
- 1990
- 1991
- 1992

- RPSfor
- 1.0
- 0.31
- 1.31
- 0.31
- 0.31
- 1.81
- 0.31
- 0.31
- 0.13
- 0.31
- 0.25
- 0.06

- Step 2: Calculate a reference RPS vector. This may be a climatology RPS that used the climatological values as forecasts. Call it RPSclim.

- RPSclim
- 1.83
- 1.33
- 1.50
- 3.00
- 1.33
- 2.00
- 1.25
- 1.00
- 1.08
- 0.83
- 1.67
- 1.67

## RPSS = 1 – 0.54/1.46

- YEAR
- 1981
- 1982
- 1983
- 1984
- 1985
- 1986
- 1987
- 1988
- 1989
- 1990
- 1991
- 1992

- RPSfor
- 1.0
- 0.31
- 1.31
- 0.31
- 0.31
- 1.81
- 0.31
- 0.31
- 0.13
- 0.31
- 0.25
- 0.06

- Step 3: Apply the formula, RPSS = 1 – mean(RPSfor)/mean(RPSclim). Note this formula is equalivent to the one a few slides back.

- RPSclim
- 1.83
- 1.33
- 1.50
- 3.00
- 1.33
- 2.00
- 1.25
- 1.00
- 1.08
- 0.83
- 1.67
- 1.67

-            = +0.63
- In words, this means the forecasts are 63% better than climatology!

## Summary

- Deterministic Forecasts
- Contingency tables
- Sample Forecast
- Statistics
- POD, FAR, etc
- ROC Area
- Ensemble Forecasts
- Brier Score
- Skill Scores
- Ranked Probability Score
- Sample Forecast calculations