# Presentation: 075155

**  **
**Simplifying the fvs statistical calculation **
**and plotting process**
- Jeff McQueen and Perry Shafran NCEP/EMC

## Overview

*** ******core script******: make-fvs.sh****** on tempest & Dew:***
**tempest:**** /export-4/tempest/wd22jm/fvs/scripts**
**dew:**** ****/meso/save/wx22mc/jeff/FVS-SCRIPTS**
*** ******Summarizes vsdb records for user selected dates, regions, variables***
*fvs s**  step*
**Computes error statistics (eg: rmse, bias...) and plots error statistics averaged by user selected bins:**** **
*fvs p**  step*
**Diurnal plots averaged by forecast hour:  ****DIU**
**Daily Time Series plots for a particular forecast hour: ****DTS**
**Upper air plots averaged by multiple vertical levels: ****UPA**
**Plots of statistics averaged by event thresholds: ****FHO**

**Notes:**
An underutilzed team of world class scientists who are developing, maintaining and evaluating the current regional operational system while working with OAR, NESDIS, NASA and university scientists to develop data assimilation, global and inline-interactive systems

## FVS Scripts & Files

*** ******Scripts located at :***
**tempest:**** ****/export-4/tempest/wd22jm/fvs/scripts**
**dew:**** ****/meso/save/wx22mc/jeff/FVS-SCRIPTS**
*Copy the following files**: *
*make-fvs.sh*
*cr8trace.sh** **** called by make-fvs.sh*
- FVS.CTL  Control file for user sets (varbs, models...) or can be created thru cr8cfg.sh  and runfvs.sh master script
*cr8cfg.sh**  **** Create FVS.CTL from exported fields set in master script (eg: runfvs.sh)*
*runfvs.sh** **** Example master script that calls :  cr8cfg.sh and *
*                        **make-fvs.sh*
*RLBL.CFG **** Region label configuration file*
*SLBL.CFG **** Statistics label configuration file*

**Notes:**
An underutilzed team of world class scientists who are developing, maintaining and evaluating the current regional operational system while working with OAR, NESDIS, NASA and university scientists to develop data assimilation, global and inline-interactive systems

## Make-fvs.sh settings

*** *****Command line: plot-type and forecast cycle**
_**make-fvs.sh  DTS 12**__**  **_**:**
* **creates daily time series plot for 12 UTC cycle*
**Plottype:  DTS, DIU, UPA, FHO**
**Forecast cycle: 00, 06, 12, 18 or nothing for all cycles**
**	**
*** *****CTL file or export variables from master script**
- FVS.CTL
- or
-     export TRACES=“NAM/218 NAMY/218”
-     export PLOTHRS=84
-     export stno=“1101 1102 1 2”
-     export PLVL=SFC
-     export RGN=G236
-     export VARB=T
-     export appstat=1
-     $scrdir/cr8cfg.sh  - creates an FVS. CTL file
-    $scrdir/make-fvs.sh DTS $cycle

## Key fields for FVS

*** ******Model(s) to verify****** ***
*eg:  NAM/218, GFS/212, SRMEAN/212*
*** ******Region(s) to verify***
*eg: G236, G216, G104/NEC...*
**Variable(s) to verify:**
*eg: T, VWND, RH, DPT, Z, SLP, APCP/24, TOTCLD, AOD, smoke, OZON1, OZON8,OZMX1, OZMX8...*
**Vertical level(s) for each variable**
*eg: SFC, P850, P700, P850-P700, ATMOS...*
**Forecast hour(s) to verify**
*eg:  36, 48, 84*
**Date/Time range:**
*Eg: 200807280000 – 200808272359*
**FVS Statistics code to compute and plot:**
*Eg: 1101 (bias), rmse (1102)...*

## DTS : DAILY TIME SERIES PLOTS for a Particular Forecast Hour NAM vs NAM-Y 2 m Temperature Errors (C)

- 36h Forecast for 12 Z cycle runs (Valid 00 Z)

**Error vs Date**

## UPPER AIR PLOTS for a particular forecast hour: UPANAM vs NAM-Y Temperature Errors (C)

- 36h Forecast (Valid 00 Z)

**Pressure Level**
**Vs error**

## Errors Averaged by Forecast hours for all Dates: DIUrnal plotsNAM-Y vs NAM RH errors (Aug. 2007)

- 2 m Relative Humidity Error (%)

**Error vs**
**Forecast **
**hour**

## Errors Averaged by threshold event for all dates: FHONAM vs SREF CSI 48 hour precip errors

**Error vs**
**threshold**

## Settings to plot errors for different regions from two different models (NAM & NAMEXPR)

*** ***FVS. CTL file format:
***======================================================***
- NAM/218 NAMEXPR/218 NAM/218      NAMEXPR/218   NAM/218        NAMEXPR/218 =
*G236        G236                G104/SWC   G104/SWC          G104/GMC     G104/GMC* = RGN
- SLP          SLP                  SLP              SLP                      SLP                SLP     = VARB
- SFC         SFC                  SFC              SFC                     SFC                SFC    = PLVL
- 36            36                     36                 36                        36                    36       =  FHR
- 1 1101 1102 =    appstat  & statistics codes
- 200708050000 200708182359 = Date Range
***======================================================***
- Will create 6 traces but 12 time-series
_***appstat=1 : put all statistics on one plot***_
_***             ***__***=0: make separate plot files for each stat type (1101 and 1102)***_
- NAM/218 SLP for CONUS region (G236) at the SFC for 36 hour forecasts from 8/5/2007- 8/1/8/2007  for stats 1101 (rmse) and 1102 (bias)
- NAMEXPR/218 SLP for CONUS region (G236) at the SFC for 36 hour forecasts from 8/5/2007- 8/1/8/2007  for stats 1101 (rmse) and 1102 (bias)
- NAM/218 SLP for SW Coast region (SWC) at the SFC for 36 hour forecasts from 8/5/2007- 8/1/8/2007  for stats 1101 (rmse) and 1102 (bias)
- NAMEPR/218 SLP for SW Coast region (G236) at the SFC for 36 hour forecasts from 8/5/2007- 8/1/8/2007  for stats 1101 (rmse) and 1102 (bias)
- NAM/218 SLP for NW Coast region (NWC) at the SFC for 36 hour forecasts from 8/5/2007- 8/1/8/2007  for stats 1101 (rmse) and 1102 (bias)
- NAMEXPR/218 SLP for NW Coast region (NWC) at the SFC for 36 hour forecasts from 8/5/2007- 8/1/8/2007  for stats 1101 (rmse) and 1102 (bias)

## Resulting trace.ctl file for fvs s/p step

- 1 2 0 2 1 1 1 0/0/0/0
- 18 0 0 NAM/218 0
- 2 2 0 2 1 1 1 0/0/0/0
- 18 0 0 NAMEXPR/218 0
- 3 2 0 2 1 1 1 0/0/0/0
- 18 0 0 NAM/218 0
- 4 2 0 2 1 1 1 0/0/0/0
- 18 0 0 NAMEXPR/218 0
- 5 2 0 2 1 1 1 0/0/0/0
- 18 0 0 NAM/218 0
- 6 2 0 2 1 1 1 0/0/0/0
- 18 0 0 NAMEXPR/218 0
- 1 3 0 2 1 1 1 0/0/0/0
- 9 36 0 0 0
- 2 3 0 2 1 1 1 0/0/0/0
- 9 36 0 0 0
- 3 3 0 2 1 1 1 0/0/0/0
- 9 36 0 0 0
- 4 3 0 2 1 1 1 0/0/0/0
- 9 36 0 0 0
- 5 3 0 2 1 1 1 0/0/0/0
- 9 36 0 0 0
- 6 3 0 2 1 1 1 0/0/0/0
- 9 36 0 0 0
- 1 15 0 2 1 1 1 0/0/0/0
- 18 0 0 0000 0
- 2 15 0 2 1 1 1 0/0/0/0
- 18 0 0 0000 0
- 3 15 0 2 1 1 1 0/0/0/0
- 18 0 0 0000 0
- 4 15 0 2 1 1 1 0/0/0/0
- 18 0 0 0000 0
- 5 15 0 2 1 1 1 0/0/0/0
- 18 0 0 0000 0
- 6 15 0 2 1 1 1 0/0/0/0
- 18 0 0 0000 0
- 1 14 0 2 1 1 1 0/0/0/0
- 18 0 0 SFC 0
- 2 14 0 2 1 1 1 0/0/0/0
- 18 0 0 SFC 02 2 0 2 1 1 1 0/0/0/0
- 3 14 0 2 1 1 1 0/0/0/0
- 18 0 0 SFC 04 14 0 2 1 1 1 0/0/0/0
- 18 0 0 SFC 0
- 5 14 0 2 1 1 1 0/0/0/0
- 18 0 0 SFC 0
- 6 14 0 2 1 1 1 0/0/0/0
- 18 0 0 SFC 0
- 1 8 0 2 1 1 1 0/0/0/0
- 18 0 0 ONLYSF 0
- 2 8 0 2 1 1 1 0/0/0/0
- 18 0 0 ONLYSF 0

- 3 14 0 2 1 1 1 0/0/0/0
- 18 0 0 SFC 0
- 4 14 0 2 1 1 1 0/0/0/0
- 18 0 0 SFC 0
- 5 14 0 2 1 1 1 0/0/0/0
- 18 0 0 SFC 0
- 6 14 0 2 1 1 1 0/0/0/0
- 18 0 0 SFC 0
- 1 8 0 2 1 1 1 0/0/0/0
- 18 0 0 ONLYSF 0
- 2 8 0 2 1 1 1 0/0/0/0
- 18 0 0 ONLYSF 0
- 3 8 0 2 1 1 1 0/0/0/0
- 18 0 0 ONLYSF 0
- 4 8 0 2 1 1 1 0/0/0/0
- 18 0 0 ONLYSF 0
- 5 8 0 2 1 1 1 0/0/0/0
- 18 0 0 ONLYSF 0
- 6 8 0 2 1 1 1 0/0/0/0
- 18 0 0 ONLYSF 0
- 1 9 0 2 1 1 1 0/0/0/0
- 18 0 0 G236 0
- 2 9 0 2 1 1 1 0/0/0/0
- 18 0 0 G236 0
- 3 9 0 2 1 1 1 0/0/0/0
- 18 0 0 G104/SWC 0
- 4 9 0 2 1 1 1 0/0/0/0
- 18 0 0 G104/SWC 0
- 5 9 0 2 1 1 1 0/0/0/0
- 18 0 0 G104/GMC 0
- 6 9 0 2 1 1 1 0/0/0/0
- 18 0 0 G104/GMC 0
- 1 10 0 2 1 1 1 0/0/0/0
- 18 0 0 SL1L2 0
- 2 10 0 2 1 1 1 0/0/0/0
- 18 0 0 SL1L2 0
- 3 10 0 2 1 1 1 0/0/0/0
- 18 0 0 SL1L2 0
- 4 10 0 2 1 1 1 0/0/0/0
- 18 0 0 SL1L2 0
- 5 10 0 2 1 1 1 0/0/0/0
- 18 0 0 SL1L2 0
- 6 10 0 2 1 1 1 0/0/0/0
- 18 0 0 SL1L2 0
- 1 12 0 2 1 1 1 0/0/0/0
- 18 0 0 SLP 0
- 2 12 0 2 1 1 1 0/0/0/0
- 18 0 0 SLP 0
- 3 12 0 2 1 1 1 0/0/0/0
- 18 0 0 SLP 0

- 4 10 0 2 1 1 1 0/0/0/0
- 18 0 0 SL1L2 0
- 5 10 0 2 1 1 1 0/0/0/0
- 18 0 0 SL1L2 0
- 6 10 0 2 1 1 1 0/0/0/0
- 18 0 0 SL1L2 0
- 1 12 0 2 1 1 1 0/0/0/0
- 18 0 0 SLP 0
- 2 12 0 2 1 1 1 0/0/0/0
- 18 0 0 SLP 0
- 3 12 0 2 1 1 1 0/0/0/0
- 18 0 0 SLP 0
- 4 12 0 2 1 1 1 0/0/0/0
- 18 0 0 SLP 0
- 5 12 0 2 1 1 1 0/0/0/0
- 18 0 0 SLP 0
- 6 12 0 2 1 1 1 0/0/0/0
- 18 0 0 SLP 0
- 1 5 1 2 1 1 1 0/0/0/0
- 15 0 0 200708050000 200708182359
- 2 5 1 2 1 1 1 0/0/0/0
- 15 0 0 200708050000 200708182359
- 3 5 1 2 1 1 1 0/0/0/0
- 15 0 0 200708050000 200708182359
- 4 5 1 2 1 1 1 0/0/0/0
- 15 0 0 200708050000 200708182359
- 5 5 1 2 1 1 1 0/0/0/0
- 15 0 0 200708050000 200708182359
- 6 5 1 2 1 1 1 0/0/0/0
- 15 0 0 200708050000 200708182359

## NAM vs NAMEXPR 36 hTemperature Errors by date for different regions 

## Example FVS.CTL files to plotmodel performance for multiple FHRs

- NAM/218 NAM/218 NAM/218 NAM/218 AM/218 NAM/218 =
- G236 G236 G236 G236 G236 G236 =
- T T T T T T =
- SFC SFC SFC SFC SFC SFC =
- 12 24 36 48 60 72 =
- 1 1101 1102 =
- 200712010000 200801282359 =

**FVS.CTL**

** ****FVS.CTL created with the following UNIX Shell ksh calls:**

- export appstat=1
- export TRACES=“NAM/218 NAM/218 NAM/218 NAM/218 NAM/218 NAM/218”
- export RGN= G236
- export PLOTHRS=“ 12 24 36 48 60 72”
- export PLVL=SFC
- export VARB=T
- export stno=“1101 1102 "
- $scrdir/cr8cfg.sh
- $scrdir/make-fvs.sh dts $cycle

## NAM T Performance by FHRWinter 07-08: Resulting Plot

## Example FVS.CTL files to plotmodel errors for different variables

- NAM/218 NAM/218 NAM/218 NAM/218 =
- G236 G236 G236 G236 =
- T DPT VWND SLP =
- SFC SFC SFC SFC =
- 00 00 00 00 =
- 0 1101 1102 =
- 200712010000 200802012359 =

**FVS.CTL**

** ****FVS.CTL created with the following UNIX Shell ksh calls:**

- export TRACES=“NAM/218 NAM/218 NAM/218 NAM/218”
- export RGN= G236
- export PLOTHRS=00
- export PLVL=SFC
- export VARB=“T DPT VWND SLP”
- export stno=“1101 1102 "
- $scrdir/cr8cfg.sh
- $scrdir/make-fvs.sh DIU $cycle

## NAM T, VWND, SLP, Z Performance by FHRWinter 07-08: Resulting Plot

## make-fvs.sh notes

***Any combination of fields can be plotted on one graph***
*models*
*regions*
*variables*
*vertical levels *
*forecast hours*
*** ******Statistics can be plotted on one plot with ***
*appstat=1  (cmb)*
*** ******Or on separate plots***
*appstat=0 (rmse, bias, etc)** *
**Output plot name: {VARB}{LVL}_{STAT}{FHR}_{MDL-TYPE}.gif**
*TSFC_cmb84_mean.gif*

## Make-fvs.sh advanced features

*** ******Region labels : edit RLBL.CFG***
*Eg: G236 CONUS*
*G104/NEC  North-East-Coast*
***Statistics labels : edit SLBL.CFG***
*Eg: 1101  Root Mean Squared Error*
* **Bias*
***Probabilistic plotting is set by Stat. type:***
*Eg: RHET **** Talagrand Ranked Historgram*
* **RHNT**** Equally Likely Ranked Histogram*
* **ESL1L2 **** Ensemble mean RMSE, Bias, statistical consistency*
*...*

## SLBL.CFG

- 2    mean  Mean
- 1001 bias  Bias
- 1002 ets   Equitable-Threat-Score
- 1003 csi   Critical-Success-Index
- 1004 far   False-Alarm-Ratio
- 1007 pod   Probability-of-Detection
- 1021 far   False-Alarm-Rate
- 1034 hit   Hit-Rate
- 1101 bias  BIAS
- 1102 rmse  Root-Mean-Squared-Error
- 1103 corr  Correlation
- 1107 sprd  Forecast Spread
- 1109 stdv  Standard-Deviation
- 1112 stdv  Standard-Deviation
- 1116 cons  Statistical-Constistancy
- 1201 bias  BIAS
- 1202 rmse  Root-Mean-Squared-Error
- 1207 sprd  Forecast Spread
- 1209 stdv  Standard-Deviation
- 1212 stdv  Standard-Deviation
- 1216 cons  Statistical-Constistancy
- 1600 RHNT  Equally-Likely-Histogram
- 1601 RHET  Talagrand-Ranked-Histogram
- 1605 OUTL  Outlier-Percent
- 1606 MRE   Missing-Rate-Error

## RLBL.CFG

- G236 CONUS
- G212 North-America
- G218 North-America
- G245 East-US
- G246 West-US
- G246 Cent-US
- SWC Southwest-Coast
- NWC Northwest-Coast
- NMT Northern-Mountains
- SMT Southern-Mountains
- SPL Southern-Plains
- NPL Northern-Plains
- MDW Midwest
- LMV Lower-Mississippi-Valley
- GMC Gulf-Mexico
- NEC Northeast
- SEC Southeast
- APL Appalachian-Mountains
- WCA Western-Canada
- ECA Eastern-Canada

## Verification Sub-domains

**includes both**
**LA, SD & SJV**

**Verification Sub-domains**

**Notes:**
Good improvement in 5X performance when Isoprene error was fixed on Aug. 5

## Models available for analysis tempest: /mmb/wd20er/vsdb/dataDescription: http://www.emc.ncep.noaa.gov/mmb/mmbpll/mmbverif/veriflog.html

- akarw	dgexx	edetnx	esatp	namcenthw		ndas	expprem	rsaspbc
- akem	eastarw	edetp	esatpbc	nameasthw		ndasexpr  prnmm         rsaspx
- Aknmm	eastem	edetpbc	esatpx	nameasthw_old	ndasr	prnmm_old  rsmh
- aknmm_old east	nmmedetpx  gdas	namexp		ndasx	rrasn	 ruc
- arwctl	east	nmm_old	ekfcc	gfs		namexpr	ndasy	 rrasnbc    rucx
- arwctlbc	ebmjc	ekfccbc	gfseo	namhihw		ngm	rrasnx	 srmean
- Arwctlx	ebmjcbc	ekfccx	hiarw	namhihw_old	nmmctl	rrasp	 srmeanbc
- arwn1	ebmjcx	ekfcn	hiem	namprhw		nmmctlbc	rraspbc	 srmeanx
- arwn1bc	ebmjn	ekfcnbc	hinmm	namprhw_old	nmmctlx	rraspx	 westarw
- arwn1x	ebmjnbc	ekfcnx	hinmm_old namr		nmmn1	rsasc	westem
- arwp1	ebmjnx	ekfcp	namwesthw 		nmmn1bc   rsascbc	westnmm
- arwp1bc	ebmjp	ekfcpbc	namwesthw_old		nmmn1x	rsascx	westnmm_old
- arwp1x	ebmjpbc	ekfcpx	nam	namx		nmmp1	rsasn
- centem	ebmjpx	esatn	nam40	namx40		nmmp1bc   rsasnbc
- centnmm	edetn	esatnbc	namakhw	namy		nmmp1x	rsasnx
- dgexo	edetnbc	esatnx	namakhw_old  ndas		prarw	rsasp

## SLIL2 Statistics types

- *************************************************************************
- * SL1L2 -- L1L2 norm statistics for scalars                             *
- *                                                                       *
- * Computation Code                      Action                          *
- *                                                                       *
- *      9002             Trace of data field values averaged over all pts*
- *        -n             Return raw data for field n (undivided sum)     *
- *        -1             Contributing record count at all trace points   *
- *         1             Return data from field 1 = N, the data count    *
- *         2             Return data from field 2 = forecast mean        *
- *         3             Return data from field 3 = observation mean     *
- *         4             Return data from field 4 = F * O mean           *
- *         5             Return data from field 5 = F ** 2 mean          *
- *         6             Return data from field 6 = O ** 2 mean          *
- *                                                                       *
- *       1101            Compute BIAS (S[S]AL1L2, SL1L2)                 *
- *       1102            Compute RMS error (S[S]AL1L2, SL1L2)            *
- *       1103            Compute Correlation (S[S]AL1L2, SL1L2)          *
- *       1104            Compute Covariance (S[S]AL1L2,SL1L2)            *
- *       1105            Compute Forecast Variance (S[S]AL1L2, SL1L2)    *
- *       1106            Compute Observation Variance (S[S]AL1L2, SL1L2) *
- *       1107            Compute Forecast Std Dev (S[S]AL1L2, SL1L2)     *
- *       1108            Compute Obs Std Dev (S[S]AL1L2, SL1L2)          *
- *       1109            Compute Error Std Dev (SAL1L2, SL1L2)           *
- *       1110            Compute BIAS + Error Std Dev (SAL1L2, SL1L2)    *
- *       1111            Compute BIAS - Error Std Dev (SAL1L2, SL1L2)    *
- *       1112            Compute BIAS**2                                 *
- *       1113            Compute -2*Covariance                           *
- *       1114            Compute Error Variance                          *
- *       1115            Compute Mean Squared error                      *
- *************************************************************************

## FHO Statistics types

- *************************************************************************
- * FHO --  Forecast, hit, observation rate statistics                    *
- *                                                                       *
- * Computation Code                      Action                          *
- *                                                                       *
- *      9002             Trace of data field values averaged over all pts*
- *        -n             Return raw data for field n (undivided sum)     *
- *        -1             Contributing record count at all trace points   *
- *         1             Return data from field 1 = N, the data count    *
- *         2             Return data from field 2 = F, fraction forecast *
- *         3             Return data from field 3 = H, fraction corret   *
- *         4             Return data from field 4 = O, fraction observed *
- *                                                                       *
- *       1001            Compute BIAS from FHO values                    *
- *       1002            Compute Equitable Threat Score from FHO values  *
- *       1003            Compute Critical Success Index from FHO values  *
- *       1004            Compute False Alarm Ratio from FHO values       *
- *       1005            Compute Postagreement from FHO values           *
- *       1006            Compute Prefigurance from FHO values            *
- *       1007            Compute Probability of Detection from FHO values*
- *       1008            Compute Threat Score from FHO values            *
- *       1009            Cumulative Forecast Prcp Vol % vs Thrshld       *
- *       1010            Cumulative Correctly Frcst Prcp Vol % vs Thrshld*
- *       1011            Cumulative Observed Prcp Vol % vs Thrshld       *
- *       1012            Forecast Prcp Vol % of Ttl vs Thrshld           *
- *       1013            Correctly Forecast Prcp Vol % of Ttl vs Thrshld *
- *       1014            Observed Prcp Vol % of Ttl vs Thrshld           *
- *       1015            Forecast Prcp # % of Ttl # vs Thrshld           *
- *       1016            Correctly Forecast Prcp # % of Ttl # vs Thrshld *
- *       1017            Observed Prcp # % of Ttl # vs Thrshld           *
- *       1018            Forecast # % of Ttl <= Thrshld vs Thrshld       *
- *       1019            Correctly Frcst # % of Ttl <= Thrshld vs Thrshld*
- *       1020            Observed # % of Ttl <= Thrshld # vs Thrshld     *
- *       1021            Compute False Alarm Rate (f-h)/(N-o)            *
- *       1022            Compute Conditional Miss Rate (o-h)/(N-f)       **                                                                       *
- *       1023            Compute Heidke Skill Score                      *
- *       1024            Compute Peirce-Hanssen-Kuipers Skill Score      *
- *       1025            Compute ODDS RATIO (OR)                         *
- *       1026            Compute log of ODDS RATIO                       *
- *       1027            Compute standard error of log of ODDS RATIO     *
- *       1028            Compute ODDS RATIO Skill Score                  *
- *       1029            Compute ODDS RATIO Skill Score Cubed            *
- *       1030            Compute log (ODDS RATIO) / STD ERROR of same    *
- *       1031            Area between ODDS RATIO ROC & chance ROC curves *
- *       1032            ROC curve parametric in OR for time series      *
- *       1033            Odds ratio Parameterized ROC area (OPRA) score  *
- *       1034            Frcst accuracy rate (hits and correct nonevents)*
- *       1035            Extreme Dependency Score (EDS)                  *
- *

- Any of the preceding codes may be computed from bias adjusted FHO    *
- *  values.  To do this, precede the 4-digit code number with one of     *
- *  the following bias adjustment specification prefixes:                *
- *                                                                       *
- *       101             Fedor Mesinger's dH/dF bias adjustment          *
- *       102             Conservation of Odds Ratio bias adjustment      *
- *       103             Fedor Mesinger's dH/dA bias adjustment          *
- *                                                                       *
- *  Example 1: -1020003 displays the bias adjusted hit count             *
- *  Example 2:  1021008 displays the bias adjusted threat score          *
- *                                                                       *
- *  Any of the preceding 10xx series that are N dependent may be         *
- *  computed using F+O-H from the Mth preceding threshold.  The first M  *
- *  thresholds use the original value of N, the total number.  Thres-    *
- *  hold must be the independent variable.  To do this, precede the      *
- *  10xx code with the following prefix:                                 *
- *                                                                       *
- *      20M   The F+O-H value from i-M threshold is used, M>0 and M<10   *
- *************************************************************************

## Probabilistic Stats

*** ****RHNT : Ranked Histogram Nearest Truth*
*Equally Likely Diagram (1600)*
*RHNT5 :  E-L diagrams for ensemble subsets *
- SREF/ETA, SREF/EKF, SREF/RSM, SREF/WRF
* **RHET: Ranked Histogram Equal to Truth*
*Talagrand Diagram (1600)*
*RHET5: Talagrand diagrams for ensemble subsets*
- Also can plot Outlier Percentage (1605) & Missing Rate Error(1606)
*ESL1L2: Partial sums for ensemble groups*
*rmse (1102) , bias (1101), spread (1107), statistical consistency(1116)*

## Ranked Histogram examples

- export TRACES=“SREFX/ETA SREFX/EKF SREFX/RSM SREFX/WRF”
- export RGN= G236
- export PLOTHRS=00
- export PLVL=P850
- export VARB=T
- export sttyp=RHET5  Talagrand diagram        export sttyp=RHNT5  equally likely
- export stno=1601                                                export stno=1600
- $scrdir/cr8cfg.sh
- $scrdir/make-fvs.sh dts $cycle

## ESL1L2 :rmse, bias, spread, statistical consistency

- export TRACES=“SREFX/ETA SREFX/EKF SREFX/RSM SREFX/WRF”
- export RGN= G236
- export PLOTHRS=00
- export PLVL=SFC
- export VARB=T
- export sttyp=ESL1L2
- export stno=1101 1102 1116 1107   bias, rmse, statistical cons., spread
- $scrdir/cr8cfg.sh
- $scrdir/make-fvs.sh dts $cycle

## Partitioned Brier Score (PBS)

- * =======================GENERAL USE CODES============================= *
- *                                                                       *
- *       1301            Compute Brier Score for weather risks           *
- *                                                                       *
- * ===================================================================== *
- * ======== Codes for three risk category PBS statistics =============== *
- *                                                                       *
- *       1302            Compute Fraction observed for NO RISK frcst     *
- *       1303            Compute Fraction observed for LOW RISK frcst    *
- *       1304            Compute Fraction observed for MOD RISK frcst    *
- *       1305            Compute Fraction observed for HIGH RISK frcst   *
- *       1306            Compute optimal Target Brier Score              *
- *                                                                       *
- * ===================================================================== *
- * =======================GENERAL USE CODES============================= *
- *                                                                       *
- *  For all non-zero risk/probability categories:                        *
- *       1307            Compute BIAS Score from PBS values              *
- *       1308            Compute Equitable Threat Score from PBS values  *
- *       1309            Compute False Alarm Rate from PBS values        *
- *       1310            Compute False Alarm Ratio from PBS values       *
- *       1311            Compute Postagreement from PBS values           *
- *       1312            Compute Prefigurance from PBS values            *
- *       1313            Compute Probability of Detection from PBS values*
- *       1314            Compute Threat Score from PBS values            *
- *                                                                       *
- * ===================================================================== *
- * ======== Codes for three risk category PBS statistics =============== *
- *                                                                       *
- *  For moderate + high risk categories:                                 *
- *       1315            Compute BIAS Score from PBS values              *
- *       1316            Compute Equitable Threat Score from PBS values  *
- *       1317            Compute False Alarm Rate from PBS values        *
- *       1318            Compute False Alarm Ratio from PBS values       *
- *       1319            Compute Postagreement from PBS values           *
- *       1320            Compute Prefigurance from PBS values            *
- *       1321            Compute Probability of Detection from PBS values*
- *       1322            Compute Threat Score from PBS values            *
- *                                                                       *
- *  For only high risk category:                                         *
- *       1323            Compute BIAS Score from PBS values              *
- *       1324            Compute Equitable Threat Score from PBS values  *
- *       1325            Compute False Alarm Rate from PBS values        *
- *       1326            Compute False Alarm Ratio from PBS values       *
- *       1327            Compute Postagreement from PBS values           *
- *       1328            Compute Prefigurance from PBS values            *
- *       1329            Compute Probability of Detection from PBS values*
- *       1330            Compute Threat Score from PBS values            *

- * =======================GENERAL USE CODES============================= *
- *                                                                       *
- *  Overall sums:                                                        *
- *       1331            Total number of forecast events (sum 2->M/2)*N  *
- *       1332            Total number observed (sum M/2+1->M)*N,         *
- *                       where M is the number of data field elements    *
- *                                                                       *
- *  For all non-zero risk/probability categories:                        *
- *       1333            Compute Heidke Skill Score                      *
- *       1334            Compute Peirce-Hanssen-Kuipers Skill Score      *
- *       1335            Compute ODDS RATIO                              *
- *       1336            Compute log of ODDS RATIO                       *
- *       1337            Compute Standard Error of log of ODDS RATIO     *
- *       1338            Compute ODDS RATIO Skill Score                  *
- *       1339            Compute ODDS RATIO Skill Score Cubed            *
- *                                                                       *
- * ===================================================================== *
- * ======== Codes for three risk category PBS statistics =============== *
- *                                                                       *
- *  For moderate + high risk categories:                                 *
- *       1340            Compute Heidke Skill Score                      *
- *       1341            Compute Peirce-Hanssen-Kuipers Skill Score      *
- *       1342            Compute ODDS RATIO                              *
- *       1343            Compute log of ODDS RATIO                       *
- *       1344            Compute Standard Error of log of ODDS RATIO     *
- *       1345            Compute ODDS RATIO Skill Score                  *
- *       1346            Compute ODDS RATIO Skill Score Cubed            *
- *                                                                       *
- *  For only high risk category:                                         *
- *       1347            Compute Heidke Skill Score                      *
- *       1348            Compute Peirce-Hanssen-Kuipers Skill Score      *
- *       1349            Compute ODDS RATIO                              *
- *       1350            Compute log of ODDS RATIO                       *
- *       1351            Compute Standard Error of log of ODDS RATIO     *
- *       1352            Compute ODDS RATIO Skill Score                  *
- *       1353            Compute ODDS RATIO Skill Score Cubed            *
- *                                                                       *
- * ===================================================================== *
- * =======================GENERAL USE CODES============================= *
- *                                                                       *
- *  For all PBS categories:                                              *
- *       1354            Compute sample climatological BS (uncertainty)  *
- *       1355            Compute Reliability term of BS decomposition    *
- *       1356            Compute Resolution term of BS decomposition     *
- *       1357            Compute Uncertainty term of BS decomposition    *
- *       1358            Compute Brier Skill Score against sample climo  *
- *       1359            Compute sample climatological frequency         *
- *       **1360            Attribute Diagram for time series of PBS data**   *
- *       1361            Compute Extreme Dependency Score (EDS)          *
- *************************************************************************