Guideline on the exchange and use of EPS verification results
Update date: 21 Mar 2024
1. Introduction
World Meteorological Organization (WMO) CBS-XIII (2005) recommended thatthe general responsibilities for a Lead Centre for Verification of EPS be added to the

Manual on the WMO Integrated Processing and Prediction System (WIPPS)(WMO-No.485) 1 and that the President designate RSMC Tokyo (JapanMeteorological Agency; JMA) as the Lead Centre.
JMA hosts two Internet sites for the Lead Centre works; a Web site and a FTPsite.

The FTP site is used for EPS producing centres to upload their statistics ofEPS verification and the Web site is used to publish them.

The Internet sites

enable EPS producers to make their own statistics of EPS verification open to theWMO Members. The Internet sites also enable the WMO Members to obtain not
only the statistics but also their visualized images.
This guideline is organized as follows. Section 2 describes definitions of theEPS verification. Section 3 describes how to record and read the statistics on theInternet sites for EPS producers and the WMO Members, respectively.

Section 4describes the rule of the report, and section 5 describes how to obtain the EPS
verification scores.

Section 6 describes how to post the statistics on the FTP site

only for EPS producers.
2. Definition of standard EPS verification
It is defined at the Appendix 2.2.35 of the Manual on the WIPPS (See AppendixA) that standard verification of an EPS are exchanged monthly to measure theforecast skill of ensemble mean, spread, continuous ranked probability score(CRPS) and probability with respect to the analysis and/or climatology. The factors
and definitions of the measure are also explained in Table 1.
There are many ways of constructing an EPS and the CBS Expert Team on
Operational Weather and Forecasting Process and Support made it clear that theintention is to verify the complete system as used operationally, and in the way inwhich the producing centre would advise its users to use the EPS. Where
time-lagged or multi-model ensemble combinations are used, the full ensemble
should be verified as used operationally. For example, if a producing centre
recommends combining ensemble members from two or more cycles of its ensemble
1 Available at https://library.wmo.int/idurl/4/35703

to form a larger time-lagged ensemble, then it is this larger time-lagged ensemble
which should be verified (rather than the latest cycle alone). (Note, nevertheless,that it is the raw ensemble which should be verified and not any post-processed orcalibrated output.)
3. The statistics of EPS verification
3.1 The verification table of ensemble mean and spread
The verification table, called the deterministic table here, is defined as follows:
A header line, which starts with “#”, should be put at the head of the deterministic
table. The header line should consist of 5 elements as follows;# <EPS centre name> <Element> <Year> <Month> <Area>
where <term> indicates a keyword which is listed up in Table 2.
The first, second, third, fourth and fifth column of the table following the headerline indicates the number, forecast time, anomaly correlation coefficient of theensemble mean, root-mean-square (RMS) error of the ensemble mean and spread(standard deviation), respectively.
Some spaces, at least one, are necessary among each column as a separator.
A comment line, which starts with “!”, can be put anywhere.
3.2 Reliability tableThe table is complied with Standardized Verification System (SVS) for long-range
forecasts (LRF), as indicated on the web-site of the Lead Centre for the Long Range
Forecast Verification System2.

The table is quoted as Table 3, below.

A header line, which starts with “#”, should be put at the head of the reliability
table. The header line should consist of 7 or 8 elements as follows;#<EPS centre name><Element><Threshold><Year><Month><Area><Forecast time> <Proxy (PR24 only)>
where <term> indicates a keyword which is listed up in Table 2. The eighth element,<Proxy (PR24 only)>, is not used for elements except for the verification ofprecipitation.
The first, second, third, fourth and fifth column of the table following the header2

http://www.bom.gov.au/wmo/lrfvs/roc.shtml

line indicates the bin number, the number of members forecasting occurrence of theevent, the number of members forecasting non occurrence of the event, observed
occurrences and observed non-occurrences, respectively.
Some spaces, at least one, are necessary among each column as a separator.
A comment line, which starts with “!”, can be put anywhere.
3.3 The verification table of continuous rank probability score (CRPS)
The verification table, called the CRPS table here, is defined as follows: A headerline, which starts with “#”, should be put at the head of the CRPS table.

Theheader line should consist of 5 or 6 elements as follows;# <EPS centre name> <Element> <Year> <Month> <Area><Proxy (PR24only)>
where <term> indicates a keyword which is listed up in Table2. The sixth element,<Proxy (PR24 only)>, is not used for elements except for the verification ofprecipitation.
The first, second, third and fourth column of the table following the header line
indicates the number, forecast time, CRPS for EPS forecast distribution, CRPS forthe control forecast in EPS, respectively.

CRPS for deterministic forecast is equal

to the mean absolute error. If CRPS for the control forecast in EPS is not being
reported, the fourth column has to be set “-1” instead.
Some spaces, at least one, are necessary among each column as a separator.
A comment line, which starts with “!”, can be put anywhere.
4. Rule of the report on the verification table
The reliability table, the CRPS table and the deterministic table should include
verification results only for one EPS reported with an ascii file(s)(See Appendix B, Cand D). The file name of the verification table should comply with the following
rule;- in case of reporting verification results for own EPS,
<Reporting centre name>_<Classification>_<Year><Month>.txt- otherwise,<Reporting centre name>_<Classification>_<Year><Month>.txt.for_<EPS centre
name>where <term> indicates a keyword which is listed up in Table 2.

If EPS producer would like to add and/or update the verification table after
reporting it, she/he has to overwrite it using the same file name.
5. Climatology
The climatology data used in the verification measures are available from the ftp
server of the WMO Lead Centre for Deterministic Forecast Verification3.
6. Stations list for verification of precipitation against observations
Verification of precipitation is recommended to be performed against
observations, which are based on the GCOS list of surface networks (GSN). The
latest GSN stations list is available from the OSCAR/Surface website4. You can get
the list in JSON format by accessing the following URL:
https://oscar.wmo.int/surface/rest/api/search/station?programAffiliation=GSN
EPS producing centres shall have the right to omit certain observation sites if thesefail a quality control.
Verification of precipitation may alternatively be against a proxy analysis. For
more details, please refer to the Manual on the WIPPS (See Appendix A).
7. How to get the verification table
JMA operates a Web site for both exchanging the statistics and publishing them.
The name and address of the Web site are EPSV and https://epsv.kishou.go.jp/EPSv/,
respectively.
8. How to post the verification table
JMA operates a FTP site which is used for EPS producers to report theirstatistics of EPS verification.

The name and address of the ftp site are FTPEPSV

and ftp://ftpepsv.kishou.go.jp, respectively.
8.1 Registration
The ftp site is allowed to access only from the registered site.
3

For more details, please see https://wmolcdnv.ecmwf.int/.

4

https://oscar.wmo.int/surface/index.html

The EPS

producer should notify the administratorof theglobal IP address(es) (for example, 11.22.33.44) of her/his own site.
JMA will send the EPS producer the login information of FTPEPSV, such as user
name, at least in one month after the notification.
8.2 Post
The EPS producer is able to login FTPEPSV with the user name and thepassword provided.

She/He put the monthly verification table on her/his home

directory of FTPEPSV.
The verification table is available by accessing EPSV (see section 7) in a day
after they are put on FTPEPSV.

And, old date verification tables on FTPEPSV

may be deleted due to the limitation of disk resources.

Table 1: Factors and definitions used in the EPS verification

1 m
Fi , jm j1

Ensemble mean~FiSpread (StandardDeviation)

1 m
Fi , jm j1~ 2FiRoot Mean Square Error

ei2

Mean Absolute Error| ei |
nAnomaly Correlation
Coefficient

fi

fi

ai

ai cosii 1nfi

2

fi

cosni

i 12

ai

ai

cosi

i 1

Brier Score

Oi ) 2( Pi2Continuous Rankˆ ( x ) dx †Pˆi ( x ) Oi
Probability Score
where Fi: forecast value, Ai: corresponding verifying value,Ci: corresponding climatological value,Pi: probability, Oi: {observed:1, or not-observed:0},m : the ensemble size,n : the number of grid points in the verification area,
cos i : cosine of latitude of grid point iei

FiPˆi ( x )

Ai ,

fi

1 m
H (xm j1Fi

Ci ,Fi , j ),

Ai

Ci ,

ˆ (x) H(xOi

Ai ),0 ( x 0),1 ( x 0)

H (x)
nXiai

X i cosii 1n.
cosii 1

† E.g. H. Hersbach, 2000: Decomposition of the continuous ranked probability scorefor ensemble prediction systems. Wea. Forecasting, 15, 559-570

Table 2 : List of keywords
ElementKeyword (Bold)<Reportin BOM,CMA,CMC,CPTEC,DW
g centre D,ECMWF,JMA,KMA,MF,NC
name>
EP,NCMRWF,RUMS,UKMO<EPS
BOM,CMA,CMC,CPTEC,DWcentre
D,ECMWF,JMA,KMA,MF,NC
name>
EP,NCMRWF,RUMS,UKMO<Classific RELTBL , ACCERRSPD oration>
CRPS<Element><Threshold>Abbreviation of reporting centre
Abbreviation of EPS producer
RELTBL
ACCERRSPD

The reliability table
The scores of ensemble
mean forecast and thespread
CRPS
Thecontinuousrank
probability
scoresofensemble forecast anddeterministic forecast
Z500anm
Z500anm Anomaly for 500hPa
T850anm
geopotential heightPMSLanm(unit:m)
TSFCanm
T850anm Anomaly for 850hPa
Z500
temperature (unit:K)
T850
PMSLanAnomaly for mean
PMSLm
sea-level pressure
W850(unit:hPa)
U850
TSFCanm Anomaly for 2-metre
V850
temperature (unit:K)
U250
Z500
500hPa geopotential
V250
height (unit:m)
PR24
T850
850hPa temperature
TSFC(unit:K)
WSFC
PMSL
Mean sea-level pressure(unit:hPa)
W850
850hPa wind speed(unit:m/s)
U(V)850
850hPa u(v) windcomponent (unit:m/s)
U(V)250250hPa u(v) windcomponent (unit:m/s)
PR2424-hour accumulated
precipitation (unit:mm)
TSFC
2-metre temperature(unit:K)
WSFC
10-metre wind speed(unit:m/s)
Z500anm,gt1sd,gt1.5sd, gt
Abbreviation of ‘greater than’T850anm,gt2sd,lt-1sd,lt lt
Abbreviation of ‘less than’
PMSLanm or -1.5sd orsd
Abbreviation of ‘Standard
TSFCanmlt-2sddeviation’W850<Year><Month><Area><Forecasttime><Proxy(PR24only)>

gt10mps,gt15 mps
Abbreviation of ‘m/s’mpsor
gt25mps
WSFC
gt10mpsorgt15mps
U850, V850, gt10P,
P
Abbreviation of ‘percentile
U250 or V250 gt25P,gt75P
point of climatological chanceor gt 90P(%)’
PR24
gt1mm,gt5m mm
Unit of precipitationm, gt10mm or
gt25mm2003, 2004, …
(Numeric) Target year
01, 02, 03, 04, 05, 06, 07, 08, (Numeric) Target month09, 10, 11 or 12
NH, SH or TR
NH
Northern
Hemisphere
extra-tropics (20N-90N)
SH
Southern
Hemisphere
extra-tropics (20S-90S)
TR
Tropics (20S-20N)
24, 48, 72, 96, 120, …
(Numeric) forecast time (unit:hour)
PROXY or blank

This keyword is used when theverificationofprecipitation
isperformed against a proxy analysis.
Otherwise, leave this field blank.

Table 3: The definition of reliability table (Source: Lead Centre for the Long-Range
Forecast Verification System)
General ROC contingency table for probabilistic forecasts of binary events withdefinitions of the different parameters. This contingency table applies when probability
thresholds are used to define the different probability bins

where,n = number of the nth probability interval or bin n; n goes from 1 to N.
Pn = lower probability limit for bin n.
Pn+1 = upper probability limit for bin n.
N = number of probability intervals or bins.

(O) being 1 when an event corresponding to a forecast in bin n, is observed as anoccurrence; 0 otherwise. The summation is over all forecasts in bin n, at all grid points i
or stations i.

(NO) being 1 when an event corresponding to a forecast in bin n, is not observed; 0otherwise. The summation is over all forecasts in bin n, at all grid points i or stations i.
Wi = 1 when verification is done at stations or single grid points; Wi = cos(Θi) atgrid-point i, when verification is done on a grid; Θi the latitude at grid point i.

Appendix A : The standard verification measures of EPS (Source: the Appendix

2.2.35 of the Manual of the WIPPS (2023 edition))

Appendix B : A sample of the content of a deterministic table--- start --# JMA PMSL 2003 10 NH
1 24 0.977 1.727 2.126
2 48 0.944 2.673 2.929
3 72 0.892 3.659 3.791
4 96 0.829 4.566 4.673
5 120 0.755 5.374 5.482
6 144 0.681 6.024 6.179
7 168 0.606 6.552 6.697
8 192 0.523 7.040 7.029
9 216 0.446 7.405 7.316! comment# JMA T850 2003 10 NH
1 24 0.969 1.032 1.104
2 48 0.936 1.477 1.493
3 72 0.893 1.893 1.907
4 96 0.840 2.283 2.338
5 120 0.774 2.671 2.726
6 144 0.705 3.008 3.054
7 168 0.634 3.293 3.295
8 192 0.567 3.509 3.451
9 216 0.506 3.676 3.563--- end ---Appendix C : A sample of the content of a reliability table--- start --# JMA PMSLanm gt1.5sd 2003 10 NH 24
1 0 25 22.05 55781.09
2 1 24 22.3 2227.46
3 2 23 26.95 1167.75
4 3 22 30.56 728.55
5 4 21 29.36 567.3
6 5 20 30.33 453.26
7 6 19 28.95 371.56
8 7 18 36.06 314.66
9 8 17 32.78 242.68
10 9 16 34.58 228.72
11 10 15 49.14 216.17
12 11 14 50.52 206
13 12 13 55.21 166.8
14 13 12 54.14 124.92
15 14 11 64.04 133.85
16 15 10 70.51 97.28
17 16 9 83.12 80.33
18 17 8 77.79 72.42
19 18 7 84.46 68.86
20 19 6 118.51 54.69
21 20 5 110.96 58.49
22 21 4 149.15 54.08
23 22 3 178.97 56.57
24 23 2 223.6 55.37
25 24 1 304.71 42.21
26 25 0 1706.48 57.91# JMA PMSLanm gt1.5sd 2003 10 NH 48
1 0 25 44.16 53376.69
2 1 24 53.03 2873.91
3 2 23 49.89 1417.19
4 3 22 35.38 960.68
5 4 21 35.32 703.11

6 5 20 43.64 581.41
7 6 19 46.73 469.66
8 7 18 52.89 424.78
9 8 17 45.92 350.8
10 9 16 44.21 336.88
11 10 15 59.71 272.31
12 11 14 63.77 230.68
13 12 13 74.05 208.44
14 13 12 67.93 206.62
15 14 11 80.56 181.75
16 15 10 89.66 165.36
17 16 9 105.09 133.48
18 17 8 109.73 135.43
19 18 7 122.6 109.76
20 19 6 125.97 110.21
21 20 5 160.89 82.79
22 21 4 187.03 81.05
23 22 3 242.3 75.05
24 23 2 258.44 51.92
25 24 1 378.68 47.92
26 25 0 1097.7 41.37--- end ---Appendix D : A sample of the content of a CRPS table--- start --# JMA Z500 2009 01 NH
1 24 5.328 7.095
2 48 9.639 13.141
3 72 14.797 20.579
4 96 20.439 28.571
5 120 26.131 36.681
6 144 31.698 44.742
7 168 36.586 51.677
8 192 40.981 57.883
9 216 44.582 63.157!comment# JMA Z500 2009 01 SH
1 24 4.657 -1
2 48 8.545 -1
3 72 12.980 -1
4 96 17.446 -1
5 120 22.148 -1
6 144 26.941 -1
7 168 31.453 -1
8 192 35.108 -1
9 216 37.920 -1--- end ---