NOAA Technical Memorandum NWS SR – 226

___________________________________________________________________________

THE TEMPERATURE AND PRECIPITATION VERIFICATION 

PROGRAM AT WFO CORPUS CHRISTI TEXAS

Gregory E. Wilk

National Weather Service

Weather Forecast Office

Corpus Christi, Texas

Scientific Services Division

Southern Region

Fort Worth, Texas

November 2005

UNITED STATES National Oceanic and National Weather Service

DEPARTMENT OF COMMERCE Atmospheric Administration David L. Johnson, Assistant

Carlos M. Gutierrezs, Secretary Conrad C. Lautenbacher Administrator for Weather Services
This publication has been reviewed

and is approved for publication by

Scientific Services Division

Southern Region

David B. Billingsley, Chief

Scientific Services Division

Fort Worth, Texas

i
TABLE OF CONTENTS

____________________________________________________

PAGE

I. Inroduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

II. Brief History of the Corpus Christi Verification Program . . . . . . . . . . . . . . . . . . . . . 1

III. Temperature Verification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

IV. Precipitation Verification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

V. Forecaster Contests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

VI Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

VII. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

VIII. References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

ii
1

I. Introduction

This paper discusses the temperature and precipitation verification programs at Weather 

Forecast Office (WFO) Corpus Christi TX (CRP). Every month, station temperature and 

precipitation verification statistics are provided to the meteorologists at CRP. Cool season 

(October through March) and warm season (April through September) temperature and 

precipitation verification statistics are also computed and presented to the operational staff. 

For each of these six month periods, each forecaster receives his or her individual verification 

statistics. Finally, a temperature and precipitation forecast contest is conducted every six 

months, with the top three forecasters announced at the end of each contest. The goals of the 

verification program are to help provide the best temperature and precipitation forecasts for the 

Corpus Christi area of responsibility, and to improve forecasts by comparing individual

forecasts with MOS forecasts. Office temperature and precipitation verification statistics for

the past several months have been some of the best in the NWS Southern Region. 

II. Brief History of the Corpus Christi Verification Program

The temperature verification program began at WFO CRP in 1998, with Corpus Christi (CRP), 

Victoria (VCT), and Laredo (LRD) as verification sites. Initially, only monthly verification 

statistics were provided. However, by the year 2000, six month cool season (October through 

March) and warm season (April through September) guidance and station statistics were 

provided, showing errors, biases, and “near hits” (the percentage of station forecasts within 

three degrees Fahrenheit of the observed value; henceforth known as 3oF or less. Additionally 

in 2000, individual temperature verification statistics began to be provided to each forecaster

for the warm and cool seasons. In 2001, an office temperature verification contest was 

initiated, and office precipitation verification began. Early in 2003, the Interactive Forecast 

Processing System (IFPS) was used to create forecasts. At this time, WFO CRP decided to go 

to a short-term and long-term forecasting strategy (rather than a public versus aviation/marine 

forecast regiment). Under the short-term/long-term concept, the short-term forecaster creates

grids of various weather elements (e. g., wind, temperature, weather) for as many as the first 

three periods (i. e., today, tonight and tomorrow), or as few as no periods (during very active 

weather), with the long-term forecaster responsible for the remaining periods. Given this 

variability of short-term forecaster responsibility, a method was devised to keep track of how 

many forecast periods the short term forecaster provided in the IFPS, allowing individual 

verification statistics to continue. In March 2004, Alice (ALI), Rockport (RKP), and Cotulla 

(COT) became verification sites, bringing the total number of sites to six. Finally, by the end 

of the 2004 warm season, a six month precipitation verification forecast contest began. A 

description of the WFO CRP temperature and precipitaion verification program is presented 

below.
2

III. Temperature Verification

A program called SOOVER is used for temperature verification for individual station statistics. 

The SOOVER program was developed by Jamie Frederick, at that time a lead forecaster at 

WFO Tulsa (now the Information Technology Officer at that office). The SOOVER program 

provides mean Model Output Statistics (MOS) and Coded Cities Forecast (CCF) absolute 

errors, mean biases, and the percentage of forecasts which fall into a particular error range (as 

specified by the office). Average absolute errors are computed using the difference between 

forecast and observed temperatures regardless of sign: 

 | Error | = (1/N) Σ | (fi - oi) | (1)

where Σ is the summation from i equals 1 to i equals N, fi is the forecast temperature for the ith 

forecast, oi is the observed temperature for the ith forecast, and N is the number of forecasts for 

that period. The average CCF and MOS bias is the arithmetic average of the differences 

between the observed temperatures and forecast temperatures for each period:

 Bias = (1/N) Σ (fi - oi) (2)

The bias indicates whether temperature forecasts have been consistently too warm or too cold. 

At this time, SOOVER tallies the aforementioned verification statistics for the CCF, and for

the MAV-MOS (generated from the Global Forecast System model, or GFS), MET-MOS

(generated from the North American Mesoscale model, or NAM), and FWC-MOS (generated 

from the Nested Grid Model, or NGM) forecasts for the first five periods. The SOOVER 

program is also dynamic in that it can compile statistics over a user-defined period of time. To 

store these forecasts, a program is run every twelve hours that extracts the MOS and CCF 

forecasts from the database and places them into a file (one file for each verification site). 

Another program (run every twelve hours) obtains the maximum and minimum temperature 

data. When SOOVER is run, the program compiles the aforementioned statistics for the 0000 

UTC cycle, 12 00 UTC cycle, and combined 0000 and 1200 UTC cycles corresponding to the 

forecasts and model runs mentioned above. 

The verification statistics output by SOOVER are presented to the forecasters every month in 

both graphical and tabular format. Figures 1 and 2 are examples of two graphs created every 

month that are placed in the office break room, so that forecasters can quickly review the 

overall results. Figure 1 shows the average absolute CCF and MOS (MAV, MET, and FWC) 

errors for a month (along with an average for all six verification sites), while Figure 2 shows a 

12 month trend of CCF and MOS absolute errors, graphically and in a tabular format.
3

Station MAV MET FWC CCF

CRP 2.38 2.23 2.42 2.14

VCT 2.18 2.56 2.84 2.26

LRD 3.21 3.26 3.43 2.17

ALI 2.49 2.54 2.87 2.25

RKP 1.88 1.83 N/A 1.95

COT 2.73 3.23 2.78 2.48

AVG 2.48 2.61 2.87 2.21

Figure 1: Average absolute temperature errors (maximum and minimum temperature errors for 

all periods combined) for the CCF, MAV, MET, and FWC for each verification site. Note that 

an average (AVG) for all six verification sites is provided in the graph. This graph is placed in 

the break room for all forecasters to easily investigate the monthly verification results.

Next, graphs of CCF and MOS absolute errors and biases for maximum and minimum 

temperatures are created for each verification site (examples shown in Figures 3 and 4), along 

with a table depicting the percentage of CCF forecasts differing from observations by 3oF or 

less (Table 1). Another table is also compiled, providing overall CCF and MOS errors, biases, 

and percentage of forecasts 3oF or less from observed in a one-page format (Table 2). A key is 

provided below the table. 

Every April and October, six month station verification statistics are computed for the October 

through March (cool) and April through September (warm) seasons, respectively, for each 

verification site. Similar to the monthly results, graphs of CCF and MOS absolute average 

errors and average biases for maximum and minimum temperatures for each station are 

presented to the staff, together with a table of the percentage of CCF forecasts 3oF or less than

observed. Also, a summary table (not shown) is provided similar to that shown in Table 2. 

Figures 5 and 6 are examples of some graphs produced for a six month period.

AVERAGE TEMPERATURE ERRORS - MAY 2005

0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

4.00

CRP VCT LRD ALI RKP COT AVG

STATION AND MODEL

ERROR (F)

MAV

MET

FWC

CCF
4

AVERAGE TEMPERATURE ERRORS - JUNE 2004 THROUGH 

MAY 2005

0.00

1.00

2.00

3.00

4.00

5.00

JUN JUL AUG SEP OCT NOV DEC JAN FEB MAR APR MAY

MONTH AND MODEL

ERROR (F)

CCF

MAV

MET

FWC

MONTH MAV MET FWC CCF

JUN 2.13 2.31 2.44 1.97

JUL 1.87 2.03 2.13 1.88

AUG 2.09 2.30 2.40 1.86

SEP 2.30 2.04 2.33 2.53

OCT 3.10 2.80 2.93 2.45

NOV 3.31 3.48 3.79 3.34

DEC 3.84 3.84 3.95 3.45

JAN 3.71 4.37 4.91 3.51

FEB 3.22 3.51 3.33 3.06

MAR 3.35 3.28 3.93 3.11

APR 3.03 2.90 3.15 2.64

MAY 2.48 2.61 2.87 2.21

Figure 2: Twelve month trend of CCF and MOS errors. This graph is actually displayed in 

color for easier interpretation, and is placed in the break room for all forecasters to easily 

investigate the monthly verification results.

Figure 3: Average absolute temperature errors for minimum (low) temperatures for the CCF, 

MAV, MET, and FWC for the verification site Victoria. A similar absolute error graph for 

each verification site is provided for maximum and minimum temperatures. 

Period MAV MET FWC CCF

1 2.50 2.74 3.45 2.61

2 2.52 2.87 3.29 2.58

3 2.45 2.93 3.47 2.63

4 2.63 3.33 3.40 2.47

5 2.89 3.38 N/A 2.59

AVG 2.60 3.05 3.40 2.58

AVERAGE ERRORS FOR MINIMUM TEMPERATURES

MAY 2005 - VICTORIA

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

1 2 3 4 5

PERIOD

ERROR (°F)

MAV

MET

FWC

CCF
5

Figure 4: Average temperature biases for maximum (high) temperatures for the CCF, MAV, 

MET, and FWC for the verification site Cotulla. A similar bias graph for each verification site 

is provided for maximum and minimum temperatures. 

Table 1: An example of a table showing the percentage of forecasts three degrees Fahrenheit 

(3oF) or less from observed, along with the percentage of forecasts falling outside of that range. 

This example is for the Laredo site. Similar tables are created for all six verification sites. 

May 2005 Forecaster error (CCF) by category (%)

LRD

 May 2005 Maximum Temperatures May 2005 Minimum Temperatures

Forecast period < -4 -3/+3 > +4 Forecast period < -4 -3/+3 > +4

1st period high 16 77 7 1st period low 16 71 13

2nd period high 17 73 10 2nd period low 13 74 13

3rd period high 13 70 17 3rd period low 17 73 10

4th period high 17 73 10 4th period low 16 77 7

5th period high 17 76 7 5th period low 14 76 10

Finally, individual temperature verification statistics are given to each forecaster for the 

previous cool or warm season. Since a short-term forecaster may contribute as many as three 

periods or as little as zero periods to the CCF during a forecast cycle, a method was devised to 

keep track of the number of periods the short-term and long-term forecaster provided. During 

the midnight shift (0000 UTC cycle/morning forecast package) and daytime shift (1200 UTC

Table 2: Absolute errors, biases, and CCF forecasts 3F or less from observed for all six 

Period MAV MET FWC CCF

1 -0.10 1.71 0.52 0.45

2 0.86 0.63 -3.00 0.87

3 1.30 1.83 0.20 1.27

4 1.61 0.76 -2.45 0.93

5 1.41 0.93 N/A 0.76

AVG 1.02 1.17 -1.18 0.86

AVERAGE BIASES FOR MAXIMUM TEMPERATURES

MAY 2005 - COTULLA

-3.0

-2.5

-2.0

-1.5

-1.0

-0.5

0.0

0.5

1.0

1.5

2.0

2.5

3.0

1 2 3 4 5

PERIOD

ERROR (°F)

MAV

MET

FWC

CCF
6

verification sites. The term “overall” for MOS errors indicates which model either had the 

highest or lowest errors for all periods or for all but one period. The term “overall” for MOS 

biases indicates what trend the model had for all periods or all but one period. Also, note the 

key below the table. This key specifies how to read the elements in each column. A similar 

table is provided for the six month verification statistics. 

 TEMPERATURE FORECAST RESULTS FOR APRIL 2005

MAXIMUM TEMPERATURE ERRORS CRP VCT LRD ALI RKP COT

Number of periods CCF error was <= 3.0F / <=2.0F 5 / 1 5 / 0 2 / 0 5 / 1 5 / 0 4 / 0

Number of periods CCF error < MAV error (3% or more) 4 5 5 4 3 4

Number of periods CCF error < MET error (3% or more) 4 5 4 5 2 4

Number of periods CCF error < FWC error (3% or more) 3 5 4 4 N/A 3

Lowest MOS errors (overall) MIX MIX MET MAV MET MAV

Highest MOS errors (overall) MIX MIX M/F FWC MAV MET

MINIMUM TEMPERATURE ERRORS CRP VCT LRD ALI RKP COT

Number of periods CCF error was <= 3.0F / <=2.0F 5 / 0 4 / 0 5 / 1 3 / 0 4 / 0 3 / 0

Number of periods CCF error < MAV error (3% or more) 5 2 4 4 3 5

Number of periods CCF error < MET error (3% or more) 3 3 3 2 2 4

Number of periods CCF error < FWC error (3% or more) 3 4 2 2 N/A 4

Lowest MOS errors (overall) MET M/E MIX MET MET MIX

Highest MOS errors (overall) MAV FWC MIX MIX MAV M/E

MAXIMUM TEMPERATURE BIASES CRP VCT LRD ALI RKP COT

Overall CCF bias trend C/SC W/SW SC/C MIX C/VC SW/N

Overall MAV bias trend C W VC SC C/VC N/SC

Overall MET bias trend SC SW/W C/VC MIX SC/C C

Overall FWC bias trend SC/C VW/W C/VC C/SC N/A MIX

MINIMUM TEMPERATURE BIASES CRP VCT LRD ALI RKP COT

Overall CCF bias trend N/SW SW SC SW/W SC/C W

Overall MAV bias trend SC SW N/SC W SC/C W/VW

Overall MET bias trend MIX W/VW MIX W/SW SC W/VW

Overall FWC bias trend C/SC VW MIX SW/W N/A W/VW

%CCF FORECASTS 3F OR LESS FROM OBSERVED CRP VCT LRD ALI RKP COT

Maximum temperature percentage range 3F or less 64-86 71-79 53-73 68-80 69-84 59-68

Minimum temperature percentage range 3F or less 57-68 52-65 61-80 44-69 63-79 55-64

CCF max temp forecast not in 3F range warm or cold? CD WM MIX MIX CD WM

CCF min temp forecast not in 3F range warm or cold? WM WM MIX WM MIX WM

Bias and Forecast Trend Key (overall trend of the biases for the periods; an exception or two are possible):

 W – Warm (bias from +1 to +2F) C – Cold (bias –1 to –2F) CT – Cooler with time WT – Warmer with time

SW – Slightly warm ( +0.2F < bias < 1.0F) SC – Slightly cool (-1.0F < bias < -0.2F)

VW – Very warm (bias > 2.0F) VC – Very cool (bias < -2.0F) NT – No Trend

 N – Neutral (-0.2F < bias <0.2F) WM – Too Warm CD - Too Cold

M/E: Mix between MAV and ETA M/F: Mix between MAV and FWC E/F Mix between ETA and FWC

Note: “MIX” means one or more of the MOS methods did not produce a distinct overall trend 

(e.g. MAV and FWC shared the highest errors), or that no conclusion in the trend could be ascertained.
7

Figure 5: Average temperature biases for maximum (high) temperatures for the CCF, MAV, 

MET, and FWC for the verification site Victoria during the period October 2004 through 

March 2005. A similar graph for each verification site is created for maximum and minimum 

temperature biases for the first five periods.

AVERAGE ERRORS FOR MAXIMUM TEMPERATURES

OCTOBER 2004 THRU MARCH 2005 - LAREDO

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

5.5

6.0

1 2 3 4 5

PERIOD

ERROR (°F)

MAV

MET

FWC

CCF

Period MAV MET FWC CCF

1 3.97 4.07 3.43 2.48

2 4.17 4.18 4.40 3.02

3 4.51 4.37 4.82 3.07

4 4.44 4.33 5.62 3.93

5 5.37 5.20 N/A 4.34

AVG 4.49 4.43 4.57 3.37

Figure 6: Average temperature errors for maximum (high) temperatures for the CCF, MAV, 

MET, and FWC for the verification site Laredo during the period October 2004 through March 

2005. A similar graph for each verification site is created for maximum and minimum 

temperature errors for the first five periods.

Period MAV MET FWC CCF

1 -0.68 0.43 0.90 -0.06

2 -0.81 0.54 1.48 -0.12

3 -1.13 -0.01 1.04 -0.28

4 -1.12 0.34 0.82 -0.70

5 -0.97 -0.07 N/A -0.79

AVG -0.94 0.25 1.06 -0.39

AVERAGE BIASES FOR MAXIMUM TEMPERATURES

OCTOBER 2004 THRU MARCH 2005 - VICTORIA

-2.0

-1.5

-1.0

-0.5

0.0

0.5

1.0

1.5

2.0

1 2 3 4 5

PERIOD

ERROR (°F)

MAV

MET

FWC

CCF
8

cycle/afternoon forecast package), the long term forecaster creates a one-line text file from the 

Graphical Forecast Editor (GFE). This file, called a “forecaster verification” file, 

provides the date and forecast cycle, the short-term forecaster number, the long-term forecaster 

number, and the number of periods done by the short-term forecaster. Every six months, a 

locally created program is run, which ingests the necessary forecaster verification files, the 

temperature forecast file that contains the CCF and MOS forecasts for each verification site, 

and the observed temperature file. The forecaster verification file identifies the short-term and 

long-term forecasters, and the number of periods the short-term forecaster created. Once all the 

forecasts are separated by forecasters, the program determines the differences between 

observed, CCF, and MOS temperatures for that period. When the calculations for all forecasts 

are completed, CCF, MAV, MET, and FWC average absolute errors, biases, and the number of 

forecasts 3F or less from observed are tabulated for the first five periods for maximum and 

minimum temperatures. These computations are done for all six verification sites, with the 

output placed in a file containing all of the individual forecaster’s verification statistics. Each 

forecaster file is then e-mailed to that forecaster so he or she can compare their CCF forecasts 

with the MOS forecasts. Tables 3 through 5 show examples of how some of the verification 

data are presented to each forecaster. 

Table 3: Maximum and minimum absolute temperature errors for an individual forecaster at 

Corpus Christi for a six-month period. A “-99.0” indicates no forecasts were available for this 

period (and is used in other tables to indicate no forecasts were available). Average absolute 

temperature errors are provided for all six verification sites.

 TEMPERATURE ERRORS FOR CORPUS CHRISTI

----------------------------------------------

MAXIMUM TEMPERATURE ERRORS

----------------------------------------------

PERIOD CCF FWC MET MAV

----------------------------------------------

 1 1.5 1.9 1.5 1.8

 2 2.6 2.7 2.4 2.4

 3 2.0 2.3 3.0 2.1

 4 2.5 2.7 3.1 2.4

 5 3.1 -99.0 3.7 3.3

----------------------------------------------

 AVE 2.4 2.4 2.8 2.5

MINIMUM TEMPERATURE ERRORS

----------------------------------------------

PERIOD CCF FWC MET MAV

----------------------------------------------

 1 1.4 1.3 2.5 1.3

 2 1.2 2.3 1.8 1.6

 3 2.4 3.6 2.7 2.7

 4 3.3 3.9 3.3 3.3

 5 2.6 -99.0 3.5 2.8

----------------------------------------------

 AVE 2.4 3.0 2.8 2.5

 
9

Table 4: Maximum and minimum temperature biases for an individual forecaster at Corpus 

Christi for a six-month period. Temperature biases are computed for all six verification sites.

TEMPERATURE BIASES FOR CORPUS CHRISTI 

----------------------------------------------

 

MAXIMUM TEMPERATURE BIASES

----------------------------------------------

 PERIOD CCF FWC MET MAV

----------------------------------------------

 1 -0.1 0.5 0.6 0.1

 2 -0.7 -0.2 1.9 0.2

 3 -0.7 0.4 1.0 -0.7

 4 0.2 0.3 1.8 0.8

 5 -1.3 -99.0 -2.7 -1.0

----------------------------------------------

 AVE -0.6 0.3 0.6 -0.2

 

 

MINIMUM TEMPERATURE BIASES

----------------------------------------------

 PERIOD CCF FWC MET MAV

----------------------------------------------

 1 0.2 -0.5 1.9 0.1

 2 -0.5 -0.5 -0.1 -0.4

 3 -0.3 -0.2 0.6 -0.2

 4 -0.5 0.4 -0.2 -0.4

 5 0.1 -99.0 1.4 -0.2

----------------------------------------------

 AVE -0.2 -0.1 0.4 -0.3

Table 5: The number and percentage of CCF and MOS maximum temperature forecasts 3

degrees F or less from observed temperatures for Period 1 during a six month period. 

Forecasts “too cold” have errors of -4 degrees F or greater; while “too warm” have errors 4 

degrees F or greater. These tables are generated for the first five periods for both maximum 

and minimum temperatures, and for all six verification sites.

MAXIMUM TEMPERATURE FORECASTS 3F FROM OBSERVED FOR PERIOD 1

 NUMBER PERCENTAGE 

------------------------------------------------------------------------

 RANGE CCF FWC MET MAV || RANGE CCF FWC MET MAV

------------------------------------------------------------------------

TOO COLD 0 2 1 1 || TOO COLD 0.0 7.7 3.8 3.8

3F OR LESS 24 23 24 24 || 3F OR LESS 92.3 88.5 92.3 92.3

TOO WARM 2 1 1 1 || TOO WARM 7.7 3.8 3.8 3.8

------------------------------------------------------------------------

IV. Precipitation Verification

For station precipitation verification, a locally created program is used to generate statistics for 

each month, for the cool season, and for the warm season. This program generates CCF, 

MAV, MET, and FWC average POP and Brier Scores (BS) for the 00Z through 12Z time 

frame (nighttime), the 12Z through 00Z time frame (daytime), and for combined daytime and 

nighttime time frames for the first five periods. The Brier Score is the mean square error of all 
10

POP forecasts. The standard National Weather Service Brier Score, defined below, is one-half 

the original score defined by Brier (1950):

BS = (1/N) Σ (fi – oi)

2

For POP verification, assuming 10 percent probability intervals (except for the 5% POP), 

where Σ is the summation from i equals 1 to i equals N, fi = forecast probability (either CCF or 

MOS, 0, 0.05, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0); for the ith case, oi = observed 

precipitation occurrence (0 or 1), and N is the number of cases. By this formula, a perfect 

Brier Score would be 0.0 (i.e, average POP of 100% for every measurable rainfall event), 

while the worst Brier Score would be 1.0 (average POP of 0% for every measurable rainfall 

event).

To present these monthly statistics to the staff, average Brier scores per period for all events 

for all six stations are shown graphically, an example of which is shown in Figure 7. 

Next, a table is generated which summarizes the remaining verification statistics (Table 6).

In this table, Brier Scores and average POPs for measurable and trace rainfall events are 

summarized for the daytime (12Z to 00Z) and nighttime (00Z to 12Z) intervals, while average 

POPs for all non rainfall events are summarized for all intervals. Similar tables are produced 

for each six month period. 

For the warm season and cool season, graphs are generated for measurable rainfall, trace 

rainfall, and no rainfall for the daytime and nighttime periods (examples are shown in Figures 

8 through 10). For all six verification sites, graphs of Brier Scores are also presented for the

Figure 7: Brier Scores for April 2005 at Victoria. Similar graphs are created for the remaining 

verification sites each month.

Period MAV MET FWC CCF

1 0.128 0.146 0.183 0.104

2 0.112 0.120 0.184 0.099

3 0.131 0.137 0.167 0.102

4 0.148 0.129 0.182 0.111

5 0.142 0.142 N/A 0.115

AVG 0.132 0.135 0.179 0.106

BRIER SCORES

APRIL 2005 - VICTORIA

0.00

0.05

0.10

0.15

0.20

1 2 3 4 5

PERIOD

SCORE

MAV

MET

FWC

CCF
11

Table 6: Example of a monthly Precipitation Summary Table for April 2005. A similar table 

is generated for the six month verification data.

 PRECIPITATION FORECAST RESULTS FOR APRIL 2005

 DAYTIME BRIER SCORES CRP VCT LRD ALI RKP COT

Numb er of periods CCF scores were better than MAV scores 3 2 2 5 1 0

Number of periods CCF scores were better than MET scores 3 4 0 1 1 0

Number of periods CCF scores were better than FWC scores 4 4 2 4 N/A 1

Best MOS scores (overall) MIX MAV M/E MET MET M/E

Worst MOS scores (overall) MIX FWC FWC MAV MAV FWC

 NIGHTTIME BRIER SCORES CRP VCT LRD ALI RKP COT

Number of periods CCF scores were better than MAV scores 4 5 2 3 0 0

Number of periods CCF scores were better than MET scores 4 5 2 1 0 5

Number of periods CCF scores were better than FWC scores 3 4 1 1 N/A 1

Best MOS scores (overall) MAV M/E MIX MIX MIX MAV

Worst MOS scores (overall) FWC FWC MIX MIX MIX MET

 DAYTIME POPS MEASURABLE RAIN CRP VCT LRD ALI RKP COT

Number of events / rainfall amount 1 / 0.09 4 / 0.43 0 / 0 2 / 0.06 0 / 0 0 / 0

Periods CCF POPs >= 30% / > = 50% 3 / 0 4 / 0 N/A 5 / 0 N/A N/A

Number of times CCF POP = 0 0 0 N/A 0 N/A N/A

Number of periods CCF POP >= MAV POP 5 2 N/A 5 N/A N/A

Number of periods CCF POP >= MET POP 4 4 N/A 3 N/A N/A

Number of periods CCF POP >= FWC POP 4 4 N/A 4 N/A N/A

Highest MOS POPs (overall) MET MAV N/A MET N/A N/A

Lowest MOS POPs (overall) FWC FWC N/A M/F N/A N/A

 NIGHTTIME POPS MEASURABLE RAIN CRP VCT LRD ALI RKP COT

Number of events / rainfall amount 3 / 0.12 6 / 0 .82 1 / 0.22 2 / 0.08 0 / 0 1 / 0.05

Periods CCF POPs >= 30% / > = 50% 0 / 0 3 / 0 3 / 0 0 / 0 N/A 5 / 1

Number of times CCF POP = 0 0 0 0 0 N/A 0

Number of periods CCF POP >= MAV POP 4 5 5 4 N/A 2

Number of periods CCF POP >= MET POP 5 5 2 3 N/A 5

Number of periods CCF POP >= FWC POP 4 4 2 4 N/A 4

Highest MOS POPs (overall) M/E M/E MET MET N/A MAV

Lowest MOS POPs (overall) FWC FWC M/F FWC N/A E/F

 DAYTIME POPS TRACE EVENTS CRP VCT LRD ALI RKP COT

Number of events 5 3 3 2 2 4

Periods CCF POPs >= 30% / >= 20% 0 / 0 0 / 0 0 / 0 0 / 0 1 / 3 0 / 4

Number of periods MAV POPs >= 30% / >= 20% 0 /0 0 / 0 0 / 0 0 / 0 0 / 0 0 / 1

Number of periods MET POPs >= 30% / >= 20% 0 /0 0 / 0 0 / 0 0 / 0 0 / 0 0 / 0

Number of periods FWC POPs >= 30% / >= 20% 0 /0 0 / 0 0 / 0 0 / 0 N/A 0 / 1

 NIGHTTIME POPS TRACE EVENTS CRP VCT LRD ALI RKP COT

Number of events 2 3 2 2 5 2

Periods CCF POPs >= 30% / >= 20% 0 / 3 0 / 1 0 / 5 0 / 5 0 / 5 0 / 5

Number of periods MAV POPs >= 30% / >= 20% 0 / 1 0 / 0 0 / 2 0 / 2 0 / 1 0 / 0

Number of periods MET POPs >= 30% / >= 20% 0 / 1 0 / 1 2 / 2 0 / 0 0 / 2 0 / 3

Number of periods FWC POPs >= 30% / >= 20% 0 / 0 0 / 0 0 / 0 0 / 0 N/A 0 / 1

 POPS FOR NO RAINFALL EVENTS (DAY AND NIGHT) CRP VCT LRD ALI RKP COT

Number of events 49 44 54 52 53 53

Periods CCF POPs >= 10% / >= 20% 0 / 0 0 / 0 0 / 0 0 / 0 0 / 0 0 / 0

Number of periods MAV POPs >= 10% / >= 20% 0 / 0 0 / 0 0 / 0 0 / 0 0 / 0 0 / 0

Number of periods MET POPs >= 10% / >= 20% 0 / 0 0 / 0 0 / 0 0 / 0 0 / 0 0 / 0

Number o f periods FWC POPs >= 10% / >= 20% 0 / 0 0 / 0 0 / 0 0 / 0 0 / 0 0 / 0

Highest MOS POPs (overall) MAV MAV M/E MAV MAV MET

Lowest MOS POPs (overall) FWC FWC FWC FWC MET M/F

Note: “MIX” means that the FWC, MET, and MAV had a better result than the other two models for at least one period.

M/E: Mix between MAV and ETA M/F: Mix between MAV and FWC E/F Mix between ETA and FWC
12

Figure 8: Example of a CCF and MOS POP graph for measurable precipitation events for a six 

month period. This graph indicates the average POP for measurable rainfall events during the 

daytime (12Z to 00Z) at Cotulla. Similar graphs are generated for all six verification sites for 

daytime and nighttime measurable rainfall events.

AVERAGE POP TRACE RAINFALL - NIGHTTIME EVENTS

2005 COOL SEASON (04OCT - 05MAR) - ALICE

0

5

10

15

20

25

30

35

40

45

1 2 3 4 5

PERIOD

POP IN PERCENT

MAV

MET

FWC

CCF

Period MAV MET FWC CCF

1 26.6 31.0 21.3 26.8

2 25.9 29.1 22.6 22.3

3 30.4 23.7 23.1 23.0

4 33.9 26.5 22.3 23.8

5 31.9 30.8 N/A 20.0

AVG 29.7 28.2 22.3 23.2

Events = 20

Figure 9: Example of a CCF and MOS POP graph for trace rainfall events for a six month 

period. This graph indicates the average POP for trace rainfall events during the nighttime 

(00Z to 12Z) at Alice. Similar graphs are generated for all six verification sites for daytime 

and nighttime trace rainfall events.

Period MAV MET FWC CCF

1 52.7 59.2 50.0 55.0

2 53.1 52.2 38.7 50.5

3 50.7 51.0 41.2 49.0

4 48.4 45.8 33.6 44.3

5 40.1 48.1 N/A 37.1

AVG 49.0 51.3 47.1 47.2

Total = 3.39 inches; Events = 21

AVERAGE POP MEASURABLE RAIN - DAYTIME EVENTS

2005 COOL SEASON (04OCT - 05MAR) - COTULLA

0

10

20

30

40

50

60

1 2 3 4 5

PERIOD

POP IN PERCENT

MAV

MET

FWC

CCF
13

AVERAGE POP NO RAINFALL - DAYTIME EVENTS

2005 COOL SEASON (04OCT - 05MAR) - COTULLA

0

5

10

15

20

25

1 2 3 4 5

PERIOD

POP IN PERCENT

MAV

MET

FWC

CCF

Period MAV MET FWC CCF

1 8.4 11.2 7.2 6.9

2 8.3 12.8 8.5 8.0

3 9.2 13.4 9.7 8.4

4 10.0 14.3 11.8 8.5

5 9.2 15.3 N/A 8.7

AVG 9.0 13.4 11.7 8.1

Events = 141

Figure 10: Example of a CCF and MOS POP graph for non rainfall events for a six month 

period. This graph indicates the average POP for non rainfall events during the daytime (12Z 

to 00Z) at Cotulla. Similar graphs are generated for all six verification sites for daytime and 

nighttime non-rainfall events.

DAYTIME BRIER SCORES

OCTOBER 2004 THRU MARCH 2005 - ALICE

0.00

0.02

0.04

0.06

0.08

0.10

0.12

1 2 3 4 5

PERIOD

SCORE

MAV

MET

FWC

CCF

Figure 11: Example of a CCF and MOS POP Brier Scores for a six month period. This graph 

indicates the scores during the daytime (12Z to 00Z) at Alice. Similar graphs are generated for 

all six verification sites for daytime and nighttime periods.

Period MAV MET FWC CCF

1 0.069 0.085 0.078 0.070

2 0.072 0.083 0.092 0.071

3 0.081 0.095 0.102 0.077

4 0.083 0.092 0.102 0.089

5 0.097 0.115 N/A 0.094

AVG 0.080 0.094 0.094 0.080
14

daytime and nighttime periods, and the combined Brier Scores for both periods (see Figure 11 

for an example). In addition, a summary table similar to the monthly summary table shown in 

Table 6 is created (for those who just want to see a quick summary rather than peruse several 

graphs).

Finally, individual precipitation verification statistics are provided to each forecaster for the 

previous six month period. Similar to the temperature verification for individual forecasters, a 

locally created program was written to ingest CCF and MOS forecast data, along with 

observed precipitation and the forecaster verification files. The forecaster verification file 

identifies the short-term and long-term forecasters, and the number of periods the short-term 

forecaster created. Once all of the data have been separated by forecasters, CCF and MOS 

precipitation verification statistics (average POP and Brier Scores) are computed for each 

forecaster for each verification site. These results are placed in an output file (separate from 

the verification file for temperatures). Each forecaster file is then e-mailed to that forecaster so 

he or she can compare their CCF forecasts with the MOS forecasts. Tables 7 through 9 show 

examples of how some of the precipitation verification data are presented to each forecaster. 

Table 7: Example of a forecaster’s precipitation verification statistics for a six month period. 

Here, average POPs for measurable rainfall are displayed for Corpus Christi. Average POPs 

for trace rainfall events and no rainfall events are also provided. Total rainfall shown is the 

amount of rain which occurred for the period the person forecast, and thus could include the 

same rainfall event in more than one period. These statistics for all six verification sites are 

given to each forecaster.

POP AVERAGES PER PERIOD FOR MEASURABLE RAINFALL AT CORPUS CHRISTI

-------------------------------------------------------------------

FCST DAY PERIOD PCPN CCF FWC MET MAV

-------------------------------------------------------------------

PERIOD 1 0.82 66.7 40.3 58.0 76.7

-------------------------------------------------------------------

PERIOD 2 3.90 60.0 25.7 36.3 63.7

-------------------------------------------------------------------

PERIOD 3 3.55 47.5 26.0 58.5 57.3

-------------------------------------------------------------------

PERIOD 4 8.24 36.7 25.0 42.2 35.8

-------------------------------------------------------------------

PERIOD 5 8.03 25.0 -99.0 12.5 33.9

-------------------------------------------------------------------

TOTALS 24.54 40.2 27.6 39.0 44.5

 

V. Forecaster Contests

Once each forecaster’s temperature and precipitation verification files have been e-mailed, 

forecaster contests are initiated to determine who provided the best temperature and 

precipitation forecasts during the six month period. Although the forecaster contests are in the 

spirit of friendly competition, the goal is to provide an incentive to produce the best possible 
15

forecasts during their shifts. Only the top three forecasters are announced for each contest. 

The first place forecaster for each contest receives a certificate; the winner 

of the temperature contest receives the “Johnny Tempo Award”, while the winner of the 

precipitation forecast receives the “Rain Man Award”.

Table 8: Example of an individual forecaster’s averages for trace rainfall events and for non 

rainfall events. Also, note the summary of periods of measurable rainfall, trace rainfall, and no 

rainfall events, as well as the number of times the person forecast a POP of ‘0’ when 

measurable rainfall occurred. Similar verification results are provided for all six verification 

sites. 

POP AVERAGES FOR TRACE EVENTS AT CORPUS CHRISTI

-------------------------------------------------------------------

FCST DAY PERIOD CCF FWC MET MAV

-------------------------------------------------------------------

AVERAGE FOR PERIOD 1 30.0 5.0 32.0 37.0

AVERAGE FOR PERIOD 2 -99.0 -99.0 -99.0 -99.0

AVERAGE FOR PERIOD 3 40.0 13.0 35.0 22.0

AVERAGE FOR PERIOD 4 25.0 13.3 19.8 34.7

AVERAGE FOR PERIOD 5 31.3 -99.0 38.8 34.8

-------------------------------------------------------------------

TRACE AVERAGES 28.8 12.3 28.4 33.8

PERIODS WITH MEASURABLE RAINFALL = 12

# OF FORECASTS WITH RAIN AND CCF '0' = 0

PERIODS WITH TRACE RAINFALL = 12

PERIODS WITH NO RAINFALL = 110

AVERAGES FOR NO RAINFALL AT CORPUS CHRISTI

-------------------------------------------------------------------

FCST DAY PERIOD CCF FWC MET MAV

-------------------------------------------------------------------

NO RAIN 1 4.5 3.9 8.3 10.8

NO RAIN 2 2.9 3.1 9.5 6.8

NO RAIN 3 4.3 4.5 9.5 10.5

NO RAIN 4 5.0 7.6 10.4 11.0

NO RAIN 5 7.8 -99.0 17.3 10.0

--------------------------------------------------------------------

AVERAGE 4.7 4.5 10.7 9.7

Table 9: Example of an individual forecaster’s Brier Scores for a six month period. Brier 

Score results are provided for all six verification sites. 

BRIER SCORES FOR ALL EVENTS AT VICTORIA 

-------------------------------------------------------------------

FCST DAY PERIOD CCF FWC MET MAV

-------------------------------------------------------------------

AVERAGE FOR PERIOD 1 0.081 0.104 0.109 0.080

AVERAGE FOR PERIOD 2 0.122 0.164 0.134 0.122

AVERAGE FOR PERIOD 3 0.136 0.214 0.174 0.149

AVERAGE FOR PERIOD 4 0.102 0.135 0.130 0.120

AVERAGE FOR PERIOD 5 0.124 -9.999 0.136 0.134

-------------------------------------------------------------------

-------------------------------------------------------------------

AVERAGES 0.114 0.154 0.137 0.123
16

The temperature forecast contest uses the six forecaster files that were created for that person’s 

verification statistics. These files contain the CCF and MOS forecasts for each day and period, 

along with the observed temperatures. Differences between the CCF forecast and observed 

temperature are computed. Next, the available MAV, MET, and FWC forecasts are averaged, 

and the difference between the average MOS forecast and observed temperature is computed. 

Differences between the CCF error and the average MOS error are compared. The CCF error 

is adjusted in the following manner. For the first period, only 20% of the error difference 

between the CCF error and average MOS error is used (let us call this difference the “error 

difference”). This error difference increases to 60% by period five (i.e., 20% for the first 

period, increasing by 10% each period, up to 60% for period five). If the CCF error is smaller 

than the average MOS error for that forecast, then that percentage error difference is subtracted 

from the CCF error. However, if the CCF error is larger than the average MOS error, then the 

percentage of that error difference is added to the CCF error. This way, the forecaster is 

rewarded for beating MOS, but punished for losing to MOS. 

After all of the CCF errors are adjusted for each forecast, the adjusted CCF errors are summed 

and averaged. The average adjusted CCF error is the forecaster’s “score” for that verification 

site. Forecaster scores are determined for all six sites and averaged to compute a final score. 

The forecaster with the lowest final score wins the coveted Johnny Tempo Award. An 

example of this certificate is shown in Figure 12. Station temperature verification results over 

the past several months indicate that WFO CRP has greatest temperature improvements over 

MOS in the Southern Region for the first three periods, and improves MOS forecasts through 

period eight. 

THE JOHNNY TEMPO AWARD

has been award to

Joe Meteorology

For the best temperature forecasts during the period April 2004 to September 2004

AT WFO CORPUS CHRISTI TX

Figure 12: Example of the Certificate for the Johnny Tempo Award given to the forecaster 

with the best temperature score for all six verification sites. This contest is done in the spirit of 

friendly competition to hopefully improve temperature forecasts.
17

Similar to the temperature forecast contest, the precipitation forecast contest compares each 

CCF forecast with observed rainfall, and an average of the available MAV, MET and FWC 

forecast. A Brier Score for each forecast is computed for the CCF and average MOS forecast. 

The differences between the CCF and average MOS score is compared (let us call this 

“difference in score”). Similar to the temperature forecast contest, for the first period only 

20% of the difference in score is used to adjust the CCF score. However, this adjustment 

increases by 10% for each period, increasing to 60% by period five. If the CCF score is better 

(worse) than the average MOS score, the CCF score is reduced (increased) by that percentage 

of the difference in score. These adjusted scores are computed for all forecasts, then averaged 

for each station. This adjusted Brier Score is computed for each verification site and averaged. 

The forecaster with the lowest final average adjusted Brier Score wins the coveted Rain Man 

Award. An example of this certificate is shown in Figure 13. Station precipitation verification 

results over the past several months indicate that WFO CRP is one of the better performing 

WFOs in Southern Region when comparing MOS and CCF forecasts. 

THE DUSTIN HOFFMAN RAIN MAN AWARD

has been award to

Joe Meteorology

For the best precipitation forecasts during the period April 2004 to September 2004

AT WFO CORPUS CHRISTI TX

Figure 13: Example of the Certificate for the Rain Man Award given to the forecaster with the 

best precipitation score for all six verification sites. This contest is done in the spirit of 

friendly competition to hopefully improve precipitation forecasts.
18

VII. Summary

In an ongoing effort to improve temperature and precipitation forecasts at WFO Corpus Christi 

TX, station verification statistics are presented to the staff every month. Also, six month 

verification statistics are generated in October and April for the warm and cool seasons, 

respectively. Since the short term forecaster can provide forecasts for as many as three periods 

or for as little as zero periods, a method was devised to keep track of the number of periods the 

short term and long term forecasters provided for that forecast cycle. This enables each 

forecaster to receive a six month summary of their temperature and precipitation verification 

statistics, and to compare their statistics with the MAV, MET and FWC guidance. As an 

additional incentive and in the spirit of friendly competition, a temperature and precipitation 

forecast contest is conducted every six months. In each contest, the top three forecasters are 

announced, with the winner of each forecast receiving a certificate. We believe that these 

procedures help WFO CRP forecasters provide the best possible forecasts to their customers by 

determining how well MAV, MET, and FWC guidance performs at each station, how station 

CCF forecasts compare with individual MOS forecasts, what MOS biases exist at each station, 

and how each forecaster’s statistics compare with MOS guidance. Verification statistics 

during the past several months has indicted that WFO CRP temperature forecasts improve 

MOS temperature forecasts for the first several periods, and that the WFO's precipitation 

statistics are among the best in the Southern Region.

VIII. Acknowledgements

The author would like to thank Ron Morales, Science and Operations Officer at WFO CRP for 

his suggestions and for his help in editing the first two drafts of this paper. Also, thanks to 

David (Rusty) Billingsley, Chief of the Science Services Division (SSD) in Southern Region, 

and Bernard Meisner Deputy SSD Chief, for their comments, suggestions and critical editing. 

Finally, I would also like to thank Mark Fox, Regional Training Officer in Southern Region for 

the verification statistics. 

IX. References

1. Brier, Glenn W., 1950: Verification of forecasts expressed in terms of probability. Mon. 

Wea. Rev., 78, 1-3. 

2. Frederick, James, 1999: AWIPS Interactive Forecast Verification. 15th International 

Conference on Interactive Information and Processing Systems for Meteorology, 

Oceanography and Hydrology, Dallas, Texas, January 10-15, 1999.