**Evaluation of Potential Forecast Accuracy Performance Measures**

**for the Advanced Hydrologic Prediction Service**

Gary A. Wick

NOAA Environmental Technology Laboratory

On Rotational Assignment with the National Weather Service

Office of Hydrologic Development

November 7, 2003

# 1. Introduction and Objectives

The primary motivation for this study was to evaluate the potential
applicability of various forecast accuracy measures as a program-level
performance measure for the Advanced Hydrologic Prediction Service
(AHPS). At the time of preparation, the only AHPS program performance
measure was number of river forecast points at which AHPS was
implemented. Clear need existed for the incorporation of additional
performance measures. Feedback from users, scientists, managers, and
administrators obtained through interviews and reviews of program
materials indicated strong interest in having one measure address the
accuracy of the forecast information generated within AHPS.

The AHPS forecast products include both deterministic and probabilistic
predictions. While accuracy measures for deterministic forecasts such as
probability of detection and false alarm rate are well established and
generally understood by the public, accuracy measures for probabilistic
forecasts are more complex. Probabilistic forecast verification is
relatively new within the hydrologic communities but various techniques
have been developed and applied within the meteorological and numerical
weather prediction communities (see, e.g. Wilks, 1995). An initial study
conducted by Franz and Sorooshian (2002) for the National Weather
Service (NWS) identified and evaluated several procedures that could be
applied to detailed, technical verification of the ensemble streamflow
predictions within AHPS.

It is important, however, to make a distinction between performance
measures at the program and science levels. At the scientific level, a
high degree of technical knowledge can be assumed enabling the use of
complex measures suitable for peer-reviewed publications. While such
measures allow the most comprehensive evaluation of forecast performance
and technical improvement, they may be difficult to communicate to an
audience with less scientific background. A program-level measure should
be constructed in such a way that it has value and can be presented to
audiences with varying technical experience. This can be challenging, as
it is still desirable to maintain scientific validity in the measure to
help ensure the integrity of the program. Application at the program
level is also enhanced if the measure can be applied uniformly over all
the hydrologic regimes covered by the program. This study builds
extensively on the previous work of Franz and Sorooshian (2002),
revisiting potential measures with specific attention to their
application at the program level.

The assessment of the measures was conducted with several specific
objectives in mind. Probabilistic measures were first reviewed to
identify the best compromises between illustration of key programmatic
capabilities, scientific merit, and ease of presentation. Existing
operational data were then used to perform sample computations of
selected measures. These tests helped identify what measures could
realistically be computed using operational data, demonstrate what
forecast outputs and verification data need to be collected and archived
regularly, and provide initial indication of how likely the measures
were to suggest program success and improvement. The review of potential
measures is presented in section 2. The operational data used to
evaluate the measures is described in section 3 and the results of the
evaluations are presented in section 4. Implications of these results on
the choice of program performance measures are then discussed in section
5 and corresponding recommendations for potential performance measures
and necessary data collection and archival are given in section 6.

# 2. Background on Probabilistic Forecast Verification

Detailed descriptions of existing probabilistic forecast verification
measures have been presented in several references (Wilks, 1995; Hamill
et al., 2000; Franz and Sorooshian, 2002; Hartmann et al., 2002). The
goal of the presentation here is to focus on the potential use of the
measures at a programmatic level. Sufficient technical details will be
provided to keep this document largely self-contained. The background
discussion will progress in order of increasing complexity of the
measures.

The majority of existing Government Performance Requirement Act (GPRA)
accuracy performance metrics are based on deterministic verification
measures such as probability of detection (frequently expressed as
accuracy) and false alarm rate. These measures, however, cannot be
applied directly to probabilistic forecasts where it is difficult to say
whether a single forecast is right or wrong. Deterministic measures will
remain important to AHPS to the extent that AHPS forecasts continue to
have deterministic elements.

## 2.1. Categorical Forecasts

The simplest probabilistic accuracy measure is constructed by
transforming a probabilistic forecast into a categorical (e.g. flood/no
flood) forecast through the selection of a probability threshold. Once
the forecasts have been categorized, probability of detection and false
alarm rate can again be computed. This was the basis for initial
discussions of a measure quantifying how often flooding occurred when
forecast with a probability exceeding a specified level. As considered,
the measure required specification of the probability threshold (e.g.,
50%, 85%), event (e.g., flooding, major flooding), and forecast period
(e.g., 30 day, 60 day, etc.).

The primary advantage of this measure is its ease of presentation to a
non-technical audience. The measure can be expressed simply as percent
accuracy as for deterministic measures. The measure also possesses a
scientific basis related to overall evaluation of probabilistic
forecasts. By evaluating the probability of detection and false alarm
rate for a series of threshold probabilities and plotting the
probability of detection against the false alarm rate it is possible to
construct what is termed a relative operating characteristics (ROC)
curve (e.g. Mason and Graham, 1999). The overall skill of the forecast
is then related to the area under the curve. These curves are currently
used as a component of forecast verification at the European Centre for
Medium-Range Weather Forecasts.

There are, however, several significant weaknesses of such a basic
measure. First, if only one probability threshold is used, the measure
does not completely address the probabilistic nature of the forecasts. A
forecast with a probability just outside the selected threshold is
awarded or penalized the same as a forecast with a high degree of
certainty. It is also not straightforward to identify a perfect score.
If a threshold probability of 50% is selected for forecasts of flooding,
the probability of detection indicating perfect forecasts should not be
100%. Forecasts of near 50% probability should verify only 50% of the
time and a perfect score should be somewhere between 50 and 100%. The
perfect score can be directly computed but this concept would be
difficult to present in programmatic briefings. Finally, the choice of a
probability threshold is arbitrary and could complicate explanation of
the measure. Discussions with several managers and administrators
indicated that the technical weakness of the measure combined with
possible confusion surrounding specification of multiple attributes made
the measure undesirable for use in the AHPS program.

## 2.2. Brier Score

An improved probabilistic accuracy performance measure can be based on
the Brier score (Brier, 1950; Wilks, 1995). While still limited to
characterizing two categories or whether or not a specific event occurs,
the Brier score fully accounts for probabilistic forecasts. The Brier
score essentially preserves the simplicity of a probability of detection
measure while eliminating the arbitrary threshold probability. The score
is formally defined as:

$\text{BS} = \frac{1}{N}\sum_{i = 1}^{N}\left( p_{i} - o_{i} \right)^{2}$
(1)

where *p~i~* is the forecast probability of the i^th^ event and *o~i~* =
1 if the event occurred and *o~i~* = 2 if it does not. As such the score
is essentially a mean square error where the error is related to the
difference between the forecast probability and the observed frequency.
The Brier score ranges between 0 for perfect forecasts and 1 for
perfectly bad forecasts. A Brier skill score can also be computed
relative to the Brier score of a reference forecast (*BS~ref~*):

*BSS* = (*BS~ref~* -- *BS*) / *BS~ref~*. (2)

The skill score gives the relative improvement of the actual forecast
over the reference forecast. A typical reference forecast would be based
on climatology. The Brier score can also be decomposed to identify the
relative effects of reliability, resolution, and uncertainty (Murphy,
1973).

Where a yes/no type measure characterizing either flooding or low flows
is of value, the Brier score can be simply presented while maintaining
scientific integrity, making it of potential interest as a programmatic
performance measure. An interview with Steven Gallagher, the Deputy
Chief Financial Officer of the National Weather Service, revealed
significant concerns with the programmatic use of any measure based on a
skill score where the method for arriving at the score requires
explanation. However since the Brier score always falls between 0 and 1,
it is possible to express the measure simply as a percent accuracy (by
subtracting the Brier score from 1) without formally referring to the
name Brier score. The relationship between the Brier score and
traditional accuracy can be illustrated with simple examples. If
flooding is forecast with perfect certainty (probability = 1) on four
occasions and flooding occurs in three of those cases, the resulting
Brier score would be 0.25 in agreement with an expected 75% accuracy.
The Brier score thus reduces to probability of detection for
deterministic cases. If each of the four forecasts were for 90%
probability, the Brier score would then be 0.21 implying 89% accuracy.
While the relationship is not as intuitive, the non-exact probabilities
are assessed in a systematic way. One additional example provides a
valuable reference. If a forecast probability of 50% is always assumed,
the Brier score will be 0.25. Poorer Brier scores would then suggest
that the corresponding forecasts add little value.

The primary weakness of the Brier score is that, by being limited to the
occurrence of a specific event such as flooding, only a small fraction
of the forecasts issued can be meaningfully evaluated. A large number of
the probabilistic forecasts for river stage or streamflow of interest
for water resource management would be essentially ignored. This concern
was voiced in particular by Steven Gallagher who favored a measure that
better addressed the overall distribution of the river forecasts.
Because of the low frequency of flooding events, it could be very
difficult to compile a dataset sufficient for complete evaluation of the
Brier score.

## 2.3. Rank Probability Score

The rank probability score and rank probability skill score (Epstein,
1969; Wilks, 1995) directly address this weakness but at the potential
cost of added complexity. Rather than being limited to the occurrence or
non-occurrence of an event, the rank probability score evaluates the
accuracy of probabilistic forecasts relative to an arbitrary number of
categories. Scores are worse when increased probability is assigned to
categories with increased distance from the category corresponding to
the observation. For a single forecast with J categories, the rank
probability score (RPS) is given by:

$\text{RPS} = \sum_{m = 1}^{J}\left\lbrack \left( \sum_{j = 1}^{m}p_{j} \right) - \left( \sum_{j = 1}^{m}o_{j} \right) \right\rbrack^{2}$
(3)

where *p~j~* is the probability assigned to the j^th^ category. To
address the multiple categories, the squared errors are computed with
respect to cumulative probabilities. For multiple forecasts, the RPS is
computed as the average of the RPS for each forecast. A perfect forecast
has an RPS = 0. Imperfect forecasts have positive RPS and the maximum
value is one less than the number of categories. As with the Brier skill
score a rank probability skill score can be computed relative to
reference forecast to provide the relative improvement of the new
forecast. The steps required for application of the rank probability
score and rank probability skill score are illustrated in Franz and
Sorooshian (2002, hereafter FS02). For the case of two categories, the
rank probability score reduces to the Brier score.

While the RPS is ideally suited for application as a scientific
performance measure as argued by FS02, several factors complicate its
use as a programmatic measure. Explanation of the score to someone with
a non-technical background would be challenging if required. It might
still be possible to map the score to a single percent accuracy figure
as for the Brier score but both the computation and interpretation are
less direct. The computation is complicated by the fact that the score
is not confined to a fixed interval. While the categories could be fixed
for all computations or the scores normalized by the maximum value to
enable comparison of different points, it is difficult to physically
interpret what such a score represents. Any relation to accuracy is
strongly influenced by the selection (number and relative width) of the
categories. Franz and Sorooshian state that the RPS alone is difficult
to interpret and is most useful for comparison of results at different
locations. Such a comparison over time has clear value for evaluating
advances in the forecasts and related science, but having a tangible
meaning for the score is also important for a programmatic measure.

Presentation as a skill score has more explicit meaning, but expression
in terms of an improvement over some reference such as climatology has
its own shortcomings. The concept of improvement over climatology can
appear less tangible than percent accuracy and selection of a meaningful
goal is less direct. The challenge of effectively communicating the
meaning of both the reference and score is likely a major contributor to
Steven Gallagher's reluctance to use a skill score. Moreover, the
climatology or other reference forecast must first be computed and this
frequently requires extensive historical data that might not be readily
available. Finally, there is the potential for the appropriate reference
or climatology to change over time.

Additional accuracy measures addressing discrimination and reliability
were advocated by FS02. Presentation of these measures was best
accomplished graphically. While of scientific importance, these measures
do not seem appropriate at the programmatic level. Hartmann et al.
(2002) also concluded that the diagrams are "probably too complex to
interpret for all but the large water management agencies and other
groups staffed with specialists."

# 3. Data

Further evaluation of the suitability of potential forecast accuracy
measures for AHPS is best achieved through application to actual data.
To accomplish this, examples of operational forecast output and
corresponding verification data are required. Two different sample data
sets were used to test possible deterministic and probabilistic accuracy
measures.

## 3.1. National Weather Service Verification Database

An initial evaluation of deterministic forecast accuracy measures was
performed using the existing National Weather Service Verification
Database. The database is accessed via a secure web interface at
[https://verification.nws.noaa.gov](https://verification.nws.noaa.gov/).
The user id and password are available to NWS employees.

The hydrology portion of the database supports verification of river
forecasts out to 3 days and flash floods. For river forecasts,
statistics on mean absolute error, root mean square error, and mean
algebraic error of forecast stage are generated interactively for a
selected set of verification sites. Data are available for 177 sites
covering every river forecast center (25 in Alaska) in monthly intervals
from April 2001. The data extend to approximately two months before the
current date. At the time of the experiments, data were available
through July 2003. The data available for flash flood verification are
more extensive in terms of both spatial and temporal extent.

The user can generate statistics for any subset of the available
verification sites for any range of months within the database. The
results can be stratified by river response period (fast: \< 24 hours,
medium: 24-60 hours, or slow: \> 60 hours), forecast period (time at
which the forecast is valid), and whether the river stage is above or
below flood stage. The data are obtained in summary tables or a comma
delimited format suitable for import into spreadsheets.

All computations are made within the web interface upon submission of a
request. Additional data such as individual stage observations or
variability about the reported mean values are not available through the
existing interface. Recovery of these values would require either direct
access to the database or extension of the online computation
capabilities. These limitations have significant implications for
practical use of the existing system for computation of AHPS program
performance measures as will be described below in Section 5.

The tests performed in this study were limited to verification sites
from the Missouri Basin, North Central, and Ohio River Forecast Centers.
Edwin Welles of the Office of Hydrologic Development contacted each of
these forecast centers to determine at what sites the basic AHPS
capabilities had been previously implemented. This enabled the
statistics to be\
further stratified into AHPS and non-AHPS categories. A listing of the
verification sites and their AHPS status is shown in Table 1.

## 3.2. Ensemble Forecast Verification Data

A small sample of the Ensemble Streamflow Prediction (ESP) system
forecasts and corresponding verification observations was used to
evaluate application of the probabilistic Brier score. This dataset was
the same as that used to conduct the study of Franz and Sorooshian
(2002). All the original files supplied by the NWS were obtained from
Kristie Franz (<franzk@uci.edu>) now with the University of California,
Irvine.

As described in detail by FS02, the data corresponded to 43 forecast
points from the Ohio River Forecast Center. Data consisted of the
forecast exceedance probabilities and individual ESP trace values,
corresponding river observations, and, for some forecast points, limited
historical observations. Separate forecasts were provided for mean
weekly stage and maximum monthly stage (30-day interval) over a period
from December 12, 2001 to March 24, 2002. Both forecast types were
issued with a 6-day lead time. An average of 11 forecasts of each type
was provided for each point. The data represented the most complete set
of forecast output, corresponding observations, and historical data that
could be obtained at the time of the study.

The observed forecast stages were provided hourly on average but in
several cases data were missing within the forecast intervals. In
particular, the observations did not extend to the end of the valid
period of the forecasts causing the last weekly forecasts and the last
five monthly forecasts to be excluded. Missing observations within other
forecast intervals forced FS02 to develop an elaborate set of rules for
treating these gaps. The same rules as described on page 9 of their
report were applied here to preserve consistency of the results.

The historical observations were not applied in this study, but the
existing data had to be supplemented with records of the flood stage at
each forecast point. These values were obtained from the individual AHPS
web pages for each point. Flood stage values could not be readily
obtained for three of the points and one additional point lacked valid
observations leaving 39 points available for further analyses. The
forecast points used and extracted flood stage values are listed in
Table 2.

# 4. Results

The forecast data sets were used to evaluate simple deterministic river
forecast accuracy measures and application of the Brier score to
probabilistic forecasts of river flooding. Details of each experiment
and the corresponding results are included in this section.

## 4.1. Deterministic Evaluations

Deterministic verification measures computed through the NWS hydrology
verification website were evaluated first. These tests were driven
primarily by the desire to learn whether a simple assessment of the
current accuracy of AHPS forecasts could be constructed rapidly from
existing data. Review of material prepared in support of the NOAA 3^rd^
quarter 2003 review revealed strong immediate interest in having a
measure to quantify any improvements in forecast accuracy resulting from
AHPS. Initial questions surrounded whether existing verification
activities could be performed separately for AHPS and non-AHPS forecast
points. The NWS verification database provided the most immediate means
of attempting such an assessment.

Computations were performed for the combination of verification points
from the Missouri Basin River Forecast Center (MBRFC), North Central RFC
(NCRFC), and Ohio RFC (OHRFC). Though there were no AHPS points in the
MBRFC subset, this combination was used to provide a comparable number
of AHPS (19) and non-AHPS (7) points. Statistics were first computed
separately for day 1, day 2, and day 3 forecasts, for fast, medium, and
slow response rivers, and for conditions corresponding to stage above
and below flood stage. It is desirable to keep the statistics separate
for each of the available categories to the extent allowed by the sample
size. This is because of the different physical processes governing the
different conditions and the potential for different contributions to
the forecast error. Preliminary results from other deterministic
forecast evaluations performed by Edwin Welles (personal communication)
clearly showed different sources for the dominant errors under high- and
low-flow conditions.

The corresponding statistics for mean absolute error for the below flood
stage forecasts are summarized in Figure 1a. The results suggest
improved forecast accuracy for the AHPS points over the non-AHPS points
for the day-1 and day-2 forecasts for fast and medium response rivers.
The AHPS forecasts have poorer accuracy, however, for the slow response
rivers and all the day-3 forecasts. The statistics for root mean squared
error (RMSE) are shown in Figure 1b. For RMSE, the AHPS forecasts also
show apparent improved accuracy for fast and medium response rivers on
day 3, but the forecasts still appear poorer for slow response rivers on
all days. The number of forecasts in each category is shown in Figure 1c
to be generally similar for the AHPS and non-AHPS points. Similar
statistics are not shown for the above flood stage forecasts because
there were too few cases for the individual fast, medium, and slow
categories (\< 30 fast and slow cases for the non-AHPS points).

To enable an assessment of the forecasts corresponding to above flood
stage observations, statistics were next examined for the combination of
the fast, medium, and slow river responses. The results are summarized
in Figure 2. While the mean absolute error for below flood stage
observations is roughly similar for AHPS and non-AHPS points (the
improvements for fast and medium response rivers are balanced by the
poorer results for slow response rivers), the AHPS forecasts appear
notably better for the above flood stage forecasts. The results reflect
38%, 41%, and 48% reduction in mean absolute error for the AHPS points
for the day-1, day-2, and day-3 above flood stage forecasts,
respectively. The AHPS below flood stage forecasts are improved by 12%
and 5% for days 1 and 2 but worsened by 4% for day 3. Relative to RMSE,
the AHPS forecasts appear improved for both above and below flood stage
forecasts for all periods.

It is not possible with the data currently available through the
verification database interface, however, to formally determine if the
suggested accuracy improvements are statistically significant. To do so,
the distribution of the individual forecast errors would be required in
addition to the mean values. While such values can clearly be computed
from the original data in the verification database, these results are
not available via the existing interface.

The results are sensitive to which of the limited number of verification
sites are included in the computation. The apparent large improvement in
the AHPS forecasts for above flood stage conditions is highly influenced
by small errors for the AHPS points from the NCRFC. Additional tests
were performed for subsets of the points including the combined points
from the NCRFC and OHRFC as well as the OHRFC points alone. This was
also done to more directly compare AHPS and non-AHPS forecasts within a
common region. For the combination of the NCRFC and OHRFC data (shown in
Figure 3), the mean absolute error results continue to suggest improved
accuracy for the AHPS points on all days for both above and below flood
stage cases. The results are less clear, however, based on RMSE and
poorer accuracy is indicated for the day-1 above flood stage AHPS
forecasts. For the OHRFC points alone (Figure 4), the results

![](media/image1.wmf){width="3.91875in"
height="2.6875in"}![](media/image2.wmf){width="3.9118055555555555in"
height="2.678472222222222in"}![](media/image3.wmf){width="3.91875in"
height="2.6875in"}

![](media/image4.wmf){width="3.8618055555555557in"
height="2.678472222222222in"}![](media/image5.wmf){width="3.88125in"
height="2.691666666666667in"}![](media/image6.wmf){width="3.88125in"
height="2.691666666666667in"}

![](media/image7.wmf){width="3.8618055555555557in"
height="2.678472222222222in"}![](media/image8.wmf){width="3.88125in"
height="2.691666666666667in"}![](media/image9.wmf){width="3.88125in"
height="2.691666666666667in"}

![](media/image10.wmf){width="3.8618055555555557in"
height="2.678472222222222in"}![](media/image11.wmf){width="3.88125in"
height="2.691666666666667in"}![](media/image12.wmf){width="3.88125in"
height="2.691666666666667in"}

are frequently worse for the AHPS forecasts, particularly for the above
flood stage cases. While these data no longer suggest improved forecast
accuracy at AHPS points, it should be emphasized that the number of
available points for these subsets is small. This sensitivity to the
small number of samples emphasizes the need for additional studies with
more data and detailed computations of statistical significance before
improved accuracy is claimed for the AHPS forecasts.

An initial test was performed in support of the long-term goal of
assessing changes in the forecast accuracies over time. The
investigation explored whether any change in the accuracy measures could
be observed over time. Preliminary generation of statistics over 3-month
intervals revealed a strong seasonal cycle in the statistics and implied
that the statistics should be evaluated over periods of at least a year.
With the present limited temporal extent of the database, only 2
year-long periods are available. Generation of statistics for the two
years showed increased errors for the AHPS points in the second year but
little significance can be attributed to the data.

All the computations were simplified in the sense that they did not
consider the exact date at which points became AHPS points. The
statistics were generated once for the entire period of April 2001 to
July 2003 and points that became AHPS points were assumed to be AHPS
points throughout. More detailed computations should treat the sites on
a point-by-point basis by individual month as necessary. Independent
compilation of the statistics outside the web interface will be
necessary.

## 4.2. Probabilistic Evaluations

The small sample of ESP forecasts and corresponding verification data
used in the work of FS02 and obtained from Kristie Franz was next used
to test application of the Brier score for evaluation of probabilistic
forecasts. In their previous work, FS02 applied the data to computation
of a rank probability score, rank probability skill score, and measures
of discrimination and reliability. A large fraction of the data could
not be used for these computations due to the lack of needed historical
data. The new computations were undertaken to determine if the
computation of the Brier score could be more easily undertaken and the
results more readily interpreted for programmatic application.

Application of the Brier score requires selection of a threshold for
which the event can be said to occur or not occur. The event of interest
was taken to be the occurrence of flooding and the threshold river stage
was defined as the flood stage. Forecast data were then evaluated based
on the forecast probability of either the weekly mean or monthly maximum
monthly river stage exceeding flood stage. While flooding was deemed to
be the event of most interest, the measure could also be evaluated for
conditions of drought or low flow if a suitable low-flow measure could
be constructed with consistency between all points.

The Brier score was computed both overall and on a point-by-point basis
for the forecasts of weekly mean and monthly maximum stage. The
resulting scores for the weekly mean stage forecasts are shown in Figure
5. Because of the low frequency of occurrence of flooding (particularly
in the weekly mean) and corresponding low flooding probability typically
assigned to forecasts, the Brier scores are extremely good (low). For
all the points combined, a Brier score of 0.00011 implies essentially
perfect accuracy. Of the 343 forecasts evaluated, flooding based on the
weekly mean value occurred only once. While the no-flood forecasts
dominate the statistics, it is interesting to note that in the one case
where flooding did occur, it was forecast with near 100% certainty. This
further supports the computed accuracy near 100%.

To reduce the dominance of no-flood cases on the computed scores, a
Brier skill score was computed based on a reference forecast that
flooding never occurs (probability of flooding is always 0%). For all
the points combined, the no-flooding reference forecast gives a Brier
score of 0.0029 resulting in a Brier skill score of 0.96. The
corresponding apparent 96% improvement in forecast accuracy resulting
from the use of the actual forecasts is due to the one case of flooding
being forecast with near perfect certainty.
![](media/image13.wmf){width="6.495138888888889in"
height="3.748611111111111in"}

The Brier scores computed from all the forecasts of monthly maximum
stage that could be evaluated are shown in Figure 6. Again the
relatively low frequency of flooding events results in generally low
Brier scores. The overall Brier score of 0.048 can be interpreted as
saying the forecasts are 95% accurate. Direct application of a Brier
score computed in this manner would be of limited use as a performance
measure since the apparent accuracy is misleadingly high and there is
little room for future improvement. Computation of a Brier skill score
based on reference forecasts that flooding never occurs, however, better
reflects the overall merit of the forecasts. For the combined points the
resulting Brier skill score is 0.17 suggesting that the operational
forecasts provide an improvement of only 17% over simple forecasts of no
flooding.

Additional information on the potential value of the forecasts can be
obtained considering only those points and forecasts where flooding
occurred. Scores computed in this manner can be interpreted similarly to
probability of detection measures. For the monthly maximum forecasts,
there are more instances in which flooding was observed (13 out of 223
forecasts) and examination of the scores for individual points reveals
that the Brier scores are poorer (higher) for the sites where flooding
was observed. If only the four sites where flooding occurred at some
time during the evaluation period (BBVK2, CARI2, PSNW2, and WLBK2) are
considered but scores are computed for all forecasts at these points,
there are 13 incidences of flooding in 27 forecasts and the
corresponding overall Brier score is 0.29. If the cases are further
limited to only those 13 forecasts where flooding occurred, the results
shown in Figure 7 are obtained. The overall Brier score of 0.54 implies
that the forecasts are only 46% effective in predicting flooding when it
does occur.

While the poor scores are the result of low forecast flooding
probabilities, the PSNW2 and overall scores are, in fact, unfairly poor
due to a likely measurement error. The Brier score of 1.0 for site PSNW2
corresponds to a perfectly bad score. For the two apparent instances of
flooding at that location, the forecast flooding probability was 0.
Detailed examination of the stage observations at that site reveals that
the conclusion that flooding occurred is based on one questionable
measurement. Half hourly observations for a period of several hours
indicate a stage consistently between 2.6 ft and 2.8 ft with one
observation in the middle of the period of\
28 ft. It seems likely that the decimal point was misplaced in this
observation. While no other observed data was examined in this detail,
additional quality control procedures may be desirable. Excluding site
PSNW2, the overall Brier score for predictions of flooding when it was
observed is 0.46 suggesting an accuracy of 54%.
![](media/image14.wmf){width="6.4944444444444445in" height="3.65in"}

The individual forecast probabilities for flooding when it occurred at
the other sites are shown in Table 3. Though flooding was forecast
correctly with perfect certainty in one instance, in all the other cases
the forecast probabilities did not exceed 38%. These results demonstrate
poor forecast skill at such longer periods and potential dependence on
non-AHPS related parameters.

![](media/image15.wmf){width="5.793055555555555in"
height="4.017361111111111in"}

# 5. Implications for Selection of Performance Measures

## 5.1. Deterministic Measures

The results of the deterministic evaluations suggest that basic
deterministic performance measures can potentially provide viable
near-term information on at least one aspect of the performance of AHPS
forecasts. While the measures clearly do not address probabilistic
content, not all forecasts generated within AHPS incorporate
probabilistic information. The 5-day outlooks on the hydrographs provide
a deterministic forecast and are one of the required base AHPS
capabilities. The initial estimates of mean absolute error and root mean
square error apparently demonstrate an improvement in forecast accuracy
at AHPS locations relative to non-AHPS locations. Such information is
potentially valuable for providing immediate support for the positive
impact of AHPS.

The initial evaluations, however, reveal several concerns that need to
be addressed for programmatic use of a similar deterministic measure. A
primary issue is how the measure should be characterized. Expression in
terms of mean absolute error or root mean square error is likely too
technical for use at the program level. Characterizing the accuracy in
terms of percent improvement over pre-AHPS or non-AHPS values provides
meaningful immediate results but is less desirable for continued
application over time. Goals for percentage improvement over pre-AHPS
values seem more arbitrary than specific accuracy targets and repeated
computations relative to non-AHPS values become less meaningful as the
number of non-AHPS points decreases. Expressing the forecast error as a
percentage relative to the observed stage is more direct and
understandable, but requires additional data.

The existing NWS verification database and web interface are inadequate
for computation of this measure. First, the interface does not enable
access to all the required information. While the observed stage is
required and used to compute the available error statistics, the values
cannot be directly accessed to enable further computation of the error
as a percentage of the observed stage. Data required to demonstrate
statistical significance of any differences are also currently
unavailable. Second, the amount of data available in the database is
limited. Additional verification sites would potentially enable a more
representative assessment of the performance of AHPS. Consideration of
both existing and any added sites, however, should address and avoid the
possibility of serial correlation. Finally, interactive generation of
the statistics will prove impractical for regular computation of the
statistics. Application of a deterministic performance measure will
require direct access to the database, construction of an enhanced
interface, or development of an independent verification system.

It may also be difficult to demonstrate continuous improvement with time
for any such measure. The presence of a strong seasonal cycle in
streamflow argues for computation of statistics over intervals of at
least a year. The existing database only supports generation of
statistics for two such intervals. Over short periods, climatological
variations in streamflow could easily overwhelm any changes in forecast
accuracy. With a limited number of verification points it is difficult
enough to obtain meaningful overall statistics, let alone for shorter
time periods.

## 5.2. Probabilistic Measures

Application of the Brier score enables one scientifically based, easily
communicated assessment of probabilistic forecasts. The initial results
emphasize, however, the importance of defining a reference for the
score. Due to the low frequency of flooding events, the forecasts appear
very accurate when directly evaluating all forecasts. One reference
considered was a perpetual forecast of no flooding. This reference could
also be changed to the climatological likelihood of flooding. For these
choices, the accuracy measure becomes a skill score and is expressed as
a percent improvement over the reference forecast. Interpretive meaning
was also enhanced by restricting computation to cases of observed
flooding as with a probability of detection measure. Under these
restrictions, the measure can be more simply expressed as percent
accuracy. Avoiding the use of a skill score is desirable for
programmatic application and the results based on flooding accuracy when
it was observed to occur were the most informative of the other
possibilities evaluated.

The results also suggest that a flooding accuracy measure might be best
applied to exceedance forecasts on a time scale of a week or less.
Forecasts of mean stage or flow over time scales of a week are less
useful since flooding will be observed in the mean only in the most
extreme cases. Evaluation of exceedance forecasts over periods
approaching a month will likely provide poor results because of a
current lack of skill over such long periods. Forecast flooding
probabilities were low for all but one of the cases where the maximum
monthly stage exceeded flood stage. The low probabilities also
illustrate the difficulty in trying to apply a probability of detection
measure based on a probability threshold as presented in section 2.1.
Only one forecast would have qualified had a probability threshold been
set higher than 40%.

The primary weakness of a Brier score associated with the occurrence of
flooding is clearly illustrated, however, in the very small number of
events in the sample data set. This is the main argument in favor of
using the more complex rank probability score that considers multiple
categories over the Brier score that addresses only the occurrence or
non-occurrence of an event. The RPS addresses the entire distribution of
the forecasts as favored by Steven Gallagher and other managers
surveyed.

Additional comparison of the relative merit of the RPS and Brier score
is possible through further interpretation of the sample RPS results
presented by FS02. In their Figure 3, they present the RPS for ten
points from the OHRFC sample dataset where they have used 8
non-exceedance probability categories (\< 0.05, 0.05-0.10, 0.10-0.25,
0.25-0.5, 0.5-0.75, 0.75-0.90, 0.90-0.95, and \> 0.95). The RPS scores
(0.6 to 1.65) can be expressed as an accuracy figure if one considers
that the maximum RPS score for 8 categories is 7 and scales the values
accordingly (0 = 100% accuracy, 7 = 0% accuracy). The range of such an
accuracy figure for the ten points would then be between approximately
91% and 76%. The apparent high accuracy is enhanced by the large central
categories (0.25-0.50 and 0.50-0.75 non-exceedance). If these category
ranges were defined differently, a different apparent accuracy figure
would result. This illustrates the difficulty in trying to interpret an
RPS as an accuracy figure. The sample application by FS02 also
illustrated the difficulty in obtaining adequate historical data to
properly define the streamflow categories for a climatological
reference. Out of the 43 sites, a meaningful rank probability skill
score could be computed for only 10. While the RPS can be computed from
the forecast traces alone, use of historically based categories can help
interpret the score when a skill score is not used. Both the Brier score
and RPS have different strengths and weaknesses for application to a
program measure.

Overall, it is apparent that programmatic application of any
probabilistic accuracy measure will require a significant new effort to
regularly collect and archive the probabilistic forecasts and
corresponding observations. The measures cannot be practically applied
to an ongoing programmatic assessment with the data available today.

# 6. Recommendations

## 6.1. Performance Measures

Based on the initial studies described in this report and the earlier
work of FS02, continued research into three forecast accuracy
performance measures is recommended.

-   Deterministic accuracy of daily river stage forecasts.

-   Accuracy of probabilistic weekly exceedance forecasts of river stage
    or streamflow.

-   Flood forecast accuracy.

The first two measures fall under the realm of overall forecasts of
river stage or streamflow and can be categorized under the general
heading of AHPS river forecast accuracy. Continued consideration of
multiple measures is important at this point because it is difficult to
capture all the relevant program characteristics within a single
measure. Moreover, it is hard to know which measures will function best
until they can be evaluated with more data.

### 6.1.1. AHPS Deterministic River Forecast Accuracy

The first recommended performance measure is a deterministic assessment
of the accuracy of the mean daily streamflow forecasts for days 1-3.
This measure is closely related to the tests described above using the
existing NWS verification database. The primary benefits of this measure
are that it captures the existing short-term forecasts, is simple to
express, and does evaluate the full range (low flow to flooding) of the
river forecasts. It also can be implemented rapidly to provide potential
support for the program. Its primary weakness is that it fails to
capture the probabilistic elements of AHPS.

This measure would serve to evaluate the hydrograph predictions
currently provided at each AHPS point. At present these are
deterministic forecasts and are the only web product for predictions on
the short-term time scales of less than a week. Inclusion of forecasts
within this short term was also recommended for consideration by Steven
Gallagher. While probabilistic forecasts are a primary focus of AHPS, as
long as deterministic forecasts remain a program element, it is
worthwhile considering a measure of their accuracy.

For use as a programmatic measure, it is best expressed as percentage
accuracy computed relative to the observed stream measure. The measure
is expressed here in terms of streamflow for application to the daily
mean. Use of stage was also considered for consistency with the
verification database and the format of the published hydrographs but
mean daily stage is more poorly defined. Presentation for days 1-3
represents a compromise over several suggestions. David Brandon of the
Colorado Basin RFC suggested evaluation of the figure for days 1-10.
Forecasts out 5 days are available in the existing AHPS hydrographs and
days 1-3 are evaluated in the verification interface. Steven Gallagher
requested a single day in the short term, possibly day 3. If ultimately
presentation of results for only a single day is desired, further tests
over a limited range of days will help indicate which day is most
useful. Initial required modifications to existing capabilities would
include enhancement of the verification database for expression as a
percent accuracy and validation in terms of streamflow.

### 6.1.2. AHPS Probabilistic River Forecast Accuracy

The second recommended performance measure is an assessment of the
accuracy of the weekly mean streamflow exceedance forecasts based on the
rank probability score. This measure would provide the single most
comprehensive evaluation of the forecasts. Through application of the
RPS it is possible to do a probabilistic assessment of the entire
distribution of river forecasts. The measure would be consistent with
and evaluate the formal weekly chance of exceedance products on the AHPS
web pages. Both of these factors are important strengths based on input
on desirable characteristics provided by Steven Gallagher. The most
significant weakness of the measure is related to whether it can be
expressed simply and meaningfully. This weakness may likely be
overshadowed, however, by the other strengths. Even should the measure
prove too complicated for use at the program level, it is highly
relevant at the science level. For use at the science level the measure
could be supplemented with discrimination and reliability measures as
advocated by FS02.

For programmatic use, it is recommended that the measure be expressed by
scaling the raw RPS to a percent accuracy figure as described
previously. Alternate referencing to a climatological forecast is less
desirable since it introduces a skill score. In addition, a raw score
can be computed directly from the forecast traces rather than extensive
historical data. For this approach the categories should be chosen
consistently between all points and in agreement with the AHPS web
products. The results will be useful as a comparative measure over time
and between points but will be hard to interpret as a physical accuracy.
Use of historically based categories, if possible, might help
interpretation even without use of a skill score.

Several possible forecast periods have been considered for presentation.
Initial recommendations are tied to the weekly forecasts because of
their current availability through the AHPS web pages. The periods
include the week 2 (first forecast period) and week 4 (third forecast
period) forecasts. An evaluation based on a composite of the week 4
through 6 forecasts was also discussed. Assessment of the longer-term
forecast accuracy is desirable for demonstrating programmatic impact but
it is unclear where there is currently sufficient skill. Application to
the week 2 forecasts provides better likelihood for initial success.
Final selection should ultimately be dictated by where there is forecast
skill and AHPS activities most influence the measure. Application to
shorter-term forecasts should be investigated if the forecasts become
available.

Evaluation in terms of the mean streamflow is specified here for
consistency with the historical records of the United States Geological
Survey (USGS). The primary long-term quantities available through the
USGS web pages include the daily mean and annual peak streamflow. The
measure could also be evaluated in terms of weekly maximum or stage
forecasts if there was stronger justification. Weekly maximum forecasts
are perhaps more relevant to flooding predictions and will form the
basis of the following flood forecast measure.

### 6.1.3. AHPS Flood Forecast Accuracy

The third recommended performance measure is an assessment of the
accuracy of predicted flooding in the weekly maximum stage exceedance
probability forecasts. The measure would be computed using the Brier
score for the event of river stage exceeding the specified flood stage
of each forecast point. As with the probabilistic river forecast
accuracy measure, this measure provides a direct evaluation of the
weekly chance of exceedance products. It can be applied simply and
requires no additional historical data. While application of the Brier
score is limited to a single event rather than the entire forecast
distribution, this is not necessarily a weakness. Use of the Brier score
should be considered complementary to the RPS rather than an alternative
as some users are likely more interested in the occurrence of specific
conditions such as flooding than the overall distribution of streamflow.

To avoid influence of the low frequency of flooding events without
introduction of a skill score, it is recommended that the measure be
computed considering only events where flooding occurred. In this manner
the measure can be interpreted like a probability of detection. For
programmatic application, the Brier score should be interpreted as a
percent accuracy as described in the trial application in section 4.2.
Selection of a forecast period for evaluation should closely follow the
discussion for the probabilistic river forecast accuracy measure.
Assurance of potential AHPS-related skill is particularly important for
a flooding measure where precipitation can have a significant impact.

## 6.2. Data Collection/Archival

Perhaps the most significant conclusion of both this study and the
previous work by FS02 is that archived records of the forecast and
observational data required to compute and evaluate the potential
performance measures are critically limited. The inability to
conclusively identify and baseline an accuracy-based performance measure
for AHPS at this time is directly related to the shortage of available
operational data. Before making a final selection of the performance
measures to be used within the program it is essential to ensure the
regular collection and archival of sufficient data to fully evaluate all
the potential measures. Archival should begin as soon as possible since
it will take time to accumulate the required data.

The required data can be broken into three categories:

-   forecast data,

-   verification data, and

-   historical data.

The primary required forecast data are the operational ensemble forecast
traces. All the original forecast values and corresponding probabilities
can be regenerated from the forecast traces. The plots generated and
displayed on the web pages are inadequate on their own. While it is
recommended that the forecast traces be regularly archived at each RFC
whenever a forecast is produced, further evaluations can be performed
using data from a subset of AHPS forecast points. Any subset should be
selected to avoid serial correlation and ensure representative sampling
over all possible river classifications such as response time. Franz and
Sorooshian (2002) also identified other additional data that might be
archived to facilitate reanalysis studies.

To enable verification of the forecasts, the observations corresponding
to the forecasts must also be regularly archived. The data must be
consistent with the forecasts both in terms of measured quantity (e.g.,
stage or streamflow) and sampling period and strategy (e.g., mean weekly
or maximum monthly). The data must also span the period corresponding to
the forecast.

Finally, historical data are also required to construct streamflow and
stage climatologies. While use of skill scores requiring climatological
references may not be desirable for use at the programmatic level, the
data can still be used to define categories for the RPS and may
facilitate applications at the science level. As noted for the
verification data, the historical data should also be fully consistent
with the forecast quantities.

## 6.3. Additional Analyses

When sufficient data has been archived, the proposed measures should all
be evaluated for common sets of forecast points and time periods.
Analysis with more comprehensive data will better identify which
measures are of most value in characterizing the program. The studies
will reveal where the forecasts have potential skill and where there is
a good chance of demonstrating improvements through AHPS activities. It
is important to ensure that the measure scores are influenced by factors
controlled by AHPS and not parameters such as certain meteorological
products that are independent. The evaluations possible at this time
were unable to consider a wide range of forecast lead times or carefully
isolate different river conditions. Ideal measure selection would
incorporate parallel computation of the different measures over an
extended period of data to illustrate where skill exists and identify
possible weaknesses under different conditions. These steps are
important if the measures are to be the primary indicator of success of
the AHPS program. Measures suggesting poor skill will be valuable for
internal evaluation of the program and management of science activities,
but could reflect unfavorably on the program if presented publicly. The
extended analyses should also serve to define the baseline for any
selected performance measures.

Alternate ways to construct streamflow and stage climatologies should
also be explored. This new activity was also recommended by FS02. As
noted above, the climatologies are still of value even if not directly
applied to the computation of skill scores. Preparing formal
climatologies of streamflow distributions requires extensive historical
data that might not be readily available at all desired locations.
Identification of extreme, narrow exceedance categories such as the
outer 1-2% requires longer historical records. One alternative to
consider would use established relations for flooding flows (Jennings et
al., 1994) as applied in threshold runoff estimates (e.g., Reed et al.,
2002). Initial flow distributions could be constructed at sites where
adequate data exists. These distributions could then be mapped to other
locations without sufficient data by adjusting the levels for 5-, 10-,
and 100-year floods as derived from the equations. Trials at locations
with data will be necessary to determine the potential accuracy of the
technique.

# 7. Concluding Notes

Several potential forecast accuracy measures were evaluated for
application as a program-level performance metric for AHPS. The relative
strengths and weaknesses of the different measures were illustrated but
the available archived forecast and verification data were inadequate to
fully demonstrate what measures would most favorably reflect on the AHPS
program. Based on the results, recommendations were made for further
evaluation of three specific measures through the creation of improved
archives of corresponding forecast and verification data. The more
comprehensive analyses should better enable selection of a final measure
for inclusion in programmatic presentations. Final selection should also
be influenced by user desires and requirements and further external
input may be desirable in these evaluations.

The AHPS forecast accuracy performance measures provide a strong
connection between the programmatic and scientific activities. Accuracy
measures could be applicable both at the programmatic and scientific
levels. They are applied here at the program level because better
forecast accuracy is identified as one objective of AHPS. Achieving the
desired forecast accuracy levels then becomes an objective for the AHPS
science activities. Appropriate science level performance measures can
be defined to quantify progress in achieving this objective. Possible
measures could range in scope from implementation of required scientific
tools to rate of accuracy improvements.

While this analysis has focused entirely on potential accuracy measures,
there are several overall potential problems that affect the application
of any such metric. The largest of these is related to sampling and
obtaining adequate statistics. Because of the episodic nature of climate
and streamflow, it may be difficult to obtain meaningful statistics over
relatively short periods of time. This is particularly relevant if one
is attempting to track year-to-year evolution of the program.
Furthermore, there is significant potential for the statistics to be
influenced by serial correlation between verification sites. Available
verification data will be further limited by avoiding possible
correlation. Any accuracy computations must also be mindful of possible
human activities not considered in the forecasts. The existence of dams
and regulation of flow could potentially affect measurements at
verification sites. Computation of performance measures must either
neglect such sites or attempt to compensate for the human activities.
Clearly the merit of any accuracy performance measure is tied to the
quality of the verification statistics that can be obtained. These
concerns, voiced strongly in interviews with Frank Richards, emphasize
the importance of incorporating additional programmatic performance
measures addressing such other topics as product use and user
satisfaction.

# Acknowledgements

This analysis was conducted as part of a NOAA rotational assignment.
Salary support was provided through the NOAA Environmental Technology
Laboratory while travel and other expenses were supported by the NWS
Office of Hydrologic Development. The many helpful suggestions and
comments of NWS and OHD staff, particularly Edwin Welles, John Ingram,
Robert Brown, and Gary Carter is gratefully acknowledged. Thanks are
also due to Marty Ralph for his encouragement for participation in the
rotational assignment program.

# References

Brier, G. W., Verification of forecasts expressed in terms of
probability, *Mon. Wea. Rev*., **78**, 1-3, 1950.

Epstein, E., A scoring system for probability forecasts of ranked
categories, *J. Appl. Meteor*., **8**, 985-987, 1969.

Franz, K. J., and S. Sorooshian, Verification of National Weather
Service Probabilistic Hydrologic Forecasts, University of Arizona,
report prepared for the National Weather Service, 46 pp., 2002.

Hamill, T. M., S. L. Mullen, C. Snyder, Z. Toth, and D. P. Baumhefner,
Ensemble forecasting in the short to medium range: Report from a
workshop, *Bull. Amer. Meteor. Soc*., **81**, 2653-2664, 2000.

Hartmann, H. C., T. C. Pagano, S. Sorooshian, and R. Bales, Confidence
Builders: Evaluating seasonal climate forecasts from user perspectives,
*Bull. Amer. Meteor. Soc*., **83**, 683-698, 2002.

Jennings, M. E., W. O. Thomas Jr., and H. C. Riggs, Nationwide summary
of U.S. Geological Survey regional regression equations for estimating
magnitude and frequency of floods for ungaged sites, *USGS
Water-Resources Investigations Rep. 94-4002*, U.S. Geological Survey,
Menlo Park, CA, 1994.

Mason, S. J., and N. E. Graham, Conditional probabilities, relative
operating characteristics, and relative operating levels, *Wea.
Forecasting*, **14**, 713-725. 1999.

Murphy, A. H., A new vector partition of the probability score, *J.
Appl. Meteorol*., 12, 595-600, 1973.

Reed, S., D. Johnson, and T. Sweeney, Application and national
geographic information system database to support two-year flood and
threshold runoff estimates, *J. Hydrologic Engineering*, **7**, 209-219,
2002.

Wilks, D. S., Statistical Methods in the Atmospheric Sciences, Academic
Press, 467 pp., 1995.