# **A Hydrologic Rainfall Product Comparison Study In the Tar River Basin, NC:** **QPE SUMS** **and MPE**

National Severe Storms Laboratory, National Weather Service Office of
Hydrologic Development, and Vieux and Associates, Inc.

5/10/2004

### Executive Summary

The efforts described herein are a spin-off of a project called IFLOW
(Inland Flooding Observation and Warning). Project IFLOW was created to
establish a research and development demonstration program for
evaluation and testing of new technologies and techniques to produce
accurate and timely identification of coastal and inland floods and
flash floods. The project was undertaken as the result of a series of
discussions between the National Oceanic and Atmospheric Administration
(NOAA) National Sea Grant Director and the NOAA Oceanic and Atmospheric
Research (OAR) Assistant Administrator. Their discussions centered on
the potential benefits that could be realized with increased
collaboration between the Sea Grant (SG) network programs and the twelve
OAR research laboratories. During an initial meeting in 1999 between the
NSSL and the SG management, the basic framework for what has now become
the IFLOW Project was devised.

As IFLOW developed, NSSL and the National Weather Service (NWS) became
aware of similar research efforts each group was undertaking in the Tar
River Basin in North Carolina. During the first meeting of the two
groups on December 12, 2001, an action item called for the creation of a
project for a transparent comparison of quantitative precipitation
estimates resulting from the NWS and NSSL approaches. Using \$30K in

seed money provided by the NWS Eastern Region Headquarters, the NWS
Office of Hydrologic Development (OHD) and NSSL joined together in late
2002 to investigate and compare respective techniques used to estimate
precipitation, MPE and QPE SUMS. NSSL's corporate partner in Project
IFLOW, Vieux & Associates, assisted the group by providing the use of
their runoff model, V*flo*™, to help assess the hydrologic impact of the
QPE provided by both MPE and QPE SUMS. The resulting study is known as
the Intercomparison Project.

The results of the Intercomparison Project are presented in this
document.

### Introduction and Methodology

The purpose of this study is to evaluate quantitative precipitation
estimates (QPE) produced by QPE SUMS, a product of the National Severe
Storms Laboratory, and the Multisensor Precipitation Estimator (MPE), a
product of the Office of Hydrologic Development. The pilot study focuses
on flood and flash flood prediction in the Tar River, NC, which during
Hurricane Floyd experienced severe flooding. Expansion to other areas is
expected. That QPE algorithms produce accurate inputs for hydrologic
modeling is critical to flood forecasting and other hydrological
services performed by the National Weather Service. The performance of
QPE SUMS and MPE rainfall estimates is evaluated using 1) 24-hour rain
gauge observations from the Cooperative Observing Network (COOP)
reported daily, and 2) streamflow simulations from a hydrologic model as
compared to streamflow observations.

The first results presented in this Intercomparison Project are based on
comparisons of QPESUMS and MPE products with rain gauge amounts. The
second results are based on a hydrologic modeling study to evaluate how
the different QPE inputs may affect streamflow simulations as compared
to streamflow observations at basin scales on the order of 100 to 1000
km^2^. Simulated hydrographs are compared to observed streamflow in
order to assess whether important rainfall rate and volume information
is captured for accurate transformation into streamflow. The hydrologic
evaluation is limited in scope in that model simulations are made only
ofor a sngle event in October 2002 at selected locations throughout the
Tar River, NC.

### Evaluation of QPESUMS and MPE using COOP Gauges

Comparisons are made between all QPE SUMS and MPE products to rain gauge
data from the cooperative observing (COOP) network. The COOP rain gauges
have a nominal reporting interval of 24 hours. The comparisons are made
daily and then lumped together for each month from October 2002 through
April 2003. The scatter plots are provided in the Appendix (see below
for description of various MPE and QPE SUMS products) and the
statistical results are summarized in Table 1.

Due to the fact that all MPE products were generated in real time at the
NWS Southeast River Forecast Center (SERFC) whereas the gauge-adjusted
QPE SUMS products were generated in a post analysis mode at NSSL, and
that the availability of radar and rain gauge data was not the same
between MPE and QPE SUMS, caution must be exercised in interpreting the
results. For example, both QPE SUMS and MPE products were hampered by
data outages. It was discovered that the amount of rain gauge data used
in MPE was consistently and significantly smaller than that used in QPE
SUMS while QPESUMS experienced prolonged periods, on the order of weeks,
of missing radar data from the Raleigh WSR-88D radar. MPE, which went
into operation at SERFC in Oct 2002, also experienced occasional radar
data outages from various sites in the WSR-88D network. It was also
discovered that a small but significant subset of rain gauge reports
used in the verification data set were also incorporated in the
gauge-adjusted products generated by both MPE and QPE SUMS. This renders
the gauge-adjusted products to show better performance than what may be
expected in a real-time environment As such, the evaluation presented
herein does not constitute a completely independent validation of the
precipitation algorithms. Nevertheless, meaningful observations and
inferences may still be made as summarized below. Because the evaluation
period covered mostly a cool season for which radar-only QPE is
generally subject to a larger error, the availability of rain gauge data
was a very important factor in multisensor QPE. To assess the
performance of MPE under the same conditions of rain gauge data
availability, MPE was also run retrospectively using the same rain gauge
data set used by QPE SUMS. This result is referred to herein as the MPE
reanalysis.

There are six MPE products represented in the scatter plots shown in
Appendix; Gageonly, Bmosaic, Lmosaic, Mmosaic, Rmosaic and Xmrg. The
Gageonly product is based only on rain gauge data and represents the
gage-only estimate. It is obtained via the Single Optimal Estimation
technique (SOE, Seo 1998a). The Rmosaic product is a mosaic of WSR-88D
Digital Precipitation Array (DPA) products from multiple sites and
represents the radar-only estimate. The mosaicking is based on radar
sampling geometry and beam blockage considerations (Breidenbach et al.
2001). The Bmosaic product is the mosaic of mean field bias-adjusted
DPAs and represents a gauge-adjusted radar precipitation estimate. The
mean field bias adjustment is based on rain gauge data via a multiscale
recursive filter (Seo et al. 1999). The Lmosaic product is the local
bias-adjusted Rmosaic field and represents a second gauge-adjusted radar
precipitation estimate. The local bias adjustment (Seo and Breidenbach
2002) is a bin-by-bin (a single bin is approximately 4x4km^2^)
application of the mean field bias algorithm. The Mmosaic product is
based on multisensor merging of rain gauge data and the Bmosaic product.
The merging is based on the bi-variate extension of SOE (Seo 1998b). The
Xmrg product represents the precipitation estimate chosen to be the
'best' by the forecaster at the RFC, and reflects manual quality control
steps, if any, taken by the forecaster. In the automatic mode, the Xmrg
product defaults to the Mmosaic product. The MPE reanalysis product
suite is the same as above except that Xmrg is replaced by Mmosaic2.
Note that the Xmrg product is not generated in reanalysis because there
is no human interaction involved. The Mmosaic2 product is the same as
the Mmosaic except that the cross correlation parameters in radar-gauge
merging were changed to assess the sensitivity to these parameters.
Generally speaking, the difference between Mmosaic and Mmosaic2 is minor
and, throughout the rest of this paper, we treat Mmosaic2 (denoted as
'Mmosaic2') as representing the 'best' MPE reanalysis product.

There are six QPE SUMS products represented in the scatter plots shown
in Appendix; LGC RAD, LGC MS, GC RAD, GC MS, TOTAL RAD and TOTAL MS. The
seventh QPE SUMS product, Gageonly, is based only on rain gauge via the
Barnes analysis (Barnes 1964). It is generally similar to the Gageonly
product of MPE and is not shown in t hi paper. The TOTAL RAD product is
the radar-only estimate as obtained from the WSR-88D Level II data and
mosaicked according to the best available radar coverage.. Te TOTAL MS
product is based on radar data for convection and calibrated satellite
data for echo deemed to be stratiform rain. A combination of radar,
model data, and lightning flashes are used in the convective-stratiform
segregation component. The LGC RAD product is the local bias-corrected
TOTAL RAD. The local gauge adjustment process computes a spatially
nonhomogeneous bias field based on the difference between hourly rain
gauge reports and multisensor QPEs. This difference field is analyzed on
a 1x1 km grid using a Barnes objective analysis scheme. The bias field
is then used to adjust the multisensor estimates. The_LGC MS product
from QPE SUMS is a local gauge correction applied to the multisensor
rainfall field (i.e. TOTAL MS). The GC RAD and GC MS products are the
mean field bias-corrected TOTAL RAD and TOTAL MS, where the mean field
bias is gven by the ratio of the sum of rain gauge data to that of the
collocateding rr-only or multisensor estimates at each hour,
respectively.

Table 1 summarizes the algorithm performance for the products from MPE
and QPE SUMS that agreed most favorably with the COOP gauges. The best
products from MPE and QPE SUMS are Mmosaic and LGC MS, respectively, and
Mmosaic2 for MPE reanalysis (see above for explanation)., Overall, the
MPE Mmosaic product has a bias (RATIO) greater than 1.0, but relatively
small root mean squared error (RMSE) and a correlation coefficient (RHO)
value closer to 1.0 than the QPE SUMS MS_LGC product. The RATIO is the
monthly bias of the rainfall estimates while the RMSE summarizes the
scatter, and RHO indicates the correlation with rain gauge amounts. The
difference in RATIOs is significant in that the MPE Mmosaic product
underestimated by approximately 25.1%. That the number of rain gauge
data available to MPE was consistently and significantly smaller than
that used in QPE SUMS is a large contributing factor to this bias. It
was also discovered that warm-season parameter settings were used in
real-time MPE throughout the evaluation period, which resulted in
exclusion of small precipitation amounts in bias estimation. Because
small precipitation amounts make up a significant fraction of total
precipitation in the cool season, such thresholding has a large impact
on long-term bias. Note that, with the same parameter settings, the
underestimation essentially disappears in Apr 2003 (i.e. the beginning
of the warm season). Note also that Mmosaic2, the MPE reanalysis product
based on the complete rain gauge data set and cool-season radar-gauge
pairing and parameter settings, shows much smaller biases (about 1%
overall). The dependence of MPE biases on rain gauge data densities
invites further investigation in regions with sparse rain gauge networks
(e.g., the Intermountain West). The QPE SUMS MS_LGC product has a very
low bias near 1.0, indicating overestimation by 0.7%. The increased
scatter in the QPE SUMS product, noted in higher RMSE and lower RHO
values, is believed to be due in part to less robust quality control of
reflectivity than that used by the WSR-88D Precipitation Processing
Subsystem (PPS, Fulton et al. 1998), which feeds radar rainfall
estimates to MPE. QPE SUMS uses raw radar reflectivity (level-II format,
provided by the CRAFT network) as input as opposed to processed rainfall
estimates. Advanced quality control steps are being investigated to
improve the quality of the QPE SUMS products. Performance of automatic
quality control of reflectivity data is also an issue with PPS: the
conspicuously poor performance of Mmosaic2 for February (see Table 1) is
due to extremely severe and extensive ground clutter returns from
anomalous propagation (AP) which disrupted gauge-based bias correction
and radar-gauge merging in MPE. It is well known (see, e.g., Seo et al.
1999) that bias correction based on rain gauge data from the current
hour alone, as practiced in QPE SUMS, is subject to large sampling
errors, resulting in significantly larger RMSE even though the bias may
be small. An additional analysis worth consideration is to 'cross-feed'
radar-only or radar-satellite estimates to MPE and QPE SUMS so that the
source(s) of larger RMSE may be ascertained.

Table 1. Statistics for the performance of the best MPE (Mmosaic from
real time, Mmosaic2 from reanalysis) and QPE SUMS products as compared
to 24-hour rain gauge reports from the cooperative observing network.
Best performance is noted in boldface. Also shown as reference is MPE
Gauge-Only (from reanalysis) .

  ----------- ----------------- ------- ------- ------- ------- ------- ------- ------- -----------
              Month             Oct     Nov     Dec     Jan     Feb     Mar     Apr     AVG

  RATIO       MPE Mmosaic       1.472   1.252   1.141   1.267   1.299   1.284   1.043   1.251

  (truth/     MPE Mmosaic2      1.062   1.042   0.952   0.946   0.953   1.029   0.959   0.991

  estimate)   QPE SUMS MS_LGC   0.985   0.839   0.981   1.056   1.076   0.979   1.037   **0.993**

              MPE Gauge-Only    1.010   1.028   0.967   1.064   1.071   1.028   1.021   **1.027**

                                                                                        

              Month             Oct     Nov     Dec     Jan     Feb     Mar     Apr     

  RMSE        MPE Mmosaic       5.183   2.872   2.361   2.881   3.582   4.016   3.094   3.427

  \(mm\)      MPE Mmosaic2      3.997   2.518   2.327   2.805   3.825   3.799   3.429   **3.243**

              QPE SUMS MS_LGC   4.783   4.111   2.441   4.574   3.741   3.693   3.753   3.871

              MPE Gauge-Only    4.520   2.565   2.569   2.511   2.936   4.001   3.975   3.297

                                                                                        

              Month             Oct     Nov     Dec     Jan     Feb     Mar     Apr     

  RHO         MPE Mmosaic       0.790   0.909   0.958   0.878   0.904   0.924   0.950   0.902

              MPE Mmosaic2      0.829   0.920   0.954   0.877   0.886   0.923   0.937   **0.904**

              QPE SUMS MS_LGC   0.815   0.815   0.948   0.664   0.879   0.931   0.927   0.854

              MPE Gauge-Only    0.782   0.915   0.943   0.893   0.927   0.914   0.913   0.898

                                                                                        
  ----------- ----------------- ------- ------- ------- ------- ------- ------- ------- -----------

### Evaluation of QPE SUMS and MPE using Hydrologic Simulations 

Evaluation of a QPE algorithm using streamflow necessarily requires a
hydrologic model to transform the precipitation input into streamflow.
The physics-based hydrologic model, V*flo*™, is used for hydrologic
evaluation of the QPE inputs. The V*flo*™ model of the Tar River basin
is developed on a 1x1-km resolution grid. Physics-based parameters are
derived from soils, land use/cover, and a digital elevation model (DEM).
The Ordered Physics-based Parameter Adjustment (OPPA) scheme developed
by Vieux and Moreda (2003) is applied to calibrate the model using
level-II radar data collected during Tropical Storm Allison (June 2001).
Adjustments were made to improve the hydrograph shapes for the 10-16
October 2002 event using level-II reflectivity for a single-radar (KRAX)
with a mean-field bias adjustment using storm totals from COOP gauges.

Hydrologic comparisons are made in the following sections by a) plotting
the storm total precipitation accumulations, b) showing individual
hydrographs at selected locations, and c) tabulating model ensemble
results that reflect the model parametric uncertainty.

**Storm Total Statistics**

After loading the QPE SUMS and MPE data into V*flo*™, storm total maps
are generated. The MPE data used in the hydrologic analysis are the
Mmosaic products produced in real-time at SERFC. Visible differences
between the products are evident. The following images show the storm
total patterns over the outline of the Tar River Basin. Fig. 1 shows the
storm total maps for several of the QPE products evaluated. The
resolution difference between MPE (4 x 4 km) and the QPE SUMS products
(1 x 1 km) is noted. There is general agreement between all the products
in generating the heaviest rainfall amounts in the headwaters of the
basin. The QPE SUMS and MPE products perform radar mosaicking based on
radar sampling geometry considerations. In this case, the mosaicking has
created a linear discontinuity north of the Tar River Basin, visible
especially in Figs. 1b-d. Other studies have indicated that these
boundaries result from calibration differences between adjacent radars.
Accurate correction of this apparent calibration discrepancy requires
further attention from both QPE SUMS and MPE developers.

  ----------------------------------------------------- -----------------------------------------------------
  ![](media/image1.jpeg){width="2.9166666666666665in"   ![](media/image2.jpeg){width="2.9166666666666665in"
  height="2.2222222222222223in"}                        height="2.2222222222222223in"}

  ![](media/image3.jpeg){width="2.9166666666666665in"   ![](media/image4.jpeg){width="2.9166666666666665in"
  height="2.2222222222222223in"}                        height="2.2222222222222223in"}
  ----------------------------------------------------- -----------------------------------------------------

Figure 1. Storm total precipitation from 0000 UTC on 10 October 2002 to
1500 UTC on 16 October 2002 from a) NEXRAD level-II data, b) MPE
Mmosaic, c) QPE SUMS MS_GC, and d) QPE SUMS MS_LGC.

**Event Hydrographs**

The following hydrographs were simulated in V*flo*™ using the
precipitation data depicted in the above storm total maps and the
calibrated parameters using results from an independent case study on
the Tar River Basin file. Hydrographs simulated for the furthest
upstream gauging location (Tar River, #1500) are shown in Fig. 2 for the
four different rainfall products depicted in Fig. 1. Simulated
hydrographs are shown at Tar River near Tar, NC, because they are least
affected by channel hydraulics, which plays a larger role at downstream
locations. All simulated hydrographs appear in solid; observed data
appear in stippled.

Similar matches to observations are achieved by using either the
level-II radar data or the MPE data as input. MPE requires a smaller
scalar (0.6) to calibrate the infiltration characteristics of the soils
as compared to using level-II data alone (it uses 0.8). The two QPE SUMS
products shown (Figs. 2c,d) require much smaller scalaers to be used for
soil infiltration, but higher degrees of soil saturation. All QPE inputs
cause an early time to peak discharge, even though they are using the
maximum scalaer of 1.5 for roughness. Because water is transferred to
the basin outlet too quickly, the model may require enhancements to
better account for channel hydraulic properties between rating curves,
soil physics, and/or subsurface runoff.

An anomalous result is seen in Fig. 2d where the QPE SUMS MS_LGC
produces a middle peak that is neither observed nor produced by the
other products. Further examination of the input data revealed that a
gauge reported \>1 in/hr at 1800 UTC on 13 October 2002, which results
in the large, middle peak in the simulated hydrograph. The QPE SUMS
product that uses a mean field bias adjustment (MS_GC; Fig. 2c) is not
affected by the erroneous gauge report. Both the QPE SUMS MS_GC and
MS_LGC products reproduce the second observed peak, but are in excess of
the peak and early in timing by \~24 hours. In these hydrographs, the
QPE SUMS products produce early timing results, whereas, the MPE and
level-II algorithms produce hydrographs with more consistent timing
compared to observed.

  ---------------------------------------------------- ----------------------------------------------------
  ![](media/image5.png){width="3.2597222222222224in"   ![](media/image6.png){width="3.2597222222222224in"
  height="2.4444444444444446in"}                       height="2.4444444444444446in"}

  ![](media/image7.png){width="3.2597222222222224in"   ![](media/image8.png){width="3.2597222222222224in"
  height="2.4444444444444446in"}                       height="2.4444444444444446in"}
  ---------------------------------------------------- ----------------------------------------------------

Figure 2. Simulated (solid) and observed (stippled) hydrographs at the
Tar River stream gauge from 0000 UTC on 10 October 2002 to 0000 UTC on
22 October 2002 using inputs from a) NEXRAD level-II data, b) MPE
(Xmrg), c) QPE SUMS MS_GC, and d) QPE SUMS MS_LGC. In this case, optimum
parameters were chosen that yielded the best hydrograph for each input.

**Parameter Ensemble Results**

For a hydrologic model to be used effectively to evaluate QPE inputs,
hydrologic uncertainty, i.e. the uncertainties associated with
transforming precipitation to streamflow using a model, should be taken
into consideration. In general, hydrologic uncertainty includes
uncertainty in model initial conditions, parametric uncertainty (i.e.
uncertainty in model parameters) and model structural uncertainty. The
purpose of this section is to assess the parametric uncertainty in
Vflo^TM^ simulations. In this hydrologic evaluation, QPE SUMS and MPE
products are used as hydrologic inputs to the V*flo*™ model and
evaluated in terms of modeled streamflow in comparison with observed.
The parametric uncertainty that is introduced in environmental modeling
is handled by running the model in ensemble mode. A score that is well
suited for assessing the performance of multicategory events using a
single observation is the Ranked Probability Score (RPS) (Wilks 1995). A
low RPS is an indication that the particular QPE is more likely to
produce a better model simulation of streamflow in the face of the
acknowledged (but not necessarily true) parametric uncertainty.
Generation of the RPS requires many model simulations for a range of
parameters. 180 simulations were performed for MPE and QPE SUMS products
for the October 2002 event. Model ensembles were created for each input
based on three sensitive parameters employed in the model: Manning
roughness factor, saturated hydraulic conductivity, and initial
fractional water content of the soils. The V*flo*™ model uses spatially
distributed parameters, so each parameter map is perturbed using scalar
multipliers that were varied from 0.25 to 1.5, in 0.25 increments for
the Manning roughness parameter; from 0.2 to 1.2 in 0.2 increments for
saturated hydraulic conductivity; and from 0.4 to 0.9, in 0.1 increments
for initial saturation. The RPS is then calculated for each of three
hydrograph objective functions: time of peak flow (Time), peak discharge
(Peak), and total volume (Volume).

The ranked probability scores (RPS) are summarized in Table 2 for the
three hydrologic variables characterizing the hydrographs: Time to peak,
Peak flow, and Volume. RPS values closest to 0 indicate the best
performance. The column to the right shows the average RPS scores for
each objective function over all stream gauges in the Tar River basin.

The QPE SUMS multisensor product with a mean field bias adjustment
(MS_GC) has the lowest overall, average RPS. This indicates that when
this product is subjected to the parametric uncertainty assumed for the
model (but not the uncertainty in the initial conditions or the model
structural errors), it provides the most probable streamflow predictions
when considering all gauging sites and the 3 characteristics of the
hydrographs. The MPE product yielded the best streamflow volume
predictions. The worst performers were the QPE SUMS gauge-only and the
level-II data. Recall that, the gauge-only products from both QPE SUMS
and MPE showed a better or mparable performance when compared to the
quasi-independent, 24-hour gauge dataset (i.e. the COOP network), but
their hydrologic performance is lacking compared to the radar and
multisensor products. This is a reflection that hydrologic processes are
highly nonlinear and scale-dependent, and indicates that the temporal
and spatial resolution of products that rely on remotely sensed radar
and satellite data are important and can improve hydrologic predictions

Table 2. Summary of ranked probability scores for the time-to-peak
(Time), peakflow (Peak), and integrated discharge volume (Volume) using
MPE and QPE SUMS estimators. Best performance noted in boldface.

  ----------- -------- ------ ------ ------ ------ ------ ------ ------ ------ ------ -------- ----------
  QPE SUMS    Gage     1500   1747   2506   2585   2770   2950   3000   3500   4000            Avg.

  TOTAL_MS    Time     0.75   1.55   2.26   3.31   1.18   2.70   1.70   3.08   1.26            1.98

  RPS         Peak     2.45   1.82   1.46   1.36   1.54   1.45   1.32   1.07   1.05            1.50

              Volume   2.47   1.74   1.66   1.55   1.34   1.01   0.98   1.01   1.09            1.43

                                                                                               1.64

  QPE SUMS    Gage     1500   1747   2506   2585   2770   2950   3000   3500   4000            

  TOTAL_RAD   Time     0.45   2.01   3.37   4.24   1.59   3.54   1.34   3.49   1.42            2.38

  RPS         Peak     1.26   1.11   0.95   0.87   1.29   1.21   1.19   0.83   0.83            1.06

              Volume   2.09   1.19   1.15   1.11   1.08   0.80   0.72   0.68   0.69            1.06

                                                                                               1.50

  MPE         Gage     1500   1747   2506   2585   2770   2950   3000   3500   4000            

  XMRG        Time     0.87   4.90   4.51   5.00   1.47   3.08   0.70   2.02   1.01            2.62

  RPS         Peak     2.13   3.31   1.51   1.34   0.98   1.04   1.53   1.56   1.33            1.64

              Volume   0.50   0.43   0.48   0.50   0.69   0.52   0.53   0.40   0.41            **0.49**

                                                                                               1.58

              Gage     1500   1747   2506   2585   2770   2950   3000   3500   4000            

  KRAX Level- Time     1.21   5.00   4.49   5.00   0.21   2.60   2.36   1.38   1.04            2.59
  II                                                                                           

  RPS         Peak     3.10   4.81   3.31   3.22   3.51   3.84   4.11   3.88   3.66            3.71

              Volume   0.29   0.99   0.90   0.79   1.51   2.24   2.36   1.77   1.51            1.37

                                                                                               2.56

  QPE SUMS    Gage     1500   1747   2506   2585   2770   2950   3000   3500   4000            

  MS_GC       Time     0.62   2.15   2.37   2.62   2.16   3.07   2.38   1.98   1.49            2.09

  RPS         Peak     1.05   0.9    0.68   0.65   1.19   1.6    1.63   0.86   0.81            **1.04**

              Volume   1.23   0.65   0.69   0.63   0.67   1.03   0.89   0.65   0.56            0.78

                                                                                      **BEST   **1.30**
                                                                                      -\>**    

  QPE SUMS    Gage     1500   1747   2506   2585   2770   2950   3000   3500   4000            

  MS_LGC      Time     1.52   2.21   1.79   2.19   1.08   1.85   1.79   2.49   1.72            **1.85**

  RPS         Peak     1.77   0.94   0.73   0.73   0.92   3.58   3.49   1.19   1.15            1.61

              Volume   1.37   0.63   0.64   0.58   0.69   3.02   2.40   0.75   0.60            1.19

                                                                                               1.55

  QPE SUMS    Gage     1500   1747   2506   2585   2770   2950   3000   3500   4000            

  RAD_GC      Time     0.43   3.92   4.58   5.00   1.10   3.00   1.07   2.72   1.26            2.56

  RPS         Peak     0.92   1.57   1.09   1.00   1.43   1.53   1.65   1.25   1.17            1.29

              Volume   0.97   0.61   0.66   0.59   0.77   0.80   0.84   0.62   0.53            0.71

                                                                                               1.52

  QPE SUMS    Gage     1500   1747   2506   2585   2770   2950   3000   3500   4000            

  RAD_LGC     Time     0.98   2.12   2.21   3.56   1.10   2.22   1.82   2.83   1.92            2.08

  RPS         Peak     1.91   1.18   0.89   0.82   1.05   3.77   3.62   1.16   1.17            1.73

              Volume   1.59   0.84   0.69   0.64   0.78   3.17   2.64   0.73   0.64            1.30

                                                                                               1.71

  QPE SUMS    Gage     1500   1747   2506   2585   2770   2950   3000   3500   4000            

  GAG         Time     1.04   4.35   4.79   5.00   0.40   2.48   1.02   3.58   2.49            2.79

  RPS         Peak     1.89   0.97   0.78   0.73   1.72   2.56   2.10   1.35   1.31            1.49

              Volume   1.93   0.74   0.62   0.66   0.55   1.56   1.13   0.59   0.56            0.93

                                                                                               1.74
  ----------- -------- ------ ------ ------ ------ ------ ------ ------ ------ ------ -------- ----------

### Conclusions 

Considerable insight into and experience with experimental methods for
hydrologic evaluation of QPE products have been gained from the
NSSL-NWS/OHD collaboration. It was found that working procedures used to
access and process operational data streams have important consequences
on the data quality and independence of verification data sets, one of
the lessons learned for future such endeavors. In addition to the
findings summarized herein, development of data acquisition, processing,
analysis and evaluation techniques, tools and experience is a
significant outcome of the project. Both NSSL and OHD have discovered
advantages and disadvantages of various QPE products, and are now in a
position to move toward second-generation QPE development. The findings
of this study may be summarized as follows:

-   Gauge-only analyses from MPE and QPE SUMS had better or comparable
    agreement with quasi-independent rain gauges accumulations from the
    COOP network as compared to all other radar-based and multisensor
    products. This result, however, is conditioned on the fact that the
    evaluation period covered mostly a cool season and that the COOP
    network in the study area is denser than the typical operational
    gauge network in many parts of the country, and hence may not be
    generalized. MPE reanalysis does show that, with cool-season
    radar-gauge pairing and parameter settings, the multisensor product
    performs better, albeit only marginally, than the gauge-only.

-   With the exception of gauge-only analysis, the best performer from
    QPE SUMS was the multisensor product that uses a spatially variable
    bias adjustment technique (MS_LGC). Similarly, MPE's Mmosaic product
    yielded the best results. This finding reaffirms the value of
    'multisensor' approach to precipitation estimation.

-   The QPE SUMS MS_LGC product had a monthly bias very near 1.0 while
    the real-time Mmosaic product indicated overall underestimation by
    25% in the cool season. However, the Mmosaic product had smaller
    scatter and a better correlation coefficient. The MS_LGC product is
    believed to be affected in part by less robust quality control
    measures than those used in the WSR-88D Precipitation Processing
    Subsystem (PPS), which generates the radar precipitation estimates
    used by MPE. Secondly, the larger root mean square error in the QPE
    SUMS MS_LGC product may have been a result of the significant
    periods of radar data outages that were not as prevalent in the MPE
    product generation. The underestimation in the MPE product was due
    in large part to the fact that the number of rain gauge data
    available to MPE in real time was consistently and significantly
    smaller than that made available to QPE SUMS in the post analysis
    mode, and that warm--season parameters were used in bias adjustment
    throughout the mostly cool-season evaluation period. MPE reanalysis
    based on the full availability of rain gauge data and cool-season
    parameters does show that the Mosaic product (referred to as
    Mmosaic2 in the text) has little monthly bias (0.7% overall) and
    significantly smaller RMSEs.

```{=html}
<!-- -->
```
-   

-   

-   

-   

-   

-   

-   . Storm total precipitation maps from MPE and QPE SUMS differ in
    resolution. The effect of this resolution difference on hydrologic
    prediction s unknown at this time and requires further
    investigation.

-   Both QPE SUMS and MPE products reveal a linear discontinuity where
    radars are being mosaicked. These differences are due to inter-radar
    calibration differences. Neither algorithm satisfactorily corrects
    for calibration differences between radars at the storm scale of
    temporal aggregation.

-   

-   

-   Observed and simulated hydrographs at an upstream gauging location
    reveal that parameters optimized for the raw, level-II data are
    similar for those obtained with MPE even though the QPE SUMS and MPE
    show better RPS scores

-   

-   

-   Regardless of the QPE input, the timing of the flood peak is
    predicted slightly earlier than observed, even for the roughest
    characterization of the land surface. This apparent model bias in
    V*flo*™ is consistent with Jones et al. (1998). This result suggests
    that the hydrologic evaluation framework initiated in this study
    must also include uncertainties in the parameterization and oructure
    of wi channel chraulic models domment by m 2.14 by Ami Jones, Baxter
    Vieux, and Michael Eilts (copy given to DJ Seo)

-   Optimized parameters for the QPE SUMS multisensor product with local
    bias adjustment (MS_LGC) could not produce realistic hydrographs due
    to an erroneous gauge report. It was discovered that the same
    product with a mean field bias adjustment (MS_GC) was less subject
    to individual gauge errors, and was capable of capturing a second
    peak in the streamflow not predicted by other estimators.

-   A hydrologic analysis was made possible based on an ensemble of
    hydrologic simulations that encompassed the uncertainty possible
    with the model parameters. This hydrologic analysis indicated that
    the gauge-only analyses did NOT yield the best hydrologic
    simulations, in contrast with the analysis based solely on rain
    gauges in the COOP network.

-   The hydrologic analysis indicated that while the MPE product yielded
    the best volume predictions, the QPE SUMS multisensor product with
    mean field bias adjustment (MS_GC) had the smallest RPS when
    averaged using all 3 hydrologic variables at all streamflow gauging
    locations.

-   

It must be acknowledged, however, that the inferences made from the
hydrologic evaluation summarized above are based only on a single event
and do not account for all sources of hydrologic uncertainty, and hence
may not be generalized.

### Future work and recommendations

It was seen that MPE and QPE SUMS products that relied heavily on rain
gauge observations tended to perform the best in the radar-gauge
comparison study. This is believed to be a result of the relatively high
density of gauging stations in the Tar River Basin and the
characteristics of cool season precipitation sampled during the study
period. It is recommended that a similar analysis as this one be
performed in a region that has a sparse rain gauge network and where
radar coverage is limited due to complex terrain.

Future work should focus on the expansion of the number of storms for
hydrologic evaluation using streamflow and rain gauges to validate QPE
products. The large number of model simulations required to produce
statistically significant parameter ensembles combined with the several
inputs being evaluated limited the number of study cases to one event.
The hydrologic analysis in this study provided additional information
about the QPE inputs to a hydrologic model that can serve to supplement
traditional evaluations that rely solely on rain gauge comparisons. This
hydrologic analysis takes more effort, but is useful for evaluating the
hydrologic performance of QPE algorithms, which is the intended
application.

REFERENCES

Barnes, S.G., 1964: A technique for maximizing details in numerical
weather map

analysis. J. Appl. Meteor., 3, 396-409.

Breidenbach, J., D.-J. Seo, P. Tilles and M. Fortune, 2001: Multisensor
precipitation

> estimation for use by the National Weather Service River Forecast
> Centers, 16^th^ Conf on Hydrology, AMS.

Fulton, R. A., J. P. Breidenbach, D.-J. Seo, and D. A. Miller, 1998: The
WSR-88D

rainfall algorithm. Weather and Forecasting, 13, 377-395.

Jones, A., B. Vieux, and M. Eilts, 1998: Aspects of calibrating a
distributed rainfall-

> runoff model using WSR-88D rainfall estimates in the Illinois Basin.
> Special Symp. on Hydrol. Paper 2.14, Amer. Meteorol. Soc.

Seo, D.-J., 1998a: Real-time estimation of rainfall fields using rain
gage data under

fractional coverage, J. Hydrol., 208, 25-36.

Seo, D.-J., 1998b: Real-time estimation of rainfall fields using radar
rainfall and rain

gage data, J. Hydrol., 208, 37-52.

Seo, D.-J., J. P. Breidenbach, and E. R. Johnson, 1999: Real-time
estimation of mean

field bias in radar rainfall data, J. Hydrol., 233.

Seo, D.-J. And J. P. Breidenbach, 2002: Real-time correction of
spatially nonuniform bias

in radar rainfall data using rain gauge measurements, J. Hydrometeorol.,
3, 93-

111\.

Vieux, B and Moreda, F., 2003 : Ordered Physics- Based parameter
adjustment

of a distributed model: In: Duan, Q., Gupta, H.V., Sorooshian, S.,

Rousseau, A.N., Turcotte, R., (Editors.) Calibration of Watershed
Models,

Water Science and Application 6, American Geophysical Union, Wash. D.C.,
P.

267-281.

Wilks, D.S., 1995. Statistical Methods in the Atmospheric Sciences: an
Introduction,

International Geophysics Series, Vol. 59, Academic Press, 464 pp.

###  

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### Appendix

Scatter plots of daily precipitation between observed and various MPE
estimates for the months of October , 2002, through April, 2003. See
text for explanation of Gageonly, Bmosaic, Lmosaic, Mmosaic, Rmosaic and
Xmrg.

![](media/image9.wmf){width="4.8125in" height="6.552083333333333in"}

Scatter plots of daily precipitation between observed and various MPE
estimates from reanalysis for the months of October , 2002, through
April, 2003. See text for explanation of Gageonly, Bmosaic, Lmosaic,
Mmosaic, Rmosaic and Mmosaic2.

![](media/image41.wmf){width="5.993055555555555in"
height="8.071527777777778in"}

![](media/image42.wmf){width="5.993055555555555in"
height="8.071527777777778in"}

![](media/image43.wmf){width="5.993055555555555in"
height="8.071527777777778in"}

![](media/image44.wmf){width="5.993055555555555in"
height="8.071527777777778in"}

![](media/image45.wmf){width="5.993055555555555in"
height="8.071527777777778in"}

![](media/image46.wmf){width="5.993055555555555in"
height="8.071527777777778in"}

![](media/image47.wmf){width="5.997916666666667in"
height="8.210416666666667in"}

Scatter plots of daily precipitation between observed and the QPE SUMS
estimated for the months of October, 2002, through April, 2003. See text
for explanation of LGC.RAD, LGC.MS, GC.RAD, GC.MS, TOTAL RAD, and TOTAL
MS.

Calibration Hydrographs (Level II)

Calibrated Events, Tropical Storm Allison and October 2002 Event
(Boundary Conditions applied) are shown below.

![](media/image23.png){width="3.0833333333333335in"
height="2.3854166666666665in"}Tropical Storm Allison

![](media/image24.png){width="3.1458333333333335in"
height="2.3854166666666665in"}

![](media/image40.png){width="3.125in" height="2.3541666666666665in"}

![](media/image31.png){width="3.1666666666666665in"
height="2.3645833333333335in"}

October 2002 (Level II)

![](media/image32.png){width="3.1770833333333335in"
height="2.3958333333333335in"}

![](media/image36.png){width="3.1770833333333335in"
height="2.3958333333333335in"}

![](media/image34.png){width="3.1875in" height="2.375in"}

![](media/image35.png){width="3.1979166666666665in"
height="2.3756944444444446in"}

![](media/image37.png){width="3.125in" height="2.35625in"}

![](media/image38.png){width="3.21875in" height="2.4381944444444446in"}

![](media/image39.png){width="3.25in" height="2.4451388888888888in"}