# Point and areal validation of forecast precipitation fields

Eddy Yates, Sandrine Anquetin, Véronique Ducrocq, Jean-Dominique

Creutin, Didier Ricard, Katia Chancibault

► To cite this version:

Eddy Yates, Sandrine Anquetin, Véronique Ducrocq, Jean-Dominique Creutin, Didier Ricard, et al. Point and areal validation of forecast precipitation fields. Meteorological Applications, 2006, 13 (1), pp.1 à 20. 10.1017/S1350482705001921. insu-00386553

HAL Id: insu-00386553

https://insu.hal.science/insu-00386553v1

Submitted on 23 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

# Point and areal validation of forecast precipitation fields

Eddy Yates<fnref n="1" />, Sandrine Anquetin<fnref n="1" />, Véronique Ducrocq<fnref n="2" />, Jean-Dominique Creutin<fnref n="1" />, Didier Ricard<fnref n="2" /> & Katia Chancibault<fnref n="1" />

Email: eddy.yates@hmg.inpg.fr

Two high resolution quantitative precipitation forecasts with different levels of realism are evaluated. Classical scores (bias, correlation and scores based on contingency tables) confirm that the two forecasts do not have the same quality. A multi-scale extension of these scores has then been made to produce a validation for hydrological purposes. Rainfall fields are integrated over surfaces of various scales. For better simulation, scores indicate an increase in the quality of the simulated precipitation for larger surfaces (typically more than 100 km<fnref n="2" />): the localisation errors are reduced by the aggregation. This helps to determine the usefulness of such forecasts for hydrological purposes.

**Keywords:** multiscale validation, mesoscale quantitative precipitation forecast

Received March 2005, revised August 2005

## I. Introduction

The south-eastern ridge of the French 'Massif Central' (see Figure 1a for location) is frequently subject to flash floods. They mainly occur during the autumn, and they affect a wide range of watershed scales (basins typically of 50–500 km<fnref n="2" /> embedded in the five main basins shown in Figure 1b). The flash floods in this region occur due to intense and stationary mesoscale convective systems. The hilly topography of the region governs the distribution of the rain and the runoff. Given the very short response time to flash floods, forecasting them must rely on quantitative precipitation forecasts.

One of the greatest challenges of Numerical Weather Prediction is to improve quantitative precipitation forecasting (QPF) significantly. As noted by Fritsch et al. (1998), QPF skill using the current operational models remains relatively low. Nevertheless, research advances have opened new perspectives and show that non-hydrostatic research models are now able to provide more realistic rainfall distributions at the meso-scale for selected case studies (Ducrocq et al. 2002, Roebber & Eise 2001, Stein et al. 2000), or during field experiments (e.g. the Mesoscale Alpine Program; Bougeault et al. 2001). Several National Weather Services plan to run such non-hydrostatic models operationally with a horizontal resolution of the order of 1–3 km by 2010.

The validation of high-resolution QPF raises two questions. The simulated rainfalls are grid fields of average rain depths over cells covering square

kilometres. Which observed rain field can be considered as a reference for such data, and what validation criteria can be used when the choice of the reference is made?

Validation of QPF implies first the definition of a reference. Each measurement device has its own weakness in estimating rainfall fields at times and spatial scales associated with flash floods. Estimates from rain gauges are only point measurements; this means that there are representativeness errors when they are used to verify simulated rainfall, which are integrated over surfaces. Some methods have been proposed to address this problem (Tustison et al. 2001, Tustison et al. 2003); the representativeness errors relating to a particular gauge network can also be evaluated for a given integration surface. In hilly regions, the density of rain gauge networks typically decreases with altitude (Kieffer-Weisse & Bois 2001); rain gauges are generally more frequent in valleys than on crests. According to kriging interpolation of rain gauge data, elevated areas often have large estimation variance. Weather radar offers an alternative means of estimating precipitation fields (Tateya et al. 1989, Fujiyoshi et al. 1996, Warner et al. 2000, Miniscloux et al. 2001, Suzuki et al. 2001). They integrate the rainfall in larger cells similar to the model cells themselves. They also provide grid fields, but radar estimates are affected by many factors including ground clutter, bright band contamination and fluctuations of Z-R relationship (Joss & Waldvogel 1990). All these factors are aggravated in mountainous areas (Andrieu et al. 1997).

<footnote n="1">¹Laboratoire d'étude des Transferts en Hydrologie et Environnement, Grenoble, France</footnote>

<footnote n="2">²Centre National de la Recherche Météorologique, Météo-France, Toulouse, France</footnote>
<figure><img src="image_3.png" /><figcaption>**Figure 1.** (a) The black rectangle indicates the study area located on the south-east edge of the Massif Central in France. (b) Relief of the study area and main hydrographic network derived from a DTM; the thick lines delineate the five main watersheds (Ardèche River (2500 km²); Cèze River (1054 km²); Gard River (1913 km²); Vidourle River (621 km²); Hérault River (2500 km²)); the thin lines indicate the main rivers; the 174 rain gauge stations used for the validation of the 2001 case are indicated by the empty circles; the main cities are also shown (black squares), as well as the cities flooded in case studies considered here (black crosses).</figcaption></figure>

Validation of QPF implies defining scores that will measure the difference between the forecast and the reference. Model verification, in general, uses traditional scores such as bias and root mean square error (RMSE) (Snook & Pielke 1995, Warner et al. 2000, Ducrocq et al. 2002). The bias measures the propensity of the model to underestimate or overestimate rainfall values on average. The RMSE indicates the dispersion. Scores based on 2 × 2 contingency tables measure the ability to predict precipitation above a given threshold at a specific location (Barnston 1992, Gandin & Murphy 1992). Some recent work bases new validation methods on the multi-scale properties of precipitation (Zepeda-Arce et al. 2000, Harris et al. 2001), or on the structure of the rain pattern, as with the object-oriented method of Ebert & McBride (2000). Other workers base their validation on hydrological responses, linking precipitation estimation and hydrologic prediction (Benoit et al. 2000, Westrick & Mass 2001).

Watershed integration allows validation to be based on areal rainfalls over given surfaces. The underlying hypothesis is that the knowledge of the areal rainfall over a watershed gives a first order clue of the hydrologic

response of the area being considered. This hypothesis is supported by a large number of hydrological modelling studies using a combined rainfall-runoff approach (Obled et al., 1994, Davis, 2001). This approach allows a scale-based evaluation of the usefulness of the forecast for hydrologic applications, as discussed by Zepeda-Arce et al. (2000). This is particularly relevant for flash-flood forecasting which spans a wide range of scales as indicated above. One aim of the present study is to see how the defaults of the simulated rainfall fields might be reduced when moving from small to large scales of integration in space. We expected that aggregating rainfall over large enough areas would eliminate the errors arising from shifts in the simulated rainfall patterns.

This paper aims to provide clues about the spatial scales that can be considered relevant when using simulated rainfall fields in hydrology. We consider two contrasted cases for which high resolution simulations of two quasi-stationary mesoscale convective systems offer different levels of realism. The reference values are derived from a network of rain gauges. The different criteria mentioned above are applied and commented
on. Results of the simulations, together with the observations available for validation, are presented in Section 2. In Section 3 we define the validation criteria that we used and we explain how we apply these criteria to an extensive set of reference values. The results of the different validations are presented and discussed in Section 4. Finally, in Section 5, we propose a multi-scale extension of the scores to obtain an assessment of the quality of the simulations depending on the scale.

## 2. Case studies: the data set and model used

### 2.1. The case studies

Two convective rainfall events that occurred on 13–14 October 1995 and 6–7 October 2001 over the Cévennes-Vivarais region in France were considered. These two meteorological events were quasi-stationary Mesoscale Convective Systems (MCS), a type of precipitating system that frequently develops over the foothills of the Massif Central. During their mature phase, both MCSs produced a typical V-shape in the infrared images (Scofield 1985). Their stationary phase lasted 9 hours for the first case and approximately 6 hours for the second case. They produced rainfall accumulations reaching respectively 330 mm and 360 mm. The first rainfall event (13–14 October 1995) led to a sudden response of the Gardon d'Anduze river (545 km²), a tributary of the Gard river (Figure 1b). In less than 9 hours, the discharge grew from a base flow of 200 m³ s⁻¹ up to a peak flow of 1400 m³ s⁻¹ (i.e. a specific peak outflow of almost 3 m³ s⁻¹ km⁻²). The second rainfall event (6–7 October 2001) was associated with an active cold front, and flooded Sommières, a city located on the Vidourle river (621 km²) (Figure 1b).

We consider these two events as representative of the flash-flood events threatening this region. They have been selected for both their stationarity and their hydrological impact.

### 2.2. Rain field estimation from the rain gauge network

For the domain presented in Figure 1b, the surface observation network is made up of 147 recording rain gauges for the 1995 event and 174 for the 2001 event. Even with a relatively large number of stations, the question of the representativeness of point observations to describe the spatial variability of the rainfall associated to the MCSs must be considered.

For each event, 11 hourly rainfall fields and their total are considered (see below). Kriging is used to interpolate rain gauge data on the study area. This geostatistical method requires the identification of a variogram that characterises the space structure of the rainfall fields. The experimental variograms are well fitted by a spherical variogram as shown by Lebel

Point and areal validation of forecast precipitation fields

<table><caption>Table 1. Main characteristics of the fitted spherical variograms used to estimate the rainfall field in the study area based on the rain gauge observation data. σ stands for the standard deviation of the measures.</caption><thead><tr><th></th><th>1995 event</th><th>2001 event</th><th></th></tr></thead><tbody><tr><td></td><td>24</td><td>22</td><td>Range (km)</td></tr><tr><td rowspan="2">Hourly</td><td>1</td><td>0.7</td><td>Sill/σ<sup>2</sup></td></tr><tr><td>0.15</td><td>0.2</td><td>Nugget/σ<sup>2</sup></td></tr><tr><td></td><td>35</td><td>50</td><td>Range (km)</td></tr><tr><td rowspan="2">Accumulated</td><td>1</td><td>0.9</td><td>Sill/σ<sup>2</sup></td></tr><tr><td>0</td><td>0.1</td><td>Nugget/σ<sup>2</sup></td></tr></tbody></table>

et al. (1987), and as shown in Figure 2. The main characteristics of the fitted spherical variograms are detailed in Table 1. Kriging provides estimators of point or block values. This means that this technique provides an estimator of the considered regionalised variable either at a point or integrated over a particular area (for instance a watershed). Kriging estimators are linear, unbiased and optimal. Linear means that the estimator is obtained through a linear combination of the values measured at a set of points randomly distributed in space (the rain gauge network in our case). Kriging aims first at equalling to zero the mean estimation error (unbiasedness). Kriging aims secondly to minimise the mean quadratic estimation error called estimation variance σ² (optimality). σ² ranges in space between 0 and the field variance indicated by the sill value (S) of the variogram, and (1 - σ²/S) is the amount of variance explained by the interpolation at a point. The estimation variance depends both on the rain gauge network configuration and on the statistical structure of the rain field featured by the variogram. In this work, we arbitrarily consider that a threshold of 40% of explained variance defines the set of reliable estimated values that can be used as reference in the validation. The variograms of each event are different, and they depend on the time duration of rainfall accumulation (see Table 1). Moreover, the gauge network is different for the two events and it is not exactly the same for each hour within an event. Therefore, the sets of reliable reference values are different for each rainfall field. When point references are considered the delineated confidence zones are different. When block references are considered, the sets of integration areas retained are also different.

The kriged rain accumulations over the 11 hours are presented for the two events in Figure 3a and Figure 4a. The confidence zones defined by the estimation variance threshold of 40% show the portions of the fields that are taken into account in the following point analyses. As indicated before, these zones depend on the available observation data and on the structure of the rain fields. The 2001 event has much wider confidence zones than the 1995 event because of the differences
<figure><img src="image_4.png" /><figcaption>Figure 2. The experimental and fitted spherical variograms: hourly variograms (1995 (a) and 2001 (b)) and 11-hour variograms (1995 (c) and 2001 (d)).</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Figure 3. 11-hour rain accumulation for the 1995 event (01 to 12UTC on 14 October 1995): (a) kriging of the 147 rain gauge (crosses) values, and (b) MesoNH simulation. The shaded surfaces correspond to points where less than 40% of variance is explained by the interpolation.</figcaption></figure>
Point and areal validation of forecast precipitation fields

<figure><img src="image_6.png" /><figcaption>Figure 4. 11-hour rain accumulation for the 2001 event (19UTC on 6 October 2001 to 06UTC on 7 October 2001): (a) kriging of the 174 rain gauge (crosses) values, and (b) MesoNH simulation. The shaded surfaces correspond to points where less than 40% of variance is explained by the interpolation.</figcaption></figure>

in the meteorological situation. In 2001 the rain fields with a larger correlation range were associated with an active cold front that crossed France during the night between 6 and 7 October. For the 1995 event, the rain fields display a shorter range of correlation as they are associated with a more localised mesoscale convective system.

## 2.3. Simulated rain fields

The two events have been simulated with the anelastic non-hydrostatic Meso-NH model (Lafore et al. 1998). It allows several grids to be run simultaneously in a two-way interactive mode. For the simulated fields presented here, two domains have been used, one using a 10 km grid mesh and the other a 2.5 km grid mesh. We consider only the surface rainfall simulated at 2.5 km.

The model relies on physical parameterisations to reproduce the different atmospheric processes such as turbulence, convection, radiative transfer, micro-physical processes and soil-atmosphere exchanges. The parameterisations used for the simulation of the two convective events are presented in Table 2. A bulk microphysical scheme (Pinty & Jabouille 1998) governs the equation of two liquid categories (cloud and rain water) and of three ice categories (graupe!, snow and primary ice). A convective parameterisation is used for the 10 km domain, whereas the 2.5 km grid mesh allows an explicit resolution of the convection.

The initial states of the simulations have been investigated in depth in order to produce realistic precipitation fields (Ducrocq et al. 2000, Ducrocq et al. 2002, Ricard 2002). Ducrocq et al. (2002) have

<table><caption>Table 2. *Physical parameterisations used for the simulations.*</caption><tr><td>Horizontal grid size (km)</td><td>10</td><td>2.5</td></tr><tr><td>Surface parameterisation: ISBA soil model (Noilhan & Planton 1989)</td><td>×</td><td>×</td></tr><tr><td>1D turbulent scheme, with prognostic TKE and mixing length following Bougeault & Lacarrère (1989)</td><td>×</td><td>×</td></tr><tr><td>3D turbulent scheme, with prognostic TKE and mixing length following Redelsperger & Sommeria (1981)</td><td>×</td><td>×</td></tr><tr><td>Convective scheme (Kain & Fritsch 1993, Bechtold et al. 2001)</td><td>×</td><td>×</td></tr><tr><td>Bulk micro-physical scheme with 3 ice and 2 liquid categories (Pinty & Jabouille 1998)</td><td>×</td><td>×</td></tr><tr><td>Radiative transfer scheme, effect of the solar and infrared radiations (Morcrette 1989)</td><td>×</td><td>×</td></tr></table>
Eddy Yates *et al.*

<table><caption>Table 3. *Times of the initial conditions of the simulations and of the studied 11-hour window.*</caption><thead><tr><th></th><th>1995</th><th>2001</th></tr></thead><tbody><tr><td>Initial conditions</td><td>22UTC, 13 Oct. 1995</td><td>18UTC, 6 Oct. 2001</td></tr><tr><td rowspan="2">11-hour window</td><td>01UTC, 14 Oct. 1995</td><td>19UTC, 6 Oct. 2001</td></tr><tr><td>12UTC, 14 Oct. 1995</td><td>06UTC, 7 Oct. 2001</td></tr></tbody></table>

used mesoscale observations such as mesonet surface observations, radar reflectivities and METEOSAT data to prescribe the initial state of four convective cases, including the 13–14 October 1995 event. The description of the humidity field in the initial state is found to be crucial for the forecast of the convective systems. The simulation of the 6–7 October 2001 case also uses this mesoscale initialisation procedure, which is applied one hour after the triggering of the convection for the two cases (Table 3).

The guess of the mesoscale initialisation procedure, as well as the boundary conditions, are provided by the operational analyses from the METEO-FRANCE ARPEGE suite for 13–14 October 1995 and from the European Centre for Medium-Range Weather Forecasts’ Integrated Forecasting System one for 6–7 October 2001.

For the two events, we extract an 11-hour window of simulated precipitation, which corresponds qualitatively to the best period of the simulated precipitation for each experiment (see Table 3). The validation is applied to the 11 hourly rainfall fields and to the corresponding accumulation field hereafter referred to as the total rain field. The choice of a basic time step of one hour is justified by hydrologic considerations. When applied to this type of storm, the watersheds of the region have typical response times of a few hours. The simulated total rain fields are plotted in Figure 3b and Figure 4b for the two considered events. The more advanced parameterisations and the mesoscale initial conditions used for the Meso-NH simulations improve the rainfall forecast (compared with the simulations of these two events by the operational model); but the Meso-NH runs do not provide the same level of quality for the two cases. Indeed, the simulation of the 1995 event gives more realistic rain fields in terms of quantity and localisation. The dynamics is also better reproduced by the model. The large discrepancy between the observed and the simulated rainfall for the 2001 event seems to be due to errors in the large-scale forcing (i.e. the timing of the active cold front is not accurately reproduced by the coupling model). Whereas the area with the maximum of the simulated precipitation for the 1995 event is very close to the observed one (error in location of less than 20 km), the heavy rainfall area simulated for the 2001 event is shifted

northward (error in location of more than 50 km) and is significantly underestimated.

# 3. Validation methods used

## 3.1. Point validation

Point validations are far more common than validations relying on values integrated in space. Point validations merely compare the values measured at the locations of the rain gauges to the values simulated at the nearest grid element.

A fairer method uses ‘super-observations’ (Ghelli & Lalaurette 2000): the observations are ‘upscaled’ to the model scales; the observations are interpolated to each model grid box. As explained in Section 2.2, the use of kriging allows computation of this extensive set of reference values.

Nevertheless, the quality of the computed reference values is uneven, as indicated by the estimation variance. The best way to deal with this problem would have been to introduce the estimation variance associated to each reference value into the definition of the validation scores. This would have led to the definition of new scores, which is considered to be beyond the scope of this study. As stated in Section 2.2, we simply chose a threshold of estimation variance over which the reference values are not included in the validation set. We therefore used the commonly used comparison criteria. They rely on both continuous and thresholded statistics.

By ‘continuous statistics’ we mean criteria that use the full range of values like distribution functions of the reference and the simulated values. This comparison is not sensitive to the space structure of the compared fields. It can be complemented by the use of scatter-diagrams, which take into account the structure. The comparison can be reduced to bulk scores, such as the bias (*BIAS*), the root mean square error (*RMSE*) (respectively the mean and the root of the quadratic mean of the differences between observed and simulated values) or the square of the correlation coefficient (*R*²). We did not use the *RMSE*, as it is very close to the *R*². The *BIAS* and *R*² scores are defined by:

$$ 
\begin{aligned}
BIAS &= \frac{1}{N} \sum_{i=1}^{N} P_i - R_i = m_p - m_R \quad \text{and} \\
R^2 &= \left( \frac{\sum_{i=1}^{N} (R_i - m_R)(P_i - m_p)}{N\sigma_R\sigma_p} \right)^2
\end{aligned}
$$

where $R_i$ and $P_i$ are respectively the ith reference and predicted values among N, $m_R$ and $m_p$ the mean and
<table><caption>Table 4. Schematic 2 × 2 contingency table for the definition of scores. ‘Ref.’ stands for ‘Reference’.</caption><thead><tr><th></th><th>Forecast &lt; threshold</th><th>Forecast ≥ threshold</th></tr></thead><tbody><tr><th>Ref. &lt; threshold</th><td>a</td><td>b</td></tr><tr><th>Ref. ≥ threshold</th><td>c</td><td>d</td></tr></tbody></table>

$\sigma_R$ and $\sigma_P$ the standard deviation of the reference and predicted rainfall values. $R^2$ is sensitive to the structure of the rain field, not to its amplitude.

By ‘thresholded statistics’ we mean binary criteria generally relying on contingency tables established for a given threshold, following the format given in Table 4. In this table, *a* is the number of points where both the reference and the simulation are under the threshold; *b* is the number of points where the forecast is above but the reference is under the threshold; *c* is the number of points where the forecast is under whereas the reference is above the threshold; *d* is the number of points where both the reference and the forecast are above the threshold. Various scores can be computed from these tables. Namely, the forecast accuracy (*ACC*), the frequency bias (*FBIAS*), the probability of detection (*POD*), the false alarm ratio (*FAR*, Mason 1989), the threat score (*TS*, Wilks 1995) and the Heidke skill score (*HSS*, Barnston 1992) can be computed according to the following expressions:

$$POD = \frac{d}{c+d}, \quad FAR = \frac{b}{b+d},$$

$$ACC = \frac{a+d}{a+b+c+d}, \quad FBIAS = \frac{b+d}{c+d},$$

$$TS = \frac{d}{b+c+d} \quad \text{and} \quad HSS = \frac{a+d-T}{a+b+c+d-T},$$

with $T = \frac{(a+c)(a+b)+(b+d)(c+d)}{a+b+c+d}$ referring to the expected number of correct random predictions. These scores can be interpreted as follow:

* POD gives the rate of well-predicted locations among the locations where the reference is over the threshold; a perfect forecast has $POD = 1$.

* FAR gives the rate of ill-predicted locations among the locations where the forecast is over the threshold; a perfect forecast has $FAR = 0$.

* ACC gives the rate of all well-predicted locations with respect to the threshold; a perfect forecast has $ACC = 1$. The POD, FAR and ACC scores need generally to be examined together.

* FBIAS compares the occurrence of the model and reference precipitation above the given threshold. It does not take into account localization errors. A perfect forecast has $FBIAS = 1$.

* TS gives the rate of well-predicted locations among locations where the reference or the simulation are above the threshold. A perfect forecast has $TS = 1$.

Point and areal validation of forecast precipitation fields

* HSS measures the ability to predict precipitation above a given threshold at a specific location; a perfect prediction has $HSS = 1$, a random prediction has this score equal to zero.

To allow statistical processing, the analysis is limited to sets containing at least 30 reference values.

## 3.2. Areal validation

As mentioned in the introduction, watershed integration allows validation to be based on areal rainfalls over given surfaces. Zepeda-Arce et al. (2000) have proposed to scan space scales with the threat score *TS*. Large scales are explored by aggregating the high resolution fields. In this paper we extend this approach to all the scores presented in Section 3.1 and we make an attempt to deal with hydrological surfaces in addition to rectangular cells.

The area under study can be divided geometrically into rectangular cells at different resolutions. The surface $d_j$ of the cells of the jth subdivision is expressed by $d_j = \frac{D_j^T}{j^2}$ with $D_T$ the surface of the domain and $j$ between 6 and 56 (as illustrated in Figure 5a with $j=10$). The areal rainfall is computed for each cell resulting in different series of reference and simulated areal rainfall values corresponding to the different resolutions. Each score is computable for each resolution.

The area of study can also be divided according to the natural drainage network. The delineation of areas constituted by two hill slopes draining water towards a river yields a set of surfaces that are sound for hydrologic applications. The example map presented in Figure 5b was obtained from a digital topography model (DTM) with a resolution of 75 m over $D_T = 32,000$ km² of the Cévennes-Vivarais region. The flow pathway algorithm based on the TOPMODEL concepts (Beven & Kirkby 1979) was used to draw the stream channels and the associated hill slopes automatically.

As this hydrologic subdivision is subjected to the natural constraints of topography, the different cells at a given scale do not have exactly the same surface. They are therefore distributed into classes to allow statistical processing. The repartition into classes has to be consistent with the statistical analysis: each class must contain enough surfaces, and the range of surface size within each class must be as narrow as possible. These two criteria can be respected by making classes with regular width with respect to the logarithm of the surface: each cell of the class $C_j$ has its surface between $10^{j-1}/10$ and $10^j/10$ km².

We must mention that our hydrologic subdivision is not yet perfect. In a given class, some pixels are not represented, and some others appear many times. Nevertheless we can apply the validation to this kind of surface.
<figure><img src="image_7.png" /><figcaption>**Figure 5.** Illustration of the two types of rectangular (a) and hydrological (b) validation surfaces used in this study. In both illustrations, the elementary surfaces are about 100 times smaller than the total domain (i.e. about 320 km²). The shading is only for clarity.</figcaption></figure>

The reference values are obtained by block-kriging the observations on each cell of each subdivision of the domain. The simulated values are obtained simply by averaging the values computed by the model in each cell. A threshold of estimation variance is used as described in Section 3.1. The cells where the estimation variance is above this threshold are not included in the validation set. To allow statistical processing, the analysis is limited to scales where this set contains at least 30 values.

The scores presented in Section 3.1 are computed for each subdivision. Only *BIAS* is not used, because the way the upscaling is done (unbiased linear estimator for the reference and simple averaging of the forecast) is linear. The *BIAS* does therefore not depend on the scale.

As noted earlier, one of the motivations of this study was to see if space integration would reduce or compensate for some errors of the model. If this is the case, the multiscale analysis should show a critical resolution, above which the scores get better.

# 4. Point validation of the simulated rain fields

## 4.1. Continuous statistics

The cumulative distribution functions (CDF) of reference and simulated values, as well as the corresponding scatter diagrams are represented in Figure 6 for the total rain fields and in Figure 7 for the hourly rain fields.

Various related statistical bulk variables are reported in Table 5.

Overall, the simulated rainfalls are too weak. According to mean values, the 2001 case produced more than twice as much rain as the 1995 case and displays a more important bias (half of the observed mean compared to one-third for 1995). According to the CDFs, the simulated values are statistically lower than the reference values except for the few highest simulated hourly rainfall intensities that can be 50 to 100% larger than the reference values (see Figure 7a and Figure 7b). This is probably due to the sampling effects of the rain gauge network and to the smoothing effects of the kriging method. This may also explain the higher variability of the simulated rainfalls. According to the normalised standard deviations, it is more pronounced for the 2001 case and more sensitive with hourly values.

The co-fluctuation between simulated and reference values (Table 5) is as expected from the maps shown in Figure 3 and Figure 4. The case of 1995 is better reproduced than that of 2001. For 1995, 23% and 46% of variance of the reference values is explained for the hourly and the total rain fields respectively. For 2001, no variance is explained.

In both cases the scatter diagrams have complex patterns and the displayed best fit lines are hardly representative. The scatter diagram for the total rain field of 1995 (Figure 6c) displays a structure near the best fit line,
Point and areal validation of forecast precipitation fields

<figure><img src="image_8.png" /><figcaption>**Figure 6.** Distribution functions of the 11 hourly accumulated reference and simulated precipitations (1995 (a) and 2001 (b) cases). Simulated versus reference 11 hourly accumulated precipitation (1995 (c) and 2001 (d) cases). The solid lines represent the best fit for each simulation. The dashed line is the 1×1 ideal curve.</figcaption></figure>

which has a slope smaller than one; it shows a set of underestimated but correctly localised predictions. For 2001 (Figure 6d), a lot of points are near the x-axis: large rainy zones were not simulated, but some points are on the ideal 1 × 1 line; this might correspond to the correct simulation (for the total rain field at least) over the north-eastern part of the domain (see Figure 4). The scatter diagram for the hourly rain fields of 1995 (Figure 7c) is more complex. Some structures show zones of overestimated rain, other rainy zones are not simulated. It is difficult to distinguish a structure from

the other points. For 2001 (Figure 7d), the diagram has a typical V-shape: some rainy zones are not simulated (points on the x-axis), and there are some zones with a strong simulated rain where little rain is observed (points on a vertical line at about 5 mm h⁻¹). That means that the simulation does not look like the observations at all.

In summary, the point validation using continuous statistics shows that the model produces rainfall values

<table><caption>Table 5. *Comparison of the statistics.*</caption><thead><tr><th></th><th colspan="2">1995</th><th colspan="2">2001</th></tr><tr><th></th><th>1h</th><th>11h</th><th>1h</th><th>11h</th></tr></thead><tbody><tr><td>Mean (mm)</td><td></td><td></td><td></td><td></td></tr><tr><td>Reference</td><td>1.5</td><td>15.7</td><td>4.0</td><td>37.5</td></tr><tr><td>Simulation</td><td>0.9</td><td>10.2</td><td>2.0</td><td>16.0</td></tr><tr><td>Bias (mm)</td><td>-0.5</td><td>-5.5</td><td>-2.0</td><td>-21.5</td></tr><tr><td>Normalised standard deviation</td><td></td><td></td><td></td><td></td></tr><tr><td>Reference</td><td>2.6</td><td>1.8</td><td>1.2</td><td>1.0</td></tr><tr><td>Simulation</td><td>3.2</td><td>1.8</td><td>2.7</td><td>1.4</td></tr><tr><td>Ratio between the normalised standard deviations</td><td>1.2</td><td>1.0</td><td>2.2</td><td>1.4</td></tr><tr><td>Correlation coefficient (R<sup>2</sup>)</td><td>0.23</td><td>0.45</td><td>0.003</td><td>0.03</td></tr></tbody></table>
<figure><img src="image_9.png" /><figcaption>**Figure 7.** Repartition functions of the 11 hourly reference and simulated precipitations (1995 (a) and 2001 (b) cases). Simulated versus reference hourly precipitations (1995 (c) and 2001 (d) cases). The solid lines represent the best fit for each simulation. The dashed line is the 1 × 1 ideal curve.</figcaption></figure>

that are too weak in general. The co-fluctuation analysis is less conclusive. The case of 1995 is obviously better reproduced than the one of 2001.

## 4.2. Statistics based on contingency tables

In the studies previously referred to, there are no general rules to guide the selection of thresholds used to compute the scores based on contingency tables. In this study, we used three thresholds corresponding to high quantiles because we were interested in the rain field features that led to flash-floods. We chose the 70, 80 and 90% quantiles of each reference distribution function (observations). The corresponding values are detailed in Table 6.

The scores computed for the 11 hourly rain fields and their total are presented in Figure 8 for the 1995 case and in Figure 9 for the 2001 case. Only the first 7 hours are represented to make the figure easier to read.

The results for the total rain fields are globally coherent with what was intuitively expected from the comparison of the maps in Figure 3 and Figure 4. For all the scores, the simulation of 1995 is closer to a 'perfect forecast' than the simulation of 2001. This result is easy to read

<table><caption>**Table 6.** Values (mm) of the thresholds used to compute the binary scores.</caption><thead><tr><th></th><th>Quantile (%)</th><th>1995 event</th><th>2001 event</th></tr></thead><tbody><tr><td rowspan="3">Hourly</td><td>70</td><td>0.36</td><td>4.55</td></tr><tr><td>80</td><td>1.44</td><td>5.97</td></tr><tr><td>90</td><td>3.59</td><td>8.46</td></tr><tr><td rowspan="3">Accumulated</td><td>70</td><td>13.1</td><td>54.0</td></tr><tr><td>80</td><td>24.1</td><td>64.0</td></tr><tr><td>90</td><td>41.0</td><td>84.1</td></tr></tbody></table>

because for each score, except for FAR, the difference for a given event between the scores obtained with the three quantiles are smaller than the differences between the scores of the two considered events (the scores of 1995 being better does not depend on the threshold). FBIAS is closer to 1 for the 1995 case (Figure 8b) than for the 2001 event (Figure 9b) indicating that the underestimation of rainfall is worse for the 2001 case. The differences in scores with the threshold show that for 1995 the two lowest thresholds provide better scores (POD, FAR and TS) than the higher threshold. This indicates that the reproduction of the precipitation patterns gets worse with higher rain accumulations. For 2001 the same comment applies and the scores obtained
Point and areal validation of forecast precipitation fields

<figure><img src="image_10.png" /><figcaption>**Figure 8.** Scores computed on 7 hourly and the 11-hour accumulated simulated precipitation for the 1995 event: ACC (a), FBIAS (b), POD (c), FAR (d), TS (e), HSS (f) (see text for a definition of these scores).</figcaption></figure>

with the threshold corresponding to the 90% quantile
are extremely bad except for ACC.

The FAR and ACC scores are misleading given the
general underestimation of the simulated rain fields.
FAR is spuriously low when the simulated values

seldom reach the threshold when the reference values
do. ACC has a lower limit equal in our case to the
quantile chosen for the threshold.

The results for the hourly rain fields demand further
comment. The two events display distinct signatures for
<figure><img src="image_11.png" /><figcaption>**Figure 9.** Scores computed on 7 hourly and the 11-hour accumulated simulated precipitation for the 2001 event: ACC (a), FBIAS (b), POD (c), FAR (d), TS (e), HSS (f) (see text for a definition of these scores).</figcaption></figure>

all the scores. 1995 is closer to a 'perfect forecast than 2001. ACC is not informative. Where the evolution in time is concerned, the only trajectories displaying a clear trend are FBIAS and FAR in 1995. FBIAS (Figure 8b) decreases from 1 down to about 0.5. FBIAS is high (more than 1) for the 90% quantile threshold at 04UTC

and 05UTC; for these times, it is below 1 for the other thresholds. FAR (Figure 8d) decreases, which is certainly linked with the decrease of FBIAS.

In conclusion, all the meaningful scores, both continuous and based on contingency tables, show the
<figure><img src="image_12.png" /><figcaption>Figure 10. The top diagrams show the evolution of the standard deviations of the reference and of the simulations and their ratio (y) for the hourly rain fields (a) and for their total (b) for 1995. Diagram (c) shows the square of the correlation coefficient between the reference and the simulation for the hourly rain fields (dashed line) and their total (solid line). Diagram (d) displays the number of reliable reference values available.</figcaption></figure>

superiority of the simulation of the 1995 case, for both the hourly and the total study. Continuous scores allow independent validation of the amount of rain and the structure of the rain field. The results of the scores based on the contingency tables depend to some extent on the choice of the threshold used to compute them. FBIAS gives a validation of the amount of rain. The other scores serve to validate the structure, but they can be biased if the amount of rain is not well reproduced.

# 5. Areal validation of the simulated rain fields

As explained in Section 3.2, the physical area of study is divided at different resolutions either geometrically into rectangles or hydrologically into sub-watersheds. For each resulting cell, the simulated areal rainfall is computed by a simple mean of the values inside the cell. Block kriging gives the reference areal rainfall over the same cell and the associated estimation variance (see Section 3.2). Only the cells where the estimation variance is under the threshold of 40% are considered as reliable. At each resolution, the two series of reference and simulated areal rainfall are compared.

## 5.1. Continuous statistics

As the bias is independent of the integration scale, we only concentrate on bulk variables such as the standard deviation and the percentage of explained variance that reflect the ability of the model to simulate the natural variability of rain fields. The results are shown in Figure 10 for the 1995 case and in Figure 11 for the 2001 case. The number of selected cells is shown in Figures 10d and 11d. By construction, the number of cells decreases as their size increases. These curves have a gradient close to -1; this indicates that the reliability of the reference values does not change much with resolution. Globally, the reliability of the statistical analysis decreases (size of the sample) with the resolution.

For the 1995 case, the model reproduces well the variability for the total rain field at small scales (Figure 10b). The ratio between the simulated and reference variances is almost one. As the scale increases (beyond 100 km²), the model displays more variability than the reference. For hourly rain fields (Figure 10a), the model displays systematically 1.5 to 2 times more variability than the reference, and it stays roughly stable over all scales. For the 2001 case, the model produces hourly
<figure><img src="image_13.png" /><figcaption>**Figure 11.** The top diagrams show the evolution of the standard deviations of the reference and of the simulations and their ratio (y) for the hourly rain fields (a) and for their total (b) for 2001. Diagram (c) shows the square of the correlation coefficient between the reference and the simulation for the hourly rain fields (dashed line) and their total (solid line). Diagram (d) displays the number of reliable reference values available.</figcaption></figure>

rain fields that are 3 to 6 times more variable than the reference, depending on the resolution (Figure 11a and Figure 11b). The problems are that the model produces more scattered rainfall and is unable to reproduce the stationarity of the storm.

The study of the correlation coefficient (Figure 10c for the 1995 case and Figure 11c for the 2001 case) confirms that the simulation of the 1995 case is better, with a difference in explained variance of at least a factor 10. For the hourly and the total rain fields of 1995, $R^2$ increases beyond 100 km$^2$. This increase in $R^2$ is exactly what was expected from the integration. For this case, there is apparently a scale of integration over which some defaults of the simulation become less penalising for hydrological applications. It would have been interesting to see how these curves get to an inflexion point further in scales. For the 2001 case, $R^2$ stays steadily low at all scales for both the hourly and the total rain fields.

The use of hydrologic surfaces instead of rectangles yields very similar results. Figure 12 displays the standard deviation ratios and the coefficients of correlation as a function of scales for the two events. The

curves show tendencies comparable to those obtained with rectangular cells. The main difference lies in the fluctuations of these curves that reflect the variability in shape of the hydrologic surfaces.

## 5.2. Scores based on contingency scores

The scores computed for the total rain fields do not change much with the resolution. They are therefore not shown.

The evolution of the thresholds used to compute the hourly scores based on contingency tables are shown in Figure 13. The values are quite low for the 1995 case because the intense rainfalls are localised in a narrow zone. Figures 14 and 15 show the evolution of the hourly scores for the 1995 and the 2001 events respectively. For each case, they present the number of surfaces where the reference (Figures 14g and 15g) and simulated (Figures 14h and 15h) areal rainfall is greater than the threshold. The scores are computed only if there are at least 30 cells with a reference areal rainfall greater than the threshold inside the confident zone. This choice introduces a difference, depending on the threshold.
<figure><img src="image_14.png" /><figcaption>**Figure 12.** Evolution of the statistical properties of the areal rainfall fields versus resolution with hydrological surfaces of validation. Hourly (dashed line) and 11-hour cumulated (solid line) ratio between the simulated and reference normalized standard deviation ('y') for the 1995 (a) and the 2001 (c) cases. Hourly (dashed line) and 11-hour cumulated (solid line) square of the correlation coefficient for the 1995 (b) and the 2001 (d) cases.</figcaption></figure>

<figure><img src="image_15.png" /><figcaption>**Figure 13.** Evolution with scales of the hourly rainfall values (mm) associated with the quantiles used to compute the binary scores for the 1995 (a) and the 2001 case (b).</figcaption></figure>
<figure><img src="image_16.png" /><figcaption>**Figure 14.** Scores computed on the 11 hourly simulated rain fields for the 1995 event versus resolution: ACC (a), FBIAS (b), POD (c), FAR (d), TS (e), HSS (f) (see text for a definition of these scores). Diagram (g) shows the number of cells where the reference areal rainfall is over the threshold ('x'). Diagram (h) shows the number of cells where the simulated areal rainfall is over the threshold ('y').</figcaption></figure>

The larger scale considered for the smallest threshold can be 3 to 8 times greater than the limit for the largest threshold.

For both cases, ACC (Figures 14a and 15a) is good for all surfaces, but see the comment on this in

Section 4.2. For the other scores, the commentaries are different for the two cases.

For the 1995 case, FBIAS (Figure 14b) does not evolve much with the scale; an increase can be noted after 100 km². The evolution with scale of POD, TS and HSS
<figure><img src="image_17.png" /><figcaption>**Figure 15.** Scores computed on the 11 hourly simulated rain fields for the 2001 event versus resolution: ACC (a), FBIAS (b), POD (c), FAR (d), TS (e), HSS (f) (see text for a definition of these scores). Diagram (g) shows the number of cells where the reference areal rainfall is over the threshold ('x'). Diagram (h) shows the number of cells where the simulated areal rainfall is over the threshold ('y').</figcaption></figure>

(Figure 14c, Figure 14e and Figure 14f), especially for the 70% and 80% quantiles, are similar to the evolution of *FBIAS*. The same comment can be made on *FAR* (Figure 14d) as the perfect forecast is when *FAR* = 0.

The evolution with the threshold is different. At a given scale, *FBIAS* gets better as the threshold increases; it is

about 1 for the 90% quantile. But *POD*, *TS* and *HSS* do not really depend on the threshold. The threshold has an influence on the *FAR*, but it must be an artifact: there is a link with *FBIAS* (if *FBIAS* is low, the simulation reaches the threshold less often than the reference; there are therefore fewer false alarms). All this leads to the conclusion that the total surface covered by heavy
rainfall is well simulated, but not its localisation. This
problem of localisation gets blurred when the scale
increases above 100 km²; it should be noted that the
same phenomenon is observed for R² for this case
(Figure 10c).

TS stays very low, when compared with the results
obtained by Zepeda-Arce et al. (2000). This is due to the
limited extent of our multi-scale study: we stop when
we have less than 30 surfaces, while Zepeda-Arce et al.
(2000) reach resolutions comparable with the size of
their total domain. There is therefore only one surface
left; TS can therefore only have a value of 0 or 1.

For the 2001 case, FBIAS does not evolve. For the 70%
quantile, the scores improve after 1000 km² (earlier for
FAR), which is more optimistic than the continuous
scores. The multi-scale study hardly reaches this surface
for the other thresholds.

In conclusion, this multi-scale approach, with scores
based on contingency tables, shows the same critical
resolution as the continuous scores for the 1995 case;
the critical resolution found for the 2001 case must be
treated cautiously, since the highest thresholds used in
the contingency tables do not allow us to reach the
considered surface: the existence of a critical surface is
therefore not confirmed.

6. Conclusions

The way in which quantitative precipitation forecasts
are validated is assessed by comparing classical scores
based on point observations with an areal validation.
To do this, we selected two cases for which the high-
resolution simulations of the rainfall fields do not have
the same quality: a subjective evaluation of the rain fields
produced suggests that the rain field in the 1995 case has
been better simulated than the one for 2001.

This is confirmed using classical scores, which em-
phasise the superiority of the 1995 simulation. The *BIAS*
is smaller for the 1995 case, the correlation much higher.
The scores based on contingency tables also confirm
this for all the thresholds we have considered, but these
scores which are based on point values of the rainfall
field have no real link with the real situations used in
hydrology.

The notion of areal rainfall, which is fundamental in
hydrology, can be dealt with by using a multi-scale
approach. The method is based on a distribution of the
rain fields among grids of the domain with different
aggregation grid sizes. The simulated and the observed
areal rainfall fields obtained are compared, and the
evolution of this comparison with the resolution of the
integrating surface is studied.

The effects of the problems of localisation should
not affect the comparison between the observed and

simulated rain fields for large surfaces of integration.
The expected improvement of the scores for large
surfaces raises the question of the existence and the value
of a critical surface.

The critical surface is perhaps at the limit of the scales
reached by the multi-scale study for the 2001 case,
but it is confirmed neither by the scores based on
contingency tables with high thresholds, nor by the
continuous scores. This means that aggregation is not
able to reduce the errors of the model for this case;
such a simulation cannot be used to force a hydrological
model in a flood forecasting system. On the contrary,
the critical surface is evaluated for better simulation at
a scale which corresponds to 16 times the surface of the
grid cell, and to the surface of medium watersheds in
the region.

The paper shows that a multi-scale study can bring
a notion of critical integrating size, which is needed
for hydrology, even if the surfaces do not have a
hydrological reality. It also shows that the use of classical
binary scores must be used with caution: they must be
studied in the context of the considered case (e.g. in our
case general underestimation of the rain) and must be
seen as complementary.

The representativeness errors have not yet been ad-
dressed. These errors and their dependence on the integ-
rating scale must be evaluated for the network of gauges
in the region.

The next step in the validation of quantitative precipita-
tion forecasts for hydrology is to show the efficiency of
these forecasts in producing a simulated river flow. This
is the object of the study by Chancibault et al. (2005).

Acknowledgments

The current study was part of the PATOM and PNRH
research programs of the CNRS-INSU, the French
Institute for the Universe Sciences. It was partly funded
by the European Research Programs HYDROTIMET
in the framework of INTERREG-IIIb Medocc and
FLOODsite in the 6th Framework Program.

References

Andrieu, H., Creutin, J.-D., Delrieu, G. & Faure, D. (1997) Use of a weather radar for the hydrology of a mountainous area. Part I: radar measurement interpretation. *J. of Hydrology* **193**: 1–25.

Barnston, A. G. (1992) Correspondence among the correla-
tion, RMSE, and Heidke forecast verification measures;
refinement of the Heidke score. *Wea. Forecasting* **7**: 699–
709.

Bechtold, P., Bazile, E., Guichard, F., Mascart, P. & Richard, E. (2001) A mass-flux convection scheme for regional and global models. *Q. J. R. Meteorol. Soc.* **127**: 869–886.
Point and areal validation of forecast precipitation fields

Benoit, R., Pellerin, P., Kouwen, N., Ritchie, H., Donaldson, N., Joe, P. & Soulis, E. D. (2000) Toward the use of coupled atmospheric and hydrologic models at regional scale. *Mon. Wea. Rev.* **128**: 1681–1706.

Beven, K. J. & Kirkby, M. J. (1979) A physically based variable contributing area model of basin hydrology. *Hydrol. Sci. Bull.* **24**: 43–69.

Bougeault, P. & Lacarrère, P. (1989) Parameterization of orography-induced turbulence in a mesobeta-scale model. *Mon. Wea. Rev.* **11**: 1872–1890.

Bougeault, P., Binder, P., Buzzi, A., Dirks, R., Houze, R., Kuettner, J., Smith, R. B., Steinacker, R. & Volkert, H. (2001) The MAP special observing period. *Bull. Amer. Meteorol. Soc.* **82**: 433–462.

Chancibault, K., Anquetin, S., Ducrocq, V. & Saulnier, G.-M. (2005) Hydrological evaluation of high resolution precipitation forecasts of the Gard flash-flood (8–9 September 2002), accepted for *Q. J. Roy. Meteor. Soc.*

Davis, R. S. (2001) Flash flood forecast and detection methods. *Severe convective storms, Meteorological monographs*.

Ducrocq, V., Lafore, J.-P., Redelsperger, J.-L. & Orain, F. (2000) Initialization of a fine-scale model for convective-system prediction: a case study. *Q. J. Roy. Meteor. Soc.* **126**: 3041–3065.

Ducrocq, V., Ricard, D., Lafore, J.-P. & Orain, F. (2002) Storm-scale numerical rainfall prediction for five precipitating events over France: on the importance of the initial humidity field. *Wea. Forecasting* **17**: 1236–1256.

Ebert, E. E. & McBride, J. L. (2000) Verification of precipitation in weather systems: determination of systematic error. *J. Hydrology* **239**: 179–202.

Fritsch, J. M., Houze, R. A. J., Adler, R., Bluestein, H., Bosart, L., Brown, J., Carr, F., Davis, C., Johnson, R. H., Junker, N., Kuo, Y. H., Rutledge, S., Smith, J., Toth, Z., Wilson, J. W., Zipser, E. & Zrnic, D. (1998) Quantitative precipitation forecasting: report of the 8th prospectus development team, U.S. Weather Research Program. *Bull. Amer. Meteorol. Soc.* **79**: 285–299.

Fujiyoshi, Y., Geng, B., Yoshimoto, N., Kanada, S. & Takeda, T. (1996) Structure and formation mechanism of a terrain trapped rain storm. *12th international commission on clouds and precipitation*, 570–572. Zurich, Switzerland.

Gandin, L. S. & Murphy, A. H. (1992) Equitable skill scores for categorical forecasts. *Mon. Wea. Rev.* **120**: 361–370.

Ghelli, A. & Lalaurette, F. (2000) Verifying precipitation forecasts using upscaled observations. *ECMWF Newsletter* **87**: 9–17.

Harris, D., Foufoula-Georgiou, E., Droegemeier, K. K. & Levit, J. J. (2001) Multiscale statistical properties of a high-resolution precipitation forecast. *J. Hydrometeorol.* **2**: 406–418.

Joss, J. & Waldvogel, A. (1990) Radar in meteorology. *Proc. Battan Memorial and 40th Anniversary Radar Meteorol. Conf.* 577–606.

Kain, J. S. & Fritsch, J. M. (1993) Convective parametrization for mesoscale models: The Kain-Fritsch scheme. *Meteorol. Monographs* **46**: 165–170.

Kieffer-Weisse, A. & Bois, P. (2001) Topographic effects on statistical characteristics of heavy rainfall and mapping in the French Alps. *J. Appl. Meteorol. 40*: 720–740.

Lafore, J.-P., Stein, J., Asencio, N., Bougeault, P., Ducrocq, V., Duron, J., Fischer, C., Héreil, P., Mascart, P., Masson, V., Pinty, J.-P., Redelsperger, J.-L., Richard, E. & Arellano, J. V.-G. d. (1998) The Meso-NH Atmospheric

Simulation System. Part I: adiabatic formulation and control simulations. *Annales Geophysicae* **16**: 90–109.

Lebel, T., Bastin, G., Obled, C. & Creutin, J.-D. (1987) On the accuracy of areal rainfall estimation: a case study. *Water Resources Research* **23**: 2123–2138.

Mason, I. (1989) Dependence of the critical success index on sample climate and threshold probability. *Aust. Meteorol. Mag.* **37**: 75–81.

Miniscloux, F., Creutin, J.-D. & Anquetin, S. (2001) Geostatistical analysis of orographic rainbands. *J. Appl. Meteorol. 40*: 1835–1854.

Morcrette, J. J. (1989) Description of the radiation scheme in the ECMWF model. *ECMWF Tech. Memo.* **165**: 536–549.

Noilhan, J. & Planton, S. (1989) A simple parametrization of land surface processes for meteorological models. *Mon. Wea. Rev.* **117**: 536–549.

Obled, C., Wendling, J. & Beven, K. (1994) The sensitivity of hydrological models to spatial rainfall patterns: an evaluation using observed data. *J. Hydrology* **159**: 305–333.

Pinty, J.-P. & Jabouille, P. (1998) A mixed-phase cloud parameterization for use in mesoscale non-hydrostatic model: simulations of a squall line and of orographic precipitations. *Proc. Conf. of Cloud Physics*, 217–220. Everett, WA, USA.

Redelsperger, J. L. & Sommeria, G. (1981) Méthode de représentation de la turbulence d'échelle inférieure à la maille pour un modèle tri-dimensionnel de convection nuageuse. *Bound.-Layer Meteorol.* **21**: 509–530.

Ricard, D. (2002) Initialisation et assimilation de données à méso-échelle pour la prévision à haute résolution des pluies intenses de la région Cévennes-Vivarais (in French). Ph.D., Université Paul Sabatier, Toulouse, France.

Roebber, P. J. & Eise, J. (2001) The 21 June 1997 flood: storm-scale simulations and implications for operational forecasting. *Wea. Forecasting* **16**: 197–218.

Scofield, R. A. (1985) Satellite convective categories associated with heavy precipitation. *Proc. Sixth Conf. on Hydrometeorology*, 42–51. Indianapolis.

Snook, J. S. & Pielke, R. A. (1995) Diagnosing a Colorado heavy snow event with a nonhydrostatic mesoscale numerical model structured for operational use. *Wea. Forecasting* **10**: 262–285.

Stein, J., Richard, E., Lafore, J. P., Pinty, J. P., Asencio, N., Cosma, S. & Volkert, H. (2000) High-resolution non-hydrostatic simulations of flash-flood episodes with grid-nesting and ice-phase parameterization. Heavy Precipitation in the Alpine Region (HERA). *Meteor. Atmos. Phys.* **72**: 203–221.

Suzuki, Y., Nakakita, E. & Ikebuchi, S. (2001) Study on the dependence properties of rainfall distributions on topography based on the altitude dependence. *Ann. J. Hydraulic Engineering* **45**: 301–306.

Tateya, K., Nakatsugawa, M. & Yamada, T. (1989) Observations and simulation of rainfall in mountainous areas. *Journal of Hydroscience and Hydraulic Engineering* **7**: 1–11.

Tustison, B., Harris, D. & Foufoula-Georgiou, E. (2001) Scale issues in verification of precipitation forecasts. *J. Geophys. Res.* **106**: 11775–11784.

Tustison, B., Foufoula-Georgiou, E. & Harris, D. (2003) Scale-recursive estimation for multisensor Quantitative Precipitation Forecast verification: A preliminary assessment. *J. Geophys. Res.* **108** (D8): 8377.
Warner, T. T., Brandes, E. A., Sun, J., Yates, D. N. & Mueller, C. K. (2000) Prediction of a flash flood in complex terrain. Part I: A comparison of rainfall estimates from radar, and very short range rainfall simulations from a dynamic model and an automated algorithmic system. *J. Appl. Meteorol.* **39**: 797–814.

Westrick, K. J. & Mass, C. F. (2001) An evaluation of high-resolution hydrometeorological modeling system for

prediction of a cool-season flood event in a coastal mountainous watershed. *J. Hydrometeorol.* **2**: 161–180.

Wilks, D. S. (1995) *Statistical Methods in the Atmospheric Sciences: An Introduction*. Academic Press.

Zepeda-Arce, J., Foufoula-Georgiou, E. & Droegemeier, K. K. (2000) Space-time rainfall organization and its role in validating quantitative precipitation forecasts. *J. Geophys. Res.* **105**: 10129–10146.