# Transferring global uncertainty estimates from gauged to ungauged catchments

François Bourgin, Vazken Andréassian, Charles Perrin, Ludovic Oudin

► To cite this version:

François Bourgin, Vazken Andréassian, Charles Perrin, Ludovic Oudin. Transferring global uncertainty estimates from gauged to ungauged catchments. Hydrology and Earth System Sciences, 2015, 19 (5), pp.2535-2546. 10.5194/hess-19-2535-2015. hal-01224248

HAL Id: hal-01224248

https://hal.sorbonne-universite.fr/hal-01224248

Submitted on 4 Nov 2015

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

<figure><img src="image_3.png" /></figure>

# Transferring global uncertainty estimates from gauged to ungauged catchments

F. Bourgin¹, V. Andréassian¹, C. Perrin¹, and L. Oudin²

¹Iirstea, Hydrosystems and Bioprocesses Research Unit (HBAN), Antony, France

²UPMC Univ. Paris 06, UMR 7619 Metis, Paris, France

Correspondence to: F. Bourgin (francois.bourgin@irstea.fr)

Received: 18 June 2014 – Published in Hydrol. Earth Syst. Sci. Discuss.: 14 July 2014

Revised: 28 March 2015 – Accepted: 20 April 2015 – Published: 29 May 2015

**Abstract.** Predicting streamflow hydrographs in ungauged catchments is challenging, and accompanying the estimates with realistic uncertainty bounds is an even more complex task. In this paper, we present a method to transfer global uncertainty estimates from gauged to ungauged catchments and we test it over a set of 907 catchments located in France, using two rainfall–runoff models. We evaluate the quality of the uncertainty estimates based on three expected qualities: reliability, sharpness, and overall skill. The robustness of the method to the availability of information on gauged catchments was also evaluated using a hydrometrical desert approach. Our results show that the method presents advantageous perspectives, providing reliable and sharp uncertainty bounds at ungauged locations in a majority of cases.

# 1 Introduction

## 1.1 Predicting streamflow in ungauged catchments with uncertainty estimates

Predicting the entire runoff hydrograph in ungauged catchments is a challenge that has attracted much attention during the last decade. In this context, the use of suitable conceptual rainfall–runoff models has proved to be useful, and because traditional calibration approaches based on observed discharge data cannot be applied to ungauged catchments, other approaches are required. Various methods have been proposed for the estimation of rainfall–runoff model parameters in ungauged catchments, as reported by the recent summary of the Prediction in Ungauged Basins (PUB) decade

(Blöschl et al., 2013; Hrachowitz et al., 2013; Parajka et al., 2013).

The estimation of predictive uncertainty is deemed good practice in any environmental modelling activity (Refsgaard et al., 2007). In hydrological modelling, the topic has been widely discussed for years, and no general agreement has yet been reached on how to adequately quantify uncertainty. In practice, various methodologies are currently used.

For gauged catchments, the methodologies include Bayesian approaches (see e.g. the review by Liu and Gupta, 2007), informal methods related to the GLUE (generalised likelihood uncertainty estimation) framework (Beven and Freer, 2001), multi-model approaches (Duan et al., 2007; Velazquez et al., 2010) and other global uncertainty quantification methods (Montanari and Brath, 2004; Solomatine and Shrestha, 2009; Weerts et al., 2011; Ewen and O'Donnell, 2012). A comprehensive review of the topic can be found in Matott et al. (2009) and Montanari (2011).

While many methods have been proposed for gauged catchments, only a few have been proposed for the estimation of predictive uncertainty on ungauged catchments. McIntyre et al. (2005) presented a GLUE-type approach consisting of transferring ensembles of parameter sets obtained on donor (gauged) catchments to target (ungauged) catchments. More recently, a framework based on constrained parameter sets was applied in several studies (Yadav et al., 2007; Zhang et al., 2008; Winsemius et al., 2009; Kapangaziwiri et al., 2012). It is a two-step procedure. The first step consists in estimating with uncertainty various summary metrics of the hydrograph, also called “signatures” of the catchments, or gathering other “soft” or “hard” information at the target ungauged catchment. The second step is the selec-
tion of an ensemble of model parameter sets: “acceptable” or “behavioural” parameter sets are those that yield sufficiently close simulated summary metrics compared to the regionalised metrics obtained during the first step. A Bayesian approach can also be used (Bulygina et al., 2011, 2012). The parameter sets are given a relative weight depending on the proximity of their summary metrics compared to regionalised metrics and depending on a priori information. The reader can refer to Wagener and Montanari (2011) for a comprehensive description of both formal and informal methods belonging to this framework.

One difficulty of the above-mentioned approaches lies in the interpretation of the uncertainty bounds obtained from the parameter ensemble predictions. As noted by McIntyre et al. (2005) and Winsemius et al. (2009), the uncertainty bounds cannot easily be interpreted as confidence intervals, and therefore it remains difficult to use them in practice. In addition, relying solely on an ensemble of model parameter sets to quantify total predictive uncertainty is often insufficient when imperfect rainfall–runoff models are used.

A pragmatic alternative consists in addressing the parameter estimation and the global uncertainty estimation issues separately. It has been argued by several authors (Montanari and Brath, 2004; Andréassian et al., 2007; Ewen and O'Donnell, 2012) that a posteriori characterisation of modelling errors of a “best” or “optimal” simulation can yield valid uncertainty bounds at gauged locations. In earlier studies, the terms “total uncertainty”, “global uncertainty” and “post-processing” approach have been used interchangeably to refer to this approach. The various sources of uncertainty are indeed lumped into a unique error term with the goal of estimating uncertainty bounds for model outputs.

As stated by Solomatine and Shrestha (2009),

> The historical model residuals (errors) between the model prediction ŷ and the observed data y are the best available quantitative indicators of the discrepancy between the model and the real-world system or process, and they provide valuable information that can be used to assess the predictive uncertainty.

Similarly, one could argue that the model residuals between the model's prediction and the observed data at neighbouring gauged locations are, perhaps, the best available indicators of the discrepancy between the model and the real-world system at the target ungauged location, under the condition that the increased uncertainty introduced by the regionalisation step compared to the calibration step is adequately taken into account.

The only attempt to apply a global uncertainty estimation approach at ungauged locations that we are aware of is the one presented by Roscoe et al. (2012). They quantified uncertainty for river stage prediction at ungauged locations by first estimating the residual errors at ungauged locations based on

residual errors at gauged locations, and then applying quantitative regression to these estimated errors.

## 1.2 Scope of the paper

The aim of this paper is to provide an estimation of the global uncertainty affecting runoff prediction at ungauged locations when a rainfall–runoff model and a regionalisation scheme are used.

To our knowledge, a framework based on residual errors and global uncertainty quantification has not yet been extensively tested in the context of prediction in ungauged catchments. This paper contributes to the search for methods able to provide uncertainty estimates when runoff predictions in ungauged catchments are sought.

# 2 Data and methods

Our objective is not to develop a new parameter regionalisation approach. Therefore, we purposely chose to use ready-to-use materials and methods and only focus on the uncertainty quantification issue. This study can be considered as a follow-up of the work by Oudin et al. (2008) on the comparison of regionalisation approaches. We only provide here an overview of the data set, the rainfall–runoff models and the parameter calibration and regionalisation approach, since the details can be found in Oudin et al. (2008).

## 2.1 Data set

A database of 907 French catchments was used. They represent various hydrological conditions, given the variability in climate, topography and geology in France. This set includes fast-responding Mediterranean catchments with intense precipitation as well as larger, groundwater-dominated catchments. Some characteristics of the data set are given in Table 1. Catchments were selected to have limited snow influence, since no snowmelt module was used in the hydrological modelling. Daily rainfall, runoff, and potential evapotranspiration (PE) data series over the 1995–2005 period were available. Meteorological inputs originate from Météo-France SAFRAN reanalysis (Vidal et al., 2010). PE was estimated using the temperature-based formula proposed by Oudin et al. (2005). Hydrological data were extracted from the HYDRO national archive (http://www.hydro.eaufrance.fr).

## 2.2 Rainfall–runoff models

Two daily, continuous lumped rainfall–runoff models were used.

- The GR4J rainfall–runoff model, an efficient and parsimonious daily lumped continuous rainfall–runoff model described by Perrin et al. (2003).
<table><caption>**Table 1.** Characteristics of the 907 catchments. *P* – precipitation, *PE* – potential evapotranspiration, *Q* – discharge.</caption><thead><tr><th rowspan="2"></th><th colspan="5">Percentiles</th></tr><tr><th>0.05</th><th>0.25</th><th>0.50</th><th>0.75</th><th>0.95</th></tr></thead><tbody><tr><td>Catchment area (km²)</td><td>27</td><td>73</td><td>149</td><td>356</td><td>1788</td></tr><tr><td>Mean annual precipitation (mm yr⁻¹)</td><td>753</td><td>853</td><td>978</td><td>1176</td><td>1665</td></tr><tr><td>Mean annual potential evapotranspiration (mm yr⁻¹)</td><td>549</td><td>631</td><td>659</td><td>700</td><td>772</td></tr><tr><td>Mean annual runoff (mm yr⁻¹)</td><td>133</td><td>233</td><td>344</td><td>526</td><td>1041</td></tr><tr><td>Q/P ratio</td><td>0.17</td><td>0.27</td><td>0.34</td><td>0.45</td><td>0.68</td></tr><tr><td>P/PE ratio</td><td>1.06</td><td>1.25</td><td>1.47</td><td>1.83</td><td>2.9</td></tr><tr><td>Median elevation (m)</td><td>76</td><td>149</td><td>314</td><td>645</td><td>1183</td></tr></tbody></table>

- The TOPMO rainfall-runoff model, inspired by TOP-MODEL (Beven and Kirkby, 1979). This version was tested on large data sets and showed performance comparable to that of the GR4J model, while being quite different (Michel et al., 2003; Oudin et al., 2008, 2010).

Using these two models rather than a single one makes it possible to draw more general conclusions. The two models use a soil moisture accounting procedure as well as routing stores. However, they differ markedly in the formulation of their functions. While the GR4J model uses two non-linear stores and a unit-hydrograph, the TOPMO model uses a linear and an exponential store, and a pure time delay.

The GR4J and TOPMO models have four and six free parameters, respectively. On gauged catchments, parameter estimation is performed using a local gradient search procedure, applied in combination with pre-screening of the parameter space (Mathevet, 2005; Perrin et al., 2008). This optimisation procedure has proved to be efficient in past applications for the conceptual models used here. As an objective function, we used the Nash and Sutcliffe (1970) criterion computed on root square transformed flows (NSVQ). This criterion was shown to yield a good compromise between different objectives (Oudin et al., 2006).

## 2.3 Regionalisation approach

By definition, no discharge data are available for calibrating parameter sets at ungauged locations. Therefore, other strategies are needed to estimate the parameters of hydrological models for prediction in ungauged catchments.

Oudin et al. (2008) assessed the relative performance of three classical regionalisation schemes over a set of French catchments: spatial proximity, physical similarity and regression. Several options within each regionalisation approach were tested and compared. Based on their results, the following choices were made here for the regionalisation approach, as they offered the best regionalisation solution.

- Poorly modelled catchments were discarded as potential donors: only catchments with a performance criterion

NSVQ in calibration above 0.7 were used as possible donors.

- The spatial proximity approach was used. It consists in transferring parameter sets from neighbouring catchments to the target ungauged catchment. The proximity of the catchments to the gauged catchments was quantified by the distances between catchment centroids.

- The output averaging option was chosen. It consists in computing the mean of the streamflow simulations obtained on the ungauged catchment with the set of parameters of the donor catchments.

- The number of neighbours was set to four and seven catchments for GR4J and TOPMO, respectively, following the work reported by Oudin et al. (2008).

# 3 Proposed approach: transfer of relative errors by flow groups

## 3.1 Description of the method

Transferring calibrated model parameters from gauged catchments to ungauged catchments is a well-established approach when parameters cannot be inferred from available data. The method presented here extends the parameter transfer approach to the domain of uncertainty estimation.

The main ideas underlying the proposed approach are to (i) treat each donor as if it was ungauged (simulating flow through the above described regionalisation approach), (ii) characterise the empirical distribution of relative errors (understood as the ratio between observed and simulated flows, i.e. considering a multiplicative model error) for each of these donors, and (iii) transfer global uncertainty estimates to the ungauged catchment.

The methodology used to transfer global uncertainty estimates can be described by the following steps, illustrated in Fig. 1.

1. Selection of catchments
Here we consider a target ungauged catchment (TUC).
<figure><img src="image_4.png" /><figcaption>**Figure 1.** Illustration of the proposed approach, in the case of *n* = 4 donors. Red catchments are first-level donors while green catchments are second-level donors. For Step 2b, the simulated discharge variable (x axis) is split into 10 equal-size groups. In Step 3, white dots represent the values of the upper and lower multiplicative coefficients for each group. See the text for the description of the four steps.</figcaption></figure>

This catchment has *n* neighbouring gauged catchments, called NGC₁, NGC₂, ..., NGC*n*, which will be considered as donors for the TUC. For the ith catchment NGCᵢ, one can also select *n* neighbouring catchments with the notation: NGCᵢ₁, NGCᵢ₂, ..., NGCᵢₙ, which

can be considered as donors for NGCᵢ. Obviously, the TUC catchment would be excluded from this set of second-order donor catchments.
2. Application of the parameter regionalisation scheme to the donor catchments NGC<sub>i</sub>

a. Apply the parameter regionalisation scheme to obtain a simulated discharge time series for each NGC<sub>i</sub> using neighbours NGC<sub>ij</sub> (with j between 1 and n).

b. Compute the relative errors of discharge reconstitution (i.e. the ratio between observed and simulated discharges) by comparing simulated and observed discharge series for catchment NGC<sub>i</sub> and create 10 groups of relative errors according to the magnitude of the simulated discharge. To ensure that each group contains the same number of points, we split the simulated discharge range into 10 subgroups of equal size, using the deciles of the simulated discharge distribution. Using several flow groups allows taking into account the possible variability of model error characteristics.

3. Computation of the multiplicative coefficients applicable to simulated discharge

a. Put together all the relative errors from the donors NGC<sub>ij</sub> (with j between 1 and n) according to the group they belong to; i.e. all relative errors of groups k of the n donors are assembled into a master group k. This is done for k between 1 and 10.

b. Compute the empirical quantiles of the relative error distribution within each master group k (with k between 1 and 10). Since relative errors were computed (i.e. ratio of simulated to observed discharge values), each quantile of relative errors can be considered a multiplicative coefficient applicable to the simulated discharge. These multiplicative coefficients will be used to obtain the prediction bounds.

4. Computation of the uncertainty bounds for the TUC

a. Apply the parameter regionalisation scheme to obtain a simulated discharge time series for the TUC using the parameter sets of the n neighbouring gauged catchments NGC<sub>1</sub>, NGC<sub>2</sub>, ..., NGC<sub>n</sub>.

b. Multiply the simulated discharge by the set of multiplicative coefficients obtained at Step 3b to obtain the uncertainty bounds. The coefficients calculated for the group k are used when the simulated discharge belongs to the group k.

Note that we based our approach on multiplicative errors and not on additive errors because using multiplicative coefficients yields prediction bounds for discharge that are always positive, whereas this might not always be the case with additive errors.

Finally, we mention that the choice to use 10 groups reflects a trade-off between the number of points available to

obtain reasonable estimates of empirical quantiles computed for each group and an adequate treatment of the variability of the characteristics of errors with the magnitude of simulated discharge. A larger (lower) number of groups could obviously be used if more (fewer) data are available (see discussion in Sect. 5.3) or based on the analysis of the statistical properties of errors.

## 3.2 Why should donors be considered as ungauged?

The first step deserves a brief explanation. The choice to treat donors as ungauged is related to the well-known fact that the performance of rainfall-runoff models decreases when they are applied at ungauged locations with a regionalisation scheme, compared to when local data are available for parameter estimation. The quadratic efficiency criterion used here is the C2M (Mathevet et al., 2006), a bounded version of the Nash and Sutcliffe (1970) efficiency (NSE) criterion. The criterion is based solely on the simulated discharges of the deterministic rainfall-runoff model and is completely independent of the application of the uncertainty method. The equations are

$$C2M = \frac{\text{NSE}}{2 - \text{NSE}}, \quad (1)$$

$$\text{NSE} = 1 - \frac{\sum_{t=1}^{T} (Q_{t}^{\text{obs}} - Q_{t}^{\text{sim}})^{2}}{\sum_{t=1}^{T} (Q_{t}^{\text{obs}} - \mu_{o})^{2}}, \quad (2)$$

where T is the total number of time steps, $Q_{t}^{\text{obs}}$ and $Q_{t}^{\text{sim}}$ are the observed and simulated discharge, respectively, at time-step t, and $\mu_{o}$ is the mean of the observed discharges. This bounded version has the advantage of avoiding large negative values which are difficult to interpret.

Figure 3 illustrates the general performance decrease for both models on our catchment set when a regionalisation scheme is used instead of a parameter estimation based on local data. As a consequence, we should expect predictive uncertainty at ungauged locations to be larger than predictive uncertainty at gauged locations, i.e. when the rainfall-runoff model is calibrated with observed discharge data. That is why donors must be considered as ungauged. We will come back to this important point in Sect. 5.

## 4 Quantitative evaluation of uncertainty bounds

We assessed the relevance of the 90% uncertainty bounds by focusing on three characteristics: reliability, sharpness and overall skill. A general introduction to probabilistic evaluation can be found in Gneiting et al. (2007), Wilks (2011), and Franz and Hogue (2011) for a more hydrological perspective.

Reliability refers to the statistical consistency of the uncertainty estimation with the observation, i.e. a 90% prediction interval is expected to contain approximately 90% of the
observations if prediction errors are adequately characterised by the uncertainty estimation. To estimate reliability, we used the coverage ratio (CR) index, computed as the percentage of observations contained in the prediction intervals.

Sharpness refers to the concentration of predictive uncertainty. The average width (AW) of the uncertainty bounds is widely used to quantify sharpness:

$$AW = \frac{1}{T} \sum_{t=1}^{T} (Q_{t}^{u} - Q_{t}^{l}), \qquad (3)$$

where $Q_t^l$ and $Q_t^u$ are, respectively, the lower and upper bounds of the prediction interval [$Q_t^l, Q_t^u$] at time-step $t$.

To ease comparison between catchments, we used the width of the 90% interval [$Q_5, Q_{95}$],

$$AW^{\text{clim}} = Q_{95} - Q_{5}, \qquad (4)$$

where $Q_5$ and $Q_{95}$ are the 5th and 95th percentiles of the flow duration curve. This value characterises the natural variability of the flows for a given catchment and has the same unit as the average width of the uncertainty bounds. It can be viewed as the average width of the uncertainty bounds of a climatological prediction, where the uncertainty bounds are constant over time and defined by the interval [$Q_5, Q_{95}$]. A graphical illustration is given in Fig. 2.

Comparing the two values AW and $AW^{\text{clim}}$ leads to the following dimensionless criterion called the average width index (AWI):

$$AWI = 1 - \frac{AW}{AW^{\text{clim}}}. \qquad (5)$$

It is positive if the uncertainty obtained by applying the rainfall-runoff model and the methodology presented here is reduced compared to the climatology, and negative otherwise.

Uncertainty bounds that are as sharp as possible and reasonably reliable are sought: sharp intervals that would consistently miss the target would be misleading, while overly large intervals that would successfully cover the observations at the expense of sharpness would be of limited value for decision making.

To complete the assessment of the prediction bounds, we used the interval score (Gneiting and Raftery, 2007). The interval score (IS) accounts for both the width of an uncertainty bound and the position of the observed value compared to the uncertainty bound. The scoring rule of the interval score at time-step $t$ is defined as

$$S_t = \begin{cases} (Q_t^u - Q_t^l) & \text{if } Q_t^l \le Q_t^{\text{obs}} \le Q_t^u \\ (Q_t^u - Q_t^l) + \frac{2}{1-\beta} (Q_t^l - Q_t^{\text{obs}}) & \text{if } Q_t^{\text{obs}} < Q_t^l \\ (Q_t^u - Q_t^l) + \frac{2}{1-\beta} (Q_t^{\text{obs}} - Q_t^u) & \text{if } Q_t^{\text{obs}} > Q_t^u \end{cases}, \qquad (6)$$

where $Q_t^{\text{obs}}$ is the value observed at time-step $t$ and $\beta$ is equal to 90% since a 90% interval is sought here. See Fig. 2 for an illustration of how $S$ is computed.

<figure><img src="image_5.png" /><figcaption>**Figure 2.** Illustration of the evaluation of the uncertainty bounds. $Q_5$ and $Q_{95}$ are the 5th and 95th percentiles of the flow duration curve. $S$ is the interval score defined at one time step for the situation where the observed value is above the upper limit of the uncertainty bound, with $k$ equal to 20 because a 90% interval is given. See the text for further details.</figcaption></figure>

$IS$ is the average value of $S_t$ over the time series:

$$IS = \frac{1}{T} \sum_{t=1}^{T} S_t. \qquad (7)$$

To ease comparison between catchments and evaluate the skill of the prediction bounds, we used the 90% interval [$Q_5, Q_{95}$] as a benchmark, similar to what we did for the sharpness index. The climatological prediction gives uncertainty bounds that are constant in time and defined by the interval [$Q_5, Q_{95}$], where $Q_5$ and $Q_{95}$ are the 5th and 95th percentiles of the flow duration curve. Thus we computed the interval skill score:

$$ISS = 1 - \frac{IS}{IS^{\text{clim}}}, \qquad (8)$$

where $IS^{\text{clim}}$ is the interval score obtained with the 90% interval [$Q_5, Q_{95}$]. Using skill scores is a very common approach in probabilistic forecasting. Dimensionless scores can thus be obtained, in much the same way as the computation of the well-known NSE criterion for assessing deterministic performance.

The interval skill score (ISS) is positive when the prediction bounds are more skilful than climatology, and negative otherwise. The best value that can be achieved is equal to 1.

# 5 Results and discussion

## 5.1 Reliability, sharpness and overall skill

Figure 4 shows the distributions of the three criteria used to evaluate the uncertainty bounds on the 907 catchments. Box-plots (5th, 25th, 50th, 75th and 95th percentiles) are used to summarise the variety of scores over the 907 catchments of the data set.
<figure><img src="image_6.png" /></figure>

**Figure 3.** Impact of the regionalisation scheme on deterministic performance, as quantified by the bounded C2M efficiency criterion. Note that in a very few cases, the performance obtained with the regionalisation scheme is better than the performance obtained with calibration. This is possible because of the output averaging option used by the regionalisation scheme.

<figure><img src="image_7.png" /></figure>

**Figure 4.** Distributions of the three performance criteria. Boxplots (5th, 25th, 50th, 75th and 95th percentiles) summarise the variety of scores over the 907 catchments of the data set.

### 5.1.1 Reliability

For both models, half of the catchments (from the lower quartile to the upper quartile) have CR values between 80 and 95 %. The median values are equal to 89 and 90 % for GR4J and TOPMO, respectively. Since a value of 90 % is expected for 90 % prediction bounds, these results suggest that the prediction bounds are in a majority of cases able to reflect the magnitude of errors when predicting runoff hydrographs in ungauged catchments, even though it is clear that the perfect value of 90 % is not reached in most cases.

The CR values fall below 70 % for around 14 % of the catchments with GR4J and 13 % with TOPMO, which indicates cases where the proposed approach yields predictive bounds that are clearly too narrow or biased (i.e. not well centred on the observations). Note that we did not find any guidance on how to properly evaluate the CR values in the literature. The results presented here may be used as a benchmark to comparatively assess the ranges of values that would be obtained in future studies.

### 5.1.2 Sharpness

Regarding sharpness, it can be seen that for GR4J, half of the catchments (from the lower quartile to the upper quartile) have AWI values between 39 and 67 %, while for TOPMO corresponding values are equal to 38 and 65 %. The median values are equal to 57 and 55 % for GR4J and TOPMO, respectively. The higher the AWI values, the lower the predictive uncertainty is. Since it would be utopic to expect that no errors will be made when predicting runoff hydrographs for ungauged catchments, we could consider here uncertainty reduction values between 30 and 80 % as quite satisfactory, even though we recognise that this statement is arbitrary since there are no widely agreed values to base our analysis on.
<figure><img src="image_8.png" /><figcaption>**Figure 5.** Distributions of the three performance criteria, obtained in two cases, (i) when the donor catchments are treated as ungauged (solid lines) and (ii) when the donor catchments are treated as gauged (dashed lines). Boxplots (5th, 25th, 50th, 75th and 95th percentiles) summarise the variety of scores over the 907 catchments of the data set.</figcaption></figure>

Note that negative values are seen for 7% of the catchments with both GR4J and TOPMO, which indicates cases where the approach yields prediction intervals whose average width is larger than the width of the [Q₅, Q₉₅] interval (Q₅ and Q₉₅ are the 5th and 95th percentiles of the flow duration curve).

### 5.1.3 Overall skill

Finally, Fig. 4c shows that the predictive skill for both models is positive for most catchments. For both models, half of the catchments (from the lower quartile to the upper quartile) have ISS values between 40 and 70%. The median values are equal to 61 and 59% for GR4J and TOPMO, respectively. While it might be argued that the unconditional climatology is not a very challenging benchmark, we consider that it is still a positive and reassuring result.

## 5.2 Do we need to treat the donor catchments as ungauged?

As mentioned earlier, a critical step of the proposed approach is to apply the regionalisation scheme to obtain a simulated discharge time series for each donor catchment (Step 2a). To assess the impact of this methodological choice, we applied the methodology described earlier to transfer uncertainty estimates, but this time the donor catchments are treated as gauged.

Similar to Fig. 4, Fig. 5 shows the distributions of the three criteria obtained in the two cases: whether or not the donor catchments are treated as ungauged. We can observe a drop in reliability for both models, whereas sharpness increases. This is because the relative errors are smaller when the donor catchments are treated as gauged, yielding narrower but less reliable prediction bounds for the target catchment. Interestingly, this results in skill scores that are quite similar: im-

provements in terms of sharpness compensate decreases in terms of reliability.

Note that reliability is generally considered as a prevailing characteristic over sharpness, since it reflects the ability of the uncertainty method to adequately reflect the magnitude of errors we might expect at locations for which prediction is done. Therefore, the benefit of treating the donor catchments as ungauged clearly appears in Fig. 5a, illustrating the theoretical argument presented in the methodological section.

## 5.3 Do we need to use groups of relative errors?

Another critical step of the proposed approach is to use 10 groups of relative errors. The groups are defined according to the magnitude of the simulated discharge (Step 2b). This was done to take into account the fact that the characteristics of errors usually change according to the magnitude of the simulated discharge. To assess the impact of this methodological choice, we again applied the methodology described earlier to transfer global uncertainty estimates, but this time using only one group instead of 10.

Figure 6 shows the distributions of the three criteria obtained in the following two cases: whether 10 groups or only one group of relative errors are used. For both models, reliability slightly increases when going from 10 groups to a single group, whereas both sharpness and skill decrease. It appears that improvements in terms of reliability are not sufficient to compensate for decreases in terms of sharpness when overall skill is considered. This can be understood by the fact that considering a single group instead of a few groups widens the uncertainty bounds on average, since the errors are generally heteroscedastic.

Obviously, although it appears that a single group is not enough to account for the variability of properties of relative errors, 10 groups may not provide significant performance gains and a compromise may be sought. The visual inspec-
<figure><img src="image_9.png" /></figure>

**Figure 6.** Distributions of the three performance criteria, obtained in two cases: (i) when 10 groups of relative errors are used (solid lines) and (ii) when only one group is used (dashed lines). Boxplots (5th, 25th, 50th, 75th and 95th percentiles) summarise the variety of scores over the 907 catchments of the data set.

<figure><img src="image_10.png" /></figure>

**Figure 7.** Impact of deterministic performance, as quantified by the bounded C2M quadratic criterion, on the three performance criteria for the 907 catchments. Note that for easier visualisation, the lower limits of the AWI (b) and ISS (c) values are set to -100% but lower AWI values are obtained in seven cases for both models, and lower ISS values are obtained in 18 and 22 cases for GR4J and TOPMO, respectively.

tion of scatter plots between relative errors and simulated discharge reveals that the shapes can be very different between catchments, hence potentially requiring different numbers of groups. Moreover, the simulation objectives, e.g. simulating intermediate or extreme flows, may also be considered when choosing the number of flow groups. Hence it appears that the number of groups may need further trial-and-error tests in specific applications to obtain the best compromise.

Although our tests reveal that the number of groups is a sensitive setting of the method, further research would be needed to evaluate whether different numbers of groups can be advised for specific objectives or conditions.

## 5.4 How does the performance of the rainfall–runoff models relate to the characteristics of uncertainty bounds?

To gain insights into the possible relationships between the performance of the deterministic rainfall–runoff simulations and the characteristics of the uncertainty bounds at ungauged locations, the three criteria used to characterise the uncertainty bounds are plotted in Fig. 7 as a function of a quadratic efficiency criterion for the 907 catchments, the C2M defined in Eq. (1).

A trend appears between deterministic performance and characteristics of the prediction bounds at ungauged locations, for the two rainfall–runoff models. The reliability in-
<table><caption>**Table 2.** Mean C2M values over the 907 catchments of the data set, with calibration (CAL), regionalisation (REGIO), and with the hydro-metrical desert (HD) defined by increasing distance (10, 20, 50, 100 and 200 km).</caption><thead><tr><th></th><th>CAL</th><th>REGIO</th><th>HD-10</th><th>HD-20</th><th>HD-50</th><th>HD-100</th><th>HD-200</th></tr></thead><tbody><tr><td>GR4J</td><td>0.67</td><td>0.51</td><td>0.49</td><td>0.46</td><td>0.43</td><td>0.41</td><td>0.35</td></tr><tr><td>TOPMO</td><td>0.59</td><td>0.47</td><td>0.46</td><td>0.44</td><td>0.41</td><td>0.39</td><td>0.34</td></tr></tbody></table>

dex exhibits greater variability compared to the sharpness index, and the stronger link is seen for the skill score. Reliability is relatively less affected by the poor deterministic performance of the simulation at an ungauged location because there are cases where poor performance at neighbouring locations leads (through the transfer of relative errors) to wide prediction bounds that are able to cover the observed values. We can also observe that skill scores and C2M scores are strongly related, which indicates that when the transfer of model parameters performs well, the transfer of global uncertainty estimates also performs well.

## 5.5 How does the method perform in data-sparse conditions?

The results presented so far were obtained with a dense network of gauging stations. To investigate the impact of the network density on our results, we applied a demanding test called the hydrometrical desert. It consists in excluding potential donors that are closer to the target ungauged catchment than a given threshold. For example, a threshold distance of 100 km means that the closest donor catchment must be at least 100 km away from the ungauged target catchment. This test results in a notable decrease of deterministic performance, as shown in Table 2, where the mean of the C2M efficiency criterion over the 907 catchments is reported for both models. Note that this is a more demanding test than a decrease of network density, because catchments retain the possibility of still having close donors in this case.

Figure 8 shows the distributions of the three criteria obtained by applying the hydrometrical desert with threshold values of 10, 20, 50, 100 and 200 km, respectively. A clear decrease appears with increasing distances. While we should expect that the sharpness of the uncertainty bounds decreases because of larger errors, and that this situation leads to a decrease of skill, the results in terms of reliability reveal the limitation of the method. With increasing distances, the method becomes less able to transfer the appropriate magnitude of the larger errors.

## 6 Conclusions

Runoff hydrograph prediction in ungauged catchments is notoriously difficult, and attempting to estimate the predictive uncertainty in that context is a further challenge. We have proposed a method allowing the transfer of global uncertainty estimates from gauged to ungauged catchments.

<figure><img src="image_11.png" /><figcaption>**Figure 8.** Impact of the hydrometrical desert on the distributions of the three performance criteria. Potential donor catchments are not retained as donors when their distance to the target catchment is below 10, 20, 50, 100 and 200 km. Boxplots (5th, 25th, 50th, 75th and 95th percentiles) summarise the variety of scores over the 907 catchments of the data set.</figcaption></figure>

The method extends the parameter transfer approach to the domain of global uncertainty estimation.

We evaluated the approach over a large set of 907 catchments by assessing three expected qualities of the uncertainty estimate: reliability, sharpness and overall skill. We applied two different rainfall-runoff models (GR4J and TOPMO) to ensure that the results presented are not model-specific. These results demonstrate that the method is generally able
to reflect model errors at ungauged locations and provide reasonable reliability.

Nonetheless, the following limitations to the study can be mentioned.

1. Although the approach seems promising on average on the large catchment set we used, it is not able to adequately quantify the predictive uncertainty for some catchments and it failed in some cases.

2. The method might not perform well in regions with sparser gauging networks than the one used here, as revealed by the application of a demanding test called the hydrometrical desert.

3. We only tested the 90% prediction intervals, whereas the method could be applied to obtain other prediction intervals. We made this choice to keep the article as simple as possible, but further work could be done in that direction.

4. We also noted that the number of flow groups used in the approach may be a sensitive setting of the method, and further research would be needed to provide more detailed guidance on this point depending on the structure of the model errors and the modelling objectives.

It is worth stressing that although we used a transfer based on spatial proximity, the approach presented in this article is not only independent of the rainfall-runoff model but also of the regionalisation scheme used to obtain deterministic prediction at ungauged locations. Any other similarity measure could be a basis for transferring residual errors, including physically based similarity measures. Accordingly, the regionalisation settings, including the output averaging option, could be adapted if deemed more appropriate.

Since we believe that uncertainty quantification has to be considered in any modelling study, further work should be devoted to the search for similarity measures that not only perform well in allowing the transfer of parameter sets from donor to target catchments, but also allow transferring modelling error characteristics.

Last, we would like to stress that the results presented in this study are expressed in terms of dimensionless measures. As such, they can provide a basis for future comparisons when prediction in ungauged catchments with uncertainty estimates is performed.

*Acknowledgements.* The authors thank Météo-France for providing the meteorological data and Banque HYDRO for the hydrological data. The financial support of SCHAPI to the first author is also gratefully acknowledged. The authors also thank the editor Ross Woods, who handled the manuscript, and the five reviewers, including Alberto Viglione and Denis Hughes, for their constructive comments, which helped improve the manuscript.

Edited by: R. Woods

## References

Andréassian, V., Lerat, J., Loumagne, C., Mathevet, T., Michel, C., Oudin, L., and Perrin, C.: What is really undermining hydrologic science today?, Hydrol. Process., 21, 2819–2822, doi:10.1002/hyp.6854, 2007.

Beven, K. and Freer, J.: Equifinality, data assimilation, and uncertainty estimation in mechanistic modelling of complex environmental systems using the GLUE methodology, J. Hydrol., 249, 11–29, 2001.

Beven, K. J. and Kirkby, M. J.: A physically based, variable contributing area model of basin hydrology, Hydrolog. Sci. Bull., 24, 43–69, doi:10.1080/02626667909491834, 1979.

Blöschl, G., Sivapalan, M., Wagener, T., Viglione, A., and Savenije, H.: Runoff Prediction in Ungauged Basins: Synthesis Across Processes, Places and Scales, Cambridge University Press, 2013.

Bulygina, N., McIntyre, N., and Wheater, H.: Bayesian conditioning of a rainfall-runoff model for predicting flows in ungauged catchments and under land use changes, Water Resour. Res., 47, W02503, doi:10.1029/2010wr009240, 2011.

Bulygina, N., Ballard, C., McIntyre, N., O'Donnell, G., and Wheater, H.: Integrating different types of information into hydrological model parameter estimation: Application to ungauged catchments and land use scenario analysis, Water Resour. Res., 48, W06519, doi:10.1029/2011wr011207, 2012.

Duan, Q., Ajami, N. K., Gao, X., and Sorooshian, S.: Multi-model ensemble hydrologic prediction using Bayesian model averaging, Adv. Water Resour., 30, 1371–1386, doi:10.1016/j.adwatres.2006.11.014, 2007.

Ewen, J. and O'Donnell, G.: Prediction intervals for rainfall-runoff models: raw error method and split-sample validation, Hydrol. Res., 43, 637–648, doi:10.2166/nh.2012.038, 2012.

Franz, K. J. and Hogue, T. S.: Evaluating uncertainty estimates in hydrologic models: borrowing measures from the forecast verification community, Hydrol. Earth Syst. Sci., 15, 3367–3382, doi:10.5194/hess-15-3367-2011, 2011.

Gneiting, T. and Raftery, A. E.: Strictly proper scoring rules, prediction, and estimation, J. Am. Stat. Assoc., 102, 359–378, doi:10.1198/016214506000001437, 2007.

Gneiting, T., Balabdaoui, F., and Raftery, A. E.: Probabilistic forecasts, calibration and sharpness, J. Roy. Stat. Soc. Ser. B, 69, 243–268, doi:10.1111/j.1467-9868.2007.00587.x, 2007.

Hrachowitz, M., Savenije, H. H. G., Bloeschl, G., McDonnell, J. J., Sivapalan, M., Pomeroy, J. W., Arheimer, B., Blume, T., Clark, M. P., Ehret, U., Fenicia, F., Freer, J. E., Gelfan, A., Gupta, H. V., Hughes, D. A., Hut, R. W., Montanari, A., Pande, S., Tetzlaff, D., Troch, P. A., Uhlenbrook, S., Wagener, T., Winsemius, H. C., Woods, R. A., Zehe, E., and Cudennec, C.: A decade of Predictions in Ungauged Basins (PUB)a review, Hydrolog. Sci. J., 58, 1198–1255, doi:10.1080/02626667.2013.803183, 2013.

Kapangaziwiri, E., Hughes, D. A., and Wagener, T.: Incorporating uncertainty in hydrological predictions for gauged and ungauged basins in southern Africa, Hydrolog. Sci. J., 57, 1000–1019, doi:10.1080/02626667.2012.690881, 2012.

Liu, Y. Q. and Gupta, H. V.: Uncertainty in hydrologic modeling: Toward an integrated data assimilation framework, Water Resour. Res., 43, W07401, doi:10.1029/2006wr005756, 2007.

Mathevet, T.: Quels modèles pluie-débit globaux au pas de temps horaire? Développements empiriques et comparaison de modèles
sur un large échantillon de bassins versants, PhD thesis, Paris,
France, 2005.

Mathevet, T., Michel, C., Andréassian, V., and Perrin, C.: A bounded version of the Nash-Sutcliffe criterion for better model assessment on large sets of basins, IAHS-AISH Publication, IAHS Red Books Series, 307, 211–219, 2006.

Matott, L. S., Babendreier, J. E., and Purucker, S. T.: Evalu-
ating uncertainty in integrated environmental models: A re-
view of concepts and tools, Water Resour. Res., 45, W06421,
doi:10.1029/2008wr007301, 2009.

McIntyre, N., Lee, H., Wheater, H., Young, A., and Wagener, T.: Ensemble predictions of runoff in ungauged catchments, Water Resour. Res., 41, W12434, doi:10.1029/2005wr004289, 2005.

Michel, C., Perrin, C., and Andreassian, V.: The exponential store: a correct formulation for rainfall-runoff modelling, Hydrolog. Sci. J., 48, 109–124, doi:10.1623/hysj.48.1.109.43484, 2003.

Montanari, A.: Uncertainty of Hydrological Predictions, in: Treatise on Water Science, edited by: Peter, W., Elsevier, Oxford, 459–478, 2011.

Montanari, A. and Brath, A.: A stochastic approach for assess-
ing the uncertainty of rainfall-runoff simulations, Water Resour.
Res., 40, W01106, doi:10.1029/2003wr002540, 2004.

Nash, J. E. and Sutcliffe, J. V.: River flow forecasting through con-
ceptual models part I – A discussion of principles, J. Hydrol., 10,
282–290, 1970.

Oudin, L., Hervieu, F., Michel, C., Perrin, C., Andreassian, V., Anctil, F., and Loumagne, C.: Which potential evapotranspi- ration input for a lumped rainfall-runoff model? Part 2 – Towards a simple and efficient potential evapotranspiration model for rainfall-runoff modelling, J. Hydrol., 303, 290–306, doi:10.1016/j.jhydrol.2004.08.026, 2005.

Oudin, L., Andreassian, V., Mathevet, T., Perrin, C., and Michel, C.: Dynamic averaging of rainfall-runoff model simulations from complementary model parameterizations, Water Resour. Res., 42, W07410, doi:10.1029/2005wr004636, 2006.

Oudin, L., Andreassian, V., Perrin, C., Michel, C., and Le Moine, N.: Spatial proximity, physical similarity, regression and ungauged catchments: A comparison of regionalization approaches based on 913 French catchments, Water Resour. Res., 44, W03413, doi:10.1029/2007wr006240, 2008.

Oudin, L., Kay, A., Andreassian, V., and Perrin, C.: Are seemingly
physically similar catchments truly hydrologically similar?,
Water Resour. Res., 46, W11558, doi:10.1029/2009wr008887,
2010.

Parajka, J., Viglione, A., Rogger, M., Salinas, J. L., Sivapalan, M., and Bloeschl, G.: Comparative assessment of predictions in ungauged basins – Part 1: Runoff-hydrograph studies, Hydrol. Earth Syst. Sci., 17, 1783–1795, doi:10.5194/hess-17-1783-2013, 2013.

Perrin, C., Michel, C., and Andreassian, V.: Improvement of a parsimonious model for streamflow simulation, J. Hydrol., 279, 275–289, doi:10.1016/s0022-1694(03)00225-7, 2003.

Perrin, C., Andreassian, V., Serna, C. R., Mathevet, T., and Le Moine, N.: Discrete parameterization of hydrological models: Evaluating the use of parameter sets libraries over 900 catchments, Water Resour. Res., 44, W08447, doi:10.1029/2007wr006579, 2008.

Refsgaard, J. C., van der Sluijs, J. P., Hojberg, A. L., and Vanrol-
leghem, P. A.: Uncertainty in the environmental modelling pro-
cess – A framework and guidance, Environ. Model. Softw., 22,
1543–1556, doi:10.1016/j.envsoft.2007.02.004, 2007.

Roscoe, K. L., Weerts, A. H., and Schroevers, M.: Estimation of the uncertainty in water level forecasts at ungauged river locations using quantile regression, Int. J. River Basin Manage., 10, 383–394, doi:10.1080/15715124.2012.740483, 2012.

Solomatine, D. P. and Shrestha, D. L.: A novel method to estimate model uncertainty using machine learning techniques, Water Resour. Res., 45, W00B11, doi:10.1029/2008wr006839, 2009.

Velazquez, J. A., Anctil, F., and Perrin, C.: Performance and reli-
ability of multimodel hydrological ensemble simulations based
on seventeen lumped models and a thousand catchments, Hy-
drol. Earth Syst. Sci., 14, 2303–2317, doi:10.5194/hess-14-2303-
2010, 2010.

Vidal, J.-P., Martin, E., Franchisteguy, L., Baillon, M., and Soubey-
roux, J.-M.: A 50-year high-resolution atmospheric reanalysis
over France with the Safran system, Int. J. Climatol., 30, 1627–
1644, doi:10.1002/joc.2003, 2010.

Wagener, T. and Montanari, A.: Convergence of approaches toward reducing uncertainty in predictions in ungauged basins, Water Resour. Res., 47, W06301, doi:10.1029/2010wr009469, 2011.

Weerts, A. H., Winsemius, H. C., and Verkade, J. S.: Estima-
tion of predictive hydrological uncertainty using quantile re-
gression: examples from the National Flood Forecasting Sys-
tem (England and Wales), Hydrol. Earth Syst. Sci., 15, 255–265,
doi:10.5194/hess-15-255-2011, 2011.

Wilks, D. S.: Statistical methods in the atmospheric sciences,
3rd Edn., Academic, Oxford, 2011.

Winsemius, H. C., Schaeфli, B., Montanari, A., and Savenije,
H. H. G.: On the calibration of hydrological models in un-
gauged basins: A framework for integrating hard and soft
hydrological information, Water Resour. Res., 45, W12422,
doi:10.1029/2009wr007706, 2009.

Yadav, M., Wagener, T., and Gupta, H.: Regionalization of con-
straints on expected watershed response behavior for improved
predictions in ungauged basins, Adv. Water Resour., 30, 1756–
1774, doi:10.1016/j.advwatres.2007.01.005, 2007.

Zhang, Z., Wagener, T., Reed, P., and Bhushan, R.: Reducing uncer-
tainty in predictions in ungauged basins by combining hy-
drologic indices regionalization and multiobjective optimization,
Water Resour. Res., 44, W00B04, doi:10.1029/2008wr006833,
2008.