Annual Journal of Hydraulic Engineering, JSCE, Vol.50, 2006, February

A PSEUDO VALIDATION ALGORITHM FOR
HYDROLOGICAL MODEL RELIABILITY ASSESSMENT
Shen CHIANG1, Yasuto TACHIKAWA2 and Kaoru TAKARA3
1 Student Member of JSCE, Graduate Student, Dept. of Civil Eng., Kyoto University(Yoshida Hon-machi, Sakyo-ku, Kyoto, 606-8501, Japan)
2 Member of JSCE, Dr. Eng., Associate Professor, DPRI, Kyoto University (Gokasho, Uji, Kyoto 611-0011, Japan.)
3 Fellow of JSCE, Dr. Eng., Professor, DPRI, Kyoto University (Gokasho, Uji, Kyoto 611-0011, Japan.)A pseudo validation algorithm, which is capable of identifying the prediction uncertainty throughrecognizing and quantifying the different uncertainty sources in a hydrologic model, is manipulated as an
instrument for hydrological model reliability assessment. For implementation, the pseudo validation
algorithm is manipulated in order to compare TOPMODEL with different vertical flux calculation
components, which have been applied to two Japanese basins. An index, which originates from the
Nash-Sutcliffe efficiency, named Model Structure Indicating Index (MSII), is used to quantify the model
reliability under different magnitudes of input uncertainty. The results show that within a small
magnitude of input uncertainty, the reliability of a five parameter TOPMODEL is worse than a six
parameter TOPMODEL. However, within larger magnitudes of the input uncertainty, the reliability of
the five parameter TOPMODEL is better than that of the six parameter TOPMODEL, this shows that thepseudo validation algorithm can be used as a reference for hydrological modeling.
Key Words: prediction uncertainty, model validation, TOPMODEL, hydrologic modeling

1. INTRODUCTION
Commonly, validation is defined as theestimation of the confidence in the ability of any
prediction model to perform with a certain
qualitative outcome for its intended purpose. Inhydrologic science, the term is often used toindicate a thorough model testing.
It is atime-consuming process that requires a massive
amount of data. Similar to the model calibration
process, the quality/representativeness of the data
used for model validation is extremely essential.
However, in most cases, the accuracy of the input
data is actually unknown. As a consequent of this,all hydrologic models are being calibrated,
validated and manipulated under an unknown
magnitude of input uncertainty. The reason why
these models are still capable of regenerating arealistic and valid watershed response data series isdue to the models’ calibration processes. This fact
has seldom been explicitly discussed.
The capability of a hydrologic model toregenerate a realistic and valid watershed response
series under a certain level of data input uncertainty
is highly related to the model structure. By
acknowledging that input data (in this study, rainfalldata) errors propagate and persist in hydrologic
models, and corrupt the parameter estimation

processes, a pseudo validation process can be
performed by using limited observation data toassess the adaptability and reliability of ahydrologic model. Instead of acquiring massive
amounts of observation data, the pseudo validation
algorithm validates the model through observing themodel behavior under different magnitudes of the
input uncertainty1). In this paper, the logic behind
the algorithm is thoroughly discussed; then thevalidity of the algorithm is verified bydemonstrating
thecomparisonsoftwo
TOPMODELs with different vertical flux structure.
Moreover, it is shown that this method is used toaccess not only the validity of the model structure,
but also the reliability of input data.
The basic idea of the algorithm is: if a model iscapable to regenerate a realistic and valid watershed
response series in the future under unknown
magnitude of input uncertainty, it must capable tocompensate the error contaminated data undercertain level of input uncertainty by using theexisted record data. To do this, first, the Monte
Carlo simulation method is applied to add bias
items into the model input data series (rainfall), andthen the rainfall realizations, parameter space, andthe model outcomes (outflow discharge) under
different bias levels are acquired. Secondly, by
examining the counter relationship between the- 103 -

model simulation outcomes, the calibration
outcomes and the observed watershed response
series (discharge), an uncertainty structure can berecognized. Finally, through this process, theparameter uncertainty, calibration uncertainty, andthe model structure uncertainty, created by the input
data uncertainty are recognized, separated, andquantified.
By applying this pseudo validation process,
TOPMODELs with different vertical flux
calculation components are compared by applying it
to two Japanese watersheds.
The Nash-Sutcliffe effeciency2) and an index
which originated from the Nash-Sutcliffe efficiency
named the Model Structure Indicating Index(MSII)1), is used to quantify and assess thereliability of the candidate models, with a larger
value of MSII indicating a hydrologic model with apoorer structure. The results show that within asmall magnitude of input uncertainty, the reliabilityof a five parameter TOPMODEL is worse than a six
parameter TOPMODEL. However, within a larger
magnitude of input uncertainty, the reliability of afive parameter TOPMODEL is better than a six
parameter TOPMODEL. This indicates that thepseudo validation algorithm can be used as anefficient
instrument
for
water
resources
management in the areas of hydrological modelingand for rainfall runoff simulations of poorly
recorded or ungauged basins.

2. DESCRIPTION OF THE ALGORITHM
AND THE LOGIC BEHIND THE
PROCESSThe straightest way to think about a pseudovalidation algorithm is that: if a model is capable ofpredicting the future conditions of a particular typeof catchment under a certain level of input error, it
must be capable of predicting the current conditions
as well, under a certain level of input error.
The algorithm1) is used to test the stability of thepredictive capability possessed by a hydrologic
model under a certain magnitude of input
uncertainty. The idea can be clarified by referringto the simplified modeling procedure depicted in
Fig.1, which is modified according to Sargent3).
In Fig.1 the problem entity is the system to bemodeled; the conceptual model is the mathematicalrepresentation of the problem entity developed for aparticular study; the computerized model is theconceptual model implemented on a computer.
The conceptual model is developed through ananalysis and modeling phase; the computerized
model is developed through a computer
programming and implementation phase, andFig.1 Algorithm for uncertainty recognition and quantification.

Fig.2 Diagram of uncertainty recognition and quantification.

inferences about the problem entity are obtained byconducting computer experiments on thecomputerized model in the experimentation phase.
Conceptual model validity is the determination ofwhether or not the theories and assumptions
underlying the conceptual model are correct and thatthe model representation of the problem entity is“reasonable” for the intended purpose of the model.
Computerized model verification ensures thatthe computer programming and implementation of
the conceptual model is correct.
This is often
referred to as the model calibration process.
Operational validity is the determination that themodel’s output behavior has sufficient accuracy forthe model’s intended purpose over the domain of themodel’s intended applicability. In most cases, thisis the model validation process. Data validity
ensures that the data necessary for model building,model evaluation and testing, and conducting themodel experiments to solve the problem are
adequate and correct3). However, the precisenessis an unknown.
Conventionally, these models require massive
amounts of data for the validation process, whichmakes model validation an intricate mission to
perform.
Input data uncertainty makes data
validity, which is located in the core position of themodeling scheme, difficult to perform. Since there
is no way to assure the accuracy of the data used for- 104 -

model calibration and validation, the subsequent
result is that the best calibrated parameter set may or
may not equal to the “effective value”, which will
make a hydrologic model works properly. A feasible
alternative to this is to recognize that within certain
levels of input uncertainty, there is a possibility thata hydrologic model would still be capable toregenerate true watershed characteristics. In thissense, both the model calibration process and error
propagation scheme induced by the model structure
must be taken into consideration.
Fig.2 depicts the schematic diagram of thepseudo validation algorithm manipulated in thisstudy4). The bias items, which are located at thecenter of the structure, dominate the whole
uncertainty propagation scheme. If the focus isonly on the changing of the bias items and its impact
on the model outcome, which is referred to as theentire uncertainty (EU) in this study, it can be seen
as a sensitivity analysis of the input data error. By
using the system uncertainty (SU), indicated by thepredictive capability of the model under input
uncertainty, the distance between system uncertainty
andentireuncertainty
indicatestheeffectiveness/goodness of the representativeness ofthe calibrated parameter sets, which can be referred
to as a measure of model divergence5). Inherent
uncertainty (IU) represents the variability of theparameter sets generated from specified input
uncertainty levels. The distance between inherent
uncertainty and entire uncertainty indicates thecapability of model to adapt itself to the specified
input uncertainty, which is dominated by the model
structure. The quantified and categorized
uncertainty: system uncertainty, entire uncertainty
and inherent uncertainty are integrated into MSII,
which enables it to evaluate the goodness of themodel structure through observing its behavior
under certain magnitudes of input uncertainty.
(Detailed algorithm for the recognition, separationand the relationship between the categorized
uncertainties is described in Chiang et al., 2005. 1) )Fig.3 depicts a schematic diagram of theuncertainty structures. In Fig.3, Qo denotes observed
discharge; Qb denotes a model outcome with thebest-fitted parameter set and Qe: denotes a modeloutcome with a parameter set within the wholeparameter space. ε SU , ε EU and ε IU represent
the difference of hydrographs amongst the observed
data, estimated with a parameter set in the parameter
space, and estimated with the best fitted parameter
set. The system uncertainty, entire uncertainty andinherent uncertainty are evaluated by using ε SU ,ε EU and ε IU with Nash-Sutcliffe efficiency. The
quantified and categorized uncertainties: system

Schematic
structures.

Fig.3

diagramoftheuncertainty

uncertainty, entire uncertainty and inherent
uncertainty are integrated into MSII, which enables
it to evaluate the goodness of the model structure
through observing its behavior under certain
magnitudes of the input uncertainty. Model
Structure Indicating Index (MSII)1) is defined as:

MSII =

IU − EUSU(1)

The difference between the entire and theinherent uncertainty is used as the numerator in theEq.(1), while the system uncertainty is used as thedenominator. The numerator is expected to be asmaller value if the model is considered to have abetter chance of reproducing a realistic watershed
response series. It is a measure of the possibilityof a model to adapt itself to the input uncertainty.
The larger the magnitude the less likely the model
will be able to adapt itself to the error contaminatedinput data, which indicates the inability of thecalibrated parameter space to drive the model toreproduce a realistic watershed response due to theinsufficient structure of the hydrologic model.
The numerator shows a measure of the ability ofa hydrologic model to adapt to the uncertainty
contained input data. The numerator in itself is notenough to reveal the model structure, a component
showing the model predictive capability is needed.
The denominator of MSII indicates theeffectiveness/goodness of the model calibration
results and the predictive capability of the model,
which therefore, enables it to reflect the calibration
scheme explicitly.
The index interprets thevariance caused by the calibration process andmodel structure in a dimensionless form through theNash-Sutcliffe
efficiency.
Duringtheimplementation of the model evaluation, the system
uncertainties less than zero are excluded from thecalculation of MSII due to its insignificance.
Hence the smaller value of the MSII represents
better model structure. The range of MSII is:
0 ≤ MSII < ∞ .

- 105 -

3. MODEL DESCRIPTION
In order to perform a pseudo validation process,
TOPMODEL is used in this study. TOPMODELis almost 30 years old and has been the subject ofnumerous applications to wide variety of catchments.
The code used herein is a modification version
based on the TOPMODEL 95.02 acquired from theofficial
websiteof
TOPMODEL(http://www.es.lancs.ac.uk/hfdg/topmodel.html).
TOPMODEL is a set of programs for rainfall-runoff
modeling in single or multiple subcatchment in asemi-distributed way and using girded elevation
data for the catchment area. It is considered to be aphysically based model as its parameters can be,theoretically measured in situ6)7). Subcatchmentdischarge is routed to the catchment outlet by using
a time-area diagram with a constant velocity
throughout the catchment area, which is a parameterthat needs to be calibrated. The infiltration excess
mechanism and evapotransportation mechanism is
not included in this study.
In TOPMODEL, there are several parametersthat need to be calibrated before model validationand implementation. Basically they are: m, thedecay factor which is the controlling rate of declinetransmissivity with increasing storage deficit; T0 ,the hydraulic transmissivity; td : the time delay
constant for vertical flux calculation; RV: theoverland flow velocity. Q0 , the initial base flow
and Sr0, the initial root zone storage deficit, which
are specified at the start of simulation.
Inside TOPMODEL, the vertical drainage qv
from unsaturated zone storage at any point of
topographic index class i is calculated either by
Eq.(2):

qvi =

Suz
Si td

(2)

where Suz is the storage in the unsaturated zone, td isa time delay constant and Si is a local storage
deficit; or by Eq.(3)8):

qvi = αK 0e − S i / m(3)where α is the effective vertical hydraulicgradient, K0 is the saturated conductivity at thesurface, and m is a model parameter controlling therate of decline of transmissivity with increasingstorage deficit. If the value of α is set to unity,
thus assuming that the vertical flux is equal to thesaturated hydraulic conductivity just at the watertable, it is eliminated as a parameter8). Eq.(2) is theequation of a linear store with a time constant Si t dthat increases with increasing depth to the watertable7). In the model, the K0 is acquired by T0 ,that means the number of the parameter are five (m,T0 , Q0 , RV and Sr0)or six (m, T0 , td, Q0 , RV and
Sr0).
The two TOPMODEL which use thedifferent vertical flux computing components areapplied for the performance of model quantitative
comparison.

4. DATA DESCRIPTION
Two Japanese watersheds are used for thepseudo validation of the two TOPMODEL: Yasu
River basin (387 km2) and Kamishiba basin (211km2). The DEM data and the rainfall data used in
the study are described as below:(1) DEM data
For TOPMODEL, two data sets are needed
before the simulation, the histogram of topographic
index and the time-distance diagram, which are bothacquired through DEMs in this study.
50m
resolution raster DEM data is used for data
extraction. The DEM algorithm manipulated inthis study for preliminary processing (depression
removing, flow direction determination and flow
accumulation value calculation) is based on thealgorithm proposed by Jenson and Domingue9).
Topographic index derivation was obtained by using
DEM algorithm proposed by Quinn10).
(2) Rainfall and discharge data
Rainfall data of the Yasu River basin wascollected from four rainfall gauging stations inside
the watershed; they are Yasu, Minakuchi, Kouka
and Oogawara. The average precipitation was
obtained by using Thiessen polygon method. The
discharge data for the Yasu River basin is collectedfrom the ground gauging station and a rating curveis applied for transferring the water level to thedischarge.
For the Kamishiba basin, a radar rainfall datawith time interval of 10 minutes is transferred to theaverage rainfall with time interval of 1 hour along
the corresponding catchment range. Discharge
data is acquired from the inflow data of theKamishiba dam.
For each watershed, one flood event is selected
for Monte Carlo simulation. Random generated bias
items are added to the original input series for every
time step in order to formulate the rainfall
realizations with specified distribution and standard
deviations.
A normal distribution with a mean

- 106 -

5 parame te r TOPMOD EL (Yasu )5 parame te r TOPMOD EL (Kamish iba)

1

1

0 .9

0 .9

0 .8

0 .8
0 .7

0 .6

En tire Un c e rtain ty

0 .5
0 .4

In h e re n t Un c e rtain ty

EU , IU , SU

EU , IU , SU

0 .7

Syste m Un c e rtain ty

0 .6

En tire Un c e rtain ty

0 .5
0 .4

In h e re n t Un c e rtain ty

0 .3

0 .3

0 .2

0 .2

0 .1

0 .1

0

Syste m Un c e rtain ty

0
1

2

3

4

5

6

7

8

9

1

2

3

in pu t u n c e rtain ty (mm/ h r)

4

5

6

7

8

9

in pu t u n c e rtain ty (mm/ h r)

Fig.4 EU, IU and SU of the five parameter TOPMODEL which
applying to the Yasu River basin.

Fig.6 EU, IU and SU of the five parameter TOPMODEL which
applying to the Kamishiba basin.

6 parameter TOPMODEL (Yasu)

6 parameter TOPMODEL (Kamishiba)

1

1

0.9

0.9
0.8

0.8
0.7

0.6
0.5

Entire Uncertainty

0.4

System Uncertainty

EU, IU, SU

EU, IU, SU

0.7
Inherent Uncertainty

0.6

Entire Uncertainty

0.5

Inherent Uncertainty

0.4

System Uncertainty

0.3

0.3

0.2

0.2
0.1

0.1
0

0

1

1

2

3

4

5

6

7

8

9

2

3

4

5

6

7

8

9

input uncertainty (mm/hr)

input uncertainty (mm/hr)

Fig.5 EU, IU and SU of the six parameter TOPMODEL which
applying to the Yasu River basin.

equals to zero and standard deviation from 1.0 to
9.0 (mm/hr) is used to generate the rainfall
realizations under specific magnitude of input
uncertainty. For each specific magnitude of input
uncertainty, 100 rainfall realizations and 100
parameter sets are derived.
10,000 modeloutcomes are generated from the combination of the
100 rainfall realizations and the 100 parameter setsof input from the two TOPMODEL which were
derived previously.

5. RESULTS
The variation of the categorized uncertainty(entire, inherent and system uncertainty) under
different levels of input uncertainty is acquired
through the pseudo validation process. Fig.4 throughFig.7 are the variation of the categorized uncertaintyof the five and six parameter TOPMODEL applied in
the Yasu River basin and the Kamishiba basin.
The results show that there is a similar tendency
within larger magnitudes of the input uncertainty,
for the five parameter TOPMODEL to outperformthe six parameter TOPMODEL. Another interesting
result is that apparently the performance of theFig.7 EU, IU and SU of the six parameter TOPMODEL which
applying to the Kamishiba basin.

TOPMODEL (no matter what the vertical
calculation component is), when applied to theKamishiba basin, is better than the TOPMODEL
applied to the Yasu River basin. This indicatesthat the application of TOPMODEL on the
Kamishiba basin is more appropriate than theapplication on the Yasu River basin. Anotherexplanation is the quality/representativeness of therainfall and discharge data, which were used for thepseudo validation process. Since the average rainfall
data acquired for the Kamishiba basin is transferredfrom the calibrated radar data, the precision ishigher than the average rainfall data acquired for the
Yasu River basin, which is transferred from four
ground gauging stations. This is a good proof thatthe quality of the data used for validation is essential,
and the proposed method well reflects this condition.
Fig.8 is the MSII of the TOPMODEL with different
vertical calculation components applied to thedifferent catchments. As described in the previous
section, a smaller magnitude of MSII indicates a
better model structure/reliability against the level of
input uncertainty.
The rapid ascending tendency of the six
parameter TOPMODEL, when applied to the Yasu

- 107 -

MSII

14
12

5 parame te r
TOPMOD EL YASU

MSII

10
8

6 parame te r
TOPMOD EL YASU

6

5 parme te r TOPMOD ELKAMISHIBA

4

6 parme te r TOPMOD ELKAMISHIBA

2
0
1

2

3

4

5

6

7

8

9

in pu t u n c e rtain ty (mm/ h r)

Fig.8 MSII of the TOPMODEL with different vertical
calculation component which applying to different catchment.

River basin, implies that it has the lowest reliability
against increasing inputs of uncertainty amongst thefour tests. The five parameter TOPMODEL, whenapplied to the Kamishiba basin, reveals that it has
the best reliability against the different levels of
input uncertainty due to its MSII mildly ascending
tendency. The reason that the five parameter
TOPMODEL is more stable than the six parameter
TOPMODEL is due to the physical bases of its
parameter. Even the time delay constant td performs
well during small magnitudes of the input
uncertainty, with increasing input uncertainties,
parameters lacking physical bases are incapable tocompensate for these discrepancies. The results also
implicitly
indicate
that
if
the
quality/representativeness of the data used for validation
process is high preciseness, the model structure
won’t matter the simulation so much.

6. CONCLUSIONS AND DISCUSSIONSIn this study, a pseudo validation algorithm was
manipulated and is used to assess TOPMODEL in
the context of different vertical flux calculation
components when applied to two Japanesecatchments. The results show that TOPMODEL
with five parameters perform better than the six
parameters TOPMODEL, both in the Yasu Riverbasin and in the Kamishiba basin under a larger
magnitude of the input uncertainty, but for smaller
magnitudes of the input uncertainty, the opposite istrue. TOPMODEL with five parameters is morestable than the one with six parameters in both
catchments. The adaptability of the TOPMODEL
when applied to the Kamishiba basin catchment wassuperior to that of the catchments in the Yasu Riverbasin.
This
indicatesthatthequality/representativeness of the observed data of the theKamishiba basin is better than that of the Yasu river
basin.

The algorithm generates the parameter set
space by introducing noise items into input datawith a specified probability distribution. This
reflects the truth that the parameter uncertainty
came from the uncertainty of data at hand and theway the model structure responds to it. This
indicates the pseudo validation algorithm is not only
capable of assessing the reliability of the hydrologic
model structure and the calibration process, but also
implicitly indicates the quality and therepresentativeness of the data used for validation
process.
According to the research results, the pseudovalidation algorithm can be an efficient instrument
for model refinement assessment.
However,various rainfall styles should also be taken into
consideration. This will be done in the following
research.
ACKNOWLEDGMENT:
A part of this study was supported by Grant-in-Aid
for Scientific Research (A)(1) 16206050 (PI: Prof. K.
Takeuchi, Yamanashi Univ.) and CREST program of
JST (PI: Prof. K. Takara, Kyoto Univ.).
REFERENCES
1)

Chiang, S., Tachikawa, Y. and Takara, K.: An approachto uncertainty identification in hydrological modeling:
Annual of Hydraulic Engineering, JSCE, Vol.49, pp.
211-216, 2005.
2) Nash, J. E., and Sutcliffe, J. V.: River flow forecasting
through conceptual models, Part 1, A discussion ofprinciples: Journal of Hydrology, Vol.10, pp. 282-290,
1970.
3) Sargent, R. G.: Validation and verifocation of simulation
models. The 1999 Winter Simulation Conference, 5 – 8
December 1999, Squaw Peak, Phoenix, AZ. In Proceedingsof the 1999 Winter Simulation Conference, pp. 39-59, 1999.
4) Chiang, S.: Hydrologic Model Comparison and Refinement
through Uncertainty Recognition and Quantification,
Doctoral Thesis, Kyoto University, 2005.
5) Sage, A. D. and Melsa, J. L.: Estimation Theory with
Applications to Communication and Control. McGraw-Hill,
New York, 1971.
6) Beven, K., and Kirkby, M. J.: A Physically based variable
contributing area model of catchments hydrology:
Hydrological Sciences Bulletin, Vol.24, pp. 43-69, 1979.
7) Beven, K.: Topmodel: A Critique: Hydrological Processes,
Vol.11(9), pp. 1069-1085, 1997.
8) Beven, K., Lamb, R., Quinn, P., Romanowicz, R. and Freer
J.: TOPMODEL, Computer models of watershed hydrology,
Water Resources Publications, pp. 627-668, 1995.
9) Jenson, S. K. and Domingue, J. O.: Extracting Topographic
Structure from Digital Elevation Data for Geographic
Information-System
Analysis,
PhotogrammetricEngineering and Remote Sensing, Vol.54, pp. 1593-1600,
1988.
10) Quinn, P., Beven, K., Chevallier, P. and Planchon, O.: The
Prediction of Hillslope Flow Paths for Distributed
Hydrological Modeling Using Digital Terrain Models:Hydrological Processes, Vol.5, pp. 59-79, 1991.
(Received September 30, 2005)- 108 -