Annual Journal of Hydraulic Engineering, JSCE, Vol.51, 2007, February
Annual Journal of Hydraulic Engineering, JSCE, Vol.51, 2007, February

IDENTIFICATION OF MODEL STRUCTURALSTABILITY THROUGH COMPARISON OF
HYDROLOGIC MODELS
Giha LEE 1, Yasuto TACHIKAWA2 and Kaoru TAKARA3
1Student Member, Graduate student, Dept. of Urban and Environmental Eng., Kyoto University(Kyoto 606-8501, Japan) E-mail:leegiha@flood.dpri.kyoto-u.ac.jp
2Member of JSCE, Dr. Eng., Associate Professor, DPRI, Kyoto University (Gokasho, Uji 611-0011, Japan)
3Fellow of JSCE, Dr. Eng., Professor, DPRI, Kyoto University (Gokasho, Uji 611-0011, Japan)

No matter how sophisticated and accurate hydrologic models may be, prediction uncertainty is
unavoidable problem in rainfall-runoff modeling and it stems from various components. Therefore, it isessential to identify and reduce sources of uncertainty for more precise agreement between model
prediction and observations in the real system. In the context of uncertainty issues, the problem of model
structural uncertainty or stability is an issue of increasing interest in recent research. This paper examinesa nature of model structural inadequacy using a single-objective global optimization method inhydrological modeling and proposes a framework to assess model structural stability through a
comparison of two hydrologic models.
Key Words: prediction uncertainty, model structural stability, single-objective global optimization

1. INTRODUCTION
The principal and intense task of modelers (e.g.,hydrologists and engineers) is to identify ahydrologic model through the estimation of an
optimal parameter set within a specific model
structure, which is suitable for given catchment
characteristics, historical data and intended
modeling purpose. The model identification process
therefore consists of two steps generally1). The firststep is the selection of a proper hydrologic model
structure and the second one is the identification ofan appropriate parameter set. However, this work is
not easy due to an influence of uncertainties
involved in modeling procedure that are also
unavoidably propagated into the prediction of model
output variables (e.g., stream flow). This model
output uncertainty originates from various sources:
input uncertainty; measurement errors; parameter
uncertainty; and model structure uncertainty.
A great deal of research has been proposed torecognize the propagation of the uncertainties whichappear in the different components of therainfall-runoff modeling into model prediction2),3),4).
Beven and Binely2) suggest a method of estimating
parameter uncertainty and its propagation,
Generalized Likelihood Uncertainty Estimation(GLUE) approach. On the other hand, Kavetski etal.4) review the influence of forcing input
uncertainty on model output in hydrologicalmodeling.
Especially, the problem of model structural
uncertainty with advanced automatic calibration
methods is an issue of increasing interest in recent
researches3),5),6). Structure error is unavoidable
problem in hydrological modeling since hydrologic
models are conversion and simplification of reality,
thus no matter how sophisticated and accurate they
may be those models only represent aspects ofconceptualization or empiricism of modelers. In
consequence, output time series of hydrologic
models are as reliable as hypothesis, structure ofmodels, quantity and quality of available forcing
data, parameter estimates7).
Gupta et al.3) pointed out that one parameter set
may be insufficient to represent the behavior of thecatchment due to the inadequacy of model
structures. In other words, a subjective selection of
objective functions for calibration of conceptualhydrologic models results in an overemphasis on
different response modes such as low and high flow
periods. This fact implies different parameter
combinations can be existent according to various
objective functions due to the presence of structural- 49 -

uncertainty.
Hydrologists have concentrated their effort on
development of more rigorous and efficient scheme
to assess the suitability of model structure forrepresenting the natural system and identifying
model structural inadequacy3),6),8). However, theirresearch is limited to improve a classical calibration
strategy, single-objective optimization algorithm,
coupled with their own conceptual model (e.g.,
SAC-SMA model). Hence, a method to identify themodel structural stability, not only limited to
lumped models but also applicable to various typeof distributed models is required.
As reported in previous studies3),5),6),8), the resultof variation of optimal parameter combination
calibrated by a single-objective optimization method
can be employed as one of the well-founded
indicators to account for model structural stability.
Accordingly, we think that a more reliable model
structure leads to the identical optimal parameter set
without regard of any objective functions selectedsubjectively. Moreover, such model structure
maintains high degree of accuracy for simulatedhydrographs when applying parameter set forvarious type and magnitude of floods in the same
study site. It means that a structurally-stable model
has high parameter transferability from event toevent. As a result, model structural stability can be
estimated as a degree of capability which enables toreduce the influence of objective functions andflood events on model parameter sets.
From this point of view, this study is conducted
to investigate answers to the following questions: 1)
What kinds of models are stable in terms of model
structure for description of rainfall-runoff
processes? 2) How can modelers identify model
structural stability and suitability? In section 2, aframework to assess the model structural stability is
proposed and different kinds of two models areevaluated through the framework. In section 3, thecomparison results with respect to the assessment ofmodel structural stability are discussed andconclusions are presented in section 4.

2. METHODOLOGY FOR MODELSTURCTURE ANALYSIS
Our purpose of this study is to establish aframework for how to assess the model structural
stability. This work is summarized by two mainprocedures. The first step is an identification ofmodel stability according to selection of objectivefunctions. The second procedure is an assessment ofmodel structural stability through the analysis ofparameter transferability. Figure 1 illustrates theschematic process of the framework for assessmentof model structural stability.
This assessment procedure is based on thefollowing ideas:

1) If hydrologic observed data (e.g., rainfall, streamflow) used for model calibration are noterroneous, the calibrated parameter set canreflect the structure of hydrologic model.
2) An ideally-structured model can be regarded as astable model which has the identical bestparameter set regardless of objective functionsand various flood events.
Therefore, analysis of the variation of single
optimal parameter sets calibrated by the single-objective global optimization method for various
flood events can be used as an indicator of thedegree of structural stability.
The Shuffled Complex Evolution (SCE-UA)
algorithm with three different objective functions(SLS, HMLE, MIA) is used to calibrate aconceptual lumped model (Storage Function
Method, SFM) and a physically based distributedmodel (Cell Distributed Runoff Model Version 3,
CDRMV3, http://fmd.dpri.kyto-u.ac.jp/~flood/produ
ct/cellModel.html)9). Five historical flood events at
Kamishiiba catchment (210㎢) located in the
Kyushu area are used to compare model structural
stability for verification of our framework. Appliedhydrologic models, optimization method andobjective functions are introduced in following
sub-sections.
(1) Hydrologic Modelsa) Conceptual lumped model, Storage Function
Method (SFM)
This model is a lumped model with the reflectionof nonlinear characteristics of hydrologic response
behavior. SFM is used for the rainfall-runoff
simulation in a small watershed usually less than
five hundred square kilometers. The form of SFM isexpressed as:
dS= re (t − Tl ) − q,dt⎧⎪ f × r ,re = ⎨⎪⎩ r ,

S = kq p∑r ≤ Rif ∑ r > R
if

SA

(1)
(2)

SA

where S = water storage; re = effective rainfall
intensity; r = rainfall intensity; q = runoff; t = time;k = storage coefficient; p = coefficient of
nonlinearity; f = primary runoff ratio; Tl = lag time;and RSA = cumulative observed rainfall from the
beginning of the studied storm. Four parameters (k,p, f and RSA ) are optimized in this model.
b) Physically based distributed model, CDRMV3
CDRMV3 is a physically based distributedhydrologic model developed by Kojima et al.9)
including discharge-stage relationship with saturated-unsaturated flow10). The model solves the one
dimensional kinematic wave equation with thedischarge-stage equation using the Lax-Wendroff
finite difference scheme according to the flow

- 50 -

Selection ofmodel structure

Sensitivity
analysis

Decision of parameters
to be optimized

1
θ OFEvent≈
1

Event 1
≈ θ OFnModel1OF1 ,, OF nθ

Event 1
≠ θ OFnModel2OF1 ,, OF nEvent 1
OF1≠

Objective Function
Selection (OFs)
Initial boundary ofparameters
Automatic Calibration

θ OFEventN≈
1

EventN≈ θ OFnModel1OF1 ,, OF nθ

EventN≠ θ OFnModel2OF1 ,, OF nEventN
OF1

θ Event 1
θ Event 2θ≠Model1,2 (Event1)
Model1,2 (Event2)

EventN

Model1,2 (EventN)

1
Index θEvent
Event 1 ,2
Index θEvent,
Event 1

Index

EventNθ Event 1,

Analysis of optimal
parameter set basedon various OFs

1, Index θEvent
EventN
2, Index θEvent
EventN, Index

EventNθ EventN

Analysis of parametertransferability fromevent to event
Qualitative andQuantitative
Assessment of model
structural stability

1
EventN
Event 1
EventN
Unstable Model Structure : θθ OFEvent≈ ≈ θ OF≠≠ θ OFOF1
1
nn
Fig.1 Schematic illustration of a framework to assess model structural stability; θ i j = optimal parameter set; Indexkj =measurement value for assessment of parameter transferability; i = Objective Function; j = storm event; k = optimal
parameter set of each event.

Ideal Model Structure :

(a)

∂h ∂q
+
= r (t )
∂t ∂x

(b)

Radar Rainfall
or Gage Rainfall
Grid Precipitation
Geo-data
Processing
Rainfall-Runoff
Simulation
surface flow
subsurface
flow
Hillsloperouting
Channelrouting(c)(d)(Option)Discharge at
Nodes and Edges

Fig.2 Schematic representation of CDRMV3 (a) Modular
structure of CDRMV3 (b) Distributed grid rainfall data (c)Slope and channel components extracted from DEM (d)Model structure for the hillslope soil layer and discharge-stage relationship.

direction map (see Figure 2). All geomorphologic
information are extracted from 250m based DEM
data. Channel routing is also carried out by thekinematic routing scheme as well as calculation ofslope elements reflecting contributing areas.
The model assumes that a permeable soil layer
covers the hillslope as illustrated in Figure 2(d). The
soil layer consists of a capillary layer in which
unsaturated flow occurs and a non-capillary layer inwhich saturated flow occurs. According to thismechanism, if the depth of water is higher than thesoil depth, then overland flow occurs. The
discharge-stage relationship is expressed by equation(3) corresponding to water levels (see Figure 2(d))
defined as:⎧vc d c (h / d c ) β ,0 ≤ h ≤ dc⎪
q = ⎨vc d c + va (h − d c ),dc ≤ h ≤ ds
⎪v d + v (h − d ) + α (h − d ) m , d ≤ hcss⎩ c c a(3)

(4)

Flow rate of each slope segment are calculated byabove governing equations combined with thecontinuity equation (4), where v c = k c i ; v a = k a i ;k c = k a / β ; α = i / n ; i is slope gradient, k c issaturated hydraulic conductivity of the capillary soil
layer, k a is hydraulic conductivity of the
non-capillary soil layer, n is roughness coefficient,d c is the depth of the capillary soil layer and d s issoil depth. Detailed explanations of model structureappear in Tachikawa et al.10). There are 5 parameters(n, k c , β , d c and d s ), which are assumed to havehomogeneous values spatially to be optimized in
CDRMV3.
(2) Shuffled Complex Evolution (SCE-UA)
Alogrithm
The Shuffled Complex Evolution Algorithm(SCE)11),12) is used to identify the best fitted
parameter set, which is a single-objective global
optimization method designed to handle high-parameter dimensionality encountered in calibrationof a nonlinear hydrologic simulation models. Thisevolutionary approach method has been performed
by a number of researchers on a variety of models
with outstanding positive results7) and has proved to
be an efficient, powerful method for the automatic
optimization13),14). SCE algorithm is basicallysynthesized by following three concepts: (1)combination of a simplex procedure with theconcepts of controlled random search approaches;(2) competitive evolution; and (3) complex shuffling.
The integration of these steps above mentioned
makes the SCE method effective and flexible12).
Initial state variables of both models are determinedby initial observed discharge assuming steady-state
condition.

- 51 -

(3) Applied Objective Functions
The aim of computer-based automatic calibration
is to find the values of the model parameters thatminimize or maximize the numerical value of theobjective functions15). In general, the mostcommonly utilized objective functions inhydrological modeling are variations of the Simple
Least Squares (SLS) function defined as:
N

SLS =

∑ (q − q (θ ))
obs
t

(5)

2

t

t =1

where qtobs is observed stream flow value at time t;
qt (θ ) is model simulated stream flow value at time tusing parameter set θ ; N is the number of flow
values available. SLS has a feature that large
discharge is emphasized due to squared errors while
low flows are neglected16), thus the parameter set
fitting around peak discharge value is likely toobtain.
Krause et al.17) proposed the Modified Index ofAgreement (MIA)18) to reduce the influence of thesquared term during high flows as putting an weighton flow values. This objective function is calculatedas:⎛ qtobs − qt (θ ) ⎞
⎜⎜
⎟⎟
qtobs
t =1 ⎝⎠n∑

MIA = 1 −n

⎛ qt (θ ) − q

∑ ⎜⎜
t =1

mean
t

q

⎝

+q

obs
t

2

(6)

−q

mean
t

mean
t

⎞
⎟
⎟
⎠

2

where qtmean is mean value of observed time series.
Sorooshian and Dracup19) proposed a different
objective function to consider entire behavior ofhydrograph,
theHeteroscedastic
Maximum
Likelihood Estimator (HMLE), which enables toestimate the most likely weights through the use of
the maximum estimation theory. This new measurecan eliminate some of the subjectivity involved in
the selection of transformation and/or a weighting
scheme by handling heteroscedastic error, so that ityields a more balanced performance over the entire
hydrograph3),5). It is calculated as:
HMLE =
min
θ ,λ

1 N
wt ε tN t =1

∑

N

∏w

t

(7)

t =1

where ε t = q tobs − qt (θ ) is the model residual at time t;
wt is the weight assigned to time t, computed as
wt = f t 2 ( λ −1) ; f t = q ttrue is the expected true flow attime t; λ is the transformation parameter which
stabilizes the variance.
In this study, above mentioned three objectivefunctions are used for the calibration trials and theanalysis of model structural stability.

illustrated in Figure 3. From Figure 3, we notice that:
1) In SFM cases, the simulated hydrographs basedon the parameters calibrated by SLS are close to
the observed ones over all events while otherparameters optimized by HMLE, MIA lead toless magnitude than the measured stream flow inlarge flood, Event 4 and 5 (peak flow > 500㎥/s).
The results show that the optimized parameter set
is dependent on objective functions. On the other
hand, the computed hydrographs in small flood,
Event 3 (peak flow < 500㎥/s) have a good
“goodness-of-fit” measurement value regardlessof objective functions.
2) In CDRMV3 cases, there is no influence ofobjective functions on model performance. In
other words, all predicted hydrographs shown in
Figure 3(b) are close to observed discharge forany objective functions. However, constantparameter set is not observed for small flood
events (Event 2, 3) while approximate values ofparameter sets are obtained for large flood events(Event 1, 4, 5), i.e., θ OF1 ≈ … ≈ θ OFn .
This result implies that the lumped model used inthis study is structurally unstable in terms ofdependency of objective function, so that we need tochange the model parameter set according to themodeling purpose. On the other hand, the problem ofsubjectivity related with selection of objectivefunctions for automatic calibration can be ignored
for the distributed hydrological modeling used here.
(2) The Assessment of Parameter Transferability
from Event to Event
From event to event, transferability of theidentified parameter set also can be guideline toassess model stability. If optimal parameter setsobtained from various flood events are occupied in asimilar location on feasible parameter space, i.e.,θ Event1 ≈ … ≈ θ EventN , undoubtedly, each optimal
parameter set could be applicable for different flood
events and model performances would be good. We
can regard that such model structure has highparameter transferability. Each model performancewith transferred parameter sets is evaluated by Peak
Discharge Ratio (PDR) and Nash-Sutcliffe (NS)statistics of the residuals as guideline indexes formeasurement of parameter transferability, definedas:(8)
PDR = Peaksim / Peakobs
N∑ (q − q (θ ))
obs
t

2

t

NS = 1 − t =N1∑ (q − q

3. RESULTS AND DISCUSSIONS

obst(1) The Influence of Objective Functions on
Model Performance
The plots of comparisons between the simulated
and the observed hydrographs according to the threeobjective functions (SLS, MIA and HMLE) are

mean
t

)

(9)

2

t =1

where Peaksim is the simulated peak discharge,
Peakobs is the observe peak discharge. PDR measurestendency of the simulated peak discharge to be larger
or smaller than the observed peak discharge. NS
measures a relative magnitude of the residual- 52 -

(a) SFM

Event3

Event4

Event5

Event4

Event5(b) CDRMV3

Event3

Fig.3 Comparison between the simulated and the observed hydrographs according to three objective functions;(a) SFM cases, (b) CDRMV3 cases.

variance to the variance of the observed stream Instead, the different identified parameter sets areapplicable to reproduce other flood events. It means
flows; the optimal value of both measures is 1.0.
The identified parameter set for each event is applied that the different parameter combinations can lead toto different events and their model performances are acceptable model performances with proper valuesplotted in the Figure 4. This figure illustrates the of NS or PDR. This effect is often called
quantified results of parameter transferability. Each “equifinality”2). Equifinality makes it difficult to
optimal parameter set of SFM is not applicable for identify a suitable model parameter set fordifferent flood events (i.e., low parameter distributed hydrological modeling. Therefore,transferability; NS, PDR are scattered far from 1.0) analysis of equifinality is an urgent problem to be
while those of CDRMV3 are applicable for solved for more reliable rainfall-runoff simulation.
simulations of different flood events (i.e., highparameter transferability; NS, PDR are plotted near 4. CONCLUSION
optimal value 1.0). Interestingly, in CDRMV3, theresults evaluated by optimal parameters of Event 2
In this paper, we have demonstrated a frameworkover entire cases tend to be inapplicable and for assessment of model structural stability through a
inaccurate; NS values are usually less than single-objective global optimization (SCE-UA)
approximately 0.75 and PDR values are method with three objective functions for various
underestimated or overestimated irregularly. flood events and compared two hydrologic models
Simulated results for Event 2 with the optimal (SFM, CDRMV3) as an example. The results underparameter sets of other events are also inapplicable. our framework lead to following conclusions:This result indicates that the observed data for Event 1) The simulated results of CDRMV3 are not
2 is unreliable, so that it is impossible to extractaffected by objective functions while theuseful information from this kind of uninformative
predicted hydrographs of SFM depend on thedata. This finding shows that the proposed
objective functions and magnitude of flood.
framework for assessment of model stability is also 2) The structural stability of CDRMV3 is superior topossible to detect a low quality observed data.
SFM with respect to parameter transferability.
The results of model stability assessment show 3) Our framework can be an indicator to detect athat CDRMV3 is more stable than SFM with respect
low quality observed data.
to independency of objective functions and flood
Through the analysis, we found that themagnitudes because the former model has physically equifinality problem still remains despite of a-based structure which reflects more reliable remarkable improvement of hydrologic modelinggeomorphological characteristics of watershed.
with respect to model structure. The analysis ofInteresting finding is that even though the equifinality and a development of a method tophysically-based distributed model has high distinguish a suitable model parameter set which has
11 ), theparameter transferability (e.g., I Eventan appropriate physical meaning are in progress.
≈ ≈ IθEvent
EvnetNθ Event1
identical single optimal parameter set over all storm
events is not observed (i.e., θ Event 1 ≠ ≠ θ EventN ). ACKNOWLEDGMENT: This study was supported- 53 -

(a) SFM(b) CDRMV3
Fig.4 Plots of parameter transferability from event to event; each point indicates the evaluated NS, PDR due toparameter transfer; bottom label = applied optimal parameter set of each event; upper label = target event.

by Grant-in-Aid for Scientific Research (C) 18560497(PI: Assoc. Prof. Y. Tachikawa, Kyoto Univ.).

REFERENCES

1) Sorooshian, S. and Gupta. V.K.: The analysis of structural
identifiability: Theory and application to conceptual
rainfall-runoff models, Water Resours. Res., Vol.21, No.4,
487-495, 1985.
2) Beven, K.J. and Binely, A.M.: The future of distributedmodels: model calibration and uncertainty prediction,
Hydrological process, Vol.6, 279-298, 1992.
3) Gupta, H.V., Sorooshian, S. and Yapo, P.O.: Toward
improved calibration of hydrologic models: Multiple andnoncommensurable measures of information, Water
Resours. Res., Vo.l34, No.4, 751-763, 1998.
4) Kavetski, D., Kuczera, G. and Franks, S.W.: Bayesiananalysis of input uncertainty in hydrological modeling: 1.
Theory, Water Resours. Res., Vol. 42, W03407, 2006.
5) Yapo, P.O., Gupta, H.V. and Sorrooshian, S.: Automatic
calibration of conceptual rainfall-runoff models: Sensitivity
to calibration data, J. Hydol., Vol.181, 23-48, 1996.
6) Boyle, D.P., Gupta, H.V. and Sorooshian, S.: Toward
improved calibration of hydrologic models: Combining thestrengths of manual and automatic methods, Water Resours.
Res., Vol.36, No.12, 3663-3674, 2000.
7) Gupta, H.V., Sorroshian, S and Yapo, P.O.: Status ofautomatic calibration for hydrologic models: comparison
with multilevel expert calibration, Journal of Hydrologic
Engineering, ASCE, Vol.4, No.2, 135-143, 1999.
8) Yapo, P.O., Gupta, H.V. and Sorrooshian, S.:
Multi-objective global optimization for hydrologic models,J. Hydol., Vol.204, 83-97, 1998.
9) Kojima, T. and Takara, K.: A grid-cell based distributed
flood runoff model and its performance, Weather radar
information and distributed hydrological modeling, IAHS
Publ. No.282, 234-240, 2003.
10) Tachikawa, Y., Nagatani, G. and Takara, K.: Developmentof stage-discharge relationship equation incorporating
saturated – unsaturated flow mechanism, Annual Journal of
Hydraulic Engineering, JSCE, Vol.48, 7-12, 2004.
11) Duan, Q., Sorooshian, S. and Gupta, V.K.: Effective andefficient global optimization for conceptual rainfall-runoff
models, Water Resours. Res., Vol.28, No.4, 1015-1031,
1992.
12) Duan, Q., Sorooshian, S. and Gupta, V.K.: Optimal use of
the SCE-UA global optimization method for calibratingwatershed models, J. Hydrol., Vol.158, 265-284, 1994.
13) Gan, T.Y. and Biftu, G.F.: Automatic calibration ofconceptual rainfall-runoff models: Optimization algorithms,
catchment conditions, and model structure, Water Resour.
Res., Vol.32, No.12, 3513-3524, 1996.
14) Yu, P.S., Yang, T.C. and Chen, S.J.: Comparison of
uncertainty analysis methods for a distributed rainfall-runoff
model, J. Hydrol., Vol.244, 43-59, 2001.
15) Sorooshian, S. and Gupta, V.K.: Model calibration,
Computer Models of Watershed Hydrology (ed. Singh,
V.P.), Water Resours. Publications, Colorado, 23-68, 1995.
16) Legates, D.R. and McCabe Jr., G.J.: Evaluating the use of“goodness-of-feet”
measuresinhydrologic
andhydroclimatic model validation, Water Resour. Res., Vol.35,
No.1, 233-241, 1999.
17) Krause, P. and Flügel, W.A.: Integrated research on thehydrological process dynamics from the Wilde Gera
catchment in Germany; Headwater Control Ⅵ: Hydrology,
Ecology and Water Resources in Headwaters, IAHS
Conference, Bergen, 2005.
18) Willmott, C.J.: On the evaluation of model performance inthe physical geography, in Spatial Statistics and Models,edited by Gaile, G.L. and Willmott, C.J., 443-460, D.
Reidel, Norwell, Mass., 1984.
19) Sorooshian, S. and Dracup, J.A.: Stochastic parameter
estimation procedures for hydrologic rainfall-runoff models:Corrolated and heteroscedastic error cases, Water Resours.
Res., Vol.16, No.2, 430-442, 1980.
(Received September 30, 2006)- 54 -