Model Parameter Estimation Experiment (MOPEX): An overview of science strategy and major results from the second and third workshops

The Model Parameter Estimation Experiment (MOPEX) is an international project aimed at developing enhanced techniques for the a priori estimation of parameters in hydrologic models and in land surface parameterization schemes of atmospheric models. The MOPEX science strategy involves three major steps: data preparation, a priori parameter estimation methodology development, and demonstration of parameter transferability. A comprehensive MOPEX database has been developed that contains historical hydrometeorological data and land surface characteristics data for many hydrologic basins in the United States (US) and in other countries. This database is being continuously expanded to include more basins in all parts of the world. A number of international MOPEX workshops have been convened to bring together interested hydrologists and land surface modelers from all over world to exchange knowledge and experience in developing a priori parameter estimation techniques. This paper describes the results from the second and third MOPEX workshops. The specific objective of these workshops is to examine the state of a priori parameter estimation techniques and how they can be potentially improved with observations from well-monitored hydrologic basins. Participants of the second and third MOPEX workshops were provided with data from 12 basins in the southeastern US and were asked to carry out a series of numerical experiments using a priori parameters as well as calibrated parameters developed for their respective hydrologic models. Different modeling groups carried out all the required experiments independently using eight different models, and the results from these models have been assembled for analysis in this paper. This paper presents an overview of the MOPEX experiment and its design. The main experimental results are analyzed. A key finding is that existing a priori parameter estimation procedures are problematic and need improvement. Significant improvement of these procedures may be achieved through model calibration of well-monitored hydrologic basins. This paper concludes with a discussion of the lessons learned, and points out further work and future strategy.

section_content: A critical step in applying a hydrologic model to a watershed or a land surface parameterization scheme (LSPS) of an atmospheric model to a specific grid element is to estimate the coefficients or constants in the model or LSPS known as parameters. These parameters are inherent in all models. While certain parameters may take on universally accepted values (e. g. gas constant, acceleration of gravity), the values of many parameters are not universally constant and may be highly uncertain. In general, parameters vary spatially so they are unique to each watershed or to a grid point, and some may even vary seasonally. Moreover, some parameters may be space-time scale dependent (Koren et al., 1999; Finnerty et al., 1997). The question of how to estimate model parameters has been receiving increasing attention from the hydrology and land surface modeling community (Franks and Beven, 1997; Bastidas et al., 1999; Gupta et al., 1999; Duan et al., 2001; Duan et al., 2003; Jackson et al., 2003; Wagener et al., 2003). 

A common approach within the hydrologic modeling community to parameter estimation is to calibrate hydrologic models to historical observations by tuning model parameters. A plethora of model calibration techniques have been reported in the literature. For a detailed review of model calibration techniques, readers are referred to Duan et al. (2003) ; Duan (2003). To conduct model calibration, a sufficient amount of historical hydrologic data is typically required. Hydrologists have the advantage of working with watersheds, many of which are well monitored with climate stations and stream gauges. 

For ungauged basins and for LSPS applications, it is difficult to obtain adequate data that are needed for model calibration. A further complication is that LSPSs are typically applied to large spatial scales and involve many grid elements. To estimate model parameters in these cases, it is necessary to assign model parameter values a priori. 

A priori parameter estimation procedures are available for many hydrologic models and LSPSs. But these procedures have not been fully validated through rigorous testing using retrospective hydrometeorological data and corresponding land surface characteristics data. This is partly because the necessary database for such testing has not been available until recently. Moreover, there is a gap in our understanding of the links between model parameters and the land surface characteristics. Generally, available information about soils (e. g. texture) and vegetation (e. g. type or vegetation index) only indirectly relates to model parameters such as hydraulic properties of soils and rooting depths of vegetation. Some models which are built using a topdown approach are by nature empirical, and no direct link has yet been established between measurable watershed characteristics and model parameters. Also it is not clear how heterogeneity associated with spatial land surface characteristics data affects those characteristics at the scale of a basin or a grid cell. Consequently, there is a considerable degree of uncertainty associated with the parameters given by existing a priori procedures. 

The Project for Intercomparison of Land-surface Parameterization Schemes (PILPS) has revealed widely discrepant simulation results by different LSPSs (see Chen et al., 1997; Wood et al., 1998; Pitman et al., 1999; Schlosser et al., 2000; Slater et al., 2001). Interestingly, the LSPSs included in the PILPS experiments were driven by the same meteorological forcing data and were required to use the same values for commonly named parameters (such as soil hydraulic properties and vegetation phenology parameters). The large scattering of model results can be partially explained by the uncertainty in the values of the parameters used in each scheme. 

The improper choice of model parameters results in poor model performance (Liston et al., 1994; Duan et al., 1995). It is necessary to develop enhanced a priori parameter estimation methodologies for hydrologic models and LSPSs. Toward this goal, a project known as the Model Parameter Estimation Experiment (MOPEX) was initiated in 1996. The MOPEX project has been an international collaborative endeavor, with the involvement of international scientists and hydrologic data assembled from different countries. MOPEX has the endorsement of several international organizations and projects including: the World Meteorological Organization (WMO) Commission on Hydrology, International Association of Hydrological Sciences (IAHS) Prediction for Ungauged Basins (PUB) Initiative (Sivapalan, 2003) and the Global Energy and Water Cycle Experiment (GEWEX). The Office of Global Programs in the National Oceanic and Atmospheric Administration (NOAA) and funding agencies in different countries have all provided financial support for scientists to participate in MOPEX activities. A series of international workshops on MOPEX have been convened over the last few years. The MOPEX workshops were designed to bring together interested international hydrologists and land surface modelers to share experience in estimation of hydrologic model parameters. Each workshop has a special focus, either in terms of hydroclimatology (i. e. humid or semi-arid) or in terms of special applications (i. e. flood forecasting). 

This paper presents an overview of the results from the second and third MOPEX workshops. For these workshops, a set of numerical experiments was constructed. The MOPEX participants were given data for 12 basins located in the southeastern quadrant of the US. Numerical test results from different modeling groups were assembled for the workshops. The paper is organized as follows. First the MOPEX rationale and science strategy are presented. Then a discussion of the objectives and numerical experiment design is given. The data sets assembled for the workshop are described, and a comprehensive analysis of the results is conducted to understand the differences in the results from the different models. Finally, further work and future strategy are discussed.
section_name: Introduction

section_content: The MOPEX science strategy involves three major steps (Fig. 1 ). The first step is to develop the necessary data sets. The second step is to use these data to develop a priori parameter estimation methodology. The third step is to demonstrate that new a priori techniques produce better model results than existing a priori techniques for basins which were not used to develop the new a priori techniques. 

Step two is accomplished using a three-path strategy illustrated in Fig. 1. The first path makes reference model runs with parameters estimated using existing a priori parameter estimation procedures. The second path makes model runs using calibrated values of selected model parameters. Within the second path, the calibrated parameters are analyzed to improve relations between model parameters and basin characteristics (i. e. climate, soils, vegetation and topographic features). These improved relations are used to estimate new a priori parameters which are then used to make model runs in the third path. The success of step two is measured by how much improvement in model performance is achieved in the third path compared with results from the reference runs in the first path. 

The MOPEX Project has assembled hydrometeorological data as well as land surface characteristics data that are needed to implement its parameter estimation strategy. Data from many basins in the US and other parts of the world are being assembled which cover a wide variety of climates. In Sections 3. 2 and 3. 3, the MOPEX data requirements and the data set used for the second and third MOPEX workshops are described in details. 

A key in implementing the MOPEX strategy is to develop systematic procedures for calibration of selected model parameters and to apply these procedures to a large number of basins in different hydroclimatic regimes. Then, empirical relations will be sought between the parameters and various characteristics of soils, vegetation and climate. Much progress has been made in the area of model calibration (Duan et al., 2003). Duan et al. (1992 Duan et al. (, 1994) ) developed a robust optimization method known as Shuffled Complex Evolution (SCE-UA) method for optimal estimation of model parameters. Yapo et al. (1998) ; Gupta et al. (1999) have extended Duan's approach in the context of multi-objective theory. 

Recently, there is a surge of interest toward producing multiple sets of model parameters, as a means to account for uncertainty related to model structure, calibration data and model parameters. These methods use Monte Carlo sampling techniques to produce a set of solutions, all of which are regarded as 'equifinal' (i. e. all of the solutions are equally valid). Examples include the Generalized Likelihood Uncertainty Estimation (GLUE) by Beven and Binley (1992), the Markov Chain Monte Carlo (MCMC) Metropolis scheme by Kuczera and Parent (1998) and the Shuffled Complex Evolution Method Metropolis (SCEM) scheme by Vrugt et al. (2003). For more on the state-of-the-art on model calibration methods, readers are referred to Duan et al. (2003). 

Numerous studies have been directed at developing improved a priori parameter estimation procedures for hydrologic models and LSPSs. Earlier examples of a priori parameter estimation procedures are from the field of soil physics, in which soil hydraulic properties (as appear in many hydrologic models and LSPSs) are related to soil texture classes in a tabular format (see e. g. Clapp and Hornberger, 1978; Cosby et al., 1984; Rawls et al., 1991; Carsel and Parrish, 1988). Many land surface modelers have directly adopted the a priori parameter estimation schemes developed by soil physicists to assign values to parameters in LSPSs (Dickinson et al., 1986; Sellers et al., 1986). Duan et al. (2001) pointed out that this practice is questionable because the tabular relations between soil hydraulic properties and soil texture classes are based on analysis of soil samples tested at laboratories. These relations may not hold up in the real world, especially over grid elements of several hundred to several thousand square kilometres. For typical hydrologic models and LSPSs, it is often the case that the relations between many of the model parameters and land surface characteristics are not obvious. One approach to solving this dilemma is to develop a priori relations between land surface characteristics and model parameters for basins where the model is appropriately calibrated (Abdulla et al., 1996; Duan et al., 1996; Merz and Bloschl, 2004; Lamb and Kay, 2004; McIntyre et al., 2004; Wagener et al., 2004). With the advent of Geographic Information Systems (GIS), many more a priori parameter estimation procedures have appeared. These schemes are model specific and are still being evolved. A number of these GIS-based schemes are being tested in the second and third MOPEX workshops and are part of the analysis presented in this paper. 

3. Design and database of the second and third MOPEX workshops
section_name: Model Parameter Estimation Experiment strategy

section_content: The second and third MOPEX workshops focused on the second step of the MOPEX strategy: data preparation and development of parameter estimation procedures. The emphasis of the workshops was on validating existing a priori procedures and on evaluating potential improvement from model calibration. Because all hydrologic models are formulated differently, parameter estimation procedures are model-specific. A challenge facing hydrologic modelers is how the knowledge gained from one model can be transferred to another model. This is also the principal reason to convene these MOPEX workshops. A specific objective of these workshops is to examine the state of a priori parameter estimation techniques and how they can be potentially improved with observations from well-monitored hydrologic basins. Particularly, we seek to answer the following questions:

(1) How do we define the relations between model parameters and basin characteristics? (2) How can model calibration be used to refine the a priori parameters? (3) How do we evaluate the uncertainty due to model structure, calibration data and model parameters?
section_name: Workshop objectives

section_content: To answer the above questions, a set of numerical experiments was designed. Data for 12 basins located in the southeastern quadrant of the United States were prepared. The data sets include hydrometeorological data as well as basin land surface characteristics data. More discussion on these data sets is given in Section 3. 3. The data were distributed to MOPEX participants via ftp and CD-ROMs. The MOPEX participants were asked to make two sets of model runs. In the first set of model runs, the participating modelers were asked to run their respective models on all 12 basins using existing a priori parameters developed for their models. The second set of model runs involved model calibration for pre-selected common data periods. After model calibration, the participants were asked to run their models using calibrated parameters for the calibration and verification data periods. All results were collected for analysis by the MOPEX workshop organizers.
section_name: Design of MOPEX numerical experiment

section_name: Description of the data set

section_content: The initial step in the MOPEX strategy is to assemble a large number of high quality data sets for a wide range of Intermediate Scale Area (ISA) river basins (500-10,000 km 2 ) throughout the world. There are strict requirements for MOPEX data sets in terms of data type, quantity and quality. The two basic data types gathered for MOPEX basins are hydrometeorological data and land surface characteristics data. The MOPEX basins should be unregulated basins and cover a variety of climate regimes. The basic hydrometeorological data required for MOPEX include daily precipitation, daily maximum and minimum temperature, daily streamflow data and climatic potential evaporation data. More desirable hydrometeorological data include hourly surface meteorological data, including precipitation, incoming long-wave and short-wave radiation, air temperature, air humidity, atmospheric pressure, and wind speed, etc. The quality of precipitation data is critically important to parameter estimation. MOPEX has established a minimum density requirement for raingauges based on basin size (Schaake et al., 2000). To ensure various hydrologic events are represented in the hydrometeorological data, MOPEX requires that the data length exceed 10 years. A desirable data length is 20 years or more. 

The basic land surface characteristics data include basin boundary, soil texture and vegetation type data. More desirable land surface data sets include high resolution (1 km or finer) Digital Elevation Model (DEM) data, seasonal land cover/land use data such as Normalized Deviation of Vegetation Index (NDVI), greenness fraction, snow cover and soil moisture climatology, etc.
section_name: MOPEX data requirements

section_content: For the second and third international MOPEX workshops, hydrometeorological data as well as basin land surface characteristics data for 12 basins in the Southeastern quadrant of the United States were assembled. Fig. 2 shows the location of the 12 basins. These basins represent a wide range of different climate, as indicated by the ratios of annual precipitation (P) and potential evapotranspiration (PE) in Fig. 3. A high value for P/PE indicates wet climate and a low value indicates dry climate (Dooge, 1997). The climatic seasonal precipitation and streamflow distributions are shown in Fig. 4. 

The hydrometeorological data sets prepared for the workshops included hourly mean areal precipitation, daily streamflow, and climatic daily potential evapotranspiration. The hourly precipitation data sets were developed by the NWS Hydrology Laboratory (HL) based on hourly and daily raingauge data from the National Climate Data Center (NCDC). The daily streamflow data were obtained from the US Geological Survey (USGS). The climatic potential evaporation data was derived from the NOAA Freewater Evaporation Atlas (Farnsworth and Peck, 1982). Also included are basin averaged hourly meteorolological forcing data, including precipitation, air temperature, wind speed, surface pressure, short-wave and long-wave radiation and specific humidity. All meteorological forcing data except precipitation were processed from the 1/88 meteorological forcing data for the conterminous US developed by the University of Washington (UW) (Maurer et al., 2001). The UW hourly meteorological data set is derived from NCDC daily precipitation, daily minimum and maximum temperature and wind speed data obtained from National Center for Environmental Predictions/National Center for Atmospheric Research (NCEP/NCAR) Global Reanalysis data (Kistler et al., 2001). The historical data from different sources span over different data periods. For this study, a common period, 1960-1998, is chosen so data from all sources are available. 

The land surface characteristics data sets assembled for this study include 1 km soil type data from the STATSGO data set (Miller and White, 1999), the 1 km vegetation type, and 5-min greenness fraction data (Loveland et al., 2000; Hansen and Reed, 2000; Gutman and Ignatov, 1998). Table 1 lists the major land surface properties of each basin including the area, elevation, and dominant soil and vegetation type. Other land surface data made available to MOPEX participants include basin boundary, elevation, monthly surface albedo and roughness length. Basin climatologic data such as monthly long-term average precipitation, streamflow and potential evapotranspiration have also been made available.
section_name: MOPEX data for the second and third MOPEX workshops

section_content: Eight hydrologic models and LSPSs have completed all of the required numerical experiments as described in Section 3. 2. A few additional groups submitted incomplete numerical experiment results which have not been included in the analysis. Table 2 lists the eight participating models. Of the eights models, the first four models (SWB, SAC, GR4J and PRMS) are watershed rainfall-runoff models, while the last three (ISBA, SWAP, and Noah models) are LSPSs. The VIC model has been used both as a watershed model and a LSPS in atmospheric models. The analysis presented below is based on the comparison of the simulated streamflow from the eight models and the corresponding observations at daily or monthly time steps. It should be emphasized that the purpose of the intercomparison study is not intended to rank the models as being 'better' or 'worse' with respect to each other. Instead, the intercomparison study was conducted to understand the differences between approaches and use this knowledge to develop new a priori parameter estimation procedures. For this reason, this paper lists all participating models in Table 2, but the analysis does not refer to individual model names directly in all subsequent text or figures.
section_name: Results and analysis

section_content: The purpose of simulations using existing a priori parameters is to establish benchmarks for the current a priori parameter estimation procedures used by the participating models. Any new a priori parameter procedures developed in the future for these models should at least perform better than the benchmarks. It should be noted that among the eight models under study, some models already have established a priori parameter estimation procedures, while others have no such systematic procedures. This discrepancy is reflected in the results discussed below. Fig. 5 displays the comparison of the simulated average annual streamflow totals from the a priori runs and the corresponding observed values. The spread of simulated streamflow annual totals is quite large between the models. None of the models were able to generate simulated streamflow values that match the observed values for all basins. The maximum overbias exceeds 400 mm/year and the maximum underbias is about 340 mm/year. 

The Nash-Sutcliffe (NS) efficiency is a commonly used goodness-of-fit measure between the simulated time series and observed time series. It is expressed as:

where Q i * and Q i are the simulated and observed values at time i, and n is the number of data points. Q is the average of observed values. A value of 1 indicates perfect fit between Q* i and Q i, while a value of !0 implies that simulated value is (on average) a poorer predictor than the long-term mean of the observations. Fig. 6 shows the NS efficiency of the daily streamflow simulations by the eight models. The NS values have been sorted from the lowest to the highest for each model. Fig. 7a and b shows the means and the standard deviations of the NS values, respectively. These figures reveal some interesting findings. Even though some models have some of the higher ranked NS values for most basins, they do not rank high for all basins. On the other hand, some models are shown to be consistent in all basins. This consistency is reflected in the low standard deviations for those models. (e. g. Models C,E,F and H). It should also be noted that models perform worse than long-term average for some basins, indicating a definitive need to improve a priori parameter estimates under those circumstances. 

Figs. 8 and 9 show the same information for all models as in Figs. 6 and 7, but are evaluated on a monthly time step. The NS statistics for watershed models on a monthly time step generally show an improvement over those on a daily time step, while three LSPSs display a degraded average performance. For some models, the model simulations produce worse statistics than the long-term average of observations for some basins. Model G has good NS statistics for most basins compared to other models. But the large negative NS statistics for two basins have dragged down the average NS statistic to 0. The fact that a model does well for most basins, but poorly for only a few, tells us that the modeler should probably focus attention on the basins with poor results when looking for enhanced a priori parameter estimates.
section_name: Simulation results using existing a priori parameters

section_content: There are several objectives in this exercise. First, we hope to quantify the potential improvement in model performance when the models are calibrated using observations, as compared to those using a priori parameters. Second, we want to make sure that there is consistency in streamflow simulations between calibration and validation data periods when the calibrated parameters are used. The ultimate objective of this exercise is to use the calibrated parameters to establish new a priori parameter estimates. 

All model groups were asked to calibrate and validate their models for all 12 basins using historical hydrologic data. Originally, a split sample approach was to be used. Years 1980 Years -1990 were to be used for calibration, while the first 19 years (1960) (1961) (1962) (1963) (1964) (1965) (1966) (1967) (1968) (1969) (1970) (1971) (1972) (1973) (1974) (1975) (1976) (1977) (1978) (1979) were to be used for validation. Because different groups used different 19-year periods for calibration, it is not possible to make a direct comparison of all eight models using the split-sample approach. However, since the differences in the calibrated model performance between the different 19-year periods were much smaller than the differences in model performance between the a priori and calibrated runs, it seemed the best way to achieve the study objectives was to use the entire 1960-1998 period to evaluate model results for both the a priori and calibrated runs. Fig. 10 shows the simulated average annual streamflow totals using calibrated parameters versus the observed average annual streamflow totals. Compared to Fig. 5, the scatter around the diagonal line is much smaller, indicating the agreement between observed and simulated streamflow is better when using calibrated parameters versus a priori parameters. Fig. 11 displays the sorted NS values for all models for the calibration period 1980-1998, while Fig. 12 shows the average NS values and standard deviations at the daily time step. All of these figures confirm that the NS values have been improved compared to the results. All NS values are now positive when calibrated parameters are used. 

Figs. 13 and 14 show the information as in Figs. 10 and 11 for all models, but the NS values are computed using monthly aggregated values.
section_name: Simulation results using calibrated parameters

section_content: Fig. 15a and b compares the daily and monthly NS values, respectively, for the entire data period where a priori and calibrated parameters are used. Both figures show that almost all of the points are on the left side of the diagonal line, indicating improvement resulting from the calibration exercise. The improvement is more apparent when examining monthly NS statistics. There are certain cases when the NS values from the calibration runs do not improve over those from the a priori runs. This is due to the fact that different modeling groups performed model calibration using different approaches. Particularly for one model (Model F), the modeler did not calibrate its model parameters to fit observed streamflow data during calibration. For another model (Model G), the modeler manually calibrated only one parameter (soil hydraulic conductivity at saturation) to get a good agreement between observed and simulated annual streamflow.
section_name: Calibration versus a priori results

section_content: It is recognized that each of the models participating in this study is an imperfect representation of the hydrologic process that occur in the real world. It seemed interesting to ask how much total information about each basin is contained in the set of all models. Accordingly, the simulated streamflow time series from all eight models are used together as independent variables to construct a multiple regression model to predict the observed streamflow. The joint correlation coefficient from this regression analysis is a measure of the total information content of all of the models, jointly. By comparing the joint correlation coefficient from the regression analysis with the simple correlation coefficients for each model we can get an idea not only of the total information content but also which models contribute most of the information. Fig. 16 shows the scatter plot of the joint correlation coefficients and individual correlation coefficients at the daily time step. All of the points lie to the left of the diagonal line, which delineates the limiting value of the regression coefficient for any individual model. The relative position of points along the abscissa indicates the contribution of individual models to the joint correlation. In Fig. 16a, it is clear that Model B contributes most to the joint correlation because most of the points associated with this model are closest to the diagonal line. In Fig. 16b, a number of models make significant contribution to the joint correlation. 

These figures point to the potential that the multimodel approach (e. g. Georgakakos et al., 2004 ) is a plausible approach to obtain improved prediction.
section_name: Joint correlation between simulated streamflow from multiple models and observations

section_content: A summary and analysis of the numerical experiment results of eight different models submitted to the second and third MOPEX workshops was presented. A number of lessons can be drawn from these results. First, the results confirm earlier statements that the existing a priori parameter estimation procedures are problematic and need improvement. 

Second, calibration results clearly demonstrate the huge potential for improvement in a priori parameter estimation. Third, different models seem to represent hydrologic processes differently and all of them are imperfect. This suggests it may be possible to improve some of the models. It also suggests that improved prediction may be possible through an ensemble of different models or, possibly, an ensemble of a given model using different parameter sets. 

Much research needs to be done to understand how model parameters are related to basin characteristics, especially considering that modelers are not sure that the presently available observable characteristics (mostly land surface characteristics) are the most relevant descriptors of the factors that control basin hydrological behavior. Further, how to use the calibrated results for improving a priori parameters is still not clear and this issue also needs addressing. Different modeling groups can learn from each other because many model parameters have similar physical interpretations and should have similarity in spacetime patterns. 

One issue that has not been examined in the workshops is the parameter transferability issue. This issue is very important for Predictions for Ungaged Basins (PUBs) and for application in land surface parameterization schemes. To study the transferability issue, data from a wide range of climatic conditions should be used. The MOPEX project is continuing to assemble data from many different countries. These data should be used to test enhanced a priori parameters. 

One of the driving forces behind the progress in parameter estimation research is the increasing array of data sources, including satellite and other advanced observational technologies. With the wealth of new data sources, it is important to investigate the ways to maximize the use of high-resolution spatio-temporal information. Meanwhile, the issue of uncertainty attributed to data errors should be addressed. 

Any improvement in parameter estimation procedures must be tied to how we represent the hydrological processes. As our knowledge of these processes advance and as the availability of distributed forcing inputs increase, improved hydrologic models are likely to emerge. This will bring new challenges in terms of parameter estimation and model calibration. Much of the work cited above has already been reported, or is in the process of being reported, by MOPEX project participants. At this moment, there is no consensus about what type of parameter estimation approach is likely to lead to success. One of the advantages of MOPEX is that we keep it open for all different approaches. With a true collaborative spirit by international scientists, enhanced a priori parameter estimation should be available. This in turn should result in improved skill in hydrologic predictions.
section_name: Lessons, conclusions and future directions

section_name: Acknowledgements

section_content: The authors would like to express our appreciation to Drs Neil McIntyre of London Imperial College and Doug Boyle of Desert Research Institute for their thoughtful review comments that have resulted in tremendous improvement to this paper. We would also like to acknowledge the support by our respective institutions.
section_name: Acknowledgements