# Making Progress in Integrated Modelling and Environmental Decision Support

Anthony J. Jakeman, Sondoss El Sawah, Joseph Guillaume, Suzanne A. Pierce

► To cite this version:

Anthony J. Jakeman, Sondoss El Sawah, Joseph Guillaume, Suzanne A. Pierce. Making Progress in Integrated Modelling and Environmental Decision Support. 9th International Symposium on Environmental Software Systems (ISESS), Jun 2011, Brno, Czech Republic. pp.15-25, 10.1007/978-3-642-22285-6_2. hal-01569228

HAL Id: hal-01569228

https://inria.hal.science/hal-01569228

Submitted on 26 Jul 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
Making Progress in Integrated Modelling and
Environmental Decision Support

Anthony J. Jakeman¹, Sondoss El Sawah¹, Joseph H.A. Guillaume¹ and
Suzanne A. Pierce²

¹Fenner School of Environment and Society and National Centre for Groundwater Research and Training, The Australian National University, Canberra

²Jackson School of Geosciences, University of Texas, Austin
{tony.jakeman, sondoss.elsawah}@anu.edu.au

**Abstract.** Integrated modelling and environmental decision support are increasingly important as society tackles some of the most complex challenges of our generation, with impacts on future generations. When integrated modelling is successful, the results can be transformational yet the core elements for generating that success are not always clear. There is an elusive element to finding the best mix of methods, models and approaches for any given problem. This raises issues for repeatability and questions regarding how the emerging metadiscipline will converge in order to consistently achieve quality results or increased understanding of the processes that lead to success. Key challenges include the need to diagnose elements that lead to successful process, training for professional and technical competencies, and increased access to stable platforms and interchangeable models and modelling tools. This paper aims to summarize some of the key process and product related challenges of integrated modelling and environmental decision support.

**Keywords:** Integrated modelling, environmental decision support, modelling tools

# 1 Introduction

Most environmental and sustainability issues we face today are known to be wicked or messy problems. Wicked problems arise in contexts where uncertainty and conflicts are rife (Rittel and Webber [1]). To manage these complex issues, we need to take a precautionary, adaptive and evidence-based approach that seeks to balance the needs of current and future generations. To help inform decision making and adaptive planning, there are several requirements:

* Engaging relevant interest groups in participatory processes to collectively frame the issues to be addressed, share knowledge and engender trust

* Modelling activities (both qualitative and quantitative) that acquire, systematize and integrate the knowledge about: the problem under scrutiny,
data, information and perspectives in order to improve system understanding
and clarify trade-off options

* Managing the major sources of uncertainty in the decision making process by identifying, ranking, communicating and purposefully reducing the crucial components

* Creating stable and accessible information and knowledge-based systems that support efficient storage, processing and (re)-use of available data, knowledge and models

* Committing sufficient resources to the overall decision making process or, at least, using available resources efficiently be they financial, technical or facilitative.

This paper focuses on the role that integrated modelling can play in effectively and efficiently meeting decision support requirements. A worthwhile question then is: how far are we as an environmental modelling and software community along this path? To help address this question, the paper has three aims:

(1) Give a brief snapshot of the integrated modelling field and some of its key achievements and weaknesses that we can improve and build upon (Sections 2 and 3)

(2) Proffer some suggestions as to where we, as a community of modellers and software developers, would like to be in a decade or so in terms of providing major progress to support the environmental decision making process (Section 4)

(3) Discuss some of the opportunities that the modelling community can embrace to achieve such progress (Section 5).

## 2 Integrated Modelling

Undoubtedly there has been a lot of activity in the environmental modelling area since the use of computers has become routine. Indeed that activity has been growing and models are increasingly used to support environmental decisions. There is really no alternative to the use of modelling, a term and activity used here in the broadest sense. Indeed modelling is unavoidable and more necessary the messier the problem.

Integrated environmental modelling can be viewed as a “metadiscipline” which integrates knowledge and practices across multiple scientific fields (e.g. hydrology, ecology, economics, various social sciences) to improve understanding of the ecological, social and economic outcomes of management decisions. The challenge is to treat integrated modelling as a process that strives for credibility and accessibility. Ravetz [2] goes so far as arguing for evaluation of the process of integrated model development rather than the product, stating that in such circumstances “*the inherently more difficult path of testing of the process may actually be more practical.*” We would argue the need for evaluation of both process and product and recognize the mutual support lent by each objective.
## 3 Where are we now?

The development and application of integrated modelling stand on a number of pillars or building blocks which constitute both the modelling process and content development (see Figure 1). These include: (1) models, (2) modelling guidelines, (3) participatory modelling, (4) modelling paradigms, (5) methods and tools, and (6) software/hardware technologies. The following sub-sections address these pillars.

<figure><img src="image_2.png" /><figcaption>Fig. 1. Pillars in the field of integrated modelling and environmental decision support</figcaption></figure>

### 3.1 The legacy of models

First of all we now have a legacy of disciplinary models for a wide range of bio-physical and socio-economic processes that is relevant for incorporating in integrated models and/or modelling. In each of hydrology, ecology, economics, and other enabling fields there is a very rich literature proposing, applying and (to a lesser extent) evaluating models of different types for different purposes utilizing a varying quantity and quality of data. In that literature there is much that can be learnt about the values of these models as component candidates for an integrated model. In the integrated modelling literature, Oxley et al. [3] discuss the difference between domain-specific research models and integrated policy-oriented models. They identify several criteria when considering a model for including as a component in an integrated model or integrated modelling exercise, such as: purpose/use of the integrated model, adequacy of represented processes, accuracy and resolution of data. Integrating, adapting and rebuilding are three strategies for incorporating existing models into an integrated platform. Each strategy has its own advantages and disadvantages in terms of development costs, flexibility of the end-product, and the
degree to which user requirements are met. Advances in software development have
provided technologies and tools that can effectively support the technical integration
of models. This will be further discussed in Sub-section 3.6.

**3.2 Protocols and Guidelines**

Secondly, in line with the dual but mutually supportive premises of accentuating the
process and evaluating the products of modelling, some accepted protocols and
guidelines exist about how one develops models and software. Adopting good
modelling practices improves the chances for the reusability and extendibility of
models. The Position Paper [4] on the development and evaluation of environmental
models is just one example of a host of literature and recommended practice that
includes Refsgaard [5], Van der Sluijs et al. [6] and Lee [7].

A weakness to date that could easily be obliterated is the need for heavier
concentration on the evaluation of models as products, most of which being reported
remains pathetically thin. Bennett et al. [8] summarize quantitative and qualitative
methods of model evaluation, which will hopefully help place the practice on a
trajectory that clarifies the level of a model’s credibility and, as a necessary reporting
condition, explicitly characterizes its limitations.

**3.3 Participatory Modelling**

Thirdly, we now have a maturing literature on the role of modelling and participatory
processes for decision support and a rapidly increasing pool of researchers dedicated
to participatory modelling. This research line is informed by a wealth of knowledge
and practices developed in psychological and social inquiry fields including: action
research, soft systems thinking and cognitive/behavioural decision making. Becker et
al. [9] provide a good summary of where and how participation enters the various
stages of integrated modelling. Bousquet and Voinov [10] is a good introduction to
“Modelling with stakeholders” and is evidence of the strong trend towards
participatory modelling. A deficiency at present, however, seems to be that there is an
imbalance in the research between there being too much science push and not enough
stakeholder pull in general. There are two interrelated causes for this perceived gap.
First, there has been little attention paid to systematically evaluating and
demonstrating the practical impacts of the process and its products on decision
making. Second, participatory modelling remains more of an art than a science.
Whereas it is recognized that the research design highly depends on the project’s
context and purpose, there is still a lack of understanding of how/why a process works
in practice to achieve a desired outcome. Recent work has started to recognise these
drawbacks, such as Mathews et al. [11]. Perhaps it is just a matter of time until the
value of the science is more recognised and this perceived imbalance reverses.
## 3.4 Paradigms for Integrated Modelling

Fourthly, we have some powerful paradigms or families for integrated modelling, each with their own suitability to different situations. These include

* Bayesian Networks

* Agent Based Models

* System Dynamics Models

* Coupled Complex Models, and

* Hybrids of these families.

The strengths and weaknesses of these paradigms in different contexts are characterized in [12], according to modelling purpose, nature of the data at one's disposal, the breadth of issues being addressed, capability of the paradigm for handling uncertainty, and disciplinary model components to be incorporated. As applications of these paradigms mount, the literature here is becoming increasingly helpful in illustrating what sort of utility they possess.

## 3.5 Methods and Tools

Fifthly, we have a myriad of methods and tools to assist with integrated modelling and the development of decision support systems [13]. These include:

* Data acquisition and analysis tools (e.g. GIS, data mining)

* Qualitative modelling methods (e.g. conceptual models)

* Scenario development methods (e.g. narratives and fuzzy cognitive maps)

* Participatory methods to elicit and synthesize multiple sources of knowledge

* Expert elicitation methods to obtain and incorporate experts opinion into models in cases where other information sources are limited or unreliable (e.g. Delphi techniques and fuzzy methods)

* Cost-benefit analysis methods

* Multi-criteria analysis tools

* Optimisation algorithms

* Sensitivity analysis

* Uncertainty analysis

* Modelling and statistical tools more generally.

## 3.6 The Software and Hardware Technology

Sixthly, the available software technology is advancing rapidly, perhaps much faster than the science of integrated participatory modelling. Many technologies have been, and can be further, leveraged to support integrated modelling. These include:

* Code coupling technologies such as APIs (e.g. interface wrappers), Open MI interfaces, and component modelling technologies (e.g. Active X)
* Digital communication and learning technologies, such as 3-D visualization and interactive gaming technologies [14]

* Distributed and parallel computing technologies, such as cloud computing.

# 4 Where would we like to be?

## 4.1 Uncertainty Management

In keeping with the emphasis on process *and* product, a primary aim should be to identify and communicate the sources of uncertainty at all stages of the decision making process. And, as far as is necessary to meet end user needs, one should for example be clarifying trade-offs, and reducing and characterizing uncertainty, such that its effect on objectives can be described by propagating each uncertainty through all stages of that process. Importantly, the existence of residual uncertainty must be recognised and measures planned to address that uncertainty as it arises.

The stages of the decision making process include project scoping, problem framing, searching for options, analysis of candidates, deliberation, implementation and monitoring and evaluation. Reducing uncertainty in problem framing and in the model used for analysis is a key outcome for the more rigorous methods recommended in the next sections. However, it is also necessary to describe and communicate the underlying, often implicitly considered, uncertainties. Scenarios and multi-model techniques [15] can be used to capture multiple frames of a system and its drivers. There are numerous techniques for addressing uncertainty in the model structure and parameters [16], the merits of which are discussed in more detail in Section 3.3. In deliberation, implementation, and monitoring and evaluation, we must be clear about uncertainties accumulated from the rest of the process and their effect on the confidence that we can have in predicted outcomes. There is some belief that discussing the uncertainty in predictions weakens the credibility and authority of recommendations. It must however be acknowledged that uncertainty exists, whether it is explicitly discussed or not, and the risk of failure of decision support will be best minimised by consciously planning how that uncertainty is communicated. By explicitly considering the sources of uncertainty throughout the process, it becomes clear that much can be done to improve the way it is addressed; and indeed many methods already exist, for example to account for and quantify uncertainties in search through optimisation and planning data acquisition for monitoring.

Note that we do have frameworks and methods to better manage uncertainty more holisically [e.g. 17,18] but we require studies that apply these frameworks with appropriate tools so as to show the way the process can be enhanced and the confidence between outcomes predicted.
## 4.2 Problem Framing

Our observation is that in general much less time is given to framing the problem under scrutiny than is desirable. Although problem framing and structuring can be more critical than the computational techniques we use to generate solutions, there is a perceived temptation to rush to “writing equations and computer code” before developing a sufficient understanding of the problem at hand. A profound investigation of the situation context is the foundation for a rigorous modelling effort to achieve higher impact results. Otherwise we take the high risk of wasting resources on developing super-elegant but less relevant and less useful models. While the upfront cost of exploring the ambiguous elements of a modelling context may seem expensive, it is often worth the extra effort because it can yield results that a community or interest groups will value.

There is a wide range of problem structuring methodologies and techniques for exploring and framing issues in ambiguous and ill-defined decision making contexts including: soft systems methodologies [19], narrative analysis and value-focused thinking. Outputs from these inquiries (mainly qualitative insights) help modellers “get a sense” of how the system works, stakeholder values and issues of concern. This knowledge provides the basis for defining modelling purpose, use, design and functional requirements.

A key challenge facing modellers is how to translate qualitative stakeholder views and values into formal model inputs (i.e. scenarios) and outputs (i.e. indicators). It is observed that in many cases the link between stakeholder frames and models are weak, and the logic information flow is obscure, and sometimes, disconnected. This weakness may defeat the modelling purpose and undermine a model’s credibility and relevance. Whereas this problem is inevitable given the different philosophical stances and natures underlying interpretive/soft framing and formal/hard modelling, there is still potential for bridging this gap, or at least transparently reporting inconsistencies. Promising steps in this direction include [20,21].

These concerns must be balanced with a determination to keep the model as simple as possible to meet the objectives. In some cases this will mean simplifying the objectives to obtain usefully accurate answers.

## 4.3 Model and Data Assurance, Uncertainty and Sensitivity

Much improved modelling practice is warranted and has several requirements. Placing more emphasis on specifying and assessing basic model limitations should be within easy reach. This is mostly a matter of the peer review community insisting more vigorously on researchers being more comprehensive in this regard. Some items in a list of limitations are easily specified and relate to the model assumptions, while others can be revealed by sensibility and sensitivity testing. Do the parameter values make sense? Are some parameters redundant? Is the fit of predictions to observations unacceptably poor under important conditions? Model performance evaluation,
beyond perfunctory sensitivity analysis and basic calibration checks, are needed to
advance the usability for real world problems.

Sensitivity analysis (SA) needs to be a common step in any complex model
evaluation, and indeed could be applied much more in the model construction stage
where alternative hypotheses are being entertained. But SA is typically applied in a
perfunctory manner, for example merely changing parameters one at a time by some
proportional amount after acceptance of the model. Saltelli and Annoni [22] illustrate
the dangers of this simplistic practice. While methods of SA need to be better
developed for complex models, including those with a strong spatial and
multidisciplinary component, use of first cut methods like that of Morris and
frequency domain analysis can be more widely applied as very useful screening tools.

Uncertainty analysis (UA) of a model, as opposed to the modelling process, is
starting to receive increased attention. But it also needs to be more pervasive, and
indeed eclectic. Much of the current focus on UA is either on sampling techniques
like pseudo Monte Carlo or on sophisticated and computationally demanding
Bayesian techniques where obtaining convergence seems to be an art. At best these
methods seem most suited to single discipline models that are not overly complex. In
many cases, simpler methods may suffice. Norton [23] for example argues and
demonstrates the ease of algebraic SA, where operations in an equation can be
combined to find the sensitivities of its output to variations in contributing factors.
The challenge for UA is to find ways of propagating uncertainties between model
components of an integrated model that have been analyzed individually in a manner
that is appropriate for that model component. Another focus in UA should be
augmented consideration of errors in input data that drives a model, on output data
that is used to calibrate and/or assess performance, and on alternative model structure
hypotheses. In the end what would be very advantageous is a catalogue of
uncertainties and a ranking of them, preferably informed by their effect on the
decision options to understand which uncertainties have crucial consequences.

4.4 Long term modelling projects

Most integrated modelling and DSS development efforts are short term projects which
are deemed completed upon the delivery of an end product in the form of a tool
and/or final results report. Therefore, there is little attention paid to the post-
implementation phases in the model's lifecycle, including: tools/results use,
maintenance and summative evaluation. This impairs our understanding of how
models are actually used in the policy domain and reduces chances for improvement.
In addition, most projects are single shot event whereas increasing returns can be
achieved through cycles of implementation. Whereas this observation has been made
frequently in the integrated modelling literature [e.g. 24,25] we cannot perceive much
improvement in this direction. Mysiak et al. [26] argue that the existing research
funding mechanisms impede long term projects and widen the perceived gap between
science and policy making.
## 4.4 Software development methodologies

A software development methodology is a prescriptive one that structures and coordinates the technical process of designing, implementing, and testing a software product. Existing software development methodologies provide a complementary framework of concepts, practices, methods and tools that can improve the effectiveness and efficiency of the modelling process and products. Using one of these methodologies enables modellers to apply mechanisms that explicitly incorporate usability aspects into the modelling process, such as: user-centred design and prototyping. To increase the awareness of the integrated modelling and software community about software development concepts, Verweij et al [27] present an IT perspective on the integrated modelling process.

# 5 How do we get there and what are the opportunities?

There is an enormous opportunity for members of the integrated modelling and environmental decision support community not only to contribute to the decision making process technically, but also to play a role as neutral broker. Teams are starting to emerge which express the combined overall scientific and participatory strengths - through engaging the interest groups, framing the problem with them and applying scientifically reputable tools.

In order to advance our metadiscipline, several strategies suggest themselves. One is to share the accumulated learnings obtained by those teams from studying messy problems. This will require mechanisms that encourage exchanges across research community boundaries to share problems and solutions.

Another strategy is to develop awareness in teams to select an integrated modelling and software approach that fits the problem well and to comprehensively apply tools that satisfactorily enhance the quality of the process and characterise the credibility of the product. Such an open mindset and capacity necessitates changes to the way that most modellers are educated. Researchers in the main tend to become modellers either from being motivated by the need to solve a problem that is in their sphere of interest or by coming through specific disciplines. In the former case many do not have modelling training. In the latter they may have more training in but tend to have preferred ways of treating problems. What is required is a move towards an education and training focus on the discipline of modelling itself that emphasizes the problem context, while also providing exposure to a broad range of problem contexts, disciplines and methods. A modeller's satisfaction would then derive from the greater insights generated from looking for, finding and applying the most relevant tools to the problem – in stark contrast, say, to the rewards of applying approaches that fit within one's comfort zone. Better modelling training is essential. To some extent we have been going backwards with so few modellers able to write serious code. This is epitomized by the advent of the interface which has prevented many from seeing what's inside the models they use.
This education of modellers can take many shapes. One is to develop the relevant majors in undergraduate degrees or create professional training programs that advance modelling capacity for mid-career levels. Another is to insist on accreditation of modellers. A community of integrated modellers exists with too few points of contact. Developing a community of practice for integrated modelling with recognized access points, such as cyberinfrastructure hubs and workshops, makes sense to us.

Participatory modelling and greater consideration and documentation of the modelling process can also support ongoing improvement of modelling practice. In some cases, modelling in a participatory setting could be considered a service rather than a product (though it is very much a research service requiring much acumen). The focus is then on the model enabling social learning and building capacity, with both the model's development and use being facilitated by its creators. This improves the modeller's capacity to evaluate the model, and helps ensure it has made an impact regardless of the fate of the product. Documentation of the modelling process to the extent that it is recoverable [28], and consideration of future re-use of the model, can also facilitate adaptive modelling, such that new work is aware of and builds on past experiences. A database of documented recoverable experiences and their evaluation can provide inspiration for better methods to tackle overlooked but often essential tasks, as well as provide evidence for revision and evolution of best practice guidelines.

What such an agenda implies about research infrastructure has been discussed in [29] and bears repeating. “Plainly a new research and education effort as broad and as discipline-spanning as the one outlined above poses some questions about the adequacy of present research, education and training arrangements. It implies time scales of the order of a decade, funding on a scale allowing continuity for a community large enough to make significant progress on broad and demanding issues, and a means of getting researchers and research users from very different backgrounds to talk to each other and learn each others’ perspectives and priorities. It also poses hard practical problems in sourcing, testing, describing, providing access to and archiving data.”

## References

1. Rittel, H., Webber, M.: Dilemmas in a General Theory of Planning. Policy Sciences 4, 155-169 (1973)

2. Ravetz, J.R.: Integrated Environmental Assessment Forum: Developing Guidelines for “Good Practice.” Darmstadt University of Technology. ULYSSES WP-97-1, ULYSSES Project (1997)

3. Oxley, T., McIntosh B.S., Winder, N., Mulligan, M., Engelen, G.: Integrated Modelling and Decision-Support Tools: a Mediterranean Example. Environmental Modelling and Software 19, 999-1010 (2004)

4. Jakeman, A.J., Letcher, R.A., Norton, J.P.: Ten Iterative Steps in Development and Evaluation of Environmental Models. Environmental Modelling and Software 21, 602-614 (2006)
5. Refsgaard, J.C., Henriksen, H.J., Harrar, W.G., Scholten, H., Kassahan, A.: Quality Assurance in Model Based Water Management – Review of Existing Practice and Outline of New Approaches. Environmental Modelling and Software 20, 1201-1215 (2005)

6. Van der Sluijs, J.P., Craye, M., Funtowicz, S., Kloprogge, P., Ravetz, J., Risbey, J.: Combining Quantitative and Qualitative Measures of Uncertainty in Model Based Environmental Assessment: the NUSAP System. Risk Analysis 25, 481-492 (2005)

7. Lee, N.: Bridging the Gap Between Theory and Practice. Environmental Impact Assessment Review 26, 57-78 (2006)

8. Bennett, N.D., Croke, B.F.W., Jakeman, A.J., Newham, L.T.H., Norton, J.P.: Performance Evaluation of Environmental Models. In: Swayne, D. et al. (eds), Proceedings International Environmental Modelling and Software Society, Ottawa, Canada, (2010)
http://www.iemss.org/iemss2010/proceedings.html

9. Becker, A., Soncini-Sessa, R., Castelletti, A., Hattermann, F.F., Willems, P., Stalnacke, P., Laurans, Y., de Lange, W.J.: How can Models Help Implementing the Water Framework Directive? In: Hattermann, F.F., Kundzewicz, Z.W. (eds.), Water Framework Directive: Model Supported Implementation. A Water Manager’s Guide. IWA Publishing, London (2010)

10. Bousquet, F., Voinov, A.: Modelling with Stakeholders. Environmental Modelling and Software 25, 1268-1281 (2010)

11. Matthews, K.B., Rivington, M., Blackstock, K.L., McCrum, G., Buchan, K., Miller, D.G.: Raising the Bar? The Challenges of Evaluating the Outcomes of Environmental Modelling and Software. Environmental Modelling and Software 26, 247-257 (2011)

12. Jakeman, A.J., Letcher, R.A., Chen, S.: Integrated Assessment of Impacts of Policy and Water Allocation Change Across Social, Economic and Environmental Dimensions. In: Hussey, K., Dovers, S. (eds) Managing Water for Australia: the Social and Institutional Challenges. CSIRO Publishing, pp. 97-112 (2007)

13. De Ridder, W., Turnpenny, J., Nilsson, M., von Ragmby, A.: A Framework for Tool Selection and Use in Integrated Assessment for Sustainable Development. Journal of Environmental Assessment Policy and Management 29, 423-441 (2007)

14. Hartig, F., Horn, M., Drechsler, M.: EcoTRADE – A Multi-Player Network Game of a tradable Permit Market for Biodiversity Credits. Environmental Modelling and Software 25, 1479-1480 (2010)

15. Rojas, R., Kahunde, S., Peeters, L., Batelaan, O., Feyen, L., Dassargues, A.: Application of a Multimodel Approach to Account for Conceptual Model and Scenario Uncertainties in Groundwater Modelling. Journal of Hydrology (in press)

16. Guillaume, J.H.A., Croke, B.F.W., El Sawah, S., Jakeman, A.J.: Implementing a Framework for Managing Uncertainty Holistically. In: Watermatex2011, San Sebastian, Spain (2011)

17. Walker, W.E., Harremoës, P., Rotmans, J., van der Sluijs, J.P., van Asselt, M.B.A., Janssen, P., von Krauss, M.P.K.: Defining Uncertainty: A Conceptual Basis for Uncertainty Management in Model-Based Decision Support. Integrated Assessment 4(1), 5-17 (2003)

18. Guillaume, J., Pierce, S.A., Jakeman, A.J.: Managing Uncertainty in Determining Sustainable Aquifer Yield. In: Groundwater 2010 Conference, Canberra, 31 October–4 November (2010)

19. Rosenhead, J.: What's the Problem: An Introduction to Problem Structuring Methods. Interfaces 26 (6), 117–131 (1996)

20. Kok, K.: The potential of Fuzzy Cognitive Maps for Semi-Quantitative Scenario Development, with an Example from Brazil. Global Environmental Change 19, 122–133 (2009)
21. Kok, K., van Delden, V.: Combining Two Approaches of Integrated Scenario Development to Combat Desertification in the Guadalentin Watershed, Spain. Environment and Planning B 36, 49-66 (2009)

22. Saltelli, A. and Annoni, P.: How to Avoid a Perfunctory Sensitivity Analysis. Environmental Modelling and Software 25, 1508-1517 (2010)

23. Norton J.P.: Algebraic Sensitivity Analysis of Environmental Models. Environmental Modelling and Software 23, 963-972 (2008)

24. Toth, F: State of the Art and Future Challenges for Integrated Environmental Assessment. Integrated Assessment. 4, 250-264 (2003)

25. Letcher, R.A., Jakeman, A.J.: Application of an Adaptive Method for Integrated Assessment of Water Allocation Issues in the Namoi River Catchment, Australia. Integrated Assessment. 4, 73-89 (2003)

26. Mysiak, J., Bazzani, G., Tamaro, M. et al.: Review of Model Based Tools with Regard to the Interaction of Water Management and Agriculture. Chapter 3 In: Vanrolleghem, P. (ed.), Decision Support for Water Framework Directive Implementation, IWA Publishing, London (2011)

27. Verweija, P.J.F.M., Knapena, M.J.R., de Wintera, W.P., Wiena, J.J.F., te Rollera, J.A., Sieberb, S., Jansen, J.M.L.: An IT Perspective on Integrated Environmental Modelling: The SIAT Case. Ecological Modelling 221, 2167-2176 (2010)

28. Checkland, P.: Model Validation in Soft Systems Practice. Systems Research 12(1), 47-54, DOI: 10.1002/sres.3850120108 (1995)

29. Jakeman, A.J., Letcher, R.A., Norton, J.P. et al.: Outstanding Research Issues in Integration and Participation for Water Resources Planning and Management. Chapter 15 In: Castelletti, A., Soncini-Sessa, R. (eds), Topics on System Analysis and Integrated Water Resource Management, Elsevier, Amsterdam, pp. 273-289 (2007)