# How do multi-criteria assessments address landscape-level problems? A review of studies and practices

Sandrine Allain, Gaël Plumecocq, Delphine Burger-Leenhardt

► To cite this version:

Sandrine Allain, Gaël Plumecocq, Delphine Burger-Leenhardt. How do multi-criteria assessments address landscape-level problems? A review of studies and practices. Ecological Economics, 2017, 136, pp.282-295. 10.1016/j.ecolecon.2017.02.011. hal-01605875

HAL Id: hal-01605875

https://hal.science/hal-01605875

Submitted on 26 May 2020

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
1 How do multi-criteria assessments address landscape-level problems? A review of

2 studies and practices

3

4 Sandrine ALLAIN

5 Gaël PLUMECOCQ

6 Delphine LEENHARDT

7

8 Professional affiliation:

9 AGIR, Toulouse University, INRA, INPT, INP-EI PURPAN

10

11 Address:

12 BP 52627, F-31326 Castanet-Tolosan cedex, France

13

14 Corresponding author: Sandrine ALLAIN

15 Contact: sandrine.allaingranet@gmail.com

16
17 **ABSTRACT:**

18 Viewing the landscape as a spatialized social-ecological system allows identification of
19 specific management challenges: integration of multiple views, multiple levels of
20 organization, complex spatial-temporal patterns and uncertainties. Multi-criteria assessments
21 (MCAs), which allow the comparison of alternative actions when multiple interests collide,
22 are considered adequate to support landscape management. However, there is no consensus
23 about how they should be applied and can integrate both multiple views and spatial
dimension. We conducted an extensive quantitative and qualitative literature review targeting
25 MCAs with a participatory and spatial approach. Our results suggest that (1) for sustainability
26 assessments, participatory and spatial approaches endorse different rationales and hybrid
27 methods are not so common; (2) within those methods, only scenario-selection methods (as
28 opposed to design methods) can integrate spatially-explicit, spatially-implicit, place-specific,
29 and overall values; and (3) current applications, which aggregate values ignoring their spatial
30 and social distribution, do not coincide with the nature of landscape-management challenges.
31 In addition, they give little importance to the structuration of information and to collective
deliberation. We conclude that, in the absence of a good match between spatiality and
33 participation, MCAs should, for now, be handled as insightful but distorted tools to explore
34 and structure landscape-level management problems.

35

36

37 **KEYWORDS:**

38 landscape management, integrated assessment, spatialization, participation, social-ecological
39 system

40
41 **1. INTRODUCTION**

42 Landscapes are complex social-ecological systems (SES) because many human and natural processes
43 mutually interact (Bastian, 2001; Cumming et al., 2012; Naveh, 2000; Wu, 2006) and because they are
44 shaped by a social history (Antrop, 2000; Pedroli et al., 2006), which promotes perceptions, values or
45 expectations that differ spatially and among individuals. This makes it challenging to collectively
46 define a desirable future for a given landscape. Many possible ways exist to address this issue; one of
47 them is multi-criteria assessment (MCA). Nonetheless, there is no consensus on how to apply this
48 method given the nature of landscape-level challenges.

49 A first challenge comes from the complexity of processes and multiple interactions. Because
50 landscapes are characterized by interdependencies between human societies and their environment that
51 originate from a coevolutionary history (Berkes et al., 2000; Costanza et al., 1998; Daily, 1997; Kallis
52 and Norgaard, 2010), they exhibit non-linear and cascading effects that make their trajectories of
53 change impossible to predict (Kinzig et al., 2006; Walker et al., 2004). It is generally acknowledged
54 that command-and-control approaches (Folke et al., 2002; Holling and Meffe, 1996) and risk
55 assessment (Linkov et al., 2014) are ill-suited in this context. Instead, scholars advocate exploring
56 possibilities for change (Berkes et al., 2002; Olsson et al., 2006) and adopting an adaptive
57 management approach (Linkov et al., 2006; Plummer, 2009).

58 Another challenge with landscape-level problems arises because the different groups concerned hold
59 different and sometimes irreconcilable values (Gómez-Sal et al., 2003; Hunziker et al., 2008;
60 Swedeen, 2006). This problem of multiple views, which is common to many decision-making
61 situations, has led to recommendations to include non-experts in evaluations of the quality of decisions
62 (Funtowicz and Ravetz, 1990) and explicitly consider incommensurabilities<fnref n="1" /> (Martinez-Alier et al.,
63 1998; Munda, 2004). Deliberative approaches and MCAs involving multiple stakeholders are
64 considered particularly well-suited to operationalize these principles (Frame and Brown, 2008; Munda,
65 2004).

<footnote n="1">1 Specific concepts are defined in the appendix.</footnote>
The landscape is a complex system that has another challenging characteristic: material resources and populations are distributed in space. Human and natural systems can interact “through” the spatial dimension: social and ecological processes increasingly overlap as perspective widens to a global scale (Alessa et al., 2008), and spatial mismatches can have far-reaching consequences (Cumming et al., 2012; Wilson et al., 1999). Likewise, human and natural systems can interact "within" the spatial dimension: processes such as species migration, farming dynamics or social exclusion are closely related to spatial patterns, such as habitat heterogeneity and the spatial distribution of crops or infrastructure networks (Benoît et al., 2012; Cumming, 2011). Accounting for complexity at the landscape level therefore requires considering these different spatial interactions. This is one reason why geographic information systems (GIS), given the wide possibilities they offer to investigate spatial relationships, have become key tools to analyze and resolve landscape-level management problems (Malczewski, 2006; Malczewski and Rinner, 2015).

Because social-ecological interactions have a spatial dimension, diverse and potentially conflicting representations of space, i.e. new types of incommensurabilities, coexist within a landscape. Because people relate to places in many different ways – not only through their actions, but also through their perceptions and history (Antrop, 2005) – they do not have the same definition of boundaries, meaningful zoning, significant places, features of identity, etc. The same occurs with expert descriptions: relevant extents, resolutions and locations differ when describing water dynamics or pollination. The ecological economics community does not formally address these types of incommensurabilities specific to spatial problems, though it is aware of “scale biases” when stakeholders express value judgments (Hein et al., 2006; McFadden, 1994; Zia et al., 2011). Incommensurabilities are not well integrated into spatial decision support systems either, because the latter are designed as “expert systems” that rely on a uniform understanding of space (Ramsey, 2009). Applying MCAs to landscape management problems raises the fundamental challenge of integrating spatiality with multiple views. In an initial step to meet this challenge, we investigate current practices of MCA reported in the scientific literature that combine a spatial approach with multi-stakeholder or participatory approaches. More specifically, we address the following issues: how, and how well, MCA practices reflect landscape-specific challenges.
94 These issues are addressed following three nested analyses that enable us to:

95 (i) Position spatial and participatory approaches within the broad scope of multi-criteria methods

96 (ii) Distinguish types of MCA methods that combine spatial and participatory approaches

97 (iii) Clarify how MCAs are applied to assess landscape-management scenarios

98 (iv) Generate suggestions for using MCAs at the landscape level.

99
100

101

# 1. MATERIALS AND METHODS

102

103

104

105

106

107

108

109

110

This literature review follows three steps (the overall method is described in Fig. 1). First, we performed a lexicometric analysis of a large sample of studies to characterize the position of sustainability assessments, participatory approaches and spatial approaches within the wide spectrum of multi-criteria methods (Section 2.2). Second, we qualitatively classified applications of multi-criteria approaches mixed with participatory and spatialization methods in the field of natural resource management (Section 2.3). Our aim was to provide a typology of existing methods with their general steps. Third, we focused on a specific type of methods arising from the typology, “scenario-selection support method”, and undertook a detailed qualitative analysis of the corresponding case-studies (Section 2.4).

111

## 1.1. Bibliographical data.

112

113

114

115

116

117

118

We generated three datasets of studies of decreasing size using the Web of Science database, corresponding to the three steps of our review. For better traceability, we summarized this selection process in a PRISMA diagram<fnref n="2" /> (Moher et al., 2009) (Fig. 1). A large dataset of abstracts (10,691) was selected to analyze recent trends (2005-2015) in sustainability assessments within multi-criteria methods. For the two subsequent steps, the time span was extended to all available years (1975-2005) to embrace a wider diversity of research; nonetheless, few records were published before 2005.

<footnote n="2">2 The “PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) statement” includes a</footnote>

<footnote n="3">3 flow chart that maps out the number of records identified, selected for analysis and excluded. The general aim is to</footnote>

<footnote n="4">4 improve the reporting of systematic reviews and to help the reader identify bias in the selected materials.</footnote>
<figure><img src="image_2.png" /><figcaption>Fig. 1 PRISMA flow diagram for each objective addressed in the review (database searched in June 2015).</figcaption></figure>

119

120

121

The three steps reflected in the PRISMA diagram are detailed in the three following sub-sections.

122

## 1.2. Step 1. Quantitative analysis of multi-criteria methods.

123

Lexicometric analysis compresses complex information contained in large numbers of texts.

124

Reorganizing and classifying segments of texts and providing textual statistics promotes understanding of the meaning and context of use of utterances (Lebart et al., 1998). We used this method to identify trends in the current use of multi-criteria methods throughout the observation of lexical similarities among abstracts.

125

126

127

128

We collected information that characterizes the heterogeneity of research referring to multi-criteria methods. We compiled the 10,691 abstracts we selected into a single textual corpus and used IRaMuTeQ software (Ratinaud and Marchand, 2012) to implement Reinert's method (1993) of textual clustering. This clustering technique proceeds from a contingency table

129

130

131
showing the presence or absence of words (except pronouns, conjunctions, and some adjectives) in a given abstract. IRaMuTeQ clusters abstracts by performing descending hierarchical classification iteratively: abstracts are divided into two significantly different groups (p-value < 0.05) according to the presence or absence of words, and then the cluster with the most abstracts is divided into two new significantly different groups, etc. Clustering stops when the number of clusters predefined by the analyst is reached or when the largest cluster contains no significantly different vocabulary. A chi-square test is then calculated for the classification of each word to assess the significance of the association between a word and a semantic class. This enabled us to obtain a broad view of multi-criteria methods and then identify subgroups relevant for analysis.

We also assessed the classification via correspondence analysis, in which variability in the vocabulary and its distribution among clusters (assessed as a chi-square value) is statistically explained by inertia. For each inertia factor, correspondence analysis provides a statistical basis for interpreting why certain clusters are similar to or different from others.

146

147 1.3. Step 2. Qualitative classification of multi-criteria assessments that use
148 participatory and spatial approaches.

149 In the second step of our analysis, we emphasize the participatory and spatial approaches of
150 MCA.

151 We collected research that links these three aspects (i.e., multi-criteria, spatialization,  
152 participation) through a database search and manual screening. “Spatialization” meant that  
153 alternatives and/or the assessment results had to be spatialized; “participation” meant that  
154 people other than the authors had to be involved at some stage of the research. We also  
155 removed all reviews, theoretical developments and studies that introduced a multi-criteria tool  
156 without applying it (even though they could provide interesting insights for interpretation).
Among the 222 studies collected, 126 met those eligibility conditions, and among them 74 belonged to the field of natural resource management. The others referred to waste- or pollution-management issues, urban or infrastructure development and health.

We systematically read the abstracts of the 74 studies dealing with natural resource management, with a special focus on their objectives and methods; when necessary, we also read the studies' methods sections. We focused more on the stages in which the inclusion of a spatial or participatory component to MCA adds to or modifies the way the method is developed. We summarized the most frequent method patterns according to their objective.

## 2.4. Step 3. Systematic qualitative analysis of case studies for scenario exploration.

To address our third objective, we analyzed a subset of the previous studies. We only investigated articles that developed a method for exploring landscape management alternatives and possibly selecting one of them. We ended up with 10 studies, some of them covered by more than one article. When necessary, we used additional articles referring to the same research project and contacted the corresponding author to clarify certain points or to obtain additional information.

Two types of characteristics were examined for each study. First, general characteristics of MCA were analyzed: the problem structure (i.e. alternatives, criteria, stakeholders and their judgments) and the decision analysis (the weighting system and the aggregation procedure used to produce the final evaluation of alternatives) (Kiker et al., 2005; Malczewski, 1999).

Second, we investigated characteristics related to the landscape viewed as a spatialized SES, focusing on participation of multiple stakeholders (i), representation of multiple levels in the assessment(ii), consideration of spatial and temporal patterns and relationships (iii), and management of uncertainties (iv).
182 We created an analytical table (provided in the appendix) with columns referring to the 10

183 studies reviewed and rows featuring the characteristics of each case-study in terms of:

184 - general characteristics (objective, type of issue addressed, area)

185 - problem structure (definition of alternatives, criteria (and indicators), stakeholders,
186 value judgments)

187 - decision rules (weighting, aggregation method)

188 - involvement of stakeholders (participants, stages with participation, differentiation
189 between participant input, participatory settings)

190 - multi-level system (levels of assessment, upscaling methods)

191 - spatiotemporal patterns (knowledge sources for spatial data, accounting for spatial
192 patterns (heterogeneity, distribution of values), use of visualization tools, accounting
193 for temporal behavior)

194 - integration of uncertainties (sensitivity analysis, uncertainties in outcomes,
195 uncertainties/inconsistencies in judgments, flexibility of tools)

196 This way of synthesizing information allowed us to identify trends and gaps in the ways

197 landscape-management options are assessed with a multi-criteria approach.
## 3. RESULTS AND DISCUSSION

### 3.1. Results of lexicometric analysis.

#### 3.1.1 Main clusters of multi-criteria methods: the vocabulary of sustainability assessments

In the abstract corpus (10,691 records), we performed a three-cluster classification, which allowed 9,450 abstracts (88%) to be classified (Fig. 2). The most significant forms of words, tool-words excluded, define three distinct “lexical worlds”.

<figure><img src="image_3.png" /><figcaption>Fig. 2 Dendrogram resulting from descending classification of the abstract corpus, showing the 20 most significant lexical forms of each cluster. Frequency is the number of abstracts classed in a given cluster that contains at least one time a given word divided by the total number of abstracts. Chi-squared values assess the significance of the association between a given word form and a cluster (a chi-squared value of 3.84 corresponds to a probability of 5% that the association of a word to a certain category occurred randomly)</figcaption></figure>
208 - integrating economic, environmental, social and institutional issues (“environmental”,
209 “economic”, “social”, “policy”, “public”)

210 - considering future consequences and uncertainties (“impact”, “assessment”, “options”,
211 “management”, “potential”)

212 - engaging the public (“stakeholder”, “public”)

213 Gasparatos et al. (2008) depicted these different aspects as shared properties of sustainability
214 assessments and added equity considerations (intra- and inter-generational) to this list. Equity
215 considerations, however, are not directly reflected by the most significant words in the
216 “sustainability assessment” cluster. The first word in this semantic field, “equity”, appears in
217 312th position (still significant for this cluster) because it occurs rarely. We conclude that
218 social justice issues are, in most cases, absent from sustainability assessments based on a
219 multi-criteria method. Sustainability remains mainly understood as the fulfilment of
220 competing goals rather than a trajectory of change that reconfigures social interactions.

221 A second cluster of abstracts focuses on optimization problems (“optimization”, “Pareto”,
222 “solution”, “design”, etc.) aiming at designing Pareto-optimal³ solutions for a limited set of
223 objectives and constraints (Linkov et al., 2006). Decision-analysis procedures (“decision”,
224 “rank”, “selection”, “aggregation”) are grouped together in the last cluster of abstracts (cluster
225 3). Considering that no optimal solution can be found, a variety of procedures exist to
226 rank/sort alternatives (Guitouni and Martel, 1998), each of them corresponding to a specific
227 definition of what makes the best compromise (the closest from an ideal point, the one that is
228 not outranked by others, etc.).

229

230 3.1.2. *Rationales underlying MCA methods: the specificity of spatial approaches*

<footnote n="5">5 3 Resource allocation is considered optimal (sensu Pareto) when an increase in one objective cannot be achieved</footnote>

<footnote n="6">6 without worsening another objective. The total number of situations that satisfy the Pareto criterion forms a</footnote>

<footnote n="7">7 boundary limit from which no improvement can be made given the available resources. The existence of such a</footnote>

<footnote n="8">8 boundary assumes that objectives are perfectly substitutable.</footnote>
We generated an additional classification to see how the sustainability assessment cluster would split up, adding two more successive phases of clustering. The abstract corpus was consequently divided into 5 classes (8,964 abstracts included in the new classification, ~84% of the corpus). We were then able to distinguish two new subgroups of abstracts: strategic business decision-making (within the “decision analysis” cluster, former cluster 3) and spatial approaches (within the “sustainability assessment” class), the latter being of special interest for our purpose. The “optimization problem” cluster, (former cluster 2) remained unaffected by this new classification. Projection on the two first axes of the correspondence analysis (Fig. 3), which summarize 69% of the total inertia, reveals that spatial assessments constitute both a marginal (approx. 1/4 of the abstracts on sustainability assessments) and specific branch of sustainability-assessment approaches.

<figure><img src="image_4.png" /><figcaption>Fig. 3: Distribution of active forms along the two first axes of the correspondence analysis (5-class clustering). These axes do not separate the classes of decision analysis theories and their application to business management.</figcaption></figure>
243 We interpret the two first factors as reflecting what is understood as a rational decision (axis
244 2, 29% of total variance) and what makes a scientifically-sound decision (axis 1, 40% of total
245 variance).

246 The first axis differentiates multi-criteria methods that rely on grounded, contextualized
247 information from those that mainly rely on mathematical equations to find a solution. By
248 "grounded", we mean that theory proceeds from the accumulation and analysis of place-based
249 data. Since they seek to capture multiple indicators and multiple views, sustainability
250 assessments imply such bottom-up view of science, providing they not only try to formulate
251 ad-hoc solutions but also create new knowledge. The opposite pole of axis 1 represents
252 another view of how science interacts with decision-making, in which there are "rules of
253 nature" that can be formalized (by scientists) and then applied to any situation (e.g. to support
254 decision-making). According to this view, generic models and algorithms are the elements
255 that validate the results obtained. Decision analysis, which focuses on finding the best
256 compromise when no optimum solution exists, and optimization methods, which focus on
257 finding solutions that maximize utility under a set of constraints, are two different methods
258 that share the assumption that mathematical formalization is the best way to reach a valid
259 solution.

260 Axis 2 reflects the opposition between substantive rationality and procedural rationality
261 behind multi-criteria methods (Simon, 1976). Under a substantive rationality hypothesis,
262 rational behaviors allow the fulfillment of a set of goals under a set of constraints (from the
263 external environment, its perception, or internal characteristics of the decision maker). This
264 realm assumes a "heroic" (Béjean et al., 1999) decision-maker who, according to Simon
265 (1990: 195): "contemplates, in one comprehensive view, everything that lies before him. [...]".
266 He has reconciled or balanced all his conflicting partial values and synthesized them into a
267 single utility function that orders, by his preference for them, all these future states of the
268 *world*". This idea is clearly embedded in optimization procedures. However, the substantive
269 rationality paradigm is also dominant in spatial assessments. The use of spatial data is thought
270 to "empower the decision-maker" (Densham, 1991) by providing both more precise and more
271 usable information. The environment (objective and subjective) is considered fixed and the
272 coexistence of different value schemes and conflicting preferences not allowed.

273 In the realm of procedural rationality, behaviors are rational because they come from an
274 appropriate individual deliberative process in the face of incomplete information, an unknown
275 future, or shifting beliefs and values (Simon, 1976). Studies related to decision analysis, since
276 they model uncertainty and the interaction of multiple views towards a compromise, typically
277 assume a procedural rationality (extended from the individual to the group) (Garmendia and
278 Gamboa, 2012; Munda, 2004). Sustainability assessments following a participatory approach
279 (Fig. 3, upper-left corner) also fall in this category because deliberation is the pivotal process
280 for integrating heterogeneous information and values (Vatn, 2005). Nonetheless, these
281 approaches often fail to accommodate the evolutionary nature of preferences and the
282 existence of non-utilitarian values (values that cannot be reflected in competing individual
283 interests). Rather, they assume that one's motivation can be captured in a unique order of
284 preferences (Béjean et al., 1999). They therefore only partially fall under procedural
285 rationality paradigm.

286 We have just demonstrated how two branches of sustainability assessment distinguish
287 themselves. Though common in their place-based anchorage, sustainability assessments
288 following a spatial approach and those following a participatory approach endorse
289 fundamentally different philosophies about what constitutes a good decision (the process or
290 the availability of information) and the context under which a decision is to be taken
291 (irreducible uncertainties and conflicting values or a well-defined decision space).
292 Nonetheless, bridges between these two branches exist and are crucial if one wants to address
293 multi-stakeholder landscape-level issues. We examine these bridges in the following section.

294

295 **3.2. How MCA combines with participatory and spatial approaches. Synthesis of**
296 **methods and objectives.**

297 In this section, we present results of our qualitative analysis that classified studies using
298 participatory and spatial approaches (Fig. 4). We distinguish three types of objectives, hence
299 three types of methods: scenario-design support, scenario-selection support, and spatial
300 assessment of real situations. The categories of both scenario-design support and scenario-
301 selection support methods are future-oriented, leading to management recommendations.

302 Method steps can be shared between real-world and future-oriented methods even when their
303 objectives differ (prescriptive as opposed to descriptive). We found two studies (3 records)
304 combining scenario-design support and scenario-selection support objectives (detailed in
305 section 3.3.2).

306 Regarding the two families of future-oriented methods, their differences in objectives are
307 reflected in different method steps (Fig. 5). Scenario-design support methods deal with a
308 location or spatial allocation problem (van Herwijnen and Janssen, 2007): Which areas should
309 be managed or preserved first (prioritization)? How best to allocate land-uses in the landscape
310 (zoning/ mapping proposals)? Those questions refer respectively to multi-attribute or multi-
311 objective problems. In the first case, every location is scored according to a set of attributes
312 (van Herwijnen and Janssen, 2007). Then, value maps are overlaid and aggregated, generally
313 with a weighted linear combination procedure (Malczewski, 2000), to define the best area(s)
314 or rank sites for a given purpose. This method is also used for suitability assessments. In the
315 second case, a set of goals are defined and multi-objective optimization algorithms produce
316 land-use plans or spatially-explicit management proposals.
Conversely, scenario-selection support methods differentiate and often rank management alternatives based on their effects (impact assessment) and sometimes their implementation (generally cost, but sometimes applicability or flexibility; e.g. Sahin and Mohamed, 2013). A scenario can be evaluated and possibly selected according to both spatially-explicit and non-spatial criteria. An aggregation procedure can be performed first on criteria (producing a landscape-level score for each criteria) or on alternatives (producing total performance maps) (see van Herwijnen, 1999, cited by van Herwijnen and Janssen, 2007), for different possible pathways).
<figure><img src="image_5.png" /><figcaption>Fig. 4. Classification of sustainability assessment methods combining a participatory and spatial approach according to their general objective for decision-making (supporting scenario design, assessing a real situation and supporting scenario selection) and the type of results produced.</figcaption></figure>

Double-headed arrows link categories that share similar method steps but differ in their objective.

* 4 records were assigned to 2 different categories.

### Scenario-design support

<figure><img src="image_6.png" /></figure>

### Scenario-selection support

<figure><img src="image_7.png" /><figcaption>Fig. 5. Classic steps in scenario design support (top) and scenario selection support (bottom). Participatory settings were used at different stages (underlined in the figure) varying between studies.</figcaption></figure>

Spatial diagnosis methods follow the same steps though objectives differ. When the aim is to establish risk, vulnerability or suitability maps, method steps are similar to the ones of the “scenario-design support” method. For value assessment, the steps follow the “scenario-selection support” method except that the input landscape(s) is (are) real.
319 The involvement of non-academic participants, including local experts, can occur at different
320 stages. In such cases, identified criteria and/or weighting are designed to reflect stakeholder
321 preferences. The issues of how to select participants, how to elicit their preferences and how
322 to combine them are the same for non-spatial evaluation (see e.g. Garmendia and Gamboa,
323 2012; Hajkowicz and Collins, 2006; Reichert et al., 2015). Also, some criteria can be
324 evaluated on a participatory basis when they integrate a sensible dimension (e.g. visual aspect
325 (Sheppard and Meitner, 2005)); risk perception (Raaijmakers et al., 2008)); place attachment
326 (Newton et al., 2012; Nordström et al., 2011) or when the evaluation of performances relies
327 on the statements of local experts (e.g. Arciniegas and Janssen, 2012).

328 Through this clarification exercise, we highlighted consistent patterns in scenario-design and
329 scenario-selection methods. The way stakeholders are involved does not appear to
330 differentiate methods. Stages with participation – and the extent of that participation – reflect
331 the diversity of scientific postures in action-research, which is independent from the
332 operational objective pursued. In contrast, the position of the MCA within the decision-
333 making process and the nature of criteria and values do distinguish scenario-design support
334 from scenario-selection support. In the former, alternatives are generally not defined a priori,
335 and the decision-making process is in its exploratory stage; also, criteria and values are
336 necessarily spatially-explicit. In the latter, alternatives are established a priori. They can be
337 realistic plans identified by managers (when the decision process is in an advanced stage) or
338 contrasting images of the future created to support learning. With scenario-selection support
339 methods, alternatives can be spatially-explicit or not (e.g. a normative change) and the
340 assessment of alternatives can accommodate both spatially-explicit and non-spatial values.
341 Assessment is therefore more comprehensive, which suits a holistic and integrative view of
342 the landscape. Because of these possibilities, we focus on the potential of scenario-selection
343 methods to compare landscape-management alternatives.
344

3.3. How MCA combining participatory and spatial approaches can help compare landscape-management alternatives. Detailed analysis of current practices.

345

346

347

3.3.1 Overview of selected studies

348

In our corpus, we found 11 studies (12 records) that implement a multi-criteria scenario-selection method at the landscape level, i.e. studies that use, at some stage, the input of non-academics to compare a set of landscape-management scenarios to highlight those best-suited (Table 1). The study of Raaijmakers et al. (2008) was excluded as there was no integration between the stakeholder assessment of alternatives (not spatial) and the simulation of their impacts using a GIS-based model. The 10 remaining studies deal with land-use planning (4), ecosystem conservation (3), water management (2) and forest management (2).

349

350

351

352

353

354

355

356

<table><caption>Table 1. General description of selected studies (LU = land-use). Italics indicate studies with a “light” participatory approach. Additional articles used for the analysis are also mentioned.</caption><thead><tr><td>Selected studies</td><td>Goal</td><td>Issue addressed</td><td>Geographic area</td></tr></thead><tbody><tr><td>Ahrens and Kantelhardt, 2009</td><td>Assessing the impact of farming production options on landscape functions</td><td>Agricultural landscape planning</td><td>Bayerisches Donauried (Germany)</td></tr><tr><td>Arciniegas et al., 2011</td><td>Designing LU plans of increased quality</td><td>LU planning</td><td>Bodegraven Polder (Netherlands)</td></tr><tr><td>Arciniegas and Janssen, 2012</td><td>Assessing impacts of water-allocation alternatives on watershed sustainability</td><td>Water management</td><td>Ichkeul Basin (Tunisia)</td></tr><tr><td>Chakroun et al., 2015</td><td>Assessing effects of forest LU management on ecosystem services</td><td>Regional LU planning</td><td>Saxony (Germany)</td></tr><tr><td>Fürst et al., 2013</td><td>Assessing impacts of alternative water-management regimes on ecological functions</td><td>Wetland ecological functioning</td><td>Wormer and Jisperveld fen meadow (Netherlands)</td></tr><tr><td>Fürst et al., 2010</td><td>Selecting a management strategy for the Snowy Plover conservation</td><td>Species conservation and coastal conservation</td><td>Florida Gulf Coast (USA)</td></tr><tr><td>Janssen et al., 2005</td><td>Assessing possible water-management interventions</td><td>Water management</td><td>Paros Island (Greece)</td></tr><tr><td>Goosen et al., 2007</td><td>Assessing costs and benefits of alternative ecological network plans</td><td>Ecological restoration</td><td>River Frome watershed (United Kingdom)</td></tr><tr><td>Linhoss et al., 2013</td><td>Designing and assessing alternative forest-management plans</td><td>Strategic forest planning</td><td>Lycksele forest (Sweden)</td></tr><tr><td>Manoli et al., 2005</td><td>Reducing conflicts and assessing alternative forest-management plans</td><td>Sustainable forest management</td><td>Lemon Landscape Unit (Canada)</td></tr><tr><td>Newton et al., 2012 (+ supporting information)</td><td></td><td></td><td></td></tr><tr><td>Nordström et al., 2010</td><td></td><td></td><td></td></tr><tr><td>Nordström et al., 2011</td><td></td><td></td><td></td></tr><tr><td>Sheppard and Meitner, 2005</td><td></td><td></td><td></td></tr><tr><td>Sheppard et al., 2006</td><td></td><td></td><td></td></tr></tbody></table>

357

358

359

360

Our sample covers the gradient of research postures that characterizes integrated assessments (Barreteau et al., 2010; van Asselt and Rijkens-Klomp, 2002). Some of the studies we analyzed hence implemented a “lighter” participatory approach, corresponding to a situation
361 of consultation or information (Barreteau et al., 2010). In these cases, stakeholders are only present at the very beginning of the process (associated with generating alternatives) or merely exist as assumed recipients/users of assessment results. They can also be absent but different values nonetheless considered through archetypical weighting schemes (in the case of Linhoss et al., 2013).

366 We identified two studies that combine scenario-selection and scenario-design approaches, and hence promote iteration for adaptive management. One (Nordström et al., 2011, 2010) used a scenario-design approach to generate and select three realistic but contrasting scenarios of forest management, which were consequently assessed. The other (Arciniegas et al., 2011; Arciniegas and Janssen, 2012) followed the opposite approach: scenario assessment was used as an exploratory phase prior to generating land-use plans. Although they are rarely used jointly, scenario-design and scenario-selection methods are not mutually exclusive.

373 **3.3.2. Results of qualitative analysis**

374 Table 2 summarizes the options adopted in the case studies reviewed.

<table><caption>Table 2. Qualitative analysis of the multi-criteria structure and incorporation of landscape-management characteristics of the 10 selected case studies. Options in bold refer to the most frequently observed practice. def.: definition; AHP: analytic hierarchy process; MAUT: multi-attribute utility theory; CP: compromise programming; MCDA: multi-criteria decision analysis; SDSS: spatial decision support system.</caption><thead><tr><th></th><th></th><th></th></tr></thead><tbody><tr><td rowspan="5">MUL TICRITERIA STRUCTURE</td><td>Alternative def.</td><td><ul><li>Directly developed by researchers according to local issues (consultation of stakeholders, literature review, etc.)</li><li>Developed by selected stakeholders or experts</li><li>Exploration-Selection process (computer-aided, matrix of scenarios)</li></ul></td></tr><tr><td>Criteria def.</td><td><ul><li>Defined by researchers (according to local issues)</li><li>Based on a scientific reference framework (more or less adapted to a specific context)</li><li>Defined by stakeholders in participatory settings (workshops, interviews)</li></ul></td></tr><tr><td>Indicator def.</td><td><strong>Defined or specified (following consultation) by experts or researchers</strong><ul><li>Defined by reference framework</li></ul></td></tr><tr><td>Stakeholder def.</td><td><strong>Absence of a formalized method</strong><ul><li>Snowball sampling, Stakeholder analysis followed by grouping and selection of participants</li></ul></td></tr><tr><td>Value judgment def.</td><td><ul><li>Importance or perceived performance (scoring/allocation of 30 points)</li><li>Preference (pairwise comparison)</li></ul></td></tr><tr><td>DECISIONS</td><td>Weighting systems</td><td><ul><li>Average of scores (for all stakeholders or at the group level)</li><li>No weights / Exploratory weights / Weighting schemes reflecting different focuses</li><li>Group average score x Group importance score</li><li>Market value (0 weight to non-marketable ecosystem services)</li></ul></td></tr></tbody></table>
<table><tr><td></td><td>Aggregation method</td><td><ul><li>When value maps are produced: weighted summation or absence of aggregated results</li><li>When assessment is not spatially-explicit: varied methods hypothesizing strong to weak compensation (AHP, MAUT, CP, Electre-II, Stochastic MCDA)</li></ul></td></tr><tr><td rowspan="4">INV OLV EME NT OF STA KEH OLD ERS</td><td>Participating stakeholders</td><td>Managers, Users of different types, Experts of different types, Representatives of interest groups, Farmers, Landowners</td></tr><tr><td>Differentiation of stakeholder input</td><td><ul><li>No</li><li>Distinct role of different stakeholder groups</li><li>Evaluation at the group level (through averaging)</li></ul></td></tr><tr><td>Stages with participation</td><td><ul><li>Criteria def. and/or Alternative def.</li><li>Assessment, Feedback, Selection of participants</li></ul></td></tr><tr><td>Participatory methods</td><td>Workshops, Interviews, Surveys/Inquiry</td></tr><tr><td rowspan="3">HIE RAR CHI CAL ORG ANI ZATION</td><td>Levels of assessment</td><td><ul><li>When assessment of criteria is spatially-explicit: basic units/ compartments + Landscape/Region</li><li>When assessment is not spatially-explicit: entire landscape/Region</li></ul></td></tr><tr><td>Upscaling/downscaling methods</td><td><ul><li>Area-weighted aggregation of basic units' normalized performance</li><li>Temporal aggregation according to statistical metrics</li><li>Corrected spatial aggregation according to landscape metrics</li></ul></td></tr><tr><td>Knowledge sources of spatial data</td><td><ul><li>Generic (from database)</li><li>Local (expert opinion, field survey, online stakeholder survey)</li><li>Simulation outputs</li></ul></td></tr><tr><td rowspan="3">SPA TIAL & TEM POR AL PAT TER NS</td><td>Spatial distribution</td><td><ul><li>No</li><li>Qualitative analysis of value maps, Use of landscape metrics, Use of spatially-explicit metapopulation models</li></ul></td></tr><tr><td>Visualization tools</td><td><ul><li>Value maps</li><li>No</li><li>Landscape visualization</li></ul></td></tr><tr><td>Temporal dynamics</td><td><ul><li>No</li><li>Statistical metrics describing temporal behavior</li></ul></td></tr><tr><td rowspan="4">UNC ERT AIN TIES</td><td>Sensitivity analysis in MCA</td><td><ul><li>No</li><li>Varying weights, Exploration of the whole weight space, Varying aggregation methods,</li></ul></td></tr><tr><td>Uncertainties in outcome</td><td><ul><li>No</li><li>Assessment under varying futures</li></ul></td></tr><tr><td>Uncertainties in judgment</td><td><ul><li>No</li><li>Consistency ratio to eliminate inconsistent stakeholders</li><li>Use of different methods to reveal preferences</li><li>Stochastic MCDA allowing undefined weights</li></ul></td></tr><tr><td>Flexibility</td><td><ul><li>No</li><li>Possible modification in the SDSS, Use of interactive devices</li></ul></td></tr></table>
380

(i) Multi-criteria structure

381

382

The case studies highlight typical limitations of MCAs in general (i.e. not specifically applied to landscape problems) about problem structuring. It is however acknowledged that the way in which problems are structured, including how to define alternatives or criteria, how to determine who is relevant and what legitimacy they have, how to elicit judgments and which decision rules to choose, is central to any MCA (Choo et al., 1999; Garmendia and Gamboa, 2012; Guitouni and Martel, 1998; Munda, 2004; Vatn, 2009).

383

384

385

386

387

388

In the 10 studies we reviewed, alternatives and criteria are either developed by researchers or by a set of stakeholders. Specific reference frameworks (for ecosystem services, ecological functions or management standards) are sometimes used. In some cases, definition of alternatives is a central part of the assessment method and results from a formalized exploration-selection-specification process (in Fürst et al., 2013; Nordström et al., 2011).

389

390

391

392

393

Often, but not always, criteria are translated into indicators (in other cases, this hierarchical structure is absent); definition of these indicators is managed by researchers and experts, without feedback from other participants.

394

395

396

The process for selecting the stakeholders included in the assessment is often not formally addressed. Two studies (Nordström et al., 2010; Sheppard and Meitner, 2005), which defined their study area as conflict-ridden (both around the issue of forest management), used stakeholder analysis to identify stakeholders and group them according to their assumed interests.

397

398

399

400

401

(ii) Weighting and aggregation

402

In 6 of the studies analyzed, criteria weights reflect stakeholder preferences. Individual elicitation of weights is preferred over collective deliberation, so that group-level values are actually the average of individual scores. The only case of using collective decision-making to

403

404
405 derive weights was found in an expert workshop in Arciniegas and Janssen (2012). Generally,
406 stakeholders are asked to score criteria according to their importance. In one study
407 (Nordström et al., 2010), stakeholders were asked to state preferences using a pairwise
408 comparison of criteria.

409 More originally, Newton et al. (2012) and Linhoss et al. (2013) used archetypical weighting
410 systems to reflect different ethical positions but not any particular stakeholder preferences.

411 Linhoss et al. (2013) produced weighting schemes for human-focused, bird-focused and
412 mixed values, while Newton et al. (2012) included a weighting scheme based on purely
413 monetary valuation.

414 In all the other cases reviewed, authors chose to use equal weights or exploratory weights
415 (weights are modified to determine how they influence the final ranking).

416 When value maps are produced as intermediate or final results of the assessment, weighted
417 summing is the common aggregation method. Studies producing spatially-aggregated,
418 spatially-implicit or non-spatial values to assess each criterion use different decision analysis
419 possibilities. The majority of those procedures assume that values associated with criteria can
420 compensate each other (compensatory procedures). Nonetheless, Ahrenz and Kantelhardt
421 (2009) specifically address the question of compensability between criteria by comparing the
422 outcomes of an additive model (multi-attribute utility theory) with those of an outranking
423 model (Electre-II).

424
425 (iii) Interaction with stakeholders

426

427 In most of the studies we collected, stakeholders of different status and with different interests
428 in the problem are considered and consulted during workshops. Interviews and surveys are
429 other observed practices. Most commonly, participating stakeholders are managers (who are
430 involved in the decision-making process) and representatives of interest groups (representing
users – farmers, timber-harvesters, hunters, etc. – or a political view – environmentalists, bankers, community leaders, etc.); they are involved in the definition of criteria and/or alternatives. Though researchers generally assign them to a category of stakeholders, Ahrenz and Kantelhardt (2009) instead let each participant choose the group to which he/she belonged. A few studies introduce hierarchy among stakeholders, either by differentiating the stages when they intervene (e.g., in Arciniegas et al. (2011), “public” stakeholders explored and designed alternatives, researchers introduced criteria, and experts defined indicators and weights) or by determining importance coefficients (in Nordström et al. (2010)), a steering group was established to define relevant stakeholder groups and their relative importance).

(iv)Multiple levels and spatio-temporal patterns

Consideration of the multi-level nature of the landscape system is generally simplified into a two-scale problem: elementary units and landscape/region. The study of Ahrenz and Kantelhardt (2009) is an exception, as authors consider an intermediate level – the farm – to derive production responses and calculate socio-economic indicators. Nonetheless, results are produced at the landscape-level only.

In the spatially-explicit assessments we analyzed, basic units or compartments are defined according to their homogeneous functioning (Chakroun et al., 2015; Janssen et al., 2005; Manoli et al., 2005) or their use (Arciniegas and Janssen, 2012). Another option is to use a cell grid (Fürst et al., 2009; Newton et al., 2012). Those units constitute the resolution of the spatial assessment. Value maps are then generated, showing the distribution of performances in the entire landscape. Subsequent spatial aggregation consists of adding the value of all units (with their size as a coefficient when it is not uniform). Some authors acknowledge the limits of such additive aggregation. For instance, Janssen et al. (2005) insist on the need to give disaggregated results to decision-makers, while Fürst et al. (2013) “corrected” the results
455 of spatial aggregation for two ecosystem services using landscape metrics⁴. Limits of spatial
456 aggregation can also be overcome through an overall evaluation: Sheppard and Meitner
457 (2005) attempted to do so by asking stakeholders to directly assess semi-realistic landscape
458 visualizations of competing alternatives.

459 Temporal patterns are conspicuous by their absence, with the notable exception of Manoli et
460 al. (2005), who first spatially aggregated each criterion considered (e.g., cost, water
461 availability) and then considered second-level criteria describing temporal behavior.

462 (v) Uncertainties

463 Finally, uncertainties are generally not considered. When they are, the most common practice
464 is to perform a sensitivity analysis on the weighting systems, which reflect uncertainties in the
465 preferences for each criteria (Chakroun et al., 2015; Janssen et al., 2005; Newton et al., 2012).

466 Linhoss et al. (2013) chose to use of a stochastic multi-criteria decision analysis in order to
467 test a case of uninformed weights (the whole weighting space is explored and a rank
468 acceptability index is calculated, describing the percentage of times one alternative ranks as
469 “most-preferred”). Uncertainties about future changes are addressed in two studies (Linhoss et
470 al., 2013; Manoli et al., 2005), in which climate is considered the main driver of change.

471 Uncertainties in judgment are addressed in the two studies that attempted to structure
472 stakeholder participation in a conflict-ridden context (Nordström et al., 2010; Sheppard and
473 Meitner, 2005) and in the study of Linhoss et al. (2013) that uses a stochastic procedure.

474 Another way to handle uncertainties is to design a flexible decision-support tool that allows
475 modifications. Some tools are also user-friendly (e.g. the touchtable of Arciniegas and Janssen
476 (2009); the "Pimp your landscape" software of Fürst et al. (2009)). It is not clear, however,
477 whether these tools can be modified by end-users.

<footnote n="9">9 4 Landscape metrics refer to measures or indices that consider the distribution of patches (or groups of patches)</footnote>

<footnote n="10">10 within a landscape mosaic. They can characterize either a landscape's composition (e.g., abundance of forested</footnote>

<footnote n="11">11 patches, evenness of different patch types) or spatial configuration (e.g., measures of patch density, shape, or</footnote>

<footnote n="12">12 connectivity).</footnote>
478 3.4. **Research gaps to fill to effectively address landscape-level management problems.**

479 **Critical discussion of observed practices.**

480 This systematic qualitative analysis enables us to identify and discuss several research gaps in the
481 application of MCA methods to assess landscape management alternatives.

482 (i) Lack of attention to problem structuring

483 Problems observed in MCA in general are also found in our analysis. Among them, the lack of
484 attention to problem structuring, which is crucial for establishing a shared platform for deliberation
485 and effectively linking assessment to decision making, is striking (Boroushaki and Malczewski, 2010;
486 Giampietro, 2003; Hajkowicz and Collins, 2006; Ramsey, 2009). This loophole is particularly
487 prevalent in the generation of alternatives, selection of stakeholders, and definition of indicators.

488 The selection of stakeholders, alternatives, criteria and indicators constitute a “compression” of the
489 information space (Giampietro, 2003) that conditions the quality of the MCA outcome. As shown by
490 this literature review, alternatives and criteria are often defined with relatively little involvement of
491 local stakeholders, although the conditions of their involvement are often not fully clarified or
492 formalized. Institutional analysis – at worst to identify the diversity of interests and “legitimate”
493 people to participate in the evaluation and at best to gain knowledge about their power relations – is a
494 noteworthy method to justify the choices made about participants, scenarios and criteria (De Marchi et
495 al., 2000; Munda, 2004; Xenarios and Tziritis, 2007). Also, if alternatives are used to expand the
496 decision-space – i.e. alternatives represent contrasting images that stimulate stakeholders’ creativity –
497 then formalized procedures (computer-assisted or not) that enable moving out of the classic set of
498 “sketched” alternatives can be useful (Groot and Rossing, 2011).

499 Defining indicators related to criteria is commonly considered the task of researchers and experts,
500 which is consistent with a positivist view of science. However, using MCA, especially in conjunction
501 with participatory methods, should force researchers to endorse a post-normal paradigm for evaluating
502 sustainability (Funtowicz and Ravetz, 1994). Following Le Bellec et al. (2012), we suggest that MCA
503 would gain in quality – under a post-normal assumption – if stakeholders were involved in the design
504 and validation of the indicator set. Also, if we acknowledge that a set of indicators reflects a specific
way of organizing information (i.e. a specific view of the problem), then it modifies the status of
indicators in MCA: indicators are not only “proxies” that provide a measurement of a criterion’s
performance but are also objects for collective deliberation (Frame and O’Connor, 2011).

(ii) Mismatches in the use of aggregation procedures

Another recurrent problem with current practices is that common aggregation procedures (i.e.
compensatory methods) are not consistent with the way value judgments are made or with
sustainability issues characterized by incommensurabilities. The first problem that was repeatedly
observed lies in the meaning of weights, which relates to the type of question stakeholders are asked
(Choo et al., 1999). In most studies that integrate stakeholder preferences, stakeholders are asked to
allocate points between criteria and then to score or rank them. Compensability is generally not
addressed as such because scientists do not ask “By how many times is this more important?” or “How
many units of one criteria would compensate for losing one unit of another?”, etc. (Choo et al., 1999;
Garmendia and Gamboa, 2012). Nonetheless, compensatory methods are usually used to aggregate
preferences. Participants could learn about the conceptual bases that distinguish, for instance,
compensatory aggregation from outranking methods through the questions they answer. Therefore,
defining a “good” question to elicit preferences could improve the general challenge of making the
assessment process more transparent for and controllable by those involved in decision-making (Bell
et al., 2001).

A second inconsistency lies in the idea that compensatory aggregation methods suit sustainability
assessments. If we consider the question of sustainability under a “non-substitutability” hypothesis
(strong sustainability, sensu Neumayer, 2003), then values of one criteria cannot compensate values of
another criteria, which excludes compensatory aggregation. Garmendia and Gamboa (2012) and
Munda (2004) suggest using outranking methods to avoid this bias. Ahrenz and Kantelhardt (2009)
tested one outranking method (Electre-II) and concluded that the transparency of the method is
limited, so that the choice between compensatory and non-compensatory methods for landscape
planning decisions should account for the context of the assessment. Likewise, the ordered weighted
averaging aggregation method allows partial compensability among criteria. In a GIS environment, it
has been used mainly for scenario-design support (e.g. prioritizing areas for forest conservation,
533 Averna Valente and Vettorazzi (2008) adjusting forest-management plans, Greene et al. (2010).

534 Nonetheless, its mathematical and computational sophistication (Aliyu and Ludin, 2015) seriously
535 limits its use in participatory settings. This trade-off between transparency and integration of
536 complexity should therefore be clearly addressed and discussed by scientists to justify their
537 aggregation choices.

538 (iii) Lack of consideration of the potential of GIS to enhance deliberation

539 In sustainability assessments, MCA methods often aim to produce group or societal solutions
540 (Boroushaki and Malczewski, 2010; Garmendia and Gamboa, 2012). In the studies selected, the
541 “group” solutions achieved are intermediate positions between the preferences of the various
542 stakeholders (e.g. a“compromise”, sensu Vodoz, 1994). Preferences are considered immutable
543 throughout the evaluation process, which can, at best, enhance the mutual understanding of
544 participants. Only one of our 10 studies effectively links spatial evaluation to deliberation (Arciniegas
545 and Janssen, 2012). In this study, maps not only provide a spatialized representation of performances,
546 but are also intermediate objects for exchanging views (for analysis and negotiation). In the other
547 studies, social learning remains understood as a “by-product” and not as a core process fundamentally
548 attached to the evaluation activity.

549 We support the rehabilitation of deliberation in spatially-explicit evaluations, understood as an
550 opportunity to dialogue, understand the position of others and eventually produce novel solutions
551 underlying a change of view. Such a deliberative perspective reconnects spatial assessments with the
552 philosophy of collaborative GIS (Boroushaki and Malczewski, 2010). Conditions to foster collective
553 deliberation encompass: establishing inclusive platforms for exchanging views (considering the
554 dominated actors and the silent mass of future generations), tracing the history of changes in the
555 evaluation matrix, engaging reiterations, and analyzing the reasons that led to a group solution or
556 impeded its emergence. This methodological posture suggests that the coupling of GIS tools with
557 MCA should turn toward the former’s potential to stimulate exploration, understanding and
558 redefinition of the decision problem by those who are involved in the evaluation (Malczewski, 2006;
559 Ramsey, 2009).
560 (iv) Forming stakeholder groups based on an *a priori* affiliation
561 Delineation of the stakeholder groups in the evaluation is crucial to allow for comparison of
562 contrasting views. This task differs from the identification of knowledge-holders (to define indicators)
563 and stakeholders (to choose diverse criteria and elicit judgments about them). The delineation of
564 groups for comparing alternatives comes later, as a way to organize the diverse opinions previously
565 explored and to identify possible alliances and conflicts. The common practice for forming groups that
566 we observed (when groups are considered) is reliance on social and/or professional affiliation (e.g.,
567 farmers, environmentalists, water managers). However, it supposes that social and/or professional
568 affiliation serves as a good proxy for value systems, i.e. value systems are considered homogenous
569 within each social or professional group.

570 Leaning on other research works, we suggest alternative guidelines to form groups of participants and
571 reflect the diversity of competing value systems. The first option is strategic: it consists in analyzing
572 stakeholders' interests and relationships to identify "key" participants, e.g. participants who have a
573 legitimate influence on the problem (e.g. De Marchi et al., 2000). In a landscape context, this option
574 also supposes to make clear what scale of interest drives one's value judgments (Zia et al., 2011). A
575 second option is to construct preferences archetypes as done in Ravier et al. (2015) from the statistical
576 clustering of individual preference trees. The various preferences archetypes reflect the diversity of
577 value systems without *a priori* assuming that these archetypes would coincide with socially instituted
578 categorization. We believe this method is useful in conflict-prone situations, which are often
579 encountered at the landscape level. A third option is to let participants define the group to which they
580 belong, as done by Ahrenz and Kantelhardt (2009). This way, part of the structuring bias is transferred
581 from researchers to the participants, which is more consistent with bottom-up approaches to multi-
582 actor problems.

583 (v) Difficulty integrating multiple levels in assessments
584 Given the complex nature of landscapes, one must account for multiple levels in the assessment. At the
585 landscape scale, two types of aggregation occur: vertical (among criteria, already well addressed by
586 the MCA literature) and horizontal (across space, a core component of spatial analysis). Landscape-
587 level MCAs mainly address the latter issue via spatially aggregative methods, such as weighted linear
588 combination, that are fully compensatory (i.e. increased performance at one location compensates for
589 decreased performance at another location). However, the overall performance of a landscape
590 management alternative cannot be reduced to the sum of performances of its spatial units at a given
591 moment. Such additive aggregation to combine attribute maps ignores the existence of spatial
592 relationships and patterns (Malczewski, 2000).

593 Two directions appear for reducing this gap. The first "corrects" the aggregation result by adding other
594 evaluation techniques. This option is illustrated by Fürst et al. (2013), who use landscape metrics to
595 improve assessment of esthetic and biological integrity services. The second acknowledges that "gaps"
596 between the multiple levels of assessment are irreducible; as a consequence, levels of assessment
597 should be explicitly distinguished. Though formal methods exist to identify gaps between
598 organizational levels (e.g. MuSIASEM, Giampietro et al., 2009), relying on management levels or
599 stakeholders' levels of interests can be more operational (see point iv). Using different criteria to
600 assess different levels of the system could however undermine the consistency and intelligibility of the
601 evaluation system. López-Ridaura et al. (2005) define disciplinary- and scale-independent criteria for
602 sustainability assessments: productivity, stability, reliability, resilience and adaptability. This idea gave
603 birth to the MESMIS framework (Astier et al., 2012), in which farm, community and regional
604 sustainability are evaluated with the same set of criteria but with different indicators. Similarly, Manoli
605 et al. (2005) use response properties (reliability, vulnerability, and resilience) to assess the temporal
606 variability of management performances. These options offer a generic multi-level framework but are
607 at odds with a bottom-up definition of criteria.

608 (vi) Uncertainties are under-considered

609 Finally, the question of uncertainties remains under-considered, or at best is reduced to a sensitivity
610 analysis of the weighting systems and flexible support tools. We assert that sensitivity to scenarios of
611 future change (as in Linhoss et al., 2013; Manoli et al., 2005), to resolution and spatial aggregation
612 units (Malczewski, 2000), and to group delineation, among others, can enrich the deliberative process.
613 Moreover, since judgments can change during the deliberative process, solutions to capture
614 uncertainties may also lie in the participatory settings (Garmendia and Stagl, 2010) and the research
615 posture (e.g. “co-design”, Barreteau et al., 2010). Specific mathematical procedures (e.g., fuzzy sets,

616 stochastic methods) can help address the vague or fuzzy nature of human judgments and information

617 (Ascough et al., 2008; Geneletti et al., 2003).

618
619 **CONCLUSIONS**

620 We critically examined the place of sustainability assessments within multi-criteria methods, trends in
621 methods that combine a participatory and spatial approach, and current practices for assessing
622 landscape-management alternatives through a systematic review of the literature, combining
623 quantitative textual statistics from large datasets and qualitative analysis of specific research.

624 Drawing upon the results of textual statistics, we suggest that conceptual assumptions constitute the
625 primary discriminating element of multi-criteria methods and that methodological elements should be
626 both adapted to those conceptual bases (i.e. the nature of the problem tackled), as well as operational
627 questions (i.e. the decision context). We offer insight into multi-criteria methods that differ from the
628 classic multi-attribute/multi-objective typology or aggregation procedures. We consider the
629 dichotomies of substantive/procedural rationality and grounded/positivist scientific legitimacy to
630 explain the diversity of methods and approaches. In this respect, though both are grounded in real-life
631 problems, spatial sustainability assessments and participatory sustainability assessments differ in what
632 they consider a “rational” decision-process. Spatial assessments focus on accumulating as much
633 information as possible to provide manager(s) with “all the elements” to make a substantively rational
634 decision. In contrast, participatory approaches recognize deliberation as the principal justification of
635 results. Consequently, cross-approaches are not so common.

636 When attempting to classify participatory spatial MCA methods, we noted that to assess a real
637 situation, support design of a management alternative or accompany selection of an alternative, a wide
638 variety of research postures towards society can be adopted. Unlike treatment of the participatory
639 aspect, treatment of the spatial aspect depends to some extent on the MCA goal. For landscape-
640 management issues, the method chosen should capture the diverse ways people relate to spaces and
641 locations. Scenario-selection methods have the greatest potential to accommodate spatially-explicit,
642 spatially-implicit, place-specific, overall visual and non-spatial values. However, putting this into
643 practice in a consistent and feasible way remains in an embryonic stage in the literature. The choice of
644 participants, criteria, indicators and competing alternatives can also support the integration of diverse
645 spatial values and multiple interest levels. Several initiatives for problem structuring could be
646 formalized to better address the spatiality of problems. As a result, current cases of landscape-level
647 MCAs resemble “patchworks” of different methods with distinct underlying hypotheses.

648 The main unaddressed issue of landscape-level MCAs is strikingly the question of distribution and
649 heterogeneity. How are benefits distributed in space, time, and between social actors? How do spatial
650 patterns and temporal behavior influence the overall “performance” of a management alternative?

651 These questions remain under-considered. It is implicitly assumed that increased overall value is
652 desirable, irrespective of social or spatial justice issues. Similarly, most studies assume that the
653 performance and acceptability of a management alternative only depends on the total extent of each
654 land cover or land-use type, irrespective of its shape, location, or configuration. Assessments are
655 grounded in a weak conception of sustainability and a reductionist view of the landscape. While

656 MCAs provide formal methods to overcome problems of vertical aggregation (how to aggregate
657 results for different criteria), horizontal aggregation (how to aggregate results from different locations)
658 is rarely considered. In addition, although (or maybe because) more complex aggregation algorithms
659 exist, they are not accessible or transparent to the wider public. Other important aspects of MCA for
660 landscape-management issues remain under-investigated: how stakeholders are involved, the group
661 values produced, the maps and other spatial representations valued, and the uncertainties addressed.

662 To conclude, we emphasize that MCA at the landscape-level do not succeed in addressing the joint
663 issues of spatiality and multiple views, and thus has important scientific loopholes to close. MCA is
664 nonetheless a promising method to structure landscape problems, to explore management options and
665 to foster social learning. Even with these objectives, it is necessary to reveal biases and ethical
666 positions implicitly assumed in the assessment process.

667

668 ACKNOWLEDGEMENTS

669 This review is part of a PhD research funded by the French Ministry of Higher Education and
670 Research. We warmly thank two anonymous reviewers whose comments helped improve and clarify
671 this manuscript.

672 References
673 Ahrenz, H., Kantelhardt, J., 2009. Accounting for farmers' production responses to environmental restrictions within landscape planning. Land Use Policy 26, 925-934.
674 doi:10.1016/j.landusepol.2008.11.003

676 Alessa, L., Kliskey, A., Brown, G., 2008. Social-ecological hotspots mapping: A spatial approach for identifying coupled social-ecological space. Landsc. Urban Plan. 85, 27-39.
677 doi:10.1016/j.landurbplan.2007.09.007

679 Aliyu, M., Ludin, A.N.B.M., 2015. A Review of Spatial Multi Criteria Analysis (SMCA) Methods for Sustainable Land Use Planning (SLUP). planning 2.
680

681 Allan, C., Curtis, A., 2003. Learning to implement adaptive management. Nat. Resour. Manag. 6, 25-30.
682

683 Allen, C.R., Fontaine, J.J., Pope, K.L., Garmestani, A.S., 2011. Adaptive management for a turbulent future. J. Environ. Manage., 92, 1339-1345. doi:10.1016/j.jenvman.2010.11.019

685 Anderies, J.M., Janssen, M.A., Ostrom, E., 2004. A framework to analyze the robustness of social-ecological systems from an institutional perspective. Ecol. Soc. 9, 18.
686

687 Antrop, M., 2005. Why landscapes of the past are important for the future. Landsc. Urban Plan., Rural Landscapes: past processes and future strategies 70, 21-34.
688 doi:10.1016/j.landurbplan.2003.10.002

690 Antrop, M., 2000. Background concepts for integrated landscape analysis. Agric. Ecosyst. Environ. 77, 17-28. doi:10.1016/S0167-8809(99)00089-4

692 Arciniegas, G., Janssen, R., 2012. Spatial decision support for collaborative land use planning workshops. Landsc. Urban Plan. 107, 332-342. doi:10.1016/j.landurbplan.2012.06.004

694 Arciniegas, G., Janssen, R., Omtzigt, N., 2011. Map-based multicriteria analysis to support interactive land use allocation. Int. J. Geogr. Inf. Sci. 25, 1931-1947.
695 doi:10.1080/13658816.2011.556118

697 Arciniegas, G.A., Janssen, R., 2009. Using a touch table to support participatory land use planning. 18th World Imacs Congr. Modsim09 Int. Congr. Model. Simul. Interfacing Model. Simul. Math. Comput. Sci. 2206-2212.
698
699

700 Ascough, J.C., Maier, H.R., Ravalico, J.K., Strudley, M.W., 2008. Future research challenges for incorporation of uncertainty in environmental and ecological decision-making. Ecol. Model. 219, 383-399. doi:10.1016/j.ecolmodel.2008.07.015

703 Astier, M., Garcia-Barrios, L., Galvan-Miyoshi, Y., Gonzalez-Esquivel, C.E., Masera, O.R., 2012. Assessing the Sustainability of Small Farmer Natural Resource Management Systems. A Critical Analysis of the MESMIS Program (1995-2010). Ecol. Soc. 17, 25. doi:10.5751/ES-04910-170325

707 Averna Valente, R. de O., Vettorazzi, C.A., 2008. Definition of priority areas for forest conservation through the ordered weighted averaging method. For. Ecol. Manag. 256, 1408-1417. doi:10.1016/j.foreco.2008.07.006

710 Barreteau, O., Bots, P., Daniell, K., 2010. A Framework for Clarifying Participation in Participatory Research to Prevent its Rejection for the Wrong Reasons. Ecol. Soc. 15, 22 p.

712 Bastian, O., 2001. Landscape Ecology - towards a unified discipline? Landsc. Ecol. 16, 757-766. doi:10.1023/A:1014412915534

714 Bejean, S., Midy, F., Peyron, C., 1999. La rationalité simonienne: Interprétations et enjeux épistémologiques (LATEC - Document de travail - Economie (1991-2003) No. 1999-14).
716 Bell, M.L., Hobbs, B.F., Elliott, E.M., Ellis, H., Robinson, Z., 2001. An evaluation of multi-criteria methods in integrated assessment of climate policy. J. Multi-Criteria Decis. Anal. 10, 229-256. doi:10.1002/mcda.305

719 Benoît, M., Rizzo, D., Marraccini, E., Moonen, A.C., Galli, M., Lardon, S., Rapey, H., 2012. Landscape agronomy: a new field for addressing agricultural landscape dynamics. Landsc. Ecol. 27, 1385-1394. doi:10.1007/s10980-012-9802-8

722 Berkes, F., Colding, J., Folke, C., 2002. Navigating Social-Ecological Systems: Building Resilience for Complexity and Change. Cambridge University Press.

724 Berkes, F., Folke, C., Colding, J., 2000. Linking Social and Ecological Systems: Management Practices and Social Mechanisms for Building Resilience. Cambridge University Press.

726 Bhave, A.G., Mishra, A., Raghuwanshi, N.S., 2014. A combined bottom-up and top-down approach for assessment of climate change adaptation options. J. Hydrol. 518, 150-161. doi:10.1016/j.jhydrol.2013.08.039

729 Boroushaki, S., Malczewski, J., 2010. Measuring consensus for collaborative decision-making: A GIS-based approach. Comput. Environ. Urban Syst. 34, 322-332. doi:10.1016/j.compenvurbsys.2010.02.006

732 Chakroun, H., Chabaane, Z.L., Benabdallah, S., 2015. Concept and prototype of a spatial decision support system for integrated water management applied to Ichkeul Basin, Tunisia. Water Environ. J. 29, 169-179. doi:10.1111/wej.12095

735 Choo, E.U., Schoner, B., Wedley, W.C., 1999. Interpretation of criteria weights in multicriteria decision making. Comput. Ind. Eng. 37, 527-541.

737 Costanza, R., d'Arge, R., De Groot, R., Farber, S., Grasso, M., Hannon, B., Limburg, K., Naeem, S., O'Neill, R.V., Paruelo, J., others, 1998. The value of ecosystem services: putting the issues in perspective. Ecol. Econ. 25, 67-72.

740 Cumming, G.S., 2011. Spatial resilience: integrating landscape ecology, resilience, and sustainability. Landsc. Ecol. 26, 899-909. doi:10.1007/s10980-011-9623-1

742 Cumming, G.S., Olsson, P., Chapin, F.S., Holling, C.S., 2012. Resilience, experimentation, and scale mismatches in social-ecological landscapes. Landsc. Ecol. 28, 1139-1150. doi:10.1007/s10980-012-9725-4

745 Daily, G., 1997. Nature's services: societal dependence on natural ecosystems. Island Press.

746 De Marchi, B., Funtowicz, S.O., Lo Cascio, S., Munda, G., 2000. Combining participative and institutional approaches with multicriteria evaluation. An empirical study for water issues in Troina, Sicily. Ecol. Econ. 34, 267-282. doi:10.1016/S0921-8009(00)00162-2

749 Densham, P.J., 1991. Spatial decision support systems. Geogr. Inf. Syst. Princ. Appl. 1, 403-412.

751 Folke, C., Carpenter, S., Elmqvist, T., Gunderson, L., Holling, C.S., Walker, B., 2002.

752 Resilience and Sustainable Development: Building Adaptive Capacity in a World of Transformations. AMBIO J. Hum. Environ. 31, 437-440. doi:10.1579/0044-7447-31.5.437

754 Frame, B., Brown, J., 2008. Developing post-normal technologies for sustainability. Ecol. Econ. 65, 225-241. doi:10.1016/j.ecolecon.2007.11.010

756 Frame, B., O'Connor, M., 2011. Integrating valuation and deliberation: the purposes of sustainability assessment. Environ. Sci. Policy 14, 1-10. doi:10.1016/j.envsci.2010.10.009

758 Funtowicz, S.O., Ravetz, J.R., 1994. The worth of a songbird: ecological economics as a post-
759 normal science. Ecol. Econ. 10, 197-207. doi:10.1016/0921-8009(94)90108-2

760 Funtowicz, S.O., Ravetz, J.R., 1990. Uncertainty and Quality in Science for Policy. Springer

761 Science & Business Media.

762 Fürst, C., Frank, S., Witt, A., Koschke, L., Makeschin, F., 2013. Assessment of the effects of

763 forest land use strategies on the provision of ecosystem services at regional scale. J. Environ.

764 Manage., Integrated land-use and regional resource management - A cross-disciplinary

765 dialogue on future perspectives for a sustainable development of regional resources 127,

766 Supplement, S96-S116. doi:10.1016/j.jenvman.2012.09.020

767 Fürst, C., Nepveu, G., Pietzsch, K., Makeschin, F., 2009. Pimp your Landscape: a software

768 application for interactive landscape management - potential and limitations. Rev. For.

769 Francaise 61, 21-36.

770 Garmendia, E., Gamboa, G., 2012. Weighting social preferences in participatory multi-criteria

771 evaluations: A case study on sustainable natural resource management. Ecol. Econ., The

772 Economics of Degrowth 84, 110-120. doi:10.1016/j.ecolecon.2012.09.004

773 Garmendia, E., Stagl, S., 2010. Public participation for sustainability and social learning:

774 Concepts and lessons from three case studies in Europe. Ecol. Econ. 69, 1712-1722.

775 doi:10.1016/j.ecolecon.2010.03.027

776 Gasparatos, A., El-Haram, M., Homer, M., 2008. Critical review of reductionist approaches

777 for assessing the progress towards sustainability. Environ. Impact Assess. Rev. 28, 286-311.

778 doi:10.1016/j.eiar.2007.09.002

779 Geneletti, D., Beinat, E., Chung, C.F., Fabbri, A.G., Scholten, H.J., 2003. Accounting for

780 uncertainty factors in biodiversity impact assessment: lessons from a case study. Environ.

781 Impact Assess. Rev. 23, 471-487.

782 Giampietro, M., 2003. Multi-Scale Integrated Analysis of Agroecosystems. CRC Press.

783 Giampietro, M., Mayumi, K., Ramos-Martin, J., 2009. Multi-scale integrated analysis of

784 societal and ecosystem metabolism (MuSIASEM): Theoretical concepts and basic rationale.

785 Energy, WESC 2006 Advances in Energy Studies 6th World Energy System Conference 5th

786 workshop on Advances, Innovation and Visions in Energy and Energy-related Environmental

787 and Socio-Economic Issues 34, 313-322. doi:10.1016/j.energy.2008.07.020

788 Gibson, C.C., Ostrom, E., Ahn, T.K., 2000. The concept of scale and the human dimensions of

789 global change: a survey. Ecol. Econ. 32, 217-239. doi:10.1016/S0921-8009(99)00092-0

790 Gómez-Sal, A., Belmontes, J.-A., Nicolau, J.-M., 2003. Assessing landscape values: a

791 proposal for a multidimensional conceptual model. Ecol. Model., Landscape Theory and

792 Landscape Modelling 168, 319-341. doi:10.1016/S0304-3800(03)00144-3

793 Greene, R., Luther, J.E., Devillers, R., Eddy, B., 2010. An approach to GIS-based multiple

794 criteria decision analysis that integrates exploration and evaluation phases: Case study in a

795 forest-dominated landscape. For. Ecol. Manag. 260, 2102-2114.

796 doi:10.1016/j.foreco.2010.08.052

797 Groot, J.C.J., Rossing, W.A.H., 2011. Model-aided learning for adaptive management of

798 natural resources: an evolutionary design perspective. Methods Ecol. Evol. 2, 643-650.

799 doi:10.1111/j.2041-210X.2011.00114.x

800 Guitouni, A., Martel, J.-M., 1998. Tentative guidelines to help choosing an appropriate

801 MCDA method. Eur. J. Oper. Res. 109, 501-521. doi:10.1016/S0377-2217(98)00073-3

802 Hajkowicz, S., Collins, K., 2006. A Review of Multiple Criteria Analysis for Water Resource
803 Planning and Management. Water Resour. Manag. 21, 1553–1566. doi:10.1007/s11269-006-

804 9112-5

805 Hein, L., van Koppen, K., de Groot, R.S., van Ierland, E.C., 2006. Spatial scales, stakeholders

806 and the valuation of ecosystem services. Ecol. Econ. 57, 209–228.

807 doi:10.1016/j.ecolecon.2005.04.005

808 Holling, C. s., Meffe, G.K., 1996. Command and Control and the Pathology of Natural

809 Resource Management. Conserv. Biol. 10, 328–337. doi:10.1046/j.1523-

810 1739.1996.10020328.x

811 Holling, C.S., 2001. Understanding the Complexity of Economic, Ecological, and Social

812 Systems. Ecosystems 4, 390–405. doi:10.1007/s10021-001-0101-5

813 Hunziker, M., Felber, P., Gehring, K., Buchecker, M., Bauer, N., Kienast, F., 2008. Evaluation

814 of Landscape Change by Different Social Groups. Mt. Res. Dev. 28, 140–147.

815 doi:10.1659/mrd.0952

816 Janssen, R., Goosen, H., Verhoeven, M.L., Verhoeven, J.T.A., Omtzigt, A.Q.A., Maltby, E.,

817 2005. Decision support for integrated wetland management. Environ. Model. Softw. 20, 215–

818 229. doi:10.1016/j.envsoft.2003.12.020

819 Kallis, G., Norgaard, R.B., 2010. Coevolutionary ecological economics. Ecol. Econ., Special

820 Section: Coevolutionary Ecological Economics: Theory and Applications 69, 690–699.

821 doi:10.1016/j.ecolecon.2009.09.017

822 Kiker, G.A., Bridges, T.S., Varghese, A., Seager, P.T.P., Linkov, I., 2005. Application of

823 multicriteria decision analysis in environmental decision making. Integr. Environ. Assess.

824 Manag. 1, 95–108. doi:10.1897/IEAM_2004a-015.1

825 Kinzig, A.P., Ryan, P.A., Etienne, M., Allison, H.E., Elmqvist, T., Walker, B.H., 2006.

826 Resilience and regime shifts: assessing cascading effects. Ecol. Soc. 11.

827 Le Bellec, F., Rajaud, A., Ozier-Lafontaine, H., Bockstaller, C., Malezieux, E., 2012.

828 Evidence for farmers’ active involvement in co-designing citrus cropping systems using an

829 improved participatory method. Agron. Sustain. Dev. 32, 703–714. doi:10.1007/s13593-011-

830 0070-9

831 Lebart, L., Salem, A., Berry, L., 1998. Exploring Textual Data, Text, Speech and Language

832 Technology. Springer Netherlands, Dordrecht.

833 Linhoss, A.C., Kiker, G.A., Aiello-Lammens, M.E., Chu-Agor, M.L., Convertino, M., Muñoz-

834 Carpena, R., Fischer, R., Linkov, I., 2013. Decision analysis for species preservation under

835 sea-level rise. Ecol. Model. 263, 264–272. doi:10.1016/j.ecolmodel.2013.05.014

836 Linkov, I., Bridges, T., Creutzig, F., Decker, J., Fox-Lent, C., Kröger, W., Lambert, J.H.,

837 Levermann, A., Montreuil, B., Nathwani, J., others, 2014. Changing the resilience paradigm.

838 Nat. Clim. Change 4, 407–409.

839 Linkov, I., Satterstrom, F.K., Kiker, G.A., Bridges, T.S., Benjamin, S.L., Belluck, D.A., 2006.

840 From optimization to adaptation: Shifting paradigms in environmental management and their

841 application to remedial decisions. Integr. Environ. Assess. Manag. 2, 92–98.

842 doi:10.1002/ieam.5630020116

843 Liu, J., Dietz, T., Carpenter, S.R., Alberti, M., Folke, C., Moran, E., Pell, A.N., Deadman, P.,

844 Kratz, T., Lubchenco, J., Ostrom, E., Ouyang, Z., Provencher, W., Redman, C.L., Schneider,

845 S.H., Taylor, W.W., 2007. Complexity of Coupled Human and Natural Systems. Science 317,

846 1513–1516. doi:10.1126/science.1144004
847 López-Ridaura, S., Keulen, H.V., Ittersum, M.K. van, Leffelaar, P.A., 2005. Multiscale

848 Methodological Framework to Derive Criteria and Indicators for Sustainability Evaluation of

849 Peasant Natural Resource Management Systems. Environ. Dev. Sustain. 7, 51–69.

850 doi:10.1007/s10668-003-6976-x

851 Malczewski, J., 2006. GIS-based multicriteria decision analysis: a survey of the literature. Int.

852 J. Geogr. Inf. Sci. 20, 703–726. doi:10.1080/13658810600661508

853 Malczewski, J., 2000. On the use of weighted linear combination method in GIS: common

854 and best practice approaches. Trans. GIS 4, 5–22.

855 Malczewski, J., 1999. GIS and Multicriteria Decision Analysis. John Wiley & Sons.

856 Malczewski, J., Rinner, C., 2015. Multicriteria decision analysis in geographic information

857 science. Springer.

858 Manoli, E., Katsiardi, P., Arampatzis, G., Assimacopoulos, D., 2005. Comprehensive water

859 management scenarios for strategic planning, in: Lekkas, T.D. (Ed.), Proceedings of the 9th

860 International Conference on Environmental Science and Technology, Vol A - Oral

861 Presentations, Pts A and B. pp. A913–A920.

862 Martinez-Alier, J., Munda, G., O'Neill, J., 1998. Weak comparability of values as a

863 foundation for ecological economics. Ecol. Econ. 26, 277–286. doi:10.1016/S0921-

864 8009(97)00120-1

865 McFadden, D., 1994. Contingent valuation and social choice. Am. J. Agric. Econ. 76, 689–

866 708.

867 Munda, G., 2004. Social multi-criteria evaluation: Methodological foundations and

868 operational consequences. Eur. J. Oper. Res. 158, 662–677. doi:10.1016/S0377-

869 2217(03)00369-2

870 Munda, G., Nijkamp, P., Rietveld, P., 1994. Qualitative multicriteria evaluation for

871 environmental management. Ecol. Econ. 10, 97–112.

872 Naveh, Z., 2000. What is holistic landscape ecology? A conceptual introduction. Landsc.

873 Urban Plan. 50, 7–26. doi:10.1016/S0169-2046(00)00077-3

874 Neumayer, E., 2003. Weak Versus Strong Sustainability: Exploring the Limits of Two

875 Opposing Paradigms. Edward Elgar Publishing.

876 Newton, A.C., Hodder, K., Cantarello, E., Perrella, L., Birch, J.C., Robins, J., Douglas, S.,

877 Moody, C., Cordingley, J., 2012. Cost-benefit analysis of ecological networks assessed

878 through spatial analysis of ecosystem services. J. Appl. Ecol. 49, 571–580.

879 doi:10.1111/j.1365-2664.2012.02140.x

880 Nitschelm, L., Aubin, J., Corson, M.S., Viaud, V., Walter, C., 2016. Spatial differentiation in

881 Life Cycle Assessment LCA applied to an agricultural territory: current practices and method

882 development. J. Clean. Prod. 112, Part 4, 2472–2484. doi:10.1016/j.jclepro.2015.09.138

883 Nordstrom, E.-M., Eriksson, L.O., Ohman, K., 2011. Multiple Criteria Decision Analysis with

884 Consideration to Place-specific Values in Participatory Forest Planning. Silva Fenn. 45, 253–

885 265. doi:10.14214/sf.116

886 Nordström, E.-M., Eriksson, L.O., Öhman, K., 2010. Integrating multiple criteria decision

887 analysis in participatory forest planning: Experience from a case study in northern Sweden.

888 For. Policy Econ. 12, 562–574. doi:10.1016/j.forpol.2010.07.006

889 Olsson, P., Gunderson, L.H., Carpenter, S.R., Ryan, P., Lebel, L., Folke, C., Holling, C.S.,
890 2006. Shooting the rapids: navigating transitions to adaptive governance of social-ecological
891 systems. Ecol. Soc. 11, 18.

892 Ostrom, E., 2009. A General Framework for Analyzing Sustainability of Social-Ecological
893 Systems. Science 325, 419-422. doi:10.1126/science.1172133

894 Pedroli, B., Pinto-Correia, T., Cornish, P., 2006. Landscape – What’s in it? Trends in
895 European Landscape Science and Priority Themes for Concerted Research. Landsc. Ecol. 21,
896 421-430. doi:10.1007/s10980-005-5204-5

897 Plummer, R., 2009. The adaptive co-management process: an initial synthesis of
898 representative models and influential variables. Ecol. Soc. 14, 24.

899 Raaijmakers, R., Krywkow, J., van der Veen, A., 2008. Flood risk perceptions and spatial
900 multi-criteria analysis: an exploratory research for hazard mitigation. Nat. Hazards 46, 307-
901 322. doi:10.1007/s11069-007-9189-z

902 Ramsey, K., 2009. GIS, modeling, and politics: On the tensions of collaborative decision
903 support. J. Environ. Manage., Collaborative GIS for spatial decision support and visualization
904 90, 1972-1980. doi:10.1016/j.jenvman.2007.08.029

905 Ratinaud, P., Marchand, P., 2012. Application de la méthode ALCESTE à de «gros» corpus et
906 stabilité des «mondes lexicaux»: analyse du «CableGate» avec IRaMuTeQ. Actes 11e Journ.
907 Int. Anal. Stat. Données Textuelles JADT 2012.

908 Ravier, C., Prost, L., Jeuffroy, M.-H., Wezel, A., Paravano, L., Reau, R., 2015. Multi-criteria
909 and multi-stakeholder assessment of cropping systems for a result-oriented water quality
910 preservation action programme. Land Use Policy 42, 131-140.
911 doi:10.1016/j.landusepol.2014.07.006

912 Reichert, P., Langhans, S.D., Lienert, J., Schuwirth, N., 2015. The conceptual foundation of
913 environmental decision support. J. Environ. Manage. 154, 316-332.
914 doi:10.1016/j.jenvman.2015.01.053

915 Reinert, M., 1993. Les "mondes lexicaux" et leur "logique" à travers l'analyse statistique d'un
corpus de récits de cauchemars." Lang. Société 66, 5-39. doi:10.3406/lsoc.1993.2632

917 Sahin, O., Mohamed, S., 2013. A spatial temporal decision framework for adaptation to sea
918 level rise. Environ. Model. Softw. 46, 129-141. doi:10.1016/j.envsoft.2013.03.004

919 Sheppard, S.R.J., Meitner, M., 2005. Using multi-criteria analysis and visualisation for
920 sustainable forest management planning with stakeholder groups. For. Ecol. Manag., Decision
921 Support in Multi Purpose Forestry Decision Support in Multi Purpose Forestry Selected papers
922 from the symposium on "Development and Application of Decision Support Tools in Multiple
923 Purpose Forest Management" 207, 171-187. doi:10.1016/j.foreco.2004.10.032

924 Simon, H.A., 1990. Alternative visions of rationality, in: Moser, P.K. (Ed.), Rationality in
925 Action: Contemporary Approaches. Cambridge University Press, pp. 189-204.

926 Simon, H.A., 1976. From substantive to procedural rationality, in: Kastelein, T.J., Kuipers,
927 S.K., Nijenhuis, W.A., Wagenaar, G.R. (Eds.), 25 Years of Economic Theory. Springer US,
928 pp. 65-86.

929 Swedeen, P., 2006. Post-normal science in practice: A Q study of the potential for sustainable
930 forestry in Washington State, USA. Ecol. Econ. 57, 190-208.
931 doi:10.1016/j.ecolecon.2005.04.003

932 van Asselt, M., Rijkens-Klomp, N., 2002. A look in the mirror: reflection on participation in
933 Integrated Assessment from a methodological perspective. Glob. Environ. Change 12, 167-
934 184. doi:10.1016/S0959-3780(02)00012-2

935 van Herwijnen, M., Janssen, R., 2007. The use of multi-criteria evaluation in spatial policy,
936 in: Asian Conference on Remote Sensing. Kuala Lumpur.

937 Vatn, A., 2009. An institutional analysis of methods for environmental appraisal. Ecol. Econ.

938 68, 2207-2215. doi:10.1016/j.ecolecon.2009.04.005

939 Vatn, A., 2005. Institutions and the Environment. Edward Elgar Publishing.

940 Vodoz, L., 1994. La prise de décision par consensus: pourquoi, comment, à quelles
941 conditions. Environ. Société 13, 57-66.

942 Walker, B., Holling, C.S., Carpenter, S.R., Kinzig, A., 2004. Resilience, adaptability and
943 transformability in social-ecological systems. Ecol. Soc. 9, 5.

944 Wilson, J., Low, B., Costanza, R., Ostrom, E., 1999. Scale misperceptions and the spatial
945 dynamics of a social-ecological system. Ecol. Econ. 31, 243-257. doi:10.1016/S0921-
946 8009(99)00082-8

947 Wu, J. (Jingle), 2006. Landscape Ecology, Cross-disciplinarity, and Sustainability Science.
948 Landsc. Ecol. 21, 1-4. doi:10.1007/s10980-006-7195-2

949 Xenarios, S., Tziritis, I., 2007. Improving pluralism in Multi Criteria Decision Aid approach
950 through Focus Group technique and Content Analysis. Ecol. Econ. 62, 692-703.
951 doi:10.1016/j.ecolecon.2006.08.017

952 Zia, A., Hirsch, P., Songorwa, A., Mutekanga, D.R., O'Connor, S., McShane, T., Brosius, P.,
953 Norton, B., 2011. Cross-Scale Value Trade-Offs in Managing Social-Ecological Systems: The
954 Politics of Scale in Ruaha National Park, Tanzania. Ecol. Soc. 16. doi:10.5751/ES-04375-
955 160407

956