# Modeling for Policy Formulation: Causal Mapping, Scenario Generation, and Decision Evaluation

Aron Larsson, Osama Ibrahim

<figure><img src="image_1.png" /></figure>

## To cite this version:

Aron Larsson, Osama Ibrahim. Modeling for Policy Formulation: Causal Mapping, Scenario Generation, and Decision Evaluation. 7th International Conference on Electronic Participation (ePart), Aug 2015, Thessaloniki, Greece. pp.135-146, 10.1007/978-3-319-22500-5_11. hal-01587633

HAL Id: hal-01587633

https://inria.hal.science/hal-01587633

Submitted on 14 Sep 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Modeling for Policy Formulation: Causal Mapping, Scenario Generation, and Decision Evaluation

Aron Larsson<fnref n="1" />,<fnref n="2" />, and Osama Ibrahim<fnref n="1" />

**Abstract.** In this paper we present a work process with associated operational research modeling and analysis tools for the policy formulation stage of the Lindblom policy cycle process model. The approach exploits the use of causal maps for problem structuring and scenario generation of policy options together with decision analysis for evaluating generated scenarios taking preferences of decision makers and stakeholders into account. The benefits of interest when exploiting this integrated modeling approach is to enable for; i) problem structuring and facilitating understanding and communication of a complex policy problem, ii) simulation of policy consequences and identification of a smaller set of policy options from a possible very large set of possible options, and iii) structured decision evaluation of the generated alternative policy options.

**Keywords:** Policy Analysis, Impact assessment, Policy Modelling, Problem Structuring, Dynamic Simulation, Causal Mapping, Scenario Planning, Decision Analysis

## 1 Introduction

A large body of public policy analysis is devoted to retrospective analysis (evaluation), which tries to understand the causes and consequences of policies after they have been implemented (Tsoukias et al. 2013). Equally important in policy analysis is the role of ex ante evaluations, i.e. prescriptive analysis involving impact assessment carried out at the early stages of policy development, cf., e.g., (Turnpenny et al. 2009). Ex ante evaluation encompasses forecasting of consequences if policies were to be implemented and prescriptions about which policies should be implemented. One important aim of ex ante decision support within the context of policy making therefore involve developing ways of facilitating for policy-makers to create policies that is consistent with their preferences while at the same time being accepted by other stakeholders, cf. (Bryson 2007). However, there is a lack of operational approaches for addressing the cognitive activity of designing or finding policy options. The long term implications of policy making imply the need to consider a vast range of possible futures, often characterized by the presence of multiple stakeholders and sometimes characterized by large uncertainties as well.

<footnote n="1">¹Dept. of Computer and Systems Sciences, Stockholm University</footnote>

<footnote n="2">²Dept. of Information and Communications Systems, Mid Sweden University</footnote>

<footnote>`{aron, osama}@dsv.su.se`</footnote>
Problem structuring are now widely acknowledged as an important feature of strategic decision support tools and there is a growing but still small body of research and practice on how to integrate such methods with other formal and/or quantitative methods for decision evaluation, see (Tsoukiàs et al. 2013). The aim of this paper is therefore to provide a basis for the selection and use of modeling and decision support tools from the operational research field in a common toolkit for ex ante policy analysis, such that there is a logical relationship between the tool used to the stage and sub-stage of the policy making process and how the modeling tools provide decision value in the policy making process.

## 2 Policy Making Process and Policy Formulation

Herein public policy is defined as a purposeful, goal-oriented action that is taken by government to deal with societal problems or to improve societal conditions for the wellbeing of its population. Decision making is the cognitive process of selecting a course of action from many possible ones which might be initially known or not known. According to Bero and Jadad (1997), policy making has some distinguishing features having an impact on the decision process:

(i) A population level decision making context;

(ii) Explicit justification is required, as policies are formally and informally evaluated by government agencies, by outside consultants, by interest groups, by the mass media, and by the public;

(iii) Effect of the existing political ideology and governance; and

(iv) Evidence of systematic reviews on public policy decisions is hard to come by.

We relate to three-stage process model for policy-making conforming to the well-established policy cycle models of Lindblom (1968) and Howlett et al. (2009). The process model includes the following steps; i) problem identification stage, ii) policy formulation stage, and iii) policy implementation stage, see Fig. 1.

<figure><img src="image_3.png" /><figcaption>Fig. 1. Policy-making process model.</figcaption></figure>
This process model acknowledges that policy-making is an ongoing, continuous process that requires continuous assessment, evaluation, and reaction. Each stage has a set of activities according to Table 1 below.

<table><caption>**Table 1.** Stages and sub-stages of the policy-making process model.</caption><thead><tr><td>Stage</td><td>Activity</td></tr></thead><tbody><tr><td>1. Problem Identification</td><td><strong>Recognition and diagnosis:</strong><br>Decision makers become aware of the fact that there is a problem and diagnosis is required to order and combine the information related to that problem.<br><br><strong>Agenda Setting:</strong><br>Focusing attention, involving interest groups, setting targets, defining and documenting the long view and/or the short view of the policy.</td></tr><tr><td>2. Policy Formulation</td><td><strong>Problem analysis:</strong><br>Identify elements of policy problem: actors, stakeholders, decision variables, parameters, links, goals, risks, limitations and associated technologies.<br>Identify problem environment: conflicting goals, inter- and intra-group negotiations. Specify decision dynamics: interrupts, feedback loops, delays, and speedups.<br><br><strong>Design and generation of policy options:</strong><br>Formulate policy proposals through political channels by policy planning organisations, interest groups and government bureaucracies. Identify the feasible options that will or might lead to desired policy consequences, conduct impact assessments of different policy options.<br><br><strong>Evaluation of policy options/alternatives:</strong><br>Compare the feasible options, considering that conflicting goals are likely to exist as well as differing acceptance levels for different stakeholders.<br><br><strong>Policy Decision:</strong><br>Decide upon policy after evaluation of the options encompassing different viewpoints and perspectives, multiple objectives, and multiple stakeholders using integrated assessments.</td></tr><tr><td>3. Policy Implementation</td><td><strong>Legitimizing and implementing a public policy:</strong><br>Policy is legitimized as a result of the public statements or actions of government officials at all levels. This includes: executive orders, rules, regulations, laws, budgets, appropriations, decisions and interpretations that have the effect of setting policy direction. Policy is implemented through the activities of public bureaucracies and the expenditure of public funds.</td></tr></tbody></table>
<table><tr><td></td><td><strong>Monitoring & Evaluation:</strong><br>Record and control of the implementation. Analysis, evaluation and feedback of the results of implementation.</td></tr></table>

Hence, a large part of the activities occurring within a policy cycle is about understanding, formulating and structuring “problems”, see also (Mingers and Rosenhead 2004; Tsoukiàs 2007). Thus, problem structuring is a key element of the public policy making process and in particular for the policy formulation stage. Modeling for problem structuring in order to find feasible policy options and decision evaluation of these options is of prime concern in the policy formulation stage, since in this stage “policy makers must ensure that there is a meaningful definition of the problem” and “need a comprehensive understanding of the problem” in order to identify different policy options (Hamilton 2010). Having this said, in the next section we focus on operational research and the decision sciences within the context of supporting policy formulation and a selection of methods for an integrated modeling approach to policy formulation decision support is suggested.

# 3 Modeling for Policy Formulation

## 3.1 Problem Analysis

Problem analysis calls for problem structuring methods (PSM) which aim to enable a better understanding of unstructured problems characterized by the existence of multiple actors, multiple perspectives, incommensurable and/or conflicting interests, intangibles and key uncertainties. The methods rely heavily on engaging with policy makers, adopting a facilitative mode of engagement, and simple, often qualitative, models (Franco and Montibeller 2010). See (Mingers and Rosenhead 2001) for an overview of the characteristics of a wide range of PSM’s.

The use of cognitive maps is among the early approaches to enable for problem understanding, originally intended for representing social science knowledge, see (Axelrod 1976). A cognitive map is a directed graph where nodes represents societal concepts or variables (such as subsidy size, traffic intensity, CO₂ emission rates) and links represent causal connections between these concepts. With respect to the simple example, two nodes “car traffic” and “emissions” and a link from the former to the latter would then mean that the former causally has an impact on the latter, negative or positive.

Multiple perspectives of the problem from different actors can be represented and debated for improving the policy-makers’ understanding of the problem situation and sharing it with associates. Several “dialects” of cognitive maps have emerged, which share focus on the qualitative aspects of problem structuring, see, e.g., (Acar and Druckenmiller 2006).

Although a cognitive map serves its purpose as a problem structuring method, its content is too informal to be used as an underlying model for simulation of how changes in one node affects other nodes through causal relations. A causal map in the sense of Acar (1983) is a diagram that maps problem variables, or factors, and defines their
interdependencies in a quantitative manner. Goals for different actors of the problem can be defined to facilitate group decision making and negotiation analysis. The primitive elements used in the causal mapping method are: Independent variables (sources of change), dependent variables (intermediate and outcome/goal variables), change transmission channels, change transfer coefficients, time lags, minimum thresholds (minimum change to be transferred), status-quo level of the map (the initial state of the system, considering 0% relative change), and goal vector representing the targeted changes in goal variables relative to status-quo values.

<figure><img src="image_4.png" /><figcaption>**Fig. 2.** Causal map model of bus ticket price subsidy policy problem with four actors, three controlling resources on the left hand side and one actor having an interest in the town commerce factor. The goal vector consists of that the three controlling actors want to reduce CO₂ emissions by 20% while the third actor wants to increase town commerce with 5%. Each link has a change transfer coefficient and a time lag. For instance, the link from “Subsidy bus tickets” into “Car traffic” has a coefficient of -0.2 and a time lag of 1 month.</figcaption></figure>

As a simple example, consider a local government wanting to reduce CO₂ emissions in order to contribute to climate targets and improve central town air quality. The upfront objectives are focused on the environmental issue. One policy option is to increase the subsidy of bus tickets, where the underlying hypothesis in terms of causality is that higher subsidies (lower ticket prices) will cause more people to take the bus instead of cars when going into town, thereby reducing emissions also enabling for more citizens to access the central town area increasing town commerce, which is the main concern of an influential stakeholder group. However, there is also a positive causal relationship between car traffic and town commerce, increasing the complexity of the policy option making. In the problem analysis, it becomes clear that for many citizens it is not the bus ticket price that cause them to avoid taking the bus, but rather
the frequency of buses. Further, reducing emissions can also be done by increasing the proportion of gas driven buses. A causal map of the policy problem can then be illustrated in Fig. 2 above.

The richer computational semantics of the causal mapping approach support simulation in ways that other varieties of cognitive mapping and causal mapping do not, see (Druckenmiller and Acar 2009). The method includes indications not only of the signs of the presumed causal influences, but also of their intensities, minimum threshold values and the time lags. Still, it is possible to utilize the non-quantitative elements of the map analogously to how cognitive maps are employed. Three classes of analyses are supported, see (Acar 1983) for detailed description:

**1- Backward analysis:**

Clarifying, testing and reassessing assumptions about the cause-effect relationships of the situation. It can be divided into:

(a) Major assumptional analysis (validity of the major aspects of the graph elements and relationships); and

(b) Minor assumptional analysis (validity of detailed qualifications and quantifications of the graph).

**2- Structural analysis:**

It includes: Graph scope, connectivity analysis, reachability analysis and goal comparative analysis (qualitative and quantitative).

**3- Forward analysis:**

Implications of a “change scenario” through simulating the transfer of change, includes:

(a) Scenario simulation: running change scenarios on the graph – simulating transfer of change over time from origins throughout the map.

(b) Goal negotiation analysis: Goal feasibility and compatibility of goals, identify scenarios realizing a goal or goals jointly). Effectiveness or efficiency – expression of outcomes in comparison with objectives and relative to change needed at origins.

Within policy formulation, a policy option would be represented by a change scenario. Feasible scenarios would be the ones who satisfy the actors by reaching their goals in different ways, and in this way scenarios can be generated/discovered (Bryant and Lempert 2010). The method cannot capture all the intricacies of a situation to its minutest details, but can sufficiently map out its principal elements and their relationships.

**3.2 Design and Generation of Policy Options**

The identification and creation of new alternatives is one of the most important aspects of any decision support. If the decision alternatives under consideration are weak, it will lead to a poor choice (Brown 2005). Thus, support in the generation of feasible options is important for policy formulation.

A policy option is represented by a scenario of change from the status-quo level of the causal map. A base line scenario is defined with initial values for the problem's key variables with zero initial relative changes. The desired state of the system is represented by a goal vector (targeted relative changes in outcome variables compared to the
base line scenario). The causal map allows triggering change transfer by a 'Pure scenario', a single change at one source, or a 'Mixed scenario', change in several sources all at once or with a time lag.

Graph change analysis allows us to investigate the dynamic consequences of entering a change in one of the graph origins, thus simulating the propagation of change throughout the causal map. Several decision analytic concepts are readily usable for scenario generation:

* "Optimality", relating scenarios to the level or degree to which the goal vector is realised.

* "Dominance", reducing the number of scenarios by discarding dominated ones.

* "Resource constraint", cost of input triggers of certain magnitude at a node. Triggering a scenario may impose on the controlling actor the supply of funds and resources.

* "Goal compatibility", goal vectors of different actors is compatible if a scenario can be found to realize them jointly.

<figure><img src="image_5.png" /><figcaption>**Fig. 3.** Example of a generated scenario reaching both the emission and the town commerce target. The scenario consists of increasing the subsidy with 15% at the initial timestep, increasing bus frequencies at the sixth, and increasing the proportion of gas driven buses at timestep 12.</figcaption></figure>

The simulation is run upon the causal map modeled, whereas the set of objectives and their target values are used for identifying feasible scenarios. Based on the simulation results of an initially large sample of scenarios generated by, e.g., full factorial design or Latin hypercube sampling, unsatisfactory scenarios not realizing the goal vec-
tor or being dominated are filtered out, while scenarios deemed efficient and “interest-
ing” according to some predefined decision rule based upon resource constraints and
goal compatibility concepts mentioned above are suggested as policy options for further
evaluation.

3.3 Evaluation of Policy Options

Arriving at a set of feasible options which are non-dominated, further discrimination
between them will call for taking preference information into account. Although, the
scenario-based dynamic simulation allows decision makers to identify alternative pol-
icy options and verify their effects over time, it does not provide the explicit decision
evaluation of policy options. Multi-criteria decision analysis (MCDA) has been exten-
sively used to support a wide variety of complex decision problems as a tool for evalu-
ating options where decisions involve the achievement of multiple objectives and con-
sidering multiple decision makers and stakeholders, see, e.g., (Zavadskas et al. 2014).
MCDA typically distinguish between a decision maker which has control of some de-
cision variable, and stakeholders, which are affected by the consequences of a decision.
This distinction is straightforwardly analogous to the actor representations of the causal
map. Applying MCDA should thus provide the relative global performance of each
alternative, and is particularly useful when selecting one out of a finite set of feasible
alternatives.

When evaluation of policy options is guided by a decision analysis approach, a relevant MCDA model is required equipped with preference elicitation methods for capturing policy makers’ and stakeholders’ preferences. Given that we have identified feasible policy options from scenario generation, there are two main tasks remaining in structuring MCDA evaluation models; i) representation of objectives in a structure, commonly a value tree, and ii) the definition of attributes to measure the achievement of objectives (Franco and Montibeller 2011).

Traditionally in MCDA, the decision process starts by structuring the problem as
an attribute tree hierarchically ordering the decision makers' aims at different abstrac-
tion levels, from fundamental objectives (such as “improved environment”) to lower
level attributes (such as “NOₓ reduction”) where the latter contributes to the former in
a hierarchical value tree. It is generally assumes that each criterion can be operational-
ized by a set of measurable attributes allowing for assessing the consequences arising
from the implementation of any particular alternative. In the next step preferential in-
formation is elicited. The relative importance of criteria is captured in weights for each
criterion at each abstraction level. At the lowest level of the value tree these objectives
are translated into attributes, with each one of them evaluating a given characteristic of
the decision options (for example, an objective ‘efficiency’ may be measured by the
attribute ‘operating cost’). The performance of each decision option against each attrib-
ute is determined and weights reflecting acceptable trade-offs of performance among
objectives are elicited from the decision-makers. Given the presence of many decision
makers or stakeholders, for a group decision a decision has to be made and there might
be disagreement with respect to what the best option is, but the group choice still has
sufficient support from the group, i.e. there is a way of representing the collective pref-
erences of the group. This differs from negotiation in that in such as process each stakeholder might simply abandon the decision process and there is no decision at all, see, e.g., (Kilgour et al. 2010).

There is no universal way of applying MCDA to group decision and negotiation processes, but the underlying use of a causal map and scenarios generated leads to that the features of interest are preference elicitation and preference aggregation. With respect to integrating multi-criteria decision analysis with causal maps and simulation of scenarios, Comes et al. (2011) presents the concept of “decision maps”, integrating problem structuring and scenario planning using causal maps with multi-criteria decision analysis, but keeps the focus on operational decision making and to a lesser extent on policy making. However, the structure and content of the map informs the building of the value tree, in an ad hoc translation, in which, the causal map can be used to elicit preferences, cf. (Comes et al. 2011). For a simple example of this, see Fig. 5 showing a value tree model based upon the causal map of Fig. 4. Montibeller and Belton (2006) investigate various ways to use causal maps as the underlying problem structuring tool and extending it with decision evaluation features and/or using the map to effectively inform the decision analysis model in the form of a multi attribute value tree.

<figure><img src="image_6.png" /><figcaption>**Fig. 4.** Value tree counterpart of causal map in Fig. 3 in accordance to the SEMPAI framework. In this example, DM Subsidy's weight for CO2 emissions is 40% so for simplicity precise numerical weights are used here.</figcaption></figure>

For simulation, in the SEMPAI<fnref n="1" /> framework reported in Hansson et al. (2011), simulation results of flooding models are combined with multi-criteria decision analysis

<footnote n="1">¹ Simulation and Evaluation with Multiple Perspectives and Agents Integrated</footnote>
where multiple stakeholders are present. Elicitation or anticipation of stakeholder preferences is done or assessed in the form of utility statements for each stakeholder or stakeholder group regarding each specific policy option and criteria weights. Stakeholders give the different outcomes a ranking order if they are unsure of their preferred choice and methods for computational decision analysis with imprecise information is promoted in order to support such statements, see (Larsson et al. 2005; Danielson et al. 2007). The result is a preference assessment for each decision maker and/or stakeholder and the value tree is constructed with stakeholders as the lowest level of the tree, see Fig. 4 for the value tree corresponding to the causal map of Fig. 3 where each decision maker has a trade-off expressed as criteria weights between cost and their benefits of concern. With respect to the model in Fig. 4, we can consider the following simple preference scheme using value functions $V(S)$ of the scenarios for each criterion delimiting the decision evaluation to the status-quo scenario called $S_0$ and the generated scenario shown in Fig. 3 called $S_1$.

<table><caption>**Table 2.** Preference schema.</caption><thead><tr><th rowspan="2">Decision maker</th><th colspan="2">CO<sub>2</sub> emissions</th><th colspan="2">Cost</th></tr><tr><th>V(S<sub>0</sub>)</th><th>V(S<sub>1</sub>)</th><th>V(S<sub>0</sub>)</th><th>V(S<sub>1</sub>)</th></tr></thead><tbody><tr><td>DM Subsidy</td><td>0</td><td>1</td><td>1</td><td>0</td></tr><tr><td>DM Bus frequency</td><td>0</td><td>1</td><td>1</td><td>0</td></tr><tr><td>DM Gas driven buses</td><td>0</td><td>1</td><td>1</td><td>0</td></tr></tbody></table>

<table><thead><tr><th rowspan="2">Stakeholder</th><th colspan="2">Town commerce</th></tr><tr><th>V(S<sub>0</sub>)</th><th>V(S<sub>1</sub>)</th></tr></thead><tbody><tr><td>ST Town commerce</td><td>0</td><td>1</td></tr></tbody></table>

Decision evaluation is then done stakeholder-wise. Each decision maker or stakeholder will receive a utility value for each scenario and the choice of selecting or discarding one scenario will be based upon its total utility through aggregating the stakeholders' utility values (see Fig. 5) together with group decision admissibility concepts such as maximum disagreement thresholds or minimum consensus thresholds (Fasth et al. 2013).

<figure><img src="image_7.png" /><figcaption>**Fig. 5.** Comparison of scenarios generated by adding part-worth utilities for each stakeholder. There is disagreement since DM Subsidy and DM Gas driven buses consider $S_1$ to be inferior to $S_0$ while the other actors have differing preferences.</figcaption></figure>
# 4 Concluding Remarks

In this paper we have outlined the linkages between the policy formulation stage of the policy making process model and contemporary decision analytic support methodologies and based upon that discussed an integrated modeling framework for computer assisted policy formulation. The approach suggested combines causal mapping for problem structuring and simulation of consequences together with identifying scenarios deemed feasible for further investigation in a multi-criteria decision evaluation. This sequence follows a logical work process conforming to the activities of the policy formulation process.

Scenario generation helps policy-makers in identifying feasible options from a possibly vast space of possible ones reaching stipulated targets, while the decision evaluation can supports an in-depth performance evaluation of policy proposals taking the preferences of stakeholders into account. The resulting integrated model for the policy problem situation can improve the process of reaching well-accepted scenarios and a transparent design of policy options while taking into account costs, benefits, resource constraints, different perspectives, and multiple objectives. In the context of policy-making the decision makers are often confronted with large and complex amounts of information, not seldom of a conflicting nature and reflect multiple interests. Problem structuring and decision analysis allows policy-makers to judge the performance of alternative policy options and from stakeholders’ different points of view of what represents a positive or negative policy impact. Hence, policy impact assessment frameworks and tools have much to gain from implementing decision analytic support methodologies.

# References

1. Acar, W. (1983). Toward a theory of problem formulation and the planning of change: causal mapping and dialectical debate in situation formulation. Ann Arbor, Michigan:U.M.I.:

2. Acar, W. and Druckenmiller, D. (2006). Endowing cognitive mapping with computational properties for strategic analysis, *Futures* **38**:993-1009.

3. Axelrod, R. (1976). The cognitive mapping approach to decision making. In Axelrod, R. (ed.), *Structure of Decision*, Princeton University Press, Princeton NJ, pp. 221-250.

4. Bero, L. A. and Jadad A. R. (1997). How consumers and policymakers can use systematic reviews for decision making, *Annals of Internal Medicine* **727**(1): 37-42.

5. Bryant, B. P. and Lempert, R. J. (2010). Thinking inside the box: a participatory, computer-assisted approach to scenario discovery, *Technological Forecasting & Social Change* **77**:34-49.

6. Bryson, J. M. (2007). What to do when stakeholders matter, *Public Management Review* **6**(1):21-53.

7. Comes, T., Hiete, M., Wijngaards, N., and Schultmann, F. (2011). Decision maps: a framework for multi-criteria decision support under severe uncertainty, *Decision Support Systems* **52**: 108-118.

8. Danielson, M., Ekenberg, L., Idefeldt, J., and Larsson, A. (2007). Using a software tool for public decision analysis: the case of Nacka municipality, *Decision Analysis* **4**(2):76-90.
9. Druckenmiller, D. A. and Acar, W. (2009). An agent-based collaborative approach to graphing causal maps for situation formulation, *Journal of the Association for Information Systems* 10(3), article 3.

10. Fasth, T., Kalinina, M., and Larsson, A. (2013). Admissibility concepts for group portfolio decision analysis, *Proceedings of GDN 2013*.

11. Hansson, K., Larsson, A., Danielson, M., and Ekenberg, L. (2011). Coping with complex environmental and societal flood risk management strategies: an integrated multi-criteria framework, *Sustainability* 3(9):1357-1380.

12. Hamilton, A. (2010). Policy formulation – critique, analysis and strategic implications, in *New Voices: Essays on the Policy Process*, JSGS Student Working Paper Series, Johnson-Shoyama Graduate School of Public Policy.

13. Franco, L. A. and Montibeller, G. (2010). Facilitated modelling in operational research. *European Journal of Operational Research* 205: 489-500.

14. Howlett, M., Ramesh, M. and Perl, A. (2009). *Studying Public Policy: Policy Cycles & Systems*, New York: Oxford.

15. Kilgour, D. M., Chen, Y., and Hipel, K. W. (2010). Multiple criteria approaches to group decision and negotiation, in Ehrgott et al. (eds.), *Trends in Multiple Criteria Decision Analysis*, Springer.

16. Larsson, A., Johansson, J., Ekenberg, L., and Danielson, M. (2005). Decision analysis with multiple objectives in a framework for evaluating imprecision, *International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems* 13:495-510.

17. Lindblom, C. (1968). *The Policy-making Process*, Prentice-Hall, Englewood Cliffs NJ.

18. Mingers, J. and Rosenhead, J. (2004). Problem structuring methods in action, *European Journal of Operational Research* 152(3):530-554.

19. Montibeller, G. and Belton, V. (2006). Causal maps and the evaluation of decision options – a review. *Journal of the Operational Research Society* 57: 779-791.

20. Tsoukiàs, A. (2007). On the concept of decision aiding process: an operational perspective, *Annals of Operations Research* 154(1):3-27.

21. Tsoukiàs, A., Montibeller, G., Lucertini, G., and Belton V. (2013). Policy analytics: an agenda for research and practice, *EURO Journal of Decision Processes* 1:115-134.

22. Turnpenny, J., Radaelli, C. M., Jordan, A., and Jacob, K. (2009). The policy and politics of policy appraisal: emerging trends and new directions. *Journal of European Public Policy* 16(4):640-653.

23. Zavadskas, E. K., Turskis, Z., Kildiene, S. (2014). State of art surveys of overviews on MCDM/MADM methods, *Technological and Economic Development of Economy* 20(1):165-179.