# Multi-stakeholder Preference Analysis in Ex-ante Evaluation of Policy Options - Use Case: Ultra Low Emission Vehicles in UK

Anton Talantsev, Osama Ibrahim, Aron Larsson

► To cite this version:

Anton Talantsev, Osama Ibrahim, Aron Larsson. Multi-stakeholder Preference Analysis in Ex-ante Evaluation of Policy Options - Use Case: Ultra Low Emission Vehicles in UK. 8th International Conference on Electronic Participation (ePart), Sep 2016, Guimarães, Portugal. pp.176-188, 10.1007/978-3-319-45074-2_14. [https://doi.org/10.1007/978-3-319-45074-2_14](https://doi.org/10.1007/978-3-319-45074-2_14)

HAL Id: hal-01637236

https://inria.hal.science/hal-01637236

Submitted on 17 Nov 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Multi-Stakeholder Preference Analysis in Ex-ante
Evaluation of Policy Options - Use Case: Ultra Low
Emission Vehicles in UK

Anton Talantsev¹, Osama Ibrahim¹ and Aron Larsson¹,²

¹Dept. of Computer and Systems Sciences, Stockholm University

²Dept. of Information and Communications Systems, Mid Sweden University

{antontal, osama, aron}@dsv.su.se

**Abstract.** While the simulation-based impact assessment of public policy proposals allows policy makers to identify the feasible policy options and verify their economic, social and environmental impacts, it does not provide the explicit evaluation of policy options. Multi-criteria decision analysis (MCDA) techniques can support an in-depth performance evaluation of policy options taking into account the preferences of decision makers and stakeholders. These preferences reflect acceptable trade-offs of performance among objectives. This study reviews multi-attribute decision-making (MADM) technique and presents a common policy appraisal format using main evaluation criteria linked to a set of measurable, context dependent attributes. We argue for a rank-based approach for eliciting preferences, select a novel method for attribute weight elicitation, and show how it can be integrated within a public policy multi-criteria evaluation framework. A use case for policymaking, 'Ultra-Low Emission Vehicles (ULEV) Uptake in UK', is used for demonstration of the proposed approach for policy decision analysis. This approach seeks to couple systems modelling and simulation of policy scenarios with MCDA, stakeholder analysis and preference elicitation. The outputs can further provide analytical insights in controversy/acceptability of policy options, and consequently guide further policy formulation and the design of better options.

**Keywords:** Public policy analysis, Multi-criteria decision analysis, Stakeholders, Preference elicitation, Decision support tools, Ultra low emission Vehicles.

# 1 Introduction

Public policy decision processes are often characterized by the presence of multiple and conflicting objectives and multiple stakeholders or decision makers who may have differing point-of-views. Using decision analysis practices to provide informed public policy decisions is not a novel approach per se. What might differ public policy decision analysis from a traditional decision analysis endeavor is that the former does not explicitly aim to reveal a clear recommendation of choice, but also to explore the problem and reveal conflicts, cf. (Quade, 1982).

© IFIP International Federation for Information Processing 2016
E. Tambouris et al. (Eds.): ePart 2015, LNCS 9249, pp. 135–146, 2015.
DOI: 10.1007/978-3-319-22500-5_11
Prescriptive approaches for the study of such decision processes have been suggested within the field of systems thinking and in the field of multi-criteria decision analysis (MCDA). In (Larsson and Ibrahim, 2015), we presented a work process with associated operational research (OR) modeling and analysis tools that supports a prescriptive analysis for policy that includes: the problem definition, ex-ante impact assessment and evaluation activities carried out at the policy formulation stage of the policymaking process. The proposed approach exploited the use of causal maps for problem structuring and scenario-based simulation for the design of policy options, together with decision analysis for evaluating generated scenarios. Having a set of feasible policy options identified, there are two main tasks remaining in structuring MCDA evaluation models; (i) representation of objectives in a structure, commonly a value tree, and (ii) the definition of attributes to measure the achievement of objectives (Franco and Montibeller 2010).

Multi-criteria evaluation can be organised with an objective to produce a single synthetic conclusion or to produce conclusions adapted to the preferences and priorities of several actors. Thus, applying MCDA should provide the relative global performance of each alternative taking preferences of decision makers and stakeholders into account. This is particularly useful when selecting one out of a finite set of feasible alternatives.

Since the rise of the graphical user interface, a flora of computer based decision analysis tools have emerged, aiming to exploit the strengths of graphic interaction with users and thereby enabling for users with less know-how of decision analysis to conduct decision modelling and evaluation, see, e.g., (Riabacke et al. 2014). A prerequisite to increase the practical aspects of decision analysis is to offer accessible tools, and of greatest importance for practical applicability of decision analysis methods seems to be how “easy” they are to use (Stewart 1992, Hülle et al. 2011).

This work is carried out within the contexts of the research project Sense4us<fnref n="1" />, which is developing a web portal that integrates public policy decision support tools to linked open data search tools and social media analytics, in order to enable easy access to information sources and knowledge creation. The aim of this paper is the design of a preference elicitation method based on a multi-criteria evaluation model for policy decision analysis. The method should: (i) not require substantial formal decision analysis knowledge; (ii) not be too cognitively demanding by forcing people to express unrealistic precision or to state more than they are able to; (iii) not require too much time; (iv) make use of the information the decision-maker is actually able to supply, and (v) be supported through a graphical user interface (GUI) accessible from different client operating systems.

This paper is structured as follows. Section 1, introduces the research problem, context, challenges and tasks. Section 2 provides a review of the state-of-the-art research related to concept and application of decision analysis and preference elicitation for policy analysis. Section 3 presents the proposed criteria model for ex-ante evaluation of policy options. Section 4 presents the proposed rank-based preference elicitation

<footnote n="1">¹ EU FP7 research project ‘Data insights for policymakers and citizens’, http://sense4us.eu/</footnote>
method. Section 5, describes the public policy use case, as real example for demonstra-
tion of the use of the proposed policy decision analysis process. Finally, Section 6,
provides the conclusions and possibilities for future work.

## 2 Related Work

Elicitation of preferences in policy making requires that there exists a basis for de-
cision, typically in the form of more or less objective so-called “impact assessment
reports” describing the impact of different alternatives on objectives but strive to be
neutral with respect to decision maker preferences. The impact assessment approaches
can be more or less formalized, and examples used together with decision analysis ap-
proaches within the context of policy making published in the literature include, e.g.,
life-cycle assessment (Miettinen and Hämäläinen 1997), quality adjusted life-years
(Drake et al. 2009, Kivunike 2014), life micro-level simulation and system dynamics
(Hansson et al. 2008, Gou et al. 2001), scenario planning (Montibeller et al. 2006), and
problem structuring using causal maps (Comes et al. 2011). One important feature of
problem structuring approaches is to facilitate the context-setting activity and defining
the environment of a decision problem to be modelled and subsequently evaluated, (see,
e.g. Belton and Stewart (2002) for an introduction to how problem structuring relates
to the decision analysis process).

Further, the use of decision analytic support in participatory/group decision pro-
cesses involving several decision makers and/or stakeholders has received increasing
attention during the last few decades. The decision makers can actually disagree on
what the best alternative is but they need to select a common alternative having suffi-
cient support from the group (Kilgour et al. 2010). Within such a context, a number of
proposed decision analytic approaches and cases of its use have been published in the
literature. For instance, Danielson et al. (2007) propose the analytical decision layer
process, incorporating the stakeholders and decision maker views in an interaction layer
and aggregating these preferences in a decision layer exploiting methods from interval
decision analysis to accommodate for differing preferences such that all decision maker
values are captured by each interval. Hansson et al. (2008) presents an MCDA approach
for evaluating flood management strategies, putting weights on the stakeholders when
aggregating their preferences. Schroeder and Lambert (2011) outlines a decision anal-
ysis process for comparison of transportation policies in the case of an emergency ex-
ploiting scenario planning in order to set the context.

Multi-Attribute Decision Making (MADM) techniques can be applied when the de-
cision consists of selecting one out of a finite set of feasible alternatives, (Belton and
Stewart, 2002). The process of MADM is an analytical process that brings together
three components; the decision objectives as measured by their associated attributes,
the decision options or alternatives and the decision maker's preferences toward the
importance of the attributes as reflected by the attribute weights.

The determination of attribute weights is an important and time consuming process.
Attribute weight elicitation methods, given the decision problem's structural elements,
include: the Balance Beam (BB), Lottery Technique (LT), Paired Comparison (PC),
Point Allocation (PA), Pricing Out (PO), Ratio Weighting (RW) and Trade-off Weighting (TW), (for a comparative review see (Crain, 2003)). These methods can be distinguished by the specific technique employed to compute the attribute weight and whether the technique is designed to determine an attribute importance weight or an attribute swing weight. Importance weights reflect the relative preference without regard to the decision problem at hand, whereas swing weights are sensitive to the values that the attributes possess for the particular decision problem. Crain (2003) emphasized the greater utility and correctness of eliciting swing weights in a MADM problem versus importance weights because of the sensitivity of the swing weight to the range of values taken on by the associated attribute.

A common practicability issue of applying decision analysis methods is that often
too hard requirements are put on the users. It can be argued that much of the applica-
bility issues in employing formal decision analysis in practical settings roots in the need
to elicit beliefs and preferences from decision makers and stakeholders. One recurrent
practicability issue reported in the literature is the need to obtain the decision-maker’s
preferences in such a way that they can be formally represented. That leads to cogni-
tively demanding methods less encouraging for decision makers to use. See, for in-
stance (Riabacke et al. 2012, Riabacke et al. 2014). One increasingly popular way of
relaxing the cognitive burden on decision makers is to rank criteria and/or alternatives
and using cardinal rankings surrogate values can be generated. Recent studies of rank
based approaches include (Sarabando et al. 2010, Wang and Zionts 2015, Danielson
and Ekenberg 2015a), showing that by allowing for the user to provide preference
strength information, the equitability of the generated surrogate numbers will improve,
(i.e. preferences of the decision maker are more properly reflected).

3 A Common Policy Appraisal Format

In MADM techniques, the decision process starts by structuring the problem as an attribute tree hierarchically ordering the decision makers' aims at different abstraction levels. It is generally assumes that each criterion can be operationalised by a set of measurable attributes allowing for assessing the consequences arising from the implementation of any particular alternative. In the next step preferential information is elicited. The relative importance of criteria is captured in weights w<sub>j</sub> for each criterion j at each abstraction level l. At the lowest level of the value tree these objectives are translated into attributes, with each one of them evaluating a given characteristic of the decision options (for example, an objective 'efficiency' may be measured by the attribute 'operating cost'). The performance of each decision option against each attribute is determined and weights are elicited from the decision-makers (Comes T., et al. 2011).

Figure 1, illustrates a multi-attribute value tree, with measurable attributes in gradi-
ent boxes, criteria and overall goal in white boxes. Dependence of attributes on the
alternatives (represented as diamonds) is shown by dashed lines.

Given a set of decision options, $A = \{a_1, ..., a_N\}$ and $M$ evaluation criteria, we can
represent a decision option $a_k$ with a vector of performance levels $(x_{k1}, x_{k2}, ..., x_{kM})$.
Conforming to multi-attribute value theory (MAVT), the global value of a decision option $a_k$ is given according to the additive value function:

$$V(a_k) = \sum_{i=1}^{M} w_i v_i(x_{ki}) \qquad (1)$$

where $v_i(x_{ki})$ is a value function representing the value of alternative $a_k$ under criterion $i$, $w_i$ is the weight of criterion $i$, $0 \le w_i \le 1$ and $\sum w_i = 1$.

This additive model cannot capture the dependencies between criteria, (i.e., being substitutes or complements of each other). We refrain from eliciting such dependencies from decision makers, to achieve the use of use of the elicitation method.

<figure><img src="image_2.png" /><figcaption>Figure 1 Multi-attribute value tree. (Comes T., et al. 2011)</figcaption></figure>

We propose a criteria model for ex-ante evaluation of EU policy interventions, based on a set of main evaluation criteria<fnref n="2" /> *Effectiveness, Efficiency, Coherence, Relevance and Added value* and the underlying measurable attributes. For each of the policy options, the observed changes on the targeted policy impact variables and the policy financial impacts are provided by the impact assessment results. Figure 2, illustrates the structure of the proposed common policy appraisal format as a multi-attribute value tree. The figure shows *k* policy options to be evaluated.

Each of the attributes underlying the “effectiveness” evaluation criterion reflects how successful a policy option is in achieving the policy goals by comparing the observed change (OC) to the targeted change (TC) for each impact variable. The attributes underlying the “efficiency” criterion are concerned with the financial impacts of a policy option, whether financial costs or benefits (cost savings). The evaluation of the impact of the action is expressed as the costs involved on governmental departments given the achieved changes. In addition costs on businesses (including small businesses), or different stakeholder groups are expressed in terms of different attributes. The efficiency evaluations can be made in a quantitative or a qualitative way. The attributes underlying the “relevance” criterion are using qualitative descriptors (or value scales) to assess the

<footnote n="2">² The European commission's better regulation guidelines for evaluations and fitness checks of EU policy interventions. http://ec.europa.eu/smart-regulation/guidelines/index_en.htm</footnote>
relevance of a policy option to the policy objectives, to the problems/needs, to the tech-
nological advances and to the citizens. The attributes underlying the “coherence” crite-
rion are using qualitative descriptors or value scales to assess the coherence of a policy
option: (i) internally if it includes multiple policy instruments, (ii) with other similar
interventions, (iii) with EU policy interventions and (iv) with international obligations.
Finally the attributes underlying the “added-value” criterion are using qualitative de-
scriptors or value scales to assess the added value of the policy option in terms of legal
impacts, increased effectiveness, complementarities and the need for continuing this
intervention.

<figure><img src="image_3.png" /><figcaption>**Figure 2** Common Policy Appraisal Format</figcaption></figure>
# 4 Weight Elicitation Method

The proposed preference elicitation method to be used is a rank-based approach complemented with intuitive ways of generating a value function for policy decision evaluation. The value of each option under each criterion from the perspective of the decision maker is then captured in a so-called value function $v(x)$ such that $v: X \to [0,1]$ where $X$ is the range of the performance indicator. Further, if we assume positive preference direction (the more the better) and let $x_{min}$ denote the worst performance level and $x_{max}$ the best, then $v(x_{min}) = 0$ and $v(x_{max}) = 1$. Two simple ways of obtaining values on intermediate performance levels are suggested.

1) Either to assign them proportionally such that $v(x) = (x - x_{min}) / (x_{max} - x_{min})$ if the preference direction is positive, or $v(x) = (x_{max} - x) / (x_{max} - x_{min})$ if the preference direction is negative. This is often labelled as “proportional scores” and is an intuitive way to generate a value function.

2) Use surrogate values derived from ranking statements according to the novel CAR method (Danielson and Ekenberg, 2015b).

For the CAR method, we use $>_{s(i)}$ to denote the strength of the rankings between criteria or alternatives, where $=_{0}$ means that they are equally ranked, $a_i >_{1} a_j$ means that option $i$ is better important than option $j$, and $a_j >_{2} a_k$ means that option $j$ is “much better” than option $k$ and so forth. This can be represented such that each option is assigned a preference position $p(i) \in \{1, ..., Q\}$ where lower position indicate stronger preference such that for two options whenever, $a_i >_{s(i)} a_k$ then $s(i) = |p(i) - p(k)|$ and the surrogate value is simply given from

$$v_i = \frac{(Q - p(i) + 1)}{Q} \qquad (2)$$

Given a slider with in total Q number of importance scale positions. Each criterion $i$ has the position $p(i) \in \{1, ..., Q\}$ on this importance scale where lower position indicate more importance, such that whenever $c_i >_{s(i)} c_j$, $s(i) = |p(i) - p(j)|$, then the surrogate weight is given by:

$$w_i = \frac{\frac{1}{p(i)} + \frac{(Q - p(i) + 1)}{Q}}{\sum_{j=1}^{N} \left( \frac{1}{p(j)} + \frac{(Q - p(j) + 1)}{Q} \right)} \qquad (3)$$

For example, Figure 3 shows four criteria and a scale with eight steps where scale position 1 is at the right end of the slider and scale 8 at the left end of the slider. The ranking visualized is $c_4 >_{2} c_3, c_3 >_{1} c_1$ and $c_1 >_{4} c_2$, yielding surrogate weights $w_1 = 0.21, w_2 = 0.06, w_3 = 0.26$, and $w_4 = 0.47$.

<figure><img src="image_4.png" /><figcaption>Figure 3 Ranking of criteria</figcaption></figure>
Further, following proportional scores value assignments for all three criteria but the Social criterion, for which we have the following ranking for options 1, 2, and 3, $a_1 >_3 a_3, a_3 >_2 a_2$ yielding values $v_3(x_{13}) = 1, v_3(x_{33}) = 0.4$, and $v_3(x_{23}) = 0$.

This can be illustrated using a multi-criteria evaluation matrix, constructed with as many columns as there are criteria and as many rows as there are decision options to be compared. Table 1 shows exemplary impact assessment results, each cell represents the performance $x_{ij}$ of option $i$ under criterion $j$ in the performance indicator used for that particular criterion. Note that: MCDA requires an evaluation of all the decision options for all the criteria, but does not require that all the evaluations take the same form. The technique can support a mix of quantitative criteria expressed by indicators, qualitative criteria expressed by descriptors, and intermediate criteria expressed by scores (e.g., a scale 0-10).

<table><caption>**Table 1:** Impact assessment matrix</caption><thead><tr><th></th><th>Criterion 1</th><th>Criterion 2</th><th>Criterion 3</th><th>Criterion 4</th></tr></thead><tbody><tr><th>Decision option 1</th><td>$x_{11}$ = 20%</td><td>$x_{12}$ = 510.5</td><td>$x_{13}$ = High</td><td>$x_{14}$ = 4</td></tr><tr><th>Decision option 2</th><td>$x_{21}$ = 35%</td><td>$x_{22}$ = 312</td><td>$x_{23}$ = Low</td><td>$x_{24}$ = 2</td></tr><tr><th>Decision option 3</th><td>$x_{31}$ = 10%</td><td>$x_{32}$ = 615</td><td>$x_{33}$ = Neutral</td><td>$x_{34}$ = 6</td></tr></tbody></table>

In Table 2 below, we denote positive preference direction with ‘↗’ and negative direction with ‘↘’ together with S or R representing “proportional scores” or “ranking” respectively for the value function model. Unequal weights are assigned to criteria (note that the sum of the weights $w_1, ..., w_n$ equals 1).

<table><caption>**Table 2:** Multi-criteria evaluation matrix</caption><thead><tr><th>Criteria</th><th>C<sub>1</sub> ↗S</th><th>C<sub>2</sub> ↗S</th><th>C<sub>3</sub> ↗R</th><th>C<sub>4</sub> ↘S</th></tr></thead><tbody><tr><th>Weights</th><td>$w_1$ = 21%</td><td>$w_2$ = 6%</td><td>$w_3$ = 26%</td><td>$w_4$ = 47%</td></tr><tr><th>Decision option 1</th><td>$v_1(x_{11})$ = 0.4</td><td>$v_2(x_{12})$ = 0.66</td><td>$v_2(x_{13})$ = 1</td><td>$v_2(x_{14})$ = 0.5</td></tr><tr><th>Decision option 2</th><td>$v_1(x_{21})$ = 1</td><td>$v_2(x_{22})$ = 0</td><td>$v_2(x_{23})$ = 0</td><td>$v_2(x_{24})$ = 0</td></tr><tr><th>Decision option 3</th><td>$v_1(x_{31})$ = 0</td><td>$v_2(x_{32})$ = 1</td><td>$v_2(x_{33})$ = 0.4</td><td>$v_2(x_{34})$ = 1</td></tr></tbody></table>

Thus, the preference elicitation methods proposed for enabling decision evaluation in the policy formulation process are:

* For decision option/alternative values: Cardinal ranking or proportional scores, complemented with direct rating.

* For criteria weights: Cardinal ranking, complemented with direct assessment. The reason for complementing the ranking approach with direct rating is that the latter provides flexibility for exploiting other preference elicitation schemes for advanced users.
## 5 Use case: Ultra Low Emission Vehicles (ULEVs) in UK<fnref n="3" />

The British Government wants to increase take up of Ultra Low Emission Vehicles (ULEVs) throughout the UK, as part of its wider plans for reducing greenhouse gas emissions. Electric cars are deemed to reduce CO2 emissions, improve urban air quality, create new jobs, and provide other benefits to society. Amongst a set of initiatives to improve the uptake of electric cars, Plug-in Car Grant, Plugged-in Places, London congestion charge, while many others are being proposed, such as creating exclusive city "green-zones", free parking for electric cars, investing in public recharging infrastructure, arranging hands-on trials for electric cars for citizens, and others. While all these policy interventions can be argued for with means-end rationality, they inevitably impact different stakeholders. Finally, when considering several policy options it's not clear how to compare their acceptability.

Analysis of the national UK stakeholders for the use case is needed to provide answers to the following questions:

1. What are the key stakeholder groups and how are they related to the policy instruments?

2. How would a policy option affect the variety of stakeholders?

3. In what extent might the stakeholders be favouring or be against the transition or a specific policy option?

4. How to accommodate this information to prescriptive decision and policy analysis?

The identified stakeholders have been grouped in order to better articulate potential opponents, reflect the national (UK) level, and reduce the number of stakeholder groups to a manageable number. The list of selected stakeholder groups includes: government, citizens likely to switch to plug-in cars, citizens unlikely to switch to plug-in cars, electricity suppliers, plug-in car manufacturers, conventional cars manufacturers, renewable energy producers, fossil fuel energy producers, power grid operators, petrol producers. "Citizens likely to buy plug-in cars" fully or partially encapsulate other stakeholder groups (citizens) with particular characteristics, such as early adopters, urban citizens, can drive, sufficient income to buy an EV, etc. Similarly, "Citizens unlikely to buy plug-in cars" can be characterised as risk-averse to new technologies, citizens with insufficient income to buy an EV, etc.

Preference elicitation implies (1) defining the direction of preference and (2) intensity of preference for each concept/stakeholder pair. The direction of preference refers to whether the higher value of a concept corresponds to a higher preference value (unidirectional, denoted by +1) or lower (oppositely directed, denoted by -1). The preference intensity is a weight showing relative importance of a concept. Each stakeholder group has been profiled in terms of their needs, goals and means for their achievement. Then, for each stakeholder assessing how increase or decrease in each concept would impact (positively, negatively, or no impact) stakeholders' needs, goals and means. In

<footnote n="3">³ Department for Transport, UK, (2015), 'Uptake of Ultra Low Emission Vehicles in the UK'. A Rapid Evidence Assessment, available on: https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/464763/uptake-of-ulev-uk.pdf (Accessed 1/4/2016)</footnote>
this way, it was relatively straightforward to define preference directions or neutrality (no interest) with respect to the policy evaluation criteria. The selection of the concepts as evaluation criteria and/or attributes for each stakeholder group, has been based on the following principles: (1) the criteria set should represent balance and diversity of aspects for a given stakeholder group, (2) the criteria in the set should be relevant and representative, and (3) preferentially independent for a focal stakeholder group.

For instance, as a representative of the “Government” stakeholder group the Office for Low Emission Vehicles (OLEV) of the British government has been selected. The following attributes were considered for evaluation of the effectiveness of policy options by the government stakeholder group: ULEV industry economic value added, new jobs created, green-house gas (GHG) / CO2 emissions, urban air pollution and local electricity demand.

The proposed policy decision analysis process supports a visual hierarchy of criteria. Figure 4, shows an example for the resulting criteria hierarchy tree with exemplary criteria and attributes weights.

<figure><img src="image_5.png" /><figcaption>**Figure 4** Visual hierarchy of the evaluation criteria for the use case</figcaption></figure>

A relative global performance of the policy options for stakeholders can be calculated through a step-wise decision evaluation process to define: (i) Stakeholder groups and/or individual stakeholders; (ii) Evaluation criteria and/or the underlying attributes, each stakeholder stipulates its own set of criteria and attributes; (iii) Scenarios/Alternative policy options are common for all stakeholders; (iv) Scenario values: using cardinal
ranking statements or surrogate value statements (a point, an interval or both) by each
stakeholder; (v) Criteria weights: using cardinal ranking statements.

Figure 5 and Figure 6 provide examples for ranking of criteria or scenario values uti-
lizing a friendly GUI.

<figure><img src="image_6.png" /><figcaption>Figure 5 Example for criteria ranking using a GUI</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>Figure 6 Example for scenario values ranking using a GUI</figcaption></figure>

**6 Conclusions**

The proposed common policy appraisal format provides a basis for computational
decision analysis for evaluation of policy options. Further, identifying the stakeholders
groups enables appraising a policy from the perspective of multiple stakeholders with
different priorities and preferences. By allowing for preference modelling, we can gain
insights into the level of disagreement on policy proposals as well as how a policy could
be efficiently refined for the mutual benefits of two or more stakeholder groups or ar-
riving at more than one competing policy options for further decision analysis. The
preference elicitation method introduced in this study, uses rank-based elicitation com-
plemented with proportional scores or direct rating in order to provide flexibility for
advanced users. Future work aims to design an integrated decision support tool that
couples systems modelling and simulation of public policy problems with MCDA tech-
niques and preference elicitation methods. The implementation of the proposed deci-
sion analysis process using an online GUI, enables policy makers and stakeholders to
provide preferential statements over their respective goals in a simple way.

7 References

1. Belton, V. and Stewart, T. J. (2002). Multiple Criteria Decision Analysis: An Integrated Approach, Kluwer Academic Publishers.

2. Comes, T., Hiete, M., Wijngaards, N., and Schultmann, F. (2011). Decision maps: a framework for multi-criteria decision support under severe uncertainty, Decision Support Systems 52: 108-118.

3. Crain, W.F. (2003). Multiattribute Weight Determination: Elicitation & Approximation. George Mason University. https://books.google.se/books?id=84IkOAAACAAJ

4. Danielson, M., Ekenberg, L., Idefeldt, J., and Larsson, A. (2007). Using a software tool for public decision analysis: the case of Nacka municipality, Decision Analysis 4(2):76-90.
5. Danielson, M. (2009). Sensitivity analysis in interval decision modelling. Engineering Letters 17(1).

6. Danielson, M. and Ekenberg, L. (2015a). The CAR method for using preference strength in multi-criteria decision making. Group Decision and Negotiation.

7. Danielson, M. and Ekenberg, L. (2015b). Using surrogate weights for handling preference strength in multi-criteria decisions. *15th International Conference on Group Decision and Negotiation, Lecture Notes in Business Information Processing* 218: 107-118.

8. Drake, J. M., Kulkarni, A. V., and Kestle, J. (2009). Endoscopic third ventriculostomy versus ventriculoperitoneal shunt in pediatric patients: a decision analysis. *Child's Nervous System* 25: 467-472.

9. Franco, L. A. and Montibeller, G. (2010). Facilitated modelling in operational research. European Journal of Operational Research 205: 489-500.

10. Gou, H. C., Liu, L., Huang, G. H., Fuller, G. A., Zou, R., and Yin, Y. Y. (2001). A system dynamics approach for regional environmental planning and management: A study for the Lake Erhai Basin. Journal of Environmental Management 61: 93-111.

11. Hansson, K., Danielson, M., and Ekenberg, L. (2008). A framework for evaluation of flood management strategies. Journal of Environmental Management 86(3): 465-480.

12. Hülle, J., Kaspar, R., and Möller, K. (2011). Multiple criteria decision-making in management accounting and control – state of the art and research perspectives based on a bibliometric study. Journal of Multi Criteria Decision Analysis 18(5-6): 253–265.

13. Kivunike, F. N., Ekenberg, E., Danielson, M., and Tusubira, F. F. (2014). Towards an ICT4D evaluation model based on the capability approach. International Journal on Advances in ICT for Emerging Regions 7(1).

14. Larsson, A. and Ibrahim, O. (2015). Modeling for policy formulation: Causal mapping, scenario generation, and decision evaluation. 7th IFIP 8.5 International Conference, ePart 2015, Lecture Notes in Computer Science 9249: 135-146.

15. Miettinen, K. and Hämäläinen, R. P. (1997). How to benefit from decision analysis in environmental life cycle assessment (LCA). European Journal of Operational Research 102: 279-294.

16. Montibeller, G., Gummer, H., and Tumidei, D. (2006). Combining scenario planning and multi-criteria decision analysis in practice. Journal of Multi-Criteria Decision Analysis 14(1-3): 5-20.

17. Quade, E. S. (1982). Analysis for Public Decisions. Elsevier, New York.

18. Riabacke, A., Larsson, A., and Danielson, M. (2014). Conceptualisation of the gap between managerial decision-making and the use of decision analytic tools. International Journal of Information Technology and Business Management 21(1): 30-46.

19. Riabacke, M., Danielson, M., and Ekenberg, L. (2012). State-of-the-art prescriptive criteria weight elicitation. Advances in Decision Science 2012, 24 p.

20. Sarabando, P. and Dias, L. C. (2010). Simple procedures of choice in multicriteria problems without precise information about the alternatives' values. Computers & Operations Research 37(12): 2239-2247.

21. Stewart, T. J. (1992). A critical survey on the status of multiple criteria decision making theory and practice. Omega 20(5-6): 569-586.

22. Wang, J. and Zionts, S. (2015). Using ordinal data to estimate cardinal values. Journal of Multi-Criteria Decision Analysis 22:185-196.