# Shaping up e-Participation Evaluation: A Multi-criteria Analysis

Leif Sundberg

<figure><img src="image_1.png" /></figure>

## To cite this version:

Leif Sundberg. Shaping up e-Participation Evaluation: A Multi-criteria Analysis. 10th International Conference on Electronic Participation (ePart), Sep 2018, Krems, Austria. pp.3-12, 10.1007/978-3-319-98578-7_1. hal-01985600

HAL Id: hal-01985600

https://inria.hal.science/hal-01985600

Submitted on 18 Jan 2019

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
# Shaping Up e-Participation Evaluation:
## A Multi-Criteria Analysis

Leif Sundberg¹

¹ Mid Sweden University, Holmgatan 10, Sweden
leif.sundberg@miun.se

**Abstract.** E-Participation is the use of internet-based technology to empower and include citizens in government decision making processes. The United Nations has evaluated e-Participation around the globe on a regular basis. This evaluation has been criticized for not corresponding to reality, thus allowing non-democratic regimes to gain high positions simply by offering web services. Against this backdrop, the purpose of this paper was to develop a method for evaluating e-Participation, which takes into account a nation's democratic system. Multi-criteria decision analysis was utilized to create a new index by combining datasets of democracy and equality with the UN e-Participation Index 2016. In the new index, authoritarian regimes have lower ranks, and it performed better than UN's index when testing it towards perceived corruption. The paper concludes by suggesting that the proposed method should be accompanied by indicators of the actual use and impact of e-Participation processes. The results of this study contribute to the improvement of e-Participation evaluation practice by introducing normative criteria of good governance.

**Keywords:** e-Participation, evaluation, democracy, equality, multi-criteria analysis.

# 1 Introduction: e-Participation evaluation and democracy

E-Participation is the use of Internet-based technology to include and empower citizens in government decision making processes. Although e-Participation is associated with the hope that new technology can enhance democracy and contribute to 'better' government, the impact on, for example, policy making has so far been limited [1].

As a research field, e-Participation is characterized as multidisciplinary, in that it brings a variety of methods and scholars together [2, 3]. Although there is no unified theory in the field, researchers have made attempts to characterize the field and set out paths for further research. Medaglia [4] argues that e-Participation research needs to move beyond technology to focus on citizens and other stakeholders. Grönlund [5] contends that the current models for describing progress in e-Participation are based on the extensive use of technology and the idea that direct democracy is the most advanced form of democracy.
Other scholars focus on success factors for good e-Participation practice. Jho and Song [6] argue that e-Participation will fail, or even be a menace to democracy, if only tech-
nological infrastructure is considered: technology needs to be accompanied by appro-
priate institutional conditions. Molinari [7] contends that five attributes are crucial to
achieving sustainable participation: juridical compliance, legitimacy, social value, effi-
ciency, and productivity. An Austrian case study concludes that to prevail with e-Par-
ticipation, the users' specific characteristics must be taken into account, including age,
skills, and gender [8]. Skills and gender are also identified as determinants for e-Partic-
ipation in the Spanish context by Vicente and Novo [9].

The United Nations evaluates e-participation every other year through the UN e-Government Survey (UNES2016) [10]. However, the UN evaluation has been criticized for not taking contextual factors into account, thus allowing non-democratic regimes to gain high positions simply by publishing services on their websites [11]. Linde and Karlsson [12] reveal that positive development in e-participation in non-democratic countries does not lead to positive effects on reducing corruption and quality of government. Karlsson [13] contends that non-democratic states with high levels of e-participation also utilize strategies for controlling citizens' internet use. Gulati et al. [14] show similar findings, suggesting that authoritarian regimes might utilize e-Government to maintain status quo. Maerz [15] argues that authoritarian regimes utilize e-government to demonstrate modernity and legitimacy, as well as gaining internal legitimacy by pretending to increase transparency and citizens' engagement. Åström et al. [16] contend that economic globalization is a strong predictor for e-participation development in non-democracies: by implementing ICT-infrastructures, countries make themselves attractive alternatives for foreign investments in technology. Kneuer and Harnisch [17] criticizes the UN surveys for not taking the nations' motives and strategic purposes of e-Government into account. Sæbø et al. [3] contend that e-Participation evaluation is dependent on articulated objectives, clear democratic ideals, and criteria that can serve as benchmarks.

Against this backdrop, the purpose of this paper is to develop a method for evaluating e-Participation that takes into account a nation’s democratic system. The paper proceeds as follows: Section 2 describes the data and research procedure. In Section 3, the results are presented and, finally, conclusions, limitations and implications for further research are described in Section 4.

## 2 Methods and materials

A point of departure in this research is the UN e-Participation Index, which is part of the UNES2016. As literature has pointed out the importance of appropriate democratic institutional conditions in e-Participation initiatives, the Economist Intelligence Unit's Democracy Index from 2016 (EIUDI2016) [18] was used as a democratic indicator. Furthermore, to account for equality and gender, the Global Gender Gap Index 2016 (GGPI2016) [19] was used. These three datasets were then merged using multi-criteria decision analysis (MCDA), which is a method that allows the evaluation and ranking of different alternatives based on a selected number of criteria that can be individually
weighted [20, 21]. A fourth dataset, the Corruption Perceptions Index 2016 (CPI2016) [21], was used to benchmark the index created through the MCDA. The reason for using the CPI2016 was that the level of corruption control in a nation correlates with its degree of civil society participation [22].

## 2.1 Description of data

<table><caption>Table 1. Description of data</caption><thead><tr><td>Dataset</td><td>Description</td><td>Source</td></tr></thead><tbody><tr><td>UN e-Government Survey 2016 (UNES2016)</td><td>The United Nations has benchmarked the e-Government and e-Participation of 193 nations in 2003-2005, 2008, 2010, 2012, 2014 and 2016. The e-Participation Index evaluates the use of online services to provide government information to citizens, interaction with stakeholders and engagement in decision-making processes.</td><td>United Nations (2016) [10]</td></tr><tr><td>The Economist Intelligence Unit Democracy Index 2016 (EIUDI2016)</td><td>Since 2004, the Economist has published regular democracy indexes. The 2016 overall index is based on five indicators: electoral process and pluralism, functioning of government, political participation, political culture, and civil liberties. The countries are then divided into four groups: full democracies, flawed democracies, hybrid regimes, and authoritarian regimes (see Table 2).</td><td>The Economist Intelligence Unit (2017) [18]</td></tr><tr><td>Global Gender Gap Index 2016 (GGPI2016)</td><td>The Global Gender Gap Index is a product of The World Economic Forum. The first measure was released in 2006. The index is based on four indicators: economic participation and opportunity, educational attainment, health and survival, and political empowerment. GGPI2016 is a relative measure of the condition of women compared to that of men in a nation and is not an indicator of the overall situation.</td><td>The World Economic Forum (2016) [19]</td></tr><tr><td>Corruption Perceptions Index 2016 (CPI2016)</td><td>The Corruption Perceptions Index by Transparency International is a measure of corruption within the public sector of nations. The index is based on surveys and expert opinions.</td><td>Transparency International (2017) [21]</td></tr></tbody></table>
In the EIUDI2016 index, countries are distributed by regime type, as seen in Table 2.

<table><caption>**Table 2.** Distribution of regimes (EIUDI2016)</caption><thead><tr><td>Regime Type</td><td>N</td></tr></thead><tbody><tr><td>1: Full democracy</td><td>19</td></tr><tr><td>2: Flawed democracy</td><td>56</td></tr><tr><td>3: Hybrid</td><td>38</td></tr><tr><td>4: Authoritarian</td><td>50</td></tr><tr><td>Total</td><td>163</td></tr></tbody></table>

## 2.2 Procedure

MCDA is a method for ranking alternatives based on several criteria with assigned weights [20, 21]. The weighted sum model (WSM) is the most straight-forward MCDA model and requires that all the criteria use the same scale. Since the scores could be expressed between 0 and 100,000, the WSM was utilized. According to the WSM, a decision problem is assumed to consist of *m* alternatives and *N* criteria. The relative weight for a criterion *C*<sub>j</sub> is expressed by *w*<sub>j</sub>. *a*<sub>ij</sub> is the value of alternative *A*<sub>i</sub> when it is evaluated in terms of a criterion. Then, the total value of *A*<sub>iScore</sub> is defined as:

$$A_{iScore} = \sum_{j=1}^{N} w_{j} a_{ij}, \text{ for } i = 1,2,3,...,m$$

The research was carried out through the following steps:

1. Definition of alternatives:

As described in 2.1, the alternatives that are to be ranked are countries listed in the UNES2016 and EIUDI2016 and GGPI2016 and CPI2016.

2. Selection and definition of criteria:

e-Participation, democracy, and gender equality were chosen as criteria to create a new index (see Figure 1).
<figure><img src="image_3.png" /><figcaption>Fig. 1. New index with criteria</figcaption></figure>

3. Assessment of scores for each attribute:
Scores were extracted from the UNES2016, EIUDI2016 and GGPI2016 datasets.
The datasets were formatted in IBM SPSS 23 and Microsoft Excel.

4. Standardization of the scores to make the criteria comparable with each other:
Each score was expressed on a scale between 0 and 100,000. Theoretical maximum and minimum were used as a global scale: 100,000 was treated as the optimal scenario for each criterion, while 0 was considered the worst-case scenario.

5. Weighting of criteria, to assign priorities to them:
At this point in the research, equal weights were used for all criteria (0.333). A weighting sensitivity analysis for the UNES2016 criteria was performed in step 7.

6. Ranking of the alternatives:
The UNES2016, EIUDI2016 and GGPI2016 were inserted in a multi-criteria decision matrix created in Excel. The weights for each set were set as equal (0.333) to calculate final scores.

<table><caption>Table 3. Sample MCDA matrix</caption><thead><tr><th>Country</th><th>UNES2016<br>(W=0.333)</th><th>EIUDI2016<br>(W=0.333)</th><th>GGPI2016<br>(W=0.333)</th><th>Total Score<br>(New index)</th></tr></thead><tbody><tr><td>Finland</td><td>91,525</td><td>9.03</td><td>0.845</td><td>88,686</td></tr></tbody></table>

For example, the total score for Finland in Table 3 is calculated by expressing the criteria between 0-100,000, followed by multiplying the score for each criterion by its weight, and then summing all the products. In this case, equal weights were applied to the criteria, which gives the following results:

$$ (91,525 * 0.333) + (90,300 * 0.333) + (84,500 * 0.333) = 88,686 $$

7. Test the ranking.
The total score in the new index was then entered into SPSS. To test how the new index fared against the UNES2016, CPI2016 was used as a benchmark using linear
regression. The hypothesis was that the total scores from the new index should correlate more strongly with the CPI2016 than would the UNES2016 would. In addition, a weight sensitivity analysis was performed using the Decerns MCDA DE software [24]. Finally, Person's correlation was utilized to show the correlations between all indexes.

# 3 Results

Table 4 presents the e-Participation (ePart) rankings of the top 11 countries in the UNES2016 and Table 5 presents the results based on the new index created from the data in UNES2016, EIUDI2016 and GGPI2016. The new index consists of 139 countries. As shown, the most notable difference is that the top 11 countries in the new index are full democracies, while the flawed democracies have been degraded. All the Nordic countries have gained positions in the new index.

<table><caption>**Table 4.** Top 11 Ranking and Scores, UNES2016</caption><thead><tr><th>UNES2016 rank<br>(New index rank)</th><th>Score</th><th>EIUDI<br>2016</th><th>GGPI<br>2016</th><th>CPI<br>2016</th><th>Regime<br>type</th></tr></thead><tbody><tr><td>1 United Kingdom (5)</td><td>1.0000</td><td>8.36</td><td>.752</td><td>81</td><td>1</td></tr><tr><td>2 Australia (3)</td><td>.98305</td><td>9.01</td><td>.721</td><td>79</td><td>1</td></tr><tr><td>2 Japan (13)</td><td>.98305</td><td>7.99</td><td>.660</td><td>72</td><td>2</td></tr><tr><td>4 South Korea (18)</td><td>.96610</td><td>7.92</td><td>.649</td><td>53</td><td>2</td></tr><tr><td>5 New Zealand (2)</td><td>.94915</td><td>9.26</td><td>.781</td><td>90</td><td>1</td></tr><tr><td>5 Netherlands (6)</td><td>.94915</td><td>8.80</td><td>.756</td><td>83</td><td>1</td></tr><tr><td>7 Spain (9)</td><td>.93220</td><td>8.30</td><td>.738</td><td>58</td><td>1</td></tr><tr><td>8 Singapore (27)</td><td>.91525</td><td>9.03</td><td>.845</td><td>89</td><td>2</td></tr><tr><td>8 Canada (7)</td><td>.91525</td><td>9.15</td><td>.731</td><td>82</td><td>1</td></tr><tr><td>8 Italy (15)</td><td>.91525</td><td>7.98</td><td>.719</td><td>47</td><td>2</td></tr><tr><td>8 Finland (1)</td><td>.91525</td><td>6.38</td><td>.712</td><td>84</td><td>1</td></tr></tbody></table>
<table><caption>**Table 5.** Top 11 Ranking and Scores, New Index</caption><thead><tr><th>New index rank<br>(UNES2016 rank)</th><th>Score</th><th>EIUDI<br>2016</th><th>GGPI<br>2016</th><th>CPI<br>2016</th><th>Regime<br>type</th></tr></thead><tbody><tr><td>1 Finland (8)</td><td>88,686</td><td>9.03</td><td>.845</td><td>89</td><td>1</td></tr><tr><td>2 New Zealand (5)</td><td>88,450</td><td>9.26</td><td>.781</td><td>90</td><td>1</td></tr><tr><td>3 Australia (2)</td><td>86,748</td><td>9.01</td><td>.721</td><td>79</td><td>1</td></tr><tr><td>4 Norway (27)</td><td>86,504</td><td>9.93</td><td>.842</td><td>85</td><td>1</td></tr><tr><td>5 United Kingdom (1)</td><td>86,180</td><td>8.36</td><td>.752</td><td>81</td><td>1</td></tr><tr><td>6 Netherlands (5)</td><td>86,085</td><td>8.80</td><td>.756</td><td>83</td><td>1</td></tr><tr><td>7 Canada (8)</td><td>85,290</td><td>9.15</td><td>.731</td><td>82</td><td>1</td></tr><tr><td>8 Sweden (28)</td><td>83,806</td><td>9.39</td><td>.815</td><td>88</td><td>1</td></tr><tr><td>9 Spain (7)</td><td>83,257</td><td>8.30</td><td>.738</td><td>58</td><td>1</td></tr><tr><td>10 Denmark (22)</td><td>82,836</td><td>9.20</td><td>.754</td><td>90</td><td>1</td></tr><tr><td>11 Iceland (50)</td><td>82,751</td><td>9.50</td><td>.874</td><td>78</td><td>1</td></tr></tbody></table>

To test how well the new index fared compared to the UNES2016 in terms of democracy, both indexes were tested against the CPI2016 by using linear regression. Figure 2 contains the results from the UNES2016 / CPI2016 dataset and Figure 3 from the new index / CPI2016 dataset. As shown, the new index has fewer outliers and has a higher correlation (see Table 6) with CPI2016 than the UNES2016 has. Furthermore, in the UNES2016 e-Participation Index, 8 authoritarian regimes can be found among the top 50 countries, with one in a top position as number 22. The full democracy with the lowest position in the UNES2016 e-Participation Index is positioned as number 74. In the new index, these rankings are changed, with the authoritarian regime moving from a ranking of 22 down to 75, and the full democracy moving up from 74 to 28. In Figures 2 and 3, the authoritarian regime ("A") and the full democracy ("D") are compared. As shown, they are closer to the regression line when using the new index. Figure 4 reveals that the UNES2016 criteria needs a weight of 59.5%, given that the weights of EIUDI2016 and GGPI2016 remain similar, for regime A to achieve a greater ranking than D. That scenario seems unlikely when considering the trade-offs such a weight would imply.
<figure><img src="image_4.png" /><figcaption>Fig. 2. UNES2016 and CPI2016</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Fig. 3. New Index and CPI2016</figcaption></figure>
<figure><img src="image_6.png" /><figcaption>Fig. 4. Weight sensitivity analysis of the UNES2016 criteria</figcaption></figure>

Finally, Table 6 shows the correlation between the different indexes. As can be seen, the New index has the strongest correlation with the CPI2016 compared to all included indexes.

<table><caption>Table 6. Correlations between indexes</caption><thead><tr><th></th><th>New index</th><th>UNES2016</th><th>EIUDI2016</th><th>GGP2016</th><th>CPI2016</th></tr></thead><tbody><tr><th>New index</th><td>1</td><td>.872**</td><td>.869**</td><td>.625**</td><td>.765**</td></tr><tr><th>UNES2016</th><td>.872**</td><td>1</td><td>.624**</td><td>.362**</td><td>.619**</td></tr><tr><th>EIUDI2016</th><td>.869**</td><td>.624**</td><td>1</td><td>.580**</td><td>.760**</td></tr><tr><th>GGP2016</th><td>.625**</td><td>.362**</td><td>.580**</td><td>1</td><td>.516**</td></tr><tr><th>CPI2016</th><td>.765**</td><td>.619**</td><td>.760**</td><td>.516**</td><td>1</td></tr></tbody></table>

<footnote>**. Correlation is significant at the 0.01 level (2-tailed).</footnote>
# 4 Concluding remarks

The purpose of this paper was to develop a method for evaluating e-Participation that takes into account a nation's democratic system. Multi-criteria analysis was utilized to create a new index by combining datasets ranking levels of democracy and equality with the UN e-Participation index 2016. In the resulting index, a country needs to have a solid democratic base with equality between men and women to rank highly. Building on this foundation, e-Participation can become a deciding criterion, inspiring nations to develop innovative participatory processes. However, no nation will gain a prominent position by focusing on technology and web services alone. By utilizing MCDA, several indexes can be combined and assigned individual weights. The study was not without limitations. In the current research stage, no method for weighting criteria has been utilized. Furthermore, although the suggested method incorporated democratic and gender equality criteria, it does not evaluate whether users actually influence processes and policies through e-Participation, as well as not accounting for local differences. Should an index measuring actual user impact be created, it could be incorporated into the model. The results of this study contribute to the improvement of e-Participation evaluation practice by introducing normative criteria for good governance.

# References

1. Coelho, T. R., Cunha, M. A., Pozzebon, M.: eParticipation and the Policy Cycle: Designing a Research Agenda. 18th Annual International Conference on Digital Government Research (Dg.o 2017). New York, NY: ACM. https://doi.org/10.1145/3085228.3085277 (2017).

2. Medaglia, R.: The challenged identity of a field: The state of the art of eParticipation research. Inf. Polity. 12, 169–181 (2007).

3. Sæbø, Ø., Rose, J., Flak, L.S.: The shape of eParticipation: Characterizing an emerging research area. Gov. Inf. Q. 25, 400–428 (2008).

4. Medaglia, R.: eParticipation research: Moving characterization forward (2006–2011). Gov. Inf. Q. 29, 346–360 (2012).

5. Grönlund, Å.: ICT Is Not Participation Is Not Democracy – eParticipation Development Models Revisited. (A. Macintosh & E. Tambouris, Eds.), Electronic Participation: Proceedings of the 1st IFIP WG 8.5 International Conference, ePart 2009. Linz, Austria (2009).

6. Jho, W., Song, K.J.: Institutional and technological determinants of civil e-Participation: Solo or duet? Gov. Inf. Q. 32, 488–495 (2015).

7. Molinari, F.: On Sustainable eParticipation. (E. Tambouris, A. Macintosh, & O. Glassey, Eds.), Electronic Participation: Proceedings of the 2nd IFIP WG 8.5 International Conference, ePart 2010. Lausanne, Switzerland. https://doi.org/10.1007/978-3-642-15158-3_11 (2010).

8. Edelmann, N., Hoechtl, J., Parycek, P.: eParticipation for Adolescent Citizens (in Austria). (A. Macintosh & E. Tambouris, Eds.), Electronic Participation: Proceedings of the 1st IFIP WG 8.5 International Conference, ePart 2009. Linz, Austria (2009).

9. Vicente, M.R., Novo, A.: An empirical analysis of e-participation. The role of social networks and e-government over citizens’ online engagement. Gov. Inf. Q. 31, 379–387 (2014).

10. United Nations.: UN E-Government Survey 2016. http://workspace.unpan.org/sites/Internet/Documents/UNPAN97453.pdf, last accessed 2018/02/21
11. Grönlund, Å.: Connecting eGovernment to Real Government - The Failure of the UN eParticipation Index. (M. Janssen, H. J. Scholl, M. A. Wimmer, & Y. Tan, Eds.), Electronic Government: Proceedings of the 10th IFIP WG 8.5 International Conference, EGOV 2011. Delft, The Netherlands (2011).

12. Linde, J., Karlsson, M.: The Dictator's New Clothes: The Relationship Between E-Participation and Quality of Government in Non-Democratic Regimes. Int. J. Public Adm. 36, 269–281 (2013).

13. Karlsson, M.: Carrots and sticks: internet governance in non-democratic regimes. Int. J. Electron. Gov. 6, 179–186 (2013).

14. Gulati, G.J. "Jeff," Williams, C.B., Yates, D.J.: Predictors of on-line services and e-participation: A cross-national comparison. Gov. Inf. Q. 31, 526–533 (2014).

15. Maerz, S.F.: The electronic face of authoritarianism: E-government as a tool for gaining legitimacy in competitive and non-competitive regimes. Gov. Inf. Q. 33, 727–735.

16. Åström, J., Karlsson, M., Linde, J., Pirannejad, A.: Understanding the rise of e-participation in non-democracies: Domestic and international factors. Gov. Inf. Q. 29, 142–150 (2012).

17. Kneuer, M., Harnisch, S.: Diffusion of e- government and e- participation in Democracies and Autocracies. Global Policy 7, 4, 548-556, (2016).

18. The Economist Intelligence Unit. Democracy Index 2016. Revenge of the "deplorables". https://infographics.economist.com/2017/DemocracyIndex/, last accessed 2018/02/21.

19. World Economic Forum.: The Global Gender Gap Report 2016. http://www3.weforum.org/docs/GGGR16/WEFalm_Gender_Gap_Report_2016.pdf, last accessed 2018/02/21.

20. Triantaphyllou, E.: Multi-Criteria Decision Making: A Comparative Study. Dordrecht, The Netherlands: Kluwer Academic Publishers (now Springer) (2000).

21. Keeney, R.L. & Raiffa, H.: Decisions with Multiple Objectives: Preferences and Value Tradeoffs, New York: Wiley (1976).

22. Transparency International.: Corruption Perceptions Index 2016. https://www.transparency.org/news/feature/corruption_perceptions_index_2016, last accessed 2018/02/21

23. Transparency International. Digging deeper into corruption, violence against journalists and active civil society. https://www.transparency.org/_view/feature/8178, last accessed 2018/02/21

24. Yatsalo, B., Didenko, V., Gritsyuk S., Sullivan, T.: Decerns: A Framework for Multi-Criteria Decision Analysis. International Journal of Computational Intelligence Systems, 8(3), 467-489 (2015).