# Measure to Improve: A Study of eParticipation in Frontrunner Dutch Municipalities

Anne Veenstra, Marijn Janssen, Andreas Boon

► To cite this version:

Anne Veenstra, Marijn Janssen, Andreas Boon. Measure to Improve: A Study of eParticipation in Frontrunner Dutch Municipalities. 3rd Electronic Participation (ePart), Aug 2011, Delft, Netherlands. pp.157-168, 10.1007/978-3-642-23333-3_14. hal-01589368

HAL Id: hal-01589368

https://inria.hal.science/hal-01589368

Submitted on 18 Sep 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Measure to Improve: A Study of eParticipation in
frontrunner Dutch Municipalities

Anne Fleur van Veenstra, Marijn Janssen, and Andreas Boon

Faculty of Technology, Policy and Management, Delft University of Technology,
Jaffalaan 5, 2628 BX Delft, the Netherlands
{a.f.e.vanveenstra,m.f.w.h.a.janssen}@tudelft.nl,

Deloitte Consulting B.V., P.O. Box 300, 1180 AH Amstelveen, the Netherlands
aboon@deloitte.nl

**Abstract.** eParticipation is still in its early phases of development, in which government organizations undertake many experiments. There is no systematic overview of these experiments, which inhibits learning from each other. Measuring and benchmarking eParticipation provide the opportunity to inspire new developments by sharing best practices. This paper presents the development of a measurement instrument that is subsequently used to investigate eParticipation in frontrunner Dutch municipalities. This instrument combines factors related to the role of citizens, the type of media deployed, and the organization of the municipality. Using these factors to measure eParticipation in seventeen municipalities we found a large focus on traditional media forms usually supporting informative and consultative practices for policy development. This study, however, has been largely limited to measuring the type of ICT that is used to facilitate eParticipation. Therefore, we recommend extending it with measurements that give insight into the actual use, effectiveness, and the inclusion of citizens’ input in policy making.

**Keywords:** eParticipation, measurement, benchmarking, measurement instrument, municipalities

## 1 Introduction

An important element in serving the citizens of any democratic nation is to understand their perceptions of the issues they find important. Yet many citizens lack an interest in and feel dissatisfied with politics [1,2]. In may parts of the world, voter turnout and civic participation have steadily been decreasing since the 1960s [3,4]. Traditional channels influencing policy-making, such as newspapers and town hall meetings, are decreasing in popularity, especially among younger citizens [4]. eParticipation has the potential to bridge the gap between citizens and the government [1], as well as being an instrument to make better and more supported policies [5]. eParticipation can be defined as the use of ICT by governments to support information provisioning to citizens and to engage and facilitate citizens to influence the government in their policy-formation and decision-making processes [1,6,7].
Many government organizations around the world undertake eParticipation projects. Due to the infancy of eParticipation [8,9], the current focus of government organizations is exploratory in nature. Current eParticipation experiments represent a divergent set of methods and philosophies within government organizations, and “comparative empirical classifications and evaluations remain the exception” [9, p. 5]. Measurements and benchmarking facilitate the uptake of new developments by showcasing best practices and stimulating the further use of these features. Although there is much research on measuring and benchmarking eGovernment [10-17], scant attention has been given to the measurement and benchmarking of eParticipation. This paper presents the development and validation of a measurement instrument for eParticipation in Dutch municipalities. Local government is an important level of government for eParticipation as citizens feel they are more personally involved and have a high expectation about the effect of participating in the decision making processes of their municipality [2,9]. Furthermore, in the Netherlands most contact citizens have with the government takes place at the municipal level [18].

The instrument presented in this paper focuses on the ability of ICT to facilitate and enable eParticipation. The measurement instrument includes three elements that, when connected, give insight into the state of eParticipation in Dutch municipalities: the role of citizens, the type of media being used, and the municipal organization. This paper is structured as follows. First, we will give an overview of existing benchmarks and their use for inspiring eParticipation developments. Then, we present the development of the measure instrument for eParticipation, followed by the outcomes of using this instrument to measure eParticipation in seventeen Dutch municipalities. Finally, we will present conclusions and recommendations for further research.

## 2 Measuring and Benchmarking eParticipation

Measurement and benchmarking have gained considerable attention in the field of eGovernment [10-17]. These measurements are often an attempt to quantify or operationalize complex and wicked problems. Breaking down such a subject into smaller and measureable parts can provide useful insights. The benefits of such measurements include being able to compare the current situation with the desired situation, analyzing the costs and benefits of investments, and providing motivation for future developments [19]. International benchmarks such as the UN Index [10,12], the OECD e-Government studies, GapGemini's Public Service Benchmark for the European Commission [13], and Accenture's Public Service Model [14] have stimulated governments to develop their eGovernment activities. While some of these benchmarks include elements of eParticipation, no benchmark exists that focuses exclusively on eParticipation. Elements of participation that are often included have a predominantly technical focus, measuring the availability of a type of media or service provided by an organization [15].

Although benchmarking is associated with the promotion of best practices and provision of incentives for improvement, it has also generated critique. Especially the criteria and indicators included as well as the method of measurement have attracted
criticism [16]. Bannister [20] criticizes scoring methods on a number of accounts: complex issues are bundled into one single score or measured by the use of proxies, the changes of scores and interpretation of criteria over time render comparison impossible, the scope and complexity involved and the associated problems of operationalizing, and the often ambiguous interpretation of the scores. Janssen et al. [21] highlighted another problem involved with benchmarking, using the slogan 'if you measure it, they will score'. This means that as soon as certain elements are operationalized in a specific way, organizations will make sure that they score high on the operationalized score, instead of looking into the underlying issues. The use of benchmarking should therefore be used as part of a wider context of assessment and quality management [17,20]. Bannister [20] further argues that a benchmark represents a trade-off between costs, scale and quality of information, and that measurement methodologies are too often guided by cost constraints.

## 3 Development of a measurement instrument for eParticipation

Due to the infancy of eParticipation [8,9], the current focus is experimental in nature and systematic studies into best practices are lacking. eParticipation is still in its early stages and few initiatives exist that connect the multiple elements of eParticipation, such as providing information, facilitating bottom-up initiatives from citizens and actively engaging citizens with government [1]. Furthermore, measuring the deployment of online media alone does not provide a full overview of the state of eParticipation as it is also necessary to understand the various roles that citizens can take on when engaging with the government as well as the role of eParticipation in the internal organization. Following the multiple elements of eParticipation, we opted for an instrument that consists of three interconnected elements of measuring eParticipation:

(1) Citizen engagement: the roles citizens take on in their contact with the government;

(2) ICT deployment: the types of online media used; and

(3) Organization: the internal organization of municipalities.

The choice for these three elements has been made to cover the 'demand' for, the 'supply' of, and the means of communication for eParticipation. The municipal website can be seen as the technological facilitator between the citizen as user and the municipal organization as facilitator and service provider. These elements have been part of some measurement instruments and frameworks as well [20,22,23,24]. In order to measure the state of eParticipation, all three elements should be taken into account as well as the relations between the three elements.

Per element of the instrument a different set of measures was identified. For example, different types of media were distinguished according to their potential for eParticipation and scores were derived reflecting potential and actual use. To ensure that the instrument has the possibility of prolonged use and enables comparisons over time, an 'ideal' score was set at 100% and subsequently the amount of this ideal realized was determined and a score accordingly attributed. Furthermore, to avoid the
pitfall mentioned by Bannister [20], scores are not aggregated on a single measure. Scores for the individual services or components that make up the elements are, for example, determined by the level of activity observed (e.g. posts per month, channel views, number of followers, etc.) and the level of interaction possible (e.g. one-way, reactions possible, discussions possible, etc.). Measurements are done based on the information gathered from the municipal websites, and by assessing and categorizing this information based on pre-defined patterns that were developed by carrying out fifteen interviews with experts in the field of eParticipation.

Measuring the state of eParticipation of a municipality was done in three steps. Firstly, the measures that can be obtained by looking at the websites of seventeen municipalities were scored. Then, using these measurements, their potential for each of the factors was determined. And finally, six municipalities were studied more in-depth by carrying out interviews with employees. These semi-structured interviews provided extra information by reflecting on the data gathered by assessing the website, and investigated the internal organization which could not be measured by looking at the website. The purpose of these interviews was to generate additional context for understanding the measurement results based on organizational factors such as the level of knowledge and resources available within a municipality.

## 2.1 Citizen engagement

The first element of the measurement instrument is the role citizens take on in their contact with the government. Our measurement instrument uses generic roles based on user patterns to identify for which roles the municipal websites cater. Distinguishing between these roles can enable municipalities to refine their eParticipation strategy based on the type of interaction they wish to promote. Six different roles are distinguished related to internet usage [26]:

- **Inactives** do not participate at all;

- **Spectators** read blogs, listen to podcasts, watch videos from other users, and read forums;

- **Joiners** maintain a social networking profile and participate in social networks;

- **Collectors** aggregate data via RSS feeds;

- **Critics** review products/services, comment on blogs/forums, and contribute to wiki's; and

- ** Creators** publish a blog/website, upload videos/music, and write articles or stories.

While the Inactive and Collector roles are useful in understanding different user types and their expectations about websites, they are not participatory roles. Therefore, these roles will be left out in our instrument. The information found on the municipal websites was assessed to determine to which extent the services cater for the user roles. For example, investments in social networking sites such as Facebook and Twitter likely invite citizens that have taken on the Joiner role. This gives insight in the degree to which municipalities facilitate specific user roles and accordingly, their type of involvement. Municipalities can use this information to determine the facilitation of specific roles and build an appropriate social strategy [26].
## 2.2 ICT deployment

The purpose of this element of the measurement instrument is to measure how well municipalities can facilitate eParticipation on their website through the use of online media. Firstly, this element includes the selection of the type of media as well as how actively these chosen media forms are used. Based on literature, the following categorization [24] and media [6,9,24,27-29] were selected for measurement:

- **Standard media:** Email, Newsletter, Short Message Service (SMS), Poll, Survey;

- **Innovative media:** Real Simple Syndication (RSS) Feed, Blog, Forum, Webcasting, Social Networks (LinkedIn, Facebook, Hyves), Twitter, Media Community (Flickr, YouTube), Chat, Interactive Map, ePetition, Wiki; and

- **Experimental media:** Simulation or Game, Virtual World, Chatbot, Group Support System.

Standard media forms have been around for a long time, innovative media contain some Web 2.0 aspects such as social networks and indicate a popular segment of media, and the experimental media are (as of yet) rarely used [24]. This categorization shifts over time and is subjective in the sense that some media that are classified as experimental may have become innovative or standard for generic situations. We found that this classification fits the current Dutch eParticipation context by discussing these issues in the interviews with the experts.

Secondly, the media types investigated in this measurement tool can be classified according to the level of participation they are usually used for. Five different levels of participation can be distinguished according to the increasing importance they entail for municipalities to act upon [8,8,24,30,31]:

- **Informing:** one-way provisioning of information about public affairs and the municipal organization;

- **Consulting:** expertise, opinions, and/or votes are used to poll for the perspective of the citizen on selected topics. Municipalities may use these as input for their decision-making but they are not obliged to act them;

- **Advising:** expanding consultation into a group discussion where citizens can deliberate together with the municipality about problems or policy alternatives, thereby carrying more weight than consulting;

- **Co-producing:** a form of participation where the agenda is set together with the citizens and a new policy or service is created together. The municipality is beholden to the results of the process; and

- **Co-decision:** the municipality asks citizens to make a binding choice about a selection of policy alternatives.

## 2.3 Organization

The third aspect of the measurement tool is the organization of the municipality. The website is the online front-office for eParticipation, so it is crucial that the offline back-office is capable of supporting this. As this cannot be studied by looking at the website, a set of interview questions for employees of the six municipalities was
defined. These open interview questions are not technically part of the measurement
tool as they do not provide any eventual score or assessment, but they can be used to
link eParticipation to the decision making processes in the municipalities. This
element included questions on whether or not people were given responsibilities
regarding eParticipation, the existence of training, and the amount of budget available
for example. Multiple eParticipation researchers [23,24,29,32] make a connection
between the different forms of participation and the different policy phases:

- **Agenda-setting;**

- Policy analysis, sometimes called Design;

- Policy-creation or decision-making;

- Implementation; and

- Monitoring or evaluation.

Analyzing the forms of participation in relation to the policy cycle is useful because it
places citizen participation in the lifecycle of municipal processes. The input from
stakeholders and the openness of the process differ as projects progress [33].
Participation of citizens plays a different role in the agenda-setting phase than in the
implementation phase. Municipalities can adopt eParticipation in those phases they
want to focus on.

## 4 Findings

The municipalities chosen for this study were known for actively exploring eParticipation opportunities and most of them can be seen as frontrunners. In total seventeen municipalities were investigated by analyzing their websites, and for six of these municipalities interviews were held. Five of these interviewed municipalities were large and one was medium-sized. During the interviews held, the employees of the six municipalities generally recognized and confirmed their scores on the measurement instrument.

### 4.1 Citizen engagement

The capabilities of municipalities to facilitate specific roles citizens' take on in their
contact with the government are shown in Fig. 1. This shows that the municipalities
generally cater to the Spectator role (i.e. listening and reading information provided
by the government) and the Joiner role (which maintains a social networking profile
and participates in social networks). The other two roles, the role of the Critic (those
that review and comment on wiki's and forums) and the role of the Creator (people
who publish their own content), are less facilitated by the capabilities of the municipal
websites.
<figure><img src="image_2.png" /><figcaption>Fig. 1. Spider graph of selected citizens’ Internet usage roles</figcaption></figure>

## 4.2 ICT deployment

<figure><img src="image_3.png" /><figcaption>Fig. 2. ICT deployment scores per media type</figcaption></figure>

The ICT deployment scores per media type – standard, innovative, and experimental – are shown in Fig. 2. Although there are some exceptions, the Standard media scores are generally the highest of the three types with scores reaching above 55% and with an average of 34%. The second most common types of media are the Innovative ones with an average score of 21%. Experimental media types are hardly – if at all – deployed by the municipalities in this study. The average score here is 6% with 7 municipalities not having any examples at all. This shows the limited level of media usage in the investigated municipalities that are considered as frontrunners, which likely has consequences for their potential to develop these media for eParticipation.
In Fig. 3, the media forms deployed are categorized according to the participatory role they facilitate. A higher occurrence of the Informing (the provisioning of information) and Consulting (facilitating input from individual citizens) levels of participation can be observed. While Advising (the facilitation of input from groups of citizens) and Co-deciding (the joint agenda-setting by citizens and municipalities) can be observed in the municipalities under study, the participation level of Co-producing (where citizens are responsible for policy-making) is observed in only a few municipalities. Note that only a selection of municipalities is shown in order to simplify the graph for viewing, but the patterns shown represents the other municipalities too.

<figure><img src="image_4.png" /><figcaption>Fig. 3. Spider graph of selected level of participation scores</figcaption></figure>

## 4.3 Organization

<figure><img src="image_5.png" /><figcaption>Fig. 4. Spider graph of selected policy cycle scores</figcaption></figure>
The data were also mapped onto the different phases of the Policy cycle. The graph pattern in Fig. 4 shows a greater focus on Monitoring and Agenda-setting. This is likely a result of the large role that Consultation and Informing play in these phases, which are currently the most common forms of participation. The lesser emphasis on Implementation is expected as there were few participation forms observed where citizens are capable of contributing in this phase.

# 5 Discussion and limitations

From the application of the measurement instrument to the Dutch municipalities it becomes clear that eParticipation in still in its early stages. Although they try to reap some of the low-hanging fruits such as deploying already common forms of ICT for purposes of eParticipation, it is clear that the municipalities participating in this study are still experimenting. As these municipalities are considered frontrunners in the Netherlands, it is likely that others lag behind. From the findings of applying our benchmarking instrument we identified two major issues that need to be addressed to further develop eParticipation in municipalities in the Netherlands: spurring the use of media forms beyond mere simple interactions, and the direct involvement of municipal administrators in more advanced forms of eParticipation.

The choice for predominantly simple media forms on municipal websites reveals the hesitance of municipalities for committing to more advanced forms of ICT and, hence, eParticipation. Insofar as municipalities have implemented media forms that can be used for interaction with citizens, they are hardly used in that way. Particularly social media such as Twitter and YouTube are very popular choices, but are only used to send information. A reason for this may be that these activities require less effort from the municipalities.

Furthermore, we observed that a major issue for the deployment of eParticipation is the degree to which municipal administrators can directly interact with citizens through social media. In case they are allowed to do so, the question what are suitable rules for this contact remains. In municipalities issues may quickly become political. This means that for the ‘real’ interactive tools, such as forums and simulation games, which need to be moderated real-time, this issue will prove a severe limitation in their use. Currently, there is a lack of skills and resources for process management to deal with new input. The role of civil officials in this interaction is also an issue which must be dealt with for eParticipation to succeed on a larger scale.

The instrument developed in this paper can be seen as a first step towards measuring the state of eParticipation in the Netherlands. This was generally confirmed by recognition of the outcomes by the interviewees. The instrument aimed, however, only at measuring the ability of ICT to facilitate and enable eParticipation, instead of on the actual use, influence, and effectiveness. Its main limitation thus the focus on the technical, ‘supply’-side of eParticipation. The measurement tool only made use of data that were visible on the website. This resulted in predominantly measuring the type of media being used. Although roles of citizens and the municipal organization were taken into account, they could only be scored on the basis of their potential through the types of media deployed. The fact that an organization is able to
deploy a certain media type and thereby has the potential to facilitate a certain user role, does not necessarily mean that they are actively engaging with citizens in an effective way.

An important reason for the limited scope of the measurement instrument was a lack of resources for in-depth assessments of a large number of municipalities. Measurements were therefore carried out in a cost-efficient manner by focusing on elements that could be measured easily. We therefore recommend for eParticipation measurements to use qualitative methods such as interviews, which allow for more substantive measurement and categorization, instead of an instrument that only measures the potential of the front-office. We also found that conducting interviews and carrying out an in-depth assessment of the state of eParticipation has the potential to inspire learning among municipalities. Furthermore, as categorization was done by experts, it was subjective. We therefore recommend that further research should be done on the development of generally accepted measurements for eParticipation.

Because of the focus on the ‘supply’-side, the actual needs and desires of citizens with regard to eParticipation were not measured in this instrument. To include measures on the demand-side, the instrument should be extended to include interviews with citizens. Another issue with the focus on the supply-side of eParticipation was that some of the experts interviewed criticized the measurement instrument for including the Informing level as this is often not seen as a form of participation, but rather as a precondition. For this reason, it is useful to keep it among the levels of participation deployed in the benchmarking tool, but it should be made clear that this is step ‘zero’ on the way towards proper participation.

An underlying difficulty encountered in this study is the current level of eParticipation in the Netherlands. Many of the limitations of the measurement instrument are related to eParticipation still being in its early stages of development. Development of a measurement instrument for the more advanced stages is be useful when there is little to measure. Therefore, extension of the measurement instrument to cover the use of citizens’ input and its effectiveness will be necessary to gain proper insight into eParticipation. Furthermore, the measurement of the more advanced stages as well as the effectiveness of eParticipation requires more in-depth research than measuring the ability of municipal websites to enable participation. The use of qualitative instruments such as interviews may be more useful for measuring eParticipation as it allows for investigating the details of enabling, stimulating, and processing the input of citizens, as well as inspiring learning from each other.

# 6 Conclusion

eParticipation is still in its infancy and few instruments exist that give an overview of the experiments that are undertaken by government organizations. To inspire further development, a measurement instrument used for assessing the level of eParticipation within municipalities was developed and applied to seventeen Dutch municipalities. While these municipalities undertook many eParticipation related experiments, these developments largely focused on informing and consulting citizens through deployment of informational online media. The interviews carried out in six of the
municipalities revealed that they have few ideas on how to design and deploy a fully-
developed eParticipation strategy. As the municipalities taking part in this study can
be characterized as frontrunners this implies that the level of eParticipation
deployment in the Netherlands is still rather low.

The instrument connects the roles citizens take on in their contact with the government, the types of online media deployed and the municipal organization. However, the instrument mainly focuses on the type of ICT being used, as this is currently how eParticipation is perceived in government organizations. For further investigations into eParticipation the measurement instrument should be extended to cover factors such as the effectiveness of the media types deployed in relation to the type of users and the inclusion of citizens' input in the decision-making processes in the organizations. Furthermore, we recommend the use of qualitative measurement instruments such as interviews to enable more in-depth assessment of eParticipation as well as to inspire learning.

References

1. Macintosh, A.: E-democracy and e-participation research in Europe. In: Chen, H., Brandt, L., Gregg, V., Traunmuller, R., Dawes, S., Hovy, E., Macintsh, A., Larson, C.A. (eds.) Digital Government: E-government Research, Case Studies, and Implementation, pp. 85-102. Springer, Berlin, Heidelberg (2008)

2. European Commission: European e-Government 2005-2007: Taking stock of good practice and progress towards implementation of the i2010 eGovernment Action Plan. Research for the Directorate General for Information Society and Media (2007)

3. Niemi, R., Weisberg, H.: Controversies in Voting Behavior. CQ Press, Washington, DC (2001)

4. Fraser, M., Dutta, S.: Throwing Sheep in the Boardroom: How Online Social Networking Will Transform Your Life, Work and World. John Wiley & Sons, West Sussex, England (2008)

5. McDermott, P.: Building open government. Government Information Quarterly 27, 401-413 (2010)

6. International Center of Excellence for Local eDemocracy (ICELE): Transformational Democracy, Third Edition (2008)

7. Macintosh, A.: eParticipation in policy-making: the research and the challenges. In: Cunningham, P., Cunningham, M. (eds.) Exploiting the Knowledge Economy: Issues, Applications and Case Studies, pp. 364-369. IOS Press, Amsterdam (2006)

8. Committee of Ministers: Recommendation CM/Rec(2009)1 of the Committee of Ministers to member states on electronic democracy (2009) https://wcd.coe.int/wcd/ViewDoc.jsp?id=1410627&Site=CM, accessed 14-03-2011

9. Albrecht, S., Kohlrausch, N., Kubicek, H., Lippa, B.: eParticipation – Electronic Participation of Citizens and the Business Community in eGovernment. Institut fur Informationsmanagement Bremen (2008)

10. UNPAN: UN e-government survey 2008: From e-government to connected governance. United Nations, New York (2008)

11. West, D.M.: E-Government and the Transformation of Service Delivery and Citizen Attitudes. Public Administration Review 64, 15-27 (2004)

12. UNPAN: Benchmarking e-Government – A Global Perspective. United Nations, New York (2002)
13. Capgemini: The User Challenge Benchmarking The Supply of Online Public Services, 7th measurement (2007)

14. Accenture: Leadership in Customer Service: New Expectations, New Experiences (2005)

15. Potnis, D.D.: Measuring e-Governance as an innovation in the public sector. Government Information Quarterly 27 (1), 41-48 (2010)

16. Janssen, M.: Measuring and Benchmarking the Back-end of E-Government. In: Wimmer, M.A. et al. (eds.) EGOV 2010, LNCS 6228, pp. 156-167 (2010)

17. Magd, H., Curry, A.: Benchmarking: achieving best value in public-sector organizations. Benchmarking: An International Journal 10, 261-286 (2003)

18. Hoogwout, M.: Super Pilots, Subsidizing or Self-Organization: Stimulating E-Government Initiatives in Dutch Local Governments. Paper presented at EGOV 2003, Prague, Czech Republic (2003)

19. Grant, G., Chau, D.: Developing a Generic Framework for E-Government. Journal of Global Information Management 13(1), 1-30 (2005)

20. Bannister, F.: The curse of the benchmark: an assessment of the validity and value of e-government comparisons. International Review of Administrative Sciences 73, 171-188 (2007)

21. Janssen, D., Rotthier, S., Snijkers, K.: If you measure it, they will score: an assessment of international eGovernment benchmarking. Information Polity 9, 121-130 (2004)

22. Aichholzer, G., Kozeluh, U.: Evaluating e-Democracy. In: EuroSpace Group (ed.) E-Gov. 2.0: pave the way for e-Participation, pp. 50-55. EuroSpace Group, Rome (2009)

23. Loukis, E., Xenakis, A.: Evaluating Parliamentary e-Participation. Paper presented at the Third International Conference on Digital Information Management, pp. 806-812. London, 13-16 November 2008

24. Ferguson, R., Griffiths, B., Miller, L.: Digital Dialogues, Second Phase Report, August 2006 – August 2007: An independent investigation into the use of online technologies to promote dialogue between central government and the public. Hansard Society (2006)

25. Islam, M.: Towards a sustainable e-Participation implementation model. European Journal of ePractice 5 (2008)

26. Li, C., Bernoff, J.: Groundswell: winning in a world transformed by social technologies. Forrester Research Inc., Harvard Business School Publishing, Boston (2008)

27. Kafentzis, K., Goulandris, V., Alexopoulos, P.: E-dialogos: an holistic approach for enhancing e-Participation in local government. In: EuroSpace Group (ed.) E-Gov. 2.0: pave the way for e-Participation, pp. 162-171. EuroSpace Group, Rome (2009)

28. Rentinck, L., Otto, W., Langenhoff, J., Sieverts E.: E-overheid lijkt consistent, maar vergeet vaak de klant. Informatie Professional 11, 12-27 (2007)

29. Tambouris, E., Liotas, N., Tarabanis, K.: A Framework for Assessing eParticipation Projects and Tools. In: Proceedings of the 40th Hawaii International Conference on System Sciences, Waikoloa, Big Island, Hawaii, 3-6 January (2007)

30. Bannister, F.: ICT Hyperbole and the Red Queen Syndrome. In: EuroSpace Group (ed.) E-Gov. 2.0: pave the way for e-Participation, pp. 115-125. EuroSpace Group, Rome (2009)

31. Parycek, P., Rupp, C.: E-democracy: ICT and policies for participation, e-Democracy in Austria. In: EuroSpace Group (ed.) E-Gov. 2.0: pave the way for e-Participation, pp. 56-62. EuroSpace Group, Rome (2009)

32. Central Office of Information: Effective public engagement: A guide for policy-makers and communication professionals (2009) http://coi.gov.uk/documents/guidance/effective-public-engagement.pdf, accessed 14-03-2011.

33. Bruijn, J.A. d., Heuvelhof, E.F. t., Veld, R.J. i.t.: Procesmanagement. Over procesontwerp en besluitvorming. Academic service, Schoonhoven (1998)