# Passive Expert-Sourcing for Policy Making in the European Union

Aggeliki Androutsopoulou, Francesco Mureddu, Euripidis Loukis, Yannis Charalabidis

► To cite this version:

Aggeliki Androutsopoulou, Francesco Mureddu, Euripidis Loukis, Yannis Charalabidis. Passive Expert-Sourcing for Policy Making in the European Union. 8th International Conference on Electronic Participation (ePart), Sep 2016, Guimarães, Portugal. pp.162-175, 10.1007/978-3-319-45074-2_13 hal-01637231

HAL Id: hal-01637231

https://inria.hal.science/hal-01637231

Submitted on 17 Nov 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Passive Expert-Sourcing for Policy Making
in the European Union

Aggeliki Androutsopoulou¹, Francesco Mureddu², Euripidis Loukis¹, and Yannis Charalabidis¹

¹Department of Information and Communication Systems Engineering,
University of the Aegean, Greece
{ag.andr,eloukis,yannisx}@aegean.gr

²Information and Communication Science Department,
Universitat Oberta de Catalunya, Spain
fmureddu@uoc.edu

**Abstract.** The public sector gradually starts exploiting the crowdsourcing ideas initially developed in the private sector. However, there is much less knowledge on efficient and effective methods and practices for public sector citizen-sourcing in comparison with private sector crowdsourcing, so extensive research is required in this area. This paper contributes to filling this research gap, by presenting an ICT-based method for ‘passive expert-sourcing’, with the latter term denoting the collection of policy relevant information, knowledge and ideas from experts, which aims at supporting policy making by the European Union (EU) by leveraging its large policy community. Its theoretical foundation is previous theoretical work on the relationships between democracy and technocracy, and also on policy networks. The main technological pillars of the proposed method are: EU policy experts’ profiling and reputation management, relevant documents’ opinion mining and relevance rating, and finally advanced visualized presentation of them. Finally, a first evaluation of the proposed method is presented, leading to encouraging results.

**Keywords:** crowd-sourcing, citizen-sourcing, public policy, technocracy, policy network, reputation management, opinion mining

## 1 Introduction

Crowd-sourcing has been initially developed in the private sector, in order to exploit external information, knowledge and ideas possessed by ‘crowds’ of individuals for problem solving or for the development of innovations [17,18], [4,5,6]. It is defined as ‘a new web-based business model that harnesses the creative solutions of a distributed network of individuals, in order to exploit ‘collective wisdom’ and mine fresh ideas from large numbers of individuals’ [4]. There has been extensive research and practice in this area, which has led to the development of efficient and effective crowd-sourcing methods and practices, and also to gaining a deeper understanding of them, their value, advantages and disadvantages, and also the specific contexts and kinds of problems for
which each of them is more appropriate for; comprehensive reviews of this research are
provided by Rouse [36], Hetmank[15], Tarrell et al.[41], Pedersen et al. [29] and Re-
chenberger et al. [35].

The public sector, motivated by the multiple ‘success stories’ of this new knowledge sourcing and innovation paradigm, and also by the increasing complexity of social problems and needs, has started making some first steps in this direction, by introducing forms of ‘citizen-sourcing’ in their policy making processes (see section 2.1 for a brief review of relevant literature). However, there is much less knowledge on efficient and effective methods and practices for public sector citizen-sourcing in comparison with private sector crowd-sourcing. It is therefore necessary to conduct extensive research in this area, in order to develop highly efficient ICT-based methods for this purpose, which enable the effective collection of policy relevant information, knowledge and ideas from citizens, and then the advanced processing of them in order to calculate useful policy analytics, which can provide substantial support for public policy making, and addressing its specific needs. In general, it is important in the area of public sector citizen-sourcing to reach a level of knowledge, efficiency, effectiveness and maturity, comparable to those of the private sector crowd-sourcing.

This paper contributes to filling this research gap, by presenting an ICT-based method for ‘passive expert-sourcing’, with the latter term denoting the collection of policy relevant information, knowledge and ideas from experts, in order to support policy making by the European Union by leveraging its large policy community. The first evaluations of citizen-sourcing initiatives [12], [23], [1] have shown that they can provide valuable insights into the perceptions of the general public concerning important social problems and government activities for addressing them, and also existing and proposed public policies for addressing them. However, it is recommended that in order to collect information and knowledge of higher quality, it is necessary to target more knowledgeable communities having strong interest and good expertise on the particular topic/policy under discussion. On this question there has been extensive political sciences research, which has revealed the importance of both democracy (i.e. political consultation with all stakeholder groups) and technocracy (i.e. specialized knowledge of experts) for the development of effective public policies for addressing the complex problems of modern societies [13], [14], [7]. At the same time this research highlights the need of a relationship and balance between them (see section 2.2 for more details on ‘Democracy vs Technocracy in Public Policy Making’). Furthermore, another stream of political sciences research has examined the emergence of policy networks, as a result of the increasing complexity of social problems, in which participate various both governmental actors and non-governmental actors (such as associations of various businesses, professions, labor unions and other interest groups), and their increasing importance for the design and implementation of public policies; each of these actors has developed valuable expertise, usually focused on its particular perspectives and concerns, which can be quite useful for public policy making [40], [33,34] (see section 2.3 for more details on ‘Public Policy Networks’). So, our research has as theoretical foundation this abovementioned previous work on the relationships between democracy and technocracy, and also on the public policy networks.

The research presented in this paper has been conducted as part of the European
research project ‘EU-Community’ (project.eucommunity.eu/), partially funded by the
‘ICT for Governance and Policy Modelling’ research programme of the EU. This paper
is structured in six sections. In the following section 2 the background of our research
is outlined (on public sector citizen-sourcing, democracy vs technocracy in public pol-
icy making, and also public policy networks). Then in section 3 is presented the meth-
odology we adopted for the design of the proposed method and then for a first evalua-
tion of it. In section 4 the proposed ICT-based method of passive expert-sourcing is
described, followed by the results of a first evaluation of it in section 5. Finally, in
section 6 conclusions are summarized and directions for further research are proposed.

2 Background

## 2.1 Public Sector Citizen-Sourcing

As mentioned previously in the Introduction, for public sector citizen-sourcing there is
a lack of research similar to the one that has been conducted for private sector crowd-
sourcing, having similar levels of breadth and depth, probably because the former is a
more recent phenomenon than the latter. Limited research has been conducted concern-
ing the application of crowd-sourcing ideas in the public sector, the development of
efficient and effective methods and practices for this purpose, and the evaluation of
them from various perspectives [24], [16], [22], [28], [39], [27], [12], [37].

Most of the existing literature on ICT-based methods for citizen-sourcing by gov-
ernment agencies is focusing on ‘active citizen-sourcing’, which aim at the use of gov-
ernment agencies’ web-sites or social media accounts, in order to pose a particular so-
sial problem or public policy direction, and solicit relevant information, knowledge,
opinions and ideas from citizens. In this direction Mergel and Desouza [27] describe
and analyze the Challenge.gov initiative the U.S. Office of Management and Budget,
which aimed at applying private sector crowdsourcing ideas in the public sector, in
order to source from citizens ideas, knowledge, and solutions for specific challenges
that government faces. This initiative was based on an ICT platform allowing U.S. fed-
eral agencies to launch contests, and at the same time citizens to find appropriate con-
tests and participate in them providing solutions, or reviewing and evaluating solutions
provided by others, voting on solutions, and even getting involved in the implementa-
tion of solutions and subsequent evaluation of them. Charalabidis and Loukis [8] and
Ferro et al. [12] propose a method for the systematic, intensive and centralized exploi-
tation of web 2.0 social media by government agencies on public policies (existing or
under development). This method is based is on a central ICT platform, which a) pub-
lishes automatically various types of policy-related content (e.g., short text long text,
images, video) in multiple social media accounts of a government agency, using the
application programming interfaces (API) of these social media, soliciting citizens’
feedback on them; and b) similarly collects automatically from them data on citizens’
interactions with this content (e.g., views, comments, ratings, votes, etc.), and makes
advanced processing of them. However, more recently, there has been some interest in
‘passive citizen-sourcing’, which aims to exploit political content that has been devel-
oped by citizens freely, without any direct stimulation or direction by government, in
various external (= not belonging to government agencies) web-sites or social media,
such as political fora, news web-sites, political blogs, Facebook, Twitter, etc. accounts
[9] [44].

Therefore, extensive further research is required in the area of public sector citizen-
sourcing, in order to develop a considerable knowledge base on it, and through it a level
of efficiency and effectiveness, comparable to those of the private sector crowd-sourc-
ing area. Our research makes a contribution in this direction, by developing an ICT-
based method of passive expert-sourcing.

## 2.2 Democracy vs Technocracy in Public Policy Making

The increasing complexity of the problems of modern societies, the globalization of the
economy and the development of technology have increased the need for and im-
portance of knowledge and expertise for the design and implementation of public poli-
cies [13], [30,31], [14], [7], [21]. This has led to the establishment of various expert
bodies (in government agencies competent for the formulation of public policies, and
also the other public policy stakeholders, such as associations of professions, labor un-
ions, businesses and other interest groups), which can have various forms, from com-
mittees to separate organizations (e.g. economic institutes). These expert bodies have
become today highly important for and influential on the formulation of public policies,
and this is termed as ‘technocracy’ [13], [21]. So today it is widely recognized that the
two fundamental bases of public policy making are democracy (i.e. political consulta-
tion with stakeholder groups) and technocracy (i.e. knowledge of experts).

However, political sciences research in this area has highlighted the need of a rela-
tionship and balance between them, as each of them needs inputs from the other. In
particular, participants in the democratic processes (various stakeholder groups, and
even active citizens) need relevant knowledge and expertise, and the lack of them can
have quite negative impacts [42]. At the same time the experts dealing with a particular
social problem/public policy also need inputs from the political process, concerning
diverse values and concerns of different stakeholder groups, and also their diverse per-
spectives, approaches and ideologies. For the above reasons Brown [7] argues that de-
mocracy and technocracy are not in conflict, and their combination is a necessity in
todays’ realities of highly complex social problems and needs, and globalization; they
generate different kinds of knowledge, which are both necessary for public policy mak-
ing. The ICT can be very useful for the required exchange of knowledge between de-
mocracy and technocracy. Our research contributes in this direction, developing an
ICT-based method for transfer of knowledge from the latter to the former.

## 2.3 Public Policy Networks

The increased complexity of social problems and needs also led governments to realize
that their ‘classical’ unilateral modes of governance are insufficient, and they need
knowledge resources and cooperation of non-state actors (initially economic actors and later other social actors as well) in order to design and implement effective policies, and this resulted in the development of public policy networks [40], [33, 34]. They are defined as sets of formal and informal institutional linkages between various both governmental actors and non-government actors (such as associations of businesses, professions, labor unions and other interest groups) structured around shared interests in public policy-making and implementation. In public policy networks the non-state actors provide to the state actors on one hand information, knowledge and expertise, and on the other hand support for the design and implementation of public policies, and legitimization of them; in return the former have the opportunity to influence the public policies (e.g. legislation, allocation of government financial resources) towards directions that are beneficial to them [2], [19], [43], [25].

A critical characteristic of a network is the density of interactions among its participants; according to [25] higher density of interactions has positive impact on the time-wise stability of the network, the development of shared values and beliefs concerning desirable policy objectives and instruments, and finally the effectiveness and outcomes of the network. Therefore, ICT can be very useful for increasing this critical characteristic at a low cost, and supporting the exchange of diverse expertise and knowledge among participants. Our research makes a contribution in this direction, developing an ICT-based method that supports the exchange of expertise and knowledge between the actors participating in a policy network.

## 3 Design and Evaluation Methodology

In order to design this ICT-based method of expert-sourcing and its supporting ICT platform thirteen workshops have been organized, with the first five of them aiming to gain a better understanding of the structure of the EU policy community, and then the next eight aiming to collect the requirements of potential users of our method and ICT platform, as part of the preparation and the implementation of the abovementioned EU-Community project. The EurActiv.Com (a leading EU policy online media network (www.euractiv.com), which participates as partner in the EU-Community project, and the Fondation EurActiv Politech (a public service foundation (www.euractiv.com/fondation) having as main mission ‘to bring together individuals and organizations seeking to shape European Union policies’, participating also as partner in this project, were the organizers of these workshops. The participants were representatives of important EU policy stakeholders (such as industry federations), members of the advisory boards of EurActiv.Com and Fondation EurActiv Politech, thematic experts in various EU policies, policy analysts, registered users of EurActiv.Com portals, and also permanent staff of various hierarchical levels from the European Commission.

In order to conduct an initial evaluation of the proposed method and its supporting ICT platform an evaluation session was organized with the participation of similar potential users. During this session the proposed method was introduced to the audience, together with the supporting ICT platform, and some first applications with their results. Then the participants had the opportunity to interact with the ICT platform by executing
a set of predefined usage scenarios, under the observation of the organizers who sup-
ported them, and recorded any comments or difficulties, and as well feedback on pos-
sible improvements. Finally, we collected evaluation data from the participants in this
session using mainly quantitative techniques, as they enable condensing and summa-
rizing a large quantity of evidence in a few numbers that enable the easier drawing of
conclusions [26], [32]. We developed our evaluation framework based on the models
developed in previous technology acceptance research [10], [38], [20], which has con-
cluded that the intention to use a new technology, is determined mainly by two factors:
its perceived ‘ease of use’ (= the degree to which potential users believe that using it
would require minimal effort) and its perceived ‘usefulness’ (= the degree to which
potential users believe that using it will enhance their job performance). So our evalu-
ation framework has been based on these three factors, which have been elaborated and
analyzed into several detailed questions, taking into account the particular objectives
and specificities of the proposed method. This elaboration has been made separately for
each the two main components that the users of the ICT platform can access: the
Euractory (which enables users’ registration and reputation calculation, rating other us-
ers and also searching for experts on a topic) and the PolicyLine (which provides a
visualization of document search results). Based on the above evaluation framework a
questionnaire was designed to be filled by the session participants; the questions of the
framework were converted to positive statements, and the respondents were asked to
provide the degree of their agreement/disagreement with each of them in a five-levels
scale (1 = totally disagree, 2 = disagree, 3 = neutral, 4 = agree, 5 = totally agree). The
questionnaire is shown in Table 1. The data collected through the questionnaire were
processed using Excel. Furthermore, after filling this questionnaire a qualitative discus-
sion was conducted with the participants on the same questions, in order to get a deeper
insight of their perceptions.

4 A Passive Expert-sourcing Method

4.1 Description

From the interviews we conducted (see first paragraph of section 3) it was concluded
that the main need of EU policy stakeholders is to be better informed on the most
knowledgeable and credible experts on a policy related topic they are interested in, and
also the most relevant documents on such a topic; it will also be useful if these docu-
ments are associated with the various stages of the E.U. policy processes. Since experts
usually do not have time to generate new content on a topic (social problem or public
policy – existing or under development) they are interesting in, the use of ‘active citi-
zen-sourcing’ would not be possible. Therefore, a ‘passive citizen-sourcing’ approach
should be adopted, based on the retrieval, processing and exploitation of already exist-
ing experts’ generated content. So, the proposed method is based on retrieving auto-
matically information from various sources about experts on policy related topics, and
then collecting the knowledge and opinions they share online through texts and postings
in multiple web-sites and social media they are using. This can be achieved by crawling
at regular time intervals the most relevant external sources of knowledgeable and cred-
ible experts on EU policies, and also of relevant documents of various types, and update
automatically the corresponding databases of a supporting ICT platform. The practical
application of this method will lead to the collection of a large amount of information
concerning policy experts and content generated by them. So, it is important to apply
automated state-of-the-art techniques for processing and classifying this content, in or-
der to extract interesting insights and knowledge from it concerning social problems
and public policies. This textual content of documents, articles and social media posts
will be processed using opinion mining and sentiment classification methods, in order
to identify subjective information, extract opinions, identify the polarity of their orien-
tation (positive, negative or neutral) and assess the relevance of them with relation to a
topic (see [10] for more details). Furthermore, for the experts it is necessary to apply
digital reputation techniques for assessing their reputation/credibility and provide a
ranking of them per topic of interest. By storing the above data in a common database,
and enabling search of it by the users and visual presentation of the results, public policy
stakeholders will be able to find useful expert knowledge on complex policy debates,
e.g. the most reputable/credible experts or the most relevant documents on a specific
topic. An overview of the proposed method is shown in Figure 1.

<figure><img src="image_2.png" /><figcaption>**Fig. 1.**An overview of the proposed method of passive expert-sourcing</figcaption></figure>

The proposed method aims to foster collaboration and knowledge sharing among the different policy stakeholders on E.U. policy topics. For this purpose, in order to enable focus on a particular policy topic of interest, our method uses the concept of ‘policy process’, under which all relevant information on experts and documents is collected and clustered. In particular, as a policy process can be modelled any ongoing or completed EU legislative procedure, or political debate in general, on a topic, while each topic can be associated with one or more policy processes. Policy processes can be initiated by any policy stakeholder in order to enable the interconnection and presentation of all relevant information and aspects of policy consultations in a structured way.

4.2 ICT Platform

An ICT platform has been designed for supporting the implementation of the above
method. It consists of two main components accessible by the users, called ‘EurActory’
and ‘PolicyLine’, with each of them including several sub-components, and also an additional component called ‘CurActory’, which is not directly accessible by the users, and includes the database storing the above information and the functionality for updating it by crawling pre-defined external sources at regular time intervals.

The EurActory component collects and maintains a directory of profiles of people with high levels of knowledge, expertise and credibility in one or more topics related with EU policies, usually having an active role in policy making processes at European level. According to their role they are categorized into the three types, which have been identified through our workshops: influencers, analysts and institutional decision makers. These people are included in the people database of the system in the CurActory component automatically by the crawlers sub-component, or manually by using the self-registration capabilities provided by the EurActory component. The crawlers component crawls at regular time intervals various external sources, which can be pre-defined websites (e.g. Euractiv.com, EUR-Lex, Europa Whoiswho directory, RSS Feeds, blogs and news sites) or social media accounts (e.g. LinkedIn, Twitter), updating the CurActory database, which makes the relevant discovered information available through the EurActory or the PolicyLine.

Furthermore, this EurActory component provides rankings of the expert profiles according to their expertise on a set of topics of interest, through the ‘reputation score’ calculated by the Reputation Management sub-component for each expert per topic, based on the following criteria:

* Self-evaluation: direct input from the user on his/her own area of expertise.

* Peer-assessment: based on endorsements from other users made through EurActory

* Business Card Reputation: based on the reputation ranking of the organization and the user's position in the organization's hierarchy

* Document Assessment: results of authored documents' assessment by their readers

* Network Value: level of influence as the sum of network connections

* Proximity trust: level of connection in social media

* Past Measurements: taking into account reputation in previous months (its stability means credibility).

* Offline Reputation: manually added for persons with no online presence

Also, the EurActory provides the following capabilities to registered users:

- Search for an expert profile, by name, EU policy or topic, which returns experts found in descending reputation score order (i.e. the most reputable first).

- View an expert profile; the profile pages can also be shared on social media.

- Create own profile and curate personal information, connect social media accounts, claim expertise topics.

- Activation of an expert profile that has been already created by the system administrators, after the discovery of it by the crawlers, and also update of profile details.

The ‘PolicyLine’ component provides state-of-the-art visualization of policy relevant documents, which are structured according to policy processes, aiming to provide to the user a better understanding of the multi-actor processes related with the EU decision making procedures and policy debates. Therefore, in the core of PolicyLine functionality is the concept of ‘policy process’ (described previously in 4.1). We can have
documents manually attached by users of the ICT Platform, to the specific ‘policy pro-
cess’, as well as automatically discovered ones by the abovementioned crawlers sub-
component, which searches on regular basis multiple web-sites and social media ac-
counts in order to find significant documents published online (media articles, reports,
tweets, policy proposals, legislative documents), authored by the experts’ categories
mentioned above.

In particular, PolicyLine provides statistical information for each policy process se-
lected by the user, such as the total number of relevant documents and the number of
visits of users on the specific policy process page. For a more detailed view, PolicyLine
offers a timeline visualization (see Figure 2), which structures the main documents
(based on relevance as well as author’s reputation) associated with this policy process
in a temporal order, and clusters them under a set of user defined stages of the particular
policy process. It also provides information with respect to their authorship (colors are
used for this purpose to reflect different authors’ categories and sub-categories), and
also shapes (such as rectangles and circles) to reflect different types of documents (e.g.
rectangles reflect the proposal documents, while general documents are represented by
circles). Documents sub-categories are defined concerning the type of organization
from which each document is originated (e.g. European Institution, National/Local
Governance, Academic Institution, Civil Society Organization, Media, etc.). The sizes
of the shapes representing these documents reflect their relevance and author’s reputa-
tion (so more relevant documents written by more reputable authors are shown bigger).

<figure><img src="image_3.png" /><figcaption>Fig. 2. PolicyLine timeline visualization</figcaption></figure>

Moreover, a user can select a particular document in order to view more details about
it, including the results from the sentiment classification provided by the opinion mining sub-component (a linguistic analysis of the textual content of a document leads to an estimation of the polarity of the underlying text – see [10] for more details) and the relevant input provided by other platform users. In particular, PolicyLine for each document provides an interface where users can rate its accuracy, value, relevance and timeliness, and also enter comments on the document, so that an informal discussion on it can be stimulated.

## 5 Evaluation Results

In Table 2 are shown the results of the processing of the data collected through the evaluation questionnaire (average ratings for all questions) for the two main components that can be accessed by the users, EurActory and PolicyLine.

<table><caption>**Table 1.** Evaluation Results</caption><tbody><tr><th rowspan="12" scope="rowgroup">EurActory</th><th colspan="2" scope="colgroup">Ease of Use Perspective</th></tr><tr><td>EurActory can be easily used without assistance</td><td>3.46</td></tr><tr><td>Creating a profile is easy</td><td>4.08</td></tr><tr><td>It is easy to access topic listings</td><td>4.15</td></tr><tr><td>It is easy to rate peers</td><td>3.75</td></tr><tr><td>Using EurActory has been a positive experience</td><td>4.08</td></tr><tr><th colspan="2" scope="colgroup">Usefulness</th></tr><tr><td>EurActory puts together information not found or collected under one roof elsewhere</td><td>3.15<br>3.38</td></tr><tr><td>EurActory allows me to be more productive</td><td>3.46</td></tr><tr><td>EurActory improves the quality of my work</td><td>3.85</td></tr><tr><td>EurActory assists me in identifying relevant experts</td><td>3.54</td></tr><tr><td>EurActory provides me with all the needed information on relevant experts</td><td>3.54</td></tr><tr><td>EurActory enables me to reinforce my expert positioning</td><td></td></tr><tr><th colspan="2" scope="colgroup">Intention to use</th></tr><tr><td>I expect to use EurActory on a regular basis in the future</td><td>3.85</td></tr><tr><td>I will advise colleagues to use EurActory</td><td>3.62</td></tr><tr><th rowspan="12" scope="rowgroup">PolicyLine</th><th colspan="2" scope="colgroup">Ease of Use Perspective</th></tr><tr><td>PolicyLine can be easily used without assistance</td><td>3.64</td></tr><tr><td>I can easily create a 'policy process'</td><td>3.69</td></tr><tr><td>I can easily add a document in the 'policy process'</td><td>3.79</td></tr><tr><td>I can easily rate/comment a document</td><td>3.5</td></tr><tr><td>I can easily get an overview of the process</td><td>3.73</td></tr><tr><td>Using PolicyLine has been a positive experience</td><td>3.71</td></tr><tr><th colspan="2" scope="colgroup">Usefulness</th></tr><tr><td>PolicyLine puts together information not found or collected under one roof elsewhere</td><td>3.29<br>3.29</td></tr></tbody></table>
<table><tr><td>PolicyLine allows me to be more productive<br/>PolicyLine improves the quality of my work</td><td>3.43</td></tr><tr><td><strong>Intention to use</strong></td><td></td></tr><tr><td>I expect to use PolicyLine on a regular basis in the future</td><td>4.14</td></tr><tr><td>I will advise colleagues to use PolicyLine</td><td>3.71</td></tr></table>

We can see that the respondents find the ease of use of the EurActory component high (the average rating of relevant questions is 3.9), and for the PolicyLine component moderate to high, but closer to the latter (the average rating of the relevant questions is 3.67). Slightly lower are their perceptions with respect to usefulness, which it is perceived as moderate to high for the EurActory component (average rating of relevant questions 3.5), and moderate to high, but closer to the former, for the PolicyLine component (average rating of relevant questions 3.3). Finally, high is the intention to use again the PolicyLine component again (average rating of relevant questions 3.9), and slightly lower for the EurActory component (average rating of relevant questions 3.75).

In the qualitative discussion with the participants of the evaluation session the latter agreed that this ICT platform, and the whole method behind it, constitute an easy to use tool for finding quickly high quality information and opinions on important policy related topics and policy formulation processes, authored by knowledgeable experts, and also debate over them with other users. Furthermore, it enables and promotes communication and exchange of knowledge among EU policy stakeholders. It also allows awareness of and also debate and criticism on policy initiatives carried out by the European Institutions. The usefulness of the EurActory component was assessed a bit higher than the PolicyLine; this probably reflects that the former is easier to use and exploit its capabilities than the latter. The participants were in general satisfied with the proposed method and its supporting infrastructure, and expressed interest in using again the functionalities of both components. However, future improvements were suggested, concerning the graphical interface and especially the timeline visualization.

# 6 Conclusions

In the previous sections of this paper an ICT-based method for 'passive expert-sourcing' was presented, which allows the collection of high quality policy relevant information, knowledge and ideas from knowledgeable experts, aiming at supporting policy making in the European Union (EU) by leveraging its large policy community. Its theoretical foundation is previous theoretical work on the relationships and the required balance between democracy and technocracy, and also on policy networks. The proposed method is based on EU policy experts' profiling and reputation management, relevant documents' opinion mining and relevance rating, and finally advanced visualized presentation of them. At the political level, its objective is to enable a better interconnection of the two important bases of modern public policy making, the democratic processes and the technocratic expertise (which is of critical importance, as discussed in more detail in section 2.2), by supporting the transfer of knowledge from the latter to the former. In particular, it aims to support the efficient and effective retrieval by
various actors of the democratic processes (e.g. representatives of stakeholder groups, journalists, government employees, active citizens, etc.) of diverse expert information, knowledge and ideas on a specific topic/policy, which is included in postings and texts authored by knowledgeable experts and published in various web-sites and social media. Furthermore, the proposed method of passive expert-sourcing aims to increase the density of interactions among the actors participating in public policy networks, which is highly important for their stability, the development of shared values and beliefs, and finally the effectiveness and the outcomes of such networks (as discussed in more detail in section 2.3), by supporting the exchange of expertise and knowledge between network participants.

Also, a first evaluation of the proposed method has been presented, which gave encouraging results, with respect to its ease of use and usefulness. However, further evaluation of this method is required, based on realistic pilot applications of it, in order to assess better its value and potential with respect to its abovementioned ambitious objectives: to what extent it enables and supports the transfer of information, knowledge and proposals from experts to the participants in the democratic processes of modern policy making, and under what conditions? to what extent it can enable and support the exchange of information, knowledge and proposals among the participants in public policy networks, and under what conditions ? to what extent can this method can assist the EU institutions to collect high quality information, knowledge, opinions and proposals from their policy networks? Research in this directions is already in progress as part of the abovementioned EU-Community project. Also, further research is required concerning the use of ICT for the transfer of knowledge in the opposite direction, from the democratic processes towards experts/technocracy (e.g. concerning diverse needs, values and concerns of different stakeholder groups on the particular social problem/public policy the experts analyze, and also existing diverse perspectives, approaches and ideologies). This is quite important for the construction of better and more multi-dimensional comprehensive and inclusive expert analyses and plans, which do not miss or neglect important aspects of the social problems or public policies they are dealing with and might be quite important for large social groups, and also do not underestimate existing diverse perspectives, approaches and ideologies.

## References

1. Androutsopoulou, A., Charalabidis, Y., Loukis, E.: Using Social Media Monitoring for Public Policy Making – An Evaluation. Proceedings of 9th Mediterranean Conference on Information Systems (MCIS 2015), October 2015, Samos, Greece (2015)

2. Atkinson, M., Coleman, W.: Strong States and Weak States: Sectoral Policy Networks in Advanced Capitalist Economies. British Journal of Political Science, 19, 47-67 (1989)

3. Bekkers, V., Edwards, A., de Kool, D.: Social media monitoring: Responsive governance in the shadow of surveillance? Government Information Quarterly, 30(4), 335-342 (2013)

4. Brabham, D. C.: Crowdsourcing as a Model for Problem Solving: An Introduction and Cases. Convergence: The International Journal of Research into New Media Technologies, 14 (1), 75-90 (2008)
5. Brabham, D. C.: Crowdsourcing: A Model for Leveraging Online Communities. In: A. Delwiche and J. Henderson (Eds.) The Routledge Handbook of Participative Cultures, Routledge (2012)

6. Brabham, D. C.: Crowdsourcing. Cambridge, MA: The MIT Press (2013)

7. Brown, M. B.: Science in Democracy: Expertise, Institutions, and Representation. Cambridge, MA: MIT Press (2009)

8. Charalabidis, Y., Loukis, E.: Participative Public Policy Making Through Multiple Social Media Platforms Utilization. International Journal of Electronic Government Re-search, 8(3), 78-97 (2012)

9. Charalabidis, Y., Loukis, E., Androutsopoulou, A., Karkaletsis, V., Triantafillou, A.: Passive Crowdsourcing in Government Using Social Media. Transforming Government: People, Process and Policy, (8)2, 283-308 (2014)

10. Charalabidis, Y., Maragoudakis, M., Loukis, E.: Opinion Mining and Sentiment Analysis in Policy Formulation Initiatives: The EU-Community Approach, IFIP eParticipation Conference EPART2015, September 2015, Thessaloniki, Greece (2015)

11. Davis, F. D.: Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly 13(3), 319-339 (1989)

12. Ferro, E., Loukis, E., Charalabidis, Y., Osella, M.: Policy Making 2.0: From Theory to Practice. Government Information Quarterly, 30(4), 359-368 (2013)

13. Fischer, F.: Technocracy and the Politics of Expertise. London: Sage (1990)

14. Harcourt, A., Radaelli C.: Limits to EU technocratic regulation?, European Journal of Political Research, 35, 107-122 (1999)

15. Hetmank, L.: Components and Functions of Crowdsourcing Systems – A Systematic Literature Review. Wirtschaftsinformatik 2013 Conference Proceedings (2013)

16. Hilgers, D., Ihl, C.: Citizensourcing: Applying the concept of open innovation to the public sector. The International Journal of Public Participation, 4(1), 67–88 (2010)

17. Howe, J.: The Rise of Crowdsourcing. Wired, 14(6). Retrieved from http://www.wired.com/wired/archive/14.06/crowds.html (2006)

18. Howe, J.: Crowdsourcing, why the power of the crowd is driving the future of business. Crown Business, New York (2008)

19. Howlett, M.: Do Networks Matter? Linking Policy Network Structure to Policy Outcome: Evidence from Four Canadian Policy Sectors 1990-2000. Canadian Journal of Political Science, 35, 2, 235-267 (2002)

20. Hsiao, C. H., Yang, C.: The intellectual development of the technology acceptance model: A co-citation analysis. International Journal of Information Management, 31(2), 128-136 (2011)

21. Kurki, M.: Democracy Through Technocracy? Reflections on Technocratic Assumptions in EU Democracy Promotion Discourse. Journal of Intervention and Statebuilding, 5(2), 211-234 (2011)

22. Linders, D.: From e-government to we-government: Defining a typology for citizen coproduction in the age of social media. Government Information Quarterly, 29, 446-454 (2012)

23. Loukis, E., Charalabidis, Y., Androutsopoulou, A.: An Analysis of Multiple Social Media Consultations in the European Parliament from a Public Policy Perspective. European Conference on Information Systems (ECIS) 2014, Tel Aviv, Israel (2014)

24. Lukensmeyer, C. J., Torres, L. H.: Citizensourcing: Citizen participation in a networked nation. In K. Yang, & E. Bergrud (Eds.), Civic engagement in a network society (pp. 207-233). Charlotte, NC: Information Age Publishing (2008)

25. Marsh, D., Smith, M.: Understanding Policy Networks: towards a Dialectical Approach. Political Studies, 48, 4-21 (2000)
26. Maylor, H., Blackmon, K.: Researching Business and Management. Palgrave-Macmillan, New York, USA (2005)

27. Mergel, I., Desouza, K. C.: Implementing Open Innovation in the Public Sector: The Case of Challenge.gov. Public Administration Review, 73(6), 882–890 (2013)

28. Nam, T.: Suggesting frameworks of citizen-sourcing via Government 2.0. Government Information Quarterly, 29, 12–20 (2012)

29. Pedersen, J., Kocsis, D., Tripathi, A., Tarrell, A., Weerakoon, A., Tahmasbi, N., Xiong, J., Deng, W., Oh, O., Vreede, G. J.: Conceptual foundations of crowdsourcing: A review of IS research. Proceedings of the 46th Annual Hawaii International Conference on System Sciences, Maui, HI, USA (2013)

30. Radaelli, C. M.: The Role of Knowledge in the Policy Process. Journal of European Public Policy, 2(2), 159-183 (1995)

31. Radaelli, C. M.: The Public Policy of the European Union: whether Politics of Expertise? Journal of European Public Policy, 6(5), 757-774 (1999)

32. Ragin, C., Amoroso, L.: Constructing Social Research: The Unity and Diversity of Method 2nd Edition. Pine Forge Press – Sage Publications, California, USA (2011)

33. Rhodes, R. A. W.: Policy Network Analysis. In M. Moran, M. Rein and R. E. Goodin (Eds.) The Oxford Handbook of Public Policy. Oxford: Oxford University Press), pp. 423-445 (2006)

34. Rhodes, R. A. W.: Understanding Governance: Ten Years On. Organization Studies, 28, 8, 1243–1264 (2007)

35. Rechenberger, T., Jung, V., Schmidt, N., Rosenkranz, C.: Utilizing the Crowd – A Literature Review on Factors influencing Crowdsourcing Initiative Success. Pacific Conference on Information Systems (PACIS) 2015 Proceedings. Paper 250 (2015)

36. Rouse, A. C.: A Preliminary Taxonomy of Crowdsourcing. Proceedings of the Austalasian Conference on Information Systems (ACIS) 2010, Paper 76. (2010)

37. Prpić, J., Taeihagh, A., Melton, J.: The Fundamentals of Policy Crowdsourcing. Policy & Internet, 7(3), 340–361 (2015)

38. Schepers, J., Wetzels, M.: A meta-analysis of the technology acceptance model: Investigating subjective norm and moderation effects. Information & Management 44, 90-103 (2007)

39. Seltzer, E., Mahmoudi, D.: Citizen Participation, Open Innovation, and Crowdsourcing: Challenges and Opportunities for Planning. Journal of Planning Literature, 28(1), 3-18 (2012)

40. Skogstad, G.: Policy Networks and Policy Communities: Conceptual Evolution and Governing Realities. Workshop on "Canada's Contribution to Comparative Theorizing" Annual Meeting of the Canadian Political Science Association, University of Western Ontario, London, Ontario (2005)

41. Tarrell, A., Tahmasbi, N., Kocsis, D., Tripathi, A., Pedersen, J., Xiong, J., Oh, O. &Vreede, G. J. d. Crowdsourcing: A Snapshot of Published Research. Proceedings of the Nineteenth Americas Conference on Information Systems (AMCIS) 2013, Chicago, Illinois, USA (2013)

42. United Nations Research Institute for Social Development. Technocratic Policy Making and Democratic Accountability, UNRISD Research and Policy Brief no. 3, United Nations Research Institute for Social Development (UNRISD), Switzerland (2004)

43. Van Waarden, F.: Dimensions and types of policy networks, European Journal of Political Research, 21, 29-52 (1992)

44. Wandhöfer, T., Taylor, S., Alani, H., Joshi, S., Sizov, S., Walland, P., Thamm, M., Bleier, A., Mutschke, P.: Engaging Politicians with Citizens on Social Networking Sites: The We-Gov Toolbox. International Journal of Electronic Government Research, 8(3), 22-43 (2012)