# Data-Driven Policy Making: The Policy Lab Approach

Anne Veenstra, Bas Kotterink

► To cite this version:

Anne Veenstra, Bas Kotterink. Data-Driven Policy Making: The Policy Lab Approach. 9th International Conference on Electronic Participation (ePart), Sep 2017, St. Petersburg, Russia. pp.100-111, 10.1007/978-3-319-64322-9_9. hal-01703333

HAL Id: hal-01703333

https://inria.hal.science/hal-01703333v1

Submitted on 7 Feb 2018

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# Data-Driven Policy Making: The Policy Lab Approach

Anne Fleur van Veenstra and Bas Kotterink

TNO Strategy & Policy, Anna van Buerenplein 1, 2595 DA The Hague, The Netherlands
{annefleur.vanveenstra, bas.kotterink}@tno.nl

**Abstract.** Societal challenges such as migration, poverty, and climate change can be considered ‘wicked problems’ for which no optimal solution exists. To address such problems, public administrations increasingly aim for *data-driven policy making*. Data-driven policy making aims to make optimal use of sensor data, and collaborate with citizens to co-create policy. However, few public administrations have realized this so far. Therefore, in this paper an approach for data-driven policy making is developed that can be used in the setting of a Policy Lab. A Policy Lab is an experimental environment in which stakeholders collaborate to develop and test policy. Based on literature, we first identify innovations in data-driven policy making. Subsequently, we map these innovations to the stages of the policy cycle. We found that most innovations are concerned with using new data sources in traditional statistics and that methodologies capturing the benefits of data-driven policy making are still under development. Further research should focus on policy experimentation while developing new methodologies for data-driven policy making at the same time.

**Keywords:** Data-driven policy making, data for policy, co-creation, Policy Lab.

## 1 Introduction

Today’s society faces complex ‘wicked problems’, such as migration, poverty, and climate change, for which not one optimal solution exists [1,2]. In order to address such problems, governments aim to realize *public sector innovation* that gears them towards becoming platforms of open governance, making optimal use of information and communication technologies (ICTs) to create public value [1]. Increasingly, ICTs are not only used for improving the daily operations of government, but also for enhancing the process of policy making [2]. Policies address societal problems by formulating and implementing laws, rules and guidelines, and policy making is the process of creating and monitoring these policies. Hence, it is often conceptualized as a policy cycle, consisting of several different phases, such as agenda setting, policy formulation, decision-making, implementation and evaluation [3]. ICTs may be used to support and enhance different phases of the policy cycle and enable experimentation [1,2].

*Data-driven policy making* uses ICTs to capture the benefits of new data sources [4,5], and to support collaboration with relevant stakeholders and citizens [2,6,7]. It builds on the notion of evidence-based policy making [see, for instance, 8,9]. In the
literature on evidence-based policy making three types of evidence are considered rel-
evant: “systematic (‘scientific’) research, program management experience (‘practice’),
and political judgement” [9, p. 1]. Data-driven policy making acknowledges the im-
portance of these types of evidence, but can be distinguished from evidence-based pol-
icy making, since it is mainly concerned with the inclusion of big and open data sources
into policy making as well as with co-creation of policy by involving citizens. Data-
driven policy making is not only expected to result in better policies, but also aims to
create legitimacy [10]. Involvement of citizens in a data-driven policy making process
is especially important since public data and statistics are increasingly met by citizens’
distrust [11].

To allow for better collaboration and involve citizens, public administrations around
the world have set up Policy Labs to allow for experimentation and facilitate the in-
volvement of relevant stakeholders [12,13]. They, thus, address the need for experi-
mentation and design-thinking to deal with wicked policy issues [1,2]. Therefore, in
this paper we develop a Policy Lab approach for data-driven policy making. First, based
on literature of public sector innovation, we identify innovations in the use of data for
policy making and co-creation of policy. Secondly, we map these innovations to differ-
ent phases of the policy cycle. And thirdly, we develop an approach that can be used to
guide data-driven policy making in a Policy Lab setting. The next section presents the
theoretical background of public sector innovation. Section 3 discusses data-driven pol-
icy making and identifies innovations. Subsequently, section 4 presents the develop-
ment of the Policy Lab approach, followed by a discussion and recommendations for
further research in section 5. Finally, section 6 presents the conclusion.

## 2 Public Sector Innovation

Public sector innovation holds that “[p]ublic policy and services need to become more
open and innovative as well as being efficient and effective” [1, p. 2], making optimal
use of ICTs [1]. As such, it encompasses a myriad of aspects. Gil-Garcia, Zhang &
Puron-Cid [14] refer to as much as fourteen aspects of smartness in government, in-
cluding evidence-based, technology savviness, openness, citizen engagement, and in-
novation. According to Millard [1], public sector innovation means that public admin-
istrations operate as a platform [15,16] and use ICT to collaborate across organizational
borders [17] and to involve citizens and other relevant stakeholders [6,7,18,19] with the
purpose of creating public value [20-22]. Over the past decades, ICTs have had a great
impact on services delivery [23], opened up public datasets [24] and increased citizens’
participation [25]. The use of ICTs for policy making can, thus, be seen as a next step
in public sector innovation [2].

The use of ICTs benefits policy making in two ways. The first is the use of new data
sources, such as (real-time) sensor data, either physical (e.g. traffic monitoring [2,4]),
or virtual (e.g. social media data [2,6]). “Data-driven decisions and intensive use of
data, through ubiquitous sensing, advanced metering and integrated applications enable
governments to make more informed decisions and improve the effectiveness of public
policies and programs” [14, p. 527]. Secondly, it requires from governments to collab-
orate across organizational borders and with citizens and businesses to enable co-crea-
tion of policies [1,6,7,16]. “Co-creation is understood as the active flow and exchange
of ideas, information, components and products across society (academia, government,
business, civil society and citizens) which allows for a better understanding of partici-
pation, engagement and empowerment in policy development” [1, p. 5].

Besides the deployment of ICTs to use new data sources and enable co-creation of policies, public sector innovation is concerned with the ability of public administrations to experiment, using innovative approaches such as gaming, simulation, and installing of sensors for do-it-yourself measurements, and deploy ‘design-thinking’ [1,2,14]. In order to do so, many public administrations have set up Policy Labs [12,13]. “Policy Labs are emerging structures that construct public policies in an innovative, design-oriented fashion, in particular by engaging citizens and companies working within the public sector” [13, p. 2]. Policy Labs exist in all shapes and sizes and on different levels of government (national, regional and municipal) [13]. The majority of Policy Labs do not focus on a specific type of policy or on a specific phase of the policy cycle, but they employ a design and experimentation based approach to policy making [13]. As such, Policy Labs can be considered as a specific instance of Living Labs, which aim to “support public open innovation processes” [26, p. 90]. While Living Labs are concerned with the involvement of private sector organizations as well as citizens in public open innovation processes in general [26], Policy Labs focus on the involvement of citizens (and also other stakeholders) into the policy making process specifically.

3 Innovations in Data-driven Policy Making

Data-driven policy making thus aims to use new data sources such as (real-time) sensor
data and new techniques for processing these data and to realize co-creation of policies,
involving citizens and other relevant stakeholders. However, realizing data-driven pol-
icy making is complex: many challenges related to the capturing, integration and re-use
of data exist [4,5], as well as to the involvement of citizens and other stakeholders in
policy making [2,6,7]. This section identifies innovations of data-driven policy making
based on literature.

**3.1 Use of new data sources in policy making**

The use of new data sources holds big promises: it is expected to offer organizations
greater operational efficiency and effectiveness, and lead to the development of new
products, services and business models [27-29]. In the context of governments, “we are
faced with a deluge of data that, when combined with new technologies and analysis
techniques, has the potential to inform decision and policy making in unprecedented
ways” [4, p. 10]. Big data is often defined as “vast datasets that cannot be analyzed
using conventional software and analytic tools” [4, p. 2]. Since many ‘big data’ sources
can be stored on a USB-stick nowadays, in the context of public administration, im-
portant characteristics of big data are not so much that they require large processing
power, but more the variety and the interoperability because of its different data sources
and formats [4]. The use of (sensor) data in policy making encompasses three steps: capturing data, integrating data from different sources, and applying these data [30]. Table 1 summarizes the main opportunities, challenges and innovations per step.

<table><caption>**Table 1.** Opportunities, challenges and innovations of new data sources for policy making.</caption><thead><tr><td>Steps of data use</td><td>Opportunities</td><td>Challenges</td><td>Innovations</td></tr></thead><tbody><tr><td>Capturing data</td><td>Availability of (real-time) sensor data [2,14,31], open data [5,31] and social media data [2,14,31].</td><td>Variety in data [1], data quality [4,5,18], reliability of data [4,5,18], and security of data [17,18].</td><td>Crowdsourcing [2,6,14]; nowcasting [32].</td></tr><tr><td>Integrating data</td><td>Cross-organizational collaboration [4,14,17]; linking new data sources to traditional statistics [4,31,33].</td><td>Interoperability [5]; lack of standardization, architectures, and portals [4,5,17]; legacy systems [4,5].</td><td>Sentiment analysis [31], location mapping [4,14,31], advanced social network analysis [14,31].</td></tr><tr><td>Application of data</td><td>Real-time monitoring of policy [31]; transparency and accountability [14].</td><td>Sense-making and interpretation [31].</td><td>Visualization techniques [19,31]; computer simulation [14,19].</td></tr></tbody></table>

Table 1 shows that public administrations increasingly see opportunities for the use of new data sources, mainly (real-time) sensor data [2,14]. These data can be physical, such as roadside monitoring, but also virtual, such as social media data. A study from 2015 finds that governments mainly make use of two types of data for data-driven policy making: “public datasets (administrative (open) data and statistics about populations, economic indicators, education, etc.) that typically contain descriptive statistics, which are now used on a larger scale, used more intensively, and linked [...] social media, sensors and mobile phones that are [...] analyzed with novel methods such as sentiment analysis, location mapping or advanced social network analysis” [31, p. 3]. Main issues are whether the data are of sufficient quality [4,5,18], and whether they are reliable and secure [4,5,17,18]. Otherwise, they may undermine the policy making process [4]. Innovations in capturing data are crowdsourcing [6], and nowcasting, which is the capturing of search engine data [32].

Regarding integration of data, to make successful use of big and open data in organizational processes, cross-boundary information integration (in between government agencies and between not-for-profit organizations and private firms and the public sector) is necessary [14,17]. The integration of data is becoming more important: linking these data sources with data sources that are traditionally used for policy making such as statistics, surveys and organizational databases is becoming the norm [31,33]. However, many challenges exist: interoperability of data and lack of standardization, architectures and portals [4,5,17]. Another issue are legacy systems that may negatively influence this linking [4,5]. Poel et al. [31] conclude that currently privately held data is of less relevance, as they are still hardly shared. Opportunities for data integration include sentiment analysis, location mapping, and social network analysis [4,14,31].
The third step in the use of new data sources is application and sense-making. While social media analysis and network analysis can be seen as forms of data integration that can be used to support the policy making, we consider the use of visualization tools and computer simulations to be applications of data to the actual process of policy making [19]. However, “[a]mong the initiatives examined, there is little use of advanced analytics or visualization techniques” [31, p. 4]. Another opportunity is to realize greater accountability [14]. Likely, the most innovative use of new datasets take place in the hidden spheres of fighting crime and terrorism [31].

## 3.2 Co-creation of policy

Another essential element of smartness in government is co-creation of policy, as ICT not only allows for collaborating with other organizations (public or private), but also with citizens [1,2,6,14]. Co-creation is the exchange of ideas and information between relevant actors, such as governments, businesses, civil society and citizens that lead to the development of policies [1,6]. Involvement of citizens in policy making is especially important since public data and statistics are increasingly met by citizens’ distrust [11]. This can take on different forms, depending on the level of involvement [2]: it may range from merely informing public administrations, for example by tapping discussion fora, opinion polls and using social media [2,6,19], to participating in decision making and in policy implementation. Table 2, which is based on Janssen & Helbig [2], summarizes the main innovations and challenges to co-creation of policies.

<table><caption>**Table 2.** Opportunities, challenges and innovations in co-creation of policies.</caption><thead><tr><td>Levels of involvement</td><td>Opportunities</td><td>Challenges</td><td>Innovations</td></tr></thead><tbody><tr><td>Informing and signaling</td><td>Citizens identify problems and set the agenda [2].</td><td>Social inclusion and overcoming exclusion [6,19,31]; lack of stability of social media [6].</td><td>Crowdsourcing [2,6]; online petitions [2]; participatory sensing [19].</td></tr><tr><td>Decision making</td><td>Citizens being involved in selecting options [2,6].</td><td>Citizens’ skills and motivation [2,16]; skills and culture of the government agency [6].</td><td>Computer simulation and serious games [2,19]; cross-platform social media analysis [6].</td></tr><tr><td>Implementation</td><td>Co-creation between governments, citizen and businesses [2]; policy evaluation [19]; transparency and accountability [6].</td><td>Privacy and security [2,6]; accuracy [6].</td><td>Camera surveillance, smart phone data, use of sensors [2]; age-ile implementation [19].</td></tr></tbody></table>

The most basic form of citizen involvement is informing and signaling, meaning that citizens’ information is used for identifying problems and setting the agenda [2]. Main challenges for this level are to make sure that different groups of citizens are represented, without excluding relevant groups [6]. Examples of this happening can be found
in literature on using social media data during disasters and disease outbreak. While nowcasting using search engine data for predicting flu outbreaks can be an accurate predictive methodology; for predicting Ebola, this method proved to be much less accurate since in the areas where the main outbreak was, internet access is still scarce [31]. Furthermore, the stability of social media is a challenge for its use in signaling problems [6]. Innovations in using citizens' ideas include crowdsourcing [2,6], online petitions [2], and participatory sensing [19].

The inclusion of citizens’ opinions in decision making refers to a higher level of involvement. This means that citizens are involved in the evaluation of policy options [2,6]. The most elaborate form of this is the organization of a referendum, but using social media or other online tools, this could be done more efficiently and effectively [1,2]. Important challenges are to ensure that both citizens’ and skills and motivation [2,16] and that civil servants’ skills and culture [6] are sufficient. Innovations in involving citizens in the choice for different policy options and decision making are computer simulations and serious games [2,19], and cross-platform social media analysis [6].

The third level of involvement is implementation of policies, which can be seen as the most immersive level of co-creation. Opportunities for co-creation include collaboration between public administrations, private companies and citizens in policy implementation [2], policy evaluation [19], and transparency and accountability [6]. Challenges include privacy and security [2,6] and accuracy [6]. Innovations in this level of involvement include camera surveillance, the use of smart phone data and sensors [2], and allowing for agile implementation, delivering faster and better innovations because of regular and short-cycle interactions [19].

# 4 The Policy Lab Approach

In the previous section we identified opportunities, challenges, and innovations based on literature of new technologies and co-creation in policy making. This section aims to present a coherent Policy Lab approach to data-driven policy making based on the innovations in these fields. Since the framework is to be used for policy making, we mapped these innovations to phases of the policy cycle [3]. Inspired by Janssen & Helbig [2], we distinguish three phases: *predictive and problem definition, design and experimentation*, and *evaluation and implementation*. Table 3 elaborates innovations and impact per phase of the Policy Lab approach, and identifies challenges.

The first phase of policy making – predictive and problem definition, (real-time) sensor data is used, comprising physical sensor data such as roadside traffic data, and virtual data such as social media data. Furthermore, innovative approaches such as crowdsourcing and nowcasting are also used to predict and identify problems. This leads to the availability of (real-time) information that allows more precise predictions than those that are merely expert based. However, experts are still important to provide context information to the trends spotted by the data. Main challenges are the availability, quality, reliability and security of the data as well as representativeness of the data
that should include viewpoints of different groups of citizens without excluding relevant groups. In a study on the use of data for policy making from 2015, over half of the cases identified were used for this first phase of policy making [31].

<table><caption>**Table 3.** Innovations, impact and challenges of data-driven policy making.</caption><thead><tr><td>Policy Cycle Phase</td><td>Innovations</td><td>Impact</td><td>Challenges</td></tr></thead><tbody><tr><td>Predictive & problem definition</td><td>Use of (real-time) sensor data from citizens (e.g. social media data, crowdsourcing), business and government for problem definition and prediction.</td><td>Problem definition based on (real-time) data from different actors, rather than merely expert based.</td><td>Capturing different data sources and ensuring data quality, reliability and security as well as representativeness of the data.</td></tr><tr><td>Design & experimentation</td><td>Using advanced analyses, such as sentiment analysis, location mapping, social network analysis, visualization, computer simulation and serious games for decision making.</td><td>Cross-organizational collaboration and involvement of citizens require more advanced analyses to be able to select policy options.</td><td>Creating an infrastructure ensuring interoperability and allowing for integration of data, in the form of standards, architectures, and portals.</td></tr><tr><td>Evaluation & implementation</td><td>Collaborative data-driven policy implementation by governments, citizens and businesses, allowing for agility of processes.</td><td>Public value creation, improved transparency and accountability, but it may also lead to more surveillance.</td><td>Accuracy of data and data models, ensuring privacy and security. Citizens' skills and motivation and skills and culture of the government agency need to be sufficient.</td></tr></tbody></table>

The second phase of policy making – design and experimentation, should ensure collaboration between government, private organizations, and citizens in the decision making process and choice for policy options. This requires the use of more advanced analytical approaches such as sentiment analysis, location mapping, social network analysis, visualization techniques, computer simulation and serious games to allow for the involvement of other stakeholders in the decision making. A major challenge for the integration of different data sources, the performance of more advanced analyses, and ensuring involvement of citizens is setting up an infrastructure that allows for interoperability and integration of data [17]. Standards, architectures and portals can be instruments for this. Traditionally, governments more often involve citizens after this phase, in the implementation, rather than in the process of decision making. This is reflected in the lower number of best practices in this phase [31].
Evaluation and implementation – the third phase of policy making, allows for joint policy implementation and co-creation of services by government, businesses and citizens. An advantage of the use of new data sets and technologies is the use of an agile approach [15] that allows for short cycles of decision making and implementation. The involvement of relevant stakeholders in the implementation and ongoing monitoring of policy creates public value [20-22]. More insight and collaboration may result in greater transparency and accountability, but also to more surveillance. Accuracy of data and data models and ensuring privacy and security are major challenges. Furthermore, co-creation of policy requires specific skills and motivation of citizens as well as specific skills and culture of the government agency [2,16]. While in traditional e-participation, citizens are involved in policy implementation, actual co-creation involving citizens in the production of services is less often found in practice [7].

These innovations are challenging and in practice most governments do use new
technologies and data sets for policy making, but they use this to enrich traditional sta-
tistical data rather than achieving co-creation [31]. Therefore, besides allowing for ex-
perimentation with policy making, new methodologies need to be developed that are
able to make use of these new data sources and technologies. Using a design science
approach [34], we developed the Policy Lab approach that can be used to guide inno-
vations in data-driven policy making, allowing for experimentation with new policies
and developing new data-driven methodologies at the same time. To validate this ap-
proach we held five internal workshops with experts that took place over the course of
2016. Furthermore, throughout this process we consulted academic and governmental
stakeholders: four representatives of three academic institutions and six representatives
of the national and local levels of government were involved. The Policy Lab approach
is graphically presented in Figure 1.

<figure><img src="image_2.png" /><figcaption>**Figure 1.** The Policy Lab approach for data-driven policy making.</figcaption></figure>
The conceptualization of the Policy Lab approach presented in Figure 1 consist of two circles. The inner circle is represents the policy making process, consisting of several phases, such as agenda setting, policy formulation, decision-making, implementation and evaluation. The outer circle of the Policy Lab approach focuses on the development of data-driven methodologies and co-creation. This approach allows the two circles to mutually influence each other: policy experiments can be used to develop and test new methodologies, that, in turn can be used for developing and evaluation policies.

# 5 Discussion

Based on the literature review, we found that most applications of new data sources, such as (real-time) sensor data are to link them to traditional statistics and few innovative methodologies are used for policy making [31]. However, “utilizing such [social] channels for policy making purposes does not constitute an established approach yet” [19]. This means that first of all instruments and methodologies for the use of these new data sets in traditional statistical and econometric methodologies should be developed [35]. Furthermore, in order for governments to become used to these methodology, they could very well use the use the ‘design-thinking approach’ of a Policy Lab that allows for experimentation. This means that the Policy Lab approach, effectively, has three pillars: using new technologies and data sources for policy making, enabling co-creation and allowing for experimentation.

The use of new datasets in traditional statistical or econometric studies is widely regarded to have a large potential for policy making. Traditional data sources are often text based or have a strong qualitative character rather than a numerical or machine generated form. Newer data sources are often human generated (social media) data, or machine generated sensor data. This can also be seen as the main distinction between data-driven and evidence-based policy. Using these newer data sources means that not only new methodologies need to deal with the size of these new data sets, but also with the variety of data, that may range from traditional statistics, to (real-time) sensor data to human generated text based social media data to images, video streams or geo-data. Statisticians and econometrists aiming to deal with these new (big) data sets, need to learn ways to incorporate them into their traditional methodologies [35].

Fundamentally, there are no contradictions between big data and traditional econometric approaches, but the two have developed independently. For example, the use of big data sets enhances statistics in prediction methods (out-of-sample), which is often not possible in traditional econometrics because data sets are not large enough [35]. Furthermore, when using big data sets it makes more sense to focus on model uncertainty than on sampling uncertainty, which is often examined in traditional econometrics. Finally, machine learning techniques such as decision tree learning may give a better picture than logistic regression [35]. Traditional statistics, in turn, provide useful methods to help variable selection in big data models such as stepwise regression penalized regressions and Bayesian techniques (including time series analysis) [35].

However, while these new methodologies could benefit from the incorporation of big data and linking them with traditional methodologies, traditional policy models are
far from obsolete. Big data mainly concerns the discovery of correlations, while policy
models present causations that have been developed based on practical experience [36].
Causation hypotheses can ultimately be confirmed using controlled or natural experi-
ments, and, thus, cannot be replaced with big data analyses alone. The degree to which
the outcomes of such combinations of big data and statistical models can be explained,
thus, represents a major issue. Therefore, the involvement of citizens and experimenta-
tion become paramount. This is even more the case in this ‘post-factual’ era, in which
citizens are critical of official statistics and data [11]. A Policy Lab setting can be used
for controlled experimentation allowing people to ‘buy into’ data, statistical methods
and data-driven policies.

Similar to the challenges that Living Labs face, the Policy Lab approach, as a spe-
cific instance of a Living Lab, presents the risk of becoming primarily focused the im-
plementation of an open innovation approach, rather than with achieving specific re-
sults [26]. While involvement of new data sources and citizens in the policy making
process are important objectives, the primary aim should be to improve policy making.
If this is not achieved, this may result in a limited application of data-driven policies
outside of the Policy Lab environment. This also means, as is the case for Living Labs,
that scaling and sustainability are major challenges [26].

Further research should thus focus on the development of these new methodologies
that allow for combination of new data sources with traditional statistical data and the
combination of big data methodologies with econometrics. Furthermore, experiments
with policy development that address wicked problems should be carried out both to
involve citizens and increase legitimacy of these policies and to capture the benefits of
these new approaches for policy makers. This means that the Policy Lab approach
should be validated and expanded based on these experiments. Finally, the issue of
scalability and sustainability should be further explored to capture the benefits of data-
driven policy making outside of the Policy Lab setting.

6 Conclusion

New data sources and ICTs have great potential for improving policy making. How-
ever, data-driven policy initiatives are scarce and the existing initiatives are, often,
cases linking (real-time) sensor data to traditional statistical analyses. Therefore, using
a design science approach, this paper develops a Policy Lab approach. Based on litera-
ture, we identified innovations in the use of new data sources and in co-creation of
policies. The involvement of citizens will likely become more important for the legiti-
macy of statistics and data and policies. Subsequently, we mapped these innovations to
the different phases of the policy cycle. Based on this overview, the Policy Lab ap-
proach draws on three aspects: using new data sources, co-creation and experimentation
with policy making focusing on real-life wicked problems. The experiments can be
used to develop data-driven policies as well as to develop new data-driven methodolo-
gies. Further research should focus on the development of methodologies for incorpo-
rating big data analyses into traditional statistical analysis and on experimentation with
policy issues, thereby validating the Policy Lab approach.
References

1. Millard, J.: Open Governance Systems: Doing More with More. Government Information Quarterly (2015).

2. Janssen, M., Helbig, N.: Innovating and changing the policy-cycle: Policy-makers be prepared! Government Information Quarterly (2015).

3. Lasswell, H.: The policy orientation. In: Lerner, D., Lasswell, H.: The Policy Sciences: recent developments in scope and method, pp. 3–15. University Press, Stanford (1951).

4. Bertot, J. C., Choi, H.: Big data and e-government: issues, policies, and recommendations. In: Proceedings of the 14th Annual International Conference on Digital Government Research, pp. 1–10. dg.o. 2013, June 17-20, Quebec City, Canada (2013).

5. Janssen, M., Charalabidis, Y., Zuiderwijk, A.: Benefits, Adoption Barriers and Myths of Open Data and Open Government. Information Systems Management 29(4), 258–268 (2012).

6. Ferro, E., Loukis, E.N., Charalabidis, Y., Osella, M.: Policy making 2.0: From theory to practice. Government Information Quarterly 30(4), 359–368 (2013).

7. Linders, D.: From e-government to we-government: Defining a typology for citizen production in the age of social media. Government Information Quarterly 29(4), 446–454 (2012).

8. Solesbury, W.: Evidence Based Policy: Where it Came and Where it's Going. ESRC UK Centre for Evidence Based Policy and Practice, Queen Mary, University of London (2001), https://www.kcl.ac.uk/sspp/departments/politicaleconomy/research/cep/pubs/papers/assets/wp1.pdf.

9. Head, B.W: Three Lenses of Evidence-Based Policy. The Australian Journal of Public Administration 67(1), 1–11 (2008).

10. Bijlsma, R.M., Bots, P.W.G., Wolters, H.A., Hoekstra, A.Y.: An Empirical Analysis of Stakeholders’ Influence on Policy Development. Ecology and Society 16(1), 51–66 (2011).

11. Davies, W.: How statistics lost their power – and why we should fear what comes next. The Guardian, 19 January (2017), https://www.theguardian.com/politics/2017/jan/19/crisis-of-statistics-big-data-democracy.

12. Williamson, B.: Governing methods: policy innovation labs, design and data science in the digital governance of education. Journal of Educational Administration and History 47(3), 251–271 (2015).

13. Fuller, M., Lochard, A.: Public policy labs in European Union Member States. EUR 28044 EN June (2016), http://publications.jrc.ec.europa.eu/repository/bitstream/JRC102665/final%20report%20w%20identifiers.pdf.

14. Gil-Garcia, J.R., Zhang, J., Puron-Cid G.: Conceptualizing smartness in government: An integrative and multi-dimensional view. Government Information Quarterly 33(3), 524–534 (2016).

15. Janssen, M., Estevez, E.: Lean government and platform-based governance – Doing more with less. Government Information Quarterly 30(S1), S1–8 (2013).

16. Meijer, A.: E-governance innovation: barriers and strategies. Government Information Quarterly 32(2), 198–206 (2015).

17. Gil-Garcia, J.R., Sayogo, D.S.: Government inter-organizational information sharing initiatives: Understanding the main determinants of success. Government Information Quarterly 33(3), 572–582 (2016).

18. Hui, G., Hayllar, M.R.: Creating Public Value in E-Government: A Public-Private-Citizen Collaboration Framework in Web 2.0. The Australian Journal of Public Administration 69(S1), S120–S131 (2010).
19. Koussouris, S., Lampathaki, F., Kokkinakos, P., Askounis, D., Misuraca, G.: Accelerating Policy Making 2.0: Innovation directions and research perspectives as distilled from four standout cases. Government Information Quarterly 32(2), 142–153 (2015).

20. Moore, M.H.: Creating Public Value Strategic Management in Government. Harvard University Press, Cambridge MA (1995).

21. Stoker, G.: Public Value Management A New Narrative for Governance? The American Review of Public Administration 36(1), 41–57 (2006).

22. Bannister, F., Connolly, R.: ICT, public values and transformative government: A framework and programme for research. Government Information Quarterly 31(1), 119–128 (2014).

23. Dawes, S.S.: The Evolution and Continuing Challenges of E-Governance. Public Administration Review 68(4), S86–102 (2008).

24. McDermott, P.: Building open government. Government Information Quarterly 27(4), 401–413 (2010).

25. Chadwick, A.: Bringing E-Democracy Back In Why it Matters for Future Research on E-Governance. Social Science Computer Review 21(4), 443–455 (2003).

26. Gasco, M.: Living Labs: Implementing open innovation in the public sector. Government Information Quarterly 34(1), 90–98 (2017).

27. Manyika, J., Chui, M., Brown, B., Bughin, J., Dobbs, R., Roxburgh, C., Hung Byers, A.: Big Data: the next frontier for innovation, competition, and productivity. McKinsey Global Institute (2011), http://www.mckinsey.com/insights/business_technology/big_data_the_next_frontier_for_innovation.

28. Cukier, K., Mayer-Schönberger, V.: Big Data: a Revolution That Will Transform how we Live, Work, and Think. Houghton Mifflin Harcourt Publishing Company, Boston (2013).

29. Chen, L., Chiang, R.H.L., Storey, V.C.: Business Intelligence and Analytics: From Big Data to Big Impact. MIS Quarterly 36(4), 1165–1188 (2012).

30. Harrison, C., Eckman, B., Hamilton, R., Hartswick, P. Kalagnanam, J. Paraszczak, J., Williams, P.: Foundations for smarter cities. IBM Journal of Research and Development 54(4), 1–16 (2010).

31. Poel, M., Schroeder, R., Treperman, J., Rubinstein, M., Meyer, E., Mahieu, B., Scholten, C., Svetachova, M.: Data for Policy: A study of big data and other innovative data-driven approaches for evidence-informed policymaking. Report about the State-of-the-Art (2015), http://media.wix.com/ugd/c04ef4_20afdcc09aa14df38fb646a33e624b75.pdf.

32. Taylor, L., Schroeder, R. & Meyer, E.: Emerging practices and perspectives on Big Data analysis in economics: Bigger and better or more of the same? Big Data & Society 1(2), 1–10 (2014).

33. Brakel, J. van den, Söhler, E., Daas, P., Buelens, B.: Social media as a data source for official statistics; the Dutch Consumer Confidence Index. Discussion Paper 2016-01, Statistics Netherlands (2016), https://www.cbs.nl/nlnl/achtergrond/2016/07/social-media-as-a-data-source-for-official-statistics-thedutch-consumer-confidence-index.

34. Hevner, A.R., March, T.S., Park, J., Ram, S.: Design science in information systems research. MIS Quarterly 28(1), 75–105 (2004).

35. Varian, H.R.: Big Data: New Tricks for Econometrics. Journal of Economic Perspectives 28(2), 3–28 (2014).

36. Cowls, J., Schroeder, R.: Causation, Correlation, and Big Data in Social Science. Policy & Internet 7(4), 447–472 (2015).