# A Digital Game to Learn About Open Data

Davide Di Staso, Fernando Kleiman, Joep Crompvoets, Marijn Janssen

## ► To cite this version:

Davide Di Staso, Fernando Kleiman, Joep Crompvoets, Marijn Janssen. A Digital Game to Learn About Open Data. 20th International Conference on Electronic Government (EGOV), Sep 2021, Granada, Spain. pp.153-164, 10.1007/978-3-030-84789-0_11 hal-04175090

HAL Id: hal-04175090

https://inria.hal.science/hal-04175090

Submitted on 1 Aug 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# A Digital Game to Learn about Open Data

Davide Di Staso¹[0000-0002-4045-1944], Fernando Kleiman¹[0000-0003-3336-3484],
Joep Crompvoets²[0000-0003-1077-597X], and Marijn
Janssen¹[0000-0001-6211-8790]

¹ Faculty of Technology, Policy and Management, Delft University of Technology,
Delft, The Netherlands

{d.distaso, f.kleiman, m.f.w.h.a.janssen}@tudelft.nl

² Public Governance Institute, KU Leuven, Leuven, Belgium
joep.crompvoets@kuleuven.be

**Abstract.** The implementation of open data policies requires the efforts of many public employees across different levels of government, who may be unaware of the benefits and risks of open data. Serious games have demonstrated potential for training in a professional environment. For this research, a collaborative digital serious game about open data was developed. A sample of 24 civil servants played the game. Pre-test and post-test surveys were used to evaluate the effects of the game on participants' perception of open data. Likert score changes between pre-test and post-test indicated that the game had a positive effect on the willingness to share public sector data. By simulating the setting of a public office and by having players make decisions about whether to open certain datasets, the game facilitated learning about the benefits and disadvantages of opening data.

**Keywords:** Serious games · Gaming · Open data · Open data policy · Covid-19.

## 1 Introduction

Open government data is data produced by governmental organisations and released to the public without any limitations on its use and redistribution [12,13]. Open data can be released either as dynamic APIs or as static datasets [21], which can be accessed via web-portals such as data.europa.eu. A number of stakeholders, such as transparency activists, entrepreneurs and government employees need open data to increase accountability, deliver better services, or support new business models [22]. Open data is one of the pillars of open government, which promotes the release of public sector data in order to enable public oversight into the government [12]. The release of open data ultimately rests on the decisions made by civil servants across different levels of government, from municipalities, to central government institutions. Public employees involved in the release of open data need to be willing to publish datasets, while at the same time being aware of the risks and benefits of doing so. Several behavioral factors can affect a civil servants' willingness to release data, such as a limited understanding of
its benefits, the perception that it might be useless, and a general risk aversion
and overestimation of security risks [16]. Serious games are a possible solution, as
they offer a powerful tool to implement policies and create behavioral change [7].
They also offer a safe environment for players to experience decision-making and
policies in a practical way [7]. In the context of open data, serious games can
facilitate learning about how data is generated [24], show ways in which datasets
can be combined to provide services [4,11] or have players decide whether or not
to share certain datasets and provide feedback on their decision [14]. Winning
Data by Kleiman et al. [14] was successful in changing civil servants' attitudes
towards open data, but requires the presence of 4 people in the same room,
which is temporarily prohibitive during the COVID-19 pandemic. Beyond the
pandemic, this limitation will remain relevant for governmental teams working
remotely or following a hybrid model. Therefore, there is a need for an online
game that lets public employees learn about the opening of data. For this re-
search, a collaborative digital serious game was developed, called Data Belt. By
digitalising the game and observing the online gameplay it is possible to produce
new insights and knowledge about open data gaming interventions. Data Belt is
based on the in-person role-playing game Winning Data [14]. Like its in-person
counterpart, Data Belt allows participants to make decisions on whether or not
to open certain datasets and encourages them to evaluate the benefits and risks
of opening data. However, in Data Belt, participants can play via video con-
ference, instead of being in the same room. No other game designed for online
multiplayer interaction on the topic of open data was found in the literature
review. The game's multiplayer interactions are designed to foster knowledge
sharing and create new insights about open data among players. A sample of
24 civil servants played remotely and the effects of the game were observed by
measuring changes in responses to pre-test and post-test questionnaires. This
research presents a preliminary analysis of the responses to the most relevant
survey items, together with statements made by players during the game session
debrief.

In the next section, barriers to open data are presented, along with examples
of the use of serious games in government and serious games for open data. Next,
the methodology for the game and experiment design are presented. Differences
between pre-test and post-test survey responses are shown and discussed, before
presenting the conclusions and limitations of this research.

## 2 Theoretical Background and Literature Review

By opening public sector data, governments can become open systems and en-
gage in feedback loops with citizens, thereby becoming more responsive to their
needs [12]. Opening data can also facilitate the creation of citizen-made digi-
tal tools for public use, such as portals to monitor lobbying activities, or even
email notifications to avoid parking violations [13]. Still, a number of barriers
and myths still exist around open public sector data [12]. Barriers to the opening
of data affect the identification of suitable datasets, the decision to release, the
publication process, usage of the dataset and the evaluation of its impact [6]. For
example, when trying to decide whether or not to release data, public employees
might not see the value of sharing a certain dataset or might overestimate the
privacy risks [12,16]. In the same context, civil servants might fear "unexpected
and unwanted responsibilities" [6] due to unclear liabilities for the consequences
of opening a dataset [6]. It should also be noted that open data decision-making
does not happen in a vacuum; civil servants might be confronted with risk-averse
stakeholders (politicians, administration officers) or they might lack the neces-
sary tools and resources to open data [17]. On the other hand, the benefits of
open data might also be exaggerated, as in the myth that releasing data will
create immediate and automatic benefits or that it is good practice to release
any sort of data unrestrictedly [12]. The barriers listed are ultimately common
issues arising from policy implementation, and the consequent need for a change
in behaviour, which serious games can facilitate [7].

## 2.1 Serious games in government

Games are uniquely positioned to facilitate learning in governmental organisa-
tions. When games are used for policy implementation, they can contribute to
sense giving and facilitate the understanding of policy documents and guidelines
that otherwise might not have clear operational value or actionable elements [7].
By playing a game, participants can experience exactly what the policy is about
on a practical level and understand it more deeply [10]. Moreover, players are
presented with a safe space, which they can use to experiment new ideas and
behaviors, some of which may be unexpected to the game designer or facilita-
tor [7]. Learning outcomes are deeply connected to the use of game mechan-
ics [2]. Game mechanics encompass everything that affects the operation of the
game [1], including how players can behave within the game and the tools, items
and attributes that they can use, such as bonus cards, incentives and penal-
ties [1]. Game mechanics are also media-independent, that is, one mechanic can
be brought over from a physical game to virtual one [1], as was done for this
study. The use of serious games in government is not new [19]. For example,
Bharosa et al. [3] developed an in-person role-playing game for civil servants
aimed at synthesising principles for service delivery. Players interacted in the
role of customer, front-office and back-office employees according to purpose-
fully flawed scripts. Participants could relate to the role they were playing and,
after the game session, identified insightful principles and rationales which can
improve professional interactions. Open data policies can benefit from a similar
bottom-up approach to explain policy contents to civil servants from different
levels of government.

## 2.2 Games about open data

There are several serious games dealing specifically with the topic of open data.
Datopolis [4] is a board game in which players need to negotiate with each
other in order to make data open and in turn build services. The game can be
played in its physical version, as well as in an online environment for virtual
tabletop games. While Datopolis can technically be played online, it was not
initially designed for this mode of interaction and the digital version is simply a
simulation of a tabletop environment. The Open Data Card Game [11] is another
game about open data in which players are divided into groups of three, asked
to freely combine different cards representing datasets, and think about ideas for
the services that can be built using the data at hand. In Datascape [24], players
are presented with a map showing different features and data sources. They are
then asked to identify the data sources that can answer the questions they are
presented with. Winning Data by Kleiman et al. [14] is an in-person role-playing
game in which one player acts as the citizen and the other three as government
employees. By processing the citizen's requests, government employees generate
datasets and decide whether or not to open them by evaluating the sensitivity
of their contents.

The examples of open data games listed until now all require the partici-
pation of multiple players. Still, none of these group games were designed for
digital gameplay, and, with the COVID-19 pandemic, in-person participation
has temporarily become prohibitive. A solution is also needed for remote or hy-
brid teams which have limited opportunities for in-person training. Multiplayer
serious games are hard to find, in part due to the additional challenges posed by
concurrent gaming and player interaction [23]. A multiplayer game is useful to
exploit the effectiveness of collaborative learning [9] and its ability to leverage
pooled knowledge and observational learning [20]. In the context of governmental
organisations, a multiplayer game can also better reflect the reality of a public
office. At the time of writing, no collaborative digital serious game on open data
could be found in the literature. This research attempts to address this gap by
creating a new game about open data for civil servants which makes extensive use
of concurrent gaming and player interaction. The methodology used to design
the game and to test its effectiveness are presented in the following section.

3 Methodology

## 3.1 Game design

By looking at the already existing Winning Data [14], which has proved to be
effective at changing civil servants' attitudes [15], a list of essential features that
had to be brought into the new, digital version of the game was drafted: (1) play-
ers must be situated in a fictional municipal office where they (2) rotate between
different roles, each of them having a specific skill, (3) at the office, players need
to process the citizen's demands, in turn generating datasets that can be opened,
partially shared, or kept closed, (4) players must receive a reward by the game if
their decision to open or close datasets matches the recommendations found in
the literature. The four essential features found in Winning Data also converge
with the seven lessons learned from board games found in Zagal et al. [25]. In
order to bring these dynamics into the digital game, the user interface and flow
of the game were sketched and a basic prototype was developed. The prototype
was then tested with university students and incremental changes were made
after each game session, with more than a dozen different iterations.

## 3.2 Game mechanics

<figure><img src="image_3.png" /><figcaption>**Fig. 1.** Screenshots representing what each player sees during gameplay. From left to the right, the civil servant, citizen, colleague, and boss. Reprinted from Di Staso, D., Kleiman, F., Crompvoets, J., Janssen, M.: Changing Civil Servants’ Awareness about Open Data Using a Collaborative Digital Game. In: DG.O2021: The 22nd Annual International Conference on Digital Government Research [In Press]. ACM, Omaha, NE, USA. Copyright 2021 Di Staso et al.</figcaption></figure>

As already described in [8], in the version played by civil servants for this research, Data Belt requires the participation of exactly four players, connected via video conference, who are assigned to the roles of civil servant, colleague, boss and citizen. Each of these roles have specific abilities which are needed by the team to complete their tasks, therefore forcing players to collaborate. The game is divided into four rounds, at the end of which players rotate into a different role; participants never play in the same role more than once. During each round, the civil servant and colleague need to process the five files brought to them by the citizen (data processing). Each correctly processed file generates a dataset, which can be released to the public to different degrees or kept closed. For each dataset, the citizen, civil servant and colleague are shown a description and asked to suggest the boss what to do with the dataset (data labelling). The suggestion is synchronised across the three devices, meaning that any of the three players can see the current pick and change it, if they want to. For example, the dataset on budget, adapted from [14], reads:

> By assessing public lawsuits, citizens can monitor problems related to public life. Tax evasion, misuse of public assets could be shared with citizens? Are there risks for individuals or security?
Through a synchronised checkbox, the citizen, civil servant, and colleague can suggest as a group to either open the dataset as is, anonymise it and then share it or keep it closed. The datasets and the labelling suggestions will be passed to the boss, who will make the final decision. The boss does not receive any special guidance or instructions; this role was created in order to place the burden of the final decision-making process on each player at least once during the game session (as noted earlier, players rotate among roles). Lastly, all players are brought to a summary screen, where they can see the datasets they processed, how the boss decided to label them, and the correct labelling according to the literature review. The dataset descriptions and labelling were adapted from Kleiman et al. [14]. In case a dataset has been labelled incorrectly, a penalty is given by the game, and the facilitator gives feedback on why the literature indicates a different labelling.

While porting the game to a digital environment, several adaptations had to be made. For example, in the data processing phase of Winning Data, the player with the role of civil servant walks to the other players' desks and passes them the files that need processing. In Data Belt, when players want to pass a file, the sender reads the file's unique code out loud and the addressee transcribes it into a keypad. In figure 1, the civil servant (second screen from the left) retrieved file 1451 (Budget) from the citizen and is processing it by putting stamps. The keypad used to retrieve files is visible in the colleague's screen (third screen from the left), who has used it to retrieve two files that will need to be opened and processed as well.

## 3.3 Pre-experiments

This research focuses on evaluating the effects of the game on participants' perception of open data in a pre-experimental setting. Having already established that a game has the potential to impact players' behaviour, there is a need to evaluate its specific effects [18]. Because the game was tested with a non-random sample participating remotely and because there was no control group, the setting in which the game was tested is a pre-experiment [5]. Data Belt aims at changing perceptions of players, which demands experimentation. A pre-experimental set-up can be appropriate to conduct a preliminary investigation of the effects of the game on its players. It converges with de Caluwé et al. [7], who identified that gaming or simulation solely developed for research purposes can leverage the benefits of experimental research, such as the possibility of applying statistical methods in a study that is, in essence, a qualitative project [7]. Additionally, games can offer insights into decision-making processes that are usually kept private and obscure and could not otherwise be studied [7].

## 3.4 Game sessions and surveys

Questionnaires from the existing literature [14] were adapted to an online environment so that they could be used for this research. Participants were arranged in groups of four people and sent the pre-test survey, which they filled some time
before the game session. At the beginning of the session, players connected via video conference and were invited to briefly introduce themselves. The facilitator then gave a 10 to 15 minutes presentation explaining the game's rules and interface. Participants then played together using smartphones, with the actual gameplay lasting 30-40 minutes. At the end of the gameplay, participants filled the post-test questionnaire, were debriefed and asked to provide their impressions about the experience. Most game sessions were organised and scheduled with governmental organisations in the Netherlands: Digicampus (12 participants), Provincie Zuid-Holland (4), ICTU foundation (4). One game session was organised as part of Brazil's National School of Public Service Innovation Week (4 participants).

The dataset used for this research only includes participants who played the finalised version of the game and who successfully completed the session without encountering game-breaking bugs or complete disconnections. This research presents responses to the items of the questionnaires focused on the learning aspect, which could be answered using a Likert scale going from one (strongly disagree) to seven (strongly agree). The idea is that the loop of processing data, deciding whether or not to open it and then receiving feedback for that decision leads to civil servants being more willing to share public sector data, while at the same time knowing more about the possible benefits of doing so (Q1, Q5) and feeling that the process is less threatening (Q6). When participants released a sensitive dataset which should have been kept private or anonymised before publishing, the game assigned a penalty and the facilitator explained the associated issues, which would often include privacy. Therefore, a change in awareness about privacy issues (Q4) was expected. Some of the items presented (Q2, Q3) ask about participants' knowledge on how to open datasets. The game represented this process in a very abstract way, and therefore we did not expect a significant effect on these items.

# 4 Results and discussion

The game was played with 24 civil servants, pre-test and post-test surveys were distributed some time before the session and immediately after the gameplay was completed. Data Belt was effective in increasing civil servants' willingness to release datasets and in facilitating learning about the general risks and benefits involved. However, findings suggest that there was no significant effect on participant's knowledge about how to open datasets. The game invited players to collaboratively decide whether or not to share some public sector data and gave feedback on their choice. Table 1 summarises the initial results of the pre-experiment. Respondent's age ranged from 25 to 53 years old, with an average of 41.6; 15 subjects were male and 9 female. A paired sample t-test was performed on each individual item, comparing pre-test and post-test responses. The results of this analysis are shown in table 1. The following paragraphs present the change observed between pre-test and post-test, discuss whether this change is consistent with our expectation and present a possible explanation behind it.
Quotes from the players during the debrief session are also presented to support the explanations.

<table><caption>**Table 1.** Most relevant survey items selected from the preliminary data analysis</caption><thead><tr><th rowspan="2">Description</th><th rowspan="2">n</th><th colspan="2">Pre-test</th><th colspan="2">Post-test</th><th rowspan="2">Δx̄</th><th rowspan="2">p</th></tr><tr><th>x̄</th><th>σ</th><th>x̄</th><th>σ</th></tr></thead><tbody><tr><td>Q1 Some public sector data can be shared</td><td>24</td><td>5.08</td><td>1.91</td><td>5.88</td><td>1.36</td><td>0.79</td><td>0.02</td></tr><tr><td>Q2 I know how to make public sector data available for others to access</td><td>24</td><td>4.12</td><td>1.75</td><td>4.21</td><td>1.72</td><td>0.08</td><td>0.79</td></tr><tr><td>Q3 People in my office know how to make public sector data available for others to access</td><td>24</td><td>4.79</td><td>1.86</td><td>4.83</td><td>1.27</td><td>0.04</td><td>0.91</td></tr><tr><td>Q4 Public sector data that results from my work cannot be shared for privacy issues</td><td>24</td><td>3.42</td><td>1.53</td><td>3.12</td><td>1.26</td><td>-0.29</td><td>0.37</td></tr><tr><td>Q5 Providing open public sector data has benefits which are difficult to explain</td><td>24</td><td>3.75</td><td>1.73</td><td>4.62</td><td>1.71</td><td>0.88</td><td>0.02</td></tr><tr><td>Q6 Providing public sector data is a threat</td><td>24</td><td>1.96</td><td>0.91</td><td>2.83</td><td>1.52</td><td>0.88</td><td>0.01</td></tr></tbody></table>

**Q1 - Some public sector data can be shared** The average score went from 5.08 to 5.88 in a statistically significant way (p = 0.02). A positive increase was expected for this item and it indicates that playing Data Belt had a positive impact on players' perceptions of data sharing. This effect can be explained by the fact that players, while labelling together each dataset, became more confident about releasing some public sector data. As stated by one of the players:

...it's spot on really...the questions...I can recognise immediately what it's about. The fact from the game...it's a bit overwhelming because of the information. There is no difference in real life, too many questions, too much information, hard, partial information, and I have to make a decision and plan some action and I'm not really confident. That's real life.

**Q2 - I know how to make public sector data available for others to access** The average Likert score went from 4.12 to 4.21 and the change was not statistically significant (p = 0.79). A significant change was not expected for
this item, as the game did not address the real-world procedures involved with
making data available to others.

**Q3 - People in my office know how to make public sector data available for others to access** For this item, the mean score went from 4.79 to 4.83 with no statistical significance (p = 0.91). For this item, just like for the previous one, a significant change was not expected.

**Q4 - Public sector data that results from my work cannot be shared for privacy issues** The mean Likert score went from 3.42 to 3.12 with no statistical significance (p = 0.37). The expectation was to see a statistically significant increase caused by players becoming more accustomed to managing privacy risks. Instead, it appears the in-game content about privacy did not affect participants' perceptions. While more research is needed to confirm this outcome, it is possible that the penalties given by the game for privacy violations were not evident enough.

**Q5 - Providing open public sector data has benefits which are difficult to explain** This item saw a positive increase from 3.75 to 4.62, which was statistically significant (p = 0.02). Open data is seen as a complex topic, and the game provided reinforcement on this aspect, therefore a positive change on this item was expected. During the debrief, one of the players said that the feedback given by the facilitator at the end of each round (when looking at the results) was helpful:

> ...the explanation you [the facilitator] gave was sometimes helpful, like why some data can be shared or cannot be shared, can be open or cannot be open. I think we can learn from that, because a lot of times people ask me to provide data, and a lot of times I cannot explain why it should be open or not.

The feedback provided by the facilitator is a fundamental aspect of the game. At the end of each round, the facilitator explains why a certain dataset might be too sensitive to be shared or why it does not actually pose a threat. The feedback highlighted hidden risks, such as data becoming personally identifiable in small datasets, even when anonymised, or hidden benefits, such as how a dataset is used by private entities or other governmental organisations.

**Q6 - Providing public sector data is a threat** The mean Likert score for this item went up from 1.96 to 2.83 in a statistically significant way (p = 0.01). This change seems to contradict the effects observed for Q1 and Q5. The assumption was that, by experiencing the process of opening data, players would perceive it as less of a threat, therefore an increase in this item's score was not expected.
Overall, changes in the responses between pre-test and post-test are coherent with the game's main focus, which is to collectively decide whether or not to open a certain dataset and then receive feedback on the decision. From the observations that player made during the debrief, it appears that game was generally well received.

# 5 Conclusion

The purpose of this research was to design a collaborative digital serious game about open data and measure its effects on civil servants. As there was no prior work on a game for open data designed for the online environment, we derived the game mechanics from another game and conducted game sessions with participants connected remotely. One of the requisites was for the new game to allow for remote participation via video conference, both due to the temporary restrictions imposed to contrast the COVID-19 pandemic and due to the need for a game than can be used by remote or hybrid government teams. Based on the in-person game Winning Data [14], the requirements and characteristics for the new, digital game were listed, and a prototype was developed. The new game, called Data Belt, was played by civil servants, who filled pre-test and post-test surveys to measure learning about open data. Data Belt used game mechanics such as collaboration, time pressure, and feedback loops to facilitate learning. Findings indicate that these game mechanics were suitable to enable learning about open data. Having multiple players collaborate to satisfy citizen's requests more closely reflected the environment of a real public office. Moreover, forcing players to make one common choice about data release fosters discussion and reflection on the contents of each dataset and on whether or not it can be shared. However, it appears that the game did not have significant effects on civil servants' concerns about privacy issues. This result is possibly explained by the penalty for sharing too much being too weak or not evident enough. After playing the game, participants were more inclined to believe that some public sector data can be shared and seemed to be more aware about the hidden benefits of opening data. At the same time, it seems that, after playing the game, participants perceived the provision of public sector data as more of a threat, a finding which seems contradictory and which requires further research. The game was designed to be played only once, therefore its scope is limited by the short gameplay time. For example, the game does not address the specific procedures and tools needed to open data, and, accordingly, there was no significant change in participant's self-reported knowledge about how to open data. Since Data Belt was designed to be played by civil servants working for different organisations in different countries, it did not detail the technical aspects involved in the opening of data, which might change from one department to another. The conclusions presented in this study are limited by the small sample size, lack of a control group and lack of data to establish if the effects of the game are still present after an extended period of time. Future research should further
investigate the effects of the game, including the contradiction observed for some
of the survey items. More research is also needed to compare the outcomes of
the in-person game with Data Belt. Finally, improvements to the game could
be explored, such as removing the requirement for a facilitator by introducing a
tutorial mode, introducing new game modes to allow smaller or larger groups to
play together, and including new game mechanics describing some of the most
common procedures involved in opening datasets.

References

1. Adams, E., Dormans, J.: Game Mechanics: Advanced Game Design. New Riders (2012)

2. Arnab, S., Lim, T., Carvalho, M.B., Bellotti, F., de Freitas, S., Louchart, S., Suttie, N., Berta, R., De Gloria, A.: Mapping learning and game mechanics for serious games analysis. British Journal of Educational Technology 46(2), 391–411 (2015). https://doi.org/10.1111/bjet.12113, https://berejournals.onlinelibrary.wiley.com/doi/abs/10.1111/bjet.12113

3. Bharosa, N., Janssen, M., Klievink, B., van Veenstra, A.F., Overbeek, S.: Guiding Integrated Service Delivery: Synthesizing and Embedding Principles Using Role-Playing Games. Electronic Journal of e-Government 8(2), pp83–92 (2010)

4. Broad, E., Tennison, J.: Datopolis board game (Mar 2015), http://datopolis.theodi.org/about/

5. Campbell, D.T.: Experimental and quasi-experimental designs for research on teaching. Handbook of Research on Teaching 5, 171–246 (1963)

6. Crusoe, J., Melin, U.: Investigating Open Government Data Barriers: A Literature Review and Conceptualization. In: Parycek, P., Glassey, O., Janssen, M., Scholl, H.J., Tambouris, E., Kalampokis, E., Virkar, S. (eds.) Electronic Government, vol. 11020, pp. 169–183. Springer International Publishing, Cham (2018)

7. de Caluwé, L., Geurts, J., Kleinlugtenbelt, W.J.: Gaming Research in Policy and Organization: An Assessment From the Netherlands. Simulation & Gaming 43(5), 600–626 (Oct 2012). https://doi.org/10.1177/1046878112439445, http://journals.sagepub.com/doi/10.1177/1046878112439445

8. Di Staso, D., Kleiman, F., Crompvoets, J., Janssen, M.: Changing Civil Servants’ Awareness about Open Data Using a Collaborative Digital Game. In: DG.O2021: The 22nd Annual International Conference on Digital Government Research [In Press]. ACM, Omaha, NE, USA. https://doi.org/10.1145/3463677.3463684

9. Dillenbourg, P., Schneider, D.: Mediating the mechanisms which make collaborative learning sometimes effective. International Journal of Educational Telecommunications 1(2), 131–146 (1995)

10. Duke, R.D.: Gaming: The Future’s Language. Sage Publications, Beverly Hills, Calif. (1974)

11. Goraya, J.: Open Data Card Game (Oct 2015), https://github.com/Atomland/open-data-card-game

12. Janssen, M., Charalabidis, Y., Zuiderwijk, A.: Benefits, Adoption Barriers and Myths of Open Data and Open Government. Information Systems Management 29(4), 258–268 (Sep 2012). https://doi.org/10.1080/10580530.2012.716740, http://www.tandfonline.com/doi/abs/10.1080/10580530.2012.716740
13. Kassen, M.: A promising phenomenon of open data: A case study of the Chicago open data project. Government Information Quarterly **30**(4), 508–513 (Oct 2013). https://doi.org/10.1016/j.giq.2013.05.012, https://linkinghub.elsevier.com/retrieve/pii/S0740624X13000683

14. Kleiman, F., Janssen, M., Meijer, S.: Evaluation of a pilot game to change civil servants' willingness towards open data policy making. In: Simulation and Gaming: Through Times and across Disciplines. pp. 286–297. Kozminski University, Warsaw (Dec 2019)

15. Kleiman, F., Janssen, M., Meijer, S., Jansen, S.J.: Changing civil servants' behaviour concerning the opening of governmental data: Evaluating the effect of a game by comparing civil servants' intentions before and after a game intervention. International Review of Administrative Sciences p. 002085232096221 (Sep 2020). https://doi.org/10.1177/0020852320962211, http://journals.sagepub.com/doi/10.1177/0020852320962211

16. Kleiman, F., Meijer, S., Janssen, M.: Behavioral factors influencing the opening of government data by civil servants: Initial findings from the literature. In: Proceedings of the 13th International Conference on Theory and Practice of Electronic Governance. pp. 529–534 (2020)

17. Luthfi, A., Janssen, M., Crompvoets, J.: Stakeholder Tensions in Decision-Making for Opening Government Data. In: Shishkov, B. (ed.) Business Modeling and Software Design, vol. 391, pp. 331–340. Springer International Publishing, Cham (2020)

18. Mayer, I., Bekebrede, G., Harteveld, C., Warmelink, H., Zhou, Q., van Ruijven, T., Lo, J., Kortmann, R., Wenzler, I.: The research and evaluation of serious games: Toward a comprehensive methodology: The research and evaluation of serious games. British Journal of Educational Technology **45**(3), 502–527 (May 2014). https://doi.org/10.1111/bjet.12067, http://doi.wiley.com/10.1111/bjet.12067

19. Mayer, I.S.: The Gaming of Policy and the Politics of Gaming: A Review. Simulation & Gaming **40**(6), 825–862 (Dec 2009). https://doi.org/10.1177/1046878109346456, http://journals.sagepub.com/doi/10.1177/1046878109346456

20. Nokes-Malach, T.J., Richey, J.E., Gadgil, S.: When is it better to learn together? Insights from research on collaborative learning. Educational Psychology Review **27**(4), 645–656 (2015)

21. Tamberer, J.: Bulk Data or an API? In: Open Government Data: The Book. Second edition edn. (2014), https://opengovdata.io/2014/bulk-data-an-api/

22. Ubaldi, B.: Open Government Data: Towards Empirical Analysis of Open Government Data Initiatives. OECD Working Papers on Public Governance 22, OECD (May 2013)

23. Wendel, V., Gutjahr, M., Göbel, S., Steinmetz, R.: Designing collaborative multiplayer serious games: Escape from Wilson Island—A multiplayer 3D serious game for collaborative learning in teams. Education and Information Technologies **18**(2), 287–308 (Jun 2013). https://doi.org/10.1007/s10639-012-9244-6, https://doi.org/10.1007/s10639-012-9244-6

24. Wolff, A., Barker, M., Petre, M.: Creating a Datascape: A game to support communities in using open data. In: 8th International Conference on Communities and Technologies. pp. 135–138. ACM, New York, NY, USA (Jun 2017)

25. Zagal, J.P., Rick, J., Hsi, I.: Collaborative games: Lessons learned from board games. Simulation & Gaming **37**(1), 24–40 (Mar 2006). https://doi.org/10.1177/1046878105282279, http://journals.sagepub.com/doi/10.1177/1046878105282279