# Provenance in Gamification Business Systems

Michelle Tizuka, Esteban Clua, Luciana Salgado, Troy Kohwalter

► To cite this version:

Michelle Tizuka, Esteban Clua, Luciana Salgado, Troy Kohwalter. Provenance in Gamification Business Systems. 20th International Conference on Entertainment Computing (ICEC), Nov 2021, Coimbra, Portugal. pp.277-288, 10.1007/978-3-030-89394-1_21. hal-04144394

HAL Id: hal-04144394

https://inria.hal.science/hal-04144394

Submitted on 28 Jun 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Provenance in Gamification business systems

Michelle Tizuka, Esteban Clua, Luciana Salgado and Troy Kohwalter

Computer Science Institute, Universidade Federal Fluminense, Niterói, Brazil
{mmtizuka}@id.uff.br,
{esteban,luciana,troy}@ic.uff.br

**Abstract.** Gamification has become increasingly popular in business contexts. This approach suggests long-term sustainable persistent systems as it increases engagement at all organizational levels. However, a gamified system generates a series of events, decisions, and interactions from the users that need to be monitored and captured for analysis. Therefore, those systems require an infrastructure capable of tracking multiple variables over time, identifying the cause-and-effect relationships between events, and analyzing the captured data to evaluate their processes engagement. This work presents a data provenance approach for modeling gamification business systems, which allows tracking, managing, and visualizing provenance data from a gamified event. We evaluated our model in a real context with our partner company, and the results provide the necessary basis to be further implemented. Therefore, project managers can visualize provenance data from a gamified activity, analyze employees' actions, and identify needs that lead to successful or unsuccessful outcomes.

**Keywords:** Gamification model, provenance, data provenance model.

## 1 Introduction

The use of gamification [1] [2] as a strategy in entrepreneur context [3] [4] has advanced as a solution with three main objectives: to increase engagement levels of key stakeholders, such as customers and employees [5] [6] [7], to improve organizational performance [8] [9], and to understand if and when gamification has positively affected users' motivations and behaviors [10].

A gamified application involves a series of activities, which also generates a series of events, decisions, and interactions from the users that need to be monitored and captured for analysis. Therefore, it requires a gamification system with an infrastructure capable of tracking and presenting multiple variables over time. Recent studies focused on gamification requirements, models, and frameworks [11-13]. However, few approaches focus on seeking to identify whether current methods coming from other areas such as games or data analytics also apply to gamification analytics or tools that collaborate to improve the analysis process by the leader. [14-17]. Seaborn and Fels [15] outline most studies did not run statistical analyses (and subsequently could not generate effect sizes). For these authors, it is essential these studies need to be replicated, comparative, and have longitudinal designs employed. Data sets are reliable when the
processes used to create them are reproducible and analyzable for defects. Within com-
puter science, the term "provenance" [19-20] mean the lineage of data, as per data prov-
enance, with research extending the conceptual model of causality and relation to in-
clude processes that act on data and agents that are responsible for those processes
providing historical records of data and its origin.

In business contexts, the provenance of data generated by complex transformations
such as workflows is of considerable value to managers as it allows tracking, managing,
and visualizing provenance data. In this view, we attempt to prospect if it is possible to
use a provenance ontology to model gamification business systems that aim to monitor
and evaluate its accuracy, provide reproducibility, and enable leaders to manage their
processes in detail. In this work we study the use of provenance approach in a gamified
system, specifically to verify if KPI monitoring can be implemented with this approach
and leverage gamification analytics requirements for entrepreneur context.

We developed qualitative research consisted of four main steps: first, we selected a
gamified enterprise application, then we mapped provenance ontology to gamification
elements and mechanics. Later we applied this model to a real entrepreneur scenario
and finally we discuss its application analysis. Our results provide the necessary basis
to help project managers to detect patterns or even historical usability records. There-
fore, project managers can visualize provenance data from a gamified activity, analyze
employees' actions, and identify steps that lead to successful or unsuccessful outcomes.
This scenario is especially representative because multiple influences may lead to suc-
cess or failure in a software project.

This paper presents six sections as follows: Section II presents related works con-
cerning gamification in the business context and the definition of provenance and ter-
minologies applied in games. Section III is where we present our gamification business
system model with a data provenance approach. Section IV exhibits the application of
the proposed model in a real entrepreneur context. Finally, Section V renders discus-
sion, limitations, and future works.

## 2 Related work

Although on the rise, many companies do not have a vision of incorporating gamification
into their system and turning it into a good communication tool and business
management solution. Several studies have indicated the need for personalizing gamified
systems to users' personalities. A cross-contextual study reveals that motivational
experiences increase these outcomes to different extents and go beyond gamified service
experiences [21]. Also, understanding how such experiences of gamified actions
influence business outcomes is critical.

2.1 Gamification in the workplace

These gamified services have appeared on the market as models, being a promising tool
adapted to the organizational context. The result, in a nutshell, is playful solutions for
converting daily tasks into an activity that aggregates elements of competition, cooper-
ation, and narrative with a range of gamification elements and mechanics willing on
satisfying needs [22]. But, as highlighted by Jorge and Sutton [23], it is impossible to
guarantee that everyone will have the same involvement and dedication. Although on
the rise, it is still common to find challenges to keep people active throughout long-
term sustainable persistent systems [24]. The "temporary" engagement does not occur
when the user only gets involved in a specific moment and cannot relate this knowledge
to other activities, which guarantees its performance in knowledge construction and
management. Actions must be rewarded and balanced according to criteria focused on
performance and quality. In addition, user profiles vary in age and roles, which can
mean differences in task assignments. These factors certainly influence how each one
interacts with the gamified system and the leader must be aware and have the challenge
of understanding how the assignment of tasks must be thought of to offer equal oppor-
tunity to all employees.

Tondello and others [25] suggest different game design elements may support different user types and developed a Gamification User Types Hexad framework based on research on human motivation, player types, and practical design experience. For instance, digital badges are known to support learning and promote engagement through peer feedback as behavioral and cognitive indicators [26-29]. While many investigations and frameworks aim to conceptualize a player, [30] states few research categorize players based on their explicit preferences for game elements and mechanics. Even though standardized languages for gamification modeling have been proposed [31], current gamification technologies neither support a common language for its design nor common interfaces for maintaining it [17].

Therefore, this complex network of data and information must be well constructed
and managed based on the choice of variables in-game elements and mechanics that
could be used for different purposes and domains [30]. For instance, behavior
understanding and analysis [32] [33], classifying users [34], understanding common
behaviors, and retention [35], among others [36]. Heilbrun [17] identified seven
available tools and assessed them for their applicability in gamification projects. The
author states that none of the gamification requirements from the groups of application
KPI monitoring, gamification adaptation, user groups, or simulation can be implemented
with those tools. Based on these results, the author concluded that suitable solutions can
be leveraged with a narrow set of analytical requirements for a small number of
scenarios. However, for most use cases, no practical solutions exist.

## 2.2 Provenance in games

Provenance is becoming more and more present in the scientific environment, both to
guarantee the origin of the data, evaluate its accuracy, provide reproducibility, and
enable customers to work with their processes in detail, collaboratively, and
participatory. According to Groth and Moreau [37], provenance is information about
entities, activities, and people involved in producing a piece of data or a thing, which
can be used to assess quality, reliability, and trustworthiness.

Glavic and others [20] use the term data item for a structural unit of data, which
targets provenance management and the notion level of detail for a data item's
granularity.

The World Wide Web Consortium standardized PROV to support the interchange of
provenance information on the Web and defines provenance as a "*record that describes*
the people, institutions, entities, and activities involved in producing, influencing, or
delivering a piece of data or a thing." [36]. The PROV ontology [37] document
expresses the PROV-DM using the W3C OWL2 Web Ontology Language (OWL2). It
provides a set of classes, properties, and constraints that can be used to represent and
interchange provenance information. Since provenance information describes how
various elements were related to or influenced by one another, it can be viewed as a
directed graph in which those elements are nodes, categorized as type (i.e., entities,
activities, agents). Directed edges represent the relations between them (e.g.,
wasGeneratedBy, wasAssociatedBy, wasAttributedTo and used). Such a graph is called
a provenance graph. Given that some provenance graphs can be extensive, the challenge
is to extract useful information and knowledge from complex provenance graphs.

Provenance in games was introduced to detect cause-and-effect relationships,
proposing a conceptual framework that collects information during a game session and
maps it to provenance terms, providing the means for post-game analysis [38-40].
Lately, [41] improved the framework providing the necessary basis to use provenance
information broadening the original approach, implementing concrete frameworks to
track, manage and display provenance data during the game, which allows developers
and analysts to understand the events and the results obtained through interactive graphs
for exploratory analysis. This integration's motivation was based on the fact that neither
of the standard practices of telemetry data used by the games industry considers the
causal relationships during the game sessions. Recently, [42] present a novel approach
for player profiling that leverages recent advances in deep learning over graph
representation at a fine-grain the player behavior in provenance data collected from a
multiplayer battle game and assess the obtained profiles through statistical analysis and
data visualization. Still, there is a need to define gamification features, which one's match
and those that do not with any class of the PROV core structure.

3 Provenance in gamification

Similar to games, we can define gamification systems using data provenance with
different use-cases in mind. Specifically, we need to map each node of a provenance
graph to users, game elements and mechanics, and their relationships. we present our
model as a PROV Core model structure (see Fig.1), detailed in the following subsections.

3.1 Entity and activities

In PROV, "things" we want to describe are called *entities* and encompass a broad
diversity of notions, including digital objects such as a file, physical things such as a
genuine product, as well as abstract concepts and ideas. Therefore, we map entities to
game elements and mechanics, such as points, badges, collectibles, and leaderboards.
They are graphically represented by an oval yellow form.
<figure><img src="image_3.png" /><figcaption>**Fig. 1.** Gamification business model using data provenance (PROV Core Structures).</figcaption></figure>

PROV says that activity occurs over time and can act upon or with entities. We map them as actions related to game mechanics, such as collecting, revealing, repairing, customizing, or generating other entities. Both entities and activities can cover a broad range of notions. They are associated with each other in three ways: activities utilize entities, activities produce entities, and activities can also consume entities. The act of using, producing, or consuming an entity may have a duration and an effect on the entity.

The term "generation" refers to the completion of a new entity's production by an activity. This entity can vary depending on the context and action used to generate it. Thus, the entity did not exist before generation and became available for usage after this step. Likewise, the term "usage" refers to the beginning of utilizing an entity by an activity. Before usage, the activity had not begun to use this entity and could not have been affected by the entity. Communication is the exchange of some unknown entity by two activities, one activity using some entity generated by the other.

In a gamification context, the time of duration of entities and activities can be seen as shortened cycles, where the end of one cycle can also be the beginning of another gamified activity. For instance, feedback (entity) should be immediately or with shortened cycles, generated as points or virtual coins (two different types of entities) to produce enjoyable gameplay, instead of vague long-term performance reports. Each of these shortened cycles can be seen as a provenance core structure that can be tracked. These virtual coins (entity) could be used in a marketplace (another entity), and after the celebration, through sharing in social media can be communicated (one activity using the social media entity by the marketplace). Another agent can receive this purchase order
and then deliver the reward. Same idea is applied on the use of digital badges, serving
as peer agile feedback.

## 3.2 Derivations

Activities utilize entities and produce entities. In some cases, using an entity influences
the creation of another in some way. This notion of "influence" is captured by
derivations, defined as transforming an entity into another, an update of an entity
resulting in a new one, or constructing a new entity based on a pre-existing entity.

According to its definition [37], PROV does not attempt to specify the conditions under which derivations exist. The focus of derivation is on connecting a generated entity to a used entity. While the basic idea is simple, the concept of derivation can be pretty subtle, being implicit the notion that the generated entity was affected in some way by the used entity. If an artifact was used by an activity that generated a second artifact, it does not always mean that the second artifact was derived from the first.

An example of derivation in gamification systems is connecting points (generated entity) to the user's progress bar (a used entity). In this case, utilizing points influences the update of another. For instance, to increase a user's level of experience. The same can occur with currency. After a mission or task is accomplished, the user's virtual coins (generated entity) accumulate on a digital account (a used entity). In this case, it also enables an entity (digital account) to be consumed by another one (marketplace). Therefore, the marketplace is the entity used to promote a new activity (trade) that generates another entity (reward). Also, badges can represent derivation, as the user's knowledge is perceived by his/her project manager. In this case, badges can contain other derivations whether the gamified system uses different categories or progress levelsPROV, "things" we want to describe are called *entities* and have some fixed aspects.

## 3.3 Agents and responsibility

Agents can be related to entities, activities, and other agents. Na agent is something that
bears some form of responsibility for an activity taking place, for the existence of an
entity, or another agent's activity. An agent may also be a particular type of entity or
activity. Thus the model can be used to express the provenance of the agents themselves.
For many purposes, a key consideration for deciding whether something is reliable
and/or trustworthy is knowing who or what *was responsible* for its production.
Therefore, in this case, we have to consider project managers, employees, and the
gamified system. Managers can be the agents responsible for creating gamification
activities, and employees, the agents receiving those activities to execute.

The gamification relation is the assignment of responsibility to the agent (employee) for executing the activity (gamification task), indicating that he/she has a role in the activity. This gamification activity corresponds to this gamified system acting as an agent, associated with other activities inside the system. Delegation is the assignment of authority and responsibility to an agent (by itself or by another agent) to carry out a specific activity as a delegate or representative. In contrast, the agent acts on behalf of retaining some responsibility for the delegated work's outcome. For instance, the manager is responsible for delegating the gamification activity to the employee. However, he/she is also responsible for monitoring and evaluating their results and
providing some feedback related to the results accomplished through their dashboard interface.

# 4 The gamification business model application

The proposed model was instantiated in the business scenario from our partner company, which develops solutions that allow leaders to better manage processes. Among the various attributions that are part of the actions of a leader, developing and putting into practice strategies aimed at motivating the employee, to achieve individual and collective goals, is part of the routine. In addition, there are several processes that a leader needs to follow. For instance, risk management process, which involves planning, organizing, directing, and controlling an organization's human and material resources, to minimize or take advantage of the risks and uncertainties about that organization. In our partner company, we took advantage of having access to complete document history for all user's problems across this scenario to create a robust definition of the model.

## 4.1 Defining metrics and targets

We defined metrics and targets related to employee behavioral activity and retrieved them temporally over a gamification activity in the risk management process. This process involves three main steps: risk analysis/assessment, risk communication, and monitoring, and critical risk analysis. Actions and interactions connect project managers and employees through the system and require quick communication between them. We highlight that project managers have an essential part in the gamified system because they are responsible for not only creating the task and defining “players”, but also monitoring and delivering progress reports, validating (or not) the achievement of the goals proposed.

Thus, the criteria we adopted for defining employee behavior towards the gamified action start with: (1) Not completing the activities in time; (2) being inactive for a period equal or greater to one week of inactivity; (3) complete the task however through a standard solution; and (4) complete the task but using an unexpected solution (see Fig.2A). Project managers can observe these metrics on a dashboard and track activity through a "mission" progress bar, besides delivering rewards and feedback as points, virtual coins, or badges. Leaders can easily visualize where possible failures along the process occur more quickly as this model allows tracking the influences that the actions of employees determine in achieving the proposed objectives.

Detailed information on how certain game elements and mechanics were thought, designed, and related to each other through the whole system is out of this study scope.

## 4.2 Gamification business model using data provenance

We used free wireframe software to represent PROV Core Structures applied to our model (see Fig. 2B). The proposed model was instantiated in the contextual scenario of risk management from our partner company and connects actions and interactions between four agents (project managers, employees, a support person, and the system). The central idea is to track all data for a specific number of gamified actions previously
parameterized by the system. In the sequence, we will briefly present the result of the
application of our approach to modeling our selected scenario.

<figure><img src="image_4.png" /><figcaption>**Fig 2.** Evaluation of the model in a real context: (A) workplace risk management process (B) gamification business model using data provenance on this context.</figcaption></figure>

Managers are responsible for creating the gamification activity (mission or chal-
lenges), for instance, to propose a solution treatment to risk. They are also responsible
for monitoring and validating solutions based on the information given by employees
and to deliver feedback. The gamification activity is 'generated' to refer to the completion
of a new entity's production by an activity (information of the mission accomplished).
This entity (information) can be only raw information or already a solution and can vary
face the context and decision points as seen in Fig.2A. Note that action was used to
generate it, i.e., it did not exist before generation and became available for usage after
this step.

Employees act as "players" on the task given, having the opportunity to receive rewards (as points and virtual coins) and feedback (badges) when concluding a task by the system. So, if the task is completed, the employee level bar is updated, given the notion of 'influence' captured by derivations. Also, likewise "usage" referred before, employees can trade currency for real available products on a virtual and personalized marketplace. Marketplace in this case was not considered an agent as it is not an automated exchange process and therefore still does not act by itself. It will depend on the fourth agent, who has been assigned as a support person, which will be the financial sector that validates and delivers reward directly to the employee (see Fig2B).
## 4.3 Considerations about the gamification model using data provenance

In this case, we suggest considering employees being all users inside a company, besides project managers, regardless of their function, gender, age, or time. Employees can perform different roles and act differently, which demands mapped attributes and paths to identify employee performance. However, this is also a possible action that a gamified system implemented with provenance data can provide, correlating gamified activity data to employee profiles and how their actions influence the results presented.

We highlight gamification features used in this case study are generalizable across the whole gamification business system for other actions. Still, there are few game elements and mechanics that are not adequate for enterprise gamification, e.g., permadeath or punish. Note that not all activity or entity is necessarily a game element or mechanic. For instance, assessment and treatment are actions inherent to the process but are not recognized as game mechanics, and entities such as information (from different types) and register suspended are not game features themselves. However, the whole integration of entities, actions, and traditional game elements are known to provide motivation and engagement (points, badges, and virtual coins) was applied to this scenario. Also, communication and feedback are highlighted and we considered them essential on the usage of provenance data analyses, as celebrating and building an interactive experience are two essential game mechanics that exist in a business context. In this case, we consider employees can send and share their performance results through external media as an option for each person to choose whether or not to perform the action, to avoid forced individual exposure.

Finally, we encourage project managers to engage, even though they are not part of it as “actors”, but to understand that they have a fundamental role in the entire process, as responsible for monitoring and qualifying the results. This process involves not only the strategy of delivering points but monitoring the progress of actions and data analysis and achieved goals. Gamification becomes a means by which they can understand which are the key performance indicators that they should monitor and be able to follow the process to more accurately assess their results and identify any failures throughout the process.

# 5 Conclusions, limitations, and future works

The diversity of data representation models and application domains has led to application or data model-dependent provenance models and prototype implementations. Some authors address provenance in the context of services and workflow management, but as far as we know, still, none has discussed its role in the gamification application domain. It is still a challenge to establish causal relationships between employees' decisions and actions. Thus, we proposed a model for gamification business systems using data provenance to highlight the fundamental PROV Core Structure.

While the core of PROV focuses on essential provenance structures commonly found in provenance descriptions, extended structures are designed to support more advanced provenance uses. The purpose of this work is twofold. First, modeling a gamification business system with data provenance. Second, evaluate the proposed model in an authentic context.
Our model suggests that entities and activities are the most visible part of a gamification system. Representing game elements or mechanics tend to be the primary focus of most gamification projects as action decision points lead to awarding rewards or feedback. These actions also discriminate whether or not a particular gamified activity has been completed. When generated, used, and associated appropriately, entities, activities, and agents can leverage a natural relational, motivational driver enabling tracking gamification data use. An essential aspect of any data gathering is that more than just capturing data to evaluate process engagement, there is a need for practitioners and designers to be cautious about ethical issues in gamification (i.e., exploitation and manipulation) [43].

Despite acknowledging the diversity of game elements and mechanics, our proposal provides the necessary groundwork to explore different ways to summarize provenance graphs into a set of generic gamification features, even those from other areas of the enterprise domain, without the knowledge required to interpret domain-specific information contained therein. We think the analysis and understanding of actions, mistakes, and fluxes of an authentic gamification event may help understand problems related to the management process, data mining of specific situations, and even understanding learning aspects in gamification contexts. Future work will be to conduct a survey with the audience involved, triangulate methods, and implement this model in our partner company application to record data provenance automatically.

# Acknowledgment

(blind)

# References

1. Deterding, S., Dixon, D., Khaled, R., and Nacke, L. "From Game Design Elements to Gamefulness: Defining Gamification", in Proceedings of the 15th international academic Mind-Trek conference: Envisioning future media environments. ACM, pp. 9–15, 2011.

2. Kapp, K. The Gamification of Learning and Instruction Fieldbook. Pfeiffer, 2013.

3. Herzig, P., Ameling, M., Wolf, B., and Schill, A. "Implementing Gamification: Requirements and Gamification Platforms.", in Gamification in Education and Business. Ed. by Torsten Reiners and Lincoln Wood. Springer, 2015.

4. Kumar, J. M., Herger, M., Gamification at Work: Designing Engaging Business Software. 2013.

5. Robson, K., Plangger, K., Kietzmann, J. H., McCarthy, I., & Pitt, L. "Game on: Engaging customers and employees through gamification". Business horizons, 59(1), pp-29-36, 2016.

6. Boulet, G. Gamification: The latest buzzword and the next fad. ELearn, v. 2012, n. 12, p. 3, 2012.

7. Koivisto, J., and Hamari, J. "The rise of motivational information systems: A review of gamification research" in International Journal of Information Management (45), pp. 191-210, 2019.

8. Werbach, K., Hunter, D., For the win: How game thinking can revolutionize your business. Wharton digital press, 2012.
9. Hamari, J., Koivisto, J., & Sarsa, H. "Does gamification work?—A literature review of empirical studies on gamification". In Proceedings of the 47th Annual Hawaii International Conference on System Sciences (HICSS), pp. 3025–3034, 2014.

10. Hagger, M. S., Chatzisarantis, N. L., & Harris, J. "From psychological need satisfaction to intentional behavior: Testing a motivational sequence in two behavioral contexts". Personality & Social Psychology Bulletin, 32(2), pp. 131–148, 2006.

11. Mora, A., Riera, D., Gonzalez, C., and Arnedo-Moreno, J. "A literature review of gamification design frameworks", in 2015 7th International Conference on Games and Virtual Worlds for Serious Applications (VS-Games), pp. 1-8. IEEE. September 2015.

12. Morschheuser, B., Werder, K., Hamari, J., and Abe, J. "How to Gamify? A Method For Designing Gamification", in Hawaii International Conference on System Sciences (HICSS), At Hawaii, USA, Volume: 50, 2017.

13. Al-Towirgi, R. S., Daghestani, L. F., and Ibrahim, L. F. "Data Mining and Gamification Techniques in Adaptive E-Learning: Promises and Challenges", International Journal of Computer Applications, (180:13), pp.49-55, 2018.

14. Heilbrunn, B., Herzig, P., and Schill, A. "Tools for Gamification Analytics: A Survey", in 2014 IEEE/ACM 7th International Conference on Utility and Cloud Computing (UCC). Dec. 2014, pp. 603–608.

15. Heilbrunn, B., Herzig, P., and Schill, A. "Towards Gamification Analytics — Requirements for Monitoring and Adapting Gamification Designs", in 44. Jahrestagung der Gesellschaft für Informatik, Informatik 2014, Big Data - Komplexität meistern, pp. 333–344, 2014.

16. Heilbrunn, B., Herzig, P., and Schill, A. "Gamification Analytics — Methods and Tools for Monitoring and Adapting Gamification Designs", in: Gamification: Using Game Elements in Serious Contexts. Ed. by Stefan Stieglitz, Christoph Lattemann, Susanne Robra-Bissantz, Rüdiger Zarnekow, and Tobias Brockmann. Cham: Springer International Publishing, pp. 31–47, 2017.

17. Heilbrunn, B. Gamification Analytics: Support for Monitoring and Adapting Gamification Designs (Doctoral dissertation, Technische Universität Dresden). 2019.

18. Seaborn, K., and Fels, D. I. "Gamification in theory and action: A survey". International Journal of Human-computer Studies, 74, pp.14–31, 2015.

19. Buneman, P., Khanna, S., Wang-Chiew, T. "Why and where: A characterization of data provenance". International conference on database theory. Springer, Berlin, Heidelberg, pp. 316-330, 2011.

20. Glavic, B., Dittrich, K. R., Kemper, A., Schöning, H., Rose, T., Jarke, M., ... and Brochhaus, C. "Data provenance: A Categorization of existing approaches. BTW'07: Datenbankssysteme". Buisness, Technologie und Web, (103), pp.227-241, 2007

21. Wolf, T., Weiger, W. H., Hammershmidt, M. "Experiences that matter? The motivational experiences and business outcomes of gamified services". Journal of Business Research, v. 106, p. 353-364, 2020.

22. Xi, N., and Hamari, J. "Does gamification satisfy needs? A study on the relationship between gamification features and intrinsic need satisfaction", in International Journal of Information Management (46), pp.210-221, 2019.

23. Jorge, C. F. B.; Sutton, M.J.D. Perspectivas em Gestão & Conhecimento, João Pessoa, v.6, Número Especial, p. 103-118, jan. 2016.

24. Fors, P., and Lennerfors, T. T. Gamification for Sustainability. The Business of Gamification: A Critical Analysis, 163, 2016.

25. Tondello, G. F., Wehbe, R. R., Diamond, L., Busch, M., Marczewski, A., & Nacke, L. E. "The gamification user types hexad scale". Proceedings of the 2016 annual symposium on computer-human interaction in play, pp. 229-243, October 2016.
26. Hatzipanagos, S., and Code, J. "Open badges in online learning environments: Peer feedback and formative assessment as an engagement intervention for promoting agency". Journal of Educational Multimedia and Hypermedia, vol. 25(2), pp.127-142, 2016.

27. Hamari, J. Do badges increase user activity? "A field experiment on the effects of gamification". Computers in human behavior, vol. 71, pp.469-478, 2017.

28. Besser, E. D., and Newby, T. J. "Feedback in a digital badge learning experience: Considering the instructor's perspective". TechTrends, vol. 64(3), pp.484-497, 2020.

29. Fanfarelli, J., Vie, S., and McDaniel, R. "Understanding digital badges through feedback, reward, and narrative: a multidisciplinary approach to building better badges in social environments". Communication Design Quarterly Review, vol. 3(3), pp.56-60, 2015.

30. Ferro, L. S. The Game Element and Mechanic (GEM) framework: A structural approach for implementing game elements and mechanics into game experiences. Entertainment Computing, 36, 100375, 2021.

31. Herzig, P., Ameling, M., and Schill, A. "Workplace Psychology and Gamification: Theory and Application", in Gamification in Education and Business. Springer, pp. 451–471, 2015.

32. Drachen, A. et al. "Guns, swords and data: Clustering of player behavior in computer games in the wild". Conference on Computational Intelligence and Games (CIG), pp. 163–170, 2012.

33. Drachen, A. et al. "Guns and guardians: Comparative cluster analysis and behavioral profiling in destiny. IEEE Conference on Computational Intelligence and Games (CIG). Anais... In: 2016 IEEE CONFERENCE ON COMPUTATIONAL INTELLIGENCE AND GAMES (CIG). set. 2016.

34. Drachen, A. et al. "A Comparison of Methods for Player Clustering via Behavioral Telemetry". Foundations of Digital Games (FDG), 2013.

35. Weber, B. G. et al. Modeling Player Retention in Madden NFL 11. Innovative Applications of Artificial Intelligence Conferences (IAAI), 2011.

36. Drachen, A. and Schubert, M. "Spatial game analytics and visualization", in IEEE Conference on Computational Intelligence in Games (CIG), 2013.

37. Moreau, L., Missier, P., Cheney, J., and Soiland-Reyes, S. PROV-N: The provenance notation. W3C Recommendation, 2013.

38. Kohwalter, T.; Clua, E.; Murta, L. "Provenance in Games". Brazilian Symposium on Games and Digital Entertainment (SBGAMES), p. 162–171, 2012.

39. Kohwalter, T.; Clua, E.; Murta, L. "Game Flux Analysis with Provenance". Advances in Computer Entertainment (ACE), p. 320–331, 2013.

40. Jacob, L. B., Kohwalter, T. C., Machado, A., and Clua, E. W. "A game design analytic system based on data provenance". In International Conference on Entertainment Computing, Springer, Berlin, Heidelberg, pp. 114-119, October 2013.

41. Kohwalter, T., Oliveira, T., Freire, J., Clua, E., and Murta, L. "Prov viewer: A graph-based visualization tool for interactive exploration of provenance data". In International Provenance and Annotation Workshop. Springer, Cham, pp. 71-82, June, 2016.

42. Melo, S. A., Kohwalter, T. C., Clua, E., Paes, A., and Murta, L. "Player Behavior Profiling through Provenance Graphs and Representation Learning". In International Conference on the Foundations of Digital Games, pp. 1-11, September 2020.

43. Kim, T.W.; Werbach, K. "More than just a game: ethical issues in gamification". Ethics and Information Technology, v. 18, n. 2, pp. 157-173, 2016. Author, F.: Article title. Journal 2(5), 99–110 (2016).