CHI Conference on Human Factors in Computing Systems Extended Abstracts, 2021

Removing Gamification:
A Research Agenda

KATIE SEABORN
Department of Industrial Engineering and Economics
Tokyo Institute of Technology

CITATION: Seaborn, K. (2021). Removing gamification: A research agenda. In Extended Abstracts of the 2021

CHI Conference on Human Factors in Computing Systems (CHI EA ‘21), in press.
https://doi.org/10.1145/3411763.3451695

The final publication will be available via ACM at https://doi.org/10.1145/3411763.3451695.

Removing Gamification: A Research Agenda
Katie Seaborn
Tokyo Institute of Technology
Tokyo, Japanseaborn.k.aa@m.titech.ac.jp
ABSTRACTThe effect of removing gamification elements from interactive systems has been a long-standing question in gamification research. Early work and foundational theories
raised concerns about the endurance of positive effects andthe emergence of negative ones. Yet, nearly a decade later,
no work to date has sought consensus on these matters.
Here, I offer a rapid review on the state of the art and what
is known about the impact of removing gamification. A
small corpus of 8 papers published between 2012 and 2020
were found. Findings suggest a mix of positive and negative
effects related to removing gamification. Significantly, insufficient reporting, methodological weaknesses, limited
measures, and superficial interpretations of “negative” results prevent firm conclusions. I offer a research agenda towards better understanding the nature of gamification
removal. I end with a call for empirical and theoretical workon illuminating the effects that may linger after systems are
un-gamified.
CCS CONCEPTS• Human-centered computing

KEYWORDS
Gamiﬁcation, Longitudinal eﬀects, Gamiﬁcation removal, Ungamify
ACM Reference format:
Katie Seaborn. 2021. Removing Gamiﬁcation: A Research Agenda. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems (CHI EA ‘21), May 8–13, 2021, Yokohama, Japan. ACM, New
York, NY, USA. 9 pages. h"ps://doi.org/10.1145/3411763.3451695

1 INTRODUCTION
Gamiﬁcation is the use of game elements in non-game systems to provide a game-like experience and inﬂuence user
motivation and engagement [8,19,40]. As an interdisciplinary topic, gamiﬁcation has been explored across a variety
of domains and areas of practice, from education to healthto marketing and more [17,22,32,40]. #e ﬁeld of gamiﬁcation research is maturing. Since the ﬁrst reviews of the literature conducted about 5-6 years ago, there has been adramatic increase in studies and surveys on gamiﬁcation.
#e ACM Digital Library, for instance, showed a 2¼-foldincrease in gamiﬁcation-related work between 2016 and thepresent1. Much of this body of work has addressed the gaps,
issues, and opportunities originally identiﬁed with increasing complexity, speciﬁcity, and rigor.
Yet, one critical question raised early on through foundational studies [42], theoretical predictions [4,5], and previous surveys [40] has remained open: the eﬀect ofremoving gamiﬁcation, or “un-gamifying” a system. In 2011,
Cunningham and Zichermann [4] noted that the most common approaches to gamifying systems targeted extrinsic
motivation, or motivation that is driven by external rewards rather than intrinsic satisfaction [5]. As suggested by
the formative work of Deci, Ryan, and colleagues [5,37,38],
extrinsic motivation may not lead to lasting behavioral
change a&er the rewarding mechanism is removed. #us,
by using game elements that tend to act as extrinsic incentives, like points [26,27] and leaderboards [21,41], users
tend to be extrinsically engaged rather than ﬁnding the system intrinsically enjoyable. As such, the system may needto stay gamiﬁed forever—otherwise, the positive eﬀects
may be lost, or even turn negative. In their seminal 2012
paper, #om et al. [42] addressed this directly by designinga study wherein gamiﬁcation was removed a&er a 10months period. #ey found that participation drasticallydecreased. #is negative result was featured by Seaborn andFels [40] in their 2015 survey paper, where they called forstudies on removing gamiﬁcation in their research agenda.
In 2017, as the ﬁeld started to mature, the dire need fordeeper studies on such behavioral pa'erns and their connection to theory was echoed by Nacke and Deterding [31]in their research agenda. Most recently, Rapp et al. [35] signaled a need for ethical reﬂection on integrating gamiﬁcation into (and presumably out of) people’s lives.
At nearly a decade a&er the initial work and ﬁve years
a&er the ﬁrst call, it is time to trace out what we know about
Calculated from a search using the keyword “gamification” conducted on
December 15, 2020, which returned 892 results between 2011 and 2015, inclusive,and 2005 results between 2016 and 2020, inclusive.
1Removing Gamification: A Research Agenda

CHI EA ’21, May 8–13, 2021, Yokohama, Japan

gamiﬁcation removal. #e stakes are high given the sheer
amount of gamiﬁcation work conducted in CHI and otherspaces. If there are negative eﬀects upon removing gamiﬁcation, they could be widescale, and if not reported, they
could be invisible. #us, the objective of this rapid review
was to understand the state of the art on gamiﬁcation removal. I asked: What is the nature of the research and ﬁndings on removing gamiﬁcation? #e contributions are
twofold: (1) to identify and describe how the question of
gamiﬁcation removal has been addressed in HCI; and (2) toprovide a consensus on the eﬀects of gamiﬁcation removal.
Taken together, this work extends theoretical and empiricalknowledge in the ﬁeld of gamiﬁcation on a key topic withwidespread and continuing implications.

Literature search

TITLE-ABS-KEY (gamification) AND (TITLE-ABSKEY (removing) OR TITLE-ABS-KEY (removal))n = 20(TS=(gamification) AND TS=(remov*)) AND
DOCUMENT TYPES:(Article OR Book OR Book
Chapter OR Proceedings Paper)n = 22(Abstract:(gamification) AND Abstract:(remov*))
OR (Title:(gamification) AND Title:(remov*)) OR(Keyword:(remov*) AND Keyword:(gamification))n =5("All Metadata":gamification AND "AllMetadata":remov*)n =5

ScopusWeb of Science
ACM Digital Library
IEEE Xplore
Limits: English-language only

Search results combined (n = 52)

Excluded (n = 46)
Articles screened based on inclusionand exclusion criteria

Duplicates (n = 17)
Did not evaluate gamification removal (n = 27)
Inaccessible (n = 2)

Included (n = 6)

Manual additions from citations ofincluded papers and expert suggestion

Included (n = 8)

2 METHODS
A rapid review [9,11,13,24] was conducted. Rapid reviews
are appropriate for focused, well-deﬁned, and time-sensitive research questions that produce descriptive statistics,
narrative summaries of qualitative data, and possibly somequantitative ﬁndings. #e research question posed for thiswork is well-suited for a rapid review, being narrowly deﬁned, timely, and geared towards producing a descriptive
and narrative synthesis of the surveyed literature. A version of the PRISMA checklist [25] adapted2 for HCI purposes was used to guide the rapid review process andstructure the reporting of results. See Figure 1 for the
PRISMA ﬂow chart. I conducted all database searches, data
extraction, and data analyses alone, which is common andadequate for rapid reviews [24]. For greater rigor and accountability, this protocol (OSF # 75cqh) was registered before data extraction on December 15, 20203.
2.1 Eligibility Criteria
Full studies involving human subjects interacting with a
gamiﬁed computer system were included; pilot studies, proposals, technical reports, surveys, and grey literature were
excluded. Only uses of gamiﬁcation in sense deﬁned
above—game elements rather than full games—were included; serious games, gameful design, and so on were excluded. Only studies that included the removal of
gamiﬁcation features and evaluated the eﬀects of doing sowere included. Inaccessible studies, such as oﬄine work,and studies not in English were excluded.

For instance, items and language specific to the medical domain, suchas patient populations, were excluded or modified as appropriate.
2

Figure 1: PRISMA flow diagram of the rapid review procedure.

2.2 Search Queries and Study Selection*eries were run in general databases (Scopus and Web of
Science) and disciplinary databases (ACM Digital Library
and IEEE Xplore) on December 15, 2020. All queries used
the terms “gamiﬁcation” and “remov*” or, when it was notpossible to use the * qualiﬁer, a combination of “removal”
and “removing.” Duplicates were removed. #e abstract
and, if necessary, full text was then reviewed against theeligibility criteria. #e related work sections of the selected
papers and the citing papers were manually reviewed for
additional relevant papers. An outside expert suggested an
additional paper. In the end, from an initial 52 papers, 7
were selected for inclusion.
2.3 Data Analysis
I extracted all data and metadata, including: the study type,research approach (quantitative, qualitative, mixed), lab or
ﬁeld, timeframe of the study, area or discipline, gamiﬁcation features, research question(s), hypothesis/es, participant data (gender, age), independent and dependent
variables and their groups/levels, measurement (subjective
and objective), data source, number of items, format of dataor responses, validation procedures, analysis methods, andmajor ﬁndings. Descriptive statistics were calculated. #eextracted ﬁndings on gamiﬁcation removal were separated3

https://osf.io/75cqh

CHI EA ’21, May 8–13, 2021, Yokohama, Japan(in the case of multiple ﬁndings per study) and then categorized as positive, negative, mixed, or neutral. Descriptive
statistics were then calculated for these categories.
3 RESULTS
From an initial 52 papers, a small corpus of 8 papers reporting on eight studies were identiﬁed as eligible. Summarized
descriptions are presented in Table 1 and Table 2. #e Moldon, Strohmaier, & Wachs [29] paper was a preprint posted
to arXiv. As it had not undergone peer review, its ﬁndings
were not included in that analysis; it was only used in describing the nature of studies. Also, the Featherstone [10]paper did not report on the results of gamiﬁcation removal,
despite the stated intention and research design, and so its
ﬁndings could not be reported.
3.1 Nature of Gamification Removal
Table 1 presents an overview of study characteristics andthe gamiﬁcation elements used. All studies were conducted
in ﬁeld. Half (4/8) had an education context. Most (6/8) used
a quasi-experimental design, with non-experimental components being lack of hypotheses, lack of randomization,lack of control groups, etc. All studies employed a longitudinal design, but timeframes varied: the shortest was 2
weeks [2] and the longest was 3 years [29]. Four studies
included a baseline, which varied in length: 9.5 months [42],5 months [15], and 3 weeks [10,33]. One [44] used a pre-test
as a baseline. Gamiﬁcation was in place for diﬀerent lengthsof time: 7 weeks [44], 6 weeks [10], 2 weeks [12,33,42], and1 week [2]. Notably, Günther, Kacperski, & Krems [15] applied the same gamiﬁcation feature at two diﬀerent times,
each for 3 months. #e timing is unknown in Moldon et al.
[29]. A clear research question was found in only 4/8 papers, but all stated the purposes for gamifying and studying
removal. Reasons for gamifying included increasing user
activity [2,15,29,42], improving performance [33,44], enhancing experience [10], and understanding the eﬀect on
motivation [12]. For removal, reasons included isolating theinﬂuence of gamiﬁcation on behavior [2,10,29,42,44] anddetermining maintenance of eﬀects over time [12,15,33].
Only 2 studies included hypotheses, despite the use of experimental approaches and inferential analyses.
Most studies did not report detailed participant demographics; only Günther et al. [15] provided gender distributions and age details (mean, standard deviation, andSeaborn

range). 2/8 reported only the total subject pool. Gamiﬁcation elements included points (5/8), leaderboards (5/8), progress indicators (5/8), rewards (3/8), badges (2/8), avatars(2/8), currency (2/8), and missions (2/8). Gamiﬁcation removal involved the full removal of all gamiﬁcation elements except in the case of Featherstone [10], wherein only
the ba'le game was removed (badges, progress bars, avatars, currency, and rewards remained).
Table 2 presents an overview of variables, measures, andmeasurement, as well as the valence of the results (positive,
negative, mixed, or leaning). Most studies used quantitative
data, i.e., logs (6/8). #e exception was García Iruela et al.
[12], who used a subjective self-report questionnaire. All
relied on quantitative data analysis, including inferential
testing. Most studies (5/8) looked at presence or absence,with 3 considering order eﬀects, and 1 considering multiple
timeframes. Most studies (6/8) focused on measures related
to evaluating the worth of gamiﬁcation for the platform
creators or purveyors, such as participation rates (3/8), performance (1/8), and app usage (1/8). User-centered alternatives included motivation for the user and course grading.
3.2 Findings on Gamification Removal
Most studies reported negative [2,12,42] or mixed results[12,15,44]. For instance, #om et al. [42] found that contributions of photos and lists decreased, as did comments
across the platform, and high-scorers stopped leaving as
many terse greetings. In García Iruela et al. [12], perceived
conﬁdence was higher in the ﬁrst two weeks of the gamiﬁed group compared to the ungamiﬁed group, but this ﬂattened on gamiﬁcation removal. In contrast, de Paula Portoet al. [33] found that gamiﬁcation increased the speed oftime sheet entry, but speeds dropped by over half a&er removal (1.99 vs .47 vs. .8). Günther et al. [15] also found thatthe positive eﬀects of gamiﬁcation remained, but with aslight, non-statistically signiﬁcant decline. Van Nulan et al.
[44] found that exam scores and course grades remained
high a&er gamiﬁcation removal, with both early- and latestarters performing be'er to the non-gamiﬁed group.
Given that three strategies (feedback, gamiﬁcation, ﬁnancial rewards) were cumulatively included over time, and theinclusion of two gamiﬁcation periods and removals, it isdiﬃcult to conclude what the role of gamiﬁcation was. Finally, as noted, Featherstone [10] did not report on the results of removal, so the impact is unknown.

Removing Gamification: A Research Agenda

4 DISCUSSION
Despite the years gone by, gamiﬁcation removal is still in
its infancy. #e current body of work is deﬁned by a mixture of strengths and weaknesses. Methodological limitations (such as a focus on quantitative measures) andweaknesses (such as poor reporting) abound. But a numberof strengths can be seen, as well. In contrast to the greater
portion of HCI and UX research [36], all studies took placein the ﬁeld. #is echoes a turn towards research in real contexts within gamiﬁcation broadly [35]. Findings on gamiﬁcation removal were mixed but lean negative. Most conﬁrm
earlier fears, with even initial positive results declining on
removal. Given the small pool and noted issues, ﬁrm conclusions cannot be drawn. More work needs to be done tostrengthen and expand—and potentially refute—these ﬁndings. As such, instead of a typical discussion, I oﬀer a more
useful format: a research agenda.
4.1

Research Agenda

4.1.1 Encore and extend: Experimentally remove gamification. In essence, this is a repeat of the original call [40] to
examine the repercussions of gamification removal. But it
is also an extension of that call, with a view to going beyond
the low-hanging fruit. While some of the below may havebeen applied to or suggested for other gamification topics,
they also need to be applied to gamification removal.
Research designs: Experiments—the more tightly controlled, the be'er—with comparative designs will beneeded. #ree designs are recommended. A pre-post design, where gamiﬁcation is removed for a single group. A
cross-over design, as in Van Nuland [44] and García Iruela
et al. [12], where the gamiﬁed and ungamiﬁed groups are
switched mid-study. And designs with a control group,
which can be those who do not experience the gamiﬁed system or those who do not have the gamiﬁcation elements
removed. Importantly, these designs should pinpoint the effects of gamiﬁcation elements and their removal. A cumulative design, such as in Günther et al. [15], may allow fora more natural and multifaceted result, but also prevents us
from knowing the exact eﬀects of gamiﬁcation itself. Research questions should be clearly stated (3/8 did not) andhypotheses should be used/stated (only 2/8 did).
Variables, measures, and measurement: Gamiﬁcation removal may be multifaceted, and so the variables understudy and how they are measured should reﬂect that. Possible individual level and group level variables should be

CHI EA ’21, May 8–13, 2021, Yokohama, Japanconsidered. For instance, in #om et al. [42], country mediated responses to gamiﬁcation and its removal in the same
system. Other possibilities from the extant literature include personality [21], game-oriented user types [43], andorientations to motivation [16], to name a few. Selecting
individuals as exemplars and extreme cases for case studies,
as in Featherstone [10], may help illustrate diﬀerences. #e
use of logs and other noninterfering and objective methodsof data collection should continue. While increasing in popularity, we should be cautious about using self-reporting
methods within gamiﬁed systems, as gamiﬁcation itself caninﬂuence reporting [14].
Data collection methods and analysis: *antitative methods can identify the presence and extent of eﬀects; the current body of work does this eﬀectively, and this should
continue. However, why there is or is no eﬀect needs to beaddressed more eﬀectively. #om et al. [42], for example,
identiﬁed a diﬀerence based on country, but did not conduct research on the reason(s) for it. Speculation and comparisons to similar research are a good starting point but
not suﬃcient to explain the speciﬁc case. *alitative methods can be used to understand the “why” and provide context where the quantitative cannot; this is where thecurrent body of work can be improved. Also, when data iscollected is paramount. Measuring at the point of removal
may be too soon to see eﬀects. García Iruela et al. [12], forinstance, show a likely novelty eﬀect. Stagger when gamiﬁcation is removed and when eﬀects are measured, as in dePorta et al. [33] and Günther et al. [15]. A cross-over design
is a simple but rigorous option [23]. Report all results: Reporting the eﬀects of removal—not just the act of removal,
as in Featherstone [10]—will be necessary. Negative results
add to knowledge just as positive results do [28]. Further, areader cannot assume a negative result when no result isreported. Negative results can resolve disputes and point to
the research trajectories that are worth pursuing, as well as
clarify the eﬀects of gamiﬁcation removal itself.
4.1.2 Interrogate the negative results. The surveyed papers,
theoretical background, and other review work have highlighted the possibility of negative results upon gamification
removal. We should not shy away from this [20]. Yet, themeaning of these “negative” results needs reconsiderationand deeper interpretation. I outline some possibilities here.

CHI EA ’21, May 8–13, 2021, Yokohama, Japan

Seaborn

Table 1: Study and gamification characteristics; papers sorted by year.
SourceArea

Research Design

Timeframe

Thom, Millen, & DiMicco(2012) [42]Workplace

Between-subjects
quasi-experimentin ﬁeld

10 months of use; data
3486 employees (1815 in Points, leaderboards; allcollected 2 weeks before and 2 the US, 287 in India)
removedweeks a"er use (4 weeks total)

Amriani, Aji, Utomo, &Junus (2014) [2]

Education

Between- andwithin-subjects
quasi-experimentin ﬁeld

2 weeks of use; 1 week
38 high school students
gamiﬁed or not gamiﬁed; order aged 16-18: 14 men andswitched between groups
24 women

Points, leaderboards, badges(“titles”), progress bar(“completion track”); allremovedFeatherstone (2017) [10]Education

Within-subjects
quasi-experimentin ﬁeld

12 weeks of use; 3 weeks oﬀ, 6 26 2nd-year undergrads
weeks on, then 3 weeks oﬀ

Badges, progress bar, avatars,
currency, rewards, ba$le;only ba$le activity removed

de Paula Porto, Ferrari, &Fabbri (2019) [33]WorkplaceWithin-subjects
quasi-experimentin ﬁeld

7 weeks of use; 3 weeks ofbaseline, 2 weeks on, then 2
weeks oﬀ

Points, leaderboards,
progress bar, avatars,
currency (“gold”), rewards,
missions, health bar, guilds;
all removed

Günther, Kacperski, &Krems (2020) [15]EnergyWithin-subjects
ﬁeld study

22 months of use; 5-month
108 university staﬀ aged Leaderboards
baseline, 3 months for each of 18-64 (M= 34, SD=8.9);
Feedback, Gamiﬁcation I,
89 men and 28 women
Rewards, Gamiﬁcation II, then
5 months oﬀ

García Iruela, Fonseca,
Hijón-Neira, & Chambel(2020) [12]Education

Within-subjects
quasi-experimentin ﬁeld

4 weeks of use; 2 weeks
169 to 113 (77 dropouts) Points, leaderboards,
gamiﬁed or not gamiﬁed; order 1st-year undergrads;
progress bar, rewards,
switched between groups
mean age of 20
missions, time limits; allremoved

Moldon, Strohmaier, &Wachs (2020) [29]So"wareWithin-subjects
Development ﬁeld study

Van Nuland, Roach, Wilson, Education& Belliveau (2014) [44]Within-subjects
experimentParticipants

13 employees in aso"ware development
company aged 21-29: 10
men and 3 women

Gamiﬁcation Elements

3 years; ~1.5 years on, thenremoved433,138 GitHub so"ware Points (# of days streak),developers
progress (# days in current
streak); all removed

13 weeks; switched at week 767 2nd-year undergrads

Competitive quiz; removed

Removing Gamification: A Research Agenda

CHI EA ’21, May 8–13, 2021, Yokohama, Japan

Table 2: Variables, measurement, and valence of findings; papers sorted by year.
SourceIV

DV

MeasureMeasurement

Valence

Thom et al. [42]Country (US vs. Indian)Participation

Objective measures: # of photos, # of lists, # ofcomments, # of terse greetings

Logs, # of type ofcomments

Negative

Amriani et al. [2]Presence or absence of
gamification; order of thegamified week

Participation

Objective measures: Course view, assignment
submission, discussion created, discussion
reply and discussion view

Logs

Negative

Featherstone [10]Presence or absence of battle App usage
games

Objective measures: attendance, handing in
assignments, completing tutorials, manualawards from staff based on spontaneous
activities (such as answering a question)

Logs

n/aa

de Paula Porto et al.
[33]

Presence or absence of
gamification

Performance

Objective measures: Delay in completing time
sheet (# of days)

Logs

Positive

Günther et al. [15]Intervention phase(cumulative)

Eco-driving
behavior

Objective measures: Average energy
consumption while driving

Logs

MixedPositive

García Iruela et al. [12] Order of the gamified week

Motivation

Subjective measure: items from the Intrinsic
Motivation Inventory (IMI) [6], includingpressure/tension, perceived choice, perceived
competence, effort/importance

Questionnaire

MixedNegative

Moldon et al. [29]Presence or absence of
gamification; streakers andnon-streakers

Participation

Objective measures: # of streaks, length of
streaks (days), activity on weekends (share ofactivity), streak after 100-day goal, streaks insocial networks based on homophily(similarity)

Logs

Negativea

Van Nuland et al. [44]Presence or absence of
gamification

Performance

Objective measures: Exam scores (mid andfinal), final grade

Exams

MixedPositivea Findings may be inconclusive due to poor research design, analysis, and/or the pre-print, pre-peer review status of the paper.

CHI EA ’21, May 8–13, 2021, Yokohama, JapanEngage with theory: A weakness of older gamiﬁcation research (cf. [31,35,40]) was a disconnect between theory andthe phenomena under study. Over time this has improvedfor certain measures, especially motivation, and certain
theories, especially Self-Determination "eory (SDT) [31].
But we need to consider other factors. For instance, somework suggests that forcing fun and games on people canmake an enjoyable activity a negative one [7,18,30]. Yet, a
reduction in performance following gamiﬁcation removal
may be interpreted in diﬀerent ways. For instance, if theuser perceived the gamiﬁcation elements as extrinsic, declines may be due to a natural regression to the mean. But
if they were intrinsically enjoyable, declines could be related to declines in satisfaction, even momentary ones.
Even further, if long-term exposure to a gamiﬁed system
leads to “oﬄoading self-regulation” [7], taking it away
could leave a person dazed and confused, without directionor the ability to self-manage.
Specify the recipient(s) of the negative eﬀects: We should
also distinguish for whom the result is negative: those employing gamiﬁcation, hoping to harness its beneﬁts by inﬂuencing or controlling others, or those on the receiving
end, who may be extrinsically or intrinsically motivated.
When it comes to extrinsic motivation, those that no longer
feel the crack of the “electronic whip” [1,7] may be less inclined to perform according to the powers that be—and may
even be happier. We will need to rethink what is being
measured and for the beneﬁt of whom, users and deployers.
Self-reports on satisfaction, aﬀect, joy, stress, and otheruser-centered measures could counterbalance the objective
user performance measures most studies relied on, collected for the beneﬁt of the overseer.
Clarify the source: Outcomes can have one or more
sources (see 4.1.1) or be confounded by other factors. For
instance, the cumulative eﬀects and/or interactions between the incentive measures applied in Günther et al. [15]
could have inﬂuenced the eﬀects of gamiﬁcation before anda&er removal. Comparisons of diﬀerent gamiﬁcation elements and/or the removal of single elements will be needed,
as in Featherstone [10]. Studies will need to be replicated in
diﬀerent contexts to pinpoint source(s), such as points in ane-health app compared to an online class. Known confounding factors, such as usability, system errors, technicalability, and a'itudes towards games should also be considered. "ese may be speciﬁc to the context, such as driving
experience in Günther et al. [15].

Seaborn

4.1.3 Harness the wild. An accessible and ecologically valid
way to answer this call may be found “in the wild.” Half ofstudies took place in an educational context, likely because
many of us are within or adjacent to such contexts. We
should continue to explore learning environments so as togenerate points of comparison. But companies and organizations are also continuing to invest in gamification [3].
Many platforms are public. As Moldon et al. [29] point out,
websites like StackOverflow even change gamification features over time, leading to publicly visible effects on user
behavior. Users also share their reactions on public forums,
especially social media. For instance, a 2016 Nike redesignof their gamified training app prompted public outcry [45].
The removal of gamification elements was not well-received by such a large portion of the user base that Nike
responded with various conciliatory measures and a fix. Using such data is robust [34,39]. Gamification researchers
can use this as a data source by identifying when platforms
have added and removed gamification features, even working with platform creators to do so.
4.1.4 Applicability to gamification research in general. This
research agenda can apply to all gamification work going
forward. The issue of removal is fundamental: in most research on gamification, people are subjected to gamification over some period of time, and then it is (presumably)
removed. Short-term work and one-off studies may be exempt, but this needs to be studied. Future and ongoing work
can incorporate modifications to research protocols, while
previous studies can be re-run.
5 CONCLUSION
Gamiﬁcation research and practice has advanced since thetopic of gamiﬁcation removal was ﬁrst raised. As this rapid
review indicates, there is a risk of reversals in positive outcomes and other negative eﬀects, as well as positive ones.
"e sheer amount of research on gamiﬁcation and gamiﬁcation platforms in use out in the world demands our a'ention. At this time, there is a small corpus from which to
draw conclusions, one subject to methodological weaknesses and limitations. We can move forward by acknowledging this and tackling a new research agenda. We canand must do be'er to understand the real eﬀects of gamiﬁcation removal in all their shades of grey.
ACKNOWLEDGMENTS
My sincere gratitude to Peter Pennefather for his guidanceand early feedback. "ank you also to the reviewers.

Removing Gamification: A Research Agenda

REFERENCES[1] Frederick E. Allen. 2011. Disneyland Uses “Electronic Whip” on Employees.
Forbes.
Retrieved
December
16,
2020
from
https://www.forbes.com/sites/frederickallen/2011/10/21/disneyland-useselectronic-whip-on-employees/[2] Afifa Amriani, Alham F. Aji, Andika Y. Utomo, and Kasiyah M. Junus. 2014. An
empirical study of gamification impact on e-Learning environment. In
Proceedings of 2013 3rd International Conference on Computer Science andNetwork
Technology(ICCSNT
2013),
265-269.
https://doi.org/10.1109/ICCSNT.2013.6967110[3] Jenny Chang. 2019. 47 Gamification Statistics You Must Know: 2020 Market
Share Analysis & Data. FinancesOnline. Retrieved December 16, 2020 fromhttps://financesonline.com/gamification-statistics/[4] Christopher Cunningham and Gabe Zichermann. 2011. Gamification by Design:
Implementing Game Mechanics in Web and Mobile Apps. O’Reilly Media,
Sebastopol, CA.
[5] Edward L. Deci, Richard Koestner, and Richard M. Ryan. 1999. A meta-analytic
review of experiments examining the effects of extrinsic rewards on intrinsic
motivation. Psychological Bulletin 125, 6, 627.
[6] Edward L. Deci and Richard M. Ryan. 2005. Intrinsic Motivation Inventory (IMI).
Retrieved
June
5,
2011
from
http://www.psych.rochester.edu/SDT/measures/IMI_description.php[7] Sebastian Deterding. 2014. Eudaimonic design, or: Six invitations to rethinkgamification.
[8] Sebastian Deterding, Miguel Sicart, Lennart Nacke, Kenton O’Hara, and Dan
Dixon. 2011. Gamification: Using game-design elements in non-gaming
contexts. In Proceedings of the 2011 CHI Conference Extended Abstracts on Human
factors in Computing Systems (CHI EA ’11), 2425-2428.
[9] Maureen Dobbins. 2017. Rapid Review Guidebook: Steps for Conducting a Rapid
Review. National Collaborating Centre for Methods and Tools, Hamilton, ON.
Retrieved from https://www.nccmt.ca/capacity-development/rapid-reviewguidebook

CHI EA ’21, May 8–13, 2021, Yokohama, Japan[19] Kai Huotari and Juho Hamari. 2012. Defining gamification - A service
marketing perspective. In Proceedings of the 16th International Academic
MindTrek Conference (MindTrek ’12), 17-22.
[20] Sami Hyrynsalmi, Jouni Smed, and Kai Kimppa. 2017. The dark side of
gamification: How we should stop worrying and study also the negative
impacts of bringing game design elements to everywhere. In Proceedings of the
1st International GamiFIN Conference (GamiFIN ’17), 96-104.
[21] Yuan Jia, Yikun Liu, Xing Yu, and Stephen Voida. 2017. Designing leaderboardsfor gamification: Perceived differences based on user ranking, application
domain, and personality traits. In Proceedings of the 2017 CHI Conference on
Human Factors in Computing Systems (CHI ’17), 1949–1960.
https://doi.org/10.1145/3025453.3025826[22] Daniel Johnson, Sebastian Deterding, Kerri-Ann Kuhn, Aleksandra Staneva,
Stoyan Stoyanov, and Leanne Hides. 2016. Gamification for health andwellbeing: A systematic review of the literature. Internet Interventions 6, 89-106.
https://doi.org/10.1016/j.invent.2016.10.002[23] Byron Jones and Michael G. Kenward. 2014. Design and Analysis of Cross-Over
Trials. CRC Press, Boca Raton, FL.
[24] Sara Khangura, Kristin Konnyu, Rob Cushman, Jeremy Grimshaw, and David
Moher. 2012. Evidence summaries: The evolution of a rapid review approach.
Systematic Reviews 1, 1, 10. https://doi.org/10.1186/2046-4053-1-10[25] Alessandro Liberati, Douglas G. Altman, Jennifer Tetzlaff, Cynthia Mulrow,
Peter C. Gøtzsche, John P. A. Ioannidis, Mike Clarke, P. J. Devereaux, Jos
Kleijnen, and David Moher. 2009. The PRISMA statement for reporting
systematic reviews and meta-analyses of studies that evaluate healthcareinterventions:
Explanation
andelaboration.
BMJ
339.
https://doi.org/10.1136/bmj.b2700[26] Elisa D. Mekler, Florian Brühlmann, Klaus Opwis, and Alexandre N. Tuch. 2013.
Disassembling gamification: The effects of points and meaning on user
motivation and performance. In CHI ’13 Extended Abstracts on Human FactorsinComputing
Systems
(CHI
EA
’13),
1137–1142.
https://doi.org/10.1145/2468356.2468559[10] Mark Featherstone. 2017. A formative evaluation of a gamification app
containing asynchronous multiplayer game elements. Proceedings of the 11th
European Conference on Games Based Learning, 798-806.

[27] Elisa D. Mekler, Florian Brühlmann, Alexandre N. Tuch, and Klaus Opwis. 2017.
Towards understanding the effects of individual gamification elements on
intrinsic motivation and performance. Computers in Human Behavior 71, 525534. https://doi.org/10.1016/j.chb.2015.08.048[11] Rebecca Ganann, Donna Ciliska, and Helen Thomas. 2010. Expediting
systematic reviews: Methods and implications of rapid reviews. Implementation
Science 5, 1, 56. https://doi.org/10.1186/1748-5908-5-56[28] Ana Mlinarić, Martina Horvat, and Vesna Šupak Smolčić. 2017. Dealing withthe positive publication bias: Why you should really publish your negative
results. Biochemia Medica 27, 3. https://doi.org/10.11613/BM.2017.030201[12] Miguel García Iruela, Manuel J. Fonseca, Raquel Hijón-Neira, and Teresa
Chambel. 2020. What happens when gamification ends? In Artificial Intelligencein Education (AIED 2020), 98-102. https://doi.org/10.1007/978-3-030-52240-7_18[29] Lukas Moldon, Markus Strohmaier, and Johannes Wachs. 2020. How
gamification affects software developers: Cautionary evidence from a quasiexperiment on GitHub. arXiv:2006.02371 [cs]. Retrieved December 15, 2020 fromhttp://arxiv.org/abs/2006.02371[13] Chantelle Garritty, Gerald Gartlehner, Barbara Nussbaumer-Streit, Valerie J.
King, Candyce Hamel, Chris Kamel, Lisa Affengruber, and Adrienne Stevens.
2021. Cochrane Rapid Reviews Methods Group offers evidence-informed
guidance to conduct rapid reviews. Journal of Clinical Epidemiology 130, 13-22.
https://doi.org/10.1016/j.jclinepi.2020.10.007[14] David Gundry and Sebastian Deterding. 2018. Intrinsic elicitation: A model anddesign approach for games collecting human subject data. In Proceedings of the
13th International Conference on the Foundations of Digital Games (FDG ’18), 1–
10. https://doi.org/10.1145/3235765.3235803[15] Madlen Günther, Celina Kacperski, and Josef F. Krems. 2020. Can electricvehicle drivers be persuaded to eco-drive? A field study of feedback,gamification and financial rewards in Germany. Energy Research & Social
Science 63. https://doi.org/10.1016/j.erss.2019.101407[30] Ethan R. Mollick and Nancy Rothbard. 2014. Mandatory fun: Consent,
gamification and the impact of games at work. Social Science Research Network,
Rochester, NY. https://doi.org/10.2139/ssrn.2277103[31] Lennart E. Nacke and Sebastian Deterding. 2017. The maturing of gamification
research.
Computersin
Human
Behavior
71,
450-454.
https://doi.org/10.1016/j.chb.2016.11.062[32] Fiona Fui-Hoon Nah, Qing Zeng, Venkata Rajasekhar Telaprolu, Abhishek
Padmanabhuni Ayyappa, and Brenda Eschenbrenner. 2014. Gamification of
education: A review of the literature. In HCI in Business, Fiona Fui-Hoon Nah(ed.). Springer International Publishing, 401-409. Retrieved April 2, 2015 fromhttp://link.springer.com/chapter/10.1007/978-3-319-07293-7_39[16] Juho Hamari and Jonna Koivisto. 2015. Why do people use gamification
services? International Journal of Information Management 35, 4, 419-431.
https://doi.org/10.1016/j.ijinfomgt.2015.04.006[33] Daniel de Paula Porto, Fabiano C. Ferrari, and Sandra C. P. F. Fabbri. 2019.
Improving project manager decision with gamification: An experience report.
In Proceedings of the XVIII Brazilian Symposium on Software Quality (SBQS ’19),
286–295. https://doi.org/10.1145/3364641.3364675[17] Juho Hamari, Jonna Koivisto, and Harri Sarsa. 2014. Does gamification work? - A literature review of empirical studies on gamification. In Proceedings of the
47th Hawaii International Conference on System Sciences (HICSS 2014), 30253034. https://doi.org/10.1109/HICSS.2014.377[34] Tongqing Qiu, Junlan Feng, Zihui Ge, Jia Wang, Jun Xu, and Jennifer Yates.
2010. Listen to me if you can: Tracking user experience of mobile network onsocial media. In Proceedings of the 10th ACM SIGCOMM conference on Internet
measurement (IMC ’10), 288–293. https://doi.org/10.1145/1879141.1879178[18] Carrie Heeter, Brian Magerko, Ben Medler, and Yu-Hao Lee. 2011. Impacts offorced serious game play on vulnerable subgroups. International Journal of
GamingandComputer-MediatedSimulations3,
3,
34–53.
https://doi.org/10.4018/jgcms.2011070103[35] Amon Rapp, Frank Hopfgartner, Juho Hamari, Conor Linehan, and Federica
Cena. 2019. Strengthening gamification studies: Current trends and future
opportunities of gamification research. International Journal of HumanComputer Studies 127, 1-6. https://doi.org/10.1016/j.ijhcs.2018.11.007

CHI EA ’21, May 8–13, 2021, Yokohama, Japan[36] Joy Robinson, Candice Lanius, and Ryan Weber. 2018. The past, present, andfuture of UX empirical research. Communication Design Quarterly 5, 3, 10–23.
https://doi.org/10.1145/3188173.3188175[37] Richard M. Ryan and Edward L. Deci. 2000. Self-determination theory and thefacilitation of intrinsic motivation, social development, and well-being.
American Psychologist 55, 1, 68-78. https://doi.org/10.1037/0003-066X.55.1.68

Seaborn[42] Jennifer Thom, David Millen, and Joan DiMicco. 2012. Removing gamificationfrom an enterprise SNS. In Proceedings of the ACM 2012 Conference on Computer
Supported
Cooperative
Work
(CSCW
’12),
1067–1070.
https://doi.org/10.1145/2145204.2145362[38] Richard M. Ryan and Edward L. Deci. 2000. Intrinsic and extrinsic motivations:Classic definitions and new directions. Contemporary Educational Psychology
25, 1, 54-67. https://doi.org/10.1006/ceps.1999.1020[43] Gustavo F. Tondello, Rina R. Wehbe, Lisa Diamond, Marc Busch, Andrzej
Marczewski, and Lennart E. Nacke. 2016. The Gamification User Types Hexad
Scale. In Proceedings of the 2016 Annual Symposium on Computer-Human
Interaction
in
Play
(CHI
PLAY
’16),
229–243.
https://doi.org/10.1145/2967934.2968082[39] Stefan Schneegass and Niels Henze. 2014. Harvesting social media for assessinguser experience. In Proceedings of the 8th Nordic Conference on Human-Computer
Interaction:
Fun,
Fast,
Foundational(NordiChi’14),
931–934.
https://doi.org/10.1145/2639189.2670243[44] Sonya E. Van Nuland, Victoria A. Roach, Timothy D. Wilson, and Daniel J.
Belliveau. 2015. Head to head: The role of academic competition inundergraduate anatomical education. Anatomical Sciences Education 8, 5, 404412. https://doi.org/10.1002/ase.1498[40] Katie Seaborn and Deborah I. Fels. 2015. Gamification in theory and action: A
survey. International Journal of Human-Computer Studies 74, 14-31.
https://doi.org/10.1016/j.ijhcs.2014.09.006[45] Chris Welch. 2016. Nike redesigned its popular running app, and users are very
angry.
The
Verge.
Retrieved
December
15,
2020
from
https://www.theverge.com/2016/8/27/12670716/nike-running-app-badredesign[41] Katie Seaborn, Peter Pennefather, and Deborah I. Fels. 2013. Reimagining
leaderboards: Towards gamifying competency models through social gamemechanics. In Proceedings of Gamification 2013: Gameful Design, Research, andApplications(Gamification2013),
107-110.
https://doi.org/10.1145/2583008.2583026