# Gamification of Online Surveys: Design Process, Case Study, and Evaluation

Johannes Harms, Stefan Biegler, Christoph Wimmer, Karin Kappel, Thomas Grechenig

► To cite this version:

Johannes Harms, Stefan Biegler, Christoph Wimmer, Karin Kappel, Thomas Grechenig. Gamification of Online Surveys: Design Process, Case Study, and Evaluation. 15th Human-Computer Interaction (INTERACT), Sep 2015, Bamberg, Germany. pp.219-236, 10.1007/978-3-319-22701-6_16 . hal-01599660

HAL Id: hal-01599660

https://inria.hal.science/hal-01599660

Submitted on 2 Oct 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Gamification of Online Surveys:
Design Process, Case Study, and Evaluation

Johannes Harms, Stefan Biegler, Christoph Wimmer, Karin Kappel,
Thomas Grechenig

INSO Research Group, Vienna Univ. of Technology, Austria
[firstname].[lastname] @ inso.tuwien.ac.at

**Abstract.** Online surveys are an important means of data collection in marketing and research, but conventional survey designs are often perceived as dull and unengaging, resulting in negative respondent behavior. Gamification has been proposed to make online surveys more pleasant to fill and, consequently, to improve the quality of survey results. This work applied gamification to an existing survey targeted at teenagers and young adults. The gamified survey was evaluated in a study with 60 participants regarding the psychological and behavioral outcomes of gamification. Results indicate that gamification successfully increased the users' perceived fun, the average time spent, as well as their willingness to use and recommend the survey, without introducing a strong bias in survey results, albeit with a lower overall response rate.

**Keywords.** Gamification; Online Surveys; Questionnaires; Evaluation

## 1 Introduction

Gamification of online surveys has been proposed to make questionnaire filling a more enjoyable experience and to improve the accuracy of survey results [6,9]. This is an important goal because online surveys have been criticized for their dullness resulting in negative respondent behavior such as speeding, random responding, premature termination, and lack of attention [9,17,23]. In contrast to these negative effects, evaluations of gamified surveys have reported diverse benefits regarding user experience, motivation, participation, amount and quality of data [6,8,9,23]. These prior works confirm the usefulness of gamified online surveys, but have remained unclear about suitable design processes. More recent work [13] has proposed (but not evaluated) a design process that unifies process models from the related disciplines of form design and gamification. This work employs and evaluates the process in a case study where two designers gamified a survey about sports and leisure activities amongst teenagers and young adults. The goals and contributions of this work are firstly, to document our application of the process and the resulting gamified design (Section 4). This will also provide qualitative results (Section 5) regarding the process's applicability and usefulness. And secondly, to evaluate the psychological and behavioral outcomes of the gamified design (Sections 6-8) in an empirical study.
## 2 Related Work

Gamification of online surveys builds on many disciplines [13]. The following section briefly discusses relevant backgrounds, concepts and methods.

**Tradition and Innovation in Surveys.** The use of forms for surveying information has a long historical tradition dating back to the 16th century when officers in Spanish provinces were equipped with questionnaires to standardize interviewing and observations [10]. These questionnaires enabled bureaucratic processes by abstracting individual life experiences into consistent, standardized representations [3,10]. This characteristic is shared with today's digital forms and online surveys, albeit with a different purpose of enabling automated data processing. Understanding the history of online surveys provides ample opportunity for innovation, as demonstrated by related work that has linked today's forms with their historic predecessors in order to derive research goals for form design [12]. The goal of this work can be described accordingly: Gamification of online surveys seeks to avoid negative historical entailments of the 'form' UI metaphor (in particular, the connotations that forms are bureaucratic and dull [9]) by adding interactive game elements to the survey.

**Form design.** The discipline of form design is highly relevant to survey gamification because online surveys typically employ form-based UIs to enable data entry. Related work has captured best practices for form design in guidelines [2] and books [16,28]. Relevant aspects have been structured into three layers of a form design process [16]. In the relationship layer, designers analyze the relationship with users, their tasks, and the usage context. In the conversation layer, designers seek to create interactions that make the conversations between users and the survey flow easily. The appearance layer describes detailed UI and graphical design.

<figure><img src="image_2.png" /><figcaption>**Fig. 1.** Gamification provides game elements as motivational affordances to produce psychological and behavioral outcomes [11,13]. These outcomes are influenced by a-priori factors such as context, tasks, user characteristics, and affect [11].</figcaption></figure>

**Gamification.** Gamification has been defined as “the use of design elements characteristic for games in non-game contexts” [7]. In this definition, the “design elements characteristic for games” can more shortly be termed game elements. The MDA framework [15] provides a way to understand game elements as MDAs, i.e., as either game mechanics, dynamics, or aesthetics. Mechanics describe the basic building blocks (data representations, algorithms, rules, interactive elements) that make up a game. Dynamics refer to the resulting run-time behavior over time. Aesthetics characterize a player’s emotional response and experience. The “non-game contexts” include business, education, health, many more listed in [11,14], and online surveys, as examined in this work.
**Gamified Online Surveys.** As a potential benefit, gamification provides motivational affordances that produce psychological (e.g., user experience, emotion, fun) and behavioral (e.g., participation, performance) outcomes [11]. In Figure 1, we additionally included a-priori factors such as context of use, user characteristics, and affect because these have been shown to significantly influence the outcomes of gamification [11]. Related work has aimed at exploring possible designs for gamified surveys and at evaluating their impact [6,8,9,23]. E.g., four designs have been compared in [9]: text-only, decoratively visual, functionally visual, and fully gamified. Evaluations have reported beneficial psychological outcomes such as a better user experience [8,9] and increased motivation [6]. Beneficial behavioral outcomes have included more participation and engagement [6,8], more feedback [23], and better data quality [8]. Despite these experienced benefits, not all gamified surveys have produced significantly positive results [9]. Furthermore, a recent literature review has shown benefits to be strongly influenced by users and context [11]. There is also a lack of comparisons of the required effort and subsequent benefits of specific game elements [11]. This calls for future studies to clearly describe the influence of survey domain and target user group(s), the game elements provided as motivational affordances, and the effort that was required for designing and implementing the gamified survey.

<figure><img src="image_3.png" /><figcaption>**Fig. 2.** Design process for gamifying online surveys. The process (a) unifies the disciplines of gamification and form design and applies them to the survey areas to be gamified, as originally proposed in [13]. Its iterations (b) follow the steps proposed in the MDA framework [15,26] and in the “three layers of form design” [16].</figcaption></figure>

# 3 A Design Process for Gamifying Online Surveys

To design a gamified online survey in this work, we chose to follow the process originally proposed in [13]. This process integrates and unifies the MDA (mechanics-dynamics-aesthetics) gamification framework [15,26] and the three layers of form design [16], applying them to the various survey areas, as visualized in Figure 2a. In addition to prior work, this work contributes a more detailed description of the process, complemented with examples (this Section), and a qualitative evaluation (Section 5).
**1. Game Elements for Inspiration.** Prior to starting with the gamification process, designers should collect game elements that can inspire their further design activities [13]. As a quick and easy starting point, they can use the pre-compiled catalogues of game elements suited for survey gamification from [23,27]. Further game elements (not all of them necessarily suited for surveys) are provided in the “gamification toolkit” [26], “ingredients of great games” [24], game mechanics listed in [1], “motivational game design patterns” [19], “game flow criteria” [25], “playful experiences” in [18], and the aesthetics in [15]. The designers should familiarize themselves with the game elements so they can inspire the subsequent design steps.

**2. Aesthetics and the Relationship Layer.** As a first step of the proposed process, designers should analyze the intended users (i.e., the survey's target population), tasks (the form schema to be filled), and context, as described in the relationship layer of form design [16]. Based on this knowledge, they can set goals regarding intended aesthetics, i.e., the intended emotional responses and user experiences that shall be elicited by the survey. Designers may set different aesthetic goals for different survey areas (introduction, questions, answers, navigation, and submission, compare the “survey areas” dimension in Figure 2a). Nonetheless, gamification should result in one coherent design; therefore a single process is proposed for all survey areas, see Figure 2b. Aesthetics from the previously compiled catalogue of MDAs can serve as inspiration. Designers can rank and choose aesthetics as deemed suitable.

For example, designers may consider the aesthetics of challenge and sensation to be suited for a survey's target users, but may deem the fellowship aesthetic unsuited for an intended single-user experience. Regarding the various survey areas, they could aim at arousing curiosity and interest in a survey's introduction page. They could seek to provide visual and auditory sensation to enhance questions and answers, but refrain from making questions challenging to answer because perceived intellectual difficulty has been shown to adversely influence respondent behavior [17]. They could decide to design navigation with a target aesthetic of gameful exploration. The submission page could be designed to reward users for their effort. Note that the above aesthetics are provided as illustrative examples – other target aesthetics are of course possible.

**3. Dynamics and the Conversation Layer.** Designers can use the MDA framework [15] to reason about which game dynamics are suited for producing the intended aesthetics. This creative thinking can be inspired by the catalogue of MDAs. Note that since game dynamics refer to the run-time behavior of a gamified system [15], the considerations in this step of the process correspond to the conversation layer of form design [16], i.e., the flow of interactions that a user is going to have with the survey.

For example, the game dynamic of time pressure has been recommended for motivating users to provide lengthy free-text answers [23], but designers should avoid creating time pressure throughout the entire survey because this could motivate users to speed. Designers may also implement feedback loops, i.e., dynamics wherein user actions affect the overall state of gameplay [15]. Feedback loops may visualize concepts such as a user's progress, status, wealth, health, points, etc.
**4. Mechanics and the Conversation & Appearance Layers.** To produce the intended dynamics and aesthetics, designers can employ suitable game mechanics and playful elements. Again, they can use the catalogue of MDAs for inspiration. Since game mechanics are the detailed building blocks and rules that make up a game [15], this step relates to detailed design activities in the conversation and appearance layers of form design. As an overall goal, re-designed questions should still represent the construct of interest and the interactive UI elements should not bias the answers given by respondents.

For example, designers may choose to employ the mechanics of points and badges to implement the dynamic of feedback, which in turn can produce the aesthetic of challenge. They may further choose to visualize a stopwatch next to free-text fields to implement the dynamic of time pressure and the same aesthetic of challenge. They may choose to employ the avatar mechanic and allow users to freely move their avatar throughout the survey and thus produce an aesthetic of exploration.

**5. Prototyping, Evaluation, and Iteration.** As typical for creative design processes [5,21], designers should work in a team, explore multiple designs in parallel, prototype, and evaluate prototypes. The overall gamification process will typically progress from deliberate vagueness during brainstorming, ideation, and sketching (primarily in steps 2–3) to increasing detail and specifity during prototyping and evaluation (primarily in step 4). Evaluations should consider both intended outcomes for the user (e.g., subjective experience) and outcomes for those who create the survey (e.g., completion rate, truthful answers). Formative evaluations can be performed with relatively few users, using test observation methods such as thinking-aloud [22]. In the authors’ experience, paper prototyping and digital mockups have worked well in the first iteration, whereas later iterations have required digital, interactive prototypes. Three iterations have sufficed to create a pleasant design with good usability.

**4 Case Study: Gamification of a Sports Survey**

An existing online survey about sports and leisure activities amongst teenagers and young adults<fnref n="1" /> was chosen as a case study because of its beneficial characteristics: The survey’s questions are easy to understand and answer; therefore domain-specific knowledge amongst test users is unlikely to bias evaluation results. It employs state-of-the art survey design using survey-monkey’s<fnref n="2" /> default style and functionality. Furthermore, the survey addresses children and teenagers as target population; related work has shown this target group to react well to gamification [20].

<footnote n="1">¹ http://jugendportal.at/befragung/bewegung-und-sport, Apr. 24th, 2015.</footnote>

<footnote n="2">² http://surveymonkey.com/, Apr. 24th, 2015. Survey Monkey is a popular, commercial tool for creating and conducting online surveys.</footnote>
## 4.1 Application of the Gamification Process

Methodologically, two designers (one senior with over five years in HCI and form design, one student in HCI) employed the design process presented in this work to gamify the sports survey. In summary, the designers held three workshops and took three iterations (each including prototyping and evaluation) to work through the different phases of the process, thus converting the conventionally-designed sports survey into a gamified one. In the first workshop, they discussed the *aesthetics* available in the catalogue in [27] and finally chose the three aesthetics of sensation, challenge, and exploration as suitable goals for the relationship layer of form design, i.e., for the intended relationship with teenage users. More specifically, they aimed at producing a design that elicits a rich *visual sensation* (in contrast to typical, text-only surveys), that includes small *challenges* in the form of micro-games (albeit without making questions too difficult to answer because this could potentially bias results), and that allows users to freely *explore and discover* the various survey areas. In the next workshop, they brainstormed possible designs using the catalogue of MDAs from [27] as inspiration. Their design activities iterated rapidly between explorative, abstract thinking (i.e., which dynamics and mechanics can produce the intended aesthetics) and specific, increasingly detailed design (i.e., sketching ideas and elaborating the conversation and appearance layers of form design). Regarding *dynamics*, they chose to implement feedback systems about the respondent's progress and about beneficial user actions. They further chose to implement the dynamic of time pressure to produce an aesthetic of challenge when users enter free-text answers. They sketched a design with the following *mechanics*: Users should steer an avatar through the survey. Feedback should be given using progress indicators and using coins as rewards for beneficial actions. In the third workshop, they produced mockups, thus addressing detailed UI design and the appearance layer of form design. An initial paper prototype was employed in an early, formative usability test and was subsequently replaced by a web-based prototypes and a final implementation.

## 4.2 Resulting Gamified Design

The resulting, gamified survey contains the same questions as the original sports survey, but features a novel design. The addition of many game elements resulted in a highly game-like appearance, as shown in Figures 3 and 4.

**Visual Design.** The overall *theme* of the gamified survey was designed to reflect the survey's topic of sports. The *graphical appearance* was designed to remind of jump'n'run games (such as Super Mario) that members of the target population are likely to be familiar with from their childhood. Survey elements such as input controls were graphically decorated in order to produce the intended aesthetic of sensation. For example, radio buttons were re-designed to include the respondent's avatar along with pictures that each represent one possible answer, as shown in Figure 4. All survey areas maintain a similar visual style but feature different interactions, as explained in the following subsections.
**Avatar.** In the first survey area, an avatar is automatically assigned to each respond-ent. The avatar's visual appearance depends on the demographic data that respondents provide about themselves, see Figure 3a for an example.

**Free Exploration.** The survey allows respondents to navigate freely between four sports disciplines that each represent a different survey area. Navigation is imple-mented through a map shown in the second survey area, see Figure 3b. When re-spondents click on a sports discipline, their avatar walks to the specified place on the map and the according survey area is subsequently shown. Once they complete a survey area, they return back to the map.

**Questions and Answers.** The survey areas of soccer, javelin throwing, long jump, and sprint, see Figure 3c-f, are micro-games that each afford and require different interactions through which respondents can answer questions. For example, the soccer game (Figure 3c) instructs respondents to perform a penalty kick by dragging and then releasing their avatar. When released, the avatar kicks the ball in the specified direction into the goal and thus selects one of two options. The other survey areas are designed in similar ways. The javelin throwing and long jump games (Figure 3d-e) map length (of jump or throw) unto answers. The sprint game (Figure 3f) creates time pressure by visualizing a decreasing amount of time during which respondents shall provide a maximum of free-text answers. To avoid bias through unintentionally wrong answers, each survey area provides instructions about the required interactions. Furthermore, respondents are asked to practice and then demonstrate their skill by providing a pre-specified answer before they can start answering real questions. Re-spondents can correct every answer before confirming it by clicking a “next”-button that leads to the next question.

**Feedback mechanisms.** Various mechanisms provide positive feedback about the respondents' progress. While filling the survey, they are awarded *coins*. The map allows respondents to enter a *shop* (Figure 3g) where they can buy accessories such as sunglasses and hats for their avatar. The shop has no other purpose than to strengthen positive reward. The last survey area – shown upon completion of the entire survey – was designed as a *medal ceremony* (Figure 3h) where each respondent is honored as winner of a sports competition.
<figure><img src="image_4.png" /><figcaption>**Fig. 3.** The gamified survey features multiple areas. Respondents (a) create an avatar, (b) freely navigate between survey areas, (c-f) play mini-games to answer questions, and (g) may buy accessories for their avatar in a shop using rewarded coins. Upon completion, they win the sports competition and are (h) rewarded a gold medal.</figcaption></figure>
<figure><img src="image_5.png" /><figcaption>Fig. 4. Input controls were visually decorated as shown in the above example.</figcaption></figure>

## 5 Lessons Learned about the Gamification Process

The two designers reported qualitative feedback regarding the applicability and use-
fulness of the gamification process, as experienced by them in the case study. Their
overall opinion was positive. They both found the process served them as a *helpful
guideline* about how to proceed with the gamification. This was especially important
for one of them whose background was more in user interface design than in gamifi-
cation. Both designers liked the *structure* provided by the process. They said they
could follow the steps proposed in the process and found no need to deviate from its
structure. We asked specifically if, in their experience, the structure of the process
succeeded in combining the MDA-phases (aesthetics, dynamics, mechanics) with the
layers of form design (relationship, conversation, appearance). They found that MDA-
aesthetics fit very well with the relationship layer because both relate to setting design
goals. One of them experienced the combinations of dynamics and mechanics with
the conversation and appearance layers to be rather intermingled. Regarding the *first
step of the process*, they found that using a *catalogue of MDAs* provided ideas and
useful inspiration. They often consulted it during their design activities and wished for
a more extensive catalogue. Regarding *subsequent steps*, they highlighted the need to
carefully *avoid bias*. For example, they were aware that their chosen target aesthetic
of challenge should not result in overly complex interactions that could bias answers.
They had therefore decided to make questions easy to answer but to produce the aes-
thetic of challenge by designing a narration of sports competition. In a similar way,
the designers reflected on their decision to include an avatar, expressing concerns that
users taking on foreign roles are likely to bias survey results. They had therefore per-
sonalized the avatars based on respondent characteristics, thus communicating that the
 avatar represents the actual respondent, and not a fictitious role. Within this context,
 they stressed the need for *formative evaluations* and said they had discovered and
 fixed many usability problems through formative usability testing and subsequent
design iterations. Both designers stated that iterative design and implementation of the
gamified survey took a lot of *time and effort* – more than they had anticipated, and
 significantly more than the non-gamified variant, see Table 1 for a quantitative com-
 parison. They found that – in addition to the gamification process – they could have
 used technical guidance and better development tools for their prototyping and im-
 plementation activities. They further suggested that future work should examine
 methods for reducing the implementation effort.
<table><caption>**Table 1.** Working time needed for design and implementation of the gamified vs. conventional survey. Note that since the designers gamified an already existing survey, the numbers do not include the time needed to plan and formulate survey questions.</caption><thead><tr><th>Activity</th><th>Working hours:</th><th>Gamified</th><th>Conventional</th></tr></thead><tbody><tr><td>Design & Meetings</td><td></td><td>57</td><td>1</td></tr><tr><td>Prototyping & Implementation</td><td></td><td>83</td><td>4</td></tr><tr><td>Testing</td><td></td><td>107</td><td>1</td></tr><tr><td>Total</td><td></td><td>247</td><td>6</td></tr></tbody></table>

# 6 Study Design

The gamified survey's psychological and behavioral outcomes were evaluated in a remote, comparative, between-subject usability test. Invitations were sent via Email and Facebook, asking to participate in a survey about sports activities. Since invitations could be forwarded without restriction, we had limited control over sample demographics. The invitations did not promise any extrinsic reward and did not disclose the study's true purpose of evaluating a gamified design. Participants were unaware of the existence of two different survey designs and were automatically assigned to one of two test conditions (gamified vs. conventional design) using a round-robin algorithm. After completing the survey, respondents were asked to also fill out a post-test questionnaire. Duplicate responses were prevented through technical measures, i.e., by setting a browser cookie. Respondent behavior and answers were logged into a database. The survey was completed after 60 participants, resulting in an equal distribution of 30 participants in both the conventional and gamified survey.

# 7 Results

A total of N=60 participants accessed the sports survey. The participants' age distribution was higher than the intended target audience of the original sports survey, see Table 2. A total of 47 participants completed the survey (24 female, 23 male). A test session was considered complete if users finished the sports survey, no matter if they also filled (N=40 out of 47) the subsequent post-test questionnaire. Quantitative results are shown in Tables 3-6, including means (M), medians (MD) and standard deviations (SD). Significant differences (p<0.05), as tested using Mann-Whitney U-Tests, are highlighted in a bold font. Qualitative results are reported in Section 7.4.

<table><caption>**Table 2.** Age distribution of participants</caption><thead><tr><th>What is your age?</th><th>0-14</th><th>14-17</th><th>17-21</th><th>22-24</th><th>&gt; 24</th><th>Unknown</th></tr></thead><tbody><tr><td>Gamified</td><td>0</td><td>3</td><td>0</td><td>5</td><td>13</td><td>9</td></tr><tr><td>Conventional</td><td>0</td><td>0</td><td>0</td><td>5</td><td>21</td><td>4</td></tr></tbody></table>
## 7.1 Respondent Behavior and Engagement

Respondent behavior was automatically logged during use. The gamified survey had a lower response rate of 70% (21 out of 30 persons), as opposed to the conventional survey with a response rate of 86% (26 out of 30 persons). We also measured the amount of time spent in the survey and the question where participants cancelled the survey. Amongst respondents who completed the survey, those working with the gamified design spent about twice as much time (19:20 ± 04:42) in comparison to those working with the conventional design (09:18 ± 04:39), see Table 3c. Amongst those who cancelled the survey, we found no significant differences regarding the question after which participants cancelled and the time after which they quit, as shown in Table 3a and b. Besides response rate and time spent, we took an additional measure of engagement by evaluating the amount of plain-text answers that respondents were willing to provide, but found no significant difference between the gamified and conventional survey, see Table 3d. We additionally investigated if respondent behavior was influenced by the following demographic factors: gender, age, self-rated health and sportiness, county, size of city, highest education and profession, relationship status, has children, living condition (i.e., lives with parents / friends / own family). None of these factors proved to have a statistically significant influence.

<figure><img src="image_6.png" /><figcaption>**Table 3.** Respondent behavior.</figcaption></figure>

## 7.2 Answers Given

We compared the answers given in response to the gamified versus non-gamified survey. For this purpose, all answers to the survey's 61 closed questions were numerically coded. The answers to 4 of 61 questions were significantly different between the two survey designs (Table 4); all other questions revealed no such influence. Interestingly, all four of these questions were negatively worded Likert questions, part of
large blocks of radiobuttons in the text-only survey and part of the javelin-throwing survey area in the gamified design. All four questions got higher answers (i.e., “agree more fully”) in the conventional survey. We further investigated the possibility of answers being systematically influenced by the gamified survey’s microgames. The javelin-throwing micro game did produce significantly different answers, as compared between the gamified (2.50 ± 1.072) vs. conventional (2.65 ± 1.130) design. There were no significant differences any of the other micro-games.

<figure><img src="image_7.png" /><figcaption>**Table 4.** Answers given. Amongst the survey's 61 closed questions, answers to the above four questions were significantly influenced by the survey's gamified vs. conventional design. All other questions showed no such influence.</figcaption></figure>

## 7.3 Self-Rated User Experience

Perceived usability and user experience were assessed upon completion of the sports survey through a post-test questionnaire that included System Usability Scale (SUS) [4] questions. It was filled out by an overall number of 40 respondents (21 gamified, 19 conventional). Pair-wise comparison of individual SUS questions (see Table 5) revealed that respondents were significantly more inclined to frequently use the gamified survey (2.81 ± 0.75), compared to the conventional version (1.16 ± 1.02). However, respondents felt significantly less confident using the gamified survey (3.1 ± 0.7), compared to the conventional version (3.79 ± 0.42). There was no significant effect of survey design on any other SUS question. Overall SUS scores for both survey versions were comparable as well, with the gamified survey scoring 77.98 points and the conventional survey scoring 79.08. Answers to further questions in the post-test questionnaire (Table 6) showed that respondents found the gamified survey (3.29 ± 0.56) significantly more *fun to use* than the conventional survey (2.32 ± 1.01). They were also significantly more inclined to *recommend* the gamified survey (3.38 ± 0.67), compared to the conventional survey (2.42 ± 0.96).
<figure><img src="image_8.png" /><figcaption>**Table 5.** System Usability Scale (SUS) scores from the post-test questionnaire.</figcaption></figure>
<figure><img src="image_9.png" /><figcaption>**Table 6.** Self-rated fun and likelihood of recommending the survey.</figcaption></figure>

## 7.4 Qualitative Results

Qualitative comments were collected from respondents using open-ended questions in the post-test questionnaire. The comments were analyzed and grouped into structured categories (see Table 7). Of 21 respondents that finished the gamified survey, every single one answered the post-test questionnaire, while 19 of 26 respondents that finished the conventional survey answered the post-test questionnaire. Respondents were also much more inclined to provide comments (both positive and negative) for the gamified survey (N=21 of 21) compared to the conventional survey (N=9 of 26).

Respondents positively commented on the novelty (9), variety (4) and interactivity (2) of the gamified survey. They found it playful (4) and fun (3). Graphics and animation also garnered positive comments (5), as well as the personalized and customizable avatar (4). Some respondents complained that the gamified survey took much longer to answer than a conventional survey might have taken (4). There also were complaints about the controls (4) and responsiveness of individual mini games (3). Several respondents also commented that they would have liked to continue playing after finishing the survey (4), which is an interesting complaint insofar as it highlights the heightened level of engagement and joy compared to a conventional survey.

Comments regarding the conventional survey were less varied: Respondents found the survey easy to use (5) and easy to answer (3), while complaining about vague or ambiguous questions (4) and boredom (3).
<table><caption>**Table 7.** Qualitative results. The table shows answers given to open-ended questions in the post-test questionnaire, structured into coded categories.</caption><thead><tr><th><strong>Gamified: Positive comments</strong></th><th><strong>N</strong></th><th><strong>Negative comments</strong></th><th><strong>N</strong></th></tr></thead><tbody><tr><td>Novelty</td><td>9</td><td>Duration</td><td>4</td></tr><tr><td>Graphics &amp; animation</td><td>5</td><td>Inability to continue playing after survey</td><td>4</td></tr><tr><td>Playfulness</td><td>4</td><td>Controls</td><td>4</td></tr><tr><td>Rich in variety</td><td>4</td><td>Responsiveness of individual games</td><td>3</td></tr><tr><td>Customizable avatar</td><td>4</td><td>Complexity</td><td>1</td></tr><tr><td>Fun</td><td>3</td><td>Sound</td><td>1</td></tr><tr><td>Interactivity</td><td>2</td><td>Amount of textual instructions</td><td>1</td></tr><tr><td>Ease of use</td><td>1</td><td></td><td></td></tr><tr><td>Anonymity</td><td>1</td><td></td><td></td></tr><tr><td>Suitability for children</td><td>1</td><td></td><td></td></tr><tr><td>No comment</td><td>0</td><td>No comment</td><td>6</td></tr></tbody><thead><tr><th><strong>Conventional: Positive comments</strong></th><th><strong>N</strong></th><th><strong>Negative comments</strong></th><th><strong>N</strong></th></tr></thead><tbody><tr><td>Ease of use</td><td>5</td><td>Vague or ambiguous questions</td><td>4</td></tr><tr><td>Clarity and ease of answering</td><td>3</td><td>Boring</td><td>3</td></tr><tr><td>Broad theme</td><td>1</td><td>Missing progress indicator</td><td>1</td></tr><tr><td>No comment</td><td>12</td><td>No comment</td><td>12</td></tr></tbody></table>

## 8 Discussion

Results indicate that gamification successfully increased the users' perceived fun, the average time spent, as well as their willingness to use and recommend the survey, without introducing a strong bias in the survey results, albeit with a lower overall response rate. This improvement in user experience is in line with related studies on gamified online surveys [6,9].

Quantitative results show that respondents found the gamified survey more fun and spent significantly more time. While an increase in time spent is in itself not necessarily a sign of heightened engagement, the fact that participants also found the gamified survey more fun and voluntarily spent more time suggests that the increased duration is the result of an improved user experience. This may prove beneficial for marketing surveys that aim at exposing users to a certain brand in a pleasant way. Furthermore, the respondents' higher willingness to use and recommend the gamified survey can be useful for viral marketing.

Given these positive outcomes, the lower overall response rate for the gamified survey was surprising and warrants further examination. Results provide two possible explanations. Firstly, higher engagement and positive feedback from those who did finish the gamified survey suggest that the gamified design may have caused polarized reactions among participants, turning those away who did not approve of the chosen design. This issue requires further examination in future work. Secondly, respondents stated they felt less confident using the gamified survey; the lack of familiarity with the novel design may have caused them to cancel. Future long-term studies are needed to investigate effects of novelty in gamified online surveys.

Overall, the gamified design barely influenced answers given by participants, as there was no significant effect of survey style for 57 of 61 questions. However, there was a significant difference for answers given in the javelin-throwing micro game that
requires further examination. One interpretation is that the gamified survey success-
fully reduced negative respondent behavior and thus reduced bias – but the gamifica-
tion may also have introduced new bias. To clarify the issue, we suggest that future
work should develop automated measures of speeding, straightlining, random re-
sponding, lack of attention, conflicting and empty answers, compare [9,17,23], and
use these measures to quantify negative respondent behavior in gamified and conven-
tional surveys.

The *qualitative comments* given by respondents reaffirm our initial expectations and motivation: That conventional surveys are often perceived as somewhat dull and boring, and that gamification is a suitable approach to make surveys more fun and engaging. Some of the comments validate specific design decisions made during the gamification process, such as the use of a customizable avatar to represent survey respondents, as well as implementation details such as graphics and animation. However, other comments demonstrate the difficulty of getting every design detail right, as demonstrated by scattered complaints about controls or the responsiveness of individual micro-games. Additionally, the abrupt ending of the gamified survey drew a large number of complaints of users who would have liked to continue playing. While an abrupt ending might be appropriate for a conventional survey, it seems inappropriate for more playful, open-ended experiences, as in our gamified survey. Comparing the comments between the gamified versus conventional survey, it becomes apparent that the gamified survey garnered both a larger number as well as more varied comments. One possible explanation for this difference in quantity and quality of respondent's comments would be the novelty of the gamified survey raising awareness of specific survey design aspects, compared to the dull familiarity of a conventional survey spurring less reflection and comment.

9 Conclusion and Future Work

Gamification is a promising way of improving user experience and increasing en-
gagement in online surveys. This work extends prior research by making the follow-
ing two contributions.

Firstly, this work documents the successful application of a recently proposed de-
sign process for gamifying online surveys and describes the resulting design. The
process was applied in a case study where two designers gamified a survey about
sports and leisure activities amongst teenagers and young adults. The designers re-
ported qualitative results supporting the practical usefulness and applicability of the
process. This indicates that other survey gamification projects can benefit from the
same or a similar process.

As a second contribution, the resulting gamified design was evaluated in a remote
online study with 60 participants. The gamified survey achieved better psychological
outcomes (respondents found the gamified survey more fun, they were more inclined
to use and recommend the gamified design, and provided more, and more positive,
qualitative feedback) and better behavioral outcomes (regarding engagement: re-
spondents voluntarily spent more time in the gamified survey). These positive results
are, however, accompanied by critical issues including a lower response rate in the
gamified survey and possibly biased answers in one specific survey area. These issues
warrant further empirical investigation.

Our future work in this area will continue in the following direction. Since a sur-
vey's gamification takes a lot of effort, we intend to examine ways of increasing ben-
efits (e.g., by identifying best practices and by seeking ways of improving behavioral
outcomes) and of reducing the required efforts (e.g., by creating re-usable design
patterns and component libraries) in order to improve the return on investment of
future survey gamifications.

References

1. Aparicio, A.F., Vela, F.L.G., Sánchez, J.L.G., Montes, J.L.I.: Analysis and application of gamification. In: Proceedings of the 13th International Conference on Interacción Persona-Ordenador. pp. 17:1–17:2. INTERACCION ’12, ACM, New York, NY, USA (2012)

2. Bargas-Avila, J., Brenzikofer, O., Roth, S., Tuch, A., Orsini, S., Opwis, K.: Simple but crucial user interfaces in the world wide web: Introducing 20 guidelines for usable web form design. In: Matrai, R. (ed.) User Interfaces, chap. 1. InTech (2010)

3. Becker, P.: Formulare als "Fließband" der Verwaltung? Zur Rationalisierung und Standardisierung von Kommunikationsbeziehungen. In: Collin, P., Lutterbeck, K.G. (eds.) Eine intelligente Maschine? Handlungsorientierungen moderner Verwaltung (19./20. Jh.). Nemos, Baden-Baden (2009)

4. Brooke, J.: SUS-a quick and dirty usability scale. Usability evaluation in industry 189, 194 (1996)

5. Buxton, B.: Sketching user experiences: getting the design right and the right design. Morgan Kaufmann Pub (2007)

6. Cechanowicz, J., Gutwin, C., Brownell, B., Goodfellow, L.: Effects of gamification on participation and data quality in a real-world market research domain. In: Gamification 2013: Proceedings of the First International Conference on Gameful Design, Research, and Applications. pp. 66–73. ACM, Stratford, Ontario, Canada (2013)

7. Deterding, S., Dixon, D., Khaled, R., Nacke, L.: From game design elements to gamefulness: Defining "gamification". In: Proceedings of the 15th International Academic Mind-Trek Conference: Envisioning Future Media Environments. pp. 9–15. MindTrek ’11, ACM, New York, NY, USA (2011)

8. Dolnicar, S., Grün, B., Yanamandram, V.: Dynamic, interactive survey questions can increase survey data quality. Journal of Travel & Tourism Marketing 30(7), 690–699 (2013)

9. Downes - Le Guin, T., Baker, R., Mechling, J., Ruyle, E.: Myths and realities of respondent engagement in online surveys. International Journal of Market Research 54(5), 1–21 (2012)

10. Eisermann, F.: Zu den Anfängen des gedruckten Formulars (Gastbeitrag). In: Formulare. Von der Wiege bis zu Bahre...: Formulare im Corporate Design. Stiebner Verlag GmbH (2007)

11. Hamari, J., Koivisto, J., Sarsa, H.: Does gamification work? – a literature review of empirical studies on gamification. In: Proceedings of the 47th Hawaii International Conference on System Sciences (2014)

12. Harms, J.: Research goals for evolving the ‘form’ user interface metaphor towards more interactivity. In: Holzinger, A., Ziefle, M., Hitz, M., Debevc, M. (eds.) Human Factors in
Computing and Informatics, Lecture Notes in Computer Science, vol. 7946, pp. 819–822.
Springer Berlin Heidelberg (2013)

13. Harms, J., Wimmer, C., Kappel, K., Grechenig, T.: Gamification of online surveys: conceptual foundations and a design process based on the MDA framework. In: Proceedings of the 8th Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational. pp. 565–568. ACM (2014)

14. Hsu, S.H., Chang, J.W., Lee, C.C.: Designing attractive gamification features for collaborative storytelling websites. Cyberpsychology, Behavior, and Social Networking 16(6), 428–435 (Jun 2013)

15. Hunicke, R., LeBlanc, M., Zubek, R.: MDA: A formal approach to game design and game research. In: Proceedings of the AAAI-04 Workshop on Challenges in Game AI. pp. 1–5 (2004)

16. Jarrett, C., Gaffney, G.: Forms that work: designing web forms for usability. Morgan Kaufmann (2008)

17. Kaminska, O., McCutcheon, A.L., Billiet, J.: Satisficing among reluctant respondents in a cross-national context. Public Opinion Quarterly 74(5), 956–984 (2010)

18. Korhonen, H., Montola, M., Arrasvuori, J.: Understanding playful user experience through digital games. In: International Conference on Designing Pleasurable Products and Interfaces. pp. 274–285 (2009)

19. Lewis, C., Wardrip-Fruin, N., Whitehead, J.: Motivational game design patterns of ‘ville games. In: Proceedings of the International Conference on the Foundations of Digital Games. pp. 172–179. FDG ‘12, ACM, New York, NY, USA (2012)

20. Mavletova, A.: Web surveys among children and adolescents: Is there a gamification effect? Social Science Computer Review (2014)

21. Mayhew, D.J.: The Usability engineering lifecycle: A practitioner’s handbook for user interface design. Morgan Kaufmann (1999)

22. Nielsen, J.: Usability engineering. Morgan Kaufmann (1993)

23. Puleston, J.: Online research–game on!: A look at how gaming techniques can transform your online research. In: Shifting the Boundaries of Research. Proceedings of the 6th ASC (Association for Survey Computing) International Conference. pp. 20–50 (2011)

24. Reeves, B., Read, J.L.: Total Engagement: Using Games and Virtual Worlds to Change the Way People Work and Businesses Compete. Harvard Business Review Press, 1st edn. (2009)

25. Sweetser, P., Wyeth, P.: Gameflow: A model for evaluating player enjoyment in games. Computers in Entertainment 3(3), 3–3 (Jul 2005)

26. Werbach, K., Hunter, D.: For the Win: How Game Thinking Can Revolutionize Your Business. Wharton Digital Press (2012)

27. Wlaschits, B.: Gamifizierung von elektronischen Umfrageformularen: Design und Experten-Evaluierung. Master’s thesis, supervised by Thomas Grechenig and Johannes Harms (2014)

28. Wroblewski, L.: Web Form Design. Filling the Blanks. Louis Rosenfeld (2008)