# The Challenge to Nurture Challenge

Heinrich Söbke, Laura Weitze

► To cite this version:

Heinrich Söbke, Laura Weitze. The Challenge to Nurture Challenge. 15th International Conference on Entertainment Computing (ICEC), Sep 2016, Wien, Austria. pp.15-23, 10.1007/978-3-319-46100-7_2. hal-01640297

HAL Id: hal-01640297

https://inria.hal.science/hal-01640297

Submitted on 20 Nov 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# The Challenge to Nurture Challenge

## Students' Perception of a Commercial Quiz App as a Learning Tool

Heinrich Söbke¹ and Laura Weitze¹

¹ Bauhaus-Universität Weimar, Bauhaus-Institute for Infrastructure Solutions (b.is), Germany
{heinrich.soebke,laura.weitze}@uni-weimar.de

**Abstract.** Commercial quiz apps show characteristics, which could contribute to a promising game-based learning tool. Among these characteristics are huge popularity, easy accessibility and adaptivity to any content domain. We used the commercial quiz app QuizUp in an explorative study to clarify acceptance and requirements for the usage of these apps for educational purposes. We developed a topic-specific question corpus. Over a period of 12 days four topic affine students used QuizUp regularly and focused on learning. We observed them in three common gaming sessions. Furthermore, three interviews and a questionnaire were conducted. A main observation has been a self-reported decrease of motivation after a phase of curiosity. In general, we conclude that successful usage of such a quiz app requires a purposeful integration in an educational setting to ensure its continuous use. Additionally, we point to a set of further relevant research topics.

**Keywords:** Quiz Apps, Educational Content, Mobile Learning, Explorative Study

## 1 Introduction

Quizzes are established as an assessment tool. Not so common, but remarkable as well are their applications as learning tools [1, 2]. The quiz format can be used regardless of the technical area of the knowledge. In this context we argued for a common technical infrastructure to facilitate quizzes, especially in educational contexts [3]. Furthermore, recently raised proliferation of mobile internet has enabled the ubiquitous and enormous spread of quiz apps (e.g. *Quizkampen* [4] or *QuizUp* [5]). In a previous study [6] we found that quiz app players enter the game with an expectation to learn. Additionally, competition with friends has been identified as one main motivation for users of such apps [6]. Thus commercial quiz apps can be considered as potential educational tool.

In 2015 the commercial quiz app *QuizUp* has been opened for user-defined topics [7]. We used this feature in two engineering university courses. As a result, we found that players can be categorized in two groups: *learners* – who accomplish their quantum of educational content and then leave the app – and *gamers* – who get stuck in the game
after having fulfilled their learning duties [8]. Further results have not drawn a clear picture. On one hand the integration of the app into the course has been appreciated by students as an enrichment, on the other hand the actual usage of the app – trackable via public ranking lists – has not been overwhelming. This ambivalence provided no answer to the question, to what extent quiz apps can contribute to intrinsic motivation in learning contexts. Furthermore, more detailed insights in usage experiences of such apps within educational settings would be helpful for the purposeful design of effective application scenarios. Therefore, the aim of the following explorative study is to contribute to a set of general conditions, necessities and requirements, which could frame the usage of game-like quiz apps as educational tools.

## 2 Study design

The employed app *QuizUp* uses multiple choice questions with three distractors. Two players can compete in a match of seven questions all belonging to a certain topic. If currently there is no other player available, the system automatically assigns as opponent a bot. A question has to be answered within 10 seconds. The less time is needed for the correct answer, the more points are earned. A match takes on average three minutes.

As a prerequisite, we have developed a corpus of 57 questions about three areas of basic knowledge in the engineering topic of *Ecological Sanitation*. (The publicly available topic in QuizUp is called “NASS” – *New Alternative Sanitation Systems*). This corpus contains questions of different difficulty levels: there are questions about simple factual knowledge like abbreviations. On the other hand there are more complex questions which require the comparison of systems. Questions differ in text length, but they cannot exceed a system provided maximum length. During the study the corpus remained unchanged.

As participants 4 students (2 male, 2 female) of engineering courses of study were recruited by means of online pin boards. Prerequisites for participation were being a student, interested in quiz apps and technical topics, the possession of a smartphone and being available for a determined time frame. As a compensation a € 25 online-shopping voucher was promised. Those four persons, who responded most quickly, have been selected. Except for one person they had no prior experience with *QuizUp*. However, all of them were active players of *Quizkampen*. Although such a small sample size clearly limits the reliability of the results, we have chosen this approach to enable a familiar environment during the gaming sessions very quickly.

Within 12 days three meetings have been scheduled. One part of each of these meetings has been an attended gaming session of 30 minutes. Additionally, these meetings have been used to perform semi-structured interviews [9] with the participants. The interviews have been roughly guided by the main categories *Content*, *Motivation* and *Learning outcome*: These topics constitute main pillars of games in educational contexts. In the last session, participants had to answer a questionnaire. After session 1 and session 2 they have been provided with extra tasks for the next meeting in order to keep
up the engagement with the app over time. In the following we refer to the period be-
tween session 1 and session 2 as *home period 1* and to the period between session 2 and
3 as *home period 2*.

**Session 1.** Here we conducted a semi-structured group interview (and asked addi-
tionally for prior knowledge, expectations and experience) and a play session of 30
minutes. Participant played against each other, they connected to each other in QuizUp
and established a chat group with the purpose of coordination. A 20-page brochure
containing an overview of the topic NASS [10] was handed out to the participants. The
task for home period 1 was to play at least one match a day. Outside of the gaming
sessions, players were free to play against other participants or bots The intention of
this task was to establish regular play and to point students to a bonus which is issued
by QuizUp to reward daily play.

**Session 2.** This session took place seven days after the first session. Besides the play session, we interviewed them about their experiences since the last sessions. The task for home period 2 was to reach 25,000 points which is equivalent to 50 matches (if students play each day and are rewarded with the bonus). Intention of this instruction was on one side to investigate, how participants react to becoming the game a task instead of being fun. On the other side, we wanted to know if the habit of daily play has been kept up. This was the case, 3 of 4 participants still declared in session 3 to have played on a daily base.

**Session 3.** In this session, held five days after session 2, we conducted an interview about their experiences in the last five days. After the play session, a questionnaire has been answered.

There have been no variations of the experimental setting during the study. The only
condition we changed was the specific task to accomplish during the home periods.

**3 Results**

Sessions, interviews and questionnaire revealed various, partially very detailed as-
pects. In the following we give an overview about selected aspects. Although categori-
zation has been partly ambiguous, we assigned each aspect to one out of three main
topics: *Content* (Knowledge has to be expressed by means of multiple choice questions (MCQs)), *Motivational aspects* (Quiz apps use game mechanics, therefore fostering of motivation, especially intrinsic motivation is a relevant aspect) and *Learning outcomes* (Learning is the purpose for employing quiz apps in this context). These were the main categories of the semi-structured interviews. Subcategories have been developed a pos-
teriori according to the found results.

**3.1 Content**

After having defined three basic learning fields, the according questions corpus has
been designed due to the recommendations of Haladyna & Rodriguez [11]. Neverthe-
less, there are further observations. In the following we describe them and suggest pos-
sible improvements.
<figure><img src="image_2.png" /><figcaption>**Fig. 1.** Sample question with references to external knowledge.</figcaption></figure>

**Text length.** Questions with short texts for question stem and answer options were preferred by participants, as they do not require much reading effort. This is especially critical as there is time pressure and limited space on the display. Long questions require more time for reading and are printed in smaller letters. Both increases the felt difficulty of answering. The allowed time to answer could be adjusted programmatically – e.g. linearly according to the length of the question stem and answer options.

**Negation.** Further, although negation in question should be avoided in general [11], one participant gave an example of such a question, which he considered as very informative: *Which is not an appropriate usage of treated stormwater? Body care – Washing machine – Toilet flushing – Cleaning*. He stated that the correct answer is not a far-fetched appropriate usage, so all options have to be considered carefully, which contributes to learning.

**Level design.** Not knowing the correct answer frustrates (One participant uttered often apparently almost in desperation “How should I know that?!”). So there is the need to order questions by complexity and release them stepwise. Questions, which refer to further, not included complex knowledge, can be seen as critical. Fig. 1 depicts such a question: the question asks for a specific sanitation system, which fulfills a particular characteristic. Each answer option names such a system. If these systems are not described by other questions, the question pool is not self-contained. External knowledge is required. However, participants have not looked up additional knowledge on their own. Our conclusion is that not self-contained question pools require embedding in a formal educational setting.
## 3.2 Motivational aspects

Motivation is a key issue for a game based learning app, because motivation encourages engagement with the app and therefore causes indirectly a learning outcome. In general, the app together with its educational content is experienced as a game, as indicated by the results of the Game Experience Questionnaire (GEQ) [12]. We included the GEQ-in-game variant in the questionnaire. Among the results (see Fig. 2) are high values especially for *Positive affect* (3.63) and *Challenge* (3.13), whereas *Tension* (1.88) is rated low. Although the sample size is by far too small for reliable quantitative findings, received values seem to be game-typical, except for the quite high value of *Negative affect* (2.88).

One participant compared the regular play of the educational topic with the healthy habit of eating fruits: “Sometimes it tastes good, sometimes not”. In the following, further observed aspects are described.

**Competition.** Matches against other persons have been indicated as more interesting as matches against bots. However, in many cases a specific bot, which imitates a perfect opponent (answering all questions within the lowest time possible) has been assigned. So there was no chance to win a match. A participant stated that such a bot assignment strategy made him quitting the gaming session. Another aspect of competition is caused by ranking lists: One participant reached more than 35.000 points in the first week. This is equivalent to 150 matches or 7.5 hours of gaming. His dedicated intention was to conquer the top rank. Another player experienced a software failure, which prevented him from having a look at the ranking lists. As a consequence, he reported a decreased motivation.

<figure><img src="image_3.png" /><figcaption>**Fig. 2.** Mean value and standard deviation of GEQ categories [12] (n=4)</figcaption></figure>
**Social Presence.** In the play sessions students competed against each other and teased each other verbally. This was described by the participants as highly motivating.

**Direct encouragement.** In the first play session participants reported to be motivated by a textual app feedback after a match. However, this effect has not been mentioned in later sessions.

**Focus and engagement.** During a match the participants almost did not talk and seemed to be quite focused, especially in the first session. As an indicator for engagement, they uttered anger about false answers and cursed from time to time. Furthermore, working with the app requires attention: percentage of correct answers lowered, when there was a distraction, e.g. a switched-on TV during the home periods.

**App-internal distraction.** Besides our educational topic, QuizUp contains the original entertainment topics. In the first home period, three of four participants played entertainment topics at the same rate as technical topics. In the second home period, the number of matches about entertainment topics even increased.

**Challenge.** Especially in the third session the decreased motivation has been mentioned. Participants stated that the game has been receipted as boring, because 90 per cent of the questions could be answered from memory. Fig. 3 mirrors the decrease of motivation over time.

<figure><img src="image_4.png" /><figcaption>Fig. 3. Course of motivation (self-assessment) over study period on a scale between 0 (unmotivated and 1 (highly motivated))</figcaption></figure>

**3.3 Learning outcome**

All participants pointed out, that they have experienced learning in quiz apps and especially during this study. The following detail aspects have been found.

**Level of knowledge.** Participants stated that mostly factual knowledge can be taught with quiz apps. Simple questions, e.g. naming abbreviations, have been learned easily and could be answered in session 3 without providing answer options. Questions about more complex topics could be answered after a while. However, especially in the case of complex topics, participants admitted, that they knew the correct answer, but did not gather the meaning of the question and missed an overview of the topic.
**Learning curve.** Already in the course of the first session effects of learning could be observed, participants could provide more and more correct answers. In the beginning of session 1, mostly 1 to 2 questions of a match (seven questions) have been answered correctly. At the end of the first session, after approximately 10 matches, this rate increased to 4 to 5 questions per match.

**Intervals of usage.** One participant, which played in the home period 2 only on two days, observed fading memorization of knowledge after longer breaks. The answers could not be retrieved easily from memory, but she had to think again about the possible answers to the questions. There are two possible interpretations for this observation. First, it could be a sign of surface learning, which does not lead to long-term memorization. On the other hand, it could be a learning strategy for deeper learning, as repeated considerations for finding the correct answers could help to think thoroughly about the question.

**Situated learning.** During the play sessions, participants started discussions about missed questions and their correct answers. These face-to-face discussions contribute to learning [13].

**External knowledge sources.** Additional sources for topic-relevant knowledge have almost not been used. The established chat group did not show discussions about the technical topic. The brochure about the topic which has been handed out to participants has not been used. In the second and third session participants had not questions about the content – despite their statement, that they had not a thorough comprehension of the content of many questions. Although, students indicated that they had a better overview about the topic compared to the beginning of the study, it has to be investigated, if and in how far deeper learning processes can be triggered by a quiz app.

**Time pressure.** The limit of ten seconds to answer the questions and its relevance for the reward seems to hinder learning. Participants reported that the urge to answer fast, leads at least partly to superficial reading and to a kind of visual pattern recognition: The shape of question and answer options is recognized, but the question is not read thoroughly. However, this would be necessary for learning.

# 4 Discussion, conclusions and outlook

The most limiting restriction of this study is the low number of participants. Even for an exploratory approach four persons are quite a small sample. However, this small number of participants led to familiar atmosphere during the gaming sessions, which may have caused a more authentic behavior. Nevertheless, all findings have to be valued in consideration of this small sample size. The relative small size of the question corpus may have influenced findings as well. The grade of challenge may have not fallen that fast in the presence of a larger corpus. However, in this case the grade of frustration may have decreased the motivation of participants. Due to such a potential ambiguity, we consider the findings as hints, which require more detailed investigations.

In general, this study confirms learning effects of quiz apps. However, there are multiple aspects still to be discussed. On a more technical level, such aspects are partially
related to the limitations and features of the studied app, *QuizUp*. So, learning algorithms – as implemented in flash card systems [14, 15] – are missing. More statistical data about answered and failed questions has been indicated as potentially being motivational. The role of time in the match process needs to be balanced carefully. Direct time pressure seems to be counterproductive. Further, the foreseen reading time should be adjusted specifically to current text length and question complexity.

In a broader range there is the process of content generation: an important aspect is the kind of knowledge which can be transformed into MCQs. Complex knowledge may have to be conveyed in other ways (e.g. lectures). Questions do not provide sufficient possibilities to teach all kinds of knowledge. An enrichment through graphics and pictures would improve current possibilities. Furthermore, we have found a relatively fast decreasing motivation among participants in this study. As a countermeasure we suggest the embedding of quiz apps into educational settings, which determine e.g. level design, regular play sessions and concrete app-related goals. In general, proven settings still have to be defined. Another important aspect, which has to be discussed is the role of entertainment topics: are they a distraction from or an enrichment for the learning process? Further, there is the basic aspect of type-dependency: the grade of motivation a quiz app can foster in a person depends on their individual preferences. Relevant in an educational context is to reach a sufficient motivation for any learner. Consequentially, a quiz app has to compete with genuine question-based learning tools like Skive [16], StudyBlue [17] or Quizlet [18]. In a further step we currently conduct a comparative study between a quiz app and a dedicated learning app regarding efficacy and acceptance. Commonly, app supported learning – no matter if game-based or not – can offer the advantages of adaptivity, ubiquitous accessibility and –at least basic – collaboration. Therefore, further research should help to mitigate current limitations.

# 5 Acknowledgements

We thank Lukas Hartung, who pursued the question design and the interviews with great commitment [19]. Parts of the activities described in this article have been supported by the German Federal Ministry of Education and Research (BMBF) under the grant agreement FKZ 033W011E. The authors gratefully acknowledge this support. Moreover, we thank the anonymous reviewers of this contribution for their valuable and helpful comments.

# 6 References

1. McDaniel, M. a., Agarwal, P.K., Huelser, B.J., McDermott, K.B., Roediger, H.L.: Test-enhanced learning in a middle school science classroom: The effects of quiz frequency and placement. J. Educ. Psychol. 103, 399–414 (2011).

2. Wiklund-Hörnqvist, C., Jonsson, B., Nyberg, L.: Strengthening concept learning by repeated testing. Scand. J. Psychol. 10–16 (2013).

3. Söbke, H., Londong, J.: One Size Could Fit It All: a Common Infrastructure for Quiz Apps. In: Busch, C. (ed.) Proceedings of the 8th European Conference on
Games Based Learning. pp. 546–554. Academic Conferences and Publishing International Limited, Reading (2014).

4. FEO Media AB: QuizClash | Challenge your friends!, http://www.quizclash-game.com/.

5. Plain Vanilla: QuizUp - Connecting people through shared interests, https://www.quizup.com/.

6. Söbke, H.: Space for seriousness? Player Behavior and Motivation in Quiz Apps. In: Chorianopoulos, K. and et al. (eds.) Entertainment Computing – ICEC 2015 14th International Conference, ICEC 2015 Trondheim, Norway, Proceedings. pp. 482–489. Springer, Cham (2015).

7. Woods, B.: QuizUp launches tools for creating your own trivia categories and questions, http://thenextweb.com/apps/2015/09/24/quizup-launches-tools-for-creating-your-own-trivia-categories-and-questions/#gref.

8. Weitze, L., Söbke, H.: Quizzing to Become an Engineer - A Commercial Quiz App in Higher Education. In: Pixel (ed.) Conference Proceeding. New Perspectives in Scienze Education. 5th Edition, Florence. pp. 225–230. libreriauniversitaria.it Edizioni (2016).

9. Schmidt, C.: The Analysis of Semi-structured Interviews. In: Flick, U., Kardorff, E. von, and Steinke, I. (eds.) A Companion to Qualitative Research. pp. 253–258. SAGE Publications, London (2004).

10. DWA: Do we need New Alternative Sanitation Systems in Germany? Deutsche Vereinigung für Wasserwirtschaft, Abwasser und Abfall e. V. (DWA), Hennef, Germany (2010).

11. Haladyna, T.M., Rodriguez, M.C.: Developing and Validating Test Items. Routledge, New York (2013).

12. IJsselsteijn, W.A., De Kort, Y.A.W., Poels, K.: The Game Experience Questionnaire: Development of a self-report measure to assess the psychological impact of digital games. Manuscript in Preparation. (2013).

13. Lave, J., Wenger, E.: Situated learning: legitimate peripheral participation. Cambridge University Press (1991).

14. Leitner, S.: So lernt man lernen. Der Weg zum Erfolg. Herder, Freiburg (2011).

15. Fritz, C., Passey, D., Morris, P.: Independent Research Report: outcomes of using phase-6. , Lancaster (2009).

16. qLearning Applications GmbH: Skive, https://goskive.com/, (2016).

17. StudyBlue Inc.: Studyblue - Conquer Your Course, https://www.studyblue.com/, (2016).

18. Quizlet LLC: Simple free learning tools for students and teachers | Quizlet, http://quizlet.com/, (2016).

19. Hartung, L.: „NASSUp" Entwicklung eines Fragenkatalogs zum Thema NASS für eine kommerzielle Quiz App, Bauhaus-Universität Weimar, Germany (2016).