# Improving Student Experience and Learning Performance with Traditional Instructional Methods and New Digital Media

Pamela Andrade, Effie L.-C. Law

► To cite this version:

Pamela Andrade, Effie L.-C. Law. Improving Student Experience and Learning Performance with Traditional Instructional Methods and New Digital Media. 18th IFIP Conference on Human-Computer Interaction (INTERACT), Aug 2021, Bari, Italy. pp.97-106, 10.1007/978-3-030-85613-7_7. hal-04292381

HAL Id: hal-04292381

https://inria.hal.science/hal-04292381v1

Submitted on 17 Nov 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
Improving Student Experience and Learning
Performance with Traditional Instructional
Methods and New Digital Media

Pamela Andrade<fnref n="1" />[0000-0002-2457-4848] and Effie L-C. Law<fnref n="2" />[0000-0002-0873-0150]

**Abstract.** Studies in Technology-enhanced learning (TEL) involve dif-
ferent instructional media and methods to understand their effects on
two main outcomes: learning performance and user experience. This pa-
per investigated the impact of three gamification elements (point, badge,
leaderboard) integrated into an educational software application on these
outcomes. To expand the scope of the related work focusing on the Higher
Education learners, our study targeted secondary school students. Re-
sults from 74 participants, who were divided into four groups (one non-
gamified and three gamified), showed that the group receiving badges
had a more positive user experience and that learning gain was observed
in all groups.

**Keywords:** Gamification · Education · User Experience · Learning Performance · Secondary Schools · Points · Badges · Leaderboards

# 1 Introduction

Several well-established pedagogical practices have been implemented in edu-
cational software in recent years to enable positive user experience (UX) and
learning gain. Among others, pedagogical theories such as Flipped Classroom,
Inquiry-Based Learning, and Learning by Questioning have shown to be bene-
ficial to improve learning performance. Based on the findings of our literature
review, we decided to ground this study in the *Learning by Questioning* ap-
proach. However, a serious drawback of using any of such powerful pedagogical
strategies for the design and development of educational software is that they do
not sustain student attention, as they can be sometimes perceived by students
as difficult or tiresome. An unfortunate consequence is that many students lose
interest in completing a task, or altogether quit their interaction with the ed-
ucational software, jeopardizing their opportunity to learn. To counteract this
possible effect, we suggest the implementation of engagement strategies to main-
tain student interest in using the educational software. In particular, we study
which specific gamification elements can potentiate UX and learning gain.

According to the related literature in Human-Computer Interaction (HCI),
further research in this specific topic could help improve the understanding of

<footnote n="1">¹ University of Leicester, United Kingdom pyas2@le.ac.uk</footnote>

<footnote n="2">² University of Leicester, United Kingdom lcl9@le.ac.uk</footnote>

<footnote>https://le.ac.uk/informatics</footnote>
which gamification elements would better enhance user perceptions of the inter-
active software. Likewise, concerning TEL, the related literature attests the need
to select appropriate pedagogical methods if the objective is to boost learning
gain. Therefore, based on the findings identified in previous studies as described
in the sections below (see Section 2), an important contribution of this research
is to compare individual user interface (UI) design elements in an educational
application, with the goal to analyse which of these elements make a significant
difference on the students' overall experience and learning performance. In ad-
dition, this study aims to investigate student perceptions of the instructional
media selected for the purposes of this research, to analyse how this selection
influence the learning experience of students in secondary schools.

Our reason for choosing this particular target group is simple: plenty has
been studied in the field of gamified educational technologies, but most of the
research has focused on the effects in Higher or Distance Education only [8] [15].
Another weakness identified in previous research is the limited breadth of con-
texts in which instructional media and methods have been studied, with the
most common topics being different from those parts of secondary school curric-
ula. Therefore, our work contributes to the understanding of how instructional
media and methods can improve the learning experience and performance of
adolescent students, by using three gamification elements to teach physics as
part of an online lesson.

## 2 Related Work

The activity planned for this study (Section 3.2) used attractive instructional
media (Section 2.1) to analyse how this makes an impact on student performance
and UX. On the other hand, the software developed for this research (Section 3.1)
used engaging instructional methods (Section 2.2) with the same aim.

### 2.1 Instructional Media

An important reason why similar research has studied the effects of instructional
media in TEL, is due to the expectation that its implementation could help
improve learning performance and user experience [4]. In general, instructional
*media* refers to materials and devices used to deliver instructions online [13].

Numerous scientific websites and resources are nowadays available on the
Internet, giving the web the potential to become a favourable environment for
complex learning [6]. In this context, multimedia tools can help students to
make the transition from a passive participation to a more responsible and self-
aware instruction. Although self-exploration and experimentation is encouraged,
online resources still need to be carefully selected and organized by teachers so
students do not waste cognitive efforts of trying to find or figure out which
ones fit best for a particular topic. Additionally, instructional media and virtual
interactions are being used in modern times to encourage students to use logical
reasoning to gain a deeper understanding of a topic. In this particular study,
the design of the learning activity (Section 3.2) included multimedia such as
illustrations, animations, and interactive experiments to encourage students to
focus on learning through instructional media.

Nonetheless, although some studies have shown that multimedia instruction is an effective method to complement science lessons [12] (in some cases even exceeding the results from many traditional pedagogical strategies), sometimes using multimedia by itself does not guarantee improved learning performance and real understanding [9] [13]. A combination of online scientific resources and other procedures (e.g. opinion exchange, group work, interactive exercises) could help to better balance critical thinking and collaborative knowledge formation.

## 2.2 Instructional Methods

Instructional *methods* refer to the techniques used to enhance cognitive processing in learners [13], which can include many types of pedagogical strategies that teachers—or software applications in this case—implement in the classroom to help students to learn the course material.

Although recent studies have pointed out the necessity of better instructional *methods* rather than focusing on instructional *media* to foster learning gain [13], findings remain inconclusive about the benefits with regard to UX. Hence, we have been motivated to explore to what extent using *Learning by Questioning* and *Gamification*—which are traditional instructional methods—can utilize the potential of the new instructional media—interactive educational software—to encourage positive experience and learning gain.

Gamification aims to create more enjoyable experiences by adding a recreational element to the interaction between students and technology [7]. In consequence, gamified software support and motivate students to perform tasks (such as questioning) promoted by the engaging nature of a gameful experience. Previous research suggests that game design elements at a surface level can be more easily manipulated independently of one another [15] [2]. Thus, badges, points, and leaderboards were selected for the software development (Section 3.1) to address the limited experimental analyses of individual game design elements and comparisons of groups thereof [11] [15].

# 3 Design of the Experiment

## 3.1 Software Development

To study how instructional methods (Section 2.2) can influence UX and learning performance, three UI design elements were implemented in Go-Lab<fnref n="3" />. The gamification elements badges, points, and leaderboards were implemented as plugins or *apps* that can be selected by teachers to add to their online lessons. The teacher view allowed the user to assign or revoke rewards by selecting students

<footnote n="3">³ Go-Lab (https://golabz.eu) is an educational platform that enables the creation and distribution of interactive online lessons.</footnote>
in a class from a dropdown menu (Fig. 3). Teachers were also able to configure
various features in the app, including a selection of prizes and languages (see the
gear icon at the bottom right corner of Fig. 3). The student view was restricted
to visualizing the learning activity and any available rewards (Fig. 1 and Fig. 2).

<figure><img src="image_3.png" /><figcaption>**Fig. 1.** Student View of Leaderboards</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>**Fig. 2.** Student View of Badges & Points</figcaption></figure>

3.2 The Learning Activity

<figure><img src="image_5.png" /><figcaption>Fig. 3. Teacher View of the App</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>Fig. 4. Initial View of the Online lesson</figcaption></figure>

Following the work of [1], an online lesson aligned with the appropriate curric-
ula was selected from the Go-Lab repository. The online lesson was designed with
various types of instructional media (Section 2.1) and covered the physics topic
of *Electric Circuits*. Moving along the phases of the learning activity (see the
side navigation menu, Fig. 4) students learned about current, tension, power, se-
rial and parallel circuits, and Ohm's law. The lesson was assembled with textual
information and several pieces of multimedia content. Also, interactive experi-
ments were available for students to practice the concepts they had learned.

# 4 Methodology

This empirical study was approved by the Research Ethics Committee of the
University of Leicester. Participants worked individually during this experiment,
which took place with the school's facilities for two sessions. A researcher and
a science teacher were present during the experiment to monitor the process,
but did not offer any unsolicited help. There were four software versions under
evaluation in this study (non-gamified [NG], gamified with points [GP], gamified
with badges [GB], and gamified with leaderboards [GL]).

**4.1 Hypotheses and Instruments**

Two main **null hypotheses** were formulated for this study:

H1: There are no significant differences in the learning performance of secondary school students among the NG, GP, GB, and GL groups.

H2: There are no significant differences in the students' experience in terms of their perceptions of software usability, user engagement and motivation among the NG, GP, GB, and GL groups.

Several data collection methods and instruments were used to test the null hy-
potheses. To analyse **learning performance** identical pre- and post-knowledge
tests were designed with questions closely related to the physics lesson part of the
experimental session (Section 3.2). The homegrown knowledge tests comprised
12 multiple-choice questions, with a single possible correct answer. To analyse
**user experience** various standardised scales measuring student perceptions of
software usability, user engagement, and motivation were used: the System Us-
ability Scale SUS [3], the User Engagement Scale Short-Form UES-SF (5-point
Likert Scale) [14], and the Situational Motivation Scale SIMS (7-point Likert
Scale) [10]. These standardised scales were selected due to their proven psycho-
metric properties and their ease of administration. Additionally, a demographic
questionnaire was used and a group discussion was held to provide insights into
the habits, preferences, and background of participants in this sample. Note that
SUS and UES-SF (subscale – usability) were used to assess the reliability of the
participants’ responses.

**4.2 Procedure and Participants**

A public school in the Basque Country in the northwest of Spain was involved in
this study. A public school in Spain refers to education that is funded by the state
and is free of charge. Students had access to good scientific and technological
facilities in the school. A total of 74 students participated in the experiment.
Each class was allocated to one of the four groups (NG, GL, GP, GB) ran-
domly. The first of the two sessions started with students filling in a demographic
questionnaire, after which students interacted with the same online physics les-
son (Section 3.2). Then, participants were asked to: complete a knowledge test,
create questions based on the lesson’s content, and complete the SIMS question-
naire. Following a schedule set by the participants’ school, students had a 24–48
hours break. Over the break, a researcher and the teacher of each participating
class evaluated the quality of the questions created by students to award them
prizes (points, badges, leaderboards) accordingly.

During the second session, participants were presented with results of their ef-
forts in the form of the gamification element under evaluation in their respective
groups (non-gamified interfaces showed no gamification element). Afterwards,
participants had the opportunity to create a second set of questions and they
completed once again the SIMS questionnaire and the knowledge test. All stu-
dents then filled in the SUS and UES-SF. To conclude the session, each class
gave feedback through a moderated discussion.

5 Results

In total, six data collection methods and instruments were used in this experi-
ment to test our two null hypotheses (see Section 4.1). The main results of this
analysis are discussed in the sections below.

5.1 Demographic data

The average age of the 74 students participating in this study was 14.56 (SD =
0.94); 36 participants were male, 33 were female, and 5 people preferred not to
specify their gender. Participants rated their IT skills on a scale 1–10 with a
mean score of 7.41 (SD = 1.22).

5.2 Learning Performance

Data were normally distributed in all groups for the scores of the pre-test and the post-test used in this study (Table 1). Parametric tests were therefore used to investigate the null hypothesis H1. The Cronbach alpha for the knowledge tests was 0.771, implying that the reliability of the knowledge test was acceptable.

Students achieved a pre-test mean of 7.05 (SD = 1.86) out of 12 possible
points and a mean of 8.12 (SD = 1.99) in the post-test. Significant differences
were found in the learning performance over all the four groups using a paired
sample t-test (t(73) = -7.028, p < 0.000)).

When comparing the two knowledge tests, a significant improvement was
found on the knowledge tests scores of the non-gamified group [NG] (t(20) =
-2.014, p = 0.009)). Likewise, students receiving points [GP], badges [GB], and
leaderboards [GL] showed significant learning gains (GP[t(17) = -3.332, p =
0.004]), GB[t(17) = -4.177, p = 0.001]), GL[t(16) = -3.801, p = 0.002]); all
<table><caption>**Table 1.** Results of the Knowledge Tests Classified per Group ([N]= Sample Size, [A]= Average, [SD]= Standard Deviation, [Dif.]= Difference)</caption><thead><tr><th>Group</th><th>N</th><th>Pre-Test(A)</th><th>SD</th><th>Post-Test(A)</th><th>SD</th><th>Dif.</th></tr></thead><tbody><tr><td>NG</td><td>21</td><td>8.00</td><td>2.00</td><td>8.62</td><td>2.16</td><td>+0.62</td></tr><tr><td>GL</td><td>17</td><td>6.82</td><td>1.56</td><td>8.12</td><td>1.69</td><td>+1.30</td></tr><tr><td>GB</td><td>18</td><td>7.11</td><td>1.94</td><td>8.67</td><td>2.08</td><td>+1.56</td></tr><tr><td>GP</td><td>18</td><td>6.11</td><td>1.41</td><td>7.00</td><td>1.57</td><td>+0.89</td></tr></tbody></table>

with a medium to large effect size (GP[r = 0.29, Cohen's d = 0.596], GB[r = 0.36, Cohen's d = 0.776], GL[r = 0.37, Cohen's d = 0.799]). Given this statistical analysis, the **null hypothesis H1** was **rejected**. Significant differences were found in the learning performance between the two knowledge tests.

## 5.3 User Experience

Three standardised questionnaires were used to assess UX in this study. Data were normally distributed in all scores of the SUS, UES-SF, and SIMS (Table 2). Parametric tests were therefore used to investigate the null hypothesis H2. Note that not all students answered all the questionnaires. Cronbach alphas for the three instruments are 0.769, 0.676 and 0.505, respectively. The range of 0.7 to 0.8 was acceptable, but 0.5 was rather poor; it implied that SIMS might not be the best instrument to be used in this context.

<table><caption>**Table 2.** Quantitative UX Results Classified per Group ([N]= Sample Size, [A]= Average, [SD]= Standard Deviation, [Dif.]= Difference)</caption><thead><tr><th>Group</th><th>N</th><th>SUS(A)</th><th>SD</th><th>N UES-SF(A)</th><th>SD</th><th>N SIMS1(A)</th><th>SIMS2(A)</th><th>Dif.</th></tr></thead><tbody><tr><td>NG</td><td>21</td><td>61.67</td><td>15.15</td><td>21</td><td>3.01</td><td>0.66</td><td>18</td><td>4.55</td><td>4.67 +0.12</td></tr><tr><td>GL</td><td>16</td><td>66.00</td><td>10.51</td><td>16</td><td>2.82</td><td>0.31</td><td>15</td><td>4.17</td><td>4.22 +0.05</td></tr><tr><td>GB</td><td>18</td><td>67.36</td><td>22.04</td><td>18</td><td>3.37</td><td>0.72</td><td>18</td><td>4.10</td><td>4.12 +0.02</td></tr><tr><td>GP</td><td>17</td><td>61.33</td><td>9.90</td><td>18</td><td>3.25</td><td>0.34</td><td>15</td><td>4.00</td><td>4.04 +0.04</td></tr></tbody></table>

A one-way ANOVA showed non-significant differences among the groups ($F(3, 68) = 1.240, p = 0.302$) on the student perception of software usability (SUS). Likewise, a paired sample t test showed non-significant differences between the students' motivation before and after the intervention as measured by the SIMS questionnaire ($t(65) = -1.001, p = 0.320$). Nevertheless, a one-way ANOVA revealed significant differences among the groups on the self-reported perception of user engagement as measured by the UES-SF ($F(3, 71) = 1.582, p = 0.039$). Results of Tukey post-hoc tests revealed that the badges [GB] showed a significant difference on the UES-SF scores compared to the non-gamified groups [NG]($p = 0.032$) with a medium effect size ($r = 0.25$, Cohen's $d = 0.521$). In addition, a Pearson correlation coefficient was computed with the SUS and
UES-SF results to assess the reliability of the participants' responses regarding usability and a positive correlation was found between the two variables ($r = 0.625, n = 72, p < 0.001$). Given these results, the **null hypothesis H2** was **partially rejected**.

# 6 Discussion

Contrary to the findings of several similar studies [15] [16], results of this experiment supported those of [1] [5], which found that badges were the gamification element with the most significant influence on user experience and learning gains.

However, the empirical results of this study suggested that the design of the gamified elements could have been improved. Firstly, one should ensure the uniformity of the assessment criteria for determining the rewards to be granted. Secondly, by enhancing the design of the gamification elements to appeal visually to the students, using appropriate positioning, size, and animation for this effect.

To trigger a comparison of different instructional media and methods that could influence learning performance and UX, this study expanded on the work of [1] by selecting students from the same age group, following the same methodological procedure, but from a different background (i.e. country and native language). Hence, future work should involve a cross-cultural analysis of these results with the aim to identify differences and similarities between behavioural patterns of students in varied socio-cultural and educational settings (e.g., private vs public education, developed vs developing countries, etc.).

Although our intention was to compare the benefits and drawbacks of instructional media and instructional methods, it is possible that the results of this study could have been influenced by external validity threats such as the *multiple treatment interference* as the same subjects were using a combination of multimedia content, gamified strategies, and questioning-based software, making it difficult to discern the effects of each in this particular sample. Also, by the different waiting time students had between sessions, which was decided entirely by the participating school depending on their available timings. Furthermore, the low reliability of SIMS suggested that an alternative instrument could have been used, and it might explain the non-significant difference in motivation between the two sessions.

# 7 Conclusion

Granted the appropriate selection of instructional media, we argue that the use of instructional methods is crucial for positively influencing learning performance and UX. Results of this study suggested that both *media* and *methods* could positively influence learning performance, whereas only instructional *methods* had an impact on (the quantitative analysis of) UX.

Further research is necessary to understand the contextual characteristics that could lead to these effects. Analysing implications from cross-cultural data on a larger scale would be the first step.
References

1. Andrade, P., Law, E.L.C., Farah, J.C., Gillet, D.: Evaluating the effects of introducing three gamification elements in stem educational software for secondary schools. In: 32nd Australian Conference on Human-Computer Interaction. pp. 220–232 (2020)

2. Bedwell, W.L., Pavlas, D., Heyne, K., Lazzara, E.H., Salas, E.: Toward a taxonomy linking game attributes to learning: An empirical study. Simulation & Gaming **43**(6), 729–760 (2012)

3. Brooke, J.: Sus: A 'quick and dirty' usability scale. In: Jordan, P.W., Thomas, B., McClelland, I.L., Weerdmeester, B.A. (eds.) Usability Evaluation in Industry, chap. 21, pp. 189–194. Taylor & Francis, London, UK (1996)

4. Clark, R.E.: Learning from media: Arguments, analysis, and evidence. IAP (2001)

5. Denny, P.: The effect of virtual achievements on student engagement. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. pp. 763–772. ACM, New York, NY, USA (2013)

6. DeSchryver, M., Spiro, R.J.: New forms of deep learning on the web: Meeting the challenge of cognitive load in conditions of unfettered exploration in online multimedia environments. In: Web Technologies: Concepts, Methodologies, Tools, and Applications, pp. 2563–2581. IGI Global (2010)

7. Deterding, S., Björk, S.L., Nacke, L.E., Dixon, D., Lawley, E.: Designing gamification: creating gameful and playful experiences. In: CHI’13 Extended Abstracts on Human Factors in Computing Systems, pp. 3263–3266 (2013)

8. Dichev, C., Dicheva, D.: Gamifying education: What is known, what is believed and what remains uncertain: A critical review. International Journal of Educational Technology in Higher Education **14**(1), 9 (2017)

9. Greenfield, P.M.: Technology and informal education: What is taught, what is learned. Science **323**(5910), 69–71 (2009)

10. Guay, F., Vallerand, R.J., Blanchard, C.: On the Assessment of Situational Intrinsic and Extrinsic Motivation: The Situational Motivation Scale (SIMS). Motivation and Emotion **24**(3), 175–213 (2000). https://doi.org/10.1023/A:1005614228250

11. Hamari, J., Koivisto, J., Sarsa, H.: Does gamification work? — a literature review of empirical studies on gamification. In: Proceedings of the 47th Hawaii International Conference on System Sciences. pp. 3025–3034. IEEE (2014). https://doi.org/10.1109/HICSS.2014.377

12. Jamison, D., Suppes, P., Wells, S.: The effectiveness of alternative instructional media: A survey. Review of educational research **44**(1), 1–67 (1974)

13. Mayer, R.E.: The promise of multimedia learning: using the same instructional design methods across different media. Learning and instruction **13**(2), 125–139 (2003)

14. O’Brien, H.L., Cairns, P., Hall, M.: A practical approach to measuring user engagement with the refined user engagement scale (ues) and new ues short form. International Journal of Human-Computer Studies **112**, 28–39 (2018). https://doi.org/10.1016/j.ijhcs.2018.01.004

15. Sailer, M., Homner, L.: The gamification of learning: A meta-analysis (2020)

16. Seaborn, K., Pennefather, P., Fels, D.I.: Reimagining leaderboards: Towards gamifying competency models through social game mechanics. In: Proceedings of the First International Conference on Gameful Design, Research, and Applications. pp. 107–110 (2013)