# Towards Cognitive Adaptive Serious Games: A Conceptual Framework

Andrew Seyderhelm, Karen L. Blackmore, Keith Nesbitt

► To cite this version:

Andrew Seyderhelm, Karen L. Blackmore, Keith Nesbitt. Towards Cognitive Adaptive Serious Games: A Conceptual Framework. 1st Joint International Conference on Entertainment Computing and Serious Games (ICEC-JCSG), Nov 2019, Arequipa, Peru. pp.331-338, 10.1007/978-3-030-34644-7_27. hal-03652035

HAL Id: hal-03652035

https://inria.hal.science/hal-03652035

Submitted on 26 Apr 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Towards Cognitive Adaptive Serious Games:
## A Conceptual Framework

Andrew J A Seyderhelm[0000-0003-0191-0367], Karen L Blackmore[0000-0002-9111-0293] and
Keith Nesbitt[0000-0001-5408-7138]

School of Electrical Engineering & Computing, University of Newcastle, Callaghan, Australia
Andrew.Seyderhelm@uon.edu.au

**Abstract.** Games and immersive training environments frequently rely on user performance measures to adapt the difficulty of tasks and behaviors, responding dynamically to changes in performance. However, users may maintain task performance while experiencing increasing levels of cognitive load. These high levels of load mean the user has no spare capacity and may fail to get the maximum benefit from the training task. While other adaptive mechanisms exist, they do not account well for cognitive load and thus may not be optimal for training tasks. In this paper we outline a conceptual framework for using real-time measures of cognitive load to dynamically adapt immersive environments. We argue that these measures have the benefit of providing a richer mix of data to base adaption on beyond simple performance metrics, and additionally provide further metrics to assess both the learner and the training material. To this end, a Cognitive Adaptive Serious Game Framework (CASG-F) is presented that draws on frameworks and theories of cognitive load and serious games. We additionally outline the range of potential mechanics and environment parameters that could potentially be adjusted to modify difficulty.

**Keywords:** serious games, conceptual model, cognitive load, adaptive.

# 1 Introduction

## 1.1 Overview

Serious games are used for many purposes; throughout this paper reference to serious games will adopt a definition specific to the purposes of education and training: "A serious game is an experience designed using game mechanics and game thinking to educate individuals in a specific content domain" (Kapp, 2012, page 15) [1]. It has been suggested that serious games provide a number of advantages over traditional learning or instructional approaches, including flexibility, creative problem solving, greater engagement and enjoyment in the material [2,3] and the ability to produce metrics that are valuable to debriefing learners and informing ongoing serious game development [4,5].

However, many serious games fail to leverage their full advantages, and this is particularly true in respect of adaptive mechanisms and dynamic difficulty adaption (DDA). Adaptive training in serious games has the benefit of being a cost-effective
method of providing training that closely approximates one-on-one tutoring and in-
creases the overall effectiveness of training [6]. Traditional games have explored DDA
in 3D games for well over a decade, and have demonstrated the efficacy of these sys-
tems on player enjoyment and engagement [7,8]. Serious games have an additional pur-
pose to provide a learning outcome for players. Thus, the mechanisms for adaption may
need to be different as the foci is on learning outcomes rather than purely entertainment,
although enjoyment and engagement are crucial components of effective learning.

A range of approaches have the potential for being useful for adapting serious games. This includes, but is not limited to, inventory and pick-up adaption [7], pedagogical agents [9], and a wide range of AI implementations. Serious games that adjust the challenge level in-line with the growing capability and knowledge of the learner are ideal for maintaining engagement, motivation and learning outcomes [3]. In this paper, we discuss DDA for adapting serious games, and extend this to consider approaches for measuring cognitive load that could be used to drive the adaptive mechanism in serious games. From this, we present a conceptual framework for cognitive adaptive serious games based on cognitive load theory and discuss future research plans to develop and assess this approach.

## 2 Core Components for Cognitive Adaptive Serious Games

2.1 Dynamic Difficulty Adjustment and Adaptive techniques

The purpose of an adaptive framework is to assess the current state of the individual learner and adjust the training to better suit their needs within the constraints of the training requirements. In a commercial video games context, there has been significant research into developing systems to implement DDA [10,11]. The aim of DDA is to provide the player with the correct level of challenge in order to make the game neither too hard nor too easy, thus increasing enjoyment [12,11]. This concept of optimal challenge for the purpose of enjoyment is implemented with the aim of assisting the player achieve a flow state [13,11]. Ideally, the benefit of DDA is that it will continuously adapt to player skill in an appropriate and subtle manner.

As previously indicated, the primary aim of a serious game is to deliver instructional,
learning, or development outcomes. As a result, DDA in this context needs to relate to
the performance of the participant against the serious games purpose rather than solely
to enhance entertainment through amplification of flow states. Our conceptual model
adopts a three-part framework towards engagement that encompasses flow along with
other aspects suited to serious games [14]. One of the crucial aspects we consider is the
role of cognitive load in the learning process, acknowledging the role of cognitive load
levels in providing insights into schema integration amongst learners [15].

## 2.2 Measuring Cognitive Load

Cognitive load is the degree to which a learning task meets, exceeds, or fails to reach the processing capacity of a learner's cognitive system [16]. This is explained through Cognitive load theory (CLT), which originated in the 1980s [17]. It is a widely accepted
concept that describes three states of the cognitive processes involved in learning; intrinsic, extraneous and germane cognitive load [18]. There are a number of methods of measuring cognitive load, principally subjective and objective measures. Subjective measures are typically completed by participants after undertaking an activity, for example the NASA Task Load Index (NASA-TLX) [19,20]. In contrast, objective measures are undertaken during an activity, measuring an observable occurrence, affect, or physiological system, and interpreting that data for cognitive load.

The conceptual framework detailed in this paper proposes the use of an adapted version of the detection-response task (DRT) [21] embedded within a serious game. The DRT has been chosen as it is registered through the International Organization for Standardization in ISO 17488:2016 as a proven and effective measure [22]. A virtual DRT, termed the “Remote DRT”, has been previously tested in-simulation, and proven to be effective [21]. An updated version of the “Remote DRT” is proposed here, potentially making a cognitive load based adoption an accurate, effective and affordable method for wide scale adoption.

## 2.3 Adapting Serious Games Through the Frame of CLT

CLT provides three aspects of cognitive load that can be manipulated to establish a sophisticated and theoretically sound adaptive framework. A practical description of how these adaptations may be implemented in serious game tasks is outlined in Table 1.

<table><caption>**Table 1.** Adaptive examples tailored to CLT</caption><thead><tr><td>Cognitive Load Element</td><td>Method for adjustment</td></tr></thead><tbody><tr><td>Intrinsic Cognitive Load (interaction elements)</td><td>Alter the task complexity [23] by increasing or decreasing the number of steps at each stage e.g. making a car automatic rather than manual until the driver has grasped steering etc.</td></tr><tr><td>Extraneous Cognitive Load (presentation of material)</td><td>The way material is presented can lead to an increase, or decrease, in extraneous load, e.g. light or weather effects may increase or decrease cognitive load.</td></tr><tr><td>Germane Cognitive Load (development of schemas)</td><td>The introduction of a “pedagogical agent” who assists the student may assist the learning process [9].</td></tr></tbody></table>

Using CLT as a framework provides a structured way for a combination of cognitive load and various performance metrics to be used to dynamically adapt a serious game to optimize learning.

# 3 Conceptual Framework

The conceptual framework detailed here is an extension of the serious games conceptual model proposed by Yusoff et al (2009) [24]. However, the Yusoff et al model does not incorporate learner motivation, affect, and prior knowledge as described in the cognitive-affective theory of learning with media (CATLM) [9]. Yusoff et al define a range
of factors, and the conceptual model outlined below adopts their definitions and adds the following components by drawing on CATLM and others [25]:

1. *Learner Knowledge & Experience* – understanding the learner's current state of intrinsic motivation, knowledge and experience is essential [6]. It is important to assess the player's ability with the serious game controls to avoid the learner struggling with the controls, rather than the learning content within it.

2. *Review, Iterate* – post-game review otherwise termed after-action review. This is critical to the learning process and is often underutilized in serious games [5,4]. It is important for the facilitator to assess the performance of the serious game itself.

3. *Play Game* – forms the start/finish of the cognitive and performance measure loop.

4. *Cognitive load and performance measures* – a constant loop driven by the cognitive measure. This process includes a performance measure, this is separate to the cognitive measure loop however they are combined to inform the DDA mechanism.

5. *Feedback* – is similar to *Reflection* described by Yusoff et al [24] and by Moreno and Mayer [9]; this is in-game feedback to help and inform the player.

6. *DDA +/-* – This is the point at which an adaption mechanism is implemented making the game easier, harder or the same as described in table 2.

7. *Game Performance* – outputs the player performance and cognitive load measures to inform the debrief process and other course requirements [5,4].

<table><caption>**Table 2.** Performance and Cognitive Load (CL) Adaption Template</caption><thead><tr><td>Achievement</td><td>Adaption</td><td>Description</td></tr></thead><tbody><tr><td>Pass assessment with low CL.</td><td>DDA increase (+)</td><td>Player is finding the task easy, so increase the challenge.</td></tr><tr><td>Pass assessment with mid CL.</td><td>DDA static</td><td>Player is in the correct level of difficulty. No change.</td></tr><tr><td>Pass assessment with High CL.</td><td>DDA static</td><td>Player is passing but finding it hard, provide in-game feedback.</td></tr><tr><td>Fail assessment with low CL.</td><td>DDA Static</td><td>Player has failed but not being mentally challenged, in-game feedback.</td></tr><tr><td>Fail assessment with mid CL.</td><td>DDA -</td><td>Cognitive load is ideal but failed assessment, indicating a lack of knowledge, reduce the difficulty slightly and add a hint.</td></tr><tr><td>Fail assessment with high CL.</td><td>DDA-</td><td>Player is struggling, make it easier and reduce complexity.</td></tr></tbody></table>

Table 2 presents the various proposed DDA adaptions including concepts discussed in the preceding section (Section 2.3). This follows on from the identification of how difficulty may be adjusted (Table 1). Together, the existing serious game conceptual models, extended to incorporate the inclusion of an adaptive mechanism based on cognitive load, results in a new Cognitive Adaptive Serious Game Framework (CASG-F) (Figure 1). Performance measures and cognitive load will work together to adapt the
in-game tasks. However cognitive load will form an adaption mechanism in its own
right, e.g. making a simulated car automatic instead of manual to reduce interaction
elements (intrinsic cognitive load), or increasing challenge by introducing additional
tasks (intrinsic) or weather effects (extraneous cognitive load).

<figure><img src="image_2.png" /><figcaption>Fig. 1. The Cognitive Adaptive Serious Game Framework (CASG-F)</figcaption></figure>

4 Future Research

The CASG-F proposed here has three broad stages in order to encompass instructional
design concepts [26] and the triadic theoretical framework [27]. The CASG-F provides
a roadmap for future experiments to incorporate these principles as best practice, and
to outline how they are being integrated into the design and development process. The
overarching aim of this research is to create an adaptive framework that can work flex-
ibly and effectively using cognitive load as a key adaption measure.

Moving forward, robust experimental studies to validate the proposed adaptive pro-
cess, and also to provide efficacy for the approach, are planned. Firstly, an experimental
study is required to assess the in-game use of the DRT cognitive load measure, and to
further validate its authenticity using a triangulation approach with an EEG combined
with the NASA-TLX. The first experiment enables statistical validation of the affect
different interventions have on the participants’ cognitive load, providing a ‘toolbox’
of adaptive techniques for future experiments. Later experimental studies will then be
required to compare adaptive serious game performance to non-adaptive variants in
order to corroborate the assumption that adaptive serious games are more effective in
enhancing human performance.
The first experiment involves a driving task in a 3D game environment. This has the benefit of using a task where the real-world efficacy of the DRT has already been established [21]. The proposed experiment will:

* Present three levels with the same driving track layout; the surrounding virtual environment will be different in each layout minimizing repetition for participants, and also provide information on how different visual environments affect cognitive and visual load.

* Using a randomized approach, participants will drive two loops of the track; one loop will have the DRT and the other will not, the EEG will remain on for both loops.

* The levels will include challenge sections, for example navigating a narrow section of road, observation tasks, following a vehicle, and more.

* Qualities of the visual environment will be manipulated in order to observe how these changes relate to visual load and cognitive load, e.g. altering in-game weather and lighting. Quantifying the effect of visual environment manipulations on cognitive load will then further contribute to the adaption 'toolbox'.

A study following this design will facilitate evaluation of the effect of the DRT itself on the cognitive load of the player participants. Experimental designs based on the CASG-F should also consider the layering of primary and secondary tasks to allow for robust measurement of cognitive load using the DRT approach [28]. To this end, the player should be given an additional task to perform requiring different cognitive processes. For example, asking the participant to count or respond to certain assets in the environment, requiring visual and cognitive discrimination, will effectively alter cognitive load [28]. Together, the CASG-F and the proposed experimental design provides an authentic first step toward a generalizable framework for developing serious games that respond dynamically to improve learning outcomes.

# 5 Conclusion

In this paper we propose a framework to realize the value of DDA and cognitive measures as applied to serious games. We then discussed how an adaptive framework can be applied through the lens of CLT. Extending existing theoretical frameworks, a conceptual model for Cognitive Adaptive Serious Games was presented that incorporates and recognizes different learning design approaches. Additionally, the proposed model has the potential to deliver real-time personalized training environments to move serious game implementations closer to the overall aim of enhancing human performance. Lastly, a comprehensive experimental design for future studies was presented where in-game cognitive load will be measured and verified by a variety of methods against a range of stimuli. This experiment design provides a roadmap for validating the real-time DRT as a cognitive load measure, and to establish a toolbox of validated methods to manipulate cognitive load based on a robust understanding of the impact of different game mechanics and mechanisms on the cognitive load of end users. Once established, this toolbox can then be applied to serious game implementations to assess the efficacy of adaptive serious games for education and training.
References

1. Kapp, K.M.: The gamification of learning and instruction: game-based methods and strategies for training and education. Wiley San Francisco, (2012)

2. Connolly, T.M., Boyle, E.A., MacArthur, E., Hainey, T., Boyle, J.M.: A systematic literature review of empirical evidence on computer games and serious games. Computers & Education **59**(2), 661-686 (2012). doi:10.1016/j.compedu.2012.03.004

3. Hamari, J., Shernoff, D.J., Rowe, E., Coller, B., Asbell-Clarke, J., Edwards, T.: Challenging games help students learn: An empirical study on engagement, flow and immersion in game-based learning. *Computers in human behavior* **54**, 170-179 (2016). doi:10.1016/j.chb.2015.07.045

4. Loh, C.S., Sheng, Y., Ifenthaler, D.: Serious games analytics: Theoretical framework. In: Serious games analytics. pp. 3-29. Springer, (2015)

5. Crookall, D.: Serious games, debriefing, and simulation/gaming as a discipline. *Simulation & gaming* **41**(6), 898-920 (2010). doi:10.1177/1046878110390784

6. Landsberg, C.R., Astwood, R.S., Van Buskirk, W.L., Townsend, L.N., Steinhauser, N.B., Mercado, A.D.: Review of Adaptive Training System Techniques. Military Psychology **24**(2), 96-113 (2012). doi:10.1080/08995605.2012.672903

7. Hunicke, R.: The case for dynamic difficulty adjustment in games. In: Proceedings of the 2005 ACM SIGCHI International Conference on Advances in computer entertainment technology 2005, pp. 429-433. ACM

8. Xue, S., Wu, M., Kolen, J., Aghdaie, N., Zaman, K.A.: Dynamic difficulty adjustment for maximized engagement in digital games. In: Proceedings of the 26th International Conference on World Wide Web Companion 2017, pp. 465-471. International World Wide Web Conferences Steering Committee

9. Moreno, R., Mayer, R.: Interactive Multimodal Learning Environments. Educational Psychology Review **19**(3), 309-326 (2007). doi:10.1007/s10648-007-9047-2

10. Zohaib, M.: Dynamic Difficulty Adjustment (DDA) in Computer Games: A Review. Advances in Human-Computer Interaction **2018**, 1-12 (2018). doi:10.1155/2018/5681652

11. Dziedzic, D., Włodarczyk, W.: Approaches to Measuring the Difficulty of Games in Dynamic Difficulty Adjustment Systems. International Journal of Human-Computer Interaction **34**(8), 707-715 (2018). doi:10.1080/10447318.2018.1461764

12. Alexander, J.T., Sear, J., Oikonomou, A.: An investigation of the effects of game difficulty on player enjoyment. Entertainment Computing **4**(1), 53-62 (2013). doi:10.1016/j.entcom.2012.09.001

13. Burns, A., Tulip, J.: Detecting flow in games using facial expressions. In: 2017 IEEE Conference on Computational Intelligence and Games (CIG) 2017, pp. 45-52. IEEE

14. Hookham, G., Nesbitt, K.: A Systematic Review of the Definition and Measurement of Engagement in Serious Games. Paper presented at the Proceedings of the Australasian Computer Science Week Multiconference on - ACSW 2019,

15. Greitzer, F.L., Kuchar, O.A., Huston, K.: Cognitive science implications for enhancing training effectiveness in a serious gaming context. *Journal on Educational Resources in Computing* (JERIC) **7**(3), 2 (2007). doi:10.1145/1281320.1281322
16. Mayer, R.E., Moreno, R.: Nine ways to reduce cognitive load in multimedia learning. Educational psychologist **38**(1), 43-52 (2003). doi:10.1207/S15326985EP3801_6

17. Paas, F., Van Gog, T., Sweller, J.: Cognitive load theory: New conceptualizations, specifications, and integrated research perspectives. Educational psychology review **22**(2), 115-121 (2010). doi:10.1007/s10648-010-9133-8

18. Paas, F., Renkl, A., Sweller, J.: Cognitive load theory and instructional design: Recent developments. Educational psychologist **38**(1), 1-4 (2003). doi:10.1207/S15326985EP3801_1

19. Hart, S.G., Staveland, L.E.: Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In: Advances in psychology, vol. 52. pp. 139-183. Elsevier, (1988)

20. Hart, S.G.: NASA-task load index (NASA-TLX); 20 years later. In: Proceedings of the human factors and ergonomics society annual meeting 2006, vol. 9, pp. 904-908. Sage publications Sage CA: Los Angeles, CA

21. Harbluk, J.L., Burns, P.C., Tam, J., Glazduri, V.: Detection response tasks: Using remote, headmounted and Tactile signals to assess cognitive demand while driving. (2013). doi:10.17077/drivingassessment.1470

22. Stojmenova, K., Sodnik, J.: Detection-Response Task-Uses and Limitations. Sensors (Basel) **18**(2) (2018). doi:10.3390/s18020594

23. Liu, P., Li, Z.: Task complexity: A review and conceptualization framework. International Journal of Industrial Ergonomics **42**(6), 553-568 (2012). doi:10.1016/j.ergon.2012.09.001

24. Yusoff, A., Crowder, R., Gilbert, L., Wills, G.: A conceptual framework for serious games. In: 2009 Ninth IEEE International Conference on Advanced Learning Technologies 2009, pp. 21-23. IEEE

25. Annetta, L.A.: The "I's" have it: A framework for serious educational game design. Review of General Psychology **14**(2), 105-113 (2010). doi:10.1037/a0018985

26. Keller, J.M.: Development and use of the ARCS model of instructional design. Journal of instructional development **10**(3), 2 (1987). doi:10.1007/BF02905780

27. Rooney, P.: A theoretical framework for serious game design: exploring pedagogy, play and fidelity and their implications for the design process. International Journal of Game-Based Learning (IJGBL) **2**(4), 41-60 (2012). doi:10.4018/ijgb1.2012100103

28. Conti, A., Dlugosch, C., Vilimek, R., Keinath, A., Bengler, K.: An assessment of cognitive workload using detection response tasks. Advances in human factors and ergonomics series. Advances in human aspects of road and rail transportation, 735-743 (2012). doi:10.1201/b12320-82