# Design and Evaluation of a Cybersecurity Awareness Training Game

Duy Huynh, Phuc Luong, Hiroyuki Iida, Razvan Beuran

► To cite this version:

Duy Huynh, Phuc Luong, Hiroyuki Iida, Razvan Beuran. Design and Evaluation of a Cybersecurity Awareness Training Game. 16th International Conference on Entertainment Computing (ICEC), Sep 2017, Tsukuba City, Japan. pp.183-188, 10.1007/978-3-319-66715-7_19 . hal-01771276

HAL Id: hal-01771276

https://inria.hal.science/hal-01771276

Submitted on 19 Apr 2018

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Design and Evaluation of a Cybersecurity
Awareness Training Game

Duy Huynh✉, Phuc Luong, Hiroyuki Iida✉, and Razvan Beuran✉

School of Information Science
Japan Advanced Institute of Science and Technology
1-1 Asahidai, Nomi, Ishikawa, Japan 923-1211
{duyhuynh, luongphuc, iida, razvan}@jaist.ac.jp

**Abstract.** Serious games are becoming more popular because they provide an opportunity for learning in a natural environment. Although many concepts included in cybersecurity awareness training are universal, many forms of training fail because they are based on rote learning and do not require users to think about security concepts. The main objective of this study is to design a candidate cybersecurity awareness training tool which provides an environment for helping users to understand security concepts while playing a game. To reach our goal, we applied a newly developed model, which is Activity Theory-based Model of Serious Games (ATMSG), to design our own game. According to the design, we implemented a game demo and assessed its gameplay. The results indicated that the story gameplay can help players improve their understanding of cybersecurity problems and resolutions.

**Keywords:** Serious Game · Cybersecurity · Game Design · Activity Theory · ATMSG

## 1 Introduction

Nowadays, the whole world is facing severe challenges posed by hackers, spammers and a large pool of attackers who are motivated by a variety of reasons. To prevent attacks from high-tech criminals, some organizations use a cybersecurity training tool to enhance its information assurance posture. However, creating an engaging training tool, which holds trainees' attention sufficiently long to impart awareness, is a considerable challenge. Moreover, many forms of training fail the studied concepts because the training is by rote learning and does not require users to think about and apply. Generally, to increase motivation and engagement of users, many trainers has designed their training tools as a serious game. A serious game is the combination between game learning sciences and digital entertainment. Similar to simulations, serious games present a virtual reality of varying fidelity that allows learners to explore, experiment, or simply engage in learning. Therefore, our research aims at designing a training tool, which helps users enhance their cybersecurity knowledge, by using a serious game design model. Moreover, we also try to find a suitable gameplay which
helps users understand the knowledge obviously. In this study, we use the Activ-
ity Theory-based Model of Serious Games (ATMSG) [1] and a story gameplay
to design the training game. The details are shown in the next sections.

## 2 Theoretical Background

The learning effects of serious games in studies across educational contexts are inconclusive [10]. One of the recommendations is to ensure that game objectives and learning objectives correspond. It is really difficult to determine whether or not a game supports the learning of students because a learning objective and outcomes are unclear. There exist several models such as RETAIN [4], DODDEL [7], and the 5/10 method [6] which support to design serious games. However, in this study, we use ATMSG [1] to implement our game design idea. The ATMSG provides a comprehensive way to investigate, in detail, how a serious game is structured, and uses activity theory as the theoretical background.

In ATMSG, educational serious games are seen as used in the context of four activities: the gaming activity, the learning activity, the intrinsic instructional activity and the extrinsic instructional activity. There is a four-steps-approach that progressively guides the user in applying the ATMSG to the design or analysis of educational serious games. These steps take the user from a high-level understanding of the activities to the concrete components which implement those activities. The user identifies game components with the help of the taxonomy of serious game components. The details of the four-steps-approach are shown in [1].

In comparison to other models, ATMSG offers a more precise model for the analysis of the educational and gaming aspects of a game, allowing users to perform a more exhaustive decomposition of components as the game unfolds, and to link these components to the overall learning objectives [1]. Moreover, the design using ATMSG is expressed as tables and charts. That facilitates game developers to understand the designer's idea. That is a reason why we choose ATMSG. In our design, we use a gameplay in story games as our main gameplay. We call it a "story gameplay". The story gameplay uses storytelling technique to lead players to experience game events. Hence, we apply this gameplay into our game to support users in clearly understanding the given problem and its solution.

## 3 Design of Cybersecurity Awareness Tranning Game

Firstly, we introduce our game design idea. Our game design idea focuses on making a cybersecurity training game for users to learn and understand obviously cybersecurity concepts. We aim at giving them some reality situations of security attack. For example, "How to recognize phishing emails?", "How to protect your information at public locations?". In those situations, users will control their character and find the solution for given problems. We will lead users through a story step by step and they must make choices, which reflect their character's
behavior. After each situation, a system will evaluate users' answer and gives them the right answer with its explanation. Following this idea, we design a game and describe it in detail by using the four-steps-approach of ATMSG. Critical points of each step are shown in the next discussion.

**Step 1 - Describe the activities:** In this step, we highlight the main aspects of activities to understand this game easily. Moreover, to make it convenient to the implementation, test and evaluation, we choose the topic “Cybersecurity awareness training in a university”. It not only makes convenient for us but also players (university students) to be familiar with given situations in the game and to keep cybersecurity concepts in mind easily. As a result, the main subject of gaming and learning activities is university students who will gain knowledge by experiencing directly real cybersecurity problems and learning to avoid and solve them. Our game demo aims at providing the training tool for students in our university (JAIST). Therefore, game resources such as images, learning content or story reflect the daily life at JAIST.

**Step 2 - Represent the game sequence:** In this step, we provide a diagram to present our game sequence by using UML. The game includes two main activities: “Problem Solving” and “Practice”. Both game activities have different gameplay. The Problem Solving will give players a new situation (story) to experience. Players must understand given situations and choose their behavior carefully to get a perfect score. If players choose a wrong answer, they cannot get a score in this event. An explanation and a right answer will be given after evaluation. While Practice gives players a quiz game, which contains questions about cybersecurity awareness. Players must answer them repeatedly and they will be punished if they choose a wrong answer. Whereas Problem Solving is commonly used to support players in understanding situations, Practice helps players remember cybersecurity concepts. Practice's gameplay is easy to understand, so in our demo, we only give an example. On the other hand, Problem Solving is more complicated than Practice, and our research tries to examine the effect of story gameplay at understanding of users. Thus, we only focus on designing Problem Solving for our game demo.

**Step 3 - Identify actions, tools and goals:** In order to understand a game sequence easily, we are going to identify components that are related to each node in the game sequence. Those nodes are composed from their actions, tools and goals. We first choose the relevant components directly from the taxonomy of serious games (Table 11 in [1]), and fill them into the three layers of each activity involved (gaming, learning, intrinsic and extrinsic instruction). The extrinsic instructional activity is performed outside by the teacher or instructor in the context of the overall learning setting. Our game purpose is not to create an open learning environment for a teacher to teach anything they want. We provide a training tool which helps players in enhancing their awareness about cybersecurity problems in a specific organization (e.g. university, company, etc.). Therefore, in our case, we do not consider an extrinsic instruction, but we simply fill nine layers in total. Almost all components were selected based on main gameplay in order to clearly show designer's ideas by the game actions, tools and
goals. Therefore, this step can help game developers understand the designer's idea thoroughly.

**Step 4 - Description of the implementation:** In this step, we provide a more detailed description of our implementation. We explain what is being done, using which tool, and with what purpose in each block of a table. We also explain how the use of such components and characteristics support the achievement of the entertainment and/or pedagogical goals of the game.

The combination of the four steps described above provides a comprehensive view of the structure of our game, from its high-level purposes and general characteristics to its concrete implementation. In this study, we used Novelty [8] to implement our design. Novelty provides simple methods to create our own visual novel game. A visual novel game [9], also known as a story game, is an interactive game. Typically, the majority of players' interaction is limited to clicking to keep the text, graphics and sound moving on while making narrative choices along the way. Our detailed design and demo is available at [3].

## 4 Evaluation and Discussion

To collect players' feedback on the demo, we created a survey which is available at [2]. The survey requires players to evaluate the demo in multiple aspects of the gameplay and educational value. They must rate on a scale from 1 (worst) to 10 (best). We had 10 participants who filled out the questionnaire. Six of them are at the beginner level in cybersecurity knowledge and other participants are at intermediate level. The details of the survey question and its results are shown in Table 1 and Table 2. The survey questions are categorized into three groups. The first group (Q1-Q6) is used to test abilities of the game demo such as clarity, game length, content, and enjoyment aspect. It helps us figure out a comfortable game setting in official implementation. The second group (Q7-Q9) aims at evaluating learning purposes and the effect of story gameplay on helping players understand cybersecurity concept. The last question is used to estimate how much this game improves players' motivation.

<table><caption>**Table 1.** Survey questions.</caption><thead><tr><td>ID</td><td>Question</td></tr></thead><tbody><tr><td>Q1</td><td>Clarity on what to do</td></tr><tr><td>Q2</td><td>Clarity of story</td></tr><tr><td>Q3</td><td>How fun it is to play the game</td></tr><tr><td>Q4</td><td>How easy it is to play the game</td></tr><tr><td>Q5</td><td>Length of the game</td></tr><tr><td>Q6</td><td>The game provides you to new material</td></tr><tr><td>Q7</td><td>The game helps you to understand a situation of the problem</td></tr><tr><td>Q8</td><td>The game helps you to understand how to solve the problem</td></tr><tr><td>Q9</td><td>Improving your understanding in cybersecurity</td></tr><tr><td>Q10</td><td>Improving the motivation to learn more about cybersecurity</td></tr></tbody></table>
<table><caption>**Table 2.** Survey results: average score per question.</caption><thead><tr><th>Level</th><th>Q1</th><th>Q2</th><th>Q3</th><th>Q4</th><th>Q5*</th><th>Q6</th><th>Q7</th><th>Q8</th><th>Q9</th><th>Q10</th></tr></thead><tbody><tr><td>Beginner</td><td>8.8</td><td>7.3</td><td>6.0</td><td>9.8</td><td>6.5</td><td>7.8</td><td>8.8</td><td>8.0</td><td>8.0</td><td>6.7</td></tr><tr><td>Intermediate</td><td>8.5</td><td>7.8</td><td>7.3</td><td>9.3</td><td>5.5</td><td>7.0</td><td>7.8</td><td>7.5</td><td>7.0</td><td>6.8</td></tr><tr><td>Total</td><td>8.7</td><td>7.5</td><td>6.5</td><td>9.6</td><td>6.1</td><td>7.5</td><td>8.4</td><td>7.8</td><td>7.6</td><td>6.7</td></tr></tbody></table>

The responses to the survey show some advantages of the game. The first advantage is that the game is very easy to play for both groups of participants (Q4, the average in total is 9.6). Besides, the game is rated highly on two aspects that are what to do (Q1, the average in total is 8.7) and the story of the game (Q2, the average in total is 7.5). However, the interest of this game is rated not so high by beginners (Q3, the average is 6.0), but it is interesting for learners who are at intermediate level (Q3, the average is 7.3). The reason is that the game provides new materials to beginners more than intermediate learners (Q6, the averages are 7.8 and 7.0). That makes the beginners feel that the game length is quite long (Q5, the average is 6.5), so they become boring when playing this game. On the contrary, the game length is suitable for intermediate learners (Q5, the average is 5.5) so they feel more comfortable than the others. According to the results of questions from 7 to 9, we have high ratings on the aspect of understanding a problem situation, how to solve the problem and improving students' understanding (Q7-Q9, the averages in total are 8.4, 7.8 and 7.6). However, those aspects are rated by beginner higher than by intermediate learners. The reason is that the game was designed to be a training tool for awareness raising for a beginner to obtain new knowledge. Therefore, this game is more helpful for beginner than intermediate learners. It seems that the game is not good at making enjoyment, so it improves the motivation of users not so well (Q10, the average in total is 6.7). In conclusion, the advantage is that the game can help the students improve their understanding of cybersecurity awareness. By giving a story, the game aims to lead players to understand cybersecurity problems and resolutions, even though it teaches an intermediate student nothing new in the material.

## 5 Conclusion

In this study, we presented the design of a security awareness training tool. By using the ATMSG paradigm, which offers a more precise model for the design of the educational and gaming aspects of a game, we present our game idea in detail. Moreover, the model ensures that game objectives and learning objectives correspond. The four-steps-approach helps game developers follow the idea of designer easily. To evaluate the suitability of the game structure and a story gameplay, we have built a game demo which is implemented by using the Novelty software. After the game was played and evaluated by university students, the result indicated that the game was rated well on the aspect of understanding

<footnote>* this aspect is rated on a scale from 1 (short) to 10 (long)</footnote>
cybersecurity problems and solutions. Therefore, the game which we developed by following this design help us avoid the rote learning of users when using the training tool. However, to improve the enjoyment of this game and the learners' motivation, we should add more game actions and game elements.

Although the game can help users in understanding the cybersecurity concept, it is not good at making enjoyment for players because it is simple to play by clicking and reading. Therefore, our future works aim at fairly changing in a gameplay which increases interaction of users such as control character, click on items, etc. to make the game more interesting. Moreover, the balance between tasks and rewards in a serious game is very important for its entertaining aspect, so we aim to find a suitable learning content structure in order to improve enjoyment in the designed game by applying some game theories on measuring the game enjoyment such as game refinement theory [5].

**Acknowledgments.** The authors wish to thank the anonymous referees for their constructive comments that helped to improve the article considerably, and the group of students who participated in the survey of this study.

## References

1. Carvalho, M. B., Bellotti, F., Berta, R., De Gloria, A., Islas Sedano, C., Baalsrud Hauge, J.: An activity theory-based model for serious games analysis and conceptual design. Computers and Education, 87, 166–181

2. CSAG Survey - Google Form. N.p., 2017. Web. 5 Jan. 2017. https://goo.gl/forms/jZUDRFcx8TMdfsTJ3

3. Cybersecurity Awareness Game Demo - Dropbox. N.p., 2017. 5 Jan. 2017. https://www.dropbox.com/sh/xpzlfjmuqaljvxm/AAAGPRHfgZimUeWmQyx3QYiHa?dl=0

4. Gunter, G.A., Kenny, R.F., Vick, E.H.: Taking educational games seriously: using the RETAIN model to design endogenous fantasy into standalone educational games. Education Tech Research Dev, 56:511–537, 2008

5. Iida, H., Takeshita, N., and Yoshimura, J.: A metric for entertainment of boardgames: Its implication for evolution of chess variants. Entertainment Computing Technologies and Applications, 65–72, (2003).

6. Van Rooij, R.: The 5/10 method: A method for designing educational games. Masters thesis, Game and Media Technology, Utrecht University (2013)

7. McMahon., M.: Using the DODDEL model to teach serious game design to novice designers. In Proceedings ascilite Auckland, 2009

8. Novelty - Visual Novel Maker. Visualnovelty.com. N.p., 2017. Web. 4 Jan. 2017, http://www.visualnovelty.com

9. Visual Novel. En.wikipedia.org. N.p., 2017. Web. 4 Jan. 2017, https://en.wikipedia.org/wiki/Visual_novel

10. Young, M.F., Slota, S., Cutter, A.B., Jalette, G., Mullin, G., Lai, B., Simeoni, Z., Tran, M., Yukhymenko, M.: Our princess is in another castle: a review of trends in serious gaming for education. Rev Educ Res, 82(1):61–89, 2012