# A Pilot Study in Cyber Security Education Using CyberAIMs: A Simulation-Based Experiment

Erjon Zoto, Stewart Kowalski, Christopher Frantz, Edgar Lopez-Rojas, Basel Katt

<figure><img src="image_1.png" /></figure>

## To cite this version:

Erjon Zoto, Stewart Kowalski, Christopher Frantz, Edgar Lopez-Rojas, Basel Katt. A Pilot Study in Cyber Security Education Using CyberAIMs: A Simulation-Based Experiment. 11th IFIP World Conference on Information Security Education (WISE), Sep 2018, Poznan, Poland. pp.40-54, 10.1007/978-3-319-99734-6_4 . hal-02125763

HAL Id: hal-02125763

https://inria.hal.science/hal-02125763

Submitted on 10 May 2019

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
# A Pilot Study in Cyber Security Education using CyberAIMs: A Simulation-Based Experiment

Erjon Zoto, Stewart Kowalski, Christopher Frantz, Edgar Lopez-Rojas, and Basel Katt

Norwegian University of Science and Technology (NTNU), Gjøvik, Norway
[erjon.zoto;stewart.kowalski;christopher.frantz;basel.katt;edgar.lopez]@ntnu.no

**Abstract.** We hardly pass any day without hearing of a new cyber attack. The recent ever-increasing occurrence of such attacks has given to researchers, practitioners and others an opportunity to raise awareness and train staff from the public and private institutions, as well as other people within the society, about the evolving nature of cyberspace threats. As a first step in this process, we aim to present main findings from a pilot study conducted with a target group of Master students with diverse backgrounds and knowledge about cyber security practices. The study was done using an agent-based simulation tool, CyberAIMs, as the core component of the experiment. Students were involved in a pre-test/post-test study in order to assess the probable change in their thinking process after using CyberAIMs. A scenario created from a real cyber case was additionally used to get the participants accustomed to the tool. The experiment is still in progress, while preliminary data indicate that there is a shift in students' perspective on the most relevant attributes affecting defense agents' performance, results that could be related to both adversarial and systems thinking processes.

**Keywords:** Agent-based simulation · Teaching · Cyber security · Adversarial thinking · Systems thinking · Training

## 1 Introduction

Cyber security events have been major headlines at an ever-increasing pace for the past recent years. Last year produced notable attacks such as WannaCry and NotPetya, while the most recent global event targeted from cyber attackers has been the Winter Olympic Games in South Korea this February.

With attacks intensifying in numbers and covering more and more unpredictable targets, researchers and practitioners are putting their best efforts in trying to raise awareness and train staff from the public and private institutions about the evolving nature of cyberspace threats. Several leading institutions from academia and beyond have already paved the way for further research related to cyber security [2] (p. 21).

In line with recent developments, the Joint Task Force on Cybersecurity Education (JTF), created in September 2015, has developed a new curriculum
volume, as part of its continuous efforts on the main purpose of developing comprehensive curriculum guidance in cyber security education [8]. The new curriculum volume introduces some new crosscutting concepts to deal with the evolving nature of cyberspace threats. These are:

- Adversarial thinking, as a process that considers the potential actions of the opposing force working against the desired result.

- Systems thinking, as a process that considers the interplay between social and technical constraints to enable assured operations.

The contribution of our work is directed towards the improvement of the adversarial and systems thinking ability in cyber security with focus on Master level students. This study was performed using an agent-based simulation tool, named CyberAIMs. The name is an acronym for Cyber Agents’ Interactive Modeling and Simulation. It also shows that each actor in cyberspace follows certain procedures and strategies according to their own aims, as part of a higher entity or on individual basis. CyberAIMs was built using NetLogo<fnref n="1" />, which is a programmable modeling environment for simulating natural and social phenomena. NetLogo is particularly well suited for modeling complex systems developing over time, with hundreds or thousands of agents, all operating independently.

We used this tool as the main component of a simulation-based experiment conducted with students of Information Security, in order to further address their adversarial and systems thinking abilities. The target group included 12 individuals in an elective Master course that were asked to answer two surveys, pre and post-experiment, as well as a scenario of a recent real-world case of a cyber attack during the experiment. Students were intended to use CyberAIMs in order to give correct answers to the questions from the scenario.

## 1.1 Learning benefits

Pastor et al. [9] have done extensive research work on the available state-of-the art simulation tools that can be used on the purpose of teaching and training. They suggest that such simulation tools should be designed to have a extremely simple user-friendly interface and, at the same time, allow the user to obtain a deep understanding of the concepts.

Adversarial thinking has already been studied as an important skill for cyber security, Hamman et al. [7] propose that cyber security students should learn about basic game theory concepts in order to improve their strategic reasoning abilities. Similar to Schneider [12], our work aims to teach cyber security to students at university level.

Systems thinking has been associated to different areas of research since several decades now, and can also be relevant for information and cyber security. There are many examples where using simulations for teaching systems thinking, such as the work from Goodwin and Franklin [6], or the contribution from Anne Badoel and Haslett [3]. Their seminal work motivated our work further in this

<footnote n="1">¹ http://ccl.northwestern.edu/netlogo/</footnote>
paper, while aiming to use simulation as part of the curriculum developed in the field of cyber security.

We aim to reflect the mechanisms behind the thinking processes above by using them within CyberAIMs, part of our recent research work done in the intersection between cyber security and related research fields.

## 1.2 Outline of the paper

We have organized the paper contents as follows. Section 2 will provide information on the main research question and methodology used. Section 3 will provide more details on the design process of the tool used and its main features. The final sections will conclude this paper by providing main insights from the pilot study and relevant discussions to help the reader get familiar with the next objectives of this research process as a whole.

# 2 Research methodology

## 2.1 Research questions

The main aim of this paper is to produce a proof of concept artifact that is able to show how a simulation tool can affect thinking processes of a group of students in cyber security. With this artifact we hope to address the new directions suggested in developing curriculum for cyber security in education. We have devised the following research question in order to achieve the objectives mentioned above:

- Research Question (RQ): How can we improve adversarial and systems thinking ability on students in cyber security?

This research question helps us understand the approaches that might help improve the learning process of adversarial and/or systems thinking for training and teaching purposes, as mentioned above. We have proposed a simulation tool, inspired from the work of Pastor et al. [9], that may prove to be useful in such case. We justify the use of the simulation tool further in section 4 by using the results from the pre-test/post-test study with the sample target. The process is explained in the next subsection below.

## 2.2 Research methodology

Besides using a simulation tool as an intuitive way to improve learning outcomes of a cyber security course, we saw the need to validate potential outcomes by conducting surveys before and after the tool was used from the target sample. The surveys included a set of similar questions and a set of different questions, according to the objectives of this study. Each student was asked to provide their ID as a means to uniquely identify them. The students were also asked if they wanted to receive via email a soft copy of their individual answers.

In the pre-simulation survey, there were three sections of open and closed questions, listed below:
- Learning from simulations

- About you

- Expectations from the model

In the first section, students were provided with two sets of statements and further asked to answer them using a 5-levelled Likert scale, with values ranging from strongly disagree to strongly agree. Table 1 gives details on all questions requiring Likert scale feedback from respondents.

<table><caption>**Table 1.** Pre-simulation survey questions using Likert scale</caption><thead><tr><th>Section</th><th>Question</th><th>Statements/Options</th></tr></thead><tbody><tr><td rowspan="2">Learning from simulations</td><td>I expect that the simulation will develop my</td><td>Problem solving skills;<br>Planning skills;<br>Understanding of cybercrime;<br>Understanding of economics theories on cybercrime;<br>Understanding of strategic mgmt. of info. security;<br>Understanding of risk management;<br>Understanding of real-world cyber scenarios;<br>Understanding of systems thinking;<br>Understanding of adversarial thinking</td></tr><tr><td>Please rate your agreement with the following</td><td>The simulation will be challenging;<br>I will enjoy learning with the simulation;<br>Building on knowledge gained from previous courses;<br>Building on knowledge gained from previous labs</td></tr><tr><td>About you</td><td>Please rate your agreement with the following</td><td>I have a background in programming;<br>I have a background in economics/management;<br>I have a background in human sciences/psychology;<br>I have a background in military/warfare strategies</td></tr><tr><td rowspan="2">Expectations from the model</td><td>Please rate the level of relevance for each attribute on the attack success rate</td><td>Defense Resources;<br>Defense Skills;<br>Defense Motivation;<br>Attack Resources;<br>Attack Skills;<br>Attack Motivation</td></tr><tr><td>Please rate the level of relevance for each attribute on the defense success rate</td><td>Defense Resources;<br>Defense Skills;<br>Defense Motivation;<br>Attack Resources;<br>Attack Skills;<br>Attack Motivation</td></tr></tbody></table>

The next section of this survey required feedback on the respondents current program of studies, home country, gender and age. It also included a question on the respondents' background using a Likert scale with same values as above.

The final section included two questions on the students' expectations related to the most relevant attributes. They had to provide answers using another 5-levelled Likert scale by rating all attributes given from highly irrelevant to highly relevant regarding their impact on each side's success rate, linked to the
probability of the simulation ending in fewer steps than the maximum available ones. Students were further asked to rank top three attributes that they thought were most relevant. They had also the opportunity to submit optional comments on the rationale behind the answers provided in this section.

In order to simplify and improve the learning outcomes, we created a scenario related to a recent real-world case of a cyber attack, that could be easily mapped into CyberAIMs and further analyzed. The scenario was the main part of the lab conducted with the students, where they were asked to answer questions by putting into practice their knowledge on the tool and the logic behind the attributes involved. Questions included calculating the defense success rate and defining the most relevant attributes related to this rate. In the next section, we will provide more details about CyberAIMs.

The post-simulation survey included two sections of open and closed ques-
tions, similar to the first and last section of the pre-survey. The perspective
changed from the expected to the real learning outcomes from using the tool.
Here, in the final section, an additional question required respondents’ feedback
on the total time of engagement with the tool as well as a concluding optional
comment on the whole experience related to the experiment.

3 CyberAIMs

CyberAIMs is an agent-based simulation tool designed in NetLogo, as shown in Figure 1. It includes two sets of agents, namely defense and attack agents. We classified each of the groups in four distinct categories, hereinafter echelons.

<figure><img src="image_3.png" /><figcaption>Fig. 1. Screenshot - CyberAIMs</figcaption></figure>

Defense echelons are: *Ind* (individuals, ordinary people, related to a specific socio-cultural context), *SMB* (small and medium businesses, with relatively
low yearly income), Corp (multinational corporations, biggest enterprises), State (state agents, part of high-level organizations and agencies).

Attack echelons are: *Kid* (the script kiddies, individual hackers, also part of a specific socio-cultural context), *Ideol* (ideological hackers, hack-tivists, acting on the basis on moral and ethical duty), *Contract* (the Contractors, organized cybercrime groups, providing illegitimate services in exchange for money or other incentives), *State* (state-sponsored attack agents, high-level organizations and agencies, heavily engaged in cyberwar events recently).

We defined further three attributes to explain the behaviour and performance of the agents within CyberAIMs. The attributes are *Resources*, the budget related to cyber activities; *Skills*, the level of training, literacy and awareness on cyber events; and *Motivation*, the level of self-motivation and incentives in a certain time.

We used various sources of data for *Resources* depending on the agents' side and echelons, including the Ponemon Report [10], and also the GCI Index [5] for the *Skills* units. As an example, an individual spending 1000 USD would have 25 units of *Resources*, while a state spending USD 1 billion would have 75 units. Meanwhile, agents from Singapore, the country with the highest GCI score, would have on average 92 units of *Skills*. Finally, we used a heuristic approach for *Motivation* in this version of CyberAIMs, which only included a four-levelled scale from Low to High. In the next versions of the tool, we intend to use various motivation theories, as explained in the last section.

The current version of CyberAIMs allows the user to define initial number of agents in each side of the battlefield and also the initial value for each of the attributes for all agents on each side. The user can choose values in a [1 100] range for the number of agents on each side, initial units of *Resources* and *Motivation*, and a [1, 93] range for the *Skills* units, as detailed in Table 2.

<table><caption>**Table 2.** Distribution of attributes' values</caption><thead><tr><th>Attribute</th><th>Side</th><th>Echelon/ Level</th><th>Range of values</th></tr></thead><tbody><tr><td rowspan="4">Resources</td><td rowspan="4">Defense</td><td>Ind</td><td>1-31</td></tr><tr><td>SMB</td><td>1-40</td></tr><tr><td>Corp</td><td>40-70</td></tr><tr><td>State</td><td>60-100</td></tr><tr><td rowspan="4">Attack</td><td rowspan="4">Attack</td><td>Kid</td><td>1-37</td></tr><tr><td>Ideol</td><td>15-37</td></tr><tr><td>Contract</td><td>35-67</td></tr><tr><td>State</td><td>60-100</td></tr><tr><td rowspan="3">Skills</td><td rowspan="3">Attack/ Defense</td><td>Low</td><td>1-30</td></tr><tr><td>Medium</td><td>31-70</td></tr><tr><td>High</td><td>71-93</td></tr><tr><td rowspan="4">Motivation</td><td rowspan="4">Attack/ Defense</td><td>Low</td><td>1-25</td></tr><tr><td>Moderate Low</td><td>26-50</td></tr><tr><td>Medium</td><td>51-75</td></tr><tr><td>High</td><td>76-100</td></tr></tbody></table>
The tool performs each run in a period of max 120 ticks. Each tick represents a fixed period of time of three days, mapping the minimum time required for an attacker to perform a successful attack [10], thus making it able to predict the behavior of agents on both sides within a year. The current version allows a random attack agent on each tick to randomly target one or more defense agents, while attacking them depends on the combined attributes' values on each side.

If the attack is performed, the defense agent on target loses to the attacking agent a certain amount of *Resources*, related to the attack agent's relative power, defined by multiplying the latter attribute values and dividing them by the sum of attribute values' products from both agents. The *Skills* units are also updated by increasing values in both sides, with the defense agent having a larger increase in terms of learning experience. *Motivation* units are also updated on the attack agents side, increasing them by the value of its relative power. If the attack is avoided, *Motivation* units are updated only on the defense side, by the value of the attack agent's relative power.

Continuous successful attacks can actually decrease defense agents' *Resources* units until losing them all. When this happens, the defense agent goes offline, meaning he does not interact anymore with the other agents. When all defense agents go offline, CyberAIMs stops running, displaying a message on the attack agents winning the game, as in Figure 1 above.

By having initial values of attributes comparable between them along with successful attacks defined by the simple product of attributes values, CyberAIMs aims to analyze the impact of initially equal attributes in the final outcome after each run. Furthermore, changing *Skills* and *Motivation* values along with *Resources* values helps create a more holistic approach to the problem in question. This is how we aimed to reflect the systems thinking concept within the tool, while *Resources* are more relevant only when comparing outcomes between attack agents of different echelons, with values of *Skills* and *Motivation* kept constant.

In terms of adversarial thinking, CyberAIMs allows the attack agents to decide if they want to attack their target opponents based on their attribute values. Thus, attack agents have full information on their opponents before taking the next step and they are able to think like their potential targets.

# 4 Study results

The target sample was composed from 12 students, attending the same course, while studying in several Masters' programs. Students had two hours of introduction to the tool developed, including the emerging concepts in cyber security curriculum related to systems and adversarial thinking. They were then asked to answer a pre-survey, followed by a scenario of a recent real-world case of a cyber attack during the experiment, and then the post-experiment survey.
## 4.1 Pre-simulation survey

We received seven surveys completed out of 12 (58.3% response rate) that will be part of the analysis below. The gender composition was two female and five male respondents. The age range of the respondents included values from 23 to 54 years old and the respondents were part of four different Masters' programs. Three students were non native, one of them being an exchange student.

Five students had a programming background, while one of them had it combined with a background in management or economics and another one had also a background in psychology or human sciences. Only one respondent had a strong military background and that was combined with a strong background in management or economics as well.

Regarding questions from the first section, six students expected the tool could help them develop their understanding of adversarial thinking and four of them agreed on the statement about systems thinking. On the other hand, only one student expected the simulation would develop his understanding on risk management.

Six students thought that the simulation would be challenging, while five of them thought they would enjoy learning with the simulation. Five students expected the simulation would build on knowledge from previous courses, while only three of them expected it would build on previous labs they attended.

In the last section, respondents answered that the defense *Motivation* was the least relevant attribute for the attack success rate, while attack *Resources* was the least relevant attribute for the defense success rate. The results show that the most relevant attributes affecting the attack success rate were *Motivation* and *Skills* of attack side and then *Resources* for the defense side. On the other side, the most relevant attributes expected to affect defense success rate were *defense Resources*, *Skills* and then *Motivation*.

Results from the first and last section were compared with results coming from the scenarios and post-simulation survey where appropriate.

## 4.2 Scenario results

The scenario was the main output of the lab conducted with the students, and we received answers from eight respondents out of 12 (66% response rate).

We prepared the scenario based on a real cyber case occurred recently, where an Iranian state-sponsored group was successful in targeting critical infrastructure entities in the US, Saudi Arabia and South Korea<fnref n="2" />.

Thus, building on this real case, we asked the students to analyze these results using the simulation tool and values mapped from the actors participating there.

First, we instructed students on using the values shown in Table 3 to define initial units of *Resources* and *Skills* for each country or entity that was part of the scenario.

<footnote n="2">² https://thehackernews.com/2017/09/apt33-iranian-hackers.html</footnote>
<table><caption>**Table 3.** Relevant values for the scenario</caption><thead><tr><th>Entity/Country</th><th>Resources</th><th>Skills</th></tr></thead><tbody><tr><td>US</td><td>98</td><td>91</td></tr><tr><td>Saudi Arabia</td><td>90</td><td>57</td></tr><tr><td>South Korea</td><td>88</td><td>78</td></tr><tr><td>Iran</td><td>84</td><td>49</td></tr><tr><td>Corporations</td><td>40-70</td><td>1-100</td></tr></tbody></table>

The values for the *Resources* are mostly related to the State agents values for each country, considering the companies were part of the critical infrastructure. Table 3 also includes values related to agents representing multinational corporations, since they could reflect same behavior with the ones targeted in the real attack. We suggested the *Skills* values above in order to think how the tool could help analyze the event explained in the scenario by using respective values from the GCI index, but we instructed the students on the best approach being to use distributed values of *Skills* and *Motivation* on both sides, so as to reflect the random distribution of values among different agents.

In the first question, we asked the students about the success rate of the defense agents involved in this scenario, whether they represented US, Saudi, Korean or corporations entities. The answers are summarized in the chart below.

<figure><img src="image_4.png" /><figcaption>**Fig. 2.** Defense Success rate (%) - Responses</figcaption></figure>

The second question asked the respondents to submit initial attribute values on both sides for at least three cases when the attack agents succeeded in their attempts to win the battle, followed by at least three cases when the defense agents succeeded in their attempt to avoid being attacked. This question was considered relevant so as to make respondents think about potential patterns drawn from the values shown here and possible correlations between attributes and the final outcome in terms of each side's success as above. It was also useful in understanding how well the respondents followed the instructions given in the first question.

Following the same logic, the third question asked respondents to rank up to three of the six defined attributes (three on each side) as the most relevant ones affecting the chances of a defense agent to survive the whole run of 120 ticks, thus
his overall success rate. The results were quite different from the same question posed in the pre-simulation survey. They showed defense *Motivation* as the most relevant attribute, followed by defense *Skills* and then attack *Motivation*. Defense *Resources* was ranked fourth overall, thus a quite different outcome from the pre-survey, while a larger sample size would provide more meaningful results on this case.

## 4.3 Post-simulation survey

We have received only four surveys completed out of 12 (33% response rate) for the post-simulation phase, while a more detailed analysis will be part of the future stages of our research.

In the first section, two respondents stated that CyberAIMs developed their understanding of economic theories in cybercrime, while the others agreed on the simulation developing their own understanding of cybercrime. Three respondents agreed on the simulation developing their understanding on strategic management of information security, and two of them stated that the simulation developed their risk management knowledge.

Three respondents agreed that the simulation developed their understanding of real world cyber scenarios. Regarding the main objective of this simulation-based experiment on learning outcomes, two out of four respondents agreed that the simulation did develop their understanding on systems thinking and, again, only two of them agreed on the statement about adversarial thinking.

All respondents thought that the simulation was challenging and that they enjoyed learning with it. Only one respondent agreed on the simulation building on knowledge from previous courses and another one agreed on the statement regarding previous labs, with the other respondents not agreeing or being neutral.

In the last section, when asked about the level of relevance of all attributes in the attack success rate, the respondents agreed that the most relevant one is attack *Motivation*. The other attributes in the top three were attack *Skills* and defense *Motivation*.

On the other hand, when asked about the most relevant attributes on the defense success rate, all respondents seemed to agree on the most relevant attribute being attack *Motivation*, followed by defense *Motivation* and then defense *Skills*.

The current results from this section, even in this preliminary stage, could define a change in perspective between the pre-study and the post-study, supported from the lab conducted using the tool.

Meanwhile, the results from the question on the respondents' engagement with the tool show that respondents spent a total time between four to five hours on learning CyberAIMs and creating useful outputs from it. Thus, somewhat between one and two regular lecture sessions were seemingly enough to change their perspective as related to systems and adversarial thinking, though a larger sample size is needed to produce statistically more significant results on this direction.
# 5 Discussion and conclusions

Only three cases were valid for the whole process from the first to the second survey out of a total of 12 students in the course (25% rate). The answers on the most relevant attributes affecting defense success rate were the only connecting dots, while Table 4 shows the different responses before and after using CyberAIMs.

<table><caption>Table 4. Most relevant attributes on the defense success rate</caption><thead><tr><th colspan="2">Pre-survey responses</th><th colspan="2">Post-survey responses</th></tr></thead><tbody><tr><td>Defense <i>Resources</i></td><td>Attack <i>Motivation</i></td><td>Defense <i>Skills</i></td><td>Defense <i>Motivation</i></td></tr><tr><td>Defense <i>Motivation</i></td><td>Defense <i>Skills</i></td><td></td><td></td></tr></tbody></table>

Overall results reflect a better understanding of systems thinking, in terms of considering as most relevant attributes *Motivation* and *Skills* instead of *Resources* of each side, along with a better understanding of adversarial thinking, while thinking of attack attributes as equal or more relevant than defense ones on the defense side performance.

## 5.1 Additional comments

The results above are prone to additional implications. Only four students were able to compute decreasing success rates of the defense side between the scenarios in the first question. According to their comments on the results, it seems that two of them were not able to follow our instructions on how to perform the analysis, while the two others could not apply them in the correct way.

The respondents further enforced these issues in their comments on the final survey, establishing a potential direction for further improvements in the whole process.

There were also useful comments received on the tool itself, including its design and the underlying features and values' distribution of the attributes, which is already incorporated in the forthcoming version of CyberAIMs, part of the future research.

## 5.2 Conclusions

This paper aims to contribute on recent research done in respect to the learning benefits of simulation tools in cyber security education. The main outcomes of our pilot study point to a shift of the respondents' perspective after using the tool, indicating that CyberAIMs can have an effect on the students' understanding of systems and adversarial thinking. The results are however preliminary, while this tool will be further improved and designed to be used for larger sample sizes of students in related programs of study and potential cyber competitions.
We are already designing another version of CyberAIMs, using another approach towards a more realistic picture of the current cyberspace, based on the work of Ablon et al. in [1]. Furthermore, we intend to look deeper into the Motivation attribute, through a more detailed literature review on the underlying theories, such as the MOMMs taxonomy [4] and the protection theory [11].

We intend to use the feedback received from the overall process in order to increase response rates and increase the usability and coverage levels of the forthcoming versions of CyberAIMs.

## References

1. Ablon, L., Libicki, M.C., Golay, A.A.: Markets for cybercrime tools and stolen data: Hackers' bazaar. Rand Corporation (2014)

2. ACM: Computer Science Curricula 2013 Curriculum Guidelines for Undergraduate Degree Programs in Computer Science. New York, NY, USA (2013)

3. Anne Bardoel, E., Haslett, T.: Success to the successful: The use of systems thinking tools in teaching ob. Organization Management Journal 1(2), 112–124 (2004)

4. Bologna, J.: Momm's (motivations, opportunities, methods, means)-a taxonomy for computer related employee theft. Assets Protection 6(3), 33–36 (1981)

5. Brahima, S.: Global cybersecurity index 2017. International Telecommunication Union (ITU) pp. 1–77 (2017)

6. Goodwin, J.S., Franklin, S.G.: The beer distribution game: using simulation to teach systems thinking. Journal of Management Development 13(8), 7–15 (1994)

7. Hamman, S.T., Hopkinson, K.M., Markham, R.L., Chaplik, A.M., Metzler, G.E.: Teaching game theory to improve adversarial thinking in cybersecurity students. IEEE Transactions on Education 60(3), 205–211 (2017)

8. Joint Task Force on Cybersecurity Education: Cybersecurity curricula 2017 - curriculum guidelines for post-secondary degree programs in cybersecurity - csec2017 v. 0.95 draft. Tech. rep. (Nov 2017)

9. Pastor, V., Díaz, G., Castro, M.: State-of-the-art simulation systems for information security education, training and awareness. In: Education Engineering (EDUCON), 2010 IEEE. pp. 1907–1916. IEEE (2010)

10. Ponemon Institute: Flipping the economics of attacks. Tech. rep. (Jan 2016)

11. Rogers, R.W.: A protection motivation theory of fear appeals and attitude change1. The journal of psychology 91(1), 93–114 (1975)

12. Schneider, F.B.: Cybersecurity education in universities. IEEE Security & Privacy 11(4), 3–4 (2013)
# Appendix

## A1. Pre-simulation Survey

Student ID number: ________

Note: this information will only be used to link your pre-simulation and post-simulation surveys and will not be retained for further analysis.

## SECTION 1: LEARNING FROM SIMULATIONS

This section aims to get information on a 5-levelled Likert scale (strongly disagree - strongly agree) basis, according to your own perceptions and expectations. The Likert scale will be replaced by numbers, as follows:

strongly disagree disagree neutral agree strongly agree

1. I expect that the simulation will develop my:

<table><thead><tr><th></th><th>-2</th><th>-1</th><th>0</th><th>1</th><th>2</th></tr></thead><tbody><tr><td>problem solving skills</td></tr><tr><td>planning skills</td></tr><tr><td>understanding of cybercrime</td></tr><tr><td>understanding of economics theories on cybercrime</td></tr><tr><td>understanding of strategic mgmt. of info. security</td></tr><tr><td>understanding of risk management</td></tr><tr><td>understanding of real-world cyber scenarios</td></tr><tr><td>understanding of systems thinking</td></tr><tr><td>understanding of adversarial thinking</td></tr></tbody></table>

2. Please rate your agreement with the following statements:

<table><thead><tr><th></th><th>-2</th><th>-1</th><th>0</th><th>1</th><th>2</th></tr></thead><tbody><tr><td>The simulation will be challenging</td></tr><tr><td>I will enjoy learning with the simulation</td></tr><tr><td>It will build on knowledge gained from previous groups</td></tr><tr><td>It will build on knowledge gained from previous labs</td></tr></tbody></table>

## SECTION 2: ABOUT YOU

This section will require some personal information from you

1. Please tell us your gender:

- Female

- Male

- Prefer not to answer

2. In what year were you born? ____

3. If you are an international student, what is your home country? ________
4. What is the name of the degree you are completing? _________________________

5. Which of the following apply to you? (Select all that apply)

- I am studying part-time

- I am studying externally (distance education)

- English is not my first language

- I am an International student

- I am working casually / part-time while studying

- I am working full-time while studying

- I am an exchange student

6. Please rate your agreement with the following statements:

<table><thead><tr><th></th><th>-2</th><th>-1</th><th>0</th><th>1</th><th>2</th></tr></thead><tbody><tr><td>I have a background in programming</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>I have a background in economics or management sciences</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>I have a background in human sciences/psychology</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>I have a background in military/warfare strategies and rules</td><td></td><td></td><td></td><td></td><td></td></tr></tbody></table>

## SECTION 3: EXPECTATIONS FROM THE MODEL

This section requires information on your perceptions and expectations on a 5-levelled Likert scale (highly irrelevant - highly relevant) basis, related to the model features explained before the lab. The Likert scale will be replaced by numbers, as follows:

<table><thead><tr><th>highly irrelevant</th><th>irrelevant</th><th>neutral</th><th>relevant</th><th>highly relevant</th></tr></thead><tbody><tr><td>0</td><td>1</td><td>2</td><td>3</td><td>4</td></tr></tbody></table>

1. Please rate the level of relevance for each attribute on the attack/defense success rate:

<table><thead><tr><th>Attack</th><th>0</th><th>1</th><th>2</th><th>3</th><th>4</th><th>Defense</th><th>0</th><th>1</th><th>2</th><th>3</th><th>4</th></tr></thead><tbody><tr><td>Defense Resources</td><td></td><td></td><td></td><td></td><td></td><td>Defense Resources</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Defense Skills</td><td></td><td></td><td></td><td></td><td></td><td>Defense Skills</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Defense Motivation</td><td></td><td></td><td></td><td></td><td></td><td>Defense Motivation</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Attack Resources</td><td></td><td></td><td></td><td></td><td></td><td>Attack Resources</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Attack Skills</td><td></td><td></td><td></td><td></td><td></td><td>Attack Skills</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Attack Motivation</td><td></td><td></td><td></td><td></td><td></td><td>Attack Motivation</td><td></td><td></td><td></td><td></td><td></td></tr></tbody></table>

2. What do you expect to be the top 3 attributes for the attack agents' success rate?

Example: a. attack resources; b. defense skills; c. defense resources;

a. _________________________

b. _________________________

c. _________________________
3. What is the rationale behind your selection above?
---------------------------------------------------------
---------------------------------------------------------
---------------------------------------------------------

4. What do you expect to be the top 3 attributes for the defense agents' success rate?

*Example: a. attack resources; b. defense skills; c. defense resources;*

a. ____________________

b. ____________________

c. ____________________

5. What is the rationale behind your selection above?
---------------------------------------------------------
---------------------------------------------------------
---------------------------------------------------------

Thank you for your participation!