<figure><img src="image_1.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Creation and Use of Virtual Simulations for Measuring Situation Awareness of Incident Commanders

Stella Polikarpus¹[0000-0002-2843-4308], Edna Milena Sarmiento-Márquez²[0000-0002-8186-9028] and Tobias Ley³[0000-0002-2804-2304]

¹ Estonian Academy of Security Sciences, Tallinn, Estonia

¹,²,³ Tallinn University, Tallinn, Estonia

stella.polikarpus@sisekaitse.ee

**Abstract.** Training and measurement of situation awareness (SA) in a dynamic decision-making context is a complex task and depends on personal factors as well as time, space, and situation. Therefore, to measure first-level incident commanders' SA several authentic and immersive virtual simulations of rescue incidents should be used. In this paper, we report on Effective Command and the dynamic decision-making assessment methodology as a theoretical framework for SA assessment and implementation of virtual reality software using sixteen different virtual simulations in an organization. First, we report four aspects needed to be implemented for virtual simulation-based assessment of SA. Next, we analysed the differences between three SA ascending levels: perception, comprehension, and prediction that might be influenced by the scenario storyline. According to prior research, we would expect the difficulty of the three SA levels to be in ascending order and we evaluated that by analysing N=665 assessment results. We confirmed such an ascending order for most scenarios, and we also identified these scenarios for which this assumption was not met. Finally, we discuss the possibilities of analysing the scenario-based SA level differences in the future when assessing SA with more automated measures. We argue that by improving the assessment of SA, we can foster the training possibilities for incident commanders' dynamic decision-making.

**Keywords:** Situation Awareness (SA), Effective Command (EC), virtual simulations, The Collaborative Authoring Process Model (CAPM).

## 1 Introduction

Rescue incident commanders need good situation awareness in a dynamic decision-making context such as fires and traffic accidents or leaks of chemicals to assess hazards and risks for humans and environment. So far it is unrealistic to measure their situational awareness in real time when they need to act in fast developing high risk situations like emergencies are. Despite the difficulties of training and assessing rescue incident commanders situation awareness is hard task for a trainers [1], it seems to be possible to do with help of virtual simulations using virtual reality technology [2]. Virtual simulations could be co-authored by trainers themselves using the *Collaborative Authoring Process Model* (CAPM), that gives step-by-step guide on how to do it in a
training organization context [1]. In The Estonian Academy of Security Sciences the Effective Command behavioural marker framework is used to assess rescue incident commanders situational awareness [3]. The framework, “focuses on five key behaviours: situational awareness, decision-making, objective setting, action behaviours and review” [4, pp. 225–226], and it is created on bases of SPAR dynamic decision-making model, meaning Situation Awareness, Plan, Action and Review [5].

The term *Situation Awareness* (SA) is common but it might mean different things to different people [6 - 7]. In this study we refer to Endsley [8, p. 65], who defines SA as a person's “perception of the elements in the environment within a volume of time and space, comprehension of their meaning, and the projection of their status in the near future”. From the definition, it becomes clear that SA levels are: SA level 1 perception (SA1); SA level 2 comprehension (SA2) and SA level 3 prediction (SA3) [9]. Endsley's definition is used as the basis to report the three levels of SA measurement, using Effective Command framework [4] in The Estonian Academy of Security Sciences [3]. It is suggested that improvements in SA2 comprehension and SA3 prediction areas would greatly impact the outcomes of dynamic decision-making assessments using virtual simulations [4]. This suggestion implies that virtual simulation-based training and assessment of rescue incident commanders' SA virtual simulations are needed to train dynamic decision-making in a safe environment without harmful consequences. We define *virtual simulations*, for this paper, as a scenario presented to the learner using software and hardware to create a virtual environment for dynamically changing emergencies. *Dynamic decision-making* could be defined “as making a series of interdependent decisions in an environment that changes over time due to the consequences of the taken decisions or due to autonomous changes in the environment” [10, p. 284]. The research on how to create these virtual simulations for rescue incident commanders' dynamic decision-making training is rather episodic and only a few articles address this issue [1, 11].

Unfortunately, from named research about creation of virtual simulations, it is not clear what are the SA level (resulting from SA assessment) when using CAPM [1] for virtual simulation scenario creation and Effective Command framework (used for simulation implementation) [4]. SA is assessed as part of the dynamic decision-making model Situation Awareness, Plan, Action and Review - SPAR in Effective Command methodology [4 - 5]. We do not know how the situation, or in other words emergency type including its dynamic situational elements (like fire, leak, traffic, victims) [1], influences the rescue incident commanders SA assessment results. There are several elements observed by assessors during simulation phase, there is a multitude of possible decisions and interactions done by incident commanders, which bring virtual simulation into a different state each time. Without knowledge about dynamic situational elements, the automated feedback to commanders' SA levels cannot be adjusted to different rescue incidents.

To the best of our knowledge, there is no research focusing on the construction of virtual simulation scenarios, except studies about CAPM. Rather, earlier research has focused on measuring the SA and its different levels [8, 12] disregarding the scenarios used for it. In this article we focus on one SA assessment method Effective Command behavioural marking framework [4]. We **aim** to find out if scenarios influence the SA
assessment results on different levels of SA. That being said, we address the following research question:

*"What are the differences in the levels of SA of rescue incident commanders as- sessed through different virtual simulation scenarios created using CAPM?"*

This paper is organised as follows: In section two we describe earlier work and the research context. In section three we explain the methods and implemented aspects. Next, we present the results by explaining the evidence supporting the use of Effective Command framework, XVR On-Scene virtual reality software and CAPM with 16 virtual simulation scenarios to assess rescue incident commanders’ SA levels in Estonia. Section five discusses the possibility to reuse the already existing assessment scenarios for the FireFront tool and the Quantitative Analysis of Situation Awareness (QASA) method in future developments, with the automating purpose of the assessments of SA based on different levels and multiple scenarios. The final section describes the limitations of the study and proposes future work directions.

## 2 Challenges in Measuring Situation Awareness

SA is a key factor in managing emergency incidents [5] and it depends on the context and personal factors [13]. As described before, the general SA levels are SA1 perception; SA2 comprehension and SA3 prediction [9]. For rescue incident commanders’ SA1– perception is about his/her ability to collect information by asking the specific question, from the right person, at the right time, and doing 360 degrees recognition of the situation effectively. SA2 – comprehension is about the commanders’ ability to collect information into a coherent operational picture. And lastly SA3 – prediction is about being able to prognose what is going to happen next in a dynamically developing rescue incident if resources are applied to stop the spread of hazards and mitigate the risks. Comprehension and prediction levels are very important for the effective use of limited resources in a timely manner.

Simulation based measurements of SA are commonly used [8] as it is almost impossible to measure SA real time when the first level incident commander has to act in fast developing high risk situations, like emergencies. In some jobs, like pilots [14], drivers [15] or teams in gas drilling industry [16], individuals work in a predetermined space (cockpit, car or drilling platform). On the contrary, rescue incident commanders need to work in any space, weather, with hundreds of tools and uncountable situations. Incident commanders are the first ones to respond to emergencies like traffic accidents, fires, or incidents with hazardous materials. They need a good level of SA for safe and fast decision making but have very limited time in a novel space to create their perception of the elements in an emergency environment like a fire or spread of hazardous materials, the health status of victims etc. For SA2 these named examples of dynamic situation elements must be put together to have an operational picture with the comprehension of their meaning. Finally, the projection of their status in the near future is needed for a dynamic decision making. Therefore, virtual simulations are useful for assessing first level rescue incident commanders’ SA [3] and enhancing the training of dynamic decision-making.
"The direct measurement of SA provides a great insight into how operators piece together the vast array of available information to form a coherent operational picture" [9, p. 17]. There are several methods to measure SA. Examples include the SART – situational awareness rating technique, the SAGAT – situation awareness global assessment technique, the SPAM - Situation Present Assessment Method [14], and the QASA - Quantitative Analysis of Situation Awareness method [12] that is used in the FireFront tool [17]. Effective Command methodology assesses key behavioural markers through criteria for dynamic decision-making [4]. Three levels of SA are assessed by two assessors, using virtual reality, based on observations of the dynamic phase of incident response and after action discussion with the trainee [3]. From earlier research there is an understanding that Effective Command methodology allows to measure different SA levels [3, 18], especially SA3 prediction, because the virtual scenario is changed dynamically by the assessors, based on the incident commander's SA1 data gathering on scene. The Effective Command methodology to measure incident commanders' SA is highly mental workload demanding for assessors. The field would benefit from automating some of the actions taken by assessors in a timely manner.

It is claimed that scenario storyline and dynamic situational elements like victims, fire, leak and traffic play an important role in SA assessments [1]. Similarly for pilots, elements like altitude, mission timing and aircraft manoeuvres play an important role in SA [13]. Endsley has divided these elements to measure different SA levels of pilots [13], but we have no information that such work has been done, so far, for rescue incident commanders' SA measurement. Therefore, the data from incident commanders' SA levels, connected to different rescue incidents, and dynamic situational elements used in virtual simulations, is essential to develop automatized SA measuring methods.

# 3 Methods

## *Approach and Procedure*

To develop virtual simulation scenarios and implement SA assessments of rescue incident commanders, we need to consider four main aspects:

1) Theoretical model to assess SA – we used Effective Command behavioural marking framework [4].

2) Platform – we used virtual reality software XVR On-Scene [19] and Effective Command web-based database [20].

3) Process model to create virtual simulations – we used The Collaborative Authoring Process Model (CAPM) [1].

4) Assessors – we selected only assessors who are certified to use the Effective Command methodology [21].

In The Estonian Academy of Security Sciences, we use data from year 2016 Effective Command behavioural marker framework [4, 21]. Effective Command methodology is based on dynamic decision-making model SPAR, it is based on research about incident commanders decision-making processes in urban fires [5]. The Effective Command methodology five key-behaviours assessment starts with three levels of SA is circular, meaning that the incident commander updates his/her SA throughout the
response phase of an emergency. Therefore, the virtual simulations need to be adoptable
to incident commanders' information collection. There are different decision-making
strategies defined, like recognition primed, value-based, procedural that incident com-
manders' use in the first stage of response [5]. Based on the initial SA, the commander
creates a plan to execute actions with a clear goal in mind, to save lives and property,
by only taking calculated risks to firefighters. He/she needs to review the emergency
response (maintaining the SA) and his/her plan to check if the plan is still feasible to
execute with the resources at scene and the current development of the situation. Be-
cause the SPAR model is a spiral, the review phase plays an important part to keep SA
constantly updated.

The second aspect: platforms to create virtual simulations for rescue incident com-
manders can be different. Nevertheless, to train all three levels of SA, XVR On-Scene
virtual reality software has been selected to create virtual simulations [2] and execute
the Effective Command criteria based assessments of SA levels. Effective Command
methodology web-based platform has eight subsections plus section for written feed-
back translated and adjusted to the Estonian context [21]. To assess SA using Effective
Command methodology three subsections for each SA level needs to be considered as
– SA1 perception, SA2 comprehension, and SA3 prediction. All subsections are meas-
ured with nine assessment criteria, based on a five-point coloured scale by two assessors
[3]. Each section was evaluated on a scale from 20 to 100 points with a threshold above
55,5 points. The score below 55,5 points is considered as fail [4]. The SA assessments
of working incident commanders in Estonia and recordings, in the form of SA formal
assessment results into Effective Command database, are done systematically and pe-
riodically by certified assessors [3, 21].

Thirdly, there is a five steps process model that explains how to build these virtual
simulations for Effective Command methodology implementation, The Collaborative
Authoring Process Model for Virtual Simulation Scenarios - CAPM [1]. First, the inci-
dent commander's virtual simulation scenario map needs to be written to communicate
the emergency type, main storyline, dilemmas for decision-making etc. The map is used
to find a co-author as well as for contracting the virtual simulations build. The second
step is scenario authoring, where the storyline is further developed in a document called
"the scenario's user manual", then an XVR On-Scene file is created using the organi-
sational virtual scenario building guidelines. Third, virtual simulations scenario tech-
nical testing appears between four Effective Command assessors. The virtual simula-
tion is played through by an expert, who has not seen it before, to test how authentic
and immersive the virtual simulation is. Fourth, the virtual scenario needs to be final-
ized based on improvements suggested in step three. Finally, "the modelling of the
threshold playthrough" is to all Effective Command assessors in an organization to
learn to use the virtual simulation and to make sure that the incident, in the form of
virtual simulation, is presented similarly to all incident commanders. The implementa-
tion of CAPM [1] allows different scenarios to be used by assessors in a systematic way
and in controlled conditions. This process model has already been adopted for the Fire-
Front tool [11] and the Quantitative Analysis of Situation Awareness (QASA) method
as well [12]. The Effective Command methodology implementation analyses in The
Estonian Academy of Security Sciences showed that in 2018 only seven different
scenarios were used, 14 scenarios in 2019, and 17 in 2020 [21]. The researchers suggested that using the CAPM allows a stable scenario creation and use for dynamic decision-making assessments [21].

The fourth aspect to implement virtual simulations-based SA assessments in an organization is to train and certify assessors. The assessors will carry out the assessments in a way that they adopt the scenario based on incident commander local recourses and change the virtual simulations-based on commander's information collection, decisions, and actions. Assessors are selected among people who have first level incident command work experience and are trained based on the curriculum where the Effective Command methodology and XVR On-Scene are thought [21]. Training the assessors in an organization is important to avoid assessment biases. The assessors are the ones who need to be able to adopt the scenario storyline to provide context to the commander being measured on all three levels of SA. Once the dynamic phase is played through in virtual simulation, they discuss the dynamic decision-making phase and give quantitative and qualitative feedback to the case on Effective Command platform [4].

## Data Collection

We want to find out whether the created virtual simulation, using CAPM, influence the SA assessment results on different levels. To use Effective Command methodology in the Estonian Academy of Security Sciences, two assessors per each incident commander are needed, and it takes around 90 min to play one scenario [3, 21]. In Estonia, the results from a working first level incident commanders' assessment were downloaded from Effective Command platform [20] on 1st July 2022. The details of assessment methodology and assessors training can be found in earlier work [3, 21]. Year 2016 and 2017, or any data having typing errors or missing data, were removed. All scenario codes being marked by assessors to the database as "Incident No", which had less than ten assessment results were deleted as well. We ended up with data from 665 individual assessments in January 2018 to July 2022.

Database download was done directly to MS Excel file. All qualitative feedback was erased as well with columns not relevant to this study like name of the assessor. Pre-work with data was registered on a table where each rescue incident commanders' assessment results were in one row. The titles for columns are to be found in the first row of data table. First column was named as "name" and second column "personnel No" of person being assessed. Followed by incident number where the scenario codes were retrieved. The fourth column was date of the assessment, the fifth was the name of the assessor who filled the certificate. Columns six to fourteen were the scores of the person assessed (Perception_(SA1); Comprehension_(SA2); Prediction_(SA3); Decision Making; Plan; Communication; Command; Review).

In The Estonian Academy of Security Sciences, first level incident commanders have been assessed from year 2016, however the CAPM was implemented in year 2018. During the time January 2018 to July 2022, 665 assessments were carried out, using sixteen different virtual scenarios which were created using CAPM. Effective Command behavioural markers assessment results (mean scores) are presented on Fig. 1 next section. There have been 21 different certified assessors logged in to fulfil the certificate, the leading assessor did 224 assessments, and four assessors did two or less
assessments. We remind the reader that assessors always worked in pairs [21] and thereby only one assessor logs into Effective Command platform to complete the certificate.

## Data Analysis Method

We used MS Excel to extract descriptive statistics like mean scores, standard deviation, etc. We also used it to assess the normality of the data, and to perform single factor ANOVA analyses and post-hoc T-tests. For scenario-based analyses, the filter function was used, in column Incident No, allowing us to see how many times same virtual simulation has been used to assess different rescue incident commanders' SA levels.

## 4 Results

All eight subsection scores of the Effective Command could be, theoretically, between 20 and 100 points. In Fig. 1 we have calculated the mean scores for each subsection over all 665 assessments. The lowest result from the eight behaviours (see Fig. 1) is *review* (M=63,9; SD =7,4), followed by *SA 3* with highest standard deviation (M=64,3; SD =8,1), followed by means score for *communicational* skills 64,6 (SD=7,5). In ascending position, 4th is *decision-making* skills (marked decision, M=65,2; SD=8), 5th is *command and control* skill (M=65,6; SD=7,6), and 6th is *plan* (M=65,7; SD=7,5), while the higher two are *SA1* with means score 67,0 and SD=7,2 and *SA2* (M=66,3; SD=7,6).

<figure><img src="image_2.png" /><figcaption>**Fig. 1.** Effective Command Behavioural Markers Mean Scores in Years 2018-2022 when First Level Rescue Incident Commanders were Assessed.</figcaption></figure>

All eight behavioural markers mean scores for all 16 scenarios, as well standard deviation (SD) and minimum and maximum results, range, and absolute value for
asymmetry (Skewness) and Kurtosis are given in Table 1. It turns out that the decision-making has the highest negative Kurtosis (-0,7) meaning the distribution is platykurtic (relatively thin and light tails, tends to be flatter than normal) [22]. However, all behaviour scores can be seen with normal distribution as the Kurtosis and Skewness are ±2 [23].

<table><caption>**Table 1.** Effective Command Behavioural Marking Framework Assessment Results Descriptive Statistics (N=665).</caption><thead><tr><th>Measurement</th><th>SA1</th><th>SA2</th><th>SA3</th><th>Decision</th><th>Plan</th><th>Communication</th><th>Command</th><th>Review</th></tr></thead><tbody><tr><td>Mean</td><td>67,0</td><td>66,3</td><td>64,3</td><td>65,2</td><td>65,7</td><td>64,6</td><td>65,6</td><td>63,9</td></tr><tr><td>Standard Deviation</td><td>7,2</td><td>7,6</td><td>8,1</td><td>8,0</td><td>7,5</td><td>7,5</td><td>7,6</td><td>7,4</td></tr><tr><td>Kurtosis</td><td>-0,5</td><td>0,1</td><td>-0,3</td><td>-0,7</td><td>0,1</td><td>-0,6</td><td>-0,4</td><td>0,0</td></tr><tr><td>Skewness</td><td>-0,3</td><td>-0,3</td><td>0,0</td><td>0,0</td><td>-0,3</td><td>0,2</td><td>0,1</td><td>0,1</td></tr><tr><td>Range</td><td>44</td><td>55</td><td>48</td><td>46</td><td>51</td><td>45</td><td>51</td><td>51</td></tr><tr><td>Minimum</td><td>42</td><td>31</td><td>40</td><td>40</td><td>35</td><td>37</td><td>40</td><td>33</td></tr><tr><td>Maximum</td><td>86</td><td>86</td><td>88</td><td>86</td><td>86</td><td>82</td><td>91</td><td>84</td></tr><tr><td>Total</td><td>665</td><td>665</td><td>665</td><td>665</td><td>665</td><td>665</td><td>665</td><td>665</td></tr></tbody></table>

Next, we checked if all 16 scenarios results had a normal distribution when SA level mean scores are used. In Table 2, the mean score for all virtual simulations, and overall SA levels, is 65,9 points which is 10,4 points higher than the threshold (55,5 points). Standard deviation was 6,9 points while the minimum result for SA three levels average was 38,3 and maximum 81,7 points. The highest absolute value for asymmetry (Skewness) is 0,7 for Scenario TM118 and Kurtosis is 1.3 for same scenario and scenario RT118 has kurtosis -1.3. In order to prove normal univariate distribution, the values for asymmetry and Kurtosis between -2 and +2 are considered acceptable [23], therefore we conclude that all scenarios used with Effective Command methodology still had normal distribution.

In Fig. 2 we present the SA average scores for all 16 scenarios as well each SA level score. The least used scenario AT121 (15 times) is at the top of Fig. 2 and most used scenario AS118 (78 times) is placed at the bottom. The general trend is that SA1 is highest, while SA3 is the lowest. There are some exceptions like scenario with the code JM218, where SA2: comprehension, is lower than SA3: prediction. AS120 and TM118 has SA2 average score lower than SA1, perception. A similar situation is with SP121, but only with a 0,1-point difference and AT118 had SA1 and SA2 with the same mean score. So out of 16 cases, we see that in five cases there are some differences from the general trend.
<table><caption>**Table 2.** Estonian First Level Incident Commanders Effective Command Situation Awareness Formal Assessment Results in Years 2018-2022 based on CAPM Virtual Simulations.</caption><thead><tr><th>No</th><th>Scenario code</th><th>Used (n)</th><th>Mean SA score</th><th>SD</th><th>Skew-ness</th><th>Kurto-sis</th><th>Min</th><th>Max</th></tr></thead><tbody><tr><td>1</td><td>AS118</td><td>78</td><td>65,7</td><td>6,3</td><td>-0,2</td><td>-0,3</td><td>50,7</td><td>81,7</td></tr><tr><td>2</td><td>AT118</td><td>62</td><td>65,2</td><td>6,6</td><td>0,1</td><td>-0,8</td><td>52,0</td><td>80,0</td></tr><tr><td>3</td><td>RT118</td><td>60</td><td>64,8</td><td>8,0</td><td>0,1</td><td>-1,3</td><td>51,7</td><td>79,0</td></tr><tr><td>4</td><td>LK118</td><td>57</td><td>65,7</td><td>6,4</td><td>-0,3</td><td>-0,4</td><td>50,3</td><td>77,7</td></tr><tr><td>5</td><td>JM118</td><td>56</td><td>66,2</td><td>6,2</td><td>-0,2</td><td>-0,8</td><td>52,0</td><td>77,3</td></tr><tr><td>6</td><td>TM118</td><td>56</td><td>65,6</td><td>8,4</td><td>-0,7</td><td>1,3</td><td>38,3</td><td>80,3</td></tr><tr><td>7</td><td>SP121</td><td>48</td><td>64,7</td><td>5,3</td><td>-0,5</td><td>-0,5</td><td>51,3</td><td>72,3</td></tr><tr><td>8</td><td>TR218</td><td>48</td><td>64,2</td><td>6,9</td><td>0,3</td><td>-0,6</td><td>52,7</td><td>79,0</td></tr><tr><td>9</td><td>AT119</td><td>41</td><td>68,9</td><td>4,8</td><td>0,0</td><td>-0,3</td><td>58,7</td><td>80,0</td></tr><tr><td>10</td><td>AS120</td><td>30</td><td>67,8</td><td>7,6</td><td>0,0</td><td>-1,2</td><td>54,7</td><td>81,7</td></tr><tr><td>11</td><td>TM218</td><td>28</td><td>68,6</td><td>6,9</td><td>0,1</td><td>-1,0</td><td>55,7</td><td>80,7</td></tr><tr><td>12</td><td>JM218</td><td>25</td><td>62,8</td><td>6,0</td><td>0,3</td><td>-1,0</td><td>55,0</td><td>74,3</td></tr><tr><td>13</td><td>TM318</td><td>24</td><td>68,0</td><td>6,4</td><td>-0,6</td><td>-0,2</td><td>53,7</td><td>78,3</td></tr><tr><td>14</td><td>JM121</td><td>21</td><td>66,0</td><td>7,0</td><td>-0,4</td><td>-1,2</td><td>53,7</td><td>75,7</td></tr><tr><td>15</td><td>TM220</td><td>16</td><td>65,0</td><td>6,9</td><td>-0,3</td><td>0,8</td><td>49,3</td><td>78,3</td></tr><tr><td>16</td><td>AT121</td><td>15</td><td>69,0</td><td>8,0</td><td>-0,5</td><td>-0,6</td><td>55,0</td><td>81,3</td></tr><tr><td></td><td>Total</td><td>665</td><td>65,9</td><td>6,9</td><td>-0,2</td><td>-0,3</td><td>38,3</td><td>81,7</td></tr></tbody></table>

To understand the differences between SA levels based on the different scenarios, we performed a single factor (one way) ANOVA. Based on earlier research [3] we hypothesized that if all 665 assessment results are considered, there is a statistically significant difference between SA levels. Results indicate that there is a statistically significant difference between SA levels scores (F=21,9 *p*=< 0,001). In Table 3 the post hoc T-test results are presented. The overall trend is that SA1 is highest, SA3 is lowest, and SA2 in between. SA level perception and prediction (Cohen's d = 0,41) as well comprehension and prediction (Cohen's d = 0,39) means differences has a small effect but perception and comprehension (Cohen's d = 0,11) has a trivial effect [24].

<table><caption>**Table 3.** Post-hoc Paired Sample T-Test Results for Situation Awareness Levels Using All CAPM Scenarios Data</caption><thead><tr><th>SA level</th><th>SA level</th><th>t</th><th>df</th><th>p</th><th>Cohen's d</th></tr></thead><tbody><tr><td>Perception -</td><td>Comprehension</td><td>2,75</td><td>664</td><td>0,006</td><td>0,11</td></tr><tr><td>Perception -</td><td>Prediction</td><td>10,51</td><td>664</td><td>&lt; ,001</td><td>0,41</td></tr><tr><td>Comprehension -</td><td>Prediction</td><td>9,95</td><td>664</td><td>&lt; ,001</td><td>0,39</td></tr></tbody></table>

As from Fig. 2, we identified some differences in the order of SA levels mean scores. We also did single factor ANOVA for each scenario. Two cases, AS118 (F=7,67; *p* < 0,001) and AT119 (F=4,02; *p*=0,02) the differences between SA levels are statistically important. It is important to note that these two scenarios are not the scenarios that did not follow the overall trend of SA level scores.
<figure><img src="image_3.png" /><figcaption>**Fig. 2.** CAPM Scenarios Situation Awareness Levels Mean Scores when Effective Command Methodology is Used (n=665)</figcaption></figure>
Therefore, we can say that Effective Command SA assessment methodology gener-
ally allows us to understand whether the three SA levels, as mean scores, are in a de-
scending sequence. Despite the exceptional scenarios (AT118, SP121, TM118, AS120,
JM218), that did not produce statistically significant differences.

To find out why some scenarios mean scores did not follow the SA levels descending rule, we counted how many individual cases did not follow the logical sequence of level scores. Table 4 presents the results of the individual SA levels’ assessment order or if there is no difference between levels scores.

<table><caption>**Table 4.** Comparison of Individuals Situation Awareness Levels Order</caption><thead><tr><td>SA level</td><td>Individual scores (n)</td><td>Comment to order of SA level scores</td></tr></thead><tbody><tr><td>SA1=SA2</td><td>133</td><td>no difference in SA levels scores</td></tr><tr><td>SA1>SA2</td><td>298</td><td>expected order</td></tr><tr><td>SA1<SA2</td><td>234</td><td>not expected order</td></tr><tr><td>SA2=SA3</td><td>155</td><td>no difference in SA levels scores</td></tr><tr><td>SA2>SA3</td><td>358</td><td>expected order</td></tr><tr><td>SA2<SA3</td><td>152</td><td>not expected order</td></tr><tr><td>SA1=SA3</td><td>126</td><td>no difference in SA levels scores</td></tr><tr><td>SA1>SA3</td><td>373</td><td>expected order</td></tr><tr><td>SA1<SA3</td><td>166</td><td>not expected order</td></tr></tbody></table>

In Table 4 we observe that all SA levels there are more individuals whose SA levels were in the expected order. We expected SA1 to have higher score than SA2 and SA2 have higher score than SA3. There was no difference between SA1 and SA2 20% of cases, 23% between SA2 and SA3 and 19% between SA1 and SA3. However, in 234 cases there was not expected meaning SA2 comprehension score being higher than SA1 perception score. In 152 cases SA2 was lower than SA3 and 166 cases SA1 was lower than SA3.

**5 Discussion**

We found 665 assessments in The Estonian Academy of Security Sciences, made by 21 assessors working in pairs from the years 2018 to July 2022. We assessed eight dynamic decision-making behaviours using Effective Command methodology, and the two lowest scores are review and SA3 (Table 1). Both, review and SA3 behaviours are connected to individual information mechanisms that are connected to abilities, experiences, and training [13]. Therefore, there is a need for more possibilities to train and assess SA in a dynamic decision-making context for rescue incident commanders. Furthermore, all dynamic decision-making behaviours had a normal distribution (Table 1), allowing us to conclude that data collected in the training organisation using virtual
simulations, has good potential to be used for learning analytics and further statistical analyses.

In our research, we asked what the differences in the levels of SA of rescue incident commanders are if being assessed through different CAPM virtual simulation scenarios. The general trend is that SA1 scores are highest and SA3 are lowest while SA2 in the middle. The same order is found earlier [3] even if not scenarios used to assess SA were created using CAPM. All mean scores are over threshold SA1 perception 67,0, SA2 comprehension 66,3 and SA3 prediction 64,3 points. There is a statistically significant difference between the SA level scores (see Table 3) and rescue incident commanders score higher in SA1 compared to SA3. However, the effect size is small (also Table 3) indicating that adding more people to be assessed using the same virtual simulation, even small differences in SA levels means might become statistically important.

Effective Command methodology together with implementation of CAPM has allowed to assess whether all SA levels and scenario mean scores are in a descending order. The results showed that SA1 has in general higher scores as, from a cognitive point of view, it is easier to collect the information (SA1) than organizing it into a coherent operational picture (SA2). It is even harder to predict the future of the situation in emergency events (SA3) while being under great stress to save lives and property. In other words Endsley states: "the three levels of SA represent ascending levels of SA"[25, p. 8] and therefore the drop of scores in SA levels is well explained with the theoretical model chosen to assess SA.

There are 16 different scenarios constructed adopting CAPM, with the most used scenario having been solved 78 times (see Table 2). The results from this study confirm that the scenario storylines affect SA scores. Similarly to earlier research we suggest applying a systematic approach to author virtual simulations for training and assessing SA in dynamic decision-making contexts [1]. A systematic approach using CAPM will help to clarify the effect of different variables in measuring SA.

Furthermore, to detect how the personal factors influence to SA levels, more individual SA assessments with different virtual simulations need to be carried out. SA depends on space and situation and these dynamical situation elements for different SA levels need to be considered while training and assessing rescue incident commanders' SA levels. In order to find a way to measure SA using fewer human resources, a larger automation is needed. The FireFront is a fully automated way to measure SA and FireFront tool [17], it also allows to distinguish actual SA and perceived SA as well as situation understanding if CAPM is used to create the scenarios for it [11]. The FireFront tool considers time and individual factors while using Quantitative Analysis of Situation Awareness (QASA) measuring method [12]. Earlier research measuring SA using QASA suggested that firefighters maintained a level of confidence in their own SA across two virtual scenarios that were used, but that confidence is unrelated to their actual SA [26]. We agree with authors as they write "In particular, it would be valuable to know whether the actual SA, in any particular situation, is influenced by the resting perceived SA" [26 p. 1481]. But to determine what a rescue incident commander needs to solve very different kinds of incidents to reduce the influence of the virtual simulation storylines (emergency/incident type) on the SA measurement.
# 6 Limitations and Future Work

As the scenario-based analyses (see Fig. 2) showed that the scenario storyline might be accountable for differences in commanders’ SA levels, further research is needed. This interesting finding – virtual simulations influence to the SA levels scores - needs to be further explored as in most research cases SA is measured just using one or two virtual simulations [12], [26]. The analysis of individual SA levels (see Table 4) showed that SA does depend on individual factors. Perhaps not all virtual simulations can pick up the SA levels well. Scenario based differences might be explained with timeline and order of presenting the dynamic situation elements like fire, victims, traffic etc. Therefore, future work should focus on this issue.

Furthermore, future research could focus on identifying the role that individual differences play in SA when the same virtual simulation is used for all incident commanders in Estonia, also by using statements asking about how different situational elements are clearly divided between SA levels. This study can be considered as a departure point to explore the factors affecting SA levels and finding strategies to improve the systematization of SA assessment as well as training dynamic decision-making.

Future work using the CAPM for scenario creation and XVR On-Scene after training output files and Effective Command database could be analysed to improve the learning analytics for virtual simulations-based training and assessment. Therefore, key factors that play a role for each SA level (dynamic situation elements like victims, fire, traffic etc.) while working incident commanders respond to virtually simulated emergencies should be identified from automated analyses of XVR On-Scene after action files to each incident type. This information could be used to automatically propose statements for Quantitative Analysis of Situation Awareness - QASA method and FireFront tool. Proposed work could support assessors and trainers as well to provide even more immersive training of dynamic decision-making using the Effective Command methodology.

The main limitation of the study is that even if the statistically significant order of SA levels was identified, the small differences in SA level mean scores do not play a practical role for incident commander overall SA (also the effect size is small or trivial). As SA assessment is such a complex task, it is very hard to detect if this difference results from personal factors or time, space, or situation (which in the context of virtual simulations-based training is called scenario storyline or incident type). Therefore, the future SA research effort should focus on providing more training possibilities including automated feedback, delivering virtual simulations for incident commanders.

This study did not consider individual differences or additional cognitive aspects related to the participants and future research can also address this matter. In addition to this, only quantitative data was used, a qualitative approach could explain what differences virtual simulations storylines have for SA levels.
# 7 Conclusion

This study presented the benefits of a strategy to use systematic approach for virtual simulations scenario creation process to automatize in future the SA measuring. It highlighted the experiences and systematic approach in The Estonian Academy of Security Sciences while laboursome assessment SA levels (665 times with 16 scenarios). With laboursome we mean that two certified assessors assess each rescue incident commander separately while adopting the virtual simulations. As virtual simulations are used with Effective Command the valuable information about SA that is depending on time, space, situation, and personal factors could be better recorded. Therefore, it is feasible to automate some of the work of assessors and to make learning analytics based on scenarios to improve the SA training possibilities with automated feedback in the future.

In this study we showed how using virtual simulations and Effective Command methodology as an assessment method for SA could be used to the different SA method: Quantitative Analysis of Situation Awareness (QASA). Four aspects: SA measurement theoretical framework, platforms, CAPM and assessors are needed to successfully implement the virtual simulations-based dynamic decision-making training in an organization. We conclude that future research on incident commanders' SA measurement topic using virtual simulations to assess different SA levels, is still very much needed. More research is required mainly for two practical reasons. First, to provide more individual and fully automated ways for incident commanders to train their SA in different incident types and to measure their actual and perceived SA. Second, to support assessors work during use of CAPM and the dynamic phase of Effective Command methodology using XVR On-Scene software.

We encourage the scientific community to further develop the virtual simulation platforms with automated learning analytics. We suggest trainers to use CAPM to create the scenarios for virtual simulations based dynamic decision-making training and assessment.

# 8 Acknowledgements

We would like to thank the Estonian Rescue Board and The Estonian Academy of Security Sciences for continues funding of rescue incidents training and SA assessments. We struggle to express our profound thankfulness for all the work of the assessors doing the assessments as well building new assessment scenarios using CAPM. Thank you, Katherine Lamb's, for your yearly efforts in training and certifying assessors.

# References

[1] S. Polikarpus and T. Ley, "Collaborative Authoring of Virtual Simulation Scenarios for Assessing Situational Awareness," in *Proceedings of the 18th ISCRAM Conference*,
2021, pp. 229–237, [Online]. Available: https://www.drrm.fralinlifesci.vt.edu/iscram2021/files/ISCRAM2021Proceedings.pdf.

[2] S. Polikarpus, M. Bøhm, and T. Ley, “Training Incident Commander’s Situational Awareness—A Discussion of How Simulation Software Facilitate Learning,” in *Digital Turn in Schools—Research, Policy, Practice*, T. Väljataga and M. Laanpere, Eds. Singapore: Springer, Singapore, 2019, pp. 219–234.

[3] S. Polikarpus, T. Ley, and K. Poom-Valickis, “Developing the Situational Awareness of Incident Commanders: Evaluating a Training Programme using a Virtual Simulation,” *Proc. Est. Acad. Secur. Sci.*, vol. 19, pp. 195–226, 2020, [Online]. Available: https://digiriiul.sisekaitse.ee/bitstream/handle/123456789/2595/Proceedings_2020_WEB.PDF?sequence=5&isAllowed=y.

[4] K. Lamb, M. Farrow, C. Olymbios, D. Launder, and I. Greatbatch, “Systematic incident command training and organisational competence,” *Int. J. Emerg. Serv.*, vol. 10, no. 2, pp. 222–234, Jun. 2021, doi: 10.1108/IJES-05-2020-0029.

[5] D. Launder and C. Perry, “A study identifying factors influencing decision making in dynamic emergencies like urban fire and rescue settings,” *Int. J. Emerg. Serv.*, vol. 3, no. 2, pp. 144–161, Oct. 2014, doi: 10.1108/IJES-06-2013-0016.

[6] D. G. Jones, “A Practical Perspective on the Utility of Situation Awareness,” *J. Cogn. Eng. Decis. Mak.*, vol. 9, no. 1, pp. 98–100, Mar. 2015, doi: 10.1177/1555343414554804.

[7] D. P. D. Jenkins, G. H. D. Walker, and N. A. P. Stanton, “What really is going on? Situation awareness literature review,” in *Distributed Situation Awareness: Theory, Measurement and Application to Teamwork*, no. 2009, Ashgate Publishing Ltd, 2009, pp. 7–34.

[8] M. R. Endsley, “Measurement of Situation Awareness in Dynamic Systems,” *Hum. Factors J. Hum. Factors Ergon. Soc.*, vol. 37, no. 1, pp. 65–84, Mar. 1995, doi: 10.1518/001872095779049499.

[9] M. R. Endsley, “Theoretical Underpinnings Of Situation Awareness: A Critical Review,” in *Situation Awareness Analysis and Measurement*, M. R. Endsley and D. J. Garland, Eds. Lawrence Erlbaum Associates Publishers, 2000.

[10] S. J. Donovan, C. D. Güss, and D. Naslund, “Improving dynamic decision making through training and self-reflection,” *Judgm. Decis. Mak.*, vol. 10, no. 4, pp. 284–295, 2015. Accessed: Jun. 07, 2020. [Online]. Available: http://eds.b.ebscohost.com.ezproxy.tlu.ee/eds/pdfviewer/pdfviewer?vid=0&sid=84c5805a-a737-4197-873e-ab8edd1d60de%2540pdc-v-sessmgr03.

[11] S. Polikarpus et al., “Authoring virtual simulations to measure situation awareness and understanding,” in *ISCRAM 2022 Conference Proceedings – 19th International Conference on Information Systems for Crisis Response and Management*, 2022, pp. 428–433. [Online]. Available: https://idl.iscram.org/files/stellapolikarpus/2022/2430_StellaPolikarpus_etal2022.pdf.

[12] G. Edgar et al., “Quantitative Analysis of Situation Awareness (QASA): modelling and measuring situation awareness using signal detection theory,” *Ergonomics*, vol. 61, no. 6, pp. 762–777, Jun. 2018, doi: 10.1080/00140139.2017.1420238.

[13] M. R. Endsley, “Toward a Theory of Situation Awareness in Dynamic Systems,” *Hum.*
*Factors J. Hum. Factors Ergon. Soc.*, vol. 37, no. 1, pp. 32–64, Mar. 1995, doi: 10.1518/001872095779049543.

[14] I. Dalinger, M. Smurov, N. Sukhikh, and E. Tsybova, "Pilot's Situational Awareness and Methods of its Assessment," *Indian J. Sci. Technol.*, vol. 9, no. 46, Dec. 2016, doi: 10.17485/ijst/2016/v9i46/107534.

[15] A. Gugliotta et al., "Are situation awareness and decision-making in driving totally conscious processes? Results of a hazard prediction task," *Transp. Res. Part F Traffic Psychol. Behav.*, vol. 44, pp. 168–179, 2017, doi: 10.1016/j.trf.2016.11.005.

[16] M. T. Crichton, "Improving team effectiveness using tactical decision games," *Saf. Sci.*, vol. 47, no. 3, pp. 330–336, 2009, doi: 10.1016/j.ssci.2008.07.036.

[17] F. Thoelen et al., "FireFront: A new tool to support training in Fireground Situation Awareness, Situation Understanding and Bias," *Int. Fire Prof.*, no. 34, pp. 34–39, 2020.

[18] S. Polikarpus and K. Danilas, "Olukorrateadlikkuse, olukorramõistmise ja kallutatuse mõõtmine siseturvalisuse valdkonnas," *Turvalisuskompass*, vol. 2, no. 1, pp. 89–107, 2022, [Online]. Available: https://digiriiul.sisekaitse.ee/bitstream/handle/123456789/2876/Turvalisuskompass 2022-1.pdf?sequence=6&isAllowed=y.

[19] XVR Simulation, "XVR Virtual Reality training software for safety and security," 2022. http://www.xvrsim.com/ (accessed Mar. 29, 2018).

[20] Effective Command, "Effective Command," *Effective Command*, 2022. https://www.effectivecommand.org/.

[21] S. Polikarpus and K. Danilas, "Eesti päästemeeskonna juhtide visiõppepõhise hindamise rakendamine ja tulemused," *Turvalisuskompass*, vol. 1, p. 204, 2021, [Online]. Available: https://digiriiul.sisekaitse.ee/handle/123456789/2844.

[22] A. Field, *Discovering Statistics Using SPSS*. 2009.

[23] D. George and M. Mallery, *SPSS for Windows Step by Step: A Simple Guide and Reference*, 17.0 updat. Boston: Pearson: Pearson, 2010.

[24] M. A. Goss-Sampson, *Statistical analysis in JASP a guide for Students*. 2020.

[25] M. R. Endsley, "Situation Awareness Misconceptions and Misunderstandings," *J. Cogn. Eng. Decis. Mak.*, vol. 9, no. 1, pp. 4–32, Mar. 2015, doi: 10.1177/1555343415572631.

[26] E. Graham, D. Catherwood, G. Sallis, D. Brookes, and A. Medley, "I always know what's going on.' Assessing the Relationship between Perceived and Actual Situation Awareness across Different Scenarios," *World Acad. Sci. Eng. Technol.*, vol. 6, no. 11, pp. 1433–1434, 2012.