# Feedback in a Training Simulator for Crisis Management Compared to Feedback in a Real-Life Exercise

Olga Druzhinina, Ebba Thora Hvannberg

► To cite this version:

Olga Druzhinina, Ebba Thora Hvannberg. Feedback in a Training Simulator for Crisis Management Compared to Feedback in a Real-Life Exercise. 3rd Human Work Interaction Design (HWID), Dec 2012, Copenhagen, Denmark. pp.124-138, 10.1007/978-3-642-41145-8_11. hal-01463368

HAL Id: hal-01463368

https://inria.hal.science/hal-01463368

Submitted on 9 Feb 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Feedback in a Training Simulator for Crisis Management Compared to Feedback in a Real-Life Exercise

Olga Druzhinina

University of Iceland
Dunhagi 5, 107 Reykjavik, Iceland

olgaromanovna86@gmail.com

Ebba Thora Hvannberg

University of Iceland
Dunhagi 5, 107 Reykjavik, Iceland

ebba@hi.is

**Abstract.** Designing feedback to trainees in a training simulator for complex cognitive domains is demanding but has not been given adequate attention. This paper aims to understand the importance of early work analysis in a real context during the design of such a simulator. More specifically, the aim of this research was to learn whether there were differences between the types of feedback given to the learner in an already planned design and a real-life training exercise. Therefore, a comparison between feedback given to trainees in a real-life training exercise and feedback planned for a virtual environment supporting training of crisis management was made. The results showed that there were several significant differences.

**Keywords:** Feedback, work analysis, human computer interaction, interaction design, virtual environment, training simulator, crisis management training

## 1 Introduction

While training for crisis management in the advent or after an airplane crash is vital, it is complex and resource demanding, involving many organisations such as medical, police, rescue, airlines and airport operators. The tasks of operational units involve field work, such as medical aid, transportation of casualties and triage of wounded. Coordination and decision making are also important tasks [1]. Training is carried out regularly in comprehensive real-life exercises, with tens or even several hundred participants, trainees, instructors and organizers. Trainees representing emergency personnel engage in activities where fast situation assessment and reaction are the main
tasks [2]. To keep the cost down, while maintaining the level of training requirements, simulators have been developed [3, 4].

Designing such simulators is a difficult task, since crisis management requires not only knowledge domain of managing crises, but also skills that enable workers to respond to unexpected and variable events and situations which neither may have apparent causes or foreseeable consequences to reactions [1]. Another aspect of the design of such a simulator is to decide the instructional design. Instructional design is a systematic practice of analysing learning requirements and based on them, designing a set of instructional specifications [5]. Instructional design models span over the lifecycle of an educational artefact through the stages of planning, designing, developing, implementing and updating where applicable. The instructional design of a crisis management simulator needs to provide trainees with experiential learning, variable uncertainty of events and situational complexity [6].

Crisis management involves teamwork in a safety critical domain requiring actors to collaborate and communicate to manage information, make decisions, plan and delegate tasks. Teams are formed at different levels. There are organizational teams, e.g. fire fighters, medical personnel and police, and there are actors from these organizations which form teams at various posts, e.g. at command centres or at the accident site. In crisis management simulation, the activities of trainees are divided into two main categories: field and command [2] [14] (Hansen & Satria, 2011) (Hansen & Satria, 2011) (Hansen & Satria, 2011) (Hansen & Satria, 2011) (Hansen & Satria, 2011). Field activities are carried out by trainees representing emergency personnel, who aim to learn to assess and respond to a situation as effectively and quickly as possible. These activities mainly involve rescue, triaging casualties, providing medical aid, transporting resources and casualties. Command activities involve the command and coordination staff, who is responsible for planning, strategizing, decision making and coordinating on a larger scale. Command activities include organizing resources, e.g. people, transportation, medical aid and hospital resources, mitigating risks, assessing situations, responding to them and planning [2]. A wide range of expertise appears during training and operations of crisis management, where responders and commanders include novices to experts [7]. Furthermore, to add to the heterogeneity some are professionals but others are volunteers from search and rescue organizations, Red Cross etc.

In addition to simulating work and situations, simulators need to include a stimulating and rewarding training environment for trainees. Experience has shown that during simulator tenders technical issues are prioritised over functionalities for supporting training [8]. To encourage better practices, Naikar and Sanderson [8] have illustrated how work domain analysis [9] can be extended to define training needs. They have argued that common techniques such as Instructional System Development (ISD) [10], which has been applied extensively by the air force in military training, is inadequate since they do not support work that is characterised by cognitive work, adaptive behaviour, random events and unpredictable situations.

Besides identifying training needs, designers of simulators have to decide how to deliver training. Training needs to be planned, carried out and assessed. Much has been written on professionals' reflection in action [11] and practitioners' ability to
self-assess or self-monitor [12]. Feedback is one of the most powerful techniques to help students learn. It provides learners with information on their performance when gaining or applying new knowledge and can help them finding the right answers [13, 14]. Feedback is an essential aspect for any learning environment, as it provides assessment of the learner's progress and results, motivates the learner, and enhances learning [15]. Either feedback can be intrinsic, that is available in a normal job situation, or extrinsic, which is available from an instructor either during or after a training exercise [16]. The latter one is also termed explicit feedback [17] which can come from a human tutor, a pedagogical agent, disembodied coach or other intelligent tutor [18]. Feedback can be provided by peers and is one of the five components of teamwork [19, 20]. Since teamwork is prevalent in crisis management, looking at such feedback becomes even more important than feedback provided in a single learner situation. As mentioned above, real-life training has many drawbacks, e.g. the costs of equipment and organization, and menace to environment [1]. However, it remains unclear whether the level of feedback delivery in a simulation, an essential part of training, may be the same or comparable as in a real-life exercise.

To meet this requirement of learning through feedback, most educational systems need to provide information on the assessment of learning to the trainee in some form [21], whether it is given by the system, or by a person, e.g. either a teacher/instructor or a fellow student. Although a simulator for crisis management training is not a game, we expect that knowledge from that field can be adopted. Researching designs of learning in games, Liu et al. [22] state that assessing the progress of a player's learning activity is one of the most rewarding interactive behaviour. To assist the player in the learning experience [23], such systems should provide appropriate feedback and they should ensure continuous flow to motivate the player to proceed [24].

Despite the substantial emphasis on including feedback in games or simulators, it seems that its design, or methods for its design, has not been given as much room as required [25]. However recently, there have been examples of methods to provide in-game assessments of objects and actions taken in real time. These assessments correspond to logged, time-stamped events that undergo statistical analyses that are displayed in real time or during a review after the game [3].

Motivated by the above need, the objective of this research is to compare, in terms of feedback, a plan for designing a training simulator to data gathered in a real-life training exercise for responding to an airplane crash. Thus, through an extensive analysis of the crisis management work domain and training exercises, we hope to learn about the kinds of feedback learners receive during training. The rest of the paper is organised as follows. After describing forms of feedback as they appear in the literature in the next section, the methodology of the research study is detailed in section 3. Section 4 describes the results of the research study which is a case study of a comparison between two different training environments. In the discussion of the results in section 5 we compare them with a feedback framework and suggest a theory of feedback. The paper concludes with section 6 presenting its contributions.
## 2 Forms of Feedback

Feedback in any learning environment may be categorized [15]. For example, feedback may be qualitative or quantitative, e.g. "Good", or "65 points" [26]. It may be negative, positive or neutral [25, 27]. Also, feedback can be corrective and/or motivational, where corrective feedback has the purpose to correct the learner when he/she has made a mistake, while motivational feedback motivates the learner, based on his/her individual preferences [28]. According to Pivec and Dziabenko [29], learning by mistake is a primary way for trainees to learn and motivating them to keep on trying.

Feedback may be in a form of elaboration or verification. Verification feedback refers to a notification of the result of the learner's answer, i.e. whether it is correct or not. Elaborative feedback provides an explanation of the result of the learner's answer and can give a hint to a solution [30, 31].

In a team, the process of providing feedback is not the same as in single person-to-person communication. According to Sonntag [32], in unidirectional teaching environment, where feedback is teacher-to-student and vice versa, feedback is a simple process, while in multidirectional environment (in a team) feedback becomes a more complex process. While Sonntag has looked at the integration of aspects within the teaching environment, the issues raised are relevant to a training simulation. To pass the right message onto students, Sonntag has emphasised a teacher's coordination of students and organization of materials and feedback in a multidirectional environment. The main concern is that students may misinterpret, underestimate or ignore bits of information [32]. In a similar manner, crisis trainees may misinterpret or not respond to information from an operational unit. Or, for example, if a trainee receives contradictory information from two other trainees, s/he may be left confused unable to proceed in such a case. Thus, unlike unidirectional feedback, multidirectional ones hold a rather arduous character due to the distributed interaction nature of the environment. Therefore, when designing scenarios, an important aspect of crisis management training is learning how to deal with wicked problems due to multi-directional feedback. Since unidirectional feedback occur in crisis handling as well, the two need to be distinguished. Another type of a teaching strategy is called peer-review, an approach when student-to-student interaction is used to provide feedback and advice [33]. Peer-review feedback exhibit similar complex behaviour of a multi-directional environment and poses a risk of credibility of information provided by the student.

Above, we have described some types of feedback in the learning context. More generally, feedback is provided in conversations, and more specifically in the context of systems, such as in the context of human computer interaction, where feedback is a report issued by the system as either a reply to the action of a user or a notification of the progress of a user's task [34]. Hence, when discussing feedback in the context of learning systems, all these aspects come into play.

Looking more closely at the system aspect, feedback in a virtual environment can be categorized according to senses, i.e. visual, haptic, or sound [22, 23, 33]. Further, there are two types of feedback in virtual environments, either immediate (real-time) feedback that can be issued to the player during the game or delayed or archival feed-
<table><caption>**Table 1.** Feedback forms as discussed in the literature</caption><thead><tr><th></th><th>Brief description</th><th>Learning</th><th>System / Virtual environment</th></tr></thead><tbody><tr><td>Qualitative</td><td>Textual</td><td>Yes</td><td></td></tr><tr><td>Quantitative</td><td>Numerical</td><td>Yes</td><td></td></tr><tr><td>Positive</td><td>Positive outcome of assessment</td><td>Yes</td><td></td></tr><tr><td>Negative</td><td>Negative outcome of assessment</td><td>Yes</td><td></td></tr><tr><td>Corrective (explicit)</td><td>Give a correct solution</td><td>Yes</td><td></td></tr><tr><td>Motivational</td><td>Encouraging to proceed</td><td>Yes</td><td></td></tr><tr><td>Unidirectional</td><td>Teacher to student or vice versa</td><td>Yes</td><td></td></tr><tr><td>Multidirectional</td><td>Teacher to a team</td><td>Yes</td><td></td></tr><tr><td>Peer-review</td><td>Among peers</td><td>Yes</td><td></td></tr><tr><td>Visual/graphical</td><td>Visual representation</td><td></td><td>Yes</td></tr><tr><td>Textual</td><td>Textual representation</td><td></td><td>Yes</td></tr><tr><td>Haptic</td><td>Touch and kinaesthetic</td><td></td><td>Yes</td></tr><tr><td>Immediate, automatic</td><td>Right away by the system</td><td></td><td>Yes</td></tr><tr><td>Delayed or archival</td><td>After the training session</td><td></td><td>Yes</td></tr></tbody></table>

back that is issued as a summary upon completion of a task or a game [15, 35]. Immediate feedback which is automatically generated by an application can be of any form mentioned above, while delayed feedback is usually given in text or graphical, form [15, 22, 25]. For example, action feedback is an immediate interactive response in real-time, while text feedback can be both immediate and delayed [22]. Immediate feedback may be provided during the execution of a task, usually when the task is complex, or after task completion if the task is relatively straightforward [25]. For example, in a car simulator, the player may get immediate haptic feedback in the form of vibration during the execution of a task.
Liu et al. [22] and Kiili [23] state that feedback is accepted more seriously by the player if it is provided immediately, instead of being issued after the game completion in the form of instructional text material. Also, according to Garzotto [36], when an action taken by the player is incorrect, it may be necessary and more effective to provide immediate than delayed feedback, since the player might later forget the incident, and, hence, not learn from delayed feedback. However, Renaud and Cooper [25] argue that archival feedback is also important, and may be presented in the form of text and/or graphically. Where appropriate, graphical feedback is more helpful than text, as it depicts the information as a whole [25]. Furthermore, feedback may be provided as an action replay in the form of a video [25]. Table 1 summarises the different forms of feedback described in this section divided into two categories depending on the source of introduction. By no means are the categories mutually exclusive, e.g. a qualitative feedback can be positive and delayed.

# 3 Research Methodology

The research method is a work analysis of trainees’ performance during the real-life exercise of emergency management. The feedback in the real-life exercise is compared to a system master plan of a virtual environment of a crisis management training simulator using categories from the literature. The real-life exercise training was conducted at an international airport in Europe and included almost 250 participants, with 90 participants creating the training scenario and serving as instructors, and the rest training for crisis management. The main exercise lasted 4-5 hours and included all organizations involved in a regular response to an aircraft incident. Seven participants of the CRISIS project served as observers, taking written notes and photos. The CITE system from VSL was used to enter notes during and after the exercise. In total, 548 observations were made for four hours at five different locations of the exercise: Emergency Operations Center (EOC), Casualty Assembly Point (CAP), Rendezvous Point (RVP), On Scene Command (OSC), and Accident Site. In addition, a few observations were made by instructors of the exercise, who were using the CITE system as a pilot.

Partners of the CRISIS project have collaboratively developed a virtual environment training simulator (CRISIS VE) for responders and commanders responsible for crisis management. The target sector is transport security, such as at for airport and railway operations. CRISIS VE was designed to support the 4C/ID instructional design model, which is meant to be used for long-period training with multiple complex tasks and learning objectives [30]. The 4C/ID instructional design model focuses on complex learning tasks and comprises four components, a) learning tasks, b) supportive information, c) just-in-time information and d) part-task practice [5]. Early on in the project, a system master plan (SMP) was written of the CRISIS VE including a detailed description of the system design and architecture, description of key concepts, such as events, variable uncertainties, and the CRISIS training cycle. Furthermore, the system master plan includes reference scenarios and example scenarios. Finally, it contains a list of roles for each of the scenarios and extensive lists of func-
<table><caption>Table 2. Feedback forms in the SMP</caption><thead><tr><td>Feedback form</td><td>Description or purpose</td></tr></thead><tbody><tr><td>Qualitative AAR</td><td>Comments by instructor</td></tr><tr><td>Quantitative AAR</td><td>A summary report</td></tr><tr><td>Immediate Visual</td><td>Verification</td></tr><tr><td>Immediate Action</td><td>Preventing mistakes</td></tr><tr><td>Verbal feedback</td><td>Communication with fellow trainees</td></tr><tr><td>Unidirectional</td><td>One-way feedback comments by instructor</td></tr><tr><td>Multidirectional</td><td>Discussion with instructor</td></tr><tr><td>Peer-reviewing</td><td>Feedback by fellow trainees</td></tr></tbody></table>

tional requirements for different modules and technologies of the simulator. Comprising well over 300 pages, the system master plan, is written after a thorough requirements analysis including site visits, interviews, questionnaires and a visit to one real-life exercise.

The classification of the feedback data from the real-life exercise and its analysis was carried out independently by two researchers. Each researcher classified data according to different feedback types, which are based on an analysis of the literature presented in section 2. Afterwards, they discussed and merged their results. Out of 548 observations noted down by observers, 49 observations on feedback were made.

# 4 A Case Study

## 4.1 An Analysis of a System Master Plan with Respect to Feedback

In this section we describe an analysis of the SMP [2] to see what requirements have been specified for the feedback available to the trainee. CRISIS VE supports functions to diagnose and to make a full After Action Review (AAR) of a trainee's action, decision and behavior based on data collected throughout the simulation game. Thus, the AAR module provides a quantitative summary report of the trainee's overall performance and details of an exercise. Based on this data, the instructor submits his/her own qualitative comments on the trainee's performance.

CRISIS VE supports **immediate visual** and **action feedbacks**. CRISIS VE supports immediate visual feedback, e.g. when a trainee places a Casualty Assembly Point (CAP) at a location, the system immediately draws a CAP object on the screen. An example of how CRISIS VE supports immediate action feedback is when the system prevents the trainee from placing the CAP in an area too close to the site of an accident, since it would endanger the casualties in case of fire spreading. Thus, the system **prevents** the trainee from making a **mistake**, but signals him or her implicitly that an incorrect action has been carried out. Furthermore, CRISIS VE facilitates, by
supporting communication with fellow trainees, learning by mistake. Trainees will be able to receive real-life **verbal feedback** through communication devices implemented in the simulator, such as radio or mobile phone, from other trainees or instructors. CRISIS VE also supports both **unidirectional** and **multidirectional feedback** between the trainee and the instructor, as well as **peer-reviewing** between all trainees.

CRISIS VE does not support sound as a form of feedback. Although it includes sounds to resemble the real-life environment [37], it does not notify the trainee of the success or failure of a task completion. While some research is underway for using haptic equipment, i.e. for sensing heart beats of casualties [38], the SMP does not include such support for physical sense feedback. **Error! Reference source not found.** presents an overview of the feedback forms found in SMP.

## 4.2 An Analysis of a Real-Life Exercise with Respect to Feedback

Using the data from a real-life exercise, this section describes feedback a trainee gave another trainee or an instructor gave a trainee or trainees. We have put feedback types, either from the literature or the real-life exercise data, in bold. Quotes from the data are presented in quotes and italics. **Error! Reference source not found.** gives an overview of the feedback types.

All 49 instances of feedback during the real-life exercise were **verbal** and **qualitative**. An AAR was presented to trainees **verbally** at a meeting after the event, but a written report was planned as is customary in such exercises.

Trainees were **not rewarded**, i.e. they were not provided with positive feedback, upon the successful completion of a task. Although there were many examples of **verification feedback** (19), such as when one commander tells the other that they "need better information on weapons", most instances were of **elaborative feedback type** (29) with an explanation of the error or the problem, such as when "airline operator discusses with Red Cross commander how to send the list" after feedback had been provided on not receiving the list although, reportedly, it had been sent. Another example of **verification feedback** was when the commander made decisions and organized the other trainees after having concluded: "transport capacity is not enough to transport all red casualties". **Elaborative feedback** occurred during every status meeting.

Many of the feedback during the exercise were of a **corrective character** (20): "Wounded still on the scene", implying that they should not be on the scene, and only a few feedback instances were **motivational in nature** (10): "EOC commander concludes that the transport capacity is not enough to transport all red casualties. She asks Resource manager for more resources". Another example of **corrective feedback** was when an instructor stated that "ambulances should already be on the scene, but they are not".

Some of the feedback is only **unidirectional** (13), e.g. in the case of providing an assessment of the situation, at RVP: "It took too long time for activating resources from the Rendezvous point to the accident site", or at the OSC post: "[We] need basic information on the status of the triage immediately in the beginning". A further example implies **negative feedback** when commander states "there should be better
flow". Another example of negative feedback happened when at the EOC it was observed "NECC (National Emergency Coordination Center) has complained about not receiving a passenger list". Feedback incidents that took place during the status meetings were **peer-reviews** with trainees explaining their current state of work and problems. Most (39) feedback instances were peer-reviews, as they involved different trainees trying to solve one task. For example: "EOC commander concludes that the transport capacity is not enough to transport all red casualties. She asks the Resource manager for more resources. The radio operator informs that OSC has not requested resources, only to move existing resources closer". There were no multidirectional feedback instances.

In two cases we noted feedback in the form of **explicit teaching**. OSC realised that the rescue coordinator had misunderstood the role of the OSC. In response, they attempted to inform the coordinator of the correct procedures: *Commander to Planning:*
"How is the flow?" - *Looking at an overview at the display. Planning to Resources:*
"Do you want to have the ambulances closer?" *Commander says:*
"Transport coordinator should resolve this" - *explains the procedure and role of transport coordinator.*

Another example of teaching trainee procedures was when a commander at OSC said: "I want to iterate that EOC should organize transport of casualties out of the scene - this is not the responsibility of OSC but an EOC."

Planning personnel at OSC communicated to the EOC: "You should talk directly to the transport coordinator" - *Commander looks at the hierarchy with plans and says that according to the hierarchy, the medical coordinator and the transport coordinator should organize the transport among themselves.*

Probing for information was noted as a trigger for feedback. At the OSC at a start of regular status meetings: "We don't have confirmation that the scene has been secured. There they come (i.e. the rescue team) (Looking through the window). How much of rescue teams did he send? Do we have numbers from security coordinator?"

During the real-life exercise, trainees were constantly burdened with various hiccups that persisted along the way, such as the equipment failure or data loss. For example, "the police officer reports a radio problem and asks for assistance for fixing it or replacing". This is important to mention since it could play a big role in determining the triggers for feedback in a simulation. There were other important triggers for feedback that took place during a real-life exercise.

Many feedback instances were triggered through discussions (22 out of 49) and information sharing (42), e.g. during regular status meetings when numbers of casualties were reported. Many instances triggered reasoning (20) and organization (22) among participants of an exercise, e.g. when the commander made decisions and organized the other trainees after having concluded that transport capacity was not enough.

More examples of when feedback initiated a discussion were triggered after an issue had been raised (11). At the EOC there was an incident when "The radio/computer operator informs that NECC has been trying to reach EOC. The EOC commander replies she is about to contact them". At the EOC, another example which showed feedback which raised an issue: "We need transport capabilities from
<table><caption>**Table 3.** Feedback Forms in a Real-Life Exercise</caption><thead><tr><td>Feedback forms</td><td>Literature / New</td></tr></thead><tbody><tr><td>Verbal qualitative</td><td>L</td></tr><tr><td>Verification</td><td>L</td></tr><tr><td>Elaborative</td><td>L</td></tr><tr><td>Corrective</td><td>L</td></tr><tr><td>Motivational</td><td>L</td></tr><tr><td>Negative</td><td>L</td></tr><tr><td>Unidirectional</td><td>L</td></tr><tr><td>Peer-reviews</td><td>L</td></tr><tr><td>Explicit teaching</td><td>N</td></tr></tbody></table>

[City excluded for the sake of anonymity] - there is nothing on this in the database. This is a problem".

It is also important to mention that on some occasions the feedback resulted in a learning outcome. For example, in two feedback cases the trainee was learning by mistake as a result of feedback, e.g. a trainee learned not to inform patients of the death incidents as they became “upset when hearing about dead patients over a near-by tetra unit”.

## 4.3 A Comparison between the Planned Design and the Real-Life Exercise

In the previous section, we presented how trainees received feedback of different types. In this section, we discuss how the design of a real-life exercise may contribute to the design of a simulation in terms of credibility of learning, by comparing it to the planned design.

Although the purpose of feedback in real-life exercise and CRISIS VE is to train participants to respond in a crisis situation, some of the feedback practices in CRISIS VE contradict with the real-life exercise. There are differences in the delayed feedback during the real-life exercise and in the training simulator. The archival feedback provided in the crisis simulator is only available in text form, and it is quantitative with the possibility that the instructor may provide qualitative comments on the performance. In the real-life exercise, the feedback which is given to the trainees as an aftermath of their performance is qualitative and is presented verbally or as text.

The immediate feedback in the real-life exercise is provided by trainees, usually in the form of elaboration, both during the task execution and/or after task completion. In the training simulator, immediate feedback is provided in the form of action by the system, and can be received through conversing with other players via communication devices. The difference here is that the trainee does not get any immediate automatic feedback in the real world, so if s/he makes a mistake, it will have consequences at a later stage of the training. The system, however, tries to prevent the trainee from making a mistake without explanations, and with no further consequences. For
example in real-life, if a trainee breaks something then other trainees will have to continue without it, and will have to use another object instead as a replacement.

While CRISIS VE supports some learning by mistake function, it does not provide feedback to the trainee on the nature of a mistake. For example, in the case of placing a CAP area too close to the accident site, the simulator does not allow for the CAP to be placed. There is no explanation on why the CAP cannot be placed. Thus, the trainee may be unsure whether it is a fault in the software or if the reason is the proximity to the accident site. Although in real-life a trainee does not get any messages of the sort, in the training environment s/he may become confused if there is no message or alert generated when something is happening unexpectedly. Cyboran [15] has explained that learners do not benefit when the system does not specify its response towards learner's action, particularly when the nature of response may not be clear or obvious. Analysing the system master plan it was unclear if incorrect actions or mistakes were logged for the AAR report.

Furthermore, the work analysis revealed that in real-life many feedback incidents are connected to unexpectedness of situations, particularly equipment failures. However, in the master plan for the CRISIS simulator, there are only some vague suggestions of triggers for unexpected incidents. While the master plan has a general design of feedback and unexpected situations, the training exercise showed that in real-life there are several typical situations that would need specifically designed feedback.

# 5 Discussion

In this paper we showed how feedback design may be overlooked if it is not supported by a work analysis that aids in explaining the relationship between feedback and learning during crisis management training. The current planned design reveals that the design of feedback, one of the most important learning tactics in education, contradicts with the real-life practices of crisis management training.

The work analysis has shown that during the real-life exercise feedback incidents, similar to typical of crisis management training, occurred. Such similar feedback incidents have been classified and grouped, and can be used in the design of a simulator for crisis management training. In preparation for the real-life exercise, exercise organisers said they were not expecting a lot of feedback given to trainees by instructors. The analysis reported in this paper showed that although not extensive, feedback was more apparent than expected. In future studies, it would be interesting to contrast this result to trainees' reflection on their own work [11] as professionals or in an after action review and see how trainees expertise might determine this reflection vs. feedback.

We analysed what the feedback implied in terms of learning in a real-life exercise, i.e. whether trainee would gain knowledge as a result of this feedback or get the message that training needs to be improved in the future. For example, for *assessment of a situation*, a fellow trainee pointed out to the trainee that the situation was deficient, and that in future situations this would need be avoided or improved. Or, for *raising an issue* or *probing for information*, the feedback indicated to the trainee that a prob-
lem had arisen that could be solved now but little or no instructions were provided. Or, in the case of explicit teaching, the trainee received instructions and guidance on how to accomplish a goal.

For this research, we have gathered empirical data to show how work analysis can incorporate and shape the design of the feedback incidents through comparison of the work analysis and an earlier design plan. In comparison, Narciss and Huth [30] carried out research, based on literature, to derive guidelines on how to design feedback incidents. In their research, they emphasise how the cognitive task analysis plays a crucial part in the design of feedback.

Narciss and Huth [30] proposed three facets of feedback, functions of feedback (cognitive, meta-cognitive or motivational), contents of feedback (evaluative or informative towards the knowledge of the result) and presentation of feedback contents (timing, schedule, adaptivity). Based on this framework, procedures are detailed on how to design feedback. Instruction contexts, such as objectives, tasks and errors contributed to the contents of feedback; and individual factors such as learning objectives, prior knowledge and skills and academic motivation contributed to functions of feedback. Narciss and Huth [30] viewed the contents of feedback in terms of evaluative and informative components. Comparing their classification to ours, the *assessment of a situation and raising an issue or probing for information* may be considered as evaluative components, while *explicit teaching* falls in the category of an informative component.

According to Gredler [39], simulations may be of two types: symbolic or experiential. Symbolic simulation presents tasks where a participant must often use or improve theoretical knowledge to solve a problem, e.g. business manager, physician, or mathematician. Experiential simulation presents tasks where a participant is trained to gain knowledge of how to operate a device or perform a set of routine tasks, which require little to no need for scientific concepts or theorems [39]. Since Narciss and Huth studied learning environments for a symbolic type of simulator where trainees solve problems by presenting knowledge of the result, such as subtraction, we do not know if their findings can be directly translated to crisis management training, which is an experiential simulator. Nonetheless, it can be valuable to compare the two types to see how well the design guidelines can be translated from one type of learning environment to another.

The research has some limitations. One limitation was that the observers may not have reported all of the incidents that occurred. Although the data set that we analysed was large, it was only from one real-life exercise. Thus, the feedback incidents retrieved are specific to that particular training scenario.

Additionally, for more detailed modelling of a learning environment in a simulation-based training, a study of gathering feedback from actual trainees, and not just observers, may need to be carried out. Castor et al. [40] performed a study for the purposes of gaining insight into the understanding of the cognitive processes of the trainees. The study was experimental in nature, and no well-defined model was derived as an outcome, although suggestions of cognitive modelling approaches were made. CRISIS VE could benefit from such an analysis in determining the functional
design of artificial intelligence components within the simulator, such as for unexpected events or computer-played characters.

# 6 Conclusion

Previous research on feedback seems to be scattered in individual areas ranging from communication to learning sciences, onwards to system related areas such as HCI, educational technologies and learning games. And thus, the theory of feedback in the context of HCI is rather weak [27]. Furthermore, what makes it difficult is that feedback is of varying abstractions along different dimensions, ranging from low lexical level to a high semantic level, from a drill type of training to training more cognitive complex tasks in disperse application domains, ranging from Sports training [41] to Medical training [42] onwards to Pilot training [16]. This research has shown that it is possible to categorize data from a real-life exercise for use in designing a simulator. Specifically, we showed how this data may be useful in recognizing what practices in real-life are frequent and need to be incorporated, and thus should be given more attention when designing a simulation. For example, many vague suggestions of unexpected situations were provided in a system master plan, while in real-life there were several typical unexpected situations.

By comparing data from a real-life training exercise to data from a system master plan, we have shown that it is necessary to pay more attention to learning aspects of a simulator, such as feedback. In particular, since the comparison revealed significant differences between the system master plan and the real-life exercise, the research has shown that while it is vital to carry out analysis of the domain tasks, roles and scenarios, the development of a learning system needs to take into account an analysis of learning tactics provided under a real-life setting.

# Acknowledgement

The research leading to these results has received funding from the European Union Seventh Framework Programme (FP7/2007-2013) under grant agreement no. [FP7-242474].

# References

1. Stolk, D., Alexandrian, D., Gros, B., Paggio, R.: Gaming and multimedia applications for environmental crisis management training. Computers in Human Behavior 17, 627-642 (2001)

2. Hansen, K.B., Satria, L.R.: System Master Plan - Deliverable 3.3. of the CRISIS project. (2011)

3. Raybourn, E.M.: Applying simulation experience design methods to creating serious game-based adaptive training systems. Interacting with Computers 19, 206-214 (2007)
4. Rudinsky, J., Hvannberg, E.T.: Consolidating models of requirements analysis for crisis management training simulator. In: Dugdale, J., Mendonça, D. (eds.) ISCRAM 2011. ISCRAM Lisbon, Portugal (2011)

5. van Merriënboer, J., Clark, R., de Croock, M.: Blueprints for complex learning: The 4C/ID-model. Educational Technology Research and Development 50, 39-61 (2002)

6. Wong, W., Rankin, A., Rooney, C.: The Variable Uncertainty Framework. Middlesex University (2011)

7. Dreyfus, H.L., Dreyfus, S.E.: Peripheral Vision: Expertise in Real World Contexts. Organization Studies 26, 779-792 (2005)

8. Naikar, N., Sanderson, P.M.: Work Domain Analysis for Training-System Definition and Acquisition. The International Journal of Aviation Psychology 9, 271-290 (1999)

9. Vicente, K.J.: Cognitive Work Analysis, Toward Safe, Productive, and Healthy Computer-Based Work. Lawrence Erlbaum Associates, London (1999)

10. Kirby, J., Hoadley, C., Carr-Chellman, A.: Instructional systems design and the learning sciences: A citation analysis. Educational Technology Research and Development 53, 37-47 (2005)

11. Schön, D.A.: The Reflective Practitioner: How Poessionals Think in Action. Basic Books New York (1983)

12. Eva, K.W., Regehr, G.: "I'll never play professional football" and other fallacies of self-assessment. Journal of Continuing Education in the Health Professions 28, 14-19 (2008)

13. Alvarez, K., Salas, E., Garofano, C.M.: An Integrated Model of Training Evaluation and Effectiveness. Human Resource Development Review 3, 385-416 (2004)

14. Salas, E., Cannon-Bowers, J.A.: THE SCIENCE OF TRAINING: A Decade of Progress. Annual Review of Psychology 52, 471-499 (2001)

15. Cyboran, V.: Designing feedback for computer-based training. Performance + Instruction 34, 18-23 (1995)

16. Hawkins, F.H.: Human Factors in Flight. Ashgate Burlington, VT (1987)

17. Hays, M., Lane, H.C., Auerbach, D., Core, M.G., Gomboc, D., Rosenberg, M.: Feedback Specificity and the Learning of Intercultural Communication Skills. Proceedings of the 2009 conference on Artificial Intelligence in Education: Building Learning Systems that Care: From Knowledge Representation to Affective Modelling. IOS Press (2009)

18. Lane, H.: Promoting Metacognition in Immersive Cultural Learning Environments Human-Computer Interaction. Interacting in Various Application Domains. In: Jacko, J.A. (ed.) Human-Computer Interaction, Interacting in various application domains, 13th International Conference, HCI International 2009, vol. LNCS 5613, pp. 129-139. Springer Berlin / Heidelberg, San Diego, CA (2009)

19. Entin, E.E., Entin, E.B.: Measures for Evaluation of Team Processes and Performance in Experiments and Exercises. Proceedings of the 2001 Command and Control, (2001)

20. Cannon-Bowers: Team performance and training in complex environments : Recent findings from applied research. Current Directions in Psychological Science 7, 83-87 (1998)

21. Moreno-Ger, P., Burgos, D., Martínez-Ortiz, I., Sierra, J.L., Fernández-Manjón, B.: Educational game design for online education. Computers in Human Behavior 24, 2530-2540 (2008)

22. Liu, K.-Y., Yang, C.-T., Chang, K.-H.: Development of a Multiplayer Online Role-Playing Game-based Learning System for Multiple Curriculums. In: Digital Game and
Intelligent Toy Enhanced Learning (DIGITEL), 2012 IEEE Fourth International Conference on, pp. 62-66. (Year)

23. Kiili, K.: Digital game-based learning: Towards an experiential gaming model. The Internet and Higher Education 8, 13-24 (2005)

24. Paras, B., Bizzocchi, J.: Game, Motivation, and Effective Learning: An Integrated Model for Eucational Game Design. DiGRA Conference: Changing Views-Worlds in Play, (2005)

25. Renaud, K., Cooper, R.: Feedback in Human Computer Interaction: Characteristics and Recommendations. South African Institue of Computer Scientists and Information Technologists. Annual Research Conference, pp. 105-114 (2000)

26. Connellan, T.K., Zemke, R.: Sustaining Knock Your Socks Off Service AMACOM, New York (1993)

27. Spink, A., Saracevic, T.: Human-computer interaction in information retrieval: nature and manifestations of feedback. Interacting with Computers 10, 249-267 (1998)

28. Pyke, J.G., Sherlock, J.J.: A Closer Look at Instructor-Student Feedback Online: A Case Study Analysis of the Types and Frequency Journal of Online Learning and Teaching 6, 110-121 (2010)

29. Pivec, M., Dziabenko, O.: Game-based learning in universities and lifelong learning : UniGame : Social Skills and Knowledge Training game concept. Journal of Universal Computer Science (J. Ucs) 10, 14-16 (2004)

30. Narciss, S., Huth, K.: How to Design Informative Tutoring Feedback for Multimedia Learning. In: Niegermann, H., Brunken, R., Leutner, D. (eds.) Instructional Design for Multimedia Learning, pp. 181-. Waxmann, Munster, Germany (2004)

31. Pridemore, D., Klein, J.: Control of feedback in computer-assisted instruction. Educational Technology Research and Development 39, 27-32 (1991)

32. Sonntag, M.: Teleteaching: From Unidirectionalism to Multidirectionalism. In: Hofer, S., Beneder, M. (eds.) IDIMT'99 7th Interdisciplinary Information Management Talks. Linz: Universitätsverlag Trauner Zadov (CZ) (1999)

33. Richer, J., Drury, J.L.: A video game-based framework for analyzing human-robot interaction: characterizing interface design in real-time interactive multimedia applications. Proceedings of the 1st ACM SIGCHI/SIGART conference on Human-robot interaction, pp. 266-273. ACM, Salt Lake City, Utah, USA (2006)

34. Perez-Quinones, M.A., Sibert, J.L.: A collaborative model of feedback in human-computer interaction. Proceedings of the SIGCHI conference on Human factors in computing systems: common ground, pp. 316-323. ACM, Vancouver, British Columbia, Canada (1996)

35. Ahn, L.v., Dabbish, L.: Designing games with a purpose. Commun. ACM 51, 58-67 (2008)

36. Garzotto, F.: Investigating the educational effectiveness of multiplayer online games for children. Proceedings of the 6th international conference on Interaction design and children. ACM, Aalborg, Denmark (2007)

37. Rudinsky, J., Hvannberg, E.T., Helgason, A.A., Petursson, P.B.: Designing soundscapes of virtual environments for crisis management training. Proceedings of the Designing Interactive Systems Conference, pp. 689-692. ACM, Newcastle Upon Tyne, United Kingdom (2012)
38. Einarsdóttir, E.M.: Simulation of Hearbeat, Respiration and Heat using Haptic Feedback and Thermal Unit Faculty of Industrial Engineering, Mechanical Engineering and Computer Science, vol. M.S., pp. 59. University of Iceland, Reykjavik (2013)

39. Gredler: Educational Games and Simulations: A Technology in Search of a (Research) Paradigm. In: Jonassen, D.H. (ed.) Handbook of research on educational communications and technology, pp. 521-540. Macmillan, New York (1996)

40. Castor, M., Sennersten, C., Gustavsson, R., Lindley, C.: Decision processes in simulation-based training for ISAF vehicle patrols. In: NATO Research and Technology Organisation, HFM-202 Symposium on "Human Modelling for Military Application". (Year)

41. Iskandar, Y.H.P., Gilbert, L., Wills, G.B.: Pedagogy in Computer-Based Sport Training. In: Advanced Learning Technologies (ICALT), 2011 11th IEEE International Conference on, pp. 403-408. (Year)

42. Cristancho, S.M., Moussa, F., Dubrowski, A.: A framework-based approach to designing simulation-augmented surgical education and training programs. The American Journal of Surgery 202, 344-351 (2011)