# Digitalizing Crisis Management Training

Monika Magnusson, Geir Ove Venemyr, Peter Bellström, Bjørn Tallak Bakken

► To cite this version:

Monika Magnusson, Geir Ove Venemyr, Peter Bellström, Bjørn Tallak Bakken. Digitalizing Crisis Management Training. 11th International Conference on Electronic Participation (ePart), Sep 2019, San Benedetto Del Tronto, Italy. pp.102-113, 10.1007/978-3-030-27397-2_9. hal-02446017

HAL Id: hal-02446017

https://inria.hal.science/hal-02446017

Submitted on 20 Jan 2020

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Digitalizing Crisis Management Training

Monika Magnusson<fnref n="1" />[0000-0002-6227-1784], Geir Ove Venemyr<fnref n="2" />, Peter Bellström<fnref n="1" />[0000-0002-0419-1647] and Bjørn Tallak Bakken<fnref n="2" />

**Abstract.** The ongoing digital transformation in government has enabled innovative changes in operational processes and service. However, while e-services and social media are widely adopted, earlier studies indicate that this transformation is still being awaited in other areas, such as crisis or disaster preparedness. Recent events such as the 2018 wildfires in several parts of Europe, as well as empirical research, highlight the need for more (systematic) training of local governments' crisis management teams. Conventional training methods are time- and space-dependent and require long-term planning, making it complicated to increase the extent of training. In this interdisciplinary study, we report on the results from the Swedish-Norwegian CriseIT project that aimed to develop information systems (IS) for crisis management training. The purpose of the article is to describe information systems designed to support local governments' crisis management training and to discuss how these artefacts could improve crisis management training practices.

**Keywords:** Crisis management training, crisis training software, computer-based training, disaster management, design science research.

## 1 Introduction

Global warming and terror attacks are just two examples of phenomena that have put crisis preparedness at the top of the agenda of government agencies world-wide. Crisis exercises are considered a key component for increasing preparedness [28]. Prior studies have detected problems with conventional training such as lack of time and other resources for planning and executing exercises and concerns that exercises are performed too seldom and/or with too few members of the organization [13]. Other problems include inadequate assessment of the effectiveness of training, difficulties in defining a suitable training content, providing feedback, and transferring the lessons learnt to future crises [23]. Researchers have also identified a lack of systematic approach to training [28] and too little focus on longitudinal learning processes [26].

In this article, we propose information systems (IS), developed in the Swedish-Norwegian R&D project CriseIT, as a complement to conventional crisis or disaster man-

<footnote n="1">¹ Karlstad University, 651 88 Karlstad, Sweden</footnote>

<footnote>{monika.magnusson, peter.bellstrom}@kau.se</footnote>

<footnote n="2">² Høgskolen i Innlandet, 2418 Elverum, Norway</footnote>

<footnote>{geir.venemyr, bjorn.bakken}@inn.no</footnote>
agement training methods. In-depth interviews in the CriseIT project with 19 respond-
ents from 16 organizations handling societal crisis at local, regional and national levels
showed that the organizations wanted more training, especially for the strategic level
[11]. Regrettably, the organizations found this difficult to accomplish in practice, due
to a lack of resources and the current training methods. For managers to devote half a
day or more to an exercise and for safety coordinators to set a date, construct a crisis
scenario etc. can be challenging tasks. Scenarios are often planned from scratch and the
ability to collaborate in planning and reuse of exercises are requested by the organiza-
tions [12]. In smaller municipalities, a safety coordinator, sometime working part-time
with the assignment, might be solely responsible for organizing the organizations’ train-
ing [14]. Few of the organizations organized any individual training.

Lukosch et al. [10] suggest that computer-based training, in comparison to traditional time- and space-dependent training, offers higher flexibility (efficacy) and improved resource-efficiency. Whether these advantages are being realized in practice remains to be seen. Studies of ongoing usage, effects and learning outcomes from computer-based training are rare, as are user need analyses [13], indicating that the field of computer-based crisis management training is still in its infancy.

Organizations interviewed in the CriseIT project were aware that different IS for
crisis training existed on the market, but few had used them. Neither were the national,
and freely provided, systems WIS (Sweden) and CIM (Norway) used for training to
any considerable extent. Limited functionality for training and low usability were men-
tioned as possible explanations. Instead, crisis training exercises were often planned,
executed and documented using regular office software. In this article, we argue for the
need for new IS for crisis management training in local government. These digital tools
should support both trainers and trainees and be well grounded in the user contexts, if
they are be useful and adopted. As current training practices are well functioning in
many aspects, the IS should complement, not replace conventional training. The overall
objective of the IS artefacts presented is to enable expanded crisis management training,
in a systematic and resource-efficient manner. The purpose of the article is to *describe
information systems artefacts designed to support local governments’ crisis manage-
ment training and to discuss how these could improve crisis management training prac-
tices*

We use the concept ‘crisis management training’ to represent both training for indi-
vidual roles in a crisis, and collaborative exercises, for “unplanned natural or man-made
events with a sudden and severe negative impact on human live [sic], the functioning
of society and/or the physical environment” ([24], p. 61).

## 2 Literature review

The fact that crises are rare often means that few in an organization's crisis management
team have actual experience of handling them. Sinclair et al. [22] argue that "[...] emer-
gency management training must include mechanisms that substitute for the practice
and experience afforded by working life in most organizations. This involves exercis-
ing" (p. 59). Exercises also serve other purposes, such as testing the viability of the
response network [19] the crisis management or disaster plan [18] or fostering relation-
ships between key personnel [24]. Based on experiences from over a hundred exercises
in twelve municipalities during a ten-year period, van Laere and Lindblom [26] argue
for a continuous training pro-gram with recurrent exercises in various formats to spur
the long-term development of an organization's crisis management capacity. They sug-
gest three development phases: (1) obtaining role understanding; (2) developing skills
and practices; and (3) mastering self-evaluation and adaption.

Despite the seeming consensus in earlier research about the necessity of training, earlier studies also indicate that there is a lack of a systemic approach to training [28]. Such an approach can be found in the management training methodology developed from a meta-analysis of training and development studies by Eduardo Salas et al. [21]. Salas et al. [21]-stress the importance of viewing training “as a system”, and not a once-off event. The methodology was intended for general ‘management’ training, but can also be used for emergency training [27]. Seven basic phases were defined by Salas et al. [21], see Fig. 1. Phase 1, *Student Need Analysis*, involves determining who needs training (audience), and what should be trained (content). In Phase 2, *Educational Competencies*, general skills-based competences for crisis management are described and compared with the skill inventory information from Phase 1. The analysis in the first two phases indicates the direction of the rest of the learning process. Phase 3, *Learning Objectives*, consists of specifying measurable training goals. After the training goals have been set, simulation scenarios are developed in Phase 4, *Trigger Events Exercises*. The scenarios and events provide opportunities to influence behaviour and practice within the relevant field of competence. In Phase 5, *Performance Measures*, process and performance measurements are developed. Without measuring performance, it is impossible to improve behaviour or competence, provide feedback, or document learning. In Phase 6, *Performance Diagnosis*, the measurements from Phase 5 are compared with the training goals defined in Phase 3 to assess if the training is effective. Finally, Phase 7, *Developmental Feedback*, involves the development of constructive feedback based on performance and process data. Feedback allows the scheme to be called training and not just simulation.

<figure><img src="image_2.png" /><figcaption>**Fig. 1.** Stages for the successful implementation of simulation based training in management education (Salas et al. [21], p. 565).</figcaption></figure>
Information systems can positively affect an organization by improving its efficacy, efficiency and/or effectiveness [6]. Higher efficacy refers to improvements in volumes or quality of the output, while improved efficiency is a measurement of the output in relation to the resources used (ibid.). Effectiveness, finally, refers to the system's contribution to the purpose of higher-level systems. If and how computer-based training indeed increase efficacy, efficacy and effectiveness remains to be seen. Among the earlier research studies from the IS field we found useful in our work are the conceptual framework for understanding training issues in Sniezek et al. [23], the generic systems modules presented by Reuter et al. [20]. Also, the flexible and user driven simulation exercises presented in Yao et al. [29], the PANDORA system for co-located, distributed or synchronous training [4] and the IS for tabletop exercises presented by Asproth et al. [3] and Araz et al. [2]. Furthermore, we have studied the participatory design process described by Lukosh et al. [10], and the serious gaming solution for individual training presented by Van de Ven et al. [25]. Moreover, Garzón and Acevedo [7] have done a meta-analysis of the impact of AR on students' learning effectiveness. These and other studies have guided the design processes presented later in the article.

## 3 Method

This study reports results from the R&D project CriseIT. CriseIT run in the border region of Värmland and Innland, during 2016-2019. The main aim was to improve crisis preparedness in the regions by developing IS in close collaboration between public organizations, universities, enterprises and non-government organizations (NGOs).

The overall research approach can be described as design science research (DSR) as the aim was to design IS artefacts. Four IS prototypes or what Hevner et al. [9] call instantiations have been developed: a training process management tool (T1), an individual training tool (T2), a web-based collaborative training tool (T3) for 'tabletop-like' exercises and an operational decision support system (T4) for 3D-based training. The prototypes are presented in the next section. Although used in the IS field for more than 30 years, there are diverse views and application of the DSR paradigm [17]. Numerous articles present theoretical and conceptual guidelines for DSR (e.g. [9], [16]). However, the body of somewhat diverse guidelines, frameworks and objectives, makes it difficult to present DSR studies without conflicting with some of the rules [17]. The DSR process model from Peffers et al. [16] represents the overall approach for our work, although the detailed DSR processes have varied between the different artefacts. Furthermore, we adhere to the seven well-known guidelines for DSR presented by Hevner et al. [9]. Through in-depth interviews with respondents from organizations at all levels in both countries and by using user-centered development methods, we especially emphasized the guideline of establishing problem relevance. Also, we have adopted the 'cycle view' of DSR first presented as three design cycles in Hevner et al. [8]. We believe that while the model of Peffers et al. [16] guides the researchers' activities, the design cycle view helps the researchers to concretize the activities, e.g. in mapping the
environment for the artefact(s) to be produced. In addition, the design cycle view well illustrates the iterative approach we used in our design processes.

# 4 The Design Processes

The cross-sector focus in this study, together with the aim of finding generic solutions for local and regional governments, meant that multi-actor design processes needed to take place. The methodology of Salas et al. [21](Fig. 1) was used to guide the design of the artefacts as were research on decision-making in crisis situations [e.g.5]. Furthermore, methods for change analysis and development of ‘work-system’ have guided the design process (e.g. [1]). Also, the results from earlier studies on crisis management training and IS for crisis training served as a starting point. Beynon Davies’ [6] extension of the model for information system success developed by DeLoan and McLean in 1992 has served as a guide for evaluations of the artefacts.

For both *the training process management tool* (T1) and *the operational decision support system* (T4) an agile approach to system development, inspired by Scrum, was used. T1 was developed through a series of 10 time-boxed iterations, called sprints. This iterative process secured a user-oriented approach, and the sprints enabled creative and synergetic meetings with the team members. Similarly, the development of T4 took place in increments or iterations, with feedback from prospective users and crisis management experts after each increment/iteration. Both T1 and T4 had been evaluated by several students groups, of which many also are practitioners, and by project stakeholders, and show promising results, for example in ensuring a systematic approach to exercises. T1 was evaluated in approximately 15 field studies (cf. [9]) in crisis handling organizations for the planning of crisis training activities. The evaluations provided feedback on utility, systems quality, ease-of-use and other parameters. T4 has so far mainly been evaluated with a focus on overall utility, interface/ease-of-use and identifying defects in what Hevner et al. [9] label functional testing. The project CriseIT2 will more systematically evaluate the artefact and further demonstrate its utility with respect to training effectiveness, ease of use (both when used locally and distributed), and costs.

For *the individual training tool* (T2) and *the collaborative training tool* (T3), an interdisciplinary research team performed in-depth, semi-structured interviews with 19 security coordinators and other personnel responsible for civil contingency management at local (municipal), regional, and national government levels in both countries. The interviews were important for the mapping of the user context, and for the identification of problems and perceived opportunities with an increased digitalization of crisis management training.

Furthermore, 17 workshop were organized. A majority of the stakeholders involved worked at municipalities on either side of the border, often as safety coordinators. Problems, opportunities and systems requirements were identified and prioritized by the stakeholder, and system prototypes were evaluated. Primarily two target groups were identified, trainers and trainees, i.e. safety coordinators responsible for planning and executing exercises and crisis management teams in local and regional government
agencies that need to train/exercise. The workshops and interviews revealed that the planning processes differ somewhat from one situation to another although several respondents stressed the importance of a systematic approach to ensure quality. For example, while there are national guidelines as well as an agreement among the respondents that the purpose and goals of an exercise should be established first, the respondents admit that is common to start defining the scenario before the purpose has been determined. The problems, opportunities and requirements identified are described in Magnusson et al. [12] and will only be briefly presented here. As for problems, several organizations experienced problems connected to organizational constraints in personnel, budget, or insufficient IS support. In addition, the expenditure of time in planning and executing current training methods were problematic, as were the lack of a systematic approach, especially to the follow-up of exercises. Some respondents found it complex to design exercises and to know the needs of training audience. Several respondents considered it difficult to keep up the organizational knowledge due to employee-turnover. New (digital) IS for crisis training were thought to enable both individual training and collaborative exercises, co-located and distributed, synchronous and asynchronous, on any device, and in short sessions. This increased flexibility, together with multimedia or gaming features, was seen as an opportunity to get more actors/trainees involved and/or to train more often. Also, new IS were considered to facilitate collaboration and reuse of exercises material, thus contributing to more resources-efficient training. Several risks and potential disadvantages of IT tools were also acknowledged, including that digital training would replace conventional training, and that technical problems would arise. Also, 17 distributed screen-sharing prototyping sessions took place in which users' needs and objectives were identified, refined and validated in what Hevner et al. [9]-describe as functional testing. T3 prototypes were also evaluated through what Hevner et al. [9]-label structural testing in three "walk-through" sessions with target users. Later, T3 was evaluated in 4 minor field studies (cf. [9]) with approximately 2-10 participants in distributed, asynchronous exercises. A mixed group of researchers and practitioners participated in the first two exercises. The next three exercises in T3 were planned and executed in real settings in the application context. One exercise was cross-border. T2 was evaluated in the functional testing of interfaces and field studies for utility and ease-of-use in more than 30 planning or training sessions. T2 and T3 will also be further evaluated during the Interreg CriseIT2 project.

## 5 The Artefacts

Four software prototypes were developed. T1 and T4 were developed in Norway; T2 and T3 in Sweden. While the prototypes were largely developed in this order, the development processes overlapped somewhat. There were several reasons for constructing four separate tools. First, one of the tools, T1, already existed as a prototype that was refined and adapted to the target group in the project. Second, two of the tools, T1 and T4, are to be commercialized and two, T2 and T4, are open/free to use. Finally, as the tools are still under construction, and in different development stages, the separation
into four different prototypes was deemed most practical. In the future, it is possible
that all or some of them will be integrated with each other (e.g. T1 with T2 and T3, or
T2 with T3). Next, we present the four prototypes.

**T1: The Training Process Management Tool MeTracker.** T1 guides the planner and offers supporting questions and suggestions to ease the planning process. In the first step, the user can create a new training process, edit, or copy established processes. Step 2 guides the user through a Training Need Analysis (TNA) for the training organization. A Main Training Objective (MTO) is established for a longitudinal training process (a series of exercises) and operationalized in Training objectives (TO). Training objectives are categorized into different generic areas of expertise (situational awareness, leadership and organisation, collaboration, decision-making and communication). TOs are then further operationalized by Evaluation Points (EP). In step 3, the training sessions and exercises are planned in detailed Activities. Activities will vary in form and duration, from basic seminars and workshops, to tabletop and input-response exercises, and full-scale exercise(s). Step 4 results in an overview of the planned activities and how the TOs are covered. It is also possible to edit, copy or establish new activities. Step 4 has several options for detailed observation (e.g. photos or video clips) and evaluation based on the predefined criteria. Step 5 is used for the final evaluation, presentation of results and to formulate lessons learned, recommendations and training reports (see Fig. 2). This includes a graphic visualization of present, wanted and achieved status, linked to each area of expertise involved in the training process. All input collected in T1 is stored, and master documents are prepared and issued on the basis of this information. The system provides templates for reports, etc.

<figure><img src="image_3.png" /><figcaption>Fig. 2. The MeTracker system.</figcaption></figure>

With its holistic view of the training process and systematic stepwise approach, T1
is based on well-known theoretical frameworks for management training [21]. It sup-
ports all the phases in Salas et al., although it does not include functionality for the
actual training assignments or exercises. By defining long-term goals and regarding
training as a processes consisting of several training activities, as well as enabling reuse,
T1 is well in line with the recommendations in van Laere and Lindblom [26]. The guidance of the process, the built-in support for training need analysis, etc., all provide a good foundation for improved quality (efficacy) as well as resource efficiency and doing the right things (effectiveness) (c.f. [6]).

**T2: The Individual Training Tool.** The individual training tool (T2) was developed to enable individual training, and has two main user groups: trainers and trainees. The tool is a responsive web application that make it possible to plan and conduct individual training sessions independently of time and place, provided that the device used has an Internet connection. T2 is still an early prototype, however it is still possible to create, share, copy, edit and search existing training assignments. An assignment could for example include a short learning material (text, video etc.) as an introduction, followed by a quiz or open-ended questions. The individual crisis training tool (T2) can be mapped as a useful tool to support Phases 2, 4 and 5 in Salas et al. [21]. It could thus be used to facilitate understanding of the (individual) role, something that van Laere and Lindholm [26] suggested to be a vital part of building up crisis management skills in organizations. The T2 tool offers good possibilities for increased individual training, i.e. improved efficacy (cf. [6]).

**T3: The Collaborative Training Tool.** The T3 tool was developed for collaborative training sessions independently of time, place and device. Like T2, the T3 tool is a (responsive) and free web-based application. In addition, T3 has two main user groups – trainers and trainees – and therefore has interfaces for both creating (planning) and conducting collaborative training sessions. T3 is built for ‘tabletop-like’, discussion-based exercises. The trainer or a group of trainers plan and create an exercise for a collaborative training session. Various manuals and guidelines from authorities at different levels in both countries were important in the design of our artefacts (e.g. [15]). While planning and creating an exercise, the trainer adds modules (sections) which in turn contain the actual content of the exercise such as text describing what had happened and tasks/challenges for the trainees. The tasks are to be solved in collaboration by the exercise team. So far, only text (‘comments’) has been used as input from the trainees but other data formats will also be tested in the future. Modules can be asynchronous or synchronous, i.e. the trainees may perform some or all of the tasks at different points of time, although within a given timeframe. The trainer decides when each module should be accessible to the trainees by giving them a date and time stamp when they should be visible. If necessary, the trainer can intervene in the exercise by communicating with the trainees in chat rooms, changing the modules, their order or pace. Fig. 3 illustrates the collaborative training tool. By clicking on the plus sign to the right, the module is expanded and its content is shown.

The crisis training tool for collaborative exercises (T3) mainly supports Phase 4 but also Phase 5 in Salas et al. [21] and recurrent exercises/series of exercises, as recommended by van Laere and Lindblom [26]. Last but not least, a free and flexible tool such as this, offering distributed and asynchronous exercises, should increase the volume of training at a low cost, thus improving both efficacy and efficiency (c.f. [6]).
<figure><img src="image_4.png" /><figcaption>Fig. 3. The Collaborative training tool.</figcaption></figure>

**T4: The Operational Decision Support System (O-DSS).** The O-DSS is based on Microsoft's Hololens Technology (https://www.microsoft.com/en-us/hololens), which is an application of Augmented Reality (AR). It is designed to assist decision makers in crisis management situations to acquire shared situation awareness, as well as to serve as an environment for training decision making at the operational level. T4 support training on decision support, communication, collaboration etc., distributed or co-located. User interaction occurs through an interface that allows the user to look at the simulated environment through a “virtual overlay” on the real world outside. Using HoloLens goggles, a large map, the size of a standard classroom, is projected “around” the participant. The map consists of a “Disaster Town/Municipal”. Participants can orientate themselves in the map by moving their heads and walking around “in the map”. Some glasses have a “Game Master” feature that allows the carrier to add objects to the map (police car, fire truck, medical car, etc.). Scenarios such as landslides and forest fires are included. Users can manipulate the simulated environment in various ways without using any other “gadgets” than their fingers and hand movements. The O-DSS allows up to four players or decision makers, who can form teams at will or join predefined teams. Each player may make decisions in order to manage an incident, usually involving allocating resources of the right type and volume to the site of the incident. Color-coding of the dome indicates status. For example, red code means that the situation is critical and resources are needed immediately. Points may be rewarded to both team and individual players, in accordance with the successfulness of their allocations. The equipment operates without any PC attached, and connects wirelessly to the internet. This tool has been used for tabletop and input-response exercises and has been tested for both tactical “on-scene command” and for staff training at operational levels. Like T3, this tool mainly supports the fourth phase in Salas et al. [21] but also the fifth and sixth phases, as it is possible to record the training sessions and use the recording for after-action reviews and improved feedback.
# 6 Discussion and conclusions

The purpose of this article is to describe information systems designed in the CriseIT project to support local governments' crisis management training and to discuss how these artefacts could improve crisis management training practices. The IS presented contribute to the computer-based training field by providing novel examples of how the entire crisis training process can be systematically supported from analysis of training needs to follow-up of lessons learnt, while also offering functionality that enable collaboration and reuse. Furthermore, both the role of the trainer and the trainee are supported, by highly flexible training software. We suggest that the IS presented here may improve crisis management training in local and regional government agencies by:

* Enabling more frequent training /exercises incl. repetition [T2, T3], thus improving the potential for increased crisis awareness and maintaining knowledge (cf. [26]).

* Facilitating the involvement of more external actors and (internal) trainees, also for preparing new employees in-between exercises [T2] or training of individual roles.

* Increasing flexibility in training/exercises with the potential to participate in training/exercises anywhere (if Internet access) and anytime (within defined time spans) in shorter sessions [T2, T3].

* Enabling cross-organizational, collaborative planning and the reuse of exercises, providing templates, auto-generating reports, offering lists of common exercise goals, etc. [T1, T3], thereby making planning more resource-efficient and in-creating the variety in exercises.

* Structuring the planning process and guiding the order of activities [T1, T3], thus ensuring quality.

* Offering in-depth support for training need analysis [T1], thereby using resources wisely and ensuring that essential knowledge gaps are filled.

* Making exercises more immersive and varied [T4].

* Providing richer (training) data, and thereby improved evaluations [T1, T2, T3, T4], thus supporting follow-up, and feedback to the trainees as well as transfer of learning.

This paper serves as an introduction to the IS designed in the CriseIT project. These constitute the first components of a digitally supported crisis management training model aimed to complement conventional crisis training methods. The individual tool and their design process will be presented in more depth in forthcoming studies. The tools will also be further evaluated and developed in the CriseIT 2 project. In addition, educational material, methods and guidelines to support the usage of the software (in planning/design of training/exercise as well as during the execution training/exercise) are being developed. Furthermore, all tools, except T1, have only had limited evaluations. The bullet list above are thus mainly visions of technology impacts at this stage. A natural next step is to test them thoroughly with local governments and their partner organizations.
# Acknowledgements

This research was partly funded by EU/Interreg, Sweden-Norway program (20200721).

# References

1. Alter, S.: The Work System Method: Connecting People, Processes, and IT for Business Results. Larkspur, CA: Work System Press (2006).

2. Araz, O. M., Jehn, M., Lant, T., Fowler, J. W.: A new method of exercising pandemic preparedness through an interactive simulation and visualization. Journal of medical systems 36(3), 1475-1483 (2012).

3. Asproth, V., Borglund, E. A., Öberg, L. M.: Exercises for crisis management training in intra-organizational settings. In: Proceedings of the 10th ISCRAM2013 (2013).

4. Bacon, L., Windall, G., MacKinnon, L. (2011): The development of a rich multimedia training environment for crisis management: using emotional affect to enhance learning. Association for Learning Technology. 19, 67-78 (2011).

5. Bakken, B.T. & Haerem, T.: Intuition in Crisis Management: The Secret Weapon of Successful Decision Makers. In: Sinclair, M. (ed.) Handbook of Intuition Research. pp. 122–132, Cheltenham: Edward Elgar (2011).

6. Beynon-Davies, P.: Business information systems. Macmillan International Higher Education (2013).

7. Garzón, J., Acevedo, J., A.: Meta-analysis of the impact of Augmented Reality on students' learning effectiveness, Educational Research Review, https://doi.org/10.1016/j.edurev.2019.04.001 (in press).

8. Hevner, A. R.: A three cycle view of design science research. Scandinavian journal of information systems. 19(2), 87-92 (2007).

9. Hevner, A. R., March, S. T., Park, J., Ram, S.: Design science in information systems research. MIS quarterly. 28(1), 75-105 (2004).

10. Lukosch, H., van Ruijven, T., Verbraeck, A.: The participatory design of a simulation training game. In: Proceedings of WSC '12 (2012).

11. Magnusson, M., Nyberg, L. & Wik, M.: Information Systems for Disaster Management Training: Investigating User Needs with a Design Science Research Approach. In: Proceedings ISCRAM2018 (2018a).

12. Magnusson, M., Pettersson, J., Bellström, P., & Andersson, H.: Developing Crisis Training Software for Local Governments – From User Needs to Generic Requirements. In B. Andersson, B. Johansson, S. Carlsson, C. Barry, M. Lang, H. Linger, & C. Schneider (Eds.), Designing Digitalization (ISD2018 Proceedings). Lund, Sweden: Lund University. ISBN: 978-91-7753-876-9. http://aisel.aisnet.org/isd2014/proceedings2018/General/6 (2018b)

13. Magnusson, M., Öberg, L. M.: Crisis Training Software and User Needs–Research Directions. In Proceedings of the ISCRAM2015 (2015).

14. Meum, T., Munkvold, B. E.: Information infrastructure for crisis response coordination: A study of local emergency management in Norwegian municipalities. In: Proceedings ISCRAM2013 (2013).

15. Myndigheten för samhällsskydd och beredskap (MSB). Övningsvägledning Grundbok – Introduktion till och grunder i övningsplanering (2016).

16. Peffers, K., Tuunanen, T., Rothenberger, M. A., Chatterjee, S.: A design science research methodology for information systems research. Journal of management information systems. 24(3), 45-77 (2007).
17. Peffers, K., Tuunanen, T. & Niehaves, B. Design science research genres: introduction to the special issue on exemplars and criteria for applicable design science research. March 2018, European Journal of Information Systems 27(2):129-139, DOI: 10.1080/0960085X.2018.1458066.

18. Perry, R. W.: Disaster exercise outcomes for professional emergency personnel and citizen volunteers. Journal of Contingencies and Crisis Management 12(2), 64-75 (2004).

19. Peterson, D. M., Perry, R. W.: The impacts of disaster exercises on participants. Disaster Prevention and Management: An International Journal 8(4), 241-255 (1999).

20. Reuter, C., Pipek, V., Müller, C.: Computer Supported Collaborative Training in Crisis Communication Management. In: Proceedings ISCRAM2009 (2009).

21. Salas, E., Wildman, J.L. Piccolo, R.F.: Using Simulation-Based Training to Enhance Management Education. Academy of Management Learning & Education 8(4), 559-573 (2009).

22. Sinclair, H., Doyle, E. E., Johnston, D. M., Paton, D.: Assessing emergency management training and exercises. Disaster Prevention and Management: An International Journal 21(4), 507-521 (2012).

23. Sniezek, J. A., Wilkins, D. C., Wadlington, P. L., Baumann, M. R.: Training for crisis decision-making: Psychological issues and computer-based solutions. Journal of Management Information Systems 18(4), 147-168 (2002).

24. Steinberger, N.: Organizing for the Big One: A Review of Case Studies and a Research Agenda for Multi-Agency Disaster Response. Journal of Contingencies & Crisis Management 24(2), 60-72 (2016).

25. van de Ven, J.G.M., Stubbe, H. & Hrehovcsik, M. (2013). Gaming for policy makers: It's serious!, Games and Learning Alliance Conference 2013 (GaLA 2013)

26. van Laere, J., Lindblom, J.: Cultivating a longitudinal learning process through recurring crisis management training exercises in twelve Swedish municipalities. Journal of Contingencies and Crisis Management, 1-12 (2018).

27. Waller, M.J., Lei, Z., Pratten, R.: Focusing on Teams in Crisis Management Education: An Integration and Simulation-Based Approach. Academy of Management Learning & Education 13(2), 208-221(2014).

28. Wilson, P. J., Gosiewska, S. Multi-agency gold incident command training for civil emergencies. Disaster Prevention and Management 23(5), 632-648 (2014).

29. Yao, X., Konopka, J.A., Hendela, A.H., Chumer, M. & Turoff, M. (2005). "Unleash physical limitations: virtual emergency preparedness planning simulation training, methodology and a case study". Proceedings from the 11th Americas Conference on Information Systems, AMCIS 2005, Omaha, Nebraska, USA, August 11-14, AIS Electroinc Library (SIDeL) s. 1642-1652.