Pandemic-Proofing Simulation-based Education: Development and Evaluation of Interactive Virtual Educational Content for Medical Trainees
Phase: NA
Conditions: COVID 19
Sponsor: Ottawa Hospital Research Institute

 
 
 
 
 Pandemic-Proofing Simulation-Based Education: 
Development and Evaluation of Interactive Virtual 
Educational Content for Medical Trainees 
 Glenn Posner, Kaitlin Endres, Michael O’Brien, Vicki LeBlanc, Fahad Alam 
 
April 15, 2023 
 

COVID-19 Pandemic Response Funding Program 
Amended Protocol April 2023 
 
Background: During the global pandemic of 2020, simulation centres were forced to close or 
significantly curtail activities in an effort to reduce the spread of COVID-19. For 4 months, medical 
students and residents were unable to participate in simulation-based training that are core to their 
established curriculum. At the University of Ottawa Skills and Simulation Centre (uOSSC), this resulted 
in the cancellation of approximately 352 sessions with an estimated loss of 28,000 learning hours. As 
regions gradually reopen, significant restrictions in group sizes mean that in-person experiential 
learning will continue to occur at a fraction of pre-pandemic times. When combined with restricted 
access to clinical environments for learners, this presents an important challenge for the preparation 
of future and current health care workers, and thus patient safety1. Many of the skills learned during 
hands-on simulation sessions are difficult to recreate trough lectures or small group discussions1. The 
goal of this project is to develop and evaluate virtual reality (VR) simulation modalities for their 
effectiveness in preparing residents to manage patients in uncertain and time-pressured conditions.  
 
Traditional theatre-based simulation sessions require the physical presence of interdisciplinary groups 
of learners caring for a physical mannequin-based patient, where social distancing is challenging. 
However, the rapidly evolving field of virtual reality (VR) applied to healthcare education2,3 may 
provide effective physically distanced solutions should another pandemic or second wave impacts our 
ability to deliver face-to-face educational content. Virtual reality is defined as “a real or simulated 
environment in which a perceiver experiences telepresence”, with telepresence defined as “the 
experience of presence in an environment by means of a communication medium”.  
 
Imagine a virtual classroom - learners wearing VR headsets in the safety of their home are immersed 
in a computer-generated simulation centre. Through their avatars, they can interact from a first-
person perspective with their environment, colleagues and instructors. The virtual patient has been in 
a car accident and the learners, by interacting with the virtual trauma bay environment, must work 
together to perform a primary trauma survey and resuscitate the patient. The interactive VR 
environment provides the team with real time information in a realistic environment that includes the 
uncertainty, noise, and time pressures of a real case. At several key points in the VR video, learners 
are presented with interactive decision points, and their choices determine how the rest of the video 
unfolds 
 
Alternatively, imagine a second immersive environment based on high-definition 360o video obtained 
from our simulation centre. Single learners, wearing a VR headset, find themselves in the 
resuscitation room, at the foot of the bed, leading a trauma team through the assessment and 
resuscitation of a patient in a virtual scenario. An example of a non-interactive VR video developed by 
our team for the obstetrical curriculum can be found at (https://youtu.be/AqdwmHPeoBY).   
 
In both the interactive VR and 360o video experiences as well as the pre- and post-test traditional 
theatre-based simulations, our study will aim to measure participants’ heart rate (HR), heart rate 
variability (HRV) and oxygen saturation levels as markers of arousal and stress responses. These will 
serve as a physiological indication for: 1) their comfort and change in comfort when leading scenarios 
and making decisions for critically ill patients in real-time and 2) their emotional engagement in their 
own learning throughout the immersive experiences.  
 

The properties of VR hypothesized to result in effective learning - over lectures, passive videos or 
discussion groups - include those of vividness (representational richness with which information is 
presented to the various senses) and interactivity (the ability to actively modify an environment). 
Increased vividness and interactivity can lead to greater emotional, motivational and cognitive 
engagement with learning materials. In turn, this greater engagement can lead to more active 
learning (analyzing, synthesizing and evaluating information rather than passively receiving it), and 
thus better knowledge retention and ability to apply this knowledge for subsequent problem solving.  
To date however, work related to VR has been targeted towards development of the technological 
resources to permit realistic and immersive experiences. Significant advances have led to this 
technology being readily affordable and accessible. However, little research has been conducted to 
determine the educational effectiveness of virtual simulations. Both interactive VR and 360o video 
provide immersive and interactive components; however, 360o video is arguably more of a passive 
experience for the learner. We felt that our constructed interactive VR simulation may provide more 
opportunity for interactivity for the learner and encourage them to participate in more active 
decision-making, but may increase extraneous cognitive load for the learner given the new 
environment. Both interactive VR and 360o video techniques require a significant investment of time 
and resources, but we do not yet know whether these techniques are equivalent in terms of learning 
or if one significantly differs from the other. There is no evidence to suggest that one method is 
superior to the other, therefore we chose to do a comparison study to investigate for any significant 
difference between these simulation experiences.   
 
 
The present application is for the development and evaluation of two VR simulations (interactive VR, 
360o video) applied to the context of Emergency Medicine. The aim of this pilot study is to develop 
two VR simulations, and to compare their effectiveness with traditional theatre-based simulations.  
Given the novelty of both interactive VR and 360o video simulations, our primary objective is to 
compare knowledge retention and application of knowledge during a paper quiz and traditional 
theatre-based emergency department simulations following preparation with either 360o video 
training or interactive VR training. The secondary objectives are to assess the usability and feasibility 
(resources), as well as the emotional engagement, of the above three modalities. 
 
 
Methods: 
Phase 1: Creation of Content: A software engineer with expertise in video-game development will be 
hired to develop the computer-generated emergency department in collaboration with the educators 
and clinicians on the team. A theatre-based trauma scenario from the current uOttawa Emergency 
Department curriculum will be adapted to the VR platform.  This virtual environment will allow a 
team of students to manage a patient simultaneously through their avatars, from a first-person 
perspective. In parallel, a well-managed depiction of the same scenario will be scripted and recorded 
using a 360-degree HD video camera. The recording will be made using the investigators as actors, 
with additional interprofessional volunteers recruited as needed.  These videos will be edited and 
then converted to an application that can be uploaded to the Oculus© platform. Both platforms will 
undergo piloting for content and usability. The amount of time, money and other resources required 
to develop each of these scenarios will be calculated. 
 
Phase 2: Evaluating the intervention:  

Participants: 40 residents and medical students will be recruited to participate in the evaluation of the 
VR modules. Participants will be randomized to one of two groups (20 participants per group) – 
interactive VR or 360o video education.  See Figure 1. 
 
Pre-Test: Prior to participating in one of the two educational interventions (360o-video or interactive 
VR), participants will complete a pre-test of their content-specific knowledge and knowledge 
application with respect to basic presentations to the emergency department. All participants will 
complete a video-recorded simulation of an emergency department case, half will complete Case A 
while the remaining half will complete Case B. These cases will be counter-balanced at post-test in 
order to account for any possible differences in case difficulty. All participants will also complete a 
knowledge test which will consist of approximately 20 multiple-choice questions with a mark out of 
20 for a minimum score of 0 and maximum score of 20. Questions will be based on Advanced Cardiac 
Life Support (ACLS) guidelines and algorithms. Team performance will be recorded for subsequent 
scoring (see measures and analysis section below). During the traditional theatre-based simulation 
sessions, we will measure the learners’ HR and HRV with the use of a Polar H10TM chest belt linked 
with a commercial capture and analysis software (EliteHRVTM) program on an iPad. This will allow us 
to measure both arousal and stress responses, as physiological indication of the learners’ emotional 
engagement.  
 
Intervention: Following the pre-test, participants will either participate in the interactive VR or the 
360o educational video. In each condition, the scenarios will be matched for duration, and will be 
followed by a debriefing session focused on the learners’ comprehension of the case, their rationales 
for decisions, as well as closing any observed gaps in performance. UOSSC-affiliated instructors with 
formalized simulation and debriefing training will lead the debriefing. The learners’ HR and HRV will 
be captured during the simulation and the debriefing sessions as a measure of engagement during the 
activity. 
 
Post-Test: Two weeks after the education session, all participants will complete a post-test of their 
knowledge and knowledge application (matched in difficulty with pre-test), as well as lead a theatre-
based simulated case. Participants who completed Case A will now complete Case B and vice versa. 
The sessions will be videotaped for subsequent rating of clinical and team performance. We will also 
capture their HR and HRV again as measures of physiological arousal and stress. 
 
Measures and Analyses: 
 
This will be an equivalency study which hypothesizes that there will not be a difference in learning 
between 360-VR or interactive VR. The primary outcome measures for this study will be changes in 
knowledge and in clinical performance from pre-test to post-test. Two experienced simulation 
instructors will view and score the videos, blinded to the group allocation and whether the scenario is 
pre or post the intervention. The raters will evaluate the learners using the Ottawa GRS scale as well 
as a checklist of expected actions for the scenario. The raters will independently rate the videos, and 
their level of agreement will be calculated using an intra-class correlations coefficient. The averaged 
scores, as well as scores on the knowledge tests, will be submitted to separate mixed-design analyses 
of variance (ANOVA) with time (pre, post) as a repeated-measure and groups (interactive VR, 360o 
video) as a between-subject variable. Post-hoc analyses will be conducted with relevant paired or 
unpaired t-tests.  
 

The secondary outcomes measures are arousal and stress responses during pre and post theatre-
based simulation sessions, educational sessions, and debriefing sessions. Mean and peak HR and HRV 
(rMSSD, pNN50) will be submitted to separate ANOVAs, with time (pre-test, debriefing, post-test) as a 
repeated measure and group (interactive VR, 360o video, theatre-based) as a between-subject 
variable. Post-hoc analyses will be conducted with relevant paired/unpaired t-tests. 
 
Another secondary outcome measure will be the cost-effectiveness of the education interventions, 
based on Levin’s cost effectiveness model4 applied to simulation-based education5. The cost 
categories will include equipment and materials (e.g. market price of equipment, maintenance cost), 
personnel costs (e.g. staff salary, volunteer time), facility costs (e.g. facility rental fee, facility 
maintenance), required client inputs (e.g. learner costs, opportunity costs), and other program costs 
(e.g. communications fees, servers for storage and retrieval of information). These will be summed 
into a dollar estimation for each modality, and compared using a one-way between-subject ANOVA, 
with group (interactive VR, 360o video) as the between variable.   
 
Outcome: After six months, we will have developed interactive, virtual educational content for 
training medical students and residents in emergency medicine using two different VR modalities. 
These modalities will be easily scalable to other programs across the Faculty of Medicine. In addition, 
the evaluation phase will result in data to help support decisions regarding which virtual modalities 
are best suited to distanced simulation, in terms of educational effectiveness (knowledge, clinical 
performance and emotional engagement) and required costs. The calculations of costs associated 
with each modality will also allow for fact-based estimations of future investments in virtual-reality 
education. 
 
Budget Request: $50,000 for 1 year 
 
Budget Justification: 
 
Item 
Price 
Total 
Computer Engineer 
80,000/year, for 6 mos. 
$40,000 
Research Assistant 
5 hrs/week for 6 mos. X 
$40.hr 
$7,200 
Oculus Quest headset 
 
$800 
Insta360 ONE X camera bundle 
with tripod 
 
$600 
InstaVR app VR platform 
$199 USD/month = 270 CDN 
x 5 mos. 
$1350 
 
 
$49,950 
 
 
Computer Engineer: A software engineer, with expertise in computer-generation of environments for 
virtual platforms, will be hired on a full-time basis for a period of 6 months. During this period of time, 
the individual will work with the team to create an interactive VR trauma bay scenario (that can 
subsequently be adapted to other clinical content), as well as develop the 360o video based on 
recordings made by the team.  
 

Research Assistant: The research assistant will support the team in the recruitment, scheduling, data 
collection, and data entry and analysis elements of the project. We will contract a skilled research 
assistant from DIME’s Research Support Unit, at their standard rate of $40/hr 
References 
 
1 Cook, David A., Ryan Brydges, Stanley J. Hamstra, Benjamin Zendejas, Jason H. Szostek, Amy T. Wang, Patricia J. Erwin, 
and Rose Hatala. "Comparative effectiveness of technology-enhanced simulation versus other instructional methods: a 
systematic review and meta-analysis." Simulation in Healthcare 7, no. 5 (2012): 308-320. 
 
2 Drummond D, Hadchouel A, Tesnière A. Serious games for health: three steps forwards. Adv Simul (Lond). 2017;2:3. 
Published 2017 Feb 4. doi:10.1186/s41077-017-0036-3 
 
3 Pasquier P, Gaudry S, Tesniere A, Mignon A. New insights into virtual medical education and assessment, Serious Games, 
and Digital Platforms. Bull Acad Natl Med. 2015;199(7):1153-1164. 
 
4 Levin, H. M., & McEwan, P. J. (2000). Cost-effectiveness analysis: Methods and applications (Vol. 4). Sage. 
 
5 Zendejas, B., Wang, A. T., Brydges, R., Hamstra, S. J., & Cook, D. A. (2013). Cost: the missing outcome in simulation-based 
medical education research: a systematic review. Surgery, 153(2), 160-176. 
 
 
 
Figure 1: Study Design