Gaming for Medical Education Research Program: An Investigator-blinded, Randomised Controlled Trial on Serious Games for Doctors, Nurses and Students.
Phase: NA
Conditions: Education, Medical
Sponsor: The University of New South Wales

 
 
FULL STUDY PROTOCOL, 
STATISTICAL ANALYSIS PLAN (SAP) 
& INFORMED CONSENT FORMS (ICF) 
 
 
The Gaming for Medical Education Research (G4MER) Program 
 
ClinicalTrials.gov ID: NCT03666468 
 
Unique Protocol ID: HC17160-LNR/17/SCHN/194 
 
 
VERSION: 1.0 
 
18th October 2018 
 
 
 

Table of Contents 
1. Personnel ................................................................................................................................ 3 
2. Introduction ............................................................................................................................ 4 
2.1 Lay Summary & Aims ..................................................................................................... 4 
2.2 Background Literature Review ........................................................................................ 5 
3. Study Protocol ........................................................................................................................ 7 
3.1 Research Design Outline .................................................................................................. 7 
3.2 Participants ....................................................................................................................... 8 
3.3 Recruitment ...................................................................................................................... 8 
3.4 Randomisation ................................................................................................................ 10 
3.5 Administration ................................................................................................................ 11 
3.6 Assessment & Outcomes Measures ............................................................................... 12 
3.7 Sample Size .................................................................................................................... 14 
4. Statistical Analysis Plan (SAP) ............................................................................................ 16 
4.1 Primary Outcomes .......................................................................................................... 16 
4.2. Secondary Outcomes ..................................................................................................... 17 
4.3 Missing Data .................................................................................................................. 17 
5. Informed Consent Forms (ICF) ........................................................................................... 18 
5.1 Risks to participants ....................................................................................................... 18 
5.2 Privacy and Confidentiality ............................................................................................ 19 
6. References ............................................................................................................................ 20 
 
 
 

1. Personnel 
 
Principal Investigator 
A/Prof Chee Y. (Keith) Ooi 
 
 
Investigators 
Dr Penelope Uther 
 
Dr Michael J. Coffey 
 
Dr Sean Kennedy 
 
Dr Adam Bartlett 
 
Dr Silas Taylor 
 
Mr Andrew Lovell-Simons 
 
Ms Janaya Perron 
 
Dr Ashlene McKay 
 
Dr Millie Garg 
 
Dr Sarah Lucas 
 
Dr Zoe Stephensen 
 
Mr Stephen Best 
 
Ms Carolyn Smith 
 
Ms Isabelle McKay 
 
Ms Madeline Sharp 
 
Ms Nerralie Shaw 
 
 

2. Introduction 
2.1 Lay Summary & Aims 
Medicine has traditionally been learned through the face-to-face lecture, clinical experience 
on hospital wards, and student self-direction to additional resources such as textbooks and 
clinical guidelines.  Recently educators have sought to improve student learning by providing 
adjuncts to these methods. The rapid development of computerized technology has seen the 
emergence of online educational packages in medical programs throughout the world.  
‘PlayMed’ is a serious computer game that has been developed to teach paediatrics to 
medical students.  It allows students to learn and apply clinical algorithms for managing 
patients in a safe, simulated environment. Preliminary data gained from a pilot study suggests 
that PlayMed improved students’ knowledge of asthma management. However, the efficacy 
of PlayMed as a learning tool has not been tested against alternative online educational tools, 
therefore we propose to conduct a randomized control trial of PlayMed using an online 
educational package as a control. Furthermore, we aim to assess the utility of PlayMed in 
different populations including nurses and doctors. 
We have targeted two common and important paediatric clinical presentations (asthma and 
seizures) as the focus of our teaching and assessment in this study.  It is likely that students 
will be exposed to both of these conditions during their clinical attachments; the acute 
management of both conditions is outlined in current NSW Health Paediatric Clinical 
Practice Guidelines (CPG); and both asthma and seizure disorders are listed in the Australian 
Curriculum Framework For Junior Doctors. The current UNSW Phase 3 Children’s Health 
course includes one lecture on each clinical problem.  
 

Under the “G4MER” Program, we plan to perform a series of randomised control trials to see 
if PlayMed is an effective teaching adjunct to our standard course(s), and if this game is more 
effective than other online packages for medical students, doctors and nurses.  We 
hypothesise that participants exposed to the game (in addition to standard teaching) will 
demonstrate superior knowledge and clinical skills compared to those exposed to the 
alternative online packages, and to those who only receive standard teaching.  Additionally, 
we hypothesise that increased time spent playing the game will improve performance in 
knowledge-based and clinical assessments.   
 
2.2 Background Literature Review 
The efficacy of both online learning and simulation-based teaching have been well 
established in Medical Education.1, 2  Online learning is most effective when students are 
required to interact either with the package or with each other, and feedback is provided.1  
Simulation is effective as it allows learners to engage in repetitive practice, in a safe 
environment, with a degree of clinical variation and a range of levels of difficulty. Timely 
feedback is also vital to effective learning.3   
Computer-based simulation games can be defined as instruction delivered via personal 
computer that immerses trainees in a decision-making exercise in an artificial environment in 
order to learn the consequences of their decisions.4  Computer-based simulation games 
combine the most effective elements of both online learning and simulation - the online 
environment provides an easily accessible, interactive yet safe space for students; feedback is 
provided to the student, and clinical variation with a range of levels of difficulty can be 
provided. 

Little empirical evidence on the effectiveness of educational video games on students’ 
educational outcomes is available.  A recent innovative study at UNSW by Dobrescu et al5 
compared the effectiveness of teaching introductory economics via a video game or 
traditional textbook learning. This study suggested that there was no difference in exam 
performance for either modality; however, the video game was considerably more enjoyable 
as a learning tool. To our knowledge there are no educational role-playing games available on 
the market which targets the education of medical or nursing students. Currently online 
learning packages for paediatric clinical practice guidelines are available through HETI 
Online (https://hetionline.cit.health.nsw.gov.au/). At present these are not part of mandatory 
training. 
The development of an online game is an expensive and resource intensive process but may 
well be an effective adjunct to current teaching programs.  We wish to test this product to see 
if further investment is worthwhile.   
 
 
 

3. Study Protocol 
3.1 Research Design Outline 
The “G4MER” Program aims to perform a series of randomised control trials on different 
groups at Sydney Children’s Hospital Randwick: 
(1) Study 1A (Formerly titled “Serious Games in Medical Education – a Randomised 
Control Trial”) involves Phase 3 medical students at UNSW having 8 weeks access to 
the game, an Online Package (OP) or NSW State Guidelines on Asthma and Seizure 
management (control). Students are then assessed using multiple choice questions 
(MCQ) and two observed structure clinical examination (OSCE) stations (detailed 
below). 
 
(2) Study 1B is also a randomised control trial similar in design to Study 1A, however it 
will involve doctors and nurses employed at Sydney Children’s Hospital. PlayMed 
will be compared against the HETI Learning Path Paediatric Clinical Practice 
Guidelines (Online Package for staff). 
 
(3) Study 2A is identical to Study 1A except participants will have 5 days access to their 
educational tool. Participants will then undergo the same assessment using multiple 
choice questions (MCQ) and two observed structure clinical examination (OSCE) 
stations. 
 
(4) Study 2B is identical to Study 1B except participants will have 5 days access to their 
educational tool. Participants will then undergo the same assessment using multiple 
choice questions (MCQ) and two observed structure clinical examination (OSCE) 
stations. 

 
The proposed study designs are all investigator blinded randomised control trials. Participants 
may only be involved in Study 1 or Study 2, not both.  
 
3.2 Participants 
Studies 1A and 2A 
Medical students in their second phase (years 3 or 4) and final phase (years five and six) of 
their medical degree (the BMed/MD program) at UNSW, currently completing their 
Children’s Health course at Sydney Children’s Hospital, will be eligible to participate in this 
study.  
Studies 1B and 2B 
Doctors and Nurses employed at Sydney Children’s Hospital will be eligible to participate in 
this study.  
 
Study investigators will not be eligible to participate. 
Participants will require a personal computer and access to the internet to participate.   
 
3.3 Recruitment 
Studies 1A and 2A 
All students enrolled in the UNSW Phase 3 Children’s Health course will be informed of the 
study during their face-to-face orientation by the Course co-ordinators (the course co-
ordinators are neither involved in the research nor in student assessment); a slide outlining 

what will be said has been attached.  Students will be told that a computer game, developed to 
teach medical students paediatrics, is being tested by randomised control trial, and that 
volunteer Children’s Health students are invited to participate. Additionally, they will be 
informed that all participants will have the opportunity to go to the simulation laboratory and 
manage a sick child as a formative assessment with feedback provided.  
Students will be informed at this time that their decision to participate, or not to participate, in 
the study, will not affect their relationship with the university or with the hospital in any way. 
Neither will the results of their study assessment contribute to their summative course 
assessment.  Feedback with the answers to the MCQs will be available at completion of 
study, which should be before their end of 6th year summative assessment.  
The participation information and consent sheet will be distributed which details including 
inclusion and exclusion criteria.  Interested students will be invited to attend an information 
session scheduled in a break in their lecture timetable, this information session will allow 
study co-ordinators to answer questions from potential participants.   
 
Studies 1B and 2B 
All medical and nursing staff at Sydney Children’s Hospital will be approached during Grand 
Rounds, lunchtime and evening teaching sessions, and via a concurrent staff email. Study 
advertisement posters will also be placed around the hospital. Recruitment will occur 
periodically throughout the calendar year (4 time points). Participants will be told that a 
computer game, developed to initially teach medical students paediatrics is being tested by 
randomised control trial, and that volunteer medical and nursing staff are also invited to 
participate. Given that the teaching tools are aimed at patient management, they are 
appropriate to administer to medical and nursing staff. Furthermore, the HETI Online 
Paediatric Clinical Practice Guidelines and NSW state guidelines are applicable to staff at 

Sydney Children’s Hospital. Additionally, they will be informed that all participants will have 
the opportunity to go to the simulation laboratory and manage a sick child as a formative 
assessment with feedback provided.  
Participants will be informed at this time that their decision to participate, or not to 
participate, in the study, will not affect their relationship with the hospital in any way. 
Feedback with the answers to the MCQs will be available at completion of study. Participants 
will be asked not to share the answers with other staff. 
 
The participation information and consent sheet will be distributed which details including 
inclusion and exclusion criteria.  Interested staff will be invited to attend an information 
session scheduled at Grand Rounds, this information session will allow study co-ordinators to 
answer questions from potential participants.   
 
All participants will have to return a signed consent form to the student coordinators prior to 
random allocation to a study group. 
 
3.4 Randomisation 
 
Students and Staff who consent to participate will be randomly allocated (within their study) 
to one of three groups (at a ratio of 1:1:1):  the game, the OP, and no intervention (ie standard 
teaching only with NSW State Guidelines on Asthma and Seizure management provided). 
Computer randomisation (https://www.sealedenvelope.com) and concealed envelopes will be 
used.  We will use a separate unique block randomisation for each study (1A, 1B, 2A and 2B) 
with block sizes of 3, 6 and 9. A strata for job (Doctor / Nurse) will be incorporated for staff 

in studies 2A and 2B. Participants will be allocated a unique identification number, and this 
will be recorded against their student or staff number and group allocation in an electronic 
database.  This will be done by the student co-ordinators at the School of Women’s and 
Children’s Health to ensure the investigators remain blinded.   
 
3.5 Administration 
 
Studies 1A and 1B 
Participants allocated to PlayMed and to the OP will be given the appropriate access for 8 
weeks (instructions provided in the study envelope). Participants allocated to the control 
guidelines will receive a print-out of the guidelines. Participants will be encouraged to engage 
with their additional educational tool as often as they wish during their eight weeks.  In the 
8th last week participants will have their knowledge and clinical performance assessed as 
outlined below.  
 
Studies 2A and 2B 
Participants allocated to PlayMed, OP and control guidelines will be given 5 days to utilise 
their assigned teaching tool.  Five days was selected as an appropriate time-frame in which 
participants would be able complete all cases of the game or online package, or read through 
the guidelines with minimal impact on their personal and academic schedules.  Participants 
will receive an email with access to their intervention from an unblinded administrator.  
Participants will have their knowledge and clinical performance assessed as outlined below. 
 

 
3.6 Assessment & Outcomes Measures 
 
Studies 1A and 2A 
Participants will be assessed for: 
1. Primary outcomes 
• Knowledge acquisition  
o Participant knowledge will be assessed using a 10-point single-best option 
multiple-choice questionnaire (MCQ); correct answers, with an explanation, 
will be provided to all participants at the end of the study via email. 
• Clinical performance 
o Participant clinical skills will be assessed via an observed structured clinical 
examination (OSCE) administered in the simulation laboratory (immediately 
after the knowledge test); participants will be tested across two clinical 
scenarios.  Strict marking criteria will be used to ensure standardisation, with 
a total possible maximum score of 30.  Individual feedback will be provided 
by assessors immediately after each scenario. 
 
2. Secondary outcomes 
• Educational experience 
o Participants will be asked to complete a short survey using 5-point Likert 
scale (strongly agree, agree, neutral, disagree, strongly disagree). Students 
will be asked for comments (free text responses) regarding their engagement 
with, and perception of the educational value, of the learning tool. 

 
The time commitment required of participants for assessment will be approximately 60 mins. 
 
Studies 1B and 2B 
Participants will be assessed for: 
1. Primary outcomes 
• Knowledge acquisition  
o Participant knowledge will be assessed using a 15-point single-best option 
MCQ; correct answers, with an explanation, will be provided to all 
participants at the end of the study via email. 
• Clinical performance 
o Participant clinical skills will be assessed via an OSCE administered in the 
simulation laboratory (immediately after the knowledge test); participants will 
be tested across two clinical scenarios.  Strict marking criteria will be used to 
ensure standardisation, with a total possible maximum score of 50.  Individual 
feedback will be provided by assessors at the conclusion of the OSCE portion 
of the assessment. 
 
2. Secondary outcomes 
• Educational experience 
o Participants will be asked to complete a short survey using 5-point Likert 
scale (strongly agree, agree, neutral, disagree, strongly disagree). Students 

will be asked for comments (free text responses) regarding their engagement 
with, and perception of the educational value, of the learning tool. 
• Future impact on clinical practice 
o Participants will also complete a short post-assessment survey 3 months after 
their formative assessment date.  The short survey contains 7 questions and 
should take 5 minutes to complete.  This survey is powered by Qualtrics 
(https://www.qualtrics.com/au/). 
 
3.7 Sample Size 
 
Study 1A 
GLIMMPSE (http://glimmpse.samplesizeshop.org/#/) was utilised for sample size calculation 
with equal group sizes (i.e ratio of groups 1:1:1). In our initial pilot study, the mean quiz 
scores were 7.0 (SD 1.4) and 6.0 (SD 1.4) for the PlayMed and control cohorts (standard 
teaching only) respectively.6 If we assume that students in the OP cohort will achieve quiz 
scores of 50% the difference between PlayMed and controls (i.e. score of 6.5), then the 
following sample size is required: 
• A total of 54 subjects (i.e. 18 in each group) to be able to reject the null hypothesis 
that the population means of the experimental and control groups are equal with 
probability (power) 0.8. The Type I error probability associated with this test of this 
null hypothesis is 0.05. 
• Assuming a total drop-out rate of 20%, we will recruit 22 subjects for each group, i.e. 
18 PlayMed, 18 OP and 18 controls.  
 

 
 
Study 2A 
Given we have no preliminary data for doctors and nurses, the same sample size estimate will 
be used for this study. We will recruit 22 subjects for each group, i.e. 22 PlayMed, 22 OP and 
22 controls. 
 
Studies 1B and 2B 
The same sample size estimates will be utilised for these studies. For each study we will 
recruit 22 subjects for each group, i.e. 22 PlayMed, 22 OP and 22 controls. 
 
 
 

4. Statistical Analysis Plan (SAP) 
 
All Studies 
Descriptive and inferential statistics will be used. Normally distributed data will be presented 
by mean and standard deviation (SD) and non-normally distributed data will be presented by 
median and interquartile range (IQR). Binary and categorical data will be presented using 
counts and percentages. R v3.4.4 will be used for all statistical analyses.  
 
4.1 Primary Outcomes 
 
All Studies 
The primary analyses will compare: (1) PlayMed vs OP; and (2) PlayMed vs control 
guidelines on their mean/median: (i) MCQ score, and (ii) OSCE scores. Comparisons will be 
made using an unpaired t-test or a Mann-Whitney test for parametric and non-parametric 
data, respectively. P-values <0.05 will be considered statistically significant. 
Effect sizes for will be calculated using Cohen’s d (d) with 95% confidence intervals (95%CI) 
using the ‘effsize’ package.7 Effect sizes will be considered small if 0.2 ≤ d < 0.5, medium if 0.5 ≤ d < 
0.8, and large if d ≥ 0.8.  
 
 
 
 

4.2. Secondary Outcomes 
 
All Studies 
Descriptive statistics will be used to explore Likert scale data and reported as percentage of 
participants who “strongly agree”, “agree”, “neutral”, “disagree”, or “strongly disagree”. 
Qualitative data will be explored thematically using a document analysis approach.  Data will 
be manually transcribed into NVIVO ™ which will assist with data immersion.  Labels will 
be created and assigned to text, which describe key meanings and ideas; with each successive 
pass codes will move from descriptive to interpretive as key themes are identified.   
 
4.3 Missing Data 
 
The primary analyses will be performed using a modified intention-to-treat (ITT) basis. For 
participants who are randomised and do not attend for assessment, a mean imputation of the 
assessment scores from their allocation group will used. A sensitivity, per-protocol analysis 
will also be performed, with no restrictions placed on the minimum time spent using the 
allocated intervention.  
 
 
 
 
 

5. Informed Consent Forms (ICF) 
 
Studies 1A and 2A 
Written consent will be gained from the Medical students using the consent forms attached to 
the application.   
 
Studies 1B and 2B 
Written consent will be gained from the medical and nursing staff using the consent forms 
attached to the application.   
 
5.1 Risks to participants 
 
The benefits outweigh the risks to participants.  Participants have access to an additional 
formative assessment; they all will also have access to an additional learning resource as all 
participants will be given access to the additional online learning tools (games and online 
package) at the completion of the study. All participants will receive feedback about the 
OSCE and the answers to the MCQs after study completion, which are further opportunities 
for learning.   
 
There are few risks to participants.  All participants would have experience of high stake 
OSCEs from previous examinations and personal development training and all will be very 
experienced in MCQ exams. We do not expect that either assessment for the purpose of this 
study will be overly stressful. All students will be made aware that the study assessments will 
not contribute to their course grades. The feedback provided after the OSCE will be 

supportive and clearly explained as formative in nature. There will be no coercion to take part 
in either assessment task. All students will have access to student support services should 
they suffer significant stress or anxiety. All staff will have access to staff support services 
should they suffer significant stress or anxiety. 
 
5.2 Privacy and Confidentiality 
 
All study participants will be de-identified and given a unique ID number at the time of 
enrolment; this will be recorded by the School of Women’s and Children’s Health student co-
ordinators and emailed to the participant.  The key linking study participants to their unique 
ID number will only be accessed by the student co-ordinators and will be stored in the 
password protected UNSW folder on the secure hospital (South Eastern Sydney Local Health 
District) hard drive.  This will be destroyed after the 7-year retention period.  Participants will 
not be able to be identified from any results published.   
 
 

6. References 
1. Cook, D. A., Levinson, A. J., Garside, S., Dupras, D. M., Erwin, P. J., & Montori, V. 
M. (2010). Instructional design variations in internet-based learning for health 
professions education: a systematic review and meta-analysis. Academic 
medicine, 85(5), 909-922. 
 
2. Cook, D. A. (2014). How much evidence does it take? A cumulative meta‐analysis of 
outcomes of simulation‐based education. Medical education, 48(8), 750-760. 
 
3. Barry Issenberg, S., Mcgaghie, W. C., Petrusa, E. R., Lee Gordon, D., & Scalese, R. J. 
(2005). Features and uses of high-fidelity medical simulations that lead to effective 
learning: a BEME systematic review. Medical teacher, 27(1), 10-28. 
 
4. Sitzmann T. (2011). ‘A met-analystic examination of the instructional effectiveness of 
computer-based simulation games. Personal Psychology 64, 489-528. 
 
5. Dobrescu LI, Greiner B, and Motta J. (2015). Learning economics concepts through 
game-play: an experiment. International Journal of Educational Research 69, 23-37.  
 
6. Coffey M, Ooi CY. PlayMed - Bridging the Gap Between Student and Doctor with Serious 
Games. Paper presented at: AMEE : An International Association for Medical Education 
2017; Helsinki. 
 
7. Cohen J. Statistical power analysis for the behavioral sciences. Hillsdale, N.J.: L. Erlbaum 
Associates; 1988.