Can an App Improve the Examination Scores of Family Medicine Residents? A Cluster Randomized Controlled Trial
Phase: NA
Conditions: Behavior
Sponsor: Jewish General Hospital

GRAD, Roland   
   
    
   
  Project: The Study Guide cluster RCT  
0  
  
 
 
 
The Study Guide Trial 
2017: The Study 
Guide Cluster 
Randomized Control 
Trial 
NCT03312218 
MARCH 2017 

GRAD, Roland   
   
    
   
  Project: The Study Guide cluster RCT  
1  
  
Each year, over 1,200 residents challenge the Family Medicine certification exam. Scores obtained by 
candidates on national licensing exams are positively associated with the quality of patient care provided 
in future practice.1-5 Therefore, educational methods to optimize residency training are vitally important.   
This research is needed to improve residency training. Our objective is to deliver a trial-validated mobile 
application as a tool for improving the knowledge of future physicians. Through this definitive trial, we 
can advance residency training for thousands of physicians, for years to come.  
  
Background  
As a guide for residents preparing for their certification exam, the College of Family Physicians of 
Canada (CFPC) elaborated the concept of Priority Topics.6 A major challenge for residents is knowledge 
retention.7,8 To address this challenge, the strategy of spaced education emerged from research in 
educational psychology.9 Spaced education takes advantage of (1) the spacing effect and (2) the testing 
effect.10,11 With the spacing effect, students can increase their retention of information by interval 
rereading of content. The testing effect refers to the following: when combined with answer feedback, the 
process of testing one’s knowledge alters learning itself, to improve recall.12-15 Greater resident 
engagement in self-learning is needed to implement spaced education. In this context, it is critical to test 
interventions to enhance such engagement. Building on the findings of our prior CIHR-funded research 
and a pilot study, this proposal is grounded in a ‘qualitative knowing’ of the context of residency training 
in Family Medicine.16  
Literature: Randomized controlled trials demonstrate that spaced education can boost long-term 
knowledge retention among students and residents in Urology.17-21 Outside of Urology, we found four 
small single-site trials of spaced education in residency (General Surgery, Oncology, Internal Medicine 
and Pediatrics).22-25 We found no published studies of spaced education in the context of Family 
Medicine. Guided by a specialized librarian, our searches were conducted iteratively in Google Scholar 
and three databases - PubMed, EMBASE, and ERIC. We provide our search strategies, and a summary of 
findings table, in the appendix.  
  
Importance  
This research is important for five reasons. (1) Foundational knowledge guides decision-making in 
practice; yet the Canadian family medicine residency is the shortest among developed nations, at just two 
years. (2) The implementation of work hour restrictions has reduced the time for learning in group 
settings. While the time to teach in residency may never be sufficient to master the body of knowledge 
required for practice, the importance of foundational knowledge is supported by observational studies 
showing correlations between certification exam scores (competence) and performance in actual practice. 
Please see the appendix for a definition of these terms. For example, Tamblyn et al. reported that family 
medicine residents achieving higher scores on their certification exam were more likely as family 
physicians in their early years of practice to prescribe fewer contraindicated drugs. Initial assessment of 
these outcomes was limited to the first 18 months of practice. Subsequently, Tamblyn et al. reported a 
sustained positive relationship between certification exam scores and performance over 4 to 7 years in 
practice.3,4 While observational studies like these do not allow for causal inference, interventions to 
improve the knowledge of future family physicians should improve health care for Canadians, given the 
importance of primary care to health systems.26 (3) Although residency training in Family Medicine does 
not yet leverage the benefit of spaced education, our pilot study reveals the feasibility of using a mobile 
app to implement this instructional approach. In line with our conceptual framework27, in a manuscript 
under review and in conference presentations we have reported that alerts from a mobile app stimulate 
residents to read exam-relevant clinical information. In this pilot study, we offered an app on the 99 
Priority Topics to first-year residents at McGill and McMaster. Respectively, 93% and 84% of all 
incoming residents were consented.28 Through the app, McMaster residents received a weekly alert to one 

GRAD, Roland   
   
    
   
  Project: The Study Guide cluster RCT  
2  
  
priority topic. McGill residents were given the same app but received no alerts, providing them with 
information on demand. After 100 days, we performed an interim analysis on the number of opened pages 
in the app, using priority topics as pairs. Intervention group residents opened more pages of information, 
per priority topic (15.7 ± 21.8 vs 9.1 ± 14.4, p < 0.0003). These findings are promising. (4) There are no 
studies of spaced education in the context of Family Medicine training. (5) Our ability to conduct the 
proposed work is enhanced through a partnership with the CFPC. Thus, our proposed research is feasible, 
innovative and supported by pilot study findings.   
  
Research question  
Among family medicine residents, does enhanced feedback through an app providing alerts and test 
questions (spaced education), compared to the same app without alerts and enhanced feedback (no 
spaced education), improve competence as measured by the certification examination?  
  
Design: We propose a cluster randomized controlled trial. For this trial, a medical school is a natural 
cluster of residents, with no crossovers. This study will be registered at clinicaltrials.gov and 
conducted to the highest standards. To ensure good reporting, our main manuscript will include all 
essential elements in accordance with standards articulated in the CONSORT statement.29 The trial 
will be coordinated by RG who will chair the steering committee comprised of co-applicants. Research 
assistants supervised by co-applicants and collaborators will manage the trial at each site. First-year 
residents will enter the trial in December 2017.  
   
What are the planned trial interventions? We will take advantage of a popular commercial app containing 
70 unique clinical cases with test questions that simulate the short answer problems on the certification 
examination. The FM Study Guide (hereafter the ‘Study Guide’) is a mobile device based platform for 
residents to test their knowledge of typical clinical cases covering Priority Topics in Family Medicine. 
Please see screenshots at http://www.familymedicinestudyguide.com/. Created in 2015 as a non-profit 
venture by co-applicant DL, 100% of app revenue is donated to charities or used to ensure app 
sustainability. Through the Study Guide, residents interact with test questions. These interactions can last 
5 minutes or less, and therefore fit easily into the lives of busy residents. For this trial, we will modify the 
app to provide a weekly alert as well as adaptive reinforcement of content based on test question 
performance. This will allow intervention group residents to take full advantage of both the spacing and 
the testing effect. However, both versions of the app will provide equivalent cases and test questions to 
intervention and control groups.   
  
Intervention: Immediately after consenting to participate, we will provide residents a coupon for a free 
copy of a modified version of the Study Guide. This will allow intervention group residents to access the 
modified app. Participants will then receive an alert to the clinical case of the week on their smartphone 
or tablet computer (iOS or Android). On tapping this alert, the app will open to allow reading of this case 
and up to 5 case-related test questions. As each question is answered, participants get immediate feedback 
by comparing their answer with the textbook answer provided on the subsequent screen. For each case, 
after the final question, participants will reflect on the correctness of their answers, using a checklist with 
two response options, as follows: (1) My answers are (largely) correct, or (2) My answers are (largely) 
incorrect. If incorrect, the adaptive system will automatically send an alert to that specific case in 7 days, 
to promote learning by repeating its questions. Correctly answered questions will also be re-presented to 
participants, but via an alert to repeat that case and its questions in 14 days. These spacing intervals 
between repetitions were based on research findings to optimize long-term retention of learning.10 Alerts 
to case-related questions will be retired (no longer sent) under two conditions: once the case and its 
questions are answered correctly twice consecutively, or by request after three attempts.   
  

GRAD, Roland   
   
    
   
  Project: The Study Guide cluster RCT  
3  
  
In the appendix, we provide an algorithm to summarize this intervention.  
  
Comparison: Residents in the control group will receive the standard version of the app at no charge, with 
no alerts. This version will provide the equivalent of clinical cases and test questions-on-demand.   
  
Participants in both groups will be encouraged to answer test questions, as each university will create an 
alias (avatar) for healthy competition. Each alias will earn one point for each attempt to answer a set of 
case-related questions. Total points earned per university will be displayed as scores on a leaderboard to 
encourage this competition.20   
  
What are the proposed arrangements for allocating participants to trial groups? One co-applicant (JK) 
will conduct a computer generated and blinded central randomization procedure of universities (clusters) 
to trial groups. As the unit of randomization, all Canadian universities will be stratified per the median 
number of residents by site, and then randomly allocated to the intervention or the control arm. This will 
minimize the effect of any inequality between clusters on the results. Following randomization, 
recruitment will begin locally as co-investigators at each site will obtain a list of their eligible residents. 
Consenting residents will then receive a coupon to install the app by the start date. The period from July 
to November 2017 will provide ample time to obtain consent. Participants will exit the study on the first 
day of their exam in the spring of 2019.    
   
What are the proposed methods for protecting against sources of bias? Trial outcomes (exam scores and 
questions completed) are objectively assessed. Residents will be blinded to the nature of the intervention. 
Our consent form will not mention our research questions or the intervention. The certification exam will 
be scored by persons unaware of group allocation.  
   
How will contamination be addressed? Participants will unlock the trial versions of the app using a 
coupon with a unique login number. We will track logins for each unique ID. With this tracking, 
intervention group residents will not access the version of the app destined for control group residents, or 
vice versa. In addition, randomization at geographically dispersed sites will limit the ability of 
intervention group residents to influence those allocated to the control group. Residents at different 
universities share no core rotations.   
   
What are the planned inclusion/exclusion criteria? All incoming family medicine residents in Canada 
will be eligible. These eligible residents will be clustered within university training programs. We will 
exclude residents from other specialties who happen to be on rotation in Family Medicine. We will 
deliver the intervention from December 2017 to April 2019 (70 weeks). We will follow each resident and 
provide tech support.   
   
What are the proposed primary and secondary outcome measures? Our primary outcome will be the 
score on the written component of the CFPC certification exam in April 2019. Co-applicant CB is 
authorized by the CFPC to provide de-identified scores for each participant. We will also test the 
hypothesis that alerts to reinforcement of content result in greater resident engagement with test 
questions, as compared with control residents not exposed to alerts to reinforcement of content.  
Therefore, our secondary outcome will be the number, percentage and time to completion of all correctly 
answered questions, per case. These data will help explain the expected improvement in exam scores.  
  
Using a study “dashboard”, we will monitor app usage over time at the group level, by examining a 
dynamic count of answered questions. In addition to counting attempts and correctly answered questions, 
the date and time of each attempt will be tracked and automatically recorded at the server. Thus, for each 

GRAD, Roland   
   
    
   
  Project: The Study Guide cluster RCT  
4  
  
case and for each participant, we will calculate a time to completion of that case. This will be defined by 
counting the number of days from the study start date in 2017 to the date that test questions for that case 
were deemed to be correctly answered.   
  
Through a baseline questionnaire, we will collect data on potential confounders at the participant level: 
age, sex, Canadian or International Medical Graduate status and whether they hold a graduate degree.  
   
What is the proposed sample size and what is the justification for the assumptions underlying the power 
calculations? We will have 80% power to detect an effect size of 0.4 (Cohen’s d). This effect corresponds 
to a 4% absolute increase in score, equivalent to two additional correct exam questions. Assumptions: 
Clusters will provide an average of 80 residents. Given this projected number, we will consent a 
minimum of 8x80=640 residents at 8 university training sites. If residents at 12 sites are consented, we 
will have adequate power to detect an effect size of 0.4. These assumptions are based on pilot study 
findings as well as actual scores obtained by all residents in Canada on the certification exams conducted 
in the spring of 2015 and the spring of 2016. We detail our calculation of the proposed sample size (and 
the design effect) in the appendix.  
  
What is the planned recruitment rate? What evidence is there that the planned recruitment rate is 
achievable? With support from co-investigators and collaborators, each university will recruit at least 
80% of their residents. The evidence for this comes from our pilot study, in which we consented more 
than 80% of residents at two university sites in Quebec and Ontario. This is not surprising; residents 
perceive a need for an app based on Priority Topics to help pass a high-stakes exam. We will report the 
number of residents eligible for participation in both study arms, their flow through the study and losses 
to follow up using a CONSORT-style flow diagram. While we anticipate a small loss of consenting 
residents to follow up, for example at McGill just two residents transfer out, on average per year.  
   
What is the proposed type of analyses? We will follow the guidelines of the CONSORT statement for 
clustered randomized trials and recommendations for their analysis when presenting and analysing our 
data.29 The primary efficacy analysis will be based on the intention-to-treat population i.e. all study 
participants will be analysed as randomized. The primary statistical analysis will evaluate the difference 
in mean exam scores between residents in the intervention and control groups using a linear mixed effects 
model. The mixed model will incorporate study site as a random factor and the following fixed factors (as 
presumed predictors of exam score): study group assignment (control / intervention), age, sex, completion 
of a graduate degree, and IMG status. The estimated fixed effects will be reported along with 95% 
confidence intervals. Confirmatory assessment of the primary endpoint will be based on the confidence 
interval for the fixed effect estimate of “study group”, implying a two-sided level of significance (against 
the null hypothesis of no effect) of 0.05.   
  
Regarding the secondary outcome of time to correct completion of test questions: For each resident, a 
mean time to event (with confidence intervals) will be calculated separately for each clinical case in the 
app, averaged and then compared across groups. These data will be described using a Kaplan-Meier 
curve. To account for clustering while adjusting for potential confounders, we will use a Cox regression 
model with a sandwich covariance structure. In the adjusted model, we will compare the hazard for this 
secondary outcome, between intervention and control groups, as of April 2019.  
  
What is the process for ethics approval? We have already initiated the approval process. At McGill, the 
IRB policy is to complete the review of an approval request once funding has been secured. We provide a 
draft of our consent form in the appendix. Residents will be told we are tracking their attempts to 

GRAD, Roland   
   
    
   
  Project: The Study Guide cluster RCT  
5  
  
complete test questions. Note that co-investigators at each university site will liaise with local officials to 
obtain the necessary approvals following IRB approval at McGill.30   
  
The risks associated with resident participation in medical education research are not negligible, and 
could theoretically include: fatigue, stress, alteration of self-concept, and loss of confidence.31 We will 
take the following steps to mitigate such risks for residents who participate. First, regarding informed 
consent, residents may decline to participate; in so doing, we will clarify that residents can still purchase 
the Study Guide. This will reduce the risk of coercion to participate. Second, de-identified examination 
scores from the CFPC will be made available to the research team only after the residency has ended. 
Thus, there is no concern that resident participation within the study will affect a program director’s 
evaluation of the resident after the study.   
  
Expertise, Experience and Participatory Approach   
Our team has the expertise to accomplish this work. Overall responsibility for this study will be assumed 
by RG, a physician-scientist with many peer-reviewed publications in the field of medical education. In 
2015, RG received the John Ruedy Award for Innovation in Medical Education from the Association of 
Faculties of Medicine of Canada. EW is director of Certification and Examinations for the CFPC. As a 
decision-maker, EW will help to ensure full consideration of study findings by the CFPC and the receptor 
community nation-wide. PP is a Senior Research Scholar and Director of Methods Development, Quebec  
SPOR-SUPPORT Unit. PP will provide expertise in participatory research with organizations. At Western 
University, DL is the founder of the Family Medicine Study Guide, and academic program director. SM is 
the Royal College Fellow contributing to this trial. TS is a Canada Research Chair in Biostatistical 
Methods for Primary Health Care Research. As the trial biostatistician, he provided statistical advice to 
develop this proposal. JK is Professor at the University of Montreal and Director of Pragmatic Trials, 
Quebec SPOR-SUPPORT Unit. JK will provide us with expertise in multi-centre cluster randomized 
trials. The web pages to enhance engagement of residents through competition between universities will 
be developed by MR, a senior family medicine resident and PhD-trained computer scientist with expertise 
in data visualization. MR will also participate in beta testing of the app. Other co-applicants and 
collaborators from across Canada demonstrate the commitment of academic Family Medicine. These 
people will mobilize their resources to optimize resident recruitment. A possible challenge in maintaining 
engagement of a large team will be mitigated by meetings between us at annual conferences such as those 
of the North American Primary Care Research Group and the CFPC’s Family Medicine Forum. The trial 
steering committee will also meet via monthly teleconference.  
  
Approach: To better inform the CFPC, we have taken a participatory approach with this organization. 
Since 2013, and through the pilot study, RG has worked cooperatively and collaboratively with 
coapplicant CB in building this proposal. With numerous awards for his contributions to medical 
education, CB works with the CFPC Committee on Examinations. RG’s work with the CFPC is inspired 
by Organizational Participatory Research (OPR), a form of Integrated Knowledge Translation that 
engages knowledge users, thereby increasing the likelihood that research findings will be relevant and 
used. Rooted in the works of Lewin and Argyris, OPR consists of undertaking research with 
participants.32,33 OPR blends research with action, thereby producing knowledge that can inform 
healthcare practice, and organizations. In OPR, organizations contribute to the project by helping 
determine the research question, methodology and methods, data collection and analysis, interpretation of 
data, and dissemination of  
results.34-37  
  

GRAD, Roland   
   
    
   
  Project: The Study Guide cluster RCT  
6  
  
Business plan and sustainability  
The CFPC has provided a letter of support (see appendix), and backed this proposal by encouraging the 
contributions of co-applicants EW and CB. Through co-applicant CB, the CFPC will provide the outcome 
data for each participant as de-identified certification exam scores.  
  
The Study Guide is a fully non-profit initiative. If trial results are positive, our findings will influence 
educational practice. The app (and more generally, the method of spaced education) will be recommended 
by program directors for thousands of medical students who enter residency worldwide, each year. This 
target audience can easily sustain the app. A timeline of study milestones, tasks and anticipated time to 
completion is provided in the appendix.