Nudging Flu Vaccination in Patients at Moderately High Risk for Flu and Flu-related Complications
Phase: NA
Conditions: Influenza, Vaccination, Health Promotion, Health Behavior, Risk Reduction
Sponsor: Geisinger Clinic

 
Nudging Flu Vaccination in Patients at Moderately High Risk for Flu and Flu-related 
Complications 
 
Study Protocol with SAP 
 
NCT05509283 
 
10/25/2022 
 

 
 
 
Study Protocol 
 
Scientific Background 
 
Almost everyone ages 6 months or older can benefit from the influenza vaccine, which can 
reduce illnesses, missed work, hospitalizations, and death by reducing the likelihood of 
contracting influenza. Flu shots are particularly important for patients at high risk of experiencing 
severe outcomes. 
 
In the 2020–21 and 2021–22 flu seasons, the study team sent messages to Geisinger patients 
in the top 10% of risk for flu and complications according to an artificial intelligence algorithm. 
Messages that disclosed patients' risk status significantly increased flu vaccination rates. 
Additionally, messages that included risk information were most effective in patients at relatively 
lower risk (those in the top 4–10%) compared with those at the highest risk (top 3%). 
 
Objectives 
 
The present work will test the effectiveness of high-risk messages in patients who are in the top 
11–20% of risk, at high risk but lower than previous studies. These communications will inform 
patients they are at high risk with either (a) no additional explanation, (b) an explanation that this 
determination comes from an analysis of their medical records, or (c) the additional explanation 
that an AI or ML algorithm made this determination. 
 
Design  
 
This study is a randomized controlled trial with 5 study arms. Patients were randomized to 
receive (or not receive) one of several messages about flu shots. 
 
Methods 
 
The sample includes 40,671 patients. Patients were randomized into the following study arms: 
 
1. Passive Control - Patients in the passive control arm did not receive additional pro-
vaccination interventions beyond the health system's normal efforts.  
2. Active Control - Patients in the active control arm received messages reminding them to 
get a flu shot without advising them of their risk status. 
3. High Risk Only - Patients in this treatment arm received messages telling them they 
were identified to be at high risk for flu complications, without specifying how or why the 
health system believes this to be the case. 
4. High Risk with Explanation Based on Medical Records - Patients in this treatment arm 
received messages telling them they were identified to be at high risk for flu 
complications via review of their medical records. 
5. High Risk with Explanation Based on Algorithm - Patients in this treatment arm received 
messages telling them they were identified to be at high risk for flu complications via 
analysis of their medical records by a computer algorithm. 
 
 
 
 

 
Power Analysis 
 
The sample size above allows 80% power to detect an increase in flu vaccination rates from 
35% to 37.1% with two-tailed alpha = .05 for any comparison between study arms. 
 
Message Modalities 
 
Patients randomized to arms 2-5 were sent messages in up to 3 modalities, depending on 
eligibility: 1. a letter, 2. a MyGeisinger (MyChart patient portal) message, and 3. a short 
message service (SMS) message.  
 
Project Status 
 
All intervention messages have been sent. 
 
Letters for the Active Control and High Risk with Explanation Based on Algorithm study arms 
were sent on 9/13/22. Due to a miscommunication with the mail room, letters for the High Risk 
Only and High Risk Based on Medical Records arms were delayed by one day (send date of 
9/14/22). 
 
Patient portal messages were sent on 9/27/22 for all study arms. 
 
Patients eligible for SMS were randomized to be sent a message on 10/12/22 or 10/13/22. 
Randomization was stratified by study arm. 
 
Data collection is ongoing within the health system, and the study team has not yet obtained or 
analyzed any outcome data from the study. 
 
Change in Primary Outcome Time Frame 
 
Our originally preregistered primary outcome time frame was within 2 weeks of the final 
message send date. However, due to the mistimed letters mentioned in the Project Status 
section, we changed the primary outcome time frame to within 6 weeks of the patient’s study 
start date, so all patients can be monitored for the same length of time. 
 
The study start date corresponds to the letter send date for the patient’s study arm: 
• 
9/13/22: Active Control and High Risk with Explanation Based on Algorithm  
• 
9/14/22: High Risk Only and High Risk Based on Medical Records 
 
Because patients in the Passive Control study arm were not sent a letter, they will be 
randomized to a study start date of 9/13/22 or 9/14/22. 
 
Analysis Exclusion Criteria 
 
Patients vaccinated before to their study start date. We will remove patients from analysis 
who received a flu shot prior to their study start date (9/13/22 or 9/14/22, depending on their 
study arm), because they could not have been influenced by our nudge messages to get a flu 
shot. 
 
Patients deceased prior to their study start date. Twice during the study, we received 
updated lists of patients who were deceased. Some had died prior to the study beginning but 

 
medical records had not yet been updated to reflect the date of death. These patients will be 
removed from all analyses. 
 
Statistical Analysis Plan 
 
Planned Analyses 
 
Primary Outcome: Received a flu vaccination within 6 weeks of the patient's study start date [ 
Time Frame: Within 6 weeks of the patient’s study start date ] 
 
Question 1: Does informing patients that they are at high risk for flu and flu-related 
complications increase the likelihood that they will get vaccinated? 
 
Analysis 1a (Confirmatory): We will test the hypothesis that patients who were sent messages 
with information about their risk status (patients randomized to message arms 3, 4, or 5) will 
exhibit improved flu vaccination rates compared with patients in the Active Control arm who 
were sent messages without risk information (arm 2).  
 
Analysis 1b (Confirmatory): We will test the hypothesis that patients who were sent messages 
with information about their risk status (those in arms 3, 4, or 5) will exhibit improved flu 
vaccination rates compared with patients assigned to the Passive Control arm (arm 1).  
 
For analyses 1a and 1b, we will run OLS regressions. For both regressions, we will include a 
binary predictor variable coding separately for baseline (arm 2 for analysis 1a and arm 1 for 
analysis 1b), and message (all patients in arms 3–5).  
 
Question 2: Do messages encouraging flu shots increase flu vaccination rates? 
 
Analysis 2 (Confirmatory): We hypothesize that patients who were sent Active Control 
messages encouraging them to get a flu shot with no information about their risk status (arm 2) 
will be more likely to get a flu shot than those in the Passive Control arm (arm 1). To assess this 
hypothesis, we will employ OLS regression with a categorical predictor variable coding for arm.  
 
Question 3: Among messages that mention patients’ risk statuses, is it most effective to a) give 
patients no information about how their risk status was determined (arm 3), b) tell patients their 
high-risk designation was based on a review of their medical records (arm 4), or c) tell patients 
their high-risk designation was based on a computer algorithm review of their records (arm 5)? 
 
Analysis 3 (Exploratory): We will evaluate this question using an OLS regression model, with a 
categorical predictor variable coding for arm (arm 3, arm 4, arm 5). We will run post-hoc tests 
using Tukey’s HSD to test for significant pairwise differences between the arms. 
 
Sensitivity Analyses and Robustness Checks  
 
Recent work suggests that OLS regressions are appropriate in randomized experiments with 
binary outcome variables such as ours (Gomilla, 2021). However, as a robustness check, we 
will also run the regressions described above as logistic regressions instead of OLS 
regressions. We may run additional sensitivity analyses or robustness checks. 
 
 
 

 
Other Pre-specified Outcomes 
 
Other Pre-specified Outcomes listed below include flu outcomes (diagnosis, complications) and 
COVID-19 vaccination. If there are any differences in these outcomes as a function of study 
arm, the mechanism would almost certainly be increased flu vaccination. Therefore, we will only 
run analyses on Other Pre-specified Outcomes for analyses above where there is a significant 
difference in flu vaccination. 
 
1. High confidence flu diagnosis 
 
Patient received a flu diagnosis via a positive polymerase chain reaction 
(PCR)/antigen/molecular test (yes/no) during the 2022-23 flu season (from the patient’s 
study start date through April 30, 2023). 
 
[Time Frame: Up to 8 months] 
 
2. "Likely flu" diagnosis 
 
Received a "high confidence flu" diagnosis (with positive PCR/antigen/molecular test) 
and/or "likely flu" diagnosis (as assessed via International Classification of Disease [ICD] 
codes or Tamiflu administration or positive PCR/antigen/molecular test) (yes/no) during 
the 2022-23 flu season (from the patient’s study start date through April 30, 2023). 
 
Note that "likely flu" is a superset of the "high confidence flu" diagnoses. 
 
[Time Frame: Up to 8 months] 
 
3. Flu complications 
 
Diagnosed with flu-related complications (yes/no) from the patient’s study start date 
through July 31, 2023. 
 
[Time Frame: Up to 11 months] 
 
4. ER visits 
 
Number of ER visits from the patient’s study start date through July 31, 2023. 
 
[Time Frame: Up to 11 months] 
 
5. Hospitalizations 
 
Number of hospitalizations from the patient’s study start date through July 31, 2023. 
 
[Time Frame: Up to 11 months] 
 
6. COVID-19 vaccination rates 
 
Received at least one COVID-19 vaccination (yes/no) during the 2022-23 flu season 
(from the patient’s study start date through April 30, 2023). 
 

 
[Time Frame: Up to 8 months]  
 
 
Additional Exploratory Analyses 
 
1. Age and gender 
 
While older patients tend to be aware of their increased vulnerability, younger patients 
may be more surprised to learn of their high-risk status. Additionally, our previous work 
suggests that males and females are differently likely to get vaccinated as a function of 
age, with younger females more likely to get vaccinated than males, and older females 
less likely to get vaccinated than older males. 
 
To test the relation between flu shots, age and gender, we will run an OLS regression 
including binned age (18–24, 35–44,45–54, 65+), gender, and their interaction.  
 
Finally, we will test for an interaction between age, gender, and study arm, as people of 
different ages and genders may be differentially receptive to different message versions. 
 
2. Timing of shot 
 
We will examine outcomes over time and run regression models to test whether 
intervention messages influenced the timing (time elapsed since the beginning of the 
intervention, September 13, 2022) of flu shots. 
 
3. Risk level 
 
We will explore heterogeneity in flu vaccination rates within the top 11–20% of risk, and 
test whether effects of study arm vary as a function of risk level.