Nudging Flu Vaccination by Making it Easy for Patients to Schedule a Flu Shot
Phase: NA
Conditions: Influenza
Sponsor: Geisinger Clinic

 
Page 1 of 5 
GENERAL INFORMATION  
Nudging flu vaccination by making it easy for patients to schedule a flu shot 
 
Christopher Chabris, Ph.D. 
cfchabris@geisinger.edu 
 
September 25, 2022 
 
IRB#: 2022-0500 
 
BACKGROUND AND OBJECTIVES 
On average, 8% of the US population gets sick from the flu each flu season (Tokars et al., 
2018). Since 2010, the annual disease burden of influenza has included 9-45 million illnesses, 
140,000-810,000 hospitalizations, and 12,000-61,000 deaths (CDC, 2020). The CDC 
recommends a flu vaccination to everyone aged 6+ months, with rare exception; almost anyone 
can benefit from the vaccine, which can reduce illnesses, missed work, hospitalizations, and 
death (CDC, 2019). Flu vaccination is especially important during the COVID-19 pandemic, to 
minimize flu/COVID-19 coinfections and conserve healthcare resources. 
 
Two past Roybal pilots focused on outreach to patients with upcoming appointments (e.g., 
Milkman et al. 2021). However, many patients in the health system do not have any 
appointments during flu season. Messaging patients without a scheduled appointment with a 
way for them to easily self-schedule a flu shot may increase flu shot uptake across the health 
system. 
 
 
The present study will test the effectiveness of a patient-facing nudge, encouraging patients 
without an upcoming appointment to get a flu shot. This study builds on Geisinger and Penn 
Medicine Roybal-funded pilots evaluating high-risk messages sent to patients with or without an 
upcoming appointment (Rosenbaum et al., 2021; Shermohammed et al., 2021), and timely 
notifications encouraging patients to get the flu shot at their upcoming appointment (e.g., 
Milkman et al. 2021). 
 
Specific Aim 
 
Evaluate via randomized controlled trial the efficacy of sending messages encouraging flu shots 
to patients with no scheduled appointment during flu season. 
 
PROCEDURES  
Research Design 
Patients will be randomly assigned to one of up to six study arms. Outcomes during the 2022–
2023 flu season will be compared between study arms. 
 
Study Population 
1) Primary target population 
Inclusion Criteria:  
• 
Included in one or both of the following two base patient lists: 
o Community Medicine Service Line (CMSL)/Marketing patient portal flu outreach list 
for Fall 2022 
o A list of patients obtained from Geisinger’s Phenomics and Clinical Data Core 
(PACDC; see Data Sources below for more on this list)  
• 
Aged 18 or older as of October 2022 

 
Page 2 of 5 
• 
Has had a Geisinger encounter in the last 2 years as of October 2022  
• 
Either of the following, as of October 2022: 
o Has a Geisinger PCP assigned in the Community Medicine, Pediatrics, or Internal 
Medicine service line  
o In the last 2 years, has completed an appointment in a Geisinger specialty on a list of 
specialties approved by system leadership for flu shot communications 
• 
Has not received a flu-shot during the 2022–23 flu season as of ~1 week prior to the 
message date (timeline may be slightly different, depending on data pull logistics), according 
to the Electronic Health Record (EHR) 
• 
Does not have a scheduled in-person primary care or in-person flu-shot-eligible specialty 
appointment in the 12 weeks following their assigned message send date, as of ~1 week 
prior to the message date (timelines may be slightly different, depending on data pull 
logistics or clinical guidance) 
o As of this writing, the team plans to define a flu-shot-eligible specialty appointments 
as appointments in departments that have historically documented or administered 
flu vaccine. However, this approach may change slightly based on changing clinical 
guidance. 
 
Exclusion Criteria:  
• 
The CMSL/Marketing list will include patients in Geisinger’s 65Forward or Community Care 
populations. If necessary due to logistical constraints, we may exclude these 65Forward 
and/or Community Care patients from our study. 
• 
Cannot be contacted via the communication modality being used in the study (e.g., patient 
portal, SMS), e.g., due to insufficient/missing contact information in the EHR or because 
they opted out  
• 
Has an allergy to flu vaccines according to any EHR allergy table known to the study team 
• 
Has a health maintenance modifier indicating they are permanently discontinued from 
receiving a seasonal flu shot 
• 
Is on a list of dismissed patients 
 
Unlike in previous pilots, with guidance from the director of Infectious Diseases at Geisinger, we 
will not explicitly exclude all patients with potential contraindications, because patients are 
expected to be screened for flu shot eligibility and prior vaccination by clinicians. However, 
clinical leadership subsequently advised us to avoid messaging patients with a flu vaccine 
allergy indicated in either of two EHR tables (an allergy table or a health maintenance modifier 
table). Therefore, we will exclude patients with flu vaccine allergies listed in these tables. 
Additionally, we will follow clinical guidance if it changes (e.g., if clinicians point out another 
allergy table that should be avoided), under the exclusion criterion “Has an allergy to flu 
vaccines in any EHR allergy table known to the study team.” 
 
Our messages will include a disclaimer stating that most people who think they shouldn’t get a 
flu shot actually can and should, even people who have been told they have allergies, and 
directing patients to talk to their doctor with any concerns. We plan to also link to a myth-busting 
website, hosted by the Geisinger Marketing team, which includes information about flu shots 
and contraindications.  
 
Recruitment and Enrollment 
All patients will be recruited and enrolled from Geisinger; Massachusetts Institute of Technology 
(MIT)/National Bureau of Economic Research (NBER) collaborators will not be involved in 
recruitment or enrollment. A conservative anticipated enrollment number for this primary target 

 
Page 3 of 5 
population is ~120,000. The minimum number of patients we plan to recruit is 60,000, with 
10,000 per arm. If, due to unforeseen circumstances, there are fewer than 60,000 patients 
available to message, we will remove one or more message arms until there are at least 10,000 
patients per arm. Only patients from this primary target population will be contacted. 
 
Detailed Study Procedures  
 
Every year, Geisinger Marketing, in collaboration with CMSL, sends messages to patients via 
the patient portal in late August or early September, along with a follow-up message later in the 
flu season. This year, the BIT is taking over the follow-up messages to test the most effective 
wording and timing for these messages.  
 
In addition to patients on the CMSL/Marketing outreach list, the study will include patients who 
are on a general list of patients generated by PACDC but who were not included in the 
CMSL/Marketing list. We are using both lists in hopes of collecting data from a broad sample of 
Geisinger patients.  
 
Participants will be randomized to receive, or not receive, a message prompting them to get a 
flu shot. The experimental arms include: 
1. Passive control: No message 
2. Active control: A message that simply states that patients can get a flu shot at Geisinger  
3. Ease: A message emphasizing the ease of getting a flu shot at Geisinger  
4. Waiting for you: A message that states the patient’s flu shot is “waiting” for them at 
Geisinger 
5. Protect yourself (rare outcomes): A message that emphasizes the rare, dangerous 
outcomes of getting the flu (e.g., hospitalization, pneumonia), and states that the flu shot 
can offer protection from those outcomes 
6. Protect yourself (frequent outcomes): A message that emphasizes the outcomes that 
frequently occur in people with the flu (e.g., fever, chills, missing important events), and 
states that the flu shot can offer protection from those outcomes 
 
Messages will include a link redirecting patients to a page where they can self-schedule a flu 
shot.  
 
The language of the attached drafts may change slightly as we are working with multiple teams 
at Geisinger, but we will not alter the content and overall message drastically.  
 
Patients will be randomized to one of a few message send dates (e.g., mid-October, mid-
November, or Mid-December; the exact dates will be determined in collaboration with the 
Marketing team). Although patients in the passive control group will not receive a message, they 
will be assigned to a send date as a comparison for those who receive messages on their 
assigned date.  
 
Although we anticipate sending messages via the patient portal, we will use SMS, email, or 
another modality if logistically necessary. 
 
All patients eligible for the study prior to the first send date will be randomized to an 
experimental arm and send date prior to the first message send date. Then, prior to each 
subsequent message send date, we will remove from the study patients who have already 
received a flu shot or who have an appointment newly scheduled in the 12 weeks after their 
scheduled message send date. 

 
Page 4 of 5 
 
The planned primary outcome will be whether the patient self-schedules a flu shot appointment 
within 4 weeks of the patient’s message send date (note that the patient needs to schedule the 
appointment in this timeframe, but the appointment can occur more than 4 weeks past the send 
date). (However, if it is determined that self-scheduling cannot be reliably extracted from the 
EHR, the primary outcome will be flu vaccination within 4 weeks of the message send date.) 
Secondary outcomes will include vaccination within 4 weeks of the message send date. 
Additional outcomes to be measured through the end of the flu season will include: rates of flu 
diagnoses (both using the most rigorous biological tests, "high confidence flu", and using 
broader criteria that also include diagnosis codes and treatment information, "likely flu"), flu 
complications, and rates of other relevant healthcare utilization outcomes such as ER visits, and 
hospitalizations. Finally, we will measure rates of COVID-19 vaccination in targeted patients. 
 
Data Sources 
 
The base sample will include patients who are on the Fall 2022 CMSL/Marketing patient-portal 
based flu outreach list, as described above.  
 
In hopes of collecting data from a broad sample of Geisinger patients, we also will screen 
patients on a separate patient list obtained from Geisinger’s Phenomics and Clinical Data 
Core (PACDC). These patients meet the following criteria: 
 
1. Had had a primary care encounter at Geisinger (outpatient or telemedicine) between 
10/1/2008 and 4/13/2022.  
2. Were born in the year 2004 or later (i.e., those turning 18 in the year 2022 and older) 
3. Were alive according to the EHR as of April 2022 when the dataset was last updated 
4. Either: 
a. Currently have a Geisinger PCP assigned 
b. Have been in the EHR since at least September 2021 AND have had at least one 
encounter in 2020–2022 
 
Based on retrospective data, the majority of patients who are on the CMSL/Marketing list will 
also be on the PACDC list. However, we plan to include patients who were on the 
CMSL/Marketing list but who were not on the PACDC list so that all eligible patients on the 
CMSL/Marketing list are sent a follow-up message (unless they are assigned to the passive 
control arm). 
 
After we obtain the base sample lists from PACDC and CMSL/Marketing, Business 
Intelligence & Advanced Analytics (BIAA), working with PACDC, will pull a dataset to help us 
determine who meets our inclusion criteria and aid in NIH demographics reporting. This dataset 
will include demographic information, including patient birth date, race, ethnicity, and gender 
and/or sex. Finally, the dataset will include death date to ensure patients are living as of the 
date the data are extracted.  
 
After the intervention is complete, we will obtain experimental outcome data for all population 
groups from BIAA & PACDC. This will include data on patient flu vaccination, diagnosis of flu, 
diagnosis of flu-like symptoms, presence of flu-associated complications, hospital visits, 
emergency department visits, and COVID-19 vaccination status. This dataset will additionally 
include patient characteristics that can serve as analysis covariates, such as: patient primary 
care provider and flu-related behavior and outcomes during previous flu seasons. 
 

 
Page 5 of 5 
We may also request claims data from the Geisinger Health Plan (GHP; e.g., for claims related 
to flu vaccination, diagnosis, and complications). GHP claims data will be pulled by BIAA or the 
GHP data team (pending approval from Research Contracts). 
 
 
STUDY DATA DETAILS  
Data Management Procedures and Confidentiality 
The complete, identified study data will be accessed only by a Geisinger data broker and by 
limited Geisinger investigators with appropriate training and a legitimate need to know (i.e., 
Geisinger Co-Is and PIs), and any appropriately trained Geisinger staff, (e.g., clinicians, data 
analysts) who have a need to review the data for purposes of their normal Geisinger role.  
 
A limited dataset containing dates of service and ZIP codes may be shared with any non-
Geisinger collaborators (e.g., Joseph Doyle and his team) under a Data Use Agreement 
(pending approval) in compliance with HIPAA’s Privacy Rule and using data security protocols 
reviewed and approved by the Geisinger Security Office, Privacy Office, and Information 
Technology department.  
 
All data will be electronic. Datasets with full identifying information will only be stored on 
Geisinger-managed, password-protected computers of the data brokers for the purpose of 
linking datasets from different sources. 
 
Identifiable data gathered for this study will be retained for at least 3 years, as required by NIH 
policy, and will be deleted after 3 years or after analysis is complete. After the data have been 
fully analyzed, the deidentified dataset will be shared along with publications from this study. 
The deidentified data will not be destroyed or removed after any prespecified period of time has 
elapsed. We intend to permanently and securely archive the deidentified dataset at a research 
repository such as Open Science Framework (OSF) in order to be consistent with the best 
practices for open and reproducible science, as well as our obligation to the public as NIA-
funded researchers. 
 
All data analysis will be conducted by Gail Rosenbaum, Henri Santos, Amir Goren, or limited 
other Geisinger investigators with appropriate training, and our non-Geisinger Collaborators. We 
will analyze the data using standard behavioral research analysis methods, including computing 
bivariate correlations, using generalized linear models, using non-parametric models for non-
normally distributed data, and entering variables as independent predictors in regression 
models to attempt to predict desired outcomes.   
 
 
RESEARCH ACTIVITIES AT OTHER SITES 
Geisinger is the lead research team for this multi-site study. Massachusetts Institute of 
Technology (MIT) and National Bureau of Economic Research (NBER) study personnel are 
considered not considered engaged in human subject research. Research activities conducted 
at MIT and NBER will be limited to data analysis, using limited data, as described above. MIT 
and NBER will not be involved in study recruitment/enrollment or intervention administration. 
 
 
 
References 
 
Centers for Disease Control and Prevention. (2019). Who needs a flu vaccine and when.  

 
Page 6 of 5 
https://www.cdc.gov/flu/prevent/vaccinations.htm (Oct 11). 
Centers for Disease Control and Prevention. (2020). Disease Burden of Influenza. 
https://www.cdc.gov/flu/ about/burden/index.html (Jan 10). 
Milkman, K. L., Patel, M. S., Gandhi, L., Graci, H. N., Gromet, D., Ho, H., … Duckworth, A. L. 
(2021). A Mega-Study of Text-Based Nudges Encouraging Patients to Get Vaccinated at An 
Upcoming Doctor’s Appointment. Available at SSRN: https://ssrn.com/abstract=3780267 
Rosenbaum, G.M., Shermohammed, M., Goren, A., Lanyado, A., Metzger, C., Yesharim, R., 
Wolk, D.M., Doyle, J., Meyer, M.N., & Chabris, C.F. (2021). Effects of AI-determined high-
risk warnings on flu vaccination rates: Two field experiments. Presented at the 11th 
Behavioral Science and Health Symposium, University of Pennsylvania, Philadelphia, 1–2 
December.  
Shermohammed, M., Goren, A., Lanyado, A., Yesharim, R., Wolk, D. M., Doyle, J., *Meyer, M. 
N., & *Chabris, C. F. (2021). Informing patients that they are at high risk for serious 
complications of viral infection increases vaccination rates. MedRxiv. 
https://doi.org/10.1101/2021.02.20.21252015  
Tokars, J.I., Olsen, S.J., & Reed, C. (2018). Seasonal incidence of symptomatic influenza in the 
United States. Clinical Infectious Diseases, 66, 1511–1518. PMID: 29206909