Alerts With Risk Information to Increase Influenza Vaccinations
Phase: NA
Conditions: Influenza
Sponsor: Geisinger Clinic

Page 1 of 7
GENERAL INFORMATION 
Alerts with Risk Information to Increase Influenza Vaccinations
Christopher Chabris, Ph.D.
cfchabris@geisinger.edu
March 13, 2023
IRB#: 2022-0502
BACKGROUND AND OBJECTIVES
On average, 8% of the US population gets sick from the flu each flu season (Tokars et al. 2018). 
Since 2010, the annual disease burden of influenza has included 9-45 million illnesses, 
140,000-810,000 hospitalizations, and 12,000-61,000 deaths (CDC 2020). The CDC 
recommends a flu vaccination to everyone aged 6+ months, with rare exception; almost anyone 
can benefit from the vaccine, which can reduce illnesses, missed work, hospitalizations, and 
death (CDC 2019). Flu vaccination is especially important during the COVID-19 pandemic, to 
minimize flu/COVID-19 coinfections and conserve healthcare resources.
One barrier to vaccination is a lack of “cues to action,” and, in particular, the lack of direct 
recommendation from medical personnel (Schmid et al. 2017); this barrier is arguably the most 
effectively overcome by a simple nudge of clinicians, compared with barriers such as negative 
attitudes toward vaccination, low perceived utility of vaccination, and less experience with 
having received the vaccine. Nudges that increase clinician actions to recommend and order flu 
shots for their patients may therefore increase vaccinations more than attempts to nudge 
patients directly (e.g., by sending them text messages). Interruptive or “hard stop” alerts can 
help promote behavior that improves patient outcomes (Powers et al., 2018). And mechanisms 
that make an alert more salient can increase attention to a nudge and thus encourage action 
(Patel et al., 2017; Kim et al. 2017). These nudges are likely to be most effective when timely, 
for instance, as alerts to clinicians during those appointments (Milkman et al. 2021). In the 
2020–21 flu season, Geisinger implemented a standard electronic health record (EHR) alert 
notifying clinicians when any outpatient they are seeing is eligible for a flu shot; yet, despite this 
alert, only a minority of patients are presently being vaccinated, especially at specialty 
appointments.
While most recover from influenza without treatment, the elderly, those with comorbidities, and 
other high-risk individuals can experience complications such as pneumonia, other respiratory 
illness, and death (Rothberg et al., 2008; Turner et al., 2003; WHO, 2010). Geisinger, a large 
health system in Pennsylvania, has partnered with Medial EarlySign (Medial; 
www.earlysign.com) to develop a machine learning (ML) algorithm to identify patients at risk for 
serious (moderate to severe) flu-associated complications based on existing EHR data. As part 
of our recent Roybal-funded pilots, Geisinger deployed this system during the 2020–21 and 
2021–22 flu seasons and sent patients in the top 10% of predicted risk special messages (in 
addition to standard health system efforts) encouraging vaccination. Interim data from both 
pilots reveal that notifying patients about their high-risk status increases vaccination, but 
vaccination is not further increased by informing patients of how their risk was determined 
(Shermohammed et al., 2021) or by informing them of contributing risk factors from their 
medical records (Rosenbaum et al., 2021). Compared with patients, clinicians may be more 
receptive to information about individual patient risk factors, given their better understanding of 
medical diagnoses contributing to flu risk. 
IRB NUMBER: 2022-0502
IRB NUMBER: 2022-0502
IRB Approved:  03/15/2023

Page 2 of 7
The present study will test the effectiveness of a clinician-facing nudge (e.g., salient, risk-based 
alert vs. a standard alert) on flu vaccination. This study builds on previous work supporting the 
effectiveness of salient alerts in encouraging flu vaccination (Kim et al., 2018; Patel et al., 2017) 
and extends it by focusing on alerts for people at high risk for flu and flu complications.
Specific Aims
Evaluate via randomized controlled trial the efficacy of a clinician-facing, EHR alert-based 
nudge promoting flu shots, alerting clinicians to their patients’ high-risk status (with or without 
personalized contributing factors). 
PROCEDURES 
Research Design
Patients from the high-risk sample (primary target population) will be randomly assigned to one 
of at least two study arms. If necessary, we will randomize at the clinician or clinic level. 
Outcomes during the 2022–2023 flu season will be compared between study arms.
Study Population
1) Patients
Inclusion Criteria: 

Age 18+

Have been determined to be in the top 20% of risk through Medial’s ML algorithm

Attend an appointment where the flu alert fires (Geisinger sets when flu alerts start and 
ends--between ~9/1/2022 and ~4/30/2023, as well as the trigger conditions for the alert, 
which includes valid departments and visits and excludes contraindications like Guillain-
Barre syndrome) 
2) Clinicians
Inclusion Criteria: 

Any Geisinger clinician who sees patient-participants in our study for an appointment 
where their flu shot alert fires 
Recruitment and Enrollment
All patients will be recruited and enrolled from Geisinger; Massachusetts Institute of Technology 
(MIT)/National Bureau of Economic Research (NBER) collaborators will not be involved in 
recruitment or enrollment. Patients meeting the primary target population criteria will be enrolled 
into one of the experimental conditions. Anticipated enrollment number for this primary target 
population is 90,000. Only Geisinger clinicians for this primary target population will see the 
salient, risk-based alert for patients who were randomized to trigger it. Clinicians of patients who 
are not part of the study population will still see the standard flu alert if their patients are eligible 
for it as part of Geisinger’s standard of care.
Detailed Study Procedures 
Patients in the top 20% of risk for flu and flu complications based on the Medial EarlySign flu 
algomarker will be randomized to trigger a standard flu alert (specifically, a best practice alert) 
or a salient, specialized alert for clinicians throughout the 2022–23 flu season, stating the 
patients’ level of risk (e.g., top 3%, top 10%, or top 20%). Among those who triggered the high-
risk alert, we may further randomize this group to be shown or not shown the top factors 
contributing to patients’ risk, based on Medial’s algorithm. The high-risk alert will also include 
IRB NUMBER: 2022-0502
IRB NUMBER: 2022-0502
IRB Approved:  03/15/2023

Page 3 of 7
text and color features to make the alert stand out among other alerts. Additionally, the high-risk 
alerts will be given priority, such that they are displayed above other unprioritized alerts (they 
will still show below high-priority alerts). The precise language and look of the alerts may 
change slightly from the attached drafts, as we are working with multiple teams at Geisinger, but 
the drafts are representative of the types of alerts that will be shown. 
Geisinger introduces flu-related alerts on September 1st.  As part of the research planning, 
randomization will occur in June 2022, in order to provide enough time to import risk scores and 
high-risk factors into Epic, so these data can be used to trigger the proper alerts. Because this 
intervention is timed to the 2022–23 flu season, the intervention period will begin ~9/1/22. 
The primary outcome will be flu vaccination at the scheduled appointment, as recorded in the 
Geisinger EHR. We may limit the timeframe of the primary outcome to end earlier in the flu 
season (e.g., December), based on data suggesting that most patients get their flu shot by the 
end of the calendar year. If we choose this timeframe, we will still examine all data collected 
throughout the flu season. Additional outcomes to be measured between the patient’s 
appointment and the end of the flu season will include: rates of flu diagnoses (both using the 
most rigorous biological tests, "high confidence flu", and using broader criteria that also include 
diagnosis codes and treatment information, "likely flu"), flu complications, and rates of other 
relevant healthcare utilization outcomes such as ER visits and hospitalizations. Finally, we will 
measure rates of COVID-19 vaccination in targeted patients.
Data Sources
All patients in this study will be in the top 20% of risk according to scores given by the Medial 
EarlySign flu algomarker. The development of this algorithm was already approved through a 
separate Geisinger IRB application (IRB number 2020-0211). Additionally, Geisinger deployed 
this system in a field study during the 2020–21 (IRB number 2020-0290) and 2021–22 flu 
seasons (IRB number 2021-0483) and contacted the identified patients with special messages 
(in addition to standard interventions conducted by the health system every flu season) to 
encourage vaccination.
In order to generate flu risk scores, Medial EarlySign has access to a de-identified, retrospective 
dataset in Geisinger’s research environment. This environment is accessed by Medial EarlySign 
staff who have undergone appropriate training and are credentialed per Geisinger’s research 
and Information Security/Access policies. This environment is protected by Geisinger 
Information Security Office standards that allow remote access to the Geisinger network in a 
‘sandbox’. All data and analysis are conducted within the secure environment and data do not 
leave the Geisinger network. The dataset contains EHR information on all patients who had a 
primary care encounter at Geisinger (outpatient or telemedicine) between 10/1/2008 and 
4/13/2022. 
Medial uses the retrospective dataset to generate flu risk scores for patients in that dataset who 
meet the following criteria: 
1.
Were born in the year 2004 or later (i.e., those turning 18 in the year 2022 and older)
2.
Were alive according to the EHR as of April 2022 when the dataset was last updated
3.
Either:
a.
Currently have a Geisinger PCP assigned
b.
Have been in the EHR since at least September 2021 AND have had at least one 
encounter in 2020–2022
IRB NUMBER: 2022-0502
IRB NUMBER: 2022-0502
IRB Approved:  03/15/2023

Page 4 of 7
After we obtain the base sample list from Medial, Business Intelligence & Advanced 
Analytics (BIAA), working with Geisinger’s Phenomics and Clinical Data Core (PACDC), will 
pull a dataset to help us determine who meets our inclusion criteria and aid in NIH 
demographics reporting. This dataset will include demographic information, including patient 
birth date, race, ethnicity, and gender and/or sex. Finally, the dataset will include death date to 
ensure patients are living as of the date the data are extracted. 
We will also obtain individual-level employee demographic information including age, gender, 
race, and ethnicity for provider-participants to aid in NIH demographics reporting. This 
information will be deidentified and aggregated prior to any external reporting or data sharing.
After the intervention is complete, we will obtain experimental outcome data for all population 
groups from BIAA & PACDC. This will include data on patient flu vaccination, diagnosis of flu, 
diagnosis of flu-like symptoms, presence of flu-associated complications, hospital visits, 
emergency department visits, and COVID-19 vaccination status. This dataset will additionally 
include patient characteristics that can serve as analysis covariates, such as: patient primary 
care provider and flu-related behavior and outcomes during previous flu seasons.
We may also request claims data from the Geisinger Health Plan (GHP; e.g., for claims related 
to flu vaccination, diagnosis, and complications). GHP claims data will be pulled by BIAA or the 
GHP data team (Pending approval from Research Contracts).
STUDY DATA DETAILS 
Data Management Procedures and Confidentiality
The complete, identified study data will be accessed only by a Geisinger data broker and by 
limited Geisinger investigators with appropriate training and a legitimate need to know (i.e., 
Geisinger Co-Is and PIs), and any appropriately trained Geisinger staff, (e.g., clinicians, data 
analysts) who have a need to review the data for purposes of their normal Geisinger role. 
A limited dataset containing dates of service and ZIP codes may be shared with any non-
Geisinger collaborators (e.g., Additional Principal Investigator Doyle and his team) under a Data 
Use Agreement (pending approval) in compliance with HIPAA’s Privacy Rule and using data 
security protocols reviewed and approved by the Geisinger Security Office, Privacy Office, and 
Information Technology department. 
All data will be electronic. Datasets with full identifying information will only be stored on 
Geisinger-managed, password-protected computers of the data brokers for the purpose of 
linking datasets from different sources.
Identifiable data gathered for this study will be retained for at least 3 years, as required by NIH 
policy, and will be deleted after 3 years or after analysis is complete. After the data have been 
fully analyzed, the deidentified dataset will be shared along with publications from this study. 
The deidentified data will not be destroyed or removed after any prespecified period of time has 
elapsed. We intend to permanently and securely archive the deidentified dataset at a research 
repository such as Open Science Framework (OSF) in order to be consistent with the best 
practices for open and reproducible science, as well as our obligation to the public as NIA-
funded researchers.
All data analysis will be conducted by Gail Rosenbaum, Henri Santos, Amir Goren, or limited 
other Geisinger investigators with appropriate training, and our non-Geisinger Collaborators. We 
IRB NUMBER: 2022-0502
IRB NUMBER: 2022-0502
IRB Approved:  03/15/2023

Page 5 of 7
will analyze the data using standard behavioral research analysis methods, including computing 
bivariate correlations, using generalized linear models, using non-parametric models for non-
normally distributed data, and entering variables as independent predictors in regression 
models to attempt to predict desired outcomes.  
RESEARCH ACTIVITIES AT OTHER SITES
Geisinger is the lead research team for this multi-site study. Massachusetts Institute of 
Technology (MIT) and National Bureau of Economic Research (NBER) study personnel are 
considered not considered engaged in human subject research. Research activities conducted 
at MIT and NBER will be limited to data analysis, using limited data, as described above. MIT 
and NBER will not be involved in study recruitment/enrollment or intervention administration.
References
Centers for Disease Control and Prevention. (2019). Who needs a flu vaccine and when. 
https://www.cdc.gov/flu/prevent/vaccinations.htm (Oct 11).
Centers for Disease Control and Prevention. (2020). Disease Burden of Influenza. 
https://www.cdc.gov/flu/ about/burden/index.html (Jan 10).
Kim, R. H., Day, S. C., Small, D. S., Snider, C. K., Rareshide, C., & Patel, M. S. (2018). 
Variations in influenza vaccination by clinic appointment time and an active choice 
intervention in the electronic health record to increase influenza vaccination. JAMA network 
open, 1(5), e181770. https://doi.org/10.1001/jamanetworkopen.2018.1770 
Milkman, K. L., Patel, M. S., Gandhi, L., Graci, H. N., Gromet, D., Ho, H., … Duckworth, A. L. 
(2021). A Mega-Study of Text-Based Nudges Encouraging Patients to Get Vaccinated at An 
Upcoming Doctor’s Appointment. Available at SSRN: https://ssrn.com/abstract=3780267
Patel, M. S., Volpp, K. G., Small, D. S., Wynne, C., Zhu, J., Yang, L., Honeywell, S., Jr, & Day, 
S. C. (2017). Using active choice within the electronic health record to increase influenza 
vaccination rates. Journal of General Internal Medicine, 32(7), 790–795. 
https://doi.org/10.1007/s11606-017-4046-6 
Powers, E. M., Shiffman, R. N., Melnick, E. R., Hickner, A., & Sharifi, M. (2018). Efficacy and 
unintended consequences of hard-stop alerts in electronic health record systems: A 
systematic review. Journal of the American Medical Informatics Association: JAMIA, 25(11), 
1556–1566. https://doi.org/10.1093/jamia/ocy112 
Rosenbaum, G.M., Shermohammed, M., Goren, A., Lanyado, A., Metzger, C., Yesharim, R., 
Wolk, D.M., Doyle, J., Meyer, M.N., & Chabris, C.F. (2021). Effects of AI-determined high-
risk warnings on flu vaccination rates: Two field experiments. Presented at the 11th 
Behavioral Science and Health Symposium, University of Pennsylvania, Philadelphia, 1–2 
December. 
Rothberg, M.B., Haessler, S.D., & Brown, R.B. (2008). Complications of viral influenza. The 
American Journal of Medicine, 121(4), 258–264. PMID: 19935413
Schmid, P., Rauber, D., Betsch, C., Lidolt, G., & Denker, M. L. (2017). Barriers of influenza 
vaccination intention and behavior - A systematic review of influenza vaccine hesitancy, 
2005 - 2016. PloS One, 12(1), e0170550. https://doi.org/10.1371/journal.pone.0170550
Shermohammed, M., Goren, A., Lanyado, A., Yesharim, R., Wolk, D. M., Doyle, J., *Meyer, M. 
N., & *Chabris, C. F. (2021). Informing patients that they are at high risk for serious 
complications of viral infection increases vaccination rates. MedRxiv. 
https://doi.org/10.1101/2021.02.20.21252015 
Tokars, J.I., Olsen, S.J., & Reed, C. (2018). Seasonal incidence of symptomatic influenza in the 
United States. Clinical Infectious Diseases, 66, 1511–1518. PMID: 29206909
Turner, D., Wailoo, A., Nicholson, K., Cooper, N., Sutton, A., & Abrams, K. (2003). Systematic 
IRB NUMBER: 2022-0502
IRB NUMBER: 2022-0502
IRB Approved:  03/15/2023

Page 6 of 7
review and economic decision modelling for the prevention and treatment of influenza A and 
B. In NIHR Health Technology Assessment Programme: Executive Summaries. NIHR 
Journals Library. PMID: 14609480
World Health Organization (2010). WHO guidelines for pharmacological management of 
pandemic (H1N1) 2009 influenza and other influenza viruses. PMID: 23741777
IRB NUMBER: 2022-0502
IRB NUMBER: 2022-0502
IRB Approved:  03/15/2023