Encouraging Flu Vaccination Among High-Risk Patients Identified by a Machine-Learning Model of Flu Complication Risk
Phase: NA
Conditions: Influenza, Vaccination, Health Promotion, Health Behavior, Risk Reduction
Sponsor: Geisinger Clinic

Page 1 of 4
GENERAL INFORMATION 
Encouraging Flu Vaccination Among High-Risk Patients Identified by a Machine-Learning 
Model of Flu Complication Risk 
Christopher Chabris, Ph.D.
cfchabris@geisinger.edu
September 9, 2020
IRB#: 2020-0290
BACKGROUND AND OBJECTIVES
On average, 8% of the US population gets sick from flu each flu season (Tokars et al. 2018). 
Since 2010, the annual disease burden of influenza has included 9-45 million illnesses, 140,000-
810,000 hospitalizations, and 12,000-61,000 deaths (CDC 2020). The CDC recommends the flu 
vaccination to everyone aged 6+ months, with rare exception; almost anyone can benefit from 
the vaccine, which can reduce illnesses, missed work, hospitalizations, and death (CDC 2019). 
Flu vaccination will be especially important for high-risk patients during the COVID-19 
pandemic so that flu cases are reduced and resources conserved.
While most recover from influenza without treatment, the elderly, those with comorbidities, and 
other high-risk individuals can experience complications such as pneumonia, other respiratory 
illness, and death. Geisinger has partnered with Medial EarlySign (Medial; www.earlysign.com) 
to develop a machine learning (ML) algorithm to identify patients at risk for serious (moderate to 
severe) flu-associated complications on the basis of their existing electronic health record (EHR) 
data. The development of this algorithm was already approved through a separate Geisinger IRB 
application (IRB number 2020-0211). Geisinger will deploy this system during the 2020-21 flu 
season and contact the identified patients with special messages (in addition to standard efforts 
made by the health system every flu season) to encourage vaccination. 
However, there is little evidence about (a) whether informing patients they are at high risk makes 
them more likely to receive vaccination; (b) how patients react to being told their risk status is 
the result of an analysis of their health records; and (c) whether informing patients their risk 
status has been determined by an "algorithm," by "machine learning," and/or by "artificial 
intelligence" (AI) will increase or decrease their likelihood of getting vaccinated. This study will 
address these gaps in the literature, which are especially important in light of the anticipated 
future growth of AI/ML system use throughout healthcare.
Medial's algorithm is an example of how interoperable health information exchange (HIE)—the 
ability for health information technology to share patient data—can improve the efficiency and 
effectiveness of healthcare. However, patients may not appreciate these benefits or the fact that 
healthcare has become substantially more integrated and collaborative. A systematic review of 
patient privacy concerns about HIE found that 15-74% of patients expressed privacy concerns, 
IRB NUMBER: 2020-0290
IRB Approved:  09/13/2021

Page 2 of 4
depending on the study, and concluded that patient perspectives remain poorly understood. A flu 
outreach message that explicitly references a review of patient medical records might backfire as 
patients react poorly to a sense they have lost control of their health records, even though in this 
case the use is intended to benefit their care.
There is conflicting evidence on how people respond to advice or information that comes from 
an algorithm or machine. Dietvorst et al. (2015) documented a pattern of "algorithm aversion," in 
which people choose inferior human over superior algorithmic forecasts, especially after they 
observed the algorithm make an error. In contrast, Logg et al. (2018) described "algorithm 
appreciation," in which people followed advice more when they thought it came from algorithms 
than when they thought it came from human beings. Finally, Bigman and Gray (2019) found 
aversion to algorithms that make "moral decisions," including a (fictitious) medical decision of 
choosing whether or not to operate on a high-risk patient. In this study, the algorithm is merely 
advising patients on taking a low-risk action (an annual flu shot) that is already the standard of 
care and heavily encouraged by Geisinger for all patients (except, of course, those for whom the 
vaccine is contraindicated), and there is no opportunity to observe an erroneous recommendation, 
so the hypothesis is that "algorithm appreciation" will cause people to react positively to being 
informed of the algorithm's role. Thus, this study will address two important research questions:
1. Does informing patients that they are at high risk for flu complications (a) increase the 
likelihood that they will receive flu vaccine; and (b) decrease the likelihood that they 
receive diagnoses of flu and/or flu-like symptoms in the ensuing flu season?
2. Does informing patients that their high-risk status was determined (a) by analyzing their 
medical records (vs. by no specified method); and (b) by an AI/ML algorithm1 analyzing 
their medical records (as opposed to via unspecified methods or human medical records 
analysis) affect the likelihood that they receive the flu vaccine and/or diagnoses of flu 
and/or flu-like symptoms in the ensuing flu season?
Our specific aims are:
1. Evaluate the effect on flu vaccination rates of informing health-system patients who are 
identified by an ML analysis of EHR data to be at high-risk for flu complications that 
they are at high risk with either (a) no additional explanation, (b) an explanation that this 
determination comes from an analysis of their medical records, and (c) the additional 
explanation that an AI or ML algorithm made this determination.
2. Evaluate the effects of the same three interventions on diagnoses of flu in the same 
patients.
1 The study will not necessarily use the terms "AI," "ML," or "algorithm" in the messages; instead, these messages 
will be designed to be readable and comprehensible by the patient audience while still including the key concepts 
that differentiate the interventions from one another.
IRB NUMBER: 2020-0290
IRB Approved:  09/13/2021

Page 3 of 4
PROCEDURES 
Research Design
Patients from the high-risk sample (primary target population) will be randomly assigned to one 
of 4 study arms. Outcomes during the 2020-2021 flu season will be compared between study 
arms.
Study Population
There are 3 separate populations to consider in the current study: the primary target population 
that will be randomized into an experimental condition and potentially contacted, and two 
secondary populations who will not experience an intervention and will only be included for data 
analysis purposes.
1) Primary target population
Inclusion Criteria: 

Aged 17 or older

Have been determined as high-risk through Medial’s ML algorithm 
Exclusion Criteria: 

Has contraindications for flu vaccination

Note: we will respect opt-out communication preferences, but will include patients in the 
study if they haven’t opted out of at least one communication modality employed
2) Secondary population A (data analysis only): Household members
Inclusion Criteria: 

Household members of primary target population

Have data in Geisinger’s electronic health records
3) Secondary population B (data analysis only): Sub-threshold risk
Inclusion Criteria: 

Same as primary target population, except that these patients’ algorithmic risk scores fall 
just below the high-risk cutoff
Exclusion Criteria: 

Same as primary target population
IRB NUMBER: 2020-0290
IRB Approved:  09/13/2021

Page 4 of 4
Recruitment and Enrollment. All patients meeting the primary target population criteria will 
be enrolled into one of the 4 experimental conditions. Anticipated enrollment number for this 
primary target population is 56,000. Only patients from this primary target population will be 
contacted. However, health record data will be accessed to assess secondary outcomes for 
household members of this target population (expected N of 234,000) and for patients whose risk 
scores were calculated by Medial’s ML algorithm and determined to fall just below the high-risk 
cutoff (expected N of 56,000). This brings the total planned enrollment to 346,000 participants.
Detailed Study Procedures. 
Eligible patients will be randomly assigned to one of 4 experimental conditions:
1. Control: In this condition, patients will receive no additional pro-vaccination 
intervention beyond Geisinger's normal efforts. Note that these efforts include a variety 
of Marketing and other system campaigns designed to encourage all patients and 
members to get their flu shot. In addition, the care gaps team has an annual campaign to 
encourage flu shots that targets patients who are determined by a non-ML assessment that 
they are at high risk for complications and we expect there to be overlap between this 
group of patients and the ML high-risk group we will be contacting. (Because the care 
gap team’s high-risk patients are not told that they are at high risk or that they have been 
targeted, we are not concerned that this ongoing campaign will interfere with our ability 
to measure the effects of high-risk communication.)
2. High Risk Only: In this condition, patients will receive messages telling them they are at 
high risk for flu complications without specifying how/why Geisinger believes this to be 
the case.
3. High Risk Based on Medical Records: In this condition, patients will receive messages 
telling them they have been identified to be at high risk for flu complications via analysis 
or review of their medical records. This is an accurate statement, since it does not specify 
that a human conducted this review or analysis. However, we anticipate that most readers 
will assume it was a human rather than an algorithm, allowing us to compare attitudes 
towards human versus machine risk determination.
4. High Risk Based on Algorithm: In this condition, patients will receive messages telling 
them they have been identified to be at high risk for flu complications via analysis of 
their medical records by AI/ML.
Because this intervention is timed to the 2020–21 flu season, the intervention period will begin 
~9/1/20 and last for three months. During that period, all identified primary target population 
participants will be randomized and will receive 1–3 separate messages (content based on their 
assigned condition) in sequence via postal letter, SMS, and patient portal modalities, using all 
modalities available for each participant. Outcomes will be measured as of 5/31/21. Data 
extraction from the EHR, data cleaning, data analysis, and manuscript preparation will begin on 
or after 6/1/21 and will include data from not only the primary target population but also the two 
secondary populations. 
IRB NUMBER: 2020-0290
IRB Approved:  09/13/2021

Page 5 of 4
The primary study outcomes will be the rates of flu vaccination and flu diagnoses during the 
2020–21 season (start of intervention through 5/31/21) by the primary target population. We will 
also measure secondary, exploratory outcomes: Rates of flu vaccination and diagnoses by fellow 
household members of targeted patients; rates of flu vaccination and diagnoses by non-targeted 
patients who were assigned a risk score that fell just below the cutoff of targeted patients; rates 
of flu complications and flu-like symptoms among targeted patients, household members, and 
those at sub-threshold risk; and rates of other relevant healthcare utilization outcomes such as ER 
visits and hospitalizations. 
Data Sources
In order to identify patients in the primary target population, Business Intelligence & Advanced 
Analytics (BIAA), working with Geisinger’s Phenomics and Clinical Data Core (PACDC), 
will provide a list of patients (including patient identifiers such as Medical Record Number) that 
were assessed by the Medial and each patient’s associated risks.
We will also obtain contact information (address, phone number, email address) to message 
participants in the primary target population from BIAA & PACDC.
After the intervention is complete, we will obtain experimental outcome data for all 3 population 
groups from BIAA & PACDC. This will include data on patient flu vaccination, diagnosis of flu, 
diagnosis of flu-like symptoms, presence of flu-associated complications, hospital visits, and 
emergency department visits. This data set will additionally include patient characteristics that 
will be useful as analysis covariates, such as: age, patient primary care provider, and flu-related 
behavior and outcomes during previous flu seasons.
STUDY DATA DETAILS 
Data Management Procedures and Confidentiality. All data will be electronic. Datasets with 
identifying information will only be stored on Geisinger-managed, password-protected 
computers of the data brokers (Amir Goren, Henri Santos, and Gail Rosenbaum) for the purpose 
of linking datasets from different sources. 
A limited data set containing dates of service and ZIP codes will be shared with any non-
Geisinger collaborators (e.g., Additional Principal Investigator Doyle and his team) under a Data 
Use Agreement in compliance with HIPAA’s Privacy Rule, and using data security protocols 
reviewed and approved by the Geisinger Security Office, Privacy Office, and Information 
Technology department.
After all data have been linked in a de-identified, coded file and analyzed, the datasets with 
identifiable information and any codes needed to link the identifiable information will be deleted. 
After the de-identified data have been fully analyzed, the de-identified dataset will be shared 
along with publications from this study. The deidentified data will not be destroyed or removed 
after any prespecified period of time has elapsed. We intend to permanently and securely archive 
the deidentified dataset at a research repository such as Open Science Framework (OSF) in order 
IRB NUMBER: 2020-0290
IRB Approved:  09/13/2021

Page 6 of 4
to be consistent with the best practices for open and reproducible science, as well as our 
obligation to the public as NIA-funded researchers.
The only study team members who will have access to identifiable and protected health 
information will be the data brokers: Amir Goren, Henri Santos, and Gail Rosenbaum. Our non-
Geisinger collaborators will have access to a limited data set which includes de-identified data as 
well as dates of flu vaccination and diagnosis. The remaining investigators will have access to 
the de-identified, coded data during data analysis.
We will analyze the data using standard behavioral research analysis methods, including 
computing bivariate correlations, using generalized linear models, using non-parametric models 
for non-normally distributed data, and entering variables as independent predictors in regression 
models to attempt to predict desired outcomes.  
RESEARCH ACTIVITIES AT OTHER SITES
Geisinger is the lead research team for this multi-site, single IRB study. Massachusetts Institute 
of Technology (MIT) and National Bureau of Economic Research (NBER) will cede to 
Geisinger’s IRB. Research activities conducted at MIT and NBER will be limited to data 
analysis, using limited data and de-identified data, as described above. MIT and NBER will not 
be involved in study recruitment/enrollment or intervention administration.
BIBLIOGRAPHY
Bigman YE, Gray K. People are averse to machines making moral decisions. Cognition. 2018 
Dec;181:21-34. doi: 10.1016/j.cognition.2018.08.003. Epub 2018 Aug 11.   PubMed 
ID: 30107256
Dietvorst BJ, Simmons JP, Massey C. Algorithm aversion: people erroneously avoid algorithms 
after seeing them err. J Exp Psychol Gen. 2015 Feb;144(1):114-26. doi: 10.1037/xge0000033. 
Epub 2014 Nov 17.   PubMed ID: 25401381
Goshen R, Choman E, Ran A, Muller E, Kariv R, Chodick G, Ash N, Narod S, Shalev V. 
Computer-Assisted Flagging of Individuals at High Risk of Colorectal Cancer in a Large Health 
Maintenance Organization Using the ColonFlag Test. JCO Clin Cancer Inform. 2018 Dec;2:1-8. 
doi: 10.1200/CCI.17.00130.   PubMed ID: 30652563
Logg, J.M., Minson, J.A., & Moore, D.A. (2019). Algorithm appreciation: People prefer 
algorithmic to human judgment. Organizational Behavior and Human Decision Processes, 151, 
90-103. https://doi.org/10.1016/j.obhdp.2018.12.005
Shena, N., et al. (2019). Understanding the patient privacy perspective on health information 
exchange: A systematic review. International Journal of Medical Informatics, 125, 1-12.
Rothberg MB, Haessler SD. Complications of seasonal and pandemic influenza. Crit Care Med. 
2010 Apr;38(4 Suppl):e91-7. doi: 10.1097/CCM.0b013e3181c92eeb. Review.   PubMed 
IRB NUMBER: 2020-0290
IRB Approved:  09/13/2021

Page 7 of 4
ID: 19935413
Tokars JI, Olsen SJ, Reed C. Seasonal Incidence of Symptomatic Influenza in the United States. 
Clin Infect Dis. 2018 May 2;66(10):1511-1518. doi: 10.1093/cid/cix1060.   PubMed 
ID: 29206909
Turner D, Wailoo A, Nicholson K, Cooper N, Sutton A, Abrams K. Systematic review and 
economic decision modelling for the prevention and treatment of influenza A and B. Health 
Technol Assess. 2003;7(35):iii-iv, xi-xiii, 1-170. Review.   PubMed ID: 14609480
WHO Guidelines for Pharmacological Management of Pandemic Influenza A(H1N1) 2009 and 
Other Influenza Viruses. Geneva: World Health Organization; 2010 Feb.   PubMed 
ID: 23741777
Zack CJ, Senecal C, Kinar Y, Metzger Y, Bar-Sinai Y, Widmer RJ, Lennon R, Singh M, Bell 
MR, Lerman A, Gulati R. Leveraging Machine Learning Techniques to Forecast Patient 
Prognosis After Percutaneous Coronary Intervention. JACC Cardiovasc Interv. 2019 Jul 
22;12(14):1304-1311. doi: 10.1016/j.jcin.2019.02.035. Epub 2019 Jun 26.   PubMed 
ID: 31255564
Centers for Disease Control and Prevention. (2020). Disease Burden of Influenza. 
https://www.cdc.gov/flu/ about/burden/index.html (Jan 10).
Centers for Disease Control and Prevention. (2019). Who Needs a Flu Vaccine and When. 
https://www.cdc.gov/flu/prevent/vaccinations.htm (Oct 11).
IRB NUMBER: 2020-0290
IRB Approved:  09/13/2021