BE IMMUNE: Behavioral Economics to IMprove and Motivate Vaccination Using Nudges Through the EHR
Phase: NA
Conditions: Behavior, Health, Flu, Flu Vaccination
Sponsor: University of Pennsylvania

 
  
BE IMMUNE: Behavioral Economics to IMprove and Motivate 
vaccination Using Nudges through the EHR 
University of Pennsylvania  
Dr. Shivan Mehta 
Dr. Amol Navathe 
Dr. Joshua Liao 
 
NCT#06057727 
Study Protocol v5 August 2024 
Approved: 10/7/2024 
 
 
 
 

2 
Protocol Version 5: August 2024 
 
Outline 
1. Abstract 
2. Overall objectives 
3. Aims 
 
3.1 Primary Outcome 
 
3.2 Secondary Outcome 
4. Background 
5. Study design 
 
5.1 Design 
 
5.2 Study duration 
 
5.3 Resources Necessary for human research participation 
 
5.4 Target population 
 
5.5. Subjects Enrolled by Penn Researchers 
 
5.6 Subjects Enrolled by Collaborating Researchers 
 
5.7 Accrual 
 
5.8 Key inclusion criteria 
 
5.9 Key exclusion criteria 
6. Subject recruitment 
7. Subject compensation 
8. Study procedures 
 
8.1 Consent 
 
8.2 Procedures 
9. Analysis plan 
10. Investigators 
11. Human research protection 
 
11.1 Data confidentiality 

3 
Protocol Version 5: August 2024 
 
 
11.2 Subject confidentiality 
 
11.3 Subject privacy 
 
11.4 Data disclosure 
 
11.5 Protected Health Information/Data Protection 
 
11.6 Data safety and monitoring 
 
11.7 Risk/benefit  
 
 
11.7.1 Potential study risks 
 
 
11.7.2 Potential study benefits 
 
 
11.7.3 Risk/benefit assessment 
 
 

4 
Protocol Version 5: August 2024 
 
1. Abstract 
Many older adults are at risk of illness, hospitalization, and death from vaccine-preventable diseases. More than 
half of older adults in the United States are not vaccinated for flu which has remained relatively constant over 
the past decade, and there are racial, ethnic, and socioeconomic disparities in care. In this study, we will 
evaluate personalized nudges to clinicians and patients to help increase flu vaccination rates during primary care 
visits among older adults at three distinct health systems, with a particular focus on population subgroups at 
high risk for vaccine noncompletion.  In a partnership between Penn Medicine and University of Washington 
(UW) Medicine, this will be a 6-month, multisite, cluster randomized, pragmatic trial with an additional 
intensification arm for high-risk patients.  We will also perform a 6-month replication trial at Lancaster General 
Health. 
2. Overall objectives 
The objectives of the study are to:  
1) To conduct a 6-month, multisite, cluster randomized, pragmatic trial to evaluate the effectiveness of 
personalized nudges to clinicians and patients relative to control to improve flu vaccine completion 
among older adults. 
2) To evaluate the effectiveness of an additional, intensification nudge intervention on reducing disparities 
in flu vaccine completion among high-risk subgroups. 
3) To evaluate heterogeneity in treatment effect across clinician, patient, and practice characteristics to 
further tailor approaches in future intervention design. 
3. Aims 
 
3.1 Primary Outcome 
The primary outcome is flu vaccination completion during the first eligible primary care visit. 
3.3 Secondary outcome 
The secondary outcome is flu vaccination completion within 3 months after the first eligible primary care visit. 
4. Background 
Older adults are at risk of serious illness and hospitalization from vaccine-preventable disease.1,2 The low 
vaccination rates have remained relatively stable over the last decade, indicating an urgent need for novel, 
scalable interventions to increase vaccination rates.3 Further, significant disparities exist relative to who 
receives a vaccination.4 Over 90% of clinicians use the electronic health record to facilitate decision making.5 
Prior work has demonstrated that EHR-based nudges can increase vaccination ordering rates, cancer screening 
rates, and statin prescribing.6-9 Building upon our prior work, we propose to use analytical methods to identify 
older adults at high-risk of flu vaccination non-participation, assess the feasibility of using a set of EHR-based 
nudge interventions at two primary sites (Penn Medicine and UW Medicine) to improve flu vaccination among 
older adults, especially those in high-risk groups. We then plan to replicate these findings in a third site, 
Lancaster General Health.   

5 
Protocol Version 5: August 2024 
 
5. Study design 
 
5.1 Design 
This study will be a multisite, cluster randomized, pragmatic trial to evaluate the effectiveness of personalized 
nudges to clinicians and patients, relative to a control, to increase flu vaccination rates among older adults in 
accordance with CDC guidelines. This will include clinician and patient level nudge interventions, with 
additional, intensified nudge interventions for patients identified as high risk for not receiving a flu vaccine. 
This cluster randomized trial will randomly assign primary care clinics in a 2:1 ratio to either a nudge 
interventions or control arm using a covariate-constrained randomization.10 The intervention arm consists of a 
multi-component nudge intervention for patients and primary care providers (PCPs). Patients at high-risk for 
non-completion of the flu vaccine will be randomized to an additional intensification nudge (compared with the 
multi-component nudge intervention alone). 
All patients scheduled for a visit at the clinics randomized to the intervention arm will receive a pre-visit text 
message sent via Way to Health (a HIPAA compliant, approved platform) that informs the patient the flu 
vaccine has been reserved for them at their upcoming visit. Those patients that do not have a cell phone in the 
EHR will receive automated voice recording (AVR). For the patient intensification nudge, we will focus on 
patient subgroups within the clinic intervention arm that were identified through our previous analyses as high 
risk for not receiving a vaccine: age ≥ 70, Black race, residence in lower income communities (lowest quartile 
according to zip code), or no documented history of flu vaccine in the year prior. Patients who meet these 
criteria will be randomized 1:1 to receive an additional nudge via a bidirectional text messaging intervention 
versus no additional nudge. We will translate patient text messages for patients who require an interpreter 
according to their language preferences in the EHR. Translation will be available for Spanish and up to 4 
additional languages based on site specific populations and technical capabilities. Our preliminary work 
identified the following five languages that account for 97.5% of the expected study sample: English, Spanish, 
Chinese, Vietnamese, and Russian.   
For clinicians within the intervention arm, a flu vaccine order will be pended on their behalf prior to the PCP 
visit. This will frame vaccination as the default choice (opt-out rather than opt-in) while reducing clinician/care 
team effort by removing the need to individually order each vaccine. The unsigned order will be visible to the 
provider during the visit and they will maintain the option to sign or cancel the order.  
In addition to the pended order, eligible clinicians within the intervention arm will receive a monthly peer-
comparison feedback message describing their flu vaccine completion rate, relative to that of their peers. We 
will deliver this feedback via email and vary feedback based on clinicians’ vaccine completion rates. PCPs 
whose completion rates are below the median rate will receive feedback relative to the median rate; PCPs 
whose completion rates are between the median and the 90th percentile will receive feedback relative to the 90th 
percentile, and those above the 90th percentile will not receive peer comparisons but will get positive feedback 
that they are high performers. This tiered approach has previously been shown to motivate clinician behavior 
and prevent regression to the mean among high performers. Figure 1 provides an overview of the study design. 
We will use this design to replicate our findings at a third site, Lancaster General Health. 

6 
Protocol Version 5: August 2024 
 
 
Figure 1. Overview of study design 
 
 
5.2 Study sites 
Penn Medicine and UW Medicine will be the primary sites for this trial, with Penn serving as the primary IRB 
site. Lancaster General Health is the study replication site.  
5.3 Study duration 
The study (including replication) is expected to take 3 years to complete including study preparation (year 1), 
trial intervention (year 2) and replication (year 3), analysis, and dissemination (year 3). The study intervention 
will run for 6 months during flu season (September – February). 
 
5.4 Resources necessary for human research participation 
The research staff spans the Penn Medicine Center for Health Care Innovation and Department of Medical 
Ethics and Health Policy to ensure Investigator support. This staff includes a study director, a project manager, 
research coordinator, and data analysts. To ensure that all staff assisting with research are adequately informed 
about the protocol and research related duties, the team will meet regularly both internally and with external 
stakeholders, including all research staff at partner sites University of Washington Medicine and Lancaster 
General Health, after which meeting minutes and task lists will be distributed. All members of the research team 
have completed CITI human subjects research training. Additionally, the protocol and related task sheets will be 
shared on a drive accessible to all staff members. The timeline has been constructed to allow for adequate time 
to conduct and complete the research. There are adequate facilities to ensure the research can proceed. 
 
5.5 Target population 
Patients ≥ 50 years of age in primary care clinics at Penn Medicine, UW Medicine, or Lancaster General 
Health, who have a scheduled new or return non-urgent/sick PCP visit at one of the study practices during the 
intervention period (September-February) and are eligible for the flu vaccination based on Health Maintenance 
data.  

7 
Protocol Version 5: August 2024 
 
 
5.6 Subjects enrolled by Penn Researchers 
Based off our power calculations and preliminary data work, Penn Medicine is expected to enroll approximately 
51,152 patients. 
 
5.7 Subjects enrolled by Collaborating Researchers 
UW Medicine plans to enroll approximately 28,705 patients. 
 
5.8 Accrual 
Patients in this trial will accrue based on their eligible visit during the study period. An eligible visit is defined 
as the patient’s first scheduled new or return non-urgent/sick visit in-person at an eligible clinical site during the 
study window. Patients in all study groups will be enrolled in the trial 4 days before their scheduled eligible 
visit.  
 
5.9 Key inclusion criteria 
Patient Inclusion Criteria 
All patients must meet the following criteria to be eligible: 
1) Age >50 years 
2) A scheduled new or return (non-urgent/sick) primary care appointment at one of the study practices at 
the Penn Medicine, UW Medicine, or Lancaster General Health  
3) Have not received their annual flu vaccine during the active intervention period (September- February) 
4) Eligible to receive the flu vaccine  
For the patient intensification nudge, at least one of the following criteria must be met to be considered high risk 
and randomized to receive the intensification nudge: 
1) Age ≥ 70 years 
2) Living in a lower income community (lowest quartile, zip-code based) 
3) Did not receive a flu vaccine in the previous calendar year 
4) Self-identifies as Non-Hispanic Black 
Clinician Inclusion Criteria 
Clinicians must meet the following criteria to be eligible to receive peer comparison feedback: 
1) Practicing physician (MD, DO) or advanced practice provider (NP, PA) with the exception of residents 
and fellows 
2) Have a minimum patient panel of at least 50 patients, and  
3) Practicing at a clinical site randomized to receive the clinic-level nudge interventions. 
 
 
5.9 Key exclusion criteria 
Patients will be excluded from the study if they: 

8 
Protocol Version 5: August 2024 
 
1) Have a documented allergy to flu vaccine  
2) Have a flu vaccine exclusion modifier in Health Maintenance  
3) Have opted out of research according to individual site guidelines and policies 
4) Have no phone number (home or mobile) listed in their chart 
6. Subject recruitment 
Information on PCPs and their patients at Penn Medicine, UW Medicine, and Lancaster General Health will be 
obtained from the electronic health record using Clarity, an Epic reporting database.     
7. Subject compensation 
No compensation will be offered in this study.  
8. Study procedures 
 
8.1 Consent 
A waiver of informed consent is requested and has been approved for this project in accordance with other, 
similar health system interventions and based on our prior meetings with the Penn IRB. This is a low-risk study 
conducted in partnership with clinical operations, and it is not feasible to consent every patient and clinician for 
a project of this scope, where the emphasis is on low cost, scalable interventions. Additionally, clinicians are 
being nudged towards evidence-based care that is standard of care within the three health systems for this 
project and based on the United States Preventive Services Task Force evidence-based guidelines. Patients and 
clinicians are not being forced into receiving a flu vaccine but, instead, this project is evaluating novel nudge 
interventions to help make it easier for clinicians and patients, especially those who are high risk, to deliver or 
receive evidence-based care. Patients and clinicians retain autonomy to not proceed with a vaccine for any 
given visit. 
 
8.2 Procedures 
--Interventions-- 
All patients in the intervention arm with a new or return, non-urgent/sick visit with a PCP will be identified 4 
days before their scheduled appointment and sent an endowed text message 72 hours prior to their appointment 
(standard messaging). The message will inform the patient that a flu shot has been reserved for them at their 
upcoming appointment. A second nudge will be delivered at 24 hours prior to the scheduled appointment 
reminding them of their upcoming appointment and encouraging the patient to ask their PCP about receiving the 
flu vaccine. 
In addition to the standard endowed text message nudge, high-risk patients randomized to receive the high-risk 
intensification nudge will also receive a bidirectional text intervention in the 72-hours prior to their visit. This 
intervention will query the patient about common questions or concerns about receiving the flu vaccine. If the 
patient responds, it will provide additional educational materials based on the patient’s specific concern(s). If 
the intervention is unable to satisfactorily address a patient’s questions or concerns, a final text will be sent 15 
minutes prior to the eligible visit to remind the patient to discuss their unaddressed concerns with their PCP. 

9 
Protocol Version 5: August 2024 
 
The first clinician nudge uses a default, pended order for the flu vaccine that will be visible to the provider 
during the visit encounter.  Clinical staff will have the option of signing the order or dismissing it if they deem it 
inappropriate for a given patient.  
The second clinician nudge is monthly peer comparison feedback that will be sent as an email facilitated by 
mail merge via Outlook or other bulk email platform per site preference. Clinicians will be told what percent of 
their eligible patients received the flu vaccine and how that compares to other peer clinicians in the intervention. 
If the completion rate is below the median rate, the clinician will receive feedback comparing their rate to the 
median.  If it is above the median but below the 90th percentile, the clinician will receive feedback comparing 
their rate to the 90th percentile. If the clinician’s rate is above the 90th percentile (defined as a top performer), 
they will receive an email informing them that they are a high performer. Residents and fellows will be 
excluded from the peer comparison analysis and messaging. 
--Data-- 
Data on PCPs and their patients at Penn Medicine, UW Medicine, and Lancaster General Health will be 
obtained from Clarity (Epic’s data reporting database). Clinician data includes gender and type of medical 
degree. Patient information includes demographic information, information about comorbid conditions 
(including diabetes, hypertension, and chronic kidney disease, and comorbid conditions needed to calculate the 
Charlson Comorbidity Index), contraindications for flu vaccine, appointment data, and flu vaccination 
administration. 
--Randomization— 
Primary care clinic level randomization will occur electronically.  We will randomize clinics 2:1 to the nudge 
arm or control arm using covariate-constrained randomization.10 Covariate-constrained randomization first 
enumerates a large number (e.g., 50,000) of possible intervention allocations. Then, a measure of covariate 
balance is computed with respect to a set of pre-specified covariates for each possible allocation. Finally, from a 
subset of possible allocations that achieve adequate covariate balance, one is randomly chosen as the final 
allocation of interventions for the study. We will implement a version of covariate-constrained randomization 
that stratifies by health system (Penn and UW) and balances the distribution of clinic size and panel risk across 
arms. The randomization procedure will be implemented in R11 using the cvcrand package.12 For the replication 
trial at Lancaster General Health, we will implement covariate-constrained randomization using the cvcrand 
package to balance the distribution of clinic size and panel risk across arms. 
Within the high-risk patient subgroups, stratifying by intervention clinics, patients will be randomized 1:1 in 
Way to Health using permuted block randomization using random block sizes of 2, 4, and 6 to receive the 
additional intensification bidirectional text messaging nudge or standard messaging. 
9. Analysis plan 
Prior to analyses, we will produce data summaries to assess data quality, data distribution, and randomization 
success. To ensure even comparison across practices and arms, we will classify eligible patients at 4 days before 
a scheduled appointment for a new or return visit with their primary care clinician as being enrolled in the 
study. The primary analysis will be performed using an intention-to-treat approach. Specifically, patients who 

10 
Protocol Version 5: August 2024 
 
do not show up to the first scheduled clinician visit will be considered enrolled and included in the primary 
analysis.  
--Power Calculations-- 
For the following power calculations, we used an adjusted Type I error of 0.025 to account for two 
comparisons: 1) clinic-level nudges versus usual care, and 2) intensification for high-risk patients versus no 
intensification within practices where clinic-level nudges are provided. Based on retrospective data from our 
health systems, we will have 85% power to detect a 7-percentage point difference in the flu vaccine completion 
rate for comparison #1. This calculation assumed N=48 participating clinics across the two sites, an average 
cluster size of n=1385 patients, and an intracluster correlation (ICC) of 0.02. For comparison #2, we will have at 
least 99% power to detect a 2-percentage point difference even if the rate of vaccination in the high-risk group 
is 50%, the most conservative rate with respect to power. This calculation assumed 1:1 individual-level 
randomization of n=1000 high-risk patients within each of N=32 clinics that will be randomized to receive 
clinic-level nudges. These power calculations used combined retrospective data from Penn and UW to estimate 
the average number of patients per clinic and the number of high-risk patients who will be randomized. We 
expect to have greater power for both comparisons at the replication site which comprises more clinics and 
patients. 
--Statistical Analysis-- 
The primary analysis will fit generalized estimating equations (GEE) with a binary outcome indicating whether 
a flu vaccine was received during the PCP visit (primary). Following the intention-to-treat framework, patients 
who do not complete their first scheduled visit will contribute a primary outcome of zero, indication a flu 
vaccine was not received during the PCP visit. We will also perform analyses based on a binary outcome that 
indicates whether a flu vaccine was received 3 months after the first scheduled visit, regardless of whether the 
visit was completed (secondary). An exchangeable working correlation will be specified at the clinic level for 
comparison #1 (evaluation of clinic-level nudges) and robust standard errors will be provided, clustered at the 
clinic level. An independence working correlation and robust standard errors without clustering will be used for 
comparison #2 (evaluation of individual-level nudges). In sensitivity analyses we will explore alternative 
specifications for the working correlation structure to examine robustness of parameter estimates and inference. 
In additional sensitivity analyses, we will censor patients with a PCP visit late in the intervention period 
(February/March) who are unlikely to complete a flu vaccine beyond March and will therefore not have a full 3 
months to obtain a vaccine and examine completion rates among these patients.   
We will examine effects of the interventions on overall rates of ordering. We will compare rates of ordering 
between the three intervention arms: 1) Control, 2) Clinic-level nudges only, and 3) Clinic-level nudges with 
intensification for high risk. We will fit a GEE with binary outcome indicating whether an order was signed or 
not. We will fit the model using the weighted and replicated approach13 to account for the fact that low-risk 
patients seen in the clinics that received nudges are consistent with both intervention arms 2) and 3). We will 
include terms for the main effects of clinic-level and individual-level nudges as well as their interaction. We 
will test the appropriate linear combinations of the parameters that correspond to pairwise differences between 
each of the three intervention arms. An independence working correlation with robust standard errors will be 

11 
Protocol Version 5: August 2024 
 
used to account for uncertainty in the weight distribution due to variability in the empirical proportion of high- 
versus low-risk patients. 
We will account for the clinic-level covariate-constrained randomization by including health system, clinic size, 
and panel risk as covariates in the models for the effect of clinic-level nudges. In models for estimating the 
effect of intensification on high-risk patients, we will control for differences across health system by including 
clinic fixed effects which will also account for the stratified randomization.  
We will perform exploratory subgroup analyses by risk group as it is defined for the individual randomization. 
For the low-risk strata, this will allow us to study the effect of the clinic-level nudges versus usual care amongst 
low-risk patients. For the high-risk strata, we will fit models that include each category used to define high risk 
(e.g., sex, race, previous year vaccination) as well as their interaction with treatment arm to determine which 
high-risk subgroups experienced the greatest benefits of the clinic- and individual-level nudges. We will also 
include a visit-level analysis with clustering by patient as an exploratory analysis.    
Investigators and analysts will be blinded to study arm assignment during the intervention and analysis phases. 
The statistician is responsible for clinic-level randomization, but subsequently will be blinded to study arm 
assignment for the remainder of the intervention phase and analysis phase. Project staff (site PIs, project 
director, project manager, and study coordinators) will be unblinded during the intervention and analysis phases 
to complete daily study activities and prepare the blinded analysis files. Unblinding for investigators, 
statisticians, and analysts will be completed by the project staff and occur at the end of study analysis once all 
of the outcomes have been ascertained.  
10. Investigators 
Shivan Mehta, MD, MBA, MSHP is a Multiple Principal Investigator (MPI) and an Associate Professor of 
Medicine and Health Policy at the Perelman School of Medicine (PSOM). He has prior experience leading 
clinical trials to deploy interventions that leverage behavioral economics and technology to improve population 
health outcomes.   
Amol Navathe, MD, PhD is a Multiple Principal Investigator (MPI), Associate Director of the Center for 
Health Incentives and Behavioral Economics, and Professor of Medicine and Health Policy at the Perelman 
School of Medicine (PSOM). He is a general internist and health economist with a research portfolio focusing 
on physician behavior. His expertise includes large‐scale policy trial design in multiple health plans and health 
care organizations around physician incentives and high‐value practice. 
Joshua Liao, MD, MSc, FACP is a Multiple Principal Investigator (MPI) and Professor and Division Chief of 
the William T. and Gay F. Solomon Division of General Internal Medicine at University of Texas Southwestern 
Medical Center.  He has prior experience designing and implementing nudge interventions in pragmatic 
randomized trials aimed at improving health care value and population health.   
Kristin Linn, PhD is a Co-Investigator and an Assistant Professor of Biostatistics at the University of 
Pennsylvania. She has prior expertise in statistical methods for sequentially randomized clinical trials, 
predictive modeling, and machine learning techniques.  
Kimberly Waddell, PhD, MSCI is a Co-Investigator and an Assistant Professor of Physical Medicine and 
Rehabilitation at the University of Pennsylvania and a Core Investigator at the Philadelphia VA Center for 

12 
Protocol Version 5: August 2024 
 
Health Equity Research and Promotion . She has prior experience with clinical trials and nudge interventions to 
improve clinician and patient decision making.  
 
11. Human research protection 
 
11.1 Data confidentiality 
Computer-based files will only be made available to personnel involved in the study through the use of access 
privileges and passwords. Wherever feasible, identifiers will be removed from study-related information. 
Precautions are already in place to ensure the data are secure by using passwords and HIPAA-compliant 
encryption. 
 
11.2 Subject confidentiality 
Data on clinicians and patients will be obtained from Epic. Any information that is obtained will be used for 
research purposes only.  Information on patients will only be disclosed within the study team and to the 
patient’s PCP.  All study staff will be reminded of the confidential nature of the data collected and contained in 
these databases. 
Information about study subjects will be kept confidential and managed according to the requirements of the 
Health Insurance Portability and Accountability Act of 1996 (HIPAA). No source documents will be printed 
or maintained in paper form at the study sites. Operational data used for daily study management will be 
stored on a secure drive maintained behind the University of Pennsylvania Health System (UPHS) firewall.  
Analytic data will be stored within Penn Leonard David Institute’s Health Services Research Data Center 
(HSRDC). HSRDC is maintained at a high-security level in accordance with federal regulations governing 
secure computer systems (e.g., the Federal Information Security Management Act-FISMA). The Way to 
Health (W2H) platform servers are based in the University of Pennsylvania Digital Academic Research 
Transformation (DART). The DART provides a secure computing environment for a large volume of highly 
sensitive data. DART requires all users of data or applications on DART servers to complete a DART-hosted 
cybersecurity awareness course annually, which stresses federal data security policies under data use 
agreements with the university. Curriculum includes HIPAA training and covers secure data transfer, 
passwords, computer security habits and knowledge of what constitutes misuse or inappropriate use of the 
server. W2H uses a role-based access control (RBAC) approach to assure that participant confidentially and 
study integrity is preserved. Access is granted by invitation only and can be revoked at any time. The W2H 
Team are employees of the University of Pennsylvania and Penn Medicine. All W2H team members have 
completed HIPAA Security training and CITI Protection of Human Subjects Research Training - ORA.  
Access to the backend database is restricted and only available to a select group of developers. The database is 
accessible only via a secure VPN (Virtual Private Network) and cannot be accessed from the public internet at 
all, i.e. authorized users can only access the databases from within Penn’s network and over a secure VPN 
channel. Data from participating sites will be transferred to Penn (the data coordinating center) according to 
the methods defined in a fully executed Data Use Agreement. All data will be covered under the appropriate 
Data Use Agreements between Penn Medicine, UW Medicine, and Lancaster General Health and follow strict 
security guidelines. 
 

13 
Protocol Version 5: August 2024 
 
 
11.3 Subject privacy 
All efforts will be made by study staff to ensure subject privacy.  Data will be evaluated in a de-identified 
manner whenever possible. 
 
11.4 Data disclosure 
Information on patients will only be disclosed within the study team and to the patient’s primary care clinician 
(to whom this information is already available).   
 
11.5 Protected Health Information/Data Protection 
• Name 
• Street address, city, county, precinct, zip code, and equivalent geocodes 
• All elements of dates (except year) for dates directly related to an individual and all ages over 89 
• Telephone and fax numbers 
• Medical record numbers 
 
11.6 Data and safety monitoring 
A Data and Safety Monitoring Board (DSMB) is an independent group of experts convened to protect the safety 
of research subjects and to ensure that the scientific goals for the project are being met. While this study 
presents very low risk, the DSMB will be appointed by and act in an advisory capacity to the National Institute 
on Aging (NIA) to monitor participant safety, data quality and to evaluate the progress of the study. The Data 
and Safety Monitoring Plan (DSMP) will be reviewed and approved by the DSMB prior to study start. Detailed 
information about safety monitoring and reporting can be found in the Manual of Operating Procedures (Section 
11) and the Data and Safety Monitoring Plan. 
11.7 Risk/benefit  
 
 
11.7.1 Potential study risks 
There are minimal risks to PCPs and patients in this trial. Ordering an annual flu vaccine is a standard of care 
for these patients. The only research-related activity is the randomization of participants to different behavioral 
nudge strategies to increase flu vaccination uptake.  Potential risks include patient frustration with messaging or 
outreach, duplicate flu vaccination within the recommended interval, or reaction requiring emergency 
department or hospital care among those with a previously known allergy or overdose of vaccine. Risks will be 
minimized by excluding any patients for which this study could be unsafe, i.e. if they have a documented 
allergy to the vaccine or documentation in their medical record that they have already received their flu vaccine 
this season. 
There is a risk of breach of data and confidentiality. To minimize the risk of breach of data and confidentiality, 
we will use secure, encrypted servers to host the data and conduct the analysis. Operational data used for daily 
study management will be stored on a secure drive maintained behind the University of Pennsylvania Health 
System (UPHS) firewall. Analytic data will be stored within Penn Leonard David Institute’s Health Services 
Research Data Center (HSRDC). HSRDC is maintained at a high-security level in accordance with federal 

14 
Protocol Version 5: August 2024 
 
regulations governing secure computer systems (e.g., the Federal Information Security Management Act-
FISMA). The Way to Health (W2H) platform servers are based in the University of Pennsylvania Digital 
Academic Research Transformation (DART). Data from participating sites will be transferred to Penn (the data 
coordinating center) according to the methods defined in a fully executed Data Use Agreement. All data will be 
covered under the appropriate Data Use Agreements between Penn Medicine, UW Medicine, and Lancaster 
General Health and follow strict security guidelines. 
 
 
11.7.2 Potential study benefits 
Receiving an annual flu vaccine is important for public health and the health of the patients. The EHR could be 
utilized to identify eligible patients, and patients at high risk for not receiving the vaccine and deliver nudges to 
help improve vaccine completion rates. The knowledge gained on how to increase flu vaccination rates could be 
applied to other populations and implemented at other health systems. PCPs may benefit from receiving 
feedback on their performance and having the vaccine order automatically ordered to help streamline the 
ordering process and prevent extra burden. 
 
 
11.7.3 Risk/benefit assessment 
The risk/benefit ratio is highly favorable given the potential benefit from eligible patients receiving a flu 
vaccine, especially those identified as high risk, that ordering and receiving a flu vaccine is within the standards 
of care in a primary care practice, and that efforts have been put into place to minimize the risk of breach of 
data.  
 
 

15 
Protocol Version 5: August 2024 
 
References 
1. 
CDC. Disease Burden of Flu. Accessed April 28, 2022, 
https://www.cdc.gov/flu/about/burden/index.html?CDC_AA_refVal=https%3A%2F%2Fwww.cdc.gov%2Fflu
%2Fabout%2Fdisease%2Fburden.htm 
2. 
Kohlhammer Y, Schnoor M, Schwartz M, Raspe H, Schäfer T. Determinants of influenza and 
pneumococcal vaccination in elderly people: a systematic review. Public health. 2007;121(10):742-751.  
3. 
CDC. Vaccination Coverage Among Adults in the United States, National Health Interview Survey, 
2016. Accessed April 28, 2022, https://www.cdc.gov/vaccines/imz-managers/coverage/adultvaxview/pubs-
resources/NHIS-2016.html 
4. 
Lu D, Qiao Y, Brown NE, Wang J. Racial and ethnic disparities in influenza vaccination among adults 
with chronic medical conditions vary by age in the United States. PloS one. 2017;12(1):e0169679.  
5. 
Henry J, Pylypchuk Y, Searcy T, Patel V. Adoption of Electronic Health Record Systems among U.S. 
Non-Federal Acute Care Hospitals: 2008-2015.11.  
6. 
Navathe A, Liao J, Delgado K, et al. Effect of Peer Comparison Feedback, Individual Audit Feedback or 
Both to Clinicians on Opioid Prescribing in Acute Care Settings: A Cluster Randomized Clinical Trial. Health 
Services Research. 2021 2021;56:55-56.  
7. 
Mehta SJ, Khan T, Guerra C, et al. A randomized controlled trial of opt-in versus opt-out colorectal 
cancer screening outreach. American Journal of Gastroenterology. 2018 2018;113(12):1848-1854.  
8. 
Adusumalli S, Jolly E, Chokshi NP, et al. Referral Rates for Cardiac Rehabilitation Among Eligible 
Inpatients After Implementation of a Default Opt-Out Decision Pathway in the Electronic Medical Record. 
JAMA network open. 2021 2021;4(1):e2033472-e2033472.  
9. 
Patel MS, Kurtzman GW, Kannan S, et al. Effect of an automated patient dashboard using active choice 
and peer comparison performance feedback to physicians on statin prescribing: the PRESCRIBE cluster 
randomized clinical trial. JAMA network open. 2018 2018;1(3):e180818-e180818.  
10. 
Moulton LH. Covariate-based constrained randomization of group-randomized trials. Clinical Trials. 
2004;1(3):297-305.  
11. 
R: A language and environment for statistical computing. R Foundation for Statistical Computing; 2019.  
12. 
Yu H, Li F, Gallis JA, Turner EL. cvcrand: A Package for Covariate-constrained Randomization and the 
Clustered Permutation Test for Cluster Randomized Trials. R Journal. 2019;9(2) 
13.  
Nahum-Shani, I., Qian, M., Almirall, D., Pelham, W. E., Gnagy, B., Fabiano, G. A., ... & Murphy, S. A. 
(2012). Experimental design and primary data analysis methods for comparing adaptive interventions. 
Psychological methods, 17(4), 457. 

16 
Protocol Version 5: August 2024