Pre-visit Questionnaire to Increase Influenza Vaccinations
Phase: NA
Conditions: Influenza
Sponsor: Geisinger Clinic

 
Pre-visit Questionnaire to Increase Influenza Vaccinations 
 
Study Protocol with Statistical Analysis Plan 
 
NCT05508698 
 
7/21/2023 
 

 
 
 
Study Protocol 
 
Scientific Background 
 
Almost everyone ages 6 months or older can benefit from the influenza vaccine, which can 
reduce illnesses, missed work, hospitalizations, and death by reducing the likelihood of 
contracting influenza.  
 
Objectives 
 
The purpose of this study is to assess, prospectively, the effect on flu vaccination rates of a pre-
visit questionnaire that asks patients to indicate their preferences for a flu shot at an upcoming 
appointment. Responses to the questionnaire are shown to clinicians via additional text in an 
existing flu shot alert. The investigators hypothesize that the pre-visit questionnaire will lead to 
increased flu vaccination compared with standard practices. 
 
Design  
 
This study is a randomized controlled trial with 2 study arms. Patients were pre-randomized to 
receive (or not receive) a one-item, pre-visit questionnaire about the flu shot. Questionnaire 
responses are shown to clinicians in an existing flu shot alert. 
 
Methods 
 
Patients were pre-randomized to two study arms in July 2022: 
 
1. Passive Control: No change to standard of care for flu vaccines 
2. Pre-visit Questionnaire: One-item questionnaire in online patient portal and additional 
information in the flu shot alert. 
 
(Note that patients new to Geisinger after pre-randomization are not included in the study.)  
 
Patients are enrolled in the study if they have a flu-shot-eligible appointment scheduled 14 days 
or less prior to their appointment.  
 
Appointments are flu-shot-eligible if the appointment department administers flu shots 
(according to an EHR classification list) at the time the appointment questionnaire becomes 
available (14 days prior to the appointment for appointments scheduled 14 days or more before 
the appointment, or on the day the appointment is scheduled for appointments scheduled less 
than 14 days in advance).  
 
For those in the passive control arm, we will retrospectively identify flu-shot-eligible 
appointments based on the departments and date ranges observed in the questionnaire arm. 
For example, if questionnaires were sent to patients in the questionnaire arm for appointments 
in a given department from 9/1/22–12/31/22, an appointment in that same department in that 
date range will be considered flu-shot-eligible for a patient in the passive control group. 
 

 
Additionally, patients are enrolled if, according to the EHR, they have not yet received a flu shot 
at the time the questionnaire becomes available (or at the time it would have become available 
for the passive control group). 
 
Power Analysis 
 
We expect that about 165,000 patients will be enrolled in the study (~82,500 per arm). This 
sample size allows 80% power to detect an increase in flu vaccination rates from 35% to 35.7% 
with two-tailed alpha = .05 for any comparison between study arms. 
 
Project Status 
 
Patients were pre-randomized to their study arms in July 2022. Enrollment was complete on 
4/1/23, after which the flu shot alerts were turned off across the system. Data collection 
continued until the primary outcome date of 4/8/23 (7 days after the final appointments). This 
updated version of the statistical analysis plan specifies slightly modified inclusion criteria for 
analysis, to correct as well as possible, and in an unbiased manner, for differences in how 
MyChart questionnaires were actually implemented relative to what was intended. No outcomes 
data were compared across study arms, and final outcomes data were not pulled or reviewed 
prior to uploading this document. 
 
 
 
Statistical Analysis Plan 
 
Planned Analyses 
 
Primary Outcome: Count of patients with a flu vaccination [ Time Frame: Up to 21 days ] 
 
Outcome Description: Patient received a flu vaccine (yes/no) between the day questionnaires 
became available for the patient’s first scheduled flu-shot-eligible appointment and 7 days 
following the appointment. Questionnaires were available 14 days prior to the appointment date 
for appointments scheduled at least 14 days in advance. Questionnaires were available on the 
day the appointment was scheduled for appointments scheduled less than 14 days in advance. 
 
Note that control patients did not receive a flu shot questionnaire, but they may have received 
other questionnaires from the health system 14 days before their appointment. We will conduct 
subgroup analyses for people who had responded to MyGeisinger pre-visit messages in the 
previous year and those who did not. 
 
Question: Does a pre-visit questionnaire that allows patients to indicate their flu shot 
preferences increase the likelihood that they will get a flu shot? 
 
Analysis (Confirmatory): We will test the hypothesis that a pre-visit flu-shot questionnaire 
increases the likelihood that patients will get a flu shot. We will run an OLS regression to 
examine whether flu vaccination differs as a function of experimental arm.  
 
Analysis Notes 
 
To identify the primary analysis sample, we will limit the dataset to the first flu shot-eligible 
appointment for which each enrolled patient was scheduled. We will then further limit the 

 
dataset to appointments where patients were expected to receive a questionnaire (or for those 
in the passive control group, where the patient would have been expected to have received a 
questionnaire), as defined further here: 
• 
The appointment was in a department and entailed an encounter type that was 
supposed to trigger a questionnaire (even if a questionnaire was not always triggered) 
• 
The patient was not marked as contraindicated for the flu vaccine in the flu shot BPA for 
the 150 days prior to the questionnaire 
• 
The patient had a “flu shot overdue” health maintenance topic (HMT) as of two days 
prior to the questionnaire launch date, ensuring sufficient time for the HMT to 
successfully trigger the questionnaire (given that a large proportion of questionnaires 
were not triggered when the HMT was too recent), OR the appointment was between 
9/1/22 and 9/15/22, at which time the questionnaires were triggered regardless of the 
HMT status 
 
Recent work suggests that OLS regressions are appropriate in randomized experiments with 
binary outcome variables such as ours (Gomila, 2021).  
 
As the treatment variation is at the individual level, we will report heteroskedasticity-robust 
standard errors. We will also explore the impact of clustering these standard errors at the clinic 
and clinic-date levels to allow for dependence across observations within these clusters. We will 
also investigate heterogeneity across clinics that vary along characteristics of interest, including 
prior-year vaccination rates. 
 
We may run additional robustness checks. These checks may include focusing on additional 
inclusion/exclusion criteria (e.g., including subsequent visits with questionnaires for a given 
patient) and subpopulations (e.g., different visit types). 
 
 
Other Pre-specified Outcomes 
 
Other Pre-specified Outcomes listed below include flu outcomes (diagnosis, complications) and 
COVID-19 vaccination. If there are any differences in these outcomes as a function of study 
arm, the mechanism would almost certainly be increased flu vaccination. Therefore, we will only 
run analyses on Other Pre-specified Outcomes for analyses above where there is a significant 
difference in flu vaccination. 
 
1. Count of patients with a high confidence flu diagnosis 
 
Patient received a flu diagnosis via a positive polymerase chain reaction 
(PCR)/antigen/molecular test (yes/no) during the 2022-23 flu season (from the patient’s 
study start date through April 30, 2023). 
 
[Time Frame: Up to 8 months] 
 
2. Count of patients with a "likely flu" diagnosis 
 
Received a "high confidence flu" diagnosis (with positive PCR/antigen/molecular test) 
and/or "likely flu" diagnosis (as assessed via International Classification of Disease [ICD] 
codes or Tamiflu administration or positive PCR/antigen/molecular test) (yes/no) during 
the 2022-23 flu season (from the patient’s study start date through April 30, 2023). 
 

 
Note that "likely flu" is a superset of the "high confidence flu" diagnoses. 
 
[Time Frame: Up to 8 months] 
 
3. Count of patients with flu complications 
 
Diagnosed with flu-related complications (yes/no) from the patient’s study start date 
through July 31, 2023. 
 
[Time Frame: Up to 11 months] 
 
4. ER visits 
 
Number of ER visits from the patient’s study start date through July 31, 2023. 
 
[Time Frame: Up to 11 months] 
 
5. Hospitalizations 
 
Number of hospitalizations from the patient’s study start date through July 31, 2023. 
 
[Time Frame: Up to 11 months] 
 
6. COVID-19 vaccination rates 
 
Received at least one COVID-19 vaccination (yes/no) during the 2022-23 flu season 
(from the patient’s study start date through April 30, 2023). 
 
[Time Frame: Up to 8 months]  
 
7. Count of patients with a flu vaccination during the 2022-2023 season 
 
Patient received a flu vaccine (yes/no) during the 2022-23 flu season (from the patient's 
appointment date through April 30, 2023) 
 
[Time Frame: Up to 8 months]  
 
 
 
Additional Exploratory Analyses 
 
1. Age and sex 
 
People of different ages and sexes may react differently to the questionnaire. To test the 
relation between flu shots, age, and sex, we will run an OLS regression including binned 
patient age (18–24, 25–34, 35–44, 45–54, 65+), sex, and their interaction.  
 
We will additionally test for an interaction between age, sex, and study arm, as people of 
different ages and sexes may be differentially receptive to different alert versions. 
 
We may also test whether alert effectiveness varies by clinician age and/or sex. 

 
 
2. Number of appointments with a flu shot questionnaire 
 
Patients may have received more than one flu-shot questionnaire during the flu season. 
We will explore whether vaccination rates increased with more exposure to the flu-shot 
questionnaire in the experimental group compared with those in the control group who 
had the same number of eligible appointments but were not sent the questionnaire. 
 
3. Appointment department/specialty 
 
Some departments give more flu shots than others. We will test if alerts with or without 
questionnaire responses are differentially effective as a function of department or 
specialty, to see if some versions are particularly helpful for under-performing 
departments or specialties. 
 
 
4. Contamination analysis 
 
Although randomization was at the patient level, the alert portion of the intervention was 
directly experienced by clinicians rather than patients. Many clinicians encountered 
patients and their assigned alerts for both experimental arms. We will explore whether 
contamination was present in our data due to clinicians’ exposure to multiple 
experimental conditions (e.g., by examining results as a function of clinic-level variation 
in the number of patients randomly assigned to each arm and/or as a function of the 
duration of exposure to the information).