Patient Portal Reminder/Recall for Influenza Vaccination in a Health System- RCT 3
Phase: NA
Conditions: Influenza, Respiratory Tract Infections
Sponsor: University of California, Los Angeles

Study Title: Patient Portal Reminder/Recall for Influenza Vaccination in a Health System- RCT 3 
Initial IRB Approval: 12/20/2017 
RCT 3 IRB Approval: 8/14/2020 
Clinical Trials Registration: NCT04533685 
 
 

Protocol 
Brief Summary: 
This trial is taking place in Los Angeles, CA among patients from primary care practices within 
the UCLA Health System. 
Despite the Advisory Committee on Immunization Practices (ACIP) recommendation in 2010 
that all people above 6 months of age should receive an annual flu vaccine, vaccination rates 
remain low. The investigators plan to assess the effectiveness of reminder/recall messages 
(direct appointment scheduling, no direct-appointment scheduling, pre-appointment, and no 
pre-appointment) encouraging influenza vaccinations sent to patient portal users, as well as the 
effectiveness of a pre-commitment prompt (pre-commitment prompt, no prompt) asking about 
a patient's intention to get the influenza vaccination, using a 2 x 2 x 2 nested factorial design. 
Methods and Procedures 
We propose a two-arm randomized controlled trial, and a 2 x 2 x 2 nested factorial design 
within the treatment arm. Patients are randomized within UCLA primary care practices to 
intervention arms sent via the portal or a control group. Study arms: 1) standard of care control 
(no reminder messages, no pre-commitment prompt, except for UCLA employees sent a 
reminder), 2) direct appointment scheduling reminder, pre-appointment reminder, pre-
commitment prompt, 3) direct appointment scheduling reminder, pre-appointment reminder, 
no pre-commitment prompt, 4) direct appointment scheduling reminder, no pre-appointment 
reminder, pre-commitment prompt, 5) direct appointment scheduling reminder, no pre-
appointment reminder, no pre-commitment prompt 6) No direct appointment scheduling 
reminder, pre-appointment reminder, pre-commitment prompt, 7) No direct appointment 
scheduling reminder, pre-appointment reminder, no pre-commitment prompt, 8) No direct 
appointment scheduling reminder, no pre-appointment reminder, pre-commitment prompt, 9) 
No direct appointment scheduling reminder, no pre-appointment reminder, no pre-
commitment prompt. 
Study Arm 
Intervention/Treatment 
 
No Intervention: Standard of care control 
 
No direct-scheduling or pre-appointment 
portal reminders + No pre-commitment 
prompt  
Participants do not receive any 
reminder/recall messages regarding influenza 
vaccination via the patient portal and do not 
receive a pre-commitment prompt asking if 
they plan to receive the influenza vaccine in 
the upcoming season 
NA 

Experimental: Direct appointment scheduling 
reminder + Pre-appointment reminder + Pre-
commitment prompt 
 
Participants receive 1) reminder/recall 
messages regarding influenza vaccination via 
the patient portal with a link enabling direct 
scheduling, 2) a pre-appointment reminder 
that mention asking the provider for a flu 
vaccine, and 3) a pre-commitment prompt 
asking if they plan to receive the influenza 
vaccine in the upcoming season.  
Behavioral: Portal Reminders for Influenza 
Vaccination: Direct scheduling link: Patients 
receive reminder/recall messages via the 
patient portal to get an influenza vaccination 
that include a direct scheduling link, enabling 
the patient to directly schedule a flu vaccination 
only visit.  
 
Behavioral: Pre-Appointment influenza 
vaccination reminder: Patients receive modified 
pre-appointment reminders for upcoming 
scheduled appointments that include language 
encouraging patients to ask the provider for the 
flu vaccine.  
 
Behavioral: Pre-commitment prompt 
Patients receive a pre-commitment prompt 
asking about their intention to get an influenza 
vaccination. 
Experimental: Direct appointment scheduling 
reminder + Pre-appointment reminder + No 
pre-commitment prompt 
 
Participants 1) reminder/recall messages 
regarding influenza vaccination via the 
patient portal with a link enabling direct 
scheduling, and 2) a pre-appointment 
reminder that mention asking the provider 
for a flu vaccine, but do not receive a pre-
commitment prompt asking if they plan to 
receive the influenza vaccine in the upcoming 
season.  
Behavioral: Portal Reminders for Influenza 
Vaccination: Direct scheduling link: Patients 
receive reminder/recall messages via the 
patient portal to get an influenza vaccination 
that include a direct scheduling link, enabling 
the patient to directly schedule a flu vaccination 
only visit.  
 
Behavioral: Pre-Appointment influenza 
vaccination reminder: Patients receive modified 
pre-appointment reminders for upcoming 
scheduled appointments that include language 
encouraging patients to ask the provider for the 
flu vaccine.  
Experimental: Direct appointment scheduling 
reminder + No pre-appointment reminder + 
Pre-commitment prompt  
 
Participants receive 1) reminder/recall 
messages regarding influenza vaccination via 
the patient portal with a link enabling direct 
scheduling, and 2) a pre-commitment prompt 
asking if they plan to receive the influenza 
vaccine in the upcoming season, but do not 
Behavioral: Portal Reminders for Influenza 
Vaccination: Direct scheduling link: Patients 
receive reminder/recall messages via the 
patient portal to get an influenza vaccination 
that include a direct scheduling link, enabling 
the patient to directly schedule a flu vaccination 
only visit.  
 
Behavioral: Pre-commitment prompt 

receive a pre-appointment reminder that 
mention asking the provider for a flu vaccine. 
Patients receive a pre-commitment prompt 
asking about their intention to get an influenza 
vaccination 
Experimental: Direct appointment scheduling 
reminder + No pre-appointment reminder + 
No pre-commitment prompt 
 
Participants receive 1) reminder/recall 
messages regarding influenza vaccination via 
the patient portal with a link enabling direct 
scheduling, but do not receive a pre-
appointment reminder that mention asking 
the provider for a flu vaccine, or a pre-
commitment prompt asking if they plan to 
receive the influenza vaccine in the upcoming 
season.  
Behavioral: Portal Reminders for Influenza 
Vaccination: Direct scheduling link: Patients 
receive reminder/recall messages via the 
patient portal to get an influenza vaccination 
that include a direct scheduling link, enabling 
the patient to directly schedule a flu vaccination 
only visit.  
 
Experimental: No direct appointment 
scheduling reminder + Pre-appointment 
reminder + Pre-commitment prompt 
 
Participants receive 1) reminder/recall 
messages regarding influenza vaccination via 
the patient portal without a link enabling 
direct scheduling, 2) a pre-appointment 
reminder that mentions asking the provider 
for a flu vaccine, and 3) a pre-commitment 
prompt asking if they plan to receive the 
influenza vaccine in the upcoming season.  
Behavioral: Portal Reminders for Influenza 
Vaccination: No direct scheduling link: Patients 
receive reminder/recall messages via the 
patient portal to get an influenza vaccination 
(no direct scheduling link included).  
 
Behavioral: Pre-Appointment influenza 
vaccination reminder: Patients receive modified 
pre-appointment reminders for upcoming 
scheduled appointments that include language 
encouraging patients to ask the provider for the 
flu vaccine.  
 
Behavioral: Pre-commitment prompt 
Patients receive a pre-commitment prompt 
asking about their intention to get an influenza 
vaccination 
Experimental: No direct appointment 
scheduling reminder + Pre-appointment 
reminder + No pre-commitment prompt 
 
Participants receive 1) reminder/recall 
messages regarding influenza vaccination via 
the patient portal without a link enabling 
direct scheduling, and 2 a pre-appointment 
reminder that mentions asking the provider 
for a flu vaccine, but do not receive a pre-
commitment prompt asking if they plan to 
Behavioral: Portal Reminders for Influenza 
Vaccination: No direct scheduling link: Patients 
receive reminder/recall messages via the 
patient portal to get an influenza vaccination 
(no direct scheduling link included).  
Behavioral: Pre-Appointment influenza 
vaccination reminder: Patients receive modified 
pre-appointment reminders for upcoming 
scheduled appointments that include language 
encouraging patients to ask the provider for the 
flu vaccine.  

receive the influenza vaccine in the upcoming 
season. 
Experimental: No direct appointment 
scheduling reminder + No pre-appointment 
reminder + Pre-commitment prompt 
 
Participants receive 1) reminder/recall 
messages regarding influenza vaccination via 
the patient portal without a link enabling 
direct scheduling, and 2)a pre-commitment 
prompt asking if they plan to receive the 
influenza vaccine in the upcoming season, 
but do not receive a pre-appointment 
reminder that mentions asking the provider 
for a flu vaccine. 
Behavioral: Portal Reminders for Influenza 
Vaccination: No direct scheduling link: Patients 
receive reminder/recall messages via the 
patient portal to get an influenza vaccination 
(no direct scheduling link included).  
 
Behavioral: Pre-commitment prompt 
Patients receive a pre-commitment prompt 
asking about their intention to get an influenza 
vaccination 
Experimental: No direct appointment 
scheduling reminder + No pre-appointment 
reminder + No Pre-commitment prompt 
 
Participants receive 1) reminder/recall 
messages regarding influenza vaccination via 
the patient portal without a link enabling 
direct scheduling, but do not a pre-
appointment reminder that mentions asking 
the provider for a flu vaccine, or a pre-
commitment prompt asking if they plan to 
receive the influenza vaccine in the upcoming 
season. 
Behavioral: Portal Reminders for Influenza 
Vaccination: No direct scheduling link: Patients 
receive reminder/recall messages via the 
patient portal to get an influenza vaccination 
(no direct scheduling link included).  
 
Spacing of R/R messages: The pre-commitment prompt will be sent in September, direct 
appointment scheduling and no direct appointment scheduling reminders will be sent one time 
per month until record of vaccine receipt appears in the patient EHR in October, November, 
and December. The pre-appointment reminders will be sent 2-4 days in advance of scheduled 
visits. 
Languages: messages will be in English.   
Health Literacy: We will use plain language <7th grade reading level per Flesch-Kincaid analysis.  
Eligibility: Our EHR will identify subjects who are eligible for either 1 or 2 influenza vaccinations 
based on ACIP algorithms (2 are recommended for children <9 years who have not had prior 
vaccination). 

Inclusion criteria 
Inclusion criteria will involve-- all UCLA primary care patients over 4 months of age who have 
made at least 1 visit to the practice within the prior 3 years and who have signed up for 
MyUCLAhealth (UCLA’s portal) and have used the portal at least once within 12 months. 
Currently this includes 298,112 patients; from these patients, index patients per household will 
be randomly selected. 
a. Active patient: We selected >1 visit in 3y based on our prior studies and the literature to 
focus the intervention on active patients; practices do not have a good mechanism to inactivate 
patients.  
b. Focusing on Portal Users: We can only send portal R/R messages to patients who have signed 
up for the portal. Since some patients sign up for the portal but never use it we will restrict our 
inclusion criteria to patients who used the portal within 12m (75% of UCLA patients). This 
standard has been applied before. We will assess intervention effects for the included 
population, and also for the entire practice, since the impact of the intervention reflects the 
extent of portal penetration.  
Exclusion criteria 
Patients will be excluded if they are not a patient at one of the 67 primary care practices within 
UCLA’s Health System, and/or if they do not meet the above specified requirements for being 
an active patient or portal user. 
Main Outcomes and Measures 
Primary Measure: Receipt of >1 influenza vaccine among index patients between 10/1/2020-
03/31/2021 as documented in the EHR which is supplemented electronically by the health 
system with influenza vaccination data from external sources (e.g., pharmacies) and claims 
data.   
Secondary Measures: Receipt of annual influenza vaccine between 10/1/2020-03/31/2021 
among all included patients in identified households.