Improving Influenza Vaccination Delivery Across a Health System by the Electronic Health Records Patient Portal
Phase: NA
Conditions: Influenza, Respiratory Tract Infections, Vaccines
Sponsor: University of California, Los Angeles

1 
 
Clinicaltrials.gov application number:  NCT03666026 
Title: Improving Influenza Vaccination Delivery Across a Health System by the Electronic Health Records 
         Patient Portal  
Unique Protocol ID17-001889 
Submission Date: 12/14/2017 
IRB Approval Date: 12/20/2017  
 
 
 

2 
 
Research Design and Methods: 
Describe in detail the design and methodology of the study. 
 
SPECIFIC AIMS 
The overall purpose of the RCTs is to evaluate the impact that R/R, delivered through the patient portal, could 
have on the flu immunization rates of UCLA Health patients. The intent is to ultimately take the information 
learned and create and disseminate an adaptable toolkit to other health systems. 
 
Specific aims are 
 
Aim #1: Adapt algorithms, educational messages, and protocols previously used for mailed or phone influenza 
vaccine R/R, to create a patient portal research platform. (Aim 1 is outlined in detail in IRB applications titled: 
Patient Portal - Patient Qualitative Interviews and Patient- Portal Provider Qualitative Interview) 
 
Aim #2: Assess the impact of portal R/R and key design features upon flu vaccination rates and costs. 
2a: Using a 4-arm RCT, clustering within practices (up to 55), compare the effectiveness of 1, 2, or 3 
portal R/R’s vs. 0 R/R’s on influenza vaccination rates 
Hyp. 2a.1 [Primary Outcome]: >1 portal R/R will increase vaccination rates vs. no R/R. 
Hyp. 2a.2: More R/R messages will raise vaccination rates (3R/R > 2R/R > 1R/R > 0R/R). 
 
 
OVERALL STUDY DESIGN 
Subjects receiving the intervention will be patients at one of the primary care practices within UCLA Health’s 
System (up to 55 practices), and will include patients of any age. A proportion of the patients from these 
practices will be selected to participate - i.e. those who meet the inclusion criteria. Randomization techniques 
will occur before each RCT. 
 
Patients will either be randomized to receive 0 (control), 1, 2 or 3 flu reminder messages via the portal. 
 
Number of Subjects 
There are approx n = 385,000 patients in the primary care registry (RCT#1) who are active/inactive portal 
users and affiliated with the primary care practices of interest for this research, and will be randomized into 
either the control, 1, 2, or 3 reminder arms (for RCT #1). 
 
Many patients are members of the same family. As such, the research team will assemble family units (using 
household address, primary telephone number, patient ID of the guarantor, and insurance member ID as the 
variables of commonality to create these family units). For RCT 1, for each family, 1 individual will be randomly 
selected as the index patient. Only index patients who are also active portal users and affiliated with a primary 
care practice (criteria explained below) will have their data analyzed as part of the primary analysis plan. 
Secondary analyses will involve all study subjects. 
 
Gender and age: We anticipate the gender distribution to be roughly equal. All ages of patients will be included 
in the study; however, patient will not receive a message until they are 6 months of age, as the flu vaccine is 
not recommended for those less than 6 months of age. 
 
Racial and ethnic origin: There are no enrollment restrictions based on race/ethnicity. 
Inclusion criteria 
 
 
A patient at one of the clinics of interest within the UCLA Health System (up to 55 practices). An individual is 
deemed a primary care patient of the UCLA Health System through the following algorithm: 
 
Assigned managed care patients (UCLAMG) + Attributed patients from other payers/ACOs 
􀀀 ≥2 PCP visits in the past 3 years; or 

3 
 
􀀀 ≥1 PCP visit with preventive service code in past 1 year (99381-99397 or G0438/G0439) 
􀀀 All visits cannot be urgent care visits (ie excludes visits after hours or on weekends, not by urgent 
care codes since UCLA does not bill accordingly) 
 
Active patient: We decided to use the algorithm outlined above and currently approved and in place by the 
UCLA Health System as the research team believes it to be a generalizable model that could be applied to 
other health systems. 
 
Exclusion criteria 
Patients will be excluded from the overall study if they are not part of UCLA's primary care registry per the 
above algorithm detailed in the inclusion criteria. 
 
Creating family units 
An overall address field will be constructed from the data pull including the address, city, state, and zip code 
fields. Primary telephone number, patient ID of the guarantor and insurance member ID will be used as other 
variables in the process to create the family units. The following steps taken to create these family units is 
described below: 
1. Start with a single entry in the contact data pull 
2. Add any other entries with the same patient ID as any existing entry to the family 
3. Add any other entries with the same address as any existing entry to the family 
4. Add any other entries with the same primary phone as any existing entry to the family 
5. Add any other entries with a patient ID matching the patient ID of guarantor as any existing entry to 
the family 
6. Add any other entries with the same insurance member ID as any existing entry to the family 
7. Repeat steps 2-6 until no new matches are found - this forms a single family 
8. Repeat steps 1-7 on the remaining entries to build each additional family until all entries have been 
associated with a family. 
Please note - when there were errors with the address and telephone number, these variables were not used 
to group families (ex. instances with 20+ entries with the same data for telephone number). 
 
Definition of active portal user: 
We can only send portal R/R messages to patients who have signed up for the portal. Since some patients 
sign up for the portal but never use it, we define an active portal user as a patient (or proxy on behalf of the 
patient) who has used the portal within 12m (~59% of UCLA patients), and who has logged in 1 or more times 
in the past 12 months (reference date for login activity will be selected by the research team and then working 
backwards by 365 days, example of a range 8/1/17 - 7/31/18 for RCT #1 excluding their initial activation login 
and any subsequent logins on the same date of account activation. 
 
Selection of index patient from each family unit 
1. If the family contains any active portal user per our definition above, the index patient is randomly selected 
from among these patients. 
2. If the family contains no active portal users, the index patient is randomly selected from all patients in the 
family. 
 
 
Selection of the index patient for primary analysis: 
1. An active portal user per the definition above 
2. The patient is affiliated with one of the primary care clinics of interest 
3. The patient was randomly selected as the index patient in their family unit 
 
We will assess primary intervention effects among eligible index patients, and secondary analyses for the 
entire primary care registry since the impact of the intervention reflects the extent of portal penetration and high 

4 
 
correlation that exists among members of the same family (n=385,000 patients in RCT #1; up to 480,000 
patients for RCT #2,(ISS-generated data extraction: of 430,000; IP-generated data extraction of approximately 
50,000 (MRNS only) 
. 
 
METHODS AND STUDY PROCEDURES 
The UCLA Health System is made up in part of approx 55 primary care practices. The leadership team at 
UCLA Health has agreed to allow these practices to participate in the 4 RCTs outlined below. 
Among the primary care practices (up to n= 55), patients who meet the inclusion criteria (outlined above and in 
section 11.1 item 4.0) will be randomized into the study arms. Below is a detailed outline of each of the four 
RCTs and the interventional components. 
 
RCT - conducted 2018-19 
We propose a randomized trial, where patients are randomized within practices to 0, 1, 2 or 3 R/R messages 
sent via the portal on a 1:1:1:1 basis. Our prior studies showed the effects of influenza R/R can vary by patient 
and practice factors (i.e. age, race, practice type). We will select 1 adult or 1 child per household to be the 
index patient for randomization since family members can affect each other’s receipt of vaccine. 
Inclusion criteria: Outlined above. 
 
Eligibility for R/R: Our EHR will identify subjects who are eligible for either 1 or 2 influenza vaccinations based 
on ACIP algorithms (2 are recommended for children <9 years who have not had a prior flu vaccination). 
Spacing of R/R messages: Based on our prior studies, we plan to send up to 3 R/R messages, spaced every 
3-4 weeks, beginning in October; Aim 1 findings may modify plans. 
 
Languages: R/R messages will be in English. Portal messages in Spanish will not be available during RCT 1 
due to technological capabilities. 
 
Health Literacy: We will use plain language <8th grade reading level per Flesch-Kincaid analysis. 
In Aim 1 we will adapt messages with input from key stakeholders (see IRB submissions titled: Patient Portal - 
Patient Qualitative Interviews and Patient- Portal Provider Qualitative Interview for full outline of protocol). Our 
message content will be rooted in decades of work by vaccine communication expertise and rooted in the 
Health Beliefs Model (see file named: Health Belief Model for multiple examples of the content that will be 
adapted for the message). The content of the R/R message will be the same for each reminder that is sent. 
Please see item 1.0 for the language to be used in the messages. Please note, the file titled RCT #1  
Reminders is the language used in the first two reminders (sent in October and November 2018), and the file 
titled RCT #1 Reminders_December is the language for the third and final reminder to be sent in December 
2018 for RCT #1. 
 
Website link: the research team is working with UCLA marketing department to develop a new flu educational 
webpage. This webpage is currently under-development, but we will embed short health educational videos 
(examples below) on the UCLA website specific for this project: 
https://www.youtube.com/watch?v=EstDvA-mr5A 
https://www.youtube.com/watch?v=QTjchoH1KYM 
 
The purpose of this website is to determine what subject areas of vaccine education (i.e. safety, effectiveness, 
general info) that patients are most interested in. All content will be approved by UCLA's marketing dept. 
 
R/R Message Content: The content of each R/R message will be the same for each study arm. As discussed 
above we may modify the R/R message content slightly based the results of the prior RCTs and the qualitative 
interviews to follow. The number of R/R’s sent will be based upon the optimal number from Aim 1 and will be 
identical for both study arms. 
 
Please see item 1.0 for examples of the messages to be used (title: RCT #3 Reminders). Example message 
for Arm 2 same as that used in RCT #1. 

5 
 
 
Toolkit Development (Aim 3) 
Toolkit and Adoption Guide: In the final year we will develop an online toolkit and adoption guide for portal-
based R/R, with a manual of operations for algorithms, messages, data management protocols and costs for 
start-up and maintenance. Guides will contain FAQs and common scenarios, references, websites, and other 
materials for health systems to personalize portal R/R. We will use diffusion of innovation methods guided by 
our National Advisory Board. The toolkit and guide will be adaptable for pandemic flu vaccination R/R. 
Please note, should the language included in any of the messages outlines for RCT 1 - RCT 4 change, and 
amendment will be submitted. 
 
MEASURES 
After each RCT, qualitative provider and patient interviews will be conducted. Those procedures and measures 
are outlined in IRB submissions titled: Patient Portal - Patient Qualitative Interviews and Patient- Portal  
 
Provider Qualitative Interview. 
Independent Variables: The key variable is study group (portal R/R vs control). 
Treatment Variables: Exposure to: (a) any portal R/R messages or (b) 1, 2 or 3 R/R’s; see Analysis for RCT 1. 
Covariates: Covariates considered for analyses (found in studies to affect flu vaccine rates) include: Practice 
variables will include type- (ex. pediatric, internal medicine, geriatric, or Med-Peds, and family medicine). 
Patient variables will include - age (6m-17y, 18-64y, 65+y), race/ethnicity, gender, primary language (by EHR), 
degree of portal use, and receipt of influenza vaccination in prior years. We recognize few UCLA patients have 
Medicaid; we will focus on low-income groups in the dissemination phase (Aim 3). 
 
Dependent Measures 
Vaccine Outcome: The primary outcome is influenza vaccination during the vaccination season as measured 
by analysis of EHR data. For children <9y who had not received prior influenza vaccination, the outcome will 
be at least 1 influenza vaccination. 
 
Process Outcomes: These will include: (a) total # visits to the practice during the study time period, (b) # flu 
vaccine or nurse visits, and (c) missed opportunities. Visits will be measured by the EHR; using ICD-10/CPT 
codes to classify visits to primary care as preventive, acute/chronic, or nurse-vaccination. Missed opportunities 
are defined as # vaccine-eligible visits during which the patient did not receive an influenza vaccination. These 
process metrics will help assess how the intervention worked. 
 
Additional Measures: see file: Summary of outcome measures RCT 1 which shows additional measures, 
grounded in the RE-AIM framework. Found in item 1.0 of this section. 
 
Costs: Since the intervention is implemented centrally we assume no added practice costs. We also assume 
practice costs/vaccination from portal R/R is identical to standard practice costs/vaccination. We will assess 
the time and costs of study and implementation personnel and non-personnel costs, distinguishing planning 
costs from intervention costs. We will measure costs using a standardized time study/resource survey sent 
weekly to all individuals working on the study that delineates (a) the # hours spent for each individual, and (b) 
research vs implementation time. We will use national salary estimates by work code from the Bureau of Labor 
Statistics to value personnel time in standard rates. We will measure non-personnel costs EHR hardware, 
software, materials. 
 
 
Cost and Cost Effectiveness 
Costs: We will assume that the actual per-dose vaccination costs (administration costs, vaccine costs, storage, 
etc.) are identical for intervention and comparison patients and equal to the average national reimbursement 
for an influenza vaccine dose. The difference in total costs between the study arms will be influenced by the 
total costs of implementing the intervention, the difference in vaccination rates within study arms, and the 
subsequent health care utilization by the population (which we can model from prior studies) 
 

6 
 
Effectiveness: For each RCT we will estimate effectiveness using model-based, standardized expected values 
(mean) at the end of each RCT. We will calculate incremental cost-effectiveness ratios (ICERs) for each 
intervention arm. 
 
For example, to compare costs in RCT #1 for Arm 3 R/R’s vs Arm 2 R/R’s, the ICER is: ICER (Arms 3 R/R vs 2 
R/R) = (costArm 3 R/R – costArm 2 R/R) / (Flu vaccine rateArm 3 R/R – Flu vaccine rateArm 2 R/R) 
In the numerator of the above equation, the costArm X is equal to [(# Flu vaccine doses given per study arm x 
average national vaccine reimbursement per dose) + average costs per patient in that Arm]. The denominator 
of the equation is the difference in model-based, standardized expected values at the end of the intervention. 
 
 
STATISTICS AND ANALYTIC PLAN 
Analytic Plan:  
The primary outcome is receipt of influenza vaccine among eligible index patients comparing the 
effectiveness of 1, 2, or 3 MyChart flu reminders to zero MyChart flu reminders. 
Secondary measures are receipt of influenza vaccine among all identified patients in the primary care registry 
(per the algorithm outlined above in the inclusion criteria), process metrics, # subsequent preventive visits, and 
costs. 
 
Power Analysis:  
Power was evaluated using a simulation study. We assumed a within-practice randomization 
plan, and expect at minimum the sample sizes reported in Table 3 (n=145,684). We further assumed, based 
on preliminary UCLA Health data, that influenza vaccination rates are 42% among portal users without receipt 
of any portal-based R/R, and 45%, 47% and 49% among users in the 1, 2, and 3 R/R arms. We estimate 1,248 
portal user families per practice (we will select 1 subject per family), based on our estimated mean family size 
of 2.38 portal users per family, and up to 55 practices. 
 
Adjusting for clustering of patients in practices, and assuming an intraclass correlation (ICC) of 1% (consistent 
with previous work), this sample size provides >90% power to detect an overall difference of 5 percentage 
points between portal-based R/R (combining 1, 2, + 3 R/R arms) and 0 R/R arm for all users and subgroups of 
interest (age and race/ethnicity). The power for the subgroups corresponds to evaluating the treatment effect 
within each subgroup. These power analyses represent conservative estimates based on the available patient 
sample—we expect larger effect sizes based upon our prior R/R studies. An estimated number of index 
subjects is 1,248 x 55 = 68,640. 
 
Statistical Analysis- Primary Outcome 
Hyp. 2a.1: Patients receiving 1, 2, or 3 portal R/R will have higher influenza vaccination rates than 0 R/R. 
Primary outcomes (patient receipt of flu vaccine) are binary; our main explanatory variable will be an indicator 
for the receipt of any portal-based R/R.  
 
We will employ intent-to-treat analyses using mixed effects logistic regression models with practice random 
effects, an approach recommended for RCTs in which the goal is to estimate the causal effects of interventions 
on individuals, adjusted for patient correlation within practice. In addition to the treatment variable, the 
covariates in the model will include patient age (6m-17y, 18-64y, 65+y), race/ethnicity, gender, primary 
language, degree of portal use, flu vaccination in prior year, and practice type (internal medicine, family 
medicine, pediatric). This method performs well in situations where the number of observations per cluster is 
large and for unequal cluster sizes. Hypothesis tests will be two-sided and a p value < 0.05 will be considered 
statistically significant. We will use SAS v9.4 (SAS Institute In c., Cary, NC). 
 
Hyp 2a.2: More R/R messages will incrementally increase vaccine rates (3 R/R > 2 R/R > 1 R/R > 0 R/R): As 
above, we will use mixed effects logistic regression models.  
The major difference for these analyses is that we will test for pairwise differences between consecutive study 
arms. With linear contrasts we can determine if each additional R/R increases vaccination rates. Bonferroni 
corrections will be applied to account for multiple comparisons. The expected sample size provides >90% 

7 
 
power to conservatively detect an OR of 1.13 between consecutive study arms (corresponding to a 3% 
increase in vaccination rates from the baseline rate of 42%). 
 
Statistical Analysis- Subgroup analysis  
We will test Hyp. 2a.1 and 2a.2 stratifying by age (6m-17y, 18-64y, 65+y) and race/ethnicity 
(Hispanic, NH Black, NH Asian, NH White). We will apply Bonferroni corrections to account for multiple 
comparisons. 
 
Statistical Analysis- Process measures  
Because other process measures (# visits, # vaccination or nurse visits, and # missed opportunities) are count 
outcomes, we will use negative binomial mixed effects models to compare incidence rates between arms, 
controlling for the patient and practice covariates mentioned above.