Improving Influenza Vaccination Delivery Across a Health System by the Electronic Health Records Patient Portal- RCT #2
Phase: NA
Conditions: Influenza, Respiratory Tract Infections
Sponsor: University of California, Los Angeles

1 
 
Clinicaltrials.gov application number:  NCT04110314 
Title: Improving Influenza Vaccination Delivery Across a Health System by the Electronic Health Records 
         Patient Portal  
IRB ID17-001889, Amendment 10 for RCT2 
Submission Date: 12/14/2017 (RCTs 1-5) IRB Approval Date: 12/20/2017   
Amendment for RCT2: Approved 10/9/2019 
Informed Consent form N/A (approved Waiver of Consent) 
 
RCT2 Protocol (includes statistical analysis plan) 
 
SPECIFIC AIMS 
The overall purpose of the RCTs is to evaluate the impact that R/R, delivered through the patient portal, could 
have on the flu immunization rates of UCLA Health patients. The intent is to ultimately take the information 
learned and create and disseminate an adaptable toolkit to other health systems. 
 
For RCT #2, we will evaluate the impact of positively framed messages versus negatively-framed messages versus control 
(standard of care) to see if either type of message has an impact on the receipt of influenza vaccination. The current 
literature is mixed as to which message type is more effective. The RCT #2 study design will enable us to evaluate the 
impact in different sub-populations which may demonstrate variability in preference. In addition, RCT #2 will allow us to 
test the impact of pre-commitment. In the field of behavioral economics, pre-commitment has shown promise as a 
means of behavior change- in this case, receipt of influenza vaccination. The study design will enable us to evaluate the 
impact of a pre-commitment question. 
 
Specific Aims for RCT 2:  
 
Aim 1 RCT #2: Using a 3-arm RCT (up to 55 practices), compare the effectiveness of flu reminder recall messages that 
have been tailored for one of 4 subpopulations- 1) children <18 years, 2) young adults (18-<65 years), 3) older adults 
(65+ years) and 4) patients with diabetes (identified by SUPREME criteria by CTSI). Within each subpopulation, one arm 
will receive positively-framed messages highlighting the benefits of flu vaccination, one arm will receive negatively 
framed messages highlighting the risks associated with not getting the flu vaccine and one arm will serve as the control 
arm (no portal reminders). 
 
Aim 2 RCT #2: In addition, we will test the impact of a "pre-commitment" question asking whether the patient plans to 
receive the flu vaccine (prior to the reminders going out). This question will be asked of half of patients per arm of the 3-
arm RCT. Previous research has indicated pre-commitment can improve the likelihood a patient follows through with a 
behavior. We intend to determine whether pre-commitment increases the likelihood of getting the flu vaccine. 
 
In summary, families will be randomized using a 3x2 factorial allocation (no reminder v. positively-framed reminder v. 
negatively-framed reminder, and no pre-commitment question v. pre-commitment question). Family members will be 
divided into 4 non-overlapping cohorts (age <18 years, 18-64 years non-diabetic, 65+ years non-diabetic, and 18+ years 
diabetic), and a single index member per family per cohort will be selected for inclusion in the study. Only active users 
affiliated with a UCLA primary care practice will be included in the study sample. For single individuals not part of a 
family, that individual will serve as the index patient. 
 
Hypotheses: 
 
Hyp 1 RCT #2: Vaccination rates will differ by receipt of no reminders, positively framed reminders and negatively 
framed reminders. 
1.1 Vaccination rates will be higher among patients receiving positively framed reminders than among patients receiving 
no reminders. 
1.2 Vaccination rates will be higher among patients receiving negatively framed reminders than among patients 

2 
 
receiving no reminders. 
1.3. Vaccination rates will be higher among patients receiving positively framed reminders than among patients 
receiving negatively framed reminders. 
Hyp 2 RCT #2: Vaccination rates will be higher among patients receiving a pre-commitment question than among 
patients not receiving the pre-commitment question. 
2.1 The arm receiving the pre-commitment question, asking about their intention to get the flu vaccine (prior to the 
reminders going out), are more likely to follow through with influenza vaccination; therefore, the pre-commitment arm 
will have higher rates of flu vaccination than the control arm for the pre-commitment component. 
 
The RCT design, subpopulations and letter components have been approved by the MyChart subcommittee, Ambulatory 
Operations Advisory Group, and the UCLA Health Primary Care Committee. 
 
METHODS AND STUDY PROCEDURES 
The UCLA Health System is made up in part of approximately 55 primary care practices. The leadership team at UCLA 
Health has agreed to allow these practices to participate in the RCT. 
Among the primary care practices (up to n= 55), patients who meet the inclusion criteria will be randomized into the 
study arms (no reminder v. positively-framed reminder v. negatively-framed reminder, and no pre-commitment 
question v. pre-commitment question). 
 
Eligibility for R/R: Our EHR will identify subjects who are eligible for either 1 or 2 influenza vaccinations based on ACIP 
algorithms (2 are recommended for children <9 years who have not had a prior flu vaccination). 
Spacing of R/R messages: Based on our prior studies, we plan to send up to 4 R/R messages, spaced every 3-4 weeks, 
beginning in October. 
 
Languages: R/R messages will be in English. Portal messages in Spanish will not be available during RCT 2 due to 
technological capabilities. 
 
Health Literacy: We will use plain language <7th grade reading level per Flesch-Kincaid analysis. 
 
R/R Message Content: The content of each R/R message are slightly modified to be tailored to each age group/condition 
(diabetes). Please see item 10.0 of the IRB system for examples of the messages to be used (title: RCT #2 Reminders).  
 
OVERALL STUDY DESIGN 
Subjects receiving the intervention will be patients at one of the primary care practices within UCLA Health’s 
System (up to 55 practices) and will include patients of any age greater than 6 months of age at the time of messages 
being sent. A proportion of the patients from these practices will be selected to participate - i.e., those who meet the 
inclusion criteria. Randomization techniques will occur before each RCT. 
 
For RCT2, we are sending enhanced reminder/recall (R/R) messages. Specifically, UCLA portal users will be divided into 4 
groups that will get tailored messages. These groups are: 1) parents of children <18 years of age, 2) adults ages 18-<65 
years of age, 3) adults ages 65 or older and 4) individuals meeting the SUPREME diabetes criteria.  Participants will be 
randomized to receiving positively-framed messages, negatively-framed messages or no portal messages regarding 
influenza vaccination.  Participants will also be randomized to either receiving a pre-commitment prompt or no pre-
commitment prompt. 
 

3 
 
 
 
Subject Population 
There are approximately n = 430,000 patients in the primary care registry (RCT#2) who are active/inactive portal users 
and affiliated with the primary care practices of interest for this research.  And additional 50,000 meet the SUPREME 
criteria for diabetes.   
 
Many patients are members of the same family. As such, the research team will assemble family units (using 
household address, primary telephone number, patient ID of the guarantor, and insurance member ID as the 
variables of commonality to create these family units). For RCT 2, for each family, 1 individual will be randomly 
selected as the index patient for each age or diabetic subgroup. Only index patients who are also active portal users and 
affiliated with a primary care practice (criteria explained below) will have their data analyzed as part of the primary 
analysis plan. 
 
Secondary analyses will involve all study subjects. 
 
Gender and age: We anticipate the gender distribution to be roughly equal. All ages of patients will be included in the 
study; however, patient will not receive a message until they are 6 months of age, as the flu vaccine is not 
recommended for those less than 6 months of age. 
 
Racial and ethnic origin: There are no enrollment restrictions based on race/ethnicity. 
 
Inclusion criteria 
 
A patient at one of the clinics of interest within the UCLA Health System (up to 55 practices). An individual is 
deemed a primary care patient of the UCLA Health System through the following algorithm: 
 
Assigned managed care patients (UCLAMG) + Attributed patients from other payers/ACOs 
 ≥2 PCP visits in the past 3 years; or 
 ≥1 PCP visit with preventive service code in past 1 year (99381-99397 or G0438/G0439) 
 All visits cannot be urgent care visits (i.e., excludes visits after hours or on weekends, not by urgent care codes 
since UCLA does not bill accordingly) 
 
Active patient: We decided to use the algorithm outlined above and currently approved and in place by the UCLA Health 
System as the research team believes it to be a generalizable model that could be applied to other health systems. 
 
Creating family units 
An overall address field will be constructed from the data pull including the address, city, state, and zip code 
fields. Primary telephone number, patient ID of the guarantor and insurance member ID will be used as other 
variables in the process to create the family units. The following steps taken to create these family units is described 
below: 

4 
 
1. Start with a single entry in the contact data pull 
2. Add any other entries with the same patient ID as any existing entry to the family 
3. Add any other entries with the same address as any existing entry to the family 
4. Add any other entries with the same primary phone as any existing entry to the family 
5. Add any other entries with a patient ID matching the patient ID of guarantor as any existing entry to the 
family 
6. Add any other entries with the same insurance member ID as any existing entry to the family 
7. Repeat steps 2-6 until no new matches are found - this forms a single family 
8. Repeat steps 1-7 on the remaining entries to build each additional family until all entries have been 
associated with a family. 
Please note - when there were errors with the address and telephone number, these variables were not used to group 
families (ex. instances with 20+ entries with the same data for telephone number). 
 
Definition of active portal user: 
We can only send portal R/R messages to patients who have signed up for the portal. Since some patients sign up for the 
portal but never use it, we define an active portal user as a patient (or proxy on behalf of the patient) who has used the 
portal within 12 months and who has logged in 1 or more times in the past 12 months (reference date for login activity 
will be selected by the research team and then working backwards by 365 days, excluding their initial activation login 
and any subsequent logins on the same date of account activation. 
 
Selection of index patient from each family unit 
For RCT2- One individual will be randomly selected as the index patient per family per stratum (e.g., if a single family 
contains eligible members in both the diabetic adult stratum and the nondiabetic younger adult stratum, one index 
patient will be selected for the former and another for the latter.  
 
Selection of the index patient for primary analysis: 
1. An active portal user per the definition above 
2. The patient is affiliated with one of the primary care clinics of interest 
3. The patient was randomly selected as the index patient in their family unit for one of the age group/diabetic groups 
 
We will assess primary intervention effects among eligible index patients, and secondary analyses for the entire primary 
care registry since the impact of the intervention reflects the extent of portal penetration and high 
correlation that exists among members of the same family (up to 480,000 patients for RCT #2, (ISS-generated data 
extraction: of 430,000; IP-generated data extraction of approximately 50,000 patients meeting the SUPREME criteria for 
diabetes (MRNS only shared with the study team by the IP group). 
 
Exclusion criteria 
Patients will be excluded from the overall study if they are not part of UCLA's primary care registry per the above 
algorithm detailed in the inclusion criteria. 
 
MEASURES 
 
Dependent Measures 
Vaccine Outcome: The primary outcome is influenza vaccination during the vaccination season as measured by analysis 
of EHR data. For children <9y who had not received prior influenza vaccination, the outcome will be at least 1 influenza 
vaccination. 
 
Process Outcomes: These will include: (a) total # visits to the practice during the study time period, (b) # flu vaccine or 
nurse visits, and (c) missed opportunities. Visits will be measured by the EHR, using ICD-10/CPT 
codes to classify visits to primary care as preventive, acute/chronic, or nurse-vaccination. Missed opportunities 
are defined as # vaccine-eligible visits during which the patient did not receive an influenza vaccination. These 

5 
 
process metrics will help assess how the intervention worked. 
 
Cost and Cost Effectiveness 
Costs: Since the intervention is implemented centrally, we assume no added practice costs. We also assume 
practice costs/vaccination from portal R/R is identical to standard practice costs/vaccination. We will assess the time 
and costs of study and implementation personnel and non-personnel costs, distinguishing planning costs from 
intervention costs. We will measure costs using a standardized time study/resource survey sent weekly to all individuals 
working on the study that delineates (a) the # hours spent for each individual, and (b) research vs implementation time. 
We will use national salary estimates by work code from the Bureau of Labor Statistics to value personnel time in 
standard rates. We will measure non-personnel costs EHR hardware, software, materials. 
 
We will assume that the actual per-dose vaccination costs (administration costs, vaccine costs, storage, 
etc.) are identical for intervention and comparison patients and equal to the average national reimbursement for an 
influenza vaccine dose. The difference in total costs between the study arms will be influenced by the total costs of 
implementing the intervention, the difference in vaccination rates within study arms, and the subsequent health care 
utilization by the population (which we can model from prior studies) 
 
Effectiveness: For each RCT we will estimate effectiveness using model-based, standardized expected values 
(mean) at the end of each RCT. We will calculate incremental cost-effectiveness ratios (ICERs) for each 
intervention arm. 
 
CONSENT PROCESS AND HIPAA AUTHORIZATION 
We are seeking a Waiver of Informed Consent for the study.  The study poses no more than minimal risk.  The research 
could not practicably be carried out without a waiver of consent because: 
It would not be possible to contact all of the participants associated with data or specimens to obtain consent. 
The size of the potential study population is so large that it would not be feasible to obtain consent. 
Requiring informed consent may introduce systematic bias into the data. 
 
Requiring consent would introduce systematic bias into each of the RCTs.  If patients know these studies will occur, it 
could influence their behavior in getting the flu vaccine prior to actual initiation of the study. Self-selection prior to the 
study of who would opt in or out would also bias the results as those who opt in may be more health conscious. This 
would in turn bias the results and the research team would be limited in the types of generalizable conclusions to be 
made. 
 
For children, we request to waive both Parental Permission and Assent. 
 
We are requesting a waiver of consent for the entire study so while there are no consent forms, the reminder recall 
messages will also be made available in Spanish when the portal technology has finished programming that capability. 
 
As of now, no translated versions of the reminders are available as the programming of this translation functionality is 
still occurring on the technical side of the patient portal but will be uploaded for approval prior to use. No Spanish 
versions will be available as the current technology available with the portal only allows for single language - English. 
 
HIPAA Authorization: 
We are seeking a total Waiver of HIPAA Authorization for the entire study.  We assure the PHI collected for this study 
from the UC records will not be reused or disclosed except as indicated in this application. 
   
The investigators agree to the following: 
- 
The protected health information requested is the minimum necessary to meet the research objectives 
- 
The protected health information that is obtained as part of this study will not be used or disclosed to any other 
person other than study personnel or to the parties listed in item Section 17.1/item 2, except as required by law. 

6 
 
- 
Study Sponsors will not be provided with personal identifying information (including PHI) to take from the study 
site at any time, including the end of the study. 
- 
Data and specimens shared with outside entities, such as study sponsors, will be coded or de-identified. 
The Waiver is necessary and the research could not be practicably conducted without access to and use of the protected 
health information because the PHI is needed to identify potential participants’ diabetic status (prior to randomization) 
and influenza vaccination status throughout the study.  It would also not be feasible to individually contact the large 
number of potential subjects in the study.  
 
PRIVACY AND CONFIDENTIALITY 
This proposal for multiple RCTs involves reminding patients within the UCLA health system, via their patient portal, that 
they are due for an annual flu vaccine. Since the messages will all be delivered via the portal, patients must have their 
own unique ID and passwords to sign in, helping to maintain their privacy. Also, patients will sign on at a time and place 
convenient to them, ensuring that they can use a private (usually home) setting. 
 
To maintain privacy we will carry out the following measures: 
RCT 2: Patients in the intervention arms of the RCT will receive up to 4 reminders via the portal, notifying them of a new 
letter from their provider in the portal system. The letter will include the first name of the patient and the patient's 
provider name will be added below the signature line. The letter is only accessible to patients once they or their patient 
proxy (in the case of a patient proxy, typically for young children or elderly patients) log into their UCLA patient portal 
(password protected). The patient population that received a generic message in RCT #1 is being subdivided into 4 
groups (children <18 years, young adults (18-<65 years), older adults (65+ years) and patients with diabetes meeting the 
SUPREME criteria. A message will only be sent after a child turns 6 months of age (the recommended minimum age for 
influenza vaccination). 
 
Confidentiality will be maintained with the following procedures: 
1) any data transfer will occur between entities at UCLA and will involve encrypted mechanisms 
2) all electronic data will be stored in locked and password protected computers, in locked rooms. Only the statistician 
will have access to identifiable data. 
3) All UCLA personnel will complete required CITI and HIPAA courses and training, and all personnel from other 
institutions will also be required to complete analogous courses and training. Verification of the completion of these 
trainings must be provided to UCLA personnel upon request. 
 
RISKS TO SUBJECTS 
The major risk is one of breach of privacy and confidentiality. We will maintain the strictest of procedures to prevent 
either of these problems as described above (i.e., staff training, maintaining strict confidentiality of data). 
 
There is always a risk of complications with vaccination, but this study does not vaccinate patients per se -- rather, it 
reminds patients (or parents of eligible minors) to receive a recommended vaccine and patients should discuss the 
benefits and risk of influenza and vaccination with their healthcare provider. 
 
The study does not represent more than minimal risk. The objective is to encourage patients to schedule a visit to see 
their primary health care provider and to make an appointment to receive a recommended vaccine. The vaccine is 
recommended by the Advisory Committee of Immunization Practices (ACIP) for all individuals over the age of 6 months 
and older on an annual basis. Therefore, indicating a patient is due for an influenza vaccination does not represent 
sensitive information. In addition, the message will only be delivered and accessible via the patient portal which is 
username and password protected. The greatest risk is one of loss of confidentiality. 
 
BENEFITS TO SUBJECTS 
The benefit to the subject is reminders to receive a recommended vaccine that offers protection from influenza, which 
can be very serious and can require hospitalization or can lead to death in some cases. 

7 
 
The benefit to society would be a better understanding of effective R/R messages and modalities that could be scaled up 
nationally and disseminated to other health systems to increase influenza vaccination (improve individual and herd 
immunity). This is particularly important for the influenza vaccine as in today's current society many have decided to 
forgo essential and recommended vaccines for themselves and their children, thereby increasing their personal risk. 
 
COSTS FOR PARTICIPATION 
Subjects will not incur any financial obligations from participation in the study. 
 
PAYMENT FOR PARTICIPATION 
No payments will be received by subjects. 
 
STATISTICS AND ANALYTIC PLAN 
Primary Analysis 
We hypothesize that patients receiving gain-framed or loss-framed portal R/R messages will have higher influenza 
vaccination rates than no R/R, and we hypothesize that patients receiving pre-commitment prompts will have higher 
influenza vaccination rates than no prompts. Primary outcomes (patient receipt of flu vaccine) are binary; our main 
explanatory variable will be an indicator for the receipt of any portal-based R/R or prompt.  
 
The analysis will be based on a 3x2 factorial design, with evaluation of a reminder framing intervention (gain-frame v. 
loss-frame v. no reminder), and a pre-commitment prompt intervention (pre-commitment v. no pre-commitment). 
Analyses will be stratified into 4 patient cohorts: <18y, 18-64 non-diabetic, 65+ non-diabetic, and 18+ diabetic. The 
pediatric age group (<18y) will be analyzed separately. 
 
We will employ intent-to-treat analyses using mixed effects log-binomial regression models with practice random 
effects, an approach recommended for RCTs in which the goal is to estimate the causal effects of interventions on 
individuals, adjusted for clustering of patients by practice. The primary model terms will be reminder arm (gain-frame v. 
loss-frame v. none), and pre-commitment arm (prompt v. no prompt). Models will adjust for the following patient 
characteristics: age, gender, race/ethnicity, primary language, primary payor, and prior receipt of influenza vaccines in 
the last two years. Evaluation of study hypotheses will be performed using model contrasts, and treatment effects will 
be reported in terms of risk ratios and 95% CIs. In the event of computational issues, we will replace the log-binomial 
specification with a log-Poisson specification, and use model-robust standard errors for inference on treatment effect 
risk ratios. 
 
Additional Analysis 
Additional analyses will include evaluation of process measures (e.g., missed opportunities), evaluation of effect 
modification by patient characteristics, comparison of the gain-frame and loss-frame arms, and evaluation of vaccination 
receipt including self-reported vaccinations. Process measures will be analyzed similarly to the primary outcome but 
using different distributions and link functions as appropriate (e.g., negative binomial distributions for number of missed 
opportunities). Effect heterogeneity will be evaluated by introducing interaction terms into the primary model 
specification. All secondary analyses will use a significance level of 0.05. All analyses will be performed using SAS v. 9.4 
(SAS Institute Inc., Cary, NC). 
 
Process measures: We assessed the percentage of patients who opened the portal reminder letter, as well as the source 
of influenza vaccination data (health system within UCLA practices, external source via data transfer, patient/proxy 
update through normal portal processes, or patient/proxy update in response to portal reminders.