Precision Vaccine Promotion in Underserved Populations
Phase: NA
Conditions: Influenza, Vaccine Hesitancy
Sponsor: University of Southern California

1 
 
 
 
 
STUDY PROTOCOL 
Project Title: Precision Vaccine Promotion in Underserved Populations 
PI: Daniella Meeker, PhD 
 
  
Supported by: 
  
National Institutes of Health 
USC Roybal Center for Behavioral Interventions in Aging 
Grant # NIA P30AG024968  
August 3, 2022 
 
 
  
  
 
 
 
 
 

2 
 
 
TABLE OF CONTENTS  
STUDY PROTOCOL 
1 
STUDY TEAM ROSTER 
5 
PARTICIPATING STUDY SITES (Los Angeles County Department of Health Services) 
6 
Study Title 
7 
1. 
STUDY OBJECTIVES 
8 
1.1 
Primary Objectives 
8 
2. 
BACKGROUND AND RATIONALE 
8 
2.1 
Background on Condition, Disease, or Other Primary Study Focus 
8 
3. 
STUDY DESIGN 
9 
4. 
SELECTION AND ENROLLMENT OF PARTICIPANTS 
9 
4.1 
Inclusion Criteria 
9 
4.2 
Exclusion Criteria 
10 
4.3 
Study Enrollment Procedures 
10 
5. 
STUDY INTERVENTIONS 
10 
5.1 
Interventions, Administration, and Duration 
10 
5.2 
Handling of Study Interventions 
12 
5.3 
Adherence Assessment 
12 
6. 
STUDY PROCEDURES 
12 
6.1 
Schedule of Evaluations 
12 
6.2 
Description of Evaluations 
13 
6.2.1 
Screening Evaluation 
13 
6.2.2 
Enrollment, Baseline, and/or Randomization 
13 
7. 
SAFETY ASSESSMENTS 
15 

3 
 
 
7.1 
Specification of Safety Parameters 
15 
7.2 
Methods and Timing for Assessing, Recording, and Analyzing Safety Parameters 
15 
7.3 
Adverse Events 
16 
7.4 
Reporting Procedures 
16 
7.5 
Safety Monitoring 
16 
8. 
INTERVENTION DISCONTINUATION 
16 
9. 
STATISTICAL CONSIDERATIONS 
17 
9.1 
General Design Issues 
17 
9.2 
Sample Size 
17 
9.3 
Interim analyses and Stopping Rules 
17 
9.4 
Outcomes 
17 
9.4.1 
Primary outcome 
17 
9.4.2 
Secondary outcomes 
17 
9.5 
Data Analyses 
18 
10. 
DATA COLLECTION AND QUALITY ASSURANCE 
18 
10.1 
Data Collection Forms 
18 
10.2 
Data Management 
18 
10.3 
Quality Assurance 
19 
Training 
19 
Quality Control Committee 
19 
Intervention fidelity 
19 
Protocol Deviations 
19 
Monitoring 
19 
11. 
PARTICIPANT RIGHTS AND CONFIDENTIALITY 
20 
11.1 
Institutional Review Board (IRB) Review 
20 

4 
 
 
11.2 
Informed Consent Forms 
20 
11.3 
Participant Confidentiality 
20 
11.4 
Study Discontinuation 
20 
12. 
COMMITTEES 
21 
13. 
PUBLICATION OF RESEARCH FINDINGS 
21 
14. 
REFERENCES 
21 
15. 
SUPPLEMENTS/APPENDICES 
21 
 
 
 

5 
 
 
STUDY TEAM ROSTER  
  
Daniella Meeker, Ph.D (Principal Investigator), University of Southern California, 2250 Alcazar Street Health Sciences Campus 
Los Angeles 90033, dmeeker@usc.edu  
 
Peter Szilagyi, MD, MPH (Co- Investigator), UCLA Pediatrics, BOX 951752, 12-353 MDCC, Los Angeles, CA 90095-1752 
PSzilagyi@mednet.ucla.edu 
 
Anshu Abhat, MD, MPH (Co-Investigator), Harbor-UCLA Medical Center, 3330 Lomita Blvd, Torrance, CA 90505, 
aabhat@dhs.lacounty.gov 
  
 
 
 
 
 
 
 
 
 
 
 

6 
 
 
PARTICIPATING STUDY SITES (Los Angeles County Department of Health Services) 
1. Antelope Valley Health Center, 335-B E. Avenue K-6 , Lancaster, 93535 
2. Bell Health Center, 6901 Atlantic Ave, Bell, 90201 
3. Bellflower Health Center, 10005 E. Flower Ave, Bellflower, 90706 
4. Curtis R. Tucker Health Center, 123 W Manchester Blvd, Inglewood, 90301 
5. East Los Angeles Health Center, 133 N Sunol Drive, Suite 150, Los Angeles, 90063 
6. East San Gabriel Valley Health Center, 1359 N. Grand Ave, Covina, 91724 
7. Edward Roybal Comprehensive Health Center, 245 S. Fetterly St, Los Angeles, 90022 
8. El Monte Comprehensive Health Center, 10953 Ramona Blvd, El Monte, 91731 
9. Glendale Health Center, 501 N. Glendale Ave., Glendale, 91206 
10. H. Claude Hudson Comprehensive Health Center, 2829 S Grand Ave, Los Angeles, 90007 
11. Harbor-UCLA Medical Center, 1000 W. Carson Street, Torrance, 90502 
12. High Desert Regional Health Center, 335 East Avenue I, Lancaster, 93535 
13. Hubert H Humphrey Comprehensive Health Center, 5850 S Main St, Los Angeles, 90003 
14. La Puente Health Center, 15930 Central Avenue, La Puente, 91744 
15. LAC+USC Medical Center, 2051 Marengo Street , Los Angeles, 90033 
16. Lake Los Angeles Community Clinic, 16921 E. Avenue O, Lake Los Angeles, 93591 
17. Littlerock Community Clinic, 8201 Pearblossom Highway, Littlerock, 93543 
18. Long Beach Comprehensive Health Center, 1333 Chestnut Blvd, Long Beach, 90813 
19. Martin Luther King, Jr. Outpatient Center, 1670 E. 120th Street, Los Angeles, 90059 
20. Mid-Valley Comprehensive Health Center, 7515 Van Nuys Blvd, Van Nuys, 91405 
21. North East Health Center, 3303 N Broadway, St. 200, Los Angeles, 90031 
22. Olive View-UCLA Medical Center, 14445 Olive View Dr., Sylmar, 91342 
23. Rancho Los Amigos National Rehabilitation Center, 7601 Imperial Highway, Downey, 90242 
24. San Fernando Health Center, 1212 Pico Street, San Fernando, 91340 
25. South Valley Health Center, 38350 40Th St E, Palmdale, 93552 
26. Torrance Health Center, 711 Del Amo Blvd., Torrance, 90502 
27. West Valley Health Center, 20151 Nordhoff St, Chatsworth, 91311 
28. Wilmington Health Center, 1325 Broad Ave, Wilmington, 90744 

7 
 
 
PRÉCIS  
Study Title 
Precision Vaccine Promotion in Underserved Populations 
Objectives 
This will be a randomized clinical trial (RCT) to test the impact of text message innovations on receipt of influenza vaccination during 
the 2022-23 season.  
Design  
Interventions will be assigned randomly in a 2 x 2 factorial design to all patients eligible for vaccines, including a “standard of care” 
control group that will receive the standard DHS campaign message. 
 
Interventions and Duration 
The intervention period will include a 6-month flu season from 2022 –2023. This pilot study will be embedded as one arm of a larger 
trial.  We will simultaneously test, using a comparative effectiveness trial design, the effect of the following 3 interventions, all 
compared to standard text messages: 1) Enhanced texting (with a callback by a trained call-center staff member to schedule a vaccine 
visit if the patient presses “1” in response to the text 2) Enhanced Bidirectional texting with a texting response from a trained call-
center staff member who will help the patient schedule a vaccine visit through a series of back-and-forth texts 3) Provision of 
transportation information with information within the text messages that highlight to patients that MediCal will pay for transportation 
to a vaccine visit  (Pilot arm). 
 
Outcome Measures  
Our  primary outcome is the end of season influenza vaccination status based on California’s statewide vaccine registry (CAIR2). 
Secondary outcomes include vaccination intentions as measured by patients’ contact with appointment systems, scheduled vaccination 
appointments (and missed appointments), and influenza hospitalizations among patients that are eligible to receive vaccinations. 
 
 

8 
 
 
1. 
STUDY OBJECTIVES 
1.1 
Primary Objectives  
The primary objectives are to 1) Conduct a Retrospective analysis of influenza and COVID-19 Messaging Strategies and Social 
Determinants of Non-Compliance. Between 2019 and 2021, DHS conducted a series of campaigns related to COVID and influenza 
vaccines with the objective of increasing community trust with geographically and demographically targeted advertisements. The 
2021 campaign messages also adopted “Flu shot reserved for you” language that was found to be comparatively effective in Milkman 
et al., 2021.1 We will also investigate disparities between the UCLA and DHS populations, 2) Develop Targeted Vaccination 
Promotion Approaches that Apply Principles of Behavioral Economics. We will use the results of Aim 1 to craft messages that (a) are 
consistent with evidence-based behavioral principles, and (b) promote “sludge reduction” mechanisms. These may include 
personalized messages about the easiest ways for patients to obtain free vaccinations - pharmacies closer to patients' homes and free 
transportation to healthcare facilities that have not been promoted at DHS in the past. Based on the retrospective analysis, we will vary 
the messenger identity and message content and tailor to specific subgroups. Additionally, we will use this time period to engage other 
large public service delivery systems that might participate in a proposal for a future multisystem RCT, 3) Conduct a Randomized 
Controlled Trial of Message Candidates in 2022-2023 Flu Season. The new messaging will be added to an ongoing RCT being 
conducted at DHS in a factorial design. The original study, limited to use of the patient portal to deliver messages, will be augmented 
with our approach and interventions, which will also incorporate paper mail, text messages, and call centers. 
2. 
BACKGROUND AND RATIONALE  
2.1 
Background on Condition, Disease, or Other Primary Study Focus 
Compliance with influenza and COVID-19 vaccinations is a critical public health issue, and underserved populations are vulnerable to 
community spread and frequently have more adverse outcomes. “Mega-studies” of insured populations with a history of vaccination 
may not be representative of heterogeneous groups of uninsured and underinsured patients. Reasons for noncompliance are 
multifactorial: well-meaning policies related to increasing vaccine accessibility may not be widely understood or disseminated, 
employer-based incentives and mandates may not be applicable, and issues related to communication, trust, and safety remain 
challenges and vary among different communities. This study will be the first large-scale randomized controlled trial testing 
personalized behavioral interventions in an underserved population. The Los Angeles County Department of Health Services 
(LACDHS) eligible population is as large as prior “Mega-studies”, though substantially less adherent to vaccination (currently 31.2% 
at LACDHS vs. 42% in Milkman et al.,) with a comparatively large proportion of Hispanic and Black patients.  
 

9 
 
 
3. 
STUDY DESIGN  
See Figure 1 for study design of the larger randomized controlled trial. The pilot arm (this study) is illustrated in green. 
 
Figure 1. Study design 
 
 
4. 
SELECTION AND ENROLLMENT OF PARTICIPANTS  
4.1 
Inclusion Criteria  
All patients over six months old that are members of an LACDHS managed care plan and have had contact with the health system in 
the past 2 years are eligible for LACDHS vaccination campaigns. In 2022, this includes over 273,000 patients. 

10 
 
 
4.2 
Exclusion Criteria  
The exclusion criterion is a patient age of < 6 months, as babies less than 6 months are not eligible for influenza vaccination. We will 
also exclude individuals who have already received their flu vaccination this season. 
4.3 
Study Enrollment Procedures  
Randomized Controlled Trial.  None of the proposed intervention strategies place patients at any additional risk of adverse outcomes, 
above and beyond the risks associated with routine immunization. Influenza vaccine is recommended by the Advisory Committee on 
Immunization Practices (ACIP) for the patient population included in this study and administration of these vaccinations is part of the 
routine practice of immunization delivery.  Therefore, verbal agreement to administer vaccines (at the patient’s medical office/clinic) 
will constitute their consent for vaccination, as is the accepted procedure for the administration of all routine vaccinations.  As with all 
vaccinations offered in a doctor’s office, patients are free to refuse vaccination.  
For the randomized controlled trial, we will be requesting a waiver of consent. Our rationale is based on our contention that we meet 
the conditions as set forth in 45 CFR 46.116, subpart D, which asks that the researchers meet the following criteria: 1) the research 
involves no more than minimal risk to the subjects; 2) the waiver or alteration will not adversely affect the rights and welfare of the 
subjects; 3) the research could not practicably be carried out without the waiver or alteration; and 4) whenever appropriate, the subjects 
will be provided with additional pertinent information after participation. Further, all patients in the RCT will be within the covered 
entity of LACDHS. Regarding subject notification, given the intervention’s focus on practice change and the anticipated very large 
number of patients from selected practices (tens of thousands), while we will update participating practices on the results of the 
intervention, we will not contact individual patients to notify them of the results.  However, results will be made publicly available 
through presentations, publication, and other dissemination efforts. 
5. 
STUDY INTERVENTIONS  
5.1 
Interventions, Administration, and Duration  
Text messages will begin to be sent at the beginning of the 2022 flu season. We will randomly spread out the calls throughout the day 
and week in order to minimize unnecessary overload for the call center when patients text back a “1” response that they would like a 

11 
 
 
call or text message to help them. We can do this by randomizing patients into batches so that there is no systematic time-of-day or 
day-of-week for any patients. We will send up to 3 reminders this flu season.  
 
Control arm: Standard Texts (n=232,950). Patient randomized to this study arm will receive up to 3 text messages, reminding them 
about the importance of influenza vaccination. The messages themselves are shown below. The standard texts will include a clinic call 
back number and patient portal self-scheduling for patients to schedule their influenza vaccines. The direct scheduling texts includes a 
direct number to an agent that can help schedule and answer questions on the phone in real time. The texts with a direct number to 
schedule will link a specified phone number to call and schedule.  This number would be answered by a central agent quickly and a 
patient could schedule their flu shot at any clinic site.  This specific phone number would not go through the multiple option menus a 
patient would normally experience when calling their clinic. 
 
Intervention arm: Standard Texts with Transportation (n=3,050). The standard texts with transportation includes phone number for 
patients to reserve transportation to a vaccination appointment based on Medi-Cal health plan transportation resources.  
 
 

12 
 
 
5.2 
Handling of Study Interventions  
As we will send up to 3 text messages during the flu season, we will search the EHR for whether or not the patient received an 
influenza vaccination this season. If yes, the patients will be dropped form the denominator of patients needing a text message 
reminder. We will search the EHR prior to each of the text message rounds.  Review of medical records is needed in order to obtain 
the information about (a) influenza vaccination dates this upcoming season, and (b) address (in order to group patients into households 
as described above. We will review the EHR prior to each round of text messages to see if patients had received an influenza this 
season.  
5.3 
Adherence Assessment  
In order to ensure that the study interventions are being reliably delivered we will implement a series of test messages to the study team 
to ensure fidelity of the messages to the study design and DHS communication quality standards prior to RCT implementation Study 
staff will conduct assessments regularly during the intervention to ensure that tests do not fail.  
6. 
STUDY PROCEDURES   
6.1 
Schedule of Evaluations 
 
 
Assessment 
Screening: 
Baseline  
Baseline, 
Enrollment,  
Randomizatio
n: (Day 1) 
Intervention 
Start 
 (Month 1) 
Continuously 
Measured or 
monitored 
Intervention 
End  
(Month 6) 
Follow-up 
Period 
Patient-level assessments 
 
 
 
 
 
 
Demographics 
X 
 
 
 
 
 
Vaccination status 
X 
X 
X 
X 
X 
X 

13 
 
 
Scheduled vaccine appts 
 
 
X 
X 
X 
 
Missed vaccine appts 
 
 
X 
X 
X 
 
Influenza 
hospitalizations 
 
 
X 
X 
X 
 
Adverse Events 
 
 
X 
X 
X 
 
6.2 
Description of Evaluations  
6.2.1 Screening Evaluation 
Consenting Procedure 
For the randomized controlled trial, we will be requesting a waiver of consent. Our rationale is based on our contention that we 
meet the conditions as set forth in 45 CFR 46.116, subpart D, which asks that the researchers meet the following criteria: 1) the 
research involves no more than minimal risk to the subjects; 2) the waiver or alteration will not adversely affect the rights and 
welfare of the subjects; 3) the research could not practicably be carried out without the waiver or alteration; and 4) whenever 
appropriate, the subjects will be provided with additional pertinent information after participation.  Further all patients in the 
RCT will be within the covered entity of LACDHS. Regarding subject notification, given the intervention’s focus on practice 
change and the anticipated very large number of patients from selected practices (tens of thousands), while we will update 
participating practices on the results of the intervention, we will not contact individual patients to notify them of the results.  
However, results will be made publicly available through presentations, publication, and other dissemination efforts. 
 
6.2.2 Enrollment, Baseline, and/or Randomization 
Enrollment 
Identification of families (i.e., households): We will group families using one of two possible methods. First, LACDHS has an 
identifier that ties family members together. It is called the “Family Budget Unit.” We will explore this method. If it does not 
work, we will use another method that we have utilized successfully in the UCLA-based study from which this study is 
derived.  We will assume that people living in the same address belong to the same family. As a secondary measure, we use a 

14 
 
 
telephone number as a linking number (i.e., if two people have the same telephone number, we assume they are from the same 
household).  
 
Baseline Assessments 
Vaccination status. We will assess baseline vaccination status at the beginning of the flu season as we will exclude individuals 
who have already received their vaccination. 
 
Randomization 
We will randomly allocate families to one of the study arms shown above and send family members the allocated text 
message. All family members will therefore receive the same text message. This is meant to reduce confusion among patients. 
For the analysis, we will randomly select one index patient per household for analysis. This follows Cochrane criteria for being 
included in meta-analyses. We will attempt to affiliate index patients with practices, and randomize patients (families) within 
these practices.   
 
The retrospective analysis will result in a randomization and personalization template that will include placeholders and 
calculations for the following items: a) patient contact information, b) experimental group assignment, c) message contact 
dates (one per month in batches of 100 patients per day), d) randomly assigned message contact channel for each contact date 
(e.g. phone, portal, email, SMS), e) message templates with substitutable text, f) personalized message content for each month 
of messages, and g) stratification and personalization data elements identified in the retrospective analysis. These templates 
will be populated on premises at LACDHS with data to be retrieved by the Informatics Core staff.  These randomized lists will 
be converted to personalized, tailored messages based on Mail Merge templates or similar DHS campaign management 
software. After lists and personalization have been completed, they will be approved by the research team in a final cross-
check. All patients that have been vaccinated will be removed from contact lists before each wave of messages is delivered. 
These templated assignments may be moved from Excel to DHS Reporting Software (ELM) or other campaign management 
software by the Informatics Core. 
 

15 
 
 
7. 
SAFETY ASSESSMENTS  
7.1 
Specification of Safety Parameters 
There are minimal risks to participants in this study. None of the intervention strategies place patients at any additional risk of adverse 
outcomes, above and beyond the risks associated with routine immunization. Influenza vaccine is recommended by Advisory 
Committee on Immunization Practices (ACIP) for the patient population included in this study and administration of these 
vaccinations is part of the routine practice of immunization delivery.  
 
Risks of immunization: Although rare, there are inherent risks to immunization with any vaccine.  While the practice-based 
immunization efforts may increase the likelihood that patients will receive recommended vaccines, the study does not increase the risk 
(per vaccine) of a vaccine-related adverse event for any individual patient. 
 
Risk of breach of confidentiality: In this study, an additional risk to subjects is the risk of a breach of confidentiality.  A number of 
policies, procedures, and technical safeguards will be in place to ensure that there is no breach of confidentiality as a consequence of 
this study.  These steps are outlined below in the “Protected Health Information” section. The study proposed is a minimal risk study, 
with a loss of confidentiality posing the greatest risk to the individual patients and participating practices, as data will be extracted 
from the EHR data of participating practices to provide study data for the research team. All study data will be prepared by the DHS-
CTSI Informatics Core in accordance with the DHS Research Oversight Board policy. Only Limited Data Sets will be transferred for 
analysis purposes; compliance with the transfer process will be managed by Drs. Meeker and Abhat as DHS Co Investigator. Any 
links between identified data and deidentified data will be stored only on DHS premises. Patient contact lists and personalized data 
elements will be exclusively maintained on premises by the Informatics Core, responsible for implementing the patient assignment 
and personalization templates.  Research data will be entered into a password-protected computer and will never be transferred to 
laptops or portable media.   
Vaccination rates as compared to the control group. Although unlikely, it is possible that the interventions have the effect of reducing 
vaccine uptake; therefore, we will monitor vaccination rates in the intervention groups as compared to control group. 
7.2 
Methods and Timing for Assessing, Recording, and Analyzing Safety Parameters 
Reports of our safety measures will be delivered to our Data Safety Monitoring Board at the cadence determined by the DSMB members 
(likely at 6 and 12 months post implementation). 

16 
 
 
7.3 
Adverse Events  
Potential adverse events include risks of side effects inherent to immunization with any vaccine. However, while the practice-based 
immunization efforts may increase the likelihood that patients will receive recommended vaccines, the study does not increase the risk 
(per vaccine) of a vaccine-related adverse event for any individual patient. Likewise, influenza vaccinations are universally 
recommended for all individuals greater than 6 months of age and is the standard of care.  It is also possible that the interventions have 
the unlikely effect of reducing vaccine uptake.  We will monitor vaccination rates as compared to the control group as a safety metric. 
7.4 
 Reporting Procedures 
The Principal Investigator will report any unanticipated events to the IRB as well as the Data Safety and Monitoring Board (DSMB) 
assembled for this study. When notified of an unanticipated event, the DSMB will convene and make a decision as to whether the study 
should continue. The IRB will also be notified of the DSMB's decision. Please see detailed Data and Safety Monitoring Plan. 
7.5 
Safety Monitoring  
Dr. Meeker as the study Principal Investigator will be responsible for monitoring participant safety, data quality/confidentiality, and 
evaluating the progress of the study on a daily basis in conjunction with study staff. Oversight for this study will be conducted by the 
Roybal Standing Data and Safety Monitoring Board. DSMB Membership will be reviewed and approved by the NIH. Should there be 
any questions regarding the independence of DSMB member(s), it will be addressed and corrected, if necessary, at that time.  
The DSMB will have the role/responsibility of reviewing the entire IRB-approved study protocol regarding subject safety and analysis, 
and participant recruitment and retention milestones. At the time of interim analysis (50% enrollment), the study team will prepare a 
safety report to be reviewed by the DSMB who will provide recommendations to the NIH for or against the trial's continuation, as well 
as any modification to the study. 
8. 
INTERVENTION DISCONTINUATION  
Following each DSMB meeting/report, the board will make recommendations to the IRB as to whether the study should continue or if 
changes to the protocol are necessary for continuation. 

17 
 
 
9. 
STATISTICAL CONSIDERATIONS  
9.1 
General Design Issues  
Design  
Randomized controlled trial  
 
Outcome Measures  
Our  primary outcome is the end of season influenza vaccination status based on California’s statewide vaccine registry (CAIR2). 
Secondary outcomes include vaccination intentions as measured by patients’ contact with appointment systems, scheduled vaccination 
appointments (and missed appointments), and influenza hospitalizations among patients that are eligible to receive vaccinations. 
9.2 
Sample Size  
Our power calculation suggests that we have 80% power to detect a 2% impact of each intervention.  
9.3 
Interim analyses and Stopping Rules 
No interim analysis will be conducted on primary or secondary outcomes. The DSMB is granted full power to recommend 
discontinuation of the study to the consolidated IRB if safety concerns are found. The board will meet prior to each flu season RCT 
implementation to review patient safety and adverse events. The DSMB members will determine the cadence of future meetings and/or 
safety reports. Following each meeting/report, the board will make recommendations to the IRB as to whether the study should continue 
or if changes to the protocol are needed.     
9.4 
Outcomes  
9.4.1 Primary outcome   
End of season influenza vaccination status based on California’s statewide vaccine registry (CAIR2). 
9.4.2 Secondary outcomes   
Vaccination intentions as measured by patients’ contact with appointment systems, scheduled vaccination appointments (and missed 
appointments), and influenza hospitalizations among patients that are eligible to receive vaccinations. 

18 
 
 
9.5 
Data Analyses  
Statistical Analysis Plan  
We will perform separate evaluations of the impact of the text enhancements (callback and bidirectional texting) on the one hand, and 
transportation messaging on the other hand. Primary outcomes will be evaluated as a test of differences between proportions 
vaccinated in control and treatment groups, and heterogeneity of treatment effects will separately analyze subgroups.  For further 
analyses, we will use multivariable log-binomial regression models with vaccination status at the end of the campaign as the outcome 
variable. Regressions will include indicators for each intervention as fixed effects, with the standard text condition as the reference 
group, and potentially clinic random effects, depending on retrospective analysis results. Models will also adjust for patient age, 
gender, race/ethnicity, insurance, and whether the patient was vaccinated in at least one of the prior two flu seasons. Secondary 
analysis will include subgroup interactions with intervention indicators to assess heterogeneity of treatment effects.  In exploratory 
analyses, we will investigate interactions between interventions, questions related to modality of message delivery, as well as whether 
encounter type, click rates on messages, and distance to the health care facility from the home address affect influenza vaccination 
rates.  
 
In addition, we will evaluate the loss to follow-through rates (scheduling an appointment, but did not complete the appointment) by 
patient demographics, as distance to the health care facility from the home address).  
 
For the evaluation of text enhancements, we will perform a 3-fold Bonferroni correction accounting for pairwise comparison of 
intervention arms, using a two-sided significance level of 0.017. For the evaluation of transportation messaging, we will use a two-
sided significance level of 0.05. All analyses will be intention-to-treat, and will be performed using SAS v. 9.4 (SAS Institute Inc., 
Cary, NC). 
10. 
DATA COLLECTION AND QUALITY ASSURANCE 
10.1 
Data Collection Forms  
Data will be collected from LACDHS electronic medical records and CAIR2 data. 
  
10.2 
Data Management  
All study data, including CAIR2 data and DHS EMR and administrative data will be prepared by the DHS-CTSI Informatics Core in 
accordance with the DHS Research Oversight Board policy. Dr. Abhat will submit all data requests to the DHS helpdesk, which 
includes verification of procedural compliance with IRB and ROB approval screening for DHS-based research. Only HIPAA-limited 
or deidentified data sets will be transferred for analysis purposes; compliance with the DHS secure data transfer process to UCLA 

19 
 
 
and/or USC will be managed by Dr. Abhat as DHS Principal Investigator. Any links between identified data and deidentified data will 
be stored only on DHS premises. Patient contact lists and personalized data elements will be exclusively maintained on premises by 
the Informatics Core, responsible for implementing the patient assignment and personalization templates.  
 
10.3 
Quality Assurance  
Training 
Staff will be trained on the permissible values present in Electronic Records, frequency of update, and expected volumes of data. All 
investigators and staff involved in the project have completed the Collaborative Institutional Training Initiative (CITI) online training 
courses related to human subject’s protections. These courses fulfill all NIH requirements for human subjects training. 
Quality Control Committee  
The Roybal Center Executive Advisory Board will serve as the quality control committee, reviewing Data and Safety reports prior to 
submitting to DSMB. 
Intervention fidelity 
Under Dr. Abhat’s oversight, the USC CTSI research coordinator will be responsible for ensuring that both experimental procedures for 
delivering the correct patient the correct message at the correct time are observed. The coordinator shall complete all DHS training 
required to execute patient messaging and implement a series of test messages to the study team to ensure fidelity of the messages to 
the study design and DHS communication quality standards. It is expected that the coordinator will take responsibility for DHS campaign 
outreach efforts to avoid redundant messaging with any existing DHS campaigns and assurance that all patients receive control campaign 
content. This will require working closely with DHS clinical staff, including clinic navigators and other personnel. Drs. Meeker and 
Abhat will ensure that all SOPs are observed and all analysis is conducted in compliance with DHS policy.   
Protocol Deviations 
Our task tracking systems, JIRA and Monday.com will be used to track and document issues. Each issue will include both an assignee 
and a reviewer. 
Monitoring 
See Data and Safety Monitoring Plan.  

20 
 
 
11. 
PARTICIPANT RIGHTS AND CONFIDENTIALITY  
11.1 
Institutional Review Board (IRB) Review  
The study protocol has been reviewed and approved by the University of California, Los Angeles Institutional Review Board (IRB) as 
the IRB of record (IRB#17-001889). USC has also agreed to rely on the UCLA IRB through the SMART IRB online reliance system 
(IRB # HS-22-00479). 
11.2 
Informed Consent Forms 
Not applicable - we have received approval for a waiver of consent for the randomized trial.   
11.3 
Participant Confidentiality  
Only HIPAA-limited or deidentified data sets will be transferred for analysis purposes;  compliance with the transfer process will be 
managed by Dr. Abhat as DHS Principal Investigator. Any links between identified data and deidentified data will be stored only on 
DHS premises. 
USC study personnel will connect to the Schaeffer Servers on desktop computers in locked offices at USC Schaeffer Center facilities. 
User accounts use 1024 bit public-private key pairs to authenticate the user; there is no password access to the servers via SSH. Users 
must be within the USC firewall, either hardwired to the USC network or authenticated through the USC VPN. All users have human 
subjects research training (either through USC CITI or NIH), as well as data protection training that is specific to our system. Unix user 
groups will control access to directories within these secure servers, such that only study investigators will have access to the data.  
11.4 
Study Discontinuation  
Following each DSMB meeting/report, the board will make recommendations to the IRB as to whether the study should continue or if 
changes to the protocol are necessary for continuation. 

21 
 
 
12. 
COMMITTEES 
Data Safety Monitoring Board: Oversight for this study will be conducted by the Roybal Standing Data and Safety Monitoring Board 
DSMB convened prior to study commencement. DSMB Membership will be reviewed and approved by the NIH. Should there be any 
questions regarding the independence of DSMB member(s), it will be addressed and corrected, if necessary, at that time.  
The DSMB will have the role/responsibility of reviewing the entire IRB-approved study protocol regarding subject safety and analysis, 
and participant recruitment and retention milestones. At the time of interim analysis (50% enrollment), the study team will prepare a 
safety report to be reviewed by the DSMB who will provide recommendations to the NIH for or against the trial's continuation, as well 
as any modification to the study. 
13. 
PUBLICATION OF RESEARCH FINDINGS  
Publication of results from our research will follow the NIH Public Access Policy, which requires that we submit to the National Library 
of Medicine’s PubMed Central an electronic version of final, peer-reviewed manuscripts upon acceptance for publication, to be made 
publicly available no later than 12 months after the official date of publication.  
14. 
REFERENCES  
[1] K.L. Milkman, L. Gandhi, M.S. Patel, H.N. Graci, D.M. Gromet, H. Ho, J.S. Kay, T.W. Lee, J. Rothschild, J.E. Bogard, I. Brody, 
C.F. Chabris, E. Chang, G.B. Chapman, J.E. Dannals, N.J. Goldstein, A. Goren, H. Hershfield, A. Hirsch, J. Hmurovic, S. Horn, 
D.S. Karlan, A.S. Kristal, C. Lamberton, M.N. Meyer, A.H. Oakes, M.E. Schweitzer, M. Shermohammed, J. Talloen, C. Warren, 
A. Whillans, K.N. Yadav, J.J. Zlatev, R. Berman, C.N. Evans, R. Ladhania, J. Ludwig, N. Mazar, S. Mullainathan, C.K. Snider, J. 
Spiess, E. Tsukayama, L. Ungar, C. Van den Bulte, K.G. Volpp, A.L. Duckworth, A 680,000-person megastudy of nudges to 
encourage vaccination in pharmacies, Proc. Natl. Acad. Sci. U. S. A. 119 (2022). https://doi.org/10.1073/pnas.2115126119. 
15. 
SUPPLEMENTS/APPENDICES 
None