Trusted Messengers: Supporting Physicians in Promoting COVID-19 Vaccination
Phase: NA
Conditions: Covid19, Vaccination
Sponsor: University of Massachusetts, Worcester

 
 
 
 
 
 
 Trusted Messengers: Intervention to Promote COVID-19 Vaccination
NCT04981392
Protocol and Analysis Plan
Version
 
Date:
 
August 1, 2022

INVESTIGATOR STUDY PLAN - REQUIRED 
 
 
 
IRB Docket #: H00023254 
Page 1 of 21 
Template Revised 3/15/2021 
Version 5: 2022/08/01 
 
1. TITLE 
 
Trusted Messengers: Supporting Physicians in Promoting COVID-19 Vaccination  
 
 
2. EXTERNAL IRB REVIEW HISTORY* 
 
N/A 
 
 
3. PRIOR APPROVALS: 
 
We have received prior approvals from the individuals at each of the clinical sites as follows: 
1. University of Massachusetts Medical Center (UMMHC) 
2. Family Health Center of Worcester (FHCW) 
3. Edward M. Kennedy Community Health Center (EMKCHC) 
 
4. OBJECTIVES* 
 
This study aims to refine and test strategies to overcome COVID-19 vaccine hesitancy, 
particularly among vulnerable populations, through a community-engaged approach.  We will 
accomplish our goals via the following specific aims: 
Aim 1 (Focus Groups): To refine and adapt tools to support effective Primary Care 
Provider (PCP) recommendations for COVID-19 vaccination and information 
dissemination by PCPs and community organizations to vulnerable patients.  Tools will 
include: (1) an online library of brief, culturally appropriate videos depicting PCP 
conversations with patients that address common concerns about the COVID-19 vaccine, 
to be disseminated by PCPs and community organizations; (2) automated PCP text 
messaging to patients recommending COVID-19 vaccination; and (3) educational 
materials for PCPs to support their conversations with patients about the COVID-19 
vaccine. 
Aim 2 (Intervention/Clinical Trial): To implement and assess the impact of the 
intervention on COVID-19 vaccine uptake through a large, pragmatic cluster randomized 
clinical trial in primary care clinics from 3 healthcare systems that serve populations 
vulnerable to both health disparities and COVID-19 vaccine hesitancy, including two 
federally qualified health center (FQHCs) and UMass Memorial Health Care (UMMHC).  
The primary outcome will be COVID-19 vaccine uptake among initially unvaccinated 
patients. 
Aim 3 (Interviews): To evaluate the intervention according to the RE-AIM framework 
(Reach-Effectiveness-Adoption-Implementation-Maintenance1), incorporating the 
perspectives of patients, primary care providers, and clinic leaders, while characterizing 
the community environment and context.  The findings of Aim 3 will be used to create an 
implementation guide to accompany the tools which will be made freely available for use 
by other physicians, healthcare systems, and community organizations. 

INVESTIGATOR STUDY PLAN - REQUIRED 
 
 
 
IRB Docket #: H00023254 
Page 2 of 21 
Template Revised 3/15/2021 
Version 5: 2022/08/01 
 
We will submit two separate IRB applications for this study.  This application is for Aim 2 
(Intervention).  The IRB application for Aim 1 (Focus Groups) and Aim 3 (Interviews) was 
submitted to the IRB on April 28, 2021 (IRB # H00023125). 
 
5. BACKGROUND* 
 
The severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) and associated coronavirus 
disease 2019 (COVID-19) pandemic have been devastating.  In the United States (US), more 
than 25 million people have been infected, resulting in more than 430,000 deaths.  Black and 
Latino communities have borne a starkly disproportionate impact of COVID-19, compounding 
existing health disparities.  The approval of multiple effective COVID-19 vaccines has raised 
hope that a return to “normal” life may be at hand.  However, realizing the benefits of a vaccine 
requires widespread acceptance and vaccine uptake.  Recent estimates suggest that as many as 
90% of the population may need to be vaccinated to achieve herd immunity.  Our work and that 
of others indicate a significant proportion of the US population are reluctant to be vaccinated.  
Overlap between factors associated with COVID-19 disease and COVID-19 vaccine hesitancy, 
including being of Black or Latino race/ethnicity, and socioeconomic disadvantage, threaten to 
severely exacerbate existing health disparities. 
Our research hypothesis is that trusted messengers, such as PCPs and community organizations, 
are essential to increasing uptake of the COVID-19 vaccine, particularly for the most hesitant 
groups.  Healthcare provider (HCP) recommendations for vaccination have consistently been 
associated with higher rates of vaccine uptake.2-8  A recent poll in Massachusetts found that 
people’s own doctors are the most trusted source of information regarding the COVID-19 
vaccine, across all demographic groups.9  In addition, a recommendation from a physician is 
highly influential in promoting uptake of other vaccines (e.g., influenza), underscoring the 
critical role of physicians in promoting COVID-19 vaccination.  Like physicians, community 
organizations have been proven to be an essential component of outreach to promote vaccination, 
particularly among hard-to-reach populations.  PCPs and community organizations need 
resources and tools to help them realize their potential role as trusted messengers to convey 
accurate, personalized recommendations for COVID-19 vaccination. 
Building on strong community partnerships, we will refine, implement, and evaluate a multi-
faceted intervention to support PCP outreach, and PCP and community organization 
dissemination of information to promote COVID-19 vaccination among vulnerable patients in 
and near Worcester.  The intervention will include: (1) a library of videos addressing common 
vaccine concerns to be disseminated by PCPs and local community organizations; (2) tools to 
implement PCP outreach and COVID-19 vaccine recommendation; and (3) educational materials 
for PCPs to support their conversations with patients about the COVID-19 vaccine.  We will 
implement and evaluate the intervention at three clinical sites via a large pragmatic clinical trial 
while simultaneously characterizing the community environment in which the intervention is 
situated.  The clinical sites include two FQHCs and UMMHC which collectively serve large 
populations of racial and/or ethnic minority groups (Black, Latino, non-primary English 
speakers) and socioeconomically disadvantaged patients.  We will work with community 
organizations to convene and engage community advisory boards to inform message and video 
development, implementation, and evaluation. 

INVESTIGATOR STUDY PLAN - REQUIRED 
 
 
 
IRB Docket #: H00023254 
Page 3 of 21 
Template Revised 3/15/2021 
Version 5: 2022/08/01 
 
This research will generate crucial evidence of how to effectively promote COVID-19 
vaccination among vulnerable populations.  Our findings can be applied to promote uptake of 
other vaccines (including possible COVID-19 vaccine “boosters”) among populations at risk of 
vaccine hesitancy and health disparities. 
 
6. INCLUSION AND EXCLUSION CRITERIA* 
 
Patient Participants 
Inclusion criteria  
• Patient at a participating clinic site 
• Age > 18 years 
 
Exclusion criteria  
• None 
 
Provider Participants  
Inclusion criteria  
• PCP at a participating clinic site 
• Age > 18 years 
 
Exclusion criteria  
• None 
 
Children 
This study will not include children.  The study is restricted to adults aged 18 and over.   
 
Prisoners 
Prisoners will not be included in the study.  We will not knowingly collect data on prisoners.  
 
Pregnant Women 
It is possible that data will be collected on pregnant women who meet inclusion criteria, however 
pregnant women are not a focus of recruitment and the intervention itself is not experimental.  
 
Non-English Speaking Subjects 
Non-English speaking subjects will be included in the study.  Patient participants will be 
recruited from primary care clinics which collectively serve a large population of non-primary 
English speakers.    
 
 
7. STUDY-WIDE NUMBER OF SUBJECTS* 
 
Patient Participants 
The intervention will be implemented at approximately 16 primary care clinics selected from 
three healthcare systems, including UMMHC, the Edward M. Kennedy Community Health 

INVESTIGATOR STUDY PLAN - REQUIRED 
 
 
 
IRB Docket #: H00023254 
Page 4 of 21 
Template Revised 3/15/2021 
Version 5: 2022/08/01 
 
Center (EMKCHC), and the Family Health Center of Worcester (FHCW).  Eligible patients seen 
at a study clinic are at least 18 years of age or older.  Based on the most recent data, there were 
approximately 152,110 adult (age > 18) patients seen in the selected primary care clinics at 
UMMHC, EMKCHC, and FHCW.    
 
Provider Participants 
Eligible providers are those who currently work at one of the primary care clinics selected from 
the three healthcare systems (UMMHC, EMKCHC, FHCW).  Using information available on 
each healthcare system’s website, we estimated that there are approximately 250 providers 
employed across the selected primary care clinics who will be eligible to participate in the 
provider education portion of the intervention.  We will collect data from a subset of these 
providers, covered in IRB docket #H00023125.   
 
8. STUDY-WIDE RECRUITMENT METHODS* 
 
Patient Participants 
Patient participants will not be recruited as there are no study-specific activities for them to 
engage in.  
 
Provider Participants 
All sites will follow local recruitment methods. Please see Section #24. Local Recruitment 
Methods for more information.  
 
 
9. STUDY TIMELINES 
 
The implementation of the intervention will start after all study materials have been finalized 
using feedback from focus groups (Aim 1) and community advisory boards.  The intervention 
will be implemented in stages.  This will allow for coordinated delivery of a physician education 
session approximately 1-2 weeks prior to each clinic starting the intervention.  The intervention 
will be implemented in blocks of matched clinics, approximately one block every 1-3 months. 
We anticipate it taking approximately 6-12 months to complete the intervention at all sites.  
Patient participants will receive intermittent text messages over a period of approximately 3 
weeks. We will examine vaccination status 12 weeks (primary outcome) and 6 months after the 
last text message to allow for possible lag between the intervention and vaccination.  Because 
the vaccination status in the electronic health record is not immediately updated, these outcomes 
will be ascertained approximately 3 months after the timepoint in question to allow time for lag 
in the clinical system updating vaccination status. Therefore, the entire duration of the 
intervention for an individual patient participant will be approximately 52 weeks.   
 
Provider participant involvement will occur over approximately 2 months.   
 
This study is planned to take place over the course of 3 years, but may be extended if needed 
until all the tasks and deliverables are complete (see Table 1).  
 

INVESTIGATOR STUDY PLAN - REQUIRED 
 
 
 
IRB Docket #: H00023254 
Page 5 of 21 
Template Revised 3/15/2021 
Version 5: 2022/08/01 
 
Table 1: Study Timeline 
Year 1 
Year 2 
Year 3 
Task 
Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 
Convene advisory panel, quarterly meetings 
X 
X X X X X X X X X X X 
IRB approval 
X 
 
 
 
 
 
 
 
 
 
 
 
Focus groups to refine messages & videos 
X 
X 
 
 
 
 
 
 
 
 
 
 
Video production & website creation 
 
X 
 
 
 
 
 
 
 
 
 
 
Baseline data collection, including interviews 
X 
X X 
 
 
 
 
 
 
 
 
 
Identification of study cohort 
 
X 
 
 
 
 
 
 
 
 
 
 
Randomization of clinics 
 
X 
 
 
 
 
 
 
 
 
 
 
Intervention – phased start 
 
 
X X X X X X 
 
 
 
 
Intervention data collection 
 
 
X X X X X X X X 
 
 
Intervention evaluation 
 
 
 
 
 
X X X X X 
 
 
Data analysis 
 
X 
 
 
 
X 
 
 
X 
 
X X 
Manuscript preparation 
Ongoing 
Implementation guide creation & dissemination 
 
 
 
 
 
 
 
 
X X X X 
 
10. STUDY ENDPOINTS* 
 
Primary Endpoint 
The primary endpoint is the receipt of a complete COVID-19 vaccination series (2 doses for 
vaccines that require 2 doses) 12 weeks after the last text message, among patients who are 
unvaccinated at the time of clinic entry into the study.  
 
Secondary Endpoints 
Secondary endpoints include receipt of incomplete (1 dose for vaccines that require 2 doses) 
COVID-19 vaccination series; receipt of a complete COVID-19 vaccination series 6 months after 
the last text message; racial/ethnic differences in vaccination rates; and the percentage of 
intervention clinic patients who initiate vaccination after the first batch of provider texts, after 
the second batch of provider texts, and after the third batch of provider texts.  
 
11. PROCEDURES INVOLVED* 
 
Please see Figure 1 for study activities.  
 
Randomization 
Participating clinics will be randomized to the intervention.  Clinics will be matched by site 
(UMMHC, EMKCHC, FHCW), baseline COVID-19 vaccination rates, and availability of 
COVID-19 vaccines in the clinic, based on advice of the statistician and feasibility.  Within each 
pair, clinics will be randomly allocated to intervention/control condition using a randomization 
generated by the study statistician.  Matched clinics will be grouped into blocks, balancing the 
blocks regarding sites and baseline vaccination rates.  

INVESTIGATOR STUDY PLAN - REQUIRED 
 
 
 
IRB Docket #: H00023254 
Page 6 of 21 
Template Revised 3/15/2021 
Version 5: 2022/08/01 
 
Intervention 
Implementation 
The details and 
timing of 
intervention 
components at a 
given study clinic 
are shown in Table 
2. 
 
Patient 
intervention  
Eligible patients 
seen at a study 
clinic randomized to the intervention will receive text messages in up to 3 different batches using 
the clinical text messaging platform at each study site.  These messages will be delivered over 
approximately 2-3 weeks. Patients will have the option to reply “NO” to the text messages to 
decline further participation at any point during the intervention.  Patients at FHC will also have 
the option to respond “VAXED” or “SCHED” (or some phrasing like these) if they have already 
been vaccinated and would like their records to reflect that, or if they would like someone to 
reach out to them to schedule a vaccination appointment. The texting at FHC will all be 
performed through their secure clinical texting system, HealthTalk AI (HTAI), which is fully 
HIPAA compliant. HTAI software is hosted in a HIPAA compliant cloud instance and their 
secure bi-directional communication is encrypted at rest and in transit. If a patient does respond 
that they would like to schedule a vaccination appointment, the Local Site Champion for the 
study at FHC will triage their information to a scheduler.  All correspondence, updating of 
medical records, and vaccine scheduling will be performed at FHC by staff performing their 
typical duties as required by their role. We are requesting a waiver of consent from the IRB. We 
will collect patient text communication preferences (‘yes’, ‘no’, or ‘null’) and filter out those 
whose text communication preference is ‘no’ so that they do not receive any texts.  We will 
follow the clinical system policy for patients whose preference is ‘null’.  
 
The first batch of text messages will be delivered at the start of the intervention at a study clinic.  
Patients who have not been vaccinated against COVID-19 will receive messages from their PCP 
recommending the COVID-19 vaccine, with information on how and where to schedule a 
vaccination, and a link to a website with additional information and resources.  The message 
content for each batch will be broken up into separate messages to ensure that the content is 
readable and not truncated by the texting system.  Text message batches #2 and #3 (if 
applicable) will be sent approximately 1 week after the prior batch and will remind patients of 
the study website.  The message content is included in appendix 1. 
 
 
Text messages sent out in each batch will be professionally translated into multiple languages 
and provided in the patient’s preferred primary language, according to the capabilities of the 
clinical text messaging systems.  Although messages will be automatically sent via a centralized 

INVESTIGATOR STUDY PLAN - REQUIRED 
 
 
 
IRB Docket #: H00023254 
Page 7 of 21 
Template Revised 3/15/2021 
Version 5: 2022/08/01 
 
system, they will indicate they are from the patient’s PCP where possible.   If it is not possible 
to include the PCP’s name, the message will refer to the organization where the patient receives 
care.  (See Appendix 1 for text message content). 
 
The website patients are directed to will be available in multiple languages via an embedded 
automatic translation feature. The provider videos will also be available in multiple languages 
via an automatic translation feature embedded in the website or video hosting platform. 
 
Provider intervention  
Providers at a study clinic randomized to the intervention will be asked to participate in a “lunch 
and learn” interactive session approximately 1-2 weeks prior to the start of the intervention.  The 
session will be conducted either via video conference (e.g., Zoom) or in-person, depending on 
the state of the pandemic.  An email inviting providers to these sessions is included in Appendix 
2. 
 
Providers will be introduced to the study and presented with underlying research supporting 
message content, as well as the importance of provider recommendations for vaccination.  They 
will be provided with an orientation to the materials on the study website during these sessions.  
Material to be presented to providers in these sessions is included in Appendix 3.   This material 
is intended to be a guide for these sessions.  Because these will be interactive sessions with 
opportunities for questions and discussions, they will necessarily not follow this PowerPoint 
verbatim.   
 
Providers at intervention clinics will be given the opportunity (separate from participation in a 
lunch-and-learn session), to decline to have text messages sent to their patients, and/or to identify 
specific patients to not send text message to.  We will ask providers if they are willing to explain 
their decisions and record that information in a de-identified manner.  We are requesting a 
waiver of consent from the IRB for this aspect of the intervention. 
 
During the first month of the intervention, providers will receive several emails over the course 
of approximately 3 weeks (the number and timing of emails received may slightly vary 
depending on feedback received during the focus groups we will conduct; we will not send more 
than 4 emails over a 4 week period).  The draft emails are included in appendix 2. The emails 
will remind providers of the availability of the website and the resources included on the website, 
and will include an attached “tip sheet” offering tips for talking with vaccine hesitant patients 
(appendix 4).  We will email providers periodically with updates as new professional resources 
or evidence related to COVID-19 vaccines or communicating with patients about COVID-19 
vaccination becomes available, as is common practice within the clinical system. Providers will 
have the option to request not to receive further emails at any point. We will give providers in 
intervention clinics prescription pads with a link to our website so that they can give “a 
prescription for trustworthy information” to vaccine hesitant patients. We have included two 
versions of this prescription pad to be used based on provider preference (appendix 5).  We are 
requesting a waiver of consent from the IRB for this aspect of the intervention.  
 
Table 2. Intervention components and timing at an individual clinic.  

INVESTIGATOR STUDY PLAN - REQUIRED 
 
 
 
IRB Docket #: H00023254 
Page 8 of 21 
Template Revised 3/15/2021 
Version 5: 2022/08/01 
 
Intervention 
Component 
 
Details (what) 
 
Recipient (who) 
 
Timing (when) 
Provider intervention 
Provider 
education 
“Lunch and learn” interactive session to introduce 
study, underlying research supporting message 
content, and importance of provider 
recommendation for vaccination (appendix 3) 
Providers at study 
clinic randomized 
to intervention 
-~ 1-2 weeks prior to 
intervention start at 
study clinic 
Provider 
education 
•  
Emails with brief educational content (appendix 
2): 
• Reminder of the study website and the 
resources available there 
• Tip sheet for responding to patient questions 
and concerns (appendix 4) 
• New scientific information as available 
Providers at study 
clinic randomized 
to intervention 
~ Weekly x 3 during 
month that 
intervention starts at 
study clinic and if 
new resources or 
information is 
available thereafter 
Patient intervention (appendix 1) 
Text message 
batch #1 
Text messages from patient’s PCP or organization 
recommending COVID-19 vaccine and the study 
website (which contains the video library and 
information on how and where to get vaccinated).  
Patients who have 
not been  
vaccinated against 
COVID-19 at time 
of study clinic entry 
Delivered in a set at 
start of intervention 
at study clinic 
Text message 
batch #2 
 
Text messages from patient’s PCP or organization 
encouraging the patient to access the study 
website.   
 
Patients who were 
unvaccinated at the 
start of the 
intervention 
Delivered as a set 
approximately one 
week after first 
message 
Text 
message batch 
#3 
 
Text messages from patient’s PCP or organization 
encouraging the patient to visit the study website 
 
Patients who were 
unvaccinated at the 
start of the 
intervention  
Delivered as a set 
approximately one 
week after the 
second message 
 
 
Control 
Eligible patient participants seen at a study clinic randomized to the control group will receive 
usual care with no text intervention.  Providers at a study clinic randomized to the control group 
will be not be invited to attend the education session or sent any education materials.  We are 
utilizing the Cluster Randomized design in an effort to control cross-group contamination.   
 
EHR Data Collection 
All Electronic Health Record (EHR) data pulls will be requested following the UMMS Data 
Science Core processes (currently located here: https://www.umassmed.edu/research-
informatics/data_services/how-to-obtain-data/).  EHR data will be pulled at non-UMMS sites 
according to their local institution-specific processes.   

INVESTIGATOR STUDY PLAN - REQUIRED 
 
 
 
IRB Docket #: H00023254 
Page 9 of 21 
Template Revised 3/15/2021 
Version 5: 2022/08/01 
 
 
COVID-19 vaccination status will be assessed via EHR on all patients seen at a study clinic to 
allow clinic matching by vaccination rate at study start.  Prior to sending a batch of text 
messages, the COVID-19 vaccination status of patients in the intervention clinic will be re-
assessed to identify those eligible to receive text messages, and their text communication 
preference will be extracted to allow those who have indicated they do not want to receive text 
messages to be filtered out.   
 
Data related to the primary and secondary outcomes will be obtained via the EHR system 
following the completion of the intervention.  
 
We will obtain additional data on all patients via the EHR to conduct supplemental analyses, 
including demographics (e.g., age, sex, race/ethnicity), text communication preference, text 
delivery status (delivered/undelivered), any response to study text messages, primary insurer, 
receipt of influenza vaccine, social vulnerability index (determined using patient address), and 
selected co-morbid medical conditions associated with risk of COVID-19 disease (e.g., 
hypertension, diabetes, obesity, congestive heart failure, chronic lung disease). 
 
For additional details, see section 13. Data Analysis and Management. 
 
 
Screenshots of the website, transcripts of the videos to be hosted on the website, factsheets that 
include video content adapted for print, and biographical sketches of the providers featured in the 
videos can be found in appendices 6, 7, 8, and 9. The factsheets were professionally translated 
and submitted to the IRB for review. 
 
12. DATA AND SPECIMEN BANKING* 
 
No specimens are being collected as part of the research.  A limited data set will be securely 
stored for potential future use by the Principal Investigator.  We acknowledge that we will be 
collecting address information which cannot be included in a limited data set.  This information 
will be used to determine the social vulnerability index and then it will be removed from the 
study data set.  The address information will not be stored for future use. A de-identified data set 
will be shared upon request in accordance with NIH requirements for data sharing.  
 
 
13. Data Analysis and Management* 
 
SAMPLE SIZE 
For details related to sample size, please see #7 Study-wide Number of Subjects. 
 
POWER ANALYSIS 
Power calculations employ two-sided hypothesis testing, 80% power, 0.05 Type I error, and 
account for within-clinic clustering and between-clinic variation in size.  If 25% of  patients are 
unvaccinated at the start of the intervention and 5% of control-clinic patients go on to receive a 

INVESTIGATOR STUDY PLAN - REQUIRED 
 
 
 
IRB Docket #: H00023254 
Page 10 of 21 
Template Revised 3/15/2021 
Version 5: 2022/08/01 
 
vaccination during the intervention period, we can detect an intervention-control difference of 
2.96%, i.e., 7.96% of intervention-clinic patients become vaccinated.  This is consistent with the 
effect size of other studies using a similar approach.10-12 
 
DATA ANALYSIS PLAN  
The primary outcome of this study is receipt of a complete COVID-19 vaccination series (2 
doses for vaccines that require 2 doses) among patients unvaccinated at a clinic’s entry into the 
study, a binary outcome.  We hypothesize that intervention-clinic patients receiving text 
messages from their provider will be more likely to receive a complete vaccination series than 
control-clinic patients.  We will compare vaccination in patients from the two randomization 
arms using generalized linear mixed modeling,13 with a random effect for clinic to account for 
within-clinic clustering.  In addition to estimating unadjusted between-randomization arm 
differences, we will adjust for key covariates, including study block as well as variables 
reflecting possible secular trends such as vaccine availability at the time of a clinic’s entry into 
the study. Patient-level covariates will be available from the electronic health record (EHR), 
including age, sex/gender, insurance type, and address which will be used to determine the social 
vulnerability index; separate models also will adjust for race/ethnicity and relevant clinical 
diagnoses, both of which may be subject to more missing data  than age and sex.  In order to fully 
describe the population who did not receive the intervention (text messages), we will collect 
text communication preference, any responses to study text messages (to capture any patients 
responding “NO” to opt out of receiving further study text messages), and text delivery status 
(delivered/undelivered). We also will adjust for provider type (e.g., resident, attending, nurse 
practitioner) and time since last visit. In supplemental analyses, we will assess effect 
modification of the intervention-control difference by a) block, to explore whether intervention-
control differences change over time; and b) provider type, to examine whether the impact of text 
message receipt varies by a patient’s provider type. Analyses will be intention-to-treat. 
Analyses for the secondary outcome of 1+ dose – also binary – will parallel the above analyses. 
For the additional secondary outcome of racial/ethnic differences in vaccination rates, we will 
add an interaction between randomization arm and race/ethnicity to the above model.  This will 
allow estimation of race/ethnic- specific vaccination rates in each arm and corresponding 
pairwise racial/ethnic differences in the intervention- control differences, as well as testing the 
statistical significance of this effect modification; we hypothesize that the beneficial impact of the 
intervention (versus control) will be larger in patients self-identifying as in minority racial 
groups than in patients self-identifying as white; that is, the gap in vaccination percentage 
between minority and non-minority patients will be smaller in the intervention arm. 
Additional secondary analyses will estimate the percentage of intervention-clinic patients who 
initiate vaccination after the first provider text, after the second provider text, and after the third 
provider text.  Related analyses will compare patients in each of these three subgroups with each 
other and with those not initiating vaccination after three texts, using generalized mixed models 
for categorical characteristics13 and linear mixed models for continuous characteristics such as 
age.14 
 
DATA SECURITY  

INVESTIGATOR STUDY PLAN - REQUIRED 
 
 
 
IRB Docket #: H00023254 
Page 11 of 21 
Template Revised 3/15/2021 
Version 5: 2022/08/01 
 
All research staff collecting, or handling data will be trained in human subjects’ procedures, 
confidentiality, and privacy protection.  All investigators and project staff are required to receive 
and complete Human Subjects and HIPAA training.  All research personnel will hold a current 
Human Subjects Training Certificate.  Clinical staff will access data to send text messages to 
eligible patients, consistent with their role and current activities in the clinical system.  They will 
provide a report of patients who the text messages were delivered to.   
 
All Electronic Health Record (EHR) data pulls at UMMHC will be requested following the 
UMMS Data Science Core processes (currently located here: 
https://www.umassmed.edu/research-informatics/data_services/how-to-obtain-data/).  We will 
follow institution-specific processes at the other clinic sites.  We will transfer data from 
participating sites to UMMS using UMMS MoveIT to transfer the information securely.  We will 
only transfer data after a data sharing agreement has been executed between the three 
organizations (UMMHC, EMKCHC, FHCW).  
 
We are requesting a waiver of HIPAA authorization for the entire conduct of the intervention.   
  
All computerized data will be kept on secured computers or network servers at Meyers Primary 
Care Institute/UMMS and/or the respective clinic sites.  These data will be accessible only to 
research staff with approved access, using confidential usernames and passwords.   
 
Patient participants will be assigned a numerical code (Study ID) for identification in study files. 
Names, addresses, and other direct identifiers will be removed from study data sets once data 
collection is complete and data accuracy is verified.  The research data set used for study 
analysis will be a limited data set (containing only date and age identifiers); the Study ID will be 
retained for researcher communication about subjects. 
 
Analyses will be performed using only limited data sets and only aggregate data will be reported.  
All data will be used for research purposes only; published data will not contain any individual 
identifiers and will be reported in the aggregate.   
 
See also #14 Provisions to Monitor the Data to Ensure the Safety of Subjects, #26 
Confidentiality and #27 Provisions to Protect the Privacy Interests of Subjects.   
DATA MANAGEMENT  
The study database will be programmed and maintained by study staff (e.g. Research Assistant, 
Project Manager, Principal Investigator).  Handling and storage of all data will be conducted as 
described above under Data Security.   
 
DATA QUALITY ASSURANCE 
The PI of the study will be responsible for oversight of the clinical trial and will review study 
data on a periodic basis to monitor and ensure the safety of study subjects. 
 
Data quality control will include regular data verification and protocol compliance checks by the 
PI and Project Manager.  Reports detailing the study progress and subject status, any adverse 

INVESTIGATOR STUDY PLAN - REQUIRED 
 
 
 
IRB Docket #: H00023254 
Page 12 of 21 
Template Revised 3/15/2021 
Version 5: 2022/08/01 
 
events, and any protocol deviations will be generated and reviewed by the PI and additional 
study team members as necessary. 
 
14. PROVISIONS TO MONITOR THE DATA TO ENSURE THE SAFETY OF SUBJECTS* 
 
The proposed study involves no more than minimal risk to participants.  Therefore, we believe 
that an intensive data and safety monitoring plan is not needed.  However, we will take steps to 
ensure the integrity of the data and to detect any adverse effects of the study on the participants. 
 
See #16 for details related to the protection against risks to study subjects.   
 
 
15. WITHDRAWAL OF SUBJECTS WITHOUT THEIR CONSENT* 
 
Patient Participants 
No patient participants will be recruited as there are no study-specific activities for them to 
engage in.  As a result, there is no need to withdraw patient participants without their consent. 
No further text messages will be sent to patients who reply “STOP” to a text message.   
 
Provider Participants 
Provider participants will engage in a one-time education session and will receive approximately 
4 emails with educational information about communicating with patients about COVID-19 
vaccination.  No further emails will be sent to any provider who asks not to receive further 
educational emails. 
 
16. RISKS TO SUBJECTS* 
 
Risk to Subjects 
Patient participants 
The proposed study poses minimal risks to patient participants. The risks that do exist are 
associated with potential loss of confidentiality.  
 
Risks associated with potential loss of confidentiality 
There is a slight risk that research records (EHR data) might be obtained by persons not 
authorized to do so.  There is a slight risk that research data files might be compromised, and 
obtained or viewed by unauthorized  persons. Our procedures for protecting against such risks 
are described below. 
 
Provider participants 
The proposed study poses minimal risks to provider participants. The risk that do exist are 
associated with participating in an educational session. 
 
Risks associated with participating in an educational session 

INVESTIGATOR STUDY PLAN - REQUIRED 
 
 
 
IRB Docket #: H00023254 
Page 13 of 21 
Template Revised 3/15/2021 
Version 5: 2022/08/01 
 
Providers will be recruited to participate in a one-time education session.  This activity can be 
time-consuming to complete and inconvenient to attend. Our procedures for protecting against 
such risks are described below. 
Protections Against Risk for Patient and Provider Participants 
 
Minimizing risks 
The training and monitoring of all study staff performance in accordance with an IRB-
approved study plan will be the responsibility of the Principal Investigator.  All efforts 
will be made to minimize risks and participant inconvenience. 
 
Protection for risks associated with potential loss of confidentiality 
The organization proposing this study has systems, oversight, experienced personnel, and an 
organizational culture that supports the appropriate use, access and storage of confidential 
information.  All persons collecting or handling data will be trained in human subjects’ 
procedures, confidentiality and privacy protection.  All investigators and project staff are 
required to receive and complete IRB and HIPAA training. 
 
Data for all participants will be kept strictly confidential.  All hard copies of research files will be 
kept in locked file cabinets or a locked file room.  Participants will be assigned a numerical code 
(Study ID) for identification in the files.  Names and other identifiers will be kept in separate 
locked files.    Individual identifier information will be removed from study data files as soon as 
possible in the data processing steps.  All computerized data will be kept on secured computers 
or networks.  These data will be accessible only to research staff, using confidential usernames 
and passwords.  Data sharing will only occur between sites with appropriate consent/waiver of 
consent, authorization/waiver of authorization, and an executed Data Use Agreement.  Statistical 
analyses will be performed using only limited datasets and only de-identified data will be 
reported.  All data will be used only for research purposes only; published data will not contain 
any individual identifiers. 
 
Protection for risks associated with participating in an educational session 
Providers will be recruited to participate in an education session.  This can be time-consuming to 
complete and inconvenient to attend.  We will minimize the inconvenience by offering to hold 
the education session at a convenient time for most providers of a given study clinic (e.g., lunch 
break) and via Zoom.  Providers will be told that participation is voluntary and that participation 
in the study will have absolutely no bearing on their employment.   
 
 
17. POTENTIAL DIRECT BENEFITS TO SUBJECTS* 
 
It is uncertain whether individual participants will directly benefit from participation. 
Some participants may learn something new about vaccination (or specifically, about the 
COVID-19 vaccine).  Some participants may be motivated to specifically discuss their 
concerns related to vaccination with their providers and/or patients. Some participants 
may become motivated to be vaccinated against COVID-19 which has substantial health 

INVESTIGATOR STUDY PLAN - REQUIRED 
 
 
 
IRB Docket #: H00023254 
Page 14 of 21 
Template Revised 3/15/2021 
Version 5: 2022/08/01 
 
benefits. Subjects may enjoy participating and may feel that doing so contributes to 
scientific knowledge in general. 
 
 
18. VULNERABLE POPULATIONS* 
 
Please refer to our answers to section #6 Exclusion and Inclusion Criteria. 
 
We will not be targeting any specific vulnerable population for this study.  Children and 
prisoners will not be included.  It is possible that a pregnant woman may meet inclusion criteria 
and it is not an explicit exclusion criterion.  However, the risks involved in participation are no 
more than risks typically associated with general care.  While we are not providing COVID 
vaccinations as part of the study, we are promoting its use.  Currently, there are no 
recommendations against vaccination for pregnant women.  Further, PCPs can exclude patients 
from participation.   
 
In the event of inclusion of pregnant women, the following three statements will hold true: 
  
1. No inducements, monetary or otherwise, will be offered to terminate a pregnancy. 
2. Individuals engaged in the research will have no part in any decisions as to the timing, 
method, or procedures used to terminate a pregnancy. 
3. Individuals engaged in the research will have no part in determining the viability of a 
neonate. 
 
While we will ask providers to participate in an education session and send them emails with 
brief educational content, providers will be informed that their decision not to participate or to 
unsubscribed from our mailing list will not impact their employment in any way.   
 
 
19. MULTI-SITE RESEARCH* 
 
This study and all participating sites will comply with the NIH Policy on the Use of a Single 
Institutional Review Board for Multi-Site Research (NOT-OD-16-094).  All identified 
participating sites have agreed to rely on the University of Massachusetts Medical School IRB.  
The PI and Project Manager at the University of Massachusetts Medical School will be 
responsible for managing all communications between participating sites and the IRB.  The 
University of Massachusetts Medical School will maintain records of the authorization/reliance 
agreements and of the communication plan.  All participating sites will, prior to initiating study 
activities, sign an authorization/reliance agreement that will clarify the roles and responsibilities 
of the IRB and participating sites. 
 
The central project office will be located at the Meyers Primary Care Institute (MPCI) at the 
University of Massachusetts Medical School (UMMS). The Principal Investigator, Dr. Fisher, 
will oversee all project activities. The MPCI project manager, with support from the MPCI 
research assistant, will coordinate all meetings, conference calls, dissemination of materials, and 

INVESTIGATOR STUDY PLAN - REQUIRED 
 
 
 
IRB Docket #: H00023254 
Page 15 of 21 
Template Revised 3/15/2021 
Version 5: 2022/08/01 
 
tracking of task completion.  As we have in previous multi-site studies, to maximize clear 
communication and efficient study management, we will utilize the following procedures: (1) a 
cross-site working group will be established for investigators and project managers; this group 
will hold regularly scheduled conference calls coordinated by staff from the central project site 
who will develop and distribute agendas and action-oriented minutes;  Dr. Fisher will participate 
in these calls; (2) detailed timelines will be developed and distributed for each aspect of the 
project; (3) tracking systems will monitor study progress.  Additional meetings will be 
coordinated regularly with study leadership and study sites for planning of implementation of 
study activities. 
 
The Principal Investigator also assures: 
• All collaborators have the most current version of the protocol. 
• Consent (where applicable) will be conducted by trained research staff listed on the IRB. 
• All required approvals will be obtained prior to the initiation of study activities.   
• Any and all modifications will be communicated to collaborators, and approved before 
the modification is implemented. 
• All engaged collaborators will safeguard data as required by local information security 
policies. 
• All local site investigators will conduct the study appropriately. 
• All non-compliance with the study protocol or applicable requirements will be reported in 
accordance with local policy. 
 
 
An organizational chart detailing the sites involved (and the preliminary study team) is included 
below (Figure 2). 
 
Figure 2. Study Team and Organization 
 
 
 
Community Advisory Board 
Science Participation Resource 
Center (SPRC) Advisory Board (~10 
members) 
Trusted Messenger Advisory Board 
(~10 members) 
 
 
 
 
University of Massachusetts Medical 
School 
Kimberly A. Fisher MD, Principal Investigator 
Co-Investigator(s) 
Programmer(s)/Analyst(s) 
                  Project Manager(s) 
  
 
Research Assistant(s) 
Interviewer(s) 
 
Clark University 
Co-Investigator 
Rosalie Torres Stone PhD 
 
 
Swedish Research Institute 
Consultant 
Barry Saver MD 
Edward M. Kennedy Community 
Health Center 
 
University of Massachusetts 
Memorial Health Center 
 
Family Health Center of 
Worcester 
Study Team 

INVESTIGATOR STUDY PLAN - REQUIRED 
 
 
 
IRB Docket #: H00023254 
Page 16 of 21 
Template Revised 3/15/2021 
Version 5: 2022/08/01 
 
 
20. COMMUNITY-BASED PARTICIPATORY RESEARCH* 
 
The study team will meet at least quarterly with each of two diverse community advisory boards 
throughout the study duration.  The advisory board meetings will serve as a forum for dialogue 
and bidirectional exchange of information with communities at risk of COVID-19 vaccine 
hesitancy and health disparities.  The advisory boards will convey up-to-date local community 
concerns regarding the COVID-19 vaccine to the study team and will provide input on all 
aspects of the study, including physician message content and delivery, and video content and 
dissemination.  Advisory boards will assist with identification of influential community leaders 
who are willing to appear in videos and assist with disseminating the link to the online video 
library via social media and other channels.  We will host continuing community forums based 
on guidance provided by the community advisory boards regarding the need and level of interest 
for ongoing dialogue about the COVID-19 vaccine among their members. 
 
The first board, the standing University of Massachusetts Medical School’s Science Participation 
Research Center’s Community Advisory Board (SPRC CAB) includes clergy who lead local 
Black and Hispanic churches, a leader in the African immigrant community, a leader in the local 
Indian community, and leaders of community organizations (e.g., Elder Services, the YWCA, 
and the United Way)  
 
We will convene a second advisory board to provide further advice and input on the proposed 
study; this board will include additional community members, both leaders and “regular” 
community members to comprise a Trusted Messengers Community Advisory Board (TMCAB) 
to complement the existing board.  Members of the TMCAB will be drawn from the community 
organizations our study team is currently working with as part of the Worcester COVID-19 
Vaccination Campaign.  These presently include but are not limited to CENTRO (the largest 
minority led, community based, multiservice, multicultural, multilingual, nonprofit organization 
in Central MA), the Southeast Asia Coalition, the African Community Education Program, the 
Massachusetts Women of Color Coalition, and Black Families Together. 
 
 
21. SHARING OF RESEARCH RESULTS WITH SUBJECTS* 
 
There are no specific plans to share results with study subjects; study procedures do not include 
any type of diagnostic testing.  All results shared in published research will be in aggregate or 
summary format and will not include identifiable information about participants.  Published 
results will be available to the greater community at large, including study subjects. 
 
 
22. SETTING 
 
The study will be conducted in primary care clinics across three health systems in central 
Massachusetts, including UMMHC, EMKCHC, and FHCW.   
 

INVESTIGATOR STUDY PLAN - REQUIRED 
 
 
 
IRB Docket #: H00023254 
Page 17 of 21 
Template Revised 3/15/2021 
Version 5: 2022/08/01 
 
The intervention will be carried out at 16 primary care clinics across three healthcare systems 
(UMMHC, EMKCHC, FHCW).  Data entry and analysis will be conducted at the Meyers 
Primary Care Institute offices.  
 
23. RESOURCES AVAILABLE 
 
All research personnel listed on this study will read the protocol and receive the appropriate 
supervision and possess the appropriate experience (both higher education and related work 
experience) needed to fulfill their roles and complete their responsibilities for this study. All 
investigators and project staff are required to receive and complete IRB and HIPAA training. 
 
The Principal Investigator (PI) will oversee all personnel and all research activities conducted 
within this study.  She will have responsibility for the overall conduct of the project at this study 
site.  She will have primary oversight of all study personnel.  She will participate in the design 
and the execution of the respective study analyses and will be responsible for the reporting of 
study results. 
 
The Co-Investigator(s) will participate in all aspects of the research and help to ensure the 
accomplishment of all study goals.  They will participate in designing, developing, and 
implementing study procedures and materials.  They will participate in project-related calls and 
meetings and help to develop deliverables and participate in manuscripts.  The Co-Investigators 
will assist the PI in research design and intervention development as well as analytic aspects of 
the study.  The Co-Investigators include clinicians and researchers with varied expertise 
including: healthcare services research, qualitative and quantitative research design, health 
communication and health literacy, and biostatistics.  
 
The Programmer/Analyst(s) will perform a range of programming and data management 
activities essential to conduct of the project.  S/he will perform analyses under the direction of 
the PI.   
 
The Project Manager(s) (PM) will assist the PI and the Co-Investigators in implementing all 
aspects of the project.  Under the direction of PI, the PM will be responsible for day-to-day 
coordination and oversight of the project, including:  developing timelines, work allocation, 
workflow plans, monitoring project progress and task completion, monitoring spending and 
effort allocation, and managing correspondence and administrative tasks.  S/he will 
monitor/manage ethics and regulatory approvals (IRB, HIPAA/DUA).  The PM will attend and 
plan for all project-related meetings as needed.  S/he will work under the direction of the PI to 
assist with all study activities, preparing IRB submissions and reports, and developing study 
materials, such as development of data collection instruments and intervention-related tools.  
S/he will be responsible for maintaining communications with all parties participating in the 
project.  S/he will maintain project documentation and will assist in developing and filing 
required project reports.  PMs at the Meyers Primary Care Institute all hold graduate-level 
degrees and have vast experience working on healthcare services research projects.  
 
The Research Assistant(s) (RA) will work under the direction of the Principal Investigator and 

INVESTIGATOR STUDY PLAN - REQUIRED 
 
 
 
IRB Docket #: H00023254 
Page 18 of 21 
Template Revised 3/15/2021 
Version 5: 2022/08/01 
 
Project Manager to assist with all study activities.  S/he will assist in the development and 
submission of IRB and HIPAA materials, and in IRB reporting over the course of the project.  
S/he will help to coordinate meetings related to the study, and record meeting minutes, products 
and action items.  S/he will assist in recruitment of participants and interviewees, consenting 
participants and scheduling interviews.  Under the direction of the PI and PM, s/he will assist 
with developing REDCap forms for data collection and management, and for tracking study 
activities. S/he will assist in managing transcription and checking the quality of the automated 
transcription. Under the PI’s direction and supervision, s/he will assist in the analysis of 
qualitative data, reviewing and coding transcripts and interviews.  
 
The SPRC CAB and TMCAB members (see #26. Community-Based Participatory Research) 
will be engaged in an advisory capacity only.  These members are neither considered study 
subjects nor study personnel, but rather an expertise resource for the study team.  They will 
neither interact with subjects nor access private identifiable information about them.  
 
All study personnel are required to undergo Human Subjects Training and hold a current CITI 
Human Subjects Training Certificate and will familiarize themselves with the study protocol and 
IRB documents.  
 
24. LOCAL RECRUITMENT METHODS 
 
Patient Participants 
Patient participants will not be recruited as there are no study-specific activities for them to 
engage in. The participating healthcare systems will implement the intervention in a fashion that 
is consistent with other healthcare system patient reminders.  Patients not vaccinated for COVID-
19 will be sent a series of text messages that will include a message from their Primary Care 
Provider, a link to schedule an appointment (if they choose to), and a link to an online library of 
videos and educational materials that they can access if they choose.  Patient participants will be 
given the option to reply “NO” to the text messages. 
 
Provider Participants  
Primary care providers at study clinics randomized to the intervention will be sent an invitation 
to participate in the “lunch and learn” interactive session via email.  The invitation email will 
inform clinic leaders and providers that their primary care clinic has been randomized to receive 
the intervention, briefly introduce the intervention, and explain their involvement in the 
intervention (appendix 2).   
 
The invitation will make it clear that participation in the education session is voluntary.  
Providers will be given the option to decline the invitation.  We will send a reminder email 1-2 
days before each session to providers who have not yet attended one of the sessions for their 
clinic.  Should a provider decline the invitation or indicate that they do not want to be contacted 
again for this study, they will be removed from our mailing list. 
 
The invitation email and subsequent reminder emails will be submitted to the IRB for review and 
approval prior to use in the study (appendix 2).  

INVESTIGATOR STUDY PLAN - REQUIRED 
 
 
 
IRB Docket #: H00023254 
Page 19 of 21 
Template Revised 3/15/2021 
Version 5: 2022/08/01 
 
 
25. LOCAL NUMBER OF SUBJECTS 
  
Please see #7. Study-Wide Number of Subjects. 
 
26. CONFIDENTIALITY  
A major risk of any research is the accidental disclosure of information; however, precautons 
will be taken to prevent this and the study team has an excellent track record of protection of 
confidential data.  The organizations proposing this study have systems, oversight, experienced 
personnel, and an organizational culture that supports the appropriate use, access and storage of 
confidential information.  All persons collecting or handling data will be trained in human 
subjects’ procedures, confidentiality, and privacy protection.  All investigators and project staff 
are required to receive and complete IRB and HIPAA training.   
 
See details related to data security under section #13 Data Analysis and Management. 
 
27. PROVISIONS TO PROTECT THE PRIVACY INTERESTS OF SUBJECTS 
 
Privacy 
Please see section #16. Risks to subject, section #26. Confidentiality, and section #30 Consent 
Process for how participant data will be protected and information on informed consent.  
 
HIPAA Authorization 
We are requesting a waiver of HIPAA Authorization for the entire conduct of the study.  See 
HIPAA waiver request for details. 
 
 
28. COMPENSATION FOR RESEARCH-RELATED INJURY 
 
Not applicable; we do not anticipate any research-related injuries.  We believe the research poses 
no more than minimal risk to subjects. 
 
 
29. ECONOMIC BURDEN TO SUBJECTS 
 
N/A 
 
 
30. CONSENT PROCESS 
 
Patient Participants 
We are requesting a waiver of consent for patient participants as there are no study-specific 
activities for them to engage in.  The participating healthcare systems will implement the 
intervention in a fashion that is consistent with other healthcare system patient reminders.  
Patients not vaccinated with the COVID-19 vaccine will be sent a series of text messages that 

INVESTIGATOR STUDY PLAN - REQUIRED 
 
 
 
IRB Docket #: H00023254 
Page 20 of 21 
Template Revised 3/15/2021 
Version 5: 2022/08/01 
 
will include a message from their Primary Care Provider, a link to schedule an appointment (if 
they choose to), and a link to online educational materials that they can access if they choose.  
Subjects will be given the option to “STOP” the text messages. 
 
Waiver of Consent 
We believe the intervention meets the following criteria to obtain a waiver of consent. We attest 
that the following statements are all true: 
1. The research involves no more than Minimal Risk to the subjects. 
2. The waiver or alteration will not adversely affect the rights and welfare of the  
subjects. 
3. The research could not practicably be carried out without the waiver or alteration. 
4. Whenever appropriate, the subjects will be provided with additional pertinent  
information after participation.  In this event, we will seek the guidance of the IRB. 
5. The research could not practicably be carried out without using Identifiable Private 
Information which is necessary to identify unique participants as participation unfolds 
over time. 
6. The research does not involve Non-Viable Neonates as subjects. 
 
Provider Participants 
We are requesting a waiver of consent for provider participation in the educational component of 
the intervention (attending a one-time educational session and receiving a series of emails with 
educational information about communicating with patients about the COVID-19 vaccine).  
Providers will be informed that participation in these activities is voluntary.  Any provider who 
does not want to take part in the education session can simply not attend.  Providers can choose 
not to open and read the emails with no consequences, or to request not to receive further emails.   
 
Waiver of Consent 
We believe the intervention meets the following criteria to obtain a waiver of consent. We attest 
that the following statements are all true: 
1. The research involves no more than Minimal Risk to the subjects. 
2. The waiver or alteration will not adversely affect the rights and welfare of the  
subjects. 
3. The research could not practicably be carried out without the waiver or alteration. 
4. Whenever appropriate, the subjects will be provided with additional pertinent  
information after participation.  In this event, we will seek the guidance of the IRB.     
5. The research could not practicably be carried out without using Identifiable Private 
Information which is necessary to identify unique participants as participation unfolds 
over time. 
6. The research does not involve Non-Viable Neonates as subjects. 
 
31. PROCESS TO DOCUMENT CONSENT IN WRITING 
 
N/A: We are requesting a waiver of consent for patient and provider participants. 
 
 

INVESTIGATOR STUDY PLAN - REQUIRED 
 
 
 
IRB Docket #: H00023254 
Page 21 of 21 
Template Revised 3/15/2021 
Version 5: 2022/08/01 
 
32. DRUGS OR DEVICES 
 
N/A 
 
REFERENCES 
 
1. 
Glasgow RE, Harden SM, Gaglio B, et al. RE-AIM Planning and Evaluation Framework: 
Adapting to New Science and Practice With a 20-Year Review. Front Public Health. 2019;7:64. 
doi:10.3389/fpubh.2019.00064. 
2. 
Dempsey AF, Abraham LM, Dalton V, Ruffin M. Understanding the reasons why mothers do or 
do not have their adolescent daughters vaccinated against human papillomavirus. Ann Epidemiol. 
2009;19(8):531-538. doi:10.1016/j.annepidem.2009.03.011. 
3. 
Dorell CG, Yankey D, Santibanez TA, Markowitz LE. Human papillomavirus vaccination series 
initiation and completion, 2008-2009. Pediatrics. 2011;128(5):830-839. doi:10.1542/peds.2011-
0950. 
4. 
Goff SL, Mazor KM, Gagne SJ, Corey KC, Blake DR. Vaccine counseling: a content analysis of 
patientphysician discussions regarding human papilloma virus vaccine. Vaccine. 
2011;29(43):7343-7349. doi:10.1016/j.vaccine.2011.07.082. 
5. 
Kessels SJ, Marshall HS, Watson M, Braunack-Mayer AJ, Reuzel R, Tooher RL. Factors 
associated with HPV vaccine uptake in teenage girls: a systematic review. Vaccine. 
2012;30(24):3546-3556. doi:10.1016/j.vaccine.2012.03.063. 
6. 
Lu PJ, Srivastav A, Amaya A, et al. Association of provider recommendation and offer and 
influenza vaccination among adults aged ≥18 years - United States. Vaccine. 2018;36(6):890-898. 
doi:10.1016/j.vaccine.2017.12.016. 
7. 
Schneeberg A, Bettinger JA, McNeil S, et al. Knowledge, attitudes, beliefs and behaviours of 
older adults about pneumococcal immunization, a Public Health Agency of Canada/Canadian 
Institutes of Health Research Influenza Research Network (PCIRN) investigation. BMC Public 
Health. 2014;14:442. doi:10.1186/1471-2458-14-442. 
8. 
Smith PJ, Kennedy AM, Wooten K, Gust DA, Pickering LK. Association between health care 
providers' influence on parents who have concerns about vaccine safety and vaccination 
coverage. Pediatrics. 2006;118(5):e1287-1292. doi:10.1542/peds.2006-0923. 
9. 
Stockwell MS, Hofstetter AM, DuRivage N, et al. Text message reminders for second dose of 
influenza vaccine: a randomized controlled trial. Pediatrics. 2015;135(1):e83-91. 
doi:10.1542/peds.2014-2475. 
10. 
Stockwell MS, Westhoff C, Kharbanda EO, et al. Influenza vaccine text message reminders for 
urban, low-income pregnant women: a randomized controlled trial. Am J Public Health. 2014;104 
Suppl 1(Suppl 1):e7-12. doi:10.2105/ajph.2013.301620. 
11. 
Cutrona SL, Golden JG, Goff SL, et al. Improving Rates of Outpatient Influenza Vaccination 
Through EHR Portal Messages and Interactive Automated Calls: A Randomized Controlled Trial. 
J Gen Intern Med. 2018;33(5):659-667. doi:10.1007/s11606-017-4266-9. 
12. 
Molenberghs G, Verbeke G. Models for Discrete Longitudinal Data. 1 ed. New York: Springer-
Verlag; 2005. 
13. 
Fitzmaurice GM, Laird NM, Ware JH. Applied longitudinal analysis. New York: Wiley, 2004.