Reducing Delay of Vaccination in Children Study
Phase: NA
Conditions: Communicable Diseases
Sponsor: Kaiser Permanente

1
The “Reducing Delays in Vaccination” (REDIVAC) Trial:  A protocol for a randomized 
controlled trial of a web-based, individually tailored, educational intervention to improve 
timeliness of infant vaccination 
Version 1.0, April 1, 2016 
Amanda F. Dempsey1; Jason Glanz2; Nicole Wagner2; Jennifer Pyrzanowski1; Komal 
Narwaney2; Carter Sevick1; Courtney Kraus2; Kathy Gleason2; Bethany Kwan1; Kenneth 
Resnicow3; Jessica Cataldi1; Sarah Brewer1
Affiliations: 
1 Adult and Child Consortium for Outcomes Research and Delivery Science (ACCORDS), 
University of Colorado Denver, Aurora, CO 
2 Kaiser Permanente Colorado, Institute for Health Research, Denver CO 
3 School of Public Health, Division of Health Behavior and Health Education, University of 
Michigan, Ann Arbor, MI 
Aim and Hypothesis 
The primary aim of this study is to conduct a three-group randomized, intervention trial to 
measure the effectiveness of the values-framed individually tailored messaging Vaccines and 
Your Baby (VAYB) intervention. The primary hypothesis to be tested is that infants of mothers 
who receive the VAYB intervention will be more up to date on vaccination and mothers will have 
lower levels of vaccine hesitancy than those receiving an untailored version of the intervention 
or those receiving usual care. A secondary aim of the project is to assess the impact of the 
intervention on vaccination attitudes.  
Methods 
Study Design  
This study includes longitudinal follow-up for a 3-armed, individually randomized clinical trial.   
Study arms include: 
1) the tailored (VAYB) intervention  
2) an untailored version of the intervention (untailored)  
3) usual care (UC).  
Study Setting 

2
Kaiser Permanente Colorado (KPCO) is a nonprofit, managed care organization. There are 
approximately 667,000 member with approximately 5,000 pregnancies annually and 140,000 
children receiving care at KPCO clinics.    
Study Population and Inclusion/Exclusion Criteria 
Inclusion 
1. Currently enrolled at KPCO 
2. Plan to use KPCO medical care for their child 
3. English speaking 
4. >18 years of age 
5. Last trimester of pregnancy, based on clinically determined expected delivery date 
Exclusion Criteria 
1. Any of the following conditions confirmed in the electronic medical record (EMR) or 
patient report 
a. fetus has a high-risk condition (e.g., fatal heart condition, trisomy 18, 
anencephaly),  
b. spontaneous or elective abortion 
c. social issues (such as domestic violence) 
d. serious health concerns for the mother  
Participants are screened for eligibility using the (EMR). Chart reviews are conducted on 
patients with health indicators of exclusion criteria in the EMR.  If exclusion criteria are 
confirmed, participants are not recruited for the study.   
Study participants can enroll from the first recruitment outreach that occurs in the last trimester 
of pregnancy to when their infant is < 2 months of age. Participants are removed from the study 
if they have a fetal demise, infant death, if the infant loses KPCO insurance coverage for greater 
than 90 days, if they request to be removed from the study or if they die. This data is obtained 
from a monthly data extraction from the EMR and patient report.     
Consent and Recruitment 
Eligible participants are sent a series of 2 letters, 3 emails, and one phone call 1-2 weeks apart. 
Participants are directed to the KPCO study registration website created specifically for this 
study. On this registration website the participants identity and eligibility are confirmed, and 
consent is electronically signed.  
After consent, participants are directed to the study website where they set up login information 
and are provided with a Baseline Questionnaire.  
The following topic areas are assessed in the Baseline Questionnaire:   
1. intention to vaccinate 
2. vaccination values 
3. logistical barriers to vaccination 

3
4. vaccine attitudes and beliefs (vaccination concerns) 
5. vaccine hesitancy (used for randomization) 
6. demographics.  
Upon completion of this questionnaire, participants are considered to be “enrolled” in the study 
and are randomized. 
Randomization 
Participants are randomized using stratified randomization along with permuted block technique 
using a 1:1:1 allocation ratio between the VAYB, untailored and usual care arms. 
Randomization occurs using software embedded in the study website immediately following 
enrollment into the study. Randomization remains in place throughout the study.  Participants 
are first stratified into either a hesitant or non-hesitant group, based on vaccine hesitancy status 
from the baseline questionnaire. Hesitancy status is assessed using a 5-item validated measure 
developed by Opel DJ (olejado). Participants are categorized based on the measure’s 
suggested (but unpublished) cutoffs. Participants from each group are then added to their own 
set of blocks that each contain 6 slots. There are 2 slots available for each of the 3 study arms. 
These slots are randomly ordered when the block is created. When all 6 slots are filled, a new 
block with 6 randomly ordered slots is added. 
Blinding 
During the consent process, study arms are described to the participant.  So, while participants 
are not explicitly informed about which study arm they are assigned to, they are not blinded to 
their study arm assignment. The project manager for the study is the only unblinded team 
member during analysis and data interpretation. She converts study data to unlabeled arms (i.e. 
arm 1, 2 or 3) allowing for the rest of the study team to be blinded to study arm assignment. 
Unblinding occurs after primary study outcome analysis. Clinics where participants receive care 
are only aware of individual’s study participation if mentioned by the patient during a clinical 
encounter.  
Interventions 
Tailored Intervention 
The VAYB intervention website contains content tailored to participants based on survey 
responses on the following topic areas:   
1. intention to vaccinate 
2. personal attitudes and beliefs about vaccines (vaccination concerns) 
3. vaccination values   
4. logistical barriers to vaccination 
5. child’s nickname and age (collected by the study team through birth date information 
from the EMR).   
The most highly tailored content is in three “Just for You” tiles that are displayed prominently on 
the landing page.  These tiles reflect the three vaccine concerns participants indicated they 
would like more information about. The tiles were further customized on vaccine values, vaccine 
intention and personal characteristics. The remaining content is lightly tailored to reflect 

4
participant’s personal characteristics and vaccine intention.  Topic areas highlighted on the 
home page further identifies additional vaccination concerns identified by participants’ survey 
responses.  
The website tailoring on participant’s vaccination concerns, values and hesitancy is refreshed 3 
times during the course of the study; when the child is 4-6 months, 10-12 months, and 13-15 
months of age when the child receives vaccinations.  
Child is 4 to 6 months of age
Participants re-answer all questions excluding the value items, and 2 of the 5 questions 
assessing vaccine hesitancy. The website tailored content is refreshed accordingly.   
Child is 10 to 12 months of age 
All questions, including values are reassessed again in the 3rd survey. The website tailored 
content is refreshed accordingly. 
Child is 13 to 15 months of age 
Vaccine hesitancy level is reassessed at a 4th survey and the content is again refreshed. 
Satisfaction with the study was also assessed at the 13-15 month survey.    
Untailored Intervention
The untailored intervention has the same text, content and design as the VAYB intervention, 
However, it does not contain any content tailoring linked to survey responses.  The same 
questionnaires administered to participants in the VAYB arm are administered to participants in 
the untailored arm (baseline, and when the child is 4-6 months, 10-12 months, and 13-15 
months of age), but the material is not used to tailor the website content.
Usual Care  
After taking the Baseline Questionnaire, participants in the usual care arm are thanked for their 
time and logged off the study website.  They receive an email containing the Vaccine 
Information Statements (VIS) for the vaccines due in the child’s first year of life. They do not 
have access to the VAYB or untailored websites.  
All arms 
Routine pediatric care is available to infants of all participants in the study. At KPCO this 
typically consists of a series well-child care visits at 2 weeks, 2 months, 4 months, 6 months, 
and 12 months of age, with an optional visit at 9 months of age if desired by the healthcare 
provider or parent.  Well child visit content is based on the Bright Futures program of American 
Academy of Pediatrics guidelines.1  The visit content is focused on the needs of the child and 
family that typically last 20 minutes or less. Parents are provided with the VIS relevant to that 
visit. Providers are often asked about vaccination and can provide additional information 
verbally.  

5
Vaccination intention:  For all time points, vaccination intention is assessed in the survey.  
Vaccination intention is assessed again following exposure to the intervention, or in the case of 
usual care after completing the survey. Post intervention vaccination intention is obtained 
through an emailed survey sent within an hour of completing the survey.  A reminder for this 
vaccination intention assessment is sent to non-responders by email after one day. 
Participants receive a $20 gift card after surveys at each timepoint (baseline, and when the child 
is 4-6 months, 10-12 months, and 13-15 months of age) are complete. 
Outcomes 
Primary:  Vaccination status 
Vaccination data is collected routinely as part of clinical care within the KPCO health system 
and will be assessed from the KPCO EMR at pre-defined ages (200 days and 489 days).  
Colorado Immunization Information System (CIIS) will be used as a secondary vaccination data 
source, though internal audits demonstrate that >95% of childhood vaccines given to KPCO 
patients are captured within the EMR.   
The primary outcome of the study is a dichotomous categorization of vaccination status (up-to-
date vs. not up-to-date) that is defined based on a continuous measure of days under-
vaccinated.  This outcome is assessed at 200 days of age to cover vaccines in the initial infant 
vaccination series and to minimize the loss to follow-up. The following 6 vaccines recommended 
by the Advisory Committee on Immunization Practices will be assessed: hepatitis B; rotavirus; 
diphtheria-tetanus-acellular pertussis; Haemophilus influenzae type b; pneumococcal conjugate 
vaccine; and polio.  
To categorize vaccination status, we will first assess the number of days under-vaccinated for 
the 2- and 4-month vaccines (combined), by calculating the difference between when a vaccine 
dose was actually administered and when a vaccine dose should have been administered 
according to the vaccination schedule recommended by the Advisory Committee on 
Immunization Practices,2 plus an additional 30 day “leeway” to account for vaccination that did 
not occur at exactly the minimal interval between doses. For example, the first dose of rotavirus 
vaccine is due at age 2 months (61 days) but is not considered late until age 92 days. Days 
undervaccinated for this dose begin accruing on day 93. The number of days under-vaccinated 
is then summed across all doses and vaccines to calculate a total number of days under-
vaccinated for each infant and can range from 0-648 days. Infants with 0 total days 
undervaccinated (assessed specifically for the 2 and 4 months vaccines) at 200 days will be 
considered up-to-date on their vaccination status; those with >1 days undervaccinated will be 
considered not up-to-date. 
Secondary Outcomes:   
1. MMR up-to-date status 
A secondary vaccination metric that is assessed is up-to-date status for measles-mumps-rubella 
(MMR) and varicella vaccine at 489 days, when delay for the first dose of these vaccines 
begins. This metric is useful because it incorporates outcomes related to parents’ decision-
making about these two vaccines recommended at 12-15 months of age that are not offered 
previously.   

6
2. Vaccination attitudes and hesitancy 
Vaccine attitudes and hesitancy was in a questionnaire online. These data are used to assess 
changes over time in vaccination attitudes and hesitancy, and how these relate to study arm, 
vaccination values, and vaccination status. Vaccination attitudes are assessed using measures 
previously developed by our team and others.3 vaccine values are assessed using a novel 
framework we have developed, and vaccine hesitancy is assessed using a 5-item validated 
measure (Olejado).  A variety of covariates and potential moderators will be assessed as part of 
this analysis including parent age, gender and insurance coverage type (such as Medicaid and 
HMO), and mother’s age, race, and ethnicity. Also included will be metrics measuring website 
engagement (VAYB and Untailored arms only) including time spent on the website, number of 
times viewing website, number and order of pages viewed, and match between stated concerns 
and website material viewed (VAYB arm only). 
Participant retention 
Even with the $20 per survey completion incentive, we expect a certain amount of drop off in 
survey participation.  Our primary outcome is vaccination status. Thus, mothers who do not 
participate in all the study surveys are eligible for primary study outcome assessment, so long 
as their child maintains coverage and continues to seek care within the KPCO health system.  
Our previous studies indicate the proportion of children who discontinue KPCO coverage after 
the birth to be ~15%, and our study is powered with this attrition in mind.4
Analysis 
Total days undervaccinated will be analyzed in two ways. First it will be done using a 
dichotomous variable (up-to-date vaccination status). Second analysis will be done as a 
continuous measure.  The dichotomous variable will be analyzed using logistic regression to 
estimate odds ratios and associated 95% confidence intervals. We will use a nonparametric 
analysis and a rank transformation approach to assess the continuous measure as total days 
undervaccinated has a highly skewed distribution. We will use a modified intention to treat 
framework. To ensure vaccination data available for assessment of outcomes, this analytic 
cohort will include infants of all randomized mothers who maintained KPCO health coverage for 
the allotted amount of time (200 days for the primary outcome, 489 days for the secondary 
outcome) with no more than 90 days of no coverage. For both vaccination behavior measures, if 
a statistically significant association is observed in the primary analysis, we will conduct 
analyses stratified by baseline vaccine hesitancy as measured by the 5-item vaccine hesitancy 
scale described above (Olejado). 
For survey measures, we will assess changes in vaccine attitudes and intention over time.  All 
survey items are measured using Likert scales and will be analyzed as linear measures.  For 
each survey outcome average change over time by arm will be assessed using a repeated 
measures ANOVA.  Mixed linear models will be used to assess the “difference in difference” 
over time in these means, by arm, controlling for the covariates. To ensure survey outcomes for 
analysis, we will use a modified intention to treat analysis that includes all participants with data 
from at least one non-baseline questionnaire.  

7
Missing data 
Nearly all vaccines provided to KPCO patients are documented in the EMR, and doses provided 
outside KPCO are documented in Colorado Immunization Information System (CIIS).  
Therefore, we expect minimal missing data for vaccination outcomes. To ensure the most 
complete record, CIIS will be cross checked for all participants to identify any vaccine doses 
given to infants outside the KPCO system that are missing from the KPCO EMR. Participants 
who do not have vaccination data present in either system will be assumed to have not gotten a 
vaccine dose elsewhere.  
For survey data, due to our recruitment strategy, we anticipate no missing data at baseline, as 
completion of the baseline survey was a criterion study enrollment. However, there may be 
missing data for subsequent surveys as completion of the subsequent surveys was not required 
to remain in the study.  For missing data in surveys beyond baseline, multiple imputation models 
will be developed for analyses involving multiple survey points where greater than 10% of 
subjects would be lost due to missing values. 
Ethics and Dissemination 
Approvals 
This study is approved by the Institutional Review Boards at the University of Colorado and 
KPCO.  
Funding  
This work was funded by the National Institutes of Health, Eunice Kennedy Shriver National 
Institute of Child Health and Human Development, # R01HD079457. 
Competing Interests 
Amanda Dempsey serves on Advisory Boards for Merck, Pfizer and Sanofi Pasteur, and has 
provided consulting services to Pfizer.  She does not receive any research funding from these 
companies. All other authors have no competing interests to declare. 

8
REFERENCES 
1. 
Bright Futures:  Guidelines for Health Supervision of Infants, Children, and Adolescents.  
2nd ed. Revised. Arlington, VA: National Center for Education in maternal and Child 
Health; 2002. 
2. 
Robinson CL, Romero JR, Kempe A, Pellegrini C. Advisory Committee on Immunization 
Practices Recommended Immunization Schedule for Children and Adolescents Aged 18 
Years or Younger - United States, 2017. MMWR Morbidity and mortality weekly report. 
2017;66(5):134-135. 
3. 
Shoup JA. Dissertation: Concerns, Attitudes, Beliefs and Intentions of Parents about 
Vaccines for their Child. Denver, CO: School of Public Affairs, University of Colorado 
Denver; 2015. 
4. 
Daley MF, Narwaney KJ, Shoup JA, Wagner NM, Glanz JM. Addressing Parents' 
Vaccine Concerns: A Randomized Trial of a Social Media Intervention. American journal 
of preventive medicine. 2018.