The VAX-MOM Study: Increasing Influenza and Tdap Vaccination of Pregnant Women
Phase: NA
Conditions: Immunization; Infection, Pregnancy Related
Sponsor: University of Rochester

Page 1 of 34 
Version Date: 6.22.2022 
 
Official Title: The VAX-MOM Study: Increasing Influenza and Tdap Vaccination of Pregnant 
Women 
 
NCT #: NCT04444518 
 
Protocol Document Date: Version 6/22/22 (the most recently approved protocol version for the 
study) 
 
 
PLEASE NOTE: This protocol document has had some sentences/sections redacted in order to 
maintain the confidentiality of participating individuals and practice sites. 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Page 2 of 34 
Version Date: 6.22.2022 
 
 
VAX-MOM Study: Protocol for Aim/Phase 2 
 
Lead Principal Investigator (URMC Site) – Cynthia Rand, MD, MPH 
Investigator (UCLA Site) – Peter Szilagyi, MD, MPH 
Investigator (RRH Site) – Elizabeth Westen, MD   
Investigator (LADHS Site) – Erin Saleeby, MD, MPH 
 
 
NOTE: The VAX-MOM study is divided into 3 main aims/phases. Each aim/phase informs the 
next, and will be submitted separately for IRB approval, as we cannot predetermine details for 
each aim/phase until the prior aim/phase has been completed. Aim/phase 1 was previously 
submitted and approved. We are now (current protocol) on aim/phase 2, which is being 
submitted for single IRB review, with the University of Rochester RSRB acting as the IRB of 
record for all study sites. 
 
1. PURPOSE OF STUDY (ALL AIMS/PHASES) 
 
Infants under 6 months of age are at increased risk of both influenza (flu) and pertussis 
disease, and pregnant women who get influenza are more likely than non-pregnant women to 
have serious complications, including hospitalizations, death, preterm labor and premature 
birth. The Advisory Committee on Immunization Practices recommends that women receive 
a flu vaccine in flu season, and tetanus toxoid, reduced diphtheria toxoid, acellular pertussis 
(Tdap) vaccine during each pregnancy (ideally between 27-36 weeks) to lower the risk for flu 
and pertussis disease for both themselves and their infants. However, only half of pregnant 
women in the US receive a flu and Tdap vaccine, respectively; only 33% of women receive 
both vaccines. Lack of vaccination stems from a combination of patient (lack of knowledge, 
vaccine hesitancy), provider (suboptimal communication skills, missed opportunities), and 
system (e.g., lack of standing orders and patient reminders) factors. An effective intervention 
is needed to improve flu and Tdap vaccination rates for pregnant women. 
 
This multi-site project has four key aims/phases. Aim/Phase 1 is to measure baseline 
flu/Tdap vaccination coverage and provider knowledge, attitudes and behaviors for flu/Tdap 
vaccination in participating OB/GYN practices within 4 health systems in New York and 
California. Aim/Phase 2 is to use a clustered randomized controlled trial (RCT) (practice-
level randomization) to measure the effect of a multi-component quality improvement (QI) 
intervention (VAX-MOM: training in communication, provider prompts, standing orders + 
feedback on rates) on vaccination rates and provider attitudes, and to measure costs of the 
intervention. Aim/Phase 3 is to develop a translational plan/toolkit for OB/GYN practices 
and health systems for maternal vaccination.  
 
2. BACKGROUND AND RATIONALE (ALL AIMS/PHASES) 
 
Burden of flu disease and pertussis 
Infants <6 months of age are more likely than any other age group to have flu 
complications that lead to hospitalization. Our group has shown that the average annual rate 
of hospitalization attributable to flu was 4.5 per 1000 among children 0 to 5 months of age.1,2 

Page 3 of 34 
Version Date: 6.22.2022 
 
Figure 1. Barriers to Vaccination 
System
Provider
Patient
*Vaccine hesitancy 
*Lack of knowledge
*Lack of visits
Costs
*Missed opportunity
*Weak endorsement
*No champion
*Suboptimal practice
operations
*No reminder/recall
Vaccine supply
*No tracking system
*No reminder system
Financing
Scattering of care
Vaccine shortages
*To be addressed
in VAX-MOM Study
Flu infection rates are similar in pregnant women to the general population, but pregnant 
women are at higher risk of severe flu complications, including hospitalization and death3-6 
and are at increased risk of adverse perinatal and neonatal outcomes.7,8 According to CDC, 
from 1990–2016 infants <1 year of age consistently had the highest reported annual 
incidence of pertussis9 as well as the greatest risk for serious disease and death from 
pertussis. Recently in the U.S., infants in the first 6 months of life had an age-specific 
pertussis incidence of 78 per 100,000; 43% of those with reported infection were 
hospitalized.10 While infants start DTaP vaccine at 2 months and vaccination rates are high, 
they are not well protected until they complete their primary series at 6 months of age. 
 
Effectiveness of maternal vaccination  
A 2016 systematic review showed that maternal flu vaccination provided “substantial 
benefit for infants and mothers,”11 with reduced rates of preterm birth and low birth weight. 
Infants <6 months of age whose mothers had received flu vaccine had a 63% reduction in 
laboratory-confirmed flu and a 29% reduction in respiratory illness with fever compared with 
controls; mothers had a reduction of 36% in the rate of respiratory illness with fever.12 
Maternal flu vaccination is 91.5% effective in preventing flu-associated hospitalization in 
newborns.13 In a cohort study of mother-infant pairs, the rate of pertussis was 43% lower 
among infants whose mothers received prenatal Tdap than among infants whose mothers did 
not;14 protection was seen only among infants whose mothers received Tdap at >27 weeks of 
gestation. These findings are consistent with earlier studies that showed maternal vaccination 
was associated with higher cord serum concentrations of pertussis antibodies15 and, that 
maternal Tdap receipt at 30-32 weeks gestation was associated with significantly higher 
infant serum concentrations of Tdap antibodies at birth and 2 months of age.16 
 
Barriers to Vaccination 
There are many barriers to influenza and Tdap vaccination for pregnant women. Our 
team has conceptualized these barriers into a widely used model; the VAX-MOM study will 
address multiple components by focusing on provider and staff communication, prompts 
and/or standing orders, provider feedback, and patient education/reminders to overcome 
these barriers (Figure 1). 
 
 
 
 
 
 

Page 4 of 34 
Version Date: 6.22.2022 
 
Patient Barriers include vaccine hesitancy, lack of knowledge of the benefits and risks of 
flu or Tdap vaccine or protection for the baby, concerns about vaccine safety generally and 
during pregnancy, low perceived susceptibility to infection, and lack of access.17,18 
Vaccination rates are lower for younger women, Blacks, those with public insurance, lower 
income, and inadequate prenatal care.19,20-22 Provider Barriers include missed opportunities 
for vaccination, lack of a vaccine recommendation or weak endorsements by health providers 
about needed vaccines (the most important predictor in many studies),19,20-22  various 
suboptimal practice operations, insufficient knowledge, financial barriers, and lack of 
reminder recall.23,24,25 System Barriers include the lack of a tracking or reminder system, as 
well as suboptimal staff training, and challenges with vaccine purchase and storage.22,26  
 
3. ADMINISTRATIVE ORGANIZATION  (AIM/PHASE 2 ONLY) 
 
NOTE: As stated at the onset of the protocol, we are requesting that the University of Rochester 
RSRB serve as the IRB of record for all study sites during aim/phase 2 of the study. The 
remainder of the protocol will describe aim/phase 2 details only.  
 
 
 
 
STUDY SITES & PROJECT STAFF OVERVIEW 
 
Project staff are affiliated with four participating health systems across two states. In New 
York, the University of Rochester Medical Center (URMC) and Rochester Regional Health 
(RRH), and in California, the University of California, Los Angeles (UCLA) and the Los 
Angeles Department of Health Services (LADHS). The University of Rochester Medical 
Center (URMC) is the lead study site, with PI Cynthia Rand overseeing all project 
activities (see Table 1). 
 
 
 
 
Aim/Phase 3:
 Translational 
Plan/Toolkit
August 2019 ----------------------------> July 2020
August 2020 ----------------------------> July 2021
August 2021 ----------------------------> July 
<-----YEAR 3----->
• Key Informant Interviews
• Provider Surveys
Translational Plan Toolkit
QI Intervention Practices
Control Practices
<-----YEAR 1----->
<-----YEAR 2----->
Aim/Phase 2:
 Practice Level QI Intervention
↓                  This protocol pertains only to Aim/Phase 2                   ↓
VAX-MOM Study Design
• EHR Analysis
Practice Level Randomization
Aim/Phase 1:
 Assess Baseline
→
→

Page 5 of 34 
Version Date: 6.22.2022 
 
Table 1: Project Sites and Leaders/Team Members 
State 
NY 
CA 
State Leader 
Cynthia Rand, MD, MPH 
(Lead PI) 
Peter Szilagyi, MD, MPH 
 (UCLA Investigator) 
Team Members 
Consultants, OB/GYN staff, Study 
Coordinator  
Consultants, OB/GYN staff, Data 
Specialists, Study Coordinator 
Health System 
URMC 
RRH 
UCLA 
LADHS 
Health System Investigator
Cynthia Rand 
Elizabeth Westen 
Peter Szilagyi 
Erin Saleeby 
# of Participating 
OB/GYN Practices  
6 
16 
7 
8 
 
Investigators and Consultants:  
Cynthia Rand, MD, MPH (URMC Lead PI), an expert on adolescent immunization 
delivery, adolescent preventive services, QI and informatics, will lead the study. She is a 
practicing pediatrician in a large urban practice where she leads QI activities and is an 
experienced health services researcher. Peter Szilagyi, MD, MPH (UCLA Investigator), 
Professor of Pediatrics, has led one of the most impactful US immunization delivery teams 
for 25 years. He studies barriers to vaccination and leads research trials aimed at reducing 
missed vaccine opportunities. Elizabeth Westen, MD (RRH Investigator), is an Ob/Gyn 
whose research interests include maternal vaccination rates. Erin Saleeby, MD, MPH 
(LADHS Investigator), is Chair and Director of Women’s Health Services at LADHS, and a 
health services researcher focusing on vulnerable populations.  
 
Christina Albertin, BSN, MPH, who has worked with both Drs. Szilagyi and Rand for 
over a decade as a senior health project coordinator, will act as a project consultant and CA 
regulatory submission liaison. Sharon Humiston, MD, MPH, a practicing pediatrician and 
expert on vaccine communication and qualitative research, will help lead the qualitative and 
educational components of the project. 
 
NY Study Team Members: 
Courtney Olson-Chen, MD, an OB/GYN and Director of Research, URMC Division of 
Maternal-Fetal Medicine, will act as the URMC health system liaison. Caitlin Dwyer, RN is a 
Nurse Manager within the Rochester Regional Health system, and will act as the RRH health 
system liaison. Robin Bender, LMSW will act as the study coordinator for both the URMC 
and RRH sites, and will additionally assist with CA site coordination. 
 
CA Study Team Members: 
 Jessica Hsu, MD is a practicing OB/GYN at UCLA who directs the department’s QI 
efforts, and will act as the UCLA health system liaison. Judy Chen, MD is a LADHS 
OB/GYN with a focus on informatics and QI, and will act as the LADHS health system 
liaison. Chi-Hong Tseng, PhD is the biostatistician for VAX-MOM, and an expert in 
pragmatic trials. Sitaram Vangala, MS, will assist with data transfer and analysis.  

Page 6 of 34 
Version Date: 6.22.2022 
 
National Advisory Board:  
This group will meet twice yearly by phone to advise on findings from KIIs, provider 
interviews, and intervention components as well as findings from the QI and patient reminder 
RCTs. Sharon Humiston, MD, MPH (project consultant) will help lead the National Advisory 
Board. Kevin Ault, MD is a Professor of OB/GYN at the University of Kansas, with a 
research interest in maternal immunization. Rebecca Perkins, MD is Associate Professor of 
OB/GYN at Boston University’s School of Medicine with expertise in implementation 
science. Mardy Sandler, LMSW, social worker, will represent patient perspectives. 
 
PRINCIPAL INVESTIGATOR (PI) OVERSIGHT PLAN 
 
Cynthia Rand, Principal Investigator, will oversee all project activities. All project 
activities (Steps 1-4 described in section 4 “Study Design & Procedures”) will be identical 
across all URMC, RRH, UCLA, and LADHS study locations. All sites will utilize the same 
site communication methods, training materials, study measures, feedback tools, online 
dissemination platforms, etc.  
 
Study staff will be grouped into distinct administrative teams, and each team will be 
responsible for defined project tasks (see Table 2). 
 
Table 2: Administrative Team Organization & Responsibilities  
Core Leadership Team (both NY & CA combined) 
Global study goals/operations  
(Weekly joint meetings across states)
C Rand*, P Szilagyi*, S Humiston, C Albertin, S Vangala, R 
Bender, A Breck 
State-Level Teams (separate teams in NY & CA) 
Daily study operations  
(Weekly meetings within each state) 
NY: C Rand*, R Bender 
CA: P Szilagyi*, A Breck, S Vangala 
Site-Level Teams 
Practice-level operations 
(Weekly meetings within each state) 
NY: C Rand*, E Westen, C Olsen-Chen, C Dwyer, R 
Bender, EHR report builders
CA: P Szilagyi*, E Saleeby, J Hsu, J Chen, A Breck, EHR 
report builders
Data Development & Analysis Team:  
Randomization, data analyses, data management  
(Meetings as needed within and between each state) 
C Rand*, P Szilagyi*, C Tseng, S Vangala, EHR report 
builders 
*Team Leader 
 
Cynthia Rand, study PI, will ensure study and regulatory compliance, by: 
 Participating in routine prescheduled meetings with all project investigators and 
project staff for the duration of the study, including: 
o Weekly Core Leadership Team Meetings (State-level Investigators, study 
coordinators, all study consultants, data specialist): discussion of global 
project goals, timeline, regulatory requirements and adherence, protocol 
requirements and adherence, measure development, e-learning content, 
data analysis, workflow details, etc. 

Page 7 of 34 
Version Date: 6.22.2022 
 
o Weekly State-level Team Meetings (State-level Investigators, study 
coordinators, data specialist as needed): discussion of day-to-day 
operations needed to support global study goals (including protocol and 
regulatory items). 
o Weekly Site-level Team Meetings (State-level Investigators, study 
coordinators, Site Investigators, OB/GYN site liaisons, site EHR report 
builders): discussion of provider/nurse workflow, development of study 
measures specific to providers/nurses, optimal measure dissemination 
procedures, identification of possible vaccine champions. 
o Data Development & Analysis Meetings (State-level Investigators, study 
coordinators, EHR report builders, data specialists): discussion of data 
collection methods, EHR data extraction methods, data analysis, data 
storage/management.  
 Keeping all study personnel abreast of prearranged study changes and subsequent 
regulatory determinations (e.g., changes to study measures, workflow procedures, 
study personnel, etc.) via a predetermined communication chain: 
o Core Leadership Team discusses necessary study changeNY study 
coordinator submits amendment to RSRBRSRB notifies Study PI and 
NY study coordinator of regulatory determination Study PI and NY 
coordinator notify Core Leadership Team As appropriate, both study 
coordinators will further notify site-level personnel and/or data specialists 
o Communication with study site personnel will most often take place 
during prescheduled meeting times, but in more urgent cases, will take 
place as soon as needed. 
o Communication will be both verbal (during or outside of weekly meetings 
depending upon level of urgency) and written as needed (via e-mail, e-
mail attachments, or the secure shared Box folder).  
 Keeping all study personnel abreast of unforeseen study events and subsequent 
regulatory determinations (e.g., protocol deviations, breaches in confidentiality, 
subject/site withdrawal, etc.) via a predetermined communication chain: 
o Study personnel reports study event to PI PI notifies NY study 
coordinator PI and NY study coordinator notify RSRB, as well as other 
study personnel as appropriate any action requested by RSRB is 
communicated to Core Leadership Team As appropriate, both study 
coordinators will further notify site-level personnel and/or data specialists 
o As described above, communication will be both verbal and written. 
o Also see section #16 “Data & Safety Monitoring Plan” 
 
 
 

Page 8 of 34 
Version Date: 6.22.2022 
 
4. STUDY DESIGN & PROCEDURES (AIM/PHASE 2 ONLY) 
 
Brief Overview of Aim/Phase 2:  
Using a clustered RCT (randomizing practices), we will allocate half of the practices within each 
health system to the VAX-MOM intervention and the other half to standard of care. We will 
measure the impact of the intervention on vaccination rates (primary outcomes) and missed 
opportunities and process metrics (secondary outcomes) (see Table 4). Finally, if VAX-MOM is 
successful, we will provide the control practices the intervention during Year 3.  
 
Design and Procedural Details of Aim/Phase 2: (also see Figure 2 for Timeline) 
 
Step 1) Deliver VAX-MOM training/intervention to randomized practices: 
 
Randomization Details:  
 
Practices will be the unit of randomization. We will assign practices using a covariate 
constrained randomization strategy as follows: First, we will use data from Aim #1 to 
obtain key factors that we expect to affect the primary outcomes. We will perform 
constrained randomization to allocate practices within each health system to 
intervention/control arms and ensure each arm has similar baseline flu and Tdap 
vaccination rates, percent of patients covered by Medicaid, number of OB providers, and 
number of patients. Specifically, we will use these variables to construct and evaluate a 
balance criterion, which is the sum of the squared difference between standardized group 
means on these variables. We will generate randomization separately by health system. 
We will generate all possible combinations of eligible practices in 2 arms (using a SAS 
macro), and define an acceptable set of randomizations that result in balanced variables 
(generally the lowest 10% on the balance criterion). From this set of randomizations, we 
will select one set at random, and then randomly assign each practice to intervention vs 
control group.  
 
Intervention Details: 
 
Once randomization has been completed, and “intervention group” practices have been 
identified, multiple quality improvement (QI) techniques will be aimed at practice 
providers, nurses, and staff.  
 
 Identification of “vaccine champion” at each intervention site: At each site, 
the affiliated medical director will select one “vaccine champion” for the practice. 
Each identified vaccine champion will be e-mailed an Information Sheet (see 
“InformationSheetVaccineChampion…” document) by their affiliated study 
coordinator, outlining the basic study goals as well as their involvement in tasks 
throughout the training and intervention adoption phases of the study (see 
“Consent Process” section). The position of vaccine champion is voluntary, and if 
they do not wish to accept the role, they may decline and another individual will 
be identified by the affiliated medical director. The vaccine champion will play a 
lead role throughout the study intervention, including: 

Page 9 of 34 
Version Date: 6.22.2022 
 
o assisting with the dissemination of e-learning training modules and the 
scheduling of a follow-up site-specific meeting 
o acting to ensure training content is being appropriately adopted into 
practice workflow procedures 
o fielding questions from site personnel regarding the logistics of study 
interventions  
o acting as an ongoing liaison between the site personnel and study staff 
o leading bi-monthly discussions regarding immunization rate feedback with 
site personnel (see step 2 for further detail) 
o planning for and completing monthly PDSA cycles (see step 2 for further 
detail) 
o monthly completion of the “Practice Time/Cost Survey” (see step 4 for 
further detail) 
 
 Delivery/discussion of training modules: Each site will receive training focused 
on 3 main areas of content (see Table 3). This training will be delivered first via 
an online e-learning platform, and next via an in-person or virtual site meeting. 
 
 

Page 10 of 34 
Version Date: 6.22.2022 
 
Table 3: Training Module Content  
Training Topic: 
Training Content Details: 
Flu & Pertussis 101 
General information regarding prevalence/severity of influenza and 
pertussis disease 
General information regarding benefits of flu/Tdap vaccination 
Benefits of flu/Tdap vaccination within the pregnant population (e.g., 
benefits of passive immunity) 
Communication 
Understanding and applying a “Presumptive Recommendation” 
Brief introduction to Motivational Interviewing (MI) and application 
with vaccine hesitant pregnant women 
Communication Tips & Tricks: Responses to common 
concerns/questions about vaccines 
Optimizing Office 
Workflow 
Optimal methods and organizational/tracking tools 
 Pre-visit planning and nurse prompts 
 EHR prompts (Checklists/Text Fields) 
 Standing orders 
 Vaccine rate feedback 
 Vaccine champion and utilization of PDSA cycles 
 Optimal vaccine documentation (e.g., vaccines given offsite) 
Putting it all together 
 “Teamwork”- Ensure entire multidisciplinary team trained in 
utilization of new methods/tools 
 “Consistency”- Ensure all tools/methods used consistently 
 “Adjust as Needed”- Use feedback reports & PDSA cycles to 
reflect on progress and fine-tune accordingly
 
 
 

Page 11 of 34 
Version Date: 6.22.2022 
 
o E-learning platform: Module content will first be disseminated to practice 
site personnel via an online e-learning platform (i.e., Articulate). Study 
coordinators will obtain e-mail addresses for the site personnel from either 
the health system liaison or the associated vaccine champion, and will then 
e-mail a training content link to all appropriate site providers, nurses, and 
staff. Attached to this initial e-mail will be an Information Sheet 
describing the study goals and training phase details, as well as their 
involvement throughout the duration of the 12-month QI intervention (see 
“TrainingPhaseEmail…” and “InformationSheetTrainingPhase…” 
documents; also see “Consent Process” section for more detail). After 
reviewing the Information Sheet and discussing any questions/concerns 
with study staff, site personnel will be asked to access and complete the 
training lessons prior to their scheduled site meeting.  
 
Embedded at the conclusion of the online training module, will be a link to 
a brief REDCap survey (see “TrainingModuleCompletionSurvey…” 
document) to obtain information regarding those who have fully 
completed the training. A list of practice personnel who have successfully 
completed the online training will be stored in a password-protected 
Excel/Word file by each study coordinator. The names will not be linked 
with any other study data. The list will be accessible only by study staff 
for the purposes of confirming those who should receive CME/CNE 
credit.  
 
Note: For intervention sites that have completed the e-learning training 
during the first half of the calendar year (i.e., prior to July), site personnel 
may be asked to additionally complete an abbreviated “flu booster” e-
learning module just prior to flu season of that same year (e.g., in 
Aug/Sept). This abbreviated version (approx. 15 minutes to complete) will 
contain content (see table 3) only applicable to influenza.  
 
o Site Meeting: Following the online training, all site personnel will meet 
with study staff either in-person or via a virtual Zoom meeting. The 
meetings (both virtual and in-person) will be audio and video recorded 
using a laptop or handheld recording device (in-person meetings) or via 
the Zoom recording option (virtual meetings) (see “Audio/Video 
Recording” and “Privacy and Confidentiality” sections for more detail). 
These archived video files will allow for the review of meeting content by 
site personnel who were unable to attend the meeting live, or by those who 
simply wish to review the meeting discussion for a second time. 
 
During the site meeting, study staff will briefly review online training 
content, allow time for the sharing of comments/questions from site 
personnel, and lead a discussion regarding the application of training 
content ideas specifically to that practice site. During this meeting, study 
staff will also present site personnel with their current immunization rates. 

Page 12 of 34 
Version Date: 6.22.2022 
 
Each designated vaccine champion will track meeting attendance, and a 
list of practice personnel who have attended the site meeting will be e-
mailed to the study coordinators and stored in a password-protected 
Excel/Word file. The names will not be linked with any other study data. 
This list will be accessed for the purposes of confirming those who should 
receive MOC credit. 
 
Note: For intervention sites that have completed their site meeting during 
the first half of the calendar year (i.e., prior to July), site personnel may be 
asked to participate in a second abbreviated “flu booster” site meeting just 
prior to flu season of that same year (e.g., in Aug/Sept). This abbreviated 
meeting (approx. 15 minutes) will discuss content (see table 3) only 
applicable to influenza.  
 
o MOC/CME/CNE Credit: To encourage the completion of online trainings 
and the attendance at site meetings, providers and nurses will be offered 
CME/CNE credit for completing learning modules, and MOC/QI credit 
for attending office systems change meetings and reviewing practice rates.  
 
Step 2) Monitor adoption of VAX-MOM training content at randomized practices: 
 
To monitor the adoption of VAX-MOM training content (changes in communication 
techniques, workflow optimization, etc.) at each practice site, each vaccine champion will 
participate in the following activities in an ongoing basis: 
 
1. Lead a discussion regarding immunization rate feedback with practice providers, 
nurses, and staff on a bi-monthly basis (every-other month). The champion will 
monitor the site personnel attending each feedback discussion as well as their 
role (e.g., Ob/Gyn, nurse, desk staff, etc.) within the practice.  They will e-mail 
study staff with a list of individual attendees for the purposes of tracking those 
receiving MOC credit. They will also report aggregate numbers on a monthly 
basis using the “Practice Time & Cost Survey.”  
 
The site-specific flu and Tdap vaccination rates will be obtained from either: 1) 
the EHR report builders for each health system (possible for all 4 health 
systems), or 2) manual chart review (possible for only NY health systems). 
Obtained rates will be sent either via secure e-mail directly to the study 
coordinator in each corresponding state, or uploaded to the secure shared Box 
folder. The study coordinator will then disseminate the vaccination rates for each 
site to all corresponding site personnel (see “RateFeedbackTemplate…” 
document).  
 
2. Develop PDSA cycles/goals on a monthly basis utilizing knowledge about current 
workflow efficiency and data from vaccine rate reports. Vaccine champions will 
utilize the PDSA template (see “PDSASurvey…” document) to guide progress 
with their respective teams. The template will help them decide upon specific 

Page 13 of 34 
Version Date: 6.22.2022 
 
intervention activities appropriate for their setting, track those involved with each 
activity, compare results from the activity to previous performance, and focus on 
changes that may need implementation during future cycles. PDSA cycle logs 
will be shared with study staff via a secure REDCap survey (same 
“PDSASurvey…” document formatted on the REDCap platform) sent to the 
vaccine champion each month by the appropriate study coordinator. 
 
During the final quarter of the 12-month intervention phase, a study team 
member will observe a subset of NY intervention sites in-person in order to 
collect qualitative data regarding workflow and communication techniques as 
described in the sites’ submitted PDSA cycle logs. Specifically, the study team 
member will observe select OB/GYN practice nurses and staff for 1-2 days per 
site, and will document the ways in which the targeted intervention techniques 
are implemented within the practice (see “InterventionObservationNotes…” 
document). Observation notes will then be compared with submitted PDSA cycle 
log information and analyzed for consistency.  
 
Of Note: Although study staff will observe OB/GYN practice nurses and staff 
during patient encounters, practice patients will NOT be the target of the 
observation, and no identifying patient information will be collected. Rather, the 
focus of observation will be the select practice personnel only. Any patient-level 
information recorded during the observation will be done so in a generic manner 
such that the patient could not be identified on the basis of the notes (e.g., 
“…when patient indicated that they did not want a Tdap vaccine, the nurse 
stated, ‘The doctor may talk to you more about that when she comes in.’”). The 
study staff will only observe the initial nurse/staff interaction with the patient, 
and will not continue to observe once the provider (doctor or midwife) enters for 
the visit.          
 
3. Complete the “Practice Time/Cost Survey” on a monthly basis (see step 4 for 
further detail).  
 
To monitor study staff effort during this phase, study staff will participate in the 
following activity in an ongoing basis: 
 
4. Complete the “Study Staff Time/Cost Survey” on a monthly basis (see step 4 for 
further detail). 
 
 
 
Step 3) Compare intervention vs. control practices: 
 
Using the “RE-AIM” framework (see Table 4) we will compare intervention practices to 
control practices using multiple data sources. We will focus on one primary outcome and 
multiple secondary outcomes. 
 
Primary Outcome: Following the conclusion of aim/phase 2 (12-month duration), we will 
assess the “Effectiveness” of the intervention by comparing intervention group flu and 
Tdap vaccination rates against control group flu and Tdap vaccination rates. 
 

Page 14 of 34 
Version Date: 6.22.2022 
 
Secondary Outcomes: Following the conclusion of aim/phase 2 (12-month duration), we 
will assess the remaining “RE-AIM” domains including “Reach,” “Adoption,” 
“Implementation,” and “Maintenance.” 
 
 
*See Step 4 for more details 
 
 
Details for EHR data extraction: 
 
 Measure Description (also see “EHRDataExtraction …” document) 
We will work with our experienced UCLA analysts (initial planning of required data 
fields and subsequent analysis of data output) and health system report builders 
(algorithm development) to extract EHR data for the purposes of evaluating basic flu and 
Tdap vaccination rates and additional secondary outcomes (see Table 5).  
 
 
 
Table 4:  Summary of Outcome Measures and Tools
RE-AIM 
Category 
Outcome Measure(s) 
Data Source
Reach 
• Number of patients seen in the practice within study period. 
 
• Number of providers/nurses/staff completing the training 
modules. 
EHR data & 
training 
attendance 
records 
Effectiveness 
• Primary Outcome: Rate of flu and Tdap vaccination  
• Secondary Outcomes:  Flu and Tdap vaccination rates by 
subgroups including (i)  insurance groups, (ii) race/ethnicity, (iii) 
number of pregnancy, (iv) flu vaccine in prior year 
EHR data 
Adoption 
• Missed opportunities by (i) provider characteristics and (ii) 
patient characteristics 
• Number and proportion of personnel involved in the VAX-MOM 
office changes (receiving MOC credit, attending in-office 
meetings) 
Provider 
Surveys, EHR 
data, training 
attendance 
records 
Implementation 
• Provider and office staff perceptions of feasibility, acceptability, 
barriers and facilitators to implementation, adherence to 
intervention, perceived time and cost, and impact on patient flow. 
• Missed opportunities for flu/Tdap vaccination. 
• Perceived strength of vaccine recommendations. 
• Perceived adherence to staff checking whether flu/Tdap is due. 
• Costs of implementing interventions 
 
Provider 
Surveys, EHR 
data & 
*Time/Cost 
Survey 
Maintenance 
• Sustainability 
Provider 
Survey 

Page 15 of 34 
Version Date: 6.22.2022 
 
 
 Measure Procedures 
 
Each health system (URMC, RRH, UCLA, and LADHS) will have its own “report 
builder.” The report builder will be in charge of collecting the electronic health record 
(EHR) data from their own health system and putting it into a format (report) that can be 
easily understood and analyzed by study staff. 
 
Bi-Monthly Feedback Reports: The report builder for each health system will create 
vaccine rate feedback reports utilizing extracted EHR data for intervention sites on a bi-
monthly basis. These reports will be sent either via secure e-mail directly to the study 
coordinator in each corresponding state, or uploaded to the secure shared Box folder. The 
study coordinator will then disseminate the vaccination rates for each site to all 
corresponding site personnel. 
 
For NY health systems, in the event an automated EHR feedback report cannot be 
generated in time for any given month, the identified vaccine champions or study 
personnel will estimate flu and Tdap vaccine rates via manual chart review. All vaccine 
champions and/or study personnel conducting manual chart review will have completed 
all appropriate eRecord training prior to accessing patient charts. Of those patients who 
delivered during the identified month, at least 20% will be reviewed. For each chart 
reviewed, the vaccine champion or study personnel will document the receipt (or lack 
thereof) of influenza and Tdap immunization as well as Covid vaccination when possible, 
and will share results with the study team via secure e-mail.  
 
Final EHR Data Analysis: Additionally, at the conclusion of aim/phase 2 (12-month 
duration) each health system report builder will create a final EHR report for both the 
intervention sites as well as the control sites capturing updated baseline information from 
2019 and 2020 (now needed due to COVID pause), as well as detailed intervention 
information from 2021 and 2022 (information indicated in Table 5 that was not captured 
during monthly reports). These final reports will be uploaded to a shared folder on the 
secure Box platform, accessible by select study staff (EHR report builders, Drs. Cindy 
Rand and Peter Szilagyi, and Study Coordinators) and the UCLA data analysis team. This 
report will allow for the comparison of vaccination rates and secondary outcomes 
Table 5. EHR Data Extraction Categories 
Primary 
 Flu vaccines received during flu season (Sept-April)  
 Tdap vaccines received during 27-36 weeks of pregnancy 
Secondary 
 Vaccine refusal (as documented in EHR) 
 Missed opportunities for flu and Tdap  
 Flu vaccine in prior season 
 Covid vaccination
Patient Demographics 
 Race/ethnicity    
 Age               
 Insurance  
 Language     Parity 
Practice Demographics 
 Site 
 Provider type (resident, midwife, MD) 

Page 16 of 34 
Version Date: 6.22.2022 
 
between intervention and control sites. In the event that the final automated EHR reports 
cannot be generated in a timely manner by the report builders, study personnel and/or a 
medical student (after being added to the study protocol) will conduct manual EHR chart 
reviews to collect the data. They will be appropriately trained in the EHR system 
requiring access, prior to conducting the chart reviews.  
 
Details for Provider & Nurse Survey: 
 
 Measure Description 
The Provider & Nurse Survey (see “ProviderNurseSurvey…” document) is an 86-
question survey consisting of Likert scale, multi-option and open-ended questions. The 
survey takes approximately 20 minutes to complete. The survey assesses opinions 
regarding flu, Tdap and COVID-19 vaccinations, patient refusal of those vaccinations, 
barriers to those vaccinations, policies and procedures within provider offices as they 
relate to vaccinations, and basic demographic information. 
 
 Measure Procedures 
At the conclusion of aim/phase 2, all providers (MD, residents, NPs, PAs and CNMs) and 
select nurses from both the intervention and control sites will be asked to complete the 
survey. Surveys will be e-mailed (see “ProviderNurseSurveyInviteEmail…” document) 
to providers and select nurses by the appropriate Study Coordinator, using the secure web 
application REDCap. E-mail addresses will have previously been obtained from the 
health system liaisons or vaccine champions and uploaded to the REDCap platform by 
the Study Coordinators. An Information Sheet (see “Consent Process” section for more 
details) will be embedded in each e-mail. 
 
After initial survey distribution, if the subject does not click on the survey link embedded 
within the e-mail, the REDCap platform will automatically disseminate a reminder e-mail 
to the subject (identical content as initial invite e-mail). They will receive this reminder e-
mail every 3 days, up to a maximum of 5 reminders. Once the subject clicks on the 
survey link, they will be redirected to the secure survey site where they may answer 
survey questions at their own pace. Subjects may stop and restart at any time, and may 
skip any questions they do not wish to answer. At the conclusion of the survey, they will 
receive an automated confirmation message to assure them that the process is complete. 
 
Details for Practice Culture Survey: 
 
 Measure Description  
The Practice Culture Survey (see “PracticeCultureSurvey…” document) is a 15-item 
Likert scale survey which takes approximately 5 minutes to complete. The survey 
assesses opinions regarding communication, decision making, stress/chaos and leadership 
domains within a practice site. 
 
 Measure Procedures 
At the conclusion of aim/phase 2, select providers (MD, residents, NPs, PAs and CNMs), 
nurses and staff from the intervention sites will be asked to complete the survey. Surveys 
will be e-mailed (see “PracticeCultureSurveyInviteEmail…” document) to personnel by 

Page 17 of 34 
Version Date: 6.22.2022 
 
the appropriate Study Coordinator, using the secure web application REDCap. E-mail 
addresses will have previously been obtained from the health system liaisons or vaccine 
champions and uploaded to the REDCap platform by the Study Coordinators. An 
Information Sheet (see “Consent Process” section for more details) will be embedded in 
each e-mail. 
 
After initial survey distribution, if the subject does not click on the survey link embedded 
within the e-mail, the REDCap platform will automatically disseminate a reminder e-mail 
to the subject (identical content as initial invite e-mail). They will receive this reminder e-
mail every 3 days, up to a maximum of 5 reminders. Once the subject clicks on the 
survey link, they will be redirected to the secure survey site where they may answer 
survey questions at their own pace. Subjects may stop and restart at any time, and may 
skip any questions they do not wish to answer. At the conclusion of the survey, they will 
receive an automated confirmation message to assure them that the process is complete. 
 
Step 4) Evaluate cost of the VAX-MOM intervention: 
 
Practice Time & Cost Survey (see “PracticeTime&CostSurvey…” document): A survey 
aimed at evaluating the amount of time and money devoted to VAX-MOM intervention 
activities will be sent by the study coordinators via REDCap to the vaccine champion for 
each site on a monthly basis. The survey will be completed by the vaccine champion 
using their best time/cost estimates pertaining to the month prior. 
 
Study Staff Time & Cost Survey (see “StudyStaffTime&CostSurvey…” document): A 
survey aimed at evaluating the time and cost efforts of VAX-MOM study staff will be 
sent to study personnel on a weekly basis via REDCap. 
 
Final Time & Cost Analysis:  
Upon completion of aim/phase 2 we will conduct a final time/cost analysis. Our two-fold 
cost measures are (a) total intervention cost to implement the VAX-MOM program 
aggregated at the four health-system level under a cost analysis and (b) cost per 
additional flu or Tdap vaccination under a subsequent cost-effectiveness analysis. 

Page 18 of 34 
Version Date: 6.22.2022 
 
Figure 2: Timeline of Aim/Phase 2 Study Procedures 
 
 
 
 

Page 19 of 34 
Version Date: 6.22.2022 
 
5. SUBJECT POPULATION 
 
During aim/phase 2, we will utilize subjects from a total of 37 participating practices across 
both NY and CA. “Subjects” refers to both a) the providers, nurses, and staff at the participating 
OB/GYN practice sites and b) data from practice patients (pregnant women) gathered via 
electronic health record (EHR). A list of all participating practices and corresponding 
demographic information is below (Table 6). 
 
Table 6: Participating Practice Sites and Demographic Information (Total Subject Pool) 
 
NEW YORK PRACTICES: 
URMC SITES 
Participating Practice 
Total Number of Providers  
Annual Deliveries  
Women’s Health Practice at Lattimore 
35 
800 
University OB/GYN 
9 
100 
Strong Perinatal Associates  
12 
200 
Community OB  
18 
300 
UR/HH Midwifery 
16 
500 
Highland Women’s Health 
2 
200 
URMC Insurance (% Private/Public/Other) 
45/55/5 
URMC Race/Ethnicity (% White/ Black/Hispanic) 
 65/25/10 
RRH SITES 
Participating Practice 
Total Number of Providers  
Annual Deliveries  
The Women's Center at RGH 
10 
511 
The Women's Center at Clinton 
4 
130 
The Women's Center at Alexander Park 
6 
144 
Clifton Springs Hospital & Clinic 
3 
13 
The Women's Center at Newark 
7 
415 
Bay Creek Midwifery 
4 
110 
The Women's Center at Victor 
4 
17 
RRH OBGYN at Henrietta 
3 
0 
Finger Lakes Medical Associates OBGYN-
Geneva
5 
256 
Finger Lakes Medical Associates OBGYN-Penn 
Yann
1 
16 
Unity OBGYN at Brockport 
5 
392 
Unity OBGYN at Ridgeway 
6 
353 
Unity OBGYN at Park Ridge 
3 
309 
Unity OBGYN at Parkway 
3 
298 
Unity OBGYN at Clinton Crossings 
6 
143 
Unity OBGYN at West Main 
3 
128 
RRH Insurance (% Private/Public/Other) 
55/45/5 
RRH Race/Ethnicity (% White/ Black/Hispanic) 
65/15/20 

Page 20 of 34 
Version Date: 6.22.2022 
 
CALIFORNIA PRACTICES: 
UCLA SITES 
Participating Practice 
Total Number of Providers  
Annual Deliveries  
Ronald Reagan OB 
13 
1792 
UCLA Faculty Practice. OBGYN at West Medical 
8 
300 
Santa Monica Faculty Practice 
8 
1500 
Midwife Clinic  
5 
157 
Santa Clarita  
1 
80 
West Lake/Thousand Oaks 
4 
472 
South Bay (Torrance) 
2 
180 
UCLA Insurance (% Private/Public/Other) 
 89/1/10 
UCLA Race/Ethnicity (% White/ Black/Hispanic) 
52/7/13 & 14% Asian 
LADHS SITES 
Participating Practice 
Total Number of Providers  
Annual Deliveries  
Olive View 
5 
≈800 
Harbor UCLA 
7 
≈1000 
MLK Outpatient Center 
3 
≈450 
Hubert Humphrey Comprehensive Health Center 
3 
≈450 
El Monte Comprehensive Health Center 
2 
≈250 
High Desert Regional Health Center 
1 
≈150 
Los Angeles County USC Medical Center 
9 
≈1000 
Hudson Comprehensive Health Center 
3 
≈500 
LADHS Insurance (% Private/Public/Other) 
 20/75/5 
LADHS Race/Ethnicity (% White/ Black/Hispanic) 
15/25/55 & 5% Asian  
 
 
Inclusion of Women and Minorities 
Every effort will be made within this project to ensure that women, racial and ethnic 
minorities will be included in all aspects of the research. Practices selected from two states 
include a broad spectrum of size, regional location as well as public and private practices. 
These variables will ensure racial/ethnic diversity among study subjects that will mirror the 
state of New York and California’s racial and ethnic distribution as shown in the enrollment 
table. In New York and California, an estimated 57% of Obstetrician/Gynecologists are 
female. Therefore, we anticipate a slightly higher percentage of females among the provider 
study subjects. Patient study subjects will all be female.  
 
Inclusion of Pregnant Women 
Although the research project involves OB/GYN practice sites, study staff will not have 
direct contact with pregnant patients, nor do study efforts place the patients’ pregnancy at 
risk. All pregnant patients, at both the intervention and control sites, will continue to receive 
their expected standard of care for the entirety of their pregnancy. 
 

Page 21 of 34 
Version Date: 6.22.2022 
 
Inclusion of Children 
Pregnant patients receiving care at eligible practices will be considered part of the study 
population. As the Advisory Committee on Immunization Practices (ACIP) recommends 
influenza and Tdap vaccination during pregnancy, the intervention, performed by primary 
care physicians within the study clinics, will include pregnant patients that are aged less than 
18 years of age.   
 
Inclusion of Employees 
Employees of URMC, RRH, UCLA and LADHS will be included as subjects. All 
Information Sheets disseminated to study subjects during this study phase will emphasize 
that the decision to participate will have no impact upon: performance evaluations, job 
advancement, or the loss/gain of benefits (e.g., salary increases, time off).  
 
6. INCLUSION/EXCLUSION CRITERIA & RECRUITMENT METHODS 
 
All Participating Sites (Intervention & Control) 
During aim/phase 1 of the study, health systems (URMC/RRH/UCLA/LADHS) within 
each state were chosen because a) their size allowed for an ample sampling of patients 
and practitioners (i.e., large subject pool) and b) they allowed for a broad spectrum of 
geographic and socio-economic diversity (e.g., privately and publicly insured, ethnic 
multiplicity, rural and urban locations, etc.). Prior to the start of the project, Drs. Rand 
and Szilagyi contacted leadership within each health system, and letters of support were 
obtained (see attached Letters of Support for each system).  
    
Intervention Sites 
As described in the “Randomization” subsection of the “Study Design & Procedures” 
portion of the protocol, study sites from the larger pool were selected to be part of the 
“intervention group” using specific constrained randomization procedures within each 
health system (see section 4 of protocol for more details). All remaining sites were 
allocated to the “control group.” 
 
Practice-Level Personnel 
 
Criteria/Recruitment Method for Vaccine Champion: 
At each site, the affiliated medical director will select one “vaccine champion” for the 
practice. The vaccine champion will be selected due to their perceived ability to play a 
lead role throughout the study intervention, including their ability to complete all 
monthly and bi-monthly tasks listed in “Step 1” of the “Study Design and Procedures” 
section. The position of vaccine champion is voluntary, and if they do not wish to accept 
the role, they may decline and another individual will be identified by the affiliated 
medical director. 
 
Criteria/Recruitment Method for Remaining Site Personnel: 
By default (and with the support of health system and practice site leaders) all remaining 
providers, nurses, and practice staff employed by the practice location, are able to 
participate in the quality improvement initiative. 

Page 22 of 34 
Version Date: 6.22.2022 
 
Patient-Level Study Subjects 
 
Criteria for EHR Data from Practice Patients (Pregnant Women): 
Baseline vaccine rates for each practice site were established during aim/phase 1 of the 
study, and contained EHR information from practice patients who were identified as 
having a live birth within the time period of summer 2018 to summer 2019. Due to a 
COVID-19 related pause in study activity, updated baseline vaccine rates will also be 
obtained during aim/phase 2 for the 2019-2020 time period. All subjects are female and 
some are <18 years of age.  
 
To establish vaccine rates for the ongoing feedback reports, EHR information will be 
collected bi-monthly (every 2 months) from intervention sites during the entirety of the 
intervention phase (see timeline depicted in Figure 2). EHR information will reflect 
practice patients who are identified as being eligible for flu or Tdap vaccine within the 
specified 2-month time period. Again, all subjects will be female and may be <18 years 
of age.  
 
To establish the final vaccine data reports (for intervention vs. control comparison), 
detailed EHR information (see Table 5) will be collected from both intervention and 
control sites for the 2021-2022 time period. 
 
7. CONSENT PROCESS 
 
Health System, Practice Site, & Site Personnel Level  
 
Health System & Practice Site Consent: 
 
Health System support (and subsequently site-level support) was previously obtained 
during aim/phase 1 of the study, as described in the “Inclusion/Exclusion Criteria & 
Recruitment Methods” section of the protocol.  
 
Site Personnel Consent: 
 
 
Delivery of Training Content & Involvement in 12-Month QI Intervention  
(All site personnel: completion of e-learning module, attendance of follow-up site 
meeting, and participation in monthly/bi-monthly discussions of rate feedback and 
PDSA cycles)  
 
This study involves commonly accepted quality improvement efforts aimed to 
improve upon the standard of care in medical settings, such as the completion of 
e-learning modules and in-person trainings specifically related to practice goals, 
the review of practice metrics (vaccine rates) by site personnel, discussion of 
office workflow procedures/efficacy, and trainings in optimal communication 
techniques.  
 
We are therefore seeking a waiver of documentation of consent and a waiver of 
HIPAA authorization for the training portion of the study. More specifically, we 
are requesting this waiver because: a) the training portion of the study is no 

Page 23 of 34 
Version Date: 6.22.2022 
 
greater than minimal risk, b) the purpose of the study/intervention and a basic 
framework of the trainings will be clearly outlined for participants in the 
Information Sheet (see “InformationSheetTrainingPhase…” document) e-mailed 
to them at the onset of the intervention phase, c) ample time will be given to 
personnel to review the Information Sheet in full and ask questions of study staff 
regarding the quality improvement effort and/or specific tasks involved in the 
training phase, and d) PHI will not be reused or disclosed to any other person or 
entity except (i) as required by law, (ii) for authorized oversight of the research 
study, or (iii) for other research for which the use or disclosure of PHI would be 
permitted by the HIPAA Privacy Rule. 
 
We will attach an Information Sheet (cited above) to the first e-learning invite e-
mail, describing the study goals and training phase details, as well as site 
personnel involvement throughout the duration of the 12-month QI intervention 
(e-mailed to site personnel just prior to the training phase). Study staff will field 
any and all questions from potential study subjects prior to their module training 
and subsequent site meeting. 
 
Assessment of Intervention Adoption: Surveys & Rate Feedback  
(Selected Vaccine Champions: completion of bi-monthly rate feedback 
discussions and monthly PDSA cycles and Practice Time & Cost Surveys) 
 
The assessment of intervention adoption by study sites involves commonly 
occurring procedures within a medical setting, including the ongoing review of 
practice metrics (vaccine rate feedback sessions), the development and assessment 
of practice goals (PDSA Cycles), and the evaluation of personnel efforts to 
achieve these goals (Time & Cost Survey). 
 
We are therefore seeking a waiver of documentation of consent and a waiver of 
HIPAA authorization for the training portion of the study. More specifically, we 
are requesting this waiver because: a) the assessment portion of the study is no 
greater than minimal risk, b) the purpose of the study/intervention and a 
description of the assessment tasks will be clearly outlined for vaccine champions 
in the Information Sheet (see “InformationSheetVaccineChampion…” document) 
e-mailed to them at the onset of the assessment phase, c) ample time will be given 
to each vaccine champion to review the Information Sheet in full and ask 
questions of study staff regarding the quality improvement effort and/or specific 
tasks involved in the assessment phase, and d) PHI will not be reused or disclosed 
to any other person or entity except (i) as required by law, (ii) for authorized 
oversight of the research study, or (iii) for other research for which the use or 
disclosure of PHI would be permitted by the HIPAA Privacy Rule. 
 
As soon as they are selected, each identified vaccine champion will be e-mailed 
(see “VaccineChampionEmail…” document) an Information Sheet (cited above) 
by their affiliated study coordinator, outlining the basic study goals as well as 
their involvement in tasks throughout the training and intervention adoption 

Page 24 of 34 
Version Date: 6.22.2022 
 
phases of the study. After the Information Sheets have been disseminated, study 
coordinators will conduct a follow-up phone call with each vaccine champion, 
reviewing their role within the project and fielding any and all questions they may 
have. The position of vaccine champion is voluntary, and if they do not wish to 
accept the role, they may decline and another individual will be identified by the 
affiliated medical director. 
 
Assessment of Intervention Adoption: In-Person Observation  
(Select Nurses and Staff: observation of workflow and communication techniques) 
 
The in-person observation of intervention adoption involves commonly occurring 
procedures within a medical setting, including the ongoing documentation and 
review of daily workflow and communication procedures.  
 
We are therefore seeking a waiver of documentation of consent and a waiver of 
HIPAA authorization for the in-person observation portion of the study. More 
specifically, we are requesting this waiver because: a) the observation portion of 
the study is no greater than minimal risk, b) the purpose of the observation will be 
clearly outlined for nurses and staff in the Information Sheet (see 
“InformationSheetInPersonObs…” document) e-mailed to them prior to the 
scheduled observation, c) ample time will be given to each nurse/staff to review 
the Information Sheet in full and ask questions of study staff regarding the 
observation, and d) PHI will not be reused or disclosed to any other person or 
entity except (i) as required by law, (ii) for authorized oversight of the research 
study, or (iii) for other research for which the use or disclosure of PHI would be 
permitted by the HIPAA Privacy Rule. 
 
Prior to in-person observation, the study coordinator will contact each site’s 
vaccine champion to determine 1-2 days that would be optimal for study staff to 
be onsite. Once the observation schedule has been determined, all nurses and staff 
who are scheduled to work on the specified day(s) will be e-mailed an 
Information Sheet (see “InformationSheetInPersonObs…” document) by the 
study coordinator, which they may review at their own pace prior to the 
observation date. They may e-mail the project coordinator with any questions, or 
bring questions to the attention of the study staff on the scheduled visit day prior 
to the start of any observation. The content of the Information Sheet will be 
reviewed again by study staff with practice nurses/staff on the visit day prior to 
the start of any documented observation (any nurses/staff not previously known to 
be working that day will be given a hard copy of the Information Sheet upon 
arrival by the study staff and given ample time to review). 
 
As stated previously, although study staff will observe OB/GYN practice nurses 
and staff during patient encounters, practice patients will NOT be the target of the 
observation, and no patient information (in either individual or aggregate form) 
will be collected. Rather, the focus of observation will be the select practice 
personnel only. Information recorded during the observation session will be 

Page 25 of 34 
Version Date: 6.22.2022 
 
written in a generic manner such that specific nurses/staff/patients will not be 
identifiable. Notes will describe the general workflow processes and 
communication methods used, rather than specific unique behaviors or exact 
dialogue. For example, study staff may document “…when patient declined to 
receive the Tdap vaccine, the practice nurse gave them an informational handout 
and told them the doctor may speak with them further.” The study staff will only 
observe the initial nurse/staff interaction with the patient, and will not continue to 
observe once the provider (doctor or midwife) enters for the visit. Although the 
patient is not the study subject, prior to each observation, study staff will explain 
to the patient in simple language the purpose of the observation (i.e., to take notes 
regarding the nurse/staff work routine), and any patient that communicates 
discomfort or preference to not be observed, will not be observed.           
 
Provider & Nurse Survey and Culture Survey Dissemination 
 
We are seeking a waiver of documentation of consent and a waiver of HIPAA 
authorization for the Provider & Nurse Survey and Culture Survey portion of the 
study. We are doing so because: a) the surveys are no greater than minimal risk (a 
brief online survey with unobtrusive questions), b) a detailed Information Sheet 
(see “InformationSheetProviderNurseSurvey…” and 
“InformationSheetPracticeCultureSurvey…” documents) will be made available 
to each subject prior to the start of the survey, c) ample time will be given to each 
subject to consider participation (subjects are notified of the survey via e-mail and 
can review the information sheet and/or ask questions of study staff for as long as 
necessary before deciding about survey completion), d) no identifiers will be 
included on the survey form (only subject IDs), and the separate document linking 
subject name to subject ID will only be made available to a limited number of 
study staff, will be kept under double-locked conditions, and will be destroyed 
three years after study completion, and e) PHI will not be reused or disclosed to 
any other person or entity except (i) as required by law, (ii) for authorized 
oversight of the research study, or (iii) for other research for which the use or 
disclosure of PHI would be permitted by the HIPAA Privacy Rule. 
 
Patient (EHR) Level  
 
EHR Data Extraction: We are seeking a waiver of consent and a waiver of HIPAA 
authorization for the EHR data extraction portion of the study. We are doing so because: 
a) we believe that this research cannot be practicably conducted without such a waiver, as 
we cannot feasibly obtain consent and HIPAA authorization from all potential subjects 
(10,000+ in total subject pool), b) site liaisons and report builders have routine access to 
patient records, c) a minimal number of patient-level data fields will be extracted from 
the EHR (just enough to complete analysis) d) a plan to protect EHR data during all data 
transfers will be implemented (see “Privacy & Confidentiality” section for details), e) any 
individual-level data will be destroyed three years after study completion, and f) PHI will 
not be reused or disclosed to any other person or entity except (i) as required by law, (ii) 

Page 26 of 34 
Version Date: 6.22.2022 
 
for authorized oversight of the research study, or (iii) for other research for which the use 
or disclosure of PHI would be permitted by the HIPAA Privacy Rule. 
 
8. AUDIO/VIDEO RECORDINGS  
 
All site meetings (both virtual and in-person) held during the training phase may be audio 
and/or video recorded using a laptop or handheld recording device (in-person meetings) or 
via the Zoom recording option (virtual meetings) (see “Privacy and Confidentiality” section 
for more detail). These archived audio/video files will allow for the review of meeting 
content by site personnel who were unable to attend the meeting live, or by those who simply 
wish to review the meeting discussion for a second time. 
 
The archived site meeting audio/video files will be stored on a secure Box platform, 
accessible only to study staff and site personnel from the corresponding practice site (each 
site will have their own Box folder ensuring meetings from other sites are not viewable to 
outside personnel). Invites to the secure Box folder will be e-mailed from study coordinators 
directly to site personnel. As soon as the video files have been uploaded to the secure Box 
platform, they will be deleted from the laptop or handheld recording device. Once the study 
has been fully completed, all video files will be deleted from the shared Box folder. 
 
9.   RISKS TO SUBJECTS 
 
Practice Level Risk: This project involves QI trainings/interventions which aim to improve 
upon the existing communication techniques and workflow procedures at each intervention 
practice site. Planned study trainings/interventions are widely accepted among medical 
practices (e.g., practice providers/nurses/staff attending an informational meeting regarding 
the risk of influenza in the pregnant population), and do not involve any novel high-risk 
changes to practice structure or procedures. Practice sites in the control group will continue 
to utilize “best-practice” guidelines. Thus, at the practice level, anticipated overall study risk 
is very low.  
 
Site Personnel & Patient Level Risk: The primary risk to site personnel and practice 
patients is the risk of a breach of confidentiality, though overall risk remains low.  
 
To reduce risk, personal information being collected from site personnel or practice patients 
(EHR data) has been minimized, and when possible is being collected in aggregate form.  
 
A number of policies, procedures, and technical safeguards (described in the “Privacy & 
Confidentiality” section of this protocol) will be in place to ensure that there is no breach of 
confidentiality as a result of this study.   
 
 
 
 
 
 
 

Page 27 of 34 
Version Date: 6.22.2022 
 
10.  POTENTIAL BENEFITS TO SUBJECTS 
 
Practice personnel will receive targeted training related to: influenza and pertussis disease as 
it pertains to the pregnant population, communication, and optimization of office workflow 
as it relates to the improvement of immunization rates within their practice. As such, practice 
personnel who participate in both the training and ongoing QI portions of the study will have 
the opportunity to apply for MOC, CME, or CNE credit. 
 
There are no anticipated benefits for practice patients beyond those inherent to the 
overarching study goals (i.e., an improvement in flu and Tdap immunization rates). 
 
11.  COSTS FOR PARTICIPATION 
 
There will be no costs incurred by participants.  
 
12.  PAYMENT FOR PARTICIPATION  
 
For completing each “PDSA Cycle” survey and “Practice Time & Cost” survey, each vaccine 
champion will receive $5. Payment will be in the form of a gift card or check, and will be mailed 
to the vaccine champion at the conclusion of the 12 month intervention phase. Payment made 
will be the cumulative total of all completed monthly surveys (12 PDSA surveys + 12 Time & 
Cost surveys = 24 total surveys), for a maximum payout of $120 (24 surveys x $5 = $120). 
Additionally, if manual chart review is necessary for any NY practice sites, each vaccine 
champion will receive $20 for each monthly review. As with the above listed surveys, payment 
will be in the form of a gift card or check, and will be mailed as a cumulative payment to the 
vaccine champion at the conclusion of the 12 month intervention phase. To receive payment, 
subjects will enter their home address information into a secure REDCap field at the conclusion 
of the last online survey. This address information will be accessible only by study staff, and will 
be deleted as soon as the payment has been mailed. 
 
For completing the “Provider & Nurse Survey” portion of the study, each subject will receive 
$40. For completing the “Practice Culture Survey” portion of the study, each subject will receive 
$10. For both surveys, payment will be in the form of a gift card or check, and will be mailed or 
e-mailed to the subject within 6 weeks of survey completion. Subjects will enter their home 
address and e-mail information into a secure REDCap field at the beginning of each online 
survey. This information will be accessible only by study staff, and will be deleted as soon as the 
payment has been sent. 
 
Additionally, subjects completing the above surveys will be entered into a drawing for a $75 gift 
card for each survey completed. Health systems will have the ability to opt out of this additional 
drawing incentive if it conflicts with their institutional policy. Disclosure of the approximate 
chances of winning, and when required per institutional policy, procedures for the inclusion of 
those who do not wish to complete the survey will be included in the survey invite e-mail and 
Information Sheets. Winners will be chosen at random using a secure computer-generated 
randomization program containing the names of all participating subjects. Winners will be 
notified within 48 hours of being selected. Payment will be sent to the winners using the contact 
information collected in the address/e-mail field of their REDCap survey.  

Page 28 of 34 
Version Date: 6.22.2022 
 
For serving as an intervention site, a single practice-level payment will be offered in the amount 
of $250. This payment will be in the form of a gift card or check, and will be disseminated to the 
vaccine champion from each intervention location. Payment may be used for any practice-wide 
item(s) deemed beneficial by the vaccine champion. If manual chart review is necessary at a 
practice site (and thus additional payment to the vaccine champion is made), the $250 practice-
level payment for that site will be reduced by the total payout amount for the manual chart 
review. 
 
13.  SUBJECT WITHDRAWALS 
 
During aim/phase 1 of this study (August 2019-July 2020) all health systems and affiliated 
practice sites agreed to participate in this quality improvement effort. While we do not anticipate 
a high level of subject/site withdrawals, any site wishing to end their involvement in the quality 
improvement initiative may do so at any time. Any practice personnel wishing to abstain from 
participation in quality improvement activities (trainings, ongoing meetings, adoption of new 
workflow procedures) are free to do so. Any vaccine champion who does not fully complete all 
monthly study activities during the 12 month intervention period, will still receive payment for 
all completed surveys prior to their withdrawal. All subjects (practice personnel) are free to 
discontinue participation at any time, without consequence.  
 
14.  PRIVACY AND CONFIDENTIALITY OF SUBJECTS AND RESEARCH DATA 
 
The following data collection and storage procedures will be implemented at each study site, to 
ensure that subject privacy and confidentiality are maintained throughout the entirety of the 
research process:  
 
Data obtained via e-mail and REDCap during the training phase: 
(Site personnel names collected during the e-learning modules and site meetings) 
 
Study coordinators will maintain a list of site personnel who have successfully completed 
their e-learning modules and who attended follow-up site meetings.  These names will be 
obtained via e-mails from the vaccine champions and/or REDCap surveys. The lists will be 
stored in password protected Excel/Word files on a password protected computer for the 
purposes of confirming requests for MOC, CME and CNE credit. No additional study 
information (e.g., future survey responses) will be contained within this file.  All files will be 
stored on a secure server within each university system (URMC/UCLA). These individual-
level training completion files will be deleted once they are no longer necessary (i.e., once all 
credit applications are in). Only aggregate data, not individual names, will be stored long-
term by study staff regarding the completion of online training content by site personnel (i.e., 
the total number of providers, nurses, and staff who did/didn’t complete the training). 
 
Data Obtained via REDCap during the intervention adoption & assessment phase: 
(PDSA cycles, Time & Cost Surveys, Provider & Nurse Survey, Practice Culture Survey) 
 
For study measures completed via a REDCap survey, all subject names and corresponding e-
mail addresses will first be collected by the study coordinators from either the previously 

Page 29 of 34 
Version Date: 6.22.2022 
 
identified health system liaisons (Courtney Olson-Chen for URMC, Caitlin Dwyer for RRH, 
Jessica Hsu UCLA and Judy Chen for LADHS) or the assigned vaccine champions (not yet 
determined) for each site. The names, e-mail addresses, site locations, and an assigned 
subject ID for each person, will be stored in a password protected Excel/Word file on a 
password protected computer. No additional study information (e.g., survey responses) will 
be contained within this file. All files will be stored on a secure server within each university 
system (URMC/UCLA).  
 
Once e-mail addresses have been obtained, each study coordinator will upload the e-mail 
addresses onto the secure REDCap platform. Surveys will automatically be disseminated to 
the appropriate subjects in the appropriate timeframe (weekly, monthly, bi-monthly) through 
previous REDCap programming completed by the study coordinator. Completed REDCap 
survey data will be exported in an Excel/Word file and stored under double-locked 
conditions on a secure server within each university system (URMC/UCLA), before transfer 
to the UCLA Health Sciences Box platform (a regulated platform frequently used by UCLA 
for data storage and transfer online) for analysis by the project’s UCLA data specialists.  
 
Surveys for which payment (gift card or check) is given will request home address 
information within the survey questionnaire. This information will temporarily remain on the 
secure REDCap platform and/or secure university server, and will be utilized by the study 
coordinators during the payment mailing process. All personal address information will be 
destroyed once payments for phase 2 have been fully completed.  
 
Data Obtained via EHRs: 
 
EHR data reports will be generated only by the specified EHR report builder for each health 
system. A minimal number of patient-level data fields (only those needed for analysis) will 
be extracted from the EHRs. For data verification purposes, identified study staff will check a 
small percentage of patient eRecord files to ensure the accuracy of generated EHR reports. 
The identified individuals will have access to EHR files and will complete all appropriate 
eRecord training prior to conducting data verification. All individual level data will be stored 
in encrypted form on the secure Box platform. Whenever possible, data will be stored in 
aggregate form. “Rate Feedback” reports received by site personnel will present 
immunization rate information in aggregate form only (no individual patient information), 
and the recipient of the report will be identified by a study ID# rather than a name. 
 
All Research Data: All individual-level research data (saved in encrypted form) will be 
stored on the secure UCLA Health Sciences Box platform (a regulated platform frequently 
used by UCLA for data storage and transfer online). Aggregate data will be stored on secure 
university servers and/or the secure Health Sciences Box platform. Analyzed data reports 
from the UCLA data specialist will be sent in a de-identified form via secure e-mail to the 
project PI for review. Any files linking subject name to subject ID# (e.g., REDCap survey 
files, personnel ID files for feedback reports) will be password protected and stored on secure 
university server systems accessible only by study staff. 
 
 

Page 30 of 34 
Version Date: 6.22.2022 
 
15. DATA / SAMPLE STORAGE FOR FUTURE USE 
 
Hard copy data: Due to the online (e.g., REDCap) nature of data collection during phase 2 of 
the study, no hard-copy documents will be created or stored.  
 
Electronic data: 
 
Site Meeting Attendance:  
Files containing site meeting attendance records will be stored in password protected 
Excel/Word files on password protected computers. No additional study information 
(e.g., survey responses) will be contained within these files. All files will be stored on a 
secure server within each university system (URMC/UCLA). Once all MOC, CME and 
CNE credit requests have been completed, attendance records will be destroyed.  
 
E-learning modules, PDSA cycles, Time & Cost Surveys, Provider & Nurse Survey, 
Practice Culture Survey: 
Survey data is stored immediately upon completion by the REDCap website. Study staff 
will then access the secure password-protected survey site to generate data reports when 
needed.  These data reports will be stored on password-protected computers within the 
locked offices of study staff (double-locked conditions) using the secure university 
servers. Any personal address information obtained from study subjects for the purposes 
of payment will be destroyed once payments for phase 2 have been fully completed. 
 
As we are seeking a waiver of documentation of consent and a waiver of HIPAA 
authorization for the survey portions of the study, there will be no need to store consent 
forms with subject names.   
 
EHR data output:  
Individual-level EHR data reports (saved in encrypted form) and aggregate EHR data 
reports will be stored on the secure UCLA Health Sciences Box platform (a regulated 
platform frequently used by UCLA for data storage and transfer online). Analyzed data 
reports from the UCLA data specialist will be sent in a de-identified form via secure e-
mail to the project PI for review.  All data will be stored 3 years beyond study 
completion. 
 
16. DATA AND SAFETY MONITORING PLAN 
 
This study presents no more than minimal risk, however there is still a risk to privacy and 
confidentiality. Prior to the start of aim/phase 1, the research team from both NY and CA met to 
create a data safety monitoring plan. PIs, team members, and representatives from each study 
clinic regularly discuss communication and action plans in the unlikely event that an adverse 
event occurs. Information on how to contact the study team via phone, mail or email is readily 
apparent to all participating providers and care team members. Though adverse events are not 
anticipated, should any occur they will be reported to the site PI (“team leader”) in each state and 
then to the IRB at the time of the event, and copies of all correspondence regarding the event 
with the IRB will be shared with the CDC, as needed. 

Page 31 of 34 
Version Date: 6.22.2022 
 
17. DATA ANALYSIS PLAN 
 
Primary outcomes (receipt of Flu and Tdap vaccines) are binary and our main explanatory 
variable will be an indicator for study arm. We will employ intent-to-treat analyses using 
generalized linear mixed models (GLMMs) with practice random effects, an approach 
recommended for group-randomized RCTs in which the goal is to estimate the causal effects of 
interventions on individuals, adjusted for clustering within groups. This method performs well in 
situations where the number of observations per cluster is large and for unequal cluster sizes. 
Models will assume a binomial distribution and a log link function in order to compare 
vaccination rates between study arms in terms of risk ratios. We will adjust for all practice-level 
variables included in the randomization balancing criterion, as well as patient-level race/ethnicity 
(and prior year vaccination status for the flu analysis). Hypothesis tests will be two-sided with 
alpha = 0.025, reflecting a 2-fold Bonferroni correction for the co-primary endpoints of Flu and 
Tdap vaccination status, ensuring a familywise type I error rate of 0.05. Analyses will be 
performed using SAS v. 9.4 (SAS Institute Inc., Cary, NC). 
 
Power Analysis: We assume national vaccine rates for flu (49%) and Tdap (54%). Adjusting 
for clustering of patients in practices, and assuming an intraclass correlation (ICC) of 1% 
(consistent with previous work), this sample size provides >80% power to detect an overall 
increase of 6.3 percentage point increase in vaccination rates between the QI intervention and 
control arms. This assumes a chi-squared test (a simplification of the planned mixed model 
analysis described above), a test-level alpha of 0.025, and a sample size of 5,000 pregnant 
women organized into 16 practices per study arm. 
 
Final Time/Cost Analysis:  
Upon completion of aim/phase 2 we will conduct a final time/cost analysis. Our two-fold 
cost measures are (a) total intervention cost to implement the VAX-MOM program aggregated at 
the four health-system level under a cost analysis and (b) cost per additional flu or Tdap 
vaccination under a subsequent cost-effectiveness analysis. To derive a policy implication 
regarding the sustainability of programs, we will estimate costs and ICER estimates from the 
health system perspective. 
 
Total Cost: To make these cost measures comparable to similar past interventions of 
reminders and educational programs, we will estimate cost with one-year time horizon, 
excluding the cost to purchase, store and administer vaccines. We will estimate the total 
intervention cost, summing non-personnel costs (e.g., EHR hardware, software, and materials) 
and personnel costs. The personnel costs will distinguish research costs from intervention costs 
(e.g., practice-level meeting and collecting EHR data). The dollar values of these personal costs 
will be calculated by multiplying “time efforts” (weekly reported by study personnel using a 
REDCap email survey) with the nationally representative “hourly-wage rates” by occupation 
codes of study personnel, derived from the Bureau of Labor Statistics.  
 
Cost-Effectiveness: We will develop a standard decision model for our cost-effectiveness 
analyses. The effectiveness measures are the rates of flu and Tdap vaccination estimated under 
Aim 2b.  
 

Page 32 of 34 
Version Date: 6.22.2022 
 
As explained above, this intervention cost will exclude the cost to purchase, store and 
administer vaccines, which is assumed to be identical among all practices. Applying the same 
assumption, the average cost for the control group (costControl) is zero. To make separate ICER 
estimates for flu and Tdap vaccination, the Tdap-vaccination-specific costStudy will be 
estimated during non-flu season (April to September) first and applied for calculating the flu-
vaccination-specific costStudy during a flu season (October to March), assuming the Tdap-
vaccination-specific costStudy is constant throughout a year. Using the developed standard 
decision model, we will conduct probabilistic analyses to generate point estimates and 95% CIs 
for ICERs and one-way sensitivity analyses to determine conditions for being lower than the 
thresholds of cost-effectiveness, e.g., healthcare-system based quality improvement interventions 
to improve flu vaccination uptake targeting general populations (median ICER $51 among 23 
interventions) and healthcare workers (median ICER $125 among 6 interventions) reported by a 
systematic review. 
 
Hyp. : Cost per additional flu or Tdap vaccination under the VAX-MOM is lower than past 
similar interventions.   
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Page 33 of 34 
Version Date: 6.22.2022 
 
18. REFERENCES 
 
1.  
Poehling KA, Edwards KM, Weinberg GA, et al. The underrecognized burden of influenza in 
young children. N Engl J Med. Jul 6 2006;355(1):31-40. 
2. 
Poehling KA, Szilagyi PG, Staat MA, et al. Impact of maternal immunization on influenza 
hospitalizations in infants. American journal of obstetrics and gynecology. Jun 2011;204(6 Suppl 
1):S141-148. 
3. 
Rasmussen SA, Jamieson DJ, Uyeki TM. Effects of influenza on pregnant women and infants. 
American journal of obstetrics and gynecology. Sep 2012;207(3 Suppl):S3-8. 
4. 
Rogers VL, Sheffield JS, Roberts SW, et al. Presentation of seasonal influenza A in pregnancy: 
2003-2004 influenza season. Obstetrics and gynecology. May 2010;115(5):924-929. 
5. 
Neuzil KM, Reed GW, Mitchel EF, Simonsen L, Griffin MR. Impact of influenza on acute 
cardiopulmonary hospitalizations in pregnant women. American journal of epidemiology. Dec 1 
1998;148(11):1094-1102. 
6. 
Cox S, Posner SF, McPheeters M, Jamieson DJ, Kourtis AP, Meikle S. Hospitalizations with 
respiratory illness among pregnant women during influenza season. Obstetrics and gynecology. 
Jun 2006;107(6):1315-1322. 
7. 
Centers for Disease C, Prevention. Maternal and infant outcomes among severely ill pregnant 
and postpartum women with 2009 pandemic influenza A (H1N1)--United States, April 2009-
August 2010. MMWR. Morbidity and mortality weekly report. Sep 9 2011;60(35):1193-1196. 
8. 
Newsome K, Alverson CJ, Williams J, et al. Outcomes of infants born to women with influenza 
A(H1N1)pdm09. Birth defects research. Jan 9 2019. 
9. 
CDC. Pertussis Incidence by Age Group and Year (1990-2016). .  
https://www.cdc.gov/pertussis/surv-reporting/cases-by-age-group-and-year.html. Accessed 
1/6/19. 
10. 
2017 Final Pertussis Surveillance Report. 2018; 
https://www.cdc.gov/pertussis/downloads/pertuss-surv-report-2017.pdf. Accessed 1/11/19. 
11. 
Nunes MC, Aqil AR, Omer SB, Madhi SA. The Effects of Influenza Vaccination during Pregnancy 
on Birth Outcomes: A Systematic Review and Meta-Analysis. American journal of perinatology. 
Sep 2016;33(11):1104-1114. 
12. 
Zaman K, Roy E, Arifeen SE, et al. Effectiveness of maternal influenza immunization in mothers 
and infants. The New England journal of medicine. Oct 9 2008;359(15):1555-1564. 
13. 
Benowitz I, Esposito DB, Gracey KD, Shapiro ED, Vazquez M. Influenza vaccine given to pregnant 
women reduces hospitalization due to influenza in their infants. Clinical infectious diseases : an 
official publication of the Infectious Diseases Society of America. Dec 15 2010;51(12):1355-1361. 
14. 
Becker-Dreps S, Butler AM, McGrath LJ, et al. Effectiveness of Prenatal Tetanus, Diphtheria, 
Acellular Pertussis Vaccination in the Prevention of Infant Pertussis in the U.S. American journal 
of preventive medicine. Aug 2018;55(2):159-166. 

Page 34 of 34 
Version Date: 6.22.2022 
 
15. 
Gall SA, Myers J, Pichichero M. Maternal immunization with tetanus-diphtheria-pertussis 
vaccine: effect on maternal and neonatal serum antibody levels. American journal of obstetrics 
and gynecology. Apr 2011;204(4):334 e331-335. 
16. 
Munoz FM, Bond NH, Maccato M, et al. Safety and immunogenicity of tetanus diphtheria and 
acellular pertussis (Tdap) immunization during pregnancy in mothers and infants: a randomized 
clinical trial. Jama. May 7 2014;311(17):1760-1769. 
17. 
Henninger M, Naleway A, Crane B, Donahue J, Irving S. Predictors of seasonal influenza 
vaccination during pregnancy. Obstetrics and gynecology. Apr 2013;121(4):741-749. 
18. 
Eppes C, Wu A, You W, Cameron KA, Garcia P, Grobman W. Barriers to influenza vaccination 
among pregnant women. Vaccine. Jun 12 2013;31(27):2874-2878. 
19. 
Kahn KE, Black CL, Ding H, et al. Influenza and Tdap Vaccination Coverage Among Pregnant 
Women - United States, April 2018. MMWR. Morbidity and mortality weekly report. Sep 28 
2018;67(38):1055-1059. 
20. 
Kriss JL, Albert AP, Carter VM, et al. Disparities in Tdap Vaccination and Vaccine Information 
Needs Among Pregnant Women in the United States. Maternal and child health journal. Oct 4 
2018. 
21. 
Henninger ML, Irving SA, Thompson M, et al. Factors associated with seasonal influenza 
vaccination in pregnant women. Journal of women's health. May 2015;24(5):394-402. 
22. 
New S, Winter K, Boyte R, et al. Barriers to Receipt of Prenatal Tetanus Toxoid, Reduced 
Diphtheria Toxoid, and Acellular Pertussis Vaccine Among Mothers of Infants Aged <4 Months 
with Pertussis - California, 2016. MMWR. Morbidity and mortality weekly report. Sep 28 
2018;67(38):1068-1071. 
23. 
National Vaccine Advisory C. The National Vaccine Advisory Committee: reducing patient and 
provider barriers to maternal immunizations: approved by the National Vaccine Advisory 
Committee on June 11, 2014. Public health reports. Jan-Feb 2015;130(1):10-42. 
24. 
Shavell VI, Moniz MH, Gonik B, Beigi RH. Influenza immunization in pregnancy: overcoming 
patient and health care provider barriers. American journal of obstetrics and gynecology. Sep 
2012;207(3 Suppl):S67-74. 
25. 
Healy CM, Rench MA, Montesinos DP, Ng N, Swaim LS. Knowledge and attitiudes of pregnant 
women and their providers towards recommendations for immunization during pregnancy. 
Vaccine. Oct 5 2015;33(41):5445-5451. 
26. 
MacDougall DM, Halperin SA. Improving rates of maternal immunization: Challenges and 
opportunities. Human vaccines & immunotherapeutics. Apr 2 2016;12(4):857-865.