CONFIDENT: A Randomized Trial to Increase COVID-19 Vaccine Confidence in Long-term Care Workers
Phase: NA
Conditions: COVID-19
Sponsor: Dartmouth-Hitchcock Medical Center

CONFIDENT: A randomized trial to increase COVID-19 vaccine
confidence in long-term care workers
Study Protocol
Principal Investigators: Glyn Elwyn and Marie-Anne Durand
Clinical trial registration number: NCT05168800
Funded by: Patient Centered Outcomes Research Institute (PCORI)
Sponsor: Trustees of Dartmouth College
Version: v2.6, January 8th, 2025
Notes: Some sections of this protocol document replicate or are similar to the
published study protocol - Stevens G, Johnson LC, Saunders CH, et al. The
CONFIDENT study protocol: a randomized controlled trial comparing two methods to
increase long-term care worker confidence in the COVID-19 vaccines. BMC Public
Health. 2023;23(1):384. Available at:
https://bmcpublichealth.biomedcentral.com/articles/10.1186/s12889-023-15266-x
Additionally, this protocol document does not explicitly indicate all protocol changes
made throughout the study. For the list of protocol changes please see the published
study results manuscript, which will be attached to the study record at
https://clinicaltrials.gov/study/NCT05168800, when it is available.
1

Table of Contents
1 Protocol Summary
5
1.1 Lay Summary
5
1.2 Scientific Abstract
6
2 Research Team
9
2.1 Principal Investigators
9
2.2 Research Team Members
9
2.3 Consultants
10
2.4 Stakeholder Advisory Group
10
3 Introduction
11
3.1 Background and Rationale
11
3.1.1
Background
11
3.1.2
Rationale
12
3.2 Aims and Objectives
13
4 Methods
15
4.1 Design
15
4.1.1
Design Overview
15
4.1.2
Randomized Controlled Trial (Aims 1 and 2)
15
4.1.3
Intervention and Implementation Mapping (Aim 3)
16
4.1.4
Process Evaluation
17
4.1.5
Controlling for Contamination
21
4.2 Setting
21
4.3 Participants
21
4.3.1
Randomized Controlled Trial (Aims 1 and 2)
21
4.3.2
Aim 3
23
4.3.3
Feasibility of Recruitment
23
4.4 Interventions and Comparator
24
4.4.1
Theory and Content Foundations
24
4.4.2
Development
25
4.4.3
Intervention 1: Dialogue-Based Webinar
26
4.4.4
Intervention 2: Social Media Website
27
4.4.5
Comparator
28
4.4.6
Adaptation of Interventions
28
4.4.7
Criteria for discontinuing or modifying allocated interventions
28
4.4.8
Strategies to improve adherence to intervention protocols
29
4.4.9
Concomitant interventions permitted during the trial
29
4.5 Outcomes
29
4.5.1
Primary Outcome Measure
29
2

4.5.2
Secondary Outcomes and Other Data
29
4.6 Sample Size and Power Calculation
32
4.7 Trial Procedure
33
4.7.1
Recruitment and Screening
33
4.7.2
Consent, Identity Verification, and Baseline Assessment
33
4.7.3
Allocation of Interventions
35
4.7.4
Blinding
36
4.7.5
Follow-up Assessments
36
4.7.6
Monitoring Enrollment
36
4.7.7
Tracking and Retaining Participants
36
4.7.8
Adherence to Protocol and Supervision
37
4.7.9
Reporting Plan and Study Termination
37
4.8 Statistical Analysis
38
4.8.1
Data Source Accuracy
38
4.8.2
Analyses Corresponding to Trial Aims 1 and 2
38
4.8.3
Data Linkage
39
4.8.4
Analyses Corresponding to Aim 3
39
4.9 Data Management
39
4.9.1
Data Management Plan
39
4.9.2
Data and Safety Monitoring Plan
43
4.9.3
Auditing of Trial Conduct
44
5 Ethics, Write-up and Dissemination
45
5.1 Research Ethics Approval
45
5.2
Protocol Amendments
45
5.3 Dissemination and Implementation in Other Settings
45
5.3.1
Academic Channels
45
5.3.2
Professional organizations and healthcare delivery systems
45
5.3.3
Social Media and Lay Press
46
5.4 Possible Barriers to Dissemination and Implementation
46
5.5 Making Results Available to Participants
46
6 Timeline
47
7 References
48
Appendix A - COVID-19 Vaccine Option GridTM
55
Appendix B - Data Analysis Methodology - Study Aims 1 and 2
58
3

List of Abbreviations
CBPR
Community Based Participatory Research
CDC
Centers for Disease Control and Prevention
COVID-19
Coronavirus Disease 2019
IHI
Institute for Healthcare Improvement
LTCW
Long-Term Care Worker
NAHCA
National Association of Health Care Assistants
PI
Principal Investigator
VCI
Vaccine Confidence Index
General Information
This document provides details regarding the setup, conduct, and analysis of the Patient
Centered Outcomes Research Institute (PCORI) funded study, “CONFIDENT: A randomized
trial to increase COVID-19 vaccine confidence in long-term care workers.”
Key Study Personnel Approval
The contact Principal Investigator (PI) (G Elwyn) and the dual PI (M-A Durand) have discussed
and approved this protocol. The investigators agree to perform the investigations and to abide
by this protocol except where significant departures from it are mutually agreed upon between
PIs, relevant Institutional Review Board, and PCORI in writing.
4

1
Protocol Summary
1.1
Lay Summary
Public health professionals think that COVID-19 vaccination is the most promising solution to
the COVID-19 pandemic. Vaccination can only be successful if a large number of people get
vaccinated. Some people are not sure about the COVID-19 vaccines. Some have used the term
‘vaccine hesitancy’ to describe this uncertainty. People are hesitant to get vaccinated for many
reasons. Some people worry that scientists developed the COVID-19 vaccines too quickly.
Other people worry there are not many research studies on the safety of the COVID-19 vaccine.
Information on the internet is often of poor quality, or misleading. There are also some people
that are actively spreading misinformation about the vaccine. As a result, many have false
beliefs and mistrust about the COVID-19 vaccines. Some groups in the U.S. have higher rates
of vaccine hesitancy. This includes a group of healthcare professionals who work in places like
nursing homes, usually called long-term care workers (LTCWs). Many LTCWs do not want or
plan to get vaccinated.
LTCWs are an important group to target because they serve a high-risk, elderly population.
They have a higher risk of passing the virus on to others. LTCWs are among the lowest paid
workers in the US. They come from diverse communities, including those that have experienced
high rates of infection, illness and death from COVID-19. There are policy mandates that were
developed in 2021 requiring LTCWs to get vaccinated, which may increase vaccine rates. The
mandates may also cause some LTCWs to leave their jobs or be unhappy if they did not want
the vaccine. It is therefore important to increase LTCWs confidence in the COVID-19 vaccines
for their own protection and wellbeing. It is not clear how best to do this. In this trial, we will
study LTCWs from nursing home and residential care settings, who are typically certified nurse
assistants and residential care assistants.
We plan to compare two promising methods to increase confidence in COVID-19 vaccines. One
method is a live webinar, which is a meeting that happens online using a video-conference
platform. In this meeting, we will ask LTCWs to look at information about the COVID-19 vaccine,
talk about concerns, and ask questions that they may have. The webinar will be led by another
LTCW. Questions will be answered by a medical doctor who has experience addressing LTCW
concerns about the COVID-19 vaccine. The other method is a website that will contain posts
derived from social media platforms, such as Facebook, YouTube, Instagram, and Tik Tok. The
posts will be short video messages and other content about the COVID-19 vaccines. Posts will
be about common questions and concerns about the COVID-19 vaccines. The posts will come
from people who are popular on social media and will include content from other LTCWs.
LTCWs in our study will look at and reply to the posts using likes and comments.
We will invite adult LTCWs to participate in the study who have concerns about the COVID-19
vaccines or who have not received a booster vaccine. We will not include people who are
pregnant or breastfeeding. We will advertise the study to LTCWs using email lists, social media,
and professional networks that include LTCWs.
5

We will compare the live webinar and social media website methods to other information that
people would usually get about the COVID-19 vaccines. We will place people in one of these
three groups (the live webinar group, the social media website group, or the usual information
group) based on chance. We will compare peoples’ level of confidence in the vaccine across
these groups. To better understand how the social media website and live webinars might
influence people, we will also measure how informed people feel about the vaccines, what they
believe about the vaccines, and their trust in information about the vaccines given by different
people and groups. We will collect this information using online surveys. We will give the first
survey to participants when they join the study. Then, we will invite them to join the live webinar,
use the social media website, or access the usual information website (CDC website). Lastly,
they will get three more surveys, 3 weeks, 3 months and 6 months after joining the study.
This study will tell us how to help people become more confident in COVID-19 vaccines. This
may be especially helpful for people who rely on information that is out of date or wrong.
1.2
Scientific Abstract
Background and Significance: Vaccination programs have the potential to reduce the incidence
of COVID-19 as well as serious illness and mortality associated with the COVID-19 pandemic.
However, success depends on widespread vaccine confidence and uptake. The speed of
vaccine development and limited research data on both efficacy and adverse effects have
caused concerns. Misinformation and disinformation have eroded confidence in the vaccine,
causing hesitancy and reduced uptake.
Long term care workers (LTCWs) report substantial vaccine hesitancy. Unvaccinated LTCWs
pose a risk of transmission to vulnerable long-term care residents. Further, over 50% of LTCWs
are from minority and socioeconomically disadvantaged groups, and may be more vulnerable to
severe disease if they become infected. While vaccine mandates have recently been
announced for workers in long-term care facilities, mandates also have the potential
consequence of job losses, workplace shortages, and reduced LTCW wellbeing. It is therefore
important to increase LTCWs’ confidence in the COVID-19 vaccines. In this trial we will recruit
LTCWs in nursing and residential care settings.
Objectives: Working with our stakeholders, we chose the following research questions:
1.
What is the comparative impact of two interventions designed to increase COVID-19
vaccine confidence (primary outcome) and influence secondary outcomes, compared to
enhanced usual practice in LTCWs?
2.
What is the most effective intervention to increase COVID-19 vaccine confidence
among different subgroups of LTCWs, including those who experience different levels of
health disparities?
3.
What are the delivery characteristics, contexts, and processes needed to sustain and
scale up the implementation of interventions designed to increase COVID-19 vaccine
confidence among LTCWs?
6

Design & Population: We will conduct a three-arm online randomized controlled trial, with
randomization at the LTCW level. We will use an effectiveness-implementation hybrid design
(type 2), guided by relevant domains of the Consolidated Framework for Implementation
Research (CFIR).
We will partner with the National Association of Health Care Assistants (NAHCA) to recruit 1800
LTCWs. Eligible LTCWs will self-identify as: 1) being at least 18 years old, 2) living in the United
States, 3) having worked in a long-term care setting in the past two years, 4) able to verify their
LTCW status, 5) not currently be pregnant or breastfeeding, and 6) able to read, write, and
understand English, and 7) being at least somewhat worried about the COVID-19 vaccines
and/or not having received any COVID-19 booster vaccine. Contingent on study results, in the
sustainability phase (see Figure 3), we will open the materials to other LTCWs.
Interventions: We selected two interactive interventions with our stakeholder partners, based on
provisional evidence of their effectiveness and Peretti-Watel’s vaccine hesitancy framework.
Interventions will be co-designed and adapted with LTCWs and the Institute for Healthcare
Improvement (IHI) co-investigators. We will include an enhanced usual practice arm.
Arm 1: Dialogue-Based Webinars
Six hundred LTCWs will attend virtual dialogue-based webinars in groups of 20. An existing
COVID-19 conversation aid (English version) will be available to all participants. Each webinar
will be co-facilitated by a LTCW and physician expert, and assisted by a communication expert.
Arm 2: Social Media Website
Six hundred LTCWs will be invited to visit a curated COVID-19 social media website,
co-designed with LTCWs and NAHCA. The site will be interactive, dynamic, and have
multi-component content derived from LTCW-nominated platforms, e.g., Facebook, YouTube,
etc. It will address frequent concerns and topics that are deemed relevant by LTCWs.
Arm 3: Enhanced Usual Practice
Six hundred LTCWs will be directed to visit the COVID-19 vaccine information on the CDC
website.
Outcomes: Our primary outcome is COVID-19 vaccine confidence as assessed by the Vaccine
Confidence Index, which measures confidence in vaccine safety, efficacy, and importance. Our
secondary outcomes include COVID-19 vaccine uptake (any dose, initial series completion,
booster completion), an adapted Net Promoter Score (likelihood of recommending COVID-19
vaccination and booster vaccination), initial vaccine, booster, and future vaccine intent, feeling
informed about the COVID-19 vaccines, identification of COVID-19 vaccine information and
misinformation, and trust in COVID-19 vaccine information provided by different people and
organizations. Exploratory outcomes will include identifying external factors that may contribute
to vaccine decisions and contextual factors experienced before and during the study period that
relate to COVID-19. We will also collect demographic information in the baseline survey,
7

including age, race, ethnicity, gender, zip code, duration of experience in long-term care,
educational attainment, health literacy, COVID-19 vaccine status, perceived influence of others,
and religiosity. Our primary, secondary, and exploratory outcomes will be collected by online
surveys at baseline (T0), 3 weeks post baseline (T1), 3 months post baseline (T2), and 6
months post baseline (T3).
8

2
Research Team
We have established an international team that combines scientific expertise in communication
in health care, dialogue-based interventions, and a track record in the areas of long-term care
facilitation and vaccine confidence and hesitancy.
2.1
Principal Investigators
Glyn Elwyn, MB BCh, MSc, PhD, FRCGP
Marie-Anne Durand, MSc, MPhil, PhD, CPsychol
2.2
Research Team Members
Lisa Johnson, MBA (Project Director)
Gabrielle Stevens, PhD (Trial Manager)
Jacqueline Pogue, MPH (Project Manager)
Catherine Saunders, PhD, MPH (Jr. Project Manager)
Renata W. Yen, PhD, MPH (Data Manager)
Peter N. Schmidt, PhD (Study Statistician)
Rachael Thomeer (Research Coordinator)
Ailyn Sierpe, MSc (Research Coordinator)
Danielle Schubbe, PhD, BS (Research Coordinator)
Rachel C. Forcino, PhD, MSc (Postdoctoral Fellow)
Jaclyn Engel (Research Assistant)
Christopher Jacobs (Project Coordinator)
Eugene C. Nelson, DSc, MPH (Advisor)
Alistair James O’Malley, PhD (Advisor)
Ruth Little, EdD, MPH (Subcontract PI) East Carolina University
Alice Bonner, PhD (Subcontract PI) Institute for Healthcare Improvement
Don Goldmann, MD, Institute for Healthcare Improvement
Jennifer Lenoci-Edwards, RN, MPH, CPPS, Institute for Healthcare Improvement
Lori Porter (Subcontract PI) National Association of Health Care Assistants
Matthew Cantrell, BS, National Association of Health Care Assistants
Lisa Houck, National Association of Health Care Assistants
9

2.3
Consultants
Heidi Larson, PhD, London School of Hygiene & Tropical Medicine
Emilie Karafillakis, MSc, London School of Hygiene & Tropical Medicine
Professor Ève Dubé, PhD, Institut National de Santé Publique du Québec
Physician Facilitators: We have engaged four physicians who will serve as facilitators of the
dialogue-based webinars to be delivered to participants in the webinar intervention arm:
Timothy Holahan, DO, CMD, University of Rochester Medical Center; Swati Gaur, MD,
Northeast Georgia Health Care; Christopher Herman, MD, affiliated with Atrium Wake Forest
Baptist Hospital; and Chetan Amin, DO, Piedmont Family Medicine in Salisbury, NC.
2.4
Stakeholder Advisory Group
We have assembled a Stakeholder Advisory Group (SAG) who will offer their guidance and will
review key study documents and examine effectiveness and implementation progress over the
course of the study. The SAG includes Terry Fulmer (John A. Hartford Foundation), Ted Goins
(Lutheran Services Carolinas), Lynn Hood (Principle LTC), Nancy Koha (Principle LTC), Karen
Ernst (Voices for Vaccines), Branden Fillbrook (Core LTCW participant representative and
trained peer), Celeste Wooten (Core LTCW participant representative and trained peer),
Rowena Sheppard (Core LTCW participant representative and trained peer), Lupita Rodriguez
(Core LTCW participant representative). Additionally, we have five additional LTCW participant
representatives to the SAG: Feng Chen, Jon Edwards, Tammy McIlnay, Katie Page, and
Mary White.
10

3
Introduction
3.1
Background and Rationale
3.1.1
Background
The COVID-19 pandemic has been affecting hundreds of millions of people worldwide for
nearly two years. Despite the emergence of new variants, vaccination programs continue to
show the most promise at mitigating the severity of illness and mortality caused by the virus1.
However, vaccination success depends on widespread uptake. Because of increased virulence
in the most recent wave and some loss of protection as the virus mutates, coverage rates need
to exceed 90%2 to minimize the risk of outbreaks of current SARS-CoV-2 variants.
The unprecedented speed at which new vaccines were developed and the use of the mRNA
technology have raised public concerns about safety3. While hesitancy about vaccines existed
before this pandemic, such concerns have been accentuated by the lack of prior data about
COVID-19 vaccines and limited follow-up data4. Additionally, political and ideological
allegiances have reinforced certain objections and resistance to vaccination5. Given that the
internet has become a prominent source of information for many people, it has also been
difficult to limit the rapid spread of misinformation or disinformation while promoting
evidence-based information6. Misinformation about the virus and the COVID-19 vaccine are a
major threat to confidence in the importance, safety, and effectiveness of vaccines, which
reinforce mistrust, and lead to vaccine hesitancy7–11.
Some subgroups of the US population, including long-term care workers (LTCWs), have higher
rates of COVID-19 vaccine hesitancy5. LTCWs include people who work in long-term care
facilities or home-based care, and include roles such as certified nurse assistants, residential
care assistants and non-clinical support. Although long-term care facilities have been
epicenters of COVID-19 outbreaks, warranting widespread vaccination, there is reported low
vaccination coverage among LTCWs12,13. LTCWs usually serve those most vulnerable to serious
complications of COVID-1914. LTCWs are also likely to be more vulnerable to complications of
COVID-19 themselves. More than 50% of LTCWs are from minority and socioeconomically
disadvantaged groups, often working multiple jobs15,16. Given the vulnerability of the
populations they serve, the possibility of transmission in their local communities, and their own
increased risk of COVID-19 morbidity/mortality, it is important to increase COVID-19 uptake
among LTCWs. In August 2021, the US Federal Government announced its intent to impose a
mandate that LTCWs be vaccinated in order to be eligible to work. This mandate has now been
expanded to include employees in all health facilities that receive federal funding. The Centers
for Medicare and Medicaid (CMS), working with the Centers for Disease Control and Prevention
11

(CDC), will be responsible for imposing the broad mandate and planned to release an interim
rule in October 2021.
While vaccine mandates have clear, positive implications for reducing COVID-19 spread and
disease severity, other potential consequences should not be overlooked. At the forefront are
job losses and workforce shortages if workers do not adhere to mandates, and reduced
workplace trust and increased vaccine hesitancy17. The likely legal challenges and conflict
created between management and employees may also have a damaging impact on
relationships at work and the wellbeing of long-term care workers. These consequences
highlight the importance of efforts to increase confidence in the COVID-19 vaccines, so
decisions to vaccinate are made willingly. Increasing vaccine confidence will also help alleviate
the concerns of those already vaccinated, but still hesitant to follow future vaccination
recommendations.
Evidence for vaccine confidence in LTCWs is limited; a cross-sectional study in Italy of LTCWs
found vaccine confidence correlated with influenza vaccine uptake18. In a global survey,
confidence in the importance of vaccines had the strongest association with vaccine uptake
compared to other factors11. For people who have been partially or fully vaccinated, increasing
confidence in the importance of the vaccine will also be vital for future uptake, adherence to
vaccination protocols, and the acceptance of scheduled boosters.
3.1.2
Rationale
The best strategy for increasing COVID-19 vaccine confidence among LTCWs remains unclear.
As demonstrated by multiple reviews (and an overview of existing reviews)19–21, addressing
vaccine hesitancy is a complex task, and there is no strong evidence supporting any one single
intervention. Nevertheless, there is some emerging evidence that identifies the features of the
interventions that are most likely to be effective. For example, Jarrett et al. suggest that
multi-component, dialogue-based interventions targeting specific unvaccinated and
vaccine-hesitant populations were most effective22. Other evidence indicates that social media
interventions can improve attitudes towards vaccines and increase uptake23–25. Lastly, shared
decision-making (SDM) interventions that involve patients and healthcare providers sharing
information and making vaccine decisions collaboratively have also been shown to improve
vaccine uptake26,27. Thus, in this study, we will co-develop and test two scalable,
multi-component interventions targeted at LTCWs – one dialogue-based that explicitly
incorporates SDM principles and a conversation aid, and the other social media-based – to
improve COVID-19 vaccine confidence and other outcomes in the US LTCW population.
This study will focus on three research questions that address compelling clinical and
implementation questions raised by the COVID-19 vaccines:
12

1.
What is the comparative impact of two interventions designed to increase COVID-19
vaccine confidence (primary outcome) and influence secondary outcomes, compared to
enhanced usual practice in LTCWs?
2.
What is the most effective intervention to increase COVID-19 vaccine confidence
among different subgroups of LTCWs, including those who experience different levels of
health disparities?
3.
What are the delivery characteristics, contexts, and processes needed to sustain and
scale up the implementation of interventions designed to increase COVID-19 vaccine
confidence among LTCWs?
3.2
Aims and Objectives
Specific Aim 1: To compare the impact of two interventions delivered online: 1) a
dialogue-based webinar using the existing COVID-19 Option Grid conversation aid and, 2) an
interactive, dynamic, and multi-component social media website, compared to enhanced usual
practice (link to Centers for Disease Control (CDC) vaccine website), on COVID-19 vaccine
confidence (primary outcome), and other secondary outcomes among LTCWs (see Figure 1).
Hypothesis 1: Each intervention will be superior to enhanced usual practice at increasing
vaccine confidence.
Hypothesis 2: The dialogue-based webinar intervention will be superior to the social media
arm at increasing vaccine confidence.
We predict superiority of the dialogue-based webinar intervention based on stronger evidence
that exists for dialogue-based approaches and SDM for improving vaccine hesitancy and/or
uptake, as compared to social media approaches. While our social media website intervention
includes elements of dialogue, this component is more explicit in the webinar intervention (see
‘Interventions and control’).
Specific Aim 2: To determine if LTCWs’ characteristics and other factors mediate and
moderate the interventions’ impact on vaccine confidence and other secondary outcomes (See
Figure 1).
Hypothesis 1: Increased perceptions of feeling informed about the vaccines, identification
of vaccine information and misinformation, and trust in vaccine information provided by
different sources will explain (mediate) the relationship between the interventions and
vaccine confidence, as well as other secondary outcomes.
Exploratory: We will conduct exploratory heterogeneity of treatment effects (HTE) analyses
to identify whether certain participant characteristics and beliefs moderate the relationship
between the interventions and vaccine confidence, as well as other secondary outcomes.
13

Variables to be explored will include, but are not limited to, vaccination status, religious
beliefs, perceived influence of others, age, race, ethnicity and personal experiences with
COVID-19.
Specific Aim 3: To explore the implementation characteristics, contexts, and processes
needed to sustain and scale up the use of interventions designed to increase vaccine
confidence among LTCWs.
Hypothesis 1: Co-developing and adapting the implementation strategy and outcomes
with LTCWs and other stakeholders will facilitate implementation.
Figure 1. Study outcomes, mediators, and moderators
14

4
Methods
4.1
Design
4.1.1
Design Overview
We will conduct an online randomized controlled parallel group trial with a hybrid-effectiveness
implementation design type 228. This design includes evaluating effectiveness in the
randomized trial alongside an exploration of implementation (tasks 1-5 of implementation
mapping)29. Contingent on the trial results, implementation will then be evaluated in a
sustainability phase after randomized trial assessments are completed. If trial results do not
support immediate implementation, we will focus our efforts on further engaging key
stakeholders to identify possible adaptations and recommendations for future implementation.
We will draw on elements of Community Based Participatory Research (CBPR), and co-design
and co-manage this study with LTCWs, the National Association of Health Care Assistant
(NAHCA), and the Institute for Healthcare Improvement (IHI).
4.1.2
Randomized Controlled Trial (Aims 1 and 2)
The randomized controlled trial will have three arms, with randomization at the individual
person level. Arm 1 is a dialogue-based webinar intervention. Arm 2 is a social media website
intervention. Arm 3 is enhanced usual practice. Intervention and comparator arms are
described in detail in section 4.4. We will use online and in-person recruitment strategies to
enroll 1,800 LTCWs to the trial. Trial data will be obtained primarily via four self-reported
participant surveys, delivered to each participant over a period of approximately six months
(Figure 2). We will also collect online activity data in each intervention arm. In the trial set-up
period, an additional online survey will be administered to a sample of ~500 people from the
general population who are demographically representative of LTCWs, to test novel trial
screening questions (e.g., COVID-19 vaccine concerns) and adapted outcome measures (e.g.,
vaccine confidence), and help refine the content of the interventions.
15

Figure 2. CONSORT Study Flow Diagram
4.1.3
Intervention and Implementation Mapping (Aim 3)
We are undertaking a process called “intervention and implementation mapping,” which is a
co-design and co-development approach to intervention design and implementation strategy29.
In Aim 3, we hypothesized that this co-development and co-learning approach would facilitate
implementation of the interventions in both the trial phase and the sustainability phase. Aim 3
will primarily involve interviews with key stakeholders, including LTCWs and others identified as
current or future adopters, implementers, and maintainers of the study interventions.
16

Aim 3 and the associated intervention and implementation mapping processes are integrated in
every stage of the broader study, including the trial setup and intervention development phase
(1), trial phase (2), and sustainability phase (3) (see section 6). Aim 3 interviews will be
conducted by research team members at IHI and the Center for Program Design and
Evaluation (CPDE) at Dartmouth College.
Much of our intervention and implementation mapping activities will occur during the
preparatory stages of the study. Guided by Fernandez’s implementation mapping approach29,
we have developed a set of tasks and criteria to operationalize implementation mapping for our
study. The steps are detailed in Table 1.
4.1.4
Process Evaluation
We will conduct a process evaluation as part of Aim 330. Using data from Aim 3 interviews, trial
participant surveys, online activity data, webinar recordings, and online observation (during and
after webinars; over the course of the social media website) from team members, trained LTCW
peers and expert facilitators, we will measure, analyze, and report the fidelity of delivering each
intervention, the dose of intervention delivered, participants’ views toward the interventions,
and the achieved versus intended reach. The process evaluation will be primarily conducted by
the CPDE at Dartmouth College, in order to retain more independence compared to the team
responsible for the outcome evaluation (Aims 1 and 2).
For each arm, we will use a maximum variation sampling approach31, to include data extremes
for analysis (with maximum diversity), through the selection of key diversity dimensions (generic
and specific to each arm). For the webinar arm, we will first record all webinars and select up to
12 webinars for process evaluation analysis, by using a range of extremes.32 To select the
webinars for analysis, key diversity dimensions may include: number of attendees, number of
messages posted in the webinar chat, timing of the webinar in the context of the overall
duration of recruitment (e.g., start of recruitment, two months into recruitment, six months into
recruitment), duration of the webinar, proportion of male/female attendees, perceived richness
of the verbal exchanges (rated at the end of each webinar by the facilitating team), presence of
conflicts/heated discussions, and number of people dropping off before the end of the webinar.
For each of the 12 selected webinars, we will measure, analyze, and report the fidelity of
delivering each intervention, the dose of intervention delivered, and reach (proportion of eligible
participants who attend the webinars). We will also conduct a thematic analysis of the chat
posts and of the transcribed discussions.
We will use the same approach for the social media platform. We will analyze up to 12
independent weeks of social media platform content, by selecting a range of extremes. To
select the days of social media platform content, key diversity dimensions may include:
specific timepoints across the study recruitment continuum (e.g., start of recruitment, two
17

months into recruitment, six months into recruitment), number of visits per day, number of
messages posted, number of likes, etc. For each of the 12 weeks of selected content, we will
measure, analyze, and report the fidelity of delivering each intervention, the dose of intervention
delivered, and reach. We will also examine how the media platform was used based on website
activity data33. The activity data will include user ID, time stamps, and page URL of each action
completed on the social media platform: reading a page, posting a comment, liking a
comment, etc. We will measure the mean total time spent on the social media website, the
mean number of pages accessed, identify the most frequently visited pages, the most
frequently liked pages or sections, etc.
18

Table 1. Summary of intervention and implementation mapping activities
Step
When
Activity
Objectives
Participants
Methods
Task 1. Conduct an intervention and implementation needs assessment
Step 1
Study set-up
Adopter,
Implementer,
Maintainer29
brainstorm
Determine who will participate in the
implementation consultations
Study team, including those at
collaborating organizations
Group meetings and
consultations
(convenience sampling)
Step 2
Study set-up
Intervention
consultations
1st round
Assist in the development of the two
study interventions. Includes asking
about their own information needs,
views and experiences with
COVID-19/vaccines, and providing
feedback on proposed intervention
content and functionalities, among
other things.
Long-term care workers (LTCWs)
on our Stakeholder Advisory
Group (SAG)
Interviews (one-on-one,
video-based)
(convenience sampling)
Adults from the general public
(demographically representative
of LTCWs) recruited by Qualtrics
Panel Services
Online survey
(purposive sampling)
Task 2A. Finalize intervention content and delivery using co-design/user-centered design principles
Step 3
Study set-up
Intervention
consultations
2nd round
(as needed)
Continuation of Step 2, to further
refine and finalize the interventions.
Informal feedback on intervention
design and user-testing of
interventions in iterative cycles.
LTCWs on our SAG, other
LTCWs not associated with the
study, and study team
members/collaborators ​
Emails, study meetings,
polls, user-testing
sessions (convenience
sampling)
Task 2B. Identify adoption and implementation outcomes, performance objectives, determinants, and change objectives
Step 4
Study set-up
Implementation
consultations
1st round
Determine what is required to
effectively implement the two
interventions during the trial, and how
implementation success should be
measured. Also explore how we might
sustain the interventions once the trial
is completed.
Study team, including those at
collaborating organizations,
consultants, and SAG members
Interviews (likely
multiple rounds,
one-on-one,
video-based or
telephone)
(convenience sampling)
19

Task 3. Select theoretical methods and design implementation strategies
Step 5
Study set-up
Implementation
collaboration
Identify necessary actions to facilitate
implementation and sustainability
using the Consolidated Framework for
Implementation Research (CFIR)34,35.
Study team, including those at
collaborating organizations,
consultants, and SAG members
Group meetings and
consultations
(convenience sampling)
Task 4. Produce implementation protocols and materials
Step 6
Study set-up
Implementation
collaboration
Using the information gleaned from
Tasks 1-3, develop implementation
protocols and materials.
Study team, including those at
collaborating organizations,
consultants, and SAG members
Group meetings and
consultations
(convenience sampling)
Task 5. Select assessment metrics
Step 7
Study set-up
and trial
phase
Implementation
collaboration
Collaboratively generate process
evaluation questions and design
evaluation assessment processes for
Aim 3 interviews during and post trial.
Study team, including those at
collaborating organizations,
consultants, and SAG members
Group meetings and
consultations
(convenience sampling)
Step 8
Trial phase
Implementation
consultations
2nd round
Determine views on the interventions,
involvement in co-designing the
implementation process (if relevant),
and views on adaptations and the
future sustainability of the
interventions, as well as other
potential ideas or approaches for
improving COVID-19 vaccine
confidence and uptake.
Trial participants, study team
members, and other key
stakeholders identified in Step 1
Interviews
(convenience and
purposive sampling)
Step 9
Trial and
sustainability
phases
Implementation
assessment
Determine the success of the
intervention implementation.
May include study team
members, trial participants, and
others TBD
Interviews, surveys
(convenience and
purposive sampling),
online activity data,
field notes
20

4.1.5
Controlling for Contamination
Access to interventions will be restricted to those assigned to each intervention arm, via the
provision of unique access links (Arm 1: Dialogue-Based Webinar) and unique user logins (Arm
2: Social Media Website). To audit for inadvertent contamination across arms, we will monitor
who registers for and accesses each intervention on a weekly basis, removing those without
legitimate access. During data cleaning we will review participants who may have been
exposed to content from more than one trial arm, and make decisions about the removal of
these data to control for the influence of contamination.
4.2
Setting
The randomized trial will be conducted entirely online, and will be open to LTCWs residing
anywhere in the United States. We will use a combination of online and in-person recruitment
strategies. In-person recruitment will occur at a convenience sample of LTC settings in North
Carolina, identified by our collaborator (NRL), who has extensive professional LTC experience
and trusted relationships with LTC leaders in the North Carolina area. Settings will include large
and smaller size for-profit and not-for-profit nursing homes and continuing care retirement
communities in both rural and urban locations. Additional settings will include the New England
region.
4.3
Participants
4.3.1
Randomized Controlled Trial (Aims 1 and 2)
Eligible participants will include vaccinated and unvaccinated LTCWs who self-identify as: 1)
being 18 years and older, 2) living in the United States, 3) having worked in a long-term care
setting in the past two years, 4) able to verify their LTCW status, 5) not currently pregnant or
breastfeeding, and 6) able to read, write, and understand English, and 7) being at least
somewhat worried about the COVID-19 vaccines and/or not having received any COVID-19
booster vaccine (see Table 2).
Eligible LTC settings will include nursing homes, skilled nursing facilities, assisted living
facilities, home health care, hospice care, and retirement communities. The two-year timeframe
will avoid excluding those who are currently unemployed, and will approximately cover the
duration of time since the COVID-19 pandemic began. To ensure a high proportion of study
participants represent our low confidence target group, we will use two criteria – worry about
the vaccines and booster uptake – to screen participants for eligibility. These criteria are based
on a preliminary analysis of panel survey data, which found associations between booster
intent, worry about the vaccines, and vaccine confidence.
21

Table 2. Screening questions and eligibility criteria
Question
Eligible Response(s)
Ineligible Response(s)
Can you read, write, and understand English? (adapted
from36)
Yes
No
How old are you?
18-19 years; 20-24 years;
25-29 years; 30-34 years;
35-39 years; 40-44 years;
45-49 years; 50-54 years;
55-59 years; 60-64 years;
65 years or older
Less than 18
Do you live in the United States?
Yes
No
This study is for long-term care workers.
We must be sure that everyone who joins is giving us the
correct information about who they are. We also need to
be sure that everyone has worked in a long-term care
setting. It is important to our study results.
The information that you provide in this survey will help us
confirm your identity and that you are a long-term care
worker. We will never contact your employer about your
participation in the study.
N/A
N/A
Have you worked in a long-term care setting in the past 2
years? This includes nursing homes, skilled nursing
facilities, assisted living facilities, home health care,
hospice care, and retirement communities.
Yes
No
Below is a list of ways we can confirm you have worked in
a long-term care setting in the past 2 years. Which of these
can you do?
Select all that apply. We will ask you to complete one of
these options later in the survey.
Provide information from
your workplace ID badge;
Provide your Certification
or Professional License
Number; Email us from
your work email account;
Provide information from
a recent pay stub (within
the last 2 years);
None of the above
Are you pregnant or breastfeeding
No
Yes
Are you worried about the COVID-19 vaccines?
If you have completed your initial COVID-19 vaccine series
(1 Johnson & Johnson shot or 2 Pfizer/Moderna shots),
have you had at least one booster shot since then?
Somewhat; Very
OR
No, I have not had any
booster shots;
I have not completed my
initial series
Not at all; A little;
AND
Yes, I have had at least
one booster shot;
Not sure
22

4.3.2
Aim 3
Eligible participants for Intervention Consultations (as described in Table 1) will include LTCWs,
including those who are members of our Stakeholder Advisory Group. We anticipate
interviewing 10-12 participants for each round of consultations, until we reach thematic
saturation.32
For Implementation Consultations (as described in Table 1), our participants will be trial
participants, study team members, and other key stakeholders identified through the “Adopters
Implementers and Maintainers” brainstorm who may play roles in facilitating implementation in
both the trial and sustainability phases. Reporting of Aim 3 activities will adhere to the
Consolidate Criteria for Reporting Qualitative Research (COREQ) checklist37.
4.3.3
Feasibility of Recruitment
We will utilize a diverse range of recruitment methods, both online and in-person. First, we plan
to recruit and engage LTCWs using methods that have been previously effective for our
partners at NACHA. We will therefore use NAHCA’s communication channels and their
networks, consisting of social media followers and contacts with NAHCA registered employers.
NAHCA membership is characterized by care workers who place value on professional
development, however this does not apply to NAHCA non-members who are NAHCA followers
on Facebook and YouTube. NAHCA has the capability to reach care workers in all 50 US states,
and has members in all states, except Alaska. NAHCA has 24,500 members, with 22,685
subscribing to their list-serv, 2,150 YouTube subscribers, and 18,263 Facebook followers.
Response rates to emails via their list-serv average a 33% open rate and a 12% click rate.
Typical engagement on their Facebook posts ranges from 5% to 15%; and their YouTube
videos reach 3,500 views on average. Using these estimates and given the survey-response
incentives ($30 per time-point), we anticipate interest from at least 2,722 LTCWs via the
list-serv and 900 LTCWs via Facebook using a conservative 5% engagement level (3,622 in
total). These numbers combined exceed the 1,800 LTCWs that we need to power the trial.
Our collaborator, Dr. Tim Holahan, Clinical Director of the American Medical Director
Association, has confirmed excellent collaborative arrangements with NAHCA to support
recruitment. Both Lori Porter (NAHCA CEO) and Matthew Cantrell (NAHCA COO) confirm that
LTCWs can be effectively engaged using their membership list outreach methods and social
media.
To further enhance diversity and reach, we will also utilize paid social media and online
advertisements for study recruitment. Lastly, in-person recruitment strategies will include
sharing recruitment materials via visits to LTC settings, conferences, and other events, and
outreach to LTC settings with requests to display study recruitment materials.
23

We also intend to make all recruitment and other study materials available via mobile
technology so that participants can easily access them on a telephone or tablet if they do not
have reliable access to desktop computing.
4.4
Interventions and Comparator
4.4.1
Theory and Content Foundations
We designed two multi-component, interactive, online interventions (arms 1 and 2), informed
by theoretical and practical information from the vaccine confidence and hesitancy literature.
Notably, our intervention development and hypothesized mediators (see Figure 3) were initially
informed by Peretti-Watel’s vaccine hesitancy framework, which conceptualizes vaccine
decision-making as a process. The framework also distinguishes between two types of vaccine
hesitant people: 1) those with poor knowledge of and/or indifference to vaccination issues, and
2) those who are interested in vaccination issues and seek more information, yet are hesitant38.
We were also informed by the Report of the SAGE Working Group on Vaccine Hesitancy and its
conceptualization of the “three Cs” of vaccine hesitancy: Confidence, Complacency and
Convenience; and, we considered all three Cs39. Additionally, our interventions were informed
by the literature on reducing vaccine hesitancy, which suggests multi-component,
dialogue-based interventions (such as conversation-based and social media interventions) are
most effective, as well as those tailored for specific populations22. The interventions were
co-designed, co-adapted, and pre-tested with LTCWs as well as NAHCA, IHI, and other
co-investigators. The control arm (enhanced usual practice) is online COVID-19 vaccine
information provided by the CDC.
24

Figure 3. Randomized controlled trial logic model
4.4.2
Development
The overall designs of the dialogue-based webinar and social media website interventions were
informed by prior studies22,23,26 and developed using participatory research approaches40,41. In
developing content for each intervention, our goal was to maximize consistency of major topics
across interventions, in order to isolate effects of each intervention delivery method.
To inform the intervention content, we initially derived topics from several sources, including
social media monitoring, news surveillance, online public opinion polls42,43, and peer-reviewed
literature39. The major topics for inclusion in both interventions were then refined via interviews
with our LTCW partners as part of Aim 3. Additional information was gathered to inform the
development of the social media website intervention, the details of which are published
25

elsewhere44. The final list of major topics includes COVID-19 in general, vaccine benefits,
vaccine risks, and vaccine development44.
We also developed community standards to ensure all participants felt welcome and
comfortable when engaging with the interventions. These standards, which are similar across
both interventions, were based on the rules of existing vaccine discussion forums and were
co-created with our LTCW partners44. Also included in both interventions is a video that we
developed featuring LTC residents voicing their views on why it is important for LTCWs to be
vaccinated.
In response to low intervention exposure in one trial arm (webinar attendance) experienced in
the first several weeks of data collection, we developed intervention ‘refreshers’ for each trial
arm. The refreshers for trial arms 1 and 2 are briefer, modified versions of each primary
intervention. They include content on new and emerging COVID-19 and vaccine-related topics,
as well as popular questions trial participants have voiced through their primary intervention
engagement. New and emerging topics were derived from several sources, including social
media monitoring, news surveillance, online public opinion polls45, and consultation with our
LTCW partners and other stakeholders. Topics for inclusion were refined primarily via a poll to
identify stakeholders’ information needs (process adapted from46). Refreshers for trial arms 1
and 2 will be updated as required when there is sufficient new and emerging COVID-19 related
public discourse.
4.4.3
Intervention 1: Dialogue-Based Webinar
LTCWs will attend one-time virtual webinars in groups of no more than 20, scheduled at
different times, and different days (including weekends). Each webinar will be led by a LTCW
peer trained in SDM principles. It will also be co-facilitated by a physician with expertise in
COVID-19 vaccination, and a communication expert with experience in SDM.
The webinar intervention agenda is as follows: i) the LTCW facilitator will introduce the session
and go over important instructions, including the set of community standards, ii) the physician
facilitator will review the COVID-19 vaccine Option GridTM conversation aid (see Appendix A)
and other related topics of interest using the principles of the SDM three-talk model47. A link to
the online Option GridTM will also be sent to participants prior to their webinar commencing iii)
participants will then be presented with the four major content topics pertaining to COVID-19
and the vaccines (consistent with those listed above in section 4.4.2) and will vote (via a poll) on
the two content topics that matter most to them. The physician facilitator will then start by
answering questions from participants related to the top voted content areas, and then open
questioning up to all topics until there are no remaining queries. The webinars will end with the
LTC resident video described earlier.
26

Throughout the webinars, the communication expert will assist with any related discussions
where needed and respond to study-related questions/issues or technical troubleshooting.
Webinar durations will vary, running up to 1 hour, or less if there are no remaining questions.
Participants will also be invited to join a facilitated online discussion in the webinar chat and
their engagement will be monitored (e.g., number of chat messages posted during each
webinar and webinar content).
A dialogue-based webinar refresher will also be delivered to all participants in this arm one
week prior to their T2 survey invitation (see Figure 2). The refresher will be a pre-recorded
webinar (video and audio-only versions) lasting ~20 minutes, sent to participants via email. The
structure of the recorded webinar will closely resemble that of the primary intervention, and will
include a review of the COVID-19 Option GridTM, question and answer discussion, and LTC
resident video. It will be available to participants as a video and an audio-only recording.
4.4.4
Intervention 2: Social Media Website
LTCWs will be invited to visit a social media website. The website is curated with popular and
topical posts from social media platforms such as Facebook, TikTok, and YouTube that are
made by medical experts, LTCWs, and other creators. The content and topics addressed on
the website are dynamic in nature, with two new posts added daily.
The top of the website homepage features the four major content topics pertaining to
COVID-19 and the vaccines (consistent with those listed above in section 4.4.2). Participants
can navigate the website by selecting from these major topics, selecting subtopics listed in the
sidebar menu or as hashtags assigned to each post, or by scrolling the remainder of the
homepage (organized as an infinite scroll of all posts, sorted by most recently uploaded). The
website supports participant interaction with features such as reactions and comments.
Participants also receive email notifications when other users react or reply to comments they
have made.
Three special website users affiliated with the study team who have worked as CNAs in LTC
settings contribute to the website as ‘Community Ambassadors’. Their role is to promote user
engagement by reacting and replying to posts and participants’ comments based on their own
experiences as CNAs. Ambassadors also provide factual information on COVID-19, the
vaccines, and boosters should participants ask any direct questions.
A social media website refresher will also be delivered to all participants assigned to this arm
one week prior to their T2 survey invitation (see Figure 2). The refresher will be an email with
previews and links (back to the full post on the social site) to a selection of featured website
content.
27

4.4.5
Comparator
LTCWs will be directed to COVID-19 vaccine information on the CDC website
(https://www.cdc.gov/coronavirus/2019-ncov/vaccines/index.html). The CDC website
addresses common questions about the COVID-19 vaccines and provides information on
related topics, such as specifics about getting vaccinated and staying up to date on the vaccines
and boosters. To address concerns of intervention inferiority, LTCWs will be given access to
active trial arms shown to have impact, after their 6 month follow-up.
An enhanced usual practice refresher will also be delivered to all participants assigned to this
arm one week prior to their T2 survey invitation (see Figure 2). The refresher will be an email
with a link to COVID-19 vaccine information on the CDC website.
4.4.6
Adaptation of Interventions
To monitor the adaptations made to the interventions over the course of the project (and
primarily in the sustainability phase), we will collect data from online activity data, field-notes,
and observations using FRAME (expanded Framework for Reporting Adaptations and
Modifications to Evidence-based interventions)48.
According to the FRAME, an adaptation is defined as a “process of thoughtful and deliberate
alteration to the design or delivery of an intervention, with the goal of improving its fit or
effectiveness in a given context”48. This is particularly relevant in the context of implementation
and thus most applicable to the second part of this project: during the sustainability phase of
the study. Understanding what, how and when adaptations are made is an essential aspect of
implementation science. In this context, adaptation will primarily be made and documented
during the sustainability phase (year 03 of the project).
Whereas the content of the interventions will be responsive to participant engagement,
significant adaptations to format and delivery during the randomized controlled trial (years 01
and 02) will be minimal and only implemented if judged essential by the research team. These
adaptations will also be documented using the FRAME guidance.
4.4.7
Criteria for discontinuing or modifying allocated interventions
Individual participants may choose to not participate in a study intervention. They can also
actively withdraw from the study at any time by either opting out of both forms of contact
(email and text), directly contacting the research team to request withdrawal, or they may
withdraw passively by choosing to not complete study surveys. Because interventions are not
administered to participants on an individual basis, it is not possible to modify allocated
interventions for any one specific person.
28

Throughout the trial, each intervention will be monitored by trained team members to identify
any breaches to community guidelines or other potential safety events. Intervention moderators
will have the ability to limit participants’ ability to interact within the interventions to varying
degrees, in the event of disruptive or unwelcome behavior.
4.4.8
Strategies to improve adherence to intervention protocols
Adherence to intervention protocols will be monitored regularly via fidelity observation grids
completed by team members involved in the delivery of each intervention. Feedback on
protocol adherence or nonadherence will be shared with relevant parties involved in
intervention delivery on a regular basis, via meetings.
4.4.9
Concomitant interventions permitted during the trial
We will not prevent participants from participating in other research studies or interventions
related to COVID-19 or the vaccines, during the study. We will, however, assess whether
participants have joined any other related research studies in each of our trial surveys.
4.5
Outcomes
We chose outcome measures with our stakeholders. These outcomes were selected to
address vaccine confidence. To minimize respondents’ burden, we will use validated
short-form questionnaires when possible. Our primary, secondary, and other outcomes will be
collected via web-based electronic surveys at baseline (T0), 3 weeks post-baseline (T1), 3
months post-baseline (T2), and 6 months post-baseline (T3) (see Table 3).
4.5.1
Primary Outcome Measure
COVID-19 vaccine confidence. We will assess participants’ confidence in the COVID-19
vaccines using an adapted version of the Vaccine Confidence Index (VCI)49 for COVID-19. The
COVID-19 VCI is a 3-item measure that assesses vaccine confidence across three domains:
safety, effectiveness, and importance (see Table 3). Each item is rated on a 5-point scale
ranging from 1 ('Strongly disagree') to 5 ('Strongly agree'). Participants will be considered
'confident' (score of '1') if they respond 'Agree' or 'Strongly agree' on all three items and 'not
confident' (score of '0') if one or more item responses is not 'Agree' or 'Strongly agree'.
4.5.2
Secondary Outcomes and Other Data
Change from baseline in COVID-19 Vaccine Confidence. Additional analysis of the VCI data will
include the proportion of positive VCI responses at subsequent evaluations and the individual
change in VCI between baseline and each follow-up evaluation. Subdomains of the VCI will be
evaluated separately from the composite score.
29

Likelihood of recommending (promoting) COVID-19 vaccination. Adapted Net Promoter Score
(NPS) questions50 will assess the likelihood that participants would recommend 1) COVID-19
vaccination to others who are unvaccinated, and 2) COVID-19 booster vaccination to a
coworker. Similar questions have been recommended previously51,52. We will adopt the
traditional scoring approach that categorizes respondents as promoters, passives, or
detractors.
COVID-19 vaccine uptake and intent. We will assess uptake of the COVID-19 vaccines (any
dose, initial series completion, and booster completion) using four questions that we
developed. For those who report not being vaccinated or boosted, intent to get a COVID-19
vaccine (initial series or booster) will be assessed using two questions broadly adapted from
prior work53. All participants will also be asked if they would get regular vaccines in the future if
they are recommended, using a single question.
Feeling informed about the COVID-19 vaccines. We will assess the degree to which
participants’ feel informed about the COVID-19 vaccines (have enough information and
understand that information), using two self-developed questions. Our operational definition of
feeling informed was influenced by the Decision Self-Efficacy Scale54.
Identification of COVID-19 vaccine information and misinformation. We will assess participants’
identification of COVID-19 vaccine-related information and misinformation, using four
questions that were shown to have low rates of correct identification in the pre-launch survey
data (see 4.1.2). Some questions were self-developed and some were adapted from prior
work43.
Trust in COVID-19 information from different sources. We will assess participants’ trust in
COVID-19 information provided by different people and organizations, using three items
broadly adapted from prior work55.
Change from baseline in secondary outcomes. Additional analysis of secondary outcomes will
include their assessment at subsequent evaluations and the individual change in secondary
outcomes between baseline and each follow-up evaluation.
As-treated analysis of primary and secondary outcomes. Primary and secondary outcomes will
also be assessed amongst participants who were exposed to their relevant primary study
intervention.
Contextual factors. We will identify contextual factors that may contribute to COVID-19 vaccine
intentions and decisions, including personal COVID-19 and vaccine experiences and
participation in other COVID-19 vaccine research, using a single self-developed question.
30

External contextual factors. Outside of the study surveys and throughout the trial, we will
monitor external factors that may impact participants’ views and actions towards the COVID-19
vaccines. This may include monitoring policy and mandate changes for LTCWs and changes in
the nature of the pandemic, among other things.
Participant characteristics. We will assess several participant characteristics including age,
gender (adapted from56,57), zip code, educational attainment (adapted from58), race and ethnicity
(adapted from58), health insurance (adapted from58,59), health literacy60,61, religiosity62, LTCW role,
duration of experience in long-term care, extent influenced by others regarding COVID-19
vaccination, and baseline vaccination status.
Intervention engagement. We will monitor the extent to which participants engage with their
assigned primary and refresher intervention content. We will collect online activity data (i.e.,
social media website user history, webinar attendance records, email click rates) and
participant self-reported engagement data via surveys. We will prioritize the use of online
activity data to minimize potential measurement error36. However, survey questions may be
used where online activity data is not available or is incomplete; for example, to determine
engagement with the webinar refresher recording (adapted from36) and enhanced usual practice
information. Data on engagement will be used to inform secondary trial analyses, as well as
Aim 3.
Process evaluation. We will conduct a process evaluation as a component of Aim 3 to inform
implementation and sustainability activities. Process evaluation questions will be administered
in all follow-up surveys (T1-T3). Acceptability of the interventions and control arm will be
determined via adapted NPS questions50 (i.e., likelihood of recommending to a coworker).
Similar approaches have previously been used for evaluating SDM interventions36,63–65. We will
also assess how new the information was that participants were exposed to, the
comprehension of and trust in the information (informed by66), the degree to which they felt
listened to and respected by those running the interventions (informed by67), and reasons for
not engaging with the primary or refresher interventions or control arm (adapted from65).
Table 3. Primary, secondary, and other outcomes
Measures
(Validated Y/N)
Time points for outcome assessment
T0
Baseline
T1
3 week
follow-up
T2
3 month
follow-up
T3
6 month
follow-up
Participant characteristics (~12 items)
✓
31

Contextual factors (1 item)
✓
✓
✓
✓
Process evaluation (including intervention exposure
and acceptability) (up to 8 questions)
✓
✓
✓
Vaccine Confidence Index49, adapted for COVID-19 (3
items) (primary outcome measure) (N)
✓
Change from baseline in COVID-19 vaccine
confidence (N)
✓
✓
✓
✓
Likelihood of recommending COVID-19 vaccination
(others not vaccinated), adapted Net Promoter Score
(NPS)68,69 for COVID-19 vaccines (1 item) (N)
✓
Likelihood of recommending COVID-19 booster
vaccination (coworker), adapted NPS68,69 for
COVID-19 vaccines (1 items) (N)
✓
COVID-19 vaccine uptake (any dose) (1 item) (N)
✓
COVID-19 vaccine uptake (initial series completion) (2
items) (N)
✓
COVID-19 vaccine uptake (booster completion) (1
item) (N)
✓
COVID-19 vaccine intent (any dose) (1 item) (N)
✓
COVID-19 vaccine intent (booster) (1 item) (N)
✓
COVID-19 vaccine intent (future vaccine
recommendations) (1 item) (N)
✓
✓
✓
✓
Feeling informed about COVID-19 vaccines (2 items)
(N)
✓
Identification of COVID-19 vaccine information and
misinformation (4 items) (N)
✓
Trust in COVID-19 vaccine information provided by
different sources55 (3 items) (N)
✓
Change from baseline in secondary outcomes
✓
✓
✓
✓
As-treated analysis of primary and secondary
outcomes
✓
✓
✓
✓
4.6
Sample Size and Power Calculation
Using historical Vaccine Confidence Index (VCI) data70 and current vaccination rates71, a VCI
threshold was determined that was significantly correlated with rates of vaccination (p=0.0001).
Assuming the analytic approaches outlined in sections 4.7.2 and later, the sample size of 1,800
32

LTCWs (600 per arm) provides 80% power to detect an 8% difference in the rate of scores
above this threshold between groups with a two-sided test at type I error rate of 0.05. The
sample size is sufficient to retain 80% power to detect a 10% difference (assuming outcomes
are randomly distributed across retained and lost subjects) after 40% attrition.
4.7
Trial Procedure
4.7.1
Recruitment and Screening
Prospective participants will be recruited via several online channels. We will co-design
engaging recruitment materials with our stakeholder partners, including brief animated videos
that describe the study. We will utilize the existing networks of our partner organizations,
stakeholders, and consultants, to share recruitment messaging. This will include email listservs
and social media platforms (e.g., Facebook groups). We will also utilize paid, targeted social
media advertising to share recruitment messaging on various platforms, including Facebook
and Instagram, among other platforms. Lastly, we will utilize in-person recruitment which will
include sharing recruitment materials during visits to LTC settings, conferences, and other
events, and outreach to LTC settings with requests to display study recruitment materials.
Recruitment messaging (i.e., emails, social media ads, posters, table tents, and business cards)
will include links to: 1) a study landing page where prospective participants can learn more
about the study via brief, plain-language study information in both video and written formats,
and, 2) a Qualtrics survey where they can proceed straight to eligibility screening. Screening
will involve prospective participants answering a series of questions to self-assess their
eligibility (see Table 2). The study landing page will also include a link to the Qualtrics survey.
4.7.2
Consent, Identity Verification, and Baseline Assessment
Those who meet the eligibility criteria will proceed to a study Information Sheet that will provide
information about the study objectives, processes, risk, benefits, and data protection and
sharing. To improve accessibility and facilitate the uptake of study information, we are also
developing an animated video version of the Information Sheet that will contain the same
information as the text version. Participants will be able to choose if they want to watch the
video or read the text version of the Information Sheet in the online Qualtrics survey platform
(or see both). At the end of the Information Sheet, prospective participants will be asked if they
have read/watched and understood the information and if they agree to take part in the study.
Those who respond ‘Yes’ to both questions will be considered to have consented to the study
and will proceed to the T0 Survey.
Because of the online nature of the trial, incentives provided, and early instances of fraudulent
activity72, we will implement several strategies to prevent, detect, and respond to fraudulent
study enrollment. The strategies we will adopt were informed by a prior review of methods73
33

and recommendations74,75 on this topic. We also engaged our stakeholders in this development
process to ensure a range of strategies feasible and acceptable to LTCWs.
First, in the baseline survey, we will add a reCAPTCHA bot detection filter73,74,76, cookie-based
settings that prevent multiple submissions from the same web browser73,74,77, and a message
discouraging duplicate survey completion73,74. After consent, participants will be asked to
provide the name of their current or former workplace and the type of long-term care setting it
is. Additionally, in order for us to verify that they have worked in a long-term care setting within
the last 2 years, they will be required to choose one of four different verification options:
1) Work ID badge. Participants will be able to directly upload a photo of their work ID badge, or,
arrange a time to speak with us to show their badge over a brief Zoom video call. We will
review badge images uploaded/shown to ensure the participant’s name, place of work, and
LTCW role (if listed) matches the information they have provided in the T0 (baseline) Survey. We
will also conduct an online search (using Google Chrome Incognito mode) of workplace names
to confirm they are a long-term care setting. Participants’ zip codes may also be used to aid in
this search, if required.
2) Recent pay stub from the past 2 years. Participants will be able to directly upload a photo of
a prior pay stub from within the past 2 years, or arrange a time to speak with us to show their
pay stub over a brief Zoom video call. We will review pay stub images uploaded/shown to
ensure the participant’s name and place of work matches the information they have provided in
the T0 (baseline) Survey. We will also conduct an online search (using Google Chrome
Incognito mode) of workplace names to confirm they are a long-term care setting. Participants’
zip codes may also be used to aid in this search, if required.
3) Certification or professional license number. Participants with certain roles (e.g., certified
nursing assistants in most states, health care providers) will be able to provide their
Certification Number or Professional License Number, alongside the state they are registered in
and their associated professional role. We will verify Certification/License Numbers by
searching state online registries (e.g., https://forms.nh.gov/licenseverificationtest/).
Participants’ names may also be used to aid in this search, if required. We will also conduct an
online search (using Google Chrome Incognito mode) of workplace names reported in the T0
(baseline) Survey to confirm they are a long-term care setting. Participants’ zip codes may also
be used to aid in this search, if required.
4) Work email account. Participants who have workplace email accounts will be able to email
us directly from their account with their first and last name, the name of their workplace, and
the name of the study. We will conduct an online search (using Google Chrome Incognito
mode) of workplace names and associated email domains to confirm they match, and that this
34

place of work matches the information they have provided in the T0 (baseline) Survey and is a
long-term care setting. Participants’ zip codes may also be used to aid in this search, if
required.
We will also employ TransUnion’s TLOxp verification service (www.tlo.com), to confirm the
identity of participants recruited online75. TLOxp aggregates publicly available databases and
records to authenticate and verify identity information. We will develop a standardized process
for identity verification using participant name, zip code, age range, cell phone number, and
email address.
We will inform those who select the option to upload an image of their badge or recent pay
stub that we only need to see their name, workplace, and LTCW role (if listed). They will be
instructed to cover up any other sensitive information or information they do not wish to share.
Reminder messages (text and email) and phone calls73,74 will be utilized to gather further
information from participants who fail to provide the requested information in the time frame
specified. Participants recruited in person will be presented with the same verification survey
questions as other participants, however, the need for follow-up contact to confirm their LTCW
status in some situations will be relaxed due to verification not being necessary for this
sub-population.
Participants who do not pass a verification check (including those who fail to provide the
requested information in the time frame specified) will be sent a message stating that they can
no longer remain in the study and will be unenrolled74.
Once participants have provided verification information or selected a verification option that
requires completion later (e.g., scheduling a video call with a member of the research team),
they will proceed to the remainder of the T0 Survey. The T0 Survey will include capturing
participant contact information and an assessment of outcomes indicated in Table 3 and
participant characteristics.
4.7.3
Allocation of Interventions
We will use the randomizer function built into the online Qualtrics survey platform to allocate
participants to a trial arm and associated intervention. At the very end of the T0 Survey, the
Qualtrics survey platform will automatically generate participants’ trial arm and they will be
presented with basic information pertaining to their assigned intervention at that time. A 1:1:1
allocation ratio will be used. After submitting their survey, those randomized to Arm 1
(Dialogue-Based Webinar) will be automatically redirected to a separate Qualtrics survey where
they can sign up for one of several pre-scheduled webinar sessions. Those randomized to Arm
2 (Social Media Website) will be automatically redirected to the Social Media Website. In order
35

to access website content, they will first be required to create a username and password for
the site. Those randomized to Arm 3 (Enhanced Usual Practice) will be automatically redirected
to a specific page(s) on the CDC website
(https://www.cdc.gov/coronavirus/2019-ncov/vaccines/index.html).
4.7.4
Blinding
Due to the nature of the interventions and intervention delivery, it is not possible to blind
facilitators, research team members or participants to the allocation. Participants may be aware
that they are in a control or intervention arm. The data analyst will be blinded to arm allocation
where possible.
4.7.5
Follow-up Assessments
Participants will be invited to complete three follow-up surveys. For all participants, the
follow-up surveys will be delivered three weeks post-baseline survey/randomization (T1), three
months post-baseline survey (T2), and six months post-baseline survey (T3).
4.7.6
Monitoring Enrollment
We will monitor trial enrollment on a daily basis. Particular attention will be paid to ensuring that
enrollment numbers do not exceed the capacity of scheduled Dialogue-Based Webinars.
Should enrollment numbers exceed the capacity of these webinars, we will pause study
enrollment across all trial arms. In this situation, prospective participants would still be
screened for eligibility but instead of proceeding to consent and the T0 Survey questions, their
contact information would be retained by the study team, and we would reach out to them
when recruitment opens up again.
We will also pay particular attention to recruitment milestones for 25%, 50%, 75% and 100%
enrollment targets. If enrollment numbers suggest we will not meet a recruitment milestone, we
will employ additional recruitment efforts, such as increased advertising and outreach to
additional people and organizations to utilize existing LTCW networks.
4.7.7
Tracking and Retaining Participants
Participant tracking and retention will be facilitated by several strategies.
Salesforce Customer Relationship Management (CRM). We will utilize Salesforce CRM software
to manage participant recruitment and retention. Selected survey data collected in Qualtrics
will be pushed into Salesforce on a real-time basis, through an automated integration. The
Salesforce platform will allow us to link participant data across all four surveys in one central
database, in order to reliably and efficiently track participants throughout the trial. Through
Salesforce, we will also automate all survey invitations and survey and intervention reminders
36

(sent via email or text message), to ensure that communication is timely and consistent across
all participants.
Multiple reminders in various formats. We will utilize several reminders for primary and refresher
intervention/usual practice information, follow-up surveys, and to obtain information needed for
participant verification For each series of reminders, we will utilize multiple forms of contact to
maximize engagement, including email, text messaging, and phone calls.
In the baseline survey we will request that each participant provide us with an email address
and cell phone number, and indicate their preferred method of contact (email or text). Preferred
methods of contact will be used for most study messaging, and where possible telephone calls
will be used for final reminders where email or text message reminders are not successful.
Compensation. Participants will be sent a $30 Amazon.com gift card after completing each
study survey, to thank them for their time. Gift cards will be distributed to participants via email
within a few days of completing a study survey.
4.7.8
Adherence to Protocol and Supervision
We will train all relevant study personnel in the trial procedures to maximize protocol
adherence. We will also ensure that all stakeholders involved in activities where they will have
direct contact with participants have received appropriate human subjects research training.
4.7.9 Reporting Plan and Study Termination
Our reporting plan will adhere to the CONSORT (Consolidated Standards of Reporting Trials)
2010 statement for reporting randomized trials78 and the SPIRIT (Standard Protocol Items:
Recommendations for Interventional Trials) Statement79, and will be sufficiently transparent and
comprehensive to allow for assessment of the study’s internal and external validity.
Each participant will remain in the trial for approximately 6 months after their initial T0 Survey
completion, or after all of their Aim 3 interviews have been conducted (during the Sustainability
Phase for Implementation Consultations). Once all participants have been sent their final T3
Survey, those originally assigned to Arm 3: Enhanced Usual Practice, will be given access to
the study interventions. Separate to the trial itself, we will monitor their engagement with the
interventions and administer a brief survey 3 months after initial exposure.
37

4.8
Statistical Analysis
4.8.1
Data Source Accuracy
Initial examination of data will include descriptive statistics and frequency distributions in order
to check planned analysis assumptions, identify missing data, and check data source
adequacy. This process will also inform any necessary variable re-coding prior to data analysis.
4.8.2
Analyses Corresponding to Trial Aims 1 and 2
All analyses pertaining to study Aims 1 and 2 will be conducted on an intention-to-treat (ITT)
basis (i.e. the arm participants were assigned to) and as-treated basis (i.e. whether they
engaged with their assigned intervention or control). A detailed data analysis methodology,
including planned statistical tests, timepoints for outcome assessment, and treatment of
missing data, is included in Appendix B.
For Aim 1, hypothesis 1.1, we will conduct one-tailed tests (superiority analysis) to compare the
impact of each of the two intervention arms against the enhanced usual practice arm on
primary and secondary outcomes. For Aim 1, hypothesis 1.2, we will conduct a two-tailed test
(equivalence analysis) of primary and secondary outcomes between the two intervention arms.
While we hypothesize that the dialogue-based webinar intervention (Arm 1) will be superior to
the social media website intervention (Arm 2), a finding of superiority, inferiority, or no
distinguishable difference will be a valuable finding.
For Aim 2, hypothesis 2.1, our mediation analyses seek to identify the relationship between the
interventions, mediator variables, and primary and secondary outcomes. We are interested in
whether interventions operate through the mediator rather than directly affecting the outcome.
If the results for Aim 1 are non-significant, we will determine whether it is a null effect of the
intervention on the mediator or a null effect of the mediator on the outcome. We will determine
mediation strength and mechanism generalizability by comparing effects across subgroups.
For Aim 2, hypothesis 2.2, our exploratory moderation analyses (also referred to as
heterogeneity of treatment effects (HTE) analyses) seek to understand whether certain
participant characteristics and beliefs influence the relationship between the interventions and
vaccine confidence, as well as other secondary outcomes. We will explore moderators
including (but not limited to) vaccination status, religious beliefs, age, race, ethnicity, perceived
influence of others, and personal experiences with COVID-19.
Because of the size, scope, and complexity of this study, exploratory analyses of relationships
within the data will be conducted to identify factors to inform future analysis (see Additional File
4). Exploratory, hypothesis-free analyses will be performed using data clustering to analyze
participants’ demographic, geographic, or temporal links, which may define statistically unlikely
38

outcome groupings.
4.8.3
Data Linkage
All trial survey data will be linked to other study data (Salesforce, online activity data) at the
participant-level, using participants’ email address and/or other unique variables that are
available. These data will be linked following exports from Salesforce and the respective online
platforms (social media website, analytics platforms, and Zoom). During the trial, a minority of
online activity data will be imported directly into Salesforce (either by automated integration of
manual input) in order to facilitate participant tracking and the distribution of tailored invitation
and reminder messages for participants in each intervention arm.
4.8.4
Analyses Corresponding to Aim 3
Using a framework analysis, guided by relevant domains of CFIR34,80, we will answer the
following questions: 1) How were characteristics of the interventions perceived by participants
and stakeholders (CFIR i); 2) How did external contextual factors influence intervention use
(CFIR ii); and, 3) What factors affected implementation (CFIR iii/CFIR iv)35. We will collect data
from relevant stakeholders. We will also use online activity data, webinar recordings,
observations, and meeting note templates from CFIR, adapted with stakeholders, as a part of
our learning community29,34,80. We will adapt Codebook Templates, double code the interviews,
and where possible integrate qualitative and quantitative data sources (e.g., interview,
observational, recording, and online activity data to inform the Process Evaluation (fidelity,
dose, acceptability, and reach of interventions). Where possible, these data will also be
triangulated with data from Aims 1 and 2, to inform interpretations and study conclusions.
4.9
Data Management
4.9.1
Data Management Plan
Collecting data. The primary source of trial data will be collected via online surveys
administered to recruited study participants. We will also use activity data from the social
media website and associated analytics software (Google Analytics and Google Cloud
Platform), (b) Zoom webinar content/recordings and registration and attendance data, and (c)
user-entered or participant-generated data from Salesforce. Data from Aim 3 interviews will be
collected via audio recordings and subsequently transcribed by a transcription company
(Civicom, Inc.). Table 4 below details all platforms and organizations that will receive study data
during the trial, and which data they will receive.
39

Table 4. Data platforms and data received
Location
Description of Data Received
Qualtrics
(Participant survey portal)
All participant survey data (e.g., screening and consent
information, contact information, participant characteristics, and
primary and secondary study outcomes).
Salesforce
(Participant management;
data will automatically
flow in from Qualtrics and
Zoom)
Participant information flowing from Qualtrics/Zoom includes
contact information, characteristics, vaccine and booster status,
trial arm assignment, Zoom registration, and survey metadata
(e.g., completion date, unique study ID). Data will also be
generated in Salesforce by research team members’ manual
entry of information (e.g., call logs) or participant-generated
information (i.e., response to a text message sent from within
Salesforce).
Mogli/Telnyx
(Integration for Salesforce
that supports text
messaging)
Participants’ first name, cell phone number, and study-related
messaging (includes unique links to surveys and interventions).
ActiveCampaign
(Email marketing software
for sending intervention
refreshers)
Participants’ first name, last name, email address, and
study-related messaging.
Zoom
(Includes Panopto for
Zoom video storage)
Participants’ name, email address, meeting registration, meeting
attended, time joined and left meeting, chat logs, meeting
recordings (audio and visual, including participant faces if they
choose to show their video).
Social Media Website
(WordPress site that will
be hosted on a
Dartmouth College
server)
Participants’ name, username and password, email address, and
any other information related to their website engagement.
Google Analytics
(Analytics of WordPress
site data)
Anonymous participant website engagement information
(aggregate data at user and site level) (e.g., total number of site
visits, location of users, number of unique page visits per user
and per page).
Google Cloud Platform
Participant website engagement information stored in Google
Analytics.
Civicom
Interview and webinar audio recordings.
40

(Transcription company)
Dartmouth SharePoint
Any raw and individual-level study data that needs to be shared
between members of the research team (at Dartmouth and
collaborating institutions) and Data and Safety Monitoring Board
(DSMB) members.
TLOxp
(Identity verification
service)
Participant information entered into the online system to conduct
identity verification. Information will include participant name, zip
code, age range, cell phone number, and email address.
Handling data. Oversight of trial data will be the responsibility of the Trial Manager and Data
Manager. Prior to data collection commencing, we will work with a third party vendor, Cloud for
Good, to implement our Salesforce platform. This will include setting up a secure integration
with Qualtrics and Zoom, so that data being generated in these platforms will flow directly into
Salesforce, in real time. The Cloud for Good team will also integrate a text messaging
application (Mogli) within Salesforce, to facilitate automated text messaging to participants. All
data in Salesforce will be backed up on a weekly basis, and stored on one or more
password-protected and encrypted external hard drives.
Access to all software and platforms where data are being handled will be password-protected,
and will only be afforded to research team members involved in data collection, management,
and/or analysis and reporting. Versions of study datasets from all originating sources will be
managed by a single version control document. This document will be maintained by the Data
Manager and Trial Manager, and will include identified and de-identified data sets used by
Dartmouth team members internally and those shared with team members at collaborating
institutions.
Describing and organizing data. Prior to the analysis of trial data, a data dictionary will be
developed for all survey variables. This dictionary will be used by anyone involved in data
analysis and reporting. The dictionary will accompany datafiles that are subsequently shared
with others outside the research team (see Sharing Data below). Data stored in Qualtrics,
Salesforce, the social media website, Google Analytics/Google Cloud Platform,, and
Zoom/Panopto will be exported in file formats (to be determined) that best support analysis.
Storing and preserving data. Any data exported from Qualtrics, Salesforce, the social media
website, Google Analytics/Google Cloud Platform, or Zoom/Panopto, will be stored on
password-protected computers or external hard drives. Data will also be stored in the
Dartmouth SharePoint system, to facilitate secure file management and sharing between
research team members at Dartmouth College and other collaborating institutions, as well as
members of the DSMB.
41

De-identification, crosswalks, and data destruction. To reduce the duration that identifying
information will be linked to other study data, at the completion of data collection, we will
download all data that was collected and stored in cloud-based systems and set up
crosswalks for each of the downloaded datasets. We will do the same for any local files used in
the process of participant management, such as tracking spreadsheets. The crosswalks will
utilize a unique and anonymous ‘Study ID’ variable and identifiable variable fields depending on
the data source. We will set up a master key file that will include only Study ID and all
identifiable variable fields. This key file will be used for all data crosswalks. The master key file
will be stored separately to the other study data in the secure SharePoint cloud system as well
as on one or more encrypted and password-protected external hard drives. Remaining
deidentified datasets will include the Study ID variable, which will be the only link between
them and the identifiable study data. The only exception to this de-identification and crosswalk
process is the video and audio recordings of webinar intervention sessions, as these will be
needed for process evaluation analyses which will continue past data collection completion.
It is not possible to create crosswalks and remove identifying information from participant data
prior to data collection completion, as many of our cloud-based systems and participant
management processes require the identifying information for our study to function correctly
(e.g., Salesforce for participant communication; study website to manage user registrations
and notifications; question responses cannot be deleted from active Qualtrics survey datasets,
Zoom video/audio recordings for process evaluation). We will also need to create weekly
back-ups of all data contained in Salesforce (including identifiers) to ensure we are protected in
the event of any data loss.
At the completion of data collection and following the downloading of data from cloud-based
systems, we will delete all participant data in cloud-based systems (excluding SharePoint) by
working with Dartmouth IT and external study vendors to the best of our ability.
Once all study publications and reports are finalized, we will delete all copies of the master key
file that contain participant identifying information from SharePoint and external hard drives
(using secure hard drive erase procedures) and with the assistance of Dartmouth IT services.
De-identified study data will be stored indefinitely.
Maintaining data. Upon completion of the project, we will prepare a full data package of all
randomized controlled trial data. The full data package will be maintained for at least seven
years. Information provided to prospective participants at the time of consent will include a
statement that de-identified data collected as part of the study may be used for other studies
and shared with researchers not part of this study team. The Principal Investigators (PIs) will be
responsible for maintaining and responding to data-related inquiries.
42

Sharing data. Any raw, individual-level data shared between members of the study team (e.g.,
survey datasets, interview recordings and transcripts) or the DSMB during and after the
completion of data collection will be done so via the secure Dartmouth SharePoint platform.
For data security purposes, non-Dartmouth team members will be required to obtain a
Dartmouth Sponsored Account in order to access the SharePoint system.
Following completion of the study, and as per PCORI’s requirements, we will make the Full
Data Package available to a PCORI-designated data repository, to facilitate data sharing with
the broader scientific community.
4.9.2
Data and Safety Monitoring Plan
A Data and Safety Monitoring Board (DSMB) will be appointed to provide additional oversight
in the trial. The attached DSMB Charter describes the DSMB’s scope and procedures in detail.
To summarize, the DSMB will meet and review data annually throughout the project (data
review in meetings 2 and 3 only - see Charter previously submitted to PCORI for details), with
three planned meetings held in total. The DSMB will include a Chair who has no direct
involvement in the trial (see DSMB Charter). The DSMB will also include the following
individuals who have not had prior involvement in the trial:
1. Board member with expertise in data safety monitoring of clinical trials
2. Board member with expertise in biostatistics and clinical trials methodology
3. Executive Secretary with experience in SDM research
The DSMB will operate independently from the study sponsor and funder.
Independence is essential to ensure that the DSMB members are objective and capable of an
unbiased assessment of the study's safety and efficacy of data. No members of the DSMB
should have financial, proprietary, professional or other interests that may affect impartial
decision making by the DSMB. The following will ensure the independence of the DSMB:
●
Members of the DSMB will not participate as investigators in the study under review.
●
Members of the DSMB must not have a direct interest in knowing or influencing trial
outcomes or have a financial, proprietary, or intellectual interest in the outcomes of the
study under review.
●
DSMB members must disclose all potential new conflicts of interests (COIs).
The DSMB will review the protocol, data collected, and advise the PIs on any potential risks
and risk mitigation plans. The DSMB recommendations will be discussed with the PIs as well
43

as the Trial Steering Group, which meets quarterly. The Trial Steering Group will consider all
DSMB recommendations and revise relevant aspects of the trial accordingly. Those
recommendations will also be fed back to PCORI annually as part of the progress reports. The
PIs will also be responsible for sending a summary of DSMB recommendations to the
Dartmouth CPHS (IRB) after each DSMB meeting. If there are adverse or unanticipated
events/problems that require reporting, PCORI and the CPHS will be notified within a day of
the DSMB meeting. If there are no adverse or unanticipated events/problems that require
reporting, DSMB meeting minutes and recommendations will be communicated to PCORI in
the interim progress report.
Trial activity will also be monitored by members of the study team on an ongoing basis, outside
of annual DSMB meetings. We believe the likelihood of Serious Adverse Events (SAEs) in the
trial that would require immediate reporting to be extremely low, as there are no invasive
procedures related to the interventions. However, it is possible that a participant may
experience psychological discomfort or distress as a result of answering certain survey
questions or through engagement in either of the two study interventions. All potential adverse
events or unanticipated events/problems, regardless of their severity, will be reported to the
PIs. All events will be discussed by the PIs and relevant members of the study team, and a
determination will be made as to whether they constitute Reportable New Information (e.g., it
increases the risk to subjects; a subject experienced unexpected harm). Any event deemed to
be Reportable New Information will be reported to Dartmouth CPHS (IRB) and the DSMB will
convene urgently and review the event in question.
4.9.3
Auditing of Trial Conduct
This trial will have no formal auditing process, however, trial conduct will be overseen by the
Dartmouth Center for the Protection of Human Subjects (CPHS) and the DSMB. Our
intervention process evaluation will also be carried out by the Center for Program Design and
Evaluation (CPDE).
44

5
Ethics, Write-up and Dissemination
5.1
Research Ethics Approval
The randomized controlled trial was initially approved by the Dartmouth CPHS on January 11th,
2022. The initial approval for Aim 3 activities was granted on September 10th, 2021.
5.2
Protocol Amendments
Throughout the course of the study, protocol modifications will be reported to the relevant
parties and groups (e.g., project investigators, PCORI, Dartmouth CPHS), where required. This
may include changes to trial procedures, materials, and planned analyses.
5.3
Dissemination and Implementation in Other Settings
The proposed research findings are highly relevant to public health as we seek to address
vaccine confidence among LTCWs. Because the interventions are scalable, study outputs will
interest a wide variety of target audiences ranging from large public health organizations like
the Centers for Disease Control, IHI, and the Agency for Healthcare Research and Quality, and
organizations dedicated to long-term care like NAHCA, American Medical Directors Association
(AMDA), and to LTCWs. We will work with each target audience to create dissemination and
implementation strategies that are tailored to their needs and interests and are understandable
to them. Evidence generated from this study is also likely to be applicable and generalizable to
future COVID-19 vaccines and to other vaccination programs.
5.3.1
Academic Channels
Although peer-reviewed scientific journals are not commonly accessed by LTCWs, they remain
the primary source of knowledge and dissemination of research findings to influence health
professionals, policy makers, and healthcare systems. Open-access journals will systematically
be used to ensure that LTCWs, their employers and other non-academic stakeholders have free
and easy access to our findings and are able to rapidly and successfully implement the
intervention(s) and/or enhanced usual practice in long-term care facilities. We will also present
the findings at up to four domestic and international scientific conferences and professional
meetings.
5.3.2
Professional organizations and healthcare delivery systems
We will prioritize working with national organizations, such as NAHCA, and IHI, and utilizing
their well-established dissemination channels to distribute our findings to LTCWs and other
community stakeholders. Specifically, IHI has a dissemination engine, and we will collaborate
with Alice Bonner from IHI in our communications, who has worked extensively with the Age
Friendly initiative, which has gained substantial traction recently. We will also create newsletters
45

and press releases using lay language. All documents will be written in partnership with our
stakeholder partners and SAG, and will be freely available on the study website.
5.3.3
Social Media and Lay Press
We will also use social media and partner with NAHCA, which consistently engages their
members through social media, to disseminate key study messages. We will work with our
SAG to plan an effective social media campaign using regular Twitter feeds, blogs on the study
website, and planned “tweet chats”. We will also reach the lay public who may not have access
to social media by using newspaper editorials, lay press, and magazine print. We have
successfully used this approach in PCORI-funded studies before.
5.4
Possible Barriers to Dissemination and Implementation
We understand that one of the greatest challenges to implementing results in routine settings is
engagement from LTCWs and the organizations that represent them. To ensure that we have
sustainable partnerships, we have reached out to recognized organization leaders in the field of
public health and long-term care facilities (IHI, NAHCA). We have also collaborated with
multiple stakeholder partners who will be involved in all aspects of the project, including its
dissemination and implementation. We will also work with influencers on social media to try
and reach a broader group. We also recognize that language may be a barrier to
implementation. Not all LTCWs will feel comfortable using an intervention written and delivered
in English. We will thus plan to translate the successful intervention(s) in the most commonly
spoken languages for this target group.
5.5
Making Results Available to Participants
In the final study survey, we will ascertain participants’ preferences for receiving (via email or
mail) information on the study’s findings. Once all study data are collected and analyzed, we
will begin developing a pictorial lay summary of the research aims, methods, and key findings.
Information on study results in the final lay summary will be emailed via the email address
participants provided at the outset of the study or directly mailed to the study participants if
that is their preference. The email will also contain a link to the full results publication online as
well as the study website, which will contain additional information about the study and its
dissemination efforts. For participants who chose to receive the summary in the mail, we will
send a copy of the published manuscript to them as well.
In addition to direct communication of aggregate study results via email or mail, participants
may also become aware of study results via the broader dissemination efforts described earlier,
targeting community stakeholder organizations (e.g., press releases, newsletters, social media
outreach).
46

6
Timeline
47

7
References
1.
Schaffer DeRoo S, Pudalov NJ, Fu LY. Planning for a COVID-19 Vaccination Program.
JAMA. 2020;323(24):2458-2459. doi:10.1001/jama.2020.8711
2.
Liu Y, Rocklöv J. The reproductive number of the Delta variant of SARS-CoV-2 is far higher
compared to the ancestral SARS-CoV-2 virus. J Travel Med. 2021;28(7).
doi:10.1093/jtm/taab124
3.
Paltiel AD, David Paltiel A, Zheng A, Schwartz JL. Speed Versus Efficacy: Quantifying
Potential Tradeoffs in COVID-19 Vaccine Deployment. Annals of Internal Medicine.
2021;174(4):568-570. doi:10.7326/m20-7866
4.
Lurie N, Saville M, Hatchett R, Halton J. Developing Covid-19 Vaccines at Pandemic
Speed. N Engl J Med. 2020;382(21):1969-1973. doi:10.1056/NEJMp2005630
5.
Dror AA, Eisenbach N, Taiber S, et al. Vaccine hesitancy: the next challenge in the fight
against COVID-19. Eur J Epidemiol. 2020;35(8):775-779. doi:10.1007/s10654-020-00671-y
6.
Naeem SB, Bhatti R. The Covid-19 “infodemic”: a new front for information professionals.
Health Info Libr J. 2020;37(3):233-239. doi:10.1111/hir.12311
7.
Roozenbeek J, Schneider CR, Dryhurst S, et al. Susceptibility to misinformation about
COVID-19 around the world. R Soc Open Sci. 2020;7(10):201199.
doi:10.1098/rsos.201199
8.
Mian A, Khan S. Coronavirus: the spread of misinformation. BMC Med. 2020;18(1):89.
doi:10.1186/s12916-020-01556-3
9.
Haase N, Schmid P, Betsch C. Impact of disease risk on the narrative bias in vaccination
risk perceptions. Psychol Health. 2020;35(3):346-365.
doi:10.1080/08870446.2019.1630561
10. Betsch C, Ulshöfer C, Renkewitz F, Betsch T. The influence of narrative v. statistical
information on perceiving vaccination risks. Med Decis Making. 2011;31(5):742-753.
doi:10.1177/0272989X11400419
11. de Figueiredo A, Simas C, Karafillakis E, Paterson P, Larson HJ. Mapping global trends in
vaccine confidence and investigating barriers to vaccine uptake: a large-scale
retrospective temporal modelling study. Lancet. 2020;396(10255):898-908.
doi:10.1016/S0140-6736(20)31558-0
12. Gharpure R, Guo A, Bishnoi CK, et al. Early COVID-19 First-Dose Vaccination Coverage
Among Residents and Staff Members of Skilled Nursing Facilities Participating in the
Pharmacy Partnership for Long-Term Care Program - United States, December
2020-January 2021. MMWR Morb Mortal Wkly Rep. 2021;70(5):178-182.
doi:10.15585/mmwr.mm7005e2
48

13. Unroe KT, Evans R, Weaver L, Rusyniak D, Blackburn J. Willingness of long-term care staff
to receive a COVID-19 vaccine: A single state survey. J Am Geriatr Soc.
2021;69(3):593-599. doi:10.1111/jgs.17022
14. Barnett ML, Grabowski DC. Nursing Homes Are Ground Zero for COVID-19 Pandemic.
JAMA Health Forum. 2020;1(3):e200369-e200369.
doi:10.1001/jamahealthforum.2020.0369
15. Van Houtven CH, DePasquale N, Coe NB. Essential Long-Term Care Workers Commonly
Hold Second Jobs and Double- or Triple-Duty Caregiving Roles. J Am Geriatr Soc.
2020;68(8):1657-1660. doi:10.1111/jgs.16509
16. Baughman RA, Stanley B, Smith KE. Second Job Holding Among Direct Care Workers and
Nurses: Implications for COVID-19 Transmission in Long-Term Care. Med Care Res Rev.
Published online November 19, 2020:1077558720974129.
doi:10.1177/1077558720974129
17. Bell S, Clarke RM, Ismail SA, et al. COVID-19 vaccination beliefs, attitudes, and behaviours
among health and social care workers in the UK: a mixed-methods study. medRxiv.
Published online 2021.
https://www.medrxiv.org/content/10.1101/2021.04.23.21255971v1.full
18. Lorini C, Collini F, Gasparini F, et al. Health Literacy, Vaccine Confidence and Influenza
Vaccination Uptake among Nursing Home Staff: A Cross-Sectional Study Conducted in
Tuscany. Vaccines (Basel). 2020;8(2). doi:10.3390/vaccines8020154
19. Dubé E, Gagnon D, MacDonald NE, SAGE Working Group on Vaccine Hesitancy.
Strategies intended to address vaccine hesitancy: Review of published reviews. Vaccine.
2015;33(34):4191-4203. doi:10.1016/j.vaccine.2015.04.041
20. Dubé E, Laberge C, Guay M, Bramadat P, Roy R, Bettinger J. Vaccine hesitancy: an
overview. Hum Vaccin Immunother. 2013;9(8):1763-1773. doi:10.4161/hv.24657
21. Jarrett C, Wilson R, O’Leary M, Eckersberger E, Larson HJ. Strategies for addressing
vaccine hesitancy – A systematic review. Vaccine. 2015;33(34):4180-4190.
doi:10.1016/j.vaccine.2015.04.040
22. Jarrett C, Wilson R, O’Leary M, Eckersberger E, Larson HJ, SAGE Working Group on
Vaccine Hesitancy. Strategies for addressing vaccine hesitancy - A systematic review.
Vaccine. 2015;33(34):4180-4190. doi:10.1016/j.vaccine.2015.04.040
23. Glanz JM, Wagner NM, Narwaney KJ, et al. Web-based Social Media Intervention to
Increase Vaccine Acceptance: A Randomized Controlled Trial. Pediatrics. 2017;140(6).
doi:10.1542/peds.2017-1117
24. Daley MF, Narwaney KJ, Shoup JA, Wagner NM, Glanz JM. Addressing Parents’ Vaccine
Concerns: A Randomized Trial of a Social Media Intervention. Am J Prev Med.
2018;55(1):44-54. doi:10.1016/j.amepre.2018.04.010
49

25. Ortiz RR, Smith A, Coyne-Beasley T. A systematic literature review to examine the potential
for social media to impact HPV vaccine uptake and awareness, knowledge, and attitudes
about HPV and HPV vaccination. Hum Vaccin Immunother. 2019;15(7-8):1465-1475.
doi:10.1080/21645515.2019.1581543
26. Scalia P, Durand MA, Elwyn G. Shared decision-making interventions: An overview and a
meta-analysis of their impact on vaccine uptake. J Intern Med. 2022;291(4):408-425.
doi:10.1111/joim.13405
27. Durand MA, Scalia P, Elwyn G. Can shared decision making address COVID-19 vaccine
hesitancy? BMJ Evid Based Med. 2022;27(3):159-161. doi:10.1136/bmjebm-2021-111695
28. Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C. Effectiveness-implementation hybrid
designs: combining elements of clinical effectiveness and implementation research to
enhance public health impact. Med Care. 2012;50(3):217-226.
doi:10.1097/MLR.0b013e3182408812
29. Fernandez ME, Ten Hoor GA, van Lieshout S, et al. Implementation Mapping: Using
Intervention Mapping to Develop Implementation Strategies. Front Public Health.
2019;7:158. doi:10.3389/fpubh.2019.00158
30. Moore GF, Audrey S, Barker M, et al. Process evaluation of complex interventions: Medical
Research Council guidance. BMJ. 2015;350:h1258. doi:10.1136/bmj.h1258
31. AncaVitcu EL, Vitcu L, Marcu A. Multi-stage maximum variation sampling in health
promotion programs’ evaluation. J Prev Med. 2007;15:5-18.
https://scholar.archive.org/work/xyowxnzsvnfobn5ksjpg7n3zuy/access/wayback/http://ww
w.jmpiasi.ro/2007/1%20vitcu.pdf
32. Guest G, Bunce A, Johnson L. How Many Interviews Are Enough? Field methods.
2006;18(1):59-82. doi:10.1177/1525822X05279903
33. Joseph-Williams N, Evans R, Edwards A, et al. Supporting informed decision making
online in 20 minutes: an observational web-log study of a PSA test decision aid. J Med
Internet Res. 2010;12(2):e15. doi:10.2196/jmir.1307
34. Kirk MA, Kelley C, Yankey N, Birken SA, Abadie B, Damschroder L. A systematic review of
the use of the Consolidated Framework for Implementation Research. Implement Sci.
2016;11(1):1-13. doi:10.1186/s13012-016-0437-z
35. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering
implementation of health services research findings into practice: a consolidated
framework for advancing implementation science. Implement Sci. 2009;4:50.
doi:10.1186/1748-5908-4-50
36. Thompson R, Stevens G, Manski R, et al. Right For Me: a pragmatic multi-arm cluster
randomised controlled trial of two interventions for increasing shared decision-making
about contraceptive methods. bioRxiv. Published online June 28, 2021.
doi:10.1101/2021.06.25.21257891
50

37. Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research
(COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care.
2007;19(6):349-357. doi:10.1093/intqhc/mzm042
38. Peretti-Watel P, Larson HJ, Ward JK, Schulz WS, Verger P. Vaccine hesitancy: clarifying a
theoretical framework for an ambiguous notion. PLoS Curr. 2015;7.
doi:10.1371/currents.outbreaks.6844c80ff9f5b273f34c91f71b7fc289
39. MacDonald NE, SAGE Working Group on Vaccine Hesitancy. Vaccine hesitancy: Definition,
scope and determinants. Vaccine. 2015;33(34):4161-4164.
doi:10.1016/j.vaccine.2015.04.036
40. Durand MA, Yen RW, O’Malley AJ, et al. What matters most: Randomized controlled trial of
breast cancer surgery conversation aids across socioeconomic strata. Cancer. Published
online November 10, 2020. doi:10.1002/cncr.33248
41. Wallerstein NB, Duran B. Using community-based participatory research to address health
disparities. Health Promot Pract. 2006;7(3):312-323. doi:10.1177/1524839906289376
42. Hamel, Lopes, Sparks, Stokes, Brodie. KFF COVID-19 vaccine monitor: April 2021. The
Kaiser Family Foundation.
43. Hamel L, Lopes L, Kirzinger A, Sparks G, Stokes M, Brodie M. KFF COVID-19 vaccine
monitor: Media and misinformation. Kaiser Family Foundation. Published online 2021.
https://www.kff.org/coronavirus-covid-19/poll-finding/kff-covid-19-vaccine-monitor-media
-and-misinformation/
44. Saunders CH, Sierpe A, Stevens G, et al. Co-Development of a Web Application
(COVID-19 Social Site) for Long-Term Care Workers (“Something for Us”): User-Centered
Design and Participatory Research Study. J Med Internet Res. 2022;24(9):e38359.
https://www.jmir.org/2022/9/e38359/
45. Polling: KFF COVID-19 VACCINE MONITOR. Accessed December 18, 2022.
https://www.kff.org/polling/
46. Donnelly KZ, Foster TC, Thompson R. What matters most? The content and concordance
of patients’ and providers' information priorities for contraceptive decision making.
Contraception. 2014;90(3):280-287. doi:10.1016/j.contraception.2014.04.012
47. Elwyn G, Durand MA, Song J, et al. A three-talk model for shared decision making:
multistage consultation process. BMJ. 2017;359:j4891. doi:10.1136/bmj.j4891
48. Wiltsey Stirman S, Baumann AA, Miller CJ. The FRAME: an expanded framework for
reporting adaptations and modifications to evidence-based interventions. Implement Sci.
2019;14(1):58. doi:10.1186/s13012-019-0898-y
49. Larson HJ, Schulz WS, Tucker JD, Smith DMD. Measuring Vaccine Confidence:
Introducing a Global Vaccine Confidence Index. PLoS Currents. Published online 2015.
doi:10.1371/currents.outbreaks.ce0f6177bc97332602a8e3fe7d7f7cc4
51

50. Reichheld FF. The one number you need to grow. Harv Bus Rev. 2003;81(12):46-54, 124.
https://www.ncbi.nlm.nih.gov/pubmed/14712543
51. U.S. Department of Health and Human Services. Vaccine Confidence Survey Question
Bank. Centers for Disease Control and Prevention.
https://www.cdc.gov/vaccines/covid-19/vaccinate-with-confidence/rca-guide/downloads/
CDC_RCA_Guide_2021_Tools_AppendixD_Surveys-508.pdf
52. Sanyal N, Nundy S. To Overcome Vaccine Hesitancy, We Need a Better Patient
Experience. Harvard Business Review. Published online January 15, 2021. Accessed
November 22, 2022.
https://hbr.org/2021/01/to-overcome-vaccine-hesitancy-we-need-a-better-patient-experie
nce
53. Latkin CA, Dayton L, Yi G, Colon B, Kong X. Mask usage, social distancing, racial, and
gender correlates of COVID-19 vaccine intentions among adults in the US. PLoS One.
2021;16(2):e0246970. doi:10.1371/journal.pone.0246970
54. Bunn H, O’Connor A. Validation of client decision-making instruments in the context of
psychiatry. Can J Nurs Res. 1996;28(3):13-27.
https://www.ncbi.nlm.nih.gov/pubmed/8997937
55. Wellcome Global Monitor 2018. Accessed September 27, 2021.
https://wellcome.org/reports/wellcome-global-monitor/2018
56. Puckett JA, Brown NC, Dunn T, Mustanski B, Newcomb ME. Perspectives from
Transgender and Gender Diverse People on How to Ask About Gender. LGBT Health.
2020;7(6):305-311. doi:10.1089/lgbt.2019.0295
57. Office of Regulatory affairs and Research Compliance, Harvard TH Chan School of Public
Health. ORARC Tip Sheet: Inclusive Demographic Data Collection. Published online 2020.
https://cdn1.sph.harvard.edu/wp-content/uploads/sites/2102/2020/04/ORARC-Tip-Sheet-I
nclusive-Demographic-Data-Collection.pdf
58. United States Census Bureau. The American Community Survey. Published online 2021.
https://www2.census.gov/programs-surveys/acs/methodology/questionnaires/2021/quest
21.pdf
59. Jerman J, Jones RK, Onda T. Characteristics of U.S. Abortion Patients in 2014 and
Changes Since 2008.; 2016.
https://www.guttmacher.org/sites/default/files/report_pdf/characteristics-us-abortion-patie
nts-2014.pdf
60. Chew LD, Bradley KA, Boyko EJ. Brief questions to identify patients with inadequate health
literacy. Fam Med. 2004;36(8):588-594. https://www.ncbi.nlm.nih.gov/pubmed/15343421
61. Wallace LS, Rogers ES, Roskos SE, Holiday DB, Weiss BD. Brief report: screening items to
identify patients with limited health literacy skills. J Gen Intern Med. 2006;21(8):874-877.
doi:10.1111/j.1525-1497.2006.00532.x
52

62. Pew Research Center. 2014 Religious Landscape Study (RLS-II) - Main Survey of
Nationally Representative Sample of Adulta - Final Questionnaire. Published online 2014.
https://www.pewforum.org/wp-content/uploads/sites/7/2015/11/201.11.03_rls_ii_question
naire.pdf
63. Van Citters AD, Gifford AH, Brady C, et al. Formative evaluation of a dashboard to support
coproduction of healthcare services in cystic fibrosis. J Cyst Fibros. 2020;19(5):768-776.
doi:10.1016/j.jcf.2020.03.009
64. Day FC, Pourhomayoun M, Keeves D, et al. Feasibility study of an EHR-integrated mobile
shared decision making application. Int J Med Inform. 2019;124:24-30.
doi:10.1016/j.ijmedinf.2019.01.008
65. Thompson RL. Family Planning Ahead. ClinicalTrials.gov identifier: NCT03500952.
Accessed November 1, 2022. https://clinicaltrials.gov/ct2/show/NCT03500952
66. Teo T. Development and validation of the E-learning Acceptance Measure (ElAM). The
Internet and Higher Education. 2010;13(3):148-152. doi:10.1016/j.iheduc.2010.02.001
67. Kaufmann R, Sellnow DD, Frisby BN. The development and validation of the online learning
climate scale (OLCS). Communication Education. 2016;65(3):307-321.
doi:10.1080/03634523.2015.1101778
68. Reichheld FF. The one number you need to grow. Harv Bus Rev. 2003;81(12):46-54.
69. Hamilton DF, Lane JV, Gaston P, et al. Assessing treatment outcomes using a single
question: the net promoter score. Bone Joint J. 2014;96-B(5):622-628.
doi:10.1302/0301-620X.96B5.32434
70. Nuzhath T, Hotez PJ, Damania A, Liu PS, Colwell B. Creation of a Global Vaccine Risk
Index. PLoS One. 2022;17(8):e0272784. doi:10.1371/journal.pone.0272784
71. CDC. COVID data tracker. Centers for Disease Control and Prevention. March 28, 2020.
Accessed November 22, 2022. https://covid.cdc.gov/covid-data-tracker
72. Stevens G, Johnson LC, Saunders CH, et al. The CONFIDENT study protocol: a
randomized controlled trial comparing two methods to increase long-term care worker
confidence in the COVID-19 vaccines. BMC Public Health. 2023;23(1):384.
doi:10.1186/s12889-023-15266-x
73. Teitcher JEF, Bockting WO, Bauermeister JA, Hoefer CJ, Miner MH, Klitzman RL.
Detecting, Preventing, and Responding to “Fraudsters” in Internet Research: Ethics and
Tradeoffs. J Law Med Ethics. 2015;43(1):116-133. doi:10.1111/jlme.12200
74. Stevens G, Washburn H, Theiler R, Woodhams E, Donnelly K, Thompson R. Enrolment
fraud in online shared decision-making research: Lessons learned from an internet-based
randomised controlled trial. Presented at: 10th International Shared Decision-Making
Conference (ISDM); Quebec City, Canada.
https://fourwaves-sots.s3.amazonaws.com/static/media/uploads/2019/06/28/isdm2019-or
53

alsessionsbooklet-2019-06-28.pdf
75. Glazer JV, MacDonnell K, Frederick C, Ingersoll K, Ritterband LM. Liar! Liar! Identifying
eligibility fraud by applicants in digital health research. Internet Interv. 2021;25:100401.
doi:10.1016/j.invent.2021.100401
76. QualtricsXM. Captcha verification question. 2015. Accessed September 20, 2022.
https://www.qualtrics.com/support/survey-platform/survey-module/editing-questions/que
stion-types-guide/advanced/captcha-verification/
77. QualtricsXM. Security survey options. 2022. Accessed September 20, 2022.
https://www.qualtrics.com/support/survey-platform/survey-module/survey-options/survey
-protection/
78. Schulz KF, Altman DG, Moher D, CONSORT Group. CONSORT 2010 statement: updated
guidelines for reporting parallel group randomised trials. Int J Surg. 2011;9(8):672-677.
doi:10.1016/j.ijsu.2011.09.004
79. Chan AW, Tetzlaff JM, Altman DG, et al. SPIRIT 2013 statement: defining standard protocol
items for clinical trials. Ann Intern Med. 2013;158(3):200-207.
doi:10.7326/0003-4819-158-3-201302050-00583
80. Qualitative Data. Accessed April 25, 2021.
https://cfirguide.org/evaluation-design/qualitative-data/
54

Appendix A
COVID-19 Vaccine Option GridTM
Note: This version of the Option Grid is dated May 2022 and has had multiple updates to content since this date. The current version is
available at https://decisions.dynamed.com/tools/vaccine-options-for-covid-19
55

56

57

Appendix B
Data Analysis Methodology - Study Aims 1 and 2
Data collection summary
Below is a summary table of primary and secondary outcomes and other data collected
(including demographic characteristics, contextual factors, and intervention engagement) and
corresponding time points for analyses.
Table 1. Time points for outcome and other data assessment
Data
Category
Name of Outcome /
Data
Time Point / Survey
T0
baseline
T1
3 weeks
post-baseline
T2
3 months
post-baseline
T3
6 months
post-baseline
Participant
characteristics
Age, gender, zip code,
race/ethnicity,
education, insurance
status, health literacy,
religiosity, extent
influenced by others,
long-term care (LTC)
role, duration of
experience in LTC,
baseline vaccination
status
✓
Contextual
factors
Personal experiences
related to COVID-19,
COVID-19 vaccines,
and other vaccines
✓
✓
✓
✓
Intervention
and usual
practice
engagement*
Engagement with each
intervention and
enhanced usual
practice information
(including refreshers)
✓
✓
✓
Primary
outcome
COVID-19 vaccine
confidence
✓
Secondary
outcomes
Change from baseline
in COVID-19 vaccine
confidence
✓
✓
✓
✓
58

COVID-19 vaccine
uptake (any dose)
✓
COVID-19 vaccine
uptake (initial series
completion)
✓
COVID-19 vaccine
uptake (booster
completion)
✓
Likelihood of
recommending
(promoting) COVID-19
vaccination
✓
Likelihood of
recommending
(promoting) COVID-19
booster vaccination to
coworker
✓
COVID-19 vaccine
intent (initial series)
✓
COVID-19 vaccine
intent (booster)
✓
COVID-19 vaccine
intent (future vaccine
recommendations)
✓
Feeling informed about
the COVID-19 vaccines
✓
Identification of
COVID-19 vaccine
information and
misinformation
✓
Trust in COVID-19
vaccine information
from different sources
✓
Change from baseline
in secondary outcomes
✓
✓
✓
✓
As treated analyses of
primary and secondary
outcomes
✓
✓
✓
✓
59

*Data on intervention engagement will be collected via a combination of online activity data and
participant survey questions
Statistical principles
Confidence intervals and p-values
P-values ≥0.0001 will be reported to 4 decimal places; p-values less than 0.0001 will be
reported as “<0.0001”. The mean, standard deviation, and any other statistics other than
quantiles, will be reported to one decimal place greater than the original data. Quantiles, such
as median, minimum, or maximum will use the same number of decimal places as the original
data. Estimated parameters, not on the same scale as raw observations (e.g. regression
coefficients) will be reported to 3 significant figures.
Adherence to interventions
Analysis of both intention-to-treat (ITT) and as-treated populations will be performed. All
participants who are randomized to a trial arm will be included in the ITT analyses. Participants
who adhere to major engagement criteria for their primary intervention will be included in the
as-treated analyses.
Analysis methodology
Aim 1
To compare the impact of two interventions delivered online: 1) a dialogue-based webinar
(webinar) using the existing COVID-19 Option Grid conversation aid and, 2) an interactive and,
dynamic, and multi-component social media web application (social media website), compared
to enhanced usual practice (link to Centers for Disease Control (CDC) vaccine website), on
COVID-19 vaccine confidence (primary outcome), and other secondary outcomes among
LTCWs. We hypothesize that each intervention will be superior to enhanced usual practice at
increasing vaccine confidence (Hypothesis 1.1) and that the dialogue-based webinar
intervention will be superior to the social media arm at increasing vaccine confidence
(Hypothesis 1.2).
Corresponding analyses
Primary outcome
The primary outcome is the binary classifier of vaccine confidence derived from the three-item
Vaccine Confidence Index (VCI), evaluated at 3 weeks post-randomization, which will be
analyzed by randomization arm. The determined binary predictors of intervention outcomes will
be compared using two-proportion z-tests to identify significant differences between each
intervention and enhanced usual practice (Arm 1 vs Arm 3 & Arm 2 vs Arm 3 - superiority
analyses) and between the intervention arms (Arm 1 vs Arm 2 - two-tailed equivalency
analysis).
Secondary outcomes
●
Change from baseline in COVID-19 Vaccine Confidence. Analysis of change from
60

baseline will be based on a binary classifier whether the average score increases (1) or
does not increase (2) at T1, T2, and T3 versus baseline.
●
Likelihood of recommending COVID-19 vaccination (others not vaccinated). This
question will be evaluated using a Net Promoter Score (NPS) approach, adding one for
each positive response, zero for neutral, and subtracting one for each negative
response, then dividing by the number of non-missing responses for each trial arm. NPS
values will be compared using Wald intervals [1].
●
Likelihood of recommending COVID-19 booster vaccination (coworker). This question
will be evaluated using the NPS approach, described above.
●
COVID-19 vaccine uptake (any dose). This will be evaluated as a binary classifier
compared across trial arms using two-proportion z-tests.
●
COVID-19 vaccine uptake (initial series completion). For participants who have received
any dose of a COVID-19 vaccine, this will be evaluated as a binary classifier compared
across trial arms using two-proportion z-tests.
●
COVID-19 vaccine uptake (booster completion). For participants who have completed
their initial vaccine series, this will be evaluated as a three category variable (No, Not
sure, Yes) compared across trial arms using chi-squared tests.
●
COVID-19 vaccine intent (initial series). For participants who are unvaccinated, we will
assess their intentions of getting a COVID-19 vaccine, this will be evaluated using the
Net Promoter Score approach, where the three responses – Yes, Not Sure, No – will be
evaluated as analogous to promoter, neutral, and detractor, respectively. The question
will be evaluated using the NPS analysis, described above.
●
COVID-19 vaccine intent (booster). For participants who have completed an initial
vaccine series, this will be evaluated using the Net Promoter Score approach, where the
three responses – Yes, Not Sure, No – will be evaluated as analogous to promoter,
neutral, and detractor, respectively. The question will be evaluated using the NPS
analysis, described above.
●
COVID-19 vaccine intent (future vaccine recommendations). Participants’ intent to get
COVID-19 vaccines regularly in the future if they are recommended will be evaluated
using the Net Promoter Score approach, where the three responses – Yes, Not Sure, No
– will be evaluated as analogous to promoter, neutral, and detractor, respectively. The
question will be evaluated using the NPS analysis, described above.
●
Feeling informed about the COVID-19 vaccines. The degree to which participants’ feel
informed about the COVID-19 vaccines (have enough information and understand that
information) will be evaluated using a continuous variable, calculated as the mean scale
61

score across two items ((1) having enough information, (2) understanding the
information) for each participant. The mean of all mean scores will be compared across
trial arms with two-sample unpaired t-tests.
●
Identification of COVID-19 vaccine information and misinformation. Subjects’
identification of COVID-19 vaccine information and misinformation will be scored on a
scale of zero to four and treated as a continuous linear variable. Results will be
compared across trial arms with two sample unpaired t-tests.
●
Trust in COVID-19 information from different sources. Subjects’ trust in COVID-19
information from different sources will be evaluated using a continuous variable,
calculated as the mean scale score across three items for each participant. The mean of
all mean scores will be compared across trial arms with two-sample unpaired t-tests.
●
Change from baseline in secondary outcomes. Change from baseline will be scored as a
binary variable with an increased value scored as one and a no change or decreased
value scored as zero. For each evaluation after baseline, each arm will be compared
using two-proportion z-tests. For vaccine and booster intent questions, where the intent
question is not repeated if the participant indicates an increase in vaccination status,
intent will be imputed from uptake. To be clear, an individual who is unvaccinated at T0,
then returns at T1 and indicates that he or she is now vaccinated, will be presented with
the vaccine intent question at T0 but not at T1. Because the individual is now
vaccinated, asking about vaccine intent at T1 would be confusing. For analyzing change
in vaccine intent, the individual whose vaccine status increases would be considered to
have positive intent at T1. A similar approach will be used for booster intent.
●
As treated analysis of primary and secondary outcomes. Primary and secondary
outcome analyses will be repeated, limited to the as treated samples.
Aim 2
To determine if LTCWs’ characteristics and other factors mediate and moderate the
interventions’ impact on vaccine confidence and other secondary outcomes.
We hypothesize that increased perceptions of feeling informed about the vaccines, identification
of vaccine information and misinformation, and trust in vaccine information provided by different
sources will explain (mediate) the relationship between the interventions and vaccine
confidence, as well as other secondary outcomes (Hypothesis 2.1).
We will also conduct exploratory heterogeneity of treatment effects (HTE) analyses to identify
whether certain participant characteristics and beliefs moderate the relationships among each of
the interventions and vaccine confidence, as well as other secondary outcomes. Variables to be
explored will include, but are not limited to, baseline vaccination status, religious beliefs,
perceived influence of others, age, race, ethnicity and personal experiences with COVID-19.
62

Corresponding analyses
Mediation analysis
We hypothesize that the relationship between participants’ assigned trial arm and primary and
secondary outcomes will be mediated by the effects of feeling informed, trust in vaccine
information, and beliefs about vaccines (measured through questions to determine each
participant’s ability to correctly classify vaccine information and misinformation). Responses for
these three variables will be evaluated at each time point versus outcomes by randomization
arm to identify differences by arm using the Kruskal-Wallis test. Significant relationships will be
explored for mediation effects in the primary and secondary outcome analyses using the method
of Barron and Kenny [2]. If this results in a significant improvement in model prediction,
hypothesis 2.1 will be supported.
Heterogeneity of treatment effects/Moderation analysis
To understand how the relationship between study interventions and outcomes are moderated
by participant characteristics including age, gender, location (level of regional analysis selected
for appropriate granularity), educational attainment, race and ethnicity, health insurance status,
health literacy, religiosity, LTCW role, duration of experience in long-term care, extent influenced
by others regarding COVID-19 vaccination, personal experiences with COVID-19 and the
vaccines, and baseline vaccination status, we will evaluate the primary and secondary
outcomes using linear and logistic regression in a mixed-effects model.
Exploratory analyses
Change from baseline in COVID-19 Vaccine Confidence. Looking deeper at the change in
vaccine confidence, analyses will consider (a) the proportion of positive (agree and strongly
agree) responses in the VCI, (b) positive responses in each of the three VCI questions.
Significant correlates with demographic, contextual, and study variables will be evaluated.
External contextual factors. Outside of the study surveys and throughout the trial, we will
monitor external factors that may impact participants’ views and actions towards the COVID-19
vaccines. This may include monitoring policy and mandate changes for LTCWs and changes in
the nature of the pandemic, among other things.
Additional analyses. Any unvalidated thresholds utilized in the primary or secondary analyses
will be evaluated for sensitivity to the choice of threshold in ROC analyses. Nonlinear
higher-dimensional effects and interactions among variables will be identified by identifying
positive or negative outcomes grouped around specific baseline characteristics that are highly
unlikely to result from random chance.
Duplicate and missing data
Participants must complete a baseline (T0) survey to be considered enrolled in the study. In the
event of duplicate complete T0 submissions, the first complete survey will be used. For
participants who complete duplicate follow-up surveys (T1, T2 or T3), the first complete survey
63

will also be used. In order to be included in primary and secondary outcome analyses,
participants must have valid primary outcome data at both T0 and T1.
The study team will endeavor to minimize missing data; however, it is possible that efforts to
ensure complete data collection within the response windows will be unsuccessful or because
participants may choose not to respond to one or more questions. Should any primary and/or
secondary outcome variables have >5% missing data, the reasons for any missing data will be
considered through an analysis for common baseline characteristics of individuals with specific
responses missing, those who were lost to follow-up, and those who did not respond to one or
more evaluations but were not lost to follow-up. Specific questions with significantly higher rates
of missing data will be reviewed for wording bias or missing response options. Based on these
considerations, a stochastic outcome set model will be created using multiple imputation and
Bayesian methods, as appropriate, to provide a high confidence range for imputed values.
Using this approach, an expected result and confidence intervals will be produced for the
missing data and the resulting analysis of the full data set, taking into account the statistical
uncertainty attributed to the missing data. If available, data from both baseline and subsequent
follow-up evaluations will be used in imputing missing results (e.g., responses from T0 and T2
evaluations could be used to impute T1 responses) through multiple imputation regression
modeling [‎3].
Response rates and the frequency of missing data in the ITT and as-treated populations will be
reported with the results.
Fraudulent enrollment
Identity verification of some participants will occur after they have completed their journey in the
study (using TransUnion’s TLOxp verification service). We will also conduct a rigorous review of
survey metadata (e.g., IP addresses) and other responses to identify duplicate survey
completions or other suspicious survey activity [4,5]. Upon discovery of compelling evidence of
fraudulent or illegitimate enrollment, such subjects will be excluded from the study prior to any
outcome analyses.
Methods to minimize bias
A naïve analysis will be conducted at the end of the study based only on the unmodified data
reflecting ITT assignments of eligible subjects to establish an initial estimator. All primary and
secondary analyses will be conducted based on the statistical analysis plan. The statistician
member of the DSMB will also independently replicate primary study analyses to confirm their
validity.
Multiple testing
The primary outcome involves three separate tests, one a two-tailed equivalence test and two
one-tailed superiority tests. In order to achieve a family-wise error rate less than 0.050, the
Bonferroni method was employed to set a p-value threshold at 0.016 for each of the three
primary outcome analyses. For each comparison, mean-value separation will be considered
significant based on a 98.4% confidence interval, which translates to +/- 2.45 times the standard
64

error for the sample mean for the equivalence test and greater than 2.15 times the standard
error for the superiority analyses.
Based on the high number of secondary outcome analyses (cross-sectional T1; change from
baseline; ITT; as-treated), results data for these outcomes will be viewed in more aggregate
form as trending in one direction or not, rather than adopting a specific Bonferroni-corrected
p-value threshold.
References
1.
B. Rocks, Interval Estimation for the “Net Promoter Score.” Am. Stat. 70, 365–372
(2016).
2.
R. M. Baron, D. A. Kenny, The moderator–mediator variable distinction in social
psychological research: Conceptual, strategic, and statistical considerations. J. Pers.
Soc. Psychol. 51, 1173–1182 (1986).
3.
T.E. Raghunathan, J.M. Lepkowski, J. Van Hoewyk, P. Solenberger, et al. A multivariate
technique for multiply imputing missing values using a sequence of regression models.
Surv. Methodol. 27, 85-96 (2001).
4.
Stevens G, Washburn H, Theiler R, Woodhams E, Donnelly K, Thompson R. Enrolment
fraud in online shared decision-making research: Lessons learned from an
internet-based randomised controlled trial. Oral presentation at the 10th International
Shared Decision-Making Conference (ISDM), Quebec City, Canada.
https://fourwaves-sots.s3.amazonaws.com/static/media/uploads/2019/06/28/isdm2019-o
ralsessionsbooklet-2019-06-28.pdf. Accessed 17 Dec 2022.
5.
Teitcher JEF, Bockting WO, Bauermeister JA, Hoefer CJ, Miner MH, Klitzman RL.
Detecting, Preventing, and Responding to “Fraudsters” in Internet Research: Ethics and
Tradeoffs. J Law Med Ethics. 2015;43:116–33.
65