Addressing COVID-19 Vaccine Hesitancy in Rural Community Pharmacies Reducing Disparities Through an Implementation Science Approach
Phase: NA
Conditions: COVID-19
Sponsor: University of North Carolina, Chapel Hill

Addressing COVID-19 Vaccine Hesitancy in Rural Community 
Pharmacies Reducing Disparities Through an Implementation Science 
Approach
NCT number NCT05926544 
Document Date 03/19/2025 

PROTOCOL 
Complete Title: Addressing COVID-19 Vaccine Hesitancy in Rural Community Pharmacies Reducing Disparities 
Through an Implementation Science Approach 
Short Title: COVID-19 Vaccine Hesitancy Counseling Intervention for Pharmacists 
Sponsor: National Institutes of Health (NIH) 
Protocol Date: 3/19/25 
Sponsor 
National Institutes of Health (NIH) 
Bethesda, MD 20892 
USA 
Study Principal Investigator  
Delesha Carpenter, PhD, MSPH 
1 University Heights, 114G Karpen Hall 
Asheville, NC 28804 
Phone: 828-250-3916 
Email: dmcarpenter@unc.edu 

Page 2 of 24 
PROTOCOL TITLE: 
Addressing COVID-19 Vaccine Hesitancy in Rural Community Pharmacies Reducing 
Disparities Through an Implementation Science Approach  
Short Title: 
COVID-19 Vaccine Hesitancy Counseling Intervention for Pharmacists 
Lead Investigator: 
Delesha Carpenter, PhD, MSPH 
University of North Carolina at Chapel Hill 
Version Date: March 19, 2025 
I confirm that I have read this protocol and understand it. 
Principal Investigator Name: 
Delesha Carpenter 
Principal Investigator Signature: 
Date: 
3/19/25 

Page 3 of 24 
Table of Contents 
ABBREVIATIONS AND DEFINITIONS OF TERMS .......................................................................................................... 4 
PROTOCOL SYNOPSIS .......................................................................................................................................................................................... 5 
BACKGROUND AND RATIONALE ......................................................................................................................................................................... 8 
1 
STUDY OBJECTIVES ............................................................................................................................................................................ 9 
2 
INVESTIGATIONAL PLAN ................................................................................................................................................................... 9 
3 
STUDY PROCEDURES ....................................................................................................................................................................... 11 
4 
STUDY EVALUATIONS AND MEASUREMENTS ................................................................................................................................ 13 
5  
    STATISTICAL CONSIDERATIONS ...................................................................................................................................................... 14 
6 
STUDY INTERVENTION .................................................................................................................................................................... 16 
7  
     SAFETY MANAGEMENT ................................................................................................................................................................... 17 
8  
  DATA COLLECTION AND MANAGEMENT ........................................................................................................................................ 18 
9  
   CONSENT PROCESS .......................................................................................................................................................................... 19 
10         REFERENCES ..................................................................................................................................................................................... 19 
APPENDIX .................................................................................................................................................................... 23 

Page 4 of 24 
Abbreviations and Definitions of Terms 
Abbreviation 
Definition 
SARS-CoV-2 
Severe Acute Respiratory Syndrome Coronavirus 2 
HPV 
Human papillomavirus 
CFIR 
Consolidated Framework for Implementation Research 
RURAL-CP 
Rural Research Alliance of Community Pharmacies 
VA 
U.S. Department of Veterans Affairs 
QUERI 
Quality Enhancement Research Initiative 
LMM 
Linear Mixed-effects Model 
GLMM 
Generalized Linear Mixed Model 
ICC 
Intraclass Correlation Coefficients 
SP 
Standardized Patient 

Page 5 of 24 
 
PROTOCOL SYNOPSIS 
  
Study Title 
Addressing COVID-19 Vaccine Hesitancy in Rural Community Pharmacies 
Reducing Disparities Through an Implementation Science Approach 
Funder 
National Institutes of Health (NIH)  
Study Rationale 
• 
The COVID-19 pandemic has disproportionally impacted rural 
communities. 
• 
When compared to urban populations, individuals living in rural areas 
are more vaccine hesitant, have a higher prevalence of comorbid 
health conditions, and are at greater risk from SARS-CoV-2 variants. 
Thus, interventions to increase vaccine uptake in rural areas are 
greatly needed. 
• 
Community pharmacists are well-positioned to address vaccine 
hesitancy with underserved, rural populations. 
• 
Because vaccination conversations are sensitive and often politically 
charged, pharmacists need implementation support, including 
training and ongoing guidance to deliver evidence-based vaccine 
hesitancy counseling interventions 
• 
Implementation facilitation, in which trained facilitators coach and 
troubleshoot problems with professionals as they implement new 
practices, increases adoption of practices with fidelity. 
• 
Facilitation generally, and virtual facilitation (e.g., video coaching) in 
particular, has not been systematically studied in community 
pharmacy settings. 
Study Objective(s) 
Primary  
• 
To evaluate whether virtual facilitation improves fidelity to the 
vaccine hesitancy counseling intervention when compared to 
standard implementation. 
Secondary 
• 
To evaluate whether virtual facilitation increases intervention 
effectiveness, defined as a higher number of vaccine hesitant 
individuals who receive the vaccine, when compared to standard 
implementation. 
Test Article 
We have refined an evidence-based vaccine hesitancy counseling 
intervention with extensive feedback from rural pharmacists, resulting in 
a 5-step counseling process (ASORT): 
• 
Ask if they would like to receive a COVID vaccination  
• 
Solicit their main vaccine concern 
• 
Offer to address their concerns 
• 
Recommend the vaccine 
• 
Try again later if they refuse or are unsure 
Study Design 
 
This is an implementation science study. We will use a randomized 
clinical trial with an adapted stepped-wedge design and 30 rural 

Page 6 of 24 
 
pharmacies. Mixed methods will provide triangulation, expansion, and 
explanation of quantitative findings. 
Subject Population 
key criteria for Inclusion 
and Exclusion: 
Data will be collected from rural pharmacies located in 7 southeastern 
states that participate in the Rural Research Alliance of Community 
Pharmacies (RURAL-CP). 
Inclusion Criteria 
1. be a member of RURAL-CP 
2. be located in a county that has an African American population of at 
least 25% or had at least 51% of the population vote for a Republican 
president in 2020. 
3. stock the Covid-19 vaccine for the duration of the study 
Each pharmacy will contribute at least 1 pharmacist and up to 4 
additional pharmacy staff members for the study. Pharmacy staff will be 
eligible if: 
• 
they are at least 18 years of age 
• 
they can read and speak English 
• 
they have been employed by the pharmacy for at least 1 month 
Exclusion Criteria 
• 
There are no exclusion criteria for pharmacy staff 
Number Of Subjects  
30 pharmacies and up to 150 subjects 
Study Duration 
Pharmacies will participate in the study for 6-12 months. 
Total anticipated study duration will be 2 years. 
Study Phases 
  
1. Screening: identifying pharmacies in the RURAL-CP network that 
meet the eligibility criteria 
2. Enrollment by invitation: inviting select pharmacies via email, 
confirming eligibility of those interested, obtaining verbal informed 
consent 
3. Standard Implementation Period: participants complete one or two 
8-week period(s), which involves: completing a 30-minute online 
training; watching a webinar; implementing the vaccine hesitancy 
counseling intervention with 10 vaccine hesitant patients per 8-week 
period; meeting with an SP once per month to have their fidelity to 
the intervention evaluated 
4. Virtual Facilitation Period: participants complete one or two 8-week 
period(s), which involves: a virtual site visit from a trained virtual 
facilitator; at least weekly calls with the facilitator; implementing the 
vaccine hesitancy counseling intervention with 10 vaccine hesitant 
patients per 8-week period; meeting with an SP once per month to 
have their fidelity to the intervention evaluated 
5. Follow-up: select pharmacies will have one 8-week follow-up period, 
during which they will continue to implement the vaccine hesitancy 
counseling intervention with 10 vaccine hesitant patients 
6. Qualitative Data Collection: select participants (12 total) will 
complete a 60-minute interview to assess feasibility and performance 
of the implementation approaches 

Page 7 of 24 
 
Efficacy Evaluations 
Primary outcome: trained study staff will use a fidelity observation guide 
to rate pharmacists’ vaccine hesitancy counseling during recorded 
counseling sessions. The fidelity measure focuses on the competence of 
the pharmacist (7 items) in their delivery of the vaccine hesitancy 
counseling intervention. Each competence item will be assessed on a 
scale from 0 to 2, with 0 = skill not demonstrated, 1 = skill needs 
development, and 2 = skill demonstrated with competence. 
Secondary outcome: using an online survey, pharmacists will document 
the following on a daily basis: 
A) how many vaccine-hesitant individuals they provided vaccine 
hesitancy counseling to; 
B) of those individuals, how many received a COVID-19 vaccine; 
C) the self-reported age, race, and gender of the individual who 
was counseled. 
Effectiveness will be calculated as the proportion of counseled individuals 
who received a vaccine, or B/A. Scores will range from 0 to 1. 
Statistical And Analytic 
Plan 
For the primary analysis of the fidelity outcome, linear mixed-effects 
models (LMM) will be used to compare the level of fidelity between the 
two implementation approaches (standard vs. virtual facilitation). We will 
report point estimates for the group mean difference along with a 95% 
confidence interval. 
Generalized linear mixed models (GLMM) will be used for analysis of the 
secondary outcome of effectiveness. 
DATA AND SAFETY 
MONITORING PLAN 
The study PIs will review interview and survey data as well as fidelity data 
on a quarterly basis for safety issues. In addition, the entire study team 
will meet monthly to discuss the study's progression and any potential 
safety or data concerns. 
 
 
 

Page 8 of 24 
 
BACKGROUND AND RATIONALE 
 
Major health crises like the COVID-19 pandemic have disproportionally impacted rural communities, which are 
frequently health professional shortage areas that lack health care infrastructure, including hospitals1-3. As one 
of the most accessible health professionals in rural areas4,5, community pharmacists are a notable exception to 
this lack of infrastructure. There are =68,000 community pharmacies in the U.S, many of which have 
convenient hours and offer free walk-in services6.  
When compared to urban populations, individuals living in rural areas are more vaccine hesitant 8,9 and have a 
higher prevalence of health conditions that increase their risk for severe COVID-19 illness and death12. Higher 
vaccine hesitancy is strongly correlated with lower vaccination rates13, with rural counties driving lower state-
level vaccination rates in the South14,15, which has some of the lowest vaccination rates in the country. Unless 
vaccine hesitancy is addressed, low vaccination rates and a lack of vaccine mandates mean these communities 
remain susceptible to SARS-CoV-2 variants. Pharmacists, one of the most trusted sources of medication 
information 16, can serve as strong allies to address vaccine hesitancy. Rural patients see their community 
pharmacist ~14 times per year; nearly three times more than they see a primary care provider17. Thus, 
pharmacists can address vaccine concerns on a monthly basis and make repeated vaccination offers. 
To engage in sensitive vaccine conversations that pharmacists have referred to as “charged”, they need 
updated information to address patients’ evolving vaccine concerns and implementation support, including 
training and ongoing guidance, to deliver evidence-based vaccine hesitancy counseling interventions18. 
A standard implementation approach, which typically involves training and dissemination of implementation 
support tools (e.g., sample workflows, counseling “cheat sheets”), is commonly used to help health 
professionals implement new practices, including HPV vaccination19. Although necessary, this standard 
approach is usually not sufficient to promote adoption of a new and complex practice with fidelity 20-22. In 
recognition of this limitation, a growing body of research shows that implementation facilitation can increase 
implementation fidelity23-26 by having trained facilitators build trusting relationships with health professionals 
to monitor their implementation progress, provide feedback, and reinforce change. To date, implementation 
facilitation has not been systematically studied in community pharmacy settings. Moreover, little is known 
about the effectiveness of a virtual facilitation approach, whereby facilitators connect with health 
professionals exclusively via telephone and video. The need to study virtual facilitation is highly relevant given 
that travel to remote rural locations is resource-intensive and has been restricted due to pandemic-related 
safety concerns. 
  
  
 
 

Page 9 of 24 
 
1 
STUDY OBJECTIVES 
The overall objective is to test the effects of a standard implementation approach and the addition of 
virtual facilitation on rural pharmacists’ ability to implement COVID-19 vaccine hesitancy counseling, 
using an incomplete stepped wedge design27. Data will be collected on the primary trial outcome of 
counseling fidelity (competence) and the secondary outcome of intervention effectiveness (vaccination 
rates). Pharmacies will implement the ASORT intervention, which was adapted from an evidence-based 
vaccine hesitancy intervention28 with extensive qualitative input from rural pharmacists29. The 
intervention will be updated frequently to address new vaccine concerns as they arise. 
We have two specific aims. 
1.1 
Aim 1 
Compared to the standard implementation approach, test whether adding virtual facilitation increases 
(a) the fidelity with which pharmacists implement the vaccine hesitancy counseling intervention and 
(b) the number of vaccine hesitant patients who receive the vaccine. Our primary outcome is 
pharmacist fidelity to the vaccine hesitancy counseling intervention and our secondary outcome is 
intervention effectiveness (i.e., the number of vaccine hesitant individuals who receive a vaccine). We 
hypothesize that: 
H1: Virtual facilitation will improve fidelity to the vaccine hesitancy counseling intervention when 
compared to standard implementation. 
H2: Virtual facilitation will increase intervention effectiveness, defined as a higher number of vaccine 
hesitant individuals who receive the vaccine, when compared to standard implementation. 
 
1.2 
Aim 2 
Conduct a cost assessment and explore potential sustainability of the implementation approaches. We 
will conduct a time-driven, activity-based cost analysis and a budget analysis of the standard 
implementation and virtual facilitation approaches. Additionally, a payer advisory board will review 
these data and advise on how to make virtual facilitation sustainable through reimbursement models. 
We do not have a hypothesis associated with this aim. 
 
2 
INVESTIGATIONAL PLAN 
2.1 
Study Design 
This is an implementation science study. We will use a randomized clinical trial with an adapted 
stepped-wedge design and 30 rural pharmacies to determine the effectiveness and incremental cost-
effectiveness of a standard implementation approach compared to the addition of virtual facilitation 
by a trained facilitator in support of the delivery of a COVID-19 vaccine hesitancy counseling 
intervention. Mixed methods will provide triangulation, expansion, and explanation of quantitative 
findings. 
In the incomplete stepped wedge cluster randomized design, each pharmacy will begin in the standard 
implementation approach condition for one or two 8-week periods and then “crossover” to the virtual 
facilitation condition for one or two 8-week periods. Length of time in each condition depends on the 
"step" to which pharmacies are randomly assigned. Figure 1 depicts the “stairstep” study design and 

Page 10 of 24 
 
pharmacy assignment. For blocks 1–4, we will continue to collect data during one 8-week follow-up 
period to evaluate the potential impact of the virtual facilitation approach on sustained intervention 
fidelity and effectiveness once virtual facilitation has been stopped. 
2.2 
Allocation to Treatment Groups  
In this stepped-wedge trial, pharmacies will be randomized in blocks at the time they begin the initial 
intervention condition (standard implementation). We will create 6 blocks of pharmacies, with 5 
pharmacies randomized to each block by the trial statistician. To ensure balance, the block 
randomization will be stratified by two measures of pharmacy size—the number of patients and 
number of pharmacists. 
 
2.3 
Number of Subjects and Study Duration 
The stepped-wedge trial includes 30 pharmacies and up to 150 subjects. Subjects will include 30 
pharmacists (1 from each pharmacy) and up to 120 pharmacy staff (up to 4 staff from each pharmacy), 
such as pharmacy technicians and cashiers. Pharmacies will participate in the study for 6-12 months, 
depending on the block to which they are randomly assigned. Total anticipated study duration will be 2 
years, with 15 pharmacies participating in year 1 and 15 participating in year 2. The study follows the 
distribution of COVID-19 vaccines in the USA, which follows the Northern Hemisphere’s “flu season”—
approximately August-February each year. We expect that the first COVID-19 vaccination season in the 
fall of 2023 will be delayed due to vaccine availability and will begin in October and run through March 
2024. In 2024, we expect that availability issues will not occur and the vaccine will be available 
during the standard flu season from August-February. All pharmacies begin the study in the fall. Block 1 
and blocks 4-6 will participate for 6 months (Oct-Mar and Aug-Jan, respectively). Blocks 2 and 3 begin 
in Oct 2023 and participate for 6 months alongside Block 1, but then have an 8-week follow-up period 
Aug-Sep 2024 (Figure 1). 
 
2.4 
Study Population 
All data will be collected from rural pharmacies that participate in the Rural Research Alliance of 
Community Pharmacies (RURAL-CP), which was established by this trial’s multiple principal 
investigators Carpenter and Curran in 2020. RURAL-CP is the first practice-based research network for 
rural community pharmacies and aims to reduce rural health disparities by supporting high-quality 
implementation research with community pharmacies. Thirty pharmacies will be recruited from a total 
of 127 RURAL-CP pharmacies located in seven states throughout the Southeast. For a pharmacy to be 
eligible for the trial, it must: be a member of RURAL-CP; be located in a county that has an African 
American population of at least 25% or had at least 51% of the population vote for a Republican 
president in 2020. The reason for this criterion is that these populations are more vaccine hesitant; and 
stock the Covid-19 vaccine for the duration of the study. Each pharmacy will contribute at least 1 
pharmacist and up to 4 additional pharmacy staff members for the study, including pharmacy 
technicians and cashiers. Pharmacy staff will be eligible if: they are at least 18 years of age; they can 
read and speak English; and they have been employed by the pharmacy for at least 1 month. There are 
no exclusion criteria for pharmacy staff. 
 
 

Page 11 of 24 
 
3 
STUDY PROCEDURES  
3.1 
Recruitment 
Study staff have identified pharmacies in the RURAL-CP network that meet the eligibility criteria. The 
program manager for RURAL-CP will inform all eligible member pharmacies about the proposed study 
via email and ask interested pharmacies to respond stating their interest. Subsequent discussions will 
explain the purpose of the study and study procedures. Pharmacies interested in participating will 
identify potential pharmacy staff respondents and provide their contact information to the PIs. These 
potential respondents will be contacted by study staff to ask them if they would like to participate in 
the study. A consent script will be created by the research team for recruitment that will include a 
description of the study and what will be expected of participants. Verbal consent will be obtained 
from all subjects. 
 
3.2 
Standard Implementation Period 
After pharmacy participants are enrolled, they will then complete one or two 8-week "standard 
implementation" period(s), depending on the step to which they are randomly assigned. At the 
beginning of this period, they will be asked to complete a 30-minute online training about COVID-19 
vaccine hesitancy counseling and watch a live or pre-recorded webinar (their choice) that provides up-
to-date information about COVID-19 vaccinations and boosters. After completion of the training and 
webinar, participants will be asked to implement the vaccine hesitancy counseling intervention with 10 
vaccine hesitant patients per 8-week period. Pharmacists will use Qualtrics or a paper survey to 
document the extent to which the intervention results in customers getting vaccinated. They will 
specifically document the number of customers they offered the intervention to, the number who 
refused, and the number who agreed to be vaccinated. No identifiable customer-level data will be 
provided by the pharmacies. 
 
Additionally, during the standard implementation period(s), trained study staff will observe and rate 
counseling fidelity two times per month for each participating pharmacy. We will use a fidelity rating 
scale to assess fidelity of delivery of the intervention during 4-8 total observed intervention sessions 
per pharmacy (4 per 8-week period). Since pharmacists have reported difficulties getting patients to 
agree to be recorded, affecting data collection, we will use standardized patients (SPs) in lieu of real 
patient conversations. Two SPs will have a total of 8 standardized scripts, each expressing different 
concerns about the Covid vaccine. The project manager will schedule a 20-minute block with an SP and 
participating pharmacist and send a Zoom link. Pharmacists will be randomized to script and order of 
the presentation of two SP scenarios. On the Zoom call, SPs will enact two of their cases, to which the 
pharmacist will ideally respond with the ASORT intervention. The encounter will be recorded using the 
secure Express Dictate mobile or desktop app and later rated for counseling fidelity. SPs will meet with 
each pharmacist one time per month, portraying different personas each time. 
 
To increase the likelihood that each participating pharmacy submits the requested study data, a fax will 
be sent to each participating pharmacy that is missing data after the first month of standard 
implementation to remind them of what is needed. This fax may be sent every 2 weeks as needed to 
acquire missing study data, with a follow-up phone call a week after the fax to non-responders. 
 

Page 12 of 24 
 
At the end of the standard implementation period(s), participating pharmacy staff (up to 5 per 
pharmacy) will complete Qualtrics surveys to assess implementation outcomes (feasibility, 
acceptability, appropriateness). 
 
3.3 
Virtual Facilitation Period 
After the implementation outcome surveys have been completed, pharmacies will "crossover" to the 
virtual facilitation period(s). This period will begin with a virtual site visit from a trained virtual 
facilitator and be followed by at least weekly calls with a local champion (a participating pharmacist) 
and at least bi-weekly calls with each participating pharmacist to provide feedback on intervention 
fidelity (from observations of interventions delivered). The virtual site visit over Zoom will establish the 
personnel and workflows at each pharmacy and allow the facilitator to establish rapport. Weekly Zoom 
calls will allow the virtual facilitator to work with a site champion to review overall implementation 
challenges associated with approaching patients, delivering the intervention, and documenting results. 
Lastly, either the facilitator or the local champion can request and schedule a Zoom call to go over any 
pressing implementation issue in need of rapid attention (e.g., technical difficulties with the website).  
 
During each 8-week virtual facilitation period (one or two periods, depending on the step to which 
pharmacies are randomly assigned), pharmacists will again be asked to implement the intervention 
with 10 vaccine hesitant patients. Like during the standard implementation period, pharmacists will 
use Qualtrics or a paper survey to document the extent to which the intervention results in customers 
getting vaccinated. Similarly, fidelity observations (4-8 total per pharmacy) will be collected using the 
same processes described in Section 3.2. Additionally, at the end of virtual facilitation period(s) 
participating pharmacy staff (up to 5 per pharmacy) will again be asked to complete Qualtrics surveys 
to assess implementation outcomes (feasibility, acceptability, appropriateness). 
 
3.4 
Follow-up 
Pharmacies randomized to blocks 1-4 will have one 8-week follow-up period. During this period, 
pharmacists will continue to use Qualtrics or a paper survey to document the extent to which the 
intervention results in customers getting vaccinated. They will specifically document the number of 
customers they offered the intervention to, the number who refused, and the number who agreed to 
be vaccinated. 
 
3.5 
Qualitative Data Collection 
After pharmacies complete both the standard and virtual facilitation periods, study staff will conduct 
qualitative interviews with the primary study participant from one high performing and one low 
performing pharmacy in each block (12 pharmacies total) to assess feasibility and performance of the 
implementation approaches. It will also assess: 1) associations observed for organizational structure 
and culture/climate measures associated with fidelity and effectiveness, 2) sustainability potential, and 
3) costs at the pharmacy level. Each interview is expected to last 60 minutes. 
 
 
 
 

Page 13 of 24 
 
4 
STUDY EVALUATIONS AND MEASUREMENTS  
4.1 
Primary Outcome  
Our primary outcome is fidelity to the vaccine hesitancy counseling intervention, assessed at the 
pharmacy level. Our fidelity measure is based on a theoretical framework of fidelity measurement30 as 
well as a validated fidelity checklist31. The fidelity measure focuses on the competence of the 
pharmacist (7 items) in their delivery of the vaccine hesitancy counseling intervention. 
 
The competence items focus on the skillfulness of intervention delivery: expressed empathy; used a 
non-confrontational manner; spoke confidently without using jargon; emphasized patient autonomy; 
reflected back patient’s statements accurately; used a respectful demeanor, and used evidence-based 
responses when responding to patient vaccine concerns. Each competence item will be assessed on a 
scale from 0 to 2, with 0 = skill not demonstrated, 1 = skill needs development, and 2 = skill 
demonstrated with competence. Competence scale scores will range from 0 to 14, with higher scores 
reflecting greater competency in the delivery of ASORT. Fidelity will be measured for each pharmacist 
approximately twice per month under the standard implementation approach and approximately twice 
per month under the virtual facilitation approach. In pharmacies with more than one pharmacist, 
fidelity ratings will be averaged to achieve a pharmacy-level measure. 
 
Trained staff will rate fidelity after reaching 80% inter-rater reliability during training32. Staff who are 
blinded to the pharmacist’s group assignment will use a fidelity observation guide to rate pharmacists’ 
vaccine hesitancy counseling during recorded counseling sessions. During virtual facilitation, ratings 
will be shared with the facilitators who provide feedback and coaching to pharmacists towards 
improving fidelity. 
 
4.2 
Secondary Outcome 
The secondary outcome of effectiveness will be assessed on a monthly basis. Pharmacists will be 
instructed to deliver vaccine hesitancy counseling to one to two vaccine-hesitant individuals each week 
(towards a target of at least 5 per month). Using Qualtrics or a paper survey, pharmacists will 
document the following on a daily basis: A) how many vaccine-hesitant individuals they provided 
vaccine hesitancy counseling to; B) of those individuals, how many received a COVID-19 vaccine; C) the 
self-reported age, race, and gender of the individual who was counseled. 
 
Effectiveness will be calculated as the proportion of counseled individuals who received a vaccine, or 
B/A. Individuals who are counseled and schedule a vaccine at a later date will not be counted in the 
numerator (B) until they have actually received a vaccine. Effectiveness scores will range from 0 to 1, 
with higher scores reflecting a greater percentage of vaccine-hesitant individuals who received a 
vaccine. During virtual facilitation, trial facilitators will be provided with bi-weekly effectiveness data so 
they can share the results with the pharmacies to assess performance and identify ways to improve 
implementation processes and intervention effectiveness. 
 
4.3 
Other Measures 
Acceptability, appropriateness, and feasibility of the vaccine hesitancy intervention and 
implementation approaches (standard, facilitation) will be measured using validated surveys33. Each 

Page 14 of 24 
 
measure includes 4 items (e.g., “intervention X seems doable” [feasibility]) measured on a 5-point 
response scale that ranges from “strongly disagree” (coded as 1) to “strongly agree” (coded as 5). For 
each pharmacy, we will average scores across the pharmacy staff members who complete the surveys. 
 
Uptake will be calculated as the number of times vaccine hesitancy counseling was offered divided by 
the number of individuals who expressed vaccine hesitancy when offered the vaccine. 
 
Sustainment will be assessed by the continued measurement of fidelity, effectiveness, and uptake 
during the “follow-up” periods. 
 
Organizational structure and context measures34: one pharmacist per pharmacy will complete the 
Organizational Structure survey35,36, which measures: location, type (e.g., independent, chain), setting 
(e.g., retail, specialty), size (weekly prescriptions, staffing), technological capacity (dispensing system), 
and services provided. The Organizational Context measure (completed by 5 pharmacy staff members) 
assesses CFIR inner setting constructs34 that reflect organizational culture/capacity for change37, 
learning climate, leadership, and resources38,39. All items are measured on a 5-point agree-disagree 
response scale, with higher scores reflecting a stronger implementation context. For each pharmacy, 
we will average scores across the five pharmacy staff members who complete the surveys. 
 
5 
STATISTICAL CONSIDERATIONS 
 
We will use the principles of intention-to-treat, using multiple imputations if needed for missingness, 
for all statistical analyses related to primary and secondary endpoints. Descriptive statistics will be 
computed for all quantitative implementation outcomes in Table 1. 
 
5.1 
Primary Outcome 
The primary outcome of fidelity is captured at the pharmacy level. For our primary analysis of the 
fidelity outcome, we will use linear mixed-effects models (LMM) to compare the level of fidelity 
between the two implementation approaches (standard approach vs. virtual facilitation). We will 
report point estimates for the group mean difference along with a 95% confidence interval. The model-
building approach will follow four analyses steps: 1) an unadjusted before/after of the effect of the 
virtual facilitation approach (ignoring period/time effect); 2) the time period (i.e., steps/blocks) to 
examine if any potential intervention effect relates only to the intervention or also to an independent 
effect of calendar time; 3) an adjustment for potential pharmacy-level confounders, such as size and 
learning climate; and 4) the interaction between period and intervention effect. 
 
5.2 
Secondary Outcome 
For the secondary outcome of effectiveness, we will use generalized linear mixed models (GLMM) to 
investigate whether pharmacies are more effective at addressing vaccine hesitancy during virtual 
facilitation when compared to standard implementation. The effectiveness outcome will be binary 
(vaccine-hesitant patient accepts vaccine after counseling = 1; vaccine-hesitant patient does not accept 
vaccine after counseling = 0). We will report the odds ratio estimate of accepting the vaccine after 
counseling for the virtual facilitation approach (facilitation versus standard) along with a 95% 

Page 15 of 24 
 
confidence interval. The model-building approach for our secondary effectiveness outcome will follow 
four analysis steps: 1) an unadjusted before/after of the effect of the virtual facilitation approach 
(ignoring period/time effect); 2) the inclusion of time period (i.e., steps) to examine if any potential 
intervention effect relates only to the intervention or also to an independent effect of calendar time; 3) 
an adjustment for patient’s age, race, and gender and potential pharmacy-level confounders, such as 
pharmacy size; 4) the possible interaction between time period and intervention. The impact of virtual 
facilitation on effectiveness could potentially change over time if vaccine acceptance rates increase 
with time and as pharmacists gain experience with counseling vaccine-hesitant patients. We aim to 
explore this question through the inclusion of an interaction between period/time and intervention 
effect in Model 4. Additionally, the intraclass correlation coefficients (ICC) will be estimated and 
reported, so this information will be available for power analyses for future investigations using similar 
designs and outcomes. In order to reduce potential bias, during the virtual facilitation period, 
pharmacists will be instructed to approach patients that they did not approach during the standard 
facilitation period, which is feasible given the large number of unvaccinated individuals in the 
communities in which RURAL-CP pharmacies are located. 
 
5.3 
 Power Analyses 
For the primary outcome of fidelity, captured at the pharmacy level, we wish to compare the fidelity 
scores under standard implementation versus fidelity scores under virtual facilitation. We expect each 
pharmacy to have approximately 4 fidelity assessments per time period (i.e., a period of 8 weeks). A 
sample of 30 pharmacies in an incomplete stepped-wedge cluster-randomized design with six periods 
(five steps), and an average of 10 fidelity assessments per pharmacy yields a total sample size of 320 
assessments, which achieves over 90% power to detect a difference between means of 0.53 with a 
standard deviation of 1 (i.e., moderate effect size). The test statistic is based on a two-sided Wald Z-
test with ICC = 0.6 and alpha = 0.05. Given that we will have repeated fidelity measures from the same 
pharmacists over time, we have specified a conservative ICC (which in a stepped wedge design does 
not impact power calculations significantly). 
 
For the secondary outcome of effectiveness, we expect each pharmacy to identify approximately 10 
vaccine-hesitant patients during each time period (i.e., a period of 8 weeks). A sample of 30 
pharmacies in an incomplete stepped-wedge cluster-randomized design with six periods (five steps) 
and an average of 27 patients per pharmacy yields a total sample size of 800 patients, which achieves 
over 90% power to detect a difference between effectiveness proportions of 10%. The proportion of 
vaccine-hesitant patients accepting the vaccine during the virtual facilitation approach is assumed to 
be 15% compared to 5% under the standard implementation period. These estimates are based on a 
review of reported changes in vaccine acceptance for evidence-based vaccine hesitancy 
interventions40. The test statistic is based on a two-sided Wald Z-test with ICC = 0.05 and alpha = 0.05. 
We note that the actual sample size should exceed 800 patients since we expect pharmacies will 
continue to identify vaccine-hesitant individuals during the follow-up periods. 
 
5.4 
 Qualitative Data Analysis 
We will use rigorous procedures41 for the analysis of qualitative interview data. Interviews will be 
transcribed by a professional transcriptionist, who will remove identifying information, and be 

Page 16 of 24 
 
imported into MAXQDA, a qualitative data analysis program. The trial team will review several 
transcripts and meet to discuss the overarching themes related to the CFIR framework (e.g., “not 
enough time” coded as an “inner context barrier”). These themes will then be incorporated into a 
codebook with definitions and example quotes to enable structured coding42,43. Using rigorous analysis 
techniques44, two researchers will use the codebook to independently code each interview and meet 
to resolve discrepancies. Inter-coder reliability will be calculated. Additionally, we will add specific 
attributes (e.g., number of scripts filled per day, pharmacy level of rurality) to each transcript, allowing 
us to examine whether fidelity and effectiveness vary by attribute. Because these interview guides and 
the quantitative data are both mapped to CFIR constructs, we will be able to conduct concurrent 
triangulation (comparing results from both data sources on the same questions) as well as elaboration 
analyses (using the qualitative data to provide depth of understanding to the quantitative findings)45,46. 
 
6 
STUDY INTERVENTION 
6.1 
Vaccine Hesitancy Counseling Intervention (ASORT) 
The vaccine hesitancy counseling intervention, known as “ASORT”, will ask pharmacists to identify and 
engage in COVID-19 vaccine hesitancy counseling with one to two vaccine-hesitant individuals weekly. 
As noted earlier, we have refined an evidence-based vaccine hesitancy counseling intervention with 
extensive feedback from rural pharmacists. These refinements resulted in a 5-step counseling process 
(ASORT; see Table 2) as well as an online resource that provides example verbiage for over 25 vaccine 
concerns that have been expressed in rural communities. The vaccine hesitancy verbiage is updated 
periodically with input from our rural patient (four rural patients; two African American, two with 
Republican party affiliation) advisory board. 
 
6.2 
Standard Implementation Approach 
The standard approach will train and prepare pharmacists to implement ASORT and provide discrete 
implementation support tools to support intervention fidelity. Specifically, a trial website will include 
numerous tools, including example vaccine hesitancy verbiage, sample workflows, marketing 
materials, and patient pamphlets. The standard approach also includes an online training module 
developed by the trial team that incorporates similar instructional design principles that have been 
used previously to develop pharmacist communication-focused modules47,48. Finally, just prior to the 
start time of each block of pharmacies, participants will attend a live webinar or watch a pre-recording 
with continuing education (CE) credit that includes interactive training on the intervention, updated 
vaccination recommendations, vaccine storage and delivery, and documentation. 
 
6.3 
Virtual Facilitation Approach 
The virtual facilitation approach will provide expert guidance from trained facilitators regarding 
intervention content and implementation processes. The facilitators will perform the following 
evidence-supported functions49–53: engaging stakeholders; building relationships; identifying and 
training a local facilitator/champion; monitoring progress; providing feedback on progress; identifying 
implementation barriers; problem-solving; re-training and coaching; and reinforcing change. 
Facilitators will attend a 16-hour virtually-delivered training in implementation facilitation provided by 
the Implementation Facilitation Learning Hub, a training center supported by the U.S. Department of 

Page 17 of 24 
 
Veterans Affairs (VA) Quality Enhancement Research Initiative (QUERI). The training teaches principles 
and techniques contained in Dollar et al.’s manual Using Implementation Facilitation to Improve 
Healthcare (Version 352), which was developed by the VA Behavioral Health QUERI, a research center 
devoted to supporting the implementation of behavioral health interventions with the VA. The training 
is highly interactive54, involving significant practice and role play. Training topics include knowledge, 
skills, core competencies of facilitators; facilitation roles and activities (e.g., assessing the site, engaging 
stakeholders, problem identification and resolution); phases of implementation; delivering facilitation 
virtually; and evaluating facilitation. After the training, the trial co-principal investigator will provide 
ongoing coaching and supervision to the trial facilitators. 
 
7. 
SAFETY MANAGEMENT  
7.1 
Risks related to qualitative interviews 
 
The risks associated with this study for pharmacy staff participating in interviews are primarily breach of 
confidentiality and distress from answering questions regarding professional practices. Participants 
might disclose to the interviewer information that could possibly have a negative effect on their 
pharmacies if it were to become public. However, as described in Section 8 below, all interviews will be 
kept confidential and transcripts will be de-identified. Written transcripts will not contain names of 
participants. To minimize distress, during the informed consent process and at the beginning of each 
interview, we will review the data security procedures used by the study to ensure confidentiality and 
data safety. 
 
7.2 
Risks associated with surveys 
The risks associated with pharmacy personnel completing the Feasibility, Acceptability, and 
Appropriateness surveys are primarily breach of confidentiality. A breach of confidentiality may mean 
that the information disclosed on the survey (e.g., staff member’s ratings of a study intervention as not 
appropriate for their setting) could lead to discrimination or could possibly have a negative effect on 
their pharmacies if it were to become public. To prevent this, the surveys are completed on Qualtrics 
and stored on a secure server at UNC. The data are not viewable by anyone outside the study team 
once they are completed on Qualtrics (but could be viewed during the completion of the survey if the 
tablet/computer is left open). All survey data will be kept completely confidential.         
 
7.3 
Risks associated with fidelity observations 
The risks for participating in the fidelity observations are minimal for pharmacists. Observation guide 
information will not be shared outside the study team, and therefore, do not pose employment risks for 
the pharmacist. Pharmacists will receive a study information sheet so they can make an informed 
decision about whether to participate. Since standardized patients will be used, there is no risk to 
customers of the pharmacies. The recording is made within the secure Express Dictate mobile app and 
encrypted recordings are only sent to study personnel, who need the encryption key and Express 
Dictate desktop software to open. Once the virtual coach has completed the observation guide, the 
recording will be permanently deleted. 
 
 

Page 18 of 24 
 
8. 
DATA COLLECTION AND MANAGEMENT  
For pharmacy staff who complete surveys directly in Qualtrics, there will be no need for study staff to 
enter survey data. Pharmacy staff who complete the paper survey will email a picture or scanned copy 
to the project manager, who will manually enter the data into Qualtrics and then delete the email. The 
fidelity observation guide data will be manually entered by study staff into a secure Qualtrics database. 
For qualitative interviews, all transcripts will be de-identified prior to analysis. The study statistician 
will perform all analyses using de-identified datasets. 
 
8.1 
Monitoring Plan 
 
The study PIs will review interview and survey data as well as fidelity data on a quarterly basis for safety
 
issues. In addition, the entire study team will meet monthly to discuss the study's progression and any
 
potential safety or data concerns. 
 
Potential and reportable events will primarily be breaches of confidentiality. Because the primary 
outcome is fidelity to the counseling intervention, we do not anticipate serious adverse events related 
to the study. In the case that an adverse event related to fidelity occurs, the PI will report the adverse 
event to the IRB and NIH in a timely fashion. In the event that the IRB takes an action that affects the 
day-to-day operations of the study, the PI will report those actions to the appropriate NIH Project 
Officer in writing. 
 
If an individual subject decides that they no longer want to participate or is not participating in the data 
collection activities and has not corresponded with study staff over a 3-month period, they will be 
withdrawn from the study. There are no criteria that will be used to stop the entire study prematurely. 
 
8.2 
Confidentiality of the data 
A secure, online survey data collection (Qualtrics) will be used to collect participant survey data. For 
interviews, participants will be informed that they have the right to refuse audio recording; in such 
cases the interviewer will take notes. For recorded interviews, the interviewer (who will be UAMS 
affiliated) will upload each digital recording to a password-protected file on a secure server at UAMS. 
Audio files will be transcribed by a professional transcriptionist who is employed by UAMS. Audio files 
will be maintained on the secure UAMS Server until transcription is complete and transcripts are 
checked for accuracy, at which time they will be destroyed. Recordings for fidelity observations will be 
made using the secure mobile app Express Dictate, will be encrypted, and will be maintained in secure, 
password-protected software or on the UNC server until observation guides are completed; then, the 
recordings will be permanently deleted. Completed survey data will be maintained in the secure 
Qualtrics system at UNC. Transcripts will be analyzed with a qualitative software package, and the 
findings generated with this program will be kept in the study folder on the secure server. To further 
protect confidentiality, we will request that no participant be referred to by name or in any way that 
would allow a person to be identified during the interview. Should this occur inadvertently, the name 
would be redacted from the transcript. 
 
De-identified data on rates of delivering vaccine hesitancy counseling and the number of hesitant 
individuals who received a vaccine will be sent to and used by the study team for analysis. All pharmacy 
participants are trained in HIPAA compliance by nature of their employment and will not introduce 
additional risk beyond the normal risk associated with clinician handling of personal health information. 
Data from pharmacies utilized to calculate quantitative implementation outcomes and effectiveness will 
be stored in secure, password protected file on a UNC Server. All data on fidelity and effectiveness will 

Page 19 of 24 
 
be recorded in aggregate and no identifying or personal health information of customers will be 
collected. 
  
9. 
CONSENT PROCESS  
Informed consent documents for this study include a specific statement that information from the trial 
will be posted on ClinicalTrials.gov. The study PIs and project manager will meet with interested 
individuals for approximately 30 minutes over videoconference to verbally inform them of the details 
of the study as described in the study information sheet provided to them via email prior to the 
meeting. The study information sheet includes all aspects of a written informed consent form, 
including consent information, purpose of the study, length of participation, what is being asked of 
participants, risks, and benefits. Prospective participants will have the opportunity to ask questions 
about the study to the study team during the meeting and will be informed of who to contact if 
additional questions come up at a later time. Once the PIs and project manager have reviewed the 
entire study information sheet and answered all questions, they will obtain verbal consent from each 
participant. Individuals who provide verbal consent will be officially enrolled. 
 
10.      REFERENCES  
1. 
Bolin JN, Bellamy GR, Ferdinand AO, et al. Rural Healthy People 2020: New Decade, Same Challenges. J 
Rural Health. 2015;31(3):326-333. 
2.  
Knapp KK, Paavola FG, Maine LL, Sorofman B, Politzer RM. Availability of primary care providers and 
pharmacists in the United States. J Am Pharm Assoc (Wash). 1999;39(2):127-135. 
3. 
Thomson K. Covid vaccine mistrust is fueling a spike in rural deaths. Here’s what’s fueling the mistrust. 
NBC. https://www.nbcnews.com/think/opinion/covid-vaccine-mistrust-fueling-spike-rural-deaths-
here-s-what-ncna1280746. Published 2021. Updated October 5, 2021. Accessed October 22, 2021. 
4. 
Lin SJ. Access to community pharmacies by the elderly in Illinois: a geographic information systems 
analysis. J Med Syst. 2004;28(3):301-309. 
5. 
Teeter BS, Mosley C, Thomas JL, et al. Improving HPV vaccination using implementation strategies in 
community pharmacies: Pilot study protocol. Res Social Adm Pharm. 2020;16(3):336-341. 
6. 
Qato DM, Zenk S, Wilder J, Harrington R, Gaskin D, Alexander GC. The availability of pharmacies in the 
United States: 2007-2015. PLoS One. 2017;12(8):e0183172. 
7.         Eccles MP, Mittman BS. Welcome to Implementation Science. Implementation Science. 2006;1(1):1. 
8. 
Hamel L, Lunna Lopes, Grace Sparks, Ashley Kirzinger, Audrey Kearney, Mellisha Stokes, and Mollyann 
Brodie. KFF COVID-19 Vaccine Monitor: September 2021. Kaiser Family Foundation. 
https://www.kff.org/coronavirus-covid-19/poll-finding/kff-covid-19-vaccine-monitor-september-
2021/. Published 2021. Accessed October 25, 2021. 
9. 
Lazer D, Ognyanova K, Baum M, et al. The COVID States Project #43: COVID-19 vaccine rates and 
attitudes among Americans. OSF Preprints2021. 

Page 20 of 24 
 
10. 
Bacci JL, Bigham KA, Dillon-Sumner L, et al. Community pharmacist patient care services: A systematic 
review of approaches used for implementation and evaluation. JACCP: JOURNAL OF THE AMERICAN 
COLLEGE OF CLINICAL PHARMACY. 2019;2(4):423-432. 
11. 
Khubchandani J, Sharma S, Price JH, Wiblishauser MJ, Sharma M, Webb FJ. COVID-19 Vaccination 
Hesitancy in the United States: A Rapid National Assessment. Journal of Community Health. 2021:1-8. 
12. 
Souch JM, Cossman JS. A Commentary on Rural-Urban Disparities in COVID-19 Testing Rates per 
100,000 and Risk Factors. J Rural Health. 2020. 
13. 
Crane MA, Faden RR, Romley JA. Disparities In County COVID-19 Vaccination Rates Linked To 
Disadvantage And Hesitancy. Health Aff (Millwood). 2021;40(11):1792-1796. 
14. 
Centers for Disease Control and Prevention. COVID Data Tracker. https://covid.cdc.gov/covid-data-
tracker/#datatracker-home. Published 2021. Accessed October 27, 2021. 
15. 
Adams K. States ranked by percentage of population fully vaccinated. Becker’s Hospital Review. 
https://www.beckershospitalreview.com/public-health/states-ranked-by-percentage-of-population-
vaccinated-march-15.html. Published 2021. Accessed October 27, 2021. 
16. 
Narhi U. Sources of medicine information and their reliability evaluated by medicine users. Pharm 
World Sci. 2007;29(6):688-694. 
17. 
Berenbrok LA, Gabriel N, Coley KC, Hernandez I. Evaluation of frequency of encounters with primary 
care physicians vs visits to community pharmacies among Medicare beneficiaries. JAMA Network 
Open. 2020;3(7):e209132-e209132. 
18. 
Curran GM, Shoemaker SJ. Advancing pharmacy practice through implementation science. Res Social 
Adm Pharm. 2017;13(5):889-891. 
19. 
Grabert BK, Kurtzman R, Heisler-MacKinnon J, et al. Implementation of quality improvement coaching 
versus physician communication training for improving human papillomavirus vaccination in primary 
care: a randomized implementation trial. Translational Behavioral Medicine. 2021. 
20. 
Allen J, Shelton R, Emmons K, Linnan L. Fidelity and its Relationship to Implementation Effectiveness, 
Adaptation, and Dissemination. In: Brownson C, Proctor, ed. Dissemination and Implementation 
Research in Health: Translating Science to Practice Oxford University Press; 2018:267-284. 
21. 
Grol R, Wensing M. Dissemination of Innovations. In: Grol. Wensing E, Davis, ed. Improving Patient 
Care: The Implementation of Change in Health Care. 2nd ed. UK: John Wiley & Sons; 2013:185-196. 
22. 
Wensing M, Fluit C, Grol R. Educational Strategies. In: Grol W, Eccles, Davis ed. Improving Patient Care: 
The Implementation of Change in Health Care UK: John Wiley & Sons; 2013:197-209. 
23. 
Bauer MS, Miller CJ, Kim B, et al. Effectiveness of Implementing a Collaborative Chronic Care Model for 
Clinician Teams on Patient Outcomes and Health Status in Mental Health: A Randomized Clinical 
Trial. JAMA Netw Open. 2019;2(3):e190230. 
24. 
Curran GM, Mukherjee S, Allee E, Owen RR. A process for developing an implementation intervention: 
QUERI Series. Implement Sci. 2008;3:17. 

Page 21 of 24 
 
25. 
Knox L. Module 2: Practice Facilitation as a Resource for Practice Improvement. In: Quality AfHRa, ed. 
pcmh.ahrq.gov: AHRQ; 2015. 
26. 
Parchman ML, Noel PH, Culler SD, et al. A randomized trial of practice facilitation to improve the 
delivery of chronic illness care in primary care: initial and sustained effects. Implement Sci. 2013;8:93. 
27. 
Brown CH, Curran G, Palinkas LA, Aarons GA, Wells KB, Jones L, et al. An overview of research and 
evaluation designs for dissemination and implementation. Annu Rev Public Health. 2017;38:1–22. 38.  
28. 
Brewer NT, Hall ME, Malo TL, Gilkey MB, Quinn B, Lathren C. Announcements versus conversations to 
improve HPV vaccination coverage: a randomized trial. Pediatrics. 2017;139(1). 
29. 
Gamble A, Curran G, Mashburn P, Rosenthal M, Thaw Z, Pepper J, et al. Rural Community Pharmacists’ 
Perspectives and Experiences with COVID19 Vaccines. [Poster]. In press; 2021. 
30. 
Carroll C, Patterson M, Wood S, Booth A, Rick J, Balain S. A conceptual framework for implementation 
fdelity. Implement Sci. 2007;2:40. 
31. 
Clark A, Breitenstein S, Martsolf DS, Winstanley EL. Assessing fidelity of a community-based opioid 
overdose prevention program: modification of the fidelity checklist. J Nurs Scholarsh. 2016;48(4):371–
7. 
32. 
Hallgren KA. Computing inter-rater reliability for observational data: an overview and tutorial. Tutor 
Quant Methods Psychol. 2012;8(1):23–34. 
33. 
Weiner BJ, Lewis CC, Stanick C, Powell BJ, Dorsey CN, Clary AS, et al. Psychometric assessment of three 
newly developed implementation outcome measures. Implement Sci. 2017;12(1):108. 
34. 
Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of 
health services research fndings into practice: a consolidated framework for advancing implementation 
science. Implement Sci. 2009;4:50. 
35. 
Smith MG, Shea CM, Brown P, Wines K, Farley JF, Ferreri SP. Pharmacy characteristics associated with 
the provision of medication management services within an integrated care management program. J 
Am Pharm Assoc. 2017;57(2):217-21 e1. 
36. 
Smith MG, Ferreri SP, Staford R, Teeter BS, Turner K, Veach S, et al. Development of community 
pharmacy characteristics questionnaire: application of cognitive interviewing. Innov Pharm. 2018;9:3.  
37. 
Scaccia JP, Cook BS, Lamont A, Wandersman A, Castellow J, Katz J, et al. A practical implementation 
science heuristic for organizational readiness: R = MC(2). J Community Psychol. 2015;43(4):484–501.  
38. 
Fernandez ME, Walker TJ, Weiner BJ, Calo WA, Liang S, Risendal B, et al. Developing measures to assess 
constructs from the Inner Setting domain of the Consolidated Framework for Implementation 
Research. Implement Sci. 2018;13(1):52. 
39. 
Helfrich CD, Li YF, Sharp ND, Sales AE. Organizational readiness to change assessment (ORCA): 
development of an instrument based on the Promoting Action on Research in Health Services (PARIHS) 
framework. Implement Sci. 2009;4:38. 

Page 22 of 24 
 
40. 
Prevention ECfD, Control. Catalogue of interventions addressing vaccine hesitancy. Germany: ECDC 
Stockholm; 2017. 
41. 
Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3:77–101. 
42. 
Patton MQ. Qualitative evaluation and research methods. San Francisco, CA: Sage; 1990. 
43. 
Moustakas C. Phenomenological research methods. Sage; 1994. 
44. 
Berends L, Johnston J. Using multiple coders to enhance qualitative analysis: the case of interviews 
with consumers of drug treatment. Addict Res Theory. 2005;13(4):373–81. 
45. 
Creswell JW, Plano-Clark VL. Designing and Conducting Mixed Methods Research. 2nd ed. Los Angeles: 
Sage Publications; 2011. p. 520. 
46. 
Schreier M. Qualitative content analysis in practice. London, UK: Sage Publications Ltd.; 2012. p. 280. 
47. 
Gilmartin-Thomas JFM, Sleath B, Cooper Bailey S, Carpenter DM, Chater A, MacAllister C, et al. 
Preparing pharmacy students to communicate effectively with adolescents. Int J Pharm Pract. 
2020;28(2):134–41. 
48. 
Stover AN, Lavigne JE, Shook A, MacAllister C, Cross WF, Carpenter DM. Development of the Pharm-
SAVES educational module for gatekeeper suicide prevention training for community pharmacy staff. 
Health Expect. 2023;26(3):1246–54. 
49. 
Bidassie B, Williams LS, Woodward-Hagg H, Matthias MS, Damush TM. Key components of external 
facilitation in an acute stroke quality improvement collaborative in the Veterans Health Administration. 
Implement Sci. 2015;10:69. 
50. 
Connolly SL, Hogan TP, Shimada SL, Miller CJ. Leveraging implementation science to understand factors 
influencing sustained use of mental health apps: a narrative review. J Technol Behav Sci. 2020;6:1–13.  
51. 
Connolly SL, Stolzmann KL, Heyworth L, Weaver KR, Bauer MS, Miller CJ. Rapid increase in telemental 
health within the department of veterans affairs during the COVID-19 pandemic. Telemed J E Health. 
2020;27. 
52. 
Ritchie MJDK, Miller CJ, Oliver KA, Smith JL, Kim B, Connolly SL, Woodward E, Ochoa-Olmos T, Day S, 
Lindsay JA, Kirchner JE. Using Implementation Facilitation to Improve Healthcare in the Veterans 
Health Administration (V3). In: Administration VH, editor. Behavioral Health Quality Enhancement 
Research Initiative (QUERI). Veterans Health Administration; 2020. p. 1–197. 
53. 
Stetler CB, Legro MW, Rycroft-Malone J, Bowman C, Curran G, Guihan M, et al. Role of “external 
facilitation” in implementation of research findings: a qualitative evaluation of facilitation experiences 
in the Veterans Health Administration. Implement Sci. 2006;1:23. 
54. 
Kirchner JE, Dollar KM, Smith JL, Pitcock JA, Curtis ND, Morris KK, et al. The next challenge in 
implementation science: transferring implementation strategy knowledge and skills. In Review; 2020. 
 
 

Page 23 of 24 
 
APPENDIX 
 
Figure 1: Stepped wedge study design 
Block  
Fall 1 2023 
Oct-Nov 
(8 weeks) 
Fall 1 2023 
Dec-Jan 
(8 weeks) 
Fall 1 2023 
Feb-March 
  (8 weeks) 
Fall 2 2024 
Aug-Sept 
(8 weeks) 
Fall 2 2024 
Oct-Nov 
(8 weeks) 
Fall 2 2024 
Dec-Jan 
(8 weeks) 
1 (5 pharms) 
Standard 
Facilitation 
Follow-up 
  
  
2 (5 pharms) 
Standard 
Standard 
Facilitation 
Follow-up 
  
3 (5 pharms) 
Standard 
Facilitation 
Facilitation 
Follow-up 
 
 
4 (5 pharms) 
  
  
  
Standard 
Facilitation 
Follow-up 
5 (5 pharms)  
  
  
  
Standard 
Standard 
Facilitation 
6 (5 pharms) 
  
  
  
Standard 
Facilitation 
Facilitation 
 
 
Table 1: Outcome measure descriptions and data collection procedures 
Measure 
Description 
Quantitative 
Qualitative†  
Primary Outcome 
Fidelity 
Degree to which the vaccine 
hesitancy counseling intervention 
was delivered as intended 
Fidelity Checklist that assesses 
adherence and competence to vaccine 
hesitancy counseling intervention. Each 
pharmacist assessed twice per month. 
Interviews with pharmacy 
personnel  
Secondary Outcome 
Effectiveness 
The proportion of vaccine hesitant 
individuals who vaccinate after 
receiving hesitancy counseling 
Monthly report completed by pharmacy 
designee using web-based reporting 
system 
Interviews with pharmacy 
personnel  
Other Measures 
Acceptability 
Stakeholder perceptions regarding 
satisfaction with and appeal of the 
vaccine hesitancy intervention and 
implementation approaches  
4-item measure completed by pharmacy 
personnel at end of standard & virtual 
facilitation approach 
Semi-structured interviews 
with pharmacy personnel   
Uptake 
How often vaccine hesitancy 
counseling was offered to hesitant 
individuals 
Monthly report completed by pharmacy 
designee using web-based reporting 
system 
Appropriateness 
Stakeholder perceptions regarding 
fit and suitability of the vaccine 
hesitancy intervention and 
implementation approaches 
4-item measure completed by pharmacy 
personnel at end of standard & virtual 
facilitation approach 
Feasibility 
Stakeholder perceptions regarding 
ease and “do-ability” of the vaccine 
hesitancy intervention and 
implementation approaches 
4-item measure completed by pharmacy 
personnel at end of standard & virtual 
facilitation approach 
Organizational 
Structure and 
Context 
Key structural aspects of pharmacy 
(e.g., size/staffing) and 
organizational capacity for change 
Structure survey completed by one 
pharmacist before randomization; 
context survey completed by ≥ 5 
pharmacy personnel 
N/A 
 
Sustainment 
Continued measurement of fidelity, 
effectiveness, and uptake during the 
“follow-up” periods 
See above entries for fidelity, 
effectiveness, and uptake 
Cost 
Costs associated with deployment of 
each implementation approach. 
Log of time and activities completed 
monthly by facilitator 
† qualitative data will be collected after the end of the virtual facilitation period with 1 high performing and 1 low performing 
pharmacies per block 

Page 24 of 24 
 
 
Table 2: Selected content from the 5-step vaccine hesitancy counseling process (ASORT) 
Step 
Recommendations & Example Verbiage 
Ask if they would like to 
receive a COVID 
vaccination  
• People are more open to talking about the COVID-19 vaccine if you ask while you’re 
doing other activities, like giving a flu shot or engaging in medication therapy 
management. 
“While I’m giving your flu shot, I just thought I’d ask if you’ve gotten your COVID 
vaccine yet.” 
• Offer praise to people who are up-to-date on their vaccination 
Solicit their main vaccine 
concern 
• People often have multiple concerns about the vaccine, but one concern will likely loom 
larger than the others, so this is the concern you’ll want to focus on first. 
“Can you tell me more about that?” 
Offer to address their 
concerns 
• People have different levels of readiness to discuss the vaccine, so it’s important to ask 
for permission to share more information about their concerns. 
• Start by validating their concerns so they know that you’re not judging them. 
“I know several other people who have had that same concern and I’ve shared some 
information with them that they’ve found useful. I’d be happy to share that same 
information with you if you want.” 
• Some people won’t be ready for more information and that’s okay. Just let them know 
that you understand.  
“Ok. No problem. Know that I’m here if you do ever want to talk.” 
• Address their concerns 
• For individuals who aren’t ready, skip to the last step. 
Recommend the vaccine 
• Share your personal experience with the vaccine and that you trust it before you 
recommend it. This can help build their trust in the vaccine. 
• After sharing your personal experience, then recommend the vaccine.  
“I wouldn’t recommend the vaccine if I didn’t think it was safe. I received it and I trust 
it. That’s why I recommend that you get the vaccine - because I care about you and 
want you to keep you safe.” 
• You can also tie your recommendation to any factors that may put them or their family 
members at higher risk for severe COVID complications.  
• If they are still unsure or refuse, then move to Step 5. 
Try again later if they 
refuse or are unsure 
• As we’ve seen throughout the pandemic, many people who say they will never get the 
vaccine have since been vaccinated. So don’t be discouraged if they refuse. React in a 
positive way and let them know you’ll check in with them again. 
“Thanks for considering it. I’ll check in with you again if I hear any new information 
about your concern.” 
• Since people can and do change their minds, it’s important to try again during one of 
their next visits to the pharmacy. 
• For regular customers, keep a list of people to follow up with or make a note in the 
pharmacy record to follow up.