Increasing HPV Vaccination Rates Through Virtual Immersive Communication Training on Recommending Immunizations: An Efficacy Study of VICTORI
Phase: NA
Conditions: Human Papilloma Virus
Sponsor: Brittany Rosen

Title: Increasing HPV Vaccination Rates through Virtual Immersive Communication Training 
on Recommending Immunizations: An Efficacy Study of VICTORI 
 
NCT: NCT04368455 
 
 
Document Date: 14 June 2022  
 
 
 

Version:  7.0
Increasing HPV Vaccination Rates through Virtual Immersive Communication
Training on Recommending Immunizations: An Efficacy Study of VICTORI
Principal Investigator: F. Joseph Real, MD, MEd
Co-investigators:
Brittany Rosen, PhD
Nick DeBlasio, MD, MEd
Nicholas Ollberding, PhD
Melissa Klein MD, MEd
Jessica Kahn, MD, MPH 
Taylor Lammert, MD
Melody Lee, MD
Emily Popler, MD
Division of General and Community Pediatrics
Cincinnati Children’s Hospital Medical Center
3333 Burnet Ave., ML 2011
Cincinnati, OH 45229
Division of Biostatistics and Epidemiology 
Cincinnati Children’s Hospital Medical Center
3333 Burnet Ave., MLC 5041
Cincinnati, OH 45229
Division of Adolescent Medicine
Cincinnati Children’s Hospital Medical Center
3333 Burnet Ave.
Cincinnati, OH 45229

Version:  7.0
Background:
Despite strong evidence the human papillomavirus (HPV) vaccine is effective in preventing certain 
anogenital cancers, only 65% of adolescent females and 56% of males in the U.S. have initiated 
the series, and only 43% of girls and 31% of boys have completed it.1 These rates fall short of 
Healthy People 2020’s objective of 80% coverage for adolescents 13-15 years of age.2 Research 
has demonstrated that the leading predictor of parents’ decisions to vaccinate their child against 
HPV is a strong clinician recommendation.3-8 However, evidence suggests that many parents of 
age-eligible adolescents are not receiving HPV vaccine recommendations9 or are receiving weak 
recommendations from clinicians.10-12 Previous interventions aimed at increasing HPV vaccination 
rates have had variable effects in part due to reach and adoption of the interventions within the 
practices.13-19 Also, the diffusion of successful interventions have been limited by lack of scalable 
designs.20 Thus, we will solve this limitation by standardizing the training component and moving 
towards developing a scalable model that translates into consistently improved HPV vaccine rates. 
Simulation-based medical education (SBME) has become an essential component of clinical 
training, demonstrating superior effectiveness in teaching a wide range of medical skills compared 
to conventional training methods.21-23 Immersive virtual reality (VR), one type of SBME, is a 
three-dimensional, computer-generated environment where users interact with graphical 
characters called avatars. Within VR, a facilitator can design scenarios based on specific 
behavioral objectives and create environments and avatars based on training needs. The technology 
facilitates deliberate practice, a personal and goal-oriented approach to skill development derived 
from Ericsson’s Theory on Expertise.24 In our preliminary study conducted in 2015, we developed 
an intervention for resident physicians to address influenza vaccine hesitancy, comprised of a self-
directed application (app) based curriculum about HPV vaccination followed by immersive VR 
simulations.25 Completion of the VR simulations, compared to receipt of standard training, led to 
a significantly lower rate of influenza vaccine refusal.26 Based on these promising preliminary 
results, we developed VR simulations focused on HPV vaccine counseling. 
Purpose:
Our approach will be to implement Virtual Immersive Communication Training on 
Recommending Immunizations (VICTORI), an intervention that includes a self-directed app-
based curriculum and VR simulations, designed to increase the strength and consistency of HPV 
vaccine recommendations among clinicians. VICTORI will be implemented using a framework 
informed by behavioral (Social Cognitive Theory, Health Belief Model, and Theory of Planned 
Behavior) and educational theory (Ericsson’s Theory on Expertise). The primary outcome will be 
an increase in HPV vaccine rates among adolescent patients. We will also assess theory-based 
mechanisms by which the intervention changes vaccination rates, including clinicians’ knowledge, 
attitudes, perceptions, self-efficacy, and strength of recommendations.
Our long-term goal is to increase HPV vaccination rates in adolescents, which will decrease rates 
of HPV-associated cancers and pre-cancers. Our short-term objective is to evaluate the efficacy of 
VICTORI, a novel VR intervention, designed to enhance the strength of clinicians’ HPV vaccine 
recommendations and improve HPV vaccine rates among 9 to 17-year-old patients. Our central 
hypothesis is that VICTORI, which includes VR simulations, will be positively associated with an 
increase in vaccine rates, and this association will be mediated by improvements in clinician 
attitudes, self-efficacy, and strength of vaccine recommendation. The rationale for the proposed 

Version:  7.0
research is that an evaluation self-directed app based curriculum about HPV vaccination and VR 
simulations will inform effective and scalable strategies to train clinicians to provide strong 
vaccine recommendations, resulting in increased HPV vaccine rates and ultimately lower rates of 
HPV-associated cancers. To accomplish our objective, we will achieve the following specific 
aims: 
1. Evaluate VICTORI for acceptability and feasibility. The working hypotheses, based on our 
preliminary data and previous experience with a vaccination-focused intervention that included 
VR simulations,26 are that: 1) clinicians (physicians) and staff (nurses and medical assistants) 
will find VICTORI (self-directed app based curriculum and VR simulations) acceptable (i.e., 
they will find it realistic, immersive, and easy to use) and will appreciate the user-centered 
components and presentation of information and 2) VICTORI will be feasible in terms of time 
required for participation and ease of incorporation of training into the outpatient setting. 
Results from this aim will be used to modify the intervention to optimize acceptability and 
feasibility.
2. Conduct a single-site intervention assessing the efficacy of VICTORI in increasing HPV 
vaccine rates. The working hypotheses are: 1) adolescent HPV vaccination rates will increase 
significantly following implementation of VICTORI compared to pre-intervention rates and 2) 
this increase will be mediated by clinicians’ more positive attitudes, higher self-efficacy, and 
stronger vaccine recommendations.
3. Determine modifiable barriers related to HPV vaccine non-completion for future 
enhancement of VICTORI accordingly. We will obtain information about barriers to vaccine 
series completion from a retrospective chart review of patients who initiated but did not 
complete the HPV vaccine series during our study.
Location
Aim 1: Evaluate VICTORI for acceptability and feasibility will take place within the Division 
of Adolescent Medicine in a conference room at CCHMC. Participants will include clinicians 
and staff from the Teen Health Center (THC). The THC at CCHMC serves as a medical home 
for 6,442 patients with just over 7,000 visits each year. The THC manages and treats medical, 
social, emotional, sexual, educational, and nutritional concerns impacting adolescent youths. 
HPV vaccination is an important preventative care action at the THC. The clinicians at the THC 
frequently discuss HPV vaccination with patients and families and are thus suitable candidates to 
provide meaningful feedback regarding the acceptability and feasibility of the proposed 
intervention. 
Aim 2: Conduct a single-site intervention assessing the efficacy of VICTORI in increasing 
HPV vaccine initiation and completion rates will take place within the Division of General and 
Community Pediatrics in a conference room at CCHMC. The PPCC is a busy, academic clinic in 
Hamilton County, Ohio with about 33,000 visits annually, serving 19,000 patients that are 
predominantly underserved with approximately 90% insured through Medicaid. Seventy-five 
percent of the patient population is African American and 5% is Latino. PPCC serves a population 
that is at high risk for HPV infection and its sequela, and therefore, we anticipate the findings to be 
generalizable to other urban primary care clinics that serve predominantly minority populations. The 

Version:  7.0
Hopple Street Neighborhood Health Center (HSNHC) will serve as a control during Aim 2. The 
HSNHC serves a similar population to the PPCC with 6000 unique patients and 13,000 patient 
visits per a year. VICTORI, our educational intervention, will be conducted in such a way as to not 
disturb or interrupt typical clinical flow.   
Aim 3: Determine modifiable barriers related to HPV vaccine non-completion for future 
enhancement of VICTORI accordingly. The procedures for this aim will take place within the 
Division of General and Community Pediatrics at CCHMC. The PPCC is a busy, academic clinic 
in Hamilton County, Ohio with about 33,000 visits annually, serving 19,000 patients that are 
predominantly underserved with approximately 90% insured through Medicaid. Seventy-five 
percent of the patient population is African American and 5% is Latino. PPCC serves a population 
that is at high risk for HPV infection and its sequela, and therefore, we anticipate the findings to be 
generalizable to other urban primary care clinics that serve predominantly minority populations.  
Hopple Street Neighborhood Health Center (HSNHC) which served as a control during Aim 2, 
serves a similar population to the PPCC with 6000 unique patients and 13,000 patient visits per a 
year. 
Duration
This study will take a total of 24 months. Aim 1 will take place over the first 6 months. Aim 2 
will take place over the entire 24-month period. A timeline has been included below. 
 
Protection of Subjects
Aim 1: Evaluate VICTORI for acceptability and feasibility
Physicians (target number = 11, maximum =18) and staff members (target number = 15, maximum 
= 30) at CCHMC THC will be recruited for this study to evaluate the acceptability and feasibility of 
VICTORI. Participants will receive $50 for their efforts. Staff will also be provided lunch. The 
VICTORI Timeline
Year 1
Year 2
Year 3
Q1
Q2
Q3
Q4
Q1
Q2
Q3
Q4
Q1
Q2
Q3
Q4
Recruitment and enrollment
Aim 
1 
feasibility 
and 
acceptability testing
Aim 1 analyses
Aim 2 baseline data collection
Aim 
2 
VICTORI 
implementation
Aim 2 post VICTORI data 
collection
Aim 2 environmental control 
data collection
Aim 2 analyses
Aim 3 retrospective chart 
review data collection
Aim 3 analysis

Version:  7.0
research materials to be obtained from physician participants are interview data collected through 
semi-structured interviews. The interviewer will record field notes during the interviews, including 
the amount of time it takes a participant to complete the VR simulations and any additional 
adjustments needed for the technology. Additionally, the interview content will include overall 
impressions of the VR simulations, perceived purpose of the VR simulations, suggestions for 
changes to the simulations, if the participant would recommend the intervention to a colleague, and 
if the participant experienced motion sickness. Interviews will last approximately 30 minutes and 
will be audio and video recorded. The research materials to be obtained from staff participants are 
data collected through focus group interviews of 5-8 individuals. A study member (JR, BR, MB, 
SM) other than the focus group facilitator will record field notes during the interviews, including 
any additional adjustments needed for the technology. Questions for the focus group will focus on 
overall content, length of training, engagement in group learning activities, and suggestions for 
changes to the intervention. Focus groups will last approximately 45 minutes and will be audio-
recorded. All semi-structured interviews and focus group interviews will be transcribed by HIPPA 
compliant transcription company. Transcribed interview and focus group data will be de-identified 
and data files will be kept in a password-protected file on a password-protected computer. Data 
collection will continue until a saturation of themes is met per qualitative research guidelines. At the 
completion of the VR simulations, physician participants will complete the MEC-Spatial Presence 
Questionnaire (MEC-SPQ), a validated tool to assess the realistic and immersive nature of virtual 
environments,27 as well as provide participant characteristics. This survey will take an additional 5 
minutes to complete. Staff participants will also complete a survey assessing participant 
characteristics. These data will be entered into Research Electronic Data Capture (REDCap)28 
upon study completed with hardcopies of surveys being destroyed. Only study personnel will have 
access to the data. 
Aim 2: Conduct a single-site intervention assessing the efficacy of VICTORI in increasing 
HPV vaccine initiation and completion rates
Physicians (n = 70) and staff (n = 40) at CCHMC PPCC will be recruited for this study to evaluate 
the effectiveness of VICTORI on HPV vaccine initiation and completion rates. Physicians from 
PPCC will receive $50 for their efforts. Staff will receive a $10 gift card for their efforts. 
Additionally, physicians (target number = 35) from HSNHC will be recruited to serve as the 
control. HSNHC participants will receive the self-directed app based curriculum component of 
VICTORI though will not undergo the VR simulations. Participants from HSNHC will receive $25 
for their efforts.
The primary outcome is the increase in the proportion of patients receiving HPV vaccination seen 
by resident and attending physicians following participation in VICTORI. These data will be 
obtained from the EPIC electronic medical record on a monthly timeline from PPCC 6 months prior 
to implementing VICTORI (baseline), the months during intervention implementation, and 6 
months after the intervention (post) for physicians, as well as 6 months after the intervention (post) 
for staff. These data will be compared with monthly HPV vaccination data from HSNHC from the 
start of the study until 22 months to serve as a control. Data will be blinded and entered into a 
REDCap database. The research materials to be obtained from participants are data collected 
through audio/video recordings during VICTORI and items from validated survey instruments to 
evaluate secondary outcomes. These secondary outcomes include physicians’ behavioral capability, 
self-efficacy, perceived benefits, barriers, and risk, as well as attitudes, subjective norms, and 
strength of recommendations. We will also collect data on staffs’ behavioral capability, perceived 

Version:  7.0
benefits, barriers, and risk, as well as attitudes, subjective norms and ability to provide consistent 
positive messaging about the HPV vaccine. We will also collect data on participant characteristics. 
Survey data will be collected pre-intervention, immediate post-intervention, and 3 months post-
intervention for physicians and pre-intervention and immediate post-intervention for staff. Data files 
will be kept in REDCap. Only study personnel will have access to the data.
 
Aim 3: Determine modifiable barriers related to HPV vaccine non-completion for future 
enhancement of VICTORI accordingly.
We will conduct a retrospective chart review to explore social determinant of health barriers to 
vaccine series completion from a retrospective chart review of patients who initiated but did not 
complete the HPV vaccine series during our study. We will identify patients within the cohort at 
PPCC and Hopple who initiated but did not complete the HPV vaccine series within a 14 month 
period following a visit with physicians that participated in the Aim 2 portion of the study.  For 
patients under the age of 15 years, the second dose of the HPV vaccine series is to be administered 
six months following the first dose. We have operationalized HPV vaccine series completion to 
occur within 14 months as this provides an adequate and pragmatic timeframe for vaccine series 
completion at a follow-up visit or annual well child visit. 
The primary outcome is the proportion of patients within a cohort at PPCC and Hopple who 
initiated but did not complete the HPV vaccine series within a 14 month period following a visit 
with physicians that participated in the Aim 2 portion of the study. These data will be obtained 
from the EPIC electronic medical record of patients who initiated the HPV vaccine series but have 
yet to complete it. Data will be entered into a REDCap database. The research materials to be 
obtained from participants are data collected through review of retrospective chart review of the 
GPC Social Risk Family Questionnaire, demographic data (DOB, age, sex, race, ethnicity, health 
insurance), visit information, and HPV vaccine dates (see appendix VICTORI Chart Review). 
Potential Benefits
Providers who participate in the VICTORI curriculum will obtain additional knowledge on how 
to appropriately counsel parents that express vaccine hesitancy.  The goal is to improve 
physician-patient communication which results in improved clinical outcomes (less vaccination 
refusal) for this high-risk population.  
Potential Risks
Aim 1: Evaluate VICTORI for acceptability and feasibility
We estimated the risks of participating in this study to be minimal since participation is limited to 
VICTORI, interviews, focus groups and a survey focused on spatial presence within the VR 
simulations. Some participants might be uncomfortable participating in the VR simulations in the 
presence of a researcher, while being recorded. However, we expect this to be minimal as it is a 
low-stakes environment. There is a small risk of breach of confidentiality through focus group and 
transcription of interview data, but we expect the risk to be minimized due to the efforts and 
protocol described.
Aim 2: Conduct a single-site intervention assessing the efficacy of VICTORI in increasing 
HPV vaccine initiation and completion rates

Version:  7.0
We estimate the risks of participating in this study to be minimal since participation is limited to the 
VICTORI intervention and survey. Some participants might be uncomfortable participating in the 
intervention in the presence of a researcher, while being recorded. However, we expect this to be 
minimal as it is a low-stakes environment. There is a small risk of breach of confidentiality through 
collection of survey data and transcription of the interview data, but we expect the risk to be 
minimized due to the efforts and protocol described.
Aim 3: Determine modifiable barriers related to HPV vaccine non-completion for future 
enhancement of VICTORI accordingly.
There is a small risk of breach of confidentiality through collection of data, but we expect the risk to 
be minimized due to the efforts and protocol described.
Risk/Benefit Analysis
Given the minimal risks and the potential benefits, the investigators believe that the benefits 
outweigh the risks. The main risk posed by this study is the potential for loss of confidentiality. 
All electronic data files will be kept on a secure, password-protected server with access 
determined by the Co-PIs and the analytic team. Hardcopies of all information will be kept in 
locked filing cabinets. Only the Co-PIs and study staff will have access to this information.
All participants will be monitored for safety during the study. We do not anticipate any adverse 
events as a result of participating in the intervention as this is not a medical intervention. We will 
review all VICTORI experiences among enrolled participants to ensure that the intervention does 
not contribute to an adverse event directly linked to the intervention experience. We will use a 
case report form that will record specifics surrounding an experienced adverse event, including 
type, severity, and relationship to the study, actions that were taken, and current status of the 
participant. Study staff will notify the Co-PIs or a qualified designee immediately of any serious 
adverse events or any adverse events that are suspected to be related to the study. 
Research Methodology
The aims for this study will be accomplished through mixed qualitative and quantitative research 
methods. Aim 1 will utilize mixed-methods research to obtain information from clinicians and 
staff about the acceptability and feasibility of VICTORI. Based on these data, adjustments and 
revisions will be made to VICTORI to optimize acceptability and feasibility. Aim 2 will be a 
single-site intervention assessing the efficacy of VICTORI (self-directed app-based curriculum 
and VR simulations) on HPV vaccine rates (primary outcome). We will assess the theory-based 
constructs that have been used in the design of VICTORI to determine whether they impact 
clinician recommendations and are mediators of the effect of VICTORI on vaccination rates. 
These constructs will include behavioral capability and self-efficacy from SCT, perceived 
benefits, barriers, and risk from HBM, and attitudes and subjective norms from TPB. The 
rationale for selecting these constructs is they have been identified as the most powerful 
predictors of clinician recommendations for and parental acceptability of HPV vaccines.7,29-49 
Aim 3 will be a single-site retrospective chart review exploring social determinants of health that 
are associated with HPV vaccine series completion. 
Aim 1: Evaluate VICTORI for acceptability and feasibility. 

Version:  7.0
Figure 1: (A) Education is administered using a head-mounted display that shows 3D 
graphics. (B) After each simulation, a learner receives feedback. 
A
B
Setting and Population: Before implementing the VR simulation, all clinicians will receive a self-
directed app based curriculum. Clinicians will receive education to increase their knowledge of 
HPV and the vaccine, perceived benefits, positive HPV vaccine attitudes, and subjective norms as 
well as decrease their perceived barriers to recommending the vaccine. The participants will 
engage in the VR simulations independently. The simulations will be audio and video recorded. 
The VR simulations, already developed through funding from APA, consist of four simulated 
scenarios 
in 
which 
participants will counsel 
caregiver avatars hesitant 
to accept the HPV vaccine 
for their child. During each 
simulated encounter, the 
participant will counsel the 
HPV 
vaccine-hesitant 
caregiver through verbal 
dialogue. 
For 
the 
participant 
to 
succeed 
(vaccination accepted), the participant has to demonstrate the following best-practice 
communication skills related to HPV vaccine 
counseling: (1) using presumptive “announcements” or short statements assuming parents are in 
agreement to vaccinate their child, (2) consistent positive messaging, (3) providing a strong 
recommendation, and (4) accurately addressing caregiver concerns without medical jargon.13,50-53 
The parent avatar’s verbal and nonverbal responses to the participant’s counseling will be driven 
by a single facilitator (either Dr. Real or Dr. Rosen; Figure 1). Each scenario will use a flow sheet 
to standardize the experience. Simulations flowsheets were developed using an iterative process 
and collaborative effort between pediatricians and technologists. Table 1 provides details 
regarding the four VR scenarios that include the most salient and important sources of hesitation.54-
58 The four-scenario intervention will take approximately 20 minutes to complete and will be 
scheduled at a time convenient for the participants. After each scenario, the facilitator will provide 
the participant with specific feedback regarding his or her use of best-practice communication 
skills based on a 
modified version 
of a previously 
developed 
rubric.26
While 
the 
staff 
participants will 
complete 
the 
same 
intervention, the 
staff will receive 
the intervention 
through 
a 
different method. Staff members will receive the intervention in a small group setting. The 
rationale for this method, which has been shown to be effective in delivering key educational 
Table 1:  Virtual Reality Simulations
Setting: Cincinnati Children’s Hospital Medical Center Pediatric Primary Care Clinic
Scenario 1
Scenario 2
Scenario 3
Scenario 4
Narrative: 11 yo 
female 
patient 
presents for well-
child care 
Characters: parent 
and 
female 
adolescent avatars 
Source 
of 
Hesitation: none
Narrative: 11 yo male 
patient presents for 
well-child care 
Characters: 
parent 
and male adolescent 
avatars 
Source of Hesitation: 
concerns 
regarding 
safety and side effects
Narrative: 11 yo female 
patient presents for well-
child care 
Characters: parent and 
female adolescent avatars 
Source of Hesitation: 
concerns 
regarding 
necessity, young age, and 
promoting sex 
Narrative: 11 yo male 
patient presents for well-
child care 
Characters: parent 
and 
male adolescent avatars 
Source 
of 
Hesitation: 
concern 
regarding 
that 
vaccine is not required by 
school and male patients 
don’ t need it.  
Key Skills: Presumptive announcement, consistent positive messaging, strong recommendation, 
emphasizes timing of second dose, and addressing specific concerns without medical jargon  

Version:  7.0
components,59,60 is to inform the staff (nurses, medical assistants, and other patient facing staff 
members—who are often the first health professional to meet with patients)61,62 about the 
communication training physicians are receiving and ensure staff members are providing 
consistent, positive vaccination messages that align with the VICTORI objectives. After watching 
a 5-minute video on evidence-based practices in recommending the HPV vaccine, participants will 
observe Dr. Real or another co-investigator clinician (a clinician) participate in the VR simulation 
while Dr. Rosen or another co-investigator facilitates the experience. Staff will be able to observe 
the VR scene and interactions through a projector set-up. A debrief of the VR observation will 
occur to reinforce background knowledge, attitudes, perceptions and consistent positive 
messaging. 
Measures and Data Analysis: To 
evaluate 
the 
acceptability 
and 
feasibility of VICTORI, we will use a 
mixed methods approach. Physicians 
will complete the MEC-SPQ, a 
validated tool to assess the realistic 
and immersive nature of virtual 
environments (see Appendix A)27 
after implementing VICTORI. The 
MEC-SPQ is simply meant to inform 
the extent of immersion experienced 
by users. It will be used to identify 
deficiencies 
in 
the 
virtual 
environment or programming that 
require modification prior to study 
implementation. 
Next, 
individual 
cognitive interviews for clinicians 
lasting approximately 30 minutes in a 
private location will be conducted. 
Interviews will be directed by either 
Dr. Real or Dr. Rosen. The interview 
items were developed from seminal 
literature in usability testing (Table 
2).63-66  For staff member participants, 
Dr. Real or Dr. Rosen will conduct a 
focus group to obtain feedback 
regarding acceptability and feasibility 
following 
demonstration 
of 
VICTORI. Questions only for staff 
are noted by superscript letter (a) in 
Table 
2. 
The 
interviews 
with 
providers will be audio and video 
recorded and transcribed by a HIPPA 
compliant transcription company and 
Table 2: Post Interview Questions
Participating in the Intervention
1.
What are your overall impressions of the VICTORI curriculum?a
2.
Name three words or characteristics that describe the VICTORI 
curriculum.
a)
RN/MAs: Name one word or characteristic that 
describes the VICTORI curriculum.a
b)
Explain why you picked these words or characteristics.a
3.
What are the three things you like least about the VICTORI 
curriculum?
4.
Would you participate in a virtual reality intervention like this 
(VICTORI) in the future?a
a)
Please share more?a
5.
Would you recommend the VICTORI curriculum to a colleague? 
Why/why not/please share more? a
6.
What information would you like to see added to the VICTORI 
curriculum? a
a)
Any changes to the current scenarios? 
b)
Any frequently encountered scenarios missing?a
7.
What information from the HPV Vaccine: Same Way, Same 
Day™ app was helpful to you before participating in the virtual 
reality training component of VICTORI? 
a)
If not, what components were missing?
8.
Did you think the duration of the VR experience was 
appropriate?a
a)
Too long? What should be removed?a
b)
Too short? What should be added?a
9.
Did you think the feedback during the VR experience was 
appropriate?a
a)
What should be removed?a
b)
What should be added?a
10. What concerns did you have while participating in the 
experience?a
The VR Device
11. How easy or difficult was the VR device to use? Tell me more. 
12. How did participating in the VR simulations make you feel?
a)
Did you feel immersed?
b)
Did it feel realistic?
c)
Did you experience any side effects? (PROBES: 
distracted, anxious, at ease, motion sickness, etc.)
13. How did you feel about the environment in which the training took 
place? What could we have done to make the environment more 
conducive to learning?
a indicates specific questions for staff

Version:  7.0
the research team.  The focus group will be audiotaped and transcribed by a HIPPA compliant 
transcription company. The interviewer will record field notes during the clinician and staff 
participation in the intervention, including the amount of time it takes to complete each component 
of the intervention as well as any possible adjustments needed for the 5-minute video on evidence-
based practices in recommending the HPV vaccine and VR simulations. The research team will 
review interview transcripts using the framework analysis approach, a multistep method for 
analyzing qualitative data, to assess the interview data and create a conceptual framework.38,67 
NVivo (QSR International, Doncaster, Australia) will be utilized to support data coding and 
analysis. Data collection will continue until a saturation of themes is met per qualitative research 
guidelines. A content analysis of the transcribed interviews will be conducted, and themes will be 
examined for evidence of needed revisions.
Expected Outcomes: These findings will provide insight from the users’ perspective regarding 
the strengths and weaknesses of the intervention. These data are required for the design of 
effective, scalable, user-centered interventions for clinicians to increase strong HPV vaccine 
recommendations to hesitant parents. These data will be used to guide revisions to the 
intervention before implementation (Aim 2). We will specifically adjust the intervention based 
on the responses to post interview questions with potential modifications including: adjusting the 
content of simulations, adjusting the order of simulations, adjusting the timing of simulations, 
adjusting the environment in which VICTORI will be administered, and/or adjusting the VR 
headset specifications (i.e. sizing). Staff will specifically provide feedback regarding small group 
learning. We plan to trial the modified version of the simulation with study team members (FR, 
BR, JK, MK, DD) prior to study implementation to ensure feedback from acceptability study has 
been successfully addressed.  
Potential Problems and Alternative Strategies: Low response rate may occur; to mitigate this 
possibility, we will work with the study team to recruit chief residents at CCHMC. Chief residents 
are a feasible alternative population as the majority previously worked at the PPCC as residents, 
and there is no risk of contamination. Furthermore, if the Food and Drug Administration releases 
new recommendations for existing vaccination, we will revise VICTORI along with the survey 
instrument to reflect these changes. 
Aim 2: Conduct a single-site intervention assessing the efficacy of VICTORI in increasing 
HPV vaccine rates. Aim 2 will utilize quantitative research methods to determine if rates of 
HPV vaccine uptake among 9 to 17-year-old boys and girls increase following the 
implementation of VICTORI for physicians and staff compared to pre-intervention rates. In 
addition, a survey instrument will be used to measure theory-based behavioral constructs 
hypothesized to mediate the intervention effect.
Setting and Population: We will recruit and implement the intervention in the CCHMC PPCC. 
There is no concern of contamination from the preliminary flu study as it was completed in 2015 
since staff members were not included and all participants graduated by spring 2017. The PPCC 
is a busy, academic clinic in Hamilton County, Ohio with about 33,000 visits annually, serving 
19,000 patients that are predominantly underserved with approximately 90% insured through 
Medicaid. Seventy-five percent of the patient population is African American and 5% is Latino. 
PPCC serves a population that is at high risk for HPV infection and its sequela, and therefore, we 

Version:  7.0
anticipate the findings to be generalizable to other urban primary care clinics that serve 
predominantly minority populations. The PPCC is staffed by approximately 20 attending 
physicians (total: 8.0 Full-time equivalent), 80 pediatric residents, 25 nurses, 21 medical 
assistants, 5 nurse care managers, 3 social work care managers, 3 social workers, 1 community 
engagement specialist, 14 community health workers, 2 customer service representatives, and 6 
psychologists. Additionally, we will recruit resident and attending physicians from the HSNHC 
who will not receive the VR intervention but will serve as controls, only receiving the self-
directed app-based curriculum. HSNHC serves a predominantly underserved population as well 
with 90% publicly insured. Seventy-three percent of the patient population is African American, 
and 5% is Latino. HSNHC is staffed by approximately 7 attending physicians (total: 2.5 Full-
time equivalent) and 28 pediatric residents. HPV initiation rates in Ohio are 57% for girls and 
55% for boys; completion rates are 42% for girls and 41% for boys.1 The completion rate in 
Hamilton County is 16% for boys and girls combined.68 The vaccination initiation rates in the 
PPCC (FY17) for 11 to 12-year-old girls was 48% (257 vaccinations initiated out of 537 visits) 
and for 11 to 12-year-old boys was 52% (289 vaccinations initiated out of 557 visits). The HPV 
vaccination initiation rate in the PPCC (FY17) for 11 to 17-year-old boys and girls was 43% 
(619 vaccinations initiated out of 1450 visits). The rationale for including adolescents 9 to 17 
years old is this will allow us to perform a stratified analysis to determine if the intervention is 
more effective during the target range (11-12 years), the catch-up range (13-17 years), or early 
vaccination range (9-10 years). In addition, the National Immunization Survey (NIS) for Teens 
assesses vaccination rates in children age 13-17 years. By including all children age 9-17 years 
within our study population, it will allow comparison with the NIS-Teen outcome measures. The 
PPCC cares for very few patients 18 years and older, and we therefore did not extend our 
population beyond 17 years. HSNHC, environmental control, had a 51% HPV initiation rate 
(FY17) for 11 to 17-year-old boys and girls (183 vaccinations out of 357 visits). The HSNHC 
serves a comparable site to the PPCC and is similarly staffed by physicians from CCHMC 
though distinct from those physicians practicing at PPCC.
 
Design and Measures: Once acceptability of VICTORI has been assessed through Aim 1 and 
modifications have been finalized, Dr. Real and Dr. Rosen will undergo fidelity training to 
ensure uniform usage of the simulation flow sheets and performance rubrics. Minimum passing 
score (i.e., performance on simulation that results in vaccine acceptance) will be established 
using the borderline group standard setting method.69-71 The borderline group method is a well-
established process for evaluating examinees that uses a checklist to assess proficiency when 
directly observing a performance test. VICTORI will be implemented using the same methods as 
described in Aim 1 for physicians and staff. VICTORI VR simulation implementation may take 
place over ZOOM based on the COVID-19 pandemic and institutional guidelines limiting in-
person research visits. VICTORI VR simulation implementation sessions will be audio and video 
recorded. VICTORI’s effect will be assessed using a repeated measures pre-post study design 
with follow-up period. We will collect monthly baseline data on HPV vaccine initiation and 
completion rates in PPCC for six months prior to implementing VICTORI. In phase I, VICTORI 
will be administered to resident and attending physicians. We will then collect HPV initiation 
and completion rates monthly during the time of intervention implementation and for the 
following six-months. To assess vaccination rates in clinic compared to HPV vaccination, we 
will also collect influenza, Tetanus, Diphtheria, and Pertussis (Tdap), and meningococcal 
conjugate (MCV4) vaccine for the six months prior and post VICTORI implementation in PPCC.  

Version:  7.0
Next, we will implement VICTORI with staff including nurses and medical assistants in groups 
of up to 15-20 participants (phase II; secondary outcome). These sessions may take place 
through a technology-based platform to allow for social distancing dependent on the COVID-19 
pandemic and institutional guidelines limiting group meetings and in-person research visits.  
Staff will watch a 5-minute video on evidence-based practices in recommending the HPV 
vaccine and observe a facilitator and clinician participating in the VICTORI VR simulations, and 
then engage in a 5-minute debriefing. Staff will complete a brief pre and immediate post survey 
assessing their perceived benefits and barriers, subjective norms, and positive messaging about 
the HPV vaccine on paper or via REDCap. We will collect an additional six months of monthly 
HPV vaccination initiation and completion data, along with influenza, Tdap, and MCV4. We 
will collect the same vaccination data from the HSNHC over the same study period to serve as a 
control. The primary outcome will be the increase in the proportion of patients receiving HPV 
vaccination seen by resident and attending physicians following participation in VICTORI. 
Additional secondary outcomes, including physicians’ behavioral capability, self-efficacy, 
perceived benefits, perceived barriers, perceived risk, attitudes, subjective norms, and strength of 
recommendations, as well as staffs’ perceived benefits and barriers, subjective norms, and 
positive messaging will be assessed via validated survey instruments created by Dr. Kahn and 
colleagues (see Appendix B and C).41,49,72-76 Physicians’ strength of recommendation will be 
self-reported, which is a common method of measuring this construct.30 In addition, this method 
(self-reported strength of recommendation) is supported as it is positively associated with 
vaccination rates documented through medical records.77 Physician and staff demographics will 
also be collected to assess for moderation or confounding. Confidentiality of the results will be 
maintained. All surveys will have unique IDs with one REDCap form, separate from the surveys 
including their email and ID. Surveys will also be administered through the REDCap Database. 
Participants will receive a total of 3 emails encouraging survey completion prior to and following 
completion of VICTORI. Participants will receive a newsletter periodically via email to enhance 
retention and provide study updates (appendix). Powerpoint slides will be presented in clinic to 
as a retention tool as well (appendix).
Data Analysis: Repeated measures generalized linear mixed-effects regression (GLMER) will be 
used to test whether vaccine initiation and completion rates of 9 to 17-year-old boys and girls 
increase in response to VICTORI. Models will include a logit link function to model the binary 
patient-level response of vaccine initiation (yes/no) and a random subject-specific intercept to 
account for patient encounters nested within physicians. Our primary outcome, determining the 
increase in the proportion of patients receiving HPV vaccination seen by resident and attending 
physicians in response to the intervention will be tested using the GLMER model including 
fixed-effect terms for study month (i.e., time) and period (pre-intervention vs. post-intervention). 
A likelihood ratio test (LRT) for period will provide a formal test of whether vaccine initiations 
increase in response to the intervention. Differences in the trajectory (i.e., change in slopes) of 
vaccine initiations between pre- and post-intervention periods will be tested by the inclusion of 
an interaction term for time and group. The inclusion of a random slope for time will be 
determined based on assessment of model fit. Polynomial terms or restricted cubic splines will 
be considered to model non-linear associations in vaccine initiation (i.e., capture the possible 
early intervention effect and subsequent decline). Additional terms as described for resident and 
attending physicians in phase I will be added to capture changes in the level and slope of vaccine 
initiation in response to training staff in phase II. Primary analyses will be conducted for 

Version:  7.0
physicians (resident and attending combined) enrolled in both the pre- and post-intervention 
periods. Secondary analyses will examine the intervention effect on completion rates, for 
resident and attending physicians separately, and for all physicians and staff combined. GLMER 
models as described above will also be used to assess whether background rates of vaccine 
initiation and completion change over the study period in the absence of the intervention at the 
control site (HSNHC). Additional secondary analyses will focus on identifying potential 
mediators of the intervention effect on PPCC providers; whether the effect of the intervention is 
mediated by changes in the strength of vaccine recommendations and relevant theory-based 
constructs. Stratified analyses and formal testing of interactions will also be used to assess 
whether the effect of the intervention differs (i.e. effect modification) according to baseline 
provider characteristics such as age, sex, and attending vs. resident status, as well as patient 
characteristics (female vs. male patient, age of patient). As discussed, providers at HSNHC will 
be receiving the self-directed app-based curriculum component of VICTORI though will not 
participate in VR simulations and thus will not engage in deliberate practice of curricular skills. 
HSNHC providers survey responses will be compared to PPCC providers to assess for relevant 
changes in theory-based constructs including physicians’ behavioral capability, self-efficacy, 
perceived benefits, barriers, and risk, as well as attitudes, subjective norms, and strength of 
recommendations that might occur following participation in the app-based curriculum only.  
Survey data will be collected before the app-based curriculum, immediately post-curriculum, and 
three months after the curriculum. This comparison will be conducted using a two-independent 
sample t-test. 
VR sessions will be reviewed as part of secondary analyses to qualitatively explore providers’ 
language around addressing particular sources of hesitancy for HPV vaccination. Language will 
be assessed through thematic analysis and summative content analysis. For example, we will 
assess provider’s responses to parental concerns regarding HPV vaccination prompting early 
sexual activity initiation. We will explore if race of the animated parent avatar influences 
providers’ word choice. We will explore if baseline provider characteristics such as age, sex, and 
attending vs. resident status, year of residency impact language. 
Transcripts from the VR simulations will be coded by a study team member to assess primary 
learning objectives. An artificial intelligence (AI) software from IBM, Watson, will be used to 
train the software through machine learning to process and code data to begin exploring the 
feasibility of adapting the VR simulation training to use AI instead of requiring a human 
facilitator. Watson is a password protected software (add URL). IBM Watson has been used in 
Medical Education research79. The coded portions of the transcripts will be used as examples for 
Watson to create “intents.” Intents are a collection of inputs, in our project they are the primary 
learning objectives, that Watson is taught to recognize. In this way, Watson recognizes that a 
participant intended to use a particular skill. A study team member will use the de-identified 
transcripts to train Watson, to recognize and code portions of transcripts from the VR 
simulations. Next, we will test Watson’s coding against members of the research team’s coded 
transcripts to test the inter-rater reliability between AI and study team members. The goal is to 
train Watson to accurately identify primary learning objectives as a first step in assessing the 
feasibility for an automated version of the VICTORI intervention. 
Power considerations: Power considerations are provided for our primary aim and outcome of 

Version:  7.0
determining whether VICTORI increases vaccine initiation. The PPCC is staffed by 80 residents 
and 20 attending physicians and reflects the maximum sample size for the intervention. To inform 
our power calculations for this proposal, we reviewed all PPCC patient visits for 11 to 17-year-old 
boys and girls from July 1, 2016 to June 30, 2017. HPV vaccination was initiated in 53% of visits 
for residents and 34% for attending physicians with greater between providervariation for 
residents. In addition, attending physicians saw more patients (n=9) in this age range on average 
per month than residents (n=2), and graphical analyses and formal regression modeling did not 
support material seasonal variation in initiation rates. Simulated outcomes based on these data 
were used to determine the minimum detectable odds ratio (OR) for vaccine initiation according 
to study period (i.e. pre- vs. post-intervention) as estimated from the proposed GLMER model. 
For resident and attending physicians combined, we assumed a total of 70 physicians having on 
average four patient encounters per month, a baseline probability of vaccine initiation of 43%, and 
a standard deviation for the subject-specific intercept of 0.55. Study power was determined to be 
80% to detect an OR=1.5 based on 10k simulations. This corresponds to a post-intervention 
vaccine initiation probability of 53%; or a 10% pre- to post-intervention difference. Thus, 
conducting this intervention at CCHMC is expected to provide sufficient power to detect a 
clinically meaningful effect size for the response.  
Potential Problems and Alternative Strategies: Insufficient recruitment may occur, but we do not 
anticipate this to be an issue as we have successfully conducted a VR intervention focused on flu 
vaccination with this population.26  In addition, our team and expert consultants have provided 
extensive recommendations to recruit providers. There is the possibility of incomplete surveys; 
scale items with <25% missing data will be included in the analysis.78 Multiple imputation and/or 
maximum likelihood estimation will be implemented should material missing values be realized 
for the survey responses and we will conduct a sensitivity analyses to determine the robustness of 
study findings to the assumptions of the imputation model. 
Aim 3: Determine modifiable barriers related to HPV vaccine non-completion for future 
enhancement of VICTORI accordingly
Setting and Population: The PPCC is a busy, academic clinic in Hamilton County, Ohio with about 
33,000 visits annually, serving 19,000 patients that are predominantly underserved with approximately 
90% insured through Medicaid. Seventy-five percent of the patient population is African American and 
5% is Latino. PPCC serves a population that is at high risk for HPV infection and its sequela, and 
therefore, we anticipate the findings to be generalizable to other urban primary care clinics that serve 
predominantly minority populations. Hopple Street Neighborhood Health Center (HSNHC) which 
served as a control during Aim 2, serves a similar population to the PPCC with 6000 unique patients 
and 13,000 patient visits per a year.
Design and Measures: We will first identify patients aged 9 to 17-year-old boys and girls who initiated the 
HPV vaccine series between 10/01/2020-06/01/2022 after seeing a provider who underwent VICTORI 
training. We will review social determinants of health screener of those who had not completed the HPV 
vaccine series by 14 months after initiation compared to those who did complete the HPV vaccine series. 
We will also collect demographic information (DOB, age, sex, race, ethnicity, health insurance; see 
appendix VICTORI Chart Review),
Based on these data, future revisions will be made to VICTORI to allow physicians to deliberately practice 
screening for and addressing barriers to vaccine series completion. Findings from this aim will provide 

Version:  7.0
information to create a data-informed enhancement of VICTORI’s simulated scenarios supporting series 
completion, which is critical to decreasing morbidity related to HPV-associated cancers. 
Data Analysis and Sample Size Calculations:

Review between n = 300 (minimum) – 720 (maximum) patient records

These sample size calculations were derived from Peduzzi et al.{Peduzzi, 1996 #3012} guideline for 
the minimum number of cases to include for a logistic regression. The formula used for the calculation 
of the minimum number of participants was N = 10 * 18 variables / 60% (the estimated proportion of 
patients who completed the HPV vaccine series). The formula used for the calculation of the 
maximum number of participants was N = 10 * 18 variables / 25% (the estimated proportion of 
patients who completed the HPV vaccine within the recommended timeframe). As a note, we will 
also examine the proportion of patients who meet the guidelines for timely series completion (N = 
400 minimum) using the following equation: N = 10 * 18 variables / 45% (the estimated proportion 
of people who completed the HPV vaccine within the recommended timeframe.

Descriptive analyses as well as data variance and distribution to characterize variables.

Logistic regression to determine factors independently associated with completing the HPV vaccine 
series including: clinic location (PPC v Hopple), type of visits at first dose, resident v attending 
physician, provider sex, age of patient at first dose, patient sex, patient race, insurance at first dose, 
and social risk screener questions.
Required dose for vaccine series completion (2 doses v 3 doses) will also be included in the 
descriptive analyses
Training of Study Personnel
All staff will undergo training in maintaining confidentiality and have completed the required 
Collaborative Institutional Training Initiative (CITI) courses and research ethics classes as 
required by CCHMC and Cincinnati Children’s Research Foundation. 
Recruitment and Consent Procedure
Aim 1: Evaluate VICTORI for acceptability and feasibility
Before recruiting any participants, approval from the CCHMC IRB will be obtained. Physicians 
working in the CCHMC THC will be sent invitations to participate in the study via email. The email 
will explain the purpose of the study, what participation entails, data to be collected, reimbursement, 
and contact information for study personnel. Participants will also be recruited in person. Staff 
working in the CCHMC THC will be recruited in person with an informational flyer explaining the 
purpose of the study, what participation entails, data to be collected, reimbursement, and contact 
information for study personnel. THC staff will be consented as a group prior to engaging in the 
study. Written consent, as appropriate, will be obtained from all participants for their participation in 
this phase of the study. 
PPCC staff will be consented as a group prior to engaging in VICTORI. Written consent will be 
obtained from each participant in accordance to CCHMC policy 41-1.4. 
The physicians will be consented using REDCap e-consent. REDCap is a secure web application for 
building and managing online surveys and databases.  While REDCap can be used to collect 
virtually any type of data (including 21 CFR Part 22, FISMA, and HIPAA-compliant 
environments), it is specifically geared to support online or offline data capture for research studies 

Version:  7.0
and operations.  The CCHMC REDCap team has templates available for research use for 
eConsenting which includes Consent to be a Research Subject and Authorization to Use or Disclose 
(Release) Health Information that Identifies You for a Research Study (HIPAA).  The REDCap 
electronic consent format does not accommodate the current CCHMC formatting which includes 
headers with logos and stamps on each page and will therefore require some modifications. 
Participants will be sent a link to review and complete the e-consent on their own, however, study 
team contact information will be available for questions. CRC will verify e-consent completion, and 
sign that consent has been obtained within one business day of participant completion. Since the 
participants will complete the consent on their own, an informed consent process note will not be 
completed for physician participants.
Copies of consent forms for staff and physicians will not be submitted to Health Information 
Management, since these participants are not patients at CCHMC.
Aim 2: Conduct a single-site intervention assessing the efficacy of VICTORI in increasing 
HPV vaccine initiation and completion rates
Before recruiting any participants, approval from the CCHMC IRB will be obtained. Physicians and 
staff working in the CCHMC PPCC and attending physicians working in the HSNHC will be sent 
invitations to participate in the study via email three times over a four-week period. Participants will 
also be recruited in person. Physicians and staff will be recruited as they were in Aim 1.
Physicians at PPCC and HSNHC will be consented using e-consent. A waiver of documentation of 
informed consent is requested since no private identifiable health information will be collected and 
the purpose of the study is to evaluate the effectiveness of VICTORI to increase HPV vaccination 
rates in clinic. This also aligns with institutional guidelines for remote consenting due to social 
distancing dependent on the COVID-19 pandemic.  A study information sheet will be provided 
using the REDCap e-consent module. REDCap is a secure web application for building and 
managing online surveys and databases.  While REDCap can be used to collect virtually any type of 
data (including 21 CFR Part 22, FISMA, and HIPAA-compliant environments), it is specifically 
geared to support online or offline data capture for research studies and operations.  The CCHMC 
REDCap team has templates available for research use for eConsenting which includes Consent to 
be a Research Subject and Authorization to Use or Disclose (Release) Health Information that 
Identifies You for a Research Study (HIPAA).  The REDCap electronic consent format does not 
accommodate the current CCHMC formatting which includes headers with logos and stamps on 
each page and will therefore require some modifications. Participants will be sent a link to review 
and complete the e-consent on their own, however, study team contact information will be available 
for questions. Participants will only be sent the surveys once the participant has electronically 
agreed to participant in the study, by checking “Yes, I consent to participate in the study”.. 
PPCC staff will be consented as a group prior to engaging in VICTORI. A waiver of documentation 
of informed consent is also requested for PPCC staff as no private identifiable health information 
will be collected and the purpose of the study is to evaluate the effectiveness of VICTORI to 
increase HPV vaccination rates in clinic. A study information sheet will be provided and reviewed 
with participants prior to the session if completed in person. If session occurs remotely, a study 
information sheet will be implemented using the REDCap e-consent module following procedures 
described for the physician participants. 

Version:  7.0
Copies of consent forms for staff and physicians will not be submitted to Health Information 
Management, since these participants are not patients at CCHMC.
Aim 3: Determine modifiable barriers related to HPV vaccine non-completion for future 
enhancement of VICTORI accordingly.
Before initiating any study procedures, approval from the CCHMC IRB will be obtained. A waiver 
of informed consent, parental permission, and assent are requested for Aim 3, as data will be limited 
to chart review in Epic. This portion of the study can be considered exempt from informed consent 
procedures as it does not involve procedures for which written consent is normally required and is 
limited to collection of data previously collected.
Additionally, a waiver of HIPAA Privacy rule is requested of research participants’ and their 
parents and guardians’ authorization for use of protected health information (PHI) for research 
purposes.  This is requested for the collection of retrospective data as the use of the PHI involves 
no more than a minimal risk to the privacy of individuals, participants will be assigned a Study 
ID in the REDCap database instead of using their name, identifying information will not be used 
in analyses, identifying information will be destroyed at study closeout, and only study team 
members will have access to the information linking the Study ID to the participant. All 
information will be stored in a secure study electronic folder and in REDCap. The research could 
not be conducted without this waiver nor could it be conducted without access to and use of the 
PHI. 
Compensation
Aim 1
Physicians and staff will receive $50 for their efforts. Staff will also be provided lunch.
Aim 2
Physicians from PPCC will receive $50 total for their efforts. PPCC physicians will receive $40 
after completing the pre-survey, the self-directed app, the VICTORI session and the immediate-post 
survey, and $10 after the 3-month post survey. Staff  will receive a $10 gift card.  Physicians from 
HSNHC will receive $25 total for their efforts. HSNHC physicians will receive $15 after 
completing the pre-survey, the self-directed app, and immediate post-survey and $10 after the 3 
month post survey. 
The electronic W-9 and ClinCard receipt templates developed by the REDCap team will be used for 
physician participants since not all participants will be seen in-person. They will be used for PPCC 
staff as well if sessions are conducted remotely. These forms are separate from study data forms in 
REDCap. ClinCards for physicians and gift cards for staff will be delivered to the participants’ 
mailboxes or work locations within one week of participation. 
Aim 3
Not applicable
Future Use

Version:  7.0
This project will be the first to provide data on the effect of deliberate practice using VICTORI 
to increase HPV vaccine rates. These data will provide the foundation for an R01 application that 
will evaluate the intervention’s effectiveness in a multisite trial. Data will also support the 
enhancement of VICTORI to address modifiable barriers associated with vaccine series 
completion. Furthermore, if effective, we plan to incorporate artificial intelligence into the VR 
design to allow automated functioning without human facilitation to promote future scalability. 
This work will positively impact public health by determining the effect of deliberate practice 
principles to train physicians to strongly recommend HPV vaccines, and thus, increasing HPV 
vaccination rates and decreasing HPV-associated cancers.
Publication
The authors plan to publish and disseminate the curriculum and its evaluation at the conclusion 
of the study in the appropriate medical education journals, on-line medical education share sites, 
and national meetings. 

Version:  7.0
References
1. Walker TY, Elam-Evan LD, Yankey D, et al. National, regional, state, and selected local area 
vaccination coverage among adolescents aged 13-17 years - United States, 2017. Morb Mortal 
Wkly Rep. 2018;67:909-917. 
2. US Department of Health and Human Services, Healthy People 2020. Immunization and 
infectious diseases. https://www.healthypeople.gov/2020/topics-objectives/topic/immunization-
and-infectious-diseases. 
3. Dorell C, Yankey D, Jeyarajah J, et al. Delay and refusal of human papillomavirus vaccine for 
girls, National Immunization Survey-Teen, 2010. Clin Pediatr. 2014;53(3):261-269. doi: 
10.1177/0009922813520070. 
4. Rosenthal SL, Weiss TW, Zimet GD, Ma L, Good MB, Vichnin MD. Predictors of HPV 
vaccine uptake among women aged 19–26: Importance of a physician's recommendation. 
Vaccine. 2011;29(5):890-895. doi: 10.1016/j.vaccine.2009.12.063. 
5. Ylitalo KR, Lee H, Mehta NK. Health care provider recommendation, human papillomavirus 
vaccination, and race/ethnicity in the US National Immunization Survey. Am J Public Health. 
2013;103(1):164-169. doi: 10.2105/AJPH.2011.300600
6. Lau M, Lin H, Flores G. Factors associated with human papillomavirus vaccine-series 
initiation and healthcare provider recommendation in US adolescent females: 2007 National 
Survey of Children's Health. Vaccine. 2012;30(20):3112-3118. doi: 
10.1016/j.vaccine.2012.02.034  
7. Kester LM, Zimet GD, Fortenberry JD, Kahn JA, Shew ML. A national study of HPV 
vaccination of adolescent girls: Rates, predictors, and reasons for non-vaccination. Matern Child 
Health J. 2013;17(5):879-885. doi: 10.1997/s10995-012-1066-z 
8. Brewer NT, Gottlieb SL, Reiter PL, et al. Longitudinal predictors of human papillomavirus 
vaccine initiation among adolescent girls in a high-risk geographic area. Sex Transm Dis. 
2011;38(3):197-204. doi: 10.1097/OLQ.0b013e3181f12dbf 
9. Stokley S, Jeyarajah J, Yankey D, et al. Human papillomavirus vaccination coverage among 
adolescents, 2007-2013, and postlicensure vaccine safety monitoring, 2006-2014 - United States. 
Morb Mortal Wkly Rep. 2014;63(29):620-624. 
10. Hughes CC, Jones AL, Feemster KA, Fiks AG. HPV vaccine decision making in pediatric 
primary care: A semi-structured interview study. BMC Pediatr. 2011;11:74. doi: 10.1186/1471-
2431-11-74 
11. Hamlish T, Clarke L, Alexander KA. Barriers to HPV immunization for African American 
adolescent females. Vaccine. 2012;30(45):6472-6476. doi: 10.1016/j.vaccine.2012.07.085 

Version:  7.0
12. McRee A, Gilkey MB, Dempsey AF. HPV vaccine hesitancy: Findings from a statewide 
survey of health care providers. J Pediatr Health Care. 2014;28(6):541-549. doi: 
10.1016/j.pedhc.2014.05.003. 
13. Brewer NT, Hall ME, Malo TL, Gilkey MB, Quinn B, Lathren C. Announcements versus 
conversations to improve HPV vaccination coverage: A randomized trial. Pediatrics. 
2017;139(1). doi: 10.1542/peds.2016-1764 
14. Staras SA, Vadaparampil ST, Livingston MD, Thompson LA, Sanders AH, Shenkman EA. 
Increasing human papillomavirus vaccine initiation among publicly insured Florida adolescents. 
J Adolesc Health. 2015;56(5):S40-S46. doi: 10.1016/j.jadohealth.2014.11.024 
15. Perkins RB, Zisblatt L, Legler A, Trucks E, Hanchate A, Gorin SS. Effectiveness of a 
provider-focused intervention to improve HPV vaccination rates in boys and girls. Vaccine. 
2015;33(9):1223-1229. doi: 10.1016/j.vaccine.2014.11.021 
16. Paskett ED, Krok-Schoen JL, Pennell, ML, et al. Results of a multilevel intervention trial to 
increase human papillomavirus (HPV) vaccine uptake among adolescent girls. Cancer Epidemiol 
Biomarkers Prev. 2016;25(4):593-602. doi: 10.1158/1055-9965.EPI-15-1243. 
17. Szilagyi PG, Albertin C, Humiston SG, et al. A randomized trial of the effect of centralized 
reminder/recall on immunizations and preventive care visits for adolescents. Acad Pediatr. 
2013;13(3):204-213. doi: 10.1016/j.acap.2013.01.002 
18. Dempsey AF, Pyrznawoski J, Lockhart S, et al. Effect of a health care professional 
communication training intervention on adolescent human papillomavirus vaccination: A cluster 
randomized trial. JAMA Pediatr. 2018;172(5):e180016. doi: 10.1001/jamapediatrics.2018.0016 
19. Zimet G, Dixon BE, Xiao S, et al. Simple and elaborated clinician reminder prompts for 
human papillomavirus vaccination: A randomized clinical trial. Acad Pediatr. 2018;18(2):S66-
S71. doi: 10.1016/j.acap.2017.11.002 
20. Gilkey MB, McRee AL. Provider communication about HPV vaccination: A systematic 
review. Hum Vaccin Immunother. 2016;12(6):1454-1468. doi: 10.1080/21645515.2015.1129090 
21. Cook DA, Hatala R, Brydges R, et al. Technology-enhanced simulation for health 
professions education: A systematic review and meta-analysis. JAMA. 2011;306(9):978-988. doi: 
10.1001/jama.2011.1234 
22. McGaghie WC, Issenberg SB, Cohen ER, Barsuk JH, Wayne DB. Does simulation-based 
medical education with deliberate practice yield better results than traditional clinical education? 
A meta-analytic comparative review of the evidence. Acad Med. 2011;86(6):706-711. doi: 
10.1097/ACM.0b013e318217e119 

Version:  7.0
23. Zendejas B, Brydges R, Wang AT, Cook DA. Patient outcomes in simulation-based medical 
education: A systematic review. J Gen Intern Med. 2013;28(8):1078-1089. doi: 10.1007/s11606-
012-2264-5 
24. Ericsson KA, Krampe RT, Tesch-Römer C. The role of deliberate practice in the acquisition 
of expert performance. Psychol Rev. 1993;100(3):363-406. 
25. Real FJ, DeBlasio D, Ollberding NJ, et al. Resident perspectives on communication training 
that utilizes immersive virtual reality. Educ Health. 2017;30(3):228-231. doi: 
10.4103/efh.EfH_9_17 
26. Real FJ, DeBlasio D, Beck AF, et al. A virtual reality curriculum for pediatric residents 
decreases rates of influenza vaccine refusal. Acad Pediatr. 2017;17(4):431-435. doi: 
10.1016/j.acap.2017.01.010
27. Vorderer P, Wirth W, Gouveia FR, et al. MEC spatial presence questionnaire (MECSPQ): 
Short documentation and instructions for application. Report to the European Community, 
Project Presence: MEC (IST-2001-37661). 2004.
28. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data 
capture (REDCap)—A metadata-driven methodology and workflow process for providing 
translational research informatics support. J Biomed Inform. 2009;42(2):377-381. 
29. Aragones A, Bruno D, Gany F. Attitudes surrounding implementation of the HPV vaccine 
for males among primary care providers serving large minority populations. J Health Care Poor 
Underserved. 2013;24(2):768-776. doi: 10.1353/hpu.2013.0088.
30. Rosen BL, Shepard AL, Kahn JA. U.S. healthcare clinicians’ knowledge, attitudes and 
practices regarding HPV vaccination: A qualitative systematic review. Acad Pediatr. 
2018;18(2S):S53-S65. doi: 10.1016/j.acap.2017.10.007.
31. Rosen BL, Ashwood D, Richardson G. School nurses' professional practice in the HPV 
vaccine decision making process. J Sch Nurs. 2016;32(2):138-148. doi: 
10.1177/1059840515583312 
32. Rosen BL, Goodson P, Thompson B, Wilson KL. School nurses' knowledge, attitudes, 
perceptions, and professional practice regarding HPV vaccine for youth. J Sch Health. 
2015;85(2):73-81. doi: 10.1111/josh.12229
33. McCave EL. Influential factors in HPV vaccination uptake among providers in four states. J 
Community Health. 2010;35(6):645-652. doi: 10.1007/s10900-010-9255-4. 
34. Young JL, Bernheim RG, Korte JE, Stoler MH, Guterbock TM, Rice LW. Human 
papillomavirus vaccination recommendation may be linked to reimbursement: A survey of 
Virginia family practitioners and gynecologists. J Pediatr Adolesc Gynecol. 2011;24(6):380-385. 
doi: 10.1016/j.jpag/2011.06.016 

Version:  7.0
 
35. Riedesel JM, Rosenthal SL, Zimet GD, et al. Attitudes about human papillomavirus vaccine 
among family physicians. J Pediatr Adolesc Gynecol. 2005;18(6):391-398.  
36. Kahn JA, Zimet GD, Bernstein DI, et al. Pediatricians' intention to administer human 
papillomavirus vaccine: The role of practice characteristics, knowledge, and attitudes. J Adolesc 
Health. 2005;37(6):502-510. 
37. Tissot AM, Zimet GD, Rosenthal SL, Bernstein DI, Wetzel C, Kahn JA. Effective strategies 
for HPV vaccine delivery: The views of pediatricians. J Adolesc Health. 2007;41(2):119-125. 
38. Kahn JA, Rosenthal SL, Tissot AM, Bernstein DI, Wetzel C, Zimet GD. Factors influencing 
pediatricians' intention to recommend human papillomavirus vaccines. Ambul Pediatr. 
2007;7(5):367-373. 
39. Kahn JA, Rosenthal SL, Jin Y, Huang B, Namakydoust A, Zimet GD. Rates of human 
papillomavirus vaccination, attitudes about vaccination, and human papillomavirus prevalence in 
young women. Obstet Gynecol. 2008;111(5):1103-1110. doi: 10.1097/AOG.0b013e31817051fa 
40. Kahn JA, Ding L, Huang B, Zimet GD, Rosenthal SL, Frazier AL. Mothers' intention for 
their daughters and themselves to receive the human papillomavirus vaccine: A national study of 
nurses. Pediatrics. 2009;123(6):1439-1445. doi: 10.1542/peds.2008-1536 
41. Kahn JA, Cooper HP, Vadaparampil ST, et al. Human papillomavirus vaccine 
recommendations and agreement with mandated human papillomavirus vaccination for 11-to-12-
year-old girls: A statewide survey of Texas physicians. Cancer Epidemiol Biomarkers Prev. 
2009;18(8):2325-2332. doi: 10.1158/1055-9965.EPI-09-0184
42. Feemster KA, Winters SE, Fiks AG, Kinsman S, Kahn JA. Pediatricians' intention to 
recommend human papillomavirus (HPV) vaccines to 11- to 12-year-old girls postlicensing. J 
Adolesc Health. 2008;43(4):408-411. doi: 10.1016/j.jadohealth.2008.06.012  
43. Conroy K, Rosenthal SL, Zimet GD, et al. Human papillomavirus vaccine uptake, predictors 
of vaccination, and self-reported barriers to vaccination. J Womens Health. 2009;18(10):1679-
1686. doi: 10.1089/jwh.2008.1329. 
44. Vadaparampil ST, Kahn JA, Salmon D, et al. Missed clinical opportunities: Provider 
recommendations for HPV vaccination for 11-12 year old girls are limited. Vaccine. 
2011;29(47):8634-8641. doi: 10.1016/j.vaccine.2011.09.006
45. Griffioen AM, Glynn S, Mullins TK, et al. Perspectives on decision making about human 
papillomavirus vaccination among 11- to 12-year-old girls and their mothers. Clin Pediatr. 
2012;51(6):560-568. doi: 10.1177/0009922812443732 

Version:  7.0
46. Mullins TL, Griffioen AM, Glynn S, et al. Human papillomavirus vaccine communication: 
Perspective of 11-12 year-old girls, mothers, and clinicians. Vaccine. 2013;31(42):4894-4901. 
doi: 10.1016/j.vaccine.2013.07.033 
47. Malo TL, Giuliano AR, Kahn JA, et al. Physicians' human papillomavirus vaccine 
recommendations in the context of permissive guidelines for male patients: A national study. 
Cancer Epidemiol Biomarkers Prev. 2014;23(10):2126-2135. doi: 10.1158/1055-9965.EPI-14-
0344. 
48. Feemster KA, Middleton M, Fiks AG, Winters S, Kinsman SB, Kahn JA. Does intention to 
recommend HPV vaccines impact HPV vaccination rates? Hum Vaccine Immunother. 
2014;10(9):2519-2526. doi: 10.4161/21645515.2014.969613. 
49. Vadaparampil ST, Malo TL, Sutton SK, et al. Missing the target for routine human 
papillomavirus vaccination: Consistent and strong physician recommendations are lacking for 
11- to 12-year-old males. Cancer Epidemiol Biomarkers Prev. 2016;25(10):1435-1446. 
50. Opel DJ, Mangione-Smith R, Robinson JD, et al. The influence of provider communication 
behaviors on parental vaccine acceptance and visit experience. Am J Public Health. 
2015;105(10):1998-2004. doi: 10.2105/AJPH.2014.302425 
51. Moss JL, Reiter PL, Rimer BK, Brewer NT. Collaborative patient-provider communication 
and uptake of adolescent vaccines. Soc Sci Med. 2016;159:100-107. 
52. Opel DJ, Heritage J, Taylor JA, et al. The architecture of provider-parent vaccine discussions 
at health supervision visits. Pediatrics. 2013;132(6):1037-1046. doi: 10.1542/peds.2013-2037 
53. Donahue KL, Hendrix KS, Sturm LA, Zimet, GD. The effect of provider communication 
about vaccination on mothers’ willingness to vaccinate their children against HPV and influenza: 
A randomized trial of illustrated health messaging vignettes. J Adolesc Health. 2016;58(2):S117-
S117. doi: 10.1016/j.jadohealth.2015.10.244 
54. Centers for Disease Control and Prevention. Human papillomavirus (HPV). 
https://www.cdc.gov/hpv/hcp/answering-questions.html. 
55. Freed GL, Clark SJ, Butchart AT, Singer DC, Davis MM. Parental vaccine safety concerns in 
2009. Pediatrics. 2010;125(4):654-659. doi: 10.1542/peds.2009-1962 
56. Centers for Disease Control and Prevention. Human papillomavirus vaccination coverage 
among adolescent girls, 2007-2012, and postlicensure vaccine safety monitoring, 2006-2013 - 
United States. Morb Mortal Wkly Rep. 2013;62(29):591-595. 
57. Holman DM, Benard V, Roland KB, Watson M, Liddon N, Stokley S. Barriers to human 
papillomavirus vaccination among US adolescents: A systematic review of the literature. JAMA 
Pediatr. 2014;168(1):76-82. doi: 10.1001/jamapediatrics.2013.2752 

Version:  7.0
58. Reiter PL, Gilkey MB, Brewer NT. HPV vaccination among adolescent males: Results from 
the National Immunization Survey-Teen. Vaccine. 2013;31(26):2816-2821. doi: 
10.1016/j.vaccine.2013.04.010 
59. Helgeson V, Cohen S, Schulz R, Yasko J. Education and peer discussion group interventions 
and adjustment to breast cancer. Arch Gen Psychiatry. 1999;56(4):340-347. doi: 
10.1001/archpsyc.56.4.340 
60. Rickheim PL, Kendall DM, Flader JL, Weaver TW. Assessment of group versus individual 
diabetes education: A randomized study. Diabetes Care. 2002;25(2):269-274. 
61. Fontenot HB, Kornides ML, McRee AL, Gilkey MB. Importance of a team recommendation 
approach on HPV vaccine uptake. J Adolesc Health. 2018;62:S24. doi: 
10.1016/j.jadohealth.2017.11.048
62. National HPV Vaccination Roundtable. Cancer prevention through HPV vaccination in your 
practice: An action guide for nurses and medical assistances. http://hpvroundtable.org/wp-
content/uploads/2018/04/RNMA-Action-Guide-WEB.pdf.
63. Slegg J. List of questions to ask website usability testers. Jennifer Slegg Service Engine 
Marketing Consultant Website. http://www.jenniferslegg.com/2007/09/24/list-of-questions-to-
ask-website-usability-testers/. 
64. Lanoue S. Writing your user test plan: Open-ended vs. specific tasks and questions. User 
Testing Blog Website. https://www.usertesting.com/blog/2015/05/18/open-ended-vs-specific-
tasks-and-questions/.
65. Web Accessibility Initiative. WAI site usability testing questions. 
http://www.w3.org/WAI/EO/Drafts/UCD/questions.html. 
66. Mosadeghi S, Reid MW, Martinez B, Rosen BT, Spiegel BM. Feasibility of an immersive 
virtual reality intervention for hospitalized patients: An observational cohort study. JMIR Ment 
Health. 2016;3(2):e28. doi: 10.2196/mental.5801 
67. Ritchie J, Spencer L. Qualitative data analysis for applied policy research. In: Bryman A, 
Brugess R, eds. Analyzing qualitative data. Routledge; 1994:731-194. 
68. Hamilton County. Community health needs assessment (CHNA) report. 
https://idph.iowa.gov/Portals/1/userfiles/91/CHNA%26HIP/2016%20CHNA%26HIPs/Hamilton
%20County%20CHNA%202-18-16.pdf. 
69. Livingston SA, Zieky MJ. Passing scores: A manual for setting standards of performance on 
educational and occupational tests. Princeton, NJ: Educational Testing Service; 1982. 

Version:  7.0
70. Wayne DB, Barsuk JH, O'Leary KJ, Fudala MJ, McGaghie WC. Mastery learning of 
thoracentesis skills by internal medicine residents using simulation technology and deliberate 
practice. J Hosp Med. 2008;3(1):48-54. doi: 10.1002/jhm.268 
71. Wayne DB, Butter J, Siddall VJ, et al. Mastery learning of advanced cardiac life support 
skills by internal medicine residents using simulation technology and deliberate practice. J Gen 
Intern Med. 2006;21(3):251-256. 
72. Kasting ML, Christy S, Sutton S, et al. Association between AFIX activities and HPV 
vaccination for Florida physicians: Analysis of a representative survey sample. Under review. 
73. Kasting ML, Lake P, Vadaparampil ST. Physicians’ current use and preferences for male 
HPV vaccine-related patient education materials. Vaccine. 2017;35(20):2613-2616. doi: 
10.1016/j.vaccine.2017.03.098 
74. Wheldon CW, Sutton SK, Fontenot HB, Quinn GP, Giuliano AR, Vadaparampil ST. 
Physician communication practices as a barrier to risk-based HPV vaccine uptake among men 
who have sex with men. J Cancer Educ. 2017. doi: 10.1007/s13187-017-1223-6 
75. Malo TL, Ali KN, Sutton SK, Perkins RB, Giuliano AR, Vadaparampil ST. The content and 
context of physicians' communication with males about human papillomavirus vaccination. Hum 
Vaccin Immunother. 2016;12(6):1511-1518. doi: 10.1080/21645515.2015.1132963 
76. Scherr CL, Augusto B, Ali K, Malo TL, Vadaparampil ST. Provider-reported acceptance and 
use of the Centers for Disease Control and Prevention messages and materials to support HPV 
vaccine recommendation for adolescent males. Vaccine. 2016;34(35):4229-4234. doi: 
10.1016/j.vaccine.2016.06.037 
77. Finney Rutten LJ, St. Sauver JL, Beebe TJ, et al. Association of both consistency and 
strength of self-reported clinician recommendation for HPV vaccination and HPV vaccine 
uptake among 11- to 12-year-old children. Vaccine. 2017;35(45):6122-6128. doi: 
10/1016/j.vaccine.2017.09.056 
78. Aday LA, Cornelius LJ. Designing and conducting health surveys: A comprehensive guide. 
3rd ed. San Francisco, CA: Jossey-Bass; 2006. 
79. Shah A SB, Chon S, Zackoff M, Cruse B, Glisson M, Schumacher D, Geis G. Using Virtual 
Reality Just-in-Time/Just-in-Place (JITP) Simulation to improve accurate senior resident 
identification of shock during the rapid cardiopulmonary assessment. Poster Presentation at 
Pediatric Academic Societies Annual Meeting, Philadelphia, PA. 2020 2020.