A Multi-modal, Physician-centered Intervention to Improve Guideline-concordant Prostate Cancer Imaging
Phase: NA
Conditions: Prostate Cancer
Sponsor: VA Office of Research and Development

 
 
 
 
 
 
 
 
 
A multi-modal, physician-centered intervention to improve guideline-concordant 
prostate cancer imaging 
Funding Agency: VA HSR&D 
Principal Investigator/Study Chair: Danil Makarov, MD, MHS 
Version 13, 02/14/2022 
 
 
 
 

Version 1  
VA Central IRB Protocol Template – version 10/26/2012 
Page 2 of 33 
 
Abstract 
 
Almost half of Veterans with localized prostate cancer (the most common non-cutaneous 
malignancy among US men) undergo inappropriate, wasteful imaging. Before widespread PSA 
screening, most incident cases were advanced, requiring radiographic staging before treatment. 
In recent years, most new prostate cancer cases have been clinically localized; there is near 
unanimous agreement that routine radiographic staging is obsolete. While there have been 
efforts to curb this practice, imaging use to stage low-risk prostate cancer remains high. Past 
studies by our research team have shown that physicians, not patients, are the drivers of 
inappropriate imaging. The objective of this study is to develop and implement an intervention to 
improve guideline-concordance across all populations – to decrease inappropriate imaging and 
increase appropriate imaging in prostate cancer patients. Building off previous research, this 
study will evaluate three multi-level interventions implemented in 10 VHAs across the US. A 
Clinical Order Check in CPRS will be used when a physician attempts to order imaging on a 
patient with the following characteristics: male, diagnosis of prostate cancer or prostate biopsy 
performed within six months of the current date, serum PSA<20ng/mL (those with higher PSAs 
all require imaging). The order check will not appear for imaging ordered for men with CPT 
codes associated with prostate cancer treatment in the past year as the intervention is tailored 
to address incident prostate cancer cases. Physicians will either be able to reverse their imaging 
order or opt out and explain the reason for doing so. Audit and feedback will be used to provide 
individual imaging feedback on prostate cancer imaging performance to physicians. This will be 
collected from Oncotrax and validated in CAPRI. Feedback will be given quarterly and will 
include recommendations by the PI. Finally, academic detailing will take place at each institution 
at the initiation of the intervention and every three months thereafter. In the event that the study 
team is unable to physically travel to the institution a Zoom meeting will be conducted covering 
the detailing agenda. The outcomes of interest are facility-level utilization of inappropriate and 
appropriate imaging, physician-level utilization of inappropriate and appropriate imaging, 
physician attitudes regarding prostate cancer imaging and the intervention, as well as the net 
cost of implementing a physician behavioral intervention. 
 

Version 1  
VA Central IRB Protocol Template – version 10/26/2012 
Page 3 of 33 
 
List of Abbreviations 
CAPRI – Compensation and Pension Record Interchange 
CDW – Corporate Data Warehouse 
CFIR – Consolidated Framework for Implementation 
CPRS – Computerized Patient Record System 
FIML – Full information maximum likelihood  
HERC – Health Economics Resource Center 
PHI – Protected health information 
TDF – Theoretical Domains Framework 
VACCR – Veterans Affairs Central Cancer Registry 
VANYHHS – VA New York Harbor Healthcare System 
VHA – Veterans Health Administration 
VINCI – VA Informatics and Computing Infrastructure  
 
 

Version 1  
VA Central IRB Protocol Template – version 10/26/2012 
Page 4 of 33 
 
Contents 
Protocol Title: ............................................................................................................................. 5 
1.0 
Study Personnel .............................................................................................................. 5 
2.0 
Introduction ..................................................................................................................... 6 
3.0 
Objectives ....................................................................................................................... 7 
4.0 
Resources and Personnel ............................................................................................... 7 
5.0 
Study Procedures ...........................................................................................................10 
5.1      Study Design ..................................................................................................................10 
5.2      Recruitment Methods .....................................................................................................14 
5.3  
Informed Consent Procedures ........................................................................................17 
5.4 
Inclusion/Exclusion Criteria .............................................................................................19 
5.5  
Study Evaluations ...........................................................................................................20 
5.6  
Data Analysis ..................................................................................................................21 
5.7  
Withdrawal of Subjects ...................................................................................................26 
6.0 
Reporting ........................................................................................................................26 
7.0 
Privacy and Confidentiality .............................................................................................27 
8.0 
Communication Plan ......................................................................................................29 
9.0 
References .....................................................................................................................30 
 

Version 1  
VA Central IRB Protocol Template – version 10/26/2012 
Page 5 of 33 
 
Protocol Title:  A multi-modal, physician-centered intervention to improve guideline-concordant 
prostate cancer imaging 
1.0 
Study Personnel 
 
Principal Investigator/Study Chair: Danil V. Makarov, MD, MHS 
 
Co-Investigators: 
o Scott Sherman, MD, MPH – VA New York Harbor 
o Craig Tenner, MD – VA New York Harbor 
o Steve Zeliadt, PhD – VA Puget Sound 
o Heather Gold, PhD – NYU School of Medicine 
o Michele Shedlin, PhD – NYU Rory Meyers College of Nursing 
 
Advisor: 
o Scott Braithwaite, MD, MSc – NYU School of Medicine 
 
Collaborators: There will be a site PI at each of our 10 participating VA 
Hospitals. They all are members of the Urology Department at their respective 
institution. 
o John Leppert, MD, MS – VA Palo Alto Healthcare System 
o Jeremy Shelton, MD, MSHS – VA Greater Los Angeles Healthcare 
System 
o Christopher Tessier, MD – VA Portland Healthcare System 
o Atreya Dash, MD VA Puget Sound Health Care System 
o Michael Risk, MD PhD – Minneapolis VA Health Care System 
o  Sara Best, MD– William S. Middleton Memorial Veterans Hospital 
o Will Lowrance, MD – VA Salt Lake City Health Care System 
o Andrew Liman, MD– VA Pittsburgh Healthcare System 
o Stephen Blakely, MD Syracuse VA Medical Center 
o Michael Leapman, MD – VA Connecticut Healthcare System 
 

Version 1  
VA Central IRB Protocol Template – version 10/26/2012 
Page 6 of 33 
 
2.0 
Introduction 
 
Almost half of Veterans with localized prostate cancer receive inappropriate, wasteful 
imaging. The VHA Blueprint for Excellence prioritizes increasing operational 
effectiveness. Prior studies seeking to limit inappropriate imaging did not assess barriers 
and achieved mixed results. Our team has explored the causes of guideline-discordant 
prostate cancer imaging and found that 1) patients with newly diagnosed prostate cancer 
have little concern for radiographic staging but rather focus on treatment, 2) physicians 
trust imaging guidelines but are apt to follow their own intuition, fear medico-legal 
consequences, and succumb to influence from colleagues who image frequently.1 In 
spite of such discrepant views, most VHA physicians suggested or supported a large-
scale effort to improve imaging use across VHA. 
We propose a multi-site, stepped wedge, cluster-randomized trial to determine the effect 
of a physician-focused behavioral intervention on VHA prostate cancer imaging use, the 
Prostate Cancer Imaging Stewardship (PCIS) intervention. The multi-level intervention, 
developed according to the Theoretical Domains Framework (TDF), combines traditional 
physician behavior change methods with novel methods of communication and data 
collection. The intervention consists of three components: 1) a system of audit and 
feedback to clinicians informing individual clinicians and their sites about how their 
behavior compares to their peers’ and to published guidelines 2) a program of academic 
detailing with the goal to educate providers about prostate cancer imaging, and 3) a 
CPRS Clinical Order Check for potentially inappropriate imaging. The intervention will be 
introduced to 10 participating geographically-distributed study sites. 
 
We will assess imaging rates for all months from the start of data collection (March 1, 
2018) prior to the intervention go live-date and 3 months following the intervention. The 
study’s specific aims seek to understand the effects of the intervention on 1) facility-level 
prostate cancer imaging rates, 2) physician experience with and perceptions of the 
intervention and its implementation, and 3) the costs of both implementing the 
intervention and affecting change in imaging use. 
 
This project seeks to describe and analyze the implementation of a behavioral 
intervention to improve prostate cancer care. This theory-based intervention builds on 
prior work identifying barriers to guideline-concordant prostate cancer imaging in VHA 
(CDA 11-257) and addresses these at three levels: individual (audit and feedback with 
VHA Cancer Care Cube data), facility (academic detailing) and system (CPRS Order 
Check). The team will assess the intervention’s cost impact and providers’ experiences 
in preparation for a subsequent large-scale VHA implementation project optimizing the 
operational effectiveness of prostate cancer imaging across VHA. The current 
application is an opportunity to leverage VHA’s state-of-the-art, integrated healthcare 
delivery system to implement a carefully designed, theory-based behavioral intervention 
to reduce harmful, inappropriate care, increase appropriate care to those who truly need 

Version 1  
VA Central IRB Protocol Template – version 10/26/2012 
Page 7 of 33 
 
it, and simultaneously save money for the healthcare system.  
 
The study will explore the existence of a causal association between a behavioral 
intervention and facility-level guideline-concordant imaging. Prior analyses of 
prostate cancer imaging guideline implementation efforts were either retrospective2 or 
lacked a control group.3 Their results may be affected by unmeasured confounding or 
secular trends. Employing a multi-site, stepped wedge cluster-randomized trial will allow 
determination of whether an intervention is causally linked with changes in imaging 
behavior.4 
3.0 
Objectives 
 
 
Aim 1: To determine whether a multi-modal, physician-focused behavioral 
intervention can improve facility-level guideline-concordant utilization of prostate 
cancer imaging. 
o H 1.1: A physician focused intervention will decrease facility-level utilization of 
guideline-discordant imaging among low-risk men because it will address the 
causes of inappropriate imaging. 
o H 1.2: A physician-focused intervention will increase facility-level utilization of 
guideline-concordant imaging among high-risk men because it will actively 
promote imaging among patients who need it most. 
 
 
Aim 2: To use mixed methods to explore physician influence on guideline-
concordant imaging. 
o H 2.1: Physicians who finished residency training more recently will be more 
likely to perform guideline-concordant imaging than their more experienced 
peers. 
o Objective 2.1: Through semi-structured interviews, the research team will explore 
physicians’ experiences with and perceptions of the intervention and how those 
perceptions relate to prostate cancer imaging use. 
 
 
Aim 3: To determine the cost and cost impact of a physician-focused behavioral 
intervention to improve guideline-concordant prostate cancer imaging. 
o H 3.1: The costs of the intervention (including physician time) and increased 
guideline-concordant imaging will be offset by savings made in reducing 
guideline-discordant prostate cancer imaging. 
4.0 
Resources and Personnel 
Management of the study will take place at VA New York Harbor Healthcare 
System (VANYHHS), the primary site. The intervention will be delivered to 10 

Version 1  
VA Central IRB Protocol Template – version 10/26/2012 
Page 8 of 33 
 
participating geographically-distributed, high volume prostate cancer treatment 
sites. Urology Chiefs at each site have agreed to participate, share expertise, 
support staff and space with our team. All administrative work, study planning, 
and data analysis will be done at VANYHHS. The PI Danil Makarov, MD, MHS; 2 
Co-Investigators: Craig Tenner, MD and Steve Zeliadt, PhD, and the research 
staff members (Project Manager, Data Manager, and Research Coordinator) will 
have access to identifiable information.  
Danil V. Makarov, MD, MHS, Principal Investigator 
As the Principal Investigator of the study, Dr. Makarov will oversee all aspects of 
the study including providing guidance to all co-investigators, study staff and site 
PI’s. Additionally, he will provide the Performance Measurement/Audit and 
Feedback to each urologist and travel to each site to conduct Academic 
Detailing.  
 
Scott Sherman, MD, MPH, Co-Investigator 
Dr. Sherman will provide both high level and logistical guidance on behavior 
change intervention implementation for a multi-site study. 
 
Craig Tenner, MD, Co-Investigator 
Dr. Tenner will be responsible for leveraging his expertise to build strong 
relationships with the local IT departments and to assist with the order check 
implementation and analysis.     
 
Steve Zeliadt, PhD, MPH, Co-Investigator 
Dr. Zeliadt will be responsible for the high level quantitative analysis and work 
with both the Data Manager and Research Staff to ensure that collection, 
cleaning and analysis are all aligned. 
 
Heather Gold, PhD, Co-Investigator, WOC-VA NY Harbor 
Dr. Gold will apply her expertise in analyzing large datasets, her history working 
with the team on prostate cancer imaging studies, and her prior research 
analyzing the cost-effectiveness of cancer treatments to this study. She is 
responsible for overseeing the cost-effective analysis of this study.  
 
Michele Shedlin, PhD, Co-Investigator, WOC-VA NY Harbor 
Dr. Shedlin will take the lead role in designing all interview and survey 
instruments and qualitative data analysis. She will also conduct in-depth 
qualitative interviews at each site following the intervention. 
 
Site Principal Investigators, 10 participating VA sites  
Site PIs at each location have committed to creating Local Clinical Advisory 
Committees, which will comprise the Site PI, Urology Chief, a local IT 
representative, and a local Research Assistant. The Site PI is responsible for 
maintaining quarterly meetings with the Local Clinical Advisory Committees, in 
addition to maintaining regular contact with PI and project manager. The 
urologists serving as Site PIs and Urology Chiefs will also have the opportunity to 
enroll in the study as participants (one site-PI is a non-urologist oncologist, Dr. 
Andrew Liman, and will not participate as a study subject).  
 

Version 1  
VA Central IRB Protocol Template – version 10/26/2012 
Page 9 of 33 
 
Research Assistants, 10 Participating VA Sites 
The Research Assistants will work directly with the Site PIs to ensure that the 
intervention components are carried out efficiently. The Research Assistants will 
act as a liaison at the participating VA site, submit the appropriate IRB 
documents for their respective sites (including R&D committee requirements), as 
well as complete other administrative responsibilities associated with the study. 
 
Scott Braithwaite, MD, MSc, Advisor  
Dr. Braithwaite will serve as an advisor for the project. He will help strategize 
research directions during periodic meetings, assist with writing manuscripts and 
dissemination of results. He will have no access to identifiable information or 
data.  
 
Hilary Oliphant, MHA, Research Manager 
The research manager will manage this project by creating and managing grant 
budgets, overseeing project-related contracts, facilitating the hiring and 
credentialing process for project staff, and communicating with project PIs and 
staff to achieve research goals. Additionally, she will oversee other aspect of the 
study including staffing and training.  
 
Shannon Ciprut, MHS, Project Manager, WOC- VA NY Harbor 
The project manager will serve as the primary liaison with research staff at 
participating sites. She will lead local project staff at VA NY Harbor in study 
activities, including participant ment, and lead and maintain regular 
communications between all parties involved including: study sites, research 
staff, and the PI. She will administer surveys to provider participating and collect 
results for analysis and may conduct interviews. She will also assist the data 
manager in data analysis and presentation.  
 
Dawn Walter, MPH, Data Manager, WOC- VA NY Harbor  
The Data Manager will ensure data integrity and security and will conduct 
preliminary analyses through active participation in the study, beginning in the 
earliest stages, to ensure the creation of secure, accurate, efficient, and 
epidemiologically sound data environments. This individual will work closely to 
train the Research Coordinator and provide expertise with these datasets 
throughout the study. 
 
Matthew Kelly, Research Coordinator  
The Research Coordinator will aid the research staff in coordinating the study. 
This individual will work closely with the data manager to extract key data points 
from chart review to compile a dataset of results.  The RC will be responsible for 
supporting the team by assisting with regulatory approvals, including IRB; 
coordinating with Site Research Assistants; and providing necessary 
administrative support to the project. He will attend weekly check-ins with the PI 
and the Project Manager. 
 
Centralized Transcript Services Program 
For the qualitative portion of the study, transcripts (text of interviews that have 
been audiorecorded) will be generated by the Centralized Transcript Services 
Program. CTSP is an internal VA service available to investigators as part of the 

Version 1  
VA Central IRB Protocol Template – version 10/26/2012 
Page 10 of 33 
 
Informatics Decision Enhancement & Analytical Sciences (IDEAS2.0) initiative 
funded through HSR&D at the VA Salt Lake City Health Care System (PI: Susan 
Zickmund, PhD). All identifying participant information will be removed before the 
transcription service has access to the secure project folder.This service is 
internal at the VA. Interviews that have been audio-recorded will be stored on a 
VA secure research drive prior to transcription. 
 
5.0 
Study Procedures 
 
5.1 Study Design 
This study aims to utilize both quantitative and qualitative methods to evaluate the 
combination of our three proposed evidence-based interventions. The study will evaluate 
three complementary, multi-level interventions for improving the rates of guideline-
concordant prostate cancer imaging at VHA. This theory-based strategy was developed 
based on preliminary data exploring barriers and facilitators to guideline-concordant 
prostate cancer imaging5 subsequently mapped to effective behavior change 
interventions.6,7 
To frame implementation of our intervention at the 10 committed study sites and to 
inform sustainability and dissemination of our findings, we used the Consolidated 
Framework for Implementation Research (CFIR), a compilation of existing 
implementation theories offering an overarching typology to promote implementation 
theory development and verification about what works in which setting and why. CFIR is 
composed of 5 domains: Intervention characteristics, Inner Setting, Outer setting, 
Individuals Involved, and Implementation Process. Each domain has within it between 4 
and 12 constructs. The domains and constructs most relevant to the current project 
include Intervention Characteristics (Evidence Strength & Quality, Trialability, 
Adaptability, Complexity, and Cost), Inner Setting (Networks, Culture), Outer Setting 
(Peer Pressure, External Policies), Individuals (Knowledge, Self-Efficacy, Individual 
stage of change), and Process (Planning, Engaging, Executing, and Evaluating). 
Qualitative analysis in Aim 2 will use a CFIR-based interview guide to explore participant 
experience and guide subsequent dissemination. 
Using a stepped wedge cluster-randomized design, the first time point will be a baseline 
measurement, where none of the study sites initiate the intervention. This is a single 
direction cross-over randomized trial where every site serves, at some point, as both 
control and an intervention site.4 This will allow for an accurate and fair reading of 
baseline imaging measures at each site. Table 1 shows a chart outlining the proposed 
project timeline. 

Version 1  
VA Central IRB Protocol Template – version 10/26/2012 
Page 11 of 33 
 
At subsequent time points, study sites initiate the intervention. The time at which each 
site initiates implementation of the intervention is randomized. Due to the staggered 
nature of the intervention, in the site randomized to receive the intervention first it will be 
ongoing for 33 months and the last site will receive a 6 month intervention. The 6 month 
minimum duration of each intervention component is consistent with prior 
implementation literature.8-10 A stepped wedge design is particularly useful when an 
intervention must be administered on a community scale (e.g. a Clinical Order Check) or 
for other financial, logistic, or ethical reasons.11 Guaranteed access to the intervention 
has been a powerful recruitment tool. 
 
 
The three interventions that each 
site will receive are: audit and 
feedback, academic detailing, and 
a clinical order check. Each 
method is described in detail 
below. 
 
Audit and Feedback: Audit and 
feedback is an effective, 
individual-level intervention for 
changing healthcare provider 
behavior, resulting in small but potentially clinically important benefits.9,12 Audit and 
feedback addresses the intervention functions of education, persuasion and 
incentivization, all of which are important for addressing beliefs about capabilities and 
consequences, knowledge, and social influence determined to be significant in our 
preliminary work.13,14 We will provide quarterly feedback on prostate cancer imaging 
performance to every participant at each study site; feedback will be given to each 
provider individually and will include his or her individual-level data as well as 
aggregated data for the local institution (for which there are more than 3 providers 
included in the aggregated sample, including both participants and non-participants) and 
VHA as a whole as collected from Oncotrax and validated using CAPRI. The data will be 
reviewed during a brief group meeting between the Site PI and participants at the clinic, 
arranged at their discretion, and will include specific recommendations for overall 
performance improvement from the Site PI for the site as a whole, based on the 
aggregate reports. Site PIs will receive de-identified individual level rates, so long as 
there are more than 3 participants enrolled at that site (including the Site PI). Individual 
participants will only see individualized reports for themselves. Participants who are not 
serving as Site PI will not have individual-level data for their colleagues and will not be 
aware of any other individual enrollment status. 
 
Academic detailing: Academic detailing (also known as educational outreach) is an 

Version 1  
VA Central IRB Protocol Template – version 10/26/2012 
Page 12 of 33 
 
individual and facility-level intervention consistently shown to affect provider behavior.8,12 
This strategy addresses the intervention functions of persuasion, modeling, and 
education which are effective methods for affecting behaviors driven by beliefs about 
capabilities, knowledge, social influences, beliefs about consequences, and 
environmental context and resources.13-15 The academic detailing sessions will take 
place at the initiation of the intervention and then regularly thereafter as determined by 
the local investigative team throughout the intervention period. The initial session will be 
performed by a member of the primary investigator team along with the local site PI; 
subsequent sessions will be performed by the local site PI to encourage sustainability. 
During the meeting, the representative from the investigator team will follow a script 
explaining that the visit is part of an experimental program to provide physicians and 
providers with up-to-date, unbiased information about imaging to stage prostate 
cancer.8,16 The representative will review summary information from the NCCN and AUA 
prostate cancer imaging guidelines and encourage the provider participants to modify 
their ordering behavior to comply with those guidelines.17 Appeals based on fear or 
coercion will be avoided. Improvement of clinical care will be emphasized above cost 
considerations. Providers will be encouraged to participate in the educational exchange 
and to discuss specific problem cases. Summaries of the guidelines and their URLs will 
be left with providers. A sample agenda for the visit will include: 
1. Review prostate cancer imaging guidelines. 
2. Demonstrate the clinical reminder. 
3. Mention the audit and feedback. 
4. Answer any questions. 
Subsequent academic detailing sessions may occur in-person, by phone, or over e-mail. At the 
discretion of the site-PI, sessions may take place during regularly scheduled department 
meetings. Attendance at these sessions will not be mandatory. Because of this, it is possible 
that non-participating providers may be exposed to the academic detailing sessions, 
should these occur during department-wide meetings. There will be no requirement to 
stay, and there will be no indication of who is enrolled and who is not at these sessions.  
 
Clinical Order Check: A Clinical Order Check is an evidence-based, systems-level 
method to affect behavior change.10,18-20 It addresses the intervention functions of 
education, enablement and incentivization which are effective methods to change 
behaviors driven by beliefs about capabilities, knowledge, social influences, beliefs 
about consequences, and environmental context and resources; all significant domains 
that have been established in our preliminary work.5,21,22 All VA facilities currently use 
locally adapted clinical reminders. We will adapt the Order Check currently in use at 
VANYHHS (implemented by Drs. Makarov, Sherman and Tenner) and adapt it for 
implementation at other study sites with guidance from their local Clinical Advisory 
Committee. This strategy is technologically simple, straightforward, and is considered to 
be a best practice within the VA IT community.23 As at VANYHHS, the reminder will be 
self-explanatory and non-intrusive to workflow. Reminder specifics include: 

Version 1  
VA Central IRB Protocol Template – version 10/26/2012 
Page 13 of 33 
 
1. Selection criteria: The Clinical Order Check will appear when a patient has the 
following characteristics: 
i. Male sex 
ii. Diagnosis of prostate cancer or prostate biopsy performed within 6 months of 
the current date 
iii.  Serum PSA<20 ng/mL. Those with higher PSAs all require imaging. 
iv.  Imaging Modality: Provider selects: bone scan or axial imaging of abdomen 
or pelvis  
The order check will not appear for imaging ordered for men with CPT codes associated 
with prostate cancer treatment in the past year as the intervention is tailored to address 
incident prostate cancer cases. 
2. Content – Based on consultations with our local physician leaders and administrators, 
we agreed on the following text for the Order Check: “Imaging not recommended to 
stage men with PSA<10, Gleason<7, and clinical stage <T3. Imaging recommended for 
high-risk cancer. Excessive imaging may harm patients and waste resources.” Below the 
popup screen are references supporting the recommendation. Local site advisory 
committees will have the opportunity to modify this text according to their practice needs 
and culture. 
3. Opt out – Providers may override the recommendation against ordering and are asked 
to explain their reasons for doing so. (The local IT representative, as part of the Local 
Clinical Advisory Committee, will be able to pull these responses from CPRS, in addition 
to the number of times the Opt Out option was utilized during the intervention period. 
Public Key Infrastructure (PKI)-encrypted email through VA Outlook will be used to share 
de-identified Clinical Order Check results between local IT representatives at 
participating sites and the central research staff. PKI is a VA-approved software used to 
secure the delivery of electronic services to VA employees, contractors, and business 
partners. The IT personnel will provide the reasons why participating providers have 
overridden (“opted out”) the order check and the quantity of times the pop-up was 
overridden by participants during the intervention period. VA Outlook is secure and PKI-
encrypted emails are only accessible to those who have been granted access. It is only 
accessible on the VA network and requires VA network ID and password.  
This feature will be implemented at the institution-level, and it will not be possible for the 
order check to distinguish among providers by enrollment status. Providers who are 
NOT participating in this study will need to simply click (“x”) out of the pop-up notification 
to proceed with their orders.  
 

Version 1  
VA Central IRB Protocol Template – version 10/26/2012 
Page 14 of 33 
 
Additional quantitative measures: Three validated instruments will be administered 
prior to the initiation of the intervention to all subjects at participating sites as one 
collective pre-intervention survey. The Organizational Readiness to Change Assessment 
(ORCA scale)24, Evidence Based Practice Attitudes Scale25 (EBPAS), and a Self-
Assessment of Contextual Fit26 will be given before the intervention is initiated, along 
with a demographic questionnaire. The first set of instruments assesses organization 
strengths and weaknesses to support implementation of evidence based practices as 
well as a provider’s feelings towards adopting new practices The CFIR constructs that 
are captured through this measurement are: evidence strength & quality, structural 
characteristics, characteristics of the individual, networks and communication, culture, 
compatibility, incentives & rewards, goals & feedback, leadership engagement, planning, 
and reflecting.  After the intervention, another survey consisting of two additional 
validated instruments will be distributed to all participants: the Adoption of Information 
Technology Innovation-Compatibility Subscale27,Level of Success Instrument28, and a 
modified Self-Assessment of Contextual Fit.26 These will measure the degree to which 
employees feel new technologies are compatible with their work, how successful the 
adoption of an innovation was, and the contextual fit of the PCIS intervention with the 
clinical department environment. The ORCA scale will also be administered within the 
post-intervention survey.  The CFIR constructs of compatibility and penetration will be 
measured through these tools. These surveys will be deidentified for analysis.  
Qualitative Component: We will be conducting a qualitative piece to assess participant 
opinions of the proposed intervention and how those perceptions relate to prostate 
cancer imaging use, among a subsample of the frontline provider study participants 
(N=25-40). This will consist of in-depth, semi-structured interviews. At the end of the 
intervention one-on-one, in-depth semi-structured interviews will take place with a 
participant and a member of the research team in person or by phone, using a CFIR-
based interview guide. We will also invite participating site-PIs and urology chiefs to 
participate in this qualitative component. In addition to the standard interview guide, we 
will also ask a short series of questions exclusively to all participating Urology Chiefs and 
site PIs (N=20-30) to explore institutional and managerial perspectives of implementation 
and attitudes towards sustainability of the intervention. We will continue interviews until 
we reach thematic saturation. 
5.2 Recruitment Methods 
 
Provider Recruitment: 
We have letters of commitment from the Urology Chiefs at all sites with their 
approval and agreement to take part in this study. Providers from all 10 
participating VAs who have the capability of ordering imaging and treat patients 
with and at-risk of prostate cancer will be invited to participate including those 
with VA clinical privileges with WOC status. We will attempt to enroll all urologists 
at the participating study sites, including the Chiefs and Site PIs (N=160; (10 

Version 1  
VA Central IRB Protocol Template – version 10/26/2012 
Page 15 of 33 
 
Chiefs and 150 frontline practitioners)We will also invite all, Physician Assistants, 
Nurse Practitioners, and other urology providers who order imaging and treat 
prostate cancer patients at participating sites to enroll (bringing N close to 200).  
Participation in the study is completely voluntary. The section Chiefs will have no 
knowledge of the extent to which providers from their clinic enroll in the research 
or not, as participation will be kept confidential among the research team. 
Responses to quantitative and qualitative surveys and interviews will be de-
identified and kept confidential and reported as aggregate. Participation will not 
affect employment at the VA, nor will it affect professional standing in any way 
including any academic appointments. 
Some aspects of the intervention are provider-specific while others are institution 
specific. We will study 2 samples of physicians: Site PIs and Urology Chiefs as 
well as frontline staff physicians. There are 20 Chiefs and Site PIs (2 at each site) 
and around 150 frontline urologists (full-time, part-time, and WOC) at the 10 sites 
(range = 3-20). There are around 200 potentially eligible providers at all 10 sites. 
We will also study other types of providers who have the ability to order imaging 
and care for patients with prostate cancer at VA including: Nurse Practitioners, 
and Physician Assistants, and other urology providers.  
The central research staff, the PI, and the Site-PI will collectively send 
informational e-mails to all eligible providers through VA Outlook. Eligible 
providers will have the opportunity to express interest or opt out of further study 
communications. Next, the central research staff, PI, and Site-PI will send 
recruitment emails that will include a study information sheet as an attachment. 
Eligible providers will have an opportunity to express interest, ask questions, 
communicate with study team members, or opt out of further study 
communications. Study contact information will be included in the e-mail to 
accept all inquiries related to the study and to go over the study and all items 
related to consent over the phone. The recruitment e-mail will contain a unique 
link, generated by the VA REDCap system, that will direct them to a portal within 
which the participants may designate consent. The study team will follow-up with 
non-responders by e-mail and by phone. Immediately following the consent 
page, the pre-intervention survey will appear. The study team will follow-up with 
participants who have consented but who have not filled out the survey by e-mail, 
one week following the indication of consent, and each week consecutively 
thereafter for up to 3 weeks (and a maximum of 3 survey reminder e-mails).  
 
During the first academic detailing session, a member of the research staff and 
the PI will travel to each study site as part of a project kick-off meeting. Since the 
session will take place at the LSI’s discretion, it is possible that this will occur 
during a Grand Rounds session or regular departmental meeting. In this case, 
there may be providers present who are eligible for the study but have not 
enrolled. During the session, the PI and study team member will let all site 

Version 1  
VA Central IRB Protocol Template – version 10/26/2012 
Page 16 of 33 
 
providers that although the study has begun, eligible providers may still opt to 
enroll in the study. The PI and study team member will invite any unenrolled, 
eligible providers to contact the study team by phone or by e-mail if they opt to 
participate at a later time. This will be conducted through a general 
announcement to the group as a whole, to avoid singling anyone out or 
identifying any participants. The providers may approach the study team in-
person during the visit or contact them later on, at their discretion. The study 
team will bring hard copies of the study information sheet to the meeting should 
anyone express interest or request the information. The study team will also 
bring hard copies of the REDCap consent portal and pre-intervention survey. If 
providers express interest and willingness during the site visit, the providers may 
fill these out in-person and return to either a local study team member or visiting 
central team member. The study team will also provide self-addressed stamped 
envelopes for providers who wish to enroll but fill out the paperwork at a later 
time. Providers may return these envelopes to the local site team or mail 
themselves to the central research team, whichever is most convenient. The 
central research team will enter the paper-based responses in REDCap. Upon 
doing so, the study team will shred the paper-based versions of the survey. The 
study team may also re-send study information sheets and the REDCap consent 
portal by e-mail to anyone who requests it. The enrollment status of meeting 
attendees will not be shared or discussed.   
Study participants will consent for the qualitative of the study during the informed 
consent process for the study as a whole. Participants will be reminded of this 
effort and invited to participate by email sent from the NY Harbor research team. 
An effort will be made to recruit Urology Chiefs (n=10), site PIs (n=10), and a 
subset of frontline practitioners (about 20-30). Reminder e-mails (up to 3 e-mails 
in 3 consecutive weeks following the initial contact) will be sent to non-
responders and include requests to schedule interviews to occur either in-person 
or by phone. Recruitment phone calls may also be made to participants’ listed 
phone numbers to non-responders from e-mail. Up to 3 phone call attempts may 
be made in 3 consecutive weeks following the initial contact; following the 
maximum attempts of contact, recruitment efforts will cease. 
For the qualitative interviews, we will recruit 20 to 30 participants across the 
sites, in addition to urology chiefs and Site-PIs in order to reach theoretical 
saturation. Interviews will cease following team consensus of theoretical 
saturation.  Participants will be invited to schedule an interview in-person or by 
phone. The study team initially plans to enroll about 30 participants beyond the 
Site-PIs and Urology Chiefs for interviews. Interviews and analyses of 
transcriptions will be ongoing throughout the qualitative portion of the study. 
Interviews will either cease early or continue enrolling until theoretical saturation 
has been achieved. Interviews that have been scheduled in advance will still 
occur. If a participant agrees to an interview after the team has ceased qualitative 
data collection, then an interview will not occur.  

Version 1  
VA Central IRB Protocol Template – version 10/26/2012 
Page 17 of 33 
 
 
After the intervention period, participants will receive an e-mail from the study 
team requesting participation in the post-intervention survey. The e-mails will 
contain new, unique links for distribution and self-initiated access to the surveys. 
Up to 3 follow-up emails will be sent to participants who did not complete the 
surveys in the 3 consecutive weeks following the initial contact regarding these 
instruments.  
 
Participants will not be compensated for their participation in this study. 
Participation in this study will not influence employment. The 10 Chiefs have 
already agreed for their departments to participate in this study but will still give 
informed consent to participate as individual study subjects. Similarly, the 10 Site 
PIs, none of whom are Chiefs, will be invited to participate in the study. The 
Urology Chiefs will have no knowledge of the extent to which staff members in 
their department elect to participate in this study. 
Subject Recruitment: 
Prostate cancer patients will not be contacted directly for participation in this 
study and the research team will not have any direct interaction with patients, 
however the NYHHS team will have access to personal health information. The 
Cancer Registry is an online tool used by cancer registrars to create and 
maintain a VA cancer patient database. Oncotrax assists registrars in saving 
information on diagnosis, treatment, and survival for newly diagnosed cancer 
cases. These data are used in quality improvement and research studies as well 
as in incidence and trends analysis.  A waiver of HIPAA authorization and a 
waiver of informed consent will be secured to identify patients in the VHA 
OncoTrax data in CDW using the ICD-9 code 185 or the ICD-10 code C61 for 
prostate cancer. Patients will be excluded if they have a history of prior 
malignancy, are over the age of 85, diagnosed at autopsy or by death certificate, 
died within 3 months of diagnosis, or not having data on at least one of the 
following: PSA, clinical stage, or Gleason score. 
5.3 Informed Consent Procedures 
Informed consent will be obtained from all providers who agree to participate at 
the 10 study sites at the beginning of the study. The consent process details the 
study procedures, including the interventions, as well as the expectations of the 
participating providers including participation in completion of surveys and 
interviews.  The study team is requesting a waiver of documentation of informed 
consent for provider participants, as this study involves very minimal risk and 
would not require informed consent outside of research. The recruitment process 
will be conducted remotely by the central site. 
The Site PI will identify eligible prostate cancer providers including urologists, 
PAs and NPs. Research staff members will identify email addresses through the 

Version 1  
VA Central IRB Protocol Template – version 10/26/2012 
Page 18 of 33 
 
VA directory as needed. The Site PI will send an informational e-mail to all 
eligible providers to briefly explain the purpose of the study intervention and an 
overview of participation. Eligible providers will have the opportunity to reach out 
to the Site PI or research team member with any potential questions or to 
express preliminary interest. The email will clearly state that participation is 
voluntary and will not affect employment at VA.  
One week following the informational e-mail, the central research team will 
follow-up with a recruitment email to those who have expressed interest and 
those who did not respond to the informational email. The team will contact each 
of the eligible providers by email (VA Outlook) and invite them to participate in 
the study through recruitment email with an information sheet attachment. The 
consent process will be conducted using a REDCap-based electronic consent 
portal.  The consent portal will be developed in REDCap, a secure, web-based, 
HIPAA-compliant, data collection platform with a user management system 
allowing project owners to grant and control varying levels of access to data 
collection instruments and data (e.g. read only, de-identified-only data views) for 
other users.  Potential participants will participate in the consent process by e-
mail approach from the study team, with an invitation for self-initiated access of 
unique link to consent portal and survey on personal portable electronic devices 
using web-links. Self-initiated accessing of consent portal may occur in clinic or 
at home; however, VA REDCap is accessible only on the VA network and login 
requires valid VA network ID.  
 
Upon receipt of the recruitment e-mail, potential participants will have the 
opportunity to speak to a study team member to go over study procedures and 
ask questions. Potential participants may also opt out of future study 
communications upon request. The central research team and/or the Site RA will 
schedule either a phone call (if NY Harbor individual) or in-person (if Site RA) 
meeting to discuss the study procedures and to provide an opportunity for the 
provider to ask questions about the study and process of consent upon the 
request of the potential participants and to non-responders by e-mail. Contact 
information will also be provided (email and phone) for prospective participants to 
contact a member of the key study personnel with questions, prior to consent. All 
individuals who will have the ability to obtain informed consent have extensive 
human subjects’ research experience.  All individuals obtaining consent will be 
members of the research team who are designated and approved for doing so. 
All members of the research team have completed numerous VA research 
trainings as well as CITI Human Subjects Protection training. Communications 
including the information sheet will explicitly state that this study is purely 
voluntary, and there is no obligation to participate. If providers decline 
participation, there will be no consequences. Communications will state that this 
study will have no effect on employment status and their supervisor will not be 
aware of his/her decision to participate or not.  

Version 1  
VA Central IRB Protocol Template – version 10/26/2012 
Page 19 of 33 
 
Participant indication of consent will be obtained using a typed signature, to be 
submitted through the REDCap portal. The information sheet will be included as 
an attachment with the recruitment email. If, when the study team speaks with 
potential participants, they don’t remember or cannot locate the information 
sheet, the study members will confirm the best email address to send it to them 
and re-send it for reference. If a participant does consent to the study, REDCap 
will send an automated completion notice that will include the information sheet 
as an attachment. If a provider declines participation, or requests to no longer be 
contacted regarding this study, the study team will not contact that individual 
again. Follow-up emails will be sent one week after initial contact to those who do 
not respond. Up to 3 follow-up emails will be sent to non-responders. The central 
research team will make phone calls to the numbers listed in the VA directory in 
attempt to reach individuals who did not respond to emails. The research team 
will end follow-up after 2 unsuccessful phone calls to non-responders. 
The research team will re-send the REDCap consent portal after the intervention 
has begun if a non-participating provider requests to be included in the study 
during or following the study kickoff site visit. The research team may send one 
follow-up email or follow-up phone call as necessary, and all recruitment efforts 
for providers at the given site will cease at that time.  
Study team members will record and receive verbal consent to audio record from 
participants for interviews that occur over the phone through Micrososft Teams. 
Participants in the qualitative portion of the study will have already consented by 
agreeing to participate in the study during the recruitment process.  
Informed consent will not be required of patients due to a waiver of HIPAA 
authorization and a waiver of informed consent. There will be no active 
participation of patients in this study; the study team will access patient electronic 
health records in order to monitor trends in imaging among prostate cancer 
patients.   
5.4 Inclusion/Exclusion Criteria 
Provider Criteria 
 
Inclusion criteria 
o Urology Chiefs and attending urologists employed through the VA 
(full-time, part-time, WOC) at one of the 10 participating sites; 
Physician Assistants and Nurse Practitioners employed through 
the VA at one of the 10 participating sites that work in the 
respective urology clinics 
o Providers may be any gender or race/ethnicity 
o Qualitative portion only: Urology Chiefs and/or frontline staff 
physicians; participating PAs & NPs having cared for at least 5 
men with incident prostate cancer within the previous 6 months 

Version 1  
VA Central IRB Protocol Template – version 10/26/2012 
Page 20 of 33 
 
 
Exclusion criteria 
o Urology Residents 
In efforts to avoid potential coercion, Urology Residents are excluded from the 
study due to the hierarchical culture of surgical training program in addition to 
preliminary findings that imply that they strictly adhere to attending preference.29  
Patient Criteria 
Patients will not be directly recruited into the study. We will obtain a waiver of 
HIPAA authorization and informed consent to analyze electronic health records 
of patients at the 10 participating sites. 
5.5 Study Evaluations 
Screening 
Physicians and other eligible providers will be identified based on their 
employment in the Urology sections at the 10 participating sites. Urology Chiefs 
and local site-PIs have already been identified.  
Questionnaires 
Prior to the initiation of the intervention, immediately following the e-consent 
process, all provider participants will be asked to complete a demographic 
questionnaire as well as validated survey measures. The demographic 
questionnaire will ask providers questions about age, race/ethnicity, and details 
about their time as urologists/providers, such as years in practice and training 
focus. Full, Part-time, or WOC VA status of the providers will be determined 
administratively. After the demographic questionnaire the participants will 
complete the ORCA scale,24 and the EBPAS scale.25 When the intervention is 
complete three additional measures will be administered to participants: Adoption 
of Information Technology Innovation-Compatibility Subscale,27 Level of Success 
Instrument,28 and a Self-Assessment of Contextual Fit.26 Participants will also 
complete the ORCA scale again as a pre/post measure. The research staff will 
prepare the VA-approved REDCap portal for virtual distribution and collection of 
results. The staff will provide access information via VA Outlook to distribute the 
survey instruments electronically, including unique links to unique portals to 
collect data from each participant. REDCap is a secure, VA-approved web 
application that facilitates the collection and entry of research data. The main 
research staff at NY Harbor will be able to access the questionnaire results 
directly through the secure REDCap project account. The main research staff will 
have exclusive access to this account. As per VHA Record Control Schedule 10-
1 8300.6, collection of records will be cutoff at the end of the fiscal year after 
completion of the research project, and results will be destroyed 6 years after 
cutoff. All responses collected will be deidentified. Results will be reported as 
aggregate. Individual responses will not be shared with the Urology Chiefs, and 
the Urology Chiefs will not be aware of who has completed the survey or not.  

Version 1  
VA Central IRB Protocol Template – version 10/26/2012 
Page 21 of 33 
 
 
Qualitative Interviews 
Qualitative interviews will be conducted using an interview guide developed using 
CFIR. The aim of the interviews is to understand the acceptability and 
sustainability of the intervention. There is a section on the interview guide that is 
specifically for the Urology Chiefs and Site PIs, as leaders of this project. These 
questions explore the institutional perspectives of intervention implementation 
and attitudes towards sustainability of the intervention. An investigator along with 
a member of the main research staff will travel to each of the participating sites to 
conduct the interview in-person, or the interviews will be held over the phone by 
a member of the central investigative team (investigator or staff member). Data 
from the interview will be collected through the VA Microsoft Teams application 
and uploaded to a secure internal research drive for transcription.  
Additionally, the central research team will collect ongoing qualitative data from 
site research team members about the implementation of the intervention every 6 
months post commencement via informal phone interviews.  Demographic 
information of LSIs and Local Rcs will be recorded. 
All evaluation materials are included as a separate attachment with the protocol.  
5.6 Data Analysis 
All data will be managed at VANYHHS by the Data Manager, Research Coordinator, and 
Project Manager. They will collect, clean, and analyze datasets. Other members of the 
main study team will analyze aggregated or de-identified data. Statistical analyses will 
be performed using the SAS 9.1.3 Service Pack 4 statistical package (SAS Institute, 
Cary, NC) and Excel 2010 (Microsoft Corporation, Seattle, WA). Qualitative data will be 
analyzed using NVivo 10 (QSR International). REDCap is a secure, VA-approved web 
application that facilitates the collection and entry of research data. A private, password 
protected REDCap project will be created for this study. Study team members will 
ascertain the input data from the questionnaire as well administered to participants. 
REDCap not only stores data but can generate descriptive statistics. Site-specific results 
will be deidentified and will not be shared with anyone beyond the central research team. 
Specifically, only aggregated data will be shared with Chief or Site PI. Site-specific 
information will be collected for analysis purposes only; results will be reported as 
aggregate. 
  
Quantitative Analysis 
Quantitative data for this study is from VHA’s Corporate Data Warehouse (CDW). VINCI, 
VHA’s secure data environment will be used to identify prostate cancer patients in CDW 
as well as their imaging tests, demographic information, and clinical history. The VINCI 
team will extract required data from CDW tables and create a work environment for our 
team on secure VINCI servers in Salt Lake City. VSSC’s Cancer Care Cube (CCC) will 
be used to generate timely data for Audit and Feedback. CCC permits rapid access to 

Version 1  
VA Central IRB Protocol Template – version 10/26/2012 
Page 22 of 33 
 
accurate nationwide VA cancer registry data30 from the clinical OncoTrax system, 
uploaded once monthly. To address potential reporting delays, we will check the validity 
of the Oncotrax data by comparing them with CAPRI data biweekly - a focused review of 
about 200 charts a week. In further efforts to avoid delays due to lags in CDW data 
availability, the local IT specialist will run a reoccurring monthly report in CPRS in 
attempt to identify patients diagnosed with prostate cancer in real-time. The results will 
yield patient name, SSN, and presumed date of prostate cancer diagnosis. The IT 
specialist will send the results of these queries to the central research team via PKI-
encrypted email on VA outlook periodically. If the IT specialist at a site is unavailable to 
produce these reports, another member of the Local Clinical Advisory Committee will 
send either a list of newly diagnosed prostate cancer patients or a list of patients 
receiving prostate biopsies at that site via PKI-encrypted email to the central research 
team. The central research team will verify and validate this data using concentrated 
chart review within CAPRI.  Due to local practice differences at the Madison VA only, the 
central research team is unable to verify receipt of imaging in CAPRI, such as bone scan 
or CT, (our primary outcome) without access to the local consult notes in CPRS.  
Therefore, the central research team will verify eligibility status of patients sent from the 
initial local CAC data pull. The central research team will send this confirmed list back to 
the local research coordinator via PKI-encrypted email for verification of imaging receipt 
and date of imaging. The local research coordinator will ascertain the imaging order and 
status via local consult notes in local CPRS system. The local research coordinator will 
then send the confirmation of imaging completed, type of imaging, and date received via 
PKI-encrypted email to the central research team. This will serve as a supplemental 
source of data to improve data validity, accuracy, and quality for both feedback reports 
and overall imaging rates. The study team will collect the following data from CAPRI: 
date of birth, zip code, marital status, race/ethnicity, Gleason score, date of diagnosis, 
PSA, clinical stage, date of medical appointments, diagnostic procedures/treatment, 
imaging ordered, and treating provider.  This will guarantee the fidelity of the data used 
for Audit and Feedback and provide insight into the timing with which data from each 
individual site are updated for future projects. CCC data will be stored securely in VINCI 
where it will be linked to CDW when necessary. We will request remote CAPRI access 
to all 10 study sites for the central research data team from the National Information 
Security Officer (ISO) following Central IRB approval.  
The sample size of 10 study sites was determined to ensure appropriate power for our 
primary outcome: differences in the rate of inappropriate prostate cancer imaging. For 
the sample size calculation of a stepped wedge trial, the key variables are the number of 
clusters (i.e., sites), 𝐼; the number of distinct time points or intervals being compared, 𝑇; 
and the number of outcome observations per time point, N (i.e., the number of individual 
patients with the outcome per cluster, per time interval).4 We assume the model, 𝑌௜௝=
𝜇+ 𝛼௜+ 𝛽௝+ 𝑋௜௝𝜃+ 𝑒௜௝, where 𝛼௜ is a random effect for cluster i such that 𝛼௜~𝑁(0, 𝜏ଶ), 𝛽௝ 
is a fixed effect corresponding to time interval j, 𝑋௜௝ is an indicator of whether the 
intervention has been implemented in cluster i at time j (1=intervention; 0=control), 𝜃 is 
treatment effect and 𝑒௜௝= ∑𝑒௜௝௞𝑁
⁄
௞
 are independent and identically distributed 𝑁(0,𝜎ଶ) 

Version 1  
VA Central IRB Protocol Template – version 10/26/2012 
Page 23 of 33 
 
and 𝜎ଶ= 𝜎ଶ௘𝑁
⁄ . Let 𝑌௜௝ be the mean for cluster i at time j. Assume testing the 
hypothesis 𝐻଴: 𝜃= 0 versus 𝐻஺: 𝜃= 𝜃஺, where 𝜃஺ is the treatment effect size. The 
approximate power for conducting a 2-tailed test of size alpha is 𝑝𝑜𝑤𝑒𝑟=
Φ ൭ቆ𝜃஺ට𝑉𝑎𝑟൫𝜃෠൯
ൗ
ቇ−𝑍ଵିఈଶ
⁄ ൱ where Φ is the cumulative standard normal distribution 
function, 𝑍ଵିఈଶ
⁄  is the (1 −𝛼2
⁄ )𝑡ℎ quantile of the standard normal distribution function 
and 𝜃෠ is the estimated effect size. The estimated number of patients exposed to the 
intervention will be 750, compared to 750 control patients, which will provide sufficient 
power for even modest improvements in imaging rates among low-risk men. Assuming 
10 time periods (Q2-Q11), an estimate for baseline imaging among men with low-risk 
prostate cancer of 40% in the usual care group, and a decrease to 20% guideline-
discordant imaging (absolute difference of -0.20), 10 clusters (sites), 15 patients with 
low-risk prostate cancer per quarter x 10 study sites x 10 quarters is estimated to impact 
1500 total patients out of 6000 patient charts reviewed (a conservative estimate based 
on analysis of VINCI data), an alpha of 0.05, and a coefficient of variation of 0.40, 
accounting for clustering, we would estimate having a power of >0.999. This is a 
conservative estimate in terms of the expected effect of the intervention on prostate 
cancer imaging rates and assumes a high coefficient of variance with outcomes highly 
correlated with site. A more conservative post-intervention rate of imaging of 28.7% 
would reduce power to 0.80. 
Similarly, we estimate that we will have sufficient power to detect increases in 
appropriate imaging among men with high-risk prostate cancer. Assuming 10 time 
periods (Q2-Q11), an estimate for baseline appropriate imaging of 66% in the usual care 
group, and an increase to 86% (for an absolute difference of +0.20), 10 clusters (sites), 5 
at patients with high-risk prostate cancer per quarter (a conservative estimate based on 
analysis of VINCI data), alpha of 0.05, a coefficient of variation of 0.40, we estimate 
power of 0.89. 
We will also perform exploratory, individual-level analyses. If providers consent to 
participate in the study, then these imaging outcomes will be linked to their survey and 
qualitative data.  For non-participants, imaging patterns alone (with no additional data) 
will be analyzed in a de-identified manner. 
Qualitative Analysis 
Qualitative data will add depth and detail, complementing the other findings to explain 
and illustrate quantitative results. In-depth interviews at the conclusion of the study will 
explore providers’ experiences with the intervention and explain the important 
implementation-related domains from CFIR.22 The exploratory nature of this component 
will permit the identification of new ideas and inform the generation of inductive 
hypotheses regarding factors motivating guideline-concordant imaging. We will also ask 
a short series of questions exclusively to all Urology Chiefs and Site PIs (N=20) to 
explore institutional and managerial perspectives of implementation and attitudes 
towards sustainability of the intervention.31 Data gathered will be critical to the plan to 

Version 1  
VA Central IRB Protocol Template – version 10/26/2012 
Page 24 of 33 
 
disseminate the intervention to other VAMCs nationally. We anticipate recruiting a 
subset of approximately 20-30 frontline providers across the 10 participating study sites 
in order to reach theoretical saturation. There are 79 practicing urologists at all 10 sites 
(10 Chiefs and 67 frontline practitioners) so we anticipate no difficulties in reaching our 
recruitment goal. Qualitative results will be analyzed using NVivo software (QSR 
International). NVivo will allow for the organization, coding, and analysis of interview 
transcripts. Results obtained from these triangulated quantitative and qualitative 
components will be integrated,32,33 a key procedure in mixed methods analysis. 
Demographic information will be collected from participants to aid in the analyses.  
Cost Analysis 
Clinical care cost data will be accessed primarily through the Health Economics 
Resource Center (HERC) Average Cost File. HERC has created estimates of the cost of 
all VA health care encounters that have taken place since October 1, 1998. These data 
are accessible approximately 6-8 months following end of the fiscal year. These data will 
allow for comparable standardized prices to be applied across all VHA facilities for all 
follow-up care activities. Cost data will also be obtained from billable private insurance 
claims. VHA’s Medical Care Recovery Program attempts to collect for care performed at 
VHA when a VA user has private insurance. We will identify subjects in our cohort who 
have billable private insurance and flag these subjects for exclusion in sensitivity 
analyses as they may be likely to be more reliant on community care than VA-users 
without private billable insurance. Preliminary analysis in VISN20 identified that out of a 
cohort of 260,743 subjects, 17,141 (6.6%) had billable insurance. This variable is 
available in the CDW. 
We will also estimate the cost of implementing the intervention using methods described 
by Liu et al. in their paper documenting organizational costs for implementing a 
depression care quality improvement intervention into VHA primary care practices.34 
These methods allow for the documentation of organizational costs associated with the 
implementation effort itself, not just the costs associated with changes in patient care, 
whose calculation we describe above. We will collect data on participant hours and 
intervention related activities directly related to the implementation effort. Implementation 
costs are related to travel, clinical informatics, training sessions, conference calls, 
development of training materials, email communication, and any other activities spent 
counseling, training, or supporting personnel at the study sites. Participants will include 
any individual (provider, study staff, administrator) participating in the implementation 
project. Data sources used for estimating costs will include interviews, surveys, e-mail 
communications, project schedules, meeting minutes, project cost records, and 
government salary information. Costs will be estimated based on length of interviews 
and surveys, email word count, time of telephone calls, and government travel budgets 
based on calculated reference ranges for such expenses, as described by Liu et al.34 
Outcomes 
The study’s Primary Outcomes determined from the above data sources will be: 

Version 1  
VA Central IRB Protocol Template – version 10/26/2012 
Page 25 of 33 
 
Specific Aim 1 
1. Facility-level utilization of bone scan or abdominal/pelvic CT or abdominal/pelvic 
MRI among men with newly diagnosed, low-risk prostate cancer. (Inappropriate 
Imaging) 
2. Facility-level utilization of bone scan or abdominal/pelvic CT or abdominal/pelvic 
MRI among men with newly diagnosed, high-risk prostate cancer. (Appropriate 
Imaging) 
Specific Aim 2 
1. Provider-level utilization of bone scan or abdominal/pelvic CT or abdominal/pelvic 
MRI among men with newly diagnosed, low-risk prostate cancer. (Inappropriate 
Imaging) 
2. Provider-level utilization of bone scan or abdominal/pelvic CT or abdominal/pelvic 
MRI among men with newly diagnosed, high-risk prostate cancer. (Appropriate 
Imaging) 
3. Provider attitudes regarding prostate cancer imaging guidelines and the 
behavioral intervention 
Specific Aim 3 
1. Net cost of implementation of physician behavioral intervention VHA. This file will 
be used to quantify facility-level workforce and imaging technology. 
Facility-level descriptors will be obtained from the 2009 VHA Oncology Services 
Survey.35 As part of the Office of Patient Care Services initiative to conduct systematic 
program reviews, Oncology Services conducted a survey of cancer care services in 
VHA. 
Missing Data 
Missing data has the potential to affect the validity of our results. We foresee 4 potential 
sources for missing data and propose strategies to deal with them. 
1. Improper transcription into Oncotrax and CCC: to overcome this, we will train our 
centrally-located, full-time Research Coordinator to identify missing data fields 
and search for this information in CAPRI. We expect the study to follow 60 
patients from 10 sites per quarter, or a manageable 12 patients per business 
day. 
2. Patients seeking care in the private sector through the Veterans Choice Act: 
Fortunately, when the VA pays for imaging tests, they are indicated in CAPRI. 
Outside healthcare providers have to send back the statement to collect payment 
so this information will be manageable to track.36 
3. Patients seeking care in the private sector through Medicare: Dr. Zeliadt found 
67% of veterans with an elevated PSA followed up within VHA in the subsequent 
24 months.37,38 We assume that follow-up will be improved among men with a 

Version 1  
VA Central IRB Protocol Template – version 10/26/2012 
Page 26 of 33 
 
cancer diagnosis already having chosen to undergo prostate biopsy within VHA. 
At the completion of the study, we will request CMS records using VIReC’s CMS-
VA linkage to identify any missing studies performed by a Medicare provider.39 
4. Patients seeking care in the private sector through private insurance: there is no 
reliable way to ensure completeness of these data. We will review CAPRI notes 
for documentation of outside care but may not be able to confirm that none took 
place. This phenomenon should occur with equal frequency in the control and the 
intervention sites and should bias our results to the null. 
 
In spite of our best efforts, it is likely that some data will remain missing. The full 
information maximum likelihood (FIML) method will be used to handle missing data as a 
less stringent assumption of missing-at-random will be made. Simulation studies have 
shown that no other missing data handling method performs better than the FIML 
regardless of the mechanism by which the missing data are generated.40,41 
5.7 Withdrawal of Subjects 
Participants will be able to withdrawal any point if they so choose without any 
consequences. Withdrawal from this study will have no impact on employment at the VA. 
Enrollment status in this study will remain confidential among personnel involved in the 
Research team. However, any data collected up to that point may still be utilized.  
6.0 
Reporting 
  
While this study is less than minimal risk, there is the possibility of loss of data 
confidentiality. Adverse events and serious adverse events are not likely to occur due to 
the nature of this intervention. The intervention does not entail greater than minimal risk 
to either provider participants or their patients. If a reportable event such as an 
unanticipated problem or protocol deviation should occur among the research team, the 
PI will be notified immediately. Research staff will record and document all details 
related to the event. The PI will determine if the event meets criteria to report to IRB, and 
this decision will be documented, signed, and dated by the PI. In the unlikely event that 
an adverse event occurs at one of the intervention sites, the Site Research Assistants 
will report it to the Site PI immediately. The Research Coordinator at each site will be 
responsible for documenting the details of the event, in addition to contacting the main 
site research staff. The Site PI will determine if the event meets criteria to report to the 
IRB. This decision will be documented and the central research study team as well as 
the main PI will be notified. It is unlikely that findings will reveal results that affect 
participants’ health or welfare in this study. There are no safety conditions that trigger 
immediate suspension of the research that we anticipate. 
 

Version 1  
VA Central IRB Protocol Template – version 10/26/2012 
Page 27 of 33 
 
7.0 
Privacy and Confidentiality 
 
There is minimal risk associated for patients involved with this study. This study will use 
patient PHI. It is necessary to have access to PHI in order to determine the rates and 
trends in prostate cancer imaging across the study sites. There is a risk of breach in 
confidentiality; however, multiple safeguards are in place to ensure information security. 
Only members of the study team performing data analysis, the Research Coordinator 
and Project Manager, will have access to central data PHI and patient records.  Only the 
designated member of the LCAC will access the locally generated data reports. A variety 
of precautions will be taken to ensure the security of patient data. PKI-encrypted email 
will be used to transmit patient data. PKI is a VA-approved software used to secure the 
delivery of electronic services to VA employees, contractors, and business partners. 
PKI-encrypted emails are only accessible to those who have been granted access and 
are only accessible on the VA network requiring a VA network ID and password. Patient 
PHI will be stored on password protected, secure sever.   
The VA’s informatics and computing infrastructure (VINCI) will be used to store and 
analyze data. Individual investigators and multiple databases may lack sufficient 
resources to ensure consistency and quality control, or a long-term commitment to data 
storage and access. Therefore, there are less consistent standards for the protection of 
Veterans data, data quality, and data access compared to a centralized repository. A 
centralized research data repository, such as the VA Informatics and Computing 
Infrastructure (VINCI), offers a number of important advantages: Consistent, defined, 
and transparent security and standards for access to data; a common point of entry for 
all investigators who use the data; tools for analysis and reporting; tighter and more 
consistent control over the standards and quality of the data included; and the ability to 
standardize and update terminology and format as technology and methodology 
improve. VINCI is a partnership between the VA Office of Information Technology (OI&T) 
and the Veterans’ Health Administration Office of Research and Development (VHA 
ORD). VINCI provides the storage and server technologies to securely host suites of 
databases integrated from select national data. These servers reside at the Austin 
Information Technology Center (AITC), located in Austin, Texas. To ensure the 
protection of Veterans data, VINCI maintains compliance with the guidelines set forth by 
Veterans Health Administration (VHA) Handbook 1200.12, Use of Data and Data 
Repositories in VHA Research and all other applicable VA and VHA policies and 
regulations. In addition, VINCI has undergone all security certification activities in 
support of obtaining an Authorization to Operate (ATO). Access to VINCI resources will 
be approved in accordance with the requirements of National Data Systems (NDS), VHA 
Handbook 1200.12, Use of Data and Data Repositories in VHA Research, and all other 
applicable VA and VHA policies and regulations. Researchers and Operations staff will 
access the data along with the tools for analysis and reporting in the secure, virtual 
working environment through a certified VHA network computer using the VA 
INTRANET (NOTE:  VINCI is not accessible through the INTERNET).  If not working 
within a VA or VHA hosted office environment containing VA network access, 

Version 1  
VA Central IRB Protocol Template – version 10/26/2012 
Page 28 of 33 
 
researchers may access VINCI through an approved Virtual Private Network (VPN) and 
Remote Desktop application. The remote computing environment will enable data 
analysis to be done directly on VINCI-CDW servers located at the Austin Information 
Technology Center, thus keeping all data from being transmitted to local PC hard drives 
or servers. 
VA REDCap is specifically designed for human subjects research. It is installed and 
accessibly only on VA network, and login requires VA network ID. The application is also 
housed on VINCI server. REDCap is a mature, secure web application for building and 
managing online surveys and databases. The VA REDCap portal is intended to be used 
by authorized VA network users for viewing and retrieving information only. Each 
member of the main study site research staff has their own individual password-
protected account on REDCap that was granted by the VA REDCap support team. It is 
located explicitly through the VA intranet. This study will have a project file that will be 
accessible only to authorized REDCap users on this study team at the main research 
site. VA REDCap login can be accessed  at: https://vhacdwweb05.vha.med.va.gov/. 
Provider privacy will also be maintained. All questionnaires and survey item individual 
responses will not be shared with any personnel beyond the central research staff. The 
responses will be directly entered into the secure REDCap project site. Participating 
providers at each site will have access to their own individual imaging rates in addition to 
aggregate institution results, through a report created by the data team . The main site 
Research staff will send the reports through VA outlook email. Participating providers will 
not be able to view other provider-specific data corresponding with other individual 
participants. In sites that have more than 3 providers, the Site PI will have access to de-
identified individual level rates to improve context, clarity, and goals for the group 
feedback meetings. Individual participants will receive institutional level data reported as 
an aggregate, conditional that there are >3 employed providers in the department to be 
included, in order to protect confidentiality. The Chief of Urology will not be aware of 
which staff members are enrolled, which study activities have been completed, or the 
content of individual responses. Identities of participating individuals for in-depth 
interviews following the intervention will be confidential and will remain within the central 
study research team in order to feasibly conduct the interview. The data collected will be 
analyzed and reported as aggregate for reporting of results and future manuscripts. 
Identities of participants in this portion of the study will not be shared with the Site PIs. 
Chiefs and other participants will not be aware who has been interviewed (other than 
themselves, if they chose to participate). Qualitative data reported will retain anonymity 
of the respondents. The site of the participant will be retained so that the central team 
members are able to analyze themes among sites by imaging volume (for example, “low 
imaging sites” versus “high imaging sites”). This information will only be reported as 
aggregated themes. Participation in this study will not influence or affect employment at 
VA.  
Interviews that have been audio-recorded will be stored on the VA desktop of approved 
study team. Interviews will be recorded during phone call interviews, and will be 
recorded by the VA Microsoft Teams application in real time. Microsoft Teams is a 

Version 1  
VA Central IRB Protocol Template – version 10/26/2012 
Page 29 of 33 
 
collaboration software that combines chat, meetings, calling, collaboration, application 
integration, and file storage into a single interface. We will use the application to create a 
meeting with dial-in information and will record the meeting directly through the 
application on VA secure desktop. Interviews will be de-identified and uploaded onto the 
secure NY Harbor research server and will be made available to the VA HSR&D 
Centralized Transcript Service Program (CTSP), an HSRD-funded initiative for 
qualitative studies at the VA (PI: Dr. Susan Zickmund). Approved staff from the VA Salt 
Lake City (VASLC) will transcribe the audio files. The VASLC has a Professional 
Transcription Service available to VA sites and monitored by their own IRB. The audio 
recordings to be transcribed by VASLC staff will be labeled by the subject's unique 
alphanumeric code and saved behind the VA Firewall in the main study site’s secure 
shared project folder. The VASLC transcription staff will be given access to a sub-folder 
within the study team’s secure project folder to be created especially for this exchange. 
Approved study staff will place a copy of the audio files in this folder for an approved 
VASLC transcriptionist to access for the purposes of transcription. The VASLC 
transcriptionist will transcribe each interview verbatim and save the completed transcript 
in the sub-folder using the same alphanumeric code. No data (audio files, in process 
transcripts, or completed transcripts) will leave the NY Harbor secure research server. 
As completed transcripts become available, approved study staff will move these files 
from the transcription sub-folder into another sub-folder that is only accessible to study 
staff, where they will be stored and accessed for qualitative analyses. We will collect site 
information within the demographic questions to be retained for analysis. This will be 
anonymized when reporting results and will be kept confidential within the NY Harbor 
research team.   
  
Project research data will only be used by authorized study personnel for the exclusive 
purposes of this project. PHI or passwords associated with data for this project will never 
be recorded or shared. Only designated VA research personnel will have access to data 
stored on VINCI, REDCap, and the secure VA research folder. If an individual is no 
longer part of the research team, the present study team will submit a modification 
through Central IRB to remove that individual from the study personnel. All passwords 
will be changed following a change in personnel, and only shared among authorized 
study team members via secure communication. The ISO and Privacy Officer will be 
notified within one hour if improper use or disclosure should occur. 
8.0 
Communication Plan 
 
Communication between the main site and the 10 participating sites is integral to the 
success of this study. As the study sites are spread across the country, it is critical that 
the Site PIs and their study team members stay in contact with the main site. Frequent 
check-ins and open lines of communication will ensure that the study is conducted 
according to the IRB-approved protocol. The main site is committed to helping the local 
sites with any aspect of study startup as well as continuing reviews or any other 
subsequent IRB and regulatory submissions. The main site research team will have 
regular weekly meetings to check-in with the PI, in addition to other research and 
administrative task-specific meetings that will regularly occur.  

Version 1  
VA Central IRB Protocol Template – version 10/26/2012 
Page 30 of 33 
 
The main study PI plans to have biannual meetings with the local Site PIs. One will 
occur in-person while the PIs attend national urologic conferences, such as the 
American Urological Society annual meeting, and another meeting will occur over a 
telephone conference line, approximately 6 months later. 
Local Site PIs have committed to creating Local Clinical Advisory Committees 
comprising the Site PI, Urology Chief, a local IT representative, and the Site Research 
Assistant. Local PIs and Research Assistants are responsible for scheduling and 
maintaining these meetings. These committees will meet quarterly to discuss the 
progress of the intervention. After each Local Clinical Advisory Committee notes and any 
updates will be delivered to the main site research staff. The project manager and 
research staff will call to check-in with the site Research Assistants after each Local 
Clinical Advisory Committee meeting and will set up remote meetings via conference or 
videoconference calls as needed. The central research staff may also participate in 
these LCAC meetings by calling in. 
Once a site has been randomized into the intervention, each participating 
provider will receive a quarterly audit and feedback report from the central 
research staff. This report will contain individual-level imaging data as well as 
data collected on imaging for their institution as a whole.  
Communication between Site PIs and Local Clinical Advisory Committee with 
participating providers at their clinic regarding the progress, status, and results 
are also an integral component to the implementation and effectiveness of the 
intervention components. The Site PI and Site Research Assistant will have 
regular communication via presentations, emails, department meetings and any 
other regular activities that occur at the given site for clinical operations  to 
encourage optimal participation in the project (clinical order check, audit and 
feedback, academic detailing) and to enhance data results. The coordinating 
staff will also document use of these communications to aid in continual 
monitoring of implementation.  
If any changes are made to the protocol or any other documents relating to the 
IRB, a message will be sent to the Site PIs as well as the Site Research 
Assistants.  
 
9.0 
References 
 
1. 
Makarov DV, Sedlander E, Braithwaite RS, et al. A qualitative study to understand 
guideline-discordant use of imaging to stage incident prostate cancer. Implementation 
science : IS. 2016;11(1):118. 

Version 1  
VA Central IRB Protocol Template – version 10/26/2012 
Page 31 of 33 
 
2. 
Makarov DV, Loeb S, Ulmert D, Drevin L, Lambe M, Stattin P. Prostate Cancer Imaging 
Trends After a Nationwide Effort to Discourage Inappropriate Prostate Cancer Imaging. 
Journal of the National Cancer Institute. 2013;105(17):1306-1313. 
3. 
Meredith LS, Yano EM, Hickey SC, Sherman SE. Primary care provider attitudes are 
associated with smoking cessation counseling and referral. Medical care. 
2005;43(9):929-934. 
4. 
Hussey MA, Hughes JP. Design and analysis of stepped wedge cluster randomized 
trials. Contemporary clinical trials. 2007;28(2):182-191. 
5. 
Makarov DV, Sedlander E, Braithwaite RS, et al. “If you’re doing something because it’s 
inexpensive but ineffective it’s still expensive:” A Qualitative Study to Understand 
Inappropriate use of Imaging for Patients with Low Risk Prostate Cancer. 
HSR&D/QUERI National Conference; July 8, 2015, 2015; Philadelphia, PA. 
6. 
Administration VH. Veterans' Diseases Associated with Agent Orange.  
http://www.publichealth.va.gov/exposures/agentorange/conditions/index.asp, 2015. 
7. 
Michie S, Johnston M, Abraham C, Lawton R, Parker D, Walker A. Making psychological 
theory useful for implementing evidence based practice: a consensus approach. Quality 
& safety in health care. 2005;14(1):26-33. 
8. 
O'Brien MA, Rogers S, Jamtvedt G, et al. Educational outreach visits: effects on 
professional practice and health care outcomes. The Cochrane database of systematic 
reviews. 2007(4):CD000409. 
9. 
Ivers N, Jamtvedt G, Flottorp S, et al. Audit and feedback: effects on professional 
practice and healthcare outcomes. The Cochrane database of systematic reviews. 
2012;6:CD000259. 
10. 
Goldzweig CL, Orshansky G, Paige NM, et al. Electronic Health Record-Based 
Interventions for Reducing Inappropriate Imaging in the Clinical Setting: A Systematic 
Review of the Evidence. 
. In: Affairs DoV, ed. Vol #05-226: Quality Enhancement Research Initiative (QUERI); 2014. 
11. 
Gail MH, Mark SD, Carroll RJ, Green SB, Pee D. On design considerations and 
randomization-based inference for community intervention trials. Statistics in medicine. 
1996;15(11):1069-1092. 
12. 
Weiner BJ, Lewis MA, Clauser SB, Stitzenberg KB. In search of synergy: strategies for 
combining interventions at multiple levels. Journal of the National Cancer Institute 
Monographs. 2012;2012(44):34-41. 
13. 
Makarov DV, Desai RA, Yu JB, et al. The population level prevalence and correlates of 
appropriate and inappropriate imaging to stage incident prostate cancer in the medicare 
population. The Journal of urology. 2012;187(1):97-102. 
14. 
Development HSR. Management Brief no. 91: A Systematic Review: Electronic Health 
Record-Based Interventions for Reducing Inappropriate Imaging in the Clinical Setting. 
Management Brief 2015; 
http://www.hsrd.research.va.gov/publications/management_briefs/default.cfm?Managem
entBriefsMenu=eBrief-no91, 2015. 
15. 
Rogers LQ, Bailey JE, Gutin B, et al. Teaching resident physicians to provide exercise 
counseling: a needs assessment. Academic medicine : journal of the Association of 
American Medical Colleges. 2002;77(8):841-844. 
16. 
Avorn J, Soumerai SB. Improving drug-therapy decisions through educational outreach. 
A randomized controlled trial of academically based "detailing". The New England 
journal of medicine. 1983;308(24):1457-1463. 
17. 
Sherman SE, Fotiades J, Rubenstein LV, et al. Teaching systems-based practice to 
primary care physicians to foster routine implementation of evidence-based depression 

Version 1  
VA Central IRB Protocol Template – version 10/26/2012 
Page 32 of 33 
 
care. Academic medicine : journal of the Association of American Medical Colleges. 
2007;82(2):168-175. 
18. 
Longman P. Best care anywhere : why VA health care would work better for everyone. 
3rd ed. San Francisco: Berrett-Koehler Publishers; 2012. 
19. 
Shelton JB, Ochotorena L, Bennett C, et al. Reducing PSA-Based Prostate Cancer 
Screening in Men Aged 75 Years and Older with the Use of Highly Specific 
Computerized Clinical Decision Support. Journal of general internal medicine. 2015. 
20. 
Morse J, Wolfe R, Niehaus L. Principles and procedures of mainting validity for mixed 
method design. In: Curry L, Shield RR, Wetle TT, eds. Improving aging and public health 
research : qualitative and mixed methods. Washington, DC: American Public Health 
Association and Gerontological Society of America; 2006:65-78. 
21. 
Michie S, Atkins L, West R. The Behaviour Change Wheel: A Guide To Designing 
Interventions. In: Silverback Publishing; 2015: http://www.behaviourchangewheel.com/. 
22. 
Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering 
implementation of health services research findings into practice: a consolidated 
framework for advancing implementation science. Implementation science : IS. 
2009;4:50. 
23. 
Rogers ES, Smelson D, Gillespie C, et al. Telephone Smoking-Cessation Counseling for 
Smokers in Mental Health Clinics: A Patient-Randomized Controlled Trial. American 
Journal of Prevention Medicine. In press. 
24. 
Helfrich CD, Li YF, Sharp ND, Sales AE. Organizational readiness to change 
assessment (ORCA): development of an instrument based on the Promoting Action on 
Research in Health Services (PARIHS) framework. Implementation science : IS. 
2009;4:38. 
25. 
Aarons GA. Mental health provider attitudes toward adoption of evidence-based 
practice: the Evidence-Based Practice Attitude Scale (EBPAS). Mental health services 
research. 2004;6(2):61-74. 
26. 
Horner RH, Salentine S, Albin RW. Self-assesment of contextual fit in schools. In: 
Eugene: University of Oregon; 2003. 
27. 
Moore GC, Benbasat I. Development of an Instrument to Measure the Perceptions of 
Adopting an Information Technology Innovation. Information Systems Research. 
1991;2(3):192-222. 
28. 
Steckler A, Goodman RM, McLeroy KR, Davis S, Koch G. Measuring the diffusion of 
innovative health promotion programs. American journal of health promotion : AJHP. 
1992;6(3):214-224. 
29. 
Sise CB, Sise MJ. Organizational rites and culture in a surgical residency program. 
Current surgery. 1989;46(5):365-371. 
30. 
Coke P, Gill T. National Cancer Care Cube. AVAHO Meeting 2014. 
31. 
Brownson RC, Jacobs JA, Tabak RG, Hoehner CM, Stamatakis KA. Designing for 
dissemination among public health researchers: findings from a national survey in the 
United States. American journal of public health. 2013;103(9):1693-1699. 
32. 
Deren S, Oliver-Velez D, Finlinson A, et al. Integrating qualitative and quantitative 
methods: comparing HIV-related risk behaviors among Puerto Rican drug users in 
Puerto Rico and New York. Subst Use Misuse. 2003;38(1):1-24. 
33. 
Moran-Ellis J, Alexander VD, Cronin A, et al. Triangulation and integration: processes, 
claims and implications. Qualitative Research. 2006;6(1):45-59. 
34. 
Liu CF, Rubenstein LV, Kirchner JE, et al. Organizational cost of quality improvement for 
depression care. Health Serv Res. 2009;44(1):225-244. 
35. 
(PCS) PCS. VHA Oncology Services Survey. 2009. 
http://vaww.infoshare.va.gov/sites/MedicalSurgical/oncology/default.aspx. 

Version 1  
VA Central IRB Protocol Template – version 10/26/2012 
Page 33 of 33 
 
36. 
QUERI Evaluates Effects of Veterans Choice Act. QUERI E-news 2015; 
http://www.queri.research.va.gov/qnews/oct15/default.cfm?QnewsMenu=article2&utm_s
ource=qnews&utm_medium=email&utm_campaign=Oct15. Accessed October 30, 2015. 
37. 
Zeliadt SB, Hoffman RM, Etzioni R, Ginger VA, Lin DW. What happens after an elevated 
PSA test: the experience of 13,591 veterans. Journal of general internal medicine. 
2010;25(11):1205-1210. 
38. 
Zeliadt SB, Buist DS, Reid RJ, Grossman DC, Ma J, Etzioni R. Biopsy follow-up of 
prostate-specific antigen tests. American journal of preventive medicine. 2012;42(1):37-
43. 
39. 
VA/CMS Data for Research Project. The Researcher's Guide to VA Data 2015; 
http://www.virec.research.va.gov/VACMS/About.asp. Accessed December 1, 2015, 
2015. 
40. 
Wothke W. Longitudinal and multi-group modeling with missing data. In: T.D. Little KUS, 
& J. Baumert, ed. Modeling longitudinal and multipl group data: Practical issues, applied 
approaches and specific examples. Mahwah, NJ: Lawrence Erlbaum Associates, Inc; 
2000:219-240. 
41. 
Enders C, Bandalos, DL. The relative performance of full information maximum 
likelihood estimation for missing data in structural equation models. Educational 
Psychology Papers and Publications. 2001.