Examining the Effectiveness of an Adaptive Implementation Intervention to Improve Uptake of the VA Suicide Risk Identification Strategy (PEC 19-303)
Phase: NA
Conditions: Suicide
Sponsor: VA Office of Research and Development



Title of project: Examining the Effectiveness of an Adaptive Implementation Intervention to Improve 
Uptake of the VA Suicide Risk Identification Strategy 
 
SPECIFIC AIMS. In the last decade, VA has made significant strides in suicide prevention, particularly for 
Veterans receiving Veterans Health Administration (VHA) care. However, most of these efforts have focused on 
downstream interventions to reduce suicidal behavior among those already identified to be at high risk (e.g., 
High-Risk Flag, post-discharge treatment policies). In contrast, more upstream efforts, such as 
population-based screening for suicide risk, have not been systematically implemented across 
VHA settings. Instead, suicide risk screening and evaluation in VHA has traditionally been limited to select 
patient cohorts or treatment settings (e.g., those with a known psychiatric disorder). However, emerging 
evidence indicates that many individuals who die by suicide are not identified as having a psychiatric disorder 
and often present for nonbehavioral health care prior to their death2-4. Given that early and accurate detection 
of suicide risk among all Veterans presenting for VHA care is a critical component of VA’s National Strategy 
for Preventing Veteran Suicide 2018-2028 8, VHA leadership mandated implementation of a national suicide 
risk identification strategy, VA Risk ID, beginning October 1, 2018. The goal of VA Risk ID is to improve the 
detection and management of suicide risk by standardizing suicide risk screening and evaluation enterprise-
wide. VA Risk ID incorporates staged evidence-based/informed tools and processes. The three stages include 
two levels of screening, followed by a Comprehensive Suicide Risk Evaluation (CSRE). VA Risk ID moves 
beyond selective screening and evaluation (i.e., based on known risk factors) and expands these practices to a 
broader population of patients presenting to a wide variety of medical settings.  
VA Risk ID is the largest implementation of population-based suicide risk screening and 
evaluation in any U.S. healthcare system to date. Given the considerable scope of this initiative, several 
strategies have been employed to support national implementation: informatic tools, educational webinars, 
facility champions, technical assistance, and clinical performance measures to monitor implementation of VA 
Risk ID to fidelity. Despite these efforts, some facilities will face challenges to implementation. In order to 
facilitate continuous quality improvement, ongoing evaluation of VA Risk ID and interventions to improve 
implementation of the three-stage process are needed. In general, multi-faceted implementation strategies 
have been shown to be most effective; however, consideration of when and for which locations certain 
implementation strategies should be provided is also necessary. For example, some facilities (i.e., early 
adopters) may not need additional intervention. Moreover, among facilities that require additional 
intervention, the dose and type of intervention needed may vary. Therefore, creating an adaptive 
implementation strategy that provides different degrees of implementation support in a step-wise fashion is 
expected to be a practical and efficient way of improving implementation of VA Risk ID to fidelity. Specifically, 
this approach will allow the Office of Mental Health and Suicide Prevention (OMHSP) to focus limited 
resources where necessary. 
Thus, the objective of this national quality improvement project is to develop an adaptive 
strategy to improve implementation of VA Risk ID to fidelity. Using a sequential multiple assignment 
randomized trial (SMART) design, two evidence-based implementation strategies will be evaluated: 1) Audit 
and Feedback (A/F); 2) A/F plus External Facilitation (A/F+EF). We will evaluate these strategies across 
several domains based on the RE-AIM QuEST mixed methods framework9. In addition to focusing on 
implementation outcomes, such as the impact of these strategies on implementation of VA Risk ID to fidelity, 
and potential barriers to adopting these strategies, we will also examine the impact of the clinical innovation 
(i.e., VA Risk ID) on patient-level outcomes. Primary Aim: Among sites that do not meet the benchmark for 
adequate performance (i.e., completion of secondary screening and CSRE for 80% or more of eligible patients) 
following 6 months of Implementation as Usual (IAU), does the addition of A/F for 10 months significantly 
improve scores on VA Risk ID performance measures compared to IAU alone? Secondary Aims: 1) Among 
sites that continue to not meet the benchmark for adequate implementation after 10 months of A/F, does the 
addition of EF significantly improve scores on VA Risk ID performance measures compared to A/F alone? 2) 
Among sites that meet the benchmark following A/F alone, is performance maintained following 
discontinuation of A/F? Outcomes for primary and secondary aims will use clinical performance measures 
extracted from VA’s External Peer Review Program (EPRP) that reflect key practice elements of VA Risk ID: 
completion of the primary suicide screen, timely completion of the secondary suicide screen and timely 
completion of the CSRE. Exploratory Aims: 1) Examine the extent to which VA Risk ID is reaching the 
intended population and compare the characteristics of Veterans reached to all Veterans eligible for screening; 
2) Examine contextual factors that may impact the a) implementation of VA Risk ID to fidelity and b) adoption 
of the implementation interventions. 3) Examine whether Veterans who receive the CSRE are more likely to 
receive a safety plan than Veterans who screen positive on the primary screen only. This project directly 

addresses VA’s highest priority of reducing Veteran suicide. Given its focus on developing and 
evaluating the impact of an adaptive strategy on implementation of VA Risk ID, this project is 
well-suited for a QUERI Partnered Evaluation.  
 
Research Plan 
A. BACKGROUND  
Screening and evaluation of suicide risk in VHA has been limited primarily to specific at-
risk patient populations. Reducing Veteran suicide is VA’s highest priority. Over the past decade VA has 
made significant strides towards this mission, particularly for Veterans receiving VHA care. Until recently, 
these efforts have largely relied on downstream programs or policies focused on improving suicide risk 
management among those already identified to be at elevated risk. For example, screening and assessment of 
suicide risk has predominately occurred in specific patient cohorts (e.g., those who screen positive for 
depression) or clinical settings (e.g., inpatient mental health). In contrast, more upstream programs such as 
population based suicide risk screening have not been systematically implemented across VHA settings.  
Many individuals who die by suicide present to nonbehavioral health care prior to their 
death. In 2016, the Joint Commission released a Sentinel Event Alert1 that prompted a shift in how healthcare 
systems approach the detection and management of suicide risk. This alert highlighted findings that a 
significant number of individuals who die by suicide were not identified as psychiatric patients nor were they 
receiving mental health care. Instead, such individuals were often seen in primary care, ED or other medical 
settings in the year and months before their death2-4. These findings underscore the importance of suicide risk 
screening and evaluation across hospital settings to identify patients with occult risk—those who may only 
disclose suicidal thoughts/behaviors if they are asked directly10. Hospital wide suicide risk screening in both 
VHA and non-VHA healthcare systems, however, has not been routinely implemented.  
 
VA Suicide Risk Identification Strategy is an enterprise-wide, population-based approach 
to detecting suicide risk among Veterans seeking VHA care. To address this gap, the Office of Mental 
Health and Suicide Prevention (OMHSP) established an interdisciplinary workgroup of subject matter experts 
to identify an evidence-informed, population-based approach to detect suicide risk among patients presenting 
to a wide range of healthcare settings. This resulted in the development of the VA Suicide Risk Identification 
Strategy (VA Risk ID). VA Risk ID is a critical first step towards a unified strategy to improve the detection and 
management of suicide risk among all Veterans presenting to VHA care including those: a) with 
unrecognized risk (universal screening); b) at-risk based on a specific diagnosis or condition (selected 
screening); and c) at elevated risk due to acute psychiatric distress (indicated screening). VA Risk ID is the 
largest implementation of population based suicide risk screening in any U.S. healthcare system to date.  
VA Risk ID presents an important opportunity to identify Veterans with unrecognized 
risk who present to general medical settings. In May of 2018, OMHSP released informational guidance 
regarding VA Risk ID, including setting-specific minimum requirements. As outlined, a key focus of the initial 
phase of implementation is identifying suicide risk among the cohort of patients who are eligible for annual 
depression and as required PTSD screening. This cohort of patients comprises approximately 76.2% of all 
Veterans receiving VHA care and represents an important opportunity to identify Veterans with unrecognized 
risk that may present to a wide range of ambulatory 
care settings, including primary care. The stages of 
VA Risk ID and their application to this cohort of 
patients are depicted in Figure 1 and are as follows: 
a primary suicide screen (i.e., Patient Health 
Questionnaire- 9 [PHQ-95], item-9 [i9]), a 
secondary suicide screen (i.e., Columbia Suicide 
Severity Rating Scale [C-SSRS6] Screener), and a 
comprehensive suicide risk evaluation (CSRE) 
standardized template.  
Several implementation strategies have been employed to support national 
implementation of VA Risk ID. To date, a multi-faceted approach has been used to support 
implementation of the three-stage screening and evaluation process. Strategies have included critical 
information technology enhancements (i.e., new informatics tools and clinical reminder updates), educational 
webinars, facility champions, weekly technical assistance calls focused on troubleshooting and building a 
community of practice, support email address, and a fallout report dashboard to help facilities track incomplete 
secondary screens and CSREs. Despite these efforts, some facilities will face challenges to implementation. To 

facilitate continuous quality improvement (QI) of VA Risk ID, ongoing evaluation of VA Risk ID and 
interventions to improve implementation are needed. 
Performance measurement is a critical component of evaluation and ongoing QI. A critical 
element of any QI program is the ability to reliably measure performance. The VA has been a pioneer when it 
comes to measuring performance as an essential mechanism in improving quality of care11. The External Peer 
Review Program (EPRP12) is the official data source for monitoring facility performance, which is used to 
prioritize the quality areas needing most attention. Moreover, there is evidence to suggest that facility 
performance measurement significantly improves overall quality of care across various conditions13-14. In 
preparing for VA Risk ID implementation, several clinical performance measures that map onto the 3-stage 
screening and evaluation process for those eligible for depression and PTSD screening were developed. These 
measures provide a standardized way of monitoring implementation of VA Risk ID to fidelity across VHA 
facilities and include: sui40 and sui51- completion of primary suicide screen as part of depression and PTSD 
screening, respectively; sui2- same day completion of the secondary screen following a positive primary 
screen; and csra1- same day completion of the CSRE following a positive secondary screen.  
Another benefit of having standardized and reliable VA Risk ID clinical performance measures is that 
they can be used to facilitate QI interventions, such as audit and feedback (A/F), particularly for sites that do 
not demonstrate adequate implementation with standard implementation support. A/F is defined as “any 
summary of clinical performance of health care over a specified period of time aimed at providing information 
to health professionals to allow them to assess and adjust their performance”15. A/F interventions can be a cost-
effective way of using EPRP performance measures to improve quality of care and reduce variability across 
facilities.  
Theoretically informed models of A/F can enhance the effectiveness of A/F interventions. 
A/F has shown small to moderate, yet worthwhile improvements in performance, and some studies have 
demonstrated large effect sizes15. Recent studies suggest that conceptualizing A/F within a theoretical 
framework may help identify ways to improve the effectiveness of A/F interventions. For example, theories of 
behavior change are potentially useful for optimizing A/F by pointing to behavior change techniques that may 
augment the effectiveness of receiving feedback about clinical practice16. Hysong and colleagues17 developed a 
model of actionable feedback, which is rooted in Feedback Intervention Theory18 (FIT). According to the model 
of actionable feedback17, three cues (timeliness, individualization, and non-punitiveness) presented in 
hierarchical order are necessary prerequisites to effective feedback and provide increased meaning to the 
feedback (e.g., is it timely and constructive), making the feedback more actionable. A fourth cue, 
customizability, while not a prerequisite, may significantly enhance the actionability of the feedback17. Overall, 
research has shown that the effects of A/F are maximized when initial performance is low and feedback is non-
punitive, provided frequently, and includes specific targets and suggested actions15,19, providing empirical 
support for the actionable feedback model11,15,19-20.  
 
Facilitation is another theoretically informed, evidence-based implementation strategy 
that may help improve uptake of VA Risk ID but requires more resources than A/F. In a 
systematic review outlining future directions for optimizing A/F, Ivers and colleagues20 highlighted the 
potential of combining A/F with coaching and facilitation to help providers move from reactions to their data 
towards planning for change. The use of facilitation as an implementation strategy has been steadily increasing 
in VHA and research suggests that implementation facilitation (IF) holds promise in improving the adoption of 
programs and practices especially among sites and settings that experience significant implementation 
barriers21-23. Briefly, IF is “a multi-faceted process of enabling and supporting individuals, groups and 
organizations in their efforts to adopt and incorporate clinical innovations into routine practices”24. This can 
include problem solving and support that occurs in the context of a recognized need for improvement or it can 
address a range of implementation challenges through other implementation strategies24. Like A/F, facilitation 
is likely to be more impactful when its application is driven by an implementation framework (e.g., integrated 
Promoting Action on Research Implementation in Health Services [i-PARIHS] framework25). Such models can 
help guide thinking about how to apply IF in a particular implementation effort24.  Though more intensive 
strategies, such as facilitation, could certainly enhance the effectiveness of A/F, they may also increase the 
intensity and cost of the intervention.  
An adaptive implementation strategy, grounded in the i-PARIHS framework, whereby 
A/F is augmented with EF when needed, may help improve uptake of VA Risk ID for low 
performing sites while focusing resources. According to the i-PARIHS framework, successful 
implementation is a function of context, innovation and recipients26. Consistent with this framework, 
successful implementation of VA Risk ID (the innovation) depends on changing multiple behaviors of multiple 
types of people (e.g., health professionals, managers, administrators – the recipients) in the complex context of 

a busy medical center setting. Behavior change is complicated, which is why multifaceted approaches that 
incorporate a variety of implementation strategies are needed to address the range of barriers that can impact 
implementation. However, consideration of when and for which facilities certain strategies should be provided 
is also necessary. Some facilities (i.e., early adopters) may not need additional intervention. Moreover, among 
facilities that require additional intervention, the dose and type of intervention needed may vary. Therefore, 
creating an adaptive implementation strategy that provides different degrees of implementation support in a 
step-wise fashion is expected to be an efficient way of improving uptake of VA Risk ID. Drawing on both the 
actionable feedback model17 and i-PARIHS framework25 we propose to test whether a staged implementation 
approach consisting of A/F followed by augmentation with EF improves uptake of VA Risk ID for facilities that 
continue to demonstrate low uptake with A/F alone. The rationale for starting with A/F as a first line 
intervention is that it is a relatively low-intensity/low-cost strategy. Because the proposed A/F intervention 
would be based on existing EPRP clinical performance measures, no additional data collection is necessary 
making it more feasible to implement on a larger scale. EF, on the other hand, requires more resources. Thus, 
beginning with a less resource intensive intervention for those who initially do not meet benchmarks and 
augmenting with a more resource intensive intervention to address barriers among sites that continue to 
perform below expectations may be a more strategic and cost-effective approach to supporting implementation 
of VA Risk ID.   
A2. Significance. This project directly addresses VA’s highest priority of reducing Veteran suicide. VA 
Risk ID represents the largest population based suicide screening and evaluation initiative in any U.S. 
healthcare system to date. By incorporating suicide screening and evaluation enterprise-wide across patient 
populations, including those with unrecognized risk as well as those with known risk, VA Risk ID embodies 
universal, selected and indicated screening and evaluation approaches that cover a wide spectrum of patients 
presenting to VHA care. This in turn will help facilitate a deeper understanding of suicide prevention from a 
population health perspective. In this vein, implementation of VA Risk ID is consistent with the paradigm shift 
proposed by the Joint Commission in the Sentinel Alert 561, which calls for healthcare systems to move beyond 
screening and evaluation of individuals with known risk. Moreover, both the practices that encompass VA Risk 
ID and the evaluation of VA Risk ID outlined in this proposal directly align with the changes to National 
Patient Safety Goal (NPSG) 157: Identifying Patients at Risk for Suicide, which will go into effect July 1, 2019. 
A3. Clinical & Implementation Innovation. VHA is the largest integrated healthcare system in the 
country and has been a pioneer in the field of suicide prevention over the last decade. VA Risk ID is further 
evidence of VA leading suicide prevention in the U.S., which includes the ability to move upstream to improve 
earlier and more accurate detection of suicide risk in all Veterans presenting to VHA care. The evaluation 
and QI approach is also innovative because it encompasses many of the foundational elements of a 
continuously learning healthcare system26 striving to deliver quality care including: a) digital capture of the 
care experience (i.e., informatic tools to facilitate implementation and health factors to facilitate evaluation); b) 
measurement (i.e., clinical performance measures); c) clinical decision support (i.e., clinical reminders to 
facilitate screening and CSRE template with built in education and support); d) optimized operations (i.e., use 
of an adaptive implementation strategy for QI while focusing resources); and e) performance transparency (i.e., 
fallout dashboard and A/F to provide timely feedback about performance and areas for improvement). This is 
also the first QI project to apply a SMART design (see Design & Setting below) to improve implementation of a 
VHA-wide suicide prevention initiative. 
 
B. RESEARCH OVERLAP 
We conducted a thorough review of research at VA, National Institute of Health, and other funding 
sources and identified no ongoing projects with significant research overlap. While we identified 
some federally funded suicide risk screening studies focused on specific at-risk patients, we did not identify any 
VA QUERI or HSR&D funded studies seeking to evaluate national implementation of population level suicide 
risk identification practices. Overlap with planned studies: This project aligns with a HSRD proposal, 
“Understanding Impact of VHA’s New Suicidal Ideation Screening Initiative: Veteran’s Perspective” that is 
being submitted in December 2018 (PI: Dobscha). The HSRD project will complement the evaluation and QI 
efforts of this proposal by gathering data directly from patients regarding their experience with VA Risk ID. Dr. 
Dobscha is PI of the HSRD project and will serve as co-I on the QUERI Partnered evaluation.  
 
B. METHODS 
C1. Evaluation Framework. RE-AIM QuEST9 will provide an overarching framework for testing the 
impact of the proposed adaptive implementation strategy. RE-AIM QuEST expands RE-AIM by allowing 
investigators to understand not only whether Reach, Effectiveness, Adoption, Implementation and 

Maintenance vary across sites and conditions, but also how and why these variations occur9. Moreover, RE-
AIM was selected because of its ability to provide an approach to evaluation that focuses both on the impact of 
the implementation interventions and the impact of the clinical innovation (Table 1). 
 
 
Table 1. Application of RE-AIM, Level of Evaluation, Operationalization, and Data Sources  
RE-AIM Domain 
 
Operationalization 
Data Sources  
Level of Evaluation: Clinical Innovation 
Reach  
The absolute number and representativeness of Veterans that received 
the primary, secondary screens and CSRE.  
Administrative (CDW) Data 
Effectiveness  
Whether Veterans who receive the CSRE are more likely to receive a 
safety plan (SP) compared to Veterans who screen positive on the 
primary screen only. 
Administrative (CDW) data 
Implementation  
 
Percentage of eligible Veterans sampled at each facility who receive the 
different stages of VA Risk ID as intended; barriers & facilitators to 
implementation to fidelity.  
EPRP measures 
Level of Evaluation: Implementation Strategy 
Effectiveness  
Effect of A/F intervention on implementation of VA Risk ID to fidelity 
compared to IAU alone. Effect of A/F + EF intervention on 
implementation of VA Risk ID to fidelity compared to A/F alone. 
EPRP measures 
Adoption  
Number of sites randomized to the implementation interventions that 
participated. Characteristics of participating/non-participating sites 
and reasons for participating/not participating. 
Key Informant (KI) & 
Debriefing Interviews 
Implementation  
 
Percent of sampled instances of implementation intervention delivered 
to fidelity (i.e., met criteria for adherence). 
Fidelity Checklists; KI & 
Debriefing Interviews 
Maintenance 
Maintenance of adequate implementation following removal of A/F; 
Sustained implementation of VA Risk ID for high performers  
EPRP measures 
a All domains will be evaluated at Baseline, Post-Intervention Phase I and II except for maintenance, which will be evaluated at post-
intervention phase I and II 
 
 C2. Design. a. Primary and secondary aims (Effectiveness and Maintenance of the implementation  
strategy) will be evaluated using a sequential multiple assignment randomized trial (SMART) design. This 
project will occur over three phases: Run-In Phase, Intervention Phase I and Intervention Phase II. The unit of 
intervention is the site and randomization will occur at the site level. Three adaptive interventions (AIs) 
are embedded in the SMART design: 1. Begin with IAU, and after six months, continue IAU if the site 
meets the pre-determined benchmark for adequate implementation. If the site does not meet the benchmark 
for adequate implementation: a) do nothing and continue to monitor or b) add A/F for 10 months as a first 
stage intervention; 2. For sites that do not meet the benchmark for adequate implementation, first intervene 
with A/F only for 10 months. Then, if the site continues to not meet benchmark for adequate implementation: 
a) continue A/F alone for another 10 months or b) augment A/F by initiating a combination strategy of A/F + 

external facilitation (A/F + EF) for 10 months; 3. For sites that meet the benchmark for adequate 
implementation after the first stage intervention (i.e., A/F for 10 months): a) continue providing A/F or b) 
discontinue A/F and go back to IAU for another 10 months. Up to 14o VHA facilities will be included in this QI 
project. Facilities will be designated into 1 of 6 groups based on performance and randomization, which will 
dictate how they move through the study phases and the sequence of interventions received: 1. High 
Performers; Continued IAU & Monitoring;2. Improved Performers; Continued A/F; 3. Improved Performers; 
A/F then step-down to IAU; 4. Low Performers; Continued A/F; 5. Low Performers; A/F + EF; 6. Low 
Performers; Continued IAU & Monitoring. b. Exploratory Aims: We will employ a mixed-methods approach 
to evaluate additional implementation and clinical outcomes addressed in the exploratory aims.  
C3. Data Sources. a. VA Risk ID Clinical Performance Measures: Outcomes for primary and 
secondary aims will include four clinical performance measures extracted from VA’s EPRP that reflect key 
practice elements of VA Risk ID: primary suicide screening (sui40 and sui51), secondary suicide screening (sui 
2) and the CSRE (csra1). These four measures are currently in pilot status as they are being validated. EPRP 
measures are routinely abstracted monthly. VA Risk ID EPRP measures at baseline evaluation (month 6), 
month 10 of Intervention Phase I and month 10 of Intervention Phase II will be used to evaluate the 
effectiveness of the implementation strategies (primary and secondary aims). A specific advantage of using 
EPRP performance measures is that the actual abstraction and collection of data occurs independently from 
the evaluation team and program office. A data use agreement will be obtained to facilitate the use of EPRP 
data for this QI project. VA Risk ID EPRP Validation Process: VHA and EPRP contractors develop the data 
collection instruments (abstraction tools), which are programmed into abstraction software. Data are collected 
monthly by the abstractors for a random sample of Veterans and the contractor monitors for irregular 
abstraction patterns. Once final quarterly data are received, question level response patterns and scoring 
algorithms may be reviewed. Inter-rater reliability (IRR) on the measures will also be reviewed at least once a 
year. Upon the IRR review, additional edits may be made to the data collection instrument. Once the measure 
and instrument are considered stable and the data is considered reliable, they can move out of a pilot 
status. We anticipate that by the start of this project, these measures will be out of pilot status as they will have 
gone through several rounds of testing and revisions. If we encounter unanticipated and significant challenges 
to validation of EPRP VA Risk ID measures, we will use the RE-AIM domain of “reach” as our primary outcome 
measure.  b. Corporate Data Warehouse (CDW): CDW is a database organized into a collection of data 
domains derived from the VHA electronic health record. It contains records of inpatient and outpatient care 
including dates and location of care, as well as associated ICD-10 codes. We will use electronically abstracted 
data to evaluate reach and effectiveness of the clinical innovation. For Exploratory Aim 1, specific sources of 
CDW data will include: 1) patient socio-demographic characteristics that allow for comparison between 
Veterans reached and all Veterans eligible for screening 2) patient responses and results from the screens for 
Depression, PTSD, and C-SSRS that are administered within Mental Health Assistant; 3) note templates 
containing the CSRE; and 4) Health Factors recording responses to the CSRE and date of CSRE 
administration. These sources of data will also serve as data sources for Exploratory Aim 3. Additionally, we 
will gather health factors specific to the suicide prevention safety plan details and dates. Free-text progress 
notes may also be abstracted if needed. c. IAU, A/F and EF Debriefing Interviews: Semi-structured 
interviews will be conducted by an independent evaluator monthly with study team members implementing 
IAU during the three project phases, A/F during Intervention Phases I and II and EF during Intervention 
Phase II. The purpose of the interviews is to gather information regarding implementation activities, program 
and implementation modifications, and implementation barriers and facilitators. The independent evaluator 
will also use this information to complete the A/F and EF Fidelity Checklists (see below). Draft interview 
guides are included in Appendix 3. d. Key Informant Interviews: Semi-structured interviews will be 
conducted with primary care (PC) and primary care mental health integration (PCMHI) leadership and 
providers after each of the three project phases via telephone. Interviews will explore barriers/facilitators to 
implementation as well as factors that impact the adoption and utility of the implementation interventions. All 
interviews will be conducted by the independent evaluator to reduce potential bias. Interviews will be audio 
recorded and recordings will be transcribed. Draft interview guides for each project phase are included in 
Appendix 3. e. IAU and EF Activity Logs; A/F and EF Adoption Logs: The IAU Activity Log contains the 
components of IAU and will be completed monthly throughout each study phase by the independent evaluator. 
The EF Activity Log24 will be completed by facilitators weekly to track the facilitation activities completed for 
each site receiving facilitation. The A/F and EF Adoption Logs will be completed weekly by the study team 
member responsible for sending out A/F and EF communications. These data will be used to determine 
whether sites adopted the implementation interventions. See Appendix 4 for draft logs. f. A/F and EF 
Fidelity Checklists: Fidelity checklists will be used to ensure that A/F and EF were delivered as intended. 

The A/F Fidelity Checklist is based on A/F best practices and will be completed during each month of 
Intervention Phases I and II by the independent evaluator. To complete the checklist, the evaluator will review 
a random sample of 25% of sites receiving A/F. If a best practice was not completed, the evaluator will contact 
the A/F team to inquire why. At the end of Phase II, the evaluator will use information gathered in the 
qualitative interviews and EF Activity Log to complete the EF checklist, which is based on core IF activities 
identified through a scoping review and rigorous consensus process27 conducted by QUERI Investigators. 
Those Investigators plan to pilot a fidelity checklist in early 2019, which will inform the final refinement of our 
EF Fidelity Checklist. Draft checklists are included in Appendix 5. 
C4. Site Selection & Recruitment. a. Site Selection: Up to 140 VAMCs across the country will be 
recruited to participate in the SMART. Sites will be allocated to various interventions based on performance 
(i.e., pre-determined benchmarks for adequate implementation- see C6). b. Patient Cohort: Although no 
patients will be recruited for this project, patient level data from EMR will be used for outcomes. The patient 
cohort of interest for this project are all Veterans eligible for annual depression and as required PTSD 
screening. c. Key Informant Interviews: A purposive sampling approach will be used to recruit up to 20 
sites that differ based on geographical location, facility complexity level, and performance level. Interviews will 
occur at three time points: Baseline, Phase I and II evaluation. Phase I and II evaluations will also include sites 
who received A/F and A/F + EF, respectively. Up to 6 types of staff/providers from these sites will be invited to 
participate in the interviews: PC and PCMHI leadership, PC Group Practice Manager, PC Physician, Nursing 
Staff, and a PCMHI mental health provider.  
C5. Randomization. a. Primary Tailoring Variable (PTV): The PTV determines the set of 
randomized intervention options. Thus, the PTV in any SMART must be a proximal measure of the outcome 
that denotes early success or failure and can be implemented in practice28. In this SMART, the PTV is a pre-
determined benchmark for adequate implementation. This benchmark will be based on C-SSRS Screener 
fallouts and CSRE fallouts. The cut-off for both must be 80% or higher (i.e., less than 20% fallouts) to be 
considered adequate implementation. This cut-off was chosen with input from the program partner and 
previous QI projects with the program office that involved setting implementation benchmarks (i.e., REACH 
VET implementation). This benchmark is meant to represent a performance target that all facilities are 
expected to work towards. Preliminary data indicates that, across facilities since October 2018, the 1st quartile 
of C-SSRS fallouts is 47.9% (range: 12.9% - 87.4%), and the 1st quartile of the CSRE fallouts is 61.2% (range: 
12.5% - 100%). 75% of sites currently have C-SSRS and CSRE fallout percentages greater than 47.9% and 
61.2%, respectively. Based on these numbers at least 80-90% of sites currently do not meet criteria for 
adequate performance. We expect that this number to decrease some as implementation continues, but we 
expect that many sites will still have significant room for improvement when this project begins. The program 
partner may adjust this benchmark based on additional data gathered.  b. Randomization procedures: 
Randomization will occur at the site level. VAMCs that are not meeting the benchmark for adequate 
implementation (See PVT above) at Baseline month 6 will be randomized 1:1 (R1) to receive A/F or continued 
IAU for 10 months. A/F sites that do not meet the benchmark for adequate implementation at the 10th 
interventional month will be randomized 1:1 (R2) to either continue A/F or augmentation with EF (A/F + EF) 
for an additional 10 months. A/F sites that meet the criteria for adequate implementation will be randomized 
1:1 (R3) to either continue A/F or discontinue A/F and return to IAU.  R1 will be stratified by facility 
complexity level (i.e., high, medium and low) and level of performance (i.e., low or moderate) at baseline. The 
Facility Complexity Model29 classifies VA medical facilities at levels 1a, 1b, 1c, 2, or 3. Level-1a facilities are the 
most complex and level-3 facilities are the least complex. Levels 1a-1c will be considered high complexity, level 
2- medium complexity and level 3- low complexity. Approximately 61% of facilities (N=85) are high 
complexity, 17% (N=24) are moderate and 22% (N=31) are low complexity. Additionally, the average of the C-
SSRS Screener and CSRE fallout rates for the 6th Baseline month will be calculated for each facility to be 
randomized (R1) and the median of these averages will be used to determine stratification by level of 
performance (i.e., low – above median fallout rate, or moderate – below median fallout rate). R2 will also be 
stratified by facility complexity level and level of performance (based on the median of the fallout rate at the 
10th month of phase 1). R3 will be stratified only by facility complexity level as all sites will have met the 
benchmark. These procedures will ensure that intervention groups are balanced for site variables that may 
correlate highly with outcomes. If less than 12 sites will be randomized at R2, stratification will only occur 
based on facility complexity, unless all facilities are at the same level of complexity in which case only median 
level of performance will be used for stratification.  
C6. Implementation Strategies/Interventions. a. Implementation As Usual (IAU): Consistent 
with the Evidence-Based System for Innovation Support Logic Model,30 we have combined tools, training and 
technical assistance (TA) with a quality assurance measure to develop a robust support system for 

implementation. VA Risk ID IAU was made available to all VHA facilities starting in July 2018. Each facility 
was required to identify a Facility Champion, who accesses the implementation support system and 
disseminates information regarding tools, training, technical assistance and quality assurance to their facility. 
Proactive TA is delivered via weekly conference calls and a support email address. These resources facilitate 
proactive, rather than reactive, problem-solving with the field .31-32 When indicated, we also offer phone calls 
with individual providers or teams to collaboratively identify flexible implementation solutions that will allow 
fidelity to be maintained, a common implementation challenge32. IAU also includes a SharePoint site which 
houses a variety of tools developed for VA Risk ID (e.g., guidance documents, education about VA Risk ID, 
Discussion Board to encourage a community of learning). Our team also conducted a three-part webinar series, 
offering training on the overall strategy and practice components. These live webinars were converted into 
trainings in the VA Talent Management System (TMS) so that employees could access them after the live 
webinar series. Finally, a fallout report was developed for quality assurance. This report provides 
information about patients who did not receive indicated levels of the screening and evaluation process (e.g., 
secondary screen not completed after a positive primary screen). b. Audit & Feedback (A/F): A/F will serve 
as the first stage implementation intervention for sites that do not meet the benchmark for adequate 
implementation following 6 months of IAU. The A/F intervention will be guided by the model of actionable 
feedback17 and other identified best practices20. The following components will be incorporated into the A/F 
design and delivery: i) individualized performance data (site level), ii) frequent delivery intervals (monthly), iii) 
comparisons with other sites, iv) graphical and text form displays of information, v) constructive, non-punitive 
tone, vi) target performance or benchmark provided, and vii) specific actions or recommendations for meeting 
the target. The audit component will include the clinical performance measures for the different VA Risk ID 
practices. These measures will be gathered monthly along with more detailed information from the fall out 
reports. A/F reports will be delivered monthly to the PC group practice manager and local manager with 
oversight of PCMHI. Various prototypes will be user-tested during the Run-In Phase and feedback from 
potential end users will be obtained to develop a final prototype for use during the Phase I intervention.  
Further refinements may be made to the prototype and other aspects of intervention delivery based on 
feedback obtained from sites receiving the A/F intervention during Phase I evaluation. Dr. Sylvia Hysong, 
nationally recognized expert in A/F interventions, will provide ongoing consultation and guidance on the A/F 
intervention for this project. c. External Facilitation (EF): The facilitation approach proposed for this study 
is grounded in the i- PARIHS framework25 described above. The recipients of VA Risk ID (i.e., the 
innovation) include providers, teams and local leadership. Facilitation will involve gaining an understanding 
of these recipients’ key characteristics such as: motivation, values and beliefs, goals, skills and knowledge, time, 
resources and support, and power and authority25. Facilitators will also learn about the inner and outer 
contexts of the recipients. The inner context includes the immediate setting for implementation (e.g., the 
clinic in which they work) and organization in which the clinic is embedded (e.g., VA medical center). The outer 
context is the VHA and the related policies, regulatory frameworks and political environment that impact its 
functioning25. Facilitation will be conducted by a team of facilitators who will employ the facilitation process (a 
set of strategies and actions) to improve uptake of VA Risk ID by the recipients. Given that it is unlikely that 
sites have an internal facilitator who is well-versed in implementation knowledge and skills24,33, we will utilize 
EF. External facilitators will be trained and mentored by an expert facilitator (Dr. Katherine Dollar) utilizing a 
QUERI-supported manual and two-day in-person training developed by Dr. Dollar and colleagues to guide IF 
efforts in VHA24. EF will occur with A/F during Intervention Phase II. To ensure successful implementation, 
external facilitators will flexibly deliver the facilitation process utilizing an integrated set of implementation 
strategies (e.g., stakeholder engagement, identification of barriers and facilitators, education, problem solving, 
the use of formative data, communication, and ongoing support).  
C7. Analyses. a. Sample Size and Power Calculations: Power was calculated using the two-sample t-
test procedure in Power and Sample Size (PASS 16). As there are two primary EPRP outcomes (sui2 and csra1), 
alpha is set to 0.025. The FY2018 first quarter numbers for a similar EPRP (sre1, timely SRE if positive PTSD 
or MDD screen) across 140 facilities was used to obtain an estimated standard deviation for sui2 and csra1 of 
0.10. Power was calculated for two scenarios given the preliminary fallout data noted above. Assuming 
proportions of 0.8 and 0.9 not adequately implementing at month 6 of the run-in phase such that N=112 and 
N=126 are eligible for randomization at R1, this would provide 80% power to detect a difference in change 
between A/F and IAU of 0.059, and 0.055, respectively. Regarding secondary aim 1, assuming alpha=0.05, 
SD=0.10, and facilities eligible for randomization at R2 of 56 and 63, there is 80% power to detect a difference 
in change between continued A/F and A/F + Facilitation of 0.076 and 0.072, respectively. Given the same 
assumptions for secondary aim 2 as used for secondary aim 1, the detectable differences in change are the same 
for examining continued A/F versus discontinued A/F.  b. Primary Aim: Using linear regression, the primary 

EPRP outcomes of change in sui2 and csra1 from the 6th month of the baseline period to the 10th month of the 
first interventional phase will each be modeled as a function of group (A/F vs. IAU), the baseline outcome 
value, the stratification variables of facility complexity (high, medium, low) and baseline performance level 
(above or below the median average fallout rate), and geographic region (West, Midwest, Southwest, Southeast 
and Northeast). Inference will be made based on the coefficient associated with the group variable and 97.5% 
confidence intervals (CI) will be reported (alpha=0.025). A similar analysis will be used to determine the effect 
of A/F on the change in the EPRP outcome for the primary screeners (sui40 and sui51), with 95% CIs reported. 
c. Secondary Aims: 1). To test the effect of the addition of EF for those who do not implement adequately at 
end of interventional phase 1, the primary EPRP outcomes of change in sui2 and csra1 from the 10th month of 
interventional phase 1 (baseline for interventional phase 2) to the 10th month of interventional phase 2 will be 
modeled as a function of group, the baseline outcome value, the stratification variables and geographic region 
(if sample sizes allow for control of this variable). Inference will be made based on the coefficient associated 
with the group variable and 95% CIs will be reported. A similar analysis will be used to determine the effect of 
the addition of EF on the change in the EPRP outcome for the primary screeners (sui40, sui51), with 95% CIs 
reported. 2). An analysis similar to that described for secondary aim 1 will be employed to investigate the effect 
of discontinuing A/F for those who implemented adequately at the end of interventional phase 1. Lastly, we will 
use a data driven approach to characterize each of the four EPRP outcomes over time for each group outlined 
in Figure 2 (group 1-6). Mixed effects models with a random intercept and slope will be used to model each of 
the four EPRP outcomes as a function of categorical group (1-6) and an interaction between group and a B-
spline transformation on time (allowing the outcome to vary smoothly over time, using 21-time points, i.e., the 
6th baseline month and every month of each interventional phase) such that each group will have its own 
trajectory. The trajectory for each group will be plotted with pointwise confidence intervals. d. Exploratory 
Aims: i. Reach:  Reach will be calculated using the following formula: actual number of Veterans screened or 
evaluated/ actual number of Veterans eligible for screening, calculated cumulatively across all three study 
phases and all sites. We will calculate reach for each assessment stage of VA Risk ID. We will examine 
representativeness of patients reached by comparing demographic characteristics and other relevant variables 
(e.g., settings in which screening was completed) between eligible Veterans who were screened and/or 
evaluated and those who were not. Given that Veterans are clustered within facility, if the intra-class 
correlation between Veterans within each facility is estimated to be greater than zero, methods appropriate for 
clustered data will be employed to compare the variables of interest. For continuous variables, linear mixed 
models will be used with a random facility effect and non-normal data will be transformed. For binary 
variables, generalized linear mixed models will be used with a logit link and for categorical variables, random 
effects multinomial models will be employed. If the ICC is zero, t-tests, Wilcoxon rank-sum, Chi-square or 
Fisher’s exact tests will be used, as appropriate.  ii. Adoption & Implementation Factors: Key informant 
interviews will be used to examine factors influencing adoption of the implementation interventions and 
barriers and facilitators of implementing VA Risk ID to fidelity. All qualitative data sources, including interview 
transcripts and documents will be compiled and managed using Nvivo V. 9.0 software. We will take a general 
inductive approach34. Specifically, data analysis will be determined by both the research objectives (i.e., 
domains of the RE-AIM framework) and multiple readings and interpretation of the raw data (i.e., content 
analysis). The goal is to establish clear links between the research objectives and the summary findings derived 
from the raw data. iii. Effectiveness- Clinical Innovation: We wish to examine whether Veterans who 
receive the CSRE are more likely to receive a safety plan within 2 weeks than those who screen positive on the 
primary screen only. To account for clustering of Veterans within facility, a mixed-effects logistic regression 
will be used to model the outcome of safety plan within 2 weeks (yes/no) as a function of group (receipt of 
CSRE/positive on primary screen only) with a random subject within facility effect. Additionally, to determine 
if receipt of a timely safety plan depends on whether Veterans are considered to be at low, moderate or high 
acute risk of suicide, this model will be repeated with the addition of a group by (categorical) acute risk 
interaction. Odds ratios for receipt of CSRE relative to positive on primary screen only will be reported for each 
level of acute risk with 95% confidence intervals.