Risk Stratified Enhancements to Clinical Care: Targeting Care for Patients Identified Through Predictive Modeling as Being at High Risk for Suicide, With the Office of Mental Health Operations
Conditions: Suicide
Sponsor: VA Office of Research and Development

 
 
 
 
 
STUDY PROTOCOL 
 
Risk Stratified Enhancements to Clinical Care: Targeting Care for Patients 
Identified Through Predictive Modeling as Being at High Risk for Suicide, 
with the Office of Mental Health Operations  
SDR 16-195 
NCT03280225 
 
 
Sara J. Landes, PhD  
Principal Investigator 
 
 
 
 
 
 
Funding Agency: HSR&D 
 
 
 
 
 
 
Version 3.0 
11/01/2018 

 
Version 3.0  
Landes IRB Protocol – Version 11/01/2018 
Page 2 of 24 
 
List of Abbreviations 
CDW:   
Corporate Data Warehouse 
CeMHOR:  
Center for Mental Healthcare and Outcomes Research 
EBP:   
Evidence-based practices 
FY:  
 
Fiscal year 
i-PARIHS:  
Integrated Promoting Action on Research Implementation in Health Services 
MAX:   
Medicaid Analytic eXtract 
MIRECC:  
Mental Illness Recovery Education Clinical Center 
NDI:   
National Death Index 
OMHO:  
Office of Mental Health Operations 
OSP:   
Office of Suicide Prevention 
PEPReC:  
Partnered Evidence-Based Policy Resource Center 
PERC:  
Program Evaluation Resource Center 
QUERI:  
Quality Enhancement Research Initiative  
RCT:   
Randomized controlled trial 
REACH VET:  Recovery Engagement and Coordination for Health – Veterans Enhanced treatment 
RE-AIM:  
Reach, effectiveness, adoption, implementation, and maintenance 
RQA:   
Rapid Qualitative Analysis 
SDR:   
Service directed research 
SMITREC:  
Serious Mental Illness Treatment Resource and Evaluation Center 
SPAN:  
Suicide Prevention Applications Network 
SPC:   
Suicide Prevention Coordinator 
VA:  
 
Department of Veterans Affairs 
VHA:   
Veterans Health Administration 
VISN:   
Veteran Integrated Service Network 

 
Version 3.0  
Landes IRB Protocol – Version 11/01/2018 
Page 3 of 24 
 
PI/Study Staff. 
Principal Investigator 
Sara J. Landes, PhD 
Center for Mental Healthcare and Outcomes Research (CeMHOR) 
Central Arkansas Veterans Healthcare System 
2200 Fort Roots Drive 
Bldg. 58 (152/NLR) 
North Little Rock, AR 72114-1706 
 
Co-I: Karen Drummond, PhD 
Center for Mental Healthcare and Outcomes Research (CeMHOR) 
Central Arkansas Veterans Healthcare System 
2200 Fort Roots Drive 
Bldg. 58 (152/NLR) 
North Little Rock, AR 72114-1706 
 
Co-I: Jacob Painter, PharmD, PhD 
Center for Mental Healthcare and Outcomes Research (CeMHOR) 
Central Arkansas Veterans Healthcare System 
2200 Fort Roots Drive 
Bldg. 58 (152/NLR) 
North Little Rock, AR 72114-1706 
 
Project Coordinator: Susan Jegley, MSW 
Center for Mental Healthcare and Outcomes Research (CeMHOR) 
Central Arkansas Veterans Healthcare System 
2200 Fort Roots Drive 
Bldg. 58 (152/NLR) 
North Little Rock, AR 72114-1706 
 
Program Analyst: Jeffery Pitcock 
Center for Mental Healthcare and Outcomes Research (CeMHOR) 
Central Arkansas Veterans Healthcare System 
2200 Fort Roots Drive 
Bldg. 58 (152/NLR) 
North Little Rock, AR 72114-1706 
 
Research Staff: Rebecca Raciborski, PhD 
Center for Mental Healthcare and Outcomes Research (CeMHOR) 
Central Arkansas Veterans Healthcare System 
2200 Fort Roots Drive 
Bldg. 58 (152/NLR) 
North Little Rock, AR 72114-1706 
 
Research Assistant: Jack Woods, MSW 

 
Version 3.0  
Landes IRB Protocol – Version 11/01/2018 
Page 4 of 24 
 
Center for Mental Healthcare and Outcomes Research (CeMHOR) 
Central Arkansas Veterans Healthcare System 
2200 Fort Roots Drive 
Bldg. 58 (152/NLR) 
North Little Rock, AR 72114-1706 
 
Research Assistant: Nyssa Curtis, MA 
Center for Mental Healthcare and Outcomes Research (CeMHOR) 
Central Arkansas Veterans Healthcare System 
2200 Fort Roots Drive 
Bldg. 58 (152/NLR) 
North Little Rock, AR 72114-1706 
 
 
 

 
Version 3.0  
Landes IRB Protocol – Version 11/01/2018 
Page 5 of 24 
 
EVALUATION OBJECTIVES 
Suicide prevention is a top priority for the Department of Veterans Affairs (VA) and the Veterans 
Health Administration (VHA). To improve suicide prevention, it is important to identify people as early as 
possible. One innovative approach recently validated in VHA is the use of predictive modeling and 
electronic medical record data that identifies Veterans at risk and thus facilitates implementation of 
targeted prevention1. The VHA predictive model has identified the top 5% of VHA patients who were at 
the highest predicted risk for suicide. These Veterans accounted for approximately 24% of all the suicide 
deaths, 37% of all reported suicide attempts, and 31% of inpatient psychiatric hospitalizations observed 
in VHA over the course of one year1,2. The model also allowed examination of different risk strata (e.g., 
the top 0.1%, 1%, or 2%).  
Veterans in the highest 0.01% of the risk stratum had suicide rates 82 times greater than the rest 
of the sample in a development model. The benefits of the model were retained in a validation sample; 
Veterans in the same risk stratum had suicide rates that were 60 times higher than the population. This 
model provides new information about who is at risk; fewer than 2% of the 5% of patients identified as 
high risk received clinical flags for being at risk in the previous 12 months. 
For those identified as high risk, VHA’s Office of Mental Health and Suicide Prevention developed 
a national suicide prevention outreach program entitled Recovery Engagement and Coordination for 
Health – Veterans Enhanced Treatment (REACH VET). REACH VET utilizes a dashboard to provide the 
names of patients identified by the model monthly to coordinators at each VA medical facility. REACH 
VET coordinators are responsible for notifying providers of the patient’s status and prompting providers 
to re-evaluate care and take appropriate clinical steps if they are not already occurring (e.g., contacting 
the patient to re-engage in care, discussing potential changes in care with the patient).  
To further strengthen REACH VET, treatment recommendations and/or augmentations are 
suggested. One recommendation is Caring Letters, an effective, low-cost suicide prevention intervention. 
Caring Letters involves the sending of recurring brief notes to patients at high risk expressing care and 
concern. It is one of the only psychosocial suicide prevention interventions that have reduced suicide 
mortality rates in a randomized controlled trial3–7. Despite positive results, Caring Letters have yet to 
become routine care – primarily because it requires considerable tracking to send letters. If these issues 
can be addressed, it is an ideal intervention to scale up for a large empirically defined high risk group.  
REACH VET is currently being implemented in VHA. Additional implementation assistance in the 
form of virtual external facilitation, an evidence-based implementation strategy, will be offered to sites 
having difficulty implementing. VISNs will have the opportunity to receive facilitation for at least 4 of their 
facilities having difficulty implementing REACH VET; the REACH VET Program Manager will discuss this 
with VISN leadership, who will decide if they want to participate in facilitation. Participating VISNs will be 
randomized to when their VISN will receive facilitation.  
The VA Serious Mental Illness Treatment, Resource, and Evaluation Center (SMITREC) and 
Partnered Evidence-Based Policy Resource Center (PEPReC) are conducting a summative evaluation of 
REACH VET’s impact on Veteran outcomes using VHA databases. The current study extends the 
breadth and depth of their evaluation with these specific aims. 
Specific Aim 1: Evaluate the impact of virtual external facilitation versus standard implementation. 
Specific Aim 1a: Conduct a formative evaluation to identify barriers and facilitators to 
implementation to define and refine the virtual external facilitation strategy. 
Specific Aim 1b: Conduct a summative evaluation of virtual external facilitation versus standard 
implementation. 

 
Version 3.0  
Landes IRB Protocol – Version 11/01/2018 
Page 6 of 24 
 
Specific Aim 1c: Collect the costs of the REACH VET and Caring Letter interventions and of the 
virtual external facilitation strategy for subsequent analysis of potential cost effectiveness. 
Specific Aim 2: Develop and evaluate the augmentation of REACH VET using Caring Letters, an 
evidence-based suicide prevention intervention. 
Specific Aim 2a: Conduct a formative evaluation of the augmentation of REACH VET with 
Caring Letters. 
Specific Aim 2b: Refine the Caring Letter intervention for scale up to the VHA-wide REACH VET 
program. 
 
BACKGROUND/CONTEXT 
Veteran suicide rates remain high despite improvements to mental health services and suicide 
prevention programs. 
 
Despite work done to strengthen VHA mental health services and suicide prevention, suicide 
rates in VHA have been stable8. These rates stand in contrast to increasing rates in other Americans, 
especially middle-aged men9,10 and in Veterans who do not utilize VHA services11,12, suggesting that VHA 
programs may have mitigated expected increases. Nevertheless, the finding that suicide rates in VHA 
remain high represents a strong call for action. Novel approaches that reduce the incidence of suicide-
related events are needed earlier, ideally before suicide-related behaviors occur.  
Predictive models improve identification of patients at risk for suicide.  
While there is agreement about the risk and protective factors for suicide that should be 
assessed13, research that has examined clinicians’ ability to predict suicide has consistently reported 
poor results14–16. Despite the difficulty in predicting risk, there is little information about multivariable 
models that clinicians can use to aid in decision-making17. To improve identification of those at risk, a 
number of big data initiatives have been developed to create predictive models that use information from 
medical and administrative records to identify patients at risk for suicide. Reports from these initiatives 
have shown that predictive modeling can identify patients at risk18, and that predictive modeling may be 
more accurate than clinical evaluation19. 
The VHA recently developed and validated a strong predictive model2. The model utilized clinical 
data from an archival VHA data set that contains comprehensive patient-by-patient, encounter-by-
encounter clinical and administrative data derived from VA’s electronic medical record. Data regarding 
vital status and cause of death were obtained from the National Death Index, a centralized national 
database of death record information on file in state vital statistics offices2. The model allowed for 
identification of the top 5% of VHA patients who were at the highest predicted risk for suicide and who 
accounted for approximately 24% of all the suicide deaths, 37% of all reported suicide attempts, and 
31% of the inpatient psychiatric hospitalizations observed in VHA over the course of one year1,2. The 
model also allowed examination of different risk strata (e.g., the top 0.1%, 1%, or 2%). 
Veterans in the highest 0.01% of the risk stratum had suicide rates 82 times greater than the rest 
of the sample in a development model. The benefits of the model were retained in a validation sample; 
Veterans in the same risk stratum had suicide rates that were 60 times higher than the population. When 
the risk stratum was adjusted to focus slightly more broadly on the highest 0.10% of the risk, Veteran 
suicide rates were 30 times higher than the population in a validation sample. These models provide new 
information about VHA patients at risk; fewer than 2% of the 5% of patients at high risk for suicide had 
received clinical flags for being at risk in the previous 12 months.  

 
Version 3.0  
Landes IRB Protocol – Version 11/01/2018 
Page 7 of 24 
 
Additional work is needed to move predictive modeling into the field to inform care. 
While predictive models allow for identification of those at risk for suicide, additional tools are 
needed to provide this information to the appropriate providers along with recommendations for how to 
use this information about patient risk. Developers of the VA predictive model noted that if development 
of a model “proved feasible, the next steps would be for the health care system to develop methods for 
informing providers about which of their patients are at high risk and for enhancing care2.” 
VHA developed REACH VET to provide risk information and suggest care enhancements to 
providers. 
VHA developed a suicide prevention outreach program that utilizes the predictive model, entitled 
Recovery Engagement and Coordination for Health – Veterans Enhanced Treatment or REACH VET. 
REACH VET coordinators at each facility are responsible for monitoring the REACH VET dashboard that 
both identifies those at high risk and identifies and tracks next steps for coordinators and providers. 
Following identification of patients at risk, coordinators notify each patient’s provider of their high-risk 
status and orient the provider to the dashboard. Providers are required to re-evaluate the patient’s care, 
determine if care enhancements are needed (e.g., re-engaging the patient in care, creating a safety 
plan), and contact the patient. REACH VET includes use of the predictive model to identify high-risk 
patients, use of the dashboard by coordinators and providers, re-evaluation of care, and outreach. 
Caring Letters is an efficacious suicide prevention approach that is extremely well suited for a 
targeted, empirically defined, high-risk population.  
While REACH VET is directly tied to the predictive model and brings needed information to the 
provider so they can reach out to the right patients before they harm themselves, it involves standard 
VHA services which are not all specifically for suicide prevention. Thus, it is critical to augment REACH 
VET with a scalable suicide-specific evidence-based practice. Caring Letters is one of the only suicide 
prevention interventions that have reduced suicide mortality rates in a randomized controlled trial (RCT). 
It is a psychosocial intervention that involves sending recurring brief notes of care and concern to high-
risk patients. The concept was developed over 40 years ago and has been tested through a variety of 
delivery modalities (e.g., standard mail, email, SMS texting). Caring Letters were sent to patients 1, 2, 3, 
4, 6, 8, 10, and 12 months after contact with services in the model that was used successfully in previous 
research studies3. Eight original studies, two follow-up studies, and one secondary analysis have been 
conducted on a variety of Caring Letter models3–7,20–22. Two RCTs demonstrated decreased suicide rates 
after Caring Letters were sent by mail or telephone. Three studies showed a statistically significant 
reduction in repeat suicide attempts. An additional six studies, including a follow-up study and a study 
reporting secondary analyses, showed mixed or non-conclusive results but also showed trends toward a 
preventative effect. Two studies did not show preventative effects for the follow-up interventions; 
however one 23 did not using an intent to treat analysis because they could not reach a lot of people and 
the other21 ended the study early and were then underpowered. 
Caring Letters are effective, non-invasive, low risk, and inexpensive. Despite positive results in 
multiple clinical trials, Caring Letters have yet to become routine care – primarily because it requires 
considerable tracking to send letters. However, if these issues are addressed, it is an ideal intervention to 
scale up for a large empirically defined high risk group. It is the ideal intervention to test with a large-
scale empirically defined high-risk group, such as those identified by the predictive model.  
VHA is now implementing REACH VET nationally.  
Two memoranda were sent to VA Network Directors and VISN Mental Health Leads to first inform 
them about the REACH VET program and direct them to identify a REACH VET coordinator and then to 
provide additional information and dates of the implementation. OMHSP is funding the staffing and 

 
Version 3.0  
Landes IRB Protocol – Version 11/01/2018 
Page 8 of 24 
 
resources for intervention development and implementation of REACH VET. The evaluation team was 
included during the planning grant to develop an evaluation plan that matched the intervention and 
implementation plan, as well as to offer input on the type of implementation plan that would facilitate 
randomized program evaluation. 
Facilitation is an evidence-based implementation strategy to support sites that have difficulty 
implementing innovative programs.  
Implementation facilitation is a multi-faceted “process of interactive problem solving and support 
that occurs in the context of a recognized need for improvement and a supportive interpersonal 
relationship24.” Facilitation has been used nationally across VHA to implement a number of different 
clinical interventions. In fact, facilitation has been further developed as a strategy through partnerships 
between VHA researchers and VHA operational partners25. Virtual external facilitation has been used 
nationally in VA to implement a low complexity intervention where a provider utilized a patient registry to 
contact patients lost to care26. The current project will examine this minimally intensive version of 
implementation facilitation, virtual external facilitation, with a moderately complex innovation targeting a 
high-risk clinical population.  
Conceptual Models 
The implementation approach has been guided by the Integrated Promoting Action on Research 
Implementation in Health Services (i-PARIHS)27 framework. The i-PARIHS framework proposes that 
successful implementation of evidence-based practices (EBPs) is the result of the facilitation of an 
innovation with recipients in the inner and outer context. Facilitation is the active ingredient with 
designated facilitators activating implementation by assessing and responding to the characteristics of 
the recipients of the innovation within their own settings. See Figure 1. 
Figure 1. Graphical representation of iPARIHS framework. 
According to i-PARIHS, the innovation is the focus 
of implementation (e.g., REACH VET). The 
construct of innovation includes characteristics of 
the innovation (e.g., relative advantage, usability, 
and trialability28–30) and the evidence related to the 
innovation. Evidence includes more than research 
evidence; it includes information obtained from 
clinical experience and patient preferences or 
experience27,31. 
The construct of recipients focuses on the 
targets for implementation, which may be individual providers or medical facilities. Key characteristics of 
recipients include: motivation, values and beliefs, goals, skills and knowledge, time, resources and 
support, local opinion leaders, collaboration and teamwork, existing networks, learning environment, 
power and authority, and presence of boundaries27. 
According to i-PARIHS, factors within the inner and outer context of the recipients affect 
implementation. Inner context includes the immediate setting for implementation (e.g., mental health 
clinic) and the organization in which that clinic is located (e.g., medical center). Outer context refers to 
the wider health system in which the organization is based and the policies, regulatory frameworks, and 
political environment that govern the way the health system functions (e.g., VHA). Examples of inner 
context constructs include: leadership support, culture, organizational priorities, evaluation and feedback 
processes, learning networks, and structure. Examples of outer context constructs include: policy drivers 

 
Version 3.0  
Landes IRB Protocol – Version 11/01/2018 
Page 9 of 24 
 
and priorities, incentives and mandates. 
Facilitation is an evidence-based strategy for achieving practice change24,32–36. The i-PARIHS 
framework identifies facilitation as the ingredient that activates the other constructs to produce successful 
implementation32,37,38. Facilitation bundles an integrated set of implementation strategies. These include 
identifying and engaging key stakeholders at all organizational levels, problem identification and 
resolution, providing assistance with technical issues, developing information exchange networks, 
academic detailing, marketing, staff training, formative evaluation, auditing and feedback, evidence-
based quality improvement, engagement of opinion leaders and clinical champions, and role modeling. 
Facilitators’ actions are dependent on a facility’s needs and the timing of the implementation process. An 
external facilitator is an expert in implementation and the relevant clinical areas38. 
METHODS 
Overview of Evaluation Methods 
This evaluation will utilize a mixed methods approach. To evaluate our aims, the project will utilize 
a stepped wedge controlled trial design. This design staggers the timing of providing implementation 
support and allows every participating VISN to receive the implementation support. The unit of 
intervention in this study is the site and randomization will occur at the VISN level for virtual external 
facilitation. Using a balancing algorithm for facility size, suicide admissions, and rural vs. urban status, 
sites will be randomized to when virtual facilitation will occur. Below we describe the conceptual models, 
national implementation plan, clinical interventions, and implementation strategy, followed by the design, 
outcomes, and analysis for each specific aim. 
Project Management 
• 
Sara J. Landes, PhD (PI, CAVHS) serves as the principal investigator for this evaluation project 
and is an Investigator at the Central Arkansas Veterans Healthcare System, VISN 16 Mental 
Illness Research Education and Clinical Center (MIRECC) and Assistant Professor of Psychiatry 
at University of Arkansas for Medical Sciences. Dr. Landes specializes in testing implementation 
strategies across a wide range of service contexts, and has received implementation science 
research funding from QUERI, NIMH, and the DOD. She is also a co-investigator on the Military 
Suicide Research Consortium’s new dissemination and implementation core. She has a history of 
partnering with the Office of Mental Health and Suicide Prevention on implementation research 
projects. Dr. Landes will head an evaluation team consisting of a qualitative investigator, cost 
evaluation investigator, project coordinator, research assistants, and a qualitative 
interviewer/coder. Karen Drummond, PhD (Co-I, CAVHS) is a Research Health Scientist at 
HSR&D Center for Mental Healthcare and Outcomes Research (CeMHOR). Dr. Drummond is a 
medical anthropologist and health services researcher with training and expertise in qualitative 
methods, rapid data analysis techniques, and formative evaluation. Dr. Drummond will help with 
qualitative data collection, analysis and dissemination of results for the evaluation. Jacob 
Painter, PhD, PharmD (Co-I, CAVHS) is an Assistant Professor at the University of Arkansas for 
Medical Sciences (UAMS) in the Division of Pharmaceutical Evaluation and Policy; and an 
investigator at the Center for Mental Healthcare & Outcomes Research (CeMHOR). His research 
focuses on economic analysis of mental health approaches for Veterans. Dr. Painter will assist in 
the development, collection, and analysis for the evaluation of costs associated with the 
intervention. Susan Jegley, MSW (Project Coordinator, CAVHS) is a Health Science Specialist 
at CeMHOR who has considerable experience as a project coordinator. For this project she will 
have primary responsibility for regulatory compliance and budget management. She will assist in 
preparation of and coordinate all IRB/R&D submissions and maintain IRB regulatory files. She will 

 
Version 3.0  
Landes IRB Protocol – Version 11/01/2018 
Page 10 of 24 
 
develop and maintain administrative tracking systems to ensure regulatory compliance and fiscal 
responsibility. She will also assist with data collection, data entry, and dissemination of results. 
Rebecca Raciborski, PhD, (Research Staff, CAVHS) has experience in economic analysis of 
mental health approaches. She will assist Dr. Painter in the data collection for evaluation of costs. 
Jeffery Pitcock, MPH (Program Analyst, CAVHS) has experience in data collection, cleaning, 
and analysis, especially using data from VA’s CDW. He will be responsible for the data analysis 
of REACH VET dashboard data. Jack Woods, MSW (Research Assistant, CAVHS) has 
experience working on a number of qualitative research studies. Mr. Woods is an experienced 
research assistant who will coordinate data management, maintain the project’s regulatory 
binder, and correspond with staff. He will coordinate evaluation activities with study personnel 
and assist in data entry and dissemination of the results. Nyssa Curtis, MA (Research 
Assistant, CAVHS) has experience working on a number of qualitative research studies. She will 
be responsible for transcription of qualitative interviews. 
• 
John McCarthy, PhD, MPH (Co-I, SMITREC) is Director of the Serious Mental Illness Treatment 
Resource and Evaluation Center (SMITREC) and Director of the National Primary Care – Mental 
Health Integration Evaluation in VHA. He has over 100 peer-reviewed publications, including first 
authorship on the manuscript that developed the predictive model that provided the foundation for 
this project. Dr. McCarthy will coordinate patient data management and analysis for the predictive 
model as well as the evaluation, working closely with PEPReC on the latter. Claire Hannemann, 
MPH (Programmer Analyst, SMITREC) is an Epidemiologist and Population Health Analyst at 
the Serious Mental Illness Treatment Resource and Evaluation Center. She received an MPH in 
Epidemiology from the University of Michigan School of Public Health, in Ann Arbor, Michigan, 
and she conducted core data management and analysis for the VA’s groundbreaking predictive 
modeling for suicide risk. Ms. Hannemann will assist Dr. McCarthy with patient data management 
and analysis for the predictive model as well as the evaluation of REACH VET implementation in 
close collaboration with the Partnered Evidence-based Policy Resource Center (PEPReC). 
• 
Craig Rosen, PhD (Co-I, VAPA) is the Deputy Director for the Dissemination and Training 
Division of the VA’s National Center for PTSD. He has expertise in implementing practices to 
enhance VA care and evaluating large scale rollouts in VA. He will serve as the site PI and 
oversee all aspects of project execution at the Palo Alto VAMC. He will supervise all study 
personnel at this site. Brandy Smith, BA (Interviewer, VAPA) is a Research Assistant at the 
Palo Alto VAMC. She has experience working on numerous VA studies conducting qualitative 
data collection and analysis. She will be involved in recruiting subjects; obtaining informed 
consent; administering survey/interview procedures; and assisting with data analysis. 
• 
Mark Reger, PhD (Co-I, VAPSHCS) is Chief of Psychology at the VA Puget Sound Health Care 
System and Associate Professor of Psychiatry at the University of Washington School of 
Medicine. He previously worked for 10 years in the Department of Defense leading military 
suicide prevention, surveillance, and research initiatives. Dr. Reger will lead the intervention team 
with Dr. Bridget Matarazzo. He will also lead the development and refinement of the Caring 
Letters component of the intervention. 
• 
Bridget Matarazzo, PsyD (Co-I, Rocky Mountain MIRECC) is Co-Director of the VA National 
Suicide Risk Management Consultation Program and Co-Director of the Rocky Mountain 
MIRECC Suicide Prevention Consultation Service. She also is an Assistant Professor in the 
Department of Psychiatry at the University of Colorado, School of Medicine. She is also the 

 
Version 3.0  
Landes IRB Protocol – Version 11/01/2018 
Page 11 of 24 
 
national clinic lead of the REACH VET program and will lead the intervention team with Dr. Mark 
Reger.  
Table 1: Members of the Evaluation Team who will consent participants and their access to PII and PHI.  
Study Staff 
Member 
Access to 
Coordinator 
PII 
Obtain 
consent from 
Coordinator 
Access to 
Provider 
PII 
Obtain 
consent from 
providers 
Access to 
Leadership 
PII 
Obtain 
consent from 
leadership 
Access to 
patient 
PHI 
Sara Landes 
Yes 
No 
Yes 
No 
Yes 
No 
Yes 
Karen 
Drummond 
Yes 
No 
Yes 
No 
Yes 
No 
No 
Jacob 
Painter 
Yes 
No 
Yes 
No 
Yes 
No 
No 
Susan Jegley 
Yes 
No 
Yes 
No 
Yes 
No 
Yes 
Jeffery 
Pitcock 
Yes 
No 
Yes 
No 
Yes 
No 
Yes 
Rebecca 
Raciborski 
Yes 
No 
Yes 
No 
Yes 
No 
Yes 
Jack Woods 
Yes 
Yes 
Yes 
Yes 
Yes 
Yes 
No 
Nyssa Curtis 
Yes 
No 
Yes 
No 
Yes 
No 
No 
Craig Rosen 
No 
No 
No 
No 
No 
No 
No 
Brandy Smith 
Yes 
Yes 
Yes 
Yes 
Yes 
Yes 
No 
John 
McCarthy 
No 
No 
No 
No 
No 
No 
Yes 
Claire 
Hannemann 
No 
No 
No 
No 
No 
No 
Yes 
Ira Katz 
No 
No 
No 
No 
No 
No 
No 
Mark Reger 
No 
No 
No 
No 
No 
No 
No 
Bridget 
Matarazzo 
Yes 
No 
Yes 
No 
Yes 
No 
No 
 
Data Use Agreements (DUAs) 
 A data use agreement was finalized with the Office of Mental Health Operations (OMHO). This 
DUA covers the REACH VET dashboard data being sent to the evaluation team. This data includes: 
names of patients identified by REACH VET and their information as displayed in the dashboard and 
actions taken by the REACH VET coordinator and providers.  
Table 2: REACH VET Dashboard Variables for Data Collection 
Patient Level Variables: 
First Name 
Last Name 
Last 4 of SSN 
Facility (location) 
Date First Showed up on Dashboard  
Coordinator Level Variables: 
First Name 

 
Version 3.0  
Landes IRB Protocol – Version 11/01/2018 
Page 12 of 24 
 
Last Name 
Is there a High Risk Suicide Flag documented in CPRS?  
Dates - Coordinator: 
Received name of patient  
Identified a primary patient provider 
Notified provider of the specific patient and program requirements  
Asked provider to re-evaluate patient’s care  
Provider Level Variables: 
First Name 
Last Name 
Dates - Provider: 
Received notification from REACH VET coordinator about the patient  
Reviewed current diagnoses and treatment plan  
Indicated enhanced care options: 
Caring Communications  
Safety Planning 
Increased monitoring of stressful life events 
Improve coping skills 
Other 
Care was reviewed and no changes are clinically indicated at this time  
Informed Veteran that they have been identified as being at risk  
Discussed care enhancement options with the Veteran  
Discussed treatment plan changes with the Veteran  
 Indicated that outreach attempts have been made but provider unable to get in touch with the Veteran  
Indicated that no outreach to Veteran is clinically indicated at this time  
NATIONAL IMPLEMENTATION PLAN 
Clinical Intervention: REACH VET 
The intervention will be provided through coordinated activities of REACH VET coordinators and 
providers at facilities. The REACH VET dashboard provides the names of patients who are identified as 
high risk by the predictive model to REACH VET coordinators at each facility. The dashboard provides 
names, contact information, risk stratum (e.g., 0.1%), and the clinical characteristics that contributed to a 
patient’s risk (e.g., inpatient psychiatric discharge, chronic non-cancer pain). It also provides prompts for 
next steps for coordinators and providers and a record of actions that have been completed. REACH 
VET coordinators are responsible for monitoring the dashboard, identifying the primary provider for each 
patient (and if none exists, working with leadership to assign one), notifying providers of the patient’s 
status on the dashboard, and ensuring the provider understands the REACH VET program and their 
responsibilities. 
The provider’s responsibilities include acknowledging notification of their patient’s high risk status 
in the REACH VET dashboard; contacting the patient; re-evaluating diagnoses, treatment plans, and 
care provided for diagnosed conditions; ensuring patients have access to care when they need it; 
ensuring patients are receiving evidence-based care; considering additional enhancement strategies; 
and when changes to the treatment plan or enhanced care is clinically indicated, discussing these 
changes with the patient. Providers are instructed to document all actions taken to support the patient in 
the REACH VET dashboard and the electronic medical record. REACH VET electronic medical record 
templates will allow consistent reporting and evaluation of provider behavior.  

 
Version 3.0  
Landes IRB Protocol – Version 11/01/2018 
Page 13 of 24 
 
The plan for national roll out of REACH VET was developed by OSP, with input from the 
evaluation team and other partners. National implementation of REACH VET began in November 2016 
and included identification of a REACH VET coordinator at each facility, REACH VET dashboard training 
for coordinators and interested providers, and rollout of the dashboard. Each facility initially received the 
top eight names of patients identified as being in the top 0.1% of risk for suicide at that facility. Over the 
following three months, facilities will receive additional names via the dashboard until all patients in the 
top 0.1% have been identified; facilities can opt to receive the full list at any time. Standard 
implementation strategies will include training and education, chart note templates to support 
documentation, an intranet site of resources for coordinators and providers, technical assistance as 
requested, and clinical consultation as requested. Barriers and facilitators will be assessed during this 
time by the evaluation team.  
Figure 2. Trial design.  
 
 
 
 
 
 
Virtual 
External Facilitation Implementation Strategy 
 
Additional implementation support will begin in April 2017. We will target 7 networks and 4 sites in 
each network not meeting the criterion for timely notification. The participating VISNs will be randomized 
to when the sites in their network will receive facilitation.  
The facilitators will be two master’s level and one doctoral level clinicians with clinical expertise in 
suicide prevention and REACH VET; the facilitators are funded through OSP and are not part of the 
evaluation team. They attended facilitation training February 1-2, 2017. The training included how to 
conduct virtual facilitation and utilized a virtual external facilitation manual developed by a workgroup of 
the Behavioral Health QUERI’s Implementation Facilitation Learning Collaborative. 
External facilitators will use a variety of strategies to facilitate implementation, including provider 
education, performance monitoring and feedback, and formative evaluation. To adapt to each facilities’ 
particular circumstances, facilitators will select from a broad range of strategies (e.g., facilitate 
stakeholder engagement, conduct provider education, facilitate performance monitoring and feedback, 
and conduct formative evaluation) based upon an assessment of each facility’s needs, barriers, and 
facilitators. 
 
Clinical Intervention: Caring Letters 
This project will evaluate methods and feasibility for the use of a Centralized Caring Letters program in 
conjunction with REACH VET. 
REACH VET Participants 
Veterans who have been flagged as high risk through REACH VET will be recruited from two VA 
facilities, including VA Puget Sound and Central Arkansas Veterans Healthcare System.  
 
 
Study Month 
Network 
Sites 
1-3 
3 – 6 
6 - 9 
9 - 12 
12 - 15 
15 - 18 
18 - 21 
21 - 24 
A 
1-4 
Facilitation 
 
 
 
 
 
 
B 
5-8 
Awaiting Facilitation 
Facilitation 
 
 
  
 
 
C 
9-12 
Awaiting Facilitation 
Facilitation 
 
 
 
 
D 
13-16 
Awaiting Facilitation 
Facilitation 
 
 
 
E 
17-20 
Awaiting Facilitation 
Facilitation 
 
 
F 
21-24 
Awaiting Facilitation 
Facilitation 
 
G 
25-28 
Awaiting Facilitation 
Facilitation 

 
Version 3.0  
Landes IRB Protocol – Version 11/01/2018 
Page 14 of 24 
 
A local Caring Letters Coordinator will contact the local REACH VET Provider who will determine 
whether Caring Letters is clinically appropriate for the patient identified by REACH VET. REACH VET 
Providers will exclude patients from the program if they had no mailing address to receive Caring Letters, 
psychosis that indicates Caring Letters could be counterproductive, adverse behavioral problems (e.g., 
patient threats towards the provider, boundary problems with a provider), or similar concerns.   
Centralized Caring Letters Intervention 
For patients appropriate for Centralized Caring Letters, a centralized administrator will track, print, and 
mail the Caring Letters on behalf of the providers at pilot sites. The provider will be notified before each 
Caring Letter is sent to avoid any potential misunderstandings with patients (e.g., the patient thanks the 
provider for the card, but the provider does not recall sending anything), and to ensure that providers 
have the chance to withdraw a patient from the program when clinically indicated (e.g., changes in 
clinical status).  
The standard VA model for REACH VET Caring Letters will be used. Detailed procedures have been 
developed for sending Caring Letters, including a template for the cards. The model was based on a 
systematic review of evidence-based approaches from the literature. The cards were tested in focus 
groups with several Veteran groups, and with high risk Veterans on a psychiatric inpatient unit. A total of 
eight Caring Letters will be sent over the course of one year (monthly for four months, and then every 
two months for the remaining period). Similar to previous studies, the letters will be printed on a flat card 
and will be sent in a sealed envelope. The envelopes will be colored like a greeting card to increase the 
likelihood they will be noticed and opened. Protocols have been developed to address patient responses 
to letters (e.g., expression of thanks, indications of risk). 
 
The goals will be to develop a centralized Caring Letter protocol in a way that remains meaningful to the 
patient, establish criteria for patient selection, adapt existing processes for sending letters, and automate 
the process for scale up. The evaluation team will document feasibility and acceptability to providers and 
patients, and resources needed.  
 
Outcomes 
This project focuses on evaluation of implementation outcomes and development and formative 
evaluation of a centralized protocol for Caring Letters. Outcomes assessed are described here by 
specific aim. 
Specific Aim 1: Evaluate the impact of virtual external facilitation versus standard implementation. 
Specific Aim 1a: Conduct a formative evaluation to identify barriers and facilitators to 
implementation to define and refine the virtual external facilitation strategy. 
Design: We will conduct formative evaluation of the implementation process, with a focus on 
barriers and facilitators related to i-PARIHS constructs (e.g., characteristics of the innovation, context, 
and recipient), with 28 facilities in 7 VISNs.  
Measures: Data sources will include qualitative interviews with site staff, including REACH VET 
coordinators, SPCs, mental health and primary care providers, and facility or VISN leadership. We will 
aim to conduct at least three interviews per facility, with a maximum of five interviews per facility. 
Interviews will be conducted by phone and be audio-recorded and transcribed. Interviews will be 
conducted to refine the virtual external facilitation strategies to be used. During virtual external 
facilitation, we will conduct monthly debriefings with the facilitators. 

 
Version 3.0  
Landes IRB Protocol – Version 11/01/2018 
Page 15 of 24 
 
Analysis: Rapid Qualitative Analysis (RQA)39 strategies will be used to ensure timely 
communication of findings to our OSP partners and to allow us to refine our virtual external facilitation 
strategies as needed. Each interview transcription will be read by two members of the qualitative team, 
who will then distill information pertaining to previously identified domains of interest based on iPARIHS 
(context, innovation, recipient), as well as emergent themes, into an episode profile. Episode profiles will 
be compared and any discrepancies resolved by referring back to the original transcript and/or 
presenting the profile to the larger team for discussion. Completed episode profiles will be entered into 
tables to create a composite matrix of data for each site, in order to be able to compare across 
participant responses. We will then share these episode profiles and site matrices with the qualitative 
coding team, REACH VET Program Manager, and external facilitators via a Lync meeting to discuss 
potential refinements to facilitation strategies.  
Specific Aim 1b: Conduct a summative evaluation of virtual external facilitation versus standard 
implementation. 
Design: All VA sites (N = 144) within VA’s 18 regional networks that were directed to implement 
REACH VET will receive standard implementation. Network directors will be asked if they would like to 
receive additional implementation help for sites in their VISN having difficulty implementing REACH VET 
(e.g., not meeting the criterion of completing at least 80% of provider notifications of patient high-risk 
status within 7 days during the last month). VISNs participating will be randomized to when the VISN 
receives virtual external facilitation at 4 of their sites having difficulty implementing.  
Measures: We will assess the impact of facilitation on REACH VET implementation using the four 
outcome measures as tracked by the REACH VET program. We will use the REACH VET Historic 
Summary Report to collect data for Coordinator assignment, Provider assignment, Care evaluation, and 
Attempted outreach all within one week of each monthly report’s release date. These metrics are those 
tracked to determine a facility’s implementation of REACH VET. They are defined as: 
• 
Coordinator Assigned – the percentage of eligible Veterans who had a Coordinator assigned 
within a week of the monthly REACH VET report being released. The denominator is the number 
of eligible Veterans within that week. The numerator is the number of Veterans with a record of a 
Coordinator assigned within or prior to that week. 
• 
Provider Assigned – the percentage of eligible Veterans who had a Provider assigned within a 
week of the monthly REACH VET report being released. The denominator is the number of 
eligible Veterans within that week. The numerator is the number of Veterans with a record of a 
Provider assigned within or prior to that week.  
• 
Care Evaluation Performed – the percentage of eligible Veterans who had a care evaluation 
performed within a week of the monthly REACH VET report being released. The denominator is 
the number of eligible Veterans within that week. The numerator is the number of Veterans with a 
record of a care evaluation performed within or prior to that week. 
• 
Attempted Outreach – the percentage of eligible Veterans with a recorded attempted outreach 
within a week of the monthly REACH VET report being released. The denominator is the number 
of eligible Veterans within that week. The numerator is the number of Veterans with a recorded 
attempted outreach within or prior to that week. 
Analysis: To compare statistical differences in outcomes between pre- and post-facilitation 
periods we will use generalized estimating equations (GEE) to control for clustering of observations 
within VAMCs. Fixed effects included a time-dependent variable identifying change in intervention 

 
Version 3.0  
Landes IRB Protocol – Version 11/01/2018 
Page 16 of 24 
 
status (pre-post) as well an independent variable of time to account for the possibility of improvement 
due to secular trends.  
 
Specific Aim 1c: Collect the costs of the REACH VET and Caring Letter interventions and of the 
virtual external facilitation strategy for subsequent analysis of potential cost effectiveness. 
Design: In order to quantify the amount of effort and time needed to offer virtual external 
facilitation, the facilitators will maintain weekly logs of activities engaged in (and time spent) related to the 
implementation. This information, along with data regarding the activities and time of staff at sites 
implementing REACH VET, will allow for preliminary data collection related to the cost of implementation 
and the cost of the intervention. 
Measures: Intervention and implementation strategy costs will include fixed and variable costs. 
Intervention and implementation strategy costs will be collected in real time using the “accountant 
perspective.” Variable intervention costs include time spent by clinical staff delivering the intervention 
(e.g., time spent using the dashboard, entering progress notes into the medical record, and contacting 
providers). Variable implementation strategy costs will be calculated based on time spent delivering 
implementation strategies. Costs will be calculated separately for each team member based on their 
respective VA salaries and fringe costs. Time spent on intervention and implementation strategy 
activities will be estimated using computerized log books to record the time spent on specific activities 
during the evaluation. Fixed intervention and implementation strategy costs include the cost of provider 
education materials, intervention training, and other materials needed for implementation strategies. 
Fixed training costs will be calculated from the duration of the trainings and salaries of trainers and 
trainees and material costs will be estimated based on project acquisition costs. 
Analyses: This data will be analyzed in a subsequent proposal. 
Specific Aim 2: Develop and evaluate the augmentation of REACH VET using Caring Letters, an 
evidence-based suicide prevention intervention. 
Specific Aim 2a: Conduct a formative evaluation of the augmentation of REACH VET with Caring 
Letters. 
Specific Aim 2b: Refine the Caring Letter intervention for scale up to the VHA-wide REACH VET 
program. 
Design. To develop the centralized Caring Letters protocol, we will implement centralized Caring 
Letters in two sites that demonstrate early success at implementing REACH VET and who have sufficient 
numbers of patients to allow for sending letters. We will conduct formative evaluation to adapt and refine 
the Caring Letters protocol and intervention.  
Qualitative Interviews. Qualitative interviews will be conducted with REACH VET coordinators 
and Caring Letters Coordinators at each of the sites. These interviews will focus on the feasibility and 
acceptability of the intervention, as well as determine the resources needed for a centralized letter 
program.  
Analysis. We will use the Rapid Qualitative Analysis (RQA)39 strategies described in Specific Aim 
1 to analyze this data. 
HUMAN SUBJECTS PROTECTION ISSUES AND DATA MONITORING PLAN 
This evaluation will involve collection of both primary and secondary data:  
1. Primary data will be collected for Aims 1a, 1c, & 2a, and will consist of data collected from 
qualitative interviews with site staff, including REACH VET coordinators, Suicide Prevention 

 
Version 3.0  
Landes IRB Protocol – Version 11/01/2018 
Page 17 of 24 
 
Coordinators, mental health and primary care providers, and facility or VISN leadership (e.g., 
VISN Mental Health Lead). During virtual external facilitation, we will also conduct monthly 
debriefings with the facilitators. 
2. Secondary data will be collected for Aims 1b & 2a from: 
a. REACH VET Dashboard Data: The Office of Mental Health Operations (see DUA). 
b. CPRS chart reviews: CAPRI National CPRS access. 
Potential Risk/Benefit Analysis 
Risks. Potential risks for this evaluation are minimal. For primary data collection, though VA staff 
members will be interviewed solely as agents of the organization, there is still a risk of breach of 
confidentiality. No data will be collected directly from VA patients. Secondary data will be collected from 
VA administrative databases by the Office of Mental Health Operations and provided to the evaluation 
team in a secure manner (e.g., via upload to VINCI). Precautions will be taken to minimize these risks. 
Benefits. There are no direct benefits to subjects participating in this study. However, this 
evaluation will help enable VA to evaluate the extent of implementation of the REACH VET program 
(e.g., the reach, adoption, and implementation fidelity), as well as determine how virtual external 
facilitation can support facilities having difficulty implementing the program. This evaluation will also 
collect initial data on the cost of the REACH VET program and cost of the virtual external facilitation 
strategy.  
Adverse & Serious Adverse Events 
Adverse events. Following guidelines put forth by NIH and VA, we define an adverse event as 
an unexpected reaction, side effect, or untoward event that occurs during the study. We also consider 
unanticipated problems or events that place participants at a greater risk of harm or discomfort than was 
previously known or recognized as adverse events. Stable, chronic conditions that are present prior to 
the study and do not worsen will not be considered. Given the nature of the study, the following adverse 
events are possible: discomfort or distress during primary data collection and breaches of confidentiality 
for data collected from participants and from databases.  
Serious adverse events. Serious adverse events are defined as death, any life-threatening 
experience, prolonged hospitalization, persistent or significant disability or incapacity, congenital 
anomaly, or the need for surgical, behavioral, social, or other intervention to prevent any of the above. 
Given the minimal risk nature of the study, and that no experimental treatment is being tested, we expect 
no serious adverse events for those participating directly in the evaluation (e.g., VA staff).  
Reporting of adverse and serious adverse events. The PI or designee will be responsible for 
evaluating any adverse events and serious adverse events and submitting the appropriate reports to the 
CAVHS Research Service within 5 working days after being made aware of the occurrence. The PI will 
evaluate whether a participant in the study should be discontinued from further participation in the study, 
for safety reasons. The PI and study staff will respond immediately to all directives from the CAVHS 
Research Service or VA concerning the protocol and the continuation of the evaluation.  
Participants/Subjects 
Planned/targeted enrollment. Primary data collection: There will be a maximum of 157 
participants (all VA staff) from whom data will be collected. Since participants will be selected based on 
their roles and some participants may serve duplicate roles within their clinics (e.g., a SPC may also 
serve as a REACH VET Coordinator), the total number of actual subjects will likely be less than 157.  
The table below illustrates the types and approximate number of VA staff from whom we will collect data 
in each group across all sites.  
Table 3: Site participants 

 
Version 3.0  
Landes IRB Protocol – Version 11/01/2018 
Page 18 of 24 
 
Setting 
Stakeholder Group 
Participants 
VISN 
MH Director 
7 
PC Director 
7 
External Facilitators 
3 
VAMC 
REACH VET Coordinators 
28 
SPCs 
28 
Primary Care Clinic 
MH Providers 
28 
PC Providers 
28 
MH Specialty Clinic 
MH Providers 
28 
Total 
 
157 
Secondary data collection: The study population for all administrative data analyses will include a 
maximum of 4,200 patients and 900 providers. These numbers are estimates based on large facilities 
(e.g., the top 0.1% of patients at a large facility is approximately 150 patients and we are including 28 
facilities) and therefore are likely overestimates. We plan to sample 10% of patients and their associated 
providers. 
Inclusion/exclusion criteria. Inclusion criteria includes VA site staff, including REACH VET 
coordinators, Suicide Prevention Coordinators, mental health providers, and facility or VISN leadership 
who are: (a) at least 18 years of age, (b) have access to a telephone, (c) English speaking, and (d) 
mentally and physically able to complete the interview. 
We will take special steps to insure that VA employees do not feel coerced to participate. VA 
employees generally need the permission of their supervisors to take time away from routine duties. 
Additionally, in order to meet with providers, appointments must be scheduled through designated clinic 
staff. Once staff members have been scheduled, they will be sent a letter with the elements of informed 
consent explaining that they have the right to decline participation. Potential subjects can contact study 
staff in advance to decline participation. At the time of each contact with study staff, they will be given the 
opportunity to decline participation in the study. The employee’s supervisor will not be informed of the 
employee’s decision and no negative employment consequences will result if an employee chooses not 
to participate. 
This study will not enroll or involve any of the special classes of research subjects, such as 
pregnant women, prisoners, institutionalized individuals, or others who may be considered vulnerable 
populations per IRB regulations.  
Participant Recruitment Methods 
Sites will be recruited via REACH VET Coordinator monthly calls and email communications. 
Within each site, staff participants will be recruited by identifying the local REACH VET Coordinator, 
Suicide Prevention Coordinator, providers identified in the REACH VET dashboard, and by contacting 
local or regional mental health leadership.  
Compensation. No compensation will be offered to participants for participating in this 
evaluation.  
Withdrawal of Subjects 
 
Participants may withdraw from the evaluation at any time by verbal or written request, which will 
be documented in the case file by the investigator. Withdrawal will not be communicated to any site staff 
in order to maintain confidentiality and avoid the potential for coercion. 
Privacy and Confidentiality (Data Management and Monitoring) 

 
Version 3.0  
Landes IRB Protocol – Version 11/01/2018 
Page 19 of 24 
 
Investigators will not disclose participants’ PII. All investigators and staff will have complied with 
all VA research training requirements prior to beginning participation in this evaluation.  
All information on individual subjects will be kept confidential on restricted-access server files 
behind the VA firewall. Study staff will obtain a list of the names and addresses of REACH VET staff from 
mental health leadership at each site in order to recruit providers to complete telephone interviews. The 
study PI and Qualitative Lead will maintain an electronic file linking identification numbers to the names 
of the provider interview participants (participant “crosswalk”). These files will be in a folder that is 
separate from other evaluation data. Access to the analytic files within VA will be limited to the principal 
investigator and those team members involved in the analyses. Individually unique user identification 
codes and passwords are necessary to access the network. Accounts and passwords comply with 
existing VA policy and procedures.  
Hard copies of participant responses will be kept in a locked area of CeMHOR space at CAVHS. 
Hard copies will be maintained in accordance with the VA Record retention schedule 
(www1.va.gov/VHAPUBLICATIONS/RCS10/rcs10-1.pdf). Only project staff with credentials and 
permission to use the data will have access to the data. 
Data Security and Sharing 
How Data Will Be Stored. Contact and consent information will be stored in a restricted-access 
folder on the password-protected HSR&D server at the Central Arkansas Veterans Healthcare System 
(CAVHS), North Little Rock, AR campus (\\R02LITHSMDC101.V16.MED.VA.GOV\SERVICES$\HSR). 
The information will remain stored and maintained per VA policies and procedures. Participants will be 
given a non-identifying code linking them to the audio recording of the interviews. The paper data will be 
kept in the study PI’s Office in Building 58, North Little Rock, AR. Qualitative interviews will be stored 
temporarily on VA approved digital audio recorders (Phillips DPM 8000). These temporary electronic files 
will be purged from the recorder upon upload to a password protected file on the secure network server 
on the CAVHS campus. Audio data will not include identifying information. Instead each participant will 
be given a corresponding number.  
Who Will Have Access. Dr. Landes and Ms. Jegley will have access to the REACH VET 
dashboard and CPRS records for screening, medical chart reviews and to conduct periodic reviews for 
AEs and SAEs. Mr. Woods will have access to the REACH VET dashboard and CPRS records for the 
Caring Letters component of this evaluation. Mr. Pitcock will have access to the REACH VET dashboard 
to analyze data related to the dashboard. Dr. Landes, Ms. Jegley, Dr. Painter, Dr. Raciborski, and Mr. 
Pitcock will have access to time tracking logs for cost evaluations. Dr. Landes, Ms. Jegley, Ms. Smith, 
Mr. Woods, and Ms. Curtis will have access to contact information of the site staff, REACH VET 
coordinators, and facilitators for interviewing purposes. Dr. Landes, Ms. Jegley, Ms. Smith, Mr. Woods, 
and Ms. Curtis will have access to the list that correlates the identifying number to the participant. This 
crosswalk file will be maintained in a separate password protected file on the secure network server on 
the CAVHS campus. All records will be accessible for inspections and copying by authorized 
representatives of VA, OHRP, and other authorized entities. If any member of the evaluation team leaves 
the study, his/her access to study data will be removed immediately. If, in spite of the implemented 
precautions, participant data is lost, stolen, or compromised, it will be reported immediately to the PI. The 
PI or designee will report the event to the IRB, the ISO, the PO, and the ACOS/R&D immediately upon 
discovery, but no longer than 1 hour after learning of the breach of confidentiality.  
How Long It Will Be Stored. VA Records Control Schedule DAA-0015-2015-0004, disposition 
authority number-0032 requires that records be maintained for 6 years following the end of the fiscal year 
after completion of the project. Audio recordings of interviews and other electronic records will be 

 
Version 3.0  
Landes IRB Protocol – Version 11/01/2018 
Page 20 of 24 
 
retained on a VA server for this period of time. All hard copies of records will be stored in locked filing 
cabinets on VA property until the close of the study, at which time they will be surrendered to Research 
Administration for storage for six years following the end of the fiscal year after completion of the project. 
Communication Plan 
All site approvals (CAVHS and VAPAHCS) will be obtained in compliance with local IRB and R&D 
procedures. For each site participating but not engaged in research (in this study, the 28 sites receiving 
virtual implementation facilitation), we will notify the Director prior to conducting evaluation activities at 
his/her facility (see Director’s Letter). Also, we will process any changes in the protocol, consent 
processes, or cessation of engagement in research for any sites in concordance with local IRB and site 
procedures. 
The PI will meet regularly with the evaluation team, which includes the Local Site Investigator 
from Palo Alto (Dr. Rosen), and through this mechanism we will facilitate all above communications.  
 
 

 
Version 3.0  
Landes IRB Protocol – Version 11/01/2018 
Page 21 of 24 
 
References 
1. McCarthy, J. F. et al. Predictive Modeling and Concentration of the Risk of Suicide: Implications for 
Preventive Interventions in the US Department of Veterans Affairs. American journal of public health 
e1–e8 (2015). 
2. Katz, I., McCarthy, J. & Bossarte, R. Development and evaluation of alternative predictive models for 
suicide among patients in the VA Healthcare System: Implications for intervention development. 
Unpublished Report. (2016). 
3. Carter, G. L., Clover, K., Whyte, I. M., Dawson, A. H. & D’Este, C. Postcards from the EDge project: 
randomised controlled trial of an intervention using postcards to reduce repetition of hospital treated 
deliberate self poisoning. BMJ 331, 805 (2005). 
4. Carter, G. L., Clover, K., Whyte, I. M., Dawson, A. H. & D’Este, C. Postcards from the EDge: 24-
month outcomes of a randomised controlled trial for hospital-treated self-poisoning. Br J Psychiatry 
191, 548–553 (2007). 
5. Carter, G. L., Clover, K., Whyte, I. M., Dawson, A. H. & D’Este, C. Postcards from the EDge: 5-year 
outcomes of a randomised controlled trial for hospital-treated self-poisoning. Br J Psychiatry 202, 
372–380 (2013). 
6. Hassanian-Moghaddam, H., Sarjami, S., Kolahi, A.-A. & Carter, G. L. Postcards in Persia: 
randomised controlled trial to reduce suicidal behaviours 12 months after hospital-treated self-
poisoning. Br J Psychiatry 198, 309–316 (2011). 
7. Motto, J. A. & Bostrom, A. G. A randomized controlled trial of postcrisis suicide prevention. Psychiatr 
Serv 52, 828–833 (2001). 
8. Kemp, J. E. Suicide Rates in VHA Patients through 2011 with Comparisons with other Americans 
and other Veterans through 2010. Available at: http://www.mentalhealth.va.gov/docs/ 
suicide_data_report_update_january_2014.pdf. (2014). 
9. Centers for Disease Control and Prevention. Suicide among adults aged 35-64 years—United 
States, 1999- 2010. 62 (17):321-325 (2013). 

 
Version 3.0  
Landes IRB Protocol – Version 11/01/2018 
Page 22 of 24 
 
10. Xu, J., Kochanek, K. D., Murphy, S. L. & Arias, E. Mortality in the United States, 2012. NCHS Data 
Brief 1–8 (2014). 
11. Katz, I. R., McCarthy, J. F., Ignacio, R. V. & Kemp, J. Suicide among veterans in 16 states, 2005 to 
2008: comparisons between utilizers and nonutilizers of Veterans Health Administration (VHA) 
services based on data from the National Death Index, the National Violent Death Reporting System, 
and VHA administrative records. American journal of public health 102, S105–S110 (2012). 
12. Hoffmire, C. A., Kemp, J. E. & Bossarte, R. M. Changes in Suicide Mortality for Veterans and 
Nonveterans by Gender and History of VHA Service Use, 2000-2010. Psychiatr Serv 66, 959–965 
(2015). 
13. Suicide Prevention Resource Center. Suicide prevention basics. (2016). 
14. Quinlivan, L. et al. Which are the most useful scales for predicting repeat self-harm? A systematic 
review evaluating risk scales using measures of diagnostic accuracy. BMJ Open 6, e009297 (2016). 
15. Murray, D. Is it time to abandon suicide risk assessment? British Journal of Psychiatry Open 2, e1–
e2 (2016). 
16. Carter, G. et al. Royal Australian and New Zealand College of Psychiatrists clinical practice guideline 
for the management of deliberate self-harm. Australian & New Zealand Journal of Psychiatry 50, 
939–1000 (2016). 
17. Galfalvy, H. C., Oquendo, M. A. & Mann, J. J. Evaluation of clinical prognostic models for suicide 
attempts after a major depressive episode. Acta Psychiatrica Scandinavica 117, 244–252 (2008). 
18. Kessler, R. C. et al. Predicting Suicides After Psychiatric Hospitalization in US Army Soldiers: The 
Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS). JAMA Psychiatry 
72, 49 (2015). 
19. Tran, T. et al. Risk stratification using data from electronic medical records better predicts suicide 
risks than clinician assessments. BMC psychiatry 14, 1 (2014). 
20. Robinson, J. et al. Can receipt of a regular postcard reduce suicide-related behaviour in young help 
seekers? A randomized controlled trial. Early Interv Psychiatry 6, 145–152 (2012). 

 
Version 3.0  
Landes IRB Protocol – Version 11/01/2018 
Page 23 of 24 
 
21. Beautrais, A. L., Gibb, S. J., Faulkner, A., Fergusson, D. M. & Mulder, R. T. Postcard intervention for 
repeat self-harm: randomised controlled trial. Br J Psychiatry 197, 55–60 (2010). 
22. Kapur, N. et al. Messages from Manchester: pilot randomised controlled trial following self-harm. Br J 
Psychiatry 203, 73–74 (2013). 
23. Vaiva, G. et al. Effect of telephone contact on further suicide attempts in patients discharged from an 
emergency department: randomised controlled study. BMJ 332, 1241–1245 (2006). 
24. Stetler, C. B. et al. Role of ‘external facilitation’ in implementation of research findings: a qualitative 
evaluation of facilitation experiences in the Veterans Health Administration. Implementation Science 
1, 23 (2006). 
25. Ritchie, M. J., Dollar, K. M., Kearney, L. K. & Kirchner, J. E. Responding to needs of clinical 
operations partners: transferring implementation facilitation knowledge and skills. Psychiatric 
Services 65, 141–143 (2014). 
26. Kilbourne, A. M. et al. Cluster randomized adaptive implementation trial comparing a standard versus 
enhanced implementation intervention to improve uptake of an effective re-engagement program for 
patients with serious mental illness. Implementation Science 8, 1–14 (2013). 
27. Harvey, G. & Kitson, A. L. Implementing evidence-based practice in healthcare: a facilitation guide. 
(Routledge, Taylor & Francis Group, 2015). 
28. Damschroder, L. J. et al. Fostering implementation of health services research findings into practice: 
a consolidated framework for advancing implementation science. Implementation Science 4, 50 
(2009). 
29. Rogers. Diffusion of innovations. (Free Press, 2003). 
30. Greenhalgh, T., Robert, G., Macfarlane, F., Bate, P. & Kyriakidou, O. Diffusion of innovations in 
service organizations: systematic review and recommendations. Milbank Quarterly 82, 581–629 
(2004). 
31. Rycroft-Malone, J. et al. An exploration of the factors that influence the implementation of evidence 
into practice. J Clin Nurs 13, 913–924 (2004). 

 
Version 3.0  
Landes IRB Protocol – Version 11/01/2018 
Page 24 of 24 
 
32. Kitson, A., Harvey, G. & McCormack, B. Enabling the implementation of evidence based practice: a 
conceptual framework. Qual Health Care 7, 149–158 (1998). 
33. Hayden, P., Frederick, L., Smith, B. & Broudy, A. Developmental Facilitation: Helping Teams 
Promote SystemsChange. Collaborative Planning Project for PlanningComprehensive Early 
Childhood Systems. 2–21 (2001). 
34. Ellis, I., Howard, P., Larson, A. & Robertson, J. From Workshop to Work Practice: An Exploration of 
Context and Facilitation in the Development of Evidence-Based Practice. Worldviews on Evidence-
Based Nursing 2, 84–93 (2005). 
35. Sharp, N. D., Pineros, S. L., Hsu, C., Starks, H. & Sales, A. E. A qualitative study to identify barriers 
and facilitators to implementation of pilot interventions in the Veterans Health Administration (VHA) 
Northwest Network. Worldviews on Evidence-Based Nursing 1, 129–139 (2004). 
36. Elwyn, G. & Hocking, P. Organisational development in general practice: lessons from practice and 
professional development plans (PPDPs). BMC Family Practice 1, (2000). 
37. Kitson, A. L. et al. Evaluating the successful implementation of evidence into practice using the 
PARiHS framework: theoretical and practical challenges. Implement Sci 3, 1 (2008). 
38. Harvey, G. et al. Getting evidence into practice: the role and function of facilitation. Journal of 
advanced nursing 37, 577–588 (2002). 
39. Beebe, J. Rapid assessment process: an introduction. (AltaMira Press, 2001).