Sust-AIns: Sustainment of Suicide Prevention Programs in American Indian Settings
Phase: NA
Conditions: Program Sustainability
Sponsor: Johns Hopkins Bloomberg School of Public Health

Page 1 of 19 
IRB#00008694 – Research Plan  
PI: E Haroz, v1, 3April2018 
Sust-AIns: Sustainment of Suicide Prevention Programs in American Indian Settings   
NCT03755206 
Document Date: April 3, 2018 

Page 2 of 19 
 
 
IRB#00008694 – Research Plan  
PI: E Haroz, v1, 3April2018 
 
JHSPH IRB Research Plan for New Data Collection 
PI Name:   
Emily E. Haroz, MA, MHS, PhD 
 
Study Title:  Sust-AIns: Sustainment of suicide prevention programs in American Indian 
Settings 
IRB No.:   
00008694 
PI Version No. / Date:  V1/April 3, 2018 
 
I. 
Aims of the Study:  Describe the aims/objectives of the research and/or the project’s research questions 
or hypotheses. 
 
The proposed study is part of the research project for a National Institute of Mental Health K01 grant to the PI. The 
overall research project is focused on understanding how to sustain evidenced-based mental and behavioral health 
programs in tribal contexts. The K01 research project has three Aims: 
 
Aim 1: To identify actionable sustainment strategies to promote the sustainment of the Celebrating Life program 
using a group modeling approach. 
Aim 2: To utilize a Community-Based Systems Dynamic modeling approach to aid in prioritizing and testing the 
impact of locally-relevant (Aim 1) and empirically supported sustainment strategies. 
Aim 3: To pilot test the sustainment strategy interventions selected in Aim 2 on the Celebrating Life program’s 
capacity for sustainability across tribal settings using mixed-methods. 
 
This IRB application focuses on Aim 1 of the K01 research project. An application for Aim 2, which is a 
secondary data analysis of existing data, will be submitted at a later date. For Aim 3, an amendment 
application will be submitted to the parent grant application (U19MH113136; JHSPH IRB# 00008138) to 
increase frequency of data collection, and a full application will be submitted to conduct the research 
study of Aim 3.  
 
II. Background and Rationale: Explain why this study is being done.  Summarize briefly what is 
already known about the issue and reference previously published research, if relevant. 
Among the most pressing issues in mental health Dissemination and Implementation research (D&I) is the 
lack of knowledge about how to sustain mental health prevention and treatment services for which there 
is substantial evidence of impact.1 Most D&I studies focus on adoption and initial implementation, with 
few studies (<20) rigorously exploring sustainment.2 The sustainment of services has been described as 
“one of the greatest, yet least understood, challenges in the field of implementation science.”3 The gap in 
knowledge is even greater for low-resource US and global contexts, as few studies have focused on these 
settings.4  
 
This project aims to address this gap by developing and testing sustainment strategies for an evidenced-
based youth suicide prevention program in a low-resource, American Indian (AI) context. Sustaining 
effective suicide prevention programs and maintaining reductions in mortality has proven challenging.5,6 
This project focuses on the Celebrating Life Surveillance and Prevention Program (CL) which combines 
active surveillance with case management and community outreach. Celebrating Life is based on evidence 
indicating long-term follow-up prevents suicide7,8 and has reduced suicide in the White Mountain Apache 
Tribe (WMAT).9 However, sustainment of these types of programs is challenging, particularly maintaining 

Page 3 of 19 
 
 
IRB#00008694 – Research Plan  
PI: E Haroz, v1, 3April2018 
the integral implementation components (i.e. community outreach, efficient follow-up, etc.). Given the 
high burden of suicide in many AI communities,10 there is an urgent need to better understand how to 
sustain effective suicide prevention programs across tribal settings.  
 
My aims leverage the NIMH Southwestern suicide prevention Hub (U19MH113136; JHSPH IRB #8138). The 
long-term goal is to increase sustainment of CL in new tribal contexts. The rationale, supported by local 
stakeholder feedback and experts in AI suicide prevention,6 is that CL sustainment is a priority, but 
methods are needed for identifying and prioritizing sustainment strategies given limited resources and the 
dynamic systems involved. The objectives of this K01 are to identify mechanisms of sustainment, select 
practical strategies to promote these, and test the effects of strategies on CL’s capacity for sustainment in 
other tribal settings.  
 
III. Study Design: 
A. Provide an overview of your study design and methods.  The study design must relate to your 
stated aims/objectives.  Details will be requested later.  If your study also involves analysis of 
existing data, please complete Section XI, “Secondary Data Analysis of Existing Data” in the 
last part of this research plan.  If your study ONLY involves analysis of existing data, please 
use the research plan template for secondary data analysis (JHSPH IRB Research Plan for 
Secondary Data Analysis of Existing Data/Specimens). 
I will use community-based systems dynamic modeling (CBSD),11,12 to select, prioritize, and test locally 
relevant and theoretically grounded sustainment strategies. Community based systems dynamic 
approaches (CBSD) involve six phases: (1) participate, (2) calibrate, (3) simulate, (4) translate, (5) 
evaluate, and (6) iterate. Aim 1 will focus on the participate phase and will involve a series of five 
Focus Group Discussions (FGDs) with stakeholders. These discussions will serve as our group modeling 
sessions, with participants building models of implementation and sustainment based on their own 
experiences. The key products of these FGDs will be explicit models with assumptions laid out 15 in the 
form of causal loop diagrams,20 as well as a prioritized list of strategies to enhance CL’s sustainment.  
 
B. Provide a sample size and a justification as to how you arrived at that number.  If you use 
screening procedures to arrive at a final sample a table may be helpful.    
Sample size is based on purposive sampling for those that knowledgeable about the Celebrating Life 
program and will include N=10-12 stakeholders from each of the following categories: tribal leaders, 
program directors, case managers, and Hub Co-Is, and be identified in collaboration with our local 
research partners.  
 
IV. Participants: 
Describe the study participants and the population from which they will be drawn.  Specify the inclusion 
and exclusion criteria.  If you plan to include children, note their ages and whether you will include 
children in foster care. Note if the participants are particularly vulnerable in terms of cognitive limitations, 
education, legal migration status, incarceration, poverty, or some combination of factors.  
 
Participants will include tribal or community leaders, site directors, program managers, case-managers, co-
Is and other personnel associated with the implementation of the CL program in tribal contexts.  
 

Page 4 of 19 
 
 
IRB#00008694 – Research Plan  
PI: E Haroz, v1, 3April2018 
A. Inclusion Criteria:  
 
1) Knowledgeable about the implementation of the Celebrating Life program 
2) Over the age of 18 
3) Available to participate in required study meetings and assessments 
4) Fluent in English 
 
B. Exclusion Criteria:  
1) Factors identified at baseline that preclude full participation including: being under the influence of a 
substance; active psychosis or mania; any other condition that makes an individual lack capacity to give 
consent. The PI  will evaluate each participate for the exclusion criteria at the start of each Focus Group 
Discussion meeting. The PI  has a Master’s degree in clinical psychology and a PhD in psychiatric epidemiology 
and is well trained in how to recognize people lacking capacity.  
 
V. Study Procedures:  
In this section, provide details of your procedures, particularly as they relate to human subjects. If 
this is a multi-center study, make the role of JHSPH clear.  If the JHSPH will serve as data 
coordinating center, indicate in the sections below which procedures JHSPH will not be 
performing.  Additional information regarding data coordinating centers is requested in a later 
section.  If your study will develop in phases, address each item below by phase. 
A. Recruitment Process: 
1. Describe how you will identify, approach, and inform potential participants about your study.  
Include details about who will perform these activities and what their qualifications are. 
This project is part of a pending K01 award with PI Haroz is leveraging an existing U19 funded 
grant focused on testing novel approaches to youth suicide prevention and dissemination and 
implementation of the Celebrating Life Suicide Surveillance and Case Management system (i.e. 
Southwest Hub for American Indian Youth Suicide Prevention Research; IRB #8138). The PI  has 
been involved many of the initial Hub meetings and been introduced to partners in this way. The PI   
also works closely with local teams. The PI will be primarily responsible for recruiting participants.  
 
Participants will be recruited from current staff. We will aim to recruit participants with a range of 
responsibilities (e.g. Supervisors, Data managers, Case managers). Potential participants from these 
areas will be recruited from the existing lists of local collaborators who are knowledgeable about the 
project and include site supervisors, case managers, hospital leadership and staff, and other 
community or tribal leaders. The investigators will help to identify potential participants from the 
local list of collaborators and staff. Hub co-Is or Hub-partner Investigators will be recruited from our 
list of co-investigators at JHU and from other Hub partner sites. For recruitment, the PI will provide 
all potential participants with an introduction script either by phone or in person, introducing the 
project and stating what is required to participate. If interested, participants will be invited to the first 
focus group discussion meeting and complete a consent form in person immediately prior to its 
beginning.  
 
2. Address any privacy issues associated with recruitment.  If recruitment itself may put 
potential participants at risk (if study topic is sensitive, or study population may be 
stigmatized), explain how you will minimize these risks. 

Page 5 of 19 
 
 
IRB#00008694 – Research Plan  
PI: E Haroz, v1, 3April2018 
The privacy issues associated with recruitment are minimal. One potential issue is whether 
participants will feel as though they have a right to refuse, as the PI is part of the investigative team 
on the overall Hub grant. It will be clear in the introduction script and consent form that participation 
is voluntary and participants can refuse to participate at any time without jeopardizing their 
employment or work on the overall research study.  
 
B. Consent Process:   
1. Describe the following details about obtaining informed consent from study participants.  If 
a screening process precedes study enrollment, also describe the consent for screening. 
a. Who will obtain informed consent, and their qualifications: 
The PI  will obtain informed consent. The PI has a PhD in psychiatric epidemiology and has completed all 
required human subjects research training.  
 
b. How, where, and when the consent discussion(s) will occur: 
Consent will be done individually and in person immediately prior to the focus group 
discussions.  
  
c. The process you will use to determine whether a potential participant meets eligibility criteria: 
We are recruiting from a known group of individuals who are knowledgeable or involved in the 
implementation of the Celebrating Life Suicide Surveillance and Case Management system. It is 
anticipated that almost all potential participants will meet eligibility requirements.  
 
d. Whether you will obtain a signature from the participant or will use an oral consent process: 
We will obtain a signature from the participant.  
 
e. Whether you will obtain a legally authorized representative’s signature for adults lacking 
capacity: 
No 
f. If children are included in the study, if and how you will obtain assent from them: 
N/A 
 
g. If children are included in the study, how you will obtain permission for them to participate from 
their parent, legal guardian, or other legal authority (if child is in foster care or under government 
supervision): 
N/A 
h. If you are seeking a waiver of informed consent or assent, the justification for this request: 
N/A 
i. 
Whether you will include a witness to the consent process and why: 
N/A 

Page 6 of 19 
 
 
IRB#00008694 – Research Plan  
PI: E Haroz, v1, 3April2018 
j. 
If the language is unwritten, explain how you will communicate accurate information to potential 
participants and whether you will use props or audio materials: 
N/A 
 
2. Identify the countries where the research will take place, and the languages that will be 
used for the consent process.   
Country 
Consent Document(s) 
(Adult Consent, Parental 
Permission, Youth Assent, etc.) 
Languages 
United States 
Adult Consent 
English 
      
      
      
      
      
      
 
 
C. Study Implementation:  
1. Describe the procedures that 
participants will undergo.  If 
complex, insert a table below 
to help the reviewer navigate.   
The proposed research focuses on 
phase 1 (“participate”) of a CBSD 
approach.11,21 We will conduct a series 
of five FGDs that will occur during 
planned U19 Hub site meetings or at 
other times that are convenient for participants. Focus Group Discussions (FGDs) will last approximately 
three to four hours each. During FGD 1, participants will be oriented to systems thinking and systems 
modeling. They will be asked to describe CL’s implementation using model building scripts (scripts 1-3). 
Following the FGD, I will add and refine the model diagram based on Implementation Science models and 
theoretical frameworks. This diagram will serve as the basis for the “implementation subsystem” and allow 
modeling of how different aspects of program implementation interact. During FGD 2, we will use model 
building scripts (scripts 1-3) to facilitate discussion of the factors involved in program sustainability. 
Participants will map out these factors to create a “sustainment subsystem.” During FGD 3, I will present 
the models back to participants and to discuss how implementation and sustainment subsystems interact 
to create our full systems dynamic model (script 4).  
 
Figure 3 provides a simplified example of this process. On the left, implementation of the CL program 
depends on tribal government support which helps to reduce suicide rates. Reduction of suicide rates in 
turn help to increase tribal government support. We then expand this model of implementation (on the 
right) to include other relevant program, practice setting, and ecological factors that influence sustainment 
of the CL program. For example, tribal government support increases local acceptance of the program, 
which increases program reach meaning that more people are registered and receive CM, thereby 
decreasing suicide rates and reinforcing tribal government support.     
 

Page 7 of 19 
 
 
IRB#00008694 – Research Plan  
PI: E Haroz, v1, 3April2018 
During FGD 4, we will generate a list of hypothesized strategies (scripts 5 & 6) for how CL sustainability can 
be enhanced based on our overall model of the combined two subsystems. For example, if program reach 
(e.g. delivering case management to all those registered in surveillance system) is thought to be a problem, 
strategies might involve increasing service capacity (i.e. increasing proportion of time allocated to case-
management, changing batching case management visits by geographic location). FGD participants will 
first generate an exhaustive list of strategies for targeted sustainment variables based on hypotheses, as 
well as, strategies known in the literature.23 Finally, we will break into setting specific groups to identify 
any setting specific considerations. Participants will then be asked to prioritize their lists based on 
applicability and feasibility within their settings. The final product will be a list of 5 general and 3 setting 
specific strategies, ranked by priority and feasibility. The models and specific strategies will serve as the 
basis for future systems science modelling based on existing data (IRB application to be submitted at a 
later date).  
 
Finally, during FGD 5, we will review CL’s mission and purpose and use the simulation results as a decision 
tool to help stakeholders select among competing strategies.18,19,21 We will then write sustainability action 
plans with specific steps for implementing each strategy or combination of strategies.24 By the end of FGD 
5, stakeholders will have selected a strategy (or set of strategies) and have an actionable sustainability plan 
for how to implement these strategies in their own contexts.  
 
Fit of CBSD approaches: All FGDs will be recorded. There will be one note taker from the local communities 
who will take notes regarding unspoken interactions and dynamics during the group sessions. At the 
completion of each FGD, we will gather anonymous feedback about the sessions using an evaluation 
survey. FGD recordings, notes and evaluation surveys will be reviewed and analyzed prior to the next 
scheduled FGD. The PI and the research associate will listen to the recordings and mark and transcribe 
sections that relate to how systems thinking approaches fit with participants existing thinking and/or the 
cultural context. 
 
2. Describe the number and type of study visits and/or contacts between the study team and 
the participant, how long they will last, and where/how they will take place. 
There will be five focus group discussions (FGDs). The same 10-12 participants will participate in 
each FGD. Each FGD will last for 3-4 hours for a total of 20 maximum hours spent on this project. 
The FGDs will take place during pre-planned Hub meetings in small conference rooms depending on 
the location of the meetings. Participant will be provided a $50.00 gift card for their participation in 
each of the FGDs.  
 
3. Describe the expected duration of the study from the perspective of the individual 
participant and duration overall. 
The five FGDs will take place over a 2-year time period. 
 
4.   Provide a brief data analysis plan and a description of variables to be derived.  
Following Community Based Systems Dynamics procedures, we will work with participants to elicit 
variables thought to help successfully implement and sustain the CL program. To do with we will 
use model building scripts in FGDs 1 and 2. We will focus on how these variables interact and 
change over time and diagram these in the form of a causal loop diagram. These causal loop 
diagrams then can serve as the basis of for stock and flow diagrams or models that can be used as 

Page 8 of 19 
 
 
IRB#00008694 – Research Plan  
PI: E Haroz, v1, 3April2018 
part of mathematical simulations using existing data (Aim 2 of the overall grant). Analysis of FGD 
recordings to document fit of this approach in the cultural context, will be conducted using the 
constant comparative method developed for use in grounded theory methodology by Glaser and 
Strauss which requires moving iteratively between codes and text to derive themes.25,26  
 
5.   
A. If the study has different arms, explain the process for assigning participants 
(intervention/control, case/control), including the sequence and timing of the assignment. 
 
N/A 
 
B. If human biospecimens (blood, urine, saliva, etc.) will be collected, provide details about who 
will collect the specimen, the volume (ml) and frequency of collection, how the specimen will be 
used, stored, identified, and disposed of when the study is over.  If specimens will be collected 
for use in future research (beyond this study), complete the “Biospecimen Repository” section 
below. 
 
N/A 
 
C. If genetic/genomic analyses are planned, address whether the data will be contributed to a 
GWAS or other large dataset.  Address returning unanticipated incidental genetic findings to 
study participants. 
 
N/A 
 
D. If clinical or laboratory work will be performed at JHU/JHH, provide the JH Biosafety 
Registration Number. 
 
N/A 
 
E. If you will perform investigational or standard diagnostic laboratory tests using human samples 
or data, clarify whether the tests are validated and/or the lab is certified (for example is CLIA 
certified in the U.S.).  Explain the failure rate and under what circumstances you will repeat a 
test.  For all human testing (biomedical, psychological, educational, etc.), clarify your plans for 
reporting test results to participants and/or to their families or clinicians.  Address returning 
unanticipated incidental findings to study participants. 
 
N/A 
 
F. If your study involves medical, pharmaceutical or other therapeutic intervention, provide the 
following information: 
 
a. Will the study staff be blind to participant intervention status?   
N/A 
b. Will participants receive standard care or have current therapy stopped? 
N/A 
c. Will you use a placebo or non-treatment group, and is that justifiable? 

Page 9 of 19 
 
 
IRB#00008694 – Research Plan  
PI: E Haroz, v1, 3April2018 
N/A 
d. Explain when you may remove a participant from the study. 
N/A 
e. What happens to participants on study intervention when the study ends? 
N/A 
f. Describe the process for referring participants to care outside the study, if needed. 
N/A 
 VI. Data Security and Confidentiality Protections: 
A. Personally Identifiable Information (PII): 
Please identify the Personally Identifiable Information (PII) that you may be collecting and 
using at any of the following stages of your study: Recruitment, Consent, and Study 
Implementation. 
Name, signature, initials, or other identifiable code 
☒ 
Geographic identifier: address, GPS location, etc. 
☐ 
Dates:  birth, death, clinical service, discharge, etc. 
☐ 
Contact information:  phone numbers, email address, etc. 
☒ 
ID:  Social Security Number, driver’s license number, etc.  
☐ 
Health record identifiers:  medical record, insurance plan number, etc. 
☐ 
Account numbers 
☐ 
Device identifiers:  e.g., implants 
☐ 
Internet identifiers:  IP address, social media accounts 
☐ 
Biometric identifiers, including finger and voice prints 
☐ 
Audio recordings 
☐ 
Video or full face photographic images 
☐ 
Genomic/genetic data 
☐ 
Any other unique identifying number, characteristic, or code (note: this 
does not mean the unique code assigned by the investigator to code the 
data) 
☐ 
Other: Click here to enter text. 
☐ 
 
B. Recruitment: 
Will you collect identifiers for the purpose of contacting potential participants?               Yes ☒   
No ☐ 
If yes, will you retain the identifiers after the recruitment contact has been made?    Yes ☒   
No ☐ 

Page 10 of 19 
 
 
IRB#00008694 – Research Plan  
PI: E Haroz, v1, 3April2018 
C. Data Collection: 
Collection of data for a research study can take on many forms.  It can be as simple as gathering the 
data with pen and paper or developing an on-line adaptive survey that changes based on the 
participant’s answers.   Regardless of the method, PII collection for the purposes of identifying the 
participants will most likely be collected.   Once collected, the raw data should go through a de-
identification process to further protect PII. 
In what form will you collect and store PII?  When you respond, think of PII collected for 
recruitment, consent, and other study purposes. 
1. Hard Copy/Paper:  Yes ☒   No ☐ 
 
If yes, please answer the following:  
a. How will the data be kept secure during transfer from study collection site to storage site? 
The PI will store the data and keep it in a locked cabinet at the Center for American Indian Health at JHU.  
 
b. Will the data be secured in a locked cabinet or room?   Yes X   No ☐ 
c. If study IDs/Codes are used, will they be stored separately from the study data?    Yes X  No ☐ 
d. Will the hard copy/paper be destroyed after data abstraction and cleaning are complete?    
Yes X  No ☐ 
If No, when do you plan to destroy the hard copies?        
 
2. Electronic:  Yes ☐   No ☒ 
 
If yes, please answer the following: 
a. Will the data be collected/stored on a portable device (laptop, mobile phone, tablet, PDA) 
protected by encryption?    Yes ☐   No ☐ 
 
b. Will study participants use personally owned devices or study-provided devices? 
Personally owned  ☐   Study provided ☐ 
 
c. Is the application/website used for data collection being developed in-house (Hopkins) or by a 
3rd party vendor? 
In-house ☐   3rd party ☐ 
 
If 3rd party, provide the name of vendor and URL:        
 
Identify Mobile Ecosystem (check all that apply)  Apple ☐  Google ☐ Website ☐ 
d. Will the data be stored on a secure server (@JHSPH/on-site)?      Yes ☐   No ☐ 
e. Will the data be stored in the Cloud/Web?    Yes ☐   No ☐ 
f. Will it be encrypted?    Yes ☐   No ☐ 
g. Will you be backing up your data?   Yes ☐   No ☐ 

Page 11 of 19 
 
 
IRB#00008694 – Research Plan  
PI: E Haroz, v1, 3April2018 
3. Audio Recording:  Yes ☒   No ☐ 
 
If yes, please answer the following: 
a. Will you store the audio recording securely in a locked cabinet/room until transcription is 
complete?   
Yes X  No ☐ 
 
b. Will you use a transcription service?    
Yes X  No ☐ 
If yes, if the PII comes from JHH/JHHS, you must use an approved vendor; otherwise, be aware 
of the data security protections that the transcription service provides. 
c. Will the audio recording be destroyed after transcription?   Yes X  No ☐ 
 
If no, why not?         
 
4. Photograph/Video:   Yes ☐   No ☒ 
 
If yes, please answer the following: 
a. Will the photographs/videos be stored securely in a locked cabinet or room?   Yes ☐  No ☐ 
 
b. Will the photograph/video be destroyed?   Yes ☐   No ☐ 
If yes, when?        
D. PII De-Identification of Data Used for this Study:  
1. When will you destroy the PII and/or the code linking the PII with the study ID?   
As soon as data collection is complete 
2. What is the method you will use to de-identify the data?   
Each PII is linked to a unique study ID. Only the study ID will be used on any written material, with the 
exception of a signed consent form. The link between PII and study ID will be stored on a password 
protected, encrypt computer maintained by the PI. This file linking the PII and study ID will be destroyed 
after data collection is completed.  
3. Is your research data governed by HIPAA (U.S. clinical data remaining within the covered entity)? 
Yes ☐  No X 
a. If yes, who is doing the de-identification?         
b. If yes, what level of de-identification will you achieve (Limited data set?  De-identified?)        
 
E. Data Storage and Analysis: 
One of the keys to protecting PII is the proper use of tools to share and conduct your analysis.  JH and 
JHSPH offers several options for you to consider.  Please select the system that you plan to use to 
protect your study data by clicking the box.  Consult JHSPH IT for assistance if needed. 
 
☐   JH Virtual Desktop:  The Hopkins Institute for Clinical and Translational Research (ICTR) 
provides a virtual Windows desktop (SAFE Desktop).  It includes productivity software 

Page 12 of 19 
 
 
IRB#00008694 – Research Plan  
PI: E Haroz, v1, 3April2018 
such as Microsoft Word and Excel, as well as statistical software, including SAS, Stata, R, 
R Studio, and Python.  100 GB of storage space is provided.  
☐    JHSPH SharePoint and File Shares:  These systems provide a managed and secure 
platform for your research project. They also provide a built-in encrypted backup solution. 
☐    JHSPH RedCAP:  These are departmentally managed applications. RedCAP is an 
application designed for collaborative research projects.   
☐    JHSPH HPCC: High Performance Computing Cluster (HPCC:  https://jhpce.jhu.edu/) can 
provide the high capacity computing required for very large data sets.   
☒    JHBox:  Johns Hopkins Box (JHBox) is a secure cloud-based file sharing service which 
enables people to collaborate and share information and may be accessed through any 
device: desktop, laptop, phone, or tablet with appropriate permissions. JHSPH IT 
recommends that investigators not use JHBox as a primary storage location, but use it 
instead for initial data collection, sharing results, and other collaborative information with 
the research team.   
☐   Independent Departmental Servers and Systems:  These servers are typically managed 
by departmental or research team IT staff.  Because these servers are not centrally 
managed by JHSPH IT, all documentation regarding data security protections will need to 
be provided by the owner/administrator of the server.  This responsibility may fall to the 
data owners (PI). 
☐   Other:  Please provide details regarding any other systems being utilized.  Examples may 
include servers and applications located at another university participating in your study or 
a 3rd party web-based application.         
 
F. Other Data Security Measures: 
In addition to the details regarding data collection, please review the following questions.  This 
additional information will be utilized to assist in the development of a comprehensive Data Security 
plan.  This would include the systems used to analyze the data, data security contacts and additional 
requirements. 
 
1. During the analysis phase, do you plan to use computer systems that are not managed by JHSPH 
or JH?  Yes ☐   No X 
If yes, please explain:        
 
2. Do you have a designated person on your research team other than the PI who is the technical 
contact for a Data Security plan?   Yes ☐   No X 
If yes, please provide a contact name:        
 
 
3. Does your sponsor have other specific data security requirements for the study data?  Yes ☐  No X 
If possible, please explain:         
 
4. Please add any other information that you believe is relevant to data security.        
 
G. Certificate of Confidentiality: 

Page 13 of 19 
 
 
IRB#00008694 – Research Plan  
PI: E Haroz, v1, 3April2018 
All NIH studies include Certificate of Confidentiality protections with the grant; the consent form 
must include the C of C language provided in our template.  Other funders may obtain C of C 
protections through NIH.  (https://humansubjects.nih.gov/coc/index)  
Does the study have Certificate of Confidentiality protections?   
Yes X  No ☐   
 
H. JHM Clinical Records: 
Will you use clinical data of 500 records or more from Johns Hopkins Hospital and its affiliates?  
  Yes ☐   No X 
  
If yes, please complete the JHM Data Security Checklist available on the JHSPH IRB website: 
www.jhsph.edu/irb and upload a copy of the checklist to the “Miscellaneous” section. 
VII.  Risks of the Study: 
A. Describe the risks, discomforts, and inconveniences associated with the study and its procedures, 
including physical, psychological, emotional, social, legal, or economic risks, and the risk of a breach of 
confidentiality.  These risks should be described in the consent documents. 
There are minimal risks associated with this study. One risk is if the participants are critical of the 
program or local context and tribal activities. During FGDs, we will refrain from using identifying 
information and ensure that no written documents have identifying information included. We will also 
stress at the beginning of the FGD the importance of maintaining confidentiality about what is said 
during the discussion. Another risk may be an inconvenience with attending five, 3-4 hour meetings 
over the course of two years. During meetings snacks, coffee and water will be provided to all 
participants to help make them more comfortable. Participants will also receive an incentive to 
participate.  
 
B. Describe the anticipated frequency and severity of the harms associated with the risks identified above; 
for example, if you are performing “x” test/assessment, or dispensing “y” drug, how often do you expect 
an “anticipated” adverse reaction to occur in a study participant, and how severe do you expect that 
reaction to be?   
We do not anticipate any harms associated with this study.  
 
C. Describe steps to be taken to minimize risks.  Include a description of your efforts to arrange for care or 
referral for participants who may need it. 
Risks are considered minimal, but cannot be fully eliminated. We will take the following steps to 
minimize risks: 
 
1. Institutional Review Boards at JHSPH and locally. The study research plan, study instruments and consent 
forms will be reviewed by an IRB at Johns Hopkins School of Public Health and by the local IRBs. 
2. Informed consent.  We will provide potential respondents a clear opportunity to understand the 
study objectives and procedures, to assess study risks and benefits, and, verbally, to give or decline 
consent or assent freely and without pressure or penalty. 
3. Focus Group discussions in a safe, private location.  The study will rely on the local expertise and 
judgment of local field staff and supervisors to identify where interviews might best be carried out.   
4. Data Storage and Protection.  No unique identifiers will be collected in data collection forms or 
handwritten notes and the location of interviews will not be recorded. 

Page 14 of 19 
 
 
IRB#00008694 – Research Plan  
PI: E Haroz, v1, 3April2018 
5. Privacy concerns: At the beginning of each FGD, the importance of not using names and identifying 
information will be reviewed so as to maintain confidentiality of participants.   
 
D. Describe the research burden for participants, including time, inconvenience, out of pocket costs, etc.  
Participants will be asked to participate in five, 3-4 hour focus group discussions.  
 
E. Describe how participant privacy will be protected during data collection if sensitive questions are 
included in interviews. 
No sensitive questions will be included in the interview. However, we will still refrain from using PII in 
the interviews so as to maintain privacy.  
 
VIII. Direct Personal and Social Benefits: 
A. Describe any potential direct benefits the study offers to participants (“payment” for participation is not a 
direct personal benefit). 
There will be no direct benefits to the study participants.  
 
B. Describe potential societal benefits likely to derive from the research, including value of knowledge 
learned. 
This research will provide knowledge about what contributes to implementation and sustainability of 
mental and behavioral health programs in tribal settings. We know very little about how to sustain 
evidence-based programs, and this research is intended to generate preliminary evidence about what 
might help contribute to sustaining these types of programs in low resource contexts.  
 
IX.  
Payment:   
A. Describe the form, amount, and schedule of payment to participants. Reimbursement for travel or other 
expenses is not “payment,” and if the study will reimburse, explain. 
The study will reimburse for travel and associated costs to attend the FGDs. In addition, participants will 
receive an incentive of $50 per FGD.  
 
B. Include the possible total remuneration and any consequences for not completing all phases of the 
research. 
If participants completed all 5 FGDs, they will receive up to $250 in gift cards. If they do not participate 
in all FGDs, they will not receive this total amount, but there will be no other consequences for not 
completing the project. Additional other amounts will include travel and accommodation costs of 
approximately $150 per participant per FGD meeting. Depending on the location of the meeting and 
who is traveling, this is not anticipated to be paid for all participants. These FGDs are taking place 
during pre-planned Hub activities (parent grant) and some participants will already be traveling and have 
costs covered by the Hub grant.  
 
X. Study Management: 
A. Oversight Plan: 
 
 
 
1. Describe how the study will be managed. 

Page 15 of 19 
 
 
IRB#00008694 – Research Plan  
PI: E Haroz, v1, 3April2018 
The study will be managed by Dr. Emily Haroz for the duration of study collection. Dr. Haroz will 
work with Drs. Barlow, Cwik, Gallo and Ms. Goklish to provide management and oversight of all 
field and data collection activities. The PI and co-investigators will ensure adherence to the IRB 
approved research plan. 
2. What are the qualifications of study personnel managing the project? 
Dr. Haroz is an Assistant Scientist in the Department of Mental Health. She has worked with the 
Center for American Indian health and field staff for the last 1.5 years. She will be assisted by Drs. 
Barlow and Cwik who are the PIs of the parent grant this K01 is leveraging (Hub grant). Ms. 
Goklish is a research associated at JHU and has a master’s in social work and has led field based 
activities for the Center for American Indian Health for the last 20 years.  
3. How will personnel involved with the data collection and analysis be trained in human subjects 
research protections?  (Use the JHSPH Ethics Field Training Guide available on the JHSPH IRB 
website: www.jhsph.edu/irb.) 
Dr. Haroz will be the primary person involved with data collection. Other faculty on the application 
have existing CITI certificates that will remain current throughout the course of this project. Any 
field staff who assist with data collection will be trained using the JHSPH Ethics Field Training Guide.  
4. If the PI will not personally be on-site throughout the data collection process, provide details about 
PI site visits, the supervision over consent and data collection, and the communication plan 
between the PI and study team. 
The PI will be personally on site throughout data collection processes.  
 
B. Recordkeeping:   
Describe how you plan to ensure that the study team follows the protocol and properly records 
and stores study data collection forms, IRB regulatory correspondence, and other study 
documentation.  For assistance, contact housecall@jhu.edu. 
Consent forms, and handwritten notes will be stored in a locked filing cabinet at JHU that will only be 
accessible by the PI. No data or forms will be accessed or allowed to be taken out of the room directly 
by anyone other than them. The PI  will take responsibility for ensuring that files are destroyed once the 
study is complete.  
 
 
C.  Safety Monitoring: 
1. Describe how participant safety will be monitored as the study progresses, by whom, and 
how often.  Will there be a medical monitor on site?  If yes, who will serve in that role? 
 
Participant safety is not considered to be at risk due to participation in this qualitative study. There 
will not be a medical monitor on site for this research. 
 
2. If a Data Safety Monitoring Board (DSMB), or equivalent will be established, describe the 
following: 
 
a.   The DSMB membership, affiliation and expertise. 
 
N/A 
 

Page 16 of 19 
 
 
IRB#00008694 – Research Plan  
PI: E Haroz, v1, 3April2018 
b. The charge or charter to the DSMB. 
 
N/A 
 
c.   Plans for providing DSMB reports to the IRB. 
 
N/A 
 
5. Describe plans for interim analysis and stopping rules, if any. 
Transcripts and related materials from each FGD will be analyzed iteratively to inform the following 
FGD.  
 
D.  Reporting Unanticipated Problems/Adverse Events (AEs) to the IRB (all studies must 
complete this section):   
Describe your plan for reporting to the IRB and (if applicable) to the sponsor.  Include your plan 
for government-mandated reporting of abuse or illegal activity.  
The Principal Investigator will report serious adverse events to the JHSPH IRB.  If there is any abuse 
reported during a focus group, the PI will report this to appropriate Tribal Authorities. With experience 
from previous studies with the program communities, the study team is knowledgeable of how to 
report this type of event without revealing study participation. 
 
NOTE: The IRB does not require PROMPT reporting of all AEs, only those that are 
unanticipated, pose risk of harm to participants or others, and are related to the 
study.  Anticipated AEs may be reported with the Progress Report. 
 
E.  Other IRBs/Ethics Review Boards:   
If other IRBs will review the research, provide the name and contact information for each 
IRB/ethics review board and its Federal Wide Assurance, if it has one (available on OHRP’s 
website at http://www.hhs.gov/ohrp/assurances).   
 
1. Phoenix Area Indian Health Service: Dr. Cynthia Claus; (602) 354-5169; Cynthia.claus@ihs.gov.  
2. Navajo Nation Human Research Review Board (FWA: 00000641): Ms. Beverly Becenti-Pigman, (928) 871-
6650. 
 
F. Collaborations with non-JHSPH Institutions:  
For studies that involve collaboration with non-JHSPH institutions, complete the chart below by 
describing the collaboration and the roles and responsibilities of each partner, including the 
JHSPH investigator.  This information helps us determine what IRB oversight is required for 
each party. Complete the chart for all multi-collaborator studies. 
 
Insert Name of Institutions in Partner column(s); add additional columns if necessary.   
 
JHSPH 
Partner 
1 
      
Partner 
2 
      

Page 17 of 19 
 
 
IRB#00008694 – Research Plan  
PI: E Haroz, v1, 3April2018 
Primary Grant Recipient 
X 
      
      
Collaborator 
      
      
      
 
For the following, indicate “P” for “Primary”, “S” for “Secondary” (as appropriate to 
role and level of responsibility.)  Add additional items if useful.  
1. 
Human subjects research ethics training for data 
collectors 
P 
      
      
2. 
Day to day management and supervision of data 
collection 
   P    
      
      
3. 
Reporting unanticipated problems to the JHSPH 
IRB/Sponsor 
   P    
      
      
4. 
Hiring/supervising people obtaining informed consent 
and/or collecting data 
  P   
      
      
5. 
Execution of plan for data security/protection of 
participant data confidentiality, as described in the Data 
Security and Confidentiality Protections section above 
    P 
    
      
      
6. 
Biospecimen processing, storage, management, 
access, and/or making decisions about future use 
N/A 
      
      
 
 
 
 
 
 
 
 

Page 18 of 19 
 
 
IRB#00008694 – Research Plan  
PI: E Haroz, v1, 3April2018 
 
 
References 
 
1. 
Proctor E, Luke D, Calhoun A, et al. Sustainability of evidence-based healthcare: research agenda, 
methodological advances, and infrastructure support. Implementation Science. 2015;10(1):88. 
2. 
Stirman SW, Kimberly J, Cook N, Calloway A, Castro F, Charns M. The sustainability of new programs and 
innovations: a review of the empirical literature and recommendations for future research. Implementation 
Science. 2012;7(1):17. 
3. 
Palinkas LA, Spear SE, Mendon SJ, et al. Measuring sustainment of prevention programs and initiatives: a study 
protocol. Implementation Science. 2016;11(1):95. 
4. 
Spoth R, Rohrbach LA, Greenberg M, et al. Addressing core challenges for the next generation of type 2 
translation research and systems: The translation science to population impact (TSci Impact) framework. 
Prevention Science. 2013;14(4):319-351. 
5. 
Knox KL, Pflanz S, Talcott GW, et al. The US Air Force suicide prevention program: implications for public health 
policy. American Journal of Public Health. 2010;100(12):2457-2463. 
6. 
Wexler L, Chandler M, Gone JP, et al. Advancing suicide prevention research with rural American Indian and 
Alaska Native populations. Journal Information. 2015;105(5). 
7. 
Luxton DD, June JD, Comtois KA. Can postdischarge follow-up contacts prevent suicide and suicidal behavior? 
Crisis. 2013. 
8. 
Robinson J, Yuen HP, Gook S, et al. Can receipt of a regular postcard reduce suicide-related behaviour in young 
help seekers? A randomized controlled trial. Early intervention in psychiatry. 2012;6(2):145-152. 
9. 
Cwik MF, Tingey L, Maschino A, et al. Decreases in Suicide Deaths and Attempts Linked to the White Mountain 
Apache Suicide Surveillance and Prevention System, 2001–2012. American journal of public health. 
2016;106(12):2183-2189. 
10. 
Health UDo, Services H. Trends in Indian health: 2014 edition. 2014. 
11. 
Hovmand P. Community based system dynamics. Springer Science & Business Media; 2013. 
12. 
Luke DA, Stamatakis KA. Systems science methods in public health: dynamics, networks, and agents. Annual 
review of public health. 2012;33:357-376. 
13. 
Carey G, Malbon E, Carey N, Joyce A, Crammond B, Carey A. Systems science and systems thinking for public 
health: a systematic review of the field. BMJ open. 2015;5(12):e009002. 
14. 
Frerichs L, Lich KH, Dave G, Corbie-Smith G. Integrating systems science and community-based participatory 
research to achieve health equity. American journal of public health. 2016;106(2):215-222. 
15. 
Peters DH. The application of systems thinking in health: why use systems thinking? Health research policy and 
systems. 2014;12(1):51. 
16. 
Powell BJ, Beidas RS, Lewis CC, et al. Methods to improve the selection and tailoring of implementation 
strategies. The journal of behavioral health services & research. 2015:1-18. 
17. 
Zimmerman L, Lounsbury DW, Rosen CS, Kimerling R, Trafton JA, Lindley SE. Participatory system dynamics 
modeling: Increasing stakeholder engagement and precision to improve implementation planning in systems. 
Administration and Policy in Mental Health and Mental Health Services Research. 2016;43(6):834-849. 
18. 
Pagea A, Atkinsonb J-A, Heffernand M, McDonnellb G, Hickiee I. A decision-support tool to inform Australian 
strategies for preventing suicide and suicidal behaviour. Public health research and practice. 2017;27(2). 
19. 
Atkinson J-A, Page A, Wells R, Milat A, Wilson A. A modelling tool for policy analysis to support the design of 
efficient and effective policy responses for complex public health problems. Implementation science. 
2015;10(1):26. 
20. 
Bala BK, Arshad FM, Noh KM. Causal Loop Diagrams. System Dynamics: Springer; 2017:37-51. 
21. 
Sterman JD. Learning from evidence in a complex world. American journal of public health. 2006;96(3):505-514. 
22. 
Schell SF, Luke DA, Schooley MW, et al. Public health program capacity for sustainability: a new framework. 
Implementation Science. 2013;8(1):15. 

Page 19 of 19 
 
 
IRB#00008694 – Research Plan  
PI: E Haroz, v1, 3April2018 
23. 
Powell BJ, McMillen JC, Proctor EK, et al. A compilation of strategies for implementing clinical innovations in 
health and mental health. Medical care research and review. 2012;69(2):123-157. 
24. 
Calhoun A, Mainor A, Moreland-Russell S, Maier RC, Brossart L, Luke DA. Peer Reviewed: Using the Program 
Sustainability Assessment Tool to Assess and Plan for Sustainability. Preventing chronic disease. 2014;11. 
25. 
Glaser BG, Strauss AL. The Discovery of Grounded Theory: Strategies for Qualitative Research. New York: Aldine 
Publishing; 1967. 
26. 
Boeije H. A purposeful approach to the constant comparative method in the analysis of qualitative interviews. 
Quality & Quantity. 2002;36:391-409. 
 

August 2, 2024 
 
Sust-AIns: Sustainment of Suicide Prevention Programs in American Indian 
Settings 
    NCT03755206 
 
Study Protocol and Statistical Analysis Plan 
 
Data sources. To calibrate the model and check our modeling assumptions, we used data from an 
annual stakeholder survey of program implementers. The survey administered the Program 
Sustainability Assessment Tool (PSAT; Luke et al., 2014), which uses the PHPCS framework to 
measure program capacity for sustainability. The PSAT was administered annually from 2018-
2022; additional items from the Sustainment Measurement System Scale (Palinkas, 2020) and 
questions assessing the impact of COVID-19 were added in 2021. While much implementation 
science research naturally focuses on implementation outcomes, we also considered sustainment 
strategies’ impact on clinical outcomes. To do so, we used data from participants of a larger study 
and data from a surveillance system, which recorded suicide-related clinical events.  
 
Modeling program sustainability. We modeled key barriers’ dynamic impact on program 
sustainability outcomes. We represented some barriers as model “stocks,” or elements that 
function as “containers” (e.g., of people, money, personal trust, etc.) whose levels can 
accumulate or deplete over time, and represented others as factors impacting stock levels (i.e., 
“inflow” or “outflow” rates from the stocks). We populated the model with data from the 
stakeholder surveys, using Python 3.7 to conduct constrained optimization for model calibration 
(Van Rossum & Drake, 2009).  
 
Modeling clinical outcomes. We used microsimulations to model client-level suicidal behaviors 
and to estimate the effects of program sustainability interventions on these health outcomes. We 
developed a multi-state model (Williams et al., 2017) to describe how people transition between 
various potential “states” of suicidal behaviors. 
 
Simulations. We conducted computational simulations of various intervention scenarios using 
the models described above. We designated December 2018 as the start of the simulated time 
frame (Month 0) and calculated outputs for the following 60 months. We conducted four 
simulation scenarios to evaluate key strategies’ impact on program sustainment outcomes and 
clinical outcomes (operationalized as the incidence of non-lethal suicidal behavior—ideation, 
self-harm, and attempts). Factors that contribute to sustainment were defined as our outcomes 
and were represented as stocks whose levels we tracked under each simulated scenario. These 
included: resources available, program services available, community trust in program, 
community and external collaborators, number of staff, and number of clients. 
 
We fitted Model 1 (Baseline) to reflect observed trends in the stakeholder survey data from 
December 2018 through October 2022, without simulating any sustainment strategies.  
 
For Model (2), we examined the impact of an Increased Funding strategy (corresponding to the 
PHPCS “funding availability” domain). We simulated an increase in funding sources by creating 

a dummy constant stock called increased funding, which flowed into resources available at a rate 
of 0.01 (initiating at Month 0). 
 
For Model (3), we examined the impact of a multifaceted approach to Enhanced Program 
Management (corresponding to the PHPCS “organizational capacity” domain). We 
conceptualized this strategy as decreasing conflicting priorities, measured by survey questions on 
whether funding comes from diversified sources, whether organizational leaders efficiently 
manage program resources and staff, and whether the organization has capacity to supervise 
program providers. We simulated this intervention by decreasing the value of conflicting 
priorities by 10% in Month 0. 
 
Finally, for Model (4), we examined the impact of a hypothetical Leadership Development 
intervention (corresponding to the PHPCS “environmental support” domain). To simulate this 
strategy, we directly increased the value of strong program leadership by 10% in Month 0. 
 
We performed sensitivity analyses to test the robustness of our assumptions and estimate 
uncertainty for our simulated results. We used Approximate Bayesian Computation to generate 
sets of system dynamic model parameters (Sunnåker et al., 2013) and then sampled multiple sets 
of parameter values using Latin hypercube sampling (Loh, 1996). 
 
 
References 
 
Loh, W. L. (1996). On Latin hypercube sampling. The annals of statistics, 24(5), 2058-2080. 
 
Luke DA. The Program Sustainability Assessment Tool: a new instrument for public health 
programs. Preventing chronic disease. 2014;11.  
 
Palinkas, Lawrence A., Chih-Ping Chou, Suzanne E. Spear, Sapna J. Mendon, Juan Villamar, 
and C. Hendricks Brown. “Measurement of Sustainment of Prevention Programs and Initiatives: 
The Sustainment Measurement System Scale.” Implementation Science 15, no. 1 (December 
2020): 71. 
 
Sunnåker M, Busetto AG, Numminen E, Corander J, Foll M, Dessimoz C. Approximate 
Bayesian Computation. PLOS Computational Biology. 2013;9(1):e1002803. 
doi:10.1371/journal.pcbi.1002803 
 
Van Buuren, S., & Groothuis-Oudshoorn, K. (2011). mice: Multivariate imputation by chained 
equations in R. Journal of statistical software, 45, 1-67. 
 
Van Rossum, G., & Drake, F.L. (2009). Python 3 Reference Manual. Scotts Valley, CA: 
CreateSpace. 
 
Williams C, Lewsey JD, Briggs AH, Mackay DF. Cost-effectiveness Analysis in R Using a 
Multi-state Modeling Survival Analysis Framework: A Tutorial. Med Decis Making. 
2017;37(4):340-352.