Investigating the Mental Health Impact of COVID-19 and Comparing the Effectiveness of Two Caring Contact Interventions on Patients, Providers, and Staff of St. Luke's Health System
Phase: NA
Conditions: Covid19, Mental Health
Sponsor: St. Luke's Health System, Boise, Idaho

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 1 of 72
Mental Health Among Patients, Providers, and Staff (MHAPPS): Investigating the 
Mental Health Impact of COVID-19 and Comparing the Effectiveness of Two 
Caring Contact Interventions 
Principal Investigator
Anna Radin, DrPH, MPH, Applied Research Scientist, St. Luke’s Health System; 208-381-8468; 
radina@slhs.org 
Confidentiality Statement
This document is confidential communication.  Acceptance of this document constitutes agreement by the recipient 
that no unpublished information contained herein will be published or disclosed without prior approval of the 
Principal Investigator.

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 2 of 72
Table of Contents
List of Acronyms ...........................................................................................................................................6
Statement of Compliance.............................................................................................................................8
Statement of Attribution for Protocol Template..........................................................................................8
Funding for the MHAPPS Trial ......................................................................................................................8
Investigator’s Signature................................................................................................................................9
1.
Protocol Summary .............................................................................................................................10
1.1 Synopsis............................................................................................................................................10
1.2 Schema / Study Flow Diagram..........................................................................................................12
2.
Research Question & Specific Aims...................................................................................................13
3.
Introduction .......................................................................................................................................13
3.1 Background.......................................................................................................................................13
3.2 Significance.......................................................................................................................................16
4.
Research Methods .............................................................................................................................17
4.1 Study Design .....................................................................................................................................17
4.1.1. Aim 1: Cross Sectional Survey...................................................................................................17
4.1.2 Aim 2: Comparative Effectiveness Trial .....................................................................................17
4.2 Randomization..................................................................................................................................17
4.3 Masking ............................................................................................................................................17
5.
Study Population................................................................................................................................17
5.1 Inclusion & Exclusion Criteria ...........................................................................................................18
5.1.1 Inclusion & Exclusion Criteria: Aim 1 .........................................................................................18
5.1.2 Inclusion & Exclusion Criteria: Aim 2 .........................................................................................18
5.2 Recruitment Strategy........................................................................................................................20
5.3 Retention..........................................................................................................................................20
5.4 Populations for Analyses ..................................................................................................................20
6.
Study Procedures ...............................................................................................................................21
6.1 Schedule of Activities........................................................................................................................22
6.2 Safety Procedures for Participants at Risk for Suicide......................................................................23

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 3 of 72
7.     Causal Framework..............................................................................................................................23
8.
Exposures and Outcomes (Variables) of Interest..............................................................................24
8.1 Exposure Variables ...........................................................................................................................24
8.1.1 Rationale for Including Selected Exposure Variables.................................................................24
8.2 Primary Outcome & Assessment......................................................................................................26
8.3
Secondary Outcomes & Assessment ..........................................................................................27
8.3.1
Suicidal ideation..................................................................................................................27
8.3.2 Scoring the C-SSRS.....................................................................................................................28
8.3.3 Risk Factors for Suicide (INQ).....................................................................................................28
8.3.4 Stress .........................................................................................................................................28
8.3.5 Alcohol and Illicit Drug Use........................................................................................................29
8.3.6 Depression.................................................................................................................................29
8.3.7 Anxiety.......................................................................................................................................29
8.3.9 Uptake of Outpatient Mental Health Services...........................................................................29
9.    Description of Interventions...............................................................................................................29
9.1 Caring Contacts with Introductory Call (CC+) ...................................................................................29
9.2 Caring Contacts without an Introductory Call (CC)...........................................................................30
9.3 Definition of Full Intervention ..........................................................................................................30
9.4 Process Evaluation............................................................................................................................30
10.
Discontinuation and Participant Withdrawal ...............................................................................31
10.1 Participant Discontinuation and Withdrawal from the Study ........................................................31
10.2 Lost to Follow-Up............................................................................................................................31
11.
Data & Safety Monitoring..............................................................................................................32
11.1 Overview of Data & Safety Monitoring Plan...................................................................................32
11.2 Role of the Data & Safety Monitoring Board (DSMB).....................................................................32
11.3 Safety Outcomes.............................................................................................................................32
11.4 Adverse Events and Serious Adverse Events ..................................................................................32
12.
Data Collection & Management ....................................................................................................33
12.1
Data Collection ...........................................................................................................................33

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 4 of 72
12.2 Data Management..........................................................................................................................35
13.
Statistical Analysis..........................................................................................................................36
13.1 General Analytic Approach.............................................................................................................36
13.2 Power & Sample Size for Aim 2 ......................................................................................................37
The total sample size for the aim 2 trial is 660 participants.  For each stratum (patients, 
providers/staff), a sample of 330 subjects (165 in each arm) allows 80% power to detect a difference 
of 5 units in the primary outcome (loneliness) if a minimum of 70% of participants are retained 
through study completion......................................................................................................................37
13.3 Duplicate Surveys ...........................................................................................................................37
13.4 Missing Data ...................................................................................................................................37
13.5 Sub-Group Analyses........................................................................................................................38
14.
Engaging Providers & People with Lived Experience with Suicide...............................................38
15.
Risk / Benefit Analysis ...................................................................................................................39
15.1 Potential Risks.................................................................................................................................39
15.2 Potential Benefits ...........................................................................................................................40
16.
Oversight for Human Subjects Protection & Regulatory Considerations.....................................40
16.1 Human Subjects Protection ............................................................................................................40
16.2 Risks to Human Subjects.................................................................................................................41
16.2.1 Characteristics of the Study Population ..................................................................................41
16.2.2 Involvement of Human Subjects..............................................................................................41
16.2.3 Protecting Individuals with Urgent Clinical Needs...................................................................41
16.2.4 Randomization.........................................................................................................................42
16.3 Informed Consent Process..............................................................................................................42
16.3.1 Documentation of Informed Consent......................................................................................43
16.4 Bioethics Consultation....................................................................................................................43
16.5 Inclusion of Women and Minorities ...............................................................................................43
16.6 Inclusion of Minors .........................................................................................................................44
16.7 Cost and Compensation for Participation.......................................................................................44
16.7.1 Participant Compensation .......................................................................................................44
16.8 Study Discontinuation and Closure.................................................................................................45

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 5 of 72
16.9 Confidentiality & Privacy ................................................................................................................45
16.9.1 Measures to Ensure Confidentiality of Shared Data................................................................46
16.10 Study Records Retention ..............................................................................................................46
16.11 Publication & Data Sharing Policy.................................................................................................46
16.12 Dissemination of Results ..............................................................................................................47
16.13 Conflict of Interest Policy..............................................................................................................47
16.14 Protocol Deviations ......................................................................................................................47
16.15 Key Roles for Study Oversight.......................................................................................................47
16.15.1 Data and Safety Monitoring Board........................................................................................48
16.16 Quality Assurance & Quality Control............................................................................................48
17.
References......................................................................................................................................49
APPENDIX B: C-SSRS Screening Questions & Scoring Criteria....................................................................72

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 6 of 72
List of Acronyms
AE
Adverse Event
AIMS
Advancing Integrated Mental Health Solutions
CFR
Code of Federal Regulations
C-SSRS
Columbia Suicide Severity Rating Scale
DSMB
Data & Safety Monitoring Board
ED
Emergency Department
GAD-7
Generalized Anxiety Disorder-7
HIPAA
Health Insurance Portability and Accountability Act of 1996
Hotline
Idaho Suicide Prevention Hotline
ICH GCP
International Council on Harmonisation Good Clinical Practice 
INL
Idaho National Laboratory
INQ
Interpersonal Needs Questionnaire 
IRB
Institutional Review Board
ITHS
Institute of Translational Health Sciences
MHAPPS
Mental Health Among Patients, Providers, and Staff (short title of trial)
NIH
National Institutes of Health
NIMH
National Institute of Mental Health
OHRP
Office of Human Research Protections
PCORI
Patient Centered Outcomes Research Institute
PHI
Protected Health Information
PHQ-9
Patient Health Questionnaire-9 
PHQ-A
Patient Health Questionnaire for Adolescents

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 7 of 72
PI
Principal Investigator
PLES
People with Lived Experience with Suicide
SAE
Serious Adverse Event
SLHS
St. Luke’s Health System
SPARC
Safety Planning Among Recipients of Care
UAP
Unanticipated Problem
US
United States
USPS Task Force 
United States Preventive Services Task Force
VA
Veterans Administration
UW
University of Washington

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 8 of 72
Statement of Compliance
The trial will be conducted in accordance with International Council on Harmonisation Good Clinical 
Practice (ICH GCP), and the Patient Centered Outcomes Research Institute (PCORI) Terms and Conditions 
of Award.  The Principal Investigators will assure that no deviation from, or changes to the protocol will 
take place without prior agreement from the funding agency and documented approval from the St. 
Luke’s Institutional Review Board (IRB), except where necessary to eliminate an immediate hazard to the 
trial participants.  All personnel involved in the conduct of this study have completed Human Subjects 
Protection and ICH GCP Training.
The protocol, informed consent form, recruitment materials, and all participant-facing materials will be 
submitted to the St. Luke’s IRB for review and approval.  Approval of the protocol and all relevant 
documents must be obtained before any participant is consented and enrolled in the study.  In addition 
to St. Luke’s Health System (SLHS) IRB approval, St. Luke’s Research Final Authorization will be in place 
before study activities begin. Any amendment to the protocol or supporting documents will require 
review and approval by the IRB before the changes are implemented to the study.  All changes to the 
consent forms will be IRB approved.  Depending on the extent of changes, the research team and/or the 
IRB will determine whether participants who provided consent using a previously approved consent 
form need to be reconsented using the revised consent form.
Statement of Attribution for Protocol Template
Significant portions of the outline and content for this protocol were adapted based on or directly 
copied from sample text provided in the National Institutes of Health Behavioral and Social Intervention 
Clinical Trial Protocol Template (v3.0 – 20180827).  Because this publicly available resource was used 
extensively in developing this protocol, we are including this statement of attribution in lieu of individual 
citations for this reference.  Other references used are cited accordingly and listed in the References 
section of this protocol.
Funding for the MHAPPS Trial
The research described in this protocol is funded through a COVID-19 Enhancement award from the 
Patient-Centered Outcomes Research Institute® (PCORI®), Award HIS-2018C3-14695.

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 9 of 72
Investigator’s Signature
The signature below constitutes the approval of this protocol and provides the necessary assurances 
that this study will be conducted according to all stipulations of the protocol, including all statements 
regarding confidentiality, and according to local legal and regulatory requirements and applicable US 
federal regulations and ICH guidelines.
Principal Investigators:
Signed: _____________________________________________  Date: ___________________________
Name: Anna K. Radin, DrPH, MPH
Title: Research Scientist, St. Luke’s Applied Research Division
Investigator Contact information
Affiliation: St. Luke’s Health System, Applied Research Division, Research Department
Address: St. Luke’s Plaza, Plaza II
  701 E Morrison Knudsen Dr., Research Department
  Boise, ID 83712
Phone: 208-381-8468
Email: Radina@slhs.org

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 10 of 72
1. Protocol Summary
1.1 Synopsis
Title
Mental Health Among Patients, Providers, and Staff (MHAPPS): Investigating the 
Mental Health Impact of COVID-19 and Comparing the Effectiveness of Two Caring 
Contact Interventions
Contract 
Number:
PCORI HIS-2018C3-14695
Study 
Description
Cross sectional survey (Aim 1) of mental distress in healthcare providers, staff, and 
patients in the COVID-19 era; and randomized controlled trial (Aim 2) comparing the 
effectiveness of two versions of a Caring Contacts intervention to reduce loneliness 
and mental distress.
Specific Aims
Aim 1:  Measure the prevalence of mental distress including loneliness, anxiety, 
depression, substance use, suicide ideation and other suicide-related risk factors in 
providers, employees, and patients served by SLHS. Describe differences associated 
with age, sex, race, ethnicity, urban or rural residence, living situation, occupation, 
local COVID-19 prevalence, perceived risk related to COVID-19, and masking and 
social distancing practices. 
Aim 2:  Compare the effectiveness of two versions of the Caring Contacts 
intervention ((1) introductory phone or video call followed by text messages (CC+), 
versus (2) text messages alone (CC)) to reduce loneliness and improve mental health 
outcomes among SLHS patients, providers, and staff experiencing mental distress 
during the COVID-19 era. 
Outcomes
Aim 2 primary outcome: loneliness (NIH Toolbox Social Relationship Scales). 1-3  Aim 2 
secondary outcomes: suicide ideation (C-SSRS) 4-13; presence of risk factors for suicide 
including those theoretically related: defeat, entrapment, perceived 
burdensomeness, and thwarted belongingness (INQ) 14,15; background risk factors 
including stress (NIH Toolbox), 1-3,16 alcohol and illicit drug use, depression (PHQ-9), 
17-19 and anxiety (GAD-7); 20,21 and attendance at mental healthcare appointments. All 
outcomes measured at 6 months following study enrollment. 
Study 
Population
(1) SLHS providers and employees, and (2) adolescent and adult patients accessing 
primary care services from SLHS primary care clinics in Idaho.  The study will enroll 
approximately 4,800 participants in the initial survey (Aim 1).  The comparative 
effectiveness trial will be comprised of a subset of 660 participants who report 
elevated loneliness, suicide ideation, or other mental distress in the Aim 1 survey.  
Enrollment will be stratified by population (providers and employees; patients) with 
165 per intervention arm in each stratum. 

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 11 of 72
Description of 
Study Sites
.  (1) SLHS employs over 14,000 providers and staff across the system. (2) SLHS 
includes over 50 primary care clinics across the state of Idaho
Study Duration
This study will last for approximately 12 months, with 6 months for recruitment and 
6 months for intervention/outcome assessment.
Participant 
Duration
Aim 1 is cross sectional (one timepoint).  
Aim 2 includes a 6-month long intervention period.  Participants will complete a 
baseline assessment and an outcomes assessment at 6 months (end of intervention).

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 12 of 72
1.2 Schema / Study Flow Diagram
Figure 1: MHAPPS Trial Flow Diagram

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 13 of 72
2. Research Question & Specific Aims
Research Question: How can health systems support mental health among providers, employees, and 
patients during the COVID-19 era while minimizing the burden on the health system? 
Specific Aims.

Aim 1:  Measure the prevalence of mental distress including loneliness, anxiety, depression, 
substance use, suicide ideation and other suicide-related risk factors in providers, staff, and 
patients served by SLHS. Describe differences associated with age, sex, race, ethnicity, urban or 
rural residence, living situation, occupation, local COVID-19 prevalence, perceived risk related to 
COVID-19, and masking and social distancing practices. 

Aim 2:  Compare the effectiveness of two versions of the Caring Contacts intervention ((1) 
introductory phone or video call (phone call) followed by text messages (CC+), versus (2) text 
messages alone (CC)) to reduce loneliness and improve mental health outcomes among SLHS 
patients, providers, and staff experiencing mental distress during the COVID-19 era. 
o
Aim 2 Hypothesis: We hypothesize that including an introductory phone call with the 
Caring Contacts intervention will yield better mental health outcomes among providers, 
staff, and patients experiencing mental distress than a Caring Contacts intervention with 
no introductory phone call.
3. Introduction
3.1 Background
The COVID-19 pandemic is occurring amid a high prevalence of mental health conditions 22 and 
increasing suicide rates 23 in the United States. Individuals are experiencing anxiety, financial hardship, 
disruption to school and work schedules, and repeated exposure to information and misinformation in 
mainstream and social media 24 as a result of the pandemic. This may increase their risk of suicide either 
directly or through proximal risk factors and can lead to or exacerbate other mental health conditions.25-
36 
Mental health conditions are common in the United States, and in 2017 nearly 1 in 5 adults ages 18 and 
older reported a mental illness in the last 12 months. 22 In adolescents ages 13-18 years, diagnostic 
interview data showed nearly 1 in 2 reporting lifetime prevalence of any mental disorder with anxiety 
disorders being the most common (31.9%). 37 An increase in social isolation and loneliness are likely to 
be significant adverse consequences of the pandemic 25, particularly in elderly individuals 38 and 
adolescents. 39 Both social isolation and loneliness are associated with anxiety, depression, self-harm, 
and lifetime suicide attempts. 40,41 Additionally, a number of the anticipated consequences of quarantine 

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 14 of 72
and social and physical distancing measures introduced to prevent community spread of the disease are 
themselves risk factors for mental health issues 25 including self-harm, alcohol and substance misuse, 
domestic and child abuse and psychosocial risks (i.e. social disconnection, loss of income, and feeling a 
burden). 25,42 Nearly 41% of 5,470 US adults who completed a web-based survey in late June 2020 
reported adverse mental or behavioral health conditions with essential workers, young adults, unpaid 
caregivers, and racial/ethnic minorities being disproportionately affected. 43  The prevalence of anxiety 
disorder symptoms was 3x higher than reported in 2019 and depression was 4x higher. 43 
Suicide is a leading cause of death in the United States and one of only three leading causes of death 
that is on the rise. 23  Suicide rates have risen by over 30% in more than half of US states since 1999, with 
Mid-Western and Intermountain West States in particular exhibiting alarmingly high increases in suicide 
rates. 23  Idaho’s suicide rate is the sixth highest in the US, 50% above the national average. 23  In 2016, 
nearly 45,000 individuals in the U.S. died by suicide, 23 roughly equivalent to one suicide every 12 
minutes. In the COVID-19 era, approximately 11% of adults reported seriously considering suicide in the 
past 30 days, and the percentage was significantly higher among 18-24-year-old respondents (25.5%). 43 
Half of suicide decedents seek healthcare – often in EDs or primary care settings – within a month of 
their death. 44 
Healthcare providers and staff face additional stress due to their increased exposure to infected 
patients 45,46 and the burden of bearing witness to the devastating effects of this disease. Healthcare 
workers who are younger, female, parents to dependent children, or report pre-existing psychological or 
physical ill health are more vulnerable to mental distress when caring for patients during a viral 
outbreak. 45,46 Interventions that support frontline staff’s mental well-being are needed to mitigate the 
added psychological distress during a pandemic. Health systems have an obligation to address mental 
distress and risk factors for suicidal ideation and behavior in the patient populations they serve and in 
clinicians and staff they employ.
Experts worldwide and at St. Luke’s Health System (SLHS) have called for research to (1) better 
understand the mental health impact of COVID-19 on the general population, and key populations 
including healthcare providers and staff, older adults, and adolescents, 25,28  and (2) to determine the 
best evidence-based mental health interventions that can be delivered virtually at scale with minimal 
resources during the COVID-19-era. 25,27,28 Additionally, the Patient-Centered Outcomes Research 
Institute (PCORI) has called for research using a comparative effectiveness framework that can provide 
timely answers to COVID-19-related challenges facing health systems, patients, caregivers, and 
clinicians.47 as well as research related to suicide prevention in adult and adolescent patients. 48
Caring Contacts is an evidence-based suicide prevention intervention and one of the only brief suicide 
prevention interventions that has demonstrated effectiveness in reducing suicidal ideation and behavior 
in randomized clinical trials. 49-51  Caring Contacts has been recommended as part of standard care for 

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 15 of 72
suicide prevention by the Joint Commission, 52 the US Department of Veterans Affairs (VA), 53 the US 
Department of Defense, 53 and the National Action Alliance for Suicide Prevention. 44 A recent high-
quality trial demonstrated a 44% reduction in the odds of suicidal ideation and a 48% reduction in the 
odds suicidal behavior among veterans receiving Caring Contacts compared to a control group. 51
Caring Contacts aim to make individuals feel supported and cared for, typically consisting of brief 
messages sent via letters 49,50 and postcards, 54,55  emails 56 or text messages 51,57 from a provider to a 
patient. Caring messages are non-demanding expressions of care for a person’s well-being and a 
reminder that help is available. 50,51,56,58 This study will use a text messaging platform to deliver Caring 
Contacts which will be cost-effective and low burden to participants and staff, sending 11 pre-scripted 
messages scheduled over a 6-month timeframe. Trained follow-up specialists at the Idaho Suicide 
Prevention Hotline (Hotline) will correspond with individuals who choose to respond to the text 
messages, although it is never required or expected of the participants to respond. Mobile technology-
based health interventions can be useful to help alleviate unmet mental health service needs in low-
resource settings and marginalized populations such as minorities and people with limited income.59-61
The proposed study will address several key gaps in the scientific literature.  While the effect of Caring 
Contacts on suicidal ideation and behavior is well documented, its effect on other types of mental 
distress including depression, anxiety and psychological stress has not been rigorously studied, 
particularly in civilian populations post-crisis or who have not been discharged with a recent suicide 
attempt. 
The theoretical basis for Caring Contacts and its hypothesized causal mechanism are both related to 
loneliness or a lack of social connection.62 The Caring Contacts intervention provides recipients with 
messages meant to reinforce the idea that someone cares about them, is thinking of them, and expects 
nothing in return. The messages facilitate a feeling of caring but also serve as a gentle reminder that 
support and resources are available. A recent meta-analysis of causal mechanisms for brief contact 
interventions such as Caring Contacts affirms that social support is the most commonly reported 
mechanism through which these interventions affect suicidal ideation and behavior. 63Loneliness is a 
well-established risk factor for suicide, 15,62 depression,36,64-66 psychological stress, 66,67 and anxiety. 
36,64,66 We therefore hypothesize that by reducing loneliness, Caring Contacts may also reduce depressive 
symptoms, stress, and anxiety, in addition to reducing suicidal ideation and behavior.  
The cross-sectional survey (Aim 1) will provide timely data on the prevalence of mental distress among 
patients, providers, and staff in the COVID-19 era in a largely rural state.  The comparative effectiveness 
trial (Aim 2) will contribute to filling the literature gap regarding whether non-demanding brief contact 
interventions such as Caring Contacts are effective in supporting individuals struggling with non-suicidal 
mental distress.  

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 16 of 72
3.2 Significance
The COVID-19 pandemic has dramatically impacted individuals and communities, with implications for 
both mental and physical health. 25-28,68 There is an urgent call for research to collect data on mental 
health effects of the COVID-19 pandemic across vulnerable groups and the population as a whole 25 
and how mental health consequences can be mitigated under pandemic conditions. 25 Additionally, 
providers need actionable research to determine the best evidence-based mental health interventions 
that can be delivered virtually at scale with minimal resources during the COVID-19 era.  25,27,28 St. Luke’s 
Health System leadership is supportive of this research and has encouraged the inclusion of providers 
and employees in key study populations. 
The proposed research fills key gaps in evidence and is aligned with PCORI research priorities. This 
study will describe the extent and character of mental distress and risk factors for suicide on key 
populations in the COVID-19 era. Study participants will include healthcare clinicians/staff and patients, 
both adult and adolescent, from a private non-profit health system. This would be the first published 
data comparing the effectiveness of two versions of the Caring Contacts intervention (one with an 
introductory phone call followed by text messages (CC+) and without an introductory phone call (CC)) 
with individuals who report loneliness or other mental distress including anxiety, depression, stress, 
emotional dysregulation, and low health-related quality of life.
This study will provide high-quality data collected with scientific rigor to determine whether CC+ or CC 
more effectively improves mental health outcomes and suicide risk factors among health care providers 
and staff, and adolescents and adults in primary care settings. Additionally, this will be the first 
randomized controlled trial to study the effects of the Caring Contacts model to support individuals 
with non-suicidal mental distress. The study population will include both urban and rural residents in a 
state with a high prevalence of suicide, 23 including a rural county with one of the highest infection rates 
of COVID-19 in the U.S. 69 
This pragmatic study is designed to optimize dissemination and implementation potential and 
scalability. The intervention design capitalizes on technology and will be delivered in partnership with 
an established community resource (Idaho Suicide Prevention Hotline) that will reach urban and rural 
populations. This approach will allow the health system to consistently deliver an effective brief social 
supportive intervention without requiring any human resources from health system providers and staff 
responding to COVID-19 demands. This models a health system-community based partnership that 
could be realistically replicated and brought to scale by other clinics and health systems faced with 
similar strains on resource and workforce capacity, including those serving rural populations. We will 
partner with the Idaho Suicide Prevention Hotline to disseminate results of this study via the national 
network of Suicide Prevention Hotlines, which regularly liaise with health systems nationwide. 

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 17 of 72
This study will have immediate and enduring public health impact, allowing health systems and 
community partners in Idaho and nationally to select the best brief, virtual, evidence-based intervention 
to provide mental health support. The interventions being compared are feasible to implement at scale 
even while quarantine and/or social and physical distancing measures are in effect from the pandemic, 
including in low-resource settings with limited access to behavioral health services. 
4. Research Methods
4.1 Study Design
4.1.1. Aim 1: Cross Sectional Survey
Aim 1 is a cross sectional survey of patients, providers, and staff to measure the prevalence of various 
measures of mental distress in the context of the COVID-19 pandemic.    
4.1.2 Aim 2: Comparative Effectiveness Trial
Aim 2 (primary analysis) is a randomized controlled trial comparing the effectiveness of CC+ vs. CC.  The 
study objectives, outcomes, and interventions all pertain to the individual level.  
4.2 Randomization
Randomization will occur at the individual level.  A list of potentially eligible participants will be 
generated based on responses to the survey conducted through Aim 1.  Survey respondents reporting 
elevated mental distress will be part of the potentially eligible study population for Aim 2. Statistical 
software will be used to randomly assign Aim 2 participants to an intervention arm immediately 
following informed consent and study enrollment.
4.3 Masking
This trial will be single masked, with most members of the study team including the lead biostatistician 
masked to the comparator group (treatment condition) randomly assigned to each participant. Due to 
the nature of the intervention, masking interventionists or study participants to the comparator 
received is not feasible. The PI and research coordination team will conduct fidelity monitoring activities 
that may reveal the treatment assignment of individual participants. 
5. Study Population
The study population for this research is adult and adolescent patients, and healthcare providers and 
staff from St. Luke’s Health System in Idaho.  Aim 1 will include up to approximately 2,400 patients and 
2,400 providers and staff.  Aim 2 will include a subset of 660 Aim 1 participants who report elevated 
levels of mental distress (330 patients, and 330 providers or staff).  SLHS employs over 14,000 people 

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 18 of 72
and serves nearly half of the population of Idaho; a recent CDC report indicated that over 40% of adults 
reported elevated levels of mental distress during the COVID-19 pandemic.  The source population is 
adequately large to meet this study’s recruitment goals.
5.1 Inclusion & Exclusion Criteria
Inclusion criteria are intentionally broad, to facilitate recruitment of a study sample that is maximally 
representative of the population of SLHS patients, providers, and staff. 
5.1.1 Inclusion & Exclusion Criteria: Aim 1
Provider & Employee Inclusion Criteria

Provider or Employee at St. Luke’s Health System

Adults ≥ 18 years of age

Proficient in spoken and written English language
Provider & Employee Exclusion Criteria

Individuals who are unable or unwilling to provide informed consent to participate

Individuals who are study staff for this study or the SPARC Trial
Patient Inclusion Criteria

Patient at a SLHS  primary care site

Current MyChart account user

Adults ≥18 years of age

Minors 12-17 years of age

Proficient in spoken and written English language
Patient Exclusion Criteria

Individuals who are unable or unwilling to provide informed consent to participate. 

Individuals who are participants in the SPARC Trial

Individuals who have not had a primary care visit in the past 12 months
5.1.2 Inclusion & Exclusion Criteria: Aim 2
Provider & Employee Inclusion Criteria

Moderate or high score for loneliness, suicide ideation, psychological stress, anxiety, or 
depression:
o
NIH Toolkit Loneliness raw score of 13 or greater or
o
C-SSRS score of 3 or greater; or

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 19 of 72
o
NIH Toolkit Perceived Stress raw score of 31 or greater for adults; or
o
GAD7 score of 11 or greater; or
o
PHQ9 score of 10 or greater

Access to a phone for the duration of the study with the ability to receive text messages and 
phone calls
Provider & Employee Exclusion Criteria

Individuals who are unable or unwilling to provide informed consent to participate

Individuals who are in acute crisis as determined by the person conducting the consent process

Individuals who are study staff for this study or the SPARC Trial

Individuals who are enrolled as participants in the SPARC comparative effectiveness clinical trial 
(SPARC Aim 1).  Providers or employees who received training related to SPARC and/or who 
completed the SPARC provider satisfaction survey are not excluded from participating in 
MHAPPS
Patient Inclusion Criteria

Moderate or high score for loneliness, suicide ideation, psychological stress, anxiety, or 
depression:
o
NIH Toolkit Loneliness raw score of 13 or greater for adults or 16 or greater for 
adolescents; or
o
C-SSRS score of 3 or greater; or
o
GAD7 score of 11 or greater; or
o
PHQ9 score of 10 or greater; or
o
NIH Toolkit Stress raw score of 31 or greater for adults or 33 or greater for adolescents

Note: validated youth versions of the NIH Toolkit assessments (loneliness and 
perceived stress), and PHQ-A tools will be used for adolescents ; the C-SSRS and 
GAD7 tools are validated for use with both adults and adolescents.

Access to a phone for the duration of the study with the ability to receive text messages and 
phone calls
Patient Exclusion Criteria

Patients who are unable or unwilling to provide informed consent/assent to participate (or 
whose legally authorized representative is unable to provide consent in the case of adolescents). 
Examples may include but are not limited to patients who present with cognitive impairment, as 
determined by the person conducting the consent process, that would preclude their ability to 
consent (i.e. acute psychosis, intoxication, or intellectual disability).

Individuals who are in acute crisis as determined by the person conducting the consent process

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 20 of 72
5.2 Recruitment Strategy
A random selection of MyChart users who have attended a primary care visit at SLHS within the past 12 
months will be recruited for the Aim 1 survey through a message including a REDCap survey link sent to 
their MyChart account or email.  A comprehensive list of providers and employees and their email 
addresses will be obtained from a contact in the SLHS Human Resources Department, Digital and 
Analytics, or another appropriate department and a random sample of providers and employees will be 
selected using appropriate software.  Those randomly selected will be recruited for the Aim 1 survey 
through an email message including a REDCap survey link.  The Aim 1 survey will include an option to 
opt out of being contacted by study staff for participation in a follow-up study if they are potentially 
eligible.  Participants who indicate interest in the follow-up study who also report elevated mental 
distress will be the source population for Aim 2.  Recruitment for Aim 2 will involve study staff 
contacting potential participants to describe the Aim 2 clinical trial, conduct informed consent, and 
enroll interested patients into the trial.  Recruitment-related emails will be either automatically sent via 
REDCap, sent from the mhapps@slhs.org email address, sent from the PI, delegated study staff, or sent 
via MyChart.  In the event that there are more eligible and interested potential participants than 
available space in the trial, potential participants will be invited to participate in the order in which they 
completed the Aim 1 Survey. 
For Aim 1, informed consent and study enrollment will be completed by the participant electronically 
through REDCap, with study staff available by phone or email to answer any questions.  For Aim 2, 
informed consent and study enrollment will be conducted by trained study staff, typically a Research 
Coordinator over the phone or via video chat. All staff conducting informed consent and study 
enrollment will be trained in human subjects’ protection and study procedures and will be delegated to 
do so by the study PI.
5.3 Retention
A variety of methods will be used to improve retention of research participants in the clinical trial. A 
contact sheet will be provided at enrollment allowing patients to share additional contact information 
(including alternative phone numbers, email addresses, or social media contact information) that may 
be used to contact participants for retention purposes.  A primary cell phone number and an email 
address are required; providing additional sources of contact on the contact sheet is optional for study 
participants.  Email, text messages, phone calls, or other forms of contact may be used for retention 
purposes or to assist with scheduling and completing six-month follow-up surveys.  Retention methods 
are further described in protocol section 10.2 Lost to Follow-Up.
5.4 Populations for Analyses
Analyses will be completed using the following populations:

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 21 of 72
1.
Intention to Treat (ITT) Analytic Population: Data for all participants that complete study 
enrollment will be included in this dataset.
2.
Safety Analysis Population: The dataset shared with the Data & Safety Monitoring Board for 
safety analysis will include data for all participants who completed study enrollment (e.g., 
the ITT Analytic Population Dataset).
3.
Per-Protocol Analysis Population: Data for a subset of participants who were retained for 
the full study, were sent all 11 caring text messages, and received a phone call if randomized 
to the CC+ intervention (as outlined in protocol section 9) will be included in this dataset.
4.
Additional populations: Additional datasets may be developed to complete sensitivity 
analyses, for example, where missing data have been imputed using different techniques.
6.  Study Procedures 
Aim 1: Cross Sectional Survey (Email REDCap survey link for providers and staff; send REDCap survey link 
via MyChart or email for patients)

Aim 1 study enrollment and informed consent

Aim 1 Survey
o
Demographics questions 
o
C-SSRS (suicide risk)
o
NIH Toolbox Social Relationship Scales (loneliness, stress)
o
INQ (defeat, entrapment, perceived burdensomeness, and thwarted belongingness)
o
PHQ-9 (depression)
o
GAD-7 (anxiety)
o
COVID-19 beliefs and practices survey (brief)
Aim 2: Baseline / Enrollment (Subset of Aim 1 participants; see below)

Aim 2 study enrollment and informed consent (phone call with Research Coordinator or 
study staff to complete via REDCap)

Aim 2 Baseline assessments (Email or text REDCap survey link)
o
Baseline survey (Socio-economic status, gender identity, sexuality, religion, and alcohol 
and illicit drug use)
o
Expanded COVID-19 beliefs and practices survey (additional questions related to impact 
of COVID-19 on the individual, and his/her/their family, friends, and community)
o
Contact Sheet to collect alternative forms of contact for study participant
Aim 2: Caring Contacts Interventions  (see description of interventions in section 9)  

CC+: 1 call,  within 2 weeks of enrollment, and 11 caring text messages, sent from Hotline 

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 22 of 72
follow-up specialists according to a standardized schedule over a 6-month period

CC:  11 personalized caring text messages sent from Hotline follow-up specialists according 
to a standardized schedule over a 6-month period
6 Month Outcome Assessment (+ 4 weeks variance window) (REDCap survey link sent via email 
or text)

Aim 2 6-Month Outcome Assessments:
o
C-SSRS (suicide risk)
o
NIH Toolbox Social Relationship Scales (loneliness, stress)
o
INQ (defeat, entrapment, perceived burdensomeness, and thwarted belongingness)
o
PHQ-9 (depression)
o
GAD-7 (anxiety)
o
COVID-19 beliefs and practices survey (brief)
6.1 Schedule of Activities
Table 1: Schedule of MHAPPS Trial Activities
Aim 1: Timepoint 0
Aim 2: Time Point 1: 
0 weeks
Aim 2 Time Point 2: 
0-2 weeks
Aim 2 Time Point 3: 
1 week – 6 months
Aim 2 Time Point 4:
6 months
Timepoints are relative to each study participant.
Aim 1 Informed Consent/Assent and 
Survey
X
Aim 2 Informed Consent/Assent and 
Enrollment
X
Aim 2 Baseline Survey & Contact Sheet
X
Randomization 
X
Phone call with Idaho Suicide 
Prevention Hotline (Hotline) 
CC+ arm 
only
Caring text messages (11 total) sent 
from Hotline
X
Aim 2 Outcome Survey
X
Safety Outcomes Reporting
X
X
X

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 23 of 72
6.2 Safety Procedures for Participants at Risk for Suicide
Participants who screen at high risk for suicide (C-SSRS score of 4-6) during the Aim 1 survey or the Aim 
2 Outcome Survey will be contacted by the Hotline within 12 hours to assess their safety and provide 
appropriate support.  The Hotline will make a minimum of 3 attempts to contact the participant by 
phone and/or text message.  Parents of minor participants who screen at moderate or high risk for 
suicide may be contacted by a provider from SLHS to inform them of their child’s risk.  Standard 
operating procedures (SOPs) for participant safety will be utilized.  These SOPs may also be utilized for 
participants whose responses to caring text messages indicate they may be at acute risk for suicide.
7.     Causal Framework
The logic model below depicts the hypothesized and known associations for variables included in this 
study. All variables included in the causal framework will be measured. The associations depicted with 
solid arrows are assumed to be true based on empirical evidence (described in protocol Section 8); 
associations depicted with dashed lines are hypothesized and will be assessed and measured as part of 
this study.  The bracket around the population variables and the proximal risk factors for suicide were 
added to visually simplify the framework.  Aim 1 of the study is designed to assess the hypothesized 
associations depicted with dashed lines between population variables and COVID-19 variables.  Aim 2 of 
the study is designed to measure the association between intervention received and primary and 
secondary outcomes, accounting for population variables and COVID-19 variables.

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 24 of 72
Figure 2: MHAPPS Logic Model
 
8. Exposures and Outcomes (Variables) of Interest
8.1 Exposure Variables
The following exposure variables will be measured and assessed as part of this study: age (date of 
birth), sex, race, ethnicity, gender identity, sexuality, religion, zip code, living situation and number of 
people at home, employment and occupation, education, school situation and extracurricular 
involvement (adolescents), caregiving status, financial security, domestic violence, food security, food 
access and eating habits related to COVID-19, physical activity, access to healthcare and mental 
healthcare services, local COVID-19 prevalence, perceived risk related to COVID-19, and masking and 
social distancing practices and beliefs. A full summary of exposure, outcomes, and process variables to 
be included in this study is included in protocol section 12, Table 2. 
8.1.1 Rationale for Including Selected Exposure Variables
Age and sex are associated with loneliness and suicidal ideation and behavior. 70 Older adults already 
experience significant impacts mental health from isolation and social disconnectedness, and COVID-19 
restrictions are predicted to magnify feelings of loneliness.70-72 Adults over 35 years old are more likely 

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 25 of 72
to die by suicide than other age categories. 73 Additionally, children and adolescents are predicted to 
experience increased depression and anxiety from loneliness and social isolation during COVID-19 
changes. 39 Sex differences have been observed for suicidal behavior. 23,73 CDC reports that suicidal 
completion is approximately 4 times more common among males than females. 73 
Though minority racial and ethnic groups have lower rates of lifetime mood disorders 74 they are 
expected to face greater mental distress from the pandemic due to disparities in infection and mortality 
rates from COVID-19. 74,75  Race and ethnicity are also associated with risk of suicidal ideation and 
behavior.  The rate of suicide is approximately three times higher among Non-Hispanic Whites and 
American Indian/Alaska Natives (16.71 and 18.37 per 100,000 respectively) than among other racial and 
ethnic groups. 73 
The Health Resources & Services Administration’s Federal Office of Rural Health Policy urban-rural 
designation for census tracts 76 will be used together with patients’ zip code to classify study participants 
as urban or rural residents. Early evidence has shown differences in COVID-19 infection percentages and 
mortality rates have been observed across urban and rural areas, with rural areas often experiencing 
lower infection rates but higher mortality. 69,77 Additionally, compared to large urban areas, rural 
residence is associated with a 45% higher rate of suicide 73 lower socioeconomic status, 78 and worse 
overall health outcomes. 78 
COVID-19 has had a significant impact on the economy, leading to widespread job loss and financial 
stress. 79  Financial stress is associated with an elevated risk for depression, suicide, and anxiety. 79  
Other determinants of socioeconomic status such as education, financial security, and food security are 
similarly related.  Lack of access to and consumption of healthy foods have also been impacted by the 
COVID-19 pandemic.80  Healthy eating habits and physical activity are associated with positive mental 
health outcomes.80,81 Additionally, people may have experienced challenges accessing both physical and 
mental healthcare services, which can lead to excess morbidity due to existing health conditions. 82 
Living situation and number of people at home may influence risk of COVID-19 exposure and degree of 
loneliness experienced. Homeless and transient individuals are at a greater risk of infection due to their 
environment. 25,83 Healthcare and other essential workers may experience stress about infecting others 
in their homes after exposure at work. 45 Having many people in a household may offer social support 
which is protective against loneliness, 84 but could also contribute to interpersonal conflict and domestic 
violence.85  Overall stress related to COVID-19 may also exacerbate domestic violence. 86
Unpaid caregivers of children, older adults, or other family members who may have physical or mental 
disabilities are twice as likely to have experienced elevated mental distress compared with non-

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 26 of 72
caregivers. 43  Both parents and adolescents have been impacted by the changes in school environment 
and extracurricular activities due to COVID-19.86,87
People in a high-risk occupation, such as healthcare workers and other essential workers may face 
increased stress and trauma from working during the COVID-19 pandemic. 45,88,89 In contrast, people 
working jobs remotely may experience increased loneliness and disconnectedness. 90 Understanding 
how the interventions administered in this study affect people of various occupations will be important 
to future implementation. 
Local COVID-19 prevalence and perceived risk related to COVID-19 may influence the protective health 
behaviors an individual engages in during the pandemic. The experience of mandatory closures of 
schools, businesses, and other facilities will likely vary for participants in “hot-spot” areas such as Ada, 
Canyon, or Blaine counties.91,92  Additionally, The higher and individual perceives their infection risk to 
be, the more likely they are to engage in social distancing and handwashing to protect from COVID-19.93
Measuring masking and social distancing beliefs and practices of respondents will be key to 
understanding the protective behaviors of participants and the degree of increased isolation 
experienced during the pandemic. Masking has been shown to be an effective measure to reduce the 
spread of COVID-19.  94-96 Despite calls from SLHS and other agencies for universal mask wearing in 
Idaho, 97 a state-wide mask order has not been issued. While social distancing orders were 
implemented,91 varying degrees of adherence98 may influence the impact of those orders on mental 
health.
Gender identity and sexuality are strongly associated with risk of suicidal ideation and behavior.  
Lesbian, gay, bisexual, transgender, or gender-nonconforming (LGBTQ+) individuals are two to seven 
times more likely than straight, cisgender peers to attempt suicide.33-38 
Religious affiliation is associated with suicidality, but the magnitude and direction of that association 
differs depending on the religion and its intersection with socio-cultural factors (e.g., sexuality).  A 
recent meta-analysis found that religiosity protects against suicidal completion, with a pooled odds 
ratio of 0.38 (95% CI: 0.21-0.71).39  Another review found that 75% of published studies identified 
religion as protective against suicidal ideation and behavior.40  
8.2 Primary Outcome & Assessment 
The primary outcome is loneliness.  Loneliness is a well-established risk factor for suicide, 15,62,63   
depression, 39,64-66 psychological stress, 66,67 and anxiety. 39,64,67 We therefore hypothesize that by 

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 27 of 72
reducing loneliness, Caring Contacts may also reduce depressive symptoms, stress, and anxiety, in 
addition to reducing suicidal ideation and behavior.  
Loneliness will be measured at 6 months as a change from baseline score using the NIH Toolbox Social 
Relationship Scales loneliness measure. The NIH Toolbox is a comprehensive set of neuro-behavioral 
measurements that quickly assess cognitive, emotional, sensory, and motor functions.3 The 
measurements were developed to be versatile, brief, and psychometrically sound. 3 The NIH Toolbox is 
composed of multiple batteries, one of which is the NIH Toolbox Emotion Battery. The Emotion Battery 
is comprised of four different domains, including the Social Relationship Scales which have items that 
have been validated to measure loneliness. 2
The loneliness measure of the NIH Toolbox Emotion battery for adults 18+ is comprised of five items 
with rating scale responses of never, rarely, sometimes, usually, and always. Responses are then used to 
calculate a raw score which is converted to a t-score 99. The version for adolescents 13-17 is comprised 
of 7 items. Both measures were validated using confirmatory factor analysis.2
8.3Secondary Outcomes & Assessment
Secondary outcomes include suicide ideation (C-SSRS) self-harm and suicidal behavior; presence of risk 
factors for suicide including those theoretically related: perceived burdensomeness, and thwarted 
belongingness (INQ); background risk factors including stress (NIH Toolbox), alcohol and illicit drug use, 
depression (PHQ-9); and anxiety (GAD-7); and attendance at mental healthcare appointments.
8.3.1
Suicidal ideation
Reduced suicidal ideation and behavior will be measured as change from baseline score at 6 months 
using the Columbia Suicide Severity Rating Scale (C-SSRS). 100-103 C-SSRS is widely used in practice and 
research and has strong psychometric properties for use with both adult and adolescent populations, 
which are summarized in a Supporting Evidence document. 102  The C-SSRS has been shown to be an 
effective tool to measure suicidality (diagnosis) and is sensitive to change over time, which allows 
measurement of the effect of treatment at 6 months. 102,104 
The C-SSRS is available in a full-length version comprised of 11 yes/no questions plus 7 multiple choice 
questions and space to collect and record brief narrative explanations. An abbreviated screener version 
of the C-SSRS consists of 5 yes/no questions related to suicidal ideation and one two-part yes/no 
question related to suicidal behavior.  Both the full-length and screener versions of the C-SSRS are 
available in a ‘lifetime-recent’ version (to establish a baseline) and a ‘since last contact’ version to be 
administered at follow-up visits.  Two versions of the C-SSRS will be used for this study: C-SSRS Lifetime-
Recent Screener (given at baseline, in accordance with standard of care at SLHS), and C-SSRS Since Last 

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 28 of 72
Contact Screener (given at 6 months).  In addition to using the C-SSRS Screeners at baseline and follow-
up, the study will incorporate three items from the full-version of the C-SSRS that are not included in the 
screener version to measure: (1) Aborted or self-interrupted suicide attempts; (2) interrupted suicide 
attempts; and (3) actual suicide attempts at baseline, 6 months.  Information on non-lethal self-harm, 
lethal means used for attempts or completions, and death by suicide will also be collected.  Vital records 
will be used to measure suicide completion.  
8.3.2 Scoring the C-SSRS
The C-SSRS lifetime-recent screener score is determined based on the highest question number (1-6B) to 
which a participant responds “yes”.  For example, a score of 5 would be assigned to a participant who 
responded “yes” to Question 5 and any or all preceding questions.  Please consult Appendix B of this 
protocol for additional C-SSRS scoring criteria.  
The additional items related to suicide attempts and lethal means will not be included in the C-SSRS 
score but will be compared across intervention groups.
8.3.3 Risk Factors for Suicide (INQ)
Reduced risk factors for suicide including perceived burdensomeness and thwarted belongingness will 
be measured at baseline and six months utilizing the Interpersonal Needs Questionnaire (INQ).14 The 
INQ was developed from the Interpersonal Theory of Suicide to measure both perceived 
burdensomeness and thwarted belongingness which are proximal causes of desire for suicide. 15 The 
INQ underwent refinement in 2012 and was reduced from 25 to 15 items. 15 Respondents indicate of 15 
questions how true each is for them on a scale of 1 to 7, with 1 being “not true at all for me” and 7 being 
“very true for me.”
The 15 items are valid and psychometrically sound for measuring perceived burdensomeness and 
thwarted belongingness in young and older adults.15,105 A study in 2014 evaluated factor structure, 
internal consistency, and concurrent predictive validity of the INQ versions with adolescent psychiatric 
inpatients and found the 15 item version to have acceptable internal consistency. 106
8.3.4 Stress
Stress level will be assessed for participants measured as a change in score from baseline at 6 months 
using the NIH Toolbox Stress and Self-Efficacy Scales Perceived Stress measure. This measure comes 
from the Emotion Battery of the NIH Toolbox and was selected because it is validated and 
psychometrically sound to measure individual perceptions about the nature of events and their 
relationship to the values and coping resources of an individual. 2 The measure is comprised of ten items 
which are scored and granted a t-score specific to adult or adolescent participants.  99

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 29 of 72
8.3.5 Alcohol and Illicit Drug Use 
Alcohol and illicit-drug use will be measured at baseline and 6 months with questions adapted from the 
Youth Risk Behavior Survey. 107  Additional questions will be included to measure self-reported changes 
in alcohol or illicit-drug use since the beginning of the COVID-19 pandemic.
8.3.6 Depression
Presence of depressive symptoms, depression, and depression severity will be screened for at baseline 
and 6 months as an indicator of mental distress. The Patient Health Questionnaire-9 (PHQ-9) is a widely 
used tool to screen for depression in primary care and other non-psychiatric settings.17,19,108 The tool is 
composed of 9 questions each with a response of 0-3 which generate a score from 0-27 with higher 
scores indicating a greater degree of depression.19 Scores are categorized in the following manner: a 
score of 5-9 is considered minimal depression, 10-14 is considered mild major, 15-19 is moderate major, 
and ≥20 is severe major.19 The PHQ-9 has been shown to be a valid and reliable measure of depression 
severity in adult and adolescent populations; this study will utilize PHQ-9 adult and PHQ-A adolescent 
versions for the corresponding age groups.17-19,108,109 
8.3.7 Anxiety
Anxiety will serve as an indicator of mental distress and will be screened for at baseline and six months. 
Symptoms of anxiety will be assessed using the GAD-7, a brief self-report scale frequently used in the 
identification of Generalized Anxiety Disorder. The GAD-7 was validated in 2006 as a valid and 
psychometrically sound measure with good sensitivity (89%) and specificity (82%). 20 The tool is similar 
to the PHQ-9 in its structure and scoring with seven items and possible scores of 0-21. Higher scores 
indicate a greater severity of generalized anxiety symptoms 20. Since the original validation, the tool has 
been confirmed as valid for detecting anxiety in both adult and adolescent populations. 21,110,111 
8.3.9 Uptake of Outpatient Mental Health Services
Current engagement and any engagement since study enrollment in outpatient mental health services 
will be measured as dichotomous variables (yes/no) at 6 months through self-report.  
9.    Description of Interventions
9.1 Caring Contacts with Introductory Call (CC+)
Caring Contacts will consist of one introductory phone or video conversation to establish a connection 
between the study participant and the follow-up specialist.  The introductory phone call will be 
unscripted, but will generally include the following elements: an introduction, if appropriate (based on 
participants’ survey scores) a safety/wellbeing check, a review and discussion of the elements of mental 
distress on which the participant scored highest, a discussion of any relevant resources that may be 
helpful to the participant, and a description of what to expect with the caring text messages that will 

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 30 of 72
follow.  Hotline follow-up specialists will attempt to complete the phone call over a two-week period 
following study enrollment.  If the phone call cannot be completed in two weeks, the Hotline will initiate 
the text message portion of the intervention.
The call will be followed by a series of 11 personalized caring contacts sent over the course of 6 
months via text message or email. 112  The frequency and cadence of caring text messages will be 
consistent for all participants and was determined following consultation with PLES Advisors to ensure it 
is culturally and age appropriate.  Caring text messages will be sent using a HIPAA-compliant program 
called Mosio according to the following schedule: weeks 1, 2, 3, 4, 6, 8, 10, 12, 16, 20, and 24.  While 
there is no expectation that participants respond to the text messages, some participants may choose to 
respond.  Hotline Follow-Up Specialists will review incoming text messages and phone calls from study 
participants and will respond according to the Hotline’s internal operating procedures. These additional 
contacts will take place outside of the structured study protocol and will not be under the purview of 
SLHS institutional policies.  Mosio will track the number and content of individual responses to study 
text messages, and this information will be shared with the study team. Response and ongoing contacts 
as part of the Hotline’s processes will not alter the timing or frequency of caring contacts established by 
the study protocol.  Data collected and stored in Mosio will become part of the study dataset and may 
be analyzed as part of process evaluation or other analyses.
9.2 Caring Contacts without an Introductory Call (CC)
Participants will be sent caring text messages according to the schedule above.  No introductory phone 
or video call from the Hotline will be made.
9.3 Definition of Full Intervention
Participants must be sent at all 11 caring text messages (all participants), and complete a phone call (if 
randomized to CC+) to be included in the Per Protocol analysis group. 
9.4 Process Evaluation
The study will include a process evaluation to assess uptake of surveys and delivery of the interventions, 
with process data collected and evaluated throughout the enrollment period.  Specifically, the study will 
measure the proportion of invited survey participants who complete surveys for Aim 1.  For Aim 2, the 
proportion of eligible participants who enroll in the clinical trial will be measured.  The proportion of 
Aim 2 participants in the CC+ arm who are successfully contacted with an introductory phone call within 
two weeks of enrollment will also be measured, along with the proportion of participants in each study 
arm who complete the full intervention.  The “dose” of each intervention received (phone call, number 

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 31 of 72
of text messages sent) will be recorded and assessed. Data will also be collected and reviewed to assess 
how well participants like the Caring Contacts messages received as part of this trial.
10.Discontinuation and Participant Withdrawal
10.1 Participant Discontinuation and Withdrawal from the Study
Participants are free to withdraw from participation in the study at any time upon written request to the 
MHAPPS email account (mhapps@slhs.org).  Additionally, study investigators may discontinue a 
participant from the study for the following reasons, but not limited to:
1.
Lost to follow-up; unable to contact subject (see section 10.2 Lost to Follow-Up)
2.
Any event or medical condition or situation occurs such that continued collection of follow-up 
study data would not be in the best interest of the participant 
3.
Any event or situation occurs in which the safety or wellbeing of study staff is compromised by 
allowing a participant to continue to participate in the research
4.
The participant meets an exclusion criterion or fails to meet an inclusion criterion (either newly 
developed or not previously recognized) that precludes further study participation
The date of discontinuation and reason for discontinuation or withdrawal from the study will be 
recorded in the MHAPPS Trial study records.  
10.2 Lost to Follow-Up
A study participant will be considered lost to follow-up if s/he fails to complete the 6-month outcome 
assessments and study staff are unable to contact the participant after at least 3 attempts.  The 
following actions must be taken before a participant will be declared lost to follow-up.

Study staff will attempt to contact the participant, re-send the REDCap survey or reschedule the 
missed phone-based assessment, counsel the participant on the importance of maintaining the 
assigned assessment schedule (if necessary), and ascertain whether the participant wishes to 
and/or should continue in the study

The study staff will make every effort to regain contact with the participant (using text, email, 
phone call, and/or alternative means of contact that the participant may have included during 
the study enrollment process).

Should the participant continue to be unreachable, s/he will be considered to have withdrawn 
from the study with a primary reason of lost to follow-up.  The participant’s data will be retained 
in study records. 

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 32 of 72
11.Data & Safety Monitoring
11.1 Overview of Data & Safety Monitoring Plan
This research will include a Data and Safety Monitoring Board (DSMB).  The DSMB will be convened and 
managed through the University of Washington’s Institute of Translational Health Sciences (UW ITHS).  
11.2 Role of the Data & Safety Monitoring Board (DSMB)
The DSMB will review and oversee the following elements:
1.
Study enrollment by population (providers/staff, and patients (adults, adolescents))
2.
Retention of study participants at 6 months
3.
Data completeness and quality
4.
Process evaluation data
5.
Safety outcomes by intervention and population
6.
Decisions related to stopping the trial early due to one or several of the elements above
11.3 Safety Outcomes
This protocol considers completed suicide, suicide attempts, and inpatient admission in the context of 
highly suicidal study participants as expected events.  These will be routinely tracked as key safety 
outcomes.  The following safety outcomes will be assessed for each participant at 3 and 6 months and 
reviewed by the Data and Safety Monitoring Board twice annually to determine whether the rate of 
safety outcomes differs by intervention arm:

Death by suicide

Attempted suicide

Interrupted or aborted suicide attempt

Psychiatric hospitalization for anxiety, depression, and/or suicidal ideation

Medical hospitalization related to self-harm or attempted suicide

Medical hospitalization related to unintentional overdose or substance use disorder
11.4 Adverse Events and Serious Adverse Events
This protocol does not include tracking of adverse events (AEs).  This protocol does not include real-time 
tracking of serious adverse events (SAEs) for several reasons.  First, as stated above, the most important 
events that would be defined as SAEs are expected safety outcomes in the context of study participants 
experiencing suicidality or other forms of mental distress.  All deaths of study participants will be 
reviewed and assessed to determine whether the cause of death is suicide.  We do not anticipate any 
SAEs beyond those listed as safety outcomes above, but unanticipated SAEs that occur will be reviewed 
by the DSMB and the IRB.  Second, this research compares two models of an evidence-based 
intervention that has been widely studied, has an established safety record, and is already in widespread 

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 33 of 72
clinical practice.  This is not an experimental, novel, or untested intervention with unknown safety 
outcomes.  Finally, the most important safety question to ascertain in the context of this trial is whether 
rates of safety outcomes or SAEs are differential across the two intervention groups.   This protocol will 
monitor safety outcomes collected at 6 months as part of routine study outcome assessments through 
regular DSMB meetings to ensure equal ascertainment of outcomes across intervention groups.  This is 
the most valid and reliable way to review safety data in the context of this pragmatic clinical trial.
12.Data Collection & Management
12.1 Data Collection
All data for this study will be collected via online REDCap surveys. Table 2 includes a list of all study 
variables, which aim they are associated with, and which tool will be used for assessment.
Table 2: MHAPPS Trial Variables & Other Data Elements
Variables
Outcomes 
Tool
Survey Collected 
Source
Loneliness
NIH Toolbox – Social Relationship 
Scales - Loneliness Scale
Aim 1 Survey; Aim 2 
Outcome Survey
REDCap
Suicidal ideation & behavior
C-SSRS Screener
Aim 1 Survey; Aim 2 
Outcome Survey
REDCap
Recent suicide attempts, suicide
completion, & self-harm; lethal 
means
6-Months Suicide Attempts Survey 
Aim 1 Survey; Aim 2 
Outcome Survey
REDCap
Perceived Burdensomeness; 
Thwarted Belongingness
INQ-15
Aim 1 Survey; Aim 2 
Outcome Survey
REDCap
Stress
NIH Toolbox – Perceived Stress 10
Aim 1 Survey; Aim 2 
Outcome Survey
REDCap
Alcohol and Illicit drug use
Youth Risk Behavior Survey 
Questions
Aim 2 Baseline 
Survey; Aim 2 
Outcome survey
REDCap
Depression
PHQ-9
Aim 1 Survey; Aim 2 
Outcome Survey
REDCap
Anxiety 
GAD-7
Aim 1 Survey; Aim 2 
Outcome Survey
REDCap
Uptake of outpatient mental 
health treatment
N/A
Aim 2 Baseline 
Survey; Aim 2 
Outcome Survey
REDCap
Exposure Variables
Survey Collected
Source
Date of birth, age in years, age 
category (adult/peds) 
Aim 1 Survey
REDCap

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 34 of 72
Sex at birth
Aim 1 Survey
REDCap
Race and ethnicity
Aim 1 Survey
REDCap
Zip code of residence 
(urban/rural)
Aim 1 Survey
REDCap
Gender identity & sexuality
Aim 2 Baseline 
Survey
REDCap
Religion
Aim 2 Baseline 
Survey
REDCap
Variables related to employment 
and occupation of self/parent
Aim 1 Survey
REDCap
Education / maternal education
Aim 1 Survey
REDCap
Living situation (permanent, 
temporary)
Aim 1 Survey
REDCap
Number of people at home
Aim 1 Survey
REDCap
Caregiving for dependent(s)
Aim 1 Survey
REDCap
Financial security and food 
security
Aim 1 Survey; Aim 2 
Baseline Survey
REDCap
Food access and eating habits 
and COVID-19
Aim 2 Baseline 
Survey
REDCap
Physical activity and COVID-19
Aim 2 Baseline 
Survey
REDCap
Perceived risk for COVID-19 & 
other COVID-19-related 
questions
Aim 1 Survey; Aim 2 
Outcome Survey
REDCap
Masking practices & beliefs
Aim 1 Survey; Aim 2 
Outcome Survey
REDCap
Social distancing practices & 
beliefs
Aim 1 Survey; Aim 2 
Outcome Survey
REDCap
School attendance, 
extracurricular activities
Aim 2 Baseline 
Survey (Youth)
REDCap
Access to healthcare and mental 
healthcare services
Aim 2 Baseline 
Survey; Aim 2 
Outcome Survey
REDCap
Domestic violence
Aim 2 Baseline 
Survey
REDCap 
Local COVID-19 prevalence
CDC or Idaho 
Department of Health 
& Welfare Prevalence 
Data 
Process Variables 
Source
Aim 1 proportion of eligible 
participants completing survey
REDCap
Aim 2 proportion of eligible 
participants enrolled
REDCap

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 35 of 72
Participant feedback on phone 
call (CC+ arm only)
Aim 2 Outcome 
Survey
REDCap
Participant feedback on content 
and timing of caring messages; 
satisfaction with caring contacts
Aim 2 Outcome 
Survey
REDCap
Rating of quality of 
“connectedness” with participant
Mosio and/or REDCap
“Dose” of follow-up contact: 
timing, type (phone vs text), and 
number of attempted and 
successful contacts from the 
Hotline
Mosio and/or REDCap
“Dose” of Caring Contact text 
messages: date/time of each text 
message sent (* 11 text 
messages)
Mosio and/or REDCap
Other Variables
Tool
Aim Collected
Source
Does cell phone on record belong 
exclusively to study participant or 
is it shared?
Aim 2 Baseline 
Survey
REDCap
Cell phone number, email 
address, contract preferences, 
mailing address, and alternative 
modes of contact
Informed Consent 
Form, Aim 1 Survey; 
Aim 2 Baseline 
Survey
REDCap
Eligibility and willingness to be 
contacted about Aim 2 
participation
Aim 1 Survey
REDCap
Death
CDC or Idaho State Vital 
Records
12.2 Data Management
This study will employ a comprehensive data management plan.  The results from the screening and 
baseline questionnaires (as well as informed consent documents) will be directly entered in REDCap. The 
patient will be assigned a unique identifier and randomly assigned to one of the two intervention arms 
immediately following informed consent and study enrollment.  Follow-up specialists at the Hotline will 
review records of new participants and follow-up with patients to initiate the CC+ or CC intervention.  
Hotline staff will record any attempt at contact or successful contact made.  
The statistics team will compile all data for each participant from REDCap on a routine basis for reports 
and to build and maintain a complete dataset.  To protect the confidentiality of participants, data and 
associated documentation will be available to the PI of the study and key personnel only, under a data-
sharing agreement that includes a commitment to: (1) use the data only for research purposes, (2) 

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 36 of 72
secure the data using appropriate computer technology, (3) destroy or return the data after analyses are 
completed, and (4) not further distribute the data outside specified members of the research team in 
compliance with SLHS Research data privacy and sharing practices and policy. 
The University of Washington Institute of Translational Health Sciences (ITHS) hosts REDCap, a secure, 
HIPAA-compliant web application, which will be used for building and managing online surveys and 
databases for this research.  ITHS provides REDCap support and an array of research data curation and 
storage support.  Other databases (such as Access, Excel) may be used for study management purposes; 
all such data will be kept on secured, password-protected computers.  
13.Statistical Analysis
13.1 General Analytic Approach
Aim 1: Descriptive statistics will be used to summarize cross sectional survey results as well as baseline 
demographic and other exposure variables.  Linear regression models or generalized linear regression 
models GLM with robust standard errors will be used to assess the association between variables of 
interest.
Aim 2: The primary analysis will compare the effectiveness of the two interventions (CC+ and CC) 
delivered to two populations (patients and providers/staff), potential confounding, effect 
modification, and mediation will be handled through study design (e.g. use of randomization to assign 
individuals to interventions), and statistically through stratification (population).   Other published 
studies related to Caring Contacts have shown that patient-centered outcomes such as suicidal ideation 
change over a 6-month period in response to receipt of these interventions. 51
Analyses will be stratified for patients and providers/staff, modeling the effects of intervention 
separately in these population groups. There is very little evidence regarding the effect of Caring 
Contacts on healthcare workers, and we hypothesize that the magnitude of effect may differ by 
population group.  Generating evidence of the effectiveness of Caring Contacts with and without an 
introductory phone call in healthcare workers is the primary goal of this heterogeneity of treatment 
effect (HTE) analysis.  Linear regression with robust standard errors will be used to determine whether 
the primary outcome differs between CC+ vs CC.
Linear regression and GLM models will be used in the analysis of the secondary outcomes. 
Robust/sandwich standard errors will be used to allow for departures of the observed standard errors 
from classic model assumptions.
All data analyses will be completed using appropriate statistical software (such as R and R Studio, SPSS, 
Stata, SAS, Python, and/or Microsoft Excel).  Statistical significance will be determined based on a type I 
error (alpha) of 0.05 (two-sided).  Confidence intervals will be reported in addition to p-values.

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 37 of 72
Variables will be measured through the Aim 1 survey or Aim 2 Baseline survey and reported using 
descriptive statistics.  Potential effect modifiers will be assessed and include age, sex, gender identity, 
sexuality, race/ethnicity, religion, employment, urban/rural residence, drug/alcohol use, suicidal 
ideation at baseline, baseline depression score, baseline anxiety score, baseline stress score, baseline 
quality of life, measures of socioeconomic status and/or other variables related to the specified 
outcomes of interest. Data will be analyzed according to both an intention to treat protocol and per 
protocol received.  Please see section 5.4 Populations for Analyses for further detail.  
This study will report the distribution of key variables to facilitate assessment of the study’s internal and 
external validity. At a minimum, this study will report the distribution of key variables within the analytic 
population (key variables are defined as those that are included in the causal diagram at the beginning 
of the Research Design section of this Research Plan). Missing data will be assessed on a regular basis 
through routine data reports. These reports will allow the investigators to assess the cause of missing or 
anomalous entries, and address these if possible. If missing values cannot be retrieved, the reason for 
the missingness will be recorded in data comments available through REDCap. Study staff will record the 
reason for any participant drop-out during the 6 months of follow-up.  
13.2 Power & Sample Size for Aim 2
The total sample size for the aim 2 trial is 660 participants.  For each stratum (patients, providers/staff), 
a sample of 330 subjects (165 in each arm) allows 80% power to detect a difference of 5 units in the 
primary outcome (loneliness) if a minimum of 70% of participants are retained through study 
completion.  
13.3 Duplicate Surveys
A data cleaning step will be included to identify surveys completed twice by the same individual.  This 
may occur, for example, when an individual is both a provider/staff person and a patient at SLHS.  If 
duplicate surveys are identified, the first will be retained, and the second will be excluded from the 
research.
13.4 Missing Data
Missing data will be tracked and prevented to the extent possible through participant retention 
measures. When missing data are unavoidable, they will be addressed through statistical analysis. With 
proper inclusion of variables associated with the probability of missingness, we will assess the nature of 
missingness; if the missing data follow a ‘missing at random’ framework, we will use multiple 
imputation. 113 The most appropriate multiple imputation technique will be determined based on the 
distributions of missing data and observed variables. 114-116 Multiple imputation methods incorporate the 
uncertainty of value estimation into parameter estimates. We will assess the sensitivity of estimates to 
the imputation approach and variables used in the imputation algorithm. 

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 38 of 72
13.5 Sub-Group Analyses
We plan to generate different effect estimates for (1) providers and staff, and (2) patients and are 
statistically powered to do so.  We also plan to produce separate effect estimates for the following 
subgroups: adult vs adolescent patients, types of providers, low baseline C-SSRS score vs moderate to 
high baseline C-SSRS score, Hispanic vs non-Hispanic, female vs male, cisgender vs transgender or 
gender-nonconforming, heterosexual vs. homosexual or bisexual, and urban vs rural; however, this 
study has not been specifically powered to identify differing treatment effects in each of these 
subgroups.  
14.Engaging Providers & People with Lived Experience with Suicide
The research team for this trial includes people with lived experience with suicide, research scientists, 
system-level medical directors and administrators, physicians, psychologists, and social workers at SLHS; 
community partners at Jannus, including the Hotline and Empower Idaho, and, the Idaho Federation of 
Families for Children’s Mental Health (IFF); suicide prevention researchers as well as dissemination, 
biostatistics, and bioethics experts at the University of Washington.
Study staff will continue to engage and expand the scope for the existing People with Lived Experience 
with Suicide (PLES) Advisory Board to support this trial.  Study staff will convene a formal PLES Advisory 
Board to include up to 15 people who experienced suicidality as adults or adolescents.  The PLES 
Advisory Board will provide input related to specific questions (such as study branding, recruitment 
strategy, the informed consent process (including readability of consent forms), retention techniques, 
and the content, frequency, and timing of the messages included in Caring Contacts), and will assist with 
the dissemination strategy.  
Study staff will continue to engage the SLHS Behavioral Health providers and a cross-functional group 
of stakeholders at SLHS for input on study design, conduct, and dissemination of study results. 
Multiple staff and providers perspectives will be represented including system- and clinic-level providers 
including, for example, patient access specialists, social workers, psychiatrists and psychologists, ED 
physicians, pediatricians, and primary care physicians, advanced practice providers, and medical 
assistants and nurses.  Process evaluation will be used to assess what providers and staff think about the 
specific messaging included in the Caring Contacts.  Study staff will also engage an existing group of 
stakeholders within the health system who are dedicated to provider and employee well-being, to solicit 
input related to the MHAPPS Trial without burdening health system staff responding to COVID-19.   
Study staff will continue to engage existing MHAPPS stakeholders at the state, regional, and 
community levels, including but not limited to public sector stakeholders, private foundations, and non-
governmental organizations.  

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 39 of 72
15.Risk / Benefit Analysis
15.1 Potential Risks
There are several potential risks to participation in this study.  Loss of confidentiality due to the 
unintended release of sensitive information is one risk. This risk will be mitigated by storing all electronic 
data on password protected servers. Data will be shared among research partners through REDCap. 
REDCap is a secure, HIPAA-compliant web-based research data management application, used for 
building and managing online surveys, and providing a secure electronic database.  REDCap is owned 
and managed by the University of Washington’s Institute of Translational Health Sciences (ITHS). 
REDCap will be used as a central location for online study data storage and participant management. 
Use of REDCap will protect against unintended release of sensitive information.
Other potential risks include psychological distress from completing study questionnaires related to 
loneliness, anxiety, depression, suicidality, other mental distress, and quality of life.  The frequency of 
contact in the follow-up intervention may also cause emotional distress.  Research participants will be 
reminded at enrollment that they may quit the survey or skip any questions (Aim 1) or leave the study 
(Aim 2) at any point with no consequences to the care they receive at SLHS (patients) or their 
employment (providers/employees).  Participants will be reminded of resources (including the Hotline 
and, if appropriate, behavioral healthcare services) that they can access as needed in the event of 
psychological distress.  Participants will not be enrolled in Aim 2 of the study if they are in an acute crisis 
or if they are unable to provide informed consent to participate. Additionally, measures will be put in 
place to support any individuals who report high levels of suicide ideation using a modified version of 
the St. Luke’s Health System Red Flag Workflow, which is the standard safety protocol for any individual 
who contacts the health system and reports suicidal ideation. In the modified version, all individuals 
endorsing acute suicide risk would be connected to the Idaho Suicide Prevention Hotline using a warm-
handoff.  Section 16.2.2 of this protocol details how study staff will assess and respond to acute crisis.  
Study staff (including anyone involved in enrollment and informed consent or delivery of the follow-up 
intervention or outcome measurements, and/or having access to patient-level data) will be trained on 
the protection of human subjects and HIPAA, with a focus on topics relevant to confidentiality. SLHS 
staff will assist participants in completing Aim 2 informed consent, and REDCap survey links will be sent 
out via text, email, or MyChart.  Study staff at SLHS, University of Washington and the Hotline will have 
access to protected health information (PHI).
The study is designed to be low burden in terms of participants’ time.  The initial survey (Aim 1) will 
consist of up to 50 multiple choice questions. Those participants who enroll in the comparative 
effectiveness trial (Aim 2) will complete a baseline survey (up to 50 questions), receive 11 caring text 
messages over a 6-month period, and depending on intervention arm, may also receive an introductory 

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 40 of 72
phone call.  Aim 2 study participants will complete an outcomes assessment survey online, which will 
consist of up to 50 questions.  Participants will be reimbursed for their time.
Participants will be under no duress or pressure to participate in or complete this study. 

Patient Participants.  Participating in this study will not impact the care they receive, and this 
will be clearly communicated to participants as part of the informed consent process.

Provider/Staff Participants. Participating in this study will not impact their employment, and 
information received in this study will not be shared outside of the study staff. This will be 
clearly communicated to participants as part of the informed consent process.
15.2 Potential Benefits
While this study is designed to improve mental health, there is no guarantee that participants will 
benefit directly from the proposed interventions. Although all participants will receive an evidence-
based intervention, patients randomized to receive the CC+ arm may benefit from the additional 
contact. 
The knowledge gained through this study is expected to advance the understanding of the burden of 
COVID-19 on patients, providers, and staff.  It is also expected to inform which version of Caring 
Contacts is most effective for reducing loneliness and mental distress among patients, providers, and 
staff.  We expect the findings of this analysis to be relevant and of interest to community members and 
people with lived experience with suicide and mental distress, the scientific community, national and 
state suicide prevention hotlines, and health system leaders.  SLHS leaders will benefit from information 
on how best to support patients, providers, and staff struggling with mental distress in the COVID-19 
era. 
16.Oversight for Human Subjects Protection & Regulatory 
Considerations
16.1 Human Subjects Protection
This study will be conducted with appropriate oversight from the St. Luke’s Health System (SLHS) 
Institutional Review Board (IRB).  The IRB will review and approve all aspects of the study once all 
criteria for approval have been met, including the protocol, informed consent process, and all relevant 
study-related documents. This includes an initial review and approval process and an annual review, as 
well as review of any modifications made prior to and after initiation of the study. All changes will be 
approved by the IRB prior to implementation. The Principal Investigator (PI) will be responsible for 
ensuring compliance with IRB regulations and procedures. All key study personnel will be trained in 
human subjects’ protection. 

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 41 of 72
16.2 Risks to Human Subjects
16.2.1 Characteristics of the Study Population

Patients: Adult (aged 18 years and older) and adolescent (aged 12 – 17 years) patients who 
report elevated loneliness, suicide ideation, or other mental distress who have sought care at a 
St. Luke’s primary care clinic in Idaho in the past 12 months will be eligible for participation in 
the proposed study. 

Providers & Employees: St. Luke’s providers and employees who report elevated loneliness, 
suicide ideation, or other mental distress will be eligible for participation in the proposed study.
16.2.2 Involvement of Human Subjects
Suicide constitutes a significant public health concern and is a leading cause of death in the United 
States.23  However, little is known about health system level interventions to prevent suicide among 
civilians, or adolescents. 117  Additionally, mental distress related to COVID-19 is a critically important but 
poorly understood aspect of the global pandemic. 25,28 This study will estimate the prevalence of several 
measures of mental distress in key populations and compare the effectiveness of two versions of the 
evidence-based Caring Contacts model. The proposed research design is a randomized clinical trial, and 
all study participants will receive an active evidence-based intervention. No one will receive a “placebo” 
or “null” treatment.   
Participation in this research involves the following:
Aim 1: Survey 

Complete an online survey (10 minutes) 
Aim 2: Comparative Effectiveness Trial 

Baseline (Month 0) 
o
Complete study enrollment and informed consent (30 minutes) 
o
Complete baseline survey (20 minutes) 

Months 0-6
o
Receive Caring Contacts intervention (6 months) 
o
1 introductory phone call (one intervention arm only) (20 minutes) 
o
11 caring text messages (both intervention arms) 

Month 6 
o
Complete online outcome assessment survey at 6 months (20 minutes) 
16.2.3 Protecting Individuals with Urgent Clinical Needs
The Aim 1 survey informed consent process will be self-completed by participants, thus study staff will 
not have an opportunity to assess the participant’s mental state prior to completion of the survey.  
Enrollment for Aim 2 (clinical trial) will involve a phone call with study staff (typically, a research 
coordinator). This protocol prioritizes individual participants’ urgent clinical needs (for example, 

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 42 of 72
imminent risk of self-harm) above research related needs or responsibilities.  Study staff will be trained 
that their first responsibility is to protect the safety and wellbeing of study participants (especially those 
experiencing suicidal crisis or another safety outcome), with duty to the research protocol taking second 
priority. For example, if a potential participant becomes agitated or indicates intent to harm themselves, 
study enrollment procedures should be halted and safety procedures (such as the modified Red Flag 
Workflow for those with suicide risk) would be put into place.  Standard operating procedures outlining 
these safety protocols will be utilized. 
Key study staff will complete training in suicide prevention, which includes recognizing when someone 
may be experiencing an acute suicidal crisis. Study staff who have completed suicide prevention training 
will be available should the need for consultation arise. Study staff will interact directly with participants 
via phone during the Aim 2 study enrollment and informed consent process.  If a potential participant 
displays signs of acute suicidal crisis during this call, study staff will follow standard operating 
procedures to keep participants as safe as possible.   
A participant who responds “yes” to question 5 and/or question 6 in the C-SSRS as part of one of the 
online surveys will be considered at high risk for suicide. In this instance, an automatic notification 
directly to study staff and the Hotline will be triggered in REDCap.  The Hotline will attempt to do a 
safety check via phone as soon as possible within 6-12 hours of survey completion.  At least three 
attempts at phone contact will be made and text messaging may be utilized as well.  All individuals who 
complete surveys will be supplied with a list of resources that they can access if they or someone they 
know needs support with mental health, suicidal crisis, or social services. 
16.2.4 Randomization 
Randomization will occur at the individual level, with each participant randomly assigned to one of the 
two intervention groups.
16.3 Informed Consent Process
Consent for participation in the initial survey (Aim 1) will be completed electronically. Potential 
participants will be asked to read and electronically sign an informed consent document prior to 
proceeding with the survey questions. 
Consent for participation in the comparative effectiveness trial (Aim 2) will be completed over the 
phone with Research Coordinators or other trained study staff. If the patient is younger than 18 years of 
age, his or her parent or guardian will provide written permission for the minor to participate in the 
study, after which the minor participant will go through an informed assent process. Those participants 
that assent to study participation prior to age 18 will be reconsented to continue participating if they 
turn 18 years old while participating in the study. Recruitment materials will be written at approximately 
an 8th grade reading level to maximize comprehension. Furthermore, the Patients with Lived Experience 

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 43 of 72
of Suicide (PLES) Advisory Board will review the informed consent documents for readability and clarity. 
Employees will be assured that their decision whether to participate in the research and any responses 
they provide will in no way impact their employment with St. Luke’s Health System. 
16.3.1 Documentation of Informed Consent
Informed consent (and assent for participants aged 12-17) will be obtained and documented for all 
study participants.  Aim 1 informed consent documentation will occur electronically.  Study staff will 
document each participants’ eligibility to participate in the clinical trial (Aim 2) prior to enrollment.  
When an Aim 2 participant signs the informed consent (or assent for participants aged 12-17) document 
electronically, the study staff will also sign a form attesting that they have screened for eligibility, 
reviewed the consent information, and responded to all questions from the participant.  The informed 
consent form will be completed in REDCap, and the consent/assent form will be combined electronically 
with the staff attestation form in a single patient record.  All primary data collection for this study will be 
done electronically.
16.4 Bioethics Consultation
The University of Washington’s Institute for Translational Health Sciences hosts a Bioethics Consultation 
Service that will be utilized for the duration of the study.  Grant resources have been set aside for a 
bioethics consultation should relevant questions arise. 
16.5 Inclusion of Women and Minorities
Efforts will be made to recruit women and minorities according to their representation in the research 
population. Given the short timeline available for MHAPPS, only English-speaking participants will be 
eligible for inclusion; this may result in an under-representation of non-English-speaking populations.  
There are no exclusion criteria based on sex/gender or minority status. Information about the 
distribution of races, ethnicities, and gender in our study population can be found in the Estimated Final 
Racial/Ethnic and Gender Enrollment Table.

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 44 of 72
Table 3: MHAPPS Estimated Final Racial/Ethnic and Gender Enrollment
Based on patient volumes and proportion of elevated PHQ-3 scores in primary care settings from July 2019 through June 30, 2020
AIM 1 (Survey)
AIM 2 (Trial)
Race
Male (N)
Female (N)
Total (N)
Male (N)
Female (N)
Total (N)
American Indian/Alaska 
Native
   
7 
   
16 
   
23 
1
3
4
Asian
   
16 
   
32 
   
48
2
5
7
Black/African American
   
11 
   
14 
   
25
2
3
5
Hawaiian/Pacific Islander
   
3 
   
6 
   
10
1
1
2
White
   
1,410 
   
2,205 
   
3,615 
156
336
492
Multi-race or Other
   
106 
   
174
   
280
17
34
51
Ethnicity
Male (N)
Female (N)
Total (N)
Male (N)
Female (N)
Total (N)
Hispanic (Latino/Latina)
   
118 
   
209 
   
328 
25
48
72
Non-Hispanic
   
1,436 
   
2,237 
   
3,672 
155
333
488
16.6 Inclusion of Minors
Adolescents face a disproportionate burden of suicidal ideation73,117 compared to adults in general and 
have been shown to have particularly elevated levels of suicidal ideation and mental distress related to 
COVID-19. 43  Additional evidence is needed to determine the most effective interventions to prevent 
suicide and address other forms of mental distress at the health system level for minors.117  Adolescents 
who screen positive for loneliness or other forms of mental distress will be included in this study in 
order to address this critical gap in the literature and begin to develop an evidence base for suicide 
prevention among adolescents in healthcare settings.
16.7 Cost and Compensation for Participation
Costs of participating in this study include the time participants spend enrolling in the study and 
completing questionnaires, and the cost of receiving text messages, emails, and phone calls as part of 
the intervention and/or outcome assessments.
16.7.1 Participant Compensation
Study participants may receive up to $80 in compensation in the form of electronic Amazon gift cards 

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 45 of 72
over a 6-month period. These funds are intended to compensate participants’ time spent discussing 
sensitive topics and are in no way meant to influence participation in the study. The compensation will 
be distributed as follows:

Aim 1 Informed consent and survey: $10 Amazon gift card

Aim 2 Informed consent and enrollment in the clinical trial: $35 Amazon gift card

Aim 2 Six-month outcome assessment: $35 Amazon gift card
16.8 Study Discontinuation and Closure
This study may be temporarily suspended or prematurely terminated if there is sufficient reasonable 
cause.  Written notification, documenting the reason for study suspension or termination, will be 
provided by the suspending or terminating party to the PIs. If the study is prematurely terminated or 
suspended, the PIs will promptly inform all study investigators, study participants, PCORI, and the SLHS 
IRB, and will provide the reason for termination or suspension. Study participants will be informed, as 
applicable, of any changes to the study schedule. 
The following circumstances may warrant termination or suspension:

Determination of unexpected significant or unacceptable risk to participants

Insufficient compliance to protocol requirements

Data that are not sufficiently complete and/or evaluable

Determination that the primary endpoint has been met

Determination of futility

Other reasonable causes not listed here
The study may resume once concerns about safety, protocol compliance, and data quality are addressed 
to the satisfaction of PCORI, the SLHS IRB, the Data and Safety Monitoring Board (DSMB), and other 
regulatory or oversight bodies.
16.9 Confidentiality & Privacy
Participant confidentiality and privacy is strictly held in trust by the participating investigators, their 
staff, the DSMB, the SLHS IRB, the Hotline, and PCORI. This confidentiality is extended to the data being 
collected as part of this study.  Data that could be used to identify a specific study participant will be 
held in strict confidence within the research team. For minor participants whose responses indicate 
suicide risk, that information may be shared with a parent or legally authorized representative. No 
personally identifiable information from the study will be released to any unauthorized third party 
without prior written approval of PCORI and the SLHS IRB.
All research activities will be conducted in as private a setting as possible. 

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 46 of 72
Authorized representatives of PCORI, the DSMB, or SLHS, including the SLHS IRB, may inspect all 
documents and records required to be maintained by the investigators, including but not limited to 
medical records for the participants of this study.  The clinical study site will permit access to such 
records for authorized review.
Study participants’ contact information will be securely stored in REDCap and/or on secure UW or SLHS 
servers for internal use during the study.  At the end of the study, all records will be kept in a secure 
location in REDCap and/or on SLHS secure servers for 10 years, in accordance with SLHS data retention 
policy.
Study participant research data, which is for purposes of statistical analysis and scientific reporting, will 
be transmitted via and stored on REDCap, a HIPAA compliant web-based research database.  At the end 
of the study, all study data will be de-identified prior to publication; research data will be archived at 
SLHS for storage for 10 years.
16.9.1 Measures to Ensure Confidentiality of Shared Data
It is PCORI policy that results and accomplishments of the research that it funds should be made 
available to the public.  The PIs will ensure all mechanism used to share data will include proper plans 
and safeguards for the protection of privacy, confidentiality, and security for data dissemination and 
reuse (e.g., all data will be thoroughly de-identified and will not be traceable to a specific study 
participant).  
16.10 Study Records Retention
Study records will be retained for 10 years, in accordance with SLHS institutional policy.  No records will 
be destroyed before that time without the written consent of PCORI and/or the SLHS Compliance 
department.
16.11 Publication & Data Sharing Policy
The PI will be responsible for developing publication procedures and resolving authorship issues.  This 
study will be conducted in accordance with all PCORI and SLHS data sharing policies and regulations.  
This trial will be registered at ClinicalTrials.gov, and the results of this trial will be submitted to 
ClinicalTrials.gov, which ensures that the public has access to the published results of this PCORI-funded 
research.  In addition, results will be submitted for publication in peer reviewed journals.  Data from this 
trial may be requested from other researchers 5 years after the completion of the primary endpoint by 
contacting the PIs.  Considerations for ensuring confidentiality of these shared data are described in 
section 16.9 Confidentiality & Privacy of this protocol.

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 47 of 72
16.12 Dissemination of Results
Any publication or presentation of the results of this study will be presented in aggregate form and will 
not include any patient identifying information. 
16.13 Conflict of Interest Policy
The independence of this study from any actual or perceived influence is critical.  Therefore, any actual 
conflict of interest of persons who have a role in the design, conduct, analysis, publication, or any aspect 
of this trial will be disclosed and managed. 
16.14 Protocol Deviations
This protocol defines a protocol deviation as any noncompliance with the clinical trial protocol, 
International Conference on Harmonization Good Clinical Practice Guidelines (ICH GCP), or SLHS IRB 
requirements.  The noncompliance may be either on the part of the participant, investigator, study staff, 
or study site staff.  Corrective actions will be developed by the site and implemented promptly in the 
event of protocol deviations, consistent with ICH GCP:

Section 4.5 Compliance with Protocol, subsections 4.5.1, 4.5.2, and 4.5.3

Section 5.1 Quality Assurance and Quality Control, subsection 5.1.1

Section 5.20 Noncompliance, subsections 5.20.1 and 5.20.2
Study staff will conduct quality assurance monitoring and internal audits on a regular basis.  Study staff 
and study site staff will be responsible for being vigilant to identify and report deviations in accordance 
with the SLHS IRB Procedures Manual.  All deviations will be addressed in study source documents and 
reported to the PIs; deviations deemed reportable based on criteria in the SLHS IRB Procedures Manual 
will be reported to the SLHS IRB.  Study staff including study site champions will be responsible for 
knowing and adhering to the IRB requirements.  
16.15 Key Roles for Study Oversight
Table 4: Key Roles for Study Oversight, MHAPPS Trial
Principal Investigators & Data & Safety Monitoring Board Leaders
Principal Investigator
Data & Safety Monitoring 
Board Chair
Anna Radin, DrPH, MPH, 
Applied Research Scientist
Ann Melvin, Chair, Data and 
Safety Monitoring Board
St. Luke’s Health System
University of Washington
Institute of Translational Health 
Sciences
208-381-8468
206-290-8294
radina@slhs.org
ann.melvin@seattlechildrens.org  
IRB and Compliance

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 48 of 72
IRB
St. Luke’s IRB: 208-381-1406
St. Luke’s IRB is the IRB of 
record for this research study.
Compliance
St. Luke’s 24/7 Compliance 
Hotline: 1-800-729-0966
St. Luke’s Health System 
maintains a compliance hotline 
that is available 24/7 to take 
compliance-related calls.
16.15.1 Data and Safety Monitoring Board
The Data and Safety Monitoring Board (DSMB) will convene at the beginning of the study to review the 
protocol, charter, and data reporting tables, then again three months after enrollment begins.  The 
DSMB will review enrollment data and to review data for safety.  Additional information on safety 
monitoring is included in the Data & Safety Monitoring section of this protocol.
16.16 Quality Assurance & Quality Control
Study staff will support each study site to perform internal quality management of study conduct, data 
collection, and documentation.  Data reports will be routinely reviewed by study staff in consultation 
with the PI in order to understand how recruitment, informed consent, and retention processes are 
going. 
Quality control (QC) measures will be implemented as follows

Informed consent – Study staff will review both documentation of the consenting process as 
well as at least 10% of the completed consent documents.  This review will evaluate compliance 
with procedures described in this protocol, accuracy, and completeness.  Feedback will be 
provided to staff at study sites to ensure proper consenting procedures are followed.

Protocol deviations – The study team will review protocol deviations on an ongoing basis and 
will implement corrective actions when the quantity or nature of deviations are deemed to 
require remediation.
Should independent monitoring of the study become necessary, the PI will provide direct access to all 
trial-related sites, source data/documents, and reports for the purpose of monitoring and auditing by 
internal SLHS auditing bodies, PCORI, and inspection by local and regulatory authorities, in compliance 
with SLHS legal guidance.

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 49 of 72
17.References
1. Klim C, Ganoczy D, Pfeiffer PN. Associations between the NIH toolbox adult social relationship scales 
and suicidal ideation. Community Ment Health J. 2020;56(1):186-192. 
https://search.ebscohost.com/login.aspx?direct=true&AuthType=ip,url,uid&db=mdc&AN=31571088&sit
e=ehost-live&scope=site. doi: 10.1007/s10597-019-00483-9.
2. Salsman JM, Butt Z, Pilkonis PA, et al. Emotion assessment using the NIH toolbox. Neurology. 
2013;80(Issue 11, Supplement 3):S76-S86. 
https://search.datacite.org/works/10.1212/wnl.0b013e3182872e11. doi: 
10.1212/wnl.0b013e3182872e11.
3. Gershon RC, Wagster MV, Hendrie HC, Fox NA, Cook KF, Nowinski CJ. NIH toolbox for assessment of 
neurological and behavioral function. Neurology. 2013;80(Issue 11, Supplement 3):S2-S6. 
https://search.datacite.org/works/10.1212/wnl.0b013e3182872e5f. doi: 
10.1212/wnl.0b013e3182872e5f.
4. Posner K, Brown GK, Stanley B, et al. The Columbia–Suicide severity rating scale: Initial validity and 
internal consistency findings from three multisite studies with adolescents and adults. American Journal 
of Psychiatry. 2011;168(12):1266-1277. http://dx.doi.org/10.1176/appi.ajp.2011.10111704. doi: 
10.1176/appi.ajp.2011.10111704.

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 50 of 72
5. Barzilay S, Yaseen ZS, Hawes M, et al. Determinants and predictive value of clinician assessment of 
short-term suicide risk. Suicide Life Threat Behav. 2019;49(2):614-626. 
https://search.ebscohost.com/login.aspx?direct=true&AuthType=ip,url,uid&db=mdc&AN=29665120&sit
e=ehost-live&scope=site. doi: 10.1111/sltb.12462.
6. Van Gerpen S, Vik T, Soundy TJ. Assessing adolescent suicide risk. S D Med. 2020;73(2):82-86. 
https://search.ebscohost.com/login.aspx?direct=true&AuthType=ip,url,uid&db=mdc&AN=32135057&sit
e=ehost-live&scope=site.
7. Erford BT, Jackson J, Bardhoshi G, Duncan K, Atalay Z. Selecting suicide ideation assessment 
instruments: A meta-analytic review. Measurement & Evaluation in Counseling & Development. 
2018;51(1):42-59. 
https://search.ebscohost.com/login.aspx?direct=true&AuthType=ip,url,uid&db=pbh&AN=127116368&si
te=ehost-live&scope=site. doi: 10.1080/07481756.2017.1358062.
8. Park CH, Kim H, Kim B, et al. Predicting planned suicide attempts with the columbia-suicide severity 
rating scale: A subanalysis of the 2013 korea national suicide survey on emergency department visitors. J 
Nerv Ment Dis. 2019;207(2):59-68. 
https://search.ebscohost.com/login.aspx?direct=true&AuthType=ip,url,uid&db=ccm&AN=134603325&s
ite=ehost-live&scope=site. doi: 10.1097/NMD.0000000000000921.
9. Núñez D, Arias V, Méndez-Bustos P, Fresno A. Is a brief self-report version of the columbia severity 
scale useful for screening suicidal ideation in chilean adolescents? Compr Psychiatry. 2019;88:39-48. 

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 51 of 72
https://search.ebscohost.com/login.aspx?direct=true&AuthType=ip,url,uid&db=mdc&AN=30471550&sit
e=ehost-live&scope=site. doi: 10.1016/j.comppsych.2018.11.002.
10. Interian A, Chesin M, Kline A, et al. Use of the columbia-suicide severity rating scale (C-SSRS) to 
classify suicidal behaviors. Arch Suicide Res. 2018;22(2):278-294. 
https://search.ebscohost.com/login.aspx?direct=true&AuthType=ip,url,uid&db=mdc&AN=28598723&sit
e=ehost-live&scope=site. doi: 10.1080/13811118.2017.1334610.
11. Katz I, Barry CN, Cooper SA, Kasprow WJ, Hoff RA. Use of the columbia-suicide severity rating scale 
(C-SSRS) in a large sample of veterans receiving mental health services in the veterans health 
administration. Suicide Life Threat Behav. 2020;50(1):111-121. 
https://search.ebscohost.com/login.aspx?direct=true&AuthType=ip,url,uid&db=mdc&AN=31441952&sit
e=ehost-live&scope=site. doi: 10.1111/sltb.12584.
12. Matarazzo BB, Brown GK, Stanley B, et al. Predictive validity of the columbia-suicide severity rating 
scale among a cohort of at-risk veterans. Suicide Life Threat Behav. 2019;49(5):1255-1265. 
https://search.ebscohost.com/login.aspx?direct=true&AuthType=ip,url,uid&db=mdc&AN=30368871&sit
e=ehost-live&scope=site. doi: 10.1111/sltb.12515.
13. Salvi J. Calculated decisions: Columbia-suicide severity rating scale (C-SSRS). Emerg Med Pract. 
2019;21(5):CD3-CD4. 
https://search.ebscohost.com/login.aspx?direct=true&AuthType=ip,url,uid&db=mdc&AN=31039299&sit
e=ehost-live&scope=site.

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 52 of 72
14. Joiner T. Interpersonal needs questionnaire (INQ-15). https://psy.fsu.edu/~joinerlab/measures/INQ-
15.pdf. 2012.
15. Van Orden KA, Cukrowicz KC, Witte TK, Joiner TE. Thwarted belongingness and perceived 
burdensomeness: Construct validity and psychometric properties of the interpersonal needs 
questionnaire. Psychol Assess. 2012;24(1):197-215. 
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3377972/. Accessed Jul 15, 2020. doi: 
10.1037/a0025358.
16. Heaton RK, Akshoomoff N, Tulsky D, et al. Reliability and validity of composite scores from the NIH 
toolbox cognition battery in adults. Journal of the International Neuropsychological Society. 
2014;20(6):588-598. https://dx.doi.org/10.1017/S1355617714000241. doi: 
10.1017/S1355617714000241.
17. Arroll, Bruce, Goodyear-Smith, Felicity, Crengle, Susan, Gunn, Jane, Kerse, Ngaire, Fishman, Tana, 
Falloon, Karen, Hatcher, Simon. Validation of PHQ-2 and PHQ-9 to screen for major depression in the 
primary care population. Annals of Family Medicine. 2010;8(4):348-353. 
https://www.clinicalkey.es/playcontent/1-s2.0-S154417091060056X. doi: 10.1370/afm.1139.
18. Na PJ, Yaramala SR, Kim JA, et al. The PHQ-9 item 9 based screening for suicide risk: A validation 
study of the patient health questionnaire (PHQ)-9 item 9 with the columbia suicide severity rating scale 
(C-SSRS). J Affect Disord. 2018;232:34-40. 

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 53 of 72
https://search.ebscohost.com/login.aspx?direct=true&AuthType=ip,url,uid&db=mdc&AN=29477096&sit
e=ehost-live&scope=site. doi: 10.1016/j.jad.2018.02.045.
19. Kroenke K, Spitzer R, Williams J. The PHQ-9 - validity of a brief depression measure. J GEN INTERN 
MED. 2001;16(9):606-613. https://www.ncbi.nlm.nih.gov/pubmed/11556941. doi: 10.1046/j.1525-
1497.2001.016009606.x.
20. Spitzer RL, Kroenke K, Williams JBW, Löwe B. A brief measure for assessing generalized anxiety 
disorder: The GAD-7. Arch Intern Med. 2006;166(10):1092-1097. Accessed Jul 15, 2020. doi: 
10.1001/archinte.166.10.1092.
21. Jordan P, Shedden-Mora MC, Löwe B. Psychometric analysis of the generalized anxiety disorder scale 
(GAD-7) in primary care using modern item response theory. PLoS One. 2017;12(8). 
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5542568/. Accessed Jul 15, 2020. doi: 
10.1371/journal.pone.0182162.
22. Bose J, Hedden SL, Lipari RN, Park-Lee E. Key substance use and mental health indicators in the 
united states: Results from the 2017 national survey on drug use and health. US Official News. 2018. 
Available from: https://www.samhsa.gov/data/sites/default/files/cbhsq-
reports/NSDUHFFR2017/NSDUHFFR2017.pdf.
23. Kegler SR, Ivey-Stephenson AZ, Stone DM, et al. Vital signs: Trends in state suicide rates--united 
states, 1999-2016 and circumstances contributing to suicide--27 states, 2015. Morbidity and Mortality 
Weekly Report. Jun 8, 2018;67:617.

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 54 of 72
24. Delcker J, Wanat Z, Scott M. The coronavirus fake news pandemic sweeping WhatsApp. Politico. Mar 
17, 2020. Available from: https://search.proquest.com/docview/2377556360.
25. Holmes EA, O'Connor RC, Perry VH, et al. Multidisciplinary research priorities for the COVID-19 
pandemic. Lancet psychiatry. 2020;7(6):547. http://urn.kb.se/resolve?urn=urn:nbn:se:uu:diva-414114.
26. Malathesh BC, Chatterjee SS, Das S. Overview of mental health issues of COVID-19: Need of the 
hour. General Psychiatry. 2020;33(3):e100233. http://dx.doi.org/10.1136/gpsych-2020-100233. doi: 
10.1136/gpsych-2020-100233.
27. Galea S, Merchant RM, Lurie N. The mental health consequences of COVID-19 and physical 
distancing: The need for prevention and early intervention. JAMA internal medicine. 2020;180(6):817-
818. http://dx.doi.org/10.1001/jamainternmed.2020.1562. doi: 10.1001/jamainternmed.2020.1562.
28. Gunnell D, Appleby L, Arensman E, et al. Suicide risk and prevention during the COVID-19 pandemic. 
Lancet Psychiatry. 2020;7(6):468-471. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7173821/. 
Accessed Jul 8, 2020. doi: 10.1016/S2215-0366(20)30171-1.
29. Carbone SR. Flattening the curve of mental ill-health: The importance of primary prevention in 
managing the mental health impacts of COVID-19. Mental Health & Prevention. 2020;19:200185. 
http://www.sciencedirect.com/science/article/pii/S2212657020300921. Accessed Jul 8, 2020. doi: 
10.1016/j.mhp.2020.200185.

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 55 of 72
30. Giallonardo V, Sampogna G, Del Vecchio V, et al. The impact of quarantine and physical distancing 
following COVID-19 on mental health: Study protocol of a multicentric italian population trial. Front 
Psychiatry. 2020;11. https://www.frontiersin.org/articles/10.3389/fpsyt.2020.00533/full. Accessed Jul 8, 
2020. doi: 10.3389/fpsyt.2020.00533.
31. Huang Y, Zhao N. Generalized anxiety disorder, depressive symptoms and sleep quality during 
COVID-19 outbreak in china: A web-based cross-sectional survey. Psychiatry Research. 
2020;288:112954. http://dx.doi.org/10.1016/j.psychres.2020.112954. doi: 
10.1016/j.psychres.2020.112954.
32. Silva Junior, Fernando Jose Guedes da, Sales, Jaqueline Carvalho e Silva, Monteiro, Claudete Ferreira 
de Souza, et al. Impact of COVID-19 pandemic on mental health of young people and adults: A 
systematic review protocol of observational studies. BMJ open. 2020;10(7):e039426. 
http://dx.doi.org/10.1136/bmjopen-2020-039426. doi: 10.1136/bmjopen-2020-039426.
33. Ornell F, Schuch JB, Sordi AO, Kessler FHP. "Pandemic fear" and COVID-19: Mental health burden 
and strategies. Revista brasileira de psiquiatria. 2020;42(3):232-235. 
https://www.ncbi.nlm.nih.gov/pubmed/32267343. doi: 10.1590/1516-4446-2020-0008.
34. Choi EPH, Hui BPH, Wan EYF. Depression and anxiety in hong kong during COVID-19. International 
journal of environmental research and public health. 2020;17(10):3740. 
https://www.ncbi.nlm.nih.gov/pubmed/32466251. doi: 10.3390/ijerph17103740.

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 56 of 72
35. Wang C, Pan R, Wan X, et al. Immediate psychological responses and associated factors during the 
initial stage of the 2019 coronavirus disease (COVID-19) epidemic among the general population in 
china. International journal of environmental research and public health. 2020;17(5):1729. 
https://search.datacite.org/works/10.3390/ijerph17051729. doi: 10.3390/ijerph17051729.
36. Loades ME, Chatburn E, Higson-Sweeney N, et al. Rapid systematic review: The impact of social 
isolation and loneliness on the mental health of children and adolescents in the context of COVID-19. 
Journal of the American Academy of Child & Adolescent Psychiatry. 2020. 
http://www.sciencedirect.com/science/article/pii/S0890856720303373. Accessed July 8, 2020. doi: 
10.1016/j.jaac.2020.05.009.
37. Merikangas KR, He J, Burstein M, et al. Lifetime prevalence of mental disorders in U.S. adolescents: 
Results from the national comorbidity survey Replication–Adolescent supplement (NCS-A). Journal of 
the American Academy of Child & Adolescent Psychiatry. 2010;49(10):980-989. 
https://search.datacite.org/works/10.1016/j.jaac.2010.05.017. doi: 10.1016/j.jaac.2010.05.017.
38. Armitage R, Nellums LB. COVID-19 and the consequences of isolating the elderly. The Lancet. Public 
health. 2020;5(5):e256. https://search.datacite.org/works/10.1016/s2468-2667(20)30061-x. doi: 
10.1016/s2468-2667(20)30061-x.
39. Loades ME, Chatburn E, Higson-Sweeney N, et al. Rapid systematic review: The impact of social 
isolation and loneliness on the mental health of children and adolescents in the context of COVID-19. 

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 57 of 72
Journal of the American Academy of Child and Adolescent Psychiatry. 2020. 
https://search.proquest.com/docview/2410349500. doi: 10.1016/j.jaac.2020.05.009.
40. Matthews T, Danese A, Caspi A, et al. Lonely young adults in modern britain: Findings from an 
epidemiological cohort study. Psychological medicine. 2019;49(2):268-277. 
https://dx.doi.org/10.1017/S0033291718000788. doi: 10.1017/S0033291718000788.
41. Elovainio M, Hakulinen C, Pulkki-Råback L, et al. Contribution of risk factors to excess mortality in 
isolated and lonely individuals: An analysis of data from the UK biobank cohort study. The Lancet. Public 
health. 2017;2(6):e260-e266. https://search.datacite.org/works/10.1016/s2468-2667(17)30075-0. doi: 
10.1016/s2468-2667(17)30075-0.
42. O'Connor RC, Nock MK. The psychology of suicidal behaviour. The Lancet. Psychiatry. 2014;1(1):73-
85. https://search.datacite.org/works/10.1016/s2215-0366(14)70222-6. doi: 10.1016/S2215-
0366(14)70222-6.
43. Czeisler ME, Lane RI, Petrosky E, et al. Mental health, substance use, and suicide ideation during the 
COVID-19 pandemic- united states , june 24-30, 2020. MMWR Morb Mortal Wkly Rep. 2020;69:1049-
1057. http://dx.doi.org/10.15585/mmwr.mm6932a1.
44. National Action Alliance for Suicide Prevention: Transforming Health Systems Initiative Work Group. 
Recommended standard care for people with suicide risk: Making health care suicide safe. . 2018.

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 58 of 72
45. Kisely S, Warren N, McMahon L, Dalais C, Henry I, Siskind D. Occurrence, prevention, and 
management of the psychological effects of emerging virus outbreaks on healthcare workers: Rapid 
review and meta-analysis. BMJ. 2020;369:m1642. 10.1136/bmj.m1642. Accessed Jul 17, 2020. doi: 
10.1136/bmj.m1642.
46. Lai J, Ma S, Wang Y, et al. Factors associated with mental health outcomes among health care 
workers exposed to coronavirus disease 2019. JAMA Network Open. 2020;3(3):e203976. 
http://dx.doi.org/10.1001/jamanetworkopen.2020.3976. doi: 10.1001/jamanetworkopen.2020.3976.
47. Patient Centered Outcomes Research Institute, (PCORI). PCORI funding opportunity for COVID-19-
related enhancements to existing PCORI-funded research projects
. https://www.pcori.org/funding-opportunities/announcement/covid-19-related-enhancements-
research-awards. Updated 2020.
48. PCORI Funding Announcement:, Pragmatic Clinical Studies To Evaluate Patient-Centered, Outcomes. 
Cycle 1 2018 funding cycle. . 
49. Motto JA, Bostrom AG. A randomized controlled trial of postcrisis suicide prevention. Psychiatric 
Services. 2001;52(6):828-833. http://dx.doi.org/10.1176/appi.ps.52.6.828. doi: 
10.1176/appi.ps.52.6.828.
50. Motto JA. Suicide prevention for high-risk persons who refuse treatment. Suicide and Life 
Threatening Behavior. 1976;6(4):223-30.

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 59 of 72
51. Comtois KA, Kerbrat AH, DeCou CR, et al. Effect of augmenting standard care for military personnel 
with brief caring text messages for suicide prevention: A randomized clinical trial. JAMA Psychiatry. 
2019;76(5):474-483. 
https://search.ebscohost.com/login.aspx?direct=true&AuthType=ip,url,uid&db=mdc&AN=30758491&sit
e=ehost-live&scope=site. doi: 10.1001/jamapsychiatry.2018.4530.
52. Detecting and treating suicide ideation in all settings. Sentinel event alert. 2016(56):1-7. 
https://search.proquest.com/docview/1768560921.
53. The Assessment and Management of Suicide Risk Work Group, Office of Safety and Value, VA / 
Office of Evidence-Based Practice, US Army Medical Command. VA/DOD clinical practice guidelines for 
the assessment and management of patients at risk for suicide. 
https://www.healthquality.va.gov/guidelines/MH/srb/VADoDSuicideRiskFullCPGFinal5088919.pdf. 
Updated 2019. Accessed April 16, 2019.
54. Matsubara T, Matsuo K, Matsuda A, et al. Combining phone and postcard brief contact interventions 
for preventing suicide reattempts: A quasi-randomized controlled trial. Psychiatry Res. 2019;279:395-
396. 
https://search.ebscohost.com/login.aspx?direct=true&AuthType=ip,url,uid&db=mdc&AN=31133416&sit
e=ehost-live&scope=site. doi: 10.1016/j.psychres.2019.05.033.
55. Vaiva G, Berrouiguet S, Walter M, et al. Combining postcards, crisis cards, and telephone contact 
into a decision-making algorithm to reduce suicide reattempt: A randomized clinical trial of a 

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 60 of 72
personalized brief contact intervention. J Clin Psychiatry. 2018;79(6). 
https://search.ebscohost.com/login.aspx?direct=true&AuthType=ip,url,uid&db=mdc&AN=30256552&sit
e=ehost-live&scope=site. doi: 10.4088/JCP.17m11631.
56. Luxton DD, Smolenski DJ, Reger MA, Relova RMV, Skopp NA. Caring E-mails for military and veteran 
suicide prevention: A randomized controlled trial. Suicide Life Threat Behav. 2020;50(1):300-314. 
https://search.ebscohost.com/login.aspx?direct=true&AuthType=ip,url,uid&db=mdc&AN=31562660&sit
e=ehost-live&scope=site. doi: 10.1111/sltb.12589.
57. Berrouiguet S, Gravey M, Le Galudec M, Alavi Z, Walter M. Post-acute crisis text messaging outreach 
for suicide prevention: A pilot study. Psychiatry Research. 2014;217(3):154-157. 
https://search.datacite.org/works/10.1016/j.psychres.2014.02.034. doi: 
10.1016/j.psychres.2014.02.034.
58. Ammerman BA, Gebhardt HM, Lee JM, Tucker RP, Matarazzo BB, Reger MA. Differential preferences 
for the caring contacts suicide prevention intervention based on patient characteristics. Arch Suicide 
Res. 2019:1-12. 
https://search.ebscohost.com/login.aspx?direct=true&AuthType=ip,url,uid&db=mdc&AN=31213148&sit
e=ehost-live&scope=site. doi: 10.1080/13811118.2019.1632231.
59. Agyapong VIO, Juhás M, Ohinmaa A, et al. Randomized controlled pilot trial of supportive text 
messages for patients with depression. BMC psychiatry. 2017;17(1):286. 
https://search.datacite.org/works/10.1186/s12888-017-1448-2. doi: 10.1186/s12888-017-1448-2.

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 61 of 72
60. Tomita A, Kandolo KM, Susser E, Burns JK. Use of short messaging services to assess depressive 
symptoms among refugees in south africa: Implications for social services providing mental health care 
in resource-poor settings. Journal of telemedicine and telecare. 2016;22(6):369-377. 
https://search.datacite.org/works/10.1177/1357633x15605406. doi: 10.1177/1357633x15605406.
61. Broom MA, Ladley AS, Rhyne EA, Halloran DR. Feasibility and perception of using text messages as an 
adjunct therapy for low-income, minority mothers with postpartum depression. JMIR mental health. 
2015;2(1):e4. https://search.datacite.org/works/10.2196/mental.4074. doi: 10.2196/mental.4074.
62. Landes SJ, Kirchner JE, Areno JP, et al. Adapting and implementing caring contacts in a department of 
veterans affairs emergency department: A pilot study protocol. Pilot Feasibility Stud. 2019;5:115. 
https://search.ebscohost.com/login.aspx?direct=true&AuthType=ip,url,uid&db=mdc&AN=31624637&sit
e=ehost-live&scope=site. doi: 10.1186/s40814-019-0503-9.
63. Milner A, Spittal MJ, Kapur N, Witt K, Pirkis J, Carter G. Mechanisms of brief contact interventions in 
clinical populations: A systematic review. BMC psychiatry. 2016;16(1):194. 
https://search.datacite.org/works/10.1186/s12888-016-0896-4. doi: 10.1186/s12888-016-0896-4.
64. Moeller RW, Seehuus M. Loneliness as a mediator for college students' social skills and experiences 
of depression and anxiety. Journal of Adolescence. 2019;73:1-13. 
http://dx.doi.org/10.1016/j.adolescence.2019.03.006. doi: 10.1016/j.adolescence.2019.03.006.

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 62 of 72
65. Cacioppo, John T, Hughes, Mary Elizabeth, Waite, Linda J, Hawkley, Louise C, Thisted, Ronald A. 
Loneliness as a specific risk factor for depressive symptoms: Cross-sectional and longitudinal analyses. 
Psychol Aging. 2006;21(1):140-51.
66. Yanguas J, Pinazo-Henandis S, Tarazona-Santabalbina FJ. The complexity of loneliness. Acta bio-
medica : Atenei Parmensis. 2018;89(2):302-314. https://www.ncbi.nlm.nih.gov/pubmed/29957768. doi: 
10.23750/abm.v89i2.7404.
67. Wang J, Lloyd-Evans B, Giacco D, et al. Social isolation in mental health: A conceptual and 
methodological review. Soc Psychiatry Psychiatr Epidemiol. 2017;52(12):1451-1461. 
https://search.datacite.org/works/10.1007/s00127-017-1446-1. doi: 10.1007/s00127-017-1446-1.
68. Experts call for COVID-19 response to include mental health research. Psychiatry & Behavioral Health 
Learning Network Web site. https://www.psychcongress.com/article/experts-call-covid-19-response-
include-mental-health-research. Accessed Jul 10, 2020.
69. Zhang CH, Schwartz GG. Spatial disparities in coronavirus incidence and mortality in the united 
states: An ecological analysis as of may 2020. The Journal of Rural Health. 2020;36(3):433-445. 
https://onlinelibrary.wiley.com/doi/abs/10.1111/jrh.12476. Accessed Jul 8, 2020. doi: 
10.1111/jrh.12476.
70. Gerst-Emerson K, Jayawardhana J. Loneliness as a public health issue: The impact of loneliness on 
health care utilization among older adults. American journal of public health (1971). 2015;105(5):1013-
1019. https://search.datacite.org/works/10.2105/ajph.2014.302427. doi: 10.2105/ajph.2014.302427.

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 63 of 72
71. Armitage R, Nellums LB. COVID-19 and the consequences of isolating the elderly. The Lancet. Public 
health. 2020;5(5):e256. https://search.datacite.org/works/10.1016/s2468-2667(20)30061-x. doi: 
10.1016/s2468-2667(20)30061-x.
72. Santini ZI, Jose PE, York Cornwell E, et al. Social disconnectedness, perceived isolation, and 
symptoms of depression and anxiety among older americans (NSHAP): A longitudinal mediation analysis. 
The Lancet. Public health. 2020;5(1):e62-e70. https://search.datacite.org/works/10.1016/s2468-
2667(19)30230-0. doi: 10.1016/s2468-2667(19)30230-0.
73. Ivey-Stephenson AZ, Crosby AE, Jack SPD, Haileyesus T, Kresnow-Sedacca M. Suicide trends among 
and within urbanization levels by sex, race/ethnicity, age group, and mechanism of death — united 
states, 2001–2015. Morbidity and mortality weekly report. Surveillance summaries (Washington, D.C. : 
2002). 2017;66(18):1-16. https://www.ncbi.nlm.nih.gov/pubmed/28981481. doi: 
10.15585/mmwr.ss6618a1.
74. Alvarez K, Fillbrunn M, Greif Green J, et al. Race/ethnicity, nativity, and lifetime risk of mental 
disorders in U.S. adults. Soc Psychiatry Psychiatr Epidemiol. 2019;54(5):553-565. 
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6586416/. Accessed Aug 6, 2020. doi: 10.1007/s00127-
018-1644-5.
75. Purtle J. COVID-19 and mental health equity in the united states. Soc Psychiatry Psychiatr Epidemiol. 
2020:1-3. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7298157/. Accessed Aug 6, 2020. doi: 
10.1007/s00127-020-01896-8.

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 64 of 72
76. Cook E. Federal office of rural health policy releases updated list of rural counties and census tracts. 
Mondaq Business Briefing. Jan 8, 2014.
77. Ramírez IJ, Lee J. COVID-19 emergence and social and health determinants in colorado: A rapid 
spatial analysis. International journal of environmental research and public health. 2020;17(11):3856. 
https://search.proquest.com/docview/2408890317. doi: 10.3390/ijerph17113856.
78. Centers for Disease Control and Prevention. CDC health disparities and inequalities report. Morbidity 
and Mortality Weekly Report (MMWR). 2013;62(3). https://www.cdc.gov/mmwr/pdf/other/su6203.pdf.
79. Panchal, Nirmita, Kamal, Rabah, Orgera, Kendal, Cox, Cynthia, Garfield, Rachel, Hamel, Liz, Muñana, 
Cailey, Chidambaram, Priya. The implications of COVID-19 for mental health and substance use. Kaiser 
Family Foundation. 2020(Coronavirus Issue Brief). https://www.kff.org/coronavirus-covid-19/issue-
brief/the-implications-of-covid-19-for-mental-health-and-substance-use/.
80. Centers for Disease Control & Prevention (CDC). Considerations for food pantries and food 
distribution sites. Coronavirus Disease 2019 (COVID-19). 2020. https://www.cdc.gov/coronavirus/2019-
ncov/community/organizations/food-pantries.html.
81. Weir K. The exercise effect. American Psychological Association Monitor on Psychology. 2011;42(11). 
https://www.apa.org/monitor/2011/12/exercise.
82. Taylor S, Landry CA, Rachor GS, Paluszek MM, Asmundson GJG. Fear and avoidance of healthcare 
workers: An important, under-recognized form of stigmatization during the COVID-19 pandemic. J 

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 65 of 72
Anxiety Disord. 2020;75:102289. 
http://www.sciencedirect.com/science/article/pii/S0887618520301031. doi: 
https://doi.org/10.1016/j.janxdis.2020.102289.
83. Tsai J, Wilson M. COVID-19: A potential public health problem for homeless populations. Lancet 
Public Health. 2020;5(4):e186-e187. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7104053/. 
Accessed Aug 6, 2020. doi: 10.1016/S2468-2667(20)30053-0.
84. Zhao X, Zhang D, Wu M, et al. Loneliness and depression symptoms among the elderly in nursing 
homes: A moderated mediation model of resilience and social support. Psychiatry Res. 2018;268:143-
151. https://pubmed.ncbi.nlm.nih.gov/30025285/. Accessed Aug 7, 2020. doi: 
10.1016/j.psychres.2018.07.011.
85. Humphreys KL, Myint MT, Zeanah CH. Increased risk for family violence during the COVID-19 
pandemic. Pediatrics. 2020;146(1). https://pubmed.ncbi.nlm.nih.gov/32317306/. Accessed Aug 7, 2020. 
doi: 10.1542/peds.2020-0982.
86. Evans ML, Lindauer M, Farrell ME. A pandemic within a pandemic — intimate partner violence 
during covid-19. The New England journal of medicine. 2020. 
https://search.proquest.com/docview/2443880657. doi: 10.1056/NEJMp2024046.
87. Salari N, Hosseinian-Far A, Jalali R, et al. Prevalence of stress, anxiety, depression among the general 
population during the COVID-19 pandemic: A systematic review and meta-analysis. Global Health. 
2020;16(1):57-w. doi: 10.1186/s12992-020-00589-w [doi].

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 66 of 72
88. Wu P, Fang Y, Guan Z, et al. The psychological impact of the SARS epidemic on hospital employees in 
china: Exposure, risk perception, and altruistic acceptance of risk. Can J Psychiatry. 2009;54(5):302-311. 
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3780353/. Accessed Jul 17, 2020.
89. Kang L, Ma S, Chen M, et al. Impact on mental health and perceptions of psychological care among 
medical and nursing staff in wuhan during the 2019 novel coronavirus disease outbreak: A cross-
sectional study. Brain, Behavior, and Immunity. 2020;87:11-17. 
http://www.sciencedirect.com/science/article/pii/S0889159120303482. Accessed Jul 17, 2020. doi: 
10.1016/j.bbi.2020.03.028.
90. Killgore WDS, Cloonan SA, Taylor EC, Dailey NS. Loneliness: A signature mental health concern in the 
era of COVID-19. Psychiatry Res. 2020;290:113117. 
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7255345/. Accessed Aug 7, 2020. doi: 
10.1016/j.psychres.2020.113117.
91. Brown R, Scholl J. With coronavirus cases climbing, little keeps idaho in stage 4 of reopening plan. 
Idaho Statesman. Jun 25, 2020. Available from: 
https://www.idahostatesman.com/news/coronavirus/article243768767.html. Accessed Aug 7, 2020.
92. Cohen R. 1 in 4 people tested in A blaine county study had antibodies to COVID-19. Boise State 
Public Radio Web site. https://www.boisestatepublicradio.org/post/1-4-people-tested-blaine-county-
study-had-antibodies-covid-19#stream/0. Updated 2020. Accessed Jul 7, 2020.

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 67 of 72
93. Bruin WBd, Bennett D. Relationships between initial COVID-19 risk perceptions and protective health 
behaviors: A national survey. American Journal of Preventive Medicine. 2020;59(2):157-167. 
https://www.ajpmonline.org/article/S0749-3797(20)30213-0/abstract. Accessed Aug 6, 2020. doi: 
10.1016/j.amepre.2020.05.001.
94. Ma Q, Shan H, Zhang H, Li G, Yang R, Chen J. Potential utilities of mask-wearing and instant hand 
hygiene for fighting SARS-CoV-2. J Med Virol. 2020. https://pubmed.ncbi.nlm.nih.gov/32232986/. 
Accessed Aug 7, 2020. doi: 10.1002/jmv.25805.
95. Konda A, Prakash A, Moss GA, Schmoldt M, Grant GD, Guha S. Aerosol filtration efficiency of 
common fabrics used in respiratory cloth masks. ACS Nano. 2020;14(5):6339-6347. 
https://pubmed.ncbi.nlm.nih.gov/32329337/. Accessed Aug 7, 2020. doi: 10.1021/acsnano.0c03252.
96. Leung NHL, Chu DKW, Shiu EYC, et al. Respiratory virus shedding in exhaled breath and efficacy of 
face masks. Nat Med. 2020;26(5):676-680. https://pubmed.ncbi.nlm.nih.gov/32371934/. Accessed Aug 
7, 2020. doi: 10.1038/s41591-020-0843-2.
97. Dutton A. Idaho health care leaders call for mask mandates to slow COVID-19 wave. Idaho 
Statesman. Jul 14, 2020. Available from: 
https://www.idahostatesman.com/news/coronavirus/article244215412.html. Accessed Aug 7, 2020.
98. Baker M. A ‘Liberty’ rebellion in idaho threatens to undermine coronavirus orders. The New York 
Times. Apr 7, 2020. Available from: https://www.nytimes.com/2020/04/07/us/coronavirus-idaho-
bundy-patriot.html. Accessed Aug 7, 2020.

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 68 of 72
99. Health Measures. Scoring instructions for NIH toolbox® emotion measures:
Raw score to T-score conversion tables. . 2020.
100. Posner K, Brent D, Lucas C, et al. Columbia-suicide severity rating scale (C-SSRS)
baseline/screening version. . 2009.
101. Posner K, Brent D, Lucas C, et al. Columbia-suicide severity rating scale (C-SSRS)
pediatric - since last contact – communities and healthcare. . 2010.
102. The Columbia Lighthouse Project/Center for Suicide Risk Assessment. The columbia suicide severity
rating scale (C-SSRS)
supporting evidence . . 2018.
103. Nilsson ME, Suryawanshi S, Gassman-Mayer C, Dubrava S, McSorley P, Jiang K. Columbia–Suicide 
severity rating scale
scoring and data analysis guide. . 2013.
104. Ionescu DF, Swee MB, Pavone KJ, et al. Rapid and sustained reductions in current suicidal ideation 
following repeated doses of intravenous ketamine: Secondary analysis of an open-label study. J Clin 
Psychiatry. 2016;77(6):719. Accessed Jul 30, 2020. doi: 10.4088/JCP.15m10056.
105. Lutz J, Fiske A. Perceived burdensomeness in older and younger adults: Evaluation of the 
psychometric properties of the interpersonal needs questionnaire. Journal of Clinical Psychology. 

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 69 of 72
2017;73(9):1179-1195. https://onlinelibrary.wiley.com/doi/abs/10.1002/jclp.22415. Accessed Aug 10, 
2020. doi: 10.1002/jclp.22415.
106. Hill RM, Rey Y, Marin CE, Sharp C, Green KL, Pettit JW. Evaluating the interpersonal needs 
questionnaire: Comparison of the reliability, factor structure, and predictive validity across five versions. 
Suicide and Life-Threatening Behavior. 2015;45(3):302-314. 
https://onlinelibrary.wiley.com/doi/abs/10.1111/sltb.12129. Accessed Aug 10, 2020. doi: 
10.1111/sltb.12129.
107. This survey is about health behavior. It has been developed so you can, tell us what you do that 
may affect your health. The information you, give will be used to improve health education for young 
people like, yourself. 2019 state and local youth risk behavior survey. . 
108. Manea L, Gilbody S, McMillan D. A diagnostic meta-analysis of the patient health questionnaire-9 
(PHQ-9) algorithm scoring method as a screen for depression. Gen Hosp Psychiatry. 2015;37(1):67-75. 
https://pubmed.ncbi.nlm.nih.gov/25439733/. Accessed Aug 11, 2020. doi: 
10.1016/j.genhosppsych.2014.09.009.
109. Richardson LP, McCauley E, Grossman DC, et al. Evaluation of the patient health questionnaire 
(PHQ-9) for detecting major depression among adolescents. Pediatrics. 2010;126(6):1117-1123. 
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3217785/. Accessed Aug 11, 2020. doi: 
10.1542/peds.2010-0852.

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 70 of 72
110. Mossman SA, Luft MJ, Schroeder HK, et al. The generalized anxiety disorder 7-item (GAD-7) scale in 
adolescents with generalized anxiety disorder: Signal detection and validation. Ann Clin Psychiatry. 
2017;29(4):227-234A. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5765270/. Accessed Aug 11, 
2020.
111. Plummer F, Manea L, Trepel D, McMillan D. Screening for anxiety disorders with the GAD-7 and 
GAD-2: A systematic review and diagnostic metaanalysis. Gen Hosp Psychiatry. 2016;39:24-31. 
https://pubmed.ncbi.nlm.nih.gov/26719105/. Accessed Aug 11, 2020. doi: 
10.1016/j.genhosppsych.2015.11.005.
112. Nelson LA. Caring texts: A strength-based, suicide prevention trial in 4 native communities. . 2016.
113. RUBIN DB. Inference and missing data. Biometrika. 1976;63(3):581-592. 
https://search.datacite.org/works/10.1093/biomet/63.3.581. doi: 10.1093/biomet/63.3.581.
114. Efron B. Missing data, imputation, and the bootstrap. Journal of the American Statistical 
Association. 1994;89(426):463-475. 
http://www.tandfonline.com/doi/abs/10.1080/01621459.1994.10476768. doi: 
10.1080/01621459.1994.10476768.
115. Lee KJ, Carlin JB. Multiple imputation for missing data: Fully conditional specification versus 
multivariate normal imputation. American journal of epidemiology. 2010;171(5):624-632. 
https://search.datacite.org/works/10.1093/aje/kwp425. doi: 10.1093/aje/kwp425.

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 71 of 72
116. Resche-Rigon M, White IR. Multiple imputation by chained equations for systematically and 
sporadically missing multilevel data. Statistical methods in medical research. 2016;27(6):1634-1649. 
https://search.datacite.org/works/10.1177/0962280216666564. doi: 10.1177/0962280216666564.
117. Mumford E, Dayal M, Joseph H. PCORI research prioritization topic brief 7: Health system 
approaches to suicide prevention. University of Chicago. 2014.

MHAPPS Trial Protocol
SLHS Applied Research
December 30,2020
 
Page 72 of 72
APPENDIX B: C-SSRS Screening Questions & Scoring Criteria
Question #
Domain
Question
Score
1.
Ideation: Wish to be dead
Have you ever wished you were dead or 
wished you could go to sleep and not 
wake up?
1 
2.
Ideation: Suicidal thoughts
Have you had any actual thoughts of 
killing yourself?
2
3.
Ideation: Suicidal thoughts with 
method (without specific plan or 
intent to act)
Have you been thinking about how you 
might do this?
3
4.
Ideation: Suicidal intent 
(without specific plan)
Have you had these thoughts and had 
some intention of acting on them?
4
5.
Ideation: Suicide intent with 
specific plan
Have you started to work out or worked 
out the details of how to kill yourself?  Do 
you intend to carry out this plan?
5
6A. 
[Lifetime-
Recent]
Suicidal Behavior (lifetime)
Have you ever done anything, started to 
do anything, or prepared to do anything 
to end your life?
[Not 
scored at 
baseline]
6B. 
[Lifetime-
Recent]
Suicidal Behavior (3 mos)
If yes to 6A, Was this within the past 3 
months?
6 
6.
[Since last 
contact]
Suicidal Behavior (since last 
contact)
Have you done anything, started to do 
anything, or prepared to do anything to 
end your life?
6