An Evaluation of Virtual Behavioral Health Integration Into Primary Care
Phase: NA
Conditions: Depression, Substance Use Disorder, Anxiety
Sponsor: Wake Forest University Health Sciences

Version Number: 1.4 
Version Date: 07-25-19 
  
1 
Protocol#: 13957 
 
TITLE: AN EVALUATION OF VIRTUAL BEHAVIORAL HEALTH INTEGRATON INTO 
PRIMARY CARE 
 
Coordinating Site: 
Carolinas Medical Center 
1000 Blythe Boulevard 
Charlotte, NC 28203 
 
Clinical Principal Investigator: 
Manuel Castro, MD  
1000 Blythe Boulevard 
Charlotte, NC 28203 
Phone: (704) 355-5375 
Email: Manuel.Castro@AtriumHealth.org 
 
Co-Principal Investigator: 
Jason Roberge, PhD, MPH 
1540 Garden Terrace, Suite 410 
Charlotte, NC 28203 
Phone: (704) 355-0268 
Email: Jason.Roberge@AtriumHealth.org 
  
 
 
 
______________________________________________________________________________ 
The study will be conducted in compliance with the protocol, ICH-GCP and any applicable 
regulatory requirements. 
Confidential 
The information provided in this document is strictly confidential and is intended solely for the 
guidance of the clinical investigation. Reproduction or disclosure of this document - whether in 
part or in full - to parties not associated with the clinical investigation, or its use for any other 
purpose, without the prior written consent of the Sponsor-Investigator is not permitted. 
Throughout this document, symbols indicating proprietary names (, TM) are not displayed. 
Hence, the appearance of product names without these symbols does not imply that these names 
are not protected. 
 
 
 

Version Number: 1.4 
Version Date: 07-25-19 
  
2 
 
Protocol Approved By: 
 
 
CLINICAL PRINCIPAL INVESTIGATOR: 
 
EXECUTIVE SPONSOR: 
 
 
_____________________________________ 
 
____________________________________ 
Manuel Castro, MD  
 
 
 
 
Ruth Krystopolski, MBA 
 
 
_____________________________________ 
 
_____________________________________ 
Date  
 
 
 
 
 
 
Date 
 
 
 
 
 
 
 
 
 
 
 
 
CO-PRINCIPAL INVESTIGATOR: 
 
 
 
 
 
 
 
 
 
_____________________________________ 
 
 
Jason Roberge, PhD, MPH  
 
 
 
 
 
 
 
_____________________________________ 
 
 
Date  
 
 
 
 
 
 
 
 
 
 

Version Number: 1.4 
Version Date: 07-25-19 
  
3 
 
PROTOCOL SUMMARY 
Study Title 
An evaluation of Virtual Behavioral Health Integration (VBHI) into Primary Care 
Study 
Design 
A pragmatic randomized trial evaluating total cost of care 
Study 
Objectives 
The primary objective is to evaluate the effectiveness of the virtual behavioral health 
integration (VBHI) program compared to usual care, on reducing the total cost of 
care reimbursed from Medicare and value-based contracts within 90 days of a 
patient’s primary care visit.  
 
Secondary objectives of the evaluation include: 
• Compare demographics, Charlson comorbidity index, and social 
determinants of health among patients (hunger vital signs, home, education, 
employment, social and emotional health) between the two arms of the study 
• Compare the 90-day acute care utilization rate after the index primary care 
visits 
• Compare the 90-day primary care utilization rate after the index primary care 
visit 
• Compare the pharmacy related 90-day total cost of care after the index 
primary care visit 
• Stratify and report objectives by index primary care visit diagnoses 
(substance use disorder, depression/dysthymia, anxiety) 
Exploratory analyses among patients who enrolled in the VBHI program. A 
subgroup of patients whose index primary care visit is within 9 months of the study 
start. Assessment at 3, 6, 9, and 12-months 
• Symptom improvement  
o Depression: 50% reduction of PHQ-9 score from initial assessment 
o Depression Remission: Percent of PHQ-9 scores <5 (for 2 
consecutive months upon graduation) 
o Suicidal ideation: Percent of PHQ-9, Question #9=0 or C-SSRS 
negative upon graduation 
o Anxiety: GAD-7, 50% reduction from initial assessment 
• Health Outcomes 
o Reduction in BMI among patients with an initial BMI ≥ 25.0 
o Reduction in HbA1c among diabetic patients with an initial A1c ≥ 9 
o Reduction in LDL cholesterol level among patients with an initial 
LDL ≥ 160 
Inclusion 
Criteria 
Inclusion criteria: 
• 18 years of age or older 
• Primary care office visit at a selected site based on preventative and 
outpatient CPT codes 
• Have a PHQ-9 and an overall score >10 or Question 9 >0 

Version Number: 1.4 
Version Date: 07-25-19 
  
4 
• Medicare or value-based contract associated with a patient in the past year at 
the time of the primary care visit 
Exclusion criteria:  
• A patient will contribute to the analysis once using their initial enrollment to 
either arm of the study 
• No insurance claim for the index primary care visit in the data warehouse 
upon evaluation 
Study 
Procedures 
Patients who present to participating primary care practices with an elevated patient 
health questionnaire (PHQ-9) score or report suicide ideation (Question 9) will be 
treated and followed per the proposed virtual behavioral health integration (VBHI) 
program or usual care. As part of the current care process, all patients with a primary 
care visit are given the PHQ-2 survey. A score ≥2 triggers the PHQ-9 survey. 
 
Of the primary care practices eligible for the VBHI program, half will be randomly 
assigned to either the control or intervention arm. Each day a list of patients with an 
elevated PHQ-9 score from the prior day is generated along with the associated 
insurance plan for the patient. All patients who meet eligibility criteria at sites where 
the VBHI program is offered will be considered exposed to the intervention; 
whereas, patients seen at sites not utilizing the VBHI program will be considered 
exposed to usual care. Patients who are in the intervention arm will be treated by the 
behavioral health care team (licensed clinicians, health coaches, a psychiatric 
pharmacist and psychiatric providers). Virtual visits are conducted either by video 
assessment or telephonically. For the primary and secondary objectives, patients will 
be followed for up to 90 days after their primary care visit. For the exploratory 
analysis, patient with a primary care visit within 9 months of the study start date will 
be followed for up to 12 months after their primary care visit. 
 
The VBHI program is defined by several key components:  
 
• Introduction to the patient follow-up process 
• Psychosocial assessment 
• Track (Residential Movement, Nutrition, Sleep Hygiene, Stress Management, 
Pain Management, Perinatal/Postpartum Mood Disorders, Substance Abuse) 
and provide treatment recommendations 
• Tracking of behavioral health symptoms (Depression, Anxiety, and Suicidal 
ideation) and provide treatment recommendations 
• Placement into an appropriate case management program, if needed 
• Navigation to additional psychiatric or substance use services, if needed  
• Motivational interviewing and brief therapeutic recommendations, if needed 
 
 

Version Number: 1.4 
Version Date: 07-25-19 
  
5 
 
 
 
 
 
LIST OF ABBREVIATIONS 
 
BHI 
Behavioral Health Integration 
CORE 
Center for Outcomes Research and Evaluation 
C-SSRS 
Columbia – Suicide Severity Rating Scale 
EDW 
Enterprise Data Warehouse 
GAD-7 
Generalized Anxiety Disorder Screener 
HbA1c 
Hemoglobin A1c 
NC 
North Carolina 
PHI 
Private Health Information 
PHQ-2 
Patient Health Questionnaire - Screener for Depressive Disorders 
PHQ-9 
Patient Health Questionnaire – Screener for Severity of Depression 
SOP 
Standard Operating Procedures 
VBHI 
Virtual Behavioral Health Integration 
 
 
Statistical 
Analysis 
Analyses will include all patients meeting the inclusion and exclusion criteria. 
Comparisons of the intervention and usual care groups will be made using univariate 
analyses such as the t-test and chi-square test. The primary outcome, the 90-day total 
cost of care, will be compared between the two groups of patients using a 
generalized linear mixed model. Results will be presented with odds ratios and 95% 
confidence intervals. 

Version Number: 1.4 
Version Date: 07-25-19 
  
6 
TABLE OF CONTENTS 
1.1. 
Hypothesis ....................................................................................................................................8 
1.2. 
Primary Objective .......................................................................................................................8 
1.3. 
Secondary Objectives..................................................................................................................8 
1.4. 
Exploratory Analyses..................................................................................................................8 
2. BACKGROUND .....................................................................................................................................9 
3. RATIONALE ..........................................................................................................................................9 
4. SUBJECT AND SITE SELECTION ..................................................................................................11 
4.1. 
Accrual – Primary Care Practices ...........................................................................................11 
4.2. 
Accrual – Subjects.....................................................................................................................11 
4.3. 
Inclusion\Exclusion Criteria ....................................................................................................12 
4.3.1. 
Inclusion Criteria ....................................................................................................................12 
4.3.2. 
Exclusion Criteria ...................................................................................................................12 
4.4. 
Evaluable Population ................................................................................................................12 
5. OVERALL DESIGN ............................................................................................................................13 
5.1. 
Variables ....................................................................................................................................13 
5.1.1. 
Primary Outcome Variable .....................................................................................................13 
5.1.2. 
Rates and Variable Descriptions ............................................................................................13 
5.2. 
Virtual Behavioral Health Integration ....................................................................................14 
5.3. 
Usual Care ..............................................................................................................................15 
5.4. 
Patient Graduation from the VBHI Program ........................................................................16 
5.5. 
Continuation of the Intervention .............................................................................................16 
6. DATA COLLECTION AND REPORTING ......................................................................................16 
6.1. 
Statistical analysis ..................................................................................................................17 
6.2. 
Data Collection Dates ...............................................................................................................18 
8. INTERVENTION PLAN .....................................................................................................................19 
8.1. 
Behavioral Health Professional Assessments .........................................................................19 
8.2. 
Follow-up Assessments .............................................................................................................20 
9. STUDY GOVERNANCE .....................................................................................................................20 
10. SAFETY RISKS AND REPORTING .................................................................................................20 
10.1. Data Quality Assurance ............................................................................................................21 
10.2. Safety Reporting to the IRB .....................................................................................................21 

Version Number: 1.4 
Version Date: 07-25-19 
  
7 
10.3. Data Monitoring by the Sponsor .............................................................................................21 
11. RESEARCH COMPLETION .............................................................................................................22 
12. ETHICAL AND LEGAL ISSUES .......................................................................................................22 
12.1. Ethical and Legal Conduct of the Study .................................................................................22 
12.2. Confidentiality ...........................................................................................................................23 
12.3. Disclosure of Data .....................................................................................................................23 
13. RETENTION OF RECORDS .............................................................................................................23 
14. PUBLICATION POLICY ....................................................................................................................23 
APPENDICES .............................................................................................................................................24 
APPENDIX 1: Patient Navigation .......................................................................................................24 
REFERENCES ............................................................................................................................................25 

Version Number: 1.4 
Version Date: 07-25-19 
  
8 
OBJECTIVES  
1.1. Hypothesis 
 
Depressed patients who visit a primary care practice that offers a virtual behavioral health integration 
(VBHI) program will have lower 90-day total costs of care compared to patients receiving primary care 
at practices without VBHI. 
1.2. Primary Objective 
 
Compare the effectiveness of the VBHI program compared to usual care, on modifying the total cost of 
care reimbursed by Medicare and value-based contracts within 90 days of a patient’s index office visit. 
1.3. Secondary Objectives 
 
For the primary and secondary objectives, patients will be followed up to 90 days. 
 
• Compare demographics, Charlson comorbidity index, and social determinants of health among 
patients (hunger vital signs, home, education, employment, social and emotional health) between 
the two arms of the study 
• Compare the patient-centric 90-day utilization rate after the index primary care visit 
• Compare the 90-day primary care utilization rate after the index primary care visit 
• Compare the pharmacy related 90-day total cost of care after the index primary care visit 
• Stratify and report outcomes by index primary care visit diagnoses (substance use disorder, 
depression/dysthymia, anxiety) 
 
1.4. Exploratory Analyses 
 
For the exploratory analysis, patients who enrolled in the VBHI program in the first 9 months from 
study start will have their symptoms (depression score reduction, depression remission, suicidal 
ideation score reduction, anxiety score reduction) and health outcomes (BMI, LDL, and HbA1c) 
assessed at 3, 6, 9, and 12-months. 
 
• Symptom improvement  
o Depression: Percent of patients with a 50% reduction of PHQ-9 score from initial 
assessment 
o Depression Remission: Percent of patients with a PHQ-9 score <5 (for 2 consecutive 
months upon graduation) 
o Suicidal ideation: Percent of patients with a PHQ-9, Question #9=0 or C-SSRS negative 
upon graduation  
o Anxiety: Percent of patients with a 50% reduction of GAD-7 score among patients with 
an initial score of > 10 
• Health Outcomes 
o Reduction in BMI among patients with an initial BMI ≥ 25.0 
o Reduction in HbA1c among diabetic patients with an initial A1c ≥ 9 
o Reduction in LDL cholesterol level among patients with an initial LDL ≥ 160 

Version Number: 1.4 
Version Date: 07-25-19 
  
9 
2. BACKGROUND 
Mental illness is highly prevalent among adults with reports as high as 20% and a common reason for visits 
to primary care.1 Affective disorders like depression and anxiety generate significant healthcare costs and 
result in social and vocational disability. Almost half of patients who are treated for major and minor 
depression are seen exclusively in primary care, with another 20% to 30% treated in both specialty and 
primary care settings.2-4 
Mental health treatment among primary care practices is typically delivered through short episodes of 
care.5 Primary care providers tend to offer less intensive mental health treatment than that provided by 
psychiatrists.6 The level of involvement of primary care providers to deliver outpatient mental health care 
often depends on: (1) lack of mental health providers in the region;7 (2) willingness to prescribe 
psychotropic medications;5 and (3) knowledge of mental health disorders.8 Clinical implications of a mental 
illness comorbidity include increased risk of suicide, psychiatric hospitalization, increased disability, 
decreased compliance with treatment of medical illness, and markedly increased utilization of medical 
services.1 If untreated, depression and anxiety disorders can contribute to high medical utilization. It has 
been reported that 24% percent of the top 10% (high) healthcare utilizers have a diagnosis of current major 
depression and 22%, from an anxiety disorder.1  
Mental health providers have increasingly targeted their efforts on delivering virtual psychiatric 
consultation in the primary care setting.1 Use of virtual care programs providing mental health services is 
on the rise, often in rural and underserved areas where access to mental health services is not present or 
limited.9 Telepsychiatry efforts are often also implemented to improve rural providers’ capacity for 
managing patients with mental health illnesses.10 Outpatient psychiatric consultation programs have utilized 
clinicians to conduct telephone consultations as an efficient means to provide consultative, supplemental, 
and follow-up care.11 Evidence has shown that telepsychiatric evaluation services can be economical and 
with levels of quality and patient satisfaction similar to those found with face-to-face evaluations.12 Few 
programs, however, have reported on efforts to integrate these multiple services into a comprehensive 
“virtual mental health clinic” that can both provide a wide range of clinical services and support providers 
by means of educational offerings. No studies to date have shown through a rigorous evaluation the effect 
of virtual behavioral health integration on total cost of care that include value-based contracts. The 
proposed virtual behavioral health integration (VBHI) program into the primary care setting is a structured 
mental illness treatment program that includes both behavioral treatment and counseling components aimed 
at reducing cost of care, developing and sustaining adaptive coping strategies, and improving short term 
acute care utilization. 
 
3. RATIONALE 
To enhance the care of patients seen in the primary care setting, Atrium Health has designed a virtual 
behavioral health integration program. A behavioral health care team comprised of licensed clinicians 
(Behavioral Health Professionals), health coaches, a consulting psychiatric pharmacist and psychiatric 
providers follow patients for up to 3 months or more depending on patient need after their primary care 
office visit. This virtual behavioral health integration (VBHI) program is currently available in 36 Atrium 
Health primary care practices providing support to over 200 providers and medical residents. The 
Behavioral health Service line is expanding the program to additional primary care practices. Atrium 

Version Number: 1.4 
Version Date: 07-25-19 
  
10 
Health and the Carolinas Physician Alliance is interested in evaluating the effectiveness of the program on 
reducing the total cost of care. 
 
Patients may be referred to the VBHI program if they are identified as depressed after administration of the 
depression screening tool (PHQ-9) or for other concerns by their primary care provider. The primary care 
provider can request assistance by a real-time in office video assessment or by a telephonic out of office 
assessment. Patients referred by a provider for ‘other concern’ instead of depression are not included in the 
evaluation.  
 
For this study, patients who have an elevated PHQ-9 score or report suicidal/homicidal ideation are eligible 
for analyses and be identified from a behavioral health patient report. Patients meeting eligibility criteria 
will accrue in both the intervention and usual care arms of the study. At intervention sites, the VBHI 
program team will reach out to patients utilizing video assessment or telephonic outreach. Patient enrolled 
in the VBHI program will receive the following core components: Introduction to the patient follow-up 
process, psychosocial assessment, tracking and treatment recommendations for (residential movement, 
nutrition, sleep hygiene, stress management, pain management, perinatal/postpartum mood disorders, 
substance abuse), tracking and treatment recommendations for behavioral health symptoms (depression, 
anxiety, suicidal ideation, etc.), placement into an appropriate case management program if needed, 
navigation to additional psychiatric or substance use services if needed, and motivational interviewing and 
brief therapeutic recommendations if needed. 
 
There may be cases where patients are eligible to enroll in full case management programs based on their 
insurance and other criteria –in these situations the VHBI team will coordinate a seamless care 
transition. For example, if a patient has Medicaid through Cardinal Innovations, a managed care 
organization providing local behavioral health care, and meets certain criteria, the patient would no longer 
require services from VHBI team as they would be duplicative.  
 
For the primary and secondary objectives, a patient’s contact with the VBHI program ends after 90 days 
following the primary care office visit, or if there is a failure to contact, or becomes enrolled with a case-
management program which provides behavioral health services, or declines services, or dies. The 
behavioral health care team will ensure the patient is enrolled in a case management program through 
contact with staff before ending contact with the patient. 
 
This research project is a parallel group, cluster randomized quality improvement evaluation which seeks to 
evaluate the total cost of care among patients in the VBHI program for up to 90 days after their primary 
care visit. The outcomes evaluation of this quality improvement intervention has been designed to integrate 
into the routine care and minimize frontline staff burden by deploying an evaluation in a real-world setting.   
 

Version Number: 1.4 
Version Date: 07-25-19 
  
11 
4. SUBJECT AND SITE SELECTION 
4.1. Accrual – Primary Care Practices 
 
Randomization will occur at the practice level with the primary care practices being identified as an 
intervention or usual care site for the duration of the study. Out of 20 primary care clinics that have 
shown interest in the VBHI program, 10 will be randomized into the intervention arm and 10 into the 
usual care arm. Not all sites in the intervention arm are starting the program immediately. There will be 
a scheduled rollout of 2-3 clinics per month. As intervention sites come on board, a corresponding 
control site will begin contributing to the evaluable population. Twelve practices indicated their 
readiness to incorporate the VBHI program into patient care in 2019 with the remaining indicating 
2020. A practice labeled as low volume had historically averaged 15 or less eligible patients per month. 
A practice labeled as high volume had historical averaged 16 or more eligible patients per month. 
 
Twelve practices were identified by the behavioral health team for a 2019 rollout based on the 
readiness to engage with the VBHI program in 2019. Practices were clustered by patient volume per 
month (low n=8, high n=4). The 8 practices with lower volumes were randomly ordered and assigned 
as either a case or control (1:1). The 4 practices with higher volumes were randomly ordered and 
assigned as either a case or control (1:1). The six primary care practices and corresponding matched 
control practice were then randomly ordered for a 2019 rollout. Each month that a practice starts using 
the VBHI program, patients start accruing from the randomly matched control practice. 
 
Eight practices were identified by the behavioral health team for a 2020 rollout based on the readiness 
to engage with the VBHI program in 2020. These practices were not ready to engage in 2019. Practices 
were clustered by patient volume per month (low n=4, high n=4). The 4 practices with lower volumes 
were randomly ordered and assigned as either a case or control (1:1). The 4 practices with higher 
volumes were randomly ordered and assigned as either a case or control (1:1). The four primary care 
practices and corresponding matched control practice were then randomly ordered for a 2020 rollout. 
Each month that a practice starts using the VBHI program, patients start accruing from the randomly 
matched control practice. 
 
4.2. Accrual – Subjects 
 
Patients 18 years of age or older at primary care practices with a PHQ-9 overall score >10 or Question 
9 >0, and with Medicare or other value-based contract where the VBHI program is offered will be 
considered exposed to the intervention; whereas, patients seen at sites not utilizing the VBHI program 
will be considered exposed to usual care. Patients who are in the intervention arm will be treated by the 
behavioral health care team (licensed clinicians, health coaches, a psychiatric pharmacist and 
psychiatric providers). Virtual visits are conducted either by video assessment or telephonically. As part 
of the current care process, all patients with a primary care visit are given the PHQ-2 survey. A score 
≥2 triggers the PHQ-9 survey. Patients who present to participating primary care practices with an 
elevated patient health questionnaire (PHQ-9) score or report suicide ideation (Question 9) will be 

Version Number: 1.4 
Version Date: 07-25-19 
  
12 
treated and followed per the proposed virtual behavioral health integration (VBHI) program or usual 
care. Each weekday a list of patients is received by the behavioral health care team. The list includes 
patients with an elevated PHQ-9 score from the prior day and the associated insurance plan for the 
patient. The insurance plan is included so that the behavioral health team can target patients with 
Medicare 
 or other value-based contracts that report total costs back to Atrium Health. On Mondays, the team 
receives a list of patients seen between Friday and Sunday. 
 
Patients included in the analysis of this study must meet the inclusion\exclusion criteria and have an 
insurance claim for the index primary care visit when they became eligible in the enterprise data 
warehouse (EDW). Patients will begin accruing based on the time the practice goes live according to 
the role out schedule and will continue for 17 months. The VBHI program is available to any patients in 
the practice but the evaluation is centered on the cohort of patients meeting the inclusion criteria. 
 
4.3. Inclusion\Exclusion Criteria 
4.3.1. Inclusion Criteria 
Eligible patients in the analyses must meet each of the following criteria: 
• 18 years of age or older 
• Primary care office visit at a selected site based on preventative and outpatient CPT codes 
• Have a PHQ-9 and an overall score >10 or Question 9 >0 
• Medicare or value-based contract associated with a patient in the past year 
4.3.2. Exclusion Criteria 
• A patient will contribute to the analysis once using their initial enrollment to either arm 
of the study 
• No insurance claim for the index primary care visit in the data warehouse upon 
evaluation 
4.4. Evaluable Population 
Patients included in the evaluable population for this project, will have their data inform the final 
outcomes assessment. All patients who meet the inclusion and exclusion criteria in both arms will be 
assessed for the primary outcome (intent to treat and by per protocol).  
 
 
 
 

Version Number: 1.4 
Version Date: 07-25-19 
  
13 
5. OVERALL DESIGN 
5.1. Variables 
5.1.1. Primary Outcome Variable 
The primary outcome variable is total cost of care. A claim for the primary care visit is first 
identified in the EDW for a patient that meets the inclusion criteria. The total cost of care is 
defined as the summation of all paid claims <= 90 days after the primary care visit. 
5.1.2. Rates and Variable Descriptions 
a. 
Demographic characteristics (age, race, gender, insurance plan) 
b. 
Charlson comorbidity index score 
If we determine that we can identify hospital and primary care visits through claims data, we will 
use that as a source over using just Atrium Health encounters for the utilization rates as specified 
below. 
 
c. 
Patient-centric defined 90-day utilization rate. This is a utilization rate among patients with an 
ED, inpatient, or observation encounter to any Atrium Health facility (<= 90 days after the 
primary care visit).  
d. 
The 90-day primary care utilization rate. This is a utilization rate among patients with a 
primary care encounter (<= 90 days after the primary care visit). 
 
e. 
Primary care visit 
i. At least one CPT code for preventative or outpatient visit 
f. Pharmacy related 90-day total cost of care. The pharmacy total cost of care is defined as the 
summation of all pharmacy paid claims within 90 days of the primary care visit among 
patients with pharmacy claim data (<= 90 days after the primary care visit). 
 
Symptom and health outcomes are assessed at 3, 6, 9, and 12-months among patients who 
enrolled in the VBHI program. 
 
g. Asses symptom improvement for depression 
i. Pre- PHQ-9 score from primary care visit 
ii. Post- PHQ-9 score 
h. Primary care visit diagnoses (primary or secondary billed diagnoses) 
i. substance use disorder 
ii. depression/dysthymia 
iii. anxiety 
i. Anxiety 
i. Pre- GAD-7 score from initial enrollment 
ii. Post- GAD-7 score 

Version Number: 1.4 
Version Date: 07-25-19 
  
14 
j. Suicidal ideation 
i. Pre- PHQ-9, Q9 score from primary care visit 
ii. Post- PHQ-9, Q9 score 
k. Remission of depressive symptoms 
i.  PHQ-9 less than 5 in 2 consecutive months 
l. BMI 
i. Pre- BMI level from primary care visit 
ii. Post- BMI level  
m. HbA1c among diabetic patients  
i. Pre- A1c level from primary care visit 
ii. Post- A1c level  
n. LDL cholesterol level  
i. Pre- LDL level from primary care visit 
ii. Post- LDL level 
 
o. Hunger vital signs – reported upon enrollment 
i. Within the past 12 months, we worried whether our food would run out before we 
got the money to buy more? 
ii. Within the past 12 months the food we bought just didn’t last and we didn’t have 
the money to get more? 
p. Home – reported upon enrollment 
i. What is your housing situation today? 
ii. Are you worries about losing your house? 
q. Education – reported upon enrollment 
i. What is the highest level of school you have finished? 
r. Employment – reported upon enrollment 
i. What is your current work situation? 
s. Social and emotional health – reported upon enrollment 
i. How often do you see or talk to people that you care about and feel close to? 
ii. Stress is when someone feels tense, nervous, anxious, or can’t sleep at night 
because their mind is troubled. How stressed are you? 
iii. Do you feel physically and emotionally safe where you currently live? 
iv. In the past year, have you been afraid of your partner or ex-partner? 
v. In the past year, have you spent more than 2 nights in a row in jail, prison, 
detention center, or juvenile correctional facility? 
 
5.2. Virtual Behavioral Health Integration 
The flow of patients from initial contact to their follow-up calls is outlined in Appendix 1. In the 
existing VBHI primary care workflow, all patients are given the PHQ-2 survey. A score ≥2 triggers the 
PHQ-9 survey. The following day the behavioral health team receives a patient report. The report 

Version Number: 1.4 
Version Date: 07-25-19 
  
15 
includes patients seen the prior day at intervention sites that had an elevated PHQ-9 score >10 or 
reported suicide ideation (Question 9 >0). The Monday report includes patients seen Friday-Sunday. 
Patients associated with Medicare or other value-based insurance plan are identified in the report. The 
behavioral health care team will reach out to these patients for enrollment utilizing video assessment or 
telephonic outreach. The patient always has the option not to be followed by the behavioral health care 
team.  
 
The behavioral health care team will complete the initial behavioral health assessment. The initial 
health assessment contains information such as preferred communication method, GAD-7, psychiatry 
provider, collateral resources, recent ED and hospital utilization, substance use disorder, medication 
barriers, appointment barriers, community resources, and crisis planning. 
 
The frequency of follow-up assessments depends on the clinical presentation and medication 
recommendation and/or adherence. The assessment includes information such as a suicide ideation 
safety screening, an appointment reminder, appointment barrier evaluations, medication reconciliation, 
medication obtainment follow-up, substance use disorder follow-up, supportive listening, 
psychoeducation, community resource follow-up, and additional crisis planning. 
 
The Columbia-Suicide Severity Rating Scale (C-SSRS) helps identify whether someone is at risk for 
suicide, and the severity and immediacy of the risk. It also identifies level of support the person needs. 
The screening tool is administered during each phone contact to measure suicide ideation. Follow-up 
patients enrolled in the VBHI program will receive follow-up phone calls. The frequency of calls is 
dependent on the clinical presentation and medication recommendation and/or adherence. A follow-up 
assessment is completed upon each phone call. 
 
5.3. Usual Care 
All patients are given the PHQ-2 survey during the primary care visit. A score ≥ 2 triggers the PHQ-9 
survey. The PHQ-9 offers providers a concise, self-administrated screening and diagnostic tool for 
Major Depressive Disorder (Table 1). Treatment can be based on the patients total PHQ-9 score. The 
patient’s provider in the primary care practice has the option to refer the patient to community based 
behavioral health services, begin medication management within the primary care setting, or encourage 
the patient to obtain a psychiatric evaluation at the nearest emergency department to assess safety 
concerns including both suicidal and homicidal ideation. For patients that have a diagnosis of 
depression or dysthymia, the PHQ-9 is administered every 4 months. There are no follow-up behavioral 
health services based on the PHQ-9 score once the patient leaves the primary care practice. 
PHQ-9 Score 
Provisional Diagnosis 
Treatment Recommendation 
0-4 
No Depression 
None 

Version Number: 1.4 
Version Date: 07-25-19 
  
16 
 
Table 
1. 
PHQ-
9 
scores and standard treatment recommendations 
 
5.4. Patient Graduation from the VBHI Program 
• Reached max benefit as deemed by the behavioral health care team 
• Depression remission – PHQ-9 <5 for 2 consecutive months 
• Unable to connect: 4 calls attempted over 12 days 
• Transition to lower level of care 
o Patients who transition to a primary care clinic not utilizing VBHI services 
o Moved out-of-state 
o Discharged from primary care clinic utilizing VBHI services 
• Transition to higher level of care 
o Psychiatry/Treatment Program 
• Patient states that they are no longer interested in services but their last PHQ-9 >5 
• Declined or no longer interested in services 
• Death 
 
5.5. Continuation of the Intervention 
Patients from participating primary care sites may continue enrolling in the VBHI program after the 
study period has ended.  
 
6. DATA COLLECTION AND REPORTING 
Patient demographics, Charlson comorbidity score, and utilization will be obtained from the electronic 
medical record and billing systems. Data from the initial assessment and follow-up contacts will be stored 
in HealtheCare, a care management platform in Cerner. Data will be retrieved by an application specialist 
on the research team. Data may be retrieved retrospective to a patient’s completion of the intervention or 
usual care arm of the study. A participation rate can be generated from the count of patients enrolled in the 
VBHI program over the count of patients meeting the inclusion criteria. 
 
5-9 
Major Depression, mild 
Reassurance and/or supportive counseling 
Education to call if deteriorates 
10-14 
Major Depression, 
moderate 
Antidepressant and/or Psychotherapy 
15-19 
Major Depression, 
moderately severe 
Antidepressant and/or Psychotherapy 
≥20 
Major Depression, severe 
Antidepressant and Psychotherapy 

Version Number: 1.4 
Version Date: 07-25-19 
  
17 
A bi-monthly report will be produced for the intervention arm. The report will be by primary care practice 
and overall showing the prior 2 weeks of activity. 
• Count of patients meeting the inclusion criteria 
• Count of patients with a HealtheCare account 
• Count of patients enrolled in the VBHI program 
• Patient list with their enrollment status in HealtheCare 
A monthly report will be produced for the intervention arm. The report will be by primary care practice and 
overall. Report to be generated 2 weeks after the end of the month showing the prior month of activity. 
• Count of patients meeting the inclusion criteria 
• Count of patients with a HealtheCare account 
• Count of patients enrolled in the VBHI program 
• Patient list with their enrollment status in HealtheCare 
 
6.1. Power Analysis 
We are working with our data management group to develop the 90-day total cost of care measure. 
With this version of the protocol we can only estimate cost savings. In the literature, studies on 
behavioral health integration tend to evaluate symptom improvement or utilization. Blue Cross Blue 
Shield of Kansas City data were used to conduct a case control program evaluation in 2018. This study 
assessed the cost effectiveness of embedding a Behavioral Health Clinician into an Existing Primary 
Care Practice. Though not virtual, we would assume a similar direction in cost saving reduction in our 
program. The results demonstrated that integrating behavioral health services into the practice was 
associated with $860.16 per member per year savings or 10.8% savings in costs.13 
 
6.2. Statistical analysis 
Prior to analysis, descriptive statistics will be calculated to screen all variables for accuracy and to 
examine distributions.   
 
All analyses will follow the intention-to-treat (“ITT”) principle, such that patients will be analyzed 
based on the assignment of their primary care site to the VBHI program or usual care.  Balance in 
baseline characteristics between the groups will be assessed using Student’s t-test for normally 
distributed data, the Wilcoxon rank sum test for ordinal data or interval data that are not normally 
distributed, and the chi-square test for categorical data. Normality will be assessed using the Shapiro 
Wilk test and Q-Q plots.  
 
The primary outcome, total cost of care, will be compared between the VBHI and usual care groups 
using a linear mixed-effects model with a random effect to account for within-practice correlation. The 
primary predictor in the model is an indicator for assignment to the VBHI program versus usual care. 
Additional baseline covariates will be included in the model if they are either strongly correlated with 
the dependent variable or show baseline imbalance between the two groups. 

Version Number: 1.4 
Version Date: 07-25-19 
  
18 
 
The secondary outcome of pharmacy-related total cost of care will be compared between treatment 
groups using the same approach as for the primary outcome. A similar approach will be used to 
compare 90-day patient-centric, primary care, and avoidable care utilization rates, except the models 
will be generalized linear mixed models with a logit link, where utilization (yes/no) is the binary 
dependent variable. 
 
Sub-analyses of the primary and secondary outcomes using the above methods will be performed on the 
data stratified by index primary care visit diagnosis. 
 
In addition, per-protocol analyses of the primary and secondary outcomes will be performed to evaluate 
the effect of VBHI program participation, in contrast to the ITT analysis evaluating the effectiveness of 
referral to the program. The per-protocol analyses will include only subjects who received and adhered 
to the treatment (VBHI program or usual care) assigned to their clinic. The analyses will be similar to 
those described above for ITT, except the primary predictor is an indicator for treatment (VBHI 
program or usual care) received rather than treatment assigned. To help address confounding due to 
incomplete adherence, the per-protocol analyses will adjust for pre-randomization and post-
randomization factors that predict adherence. To identify such factors, a separate, multivariable logistic 
regression analysis will be performed on the full dataset with adherence (yes/no) as the dependent 
variable. 
 
Exploratory analyses will evaluate baseline to 3-, 6-, 9-, and 12-month changes in symptom 
improvement and health outcomes using generalized linear mixed models with a logit link, where the 
dependent variable is binary (e.g., 50% reduction of PHQ-9 score from initial assessment (yes/no)). 
Random effect terms in the model will account for within-practice clustering and within-patient 
correlation. Fixed effects will be the treatment group indicator, time indicator, and interaction between 
treatment group and time. 
 
All tests will be two-sided and the data will be analyzed using SAS Enterprise Guide 7.1 (SAS Institute 
Inc., Cary, NC, USA). 
 
 
6.3. Data Collection Dates 
The implementation of the VBHI program will begin in August 2019. 
 
 
 
 
 
 
 
 
 
 
 

Version Number: 1.4 
Version Date: 07-25-19 
  
19 
7. PROJECT TIMELINE 
 
2019  
  
   
   
   
   
   
   
 2020  
   
   
2021   
   
 
   
May  June  July  Aug  Sept  Oct  Nov  Dec  
Jan  
Feb  Dec  
Jan  
Feb  Mar  
Apr-
June  July  Aug  
Identify Key personnel 
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
 
 
Draft Charter  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
 
 
Draft Protocol  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
 
 
Initial IRB Submission  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
 
 
Registration with Clinical 
Trials.gov  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
First Patient on Study  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
 
 
Enrollment Duration 
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
 
 
Last Patient on Study  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
 
 
Last Patient Follow-up  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
 
 
Waiting on claims/ work 
on manuscript  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
 
 
Analysis/ Report  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
 
 
 
 
 
8. INTERVENTION PLAN 
8.1. Behavioral Health Professional Assessments 
The initial patient assessment by the behavioral health care team captures components such as: 
• Insurance 
• Primary care provider 
• Outpatient Psychiatry provider 
• Collateral resource 
• Recent visit summary 
• Substance abuse 
• Medication barriers 
• Appointment Barriers 
• Transportation needs 
• Access to medications 
• Community Resources 
• Columbia-Suicide Rating Scale to assess for suicide ideation 

Version Number: 1.4 
Version Date: 07-25-19 
  
20 
8.2. Follow-up Assessments 
The follow-up contact assessments are comprised of components such as: 
• Columbia-Suicide Rating Scale to assess for suicide ideation 
• Appointment reminder\follow-up 
• Medication reconciliation 
• Medication access barriers 
• Substance use disorder follow-up 
• Supportive listening 
• Psychoeducation 
• Utilization of community resources 
• Crisis planning and adherence to the plan by the patient 
9. STUDY GOVERNANCE 
This quality improvement trial will be conducted at Atrium Health. It will be run jointly by the Center for 
Outcomes Research and Evaluation (CORE) and the Behavioral Health Service Line. Manuel Castro, MD, 
(Psychiatry) will serve as the clinical principal investigator with oversight from the Executive Committee (EC). 
Jason Roberge, PhD, MPH will serve as co-principal investigator on behalf of CORE. The EC will consist of 
leaders across the System involved in the trial, quality improvement, and implementation (Table 2). The EC 
will have the overall responsibility of trial oversight and direction. The EC will support dissemination of project 
findings and next steps. The EC will receive progress reports from the team. When appropriate, ad hoc 
committee meetings will be scheduled to discuss pressing concerns. 
 
Table 2. Executive Committee 
James Rachal 
Atrium Health Executive Leadership – Psychiatry 
Ruth Krystopolski 
Atrium Health Executive Leadership – Population Health 
Wayne Sparks 
Atrium Health Executive Leadership – Psychiatry 
Erica Todd 
Atrium Health Executive Leadership – Behavioral Health 
Martha Whitecotton 
Atrium Health Executive Leadership – Behavioral Health 
John Franko 
Atrium Health Executive Leadership – Primary Care 
Andrew McWilliams 
Atrium Health Executive Leadership – CORE Medical Director 
 
10. SAFETY RISKS AND REPORTING 
The data collection and intervention for this project presents no more than minimal risk to patients. The 
VBHI program is currently offered at 36 primary care clinics within Atrium Health. This study is 
evaluating the VBHI program among additional clinics that will offer the program to their patients. 
However, patients who present to the primary care setting with a behavioral health crisis are considered a 
high-risk population. The implementation of the VBHI program and its components complement ongoing 
patient care through virtual patient navigation within Atrium Health. The addition of an evaluation design 
that aligns with existing patient care expansion thus confers minimal additional risk to patients. 

Version Number: 1.4 
Version Date: 07-25-19 
  
21 
Other potential risks of participation in this project include, the risk of health information disclosure. There 
is always the risk of disclosure of a patient’s private health information (PHI) or medical information. 
However, the processes identified in this protocol to enable the execution of this project, do not increase 
inherent risk of disclosure. Atrium Health utilizes several hard and soft safety controls in the protection of 
patient information and medical records. Security controls include, but are not limited to, multiple system 
firewalls, access restrictions to patient records and information, locked offices and buildings housing 
research and patient data, and multiple layers of username and password protected computer and system 
access. Statistical tools and programs that are developed are stored on secure restricted access file shares.  
Programs have versioning controls turned on and access to the file shares are restricted.  Access is granted 
through a request process that requires multiple authorizations and is controlled centrally by our 
Information and Analytics Service Department.  
The CORE project team will ensure that appropriate handling of patient PHI follows standard Atrium 
Health procedure. Confidentiality will be maintained according to ICH E6; 4.8.10, part 0: “Records 
identifying the subject will be kept confidential and, to the extent permitted by the applicable laws and/or 
regulations, will not be made publicly available. If the results of the study trial are published, the patient’s 
identity will remain confidential.’ In the event of PHI disclosure, the appropriate internal departments will 
be informed and processed per legislation and privacy regulations. 
 
10.1. 
Data Quality Assurance 
 
This study will be organized, performed, and reported in compliance with the study protocol, 
standard operating procedures (SOPs) of the CORE and Atrium Health, and other applicable 
regulations and guidelines (e.g. GCP).  
 
10.2. 
Safety Reporting to the IRB 
 
All events occurring during the conduct of a protocol and meeting the definition of a reportable 
safety event per the IRB guidelines, will be reported to the IRB within 10 working days of the 
Investigator learning of the event, per their requirements. 
 
Major protocol deviations that result in a threat to subject safety or the integrity of the study will be 
reported to the IRB per their requirements. 
 
10.3. 
Data Monitoring by the Sponsor 
The conduct of this project will abide by standard operating procedures set forth by both Atrium 
Health and CORE. The Principal Investigators, statistician, and other team members will meet as 
needed to review enrollment and retention, study progress, and validity/integrity of the data. 
Documentation of these meetings will be kept with study records. 

Version Number: 1.4 
Version Date: 07-25-19 
  
22 
11. RESEARCH COMPLETION 
The Principal Investigators have the right to close the project at any site at any time. 
For any closure, the following applies: 
• Closure should occur only after consultation between involved parties. 
• In case of a partial study or site closure, patients still participating in the VBHI program, or those 
who are considered in follow-up, must be taken care of in an ethical manner. 
 
The study will be considered complete when one or more of the following conditions is met: 
• 
The enrollment period has ended, and the data collection period is complete. 
• 
All subjects have dropped out or discontinued from the study after the enrollment period is 
completed, but prior to data collection cutoff as described in section 8.3. 
• 
The IRB or Principal Investigator discontinues the study. 
• 
The Principal Investigator defines an administrative or clinical cut-off date. 
Upon study completion, a final report will be presented to the Executive Committee and all key 
stakeholders. The final report will detail all findings including primary, secondary and post-hoc outcomes. 
The team will also prepare a manuscript for publication focused on outcomes and feasibility of 
implementation of the VBHI program.   
 
12. ETHICAL AND LEGAL ISSUES 
12.1. 
Ethical and Legal Conduct of the Study 
The procedures set out in this protocol, pertaining to the conduct, evaluation, and documentation of this 
study, are designed to ensure that the Investigators abide by Good Clinical Practice (GCP) guidelines 
and under the guiding principles detailed in the Declaration of Helsinki. The study will also be carried 
out in keeping with the applicable local laws and regulation(s). 
 
Documented approval from appropriate agencies (e.g. IRB) will be obtained before the start of the 
study, per GCP, local laws, regulations, and organizations.  
 
Strict adherence to all specifications laid down in this protocol is required for all aspects of study 
conduct; the Investigators may not modify or alter the procedures described in this protocol. 
 
Modifications to the study protocol will not be implemented without consulting the Principal 
Investigator and the IRB, as applicable. The Principal Investigator must assure that all study personnel, 
including co-investigators and other study staff members, adhere to the study protocol and all 
applicable regulations and guidelines regarding research both during and after study completion. 
 

Version Number: 1.4 
Version Date: 07-25-19 
  
23 
The Principal Investigator will be responsible for assuring that all the required data will be collected 
and properly documented. 
12.2. 
Confidentiality 
All records identifying the subject will be kept confidential and, to the extent permitted by the 
applicable laws and/or regulations, will not be made publicly available. 
12.3. 
Disclosure of Data 
The Principal Investigator, his associates and co-workers, and the appropriate regulatory agencies may 
use the information and data included in this protocol as necessary for the conduct of the study. 
Information contained in this study, and data and results from the study are confidential and may not be 
disclosed without the written permission of the Principal Investigator.  
 
 
13. RETENTION OF RECORDS 
Essential documentation including all IRB correspondence, will be retained for at least 2 years after the 
investigation is completed. Documentation will be readily available upon request. 
 
14. PUBLICATION POLICY 
The Principal Investigator or designee must send any draft manuscript, abstract, or conference presentation 
to members of the project Executive Committee for feedback and transparency, prior to submission of the 
final version. The Principal Investigator will be responsible for all relevant aspects regarding data reporting 
and publication.  
 
The Principal Investigator or designee will ensure that the information and results regarding the study will 
be made publicly available on the internet at www.clinicaltrials.gov. 
 
 

Version Number: 1.4 
Version Date: 07-25-19 
  
24 
APPENDICES 
APPENDIX 1: Patient Navigation 
 
This diagram shows the patient contact points among the population. 
 
Primary care visit
Patient at practice 
using the 
VBHI program
Call Back 
Protocol
Follow-up phone 
calls until 
graduation
• Reached max benefit as deemed by the behavioral health care team
• Depression remission – PHQ-9 <5 for 2 consecutive months
• Unable to connect: 4 calls attempted over 12 days
• Transition to lower level of care
• Patient transitions to a primary care clinic not utilizing VBHI services
• Moved out-of-state
• Discharged from primary care clinic utilizing VBHI services
• Transition to higher level of care
• Psychiatry/Treatment Program
• No longer interested in services but last PHQ-9 >5
• Declined services
• Death
Patient completes 
PHQ-2
Score < 2 
Score   
Patient completes 
PHQ-9
No further action
Not enrolled in 
Study
Score < 10 and Q9 = 0
Patient at practice 
using 
usual care
Usual care
No VBHI
Score      or Q9 > 0
Patient meets 
inclusion criteria and 
is on the daily report
 
 
 
 

Version Number: 1.4 
Version Date: 07-25-19 
  
25 
REFERENCES 
 
1. 
Hirschfeld RM. The Comorbidity of Major Depression and Anxiety Disorders: Recognition and 
Management in Primary Care. Prim Care Companion J Clin Psychiatry 2001;3:244-54. 
2. 
Wang PS, Demler O, Olfson M, Pincus HA, Wells KB, Kessler RC. Changing profiles of service sectors 
used for mental health care in the United States. Am J Psychiatry 2006;163:1187-98. 
3. 
Wang PS, Lane M, Olfson M, Pincus HA, Wells KB, Kessler RC. Twelve-month use of mental health 
services in the United States: results from the National Comorbidity Survey Replication. Arch Gen Psychiatry 
2005;62:629-40. 
4. 
Katon W, Robinson P, Von Korff M, et al. A multifaceted intervention to improve treatment of 
depression in primary care. Arch Gen Psychiatry 1996;53:924-32. 
5. 
Olfson M, Kroenke K, Wang S, Blanco C. Trends in office-based mental health care provided by 
psychiatrists and primary care physicians. J Clin Psychiatry 2014;75:247-53. 
6. 
Van Voorhees BW, Cooper LA, Rost KM, et al. Primary care patients with depression are less accepting 
of treatment than those seen by mental health specialists. J Gen Intern Med 2003;18:991-1000. 
7. 
Thomas KC, Ellis AR, Konrad TR, Holzer CE, Morrissey JP. County-level estimates of mental health 
professional shortage in the United States. Psychiatr Serv 2009;60:1323-8. 
8. 
Loeb DF, Bayliss EA, Binswanger IA, Candrian C, deGruy FV. Primary care physician perceptions on 
caring for complex patients with medical and mental illness. J Gen Intern Med 2012;27:945-52. 
9. 
Neufeld JD, Yellowlees PM, Hilty DM, Cobb H, Bourgeois JA. The e-Mental Health Consultation 
Service: providing enhanced primary-care mental health services through telemedicine. Psychosomatics 
2007;48:135-41. 
10. 
Ricci MA, Caputo MP, Callas PW, Gagne M. The use of telemedicine for delivering continuing medical 
education in rural communities. Telemed J E Health 2005;11:124-9. 
11. 
Worth JL, Stern TA. Benefits of an Outpatient Psychiatric TeleConsultation Unit: Results of a 1-Year 
Pilot. Prim Care Companion J Clin Psychiatry 2003;5:80-4. 
12. 
Hyler SE, Gangure DP, Batchelder ST. Can telepsychiatry replace in-person psychiatric assessments? A 
review and meta-analysis of comparison studies. CNS Spectr 2005;10:403-13. 
13. 
Ross KM, Klein B, Ferro K, McQueeney DA, Gernon R, Miller BF. The Cost Effectiveness of 
Embedding a Behavioral Health Clinician into an Existing Primary Care Practice to Facilitate the Integration of 
Care: A Prospective, Case-Control Program Evaluation. J Clin Psychol Med Settings 2019;26:59-67.