An Evaluation of Virtual Psychiatric Transition of Care Offered in Behavioral Health Settings
Phase: NA
Conditions: Mental Health Issue
Sponsor: Wake Forest University Health Sciences

 
 
Protocol#: 18188 
 
TITLE: AN EVALUATION OF VIRTUAL PSYCHIATRIC TRANSITION OF CARE OFFERED IN 
BEHAVIORAL HEALTH SETTINGS 
 
Coordinating Site: 
Carolinas Medical Center 
1000 Blythe Boulevard 
Charlotte, NC 28203 
 
Clinical Principal Investigator: 
Manuel Castro, MD  
1000 Blythe Boulevard 
Charlotte, NC 28203 
Phone: (704) 355-5375 
Email: Manuel.Castro@AtriumHealth.org 
 
Co-Principal Investigator: 
Jason Roberge, PhD, MPH 
1300 Scott Avenue 
Charlotte, NC 28204 
Phone: (704) 898-3855 
Email: Jason.Roberge@AtriumHealth.org 
  
 
 
 
______________________________________________________________________________ 
The study will be conducted in compliance with the protocol, ICH-GCP and any applicable 
regulatory requirements. 
Confidential 
The information provided in this document is strictly confidential and is intended solely for the 
guidance of the clinical investigation. Reproduction or disclosure of this document - whether in 
part or in full - to parties not associated with the clinical investigation, or its use for any other 
purpose, without the prior written consent of the Sponsor-Investigator is not permitted. 
Throughout this document, symbols indicating proprietary names (, TM) are not displayed. 
Hence, the appearance of product names without these symbols does not imply that these names 
are not protected. 
 
 
 

Version Number: 1.1 
Version Date: 10-08-21 
  
2 
Protocol Approved By: 
 
 
CLINICAL PRINCIPAL INVESTIGATOR: 
 
EXECUTIVE SPONSOR: 
 
 
____________________________________ 
Manuel Castro, MD  
 
 
 
 
James Rachal, MD 
 
 
_____________________________________ 
 
_____________________________________ 
Date  
 
 
 
 
 
 
Date 
 
 
 
 
 
 
 
 
 
 
 
 
CO-PRINCIPAL INVESTIGATOR: 
 
 
 
 
 
 
 
 
 
_____________________________________ 
 
 
Jason Roberge, PhD, MPH  
 
 
 
 
 
 
 
_____________________________________ 
 
 
Date  
 
 
 
 
 
 
 
 
 
 

Version Number: 1.1 
Version Date: 10-08-21 
  
3 
 
PROTOCOL SUMMARY 
Study Title 
An Evaluation of Virtual Psychiatric Transition of Care (VPTC) Offered in 
Behavioral Health Settings 
Study 
Design 
A stepped-wedge trial assessing 30-day acute care utilization 
Study 
Objectives 
The primary objective is to evaluate the effectiveness of a post inpatient discharge 
virtual psychiatric care team compared to standard care, to reduce 30-day all cause 
non-elective acute care utilization (ED, observation, and inpatient encounters). 
 
Secondary objectives of the evaluation include: 
• Compare the effectiveness of the VPTC program compared to standard care, 
on 60-day all cause non-elective acute care utilization (ED visits, 
observation, and inpatient encounters). 
• Compare the effectiveness of the VPTC program compared to standard care, 
on 30-day behavioral health acute care utilization (ED visits, observation, 
and inpatient encounters). 
• Examine the percent of patients that had a behavioral health office visit 
within 7 days of discharge to standard of care 
• Examine the count of all cause avoidable ED visits within 30 days of 
discharge 
Exploratory analyses: 
• Count of patients: Identified, Refused, Enrolled, Graduated, Did not graduate 
and reason 
• Characterize: demographics and Charlson comorbidity score 
• Assess symptom improvement  
o Suicidal ideation: 
▪ Percent of patients with a PHQ-9, Question 9=0 upon 
graduation among patients with an initial Q9>0 
o Depression reduction:  
▪ Percent of patients with a 5 point reduction of their PHQ-9 
score who had an initial PHQ-9 score ≥ 10 
▪ Show the distribution of PHQ-9 score reduction among 
participants 
o Depression remission: Percent of patients with a PHQ-9 score <5 
upon graduation who had an initial PHQ-9 score ≥ 5 
Inclusion 
Criteria 
Inclusion criteria: 
• Residency or shelter in North Carolina 
• 18 years of age or older on the day of discharge 

Version Number: 1.1 
Version Date: 10-08-21 
  
4 
• Access virtual care (phone/video) 
• Discharged from a behavioral health hospital or a behavioral health unit 
Exclusion criteria: None 
Study 
Procedures 
The VPTC program has a focus on removing obstacles and social determinants that 
often impede patients from engaging in post hospitalization care. This program 
would support vulnerable patients, engaging them in virtual services to assist their 
transition to appropriate and patient centered outpatient care.  
 
This study is comprised of a pilot and a pragmatic trial.  
 
For the pilot, the Center for Outcomes Research and Evaluation (CORE) team will 
lead a qualitative evaluation of the psychiatric virtual transition of care (VPTC) 
program’s implementation. The study team will gather first-hand accounts from 
patients and clinicians regarding their program experiences and program utilization. 
Changes to the program can be made as needed to optimize enrollment and 
implementation. The qualitative evaluation will assess the strengths and weaknesses 
of the VPTC program so modifications can be implemented. The pilot location will 
not be a part of the trial. 
 
For the trial, the VPTC program will be compared to standard care through a cluster 
randomized by site, stepped-wedge design. Behavioral Health facility or units that 
have discharged patients meeting inclusion criteria will be randomly assigned as 
either an intervention or standard care site. Intervention sites will be those using the 
VPTC program. This evaluation design will integrate into current workflows for the 
measurement of outcomes. Prior to discharge, patients may be referred to the VPTC 
program if they report residency or shelter in North Carolina, are 18 years of age or 
older on the day of discharge and can access virtual care (phone/video). 
The VPTC program is defined by several key components:  
• Introduction to the patient follow-up process 
• Psychiatric consultation 
• Medication management services with pharmacological consultation 
• following of Health Coaching participation (Movement, Nutrition, Sleep 
Hygiene, Stress Management, Pain Management, Perinatal/Postpartum 
Mood Disorders, Substance abuse) and provide treatment 
recommendations 
• Tracking of behavioral health symptoms (Depression, Anxiety, Suicidal 
ideation) and provide treatment recommendations 
• Health coaching; Psychoeducational support 
• Treatment and safety planning 
• Appointment reminders 
• Coordination for financial assistance 
• Assistance connecting with an outpatient care provider 
• Placement into an appropriate case management program, if needed 

Version Number: 1.1 
Version Date: 10-08-21 
  
5 
 
 
 
 
 
LIST OF ABBREVIATIONS 
 
BHI 
Behavioral Health Integration 
CORE 
Center for Outcomes Research and Evaluation 
C-SSRS 
Columbia – Suicide Severity Rating Scale 
EDW 
Enterprise Data Warehouse 
VPTC 
Virtual Psychiatric Transition of Care 
NC 
North Carolina 
PHI 
Private Health Information 
PHQ-2 
Patient Health Questionnaire - Screener for Depressive Disorders 
PHQ-9 
Patient Health Questionnaire – Screener for Severity of Depression 
SOP 
Standard Operating Procedures 
 
 
• Navigation to additional psychiatric or substance use services, if needed 
• Motivational interviewing and brief therapeutic recommendations, if 
needed 
Statistical 
Analysis 
Analyses will include all patients identified as having been discharged and meeting 
the inclusion criteria. Baseline comparisons of the intervention and usual care groups 
will be made using univariate analyses. The primary analysis will be conducted 
using generalized linear mixed models with fixed effects for intervention, and 
intervention multiply by the time, accounting for the cluster randomized trial design 
with a random cluster (hospitals) or time (steps) effect. Results will be presented 
with odds ratios and 95% confidence intervals. 

Version Number: 1.1 
Version Date: 10-08-21 
  
6 
TABLE OF CONTENTS 
1.1. 
Hypothesis............................................................................................................................... 8 
1.2. 
Primary Objective .................................................................................................................. 8 
1.3. 
Secondary Objectives ............................................................................................................. 8 
1.4. 
Exploratory Analyses ............................................................................................................. 8 
2. BACKGROUND ................................................................................................................................ 9 
3. RATIONALE ..................................................................................................................................... 9 
4. SUBJECT AND SITE SELECTION .............................................................................................. 10 
4.1. 
Accrual .................................................................................................................................. 10 
4.2. 
Accrual – Subjects ................................................................................................................ 10 
4.3. 
Participating sites ................................................................................................................. 10 
4.4. 
Inclusion\Exclusion Criteria ................................................................................................ 11 
4.4.1. 
Inclusion Criteria ............................................................................................................... 11 
4.4.2. 
Exclusion Criteria .............................................................................................................. 11 
4.5. 
Evaluable Population ........................................................................................................... 11 
5. OVERALL DESIGN........................................................................................................................ 11 
5.1. 
Variables ............................................................................................................................... 11 
5.1.1. 
Primary Outcome Variable ................................................................................................. 11 
5.1.2. 
Secondary Outcome Variables ........................................................................................... 11 
5.1.3. 
Exploratory Variables ........................................................................................................ 12 
5.2. 
Trial Design .......................................................................................................................... 12 
5.2.1. 
Justification for stepped wedge cluster randomized control trial ......................................... 12 
5.2.2. 
Virtual Psychiatric Transition of Care Program .................................................................. 13 
5.2.2.1. Zero Suicide Pathway ........................................................................................................ 13 
5.2.2.2. VPTC Pathway .................................................................................................................. 13 
5.3. 
Patient Graduation from the VPTC Program .................................................................... 14 
5.4. 
Continuation of the Intervention ......................................................................................... 14 
6. QUALITATIVE REVIEW .............................................................................................................. 14 
7. DATA COLLECTION AND REPORTING ................................................................................... 15 
7.1. 
Sample Size Analysis ......................................................................................................... 16 
7.2. 
Statistical analysis .............................................................................................................. 16 
7.3. 
Missing Data ...................................................................................................................... 17 
7.4. 
Training ............................................................................................................................. 17 
7.5. 
Data Collection Dates ........................................................................................................... 17 

Version Number: 1.1 
Version Date: 10-08-21 
  
7 
8. INTERVENTION PLAN ................................................................................................................. 18 
8.1. 
Behavioral Health Professional Assessments ...................................................................... 18 
8.2. 
Follow-up Assessments ...................................................................................................... 18 
8.3. 
Psychiatric Virtual Appointments ....................................................................................... 18 
9. STUDY GOVERNANCE ................................................................................................................ 19 
10. SAFETY RISKS AND REPORTING ............................................................................................. 19 
10.1. 
Data Quality Assurance ...................................................................................................... 20 
10.2. 
Safety Reporting to the IRB ............................................................................................... 20 
10.3. 
Data Monitoring by the Sponsor ......................................................................................... 20 
11. RESEARCH COMPLETION ......................................................................................................... 20 
12. ETHICAL AND LEGAL ISSUES................................................................................................... 21 
12.1. Ethical and Legal Conduct of the Study .............................................................................. 21 
12.2. Confidentiality ...................................................................................................................... 21 
12.3. Disclosure of Data................................................................................................................. 21 
13. RETENTION OF RECORDS ......................................................................................................... 22 
14. PUBLICATION POLICY ............................................................................................................... 22 
15. REFERENCES ................................................................................................................................ 22 

Version Number: 1.1 
Version Date: 10-08-21 
  
8 
OBJECTIVES  
1.1. Hypothesis 
 
Patients discharged from behavioral health hospitals or units that offer a virtual psychiatric transition of 
care (VPTC) program will have a lower 30-day all cause acute care utilization rate (ED, observation, or 
inpatient) compared to patients discharged from behavioral health hospitals or units that do not offer the 
VPTC program. 
1.2. Primary Objective 
 
The primary objective is to evaluate the effectiveness of a virtual psychiatric care team compared to 
standard care, to reduce 30-day all cause non-elective acute care utilization (ED, observation, and 
inpatient encounters). 
1.3. Secondary Objectives 
 
• Compare the effectiveness of the VPTC program compared to standard care, on 60-day all cause 
non-elective acute care utilization (ED, observation, and inpatient encounters). 
• Compare the effectiveness of the VPTC program compared to standard care, on 30-day 
behavioral health acute care utilization (ED, observation, and inpatient encounters). 
o Behavioral health: ED visit with a telepsychiatric consult; inpatient or observation 
discharge from a behavioral health hospital or unit) 
• Examine the percent of patients that had a behavioral health office visit within 7 days of 
discharge 
• Examine the count of avoidable ED visits within 30 days of discharge 
 
1.4. Exploratory Analyses 
 
• Qualitative review of the program 
• Count of patients: Identified, Refused, Enrolled, Graduated, Did not graduate and reason 
• Characterize: demographics and Charlson comorbidity score 
• Assess symptom improvement 
o Suicidal ideation: 
▪ Percent of patients with a PHQ-9, Question 2=0 upon graduation among patients 
with an initial Q2>0 
o Depression reduction:  
▪ Percent of patients with a 5 point reduction of their PHQ-9 score who had an initial 
PHQ-9 score ≥ 10 
▪ Show the distribution of PHQ-9 score reduction among participants 
o Depression remission: Percent of patients with a PHQ-9 score <5 upon graduation who had 
an initial PHQ-9 score ≥ 5 

Version Number: 1.1 
Version Date: 10-08-21 
  
9 
2. BACKGROUND 
The successful transition to outpatient care after behavioral health inpatient services or discharge from an 
emergency department (ED) is critical to reducing suicide risk and deterioration of mental health due to 
disease. Social determinants, limited network of outpatient care providers and disease progression often 
impede patients from accessing outpatient care after discharge. 2018 data from The National Committee for 
Quality Assurance demonstrated that 27.5% of Medicare patients Health Care attended their outpatient 
follow up appointment after discharge from inpatient psychiatric care. The percent for Medicaid patients 
was slightly higher at 35.8% and privately insured patients had a show rate of 46.6%. After interviewing 
patients who were readmitted to Atrium Health Behavioral Health (AH BH) inpatient hospitals, it was 
determined that additional support in that critical period after discharge is needed to assist patients in 
successfully transitioning to outpatient care.  
 
The Virtual Psychiatric Transition of Care (VPTC) program offers support and services to patients in the 
immediate post discharge period to minimize suicide risk, decrease avoidable ED and inpatient care, while 
helping patients successfully connect with an outpatient care provider to maintain long term success in 
disease management. The VPTC program provides a robust virtual care delivery system for patients as they 
transition from inpatient to outpatient care by wrapping around services and support during high-risk 
transition periods. 
 
3. RATIONALE 
The foundational idea for the VPTC program came from the Atrium Health Hospital at Home care model 
which has garnered wide acclaim and financial support in the acute care world. A behavioral health care 
team comprised of licensed clinicians (Behavioral Health Professionals), health coaches, a consulting 
psychiatric pharmacist and psychiatric providers follow patients for 45 days or more depending on patient 
need after their acute care encounter. Patient enrolled in the VPTC program will receive the following core 
components: Introduction to the patient follow-up process, psychosocial assessment, tracking and treatment 
recommendations for (residential movement, nutrition, sleep hygiene, stress management, pain 
management, perinatal/postpartum mood disorders, substance abuse), tracking and treatment 
recommendations for behavioral health symptoms (depression, anxiety, suicidal ideation, etc.), placement 
into an appropriate case management program if needed, navigation to additional psychiatric or substance 
use services if needed, and motivational interviewing and brief therapeutic recommendations if needed.  
 
For patients that have attempted suicide, they will begin the VPTC program following the zero suicide 
pathway. The Atrium Health Behavioral Health Service Line created the ‘Zero Suicide pathway’, in 2019 
for those patients identified as “high risk” for suicide when leaving inpatient units. Zero Suicide is a 
national movement sponsored by the 2012 National Strategy for Suicide Prevention, National Alliance for 
Suicide Prevention, Suicide Prevention Resource Center and the Substance Abuse and Mental Health 
Services Administration (SAMHSA). It is a programmatic approach to keep patients from falling through 
the cracks in a fragmented and disconnected healthcare system and is based on the belief that suicide is 
preventable, and suicide deaths are not fated. Since access to mental health services is constrained in our 
community, patients often fall through the cracks when transitioning between points of care.   
 

Version Number: 1.1 
Version Date: 10-08-21 
  
10 
4. SUBJECT AND SITE SELECTION 
4.1. Accrual  
 
For this study, all patients discharged that meet the inclusion criteria are eligible for analyses. Patients 
meeting eligibility criteria will accrue in both the intervention and usual care arms of the study. 
 
A weekly list will be generated indicating patients discharged the prior week at any of the participating 
locations. The list contains inpatient and observation encounters. The list will identify discharged 
patients at facilities throughout the project, not just once the facility has implemented the VPTC 
program. For the purposes of this project, we are not enrolling towards a targeted patient accrual. We 
will enroll patients during a pre-determined data collection period, See Section 5. 
4.2. Accrual – Subjects 
 
Patients 18 years of age or older at time of discharge, a resident of North Carolina, reported a telephone 
number in the electronic health record, and discharged from a behavioral health hospital or a behavioral 
health unit will be considered exposed to the intervention, whereas patients seen at locations not 
utilizing the VPTC program will be considered exposed to usual care. Patients who are in the 
intervention arm will be treated by the behavioral health care team (licensed clinicians, health coaches, 
a psychiatric pharmacist, and psychiatric providers). Virtual visits are conducted telephonically. Each 
week a list of discharged patients will be received by the behavioral health care team.  
 
Patients included in the analysis of this study must meet the inclusion\exclusion criteria. Patients will 
begin accruing based on the time the location goes live according to the role out schedule. The VPTC 
program is available to any behavioral health patient, but the evaluation is centered on the cohort of 
patients meeting the inclusion criteria. 
 
4.3. Participating sites 
 
The program has been piloted at AH Stanly as part of usual care. AH Stanly will not be included in the 
quantitative assessment. As part of the research design and rollout of this project, randomization will 
occur at the site level. CORE will randomize the facility participation order in advance of project 
kickoff. Site enrollment will begin with 1 facility, adding on 1 additional facility on a rolling basis in 3-
month intervals, until all 7 facilities are active in using the VPTC program. Once added, sites will begin 
utilizing the program in their daily practice among the target patient population.  
 
The following Atrium Health sites will be randomized for participation in this project. 
 
• AH Kings Mountain Psychiatry 
• AH BH Davidson - Fraser Fir 
• AH BH Davidson - River Birch 
• AH BH Charlotte - ED OBS 

Version Number: 1.1 
Version Date: 10-08-21 
  
11 
• AH BH Charlotte - North and South 
• CMC Main 6B 
• AH BH Davidson - Mountain Laurel 
 
4.4. Inclusion\Exclusion Criteria 
4.4.1. Inclusion Criteria 
Eligible patients in the analyses must meet each of the following criteria: 
• Residency or shelter in North Carolina 
• 18 years of age or older on the day of discharge 
• Access virtual care (phone/video) - having a phone number in the electronic health record 
• Discharged from a behavioral health hospital or a behavioral health unit 
4.4.2. Exclusion Criteria 
• None 
4.5. Evaluable Population 
Patients included in the evaluable population for this project, will have their data inform the final 
outcomes assessment. All patients who meet the inclusion and exclusion criteria in both groups of the 
study will be assessed for the primary outcome (intent to treat).  
 
5. OVERALL DESIGN 
5.1. Variables 
5.1.1. Primary Outcome Variable 
The primary outcome variable is 30-day all cause non-elective acute care post discharge utilization 
(ED, inpatient, and observation encounters). 
5.1.2. Secondary Outcome Variables 
a. 
Demographic characteristics (age, race, gender, insurance plan) 
b. 
Charlson comorbidity index score 
c. 
60-day all cause non-elective acute care utilization (ED visits, observation, and inpatient 
encounters). 
d. 
30-day behavioral health acute care utilization (ED visits, observation, and inpatient 
encounters). 
e. 
A behavioral health office visit within 7 days of discharge 
f. 
An avoidable ED visit within 30 days of discharge 

Version Number: 1.1 
Version Date: 10-08-21 
  
12 
5.1.3. Exploratory Variables 
g. Assess symptom improvement for depression 
i. Initial- PHQ-9 total score 
ii. Post- PHQ-9 total score 
h. Assess suicidal ideation 
i. Initial- PHQ-9, Q9 score 
ii. Post- PHQ-9, Q9 score 
5.2. Trial Design 
5.2.1. Justification for stepped wedge cluster randomized control trial 
The primary aim is to assess the effectiveness of a structured multidisciplinary care plan 
(intervention), compared with usual care (control) on 30-day all cause non-elective acute care 
utilization (ED visits, observation, and inpatient encounters). The VPTC program will be 
delivered at the level of location or cluster, thus minimizing contamination and the risk of 
selection bias, if subjects were randomized at the individual level. Therefore, the unit of 
randomization and analysis for this study will be the cluster. Considering logistical constraints 
and design rigor, the intervention will be rolled out sequentially instead of at the same time.  
Moreover, substantial and differential cluster-effects may exist because of distinct volume and 
characteristics of patients in each hospital. Thus, the stepped wedge design is advantageous 
compared with other alternative designs. 
 
The stepped wedge design implies a baseline period, in which no clusters are exposed to the 
intervention. Then, at the chosen time interval of three months, one location will be randomized 
to cross from the control to intervention (Figure 1). This process continues until all seven 
locations have crossed over to the intervention. The order of this cross over process will be 
randomized. We will use randomization to assign location order so that if a high or low volume 
location is assigned cluster 1, a high or low volume location is assigned cluster 7. The single ED 
location will be assigned to cluster 4. SAS Enterprise Guide version 8.2 will be used for the 
randomization. Allocation will be communicated only to the behavioral health team initially to 
prevent contamination. Locations will learn of allocation in the month preceding roll-out. 
Patients in included locations will be blinded to the intervention or control period.  
 
An example of the roll out schedule 
Location 
August 
2021 
November 
2021 
February 
2022 
May 
2022 
August 
2022 
November 
2022 
February 
2023 
May 
2023 
Location 1 
0 
1 
1 
1 
1 
1 
1 
1 
Location 2 
0 
0 
1 
1 
1 
1 
1 
1 
Location 3 
0 
0 
0 
1 
1 
1 
1 
1 
Location 4 
0 
0 
0 
0 
1 
1 
1 
1 
Location 5 
0 
0 
0 
0 
0 
1 
1 
1 
Location 6 
0 
0 
0 
0 
0 
0 
1 
1 
Location 7 
0 
0 
0 
0 
0 
0 
0 
1 
1=Virtual Psychiatric Transition of Care Program; 0=Usual Care 
 
 

Version Number: 1.1 
Version Date: 10-08-21 
  
13 
5.2.2. Virtual Psychiatric Transition of Care Program 
 
5.2.2.1. 
Zero Suicide Pathway 
Depending on the patient’s reason for admission, patients may start the VPTC program in the 
zero suicide pathway. Patients that meet criteria for this higher level of care will have attempted 
suicide. Patients are identified from the C-SSRS screening tool along with referrals from the 
treatment team and treating psychiatrist. Identified patients are enrolled a day or two prior to 
discharge and then contacted within 24 hours of discharge, to assess for suicidality and self-harm 
and to review safety plans and discharge instructions. Patients are contacted daily, to every few 
days until they no longer need the higher-level of outreach and will be transferred to the VPTC 
pathway. VPTC clinicians will continue to outreach with the patient for the remaining time in the 
VPTC program. 
5.2.2.2. 
VPTC Pathway 
If a patient has not attempted suicide as the reason for admission, VPTC clinicians will begin 
outreach with the patient within the first 24-72 hours after discharge. The first outreach with the 
patient will be completed by a licensed clinician who will complete a brief psychosocial 
assessment with the patient. After the initial outreach is completed a health coach will then reach 
out to the patient to set a weekly schedule. The weekly outreaches with the VPTC health coach 
will be patient centered and patient specific. These outreaches will help support patients through 
goal setting/treatment plans, medication support, linkage and support to community resources, 
and safety planning. These outreaches will also help patients identify any barriers they may have 
in attending their follow up care with their outpatient community providers. Throughout the 
outreaches at any point and time if a patient has a question or concern about their medication the 
VPTC licensed practicing nurse can contact the patient to help address those needs. At each 
outreach patient’s safety is also assessed. At any point in the outreach if a patient endorses any 
safety concerns that patient will be assessed by a licensed qualified clinician to identify those 
safety concerns and create a safety plan with the patient. 
 
If the patient is scheduled with a VPTC psychiatric provider for a hospital follow-up 
appointment, an LPN (Licensed Practical Nurse) will outreach to the patient before that visit to 
complete the nursing intake. The patient will then have their hospital follow up visit either with 
their community provider, or the VPTC psychiatric provider. The VPTC psychiatric provider will 
review medications that the patient started on the inpatient unit or emergency department and 
provide any further education, adjustments, and support. The provider is able to order the 
medications. The VPTC program will also have support from a psychiatric provider through 
weekly treatment team huddles to help support patients and identify any barriers they may be 
facing. 
 

Version Number: 1.1 
Version Date: 10-08-21 
  
14 
There may be cases where patients are eligible to enroll in full case management programs based 
on their insurance or other criteria. In these situations, the VPTC team will coordinate a care 
transition and the patient will be considered to have completed the program. For example, if a 
patient has Medicaid through a local management entity, a managed care organization providing 
local behavioral health care, and meets certain criteria, the patient will not require services from 
the VPTC team as they would be duplicative.  
5.3. Patient Graduation from the VPTC Program 
• Reached max benefit as deemed by the behavioral health care team 
• Unable to connect: 4 calls attempted over 12 days 
• Transition to lower level of care 
o Patients who transition to a primary care clinic 
• Transition to higher level of care 
o Psychiatry/Treatment Program 
o Readmission to an Inpatient Facility 
• Declined or no longer interested in services 
• Death 
 
5.4. Continuation of the Intervention 
Patients may continue enrolling in the VPTC program after the study period has ended.  
 
6. QUALITATIVE REVIEW 
A qualitative evaluation will be conducted among the care team and patients at the pilot location. This 
evaluation aims to evaluate the implementation of the psychiatric virtual transition of care program and the 
nature of the targeted patient population. The qualitative data collection for this study will be guided by the 
Consolidated Framework for Implementation Research (CFIR). The CFIR incorporates several established 
implementation theories into a common framework. Composed of a menu of 39 constructs that fall into 5 
domains (intervention characteristics, outer setting, inner setting, characteristics of individual, and process), 
the CFIR is used to design, guide, and evaluate health care delivery interventions by identifying potential or 
existing barriers and facilitators to implementation (Damschroder et al. 2009). Through the aggregation of 
corroborated constructs, the CFIR promotes a systematic and consistent use of implementation measures, 
which help render study findings intelligible and generalizable. The framework is well-suited to projects 
like the VPTC program as it allows for complex theory-building and cross-study comparisons using a 
common set of constructs while obviating the need to use multiple implementation frameworks.  
 
 
 
 
 
 
Semi-structured CFIR-guided interviews will be conducted with 12 VPTC patients/care takers and 12 VPTC 
team members. This sample size was arrived at after considering the project’s narrow study aim (Malterud 
et al., 2016). Clinician interviews will include a diversity of frontline staff, including, but not limited to, 
nurses, health coaches, psychiatrists, and discharge planners.    
 
 
 
 
 

Version Number: 1.1 
Version Date: 10-08-21 
  
15 
Patient-interviewees will have been enrolled in the VPTC program and be targeted using a convenience 
sample, a non-probability sampling strategy commonly used for studying reclusive and reluctant 
populations (Bernard, 2017; Guest et al., 2006). A VPTC team member will reach out to the patient and ask 
if contact can be made to discuss their thoughts on the VPTC program. A researcher will call to follow-up 
with and schedule an interview with the patient. During recruitment calls, potential patient-interviewees will 
be assured that their receiving of care is not in any way contingent on consenting to an interview. This 
assertion will be reaffirmed by the interviewer before the beginning of interviews. Patients that complete an 
interview will be compensated for their time with a $25 Clincard. 
 
 
 
 
Clinician-interviewees will be recruited through snowball sampling: The VPTC coordinator will identify 
and refer clinicians and other relevant staff for interviews, these interviewees will in turn be asked to 
identify other potential respondents. A researcher will send up to two weekly emails to introduce clinicians 
to the qualitative review and encourage them to contact us to schedule an interview. A researchers will 
schedule an appointment with those who reply to the email and are interested. All interviews will be 
conducted via telephone and will be audio recorded.  
 
 
 
 
 
 
 
 
 
The recorded interviews will be de-identified and then parsed and analyzed using ATLAS.ti 9. The 
analytical approach will be inductive—building theory on patterns found in the data—as such analyses are 
typically well-suited for exploratory examinations of novel or under-studied social phenomena. Coding will 
begin at the question-level, allowing the qualitative team to compare all interviewee answers to a specific 
question. Next. a vertical analysis across interviews will be conducted using the constant comparative 
method, a “scrutiny-based technique” (Ryan and Bernard, 2003: 101) foundational to grounded theory, an 
approach in which theory building is based in empirical analysis rather than deductive inference. Using this 
comparative method, codes will be produced inductively and iteratively, as early codes often have a 
contingent and provisional quality and must be delineated and refined through a continual comparison with 
preceding and subsequent data (Boeije, 2002).  During these vertical readings, codes will be used to 
paraphrase and summarize passages. 
 
 
 
 
 
Themes will be developed using the constant comparative method, which provides a framework to 
understand the commonalities and differences within and among interviews, as the method is also a method 
of identifying contrasts. (Bradley, Curry & Devers 2007). Themes will be inferred from particular, 
reoccurring instances—through the repeated parsing and partitioned of interview data according to CFIR 
constructs and the relevance to our research questions. Once barriers and facilitators of implementation have 
been identified using the CFIR framework, recommendation will be drafted to optimize implementation of 
the VPTC program.  
 
The most immediate risk of the aforementioned data collection procedures to interviewees is the potential 
for embarrassment, should their unredacted interview data be shared. Several measures will be put in place 
to protect the identity and confidentiality of interviewees: recordings of the interviews will be stored on a 
password-protected computer accessible only to pertinent project staff; transcriptions of the interviews will 
be de-identified, i.e. the names of interviewees, employers, and care providers will be redacted; and 
interview quotes used in subsequent presentations, reports, and publications will be anonymized. 
 
7. DATA COLLECTION AND REPORTING 
Patient demographics, Charlson comorbidity score, and utilization will be obtained from the electronic 
medical record and billing systems. Data from the initial assessment and follow-up contacts will be stored 
in HealtheCare, a care management platform in Cerner. Data will be retrieved by an application specialist 
on the research team. Data may be retrieved retrospective to a patient’s completion of the intervention or 

Version Number: 1.1 
Version Date: 10-08-21 
  
16 
usual care arm of the study. A participation rate can be generated from the count of patients enrolled in the 
VBHI program over the count of patients meeting the inclusion criteria. 
 
A bi-monthly report will be produced for the intervention arm. The report will be by primary care practice 
and overall showing the prior 2 weeks of activity. 
• Count of patients meeting the inclusion criteria 
• Count of patients with a HealtheCare account 
• Count of patients enrolled in the VBHI program 
• Patient list with their enrollment status in HealtheCare 
A monthly report will be produced for the intervention arm. The report will be by primary care practice and 
overall. Report to be generated 2 weeks after the end of the month showing the prior month of activity. 
• Count of patients meeting the inclusion criteria 
• Count of patients with a HealtheCare account 
• Count of patients enrolled in the VBHI program 
• Patient list with their enrollment status in HealtheCare 
 
7.1. Sample Size Analysis 
Calculations of sample size were based on primary analysis of 30-day non-elective acute care utilization 
on a binary group. Our preliminary data shows that average unique patients in each hospital (cluster 
size) per three months ranged from 159 to 300, we expected an enrollment rate of 40%. Thus, the 
average effective cluster size is 96 with a standard deviation of 15. Conservatively, we would at least 
expect an average cluster size of 81. With the assumption of detecting a change from 20% in the control 
period to 14% in the intervention period for 30-day non-elective acute care utilization, a mean cluster 
size of 81 with seven clusters and seven steps (8 time periods inclusive of a control period) would 
achieve a power of 0.89 if intra-class correlation coefficients (ICC)=0.20 and two-side α=0.05. Holding 
other parameters constant, the larger ICC is, the higher the power we will obtain. The coefficient of 
variation (CV) of sample size is approximately 0.15. To account for this varying cluster size, we inflated 
the number of patients by 10% resulting in an average cluster size of 89 patients per three months, 
resulting in a sample size of 2,495 patients among the intervention group and 4,990 patients overall 
inclusive of our control population. In this pragmatic trial, those assigned to the intervention but did not 
enroll in VPTC during the intervention period (around 60%) would also be counted in the intervention 
arm. It is within our expectation that intervention arm eventually has smaller reduction in 30-day non-
elective acute care utilization (i.e. 0.4*0.14 +0.6*0.20=17.6%, difference=2.4%). However, we still 
ensure a power above 0.80 as long as over 2,495 patients actually received VPTC. The power was 
calculated by using the PASS 15. 
 
7.2. Statistical analysis 
Distributions of baseline characteristics for locations and patients will be compared between the 
intervention or control periods to assess effectiveness of the randomization, and statistical or clinical 
differences will be adjusted in sensitivity analyses. All analyses for treatment and control periods 
comparisons will use an intention-to-treat approach and results will be reported used the CONSORT 
extension to cluster randomized trials.  
 
The primary outcome for this study is 30-day all cause non-elective acute care utilization. 
Characteristics of the locations and individuals will be summarized by levels of randomization steps, 
locations, and individuals. The count, rates and 95% confidence intervals of 30-day non-elective acute 

Version Number: 1.1 
Version Date: 10-08-21 
  
17 
care utilization in the control period will be presented and compared with that in the treatment period. 
The primary analysis will be conducted using generalized linear mixed models with fixed effects for 
intervention, and intervention multiply by the time, accounting for the cluster randomized trial design 
with a random cluster (hospitals) or time (steps) effect. We will estimate and report the intra class 
correlation coefficients for all outcome measures to assess assumptions for sample size analyses and for 
future investigations using similar designs and outcomes. The primary analyses will follow the 
intention to treat principle with an additional per protocol analysis including those who actually used 
VPTC. The secondary analyses will include within cluster comparisons of intervention and control 
periods (not adjusting for any confounding effect of time), as well as comparisons of a series of 
(unbalanced) quasi-parallel cluster trials between those enrolled and unenrolled clusters at each 
intervention enrollment period. The same analyses plan will also be applied to secondary outcomes. 
 
7.3. Missing Data 
All the related available electronic data will be pulled without missing any subjects. Thus, the missing 
of outcome variables are less likely to occur. However, the missing information on covariables is 
unavoidable. Sensitivity analyses are recommended for trials with missing data. We will compare 
baseline characteristics such as age and gender between patients with complete follow-up data to those 
with missing data by intervention period to assess potential biases that may exist in the complete case 
analysis. We will conduct sensitivity analyses for the primary and secondary outcomes using several 
methods which have different missing data assumptions: (1) complete case analyses which assumes 
missing completely at random; (2) multiple imputation using M=10 imputations, which assumes 
missing at random; and (3) assigning poor scores and good scores for missing values differentially by 
treatment group, which aligns with non-ignorable missingness (the data missingness is related to the 
actual value).  
 
7.4. Training 
Beginning two to three weeks prior to first patient enrolment at the respective location, the VPTC 
program coordinator and those responsible for discharge will begin training on the materials supporting 
the program. This will include communication touchpoints between the coordinator and discharge 
facilitator.  
 
7.5. Data Collection Dates 
The implementation of the VPTC program is expected to begin in November 2021 with last enrollment 
in July 2023. 
 
 
 
 
 
 
 
 

Version Number: 1.1 
Version Date: 10-08-21 
  
18 
8. INTERVENTION PLAN 
8.1. Behavioral Health Professional Assessments 
The initial patient assessment by the behavioral health care team captures components such as: 
• Insurance 
• Primary care provider 
• Outpatient Psychiatry provider 
• Collateral resource 
• Recent visit summary 
• Substance abuse 
• Medication barriers 
• Appointment Barriers 
• Transportation needs 
• Access to medications 
• Community Resources 
• Columbia-Suicide Rating Scale to assess for suicide ideation 
• Review Hospital Discharge plan  
• Crisis plan and follow up 
8.2. Follow-up Assessments 
The follow-up contact assessments are comprised of components such as: 
• Columbia-Suicide Rating Scale to assess for suicide ideation 
• Appointment reminder\follow-up 
• Medication reconciliation 
• Medication access barriers 
• Substance use disorder follow-up 
• Supportive listening 
• Psychoeducation 
• Utilization of community resources 
• Crisis planning and adherence to the plan by the patient 
• Abnormal Involuntary Movement Scale (AIMS) 
• PHQ-9 screening tool 
• GAD-7 screening tool 
8.3. Psychiatric Virtual Appointments 
• Medication reconciliation 
• Review of history of treatment  
• Review of current treatment 
• Review Discharge plan  

Version Number: 1.1 
Version Date: 10-08-21 
  
19 
9. STUDY GOVERNANCE 
This quality improvement trial will be conducted at Atrium Health. It will be run jointly by the Center for 
Outcomes Research and Evaluation (CORE) and the Behavioral Health Service Line. Manuel Castro, MD, 
(Psychiatry) will serve as the clinical principal investigator with oversight from the Executive Committee 
(EC). Jason Roberge, PhD, MPH will serve as co-principal investigator on behalf of CORE. The EC will 
consist of leaders across the System involved in the trial, quality improvement, and implementation (Table 
2). The EC will have the overall responsibility of trial oversight and direction. The EC will support 
dissemination of project findings and next steps. The EC will receive progress reports from the team. When 
appropriate, ad hoc committee meetings will be scheduled to discuss pressing concerns. 
 
Table 2. Executive Committee 
James Rachal 
Atrium Health Executive Leadership – Psychiatry 
Eva McNeill 
Atrium Health Executive Leadership – Behavioral Health 
Wayne Sparks 
Atrium Health Executive Leadership – Psychiatry 
Christine Zazzaro 
Atrium Health Executive Leadership – Behavioral Health 
Martha Whitecotton 
Atrium Health Executive Leadership – Behavioral Health 
Andrew McWilliams 
Atrium Health Executive Leadership – CORE Medical Director 
 
10. SAFETY RISKS AND REPORTING 
The data collection and intervention for this project presents no more than minimal risk to patients. This 
study is evaluating the VPTC program among discharged patients. The program itself is similar to existing 
virtual programs managed by the behavioral health service line for patients discharged from the emergency 
room.  The implementation of the VPTC program and its components complement ongoing patient care 
through virtual patient navigation within Atrium Health. The addition of an evaluation design that aligns 
with existing patient care expansion thus confers minimal additional risk to patients. 
Other potential risks of participation in this project include the risk of health information disclosure. There 
is always the risk of disclosure of a patient’s private health information (PHI) or medical information. 
However, the processes identified in this protocol to enable the execution of this project, do not increase 
inherent risk of disclosure. Atrium Health utilizes several hard and soft safety controls in the protection of 
patient information and medical records. Security controls include, but are not limited to, multiple system 
firewalls, access restrictions to patient records and information, locked offices and buildings housing 
research and patient data, and multiple layers of username and password protected computer and system 
access. Statistical tools and programs that are developed are stored on secure restricted access file shares.  
Programs have versioning controls turned on and access to the file shares are restricted. Access is granted 
through a request process that requires multiple authorizations and is controlled centrally by our 
Information and Analytics Service Department.  
The CORE project team will ensure that appropriate handling of patient PHI follows standard Atrium 
Health procedure. Confidentiality will be maintained according to ICH E6; 4.8.10, part 0: “Records 
identifying the subject will be kept confidential and, to the extent permitted by the applicable laws and/or 

Version Number: 1.1 
Version Date: 10-08-21 
  
20 
regulations, will not be made publicly available. If the results of the study trial are published, the patient’s 
identity will remain confidential.’ In the event of PHI disclosure, the appropriate internal departments will 
be informed and processed per legislation and privacy regulations. 
10.1. 
Data Quality Assurance 
This study will be organized, performed, and reported in compliance with the study protocol, 
standard operating procedures (SOPs) of the CORE and Atrium Health, and other applicable 
regulations and guidelines (e.g. GCP).  
 
10.2. 
Safety Reporting to the IRB 
All events occurring during the conduct of a protocol and meeting the definition of a reportable 
safety event per the IRB guidelines, will be reported to the IRB within 10 working days of the 
Investigator learning of the event, per their requirements. 
 
Major protocol deviations that result in a threat to subject safety or the integrity of the study will be 
reported to the IRB per their requirements. 
 
10.3. 
Data Monitoring by the Sponsor 
The conduct of this project will abide by standard operating procedures set forth by both Atrium 
Health and CORE. The Principal Investigators, statistician, and other team members will meet as 
needed to review enrollment and retention, study progress, and validity/integrity of the data. 
Documentation of these meetings will be kept with study records. 
11. RESEARCH COMPLETION 
The Principal Investigators have the right to close the project at any site at any time. 
For any closure, the following applies: 
• Closure should occur only after consultation between involved parties. 
• In case of a partial study or location closure, patients still participating in the VPTC program, or 
those who are considered in follow-up, must be taken care of in an ethical manner. 
 
The study will be considered complete when one or more of the following conditions is met: 
• 
The enrollment period has ended, and the data collection period is complete. 
• 
All subjects have dropped out or discontinued from the study after the enrollment period is 
completed, but prior to data collection cutoff. 
• 
The IRB or Principal Investigator discontinues the study. 
• 
The Principal Investigator defines an administrative or clinical cut-off date. 

Version Number: 1.1 
Version Date: 10-08-21 
  
21 
Upon study completion, a final report will be presented to the Executive Committee and all key 
stakeholders. The final report will detail all findings including primary, secondary and post-hoc outcomes. 
The team will also prepare a manuscript for publication focused on outcomes and feasibility of 
implementation of the VPTC program.   
 
12. ETHICAL AND LEGAL ISSUES 
12.1. 
Ethical and Legal Conduct of the Study 
The procedures set out in this protocol, pertaining to the conduct, evaluation, and documentation of this 
study, are designed to ensure that the Investigators abide by Good Clinical Practice (GCP) guidelines 
and under the guiding principles detailed in the Declaration of Helsinki. The study will also be carried 
out in keeping with the applicable local laws and regulation(s). 
 
Documented approval from appropriate agencies (e.g. IRB) will be obtained before the start of the 
study, per GCP, local laws, regulations, and organizations.  
 
Strict adherence to all specifications laid down in this protocol is required for all aspects of study 
conduct; the Investigators may not modify or alter the procedures described in this protocol. 
 
Modifications to the study protocol will not be implemented without consulting the Principal 
Investigator and the IRB, as applicable. The Principal Investigator must assure that all study personnel, 
including co-investigators and other study staff members, adhere to the study protocol and all 
applicable regulations and guidelines regarding research both during and after study completion. 
 
The Principal Investigator will be responsible for assuring that all the required data will be collected 
and properly documented. 
12.2. 
Confidentiality 
All records identifying the subject will be kept confidential and, to the extent permitted by the 
applicable laws and/or regulations, will not be made publicly available. 
12.3. 
Disclosure of Data 
The Principal Investigator, his associates and co-workers, and the appropriate regulatory agencies may 
use the information and data included in this protocol as necessary for the conduct of the study. 
Information contained in this study, and data and results from the study are confidential and may not be 
disclosed without the written permission of the Principal Investigator.  
 
 

Version Number: 1.1 
Version Date: 10-08-21 
  
22 
13. RETENTION OF RECORDS 
Essential documentation including all IRB correspondence, will be retained for at least 2 years after the 
investigation is completed. Documentation will be readily available upon request. 
 
14. PUBLICATION POLICY 
The Principal Investigator or designee must send any draft manuscript, abstract, or conference presentation 
to members of the project Executive Committee for feedback and transparency, prior to submission of the 
final version. The Principal Investigator will be responsible for all relevant aspects regarding data reporting 
and publication.  
 
The Principal Investigator or designee will ensure that the information and results regarding the study will 
be made publicly available on the internet at www.clinicaltrials.gov. 
 
15. REFERENCES 
Damschroder, Laura J., David C. Aron, Rosalind E. Keith, Susan R. Kirsh, Jeffery A. Alexander, and Julie 
C. Lowery. "Fostering implementation of health services research findings into practice: a consolidated 
framework for advancing implementation science." Implementation science 4, no. 1 (2009): 1-15. 
 
Malterud, K., Siersma, V. D., & Guassora, A. D. (2016). Sample size in qualitative interview studies: 
guided by information power. Qualitative Health Research, 26(13), 1753-1760. 
https://doi.org/10.1177/1049732315617444 
 
Bernard, H. R. (2017). Research methods in anthropology: Qualitative and quantitative approaches. 
Lanham, Maryland: Rowman & Littlefield. 
 
Guest, G., Bunce, A., & Johnson, L. (2006). How many interviews are enough? An experiment with data 
saturation and variability. Field Methods, 18(1), 59-82. 
https://doi.org/10.1177/1525822X05279903 
 
Ryan, G. W., & Bernard, H. R. (2003). Techniques to identify themes. Field methods, 15(1), 85-109. 
 https://doi.org/10.1177/1525822X02239569 
 
Boeije, H. (2002). A purposeful approach to the constant comparative method in the analysis of qualitative 
interviews. Quality and quantity, 36(4), 391-409. 
https://doi.org/10.1023/A:1020909529486 
 

Version Number: 1.1 
Version Date: 10-08-21 
  
23 
Bradley, E. H., Curry, L. A., & Devers, K. J. (2007). Qualitative data analysis for health services research: 
developing taxonomy, themes, and theory. Health services research, 42(4), 1758-1772. 
https://doi.org/10.1111/j.1475-6773.2006.00684.x