An Innovative Mental Health Virtual Ward: Evaluation of Patient-Centered Outcomes
Phase: NA
Conditions: Mental Health Disorder, Mental Health Impairment, Psychiatric Emergency, Crisis; Emotional
Sponsor: University of Manitoba

1 
Virtual Ward Cohort Study Protocol 
Version date: January 31, 2024 
An Innovative Mental Health Virtual Ward: Evaluation of Patient-Centred Outcomes 
HS26297 (H2024:027) 
January 31, 2024 

2 
Virtual Ward Cohort Study Protocol 
Version date: January 31, 2024 
Introduction 
In response to the changes in healthcare systems delivery accelerated by COVID-19 
public health restrictions, novel virtual alternatives to in-person care were launched to support 
the stabilization of individuals in mental health crisis. In Manitoba, Canada, a Virtual Acute 
Psychiatric Unit (Castillo et al., 2023) and Virtual Crisis Stabilization Unit (Mental Health Virtual 
Ward; Pullia et al., 2023) were launched early in the pandemic for residents of Winnipeg (the 
province’s largest city) requiring intensive mental health support, as a strategy to divert need 
for in-person care in hospital and the community-based Crisis Stabilization Unit. After 2 years, 
the Virtual psychiatric unit closed due to staffing issues, and the Mental Health Virtual Ward 
continued to operate at 6-‘bed’ capacity with Government funding. Effective November 2023, 
these programs have merged and expanded to the entire province as part of a Provincial Virtual 
Crisis Service that will offer emergency telemental health assessment to rural health centres 
and intensive follow-up virtual crisis stabilization support post-crisis visit to individuals across 
Manitoba. The latter program component is referred to as the Mental Health Virtual Ward (MH 
vWard) with significant potential to increase access to needed care and provide person-
centered alternatives to hospital. As this program is unique, long-term outcomes have not been 
studied. This research proposes to prospectively measure demographic, recovery, service 
delivery, and systems use outcomes in a cohort of MH vWard admissions, and 2) to establish a 
cohort of admissions to use as a historical comparison/baseline for more rigorous evaluation in 
the future. 
Background 
There is a Need for Accessible Mental Health Crisis Care in Manitoba. Manitoba has 
the highest prevalence of any mental disorder compared to other Canadian provinces, with 
major depressive disorder and suicidal ideation being especially prevalent among Manitobans 
(Palay et al., 2019). Crisis and emergency systems are common entry points to the mental 
health systems and serve to support individuals with existing mental health difficulties during 
times of worsening or increased need (Butler et al., n.d.).  Winnipeg emergency departments 
and urgent care centres receive approximately 10,000 visits for mental health concerns yearly, 
with nearly half of those at HSC alone (Shared Health, 2018). The Crisis Response Centre – open 
24 hours a day for individuals to walk-in and receive mental health assessment and support – 
has over 6000 annual visits and approximately 17,000 calls to the mobile crisis line (Crisis 
Response Services, 2022). Depending on the level of need, individuals experiencing mental 
health crises require intervention options ranging from hospital to community systems. 
Hospital bed shortages and corresponding long ED wait times to admission (upwards of 40 
hours in Manitoba; Canadian Institute for Health Information, 2016) have necessitated 
alternative solutions for those needing higher intensity support.  
The virtual units are modelled after inpatient services based on the concept of a Virtual 
Ward or Hospital-at-home (Arsenault-Lapierre et al., 2021). Intensive daily care is provided 
remotely to individuals in their own homes or other place of their choosing. Care is delivered by 
telephone, email, videoconferencing, and text messaging, with daily on-line live hosted groups 
offered on a topic related to mental wellness and recovery. After hours support is available 
through a 24-hour a day crisis line and individuals can also access in person support at the Crisis 

3 
Virtual Ward Cohort Study Protocol 
Version date: January 31, 2024 
Response Centre at any time with continuity of care ensured through use of a common 
electronic patient record. Although the virtual acute psychiatry unit (staffed by psychiatrists 
and physician assistants) closed after 2 years due to staffing limitations, the Mental Health 
Virtual Ward has continued to operate 6 “beds” supported by a multidisciplinary team, with 
approximately 350 admissions/year, and ongoing high demand despite the in-person CSU 
returning to full capacity in August 2021. The newly expanded province-wide program is a 
merging of the virtual units, to have a single 12-bed MH vWard that provides multidisciplinary 
care from a team of clinicians, nurses, and psychiatrists enabling management of a wide range 
of patients.   
Rationale 
Further evaluation needs to determine if this novel service is impacting meaningful 
outcomes (i.e., recovery, functioning, systems use). While systems level outcomes such as 
avoided emergency department visits and hospitalization are often prioritized by decision 
makers and researchers, they may not be the only or most meaningful outcomes for patients 
and families (dosReis et al, 2022; Frank et al., 2014). For example, in previous work done by the 
PI on the impact of a post-discharge transitional program for patient with mood and anxiety 
disorders, it was discovered that many of the components of the planned program either didn’t 
work or didn’t matter to patients and families (Bhattacharyya et al, 2015). Identifying the most 
meaningful outcomes for both the systems and the patients, and evaluating if the MH vWard is 
having a positive impact on these outcomes is critical to inform ongoing improvement and to 
make sure that the care is meeting the needs of the people who receive it in the best way 
possible. We also need to evaluate this program to support ongoing investment in the 
resources needed to run it, and to support its scale and spread.  
Objectives 
The objectives of this project are to: (1) prospectively measure demographic, recovery, 
service delivery, and systems use outcomes in a cohort of MH vWard admissions, and (2) 
establish this cohort for use in future research. 
Methods 
Study Design 
Objective 1 involves the prospective collection of data from a cohort of MH vWard 
admissions over approximately 1 year to measure demographic, recovery, service delivery, and 
systems. Outcomes for consenting individuals will be assessed at time of admission (Time 1), 1 
week after discharge (Time 2), 4 weeks after discharge (Time 3), and 6 months after discharge 
(Time 4). Data on health systems use during the 3 months prior to admission (i.e., pre-test data) 
and 6 months after admission will be collected in order to identify changes in post-intervention 
systems use. Within the prospective cohort study, we will also establish the requirements for 
the cohort to be used for future research (Objective 2). 
Patient and Caregiver Advisory Committee (PCAC). This study includes a PCAC who will 
be involved in the initial selection of outcomes and will give input into the overall study 
procedures to incorporate the voices of people with lived experience. Given that this study 

4 
Virtual Ward Cohort Study Protocol 
Version date: January 31, 2024 
collects data from individuals experiencing a mental health crisis, the input from a PCAC will be 
essential to determine optimal data collection protocols that increase recruitment and 
retention and overall data quality. The PCAC will consist of approximately 6 
individuals/caregivers from across Manitoba who have lived experience with crisis services. The 
PCAC will meet regularly with the study team to achieve project objectives. Oversight of the 
PCAC will be through our patient engagement lead and partnership with Peer Connections 
Manitoba.   
Exposure to Intervention 
Individuals referred to the MH vWard receive an intensive level of daily support that 
approximates what would be provided through receipt of in-person services but they remain in 
the community, usually at their own homes or with family/friends. Care in the MH vWard is 
delivered by any combination of telephone, email, videoconferencing, and text messaging. 
Individuals have the option of opting in to the use of a web- or app-based tool to complement 
their care delivery and support self-management. The typical duration of admission is 5 days 
with flexibility based on individual needs. The program is ‘open’ 8am to 10pm. All individuals 
develop an after-hours crisis plan that varies depending where they are located and what the 
local resources are. 
 Program components are as follows: 
Individual therapy and care planning. Daily 1:1 sessions with a clinician or psychiatry 
team member to assess and monitor, develop and progress on recovery goals, discuss referrals, 
etc.  
Engagement of formal and informal supports. Direct and indirect communication with 
community providers for collateral and collaboration, as well as involvement of informal 
supports (i.e. family, friends) through engagement during 1:1 sessions or additional contacts. 
Medication reconciliation and management. Full medication review and reconciliation 
with program nurse at time of admission. Psychiatry team will manage medications during stay 
if relevant. Discharge medication reconciliation is completed when appropriate.  
Group programming. Daily on-line, live hosted groups are provided for all individuals 
admitted. These will cover a range of content including skills from Cognitive Behavioural 
Therapy, medication education, etc.  
Referrals for follow-up services. Referrals to community-based services for ongoing 
support post-discharge. 
After hours support. All individuals will create their own individualized after hours 
support plan at the time of referral and/or intake. Depending where the individual resides this 
will include crisis lines, in person options, etc.  
Telus Home Health Monitoring (HHM) App.  At intake, individuals may opt in to using 
the Telus HHM app which is custom designed to complement the services provided by the 
program. This system has been approved and adopted by Shared Health for the remote home 
monitoring of patients. The app pushes periodic surveys to admitted individuals that prompts 
them to track symptoms and monitor their progress, select availability for 1:1 support and 
group programming, and work through their crisis plan if needed. The app also allows the 
custom selection of educational resources that the individual can reference as part of their 

5 
Virtual Ward Cohort Study Protocol 
Version date: January 31, 2024 
treatment plan. Alerts can be sent to the clinical team based on any pre-determined criteria 
that the team identifies (e.g. a symptom score above a certain threshold). The clinical team has 
access to a dashboard through which an individual’s use of the app is monitored during 
program hours. Individuals are active on the app for the duration of their admission; after 
which they retain access to the materials for an additional 30 days but no longer are monitored 
by the clinical team.  
Integration with in-person services. In the case that an admitted individual requires 
their care to step up to in-person, the MH vWard is integrated with emergency services and 
inpatient units. Whenever possible, individuals are admitted directly from their home to the 
inpatient unit to avoid an intermediate emergency visit. If needed due to acute safety or other 
concerns, local emergency services (911, RCMP) are activated and the team communicates 
directly with the receiving site to ensure continuity of care.  
Participants 
Eligibility 
All individuals admitted to the virtual ward during the study period are eligible for 
participation in the study. There are no exclusion criteria. 
Sample Size 
A sample size calculation was made based on the change score (paired t-test) of the 
study’s primary outcome measure for recovery (WHO-5 Well-being Index). The minimum 
clinically significant difference for the WHO-5 Well-being Index has been established (Topp et 
al., 2015). Parameter information for the power calculation was provided by a study that used 
the WHO-5 Well-Being Index in a psychiatric inpatient population (Newnham et al., 2010). The 
estimated standard deviation of the change score for the present study was determined to be 
4.9 based a standard deviation of 5.5 and a rwithin of 0.6. Additional parameters included a two-
tailed test at α=0.05, a type II error rate of 20%, and a medium to large effective size of 0.65. 
Based on these parameters, the required sample size is 452.  
Drawing on Mental Health Virtual Ward admission statistics in the 2022 calendar year, 
based on the 6-bed unit, there were about 350 admissions. With the program growing to 12 
beds, it is expected the total number of admissions would double to 700 over 1 year. We 
anticipate about 10-15% of individuals may choose not to participate in the study so we are 
expecting a sample of around 595 admissions. With participant incentives (gift cards) for 
responding to surveys and follow up contacts by our team, we hope for an 80% response rate 
for outcome surveys based on completion rates described by Gilbody et al. (2015) who used a 
similar method for retention in a trial of an on-line mental health intervention. We estimate 
that this would result in outcome data at Time 2 for 476 admissions with complete pre and post 
data over 1 year. Recruitment could be extended by several months if we are failing to meet 
targets within the 1-year timeframe.  
Recruitment 
Recruitment script at time of referral & Referral Confirmation/Welcome Email 

6 
Virtual Ward Cohort Study Protocol 
Version date: January 31, 2024 
The recruitment process begins with a welcome email that is sent by the program. From 
here, interested individuals may click on a link in the referral confirmation/welcome email and 
self-proceed to review the online consent form and enroll in the study. For those who do not 
complete this step prior to their admission intake (typically 24-48 hours from receipt of 
referral), a clinician will follow up to inquire about their interest during this meeting. Those 
indicating interest at that point will be directed to the same information via email. For those 
with no email address, permission will be obtained from patients by a clinician for a research 
team member to reach out for the purposes of discussing the study and enroll over the phone. 
The first month of the study will be a feasibility period to test out our recruitment strategies. 
The researchers will seek feedback and suggestions to troubleshoot recruitment barriers from 
the clinical team members and the PCAC during this time. Recruitment strategies will be 
adjusted as required.   
Data Collection 
Data will be collected through self-report surveys at designated time points, as well as chart 
review to collect service delivery and healthcare utilization data.   
Participant Incentives. Participants will be offered honoraria as incentives for study 
participation. Participants will be offered $5 gift cards at each follow-up data collection time 
points (Time 2, 3, and 4). Gift cards will be administered electronically (unless physical cards are 
requested) by the RA.  
Variables & Data Sources/Measures 
Variables of interest for the study include participant demographics, along with service 
delivery, recovery, and systems use outcomes.  
Demographic Variables (Time 1). The Demographic Questionnaire and SINCERE questionnaire 
will be completed at Time 1 (admission). The demographic questionnaire will include variables 
related to sociodemographic information: age, gender, education, income, employment status, 
relationship status, living situation, urban/rural residence, race/ethnicity, and clinical reason for 
referral/diagnosis. The data will be collected from participant self-report. 
Recovery Outcomes (Time 1, 2, 3, and 4).   Recovery will be measured using 2 questionnaires 
that were co-selected with input from the PCAC: the WHO-5 Well-Being Index and the 
Maryland Assessment of Recovery Scale- 12 item (MARS-12). The Who-5 Well-Being Index is a 
5-item measure that assesses subjective psychological well-being” (Topp et al., 2014, p. 167). It
has been used to monitor global recovery among psychiatric inpatients showing good sensitivity
to change with an established MCID that correlates to other validated measures of psychiatric
impairment (Topp et al., 2014). MARS-12 is a shortened version of the MARS-24 designed to
measure recovery orientation among individuals with mental illness (Drapalski et al., 2012).
Service Delivery Outcomes (Discharge). Service delivery outcome data are: length of stay, and
outcome of stay (discharge, transfer to in person CSU/hospital, emergency activation, and lost

7 
Virtual Ward Cohort Study Protocol 
Version date: January 31, 2024 
to follow-up. These data will be provided by the clinical team lead as a report approximately 
monthly.  
Systems Use Outcomes (Time 1, 2, 3, 4). Systems use outcomes will include: CRC use, 
Emergency Department use, and psychiatric hospitalizations.  We will collect both self-report 
and health administrative data to assess for recall bias and to enable some systems use 
outcomes for individuals who are lost to follow-up in the study. At baseline, systems use in the 
3 months prior to admission will be collected. At time 1 and 2, we will collect systems use since 
discharge from the MH vWard. At time 4, we will collect prior 1-month utilization, including if 
they utilized the MH vWard again during that time period. Self-report systems variables will be 
collected using a modified version of the Service Utilization based on the Client Service Receipt 
Inventory (CSRI). Corresponding health administrative data will be extracted from the 
electronic medical records of participants by a research team member. Reliable health 
administrative data will only be available for Winnipeg-based participants as few rural health 
care facilities utilize the available electronic medical records (most are paper-based).  
Time Points 
Data will be collected at 4 different time points: 
Time 1 (baseline): 
•
Participants are to complete baseline questionnaires within 72 hrs of admission
•
If a participant does not complete questionnaires within 72 hrs of admission, the
participant will be excluded from further data collection.
Time 2 (primary outcome): 
•
Participants are administered questionnaires 1 week AFTER discharge, with + 1 week of
leeway for completion (within 7-14 days of discharge).
Time 3 (secondary outcome): 
•
Participants are administered questionnaires 5 weeks AFTER discharge, with +/- 1 week
of leeway for completion (within 4-6 weeks of discharge).
Time 4 (secondary outcome): 
•
Participants are administered questionnaires 6 months AFTER discharge, with a +/- 2
weeks of leeway for completion (within 6 months -2 weeks and 6 months +2 weeks).
Time 1 
(admission) 
Discharge 
Time 2 
(1 week post 
discharge) 
Time 3 
(5 weeks 
post 
discharge) 
Time 4 
(6 mos post 
discharge) 
Demographic 
data 
(Demographic 
Questionnaire; 
SINCERE) 
X 
Recovery 
outcomes 
(WHO-5, 
X 
X 
X 
X 

8 
Virtual Ward Cohort Study Protocol 
Version date: January 31, 2024 
MARS-12; 
Global 
Recovery 
Questions) 
Service 
Delivery 
outcomes 
(Data capture 
sheet) 
X 
Systems use 
outcomes 
(Service 
Utilization; 
data capture 
sheet) 
X 
X 
X 
X 
•
For self-report surveys at Time 1, 2 and 3, data may be collected through the Telus HHM
App as they will also be utilized for assessment and care planning purposes (for those
utilizing the App as part of their care), RedCap (for those not utilizing the App), or by
phone or in person, for those with this preference or who lack an email/internet
connection. All participants will complete questionnaires at Time 4 on REDCap.
Participants completing questionnaires electronically (Telus, REDCap) will receive up to 2 
reminders by phone or email  at Time 2, 3, and 4, if needed.  
Participants completing questionnaires over the phone or in person with the RA will receive 
up to 2 phone calls to connect for the purpose of completing the questionnaire together.  
All data entered into the Telus HHM App for consenting participants will be output in an 
electronic report by a clinical team member and shared with the research team. The research 
team member will then manually enter these responses into the REDCap survey.   
Data Storage 
Clinicians will upload PDF reports containing questionnaire responses generated from the 
Telus App into a shared folder on the Shared Health Network drive, accessible by the PI. The PI 
will generate hard copies of the reports for the RA to extract the data from. Once data are 
manually entered into REDCap and verified, hard copies will be destroyed using confidential 
shredding. All responses will be stored in REDCap along with identifying data and consent 
information.  
Statistical Analyses 
Data collected on the cohort will be quantitatively analyzed using descriptive statistics 
to summarize the baseline characteristics and service delivery outcomes. Change in primary 
outcome measure (assuming a continuous measure) over time will be assessed with a repeated 
measures ANOVA using a general linear model with sociodemographics, clinical variables, and 
length of stay entered as time-invariant covariates. Maximum likelihood estimation will be 

9 
Virtual Ward Cohort Study Protocol 
Version date: January 31, 2024 
used to calculate marginal means at each time point for the same cohort of MH vWard 
participants. Significant main effects of time will be statistically evaluated. Secondary outcomes 
will be assessed with similar methods depending on types of variables. Depending on the 
number of outcomes and relationships between outcomes, multivariate modelling may be 
used. 

10 
Virtual Ward Cohort Study Protocol 
Version date: January 31, 2024 
References 
1. Castillo A, Shterenberg R, Bolton J, Dewa C, Pullia K, Hensel J. Virtual Acute Psychiatric
Ward: Evaluation of Outcomes and Cost Savings. Psychiatric Online. April 2023. Doi:
https://doi.org/10.1176/appi.ps.20220332
2. Pullia K, Clavio A, Bolton JM, Hunzinger E, Svenne S, Hensel J (revisions submitted).
Providing Crisis Care in a Pandemic: A Virtual Based Crisis Stabilization Unit. Submitted
to: Frontiers in Psychiatry.
3. Palay J, Taillieu TL, Afifi TO, et al. Prevalence of Mental Disorders and Suicidality in
Canadian Provinces. Can J Psychiatry. Nov 2019;64(11):761-769.
doi:10.1177/0706743719878987
4. Butler A, Adair C, Jones W, et al. Towards quality mental health services in Canada: a
comparison of performance indicators across 5 provinces. Vancouver, BC: Centre for
Applied
5. Shared Health (2018). Shared Health Mental Health Program STATs 2018.
6. Arsenault-Lapierre G, Henein M, Gaid D, Le Berre M, Gore G, Vedel I. Hospital-at-home
interventions vs in-hospital stay for patients with chronic disease who present to the
emergency department: A systematic review and met-analysis. JAMA Network Open.
June 2021. Doi: 10.1001/jamanetworkopen.2021.11568
7. dosReis, S. C., Alejandro, A.-R., & Slejko, J. F. (2022). Health Technology Assessment:
Evidence Generation Methods For Patient-Driven Values. Health Affairs. Retrieved from:
https://www.healthaffairs.org/do/10.1377/forefront.20220311.897007
8. Frank L, Basch E, Selby JV, Institute P-COR. The PCORI perspective on patient-centered
outcomes research. JAMA. Oct 15 2014;312(15):1513-4. doi:10.1001/jama.2014.11100
9. Bhattacharyya O, Hayden L, Hensel J. Health Services and Designing for Uncertainty:
How a “lean startup” approach can help create an effective community-based program.
Stanford Social Innovation Review. 2015;doi:10.48558/kw2z-6a08
10. Gilbody S, Littlewood E, Hewitt C, Brierley G, Tharmanathan P, Araya R, et al.
Computerised cognitive behaviour therapy (cCBT) as treatment for depression in
primary care (REEACT trial): large scale pragmatic randomised controlled trial. Br Med J
2015;351:h5627
11. Topp C, Ostergaard S, Sondergaard S, Bech P. The WHO-5 well-being index: A systematic
review of the literature. Psychotherapy and Psychosomatics. 2015. Doi:
10.1159/000376585
12. Newnham E, Hooke G, Page A. Monitoring treatment response and outcomes using the
World Health Organization’s wellbeing index in psychiatric care. Journal of Affective
Disorders. April 2010. Doi: 10.1016/j.jad.2009.06.005
13. Drapalski A, Medoff D, Unick G, Velligan D, Dixon L, Bellack A. Assessing recovery of
people with serious mental illness: Development of a new scale. Psychiatric Services
63.1 (2012): 48-53.