JMIR FORMATIVE RESEARCH

Toh et al

Original Paper

Mobile Behavioral Health Coaching as a Preventive Intervention
for Occupational Public Health: Retrospective Longitudinal Study
Sean Han Yang Toh, BSocSci; Sze Chi Lee, MSocSci; Oliver Sündermann, PhD
Intellect Private Limited Company, Singapore, Singapore

Corresponding Author:
Oliver Sündermann, PhD
Intellect Private Limited Company
171 Tras St, #02-179 Union Building
Singapore, 079025
Singapore
Phone: 65 93571995
Email: oliver@intellect.co

Abstract
Background: Researchers have recently proposed that behavioral health coaching (BHC) is effective in promoting proactive
care among employees. However, to qualify as a preventive workplace intervention, more research is needed to evaluate whether
BHC can further elevate well-being among moderately mentally healthy employees.
Objective: Using real-world data, this study evaluates the preliminary effectiveness of app-based BHC against a nonrandomized
control group with open access to self-help tools in improving well-being (ie, mood levels and perceived stress). The study also
explores the active ingredients of BHC and dose-response associations between the number of BHC sessions and well-being
improvements.
Methods: Employees residing across Asia-Pacific countries (N=1025; mean age 30.85, SD 6.97 y) who reported moderately
positive mood and medium levels of perceived stress in their first week of using the mental health app Intellect were included inthis study. Users who were given access by their organizations to Intellect’s BHC services were assigned to the “Coaching”
condition (512/1025, 49.95%; mean age 31.09, SD 6.87 y), whereas other employees remained as “Control” participants (513/1025,
50.05%; mean age 30.61, SD 7.06 y). To evaluate effectiveness, monthly scores from the validated mood and stress sliders were
aggregated into a composite well-being score and further examined using repeated-measure conditional growth models. Postcoachingitems on “Perceived Usefulness of the BHC session” and “Working Alliance with my Coach” were examined as active ingredientsof BHC using 1-1-1 multilevel mediation models. Finally, 2-way repeated-measure mixed ANOVA models were conducted to
examine dose-response effects on well-being improvements between groups (coaching and control) across time.
Results: Growth curve analyses revealed significant time by group interaction effects for composite well-being, where “Coaching”
users reported significantly greater improvements in well-being than “Control” participants across time (composite well-being:
F1,391=6.12; ηp2=0.02; P=.01). Among “Coaching” participants, dependent-sample 2-tailed t tests revealed significant improvements
in composite well-being from baseline to 11 months (t512=1.98; Cohen d=0.17; P=.049). Improvements in “Usefulness of the
BHC session” (β=.078, 95% Cl .043-.118; P<.001) and “Working Alliance” (β=.070, 95% Cl .037-.107; P<.001) fully mediatedwithin-level well-being enhancements over time. Comparing against baseline or first month scores, significant time by groupinteractions were observed between the second and sixth months, with the largest effect size observed at the fifth month mark(first month vs fifth month: F1,282=15.0; P<.001; ηp2=0.051).
Conclusions: We found preliminary evidence that BHC is an effective preventive workplace intervention. Mobile-based coaching
may be a convenient, cost-effective, and scalable means for organizations and governments to boost public mental health.
(JMIR Form Res 2023;7:e45678) doi: 10.2196/45678
KEYWORDS
mobile health apps; mHealth apps; behavioral health coaching; behavioral coaching; app-based coaching; self-help; employees;
well-being; mood; stress; public health; preventive interventions; positive psychology

https://formative.jmir.org/2023/1/e45678

XSL• FO
RenderX

JMIR Form Res 2023 | vol. 7 | e45678 | p. 1
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Introduction
Background
Mental health conditions in the workforce are increasingly
common and deleterious to employees’ well-being, productivity,
presenteeism, compensation claims, and many otheroccupational outcomes [1-3]. Over the last 2 decades, annual
productivity losses caused by these conditions have amounted
to approximately US $225 billion for all organizations in the
United States [4], £13 billion (US $16.3 billion) in the United
Kingdom [5], and US $1.4 to US $1.8 billion across Asia-Pacificcountries [6]. To date, most efforts to reduce the burden of these
conditions have focused on implementing reactive measuresand ensuring immediate treatment for those with manifest
difficulties [7] but not proactive care [8]. Not only are these
employee assistance programs (EAPs) typically costly andheavily underused (eg, <1% of 10,000 Asian employees attended
EAPs [9]; uptake rates are 2.6%-3.5% on average across 133
EAPs in Canada [10]), but cost-effective models suggest that
only a third of the overall burden would be alleviated even if
treatment were to be delivered successfully to all struggling
employees [11]. Consequently, maximizing employees’
well-being through proactive and preventive care may befundamental to reducing these cost burdens [12].
Recent studies have supported the effectiveness of lower-cost
preventive programs in improving well-being across generalizedworking populations [13]. These interventions can range fromfully self-guided mobile interventions [14-19] to personalized
behavioral health coaching (BHC) services [20-22]. Contrary
to the mixed findings on self-help features [23], BHC
interventions have shown consistent improvements in multiple
individual-level outcomes. A recent meta-analysis [24] of 18
BHC studies conducted in different samples (ie, students,
employees, and the community) revealed significant
enhancements in work performance (Hedges g=0.19; n=6) andattitudes (Hedges g=0.54; n=7), coping self-efficacy (Hedges

Toh et al
g=0.43; n=10), stress and affectivity (Hedges g=0.46; n=10),
and self-regulation (Hedges g=0.74; n=11). These findings werelargely similar to those of a more recent systematic review(n=18) that examined the impact of coaching in purelyorganizational settings [25]. This review also demonstrated
small effect sizes on skill-based outcomes (ie, leadership skills,
technical skills, and general competency; Hedges g=0.26; n=10),
medium effect sizes on well-being (eg, positive emotions,self-efficacy, life satisfaction, and reduction in stress; Hedges
g=0.46; n=10), and large effect sizes on organizational outcomes(eg, employees’ goal attainment and returns on investments;
Hedges g=1.15; n=3).
By definition, BHC strives to achieve these benefits through avariety of evidence-based approaches including cognitive
behavioral therapy (CBT), mindfulness, solution-oriented focus,
and positive psychology [26,27]. On the basis of a collaborative,
results-oriented relationship, behavioral health care providers
are well positioned to address emotional difficulties or life
challenges and facilitate positive behavior changes [28,29].
During the course of BHC, well-being enhancements were
believed to be a central tenet in building enduring personal
resources (physical, intellectual, social, and psychological) [30],which in turn help clients improve their coping abilities andgoal striving after each session [31]. The experience of increased
positive emotions often leads to the broadening and building ofthought-action repertoires. This strengthens metacognition andcreativity, which then enables the client or employee toovercome challenges more efficiently and further incites anupward spiral of positive affect and widely improves outcomes[31]. In a similar fashion, decreases in perceptions of stresssignificantly predicted proactive and preventive coping with
future work challenges [32]. Such experiences free up cognitiveand self-regulatory resources [33], motivating the client or
employee to be further engaged with performance andself-actualization in coaching. Integrating these findings, Figure
1 illustrates these mechanisms.

Figure 1. Well-being improvements, a key facet in driving effectiveness of coaching.

Despite growing research on the general effectiveness of BHC,
potentially important questions related to its preventive
effectiveness still remain. Grant et al [34] observed 3 large-scalecoaching studies whose participants demonstrated
higher-than-average levels of psychopathology [35]. Even
though these participants were recruited from nonclinicalcommunity samples, at least a quarter of them displayed
clinically elevated scores for a variety of psychiatric symptoms[36,37], psychological distress, anxiety, and depressive
symptoms at baseline [34]. In supporting the observation by
Grant et al [34], most of the studies reviewed by the recent
meta-analyses either did not exclude employees at clinically
https://formative.jmir.org/2023/1/e45678

XSL• FO
RenderX

significant risk [38] or did not explicitly state any exclusion
criteria [39-41]. Possibly, research is still at an early stage inempirically evaluating BHC as a preventive intervention for
positive well-being that tries to sustain or promote mental
wellness in healthy individuals at the public health level [42].
This lack of differentiation between psychotherapy and coachingcan potentially be detrimental to clients’ well-being astechniques normally executed in BHC (eg, solution-focusedstrategies and expressive-experiential strategies) may
inadvertently cause minimal harm to some at-risk clients. The
failure to address clinical levels of stress, anxiety, or depressionin BHC may have escalated mental health problems and
JMIR Form Res 2023 | vol. 7 | e45678 | p. 2
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
employee burnout previously [43]. Evaluating the success of
BHC as a preventive workplace intervention may then help
promote this differentiation. Second, although increments inwell-being may catalyze personal resource development during
the course of BHC, few studies to our knowledge have examined
BHC features that directly cultivate well-being. Working
alliance, coach and client personalities, clients’ perceived
usefulness of the coach, and clients’ expectancy werecoaching-related factors that directly facilitated clients’ resource
development and eventually coaching success [44-47]. A
plausible question is which of these factors would also cultivate
well-being [48,49]. Identifying these predictors may provide
insights on how training for behavioral health care providers
can be further enhanced or tailored to maximize client outcomes.
Another relatively unexplored dimension is estimating thedose-response relationship between the number of BHC sessions
and well-being improvements. The few coaching studies on
preclinical employee samples demonstrated significant
improvements in various well-being outcomes after 4 to 6
sessions [50,51]. Other studies have revealed similar
improvements after 30 to 40 days on average but without
knowing the number of sessions [52]. Clearer knowledge on
the short- (ie, <4 sessions) and long-term (ie, >6 sessions) effectsof BHC can inform expectations and guide treatment decisions
for organizations and employees.

Objectives
The primary objective of this retrospective longitudinal study
was to evaluate the preventive effectiveness of a BHC
intervention against a no-coaching control group using a sampleof employees with at least moderate mental health over a
standardized period of 12 months. To our knowledge, this is
the first study that tested the effectiveness of app-based BHC
on moderately healthy employees. Should BHC participants
show significant improvements in well-being overtime, we also
explored whether clients’ “perceived usefulness of BHC session”
and their “working alliance” with the coach mediated thesechanges. Finally, we also examined the dose-response
relationship between the number of BHC sessions andimprovements in well-being, if any.

Methods
Study Design
This retrospective study used existing data from pilotcommercial collaborations between Intellect and its business

https://formative.jmir.org/2023/1/e45678

XSL• FO
RenderX

Toh et al
partners. Interested employees were enrolled from October
2020, when Intellect’s coaching platform was launched, toOctober 2022.

Participants
The Intellect app is publicly available on the Google Play andApple App stores and has been downloaded >3,500,000 times
since its release in October 2019. Interested employees fromthe commercial pilots voluntarily downloaded Intellect throughthese app stores after consenting to Intellect’s terms of service
and privacy policy. All users were instructed to set up theirprofiles by completing an onboarding survey regarding theirage and gender, the name of their organization, and theirpersonal goals for using Intellect. Although all registered users
had access to all self-care features, only a subsample of themgained additional access to BHC services depending on theirorganization. Registered users were further screened for study
eligibility, which included (1) being aged ≥18 years and (2)having scored an average daily rating of “≥2” on the mood slider
and “≤3” on the stress slider (see the following section) in theirfirst week of use. To compute monthly changes in well-being,
eligible participants were also expected to (3) attempt thewell-being scales every 4 weeks. Users in the “Coaching”
condition would (4) have participated in at least one BHC
session every month, whereas the remaining users in the“Control” condition may or may not have engaged withIntellect’s self-care features during the month. All “Coaching”
users were prompted to complete a 4-item postcoaching
feedback form after each BHC session. In accordance withIntellect’s risk assessment procedures, it may be reasonable toassume that these “Coaching” users were “mentally healthy.”
As a qualifying criterion, each behavioral health coach would
have been trained to detect symptoms of anxiety and depression
and accumulated a minimum of 300 client hours before they
were employed at Intellect. Users who were identified as
showing elevated symptoms of anxiety and depression in thefirst session based on each provider’s professional judgment
would be referred out of the BHC program to see an internal(in-app) or external licensed therapist. These users were notconsidered as having completed 1 coaching session. Therefore,
they are not present in this retrospective cohort. The overall
participant flowchart is shown in Figure 2. The resulting sample
of 1025 participants comprised predominantly women with amean age of 30.85 (SD 6.97; range 18-61) years. To maintainthe real-world evaluation of this data set, participants received
neither any in-app reminders to engage with the app normonetary reimbursement for their participation.

JMIR Form Res 2023 | vol. 7 | e45678 | p. 3
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Toh et al

Figure 2. Participant allocation chart.

Intervention
Self-Guided Features
Intellect is a consumer-based mental health app that provides
all registered users with free access to a variety of self-guided
features, some of which have been validated in previous
randomized controlled trials [18,19,53,54]. On the “Home” tab,participants can access any of the 3 self-help features: “Learning
Paths,” “Rescue Sessions,” or “Guided Journals” (Figure 3).
“Learning Paths” contains psychoeducation on different aspectsof mental health. An example is the “Self-Esteem LearningPath,” where the program guides users to identify andcognitively restructure their negative thoughts related toself-perceptions. A “Rescue Session” targets a specific themeof adversity, such as “procrastination,” “irregular sleep,”“burnout,” or “loneliness.” Depending on the adversity, thetheoretical foundations of these sessions can range fromhttps://formative.jmir.org/2023/1/e45678

XSL• FO
RenderX

mindfulness and self-compassion practices to CBT and active
behavior change techniques (eg, self-monitoring and social
support). Intellect’s “Guided Journals” comprise 6 themes:“gratitude,” “reflection,” “problem-solving,” “goal-setting,”“sleep,” and “self-affirmation.” Each journal provides specificguidance to each participant on how to write a relevant entry.
For example, the “gratitude” journal encourages each participant
to recall something that they can be thankful for on that day but
not usually on other days. The fourth feature, “Toolkit,” maybe most suitable for participants who prefer a standardized
schedule for self-care. This feature, which is accessible on the“Daily” tab, contains brief mental and physical exercises
dedicated to 3 parts of the day (Figure 3). The morning section
includes deep breathing exercises guided by therapist-led audio
messages. The afternoon section provides grounding techniquesthat increase mindful focus on the users’ surroundings. Finally,
the evening section focuses on improving sleep throughmeditation and cognitive defusion.

JMIR Form Res 2023 | vol. 7 | e45678 | p. 4
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Toh et al

Figure 3. Intellect’s self-help exercises (learning paths, rescue sessions, toolkits, and journaling).

BHC Component
Intellect also offers personalized one-on-one app-based coaching
to support each participant’s personal and professional
development. Behavioral health care providers across the
Asia-Pacific region are connected to the platform, where eachof them receives professional certification either in counselingfrom registered institutions or in coaching licensed by theInternational Coaching Federation [55]. The former grouphttps://formative.jmir.org/2023/1/e45678

XSL• FO
RenderX

constitutes approximately 85% of all Intellect’s providers.
Specifically, these coaches are accredited at a minimum with amaster’s degree in counseling and are officially licensed bytheir national counseling entities (eg, the Singapore Association
for Counselling) to practice professionally. A minimum of 300
client hours are also required to qualify as a behavioral healthcoach for Intellect. Intellect’s diverse coaching network meansthat providers have different specialties. Most of Intellect’s
providers are trained in CBT or problem-solving strategies and
JMIR Form Res 2023 | vol. 7 | e45678 | p. 5
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
third-wave CBT approaches such as mindfulness, dialecticalbehavior therapy, or acceptance and commitment therapies.
Participants supplied with “coaching” credits by theirorganizations gain access to BHC services via the “Coach” tab.
Upon accessing the tab, users are automatically directed to select3 areas of well-being that they may be interested to work on
with a coach (ie, “anxiety & worry,” “career & work,”“emotional regulation,” “low mood,” “health and lifestyle,”“relationships,” “self-confidence,” and “stress & burnout”) andtheir preferred language of communication. Coach selection isthen optimized through a set of algorithms that makerecommendations based on the participants’ motivations, goals,
app engagement patterns, preferred language, and coachingstyle. Each coaching session, which lasts approximately 30
minutes, is conducted via video between the coach and the user
on the same tab. Typically, in the first session, the coachattempts to fully understand each user’s topical struggles andformulate clear goals for the coaching journey. For subsequentsessions, the coach is free to explore and select theevidence-based modalities and exercises that they think would
work best to fulfill the user’s set goals and presenting struggles.
Hence, the delivery of Intellect’s BHC program is flexible andnot in a standardized modular format. Between each session,
the coach and user may continue to interact via SMS text

https://formative.jmir.org/2023/1/e45678

XSL• FO
RenderX

Toh et almessage and audio messaging, usually to clarify homework or
provide additional motivational support. Coaching sessions are
typically arranged a minimum of a week apart to allow for moretime for implementing coaching exercises and reflection. Thesefeatures are illustrated in Figure 4.
The collaborative and supportive relationship between the coachand client is thought to be critical in attaining valued outcomes[56]; hence, all providers were trained in areas of active
listening, goal setting, empowerment, scaffolding behavior
changes, providing constructive feedback, emotional validation,and maintaining accountability of change. During a typicalsession, the user may be encouraged to lead discussion topics
related to the mutually agreed upon goals as this increases
responsibility over their personal development at a comfortablepace. In response, the coach takes a future-focused stance whenfacilitating the discussion, empowering the user to discoverpotentially novel solutions. As a form of preventive care, BHC
actively seeks to develop growth mindsets toward positive
behavior change. Once this is accomplished within the session,both the coach and user proceed to formulate a feasible action
plan beyond the session. The coach continues to provide support
in monitoring progress and reframing unhelpful thinking styles
while emphasizing accountability of change for the user.

JMIR Form Res 2023 | vol. 7 | e45678 | p. 6
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Toh et al

Figure 4. Screenshots of Intellect’s behavioral health coaching, accessible on the Coach tab.

Measures
Overview
Participants rated their emotions and perceived stress levels
using a mood and stress slider, respectively. These sliders werelocated at the top of the “Home” tab. Participants were able toredo their ratings at any period of the day, and these scores wererefreshed on a daily basis.

https://formative.jmir.org/2023/1/e45678

XSL• FO
RenderX

Perceived Stress
The self-developed stress slider was rated on a 5-point scale(1=“Very Low,” 2=“Low,” 3=“Medium,” 4=“High,” and5=“Very High”). Previously, we examined the convergent
validity of this self-developed item using a separate large sample
of working professionals (N=997) recruited from the web-based
Rakuten platform in an unpublished study. The Rakuten platformfunctions similarly to the Amazon Mechanical Turk platformin the United States. We found significant positive correlations
JMIR Form Res 2023 | vol. 7 | e45678 | p. 7
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH(r=0.390; P<.001) and acceptable factor loadings (λ=0.68)
between the stress slider and Perceived Stress Scale items [57].
On this scale, we considered scores of ≥4 to be “unhealthy” andscores of ≤3 to be “healthy.” This is consistent with ourscreening criterion.
Mood Levels
Similar to the stress slider, the self-developed mood slider was
rated on a 5-point scale (0=“Terrible,” 1=“Very Bad,”
2=“Alright,” 3=“Good,” and 4=“Fantastic”). Higher total scoresindicate a more positive mood. Previously, significant negative
correlations between the mood slider and Patient Health
Questionnaire–4 items (r=−0.501; P<.001) and high factor
loadings (λ=0.72) indicated good support for convergent validity[58]. On this scale, we considered scores of ≥2 to be “healthy”
and scores of ≤1 to be “unhealthy.” This is also consistent with
our screening criterion.
Composite Well-Being Score
As the use of single-item measures may be less reliable inassessing a multidimensional construct such as well-being [59],participants’ scores on the mood and stress sliders were
aggregated to form a composite well-being score. All stressscores were reverse coded such that higher aggregate scores oncomposite well-being indicated greater mental health (ie, bettermood and lower stress). In line with the scoring systems of the2 items, we considered scores of ≥5 (ie, 3 on stress and 2 onmood) to be “healthy” and scores of ≤4 to be “unhealthy.”
Postcoaching Feedback
To evaluate acceptability, “Coaching” users were prompted tocomplete a self-developed 4-item feedback form immediately
after a coaching session. A total of 2 items on “Perceived
Usefulness of Coaching” asked whether participants (1) foundthe session helpful and (2) felt closer to their personal goals
after the session. The remaining 2 items on “Perceptions ofWorking Alliance” asked whether (1) participants felt supported
and understood by their respective coaches and (2) the coachgenerally helped them gain new insights and skills. Each item
is rated on a 5-point scale (1=“Strongly Disagree,” 3=“Neutral,”and 5=“Strongly Agree”). Among the coaching users, this form
showed excellent internal consistency (Cronbach α=.81).

Statistical Analyses
Descriptive statistics, including the average retention period foreach group, were calculated using SPSS Statistics (version 28.0;
IBM Corp) [60]. For intervention effects on primary outcomes,a mixed model for repeated-measure conditional growth model
analysis was conducted to test whether differences in the change
trajectories on well-being between participants could be
explained by group membership (condition). Analyses wereconducted using the lme4 package on R (R Foundation for

https://formative.jmir.org/2023/1/e45678

XSL• FO
RenderX

Toh et al
Statistical Computing) [61]. The use of multilevel mixed models
allows for not only missing data but also time-varying covariates[62]. For samples with high attrition, multilevel models can beapproximated using maximum likelihood estimators. Theseestimators provide unbiased parameter estimates as long as themissing or dropout data are random [63-66]. As the intervention
was conducted in observational (nonexperimental) settings, it
is expected for attrition rates to be very high in this sample.
Hence, the use of multilevel models may be a reliable method
to estimate the true intervention effect of Intellect’s BHC
services against a “no-coaching” control group over the periodof evaluation (ie, 12 mo) [67]. In the mixed model, group, time,
time by group interaction, and covariates (variables presenting
significant differences between the “Coaching” and “Control”
groups at baseline, demographic characteristics, and number oftimes each self-guided feature [ie, learning paths, rescue
sessions, journaling sessions, and toolkits] was engaged
with/mo) were fitted as fixed effects in the model. Participants
were included as random intercepts. The “Group” variable wascoded binarily (0=“Control”; 1=“Coaching”), whereas “Time”
indicates the number of months since participants started using
the app. Before these variables were included in the model ormodels, mean centering was necessary to compute a moreaccurate cross-level interaction by separating within- frombetween-subject effects [68,69]. “Group,” “Time,” anddemographic variables (age and gender) were all grand meancentered to provide meaning to their intercepts. The grand meancentering of “Group” and “Time” also reduces multicollinearity
with their interaction term through decorrelating predictor
variables [70]. As purely within-subject variables, frequencies
of engaging with each self-guided feature were group meancentered.
To examine whether changes in “perceptions of coach” and“perceived usefulness of the BHC session” mediated therelationship between time and primary outcomes for the“Coaching” users, 1-1-1 multilevel mediation models [71] were
estimated using the MLmed function in SPSS [72]. Similar tolinear mixed models, multilevel mediation allows all participants
to be included regardless of missing data. As mentioned
previously, the restricted maximum likelihood estimators ofthese models also allow for the approximation of unbiased
parameter estimates in the presence of high attrition. Essentially,
multilevel mediation models have the advantage to exploit all
valid data points [73]. They also extend classic mediation
models to account for the nested nature of longitudinal data (ie,time nested within individuals). Mediators were group meancentered to evaluate within-subject effects, whereas parameters
were estimated using restricted maximum likelihood estimation.
Each model also included the same covariates as in the linear
mixed models. Figure 5 illustrates the 1-1-1 multilevel mediation
model.

JMIR Form Res 2023 | vol. 7 | e45678 | p. 8
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Toh et al

Figure 5. Illustration of the 1-1-1 mediation model. Path a represents the effect of time on clients’ perceptions of behavioral health coaching (BHC).
Path b represents the changes in well-being when client ratings of BHC change and time is held constant. Path c′ represents the direct effect of time on
well-being when controlling for the effect of changes in clients’ perceptions. Path a*b represents the within-group indirect effect of time on well-being
represented by changes in the clients’ perceptions of BHC.

Finally, to test for well-being improvements between specifictime points, 2 × 2 repeated-measure mixed ANOVA models
with time (first month [T1] and second, third, fourth, fifth, sixth,
seventh, eighth, ninth, 10th, 11th, or 12th month [T2]) as thewithin-subject factor and group (coaching and control) as the
between-subject factor were conducted to assess the differencein participants’ well-being scores. In each ANOVA model,
demographic variables (age and gender), quantity, and type ofself-guided feature engagement were controlled as covariates.
A significant time by group interaction effect indicates groupdifferences in improvement over the 2 time points. Any
significant interaction effects were further examined using
Bonferroni-adjusted contrast analyses to assess the locus ofdifferences for each group across the time points.

Ethical Considerations
This study received an advisory review from the Advarra
Institutional Review Board and was deemed to be of no morethan minimal risk (protocol Pro00069682). Employees
downloaded the app after having agreed to Intellect’s terms ofservice and privacy policy, which included consent to use
anonymized data for research purposes. All retrospective cohort
data were deidentified.

Results
Participants
Of the total 1025 users, participants were predominantly woman(n=626, 61.07%; mean age 30.85, SD 6.97 y) who completed
close to 5 monthly assessments on average (mean 4.92, SD
6.07). At baseline, the average participant experienced moderate

https://formative.jmir.org/2023/1/e45678

XSL• FO
RenderX

levels of perceived stress (mean 2.39, SD 0.762) and mood(mean 2.98, SD 0.966). Within the control group, 71.3%(366/513) of the participants scored moderately for mood,
whereas the remaining 27.1% (139/513) and 1.6% (8/513)
experienced “Good” and “Fantastic” mood at baseline,
respectively. Similarly, 41.5% (213/513) of the control
participants experienced moderate levels of stress, whereas theremaining 47.4% (243/513) and 11.1% (57/513) scored “Low”
and “Very Low” for baseline stress levels, respectively. Within
the intervention group, 68.8% (352/512) of the participants
experienced moderate mood levels, whereas the remaining 25%(128/512) and 6.3% (32/512) scored “Good” and “Fantastic”
for mood levels, respectively. For stress levels, 58.4% (299/512)of the intervention participants scored “Moderate,” whereas
35.5% (182/512) and 6.1% (31/512) scored “Low” and “VeryLow” at baseline, respectively. Table 1 presents the main
descriptive statistics and baseline scores of the groups. Apartfrom the lower number of monthly assessments completed on
average by “Control” participants (mean difference −1.76, SD
0.180; t4662=−9.82; P<.001), independent-sample 2-tailed t tests
and chi-square analyses did not reveal significant differencesin other baseline characteristics (see Table 1).
Independent-sample 2-tailed t tests and chi-square analyses also
revealed no significant differences in any demographic (ageand gender) measures between participants who retained their
engagement with Intellect and those who withdrew from theapp over the 12 months (P>.05 in all cases). This result alsosuggests that missing data were random and the use of restrictedmaximum likelihood estimators in our multilevel models wasadequate [66].

JMIR Form Res 2023 | vol. 7 | e45678 | p. 9
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Toh et al

Table 1. Baseline descriptive statistics for demographics (N=1025).
Variable

Coaching (n=512)

Control (n=513)

P valueAge (y), mean (SD)31.09 (6.87)

30.61 (7.06).27

Monthly assessments completed, mean (SD)5.62 (6.47)

3.86 (5.24)<.001

Perceived stress2.39 (0.729)

2.39 (0.792).92

Positive mood

2.94 (1.07)

3.02 (0.852).48

Composite well-being

5.64 (0.151)

5.43 (0.164).35

Well-being (averaged across the first week), mean (SD)

Gender, n (%)

.16

Woman

326 (63.7)

300 (58.5)

Man

170 (33.2)

194 (37.8)

Nonbinary

7 (1.4)14 (2.7)

Other

4 (0.8)5 (1)Engagement With Self-Guided Features
As shown in Table 2, there were no significant differences in
the average number of self-guided features engaged with permonth between the 2 groups (P>.05 in all cases). Relative tothose in the “Control” group (mean 5.18, SD 21.5), “Coaching”
participants engaged with and completed an average of 5.06
(SD 9.54) self-guided features per month. Similarly, there were
almost no significant differences in the average number offeatures engaged with per month between the 2 groups except
for the first month (learning paths: mean difference −0.098, SD

0.046, t1022=−2.13; P=.03; journaling sessions: mean difference−0.293, SD 0.118, t1022=−2.48; P=.01; rescue sessions: meandifference −0.502, SD 0.132, t1022=−3.81; P<.001). Only atbaseline were “Coaching” participants significantly moreengaged with nearly all self-guided features (learning paths:
mean 0.82, SD 0.755; journaling sessions: mean 0.88, SD 2.04;
rescue sessions: mean 1.08, SD 2.63) than “Control” participants(learning paths: mean 0.72, SD 0.714; journaling sessions: mean
0.59, SD 1.73; rescue sessions: mean 0.58, SD 1.40). The
descriptive results of each self-guided feature can be found in
Tables S1-S4 in Multimedia Appendix 1.

Table 2. Monthly average number of self-guided features engaged with between the groups.
Period by month

Coaching (n=512), mean (SD)Control (n=513), mean (SD)P valueFirst month (N=1025, 100%)

4.59 (6.87)

3.79 (11.7).19

Second month (N=1025, 100%)

3.94 (7.99)

3.08 (11.7).17

Third month (n=709, 69.17%)

3.60 (8.40)

3.89 (15.3).75

Fourth month (n=456, 44.49%)

4.08 (8.57)

5.91 (20.3).18

Fifth month (n=301, 29.37%)

4.89 (9.92)

7.51 (26.2).21Sixth month (n=218, 21.27%)

5.27 (10.3)

7.25 (30.5).48

Seventh month (n=156, 15.22%)

6.79 (12.0)

11.5 (37.3).23

Eighth month (n=123, 12%)

6.61 (11.8)

12.5 (42.1).23

Ninth month (n=92, 8.98%)

8.37 (17.3)

16.8 (52.5).25

10th month (n=75, 7.32%)

7.31 (11.2)

20.9 (57.9).09

11th month (n=60, 5.85%)

7.50 (10.1)

21.5 (59.8).1412th month (n=53, 5.2%)

7.63 (11.8)

26.9 (79.9).14Effects of BHC on Well-Being
Overview
The fixed effects of time, group, time × group interactions, andcovariates (ie, self-guided features) are presented in Table 3.
Significant effects of time × group interaction were found forcomposite well-being, indicating that change patterns forwell-being scores differed significantly between the “Coaching”
and “Control” participants across time (well-being: F1,391=6.12;
https://formative.jmir.org/2023/1/e45678

XSL• FO
RenderX

ηp2=0.02; P=.01). Significant main effects were also found forgroup (well-being: F1,354=7.40; ηp2=0.02; P=.007), with the“Coaching” group generally exhibiting greater well-being thanthe “Control” group on average over the 12 months. There wasno significant main effect of time on well-being (F1,391=2.35;
P=.13). Simple slope analyses were conducted on SPSS to probe
the interaction effect, namely at high (1 SD above the mean at
11 mo) and low (1 SD below the mean at baseline) levels of“Time.” Figure 6 shows the best-fit lines indicating changes in
JMIR Form Res 2023 | vol. 7 | e45678 | p. 10(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Toh et al

well-being over the months as a function of group. Probing
revealed a significant improvement in composite well-beingfrom baseline to 11 months for the “Coaching” group (t512=1.98;

Cohen d=0.17; P=.049), whereas no significant changes wereobserved for “Control” participants over the same period(t513=−1.50; Cohen d=−0.13; P=.14).

Table 3. Fixed effects on well-being.
β (SE; 95% Cl)

Partial η2

P valueTime−.001 (.014; −.012 to .010)

0.00597

.13

Group

.362 (.103; .239 to .485)

0.02

.007

Time × group.067 (.027; .047 to .088)

0.02

.049

Learning paths.005 (.071; −.075 to .077)

0.0000409.55

Rescue sessions−.076 (.027; −.106 to –.052)

0.02

.005

Journaling.023 (.012; −.001 to .048)

0.00436.06

Toolkits.000 (.010; −.010 to .010)

0.00000256

.98

Age

.015 (.006; .008 to .023)

0.02

.01

Gender−.019 (.082; −.126 to .88)

0.000113.84

Primary outcome and effectWell-being

Figure 6. Changes in well-being over time (mo) as a function of group. BHC: behavioral health coaching.

Multilevel Mediation Models and Coaching Service
Satisfaction
Table 4 presents the path coefficients and the direct and indirect
effects of the postcoaching feedback items on well-being scores.
Among the large subsample of “Coaching” users who completed
the postcoaching feedback items (341/512, 66.6%; mean age
31.52, SD 7.21 y), these participants developed more positive
perceptions of their coach (ie, “Perceptions of My Coach”) andfound the BHC sessions to be increasingly useful (ie, “Perceived
Usefulness of Coaching”) over time. Increases in both
“Perceived Usefulness of Coaching” (well-being: β=.078, 95%
https://formative.jmir.org/2023/1/e45678

XSL• FO
RenderX

Cl .043-.118; P<.001) and “Working Alliance” (well-being:
β=.070, 95% Cl .037-.107; P<.001) fully mediated thewithin-level improvements in well-being during the intervention
period (Table 4). Across the sessions, 88.3% (301/341) and
84.2% (287/341) of these participants averaged a high score of≥8 out of 10 on “Perceived Usefulness of Coaching” and“Perceptions of Working Alliance,” respectively, implying ahigh level of satisfaction with the BHC intervention. The highaverage ratings on “Perceived Usefulness of Coaching” were
supported by high scores on its individual items. On“Helpfulness” of coaching, 0.6% (2/341), 5% (17/341), 15%(51/341), and 79.5% (271/341) of these participants averaged
JMIR Form Res 2023 | vol. 7 | e45678 | p. 11(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Toh et ala rating of 1, ≤3, ≤4, and ≤5 across the monthly sessions,
respectively. On “Goal Attainment,” 0.3% (1/341), 1.2%
(4/341), 12.9% (44/341), 26.1% (89/341), and 59.5% (203/341)of these participants averaged a rating of 1, ≤2, ≤3, ≤4, and ≤5
across the monthly sessions, respectively. Similarly, highmonthly average ratings for “Perceptions of Working Alliance”
were supported by high scores on its items. On “Supportivenessof my Coach,” 0.3% (1/341), 2.6% (9/341), 17.3% (59/341),
and 79.7% (272/341) of these participants averaged a monthly
rating of 1, ≤3, ≤4, and ≤5, respectively. Finally, on “Initiativeof my Coach,” 0.6% (2/341), 0.6% (2/341), 8.2% (28/341),
25.2% (86/341), and 65.4% (223/341) of these participants
averaged a monthly rating of 1, ≤2, ≤3, ≤4, and ≤5, respectively.

Table 4. Multilevel coefficients and indirect effects of postcoaching feedback items on well-being.
Mediator

Path aa

Path bb

Path c’cIndirect effects
Path “ab”d (95% Cl)aPerceived usefulness of coaching

0.118e0.664e−0.0620.078e (0.043-0.118)

Perceptions of working alliance0.105e0.671e−0.0570.070e (0.037-0.107)

The effect of time on the mediator (Perceived Usefulness of Coaching or Perceptions of Working Alliance).

b

The effect of the mediator on well-being.

c

The direct effect of time on well-being after controlling for the indirect effects of the mediator.

d

The indirect effects of the mediator on the relationship between time and well-being.

eP<.001.

Mixed ANOVAs (Exploratory Analyses)
Significant time by group interactions were observed between
the second and sixth months, contrasting with participants’
well-being scores at baseline or first month, with the largest
effect size observed at the fifth month mark (first month vssecond month: F1,1004=10.04; P=.001; ηp2=0.010; first monthvs third month: F1,690=9.23; P=.002; ηp2=0.013; first month vsfourth month: F1,436=11.3; P<.001; ηp2=0.028; first month vsfifth month: F1, 282=15.0; P<.001; ηp2=0.051; first month vssixth month: F1,200=7.01; P=.009; ηp2=0.034). There were no
significant time by group interactions between the seventh to12th month scores and baseline scores across both groups (P>.05
in all cases). Results from post hoc pairwise comparisonsrevealed significant increases in well-being from the first monthleading to the sixth month in the “Coaching” group (first monthvs second month: mean difference 0.231, SD difference 0.043;
P<.001; first month vs third month: mean difference 0.666, SDdifference 0.102; P<.001; first month vs fourth month: meandifference 0.237, SD difference 0.081; P<.001; first month vsfifth month: mean difference 0.420, SD difference 0.069;
P<.001; first month vs sixth month: mean difference 0.816, SDdifference 0.204; P<.001). In contrast, the “Control” participants
did not experience any significant well-being improvements
across any time points (P>.05 in all cases).

Discussion
Principal Findings
This study demonstrated the preliminary effectiveness of a
retrospective BHC intervention in improving the well-being ofrelatively mentally healthy employees. Consistent with the few
face-to-face BHC interventions that manifested several
well-being benefits (ie, self-reported physical health, burnout,
psychological distress, and positive and negative affect) in
https://formative.jmir.org/2023/1/e45678

XSL• FO
RenderX

employees with minimal mental health problems at baseline[74,75], our findings also showed that web-based, one-on-one
BHC may replicate some of these benefits. As face-to-faceprograms incur higher costs than web-based interventions,which, in turn, limits their scalability and feasibility in alarge-scale rollout [76], mobile-based or web-based coachingsessions may be a more convenient, cost-effective, and scalable
means for organizations and governments to promote public
mental health. In line with previous research [77], our results
also showed that having free access to self-help tools may notnecessarily be sufficient to sustain mental wellness among
healthy employees, as indicated by our finding that thewell-being levels of noncoaching users declined over time
(Figure 6). Previously, the role of human contact and therapeutic
alliance with a professionally trained coach has improved
participants’ adherence rates to the intervention [78,79]. Thisis also supported by the higher number of monthly assessmentscompleted by the average “Coaching” participant in this study.
Compared with coaching, self-help features were less able tobe contextualized to the needs of every individual user, which
may then lead to lower engagement motivation with the app[79,80]. Given that mentally healthy individuals are alreadyprone to perceive a lower need for support and intervention[81], self-guided features that are often heavily reliant on users’
self-awareness and motivation at baseline may be limited intheir capacity to challenge growth and cultivate health andwell-being–promoting behaviors and thoughts. Comparatively,
most behavioral health coaches are professionally trained andcommitted to increasing clients’ introspection on their needs
and goals after accounting for their background factors (eg,specific vulnerabilities, working habits, communication styles,
and content of stressors) [46]. Even for flourishing clients withminimal needs, behavioral health coaches could leverage theircharacter strengths and encourage positive habit formation tomaintain wellness [82,83]. Although it is beyond the scope of
this study, future comparative research may consider delineating
the mechanisms that derive higher preventive effectiveness in
JMIR Form Res 2023 | vol. 7 | e45678 | p. 12(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
BHC as compared with self-help interventions. Finally, owing
to the nonsignificant between-group differences in the monthly
engagement rates of self-guided features, we infer that these
positive effects were more likely to be derived from the coachingsessions.
Among the “Coaching” participants, our study also demonstrated
that clients’ perceptions on the “usefulness of BHC sessions”
and “working alliance” significantly mediated theirimprovements in well-being over the course of BHC. First, ourfindings that these perceptions were developed across time (“a”path; Table 4) are consistent with the notion that coaching is areflective, gradual process that transforms behavior bycultivating the client’s self-awareness of a need to thrive.
Accordingly, a stable “working alliance” involves an ongoing,mutual exchange of sincerity, respect, and trust between thecoach and the client [84-87]. Even for coaches who showedgenuine interest in the client’s well-being and goals from thefirst session, establishing clear agreements and maintaining
accountability of changes in later sessions were necessarycomponents of a client-rated effective alliance [88,89]. These
positive developments in alliance, when combined with effective
coaching techniques, may incrementally contribute to the higher
client-perceived usefulness of coaching sessions as they progress
toward attaining their goals [90,91]. Our findings that theseprocesses fully mediated improvements in well-being, which
in turn propels other “active ingredients” of BHC, highlight theimportance of fostering these components in current BHC
practices. Future work may continue to explore other mediatorsof change in well-being and suggest enhancements to BHC
training.
In line with recent web-based coaching studies [50,51], ourstudy revealed larger effect sizes for employees who completed
4 to 6 BHC sessions. We extend these findings by also showingthat significant increments in well-being were possible after
only 2 to 3 sessions. These numbers differed from therecommended range of CBT sessions for patients with mental
health diagnoses (ie, 8-20 sessions [92]), presuming that healthyfunctioning individuals may find it relatively easier to adhere
to the intervention. Imminent improvements after the initial
sessions can be attributed to a variety of reasons, such as
establishing a trustable working alliance [93], enhanced
self-awareness, and relief of the user’s own presenting struggles
during problem construction and goal formulation [94]. Gaining
clarity and resolution of their problems subsequently enhances
motivation, interest, and hope for intervention success in theclient. This, in turn, strengthens participant adherence in later
sessions, which is helpful in maximizing outcomes [95].
However, the larger effect sizes observed for this treatment
length may also be due to higher levels of motivation or different
presenting problems that may not require more sessions to reacha resolution. These potential confounding variables were notassessed in our study. For the same reasons, the absence ofsignificant improvements starting from the seventh session may
not necessarily imply a lack of long-term effects of a BHC
intervention. As this was a retrospective cohort study, we
recommend that future researchers use more rigorous research
designs, such as a 2-armed randomized controlled trial collecting

https://formative.jmir.org/2023/1/e45678

XSL• FO
RenderX

Toh et alobjective well-being data from wearable devices to assess thedose-response relationship.

Strengths and Limitations
Despite the large sample size and statistically rigorous
methodologies, this study also has several limitations. First, thelack of randomization procedures precluded any possible
conclusions on the causality of the intervention. Second,although this study used real-world data that not only enhanced
external validity but also allowed for an appropriate assessmentof the acceptability and preliminary impact of Intellect’s
coaching, users were not required to complete some assessments(ie, postcoaching feedback), which limited the evaluation of
possible mediators to a small subsample. Third, participants
may have used other well-being apps in conjunction withIntellect’s BHC or self-guided features during the study, which
may have confounded our findings. Fourth, although ourself-developed items were validated using an external sample(n=997), existing surveys that were validated thoroughly andcross-culturally (ie, the Patient Health Questionnaire–2 andPerceived Stress Scale) may have increased the reliability of
our findings. Fifth, the use of single-item scores on mood andstress as one of the screening criteria for “mentally healthy”
employees is much less reliable in comparison with more robust
screening procedures using clinically validated scales (eg, thePatient Health Questionnaire–9). Even if participants in theintervention group had been subjected to some form of risk
assessment by their providers, no formal assessments wereconducted in this study. This may have affected the homogeneityof the groups, which would bias the group comparison. Sixth,
despite having already controlled for self-guided feature
engagement as a covariate in our main analyses, the average
improvements in well-being could still be influenced by possible
web-based effects between the BHC intervention and self-guided
features. These limitations, together with the small effect sizes,
warrant the consideration of our results as only preliminary.
Future research aiming to replicate our findings may benefit
from a more rigorous design, such as including randomized
treatment and control groups and assessing all participants using
brief psychometrically validated instruments at standardized
time points. For instance, with a 3-armed randomized controlled
design, researchers have a better chance of parsing the main
effects of the BHC intervention, self-guided feature engagement,
and no treatment on well-being outcomes separately. Unlike
real-world studies, a controlled research study can also prevent
its participants from receiving alternative interventions (ie,self-guided features and other well-being apps) in parallel andminimize confounding effects. Finally, this study has limited
external validity to the working population at large as the sample
consisted mainly of relatively young (mean age 30.85, SD 6.97y) woman employees (626/1025, 61.1%) residing acrossAsia-Pacific countries. The transferability of our findings canbe further strengthened if researchers can replicate our study
with more diverse samples (eg, European or Western employees,
Asian male employees, and older employees).

Conclusions
Using a large sample of relatively mentally healthy employees
residing across the Asia-Pacific region, this study evaluated the
JMIR Form Res 2023 | vol. 7 | e45678 | p. 13(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
real-world effectiveness of Intellect’s professional coaching
services against a nonrandomized control group with open
access to Intellect’s self-help features in further improving
well-being. The findings indicated significant improvements incomposite well-being among healthy employees who
participated in coaching, and no significant between-groupdifferences were found in the monthly engagement rates ofself-help features. Within a subsample of the intervention group,clients’ positive perceptions of “working alliance” and“usefulness of coaching” mediated the improvements in

Toh et al
well-being. Our findings are useful given the potential for
scaling up easily accessible, cost-effective coaching services tomany organizations that currently aim to promote preventive
and proactive care for their employees. Given the lack ofrandomized groups, comprehensive psychometric instruments,
a more diverse sample, and other possible confounds, this study
provides only preliminary evidence that BHC can improve
occupational welfare, which translates to significant
improvements in public mental health.

Acknowledgments
The authors would like to thank all the employees and coaches who were involved in this retrospective study.

Data Availability
The data analyzed in this study are not publicly available because of the restricted permissions and confidentiality shared between
Intellect and its commercial partners. The data will only be available upon reasonable request made to the corresponding author.

Authors' Contributions
SHYT researched the literature and conceived the study. SHYT was also responsible for all the retrospective data analyses. OS
supervised the study, whereas SHYT led the commencement until the completion of the manuscript. SCL had a consulting rolein data analysis. All authors reviewed, edited, and approved the final version of the manuscript.

Conflicts of Interest
OS holds the position of Clinical Director at Intellect Pte Ltd; he also reports holding equity in Intellect. SCL is employed as
Clinical Research Lead at Intellect, and SHYT is a Research Associate in Intellect’s Clinical team.

Multimedia Appendix 1
Tables comparing the differences in engagement with each self-guided feature (learning paths, journaling sessions, rescue sessions,and toolkits) between groups over time.
[DOCX File , 26 KB-Multimedia Appendix 1]

References
1.
2.
3.

4.

5.
6.
7.

8.

9.

Glozier N. Mental ill health and fitness for work. Occup Environ Med 2002 Oct;59(10):714-720 [FREE Full text] [doi:10.1136/oem.59.10.714] [Medline: 12356935]
Harvey SB, Henderson M, Lelliott P, Hotopf M. Mental health and employment: much work still to be done. Br J Psychiatry
2009 Mar 02;194(3):201-203 [doi: 10.1192/bjp.bp.108.055111] [Medline: 19252144]
Stratton E, Lampit A, Choi I, Calvo RA, Harvey SB, Glozier N. Effectiveness of eHealth interventions for reducing mental
health conditions in employees: a systematic review and meta-analysis. PLoS One 2017 Dec 21;12(12):e0189904 [FREEFull text] [doi: 10.1371/journal.pone.0189904] [Medline: 29267334]
Stewart WF, Ricci JA, Chee E, Morganstein D. Lost productive work time costs from health conditions in the United States:
results from the American Productivity Audit. J Occup Environ Med 2003 Dec;45(12):1234-1246 [doi:10.1097/01.jom.0000099999.27348.78] [Medline: 14665809]
Black CM. Working for a Healthier Tomorrow: Dame Carol Black's Review of The Health of Britain's Working AgePopulation. London, UK: Stationery Office Books; 2008.
Hu TW. The economic burden of depression and reimbursement policy in the Asia Pacific region. Australas Psychiatry
2004 Oct 01;12 Suppl(1_suppl):S11-S15 [doi: 10.1080/j.1039-8562.2004.02100.x-1] [Medline: 15715825]
Deady M, Johnston DA, Glozier N, Milne D, Choi I, Mackinnon A, et al. Smartphone application for preventing depression:
study protocol for a workplace randomised controlled trial. BMJ Open 2018 Jul 13;8(7):e020510 [FREE Full text] [doi:10.1136/bmjopen-2017-020510] [Medline: 30007927]
Weber S, Lorenz C, Hemmings N. Improving stress and positive mental health at work via an app-based intervention: alarge-scale multi-center randomized control trial. Front Psychol 2019 Dec 6;10:2745 [FREE Full text] [doi:10.3389/fpsyg.2019.02745] [Medline: 31866915]
Long T, Cooke FL. Employee assistance programmes in China: a state‐of‐the‐art review and future research agenda.
Asia Pac J Hum Resour 2022 Jul 24;61(1):3-31 [FREE Full text] [doi: 10.1111/1744-7941.12345]

https://formative.jmir.org/2023/1/e45678

XSL• FO
RenderX

JMIR Form Res 2023 | vol. 7 | e45678 | p. 14(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
10.
11.

12.

13.
14.
15.

16.

17.

18.

19.

20.
21.

22.

23.

24.

25.
26.

27.

28.
29.
30.
31.
32.

Csiernik R, Chaulk P, McQuaid S. A process evaluation of a Canadian public sector employee assistance program. J
Workplace Behav Health 2012 Jul;27(3):160-180 [FREE Full text] [doi: 10.1080/15555240.2012.701169]
Andrews G, Issakidis C, Sanderson K, Corry J, Lapsley H. Utilising survey data to inform public policy: comparison of
the cost-effectiveness of treatment of ten mental disorders. Br J Psychiatry 2004 Jun;184(6):526-533 [doi:10.1192/bjp.184.6.526] [Medline: 15172947]
Berkels H, Henderson J, Henke N, Kuhn K, Lavikainen J, Lehtinen V. Mental health promotion and prevention strategies
for coping with anxiety, depression and stress related disorders in Europe (2001-2003). European Commission. 2003 Jul.
URL: https://ec.europa.eu/health/ph_projects/2001/promotion/fp_promotion_2001_frep_02_en.pdf [accessed 1999-11-30]
Aryana B, Brewster L, Nocera JA. Design for mobile mental health: an exploratory review. Health Technol 2019;9(4):401-424[FREE Full text] [doi: 10.1007/s12553-018-0271-1]
Kaplan S, Bradley-Geist JC, Ahmad A, Anderson A, Hargrove AK, Lindsey A. A test of two positive psychology interventions
to increase employee well-being. J Bus Psychol 2014;29(3):367-380 [FREE Full text] [doi: 10.1007/s10869-013-9319-4]
Ebert DD, Heber E, Berking M, Riper H, Cuijpers P, Funk B, et al. Self-guided internet-based and mobile-based stressmanagement for employees: results of a randomised controlled trial. Occup Environ Med 2016 May 16;73(5):315-323 [doi:10.1136/oemed-2015-103269] [Medline: 26884049]
Bücker L, Schnakenberg P, Karyotaki E, Moritz S, Westermann S. Diminishing effects after recurrent use of self-guided
internet-based interventions in depression: randomized controlled trial. J Med Internet Res 2019 Oct 02;21(10):e14240[FREE Full text] [doi: 10.2196/14240] [Medline: 31579014]
Fischer R, Bortolini T, Karl JA, Zilberberg M, Robinson K, Rabelo A, et al. Rapid review and meta-meta-analysis ofself-guided interventions to address anxiety, depression, and stress during COVID-19 social distancing. Front Psychol 2020
Oct 28;11:563876 [FREE Full text] [doi: 10.3389/fpsyg.2020.563876] [Medline: 33192837]
Ong WY, Sündermann O. Efficacy of the mental health app "Intellect" to improve body image and self-compassion inyoung adults: a randomized controlled trial with a 4-week follow-up. JMIR Mhealth Uhealth 2022 Nov 21;10(11):e41800[FREE Full text] [doi: 10.2196/41800] [Medline: 36409542]
Toh SH, Tan JH, Kosasih FR, Sündermann O. Efficacy of the mental health app intellect to reduce stress: randomized
controlled trial with a 1-month follow-up. JMIR Form Res 2022 Dec 14;6(12):e40723 [FREE Full text] [doi: 10.2196/40723][Medline: 36515984]
Bozer G, Jones RJ. Understanding the factors that determine workplace coaching effectiveness: a systematic literature
review. Eur J Work Organ Psychol 2018;27(3):342-361 [FREE Full text] [doi: 10.1080/1359432x.2018.1446946]
Corbu A, Peláez Zuberbühler MJ, Salanova M. Positive psychology micro-coaching intervention: effects on psychologicalcapital and goal-related self-efficacy. Front Psychol 2021 Feb 11;12:566293 [FREE Full text] [doi:10.3389/fpsyg.2021.566293] [Medline: 33643121]
van Nieuwerburgh C, Barr M, Fouracres AJ, Moin T, Brown C, Holden C, et al. Experience of positive psychology coaching
while working from home during the COVID-19 pandemic: an Interpretative Phenomenological Analysis. Coaching: Int
J Theory Res Pract 2022;15(2):148-165 [FREE Full text] [doi: 10.1080/17521882.2021.1897637]
Kranenburg LW, Gillis J, Mayer B, Hoogendijk WJ. The effectiveness of a nonguided mindfulness app on perceived stressin a nonclinical Dutch population: randomized controlled trial. JMIR Ment Health 2022 Mar 18;9(3):e32123 [FREE Fulltext] [doi: 10.2196/32123] [Medline: 35302504]
Theeboom T, Beersma B, van Vianen AE. Does coaching work? A meta-analysis on the effects of coaching on individuallevel outcomes in an organizational context. J Posit Psychol 2014;9(1):1-18 [FREE Full text] [doi:10.1080/17439760.2013.837499]
Jones RJ, Woods SA, Guillaume YR. The effectiveness of workplace coaching: a meta-analysis of learning and performanceoutcomes from coaching. J Occup Organ Psychol 2016;89(2):249-277 [FREE Full text] [doi: 10.1111/joop.12119]
Hill B, Richardson B, Skouteris H. Do we know how to design effective health coaching interventions: a systematic reviewof the state of the literature. Am J Health Promot 2015 May;29(5):e158-e168 [doi: 10.4278/ajhp.130510-LIT-238] [Medline:
24720388]
Shih E, Aylward BS, Kunkle S, Graziani G. Association between care modality and use with treatment response among
members accessing virtual mental health services: real-world observational study. JMIR Form Res 2022 Jul 22;6(7):e36956[FREE Full text] [doi: 10.2196/36956] [Medline: 35867401]
Carvalho M, Matos MG, Anjos MH. Cognitive-behavioural coaching: Applications to health and personal development
contexts. EC Psychol Psychiatr 2018;7(3):29 [FREE Full text]
Grant AM, Green RM. Developing clarity on the coaching-counselling conundrum: implications for counsellors andpsychotherapists. Couns Psychother Res 2018 Sep 14;18(4):347-355 [FREE Full text] [doi: 10.1002/capr.12188]
Grant AM, O’Connor SA. Broadening and building solution-focused coaching: feeling good is not enough. Coaching: Int
J Theory Res Pract 2018 Jun 18;11(2):165-185 [FREE Full text] [doi: 10.1080/17521882.2018.1489868]
Fredrickson BL. The broaden-and-build theory of positive emotions. Philos Trans R Soc Lond B Biol Sci 2004 Sep
29;359(1449):1367-1378 [FREE Full text] [doi: 10.1098/rstb.2004.1512] [Medline: 15347528]
Gan Y, Yang M, Zhou Y, Zhang Y. The two-factor structure of future-oriented coping and its mediating role in student
engagement. Pers Individ Dif 2007 Sep;43(4):851-863 [FREE Full text] [doi: 10.1016/j.paid.2007.02.009]

https://formative.jmir.org/2023/1/e45678

XSL• FO
RenderX

Toh et al

JMIR Form Res 2023 | vol. 7 | e45678 | p. 15(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
33.
34.
35.
36.

37.
38.
39.
40.

41.
42.
43.
44.
45.
46.
47.
48.
49.
50.

51.

52.

53.

54.

55.
56.

57.
58.

Sirois FM, Molnar DS, Hirsch JK. Self-compassion, stress, and coping in the context of chronic illness. Self Identity 2015
Jan 09;14(3):334-347 [FREE Full text] [doi: 10.1080/15298868.2014.996249]
Grant AM, Curtayne LS, Burton G. A randomised controlled study of 360 degree feedback based workplace coaching withhigh school teachers. J Posit Psychol 2009 Sep;4(5):396-407 [FREE Full text] [doi: 10.1080/17439760902992456]
Grant A. A languishing–flourishing model of goal striving and mental health for coaching populations. In: Passmore J, Tee
D, editors. Coaching Researched: A Coaching Psychology Reader. Hoboken, NJ: John Wiley & Sons; Oct 27, 2020:65-82
Green S, Oades L, Grant A. An evaluation of a life-coaching group program: Initial findings from a waitlist control study.
In: Cavanagh M, Grant A, Kemp T, editors. Evidence-Based Coaching: Volume 1, Theory, Research and Practice fromthe Behavioural Sciences. Queensland, Australia: Australian Academic Press; 2005:127-141
Spence GB, Grant AM. Professional and peer life coaching and the enhancement of goal striving and well-being: an
exploratory study. J Posit Psychol 2007;2(3):185-194 [FREE Full text] [doi: 10.1080/17439760701228896]
Spence GB, Cavanagh MJ, Grant AM. The integration of mindfulness training and health coaching: an exploratory study.
Coaching: Int J Theory Res Pract 2008 Sep;1(2):145-163 [FREE Full text] [doi: 10.1080/17521880802328178]
Gyllensten K, Palmer S. Can coaching reduce workplace stress? Int J Evid Based Coach Mentor 2005 Jul 01;1(1):15-17[doi: 10.53841/bpstcp.2005.1.1.15]
Finn F. Leadership development through executive coaching: the effects on leaders' psychological states and transformational
leadership behaviour. Queensland University of Technology. 2007. URL: https://eprints.qut.edu.au/17001/1/
Fran%20Finn%20Thesis.pdf [accessed 2022-11-10]
Moen F, Allgood E. Coaching and the effect on self-efficacy. Organ Dev J 2009;27(4):69-81 [FREE Full text]
Biswas-Diener R. Personal coaching as a positive intervention. J Clin Psychol 2009 May;65(5):544-553 [doi:10.1002/jclp.20589] [Medline: 19294747]
Lilienfeld SO. Psychological treatments that cause harm. Perspect Psychol Sci 2007 Mar;2(1):53-70 [doi:10.1111/j.1745-6916.2007.00029.x] [Medline: 26151919]
McKenna DD, Davis SL. Hidden in plain sight: the active ingredients of executive coaching. Ind Organ Psychol 2015 Jan
07;2(3):244-260 [FREE Full text] [doi: 10.1111/j.1754-9434.2009.01143.x]
Baron L, Morin L. The coach-coachee relationship in executive coaching: a field study. Hum Resour Dev Q 2009
Dec;20(1):85-106 [FREE Full text] [doi: 10.1002/hrdq.20009]
de Haan E, Duckworth A. Signalling a new trend in executive coaching outcome research. Int Coach Psychol Rev 2013
Mar 01;8(1):6-19 [FREE Full text] [doi: 10.53841/bpsicpr.2013.8.1.6]
Pandolfi C. Active ingredients in executive coaching: a systematic literature review. Int Coach Psychol Rev 2020;15(2):6-30[FREE Full text] [doi: 10.53841/bpsicpr.2020.15.2.6]
Kauffman C. Positive psychology: the science at the heart of coaching. In: Stober DR, Grant AM, editors. Evidence Based
Coaching Handbook: Putting Best Practices to Work for Your Clients. Hoboken, NJ: John Wiley & Sons; 2006:219-253
Biswas-Diener R. Practicing Positive Psychology Coaching: Assessment, Activities, and Strategies for Success. Hoboken,
NJ: John Wiley & Sons; 2010.
Lungu A, Boone MS, Chen SY, Chen CE, Walser RD. Effectiveness of a cognitive behavioral coaching program delivered
via video in real world settings. Telemed J E Health 2021 Jan;27(1):47-54 [FREE Full text] [doi: 10.1089/tmj.2019.0313][Medline: 32311301]
Sagui-Henson S, Prescott MR, Corcoran JB, Pillai S, Mattila L, Mathur S, et al. Effectiveness of evidence-based telecoachingdelivered through an employer-sponsored mental health benefits platform. Telemed J E Health 2022 Apr;28(4):486-494[FREE Full text] [doi: 10.1089/tmj.2020.0555] [Medline: 34191616]
Auer EM, Hutchinson D, Eatough E, Carr EW, Sinar EF, Kellerman G. The buffering effects of virtual coaching duringcrisis: a quasi-experimental study of changes in well-being, work, and social outcomes before and during the COVID-19
pandemic. Int J Evid Based Coach Mentor 2022;20(2):3-19 [FREE Full text]
Kosasih FR, Yee VT, Toh SH, Sündermann O. Efficacy of Intellect's self-guided anxiety and worry mobile health programme:a randomized controlled trial with an active control and a 2-week follow-up. PLOS Digit Health 2023 May 24;2(5):e0000095[FREE Full text] [doi: 10.1371/journal.pdig.0000095] [Medline: 37224139]
Yang Toh SH, Lee SC, Kosasih FR, Lim JW, Sündermann O. Preliminary effectiveness of an evidence-based mobile
application to promote resilience among working adults in Singapore and Hong Kong: Intensive longitudinal study. Digit
Health 2023 May 30;9:20552076231178616 [FREE Full text] [doi: 10.1177/20552076231178616] [Medline: 37274370]
Passmore J. The Coaches' Handbook: The Complete Practitioner Guide for Professional Coaches. New York, NY: Routledge;
Oct 2020.
Richter S, van Zyl LE, Roll L, Stander M. Positive psychological coaching tools and techniques: a systematic review andclassification. Front Psychiatry 2021 Jul 09;12:667200 [FREE Full text] [doi: 10.3389/fpsyt.2021.667200] [Medline:
34305674]
Warttig SL, Forshaw MJ, South J, White AK. New, normative, English-sample data for the short form Perceived Stress
Scale (PSS-4). J Health Psychol 2013 Dec;18(12):1617-1628 [doi: 10.1177/1359105313508346] [Medline: 24155195]
Kroenke K, Spitzer RL, Williams JB, Löwe B. An ultra-brief screening scale for anxiety and depression: the PHQ-4.
Psychosomatics 2009 Nov;50(6):613-621 [doi: 10.1176/appi.psy.50.6.613] [Medline: 19996233]

https://formative.jmir.org/2023/1/e45678

XSL• FO
RenderX

Toh et al

JMIR Form Res 2023 | vol. 7 | e45678 | p. 16(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
59.
60.
61.
62.
63.

64.
65.
66.
67.
68.

69.
70.

71.

72.
73.
74.

75.
76.

77.

78.
79.

80.

81.
82.
83.

Fuchs C, Diamantopoulos A. Using single-item measures for construct measurement in management research: conceptualissues and application guidelines. Die Betriebswirtschaft 2009;69(2):195 [FREE Full text]
Field A. Discovering Statistics Using IBM SPSS Statistics. Thousand Oaks, CA: Sage Publications; 2013.
Bates D, Mächler M, Bolker B, Walker S. Fitting linear mixed-effects models using lme4. J Stat Softw 2015;67(1):1-48[FREE Full text] [doi: 10.18637/jss.v067.i01]
Hutchinson S, Wei L, Louwerse M. Avoiding the language-as-a-fixed-effect fallacy: how to estimate outcomes of linear
mixed models. Proc Annu Meet Cogn Sci Soc 2014:2393-2398 [FREE Full text]
Kwok OM, Underhill AT, Berry JW, Luo W, Elliott TR, Yoon M. Analyzing longitudinal data with multilevel models: an
example with individuals living with lower extremity intra-articular fractures. Rehabil Psychol 2008 Aug;53(3):370-386[FREE Full text] [doi: 10.1037/a0012765] [Medline: 19649151]
Everitt B. A Monte Carlo investigation of the likelihood ratio test for the number of components in a mixture of normal
distributions. Multivariate Behav Res 1981 Apr 01;16(2):171-180 [doi: 10.1207/s15327906mbr1602_3] [Medline: 26825421]
Little TD. Longitudinal Structural Equation Modeling. New York, NY: Guilford press; 2013.
Curran PJ, Obeidat K, Losardo D. Twelve frequently asked questions about growth curve modeling. J Cogn Dev
2010;11(2):121-136 [FREE Full text] [doi: 10.1080/15248371003699969] [Medline: 21743795]
Twisk JW. Applied Multilevel Analysis: A Practical Guide for Medical Researchers. Cambridge, MA: Cambridge university
press; 2006.
Curran PJ, Bauer DJ. The disaggregation of within-person and between-person effects in longitudinal models of change.
Annu Rev Psychol 2011 Jan 10;62(1):583-619 [FREE Full text] [doi: 10.1146/annurev.psych.093008.100356] [Medline:
19575624]
Bolger N, Laurenceau JP. Intensive Longitudinal Methods: An Introduction to Diary and Experience Sampling Research.
New York, NY: The Guilford Press; 2013.
Hayes AF. PROCESS: a versatile computational tool for observed variable mediation, moderation, and conditional processmodeling. University of Kansas. 2012. URL: https://imaging.mrc-cbu.cam.ac.uk/statswiki/FAQ/
SobelTest?action=AttachFile&do=get&target=process.pdf [accessed 2023-09-20]
Bauer DJ, Preacher KJ, Gil KM. Conceptualizing and testing random indirect effects and moderated mediation in multilevel
models: new procedures and recommendations. Psychol Methods 2006 Jun;11(2):142-163 [doi: 10.1037/1082-989X.11.2.142][Medline: 16784335]
Hayes AF, Rockwood NJ. Conditional process analysis: concepts, computation, and advances in the modeling of thecontingencies of mechanisms. Am Behav Sci 2019 Jul 17;64(1):19-54 [FREE Full text] [doi: 10.1177/0002764219859633]
Hauk N, Göritz AS, Krumm S. The mediating role of coping behavior on the age-technostress relationship: a longitudinal
multilevel mediation model. PLoS ONE 2019 Mar 5;14(3):e0213349 [doi: 10.1371/journal.pone.0213349]
Duijts SF, Kant I, van den Brandt PA, Swaen GM. Effectiveness of a preventive coaching intervention for employees atrisk for sickness absence due to psychosocial health complaints: results of a randomized controlled trial. J Occup Environ
Med 2008 Jul;50(7):765-776 [doi: 10.1097/JOM.0b013e3181651584] [Medline: 18617832]
Grant AM. Reality TV gets positive: psychological reflections on making Australia happy. Int Coach Psychol Rev 2011
Sep 01;6(2):229-239 [FREE Full text] [doi: 10.53841/bpsicpr.2011.6.2.229]
Solomon D, Proudfoot J, Clarke J, Christensen H. e-CBT (myCompass), antidepressant medication, and face-to-face
psychological treatment for depression in Australia: a cost-effectiveness comparison. J Med Internet Res 2015 Nov
11;17(11):e255 [FREE Full text] [doi: 10.2196/jmir.4207]
Strudwick G, Sockalingam S, Kassam I, Sequeira L, Bonato S, Youssef A, et al. Digital interventions to support population
mental health in Canada during the COVID-19 pandemic: rapid review. JMIR Ment Health 2021 Mar 2;8(3):e26550 [doi:10.2196/26550]
Andersson G, Carlbring P, Berger T, Almlöv J, Cuijpers P. What makes internet therapy work? Cogn Behav Ther 2009;38
Suppl 1:55-60 [doi: 10.1080/16506070902916400] [Medline: 19675956]
Mehrotra S, Kumar S, Sudhir P, Rao GN, Thirthalli J, Gandotra A. Unguided mental health self-help apps: reflections onchallenges through a clinician's lens. Indian J Psychol Med 2017 Sep;39(5):707-711 [FREE Full text] [doi:10.4103/IJPSYM.IJPSYM_151_17] [Medline: 29200577]
Mehrotra S, Sudhir P, Rao G, Thirthalli J, Srikanth TK. Development and pilot testing of an internet-based self-help
intervention for depression for Indian users. Behav Sci (Basel) 2018 Mar 22;8(4):36 [FREE Full text] [doi:10.3390/bs8040036] [Medline: 29565278]
Daniel K, Haidt J. What is the good life? PositivePsychology. 2009. URL: https://positivepsychology.com/
what-is-positive-psychology-definition/#google_vignette [accessed 2022-11-13]
McQuaid M. Positive psychology coaching. Organ Superv Coach 2017;24(3):283-296 [FREE Full text]
Kemp TJ. Positive psychology coaching and industrial-organisational psychology: blending the science of human flourishing
to build sustainable enterprise for an unpredictable future. In: Positive Psychology Coaching in the Workplace. Cham,
Switzerland: Springer; 2021:61-84

https://formative.jmir.org/2023/1/e45678

XSL• FO
RenderX

Toh et al

JMIR Form Res 2023 | vol. 7 | e45678 | p. 17(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
84.

85.

86.
87.
88.

89.

90.

91.

92.
93.
94.

95.

Toh et al

Oades LG, Crowe TP, Nguyen M. Leadership coaching transforming mental health systems from the inside out: The
Collaborative Recovery Model as person-centred strengths based coaching psychology. Int Coach Psychol Rev 2009 Mar
01;4(1):25-36 [FREE Full text] [doi: 10.53841/bpsicpr.2009.4.1.25]
Stander F. Strength coaching as an enabler of positive athlete outcomes in a multi-cultural sport environment. In: van Zyl
LE, Stander MW, Odendaal A, editors. Coaching Psychology: Meta-Theoretical Perspectives and Applications in Multicultural
Contexts. Cham, Switzerland: Springer; 2016:279-298
Dyess SM, Sherman R, Opalinski A, Eggenberger T. Structured Coaching programs to develop staff. J Contin Educ Nurs
2017 Aug;48(8):373-378 [doi: 10.3928/00220124-20170712-10]
Yeager J, Britton K. Positive psychology coaching for sports leaders. In: Brady A, Grenville-Cleave B, editors. Positive
Psychology in Sport and Physical Activity. Oxfordshire, UK: Routledge; 2017:243-255
Kivlighan DM, Hill CE, Gelso CJ, Baumann E. Working alliance, real relationship, session quality, and client improvementin psychodynamic psychotherapy: a longitudinal actor partner interdependence model. J Couns Psychol 2016
Mar;63(2):149-161 [doi: 10.1037/cou0000134] [Medline: 26689627]
Shafran N, Kivlighan DM, Gelso CJ, Bhatia A, Hill CE. Therapist immediacy: the association with working alliance, real
relationship, session quality, and time in psychotherapy. Psychother Res 2017 Nov 19;27(6):737-748 [doi:10.1080/10503307.2016.1158884] [Medline: 27092970]
Grover S, Furnham A. Coaching as a developmental intervention in organisations: a systematic review of its effectivenessand the mechanisms underlying it. PLoS One 2016 Jul 14;11(7):e0159137 [FREE Full text] [doi:10.1371/journal.pone.0159137] [Medline: 27416061]
Vermeiden M, Reijnders J, van Duin E, Simons M, Janssens M, Peeters S, et al. Prospective associations between working
alliance, basic psychological need satisfaction, and coaching outcome indicators: a two-wave survey study among 181
Dutch coaching clients. BMC Psychol 2022 Nov 15;10(1):269 [FREE Full text] [doi: 10.1186/s40359-022-00980-9][Medline: 36380365]
Barrington J. Cognitive behaviour therapy: standards for training and clinical practice. Behav Change 2012 Feb
22;23(4):227-238 [doi: 10.1375/bech.23.4.227]
Spencer J, Goode J, Penix EA, Trusty W, Swift JK. Developing a collaborative relationship with clients during the initial
sessions of psychotherapy. Psychotherapy (Chic) 2019 Mar;56(1):7-10 [doi: 10.1037/pst0000208] [Medline: 30816757]
Jacobus SI. Therapists' documentation of change in clients' problems and relationships in long-term psychotherapy. Universityof Massachusetts. 1994. URL: https://scholarworks.umass.edu/cgi/viewcontent.cgi?article=4252&context=dissertations_1[accessed 2023-08-09]
Braun L, Freund J, Thielecke J, Baumeister H, Ebert DD, Titzler I. Barriers to and facilitators of engaging with and adhering
to guided internet-based interventions for depression prevention and reduction of pain-related disability in green professions:
mixed methods study. JMIR Ment Health 2022 Nov 09;9(11):e39122 [FREE Full text] [doi: 10.2196/39122] [Medline:
36350684]

Abbreviations
BHC: behavioral health coaching
CBT: cognitive behavioral therapy
EAP: employee assistance program

Edited by A Mavragani; submitted 12.01.23; peer-reviewed by M Attridge, J Wang; comments to author 27.07.23; revised version
received 17.08.23; accepted 31.08.23; published 20.10.23
Please cite as:Toh SHY, Lee SC, Sündermann O
Mobile Behavioral Health Coaching as a Preventive Intervention for Occupational Public Health: Retrospective Longitudinal StudyJMIR Form Res 2023;7:e45678
URL: https://formative.jmir.org/2023/1/e45678
doi: 10.2196/45678
PMID: 37862086©Sean Han Yang Toh, Sze Chi Lee, Oliver Sündermann. Originally published in JMIR Formative Research(https://formative.jmir.org), 20.10.2023. This is an open-access article distributed under the terms of the Creative Commons
Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction
in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete

https://formative.jmir.org/2023/1/e45678

XSL• FO
RenderX

JMIR Form Res 2023 | vol. 7 | e45678 | p. 18(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Toh et albibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license
information must be included.

https://formative.jmir.org/2023/1/e45678

XSL• FO
RenderX

JMIR Form Res 2023 | vol. 7 | e45678 | p. 19(page number not for citation purposes)