JMIR FORMATIVE RESEARCH

Anton et al

Original Paper

Evaluation of a Commercial Mobile Health App for Depression
and Anxiety (AbleTo Digital+): Retrospective Cohort Study
Margaret T Anton*, PhD; Heidi Mochari Greenberger*, MPH, PhD; Evie Andreopoulos*, MA; Reena L Pande*, MSc,
MD
AbleTo, Inc, New York, NY, United States*

all authors contributed equally

Corresponding Author:
Margaret T Anton, PhD
AbleTo, Inc
320 W, 37th Street, 5th floor
New York, NY, 10018
United States
Phone: 1 (347) 926 5527
Email: margaret.anton@ableto.com

Abstract
Background: Digital solutions, such as web-based and mobile interventions, have the potential to streamline pathways to mental
health services and improve access to mental health care. Although a growing number of randomized trials have established theefficacy of digital interventions for common mental health problems, less is known about the real-world impact of these tools.
AbleTo Digital+, a commercially available mental health app for depression and anxiety, offers a unique opportunity to understandthe clinical impact of such tools delivered in a real-world context.
Objective: The primary aim of this study is to examine the magnitude of change in depression and anxiety symptoms amongindividuals who used AbleTo Digital+ programs. The secondary aim is to evaluate Digital+ module completion, including theuse of 1:1 coaching.
Methods: In this retrospective cohort study, we analyzed previously collected and permanently deidentified data from a
consecutive cohort of 1896 adults who initiated using one of the three Digital+ eight-module programs (depression, generalized
anxiety, or social anxiety) between January 1 and June 30, 2020. Depression, generalized anxiety, and social anxiety symptoms
were assessed within each program using the Patient Health Questionnaire-9, the Generalized Anxiety Disorder-7, and the Social
Phobia Inventory, respectively. Linear mixed effects models were built to assess the association between module completion andsymptom change among users who completed at least four modules and had at least mild baseline symptom elevations, controllingfor age, gender, and baseline symptom severity. Digital+ use, including module completion, 1:1 coaching calls, and in-app coachmessaging, was also evaluated.
Results: Significant effects were observed among depression (Cohen d=1.5), generalized anxiety (Cohen d=1.2), and social
anxiety (Cohen d=1.0) program participants who completed at least four modules and had mild baseline elevations (n=470).
Associations between module completion and change in depression (β=−1.2; P<.001), generalized anxiety (β=−1.1; P<.001),
and social anxiety (β=−2.4; P<.001) symptom scores retained significance with covariate adjustment. Participants completed anaverage of 2.6 (SD 2.7) modules. The average total length of app use was 52.2 (SD 83.5) days. Approximately two-thirds of theusers engaged in at least 1 coaching call (66.82%, 1267/1896) or in-app text messaging (66.09%, 1253/1896). Participants whocompleted at least four modules participated in significantly more coaching calls per module (mean 1.1, SD 0.7) than users whocompleted fewer than four modules (mean 1.0, SD 1.2; t1407=−2.1; P=.03).
Conclusions: This study demonstrated that AbleTo Digital+ users experienced significant reductions in depression, generalized
anxiety, and social anxiety symptoms throughout the program.
(JMIR Form Res 2021;5(9):e27570) doi: 10.2196/27570
KEYWORDS
digital mental health; mHealth; iCBT; coaching; depression; generalized anxiety; social anxiety; mobile phone

https://formative.jmir.org/2021/9/e27570

XSL• FO
RenderX

JMIR Form Res 2021 | vol. 5 | iss. 9 | e27570 | p. 1
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Introduction
Background
Effective mental health interventions exist for depression andanxiety [1,2], yet more than 50% of people in need of these
services do not receive them [3]. A number of systemic- andindividual-level barriers lead to these service gaps, including
mental health workforce shortages, stigma, and financial barriers[4,5]. In addition, even when individuals are able to accesstraditional face-to-face services, the lag between symptom onset
and treatment initiation is, on average, 11 years [6,7]. As therates of anxiety and depression continue to increase [8], hastenedin part by the current COVID-19 pandemic [9,10], access gaps
are likely to worsen in an already strained, fragmented, andunderresourced mental health care system [11]. These gaps inservice have large societal, familial, and individual costs [12].
For example, in the United States, the estimated annual
economic burden of depression alone is US $210 billion [13,14].
Digital solutions, including web-based and mobile behavioral
health care interventions, have great potential to streamline
pathways to mental health services and improve access to quality
mental health care [15]. Stand-alone and adjunctive digital tools
can provide timely mental health support that fits individuals’
daily lives from the comfort of their own homes. As such, theseinterventions have the potential to mitigate some of the mostcommon barriers to mental health care, including transportation,
time constraints, and stigma. In addition, digital interventions
offer a cost-effective and scalable alternative to traditional
treatment modalities.
A growing body of research supports the efficacy of digitalinterventions (ie, mobile interventions and web-based
interventions), particularly for common mental health problems
such as depression and anxiety [15-19]. Indeed, between 2009
and 2015, the National Institute of Mental Health awarded 404
grants to technology-enabled interventions, and more than 100
randomized trials have been conducted [20]. However, questions
remain about the effectiveness of these tools, particularly inreal-world settings [21], and their reach is often limited. Few
mental health apps developed in the context of university-based
research are being widely used, limiting their potential [22,23].
On the other hand, more than 10,000 commercially developed
mobile apps focused on mental health are now widely available
and easily accessible [24]. However, the vast majority of thesetools lack rigorous testing or evaluation. A systematic review
suggests that as few as 3% of the commercially available mobile
mental health apps have any peer-reviewed evidence [25], andthe apps most commonly downloaded and used contain minimal
evidence-based content [23]. For example, Wasil et al [23]examined the proportion of evidence-based content based onmonthly use data of common apps. They found that common
treatment elements, such as exposure, reached only 2% of users.
As a result, numerous efforts have been made to evaluate
commercially available apps and create app repositories withstandard reporting, and at least 45 frameworks have beendeveloped to assist consumers in identifying the most effective
and appropriate apps [24-26]. However, these efforts have fallen
short. To date, personal searches on commercial app stores
https://formative.jmir.org/2021/9/e27570

XSL• FO
RenderX

Anton et alremain the most common method for finding mental health apps[27], and consumers are looking for new approaches to findthese tools [28].
Partnering with health plans and employers to help disseminate
these tools has the potential to help guide consumers toevidence-based apps that have the greatest potential for impact.
There have been calls for health plans to incorporate digitalsolutions into their behavioral health resources and help theirmembers navigate and discover apps that best meet their needs[29]. This approach may improve behavioral health care use bydirecting users to additional mental health services. In addition,
employers are uniquely situated to link their employees tobehavioral health resources. Adults in the United States spend
most of their time at work, and mental health problems are often
costly to employers because of lost workdays and decreased
productivity. Preliminary research suggests a US $2-$4 returnon investment for every dollar that employers spend on mental
health resources [30]. However, little is known about the effectsof these approaches.

Objectives
There is a need to better understand the impact of digitalinterventions disseminated to real-world users in these new
ways. AbleTo’s Digital+ program (AbleTo Digital+ has
previously been reported on as Joyable [31]. Joyable, Inc, wasacquired by AbleTo, Inc, in March 2019) provides a unique
opportunity to understand the impact of such tools. Digital+ isa web- and mobile-based platform with personalized coachingfor anxiety and depression made available to users throughhealth plans and employer partners. Users are directed to oneof three different programs based on their symptom presentationand goals. All Digital+ programs are based on cognitive
behavioral principles and include psychoeducation, brief
activities (<10 minute), weekly symptom tracking, and 1:1
coaching via phone and in-app messaging. Programs include(1) depression, an eight-module program focused on behavioral
activation; (2) generalized anxiety, an eight-module program
focused on worry exposures and distress tolerance; and (3) social
anxiety, an eight-module program focused on exposure to fearedsocial situations [31]. A critical next step is to better understandthe impact of this tool on real-world users. The primary objectiveof this study is to examine the magnitude of changes in
depression and anxiety symptoms among individuals who usedthe Digital+ programs. The secondary aim is to evaluate Digital+
use across the eight modules of each program and the use of
1:1 coaching.

Methods
Participants
In this retrospective cohort study, we analyzed previously
collected and permanently deidentified data from a consecutive
cohort of 1896 adults enrolled in an AbleTo Digital+ program
between January 1 and June 30, 2020. Participants were aged
≥18 years and were required to have access to their own devices.
Participants were excluded if they reported active suicidal
ideation or psychosis. All study procedures were submitted to
the Sterling institutional review board, Atlanta, Georgia, UnitedStates, and deemed exempt.
JMIR Form Res 2021 | vol. 5 | iss. 9 | e27570 | p. 2
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Procedure
AbleTo, a technology-enabled virtual behavioral health careorganization, partners with employers and health plans to make
Digital+ accessible to employees and covered members in their
networks. Employees and covered members are made aware of
Digital+ through marketing campaigns and employer
communications that explain the potential benefits of engaging
in evidence-based, time-bound digital programs. A variety ofengagement strategies were used to make individuals aware of
the program and facilitate enrollment. These methods included
divulging information about Digital+ on an employer’s benefits
webpage, via posters, flyers, and table tents in office settings,
and through email and text campaigns. Interested participants
were then directed to access Digital+ via the web or a mobile
app to complete a brief survey (approximately 2 minutes) todetermine program appropriateness and create an account.

Description and Structure of the Program
During enrollment, users completed a series of questions
designed to assess their appropriateness for Digital+ andgenerate initial program recommendations. Screening questions
to assess participant goals and primary presenting problems
were used to determine which of Digital+’s three programs—(1)depression, (2) generalized anxiety, or (3) social anxiety—would
best fit the users’ needs. Users had the opportunity to review
the recommendation and select a different program if they feltthat it was not the correct fit. Users then completed one of threestandard assessments to establish baseline symptom severity
and screening questions about suicidality and psychosis designed
to assess safety and risk. Users were also asked to report any
current or prior history of common, high-risk, comorbid
psychiatric challenges (ie, substance use, eating disorders,bipolar disorder, and posttraumatic stress disorder) to further
evaluate the clinical complexity.

https://formative.jmir.org/2021/9/e27570

XSL• FO
RenderX

Anton et alThose deemed inappropriate for Digital+ (eg, active suicidal
ideation and psychosis) were routed to various resources
depending on their employer or health plan. These resources
included, but were not limited to, the National Suicide Hotline,
a Digital+ in-network provider matching service, or linkage totheir health plan to assist in connecting individuals with a higher
level of care. In addition, for some partners, individuals were
offered the opportunity to enroll in a purely self-guided,
five-module skills-based program if they did not wish to engage
with a coach or in any of the primary Digital+ programs.
Individuals who endorsed active suicidal ideation or psychosis
and those in the five-module skills-based program were excluded
from this analysis.
Digital+ programs included eight content modules, a 1:1
coaching option, and mood and symptom assessment. All
psychoeducational materials and activities were based on corecomponents of cognitive behavioral therapy (CBT), including
cognitive restructuring, gradual exposure, and behavioral
activation. Each module consisted of 4 to 6, approximately
10-minute activities, including deep breathing, progressive
muscle relaxation, cognitive restructuring, and behavioral
activation. The activities were organized to help users learn
about CBT, depression, or anxiety; develop skills to challenge
maladaptive thoughts; and practice newly acquired skills in
real-life settings. Each module was designed to be completed
within approximately 1 week, but users were not required tocomplete them in that amount of time. New activities become
available once the previous activity is completed. Once thecontent was made available to users, they were able to revisit
previous modules and activities and complete them as manytimes as desired. At the end of each module, users completed
a standard assessment (refer to the Measures section for more
information) before proceeding to a new module. Users were
presented with feedback on their scores, and these scores wereaccessible to their coaches (see Table 1 for a description of thecontent of each program).

JMIR Form Res 2021 | vol. 5 | iss. 9 | e27570 | p. 3
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Anton et al

Table 1. Description of AbleTo Digital+ programs.
ModulePrograms
Depression

1

Number of
Activities

Content

6••

Generalized anxiety

Social anxiety

Number of
Activities

Number of
Activities

Content

Psychoeducation about 6
depression
Introduction to distress
tolerance skills (eg, deep
breathing)••Content

Psychoeducation about 4stress and anxiety
Introduction to distress
tolerance skills (eg, deep
breathing)••

Psychoeducation aboutsocial anxiety
Introduction to distress
tolerance (eg, deep
breathing)

2

4

•

Introduction to behav5
ioral activation andplanning for first activity•

Introduction to automat- 4
ic thoughts and cognitive
distortions•

Introduction to automatic thoughts and cognitive
distortions

3

4

•

Introduction to automat- 4
ic thoughts and cognitive
distortions•

Psychoeducation about 5
avoidance and exposure
Completion and review
of first exposure activity•

Psychoeducation about
avoidance and exposure
Completion and review
of first exposure activity

•

•

4

3

•

Introduction to values5
based behavioral activation, planning and completion of an activation
exercise•

Practicing cognitive
skills, planning, andcompleting a worrybased exposure

4•

Creating a fear hierarchy
and planning and completing an exposure activity

5

4

•

Plan and complete an
additional values-based
activation

5•

Plan and complete an
additional exposure andmindfulness activity

4•

Pick a new exposure
from the fear hierarchy,
plan, and complete theexposure.

6

4•

Plan and complete an
additional values-based
activation

5•

Plan and complete an
additional worry-based
exposure

4•

Pick a new exposure
from the fear hierarchy,
plan, and complete theexposure

7

4•

Plan and complete an
additional values-based
activation

5•

Plan and complete an
additional exposure andmindfulness activity

4•

Pick a new exposure
from the fear hierarchy,
plan, and complete theexposure

8

6•

Plan and complete an
5
additional final activation
Introduction to core beliefs and preparation formaintaining gains•

Plan and complete final 4
exposure
Introduction to core beliefs and preparation formaintaining gains•

Plan and complete final
exposure
Introduction to core beliefs and preparation formaintaining gains•

Coaching
All users were assigned a coach during program enrollment.
Coaches are nonlicensed professionals with bachelor’s degrees
and relevant work experience, course work, or certification incoaching. They receive intensive training in the principles of
behavior change, motivational interviewing, Digital+ programcontent, and crisis intervention. Coaches are trained to focus
on motivation and engagement by using motivational
interviewing and validating the participants’ experiences withdepression or anxiety. They also help users understand how
program content and activities align with their individual goals,
reinforce activity completion and skill use, and provide
additional accountability through reminders and encouragement.
Coaches are trained in crisis intervention and use the question,
https://formative.jmir.org/2021/9/e27570

XSL• FO
RenderX••

persuade, and refer method of crisis intervention. If a user needs
a higher level of care, coaches will refer to external resources
to better meet the user’s needs. Coaches receive ongoingsupervision from a motivational interviewing certified trainer
and one-on-one supervision to monitor competence in program
knowledge and client service. A subset of coaching calls isreviewed to assess adherence to motivational interviewing
protocols and monitor quality.
Users have the option to engage in coaching by scheduling an
initial kickoff call with their coach during program enrollment,
and a welcome message is sent to all users. Kickoff calls are 30
minutes and are used to orient users to the program, establish
goals, and set expectations. If a user does not schedule a kickoff
call during enrollment, coaches make three attempts to engage
JMIR Form Res 2021 | vol. 5 | iss. 9 | e27570 | p. 4
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
users. After three attempts with no contact, the proactive reachout from the coaches is suspended. Users, however, continue
to have access to coaches and can schedule calls or message
coaches throughout the program, regardless of their initial
preference for coaching, and coaches respond within one
business day. Subsequent weekly 15-minute coaching calls areavailable to users throughout the eight-module program andfocus on reviewing activities, clarifying goals, and encouraging
ongoing use. Users also have access to in-app text messaging.

Measures
Participant Characteristics
Consistent with the standard program experience, users provided
basic demographic information (ie, age and gender) andindicated any current or recent serious behavioral healthconcerns by selecting all that applied from a list of common
comorbid mental health problems, including substance use,bipolar disorder, eating disorder, and posttraumatic stressdisorder.

Outcome Measures
Participants in the depression program completed the Patient
Health Questionnaire-9 (PHQ-9) [32] during enrollment andafter completion of each module. The PHQ-9 is a 9-item
self-report measure designed to evaluate the presence ofdepressive symptoms during the past two weeks. Items are rated
on a scale ranging from 0 (not at all) to 3 (nearly every day).
Total scores range from 0 to 27, and cut-off scores for mild,moderate, moderately severe, and severe depressive symptoms
are 5, 10, 15, and 20, respectively.
Participants in the generalized anxiety program completed theGeneralized Anxiety Disorder-7 (GAD-7) scale [33] during
enrollment and after completion of each module. The GAD-7
is a 7-item self-report measure of anxiety. Items are rated on ascale ranging from 0 (not at all) to 3 (nearly every day). Total
scores range from 0 to 21, and the cut-off scores for mild,moderate, and severe anxiety symptoms are 5, 10, and 15,
respectively.
Participants in the social anxiety program completed the Social
Phobia Inventory (SPIN) [34] during the enrollment and after
completion of each module. The SPIN is a 17-item self-reportquestionnaire that assesses social anxiety symptoms during thepast week. Items are rated on a scale ranging from 0 (not at all)to 4 (extremely). Total scores range from 0 to 68, and the cut-off
scores for mild, moderate, severe, and very severe are 20, 31,
41, and 51, respectively [35,36].

Anton et al
examining the dates of the first and last activity use. In addition,
patterns of coaching use were evaluated using the number ofcoaching calls completed per module and the number ofincoming messages per module.

Statistical Analysis
Baseline demographic and clinical characteristics were reported
as frequencies and percentages for categorical variables andmeans and SDs for continuous variables. Outcome measuresthroughout time were reported as means and SDs. Baselinecharacteristics were compared between those who completed
all eight modules and those who discontinued use before
completion using a two-sample, two-tailed t test for continuous
variables and chi-square tests for categorical variables. Linear
mixed models with a random intercept and slope wereconstructed using restricted maximum likelihood estimation
procedures to examine the association between module
completion and depression, anxiety, and social anxiety symptomchange throughout the Digital+ programs.
First, models were estimated separately for each outcome
measure, and the covariance structure that provided the bestmodel fit was identified. On the basis of model fit indices, thebest-fitting model was carried forward into the covariate models
to examine the associations between baseline participantcharacteristics (ie, baseline symptom severity, age, and gender)and changes in symptom severity throughout time. Participants
with at least mild symptom severity at baseline (ie, PHQ-9 ≥5,
GAD-7 ≥5, or SPIN ≥20) who completed at least four modules
were included in these analyses. Consistent with establishedstandards for internet-based interventions [37], users who hadcompleted four modules were included because they were
exposed to all primary program content at this point (ie,psychoeducation, cognitive restructuring, and practice activities).
In addition, we examined within-subject effect sizes and theassociation between module completion and symptom reduction
using linear mixed models among users who enrolled with atleast mild baseline symptom elevations, regardless ofintervention exposure (n=1694). Digital+ module completion,completed coaching calls, and incoming and outgoing messages
were reported using frequencies and means. The association
between module completion and baseline scores was assessedusing unadjusted linear regression. Program retention or timespent using Digital+ was defined as the number of days between
completing the first and last recorded activities. All analyses
were conducted using SAS 9.4.

Results

Digital+ Use

Participants

Patterns of Digital+ activity and module completion were
explored using passive data collected through a digital platform.
These data included whether the modules were started and/or
completed. Time spent using Digital+ was assessed by

A total of 1896 participants were enrolled and initiated the useof Digital+ between January 1, 2020, and June 30, 2020. The
baseline characteristics of the users are shown in Table 2.

https://formative.jmir.org/2021/9/e27570

XSL• FO
RenderX

JMIR Form Res 2021 | vol. 5 | iss. 9 | e27570 | p. 5
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Anton et al

Table 2. Baseline sample characteristics (N=1896)a.
Variables

Total (N=1896)

Depression (n=560)

Generalized anxiety (n=974)

Social anxiety (n=362)

Age (years), mean (SD)37.4 (11.8)

37.1 (12.2)

38.2 (11.7)

35.9 (11.6)

Female1196 (63.1)

356 (63.6)

636 (65.4)

204 (56.4)

Male

470 (24.8)

147 (26.3)

231 (23.7)

92 (25.4)

Nonbinary

14 (0.7)

7 (1.3)2 (0.2)5 (1.4)Not disclosed

215 (11.3)

50 (8.9)104 (10.7)

61 (16.9)

Missing

1 (0.1)0 (0)1 (0.1)0 (0)Employer1359 (71.7)

399 (71.3)

667 (68.5)

293 (80.9)

Health plan

537 (28.3)

161 (28.8)

307 (31.5)

69 (19.1)

Gender, n (%)

Source, n (%)

Patient-reported psychiatric historya, n (%)
Substance use

29 (1.5)

15 (2.7)

4 (0.4)10 (2.8)

Eating disorder

145 (7.6)

61 (10.9)

56 (5.8)

28 (7.7)

Bipolar disorder

83 (4.4)

36 (6.4)

31 (3.2)

16 (4.4)

Passive suicidal ideation

95 (5)

44 (7.9)31 (3.2)

20 (5.5)

Posttraumatic stress disorder

117 (6.2)

39 (7)

57 (5.9)21 (5.8)aIndividuals could endorse more than one psychiatric difficulty. In total, 17.51% (332/1896) of users reported at least one current or recent psychiatricchallenge.

Of those enrolled, 29.54% (560/1896) enrolled in the depression
program, 51.37% (974/1896) enrolled in the generalized anxiety
program, and 19.09% (362/1896) enrolled in the social anxiety
program. At baseline, 95.2% (533/560) of users met the criteria
for at least mild depression on the PHQ-9 (ie, total score ≥5).
The criteria for at least mild anxiety on the GAD-7 scale (ie,total score ≥5) were met by 90.3% (879/974) of users in thegeneralized anxiety program. The criterion for social anxiety(ie, total score ≥20) was met by 77.9% (282/362) of users inthe social anxiety program.
Users were asked to select any current or recent psychiatricconcerns from a list of common comorbidities during programenrollment. At least one current or recent psychiatric disorder
was endorsed by 17.51% (332/1896) of users. Users who
endorsed at least one recent or current psychiatric concern had
significantly higher baseline PHQ-9 (14.8 vs 11.8; P<.001),
GAD-7 (14.2 vs 11.0; P<.001), and SPIN (35.7 vs 30.0; P=.002)
scores than users who did not endorse any recent or current
psychiatric concerns.

Clinical Outcomes
Descriptive statistics for the outcome measures throughout time
are summarized in Table 3 for all the users.

https://formative.jmir.org/2021/9/e27570

XSL• FO
RenderX

Among users who completed at least half of the program content
and had mild baseline symptom elevations (n=470), there weresignificant reductions in PHQ-9, GAD-7, and SPIN scores.
These reductions corresponded to large within-group effectsizes across all three programs among those who completed atleast four modules. The effect sizes for depression, generalized
anxiety, and social anxiety symptoms were 1.5, 1.2, and 1.5,
respectively. Similarly, the within-group effect sizes among
those participants who completed all eight modules were large
across all three programs. The effect sizes for depression,
generalized anxiety, and social anxiety symptoms were 1.7, 1.7,
and 1.4, respectively.
Of those who completed at least four modules in the depression
program and had at least moderate depressive symptoms atbaseline (PHQ-9 ≥10; mean 15.0, SD 3.6), 75.7% (87/115) ofparticipants met the criteria for treatment response (PHQ-9 <10;mean 6.6, SD 4.8). In addition, 73.9% (122/165) of participantsin the generalized anxiety program who completed four modules
and had at least moderate symptoms (GAD-7 ≥10) at baseline(mean 15.0, SD 3.1) met the criteria for response (GAD-7 <10)
at their last assessment (mean 7.0, SD 4.6). For the social anxiety
program, among users who completed at least four modules andhad SPIN scores of 20 and higher at baseline (mean 37.0, SD
11.0), 42% (31/74) of participants achieved a response(SPIN<20) by their last assessment (mean 24.9, SD 14.5).

JMIR Form Res 2021 | vol. 5 | iss. 9 | e27570 | p. 6
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Anton et al

Table 3. Symptom severity by module completed (N=1896).
OutcomeModule 2 Module 3 Module 4 Module 5 Module 6 Module 7 Module 8 Users who com- Users who com(n=976)(n=700)(n=525)(n=430)(n=343)(n=305)(n=267)pleted ≥4 mod- pleted all 8
ules
modules
Effectsize(Cohend)P
value

Effectsize(Cohen
d)

P
value

135
(24.1)

119(21.3)

102(18.2)

N/Aa

N/A

N/A

N/A

7.20 (4.8) 6.3 (4.9)

5.5 (4.7)

5.0 (5.0)

4.8 (4.8)

1.5<.001 1.7<.001

177 (77)122(67.4)

94 (58.9)

64 (47.4)

49 (41.2)

33 (32.3)

N/A

N/A

N/A

N/A

130
(47.6)

77 (33.5)

46 (25.4)

29 (18.5)

22 (16.3)

18 (15.1)

13 (12.8)

N/A

N/A

N/A

N/A

Generalized
974 (51.4)
anxiety, n (%)

500(51.3)

342(35.1)

248(25.5)

206(21.2)

159(16.3)

136 (14)126(12.9)

N/A

N/A

N/A

N/A

GAD-7c,mean (SD)

11.4 (5.2)

10.13
(4.9)

8.14 (4.2) 7.4 (4.3)

6.87 (4.3) 6.59 (4.5) 5.1 (3.8)

4.7 (3.6)

1.2<.001 1.7<.001

GAD-7 >4, n(%)

879 (90.3)

432(86.4)

270 (79)180(72.6)

141(68.5)

101(63.5)

76 (52.8)

61 (48.4)

N/A

N/A

N/A

N/A

GAD-7 >9, n(%)

589 (60.5)

262(52.4)

116(33.9)

67 (27)44 (21.4)

32 (20.1)

16 (11.1)

13 (10.3)

N/A

N/A

N/A

N/A

Social anxiety, 362 (19.1)
n (%)

203
(56.1)

128(35.3)

96 (26.5)

67 (18.5)

51 (14.1)

40 (11.1)

39 (10.8)

N/A

N/A

N/A

N/A

SPINd, mean(SD)

31.1 (13.8) 31.2
(13.9)

26.1(14.9)

24.8(14.3)

21.5(13.4)

18.9(11.8)

16.7(11.8)

14.7(11.6)

1.0<.001 1.7<.001

SPIN >19, n(%)

282 (77.9)

77 (60.2)

58 (60.4)

34 (50.8)

22 (43.1)

15 (37.5)

11 (28.2)

N/A

N/A

N/A

Depression, n(%)

aModule 1(N=1896)560 (29.5)

273(48.8)

230(41.1)

181(32.4)

PHQ-9b, mean 12.5 (5.6)(SD)

9.9 (5.3)

8.4 (4.9)

PHQ-9 >4, n(%)

533 (95.2)

231(84.6)

PHQ-9 >9, n(%)

368 (65.7)

155(76.4)

157 (28)

N/A

N/A: not applicable.

b

PHQ-9: Patient Health Questionnaire-9.

c

GAD-7: Generalized Anxiety Disorder-7.

d

SPIN: Social Phobia Inventory.

Model-implied estimates for PHQ-9, GAD-7, and SPIN scoresby program module are presented in Figure 1. In the linear
mixed model to evaluate associations between depression
program module completion and symptom reduction amongusers with at least mild baseline symptom severity (PHQ-9 ≥5)and exposure to at least four modules (n=169), significant fixedhttps://formative.jmir.org/2021/9/e27570

XSL• FO
RenderX

effects of module completion (F1972=319.4; P<.001) andbaseline PHQ-9 scores (F1164=454.9; P<.001) but not age(F1164=1.2; P=.19) or gender (F2164=0.82; P=.44) were observed.
There was also significant variability around the mean intercept(P=.001) and slope (P<.001) and significant residual variance(P<.001).

JMIR Form Res 2021 | vol. 5 | iss. 9 | e27570 | p. 7
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Anton et al

Figure 1. Estimated mean outcome scores by module for users who completed at least four modules. (A) depression program; (B) generalized anxiety
program; and (C) social anxiety program. GAD-7: Generalized Anxiety Disorder-7; PHQ-9: Patient Health Questionnaire-9; SPIN: Social Phobia
Inventory.

Similarly, there were significant fixed effects of anxiety program
module completion (F11,258=332.5; P<.001) and baseline GAD-7
scores (F1222=492.2; P<.001) but not age (F1222=0.17; P=.68)
or gender (F2222=0.40; P=.67) on changes in symptom severity
throughout time among users who had at least mild baseline
generalized anxiety disorder symptoms (GAD-7 >5) andcompleted at least four modules of content (n=227). There wasalso significant variability around the mean intercept (P=.02)
and slope (P<.001) and significant residual variance (P<.001).

were 0.58, 0.50, and 0.44 for depression, generalized anxiety,and social anxiety symptom scores, respectively. There was asignificant association between the number of modules
completed and symptom reduction on the PHQ-9 (P<.001),
GAD-7 (P<.001), and SPIN (P<.001) scores. Across all threeprograms, higher baseline symptom severity was significantly
associated with slower symptom reduction (P<.001). There
were no significant relationships among age, gender, andsymptom reduction.

In the linear mixed effects model among social anxiety program
users with elevated baseline symptoms (SPIN >20) exposed tofour or more digital modules (n=74), there were significant
fixed effects of module completion (F1366=114.9; P<.001),
baseline SPIN scores (F169=326.3; P<.001), and age (F169=9.2;
P=.004), but not gender (F269=1.3; P=.27). There was alsosignificant variability around the mean intercept (P=.02) andslope (P<.001), and there remained significant residual variance(P<.001).

Digital+ Use

To understand the impact of the intervention as delivered, we
examined primary outcomes among all users who enrolled withat least mild baseline symptom elevations, regardless ofintervention exposure (n=1694). The within-person effect sizes
https://formative.jmir.org/2021/9/e27570

XSL• FO
RenderX

Of those who enrolled, 27.9% (529/1896) completed at least
half of the program content (ie, four modules), and 13.08%(248/1896) completed all eight modules in their respective
programs. Among users who completed all eight modules in
one program and those who did not, there were no significant
differences in mean age (38.5 vs 37.3 years; P=.13), gender(26.21% female (313/1196) vs 24.59% (116/470) male; P=.14),
or mean baseline scores on the GAD-7 (12.1 vs 11.3; P=.14)
or SPIN (30.1 vs 31.2; P=.66). However, there were significant
differences in baseline PHQ-9 scores (P=.01) between
individuals who completed all program content (mean 11.2, SD
5.1) and those who discontinued (mean 12.8, SD 5.7).

JMIR Form Res 2021 | vol. 5 | iss. 9 | e27570 | p. 8
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
Digital+ users completed 2.6 (SD 2.7) modules on average.
Participants in the depression program (mean 2.9, SD 3.0)
completed significantly (F2=4.2; P=.01) more modules thanthose in the generalized (mean 2.5, SD 2.7) and social anxiety(mean 2.5, SD 2.4) programs. On average, the length of timefrom first to last activity was 52.2 (SD 83.5) days, and of the1896 participants who initiated the use of Digital+, 45.09%(855/1896) were retained at 30 days and 26.85% (509/1896)
were retained at 60 days.
Higher baseline depressive symptomatology was associatedwith fewer modules completed (β=−.1; P=.02). However, higher
baseline generalized anxiety symptomatology was associatedwith more modules completed (β=.1; P=.01). Gender was alsosignificantly associated with the number of modules completed(F3=4.3; P=.01), with women completing 2.7 modules (SD 2.7)
and men completing 2.6 modules (SD 2.7) on average. However,there was no significant relationship between the number ofmodules completed and baseline SPIN scores, age, or endorsing
at least one recent or current psychiatric concern.

Coaching
Overall, 66.82% (1267/1896) of participants engaged in at least
one coaching call. On average, users scheduled 3.8 (SD 4.4)
calls and completed 3.1 (SD 4.2) calls throughout theirprograms. Those who completed at least four modules (mean1.1, SD 0.7) completed significantly more calls per module thanthose who completed fewer than four modules (mean 1.0, SD
1.2; t1407=−2.1; P=.03). In addition, users who completed atleast one coaching call (mean 3.4, SD 2.8) completed
significantly more modules than users who never engaged incoaching calls (mean 1.0, SD 1.5; t1894=−20.3; P<.001).
There was a significant relationship between age and the numberof calls completed, such that older participants completed morecalls than younger participants (β=.02; P=.01). In addition,baseline depression (β=−.1; P=.03) and generalized anxiety(β=.1; P=.01) symptom severity scores were associated with
the number of calls completed. In the depression program, users
with higher baseline levels of depressive symptomatology
completed fewer calls than those with lower baseline
symptomatology. In contrast, in the generalized anxiety
program, users with higher baseline levels of anxiety symptoms
completed more calls than those with lower baseline symptom
severity. There were no significant associations among program(P=.21), gender (P=.16), or current or recent psychiatricdifficulties (P=.89); baseline social anxiety symptom severity(P=.92); and the number of calls completed.
In-app messaging was used by most participants, with 66.09%(1253/1896) of users sending at least one in-app coach text
message and an average of 4.5 (SD 11.1) messages sent tocoaches. Users who completed fewer than four modules sent
approximately one message to every seven messages coaches
sent, whereas users who completed at least four modules sent
approximately one message to every four messages fromcoaches (14.28% vs 25.75%; P<.001). There were no significant
associations between program (P=.13), gender (P=.51), baseline
depression (P=.68), or social anxiety (P=.76) symptom severity
and the number of messages sent to coaches. However, there
https://formative.jmir.org/2021/9/e27570

XSL• FO
RenderX

Anton et al
was a significant relationship between baseline generalized
anxiety symptom severity and the number of messages users
sent to coaches, such that higher baseline anxiety was associatedwith more sent messages (β=.2; P<.001). Age was alsosignificantly related to the number of messages sent, such thatolder users sent coaches significantly fewer messages than
younger users (β=−0.1; P=.03). Finally, there was also asignificant relationship between user-endorsed recent or current
psychiatric concerns and the number of incoming messages
from users (F1=4.24; P=.04), such that those who endorsed atleast one concurrent psychiatric challenge sent coaches
significantly more messages (mean 5.6, SD 10.4) than those
who did not report any recent or current psychiatric concerns(mean 4.2, SD 11.3).

Discussion
Principal Findings
This study adds to a small yet growing body of literature
examining the magnitude of symptom reduction among usersof commercially available digital mental health apps in areal-world context. Overall, Digital+ programs demonstrated
significant reductions in depressive, generalized anxiety, andsocial anxiety symptoms throughout the program. The
magnitude of these effects appeared to grow as more modules
were completed; however, users who completed at least half ofthe program content also experienced significant and large
reductions in symptomatology. These improvements werelargely consistent across participant characteristics (ie, age andgender). In all three programs, those with more severe baseline
symptomatology experienced smaller symptom reductions thanthose with lower baseline symptom severity. Of the 1896
AbleTo Digital+ users who initiated use between January 1,
2020, and June 30, 2020, 27.90% (529/1896) completed halfof the program content, and 13.08% (248/1896) completed all
the program content, with a 30-day retention rate of 45.09%(855/1896). Completion rates were consistent across age, gender,
and baseline generalized and social anxiety symptom severity.
Users with more severe baseline depressive symptomatology
completed fewer modules than those with lower baseline scores.
Almost all Digital+ users endorsed interest in 1:1 coaching, andapproximately two-thirds of users engaged in at least one
coaching call (66.2%, 1267/1896) and/or in-app text messaging(66.09%, 1253/1896). Users who completed at least four
modules completed significantly more coaching calls per module
than users who completed fewer than four modules.

Comparison With Previous WorkRegarding clinical outcomes, the magnitude of symptom
reduction among participants in Digital+ programs wascomparable with or larger than those seen in the broader
literature for internet-based CBT [38-40] and similar digitalproducts for anxiety and depression [41,42]. For example, arecent meta-analysis suggests that in randomized trials,
smartphone apps had moderate positive effects on depression[18] and anxiety symptomatology [17]. Also consistent withbroader literature, these findings suggest comparable outcomes
across several key demographic variables, including age andgender. Thus, this study adds to the growing body of literature
JMIR Form Res 2021 | vol. 5 | iss. 9 | e27570 | p. 9
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
supporting the efficacy of these interventions for a broad range
of users [18]. Notably, the magnitude of symptom improvement
is similar to effect sizes seen in studies of face-to-face CBT[43,44], although it is important to note that individuals engaging
in Digital+ may differ from those presenting to face-to-facetreatment.
AbleTo Digital+’s completion rate was comparable with or
higher than that reported in a previous meta-analysis of similar
digital interventions (ie, 0.5%-28.6%) [45], and the 45% 30-day
retention rate was approximately 15 times higher than the 3.3%
30-day retention rate previously reported for other commercial
mental health apps without human support, which is known tobe key for improving engagement [46]. Although the use andretention rates were relatively high compared with othercommercially available apps, there was still a substantial dropoff, particularly earlier in the programs. For digital interventions
to reach their full potential, optimizing engagement is necessary
and remains a top research priority [47,48]. We hypothesized
several reasons for premature dropouts. First, Digital+ may not
be the appropriate intervention for some individuals, and theyprematurely drop out because the intervention did not meet theirneeds. A key priority should be to link individuals to theappropriate level of care, and strategies are needed to link users
to appropriate services. Digital+ coaches were trained to assist
users in finding a more appropriate level of care if needed bylinking them to their health plan resources or a Digital+
in-network provider matching service, which helped users locateand schedule appropriate care. Alternatively, it is possible thatsome individuals left the program owing to feeling better orwhat has been termed happy abandonment [49]. Although some
Digital+ users experienced significant symptom reduction whencompleting only half of the program content, the long-term
impact of premature dropout among these individuals remains
unknown. Finally, given the finding that individuals with more
severe baseline depression were more likely to discontinue using
Digital+, motivation may be a primary target for sustained
engagement. Future research is required to explore the reasonsfor and predictors of premature dropout, including both userand program-related factors and the clinical implications ofpremature dropout.
Human support has consistently been shown to improveengagement in and sustained use of digital interventions [50-52];
however, much remains unknown about the aspects of coaching
most closely linked to improved engagement [53]. Researchhas demonstrated that coaching focuses on reminding users toengage, and providing personalized feedback on completed
content can boost engagement [53-55]. However, questions
remain about the timing, intensity, and structure of coachingassociated with higher engagement. More intensive coachingearlier in Digital+ programs may be needed to decreasepremature dropout, particularly among groups of users who are
more prone to drop out. In addition, one-third of the users did
not engage in 1:1 coaching. Given the relationship between
coaching and engagement, strategies are needed to promotecoaching initiation among those most likely to benefit fromadditional support. Future research is also necessary to better
understand the impact of coaching interactions on engagement
and outcomes and develop and test tailored coaching strategies,
https://formative.jmir.org/2021/9/e27570

XSL• FO
RenderX

Anton et al
including testing different coaching techniques, methods, anddoses.
Although users with a broad range of symptom presentations
benefited from Digital+, there was a consistent finding acrossprograms that those with more severe baseline symptomatology
experienced less improvement throughout the programs. In thecase of depression, these individuals were also more likely to
discontinue use prematurely. These findings are consistent withand add to the randomized controlled trial literature suggestingthat digital interventions have the most consistent benefit for
those experiencing mild to moderate symptoms at the time ofinitiation [18,56]. As digital mental health interventions continue
to evolve, it is critical that we better understand who is most
likely to benefit from these interventions to direct patients tothe appropriate level of care and optimize outcomes.
Several stepped and staged care models that incorporate digitalinterventions into the broader behavioral health care system are
emerging. For example, the United Kingdom’s National Health
System’s Improving Access to Psychological Treatments [57]and Australia’s Mindspot Clinics [58] have integrated
computerized and digital interventions into their suite oftreatment options to improve efficiency and access to care. In
both of these systems, digital interventions are considered
first-line interventions that can be effective for those with mild
to moderate symptoms and serve as a gateway to additional ormore intensive interventions, if needed. Incorporatinginterventions such as Digital+ into the behavioral health caresystem offerings has the potential to reserve more intensive
interventions for those who need them the most, decrease wait
times, and mitigate worsening workforce shortages. As we learn
more about who is likely to engage and benefit, we can use
data-driven approaches combined with personal preferences todirect people to timely and appropriate services.
Finally, despite a growing body of literature supporting theefficacy of these interventions and consumer interest, consumers
continue to struggle to identify high-quality, evidence-based
products [27,28]. Novel approaches to disseminating tools suchas Digital+ are needed to help people identify the tools best
supported by research evidence that are most likely to help them.
Research suggests that the interest in digital mental healthinterventions outpaces uptake and that individuals desire easieraccess to information on effective tools and look to trusted
providers for information on which tools to use [28]. Healthplans and employers may be uniquely situated to help streamline
the dissemination of these tools and direct users to the best
possible services by adding them to their suite of behavioral
health care offerings. This approach can boost awareness andenhance credibility. This study demonstrates the potential ofthis approach; however, future research is needed to continue
to evaluate the impact of these dissemination methods,
particularly concerning improving access and decreasing healthcare and workplace costs.

Limitations
The results of this study must be interpreted in light of some
limitations. First, the lack of a control group means that we wereunable to account for the natural remission of symptoms, theeffect of missing data, or fully understand the impact of 1:1
JMIR Form Res 2021 | vol. 5 | iss. 9 | e27570 | p. 10(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
coaching on engagement and outcomes. This limitation,
however, was mitigated to some extent by the weekly collectionof symptom severity scores and conservative statistical modelingthat incorporated those who did not complete all intervention
content [59]. In addition, the real-world context of this study isseen as a primary strength in that it establishes proof of concept
for a health plan and employer rollout of mental health apps
while establishing clinical impact.
Second, only a small number of baseline characteristics werecollected to minimize user burden. However, this limited ourability to evaluate a broader set of factors that might be related
to engagement and treatment response. The results suggest thatabove and beyond the known participant characteristics (eg,age, gender, and baseline symptom severity), there wassignificant variability unaccounted for in both engagement andtreatment response. Although age and gender were notassociated with treatment response, indicating the potential
broad applicability of Digital+, additional information on
participant demographics, including race or ethnicity andsocioeconomic status, may help us better understand who ismost likely to benefit. In addition, other data such as treatment
history, concurrent treatment, and psychiatric medication use
were not available. Collecting these data in the future would
allow us to better understand how psychiatric treatment history
affects program retention and response and potentially informthe integration of Digital+ into staged or stepped care models.
Third, these analyses precluded examining the impact of suicidal
ideation and/or psychosis on outcomes, app use, or 1:1 coaching.
Future research is needed to better understand the impact ofthese risk factors. Fourth, all users needed access to a device

Anton et al(eg, smartphone, tablet, or computer) and internet access toparticipate in Digital+. Although 81% of adults in the UnitedStates own a smartphone [60], this may limit the generalizabilityof these results to individuals who do not have access to a device
or the internet. Finally, participants were not followed beyond
the program period, making it difficult to draw conclusions
about the sustained impact of Digital+ programs. This may beparticularly important for users who discontinued use before
full program completion yet showed symptom improvement.
Future research is needed to understand the long-term clinicalimpact of tools such as Digital+.

Conclusions
This study demonstrated that Digital+ users experienced
significant reductions in depression, generalized anxiety, andsocial anxiety symptoms throughout the programs, independentof user age, gender, and baseline symptom severity. Overall,
30-day retention rates were significantly higher than previously
reported rates for other commercially available mental healthapps, particularly self-guided ones. In addition, users whocompleted at least half of the program content completed more
1:1 coaching calls than users who completed fewer than half ofthe program content. Participants who enrolled in Digital+
through their employees and health plan benefits experienced
clinically significant symptom reductions. Digital+ may offera scalable, low-cost additional service that may help mitigate
workforce shortages and other common barriers to treatment.
Future research is needed to continue to identify those who aremost likely to benefit from these apps and examine how best
to integrate digital interventions such as Digital+ into the broader
behavioral health care system.

Conflicts of Interest
MTA is an employee of AbleTo. HMG is an employee of and holds an equity interest in AbleTo. EA is an employee of AbleTo.
RLP is an employee of and holds an equity interest in AbleTo; she serves as AbleTo’s Chief Medical Officer.

References
1.
2.

3.
4.

5.

6.
7.
8.

Cuijpers P, van Straten A, Andersson G, van Oppen P. Psychotherapy for depression in adults: a meta-analysis of comparative
outcome studies. J Consult Clin Psychol 2008 Dec;76(6):909-922. [doi: 10.1037/a0013075] [Medline: 19045960]
Hofmann SG, Smits J. Cognitive-behavioral therapy for adult anxiety disorders: a meta-analysis of randomized
placebo-controlled trials. J Clin Psychiatry 2008 Apr;69(4):621-632 [FREE Full text] [doi: 10.4088/jcp.v69n0415] [Medline:
18363421]
Mental illness. National Institute of Mental Health. URL: https://www.nimh.nih.gov/health/statistics/mental-illness.
shtml#part_154787 [accessed 2021-08-18]
Mohr DC, Ho J, Duffecy J, Baron KG, Lehman KA, Jin L, et al. Perceived barriers to psychological treatments and theirrelationship to depression. J Clin Psychol 2010 Apr;66(4):394-409 [FREE Full text] [doi: 10.1002/jclp.20659] [Medline:
20127795]
Weissman J, Russell D, Jay M, Beasley JM, Malaspina D, Pegus C. Disparities in health care utilization and functional
limitations among adults with serious psychological distress, 2006-2014. Psychiatr Serv 2017 Jul 01;68(7):653-659. [doi:10.1176/appi.ps.201600260] [Medline: 28412896]
Lake J, Turner MS. Urgent need for improved mental health care and a more collaborative model of care. Perm J
2017;21:17-024 [FREE Full text] [doi: 10.7812/TPP/17-024] [Medline: 28898197]
Mental health care matters. National Alliance on Mental Illness. URL: https://www.nami.org/NAMI/media/NAMI-Media/
Infographics/NAMI-Mental-Health-Care-Matters-FINAL.pdf [accessed 2021-08-18]
Major depression. National Institute of Mental Health. URL: https://www.nimh.nih.gov/health/statistics/major-depression[accessed 2021-08-18]

https://formative.jmir.org/2021/9/e27570

XSL• FO
RenderX

JMIR Form Res 2021 | vol. 5 | iss. 9 | e27570 | p. 11(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
9.

10.

11.

12.
13.

14.

15.

16.

17.

18.

19.

20.
21.
22.
23.

24.

25.

26.
27.

28.

29.

Czeisler M, Lane RI, Petrosky E, Wiley JF, Christensen A, Njai R, et al. Mental health, substance use, and suicidal ideation
during the COVID-19 pandemic - United States, June 24-30, 2020. MMWR Morb Mortal Wkly Rep 2020 Aug
14;69(32):1049-1057 [FREE Full text] [doi: 10.15585/mmwr.mm6932a1] [Medline: 32790653]
Ettman CK, Abdalla SM, Cohen GH, Sampson L, Vivier PM, Galea S. Prevalence of depression symptoms in US adults
before and during the COVID-19 pandemic. JAMA Netw Open 2020 Sep 01;3(9):e2019686-e201968- [FREE Full text][doi: 10.1001/jamanetworkopen.2020.19686] [Medline: 32876685]
Rouse WB, Johns MM, Pepe KM. Service supply chains for population health: overcoming fragmentation of service delivery
ecosystems. Learn Health Syst 2019 Feb 5;3(2):e10186-e1018- [FREE Full text] [doi: 10.1002/lrh2.10186] [Medline:
31245604]
The ripple effect of mental illness. National Alliance on Mental Illness. 2020. URL: https://nami.org/NAMI/media/
NAMI-Media/Infographics/NAMI-Impact-Ripple-Effect-FINAL.pdf [accessed 2021-08-18]
Economic burden among patients with major depressive disorder: an analysis of healthcare resource use, work productivity,and direct and indirect costs by depression severity. Am J Manag Care. URL: https://ajmc.s3.amazonaws.com/_media/_pdf/
AJMC_A896_02_2019_EconomicBurden.pdf [accessed 2021-08-18]
Greenberg PE, Fournier A, Sisitsky T, Pike CT, Kessler RC. The economic burden of adults with major depressive disorderin the United States (2005 and 2010). J Clin Psychiatry 2015 Feb;76(2):155-162 [FREE Full text] [doi:10.4088/JCP.14m09298] [Medline: 25742202]
Ebert DD, Cuijpers P, Muñoz RF, Baumeister H. Prevention of mental health disorders using internet- and mobile-based
interventions: a narrative review and recommendations for future research. Front Psychiatry 2017 Aug 10;8:116 [FREEFull text] [doi: 10.3389/fpsyt.2017.00116] [Medline: 28848454]
Andersson G, Carlbring P, Titov N, Lindefors N. Internet interventions for adults with anxiety and mood disorders: anarrative umbrella review of recent meta-analyses. Can J Psychiatry 2019 Jul;64(7):465-470 [FREE Full text] [doi:10.1177/0706743719839381] [Medline: 31096757]
Firth J, Torous J, Nicholas J, Carney R, Rosenbaum S, Sarris J. Can smartphone mental health interventions reduce symptomsof anxiety? A meta-analysis of randomized controlled trials. J Affect Disord 2017 Aug 15;218:15-22 [FREE Full text] [doi:10.1016/j.jad.2017.04.046] [Medline: 28456072]
Firth J, Torous J, Nicholas J, Carney R, Pratap A, Rosenbaum S, et al. The efficacy of smartphone-based mental healthinterventions for depressive symptoms: a meta-analysis of randomized controlled trials. World Psychiatry 2017
Oct;16(3):287-298 [FREE Full text] [doi: 10.1002/wps.20472] [Medline: 28941113]
Romijn G, Batelaan N, Kok R, Koning J, van Balkom A, Titov N, et al. Internet-delivered cognitive behavioral therapy foranxiety disorders in open community versus clinical service recruitment: meta-analysis. J Med Internet Res 2019 Apr
17;21(4):e11706 [FREE Full text] [doi: 10.2196/11706] [Medline: 30994462]
Technology and the future of mental health treatment. National Institute of Mental Health. URL: https://www.nimh.nih.gov/
health/topics/technology-and-the-future-of-mental-health-treatment/index.shtml [accessed 2021-08-18]
Torous J, Powell A. Current research and trends in the use of smartphone applications for mood disorders. Internet Interv
2015 May;2(2):169-173. [doi: 10.1016/j.invent.2015.03.002]
Buss J, Lorenzo-Luaces L, Banks G, Horani D, Rutter L, Wasil A. Availability of internet-based cognitive behavioral
therapy (iCBT) for depression: a systemic review. PsyArXiv. Preprint posted May 20, 2020. [doi: 10.31234/osf.io/2jwgd]
Wasil AR, Gillespie S, Patel R, Petre A, Venturo-Conerly KE, Shingleton RM, et al. Reassessing evidence-based content
in popular smartphone apps for depression and anxiety: developing and applying user-adjusted analyses. J Consult Clin
Psychol 2020 Nov;88(11):983-993. [doi: 10.1037/ccp0000604] [Medline: 32881542]
Torous J, Andersson G, Bertagnoli A, Christensen H, Cuijpers P, Firth J, et al. Towards a consensus around standards for
smartphone apps and digital mental health. World Psychiatry 2019 Feb;18(1):97-98 [FREE Full text] [doi:10.1002/wps.20592] [Medline: 30600619]
Larsen ME, Huckvale K, Nicholas J, Torous J, Birrell L, Li E, et al. Using science to sell apps: evaluation of mental healthapp store quality claims. NPJ Digit Med 2019 Mar 22;2:18 [FREE Full text] [doi: 10.1038/s41746-019-0093-1] [Medline:
31304366]
Henson P, David G, Albright K, Torous J. Deriving a practical framework for the evaluation of health apps. Lancet Digit
Health 2019 Jun;1(2):e52-e54 [FREE Full text] [doi: 10.1016/S2589-7500(19)30013-5] [Medline: 33323229]
Schueller SM, Neary M, O'Loughlin K, Adkins EC. Discovery of and interest in health apps among those with mental
health needs: survey and focus group study. J Med Internet Res 2018 Jun 11;20(6):e10141 [FREE Full text] [doi:10.2196/10141] [Medline: 29891468]
Lipschitz J, Miller CJ, Hogan TP, Burdick KE, Lippin-Foster R, Simon SR, et al. Adoption of mobile apps for depression
and anxiety: cross-sectional survey study on patient interest and barriers to engagement. JMIR Ment Health 2019 Jan
25;6(1):e11334 [FREE Full text] [doi: 10.2196/11334] [Medline: 30681968]
Medicaid and digital health: findings from the Deloitte 2018 survey of us health care consumers. Deloitte. 2018. URL:
https://www2.deloitte.com/us/en/insights/industry/public-sector/mobile-health-care-app-features-for-patients.html [accessed2021-08-18]

https://formative.jmir.org/2021/9/e27570

XSL• FO
RenderX

Anton et al

JMIR Form Res 2021 | vol. 5 | iss. 9 | e27570 | p. 12(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
30.
31.

32.
33.
34.

35.
36.
37.
38.

39.

40.

41.

42.

43.
44.

45.

46.

47.

48.

49.

50.

51.

Digital tools and solutions for mental health: an employer's guide. Northeast Business Group on Health and One Mind
PsyberGuide. 2020. URL: https://online.flippingbook.com/view/911432/ [accessed 2021-08-18]
Dryman MT, McTeague LM, Olino TM, Heimberg RG. Evaluation of an open-access CBT-based internet program forsocial anxiety: patterns of use, retention, and outcomes. J Consult Clin Psychol 2017 Oct;85(10):988-999. [doi:10.1037/ccp0000232] [Medline: 28650193]
Kroenke K, Spitzer RL, Williams JB. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med 2001
Sep;16(9):606-613 [FREE Full text] [doi: 10.1046/j.1525-1497.2001.016009606.x] [Medline: 11556941]
Spitzer RL, Kroenke K, Williams JB, Löwe B. A brief measure for assessing generalized anxiety disorder: the GAD-7.
Arch Intern Med 2006 May 22;166(10):1092-1097. [doi: 10.1001/archinte.166.10.1092] [Medline: 16717171]
Connor KM, Davidson JR, Churchill LE, Sherwood A, Foa E, Weisler RH. Psychometric properties of the Social Phobia
Inventory (SPIN). New self-rating scale. Br J Psychiatry 2000 Apr;176:379-386. [doi: 10.1192/bjp.176.4.379] [Medline:
10827888]
Chukwujekwu C, Emmanuel O. Validation of the Social Phobia Inventory (SPIN) in Nigeria. J Psychiatry Psychiatric
Disord 2018 Apr 27;2(2):49-54. [doi: 10.26502/jppd.2572-519x0040]
Lauria-Horner B. The Primary Care Toolkit for Anxiety and Related Disorders: Quick, Practical Solutions for Assessment
and Management. Canada: Brush Education Inc; Aug 10, 2017.
Christensen H, Griffiths KM, Farrer L. Adherence in internet interventions for anxiety and depression. J Med Internet Res
2009 Apr 24;11(2):e13 [FREE Full text] [doi: 10.2196/jmir.1194] [Medline: 19403466]
Etzelmueller A, Vis C, Karyotaki E, Baumeister H, Titov N, Berking M, et al. Effects of internet-based cognitive behavioral
therapy in routine care for adults in treatment for depression and anxiety: systematic review and meta-analysis. J MedInternet Res 2020 Aug 31;22(8):e18100 [FREE Full text] [doi: 10.2196/18100] [Medline: 32865497]
Josephine K, Josefine L, Philipp D, David E, Harald B. Internet- and mobile-based depression interventions for people withdiagnosed depression: a systematic review and meta-analysis. J Affect Disord 2017 Dec 01;223:28-40. [doi:10.1016/j.jad.2017.07.021] [Medline: 28715726]
Bergström J, Andersson G, Ljótsson B, Rück C, Andréewitch S, Karlsson A, et al. Internet-versus group-administeredcognitive behaviour therapy for panic disorder in a psychiatric setting: a randomised trial. BMC Psychiatry 2010 Jul 02;10:54[FREE Full text] [doi: 10.1186/1471-244X-10-54] [Medline: 20598127]
Mohr DC, Tomasino KN, Lattie EG, Palac HL, Kwasny MJ, Weingardt K, et al. IntelliCare: an eclectic, skills-based app
suite for the treatment of depression and anxiety. J Med Internet Res 2017 Jan 05;19(1):e10 [FREE Full text] [doi:10.2196/jmir.6645] [Medline: 28057609]
Yu JS, Szigethy E, Wallace M, Solano F, Oser M. Implementation of a guided, digital cognitive behavioral program foranxiety in primary care: preliminary findings of engagement and effectiveness. Telemed J E Health 2018 Nov;24(11):870-878.
[doi: 10.1089/tmj.2017.0280] [Medline: 29480752]
Hans E, Hiller W. A meta-analysis of nonrandomized effectiveness studies on outpatient cognitive behavioral therapy for
adult anxiety disorders. Clin Psychol Rev 2013 Dec;33(8):954-964. [doi: 10.1016/j.cpr.2013.07.003] [Medline: 23988455]
Stewart RE, Chambless DL. Cognitive-behavioral therapy for adult anxiety disorders in clinical practice: a meta-analysis
of effectiveness studies. J Consult Clin Psychol 2009 Aug;77(4):595-606 [FREE Full text] [doi: 10.1037/a0016032][Medline: 19634954]
Fleming T, Bavin L, Lucassen M, Stasiak K, Hopkins S, Merry S. Beyond the trial: systematic review of real-world uptake
and engagement with digital self-help interventions for depression, low mood, or anxiety. J Med Internet Res 2018 Jun
06;20(6):e199 [FREE Full text] [doi: 10.2196/jmir.9275] [Medline: 29875089]
Baumel A, Muench F, Edan S, Kane JM. Objective user engagement with mental health apps: systematic search andpanel-based usage analysis. J Med Internet Res 2019 Sep 25;21(9):e14567 [FREE Full text] [doi: 10.2196/14567] [Medline:
31573916]
Hollis C, Sampson S, Simons L, Davies EB, Churchill R, Betton V, et al. Identifying research priorities for digital technology
in mental health care: results of the James Lind Alliance Priority Setting Partnership. Lancet Psychiatry 2018
Oct;5(10):845-854. [doi: 10.1016/S2215-0366(18)30296-7] [Medline: 30170964]
Taylor CB, Ruzek JI, Fitzsimmons-Craft EE, Sadeh-Sharvit S, Topooco N, Weissman RS, et al. Using digital technology
to reduce the prevalence of mental health disorders in populations: time for a new approach. J Med Internet Res 2020 Jul
24;22(7):e17493 [FREE Full text] [doi: 10.2196/17493] [Medline: 32706665]
Glover AC, Schueller SM, Winiarski DA, Smith DL, Karnik NS, Zalta AK. Automated mobile phone-based mental healthresource for homeless youth: pilot study assessing feasibility and acceptability. JMIR Ment Health 2019 Oct 11;6(10):e15144[FREE Full text] [doi: 10.2196/15144] [Medline: 31605516]
Ben-Zeev D, Scherer EA, Gottlieb JD, Rotondi AJ, Brunette MF, Achtyes ED, et al. mHealth for schizophrenia: patientengagement with a mobile phone intervention following hospital discharge. JMIR Ment Health 2016 Jul 27;3(3):e34 [FREEFull text] [doi: 10.2196/mental.6348] [Medline: 27465803]
Mohr DC, Cuijpers P, Lehman K. Supportive accountability: a model for providing human support to enhance adherence
to eHealth interventions. J Med Internet Res 2011 Mar 10;13(1):e30 [FREE Full text] [doi: 10.2196/jmir.1602] [Medline:
21393123]https://formative.jmir.org/2021/9/e27570

XSL• FO
RenderX

Anton et al

JMIR Form Res 2021 | vol. 5 | iss. 9 | e27570 | p. 13(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
52.
53.

54.

55.

56.

57.
58.

59.

60.

Anton et al

Schueller SM, Tomasino KN, Mohr DC. Integrating human support into behavioral intervention technologies: the efficiency
model of support. Clin Psychol Sci Pract 2017 Mar;24(1):27-45. [doi: 10.1111/cpsp.12173]
Zarski A, Lehr D, Berking M, Riper H, Cuijpers P, Ebert DD. Adherence to internet-based mobile-supported stressmanagement: a pooled analysis of individual participant data from three randomized controlled trials. J Med Internet Res
2016 Jun 29;18(6):e146 [FREE Full text] [doi: 10.2196/jmir.4493] [Medline: 27357528]
Titov N, Dear BF, Johnston L, Lorian C, Zou J, Wootton B, et al. Improving adherence and clinical outcomes in self-guided
internet treatment for anxiety and depression: randomised controlled trial. PLoS One 2013 Jul 3;8(7):e62873 [FREE Fulltext] [doi: 10.1371/journal.pone.0062873] [Medline: 23843932]
Mohr DC, Duffecy J, Ho J, Kwasny M, Cai X, Burns MN, et al. A randomized controlled trial evaluating a manualized
TeleCoaching protocol for improving adherence to a web-based intervention for the treatment of depression. PLoS One
2013 Aug 21;8(8):e70086 [FREE Full text] [doi: 10.1371/journal.pone.0070086] [Medline: 23990896]
Arean PA, Hallgren KA, Jordan JT, Gazzaley A, Atkins DC, Heagerty PJ, et al. The use and effectiveness of mobile appsfor depression: results from a fully remote clinical trial. J Med Internet Res 2016 Dec 20;18(12):e330 [FREE Full text][doi: 10.2196/jmir.6482] [Medline: 27998876]
The improving access to psychological therapies manual. The National Collaborating Centre for Mental Health. 2018.
URL: https://www.england.nhs.uk/wp-content/uploads/2020/05/iapt-manual-v4.pdf [accessed 2021-08-18]
Titov N, Dear BF, Staples LG, Bennett-Levy J, Klein B, Rapee RM, et al. MindSpot clinic: an accessible, efficient, andeffective online treatment service for anxiety and depression. Psychiatr Serv 2015 Oct;66(10):1043-1050. [doi:10.1176/appi.ps.201400477] [Medline: 26130001]
Karin E, Dear BF, Heller GZ, Gandy M, Titov N. Measurement of symptom change following web-based psychotherapy:
statistical characteristics and analytical methods for measuring and interpreting change. JMIR Ment Health 2018 Jul
12;5(3):e10200 [FREE Full text] [doi: 10.2196/10200] [Medline: 30001999]
Mobile fact sheet. Pew Research Center. 2021. URL: https://www.pewresearch.org/internet/fact-sheet/mobile/ [accessed2021-08-18]

Abbreviations
CBT: cognitive behavioral therapy
GAD-7: Generalized Anxiety Disorder-7
PHQ-9: Patient Health Questionnaire-9
SPIN: Social Phobia Inventory

Edited by J Torous; submitted 02.02.21; peer-reviewed by J Yu, A Rotondi; comments to author 29.03.21; revised version received
29.04.21; accepted 26.07.21; published 21.09.21
Please cite as:
Anton MT, Greenberger HM, Andreopoulos E, Pande RL
Evaluation of a Commercial Mobile Health App for Depression and Anxiety (AbleTo Digital+): Retrospective Cohort StudyJMIR Form Res 2021;5(9):e27570
URL: https://formative.jmir.org/2021/9/e27570
doi: 10.2196/27570
PMID:©Margaret T Anton, Heidi Mochari Greenberger, Evie Andreopoulos, Reena L Pande. Originally published in JMIR Formative
Research (https://formative.jmir.org), 21.09.2021. This is an open-access article distributed under the terms of the Creative
Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, andreproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The
complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright andlicense information must be included.

https://formative.jmir.org/2021/9/e27570

XSL• FO
RenderX

JMIR Form Res 2021 | vol. 5 | iss. 9 | e27570 | p. 14(page number not for citation purposes)