JMIR MHEALTH AND UHEALTH

Arevian et al

Original Paper

Patient and Provider Cocreation of Mobile Texting Apps to Support
Behavioral Health: Usability Study
Armen C Arevian, MD, PhD; Jennifer O'Hora, PhD; James Rosser, LCSW; Joseph D Mango, MFA; David J Miklowitz,
PhD; Kenneth B Wells, MD, MPH
Jane and Terry Semel Institute for Neuroscience and Human Behavior, University of California Los Angeles, Los Angeles, CA, United States

Corresponding Author:
Armen C Arevian, MD, PhD
Jane and Terry Semel Institute for Neuroscience and Human Behavior
University of California Los Angeles
10920 Wilshire Blvd, Suite 300
Los Angeles, CA,
United States
Phone: 1 310 794 3732
Fax: 1 310 794 3724
Email: AArevian@mednet.ucla.edu

Abstract
Background: Mobile technologies hold potential for improving the quality of care and engagement of patients. However, there
are considerable challenges in ensuring that technologies are relevant, useful, and engaging. While end users such as patients andproviders are increasingly involved in the design of health technologies, there are limited examples of their involvement in directly
creating technologies for their personal use.
Objective: We aim to evaluate the feasibility and acceptability of patients and providers creating mobile texting apps to support
treatment goals.
Methods: In an 11-month usability study, we enrolled 4 providers and 28 patients in an intensive outpatient program forobsessive-compulsive disorder. Patients and providers created their own mobile texting apps using a visual app development
platform. A subsample of 10 patients and 4 providers completed a usability measure.
Results: Participants created a total of 360 unique mobile text messages (1787 total messages sent). There were 4 types ofmessages identified, including personalized reminders, clinical exposures, interactive prompts, and encouraging/informational
messages. A total of 9 out of 10 (90%) patients agreed that the messages were relevant to their recovery, and 8 out of 10 (80%)
agreed that the messages were effective at helping complete treatment plans.
Conclusions: Enabling patients and providers to cocreate apps for their own use by using a visual application platform is feasible
and holds potential for increasing the relevance, sustainability, and effectiveness of digital health technologies.
(JMIR Mhealth Uhealth 2020;8(7):e12655) doi: 10.2196/12655
KEYWORDS
mobile health; community-based participatory research; app development; technology platforms; personalized medicine; behavioral
health; mobile phone

Introduction
Digital technologies such as mobile apps are increasingly used
to improve the quality of care and engagement of patients [1],especially for chronic disease management, in which
engagement of individuals over time is challenging [2]. Thereare considerable barriers in ensuring that technologies are
relevant, useful, and lead to sustained use at the patient,provider, and institutional levels [3,4]. Involving broader
http://mhealth.jmir.org/2020/7/e12655/

XSL• FO
RenderX

stakeholders, including patients and providers, is increasingly
seen as critical in addressing these challenges. However, theirinvolvement in the development process is typically limited tothe design of apps, often providing feedback on the content,
functionality, and visual appearance. Translating these designs
into functioning apps is often undertaken separately by computer
programmers, resulting in the removal of the end users fromthis aspect of development, as well as additional cost,complexity, and time. There have not been previous reports ofa systematic approach to engaging patients and providers as the
JMIR Mhealth Uhealth 2020 | vol. 8 | iss. 7 | e12655 | p. 1
(page number not for citation purposes)

JMIR MHEALTH AND UHEALTH
direct creators of mobile health apps that they use to support
their health needs [5].
The technology adoption model identifies several intrinsic andextrinsic factors related to technology use, including perceived
ease of use and relevance [6]. Both of these factors may be
negatively impacted when apps are not created for specificindividuals or local groups but rather made to address broader
diseases (eg, diabetes management), conditions (eg, weight loss,
stress reduction), or approaches (eg, cognitive behavioral
therapy). At the same time, direct engagement by patients has
been recognized as a key determinant of health behavior change,
better health outcomes, and satisfaction [7].

Arevian et al
Participatory approaches, including participatory design andcommunity-partnered participatory research [8,9], were created
to involve stakeholders such as patients, providers, andcommunity leaders, and are increasingly used in a variety ofsettings, including those involving digital health [5]. However,
the translation of designs to app creation is often undertaken
separately by technical individuals such as computer
programmers. In addition, existing participatory methods often
aim to create an app for a population whose users were not
involved in the process. This is different than a systematic
approach to involving individuals in the creation of an app for
their personal use, as seen in Figure 1.

Figure 1. Comparison of two approaches to the participatory technology development process for mobile apps. A) Stakeholders cocreate an app for agroup of end users. B) Patients and providers cocreate apps to support themselves.

A major assumption of the present study is that when individuals
are involved in creating the apps that they will personally use,
their engagement and the perceived usefulness of the apps will
be enhanced. We created the Chorus application platform(Chorus Innovations Inc) [10] to facilitate this process ofparticipatory technology development, in which stakeholders
can be directly involved in both designing and creating mobile
apps. An application platform provides modules and functions
that can be reused to create specific applications to reduce
development time, lower cost, and improve scalability [11].
Furthermore, Chorus is a no-code application platform that uses
a visual interface to configure apps without the need forcomputer programming, with the objective of supporting a broad
population of stakeholders as app creators [12]. Chorus has beenused to create mobile web, text messaging, and interactive voiceapps for more than 50 research and clinical projects, including
ongoing studies of automated text messaging interventions forasthma [13] and healthy lifestyle [14].
Our group previously used the Chorus platform to facilitate aparticipatory development process to cocreate a mobile texting
app to support resiliency (B-RESILIENT) [15]. We conducteda series of partnered workgroups with stakeholders, including
patients, community leaders, and academic researchers. In theseworkgroups, key needs of the community, selected and adapted
content to include in the app, and privacy and other concerns
related to trust of the technology were discussed. The
workgroups then created an app that addressed these needs.
It remains an open question to what extent it is feasible forindividual patients and providers to directly create personal
mobile apps for their own use in real-world clinical settings.
The goal of this pilot study was to evaluate the usability of the
http://mhealth.jmir.org/2020/7/e12655/

XSL• FO
RenderX

Chorus platform and the feasibility of a participatory
development process in the creation of mobile texting apps bypatients and providers, as well as to describe the kinds of mobile
texting apps created by these participants. The study was
conducted with patients in an intensive outpatient treatment
program and their providers.

Methods
Setting and ParticipantsThis study was conducted in the Obsessive-Compulsive Disorder(OCD) Intensive Outpatient Program (IOP) of University ofCalifornia Los Angeles (UCLA) Health. Patients with OCD
attended the IOP program 3 to 5 weekdays per week. The IOP
uses cognitive behavioral therapy (CBT) techniques, including
exposure to situations and stimuli that trigger anxiety symptoms,
known as Exposure and Response Prevention (ERP) [16]. The
core of ERP is intentional and planned exposure to stimuli thattrigger anxiety responses, with progressively increasing levelsof stimulus threat as treatment progresses.
Key clinical challenges were identified in a workgroup meetingwith IOP staff and research personnel at the beginning of thestudy. The IOP staff identified low patient satisfaction withsupport after clinic hours and the difficulty of completing
exposure tasks outside of the clinic setting, a common barrier
in OCD treatment [17].
The Chorus platform was implemented by the IOP as part ofclinical care rather than as part of a research study. Patients who
enrolled in the IOP were consented by clinic staff for use of text
messaging as part of their care. Those that agreed were thenconsented by research staff to participate in this study. The
JMIR Mhealth Uhealth 2020 | vol. 8 | iss. 7 | e12655 | p. 2
(page number not for citation purposes)

JMIR MHEALTH AND UHEALTH
scope of the research study was to evaluate the clinic’s
implementation and use of the Chorus platform by analyzing
text messages, administering usability measures, and conducting
workgroups. All research methods were approved by the UCLA
Institutional Review Board.
Inclusion criteria for study participation included being an active
patient of the IOP clinic, having a mobile phone (smartphone
not required), and being willing to receive text messages.
Patients used their personal mobile phones and interacted with
their app through text messaging. There was no app todownload. There were no exclusion criteria. We chose thesebroad criteria to reflect the typical population from this clinic.
Technical skills such as computer programming were not arequirement for participation.
During the pilot period, there were 4 PhD psychology providers
who participated in the study. The number of patients active inthe clinic at any given time varied between 6 and 8 patients.
Patients participated in the study for as long as they wereenrolled in the IOP (average stay of 6-8 weeks). No participants
disenrolled early from the study.

Participatory Technology Development
The clinic implemented a process of participatory technology
development (Figure 1) to facilitate patients and providers to

http://mhealth.jmir.org/2020/7/e12655/

XSL• FO
RenderX

Arevian et al
cocreate personal mobile text messaging apps using the Chorus
platform created by Arevian et al [15]. Providers were given
log-in access to the platform and used the visual web interface
with their patients to cocreate the texting apps, as seen in Figure
2. This included specifying the content of messages, logic tohandle responses from the patient if needed, and scheduled times
to send the messages to the patient from the platform. The
content and functionality of the apps were entirely driven by
the patient and provider. The messaging did not facilitate directcommunication between patients and providers. Instead, theapp was configured to automatically send the messages to thepatient at a later time. If the patient replied to the message, theywould receive an automated preconfigured (by the patient andprovider) response from the app.
Initial training for clinic psychotherapists to use Chorus was asingle 1-hour session. One therapist requested an additional
training session. Therapists were not provided specific guidanceon the types of apps to create other than to address the clinicalneeds of their patients. Therapists reviewed the app’s
functionality with patients as needed as part of the cocreation
process. The Chorus platform was provided through theInnovation Lab at the Jane and Terry Semel Institute for
Neuroscience and Human Behavior at UCLA.

JMIR Mhealth Uhealth 2020 | vol. 8 | iss. 7 | e12655 | p. 3
(page number not for citation purposes)

JMIR MHEALTH AND UHEALTH

Arevian et al

Figure 2. Visually creating apps with Chorus. A) Creation of text messaging content visually with a simulated phone. B) Visual interface to configure
the logic that guides subsequent messages to send based on user’s response. C) Screenshot of an example mobile texting interaction as seen by patients.

Measures and Analysis
Demographic details (age, sex, race) and clinical outcomes wereextracted from patients’ medical records. We report the
Yale-Brown Obsessive-Compulsive Scale (Y-BOCS) score todescribe the clinical severity of this population. Y-BOCS is ameasure of OCD symptom severity and was assessed by study
staff on admission as part of routine care [18].
http://mhealth.jmir.org/2020/7/e12655/

XSL• FO
RenderX

Feasibility and acceptability were evaluated through a written
questionnaire that study staff administered to patients. The
measure included items of perceived ease of use, perceived
usefulness, self-efficacy, and patient-centeredness (seeMultimedia Appendix 1) [6,19]. Participants indicated theiragreement with statements in the survey using a Likert scaleranging from 1 to 7, where 1 indicated they strongly disagreed,
4 indicated they were neutral, and 7 indicated they strongly
JMIR Mhealth Uhealth 2020 | vol. 8 | iss. 7 | e12655 | p. 4
(page number not for citation purposes)

JMIR MHEALTH AND UHEALTH
agreed with the statement. The survey also included 2
open-ended prompts to list the most positive and negative
aspects of their apps, the platform, or the process of participatory
development. The survey was administered to 14 participants(10 patients and 4 therapists) during a 2-month evaluation periodof the pilot.
We conducted a total of 7 workgroups with patients and 7 with
providers to discuss experiences related to the app development
and participatory process. Each workgroup typically had 2 to
7 participants and lasted 30 minutes to 1 hour. Workgroupsessions were audiorecorded and reviewed by study staff.
Qualitative analyses were conducted by reviewing transcripts(for workgroups) and message logs (for app content) to groupphrases or messages together and identify themes [20]. Themes
were cross-checked by 2 study staff and reviewed with clinic
staff for validity. Representative quotes and messages aretabulated in the results below. App use metrics were extracted
from the Chorus activity logs for all participants.
Retention rates for the use of the mobile texting apps were
calculated using the date of the last message sent or received
by the patient from the Chorus logs and comparing this to thedate of termination from the clinical program.

http://mhealth.jmir.org/2020/7/e12655/

XSL• FO
RenderX

Arevian et al

Results
We conducted a rolling enrollment over an 11-month period
between July 2015 and May 2016. A total of 28 patients and 4
providers were enrolled in the study during this period. The
providers were all PhD-level psychotherapists specializing in
the treatment of anxiety disorders. Of the 28 patients, 11 werefemale (39%). Mean age was 33 years (SD 14.2; range 18-69).
Patients were mostly white (26/28, 93%) and had severe OCD
symptom burden on admission (Y-BOCS mean score of 38 outof 40, indicating extreme level of symptoms).
A total of 1787 messages were sent to patients during theevaluation period (360 unique messages) and 80 responses werereceived from patients. Patients received an average of 51
messages (SD 55). All patients responded at least once to a text
message. The number of messages sent did not vary by day of
the week. Most text messages were sent after clinic hours(between 2 PM and 10 PM).
We identified 4 basic types of messages created, including
personalized reminders (ie, prompting of a clinical goal to beachieved at home), messages serving as clinical exposures (ie,intended to evoke feared stimuli), interactive prompts (messages
that requested a response from the patient), and encouraging or
informational messages (Textbox 1). A common theme of themessages was to use personalization and humor to create
messages related to treatment engagement.

JMIR Mhealth Uhealth 2020 | vol. 8 | iss. 7 | e12655 | p. 5
(page number not for citation purposes)

JMIR MHEALTH AND UHEALTH

Arevian et al

Textbox 1. Content of mobile texting apps.
Interactive•“Were you able to sit with the anxiety and contamination without washing today? Press 1 for yes and 2 for no”•“Hey there {patient name}, did you meet your goal today? TEXT 1 for yes or 2 for no”•“Don't forget to take the Namenda tonight. Press 1 if you took the medication”•“Hey, did you go to the store today?! TEXT: 1 for yes, 2 for no”

Personalized reminders•“Hi {patient name}, No more showering. Life is too short. Lean into the grubbiness. You can do this”•“It's 10pm, time to give up your devices. They turn into pumpkins!”•“Good morning {patient name}, here is your sleep diary reminder! {link to diary website}”•“Ready, set, go - shower in under 12. Otherwise get ready to shower at [University of California Los Angeles] in 6 minutes”•“Leave those faucets alone {patient name}. You don't need them”•“C'mon tin ribs, get to the gym and feel the burn!”

Message as the exposure (exposure type)•“Remember, Typhoid {patient name}, No handwashing. You are a walking petri dish!” (contamination)•“Showering once a day means you have some pretty gross stuff on you” (contamination)•“gr8t job this week - don't forget two exercise this weekend :)” (spelling)•“{expletive}” (unwanted thoughts)

Encouraging/informational•“Gold star Mr. Jelly Legs. Be proud of yourself”•“Remember to try to go as long as possible without washing, even if you feel contaminated. You are stronger than your [obsessive-compulsive
disorder]!”•“Very good padawan”•“Exercise combats health conditions and diseases. It can also improve mood”•“That's awesome!! Great Job!”

Results from the usability survey for patients and providers arepresented in Table 1. Individuals responded to how much theyagreed with the statements listed in Table 1 using a Likert scale,where 1=strongly disagree, 4=neutral, and 7=strongly agree.
Representative open-ended responses for positive and negative
features of the system are included in Textbox 2. Regarding theuse of the Chorus platform, 11 out of 14 (79%) participants (10patients and 4 providers) agreed that Chorus was simple to use,
12 out of 14 (86%) felt comfortable using Chorus, 13 out of 14(93%) felt comfortable using the text messaging app, and 11
out of 14 (79%) felt they (or their patients) were more engaged
as a result. All providers (4/4, 100%) and most patients (8/10,80%) agreed or responded neutrally to the statement “I caneffectively develop messages with my provider (patient) using
this messaging application.” Of the 10 patients surveyed, 9(90%) agreed that the messages were relevant to them and theirrecovery, and 8 (80%) agreed that the messages were effective
in helping them complete treatment exercises at home.
Overall, engagement with the cocreated apps was sustained over
the time the patient was engaged in the clinic (Figure 3). The
average duration of mobile app use was 25.1 days (SD 15.1),

http://mhealth.jmir.org/2020/7/e12655/

XSL• FO
RenderX

which was 6.7 days (SD 8.8) shorter than the average length ofstay in the clinical program (mean 31.8 days, SD 13.3).
Key themes that emerged from provider workgroups wererelated to the effects of the app development process on
engagement and between-session homework completion (eg,“Follow through on homework is better”; “One of the patients
I had, he really loved it. He said he felt more accountable. And
it involved his wife because he showed his wife so she was kindof onboard with the whole treatment. And he said it felt like[my provider] was in the room...”). Themes emerging frompatient workgroups included discussion of personalization oftreatment (eg, “I tell [my provider] what kind of topic I want.
And then we talk about what times are good. But then she makes
up the specifics”), motivation (eg, “Helps the motivation
continue”; “It helps remind you on what you need to be doing”),
and connection after hours (eg, “You’re [in the IOP clinic] foran allotted amount of time and then you have to be out into thereal world… This supplements [the time in the IOP clinic]”).
Providers stated that during the final week that patients are in
the IOP, providers are focused on discharge planning andtherefore less focused on app use by the patients.

JMIR Mhealth Uhealth 2020 | vol. 8 | iss. 7 | e12655 | p. 6
(page number not for citation purposes)

JMIR MHEALTH AND UHEALTH

Arevian et al

Table 1. Usability and satisfaction with the Chorus platform and mobile texting apps.
MeasureScore, mean (SD)Patients (n=10)

Providers (n=4)

6.0 (1.5)

5.8 (0.5)

This messaging application has all the functions and capabilities I expect it to have.

5.7 (1.6)

5.3 (1.0)

I feel (I am/my patients are) more engaged in my treatment as a result of using this messaging application.

5.8 (1.2)

6.0 (1.4)

I can effectively develop messages with my provider (patient) using this messaging application.

6.0 (1.3)

6.8 (0.5)

I (my patient) was directly involved in creating the text messages I would receive.

4.8 (2.2)

5.5 (3.0)

Perceived ease of use
It is simple to use this messaging application.
Perceived usefulness

Self-efficacy

Receiving these text messages is effective in helping me (my patient) complete the treatment plans at home. 5.7 (1.1)

7.0 (0)Patient-centeredness
Information provided by the text messages is relevant to me (my patient) and my (patient’s) recovery.

6.2 (1.0)

7.0 (0)Overall, I am satisfied with this messaging application.

5.8 (1.4)

5.5 (0.6)

Textbox 2. Illustrative quotes of positive and negative aspects of Chorus.
Positive aspects•“It's nice to have it remind me of all the things I need to accomplish”•“Helps the motivation continue”•“Get out of comfort zone”•“I feel more encouraged in treatment”•“It is simple”•“Very straightforward”•“Simple to create”•“Fun texts”•“Patients love it”•“Patients like interaction”

Negative aspects•“Harder to use if you're a bad texter”•“Some bugs”•“Disappointment it's not a text from a friend”•“Folders needed [to organize messages]”

http://mhealth.jmir.org/2020/7/e12655/

XSL• FO
RenderX

JMIR Mhealth Uhealth 2020 | vol. 8 | iss. 7 | e12655 | p. 7
(page number not for citation purposes)

JMIR MHEALTH AND UHEALTH

Arevian et al

Figure 3. Patient engagement. Percentage of participants enrolled in the Intensive Outpatient Program clinic (black line) compared with the percentageof participants actively using their mobile app (grey line) by number of days in the program.

Discussion
This is the first study of a systematic approach for patients andtheir providers to directly cocreate mobile texting apps for theirown use as part of clinical care. We implemented a web-based
application development platform (Chorus) and participatory
development process in an outpatient clinical setting for patientswith severe OCD. Providers required minimal training and wereable to cocreate and use apps the same day that they were trainedon how to use the platform. Providers and patients determinedthe content, frequency, and timing of notifications. Despite thehigh symptom burden experienced by patients, both patientsand providers reported that the Chorus platform was easy to use
and that it helped patients engage in treatment goals. A total of
9 out of 10 patients agreed that the messages were relevant tothem. We observed sustained use of the cocreated apps
throughout patients’ time in the IOP. In addition, while thepatients and providers were not given specific direction as tothe types of messages to create, the 4 categories of messages
identified were consistent with key principles of CBT and ERP
implemented by the clinic.
There is increasing interest in including patients and providers
at various levels of technology development. The Nightscout
project is an example of an app first created by a patient’s parent
for diabetes management [21]. The parent, who was a computer
programmer, identified a need and created a technology toaddress it, resulting in an open source app for monitoring
glucose levels. While this project involved an end user in its
development, our approach differs in that computer
programming is not required so that apps can be created byindividuals without technical backgrounds, including for theirown use. Torous and Roux [22] describe an individual patient
http://mhealth.jmir.org/2020/7/e12655/

XSL• FO
RenderX

with schizophrenia working with their provider to create acustom symptom-tracking system. Though the patient andprovider were not creating a mobile app, they used spreadsheets
to graph symptom counts recorded with a manual tally counterby the patient.
Part of the challenge of involving patients in creating apps fortheir own use is the high technical and financial barriers tocreating apps. The use of application platforms may offer several
advantages. Visual development interfaces mean that individuals
without knowledge of computer programming can create apps.
In addition, compliance, auditing, and security requirements,
such as those in the Health Insurance Portability andAccountability Act, can be handled centrally by the platform
without the need to repeat these functions and organizational
review processes for each individual app. There are several
examples of no-code platforms used in health care and researchoutside of mobile app development. For example, REDCap(Vanderbilt University) [23] and Qualtrics (SAP SE) [24] arecommonly used visual development platforms for creating online
surveys. Squarespace (Squarespace Inc) [25] and Wix (Wix.com
Inc) [26] are platforms to visually create and host websites. This
is in addition to platforms for use by developers and institutions
with technical expertise, including platforms to facilitate device
sensing [27], data collection [28], and data storage and analysis[29].
The direct development of technologies by patients, providers,and other stakeholders has several implications. First, thisparticipatory process aims to increase the level of patient
involvement and the relevance of health technologies, which isconsistent with recommendations from behavior change theories
and guidelines for patient-centered care [7,8,30]. Second, thisJMIR Mhealth Uhealth 2020 | vol. 8 | iss. 7 | e12655 | p. 8
(page number not for citation purposes)

JMIR MHEALTH AND UHEALTH
process also has implications for equity and power sharing, a
key principle from participatory approaches such ascommunity-partnered participatory research. If patients areprimarily involved at the design stage only (as is the current
state of practice), the ability to create and maintain technologies
still rests with computer programmers and server infrastructure
staff. When patients and providers directly create and maintaintheir own apps, they may be better able control what is created,change the apps over time as their needs and priorities change,and directly benefit from their own apps.
Our approach is consistent with recent recommendations
regarding how health information technologies may be evaluatedmore effectively within health systems. Learning health systems[31], agile science [32], and responsive research [33] each aim
to support iterative intervention development, learning throughimplementation in real-world settings, and flexible evaluation
options such as N-of-1 and pragmatic trials. End users are an
untapped talent pool. According to the US census, only 0.1%of the US population are computer programmers [34]. By
exploring approaches and technology platforms that do not
require programming skills, we are better able to tap into theexpertise and capacity of individuals who have lived
experiences. An important future consideration is how clinicaltraining programs may be modified to prepare clinicians andArevian et al
other staff for increased patient involvement in healthtechnology development [35].
This study has several limitations. It was conducted at a single
site with a limited number of patients, most of whom were white,
young (mean 33 years), and had severe OCD symptoms. The
clinical program was also suited to texting interventions, given
its focus on CBT and exposure-driven treatment. Implementationin larger samples, at other sites, and in other clinical conditions
would allow further evaluation of the generalizability of thisapproach, including how it may be adapted to other technology
platforms [11]. This study did not evaluate the impact of theapp development and cocreation process on clinical outcomes,
a topic to be explored in randomized controlled trials. Whilepatients engaged with the app for the majority of their lengthof stay in the IOP, we did observe a discontinuation of the app
an average 6.7 days prior to discharge from the IOP. This may
be due to the reduced focus on exposures and interventions in
the final week of the IOP, with a transition to discharge planning
reported by providers.
Technology approaches that use flexible, user-driven platforms
to engage a broader set of individuals in development may holdpotential for increasing the relevance and sustainability oftechnology interventions, which may in turn lead to improved
patient engagement and outcomes.

Acknowledgments
The authors would like to thank the patients and staff of the Resnick Neuropsychiatric Hospital Intensive Outpatient Programsfor their engagement and support of this project. ACA, KBW, and JO were supported by California State Behavioral HealthCenter of Excellence (SB-852). ACA and KBW were supported by NIMH P30 MH082760-05S1. ACA was supported by NIMH
R01 MH093676-02S1.

Conflicts of Interest
ACA is the founder of Chorus Innovations Inc, Arevian Technologies, and Open Science Initiative. ACA developed the Chorus
platform, which is licensed to Chorus Innovations Inc from the University of California Los Angeles.

Multimedia Appendix 1
Patient Usability Measure.
[PDF File (Adobe PDF File), 47 KB-Multimedia Appendix 1]

References
1.
2.

3.

4.
5.

6.

Zhao J, Freeman B, Li M. Can Mobile Phone Apps Influence People's Health Behavior Change? An Evidence Review. J
Med Internet Res 2016 Oct 31;18(11):e287 [FREE Full text] [doi: 10.2196/jmir.5692] [Medline: 27806926]
de Jongh T, Gurol-Urganci I, Vodopivec-Jamsek V, Car J, Atun R. Mobile phone messaging for facilitating self-managementof long-term illnesses. Cochrane Database Syst Rev 2012 Dec 12;12:CD007459 [FREE Full text] [doi:10.1002/14651858.CD007459.pub2] [Medline: 23235644]
Mohr DC, Weingardt KR, Reddy M, Schueller SM. Three Problems With Current Digital Mental Health Research . . . andThree Things We Can Do About Them. Psychiatr Serv 2017 May 01;68(5):427-429 [FREE Full text] [doi:10.1176/appi.ps.201600541] [Medline: 28412890]
Gagnon M, Ngangue P, Payne-Gagnon J, Desmartis M. m-Health adoption by healthcare professionals: a systematic review.
J Am Med Inform Assoc 2016 Jan;23(1):212-220. [doi: 10.1093/jamia/ocv052] [Medline: 26078410]
Unertl KM, Schaefbauer CL, Campbell TR, Senteio C, Siek KA, Bakken S, et al. Integrating community-based participatory
research and informatics approaches to improve the engagement and health of underserved populations. J Am Med Inform
Assoc 2016 Jan;23(1):60-73 [FREE Full text] [doi: 10.1093/jamia/ocv094] [Medline: 26228766]
Davis FD. User acceptance of information technology: system characteristics, user perceptions and behavioral impacts.
International Journal of Man-Machine Studies 1993 Mar;38(3):475-487. [doi: 10.1006/imms.1993.1022]

http://mhealth.jmir.org/2020/7/e12655/

XSL• FO
RenderX

JMIR Mhealth Uhealth 2020 | vol. 8 | iss. 7 | e12655 | p. 9
(page number not for citation purposes)

JMIR MHEALTH AND UHEALTH
7.
8.
9.

10.
11.
12.

13.
14.
15.

16.
17.
18.
19.
20.
21.

22.
23.
24.
25.
26.
27.

28.
29.

30.
31.

32.

33.

Epstein R, Street R. The values and value of patient-centered care. Ann Fam Med 2011;9(2):100-103 [FREE Full text][doi: 10.1370/afm.1239] [Medline: 21403134]
Jones L, Wells K. Strategies for academic and clinician engagement in community-participatory partnered research. JAMA
2007 Jan 24;297(4):407-410. [doi: 10.1001/jama.297.4.407] [Medline: 17244838]
Wells KB, Jones L, Chung B, Dixon EL, Tang L, Gilmore J, et al. Community-partnered cluster-randomized comparative
effectiveness trial of community engagement and planning or resources for services to address depression disparities. J
Gen Intern Med 2013 Oct;28(10):1268-1278 [FREE Full text] [doi: 10.1007/s11606-013-2484-3] [Medline: 23649787]
Arevian AC. Chorus Participatory Mobile Framework. URL: https://chorus.semel.ucla.edu/ [accessed 2020-06-01]
Reviews for Enterprise High-Productivity Application Platform as a Service. Gartner. 2019. URL: https://www.gartner.com/
reviews/market/enterprise-high-productivity-application-paas [accessed 2020-06-01]
Bloomberg J. The Low-Code/No-Code Movement: More Disruptive Than You Realize. Forbes. 2017 Jul 20. URL: https://www.forbes.com/sites/jasonbloomberg/2017/07/20/the-low-codeno-code-movement-more-disruptive-than-you-realize/#6c0e2dd9722a, [accessed 2020-06-01]
Teach S. Breathe With Ease: A Unique Approach to Managing Stress (BEAMS). clinicaltrials.gov. 2019. URL: https://clinicaltrials.gov/ct2/show/NCT02374138, [accessed 2020-03-01]
Brown A. Financial Incentives and Text Messaging to Improve African American Womens' Glycemic Control(FRIENDS-Text). clinicaltrials.gov. 2015. URL: https://clinicaltrials.gov/ct2/show/NCT02384265 [accessed 2020-03-01]
Arevian AC, O'Hora J, Jones F, Mango J, Jones L, Williams PG, et al. Participatory Technology Development to EnhanceCommunity Resilience. Ethn Dis 2018;28(Suppl 2):493-502 [FREE Full text] [doi: 10.18865/ed.28.S2.493] [Medline:
30202203]
Koran L, Hanna G, Hollander E, Nestadt G, Simpson H, American Psychiatric Association. Practice guideline for thetreatment of patients with obsessive-compulsive disorder. Am J Psychiatry 2007 Jul;164(7 Suppl):5-53. [Medline: 17849776]
Abramowitz JS, Franklin ME, Zoellner LA, DiBernardo CL. Treatment compliance and outcome in obsessive-compulsive
disorder. Behav Modif 2002 Sep;26(4):447-463. [doi: 10.1177/0145445502026004001] [Medline: 12205821]
Koen N, Stein D. Obsessive-Compulsive Disorder. In: Zigmond MJ, Rowland LP, Coyle JT, editors. Neurobiology of Brain
Disorders: Biological basis of neurological and psychiatric disorders. Amsterdam, Netherlands: Elsevier; 2015:621-638.
Bevan N, Kirakowski J, Maissel J. What is Usability? In: Human Aspects in Computing: Design and use of interactive
systems and information management. Amsterdam, Netherlands: Elsevier Science Ltd; 2001.
McQuarrie EF, Marshall C, Rossman GB. Designing Qualitative Research. Journal of Marketing Research 1990
Aug;27(3):370. [doi: 10.2307/3172595]
Lee JM, Hirschfeld E, Wedding J. A Patient-Designed Do-It-Yourself Mobile Technology System for Diabetes: Promiseand Challenges for a New Era in Medicine. JAMA 2016 Apr 12;315(14):1447-1448. [doi: 10.1001/jama.2016.1903][Medline: 27115262]
Torous J, Roux S. Patient-Driven Innovation for Mobile Mental Health Technology: Case Report of Symptom Tracking
in Schizophrenia. JMIR Ment Health 2017 Jul 06;4(3):e27 [FREE Full text] [doi: 10.2196/mental.7911] [Medline: 28684386]
REDCap.: REDCap; 2004. URL: http://projectredcap.org [accessed 2020-02-29]
Qualtrics.: Qualtrics URL: http://qualtrics.com [accessed 2020-02-29]
Squarespace.: Squarespace URL: http://squarespace.com [accessed 2020-03-01]
Wix.: Wix URL: http://wix.com [accessed 2020-03-01]
Torous J, Kiang MV, Lorme J, Onnela J. New Tools for New Research in Psychiatry: A Scalable and Customizable Platformto Empower Data Driven Smartphone Research. JMIR Ment Health 2016 May 05;3(2):e16 [FREE Full text] [doi:10.2196/mental.5165] [Medline: 27150677]
Tangmunarunkit H, Hsieh CK, Longstaff B, Nolen S, Jenkins J, Ketcham C, et al. A general extensible end-to-end
participatory sensing platform. ACM Trans Intell Syst Technol 2015;6(3):1-21 [FREE Full text] [doi: 10.1145/2717318]
Chen C, Haddad D, Selsky J, Hoffman JE, Kravitz RL, Estrin DE, et al. Making sense of mobile health data: an open
architecture to improve individual- and population-level health. J Med Internet Res 2012 Aug 09;14(4):e112 [FREE Fulltext] [doi: 10.2196/jmir.2152] [Medline: 22875563]
Ryan RM, Deci EL. Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being.
Am Psychol 2000 Jan;55(1):68-78. [doi: 10.1037//0003-066x.55.1.68] [Medline: 11392867]
Patrick K, Hekler E, Estrin D, Mohr DC, Riper H, Crane D, et al. The Pace of Technologic Change: Implications for Digital
Health Behavior Intervention Research. Am J Prev Med 2016 Nov;51(5):816-824. [doi: 10.1016/j.amepre.2016.05.001][Medline: 27745681]
Hekler EB, Klasnja P, Riley WT, Buman MP, Huberty J, Rivera DE, et al. Agile science: creating useful products forbehavior change in the real world. Transl Behav Med 2016 Jun;6(2):317-328 [FREE Full text] [doi:10.1007/s13142-016-0395-7] [Medline: 27357001]
Riley WT, Glasgow RE, Etheredge L, Abernethy AP. Rapid, responsive, relevant (R3) research: a call for a rapid learning
health research enterprise. Clin Transl Med 2013 May 10;2(1):10 [FREE Full text] [doi: 10.1186/2001-1326-2-10] [Medline:
23663660]

http://mhealth.jmir.org/2020/7/e12655/

XSL• FO
RenderX

Arevian et al

JMIR Mhealth Uhealth 2020 | vol. 8 | iss. 7 | e12655 | p. 10(page number not for citation purposes)

JMIR MHEALTH AND UHEALTH
34.
35.

Arevian et al

Labor Force Statistics. US Census Bureau. URL: https://www.census.gov/topics/employment/labor-force.html [accessed2020-07-04]
Torous J, Chan S, Luo J, Boland R, Hilty D. Clinical Informatics in Psychiatric Training: Preparing Today's Trainees for
the Already Present Future. Acad Psychiatry 2018 Oct;42(5):694-697 [FREE Full text] [doi: 10.1007/s40596-017-0811-4][Medline: 29047074]

Abbreviations
CBT: cognitive behavioral therapy
ERP: Exposure and Response Prevention
IOP: intensive outpatient program
OCD: obsessive-compulsive disorder
UCLA: University of California Los Angeles
Y-BOCS: Yale-Brown Obsessive-Compulsive Scale

Edited by G Eysenbach; submitted 30.10.18; peer-reviewed by D Chao, T Irizarry, A Pratap, K Usop; comments to author 31.03.19;
revised version received 05.08.19; accepted 29.03.20; published 29.07.20
Please cite as:
Arevian AC, O'Hora J, Rosser J, Mango JD, Miklowitz DJ, Wells KB
Patient and Provider Cocreation of Mobile Texting Apps to Support Behavioral Health: Usability StudyJMIR Mhealth Uhealth 2020;8(7):e12655
URL: http://mhealth.jmir.org/2020/7/e12655/
doi: 10.2196/12655
PMID: 32723714©Armen C Arevian, Jennifer O'Hora, James Rosser, Joseph D Mango, David J Miklowitz, Kenneth B Wells. Originally publishedin JMIR mHealth and uHealth (http://mhealth.jmir.org), 29.07.2020. This is an open-access article distributed under the terms of
the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use,
distribution, and reproduction in any medium, provided the original work, first published in JMIR mHealth and uHealth, isproperly cited. The complete bibliographic information, a link to the original publication on http://mhealth.jmir.org/, as well as
this copyright and license information must be included.

http://mhealth.jmir.org/2020/7/e12655/

XSL• FO
RenderX

JMIR Mhealth Uhealth 2020 | vol. 8 | iss. 7 | e12655 | p. 11(page number not for citation purposes)