JMIR FORMATIVE RESEARCH

Ditton et al

Original Paper

Medical Student Experiences of Engaging in a Psychological
Flexibility Skill Training App for Burnout and Well-being: Pilot
Feasibility Study
Elizabeth Ditton1,2,3, BA (Psych) Hons, MA Psych (Clin); Brendon Knott4, BSc (Psych) Hons, MA Psych (Clin);
Nicolette Hodyl1,2,5, BSc Hons, GradCert Biostats, PhD; Graeme Horton3, BS, MB, MEnvStud, PhD; Frederick Rohan
Walker1,2,3*, BA (Psych) Hons, PhD; Michael Nilsson1,2,3,6*, MD, PhD
1Centre for Rehab Innovations, University of Newcastle, Callaghan, Australia

2

Hunter Medical Research Institute, New Lambton Heights, Australia

3

College of Health, Medicine and Wellbeing, University of Newcastle, Callaghan, Australia

4

Contextual Interventions, Newcastle, Australia

5

New South Wales Regional Health Partners, Newcastle, Australia

6

Lee Kong Chian School of Medicine, Nanyang Technological University, Singaport, Singapore*

these authors contributed equally

Corresponding Author:
Elizabeth Ditton, BA (Psych) Hons, MA Psych (Clin)
Centre for Rehab Innovations
University of Newcastle
University Drive
Callaghan, 2300
Australia
Phone: 61 2 404 20738
Email: elizabeth.ditton@uon.edu.au

Abstract
Background: Medical students are at higher risk of burnout than the general population. Interventions that facilitate adaptive
coping behaviors (eg, Psychological Flexibility) in the context of inherent stressors associated with medical training could mitigate
burnout risk and improve well-being. Delivering these interventions using smartphone apps offers advantages such as accessibility,
scalability, mitigation of time and stigma barriers, and facilitation of individual tailoring (individualization). There is a need forfeasibility trials with medical students in this emerging field. Formal evaluations of user experiences of app-based psychologicalskill training are required to identify barriers to and facilitators of engagement and optimize intervention development before
implementation in efficacy trials and real-world settings.
Objective: This study aimed to assess the feasibility of delivering an individualized Psychological Flexibility skill training
intervention (Acceptance and Commitment Training [ACTraining]) to medical students using an app-based delivery format. We
further aimed to explore how formal evaluation of user experiences might inform and guide the development of this app before
implementation in an efficacy trial and future research involving app-delivered psychological skill training for medical students.
Methods: This single-arm study was an early-phase feasibility trial of a stand-alone ACTraining app conducted with a sampleof Australian medical students (n=11). We collected app usability and user experience data across a broad range of domains (eg,perceived helpfulness and relevance, learning experiences, and self-efficacy) using self-report questionnaires (quantitative andqualitative) and behavioral engagement outcomes.
Results: Behavioral engagement data demonstrated that the app delivered the assessment procedures and individualized
ACTraining intervention to medical students as intended. The subjective feedback provided by students who actively engaged
with the app was generally positive across several indicators, including usability, perceived relevance and helpfulness, accessibility,
maintenance of privacy, and opportunity for self-reflection. Disengagement from the app was an identified challenge throughout
the trial. Participant feedback identified several factors that may have affected engagement, such as time, expectations regarding
app interface functioning, and individual differences in confidence and self-efficacy when implementing skills.

https://formative.jmir.org/2023/1/e43263

XSL• FO
RenderX

JMIR Form Res 2023 | vol. 7 | e43263 | p. 1
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Ditton et al

Conclusions: This study reports user experience data that have been largely absent from the literature on digital psychologicalinterventions for medical students. Our findings demonstrate the preliminary feasibility of an app-delivered ACTraining intervention
for medical student well-being and burnout and support the value of future assessment of the efficacy of this approach with larger
samples. We consider subjective feedback from medical students in relation to observed engagement and propose how thisinformation might be used to inform the development of this app and future research in this nascent field.
(JMIR Form Res 2023;7:e43263) doi: 10.2196/43263
KEYWORDS
medical students; burnout prevention; app; feasibility; intervention engagement; psychological flexibility; acceptance andcommitment therapy; mobile phone

Introduction
Medical Student Stress and Psychological Health
The stress of medical training can have adverse effects on
students’ psychological health [1-3]. Worldwide, medical
students are at an elevated risk of psychological ill healthcompared with the general population [1], demonstrating highprevalence of burnout (44.2% [4]), depression (27.2% [2]), andsuicidal ideation (11.1% [2]). Burnout risk increases as training
progresses [5] and can persist throughout a physician’s
professional life [6]. The detrimental consequences of burnout
and psychological ill health among medical students andphysicians are well documented and include poor physical health[7], diminished life satisfaction [6], and disruption to healthcare service delivery (eg, impaired performance [8] and reduced
empathy [5] and work engagement [9]).

Adaptive Psychological Skill Training
Medical educators report increasing concern regarding thepsychological health of students [10] and recognize the needfor early intervention strategies that can prevent burnout rather
than treat it after it has already emerged [10-12]. In addition toorganizational approaches that mitigate modifiable external
stressors (eg, improving learning environments and addressing
student mistreatment [13]), there is support for initiatives thatbuild individual psychological and behavioral resources forcoping with unmodifiable stressors encountered during medical
education and practice [10,12,14,15] (eg, academic pressuresand exposure to illness and death [5,6]). This intervention
strategy has the potential to buffer medical students against
burnout and psychological ill health by facilitating adaptive
responses to stressful experiences [6,15,16]. Beyond simply
preventing psychological ill health, adaptive psychological skill
training can improve well-being [17,18]. Well-being is a stateof personal thriving and vitality that is an essential componentof complete psychological health [19,20] and a key outcome ofinterest to medical educators [10]. There is support for theacceptability of adaptive psychological skill-building
interventions among medical students [3] and physicians
experiencing burnout, with the latter having rated such training
as being as important as learning clinical skills [21].

App-Delivered Interventions
The increased interest in medical student psychological healthhas resulted in a growing number of studies evaluating
individual skill-building interventions [3]. A notable gap in thisemerging field is the absence of studies evaluating the delivery
https://formative.jmir.org/2023/1/e43263

XSL• FO
RenderXof such interventions using stand-alone smartphone apps [1,3].
Although the preliminary feasibility of app-delivered
interventions (eg, mindfulness) has been demonstrated amongsmall samples of medical residents [22,23], there are no studies
evaluating feasibility among medical students. Medical students
use smartphones frequently and are accustomed to engaging
with technology [1,24,25] and, thus, app-delivered interventions
have the potential to be accessible [1,26] and scalable [12].
Apps could facilitate medical students’ access to psychologicalskill training anytime and anywhere [27], which is particularly
important when the availability of psychological health carepersonnel is limited or face-to-face delivery is not practical.
App-delivered interventions may also offset known barriers to
medical student engagement in psychological interventions by
maintaining anonymity and privacy for students concerned
about mental health stigma [1,6] and delivering brief
intervention components that minimize participation-related
time pressures [28]. Finally, app-delivered interventions provide
a potential solution to the problem of heterogeneity and variation
in individual responses to psychological interventions [29] byfacilitating the tailoring of training to individual medical student
needs (ie, individualization) [27]. Individualized psychologicalapps have demonstrated greater effectiveness than
nonindividualized apps with respect to well-being outcomesamong heterogeneous university student samples [27].
Despite the potential advantages of app-delivered interventions
for medical student psychological health, there are several
challenges that could affect the feasibility of this strategy.
Maintaining individual participation in psychological healthapps is a known problem generally, with real-world user
engagement diminishing substantially within 2 to 4 weeks ofsign-up [30]. Engagement can be affected by a range of usability
and acceptability factors, including the quality and functionalityof the interface [1,31], training content relevance [26], thecontext in which the intervention is experienced [31], timeburden [22], and user experiences [1] (eg, affective responses,
perceived effectiveness, and self-efficacy [31]).
Given the infancy of this emerging field and interventiondelivery mode, feasibility assessment is an important precursor
to successful implementation in larger efficacy trials [32]. Thereis a need for studies that formally evaluate medical students’subjective experiences of engaging in app-delivered
psychological interventions as well as their perceptions ofinterface usability [1,33]. The absence of this information fromthe literature has been noted as a hindrance to progressing
digitally delivered psychological skill interventions beyond the
JMIR Form Res 2023 | vol. 7 | e43263 | p. 2
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
prototype phase to effective, fully developed products that could
be successfully implemented in medical education settings [1].
The collection of these data can provide information about how
psychological skill training apps are experienced in real-worldcontexts among medical student end users [32,34], unexpected
methodological issues and barriers [32], and factors that may
affect engagement [34,35].

This Study
This study is a small-scale feasibility assessment of an adaptive
psychological skill training app for medical student burnout
prevention and well-being using an evidence-informed
behavioral model (Psychological Flexibility).

Intervention Model: Psychological Flexibility
There are no standardized psychological skill training
interventions available for medical student burnout andpsychological well-being [3]. However, Psychological
Flexibility is an adaptive behavioral skill set that has been shown
to protect against adverse mental health outcomes in stressful
situations [18,36-41] and is emerging as a promising intervention
target [17,18,37,42]. The model encompasses 6 modifiablebehavioral flexibility (and corresponding inflexibility) processes:
present-moment awareness (nonawareness), experiential
acceptance (avoidance), cognitive defusion (fusion),self-as-context (self-as-content), contact with values (noncontact
with values), and committed action toward values (inaction)[43]. When exposed to stressful situations, individuals who arehigh in Psychological Inflexibility tend to respond in ways thatare rigidly driven by their internal experiences (eg, fusion withunhelpful thoughts, rules, or self-stories, and avoidance ofemotional discomfort). This can increase their risk of adverse
psychological outcomes by restricting coping repertoires,
disrupting engagement in personally meaningful behaviors, andexacerbating distress [36,44,45]. Conversely, as psychologically
flexible individuals are more able to adopt an open andnonevaluative stance toward internal experiences, they are lesslikely to interpret emotionally uncomfortable stressors as
threatening to their well-being [36]. By focusing less on
avoiding or reducing discomfort and more on connecting withand accepting their direct experiences, psychologically flexible
individuals are able to recognize a broader range of situational
opportunities available to them when responding to stressful
conditions [40,46]. This facilitates purposeful action toward
personally held values and related goals [43]. Early evidence
supports the potential relevance of Psychological Flexibility
skills to medical students, who are less likely to experienceburnout when they adopt experiential acceptance [6,47] andengage in values-driven actions [48]. Conversely, low
Psychological Flexibility among medical students is associatedwith an elevated risk of burnout [49], reduced life satisfaction,and greater personal distress when seeing others in harm [50].

Acceptance and Commitment Training
Psychological Flexibility skills can be developed andstrengthened using Acceptance and Commitment Training(ACTraining) [43]. ACTraining interventions implemented in
medical education settings have demonstrated beneficial impacts
such as reduced burnout among distressed medical students [51]https://formative.jmir.org/2023/1/e43263

XSL• FO
RenderX

Ditton et al
and improved well-being and diminished psychological distressamong female medical students [52]. ACTraining is well suited
to an app-based format as it teaches Psychological Flexibility
skills using metaphors and experiential activities that can bedelivered as brief and practical training components [27]. By
facilitating opportunities to practice during the course of medical
students’ everyday lives, app-delivered ACTraining
interventions could strengthen and generalize Psychological
Flexibility skills [27]. Furthermore, as ACTraining components
have functional links with their corresponding Psychological
Flexibility processes [43,45,53], interventions can be
individualized within an app by identifying which skill a student
requires the most assistance with in a particular moment andaligning the delivery of targeted components with these needs[27]. This is important given the heterogeneity of Psychological
Flexibility skill profiles between individuals and the fact thatthe 6 processes can vary independently of one another[44,54,55], suggesting that medical students are likely to require
training in different skills at different times and in different
situations [44].
At the content level, ACTraining offers potential feasibility
benefits with respect to its acceptability and relevance to medical
students. ACTraining adopts a normalizing stance toward
uncomfortable internal responses to stressors (ie, thoughts,
emotions, and physical sensations) [43,56], providing
opportunities to learn adaptive behavioral skills in a
nonstigmatizing way. Furthermore, as ACTraining encourages
action driven by awareness of an individual’s uniqueexperiences, values, and goals [43], students can learn to apply
the skills in ways that are personally relevant. However, aspectsof ACTraining (eg, self-reflection and bringing attention touncomfortable thoughts and emotions) may be challenging whenexperienced in a self-directed app-based context without guided
support [26]. Evaluating medical students’ subjective
experiences of engaging with ACTraining content is an
important aspect of feasibility assessment, particularly given
the potential impact on app engagement [26]. To our knowledge,
no feasibility data of this nature have been reported previously.

Study Aims and Objectives
We developed an individualized Psychological Flexibility skill
training app (using an ACTraining intervention approach) for
medical student burnout and well-being. The aims of this study
were to assess the feasibility of delivering an ACTraining
intervention to medical students using an app-based delivery
format, and explore how formal evaluation of subjective user
experiences might inform and guide the development of thisapp before implementation in a planned efficacy trial as well
as future research regarding psychological skill training apps
for medical students. This study was an early implementation
trial conducted with a small sample of Australian medical
students who were interested in engaging in an app-based
burnout prevention and well-being intervention. We collectedbehavioral engagement data and detailed student feedback
regarding the ACTraining intervention content, usability of theapp interface, and experiences of engaging with the intervention
across a range of acceptability domains (eg, affective responses,
perceived learning, self-efficacy, and engagement barriers andfacilitators).
JMIR Form Res 2023 | vol. 7 | e43263 | p. 3
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Ditton et al

Methods

all members of the sampling frame met this criterion. There
were no ineligibility criteria.

Study Design

All aspects of the study were implemented on the web. A digital
invitation was sent from a Joint Medical Program administrative
account to the students’ university email accounts, and afollow-up invitation was sent a few days before the final
enrollment date. Students also received a verbal invitation at
the end of a web-based class delivered by a member of theresearch team. Participation in the study was voluntary. Students
were given a URL and QR code that allowed them to accessthe enrollment website (hosted on a secure web-based survey
and database platform, REDCap [Research Electronic Data
Capture; Vanderbilt University]) that provided information
about the study and allowed students to enroll using an e-consent
process. After completing a brief demographic questionnaire,consented students were allocated a unique (4-digit) participant
ID and provided links to download the app via the App Store(Apple) and Play Store (Android). After downloading the app,students created an account. All further assessment, datacollection, and intervention procedures were conducted via theapp, which students accessed independently during the study
period. Self-report outcomes were measured at 3 time points:
baseline (t0), following stage 1 of the intervention (t1), andfollowing stage 2 of the intervention (t2). Owing to the timecommitment involved in the study, an Aus $30 (US $20.43)
gift voucher was offered to participants who completed theintervention and self-report outcome measures. Figure 1 shows
the participant flow diagram.

This study was a single-arm feasibility trial of a 2-stage
individualized ACTraining app.

Participants, Recruitment, and Study Setting
We aimed to recruit approximately 10 to 15 participants,
consistent with sample size recommendations for feasibility
studies [35]. As the app had not been trialed previously, we
limited the number of participants to allow for a smaller-scale,detailed end-user evaluation before broader implementation ina planned larger efficacy trial [57]. Recruitment occurred overa 3-week period in June 2021.
The sampling frame was all students enrolled in year 3 of the
Joint Medical Program at the University of Newcastle andUniversity of New England (Australia) during semester 1 of
2021 (N=170). Year 3 students were selected for pragmatic
reasons (eg, timing of the study in relation to academic demands
and the impact of the COVID-19 pandemic), as recommended
by the University of Newcastle head of medical student
well-being (GH). Students were eligible for inclusion if they
had regular access to a reliable internet connection and anelectronic device compatible with app use (smartphone ortablet). Although the ability to understand English was arequirement for using the app, this was not stated explicitly as

https://formative.jmir.org/2023/1/e43263

XSL• FO
RenderX

JMIR Form Res 2023 | vol. 7 | e43263 | p. 4
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Ditton et al

Figure 1. Participant flow diagram.

Ethical Considerations
Ethics approval for this study was granted by the University of
Newcastle Human Research Ethics Committee on January 21,
2021 (approval ID: H-2020-0311) and ratified by the Universityof New England Human Research Ethics Committee on
February 11, 2021. To ensure the privacy and anonymity ofparticipant data, 2 password-protected databases were used tostore personal and demographic information (in an identifiable
Participant Information Database) separately from the outcome
data (stored in an anonymous Study Database). Outcome data
were deidentified, with participant ID used as a linkage key.
Only the lead author had access to the Participant Information
Database. Members of the research team did not have access to
identifying participant information. To protect privacy while
using the app, students input their unique participant ID as theirusername when creating and accessing their accounts.
Before providing consent, students were informed that (1) datacollected during the study would be anonymous and retained
and stored for a minimum period of 5 years from completionof the research, in accordance with the University of Newcastle
research data management policies; (2) deidentified data would
be analyzed, summarized, and presented in various formats (eg,https://formative.jmir.org/2023/1/e43263

XSL• FO
RenderX

peer-reviewed journal publications); (3) nonidentifiable data
may be used for future ethically approved research; and (4) if
students elected to withdraw from the study, data provided up
to the point of withdrawal would be retained and included in
the study in a deidentified and anonymized form.

Intervention
Overview
The intervention involved the delivery of individualized
ACTraining via an app (“BiSi: Build it. Sustain it.”) developedby clinical psychologists (ED and BK). Existing Psychological
Flexibility training activities and concepts were adapted to suitthe context and target participant group. The app was createdin a low-code platform (Cogniss) that facilitated the deliveryof the outcome measures and intervention and the collection of
study data (including participants’ responses to outcome
measures and use data). Intervention content was delivered inaudio format with accompanying images and some supporting
written text. The ACTraining intervention was delivered in 2stages. Further information regarding the Build it. Sustain it.
app and intervention protocol has been published elsewhere[57].
JMIR Form Res 2023 | vol. 7 | e43263 | p. 5
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Ditton et al

Stage 1: Introductory Module

perspective-shifting techniques, and behavioral activities).
Participants were given 1 week to complete stage 1.

During stage 1, participants used the app to complete an
introductory module that provided a conceptual framework fromwhich to understand the full Psychological Flexibility skill set.
The module was divided into 7 sections (<10 min each), whichwere presented in a fixed order and took approximately 1 hourto complete. Progress was saved at the end of each section,
allowing participants to pause and resume the module if they
wished to complete it over multiple sittings. Section 1 content
included an introduction to burnout (definition, recognizing thesigns, and normalizing and destigmatizing) and well-being(definition and importance of choosing behaviors that support
career sustainability). Sections 2 to 7 provided education about
each Psychological Flexibility skill set (see the screenshot in
Multimedia Appendix 1), including the role of psychologicallyinflexible behaviors in burnout and psychological distress andhow flexibility processes may facilitate more adaptive responses
and psychologically healthy outcomes. Although the focus ofstage 1 was predominantly conceptual, each Psychological
Flexibility process was accompanied by at least one skill practiceactivity (including personal value identification, metaphors,
mindfulness practice, awareness practice, thought defusion,

Stage 2: Individualized Skill Training
Stage 2 was designed to provide students with on-demand accessto a library of short (3-8 min each) practical Psychological
Flexibility skill activities (20-25 activities per process), whichwere targeted to their specific training needs each time they
accessed the app. Completion of stage 1 unlocked access to thestage 2 dashboard (see the screenshot in Multimedia Appendix2), which students could access at any time. Although there wasno requirement to complete a specific number of activities,
participants were sent daily reminder emails and encouraged topractice regularly to optimize skill learning. Activities focusedon normalizing challenging internal experiences, strengthening
present-moment awareness, recognizing the influence of internal
experiences (including thoughts and emotions) on choices andactions, learning to alter unworkable responses to internal
experiences, setting values-based goals, and developing more
flexible and value-aligned behavioral repertoires [43,45,58].
Table 1 provides a summary of the aims and an example activity
for each Psychological Flexibility skill set.

Table 1. Aims and example activities for each Psychological Flexibility skill set.
Skill set

Aim

Example activity

Present-moment Teach students to bring purposeful awareness to their presentGuided mindfulness activity in which students practice attending
awareness
moment experiences and recognize how these experiences influ- to specific aspects of their present-moment experience (eg,ence their choices (in everyday life and stressful situations)sounds, physical sensations, thoughts, emotions, and breathing)
Defusion

Teach students to distinguish between their direct experiences
and their thoughts about their experiences and respond more
flexibly to the latter by changing the contexts in which they experience their thoughts

Students practice responding flexibly to a thought by choosing
an action that contradicts a thought they are having (eg, they are
asked to think “I can’t lift my arm” and to also lift their arm).

Self-as-context(perspective
taking)Teach students to recognize that their thoughts and emotions do
not define their identity but are transient aspects of their experience, which they can observe from a more stable and constantsense of “self”

Students identify a “self-story” that causes distress or limits their
response options in certain situations and practice choosing actions that expand their behavioral repertoires outside of thesestories.

Acceptance(willingness)

Teach students to identify unhelpful avoidance behaviors andStudents practice observing an uncomfortable sensation (eg, anpractice open, flexible, and adaptive ways of responding to inter- urge or an itch) with openness, curiosity, and willingness (eg,nal discomfort
notice the components of the sensation and allow them to comeand go without buying into evaluations of the sensations or attempting to alter them).

ValuesTeach students to identify their own personally held values andnotice how connection or disconnection with these values cancontribute to well-being or burnout experiences, respectively

Committed action

Teach students to choose effective and purposeful behaviors that Students select a personally held value that feels most importantalign with personally held values even when internal discomfort to them in that moment and commit to taking one small andis present; use direct experiences to recognize the difference be- achievable action toward that value during the day.
tween well-being–oriented action and inflexible persistence that
could increase burnout risk

Individual tailoring of Psychological Flexibility skill activity
delivery was implemented using a method adapted from asimilar previous study [27]. At each app log-in, participants
responded to a single-item screening question: which of thefollowing are you having the most difficulty with today? (seethe screenshot in Multimedia Appendix 3). Each response option
corresponded to 1 of the 6 Psychological Flexibility processes(Textbox 1). After selecting a response, participants weredirected to a dashboard that provided access to all skill activities
https://formative.jmir.org/2023/1/e43263

XSL• FO
RenderX

Students choose one of their own personally held values andbriefly reflect on how their behaviors have aligned with thatvalue during the day.

available for the Psychological Flexibility process they had
identified as having the most difficulty with on that occasion(presented in a list). Students were then able to practice one of
the available skill activities, either by choosing from the list or
by allowing the app to make a random selection for them (seethe screenshot in Multimedia Appendix 4). Upon completionof an activity, participants had the option to complete anotheractivity within the same Psychological Flexibility skill set. These

JMIR Form Res 2023 | vol. 7 | e43263 | p. 6
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
steps were repeated until the participants elected to discontinue.

Ditton et al
Participants had access to stage 2 for 3 weeks.

Textbox 1. Stage 2 individualization screening question options and corresponding Psychological Flexibility processes.
•

Struggling with your feelings: acceptance (willingness)•

Unable to do what matters to you: committed action•

Stuck in your thoughts: defusion•

Stuck in autopilot or struggling to stay in the present moment: present-moment awareness•Disconnected from a sense of meaning or purpose: values•

Stuck in stories about who you are or who you should be: self-as-context (perspective taking)Engagement
We integrated design elements aimed at enhancing medical
students’ engagement with the app-based intervention. In
addition to individualization [27], this included the provisionof daily reminder emails [59], offering a variety of training
activities for each skill set [27,60], and providing opportunities
to engage in self-reflection while engaging in the educationaland skill practice components of the app [60].

https://formative.jmir.org/2023/1/e43263

XSL• FO
RenderX

Psychological Outcome Measures
Overview
We administered self-report psychological outcome measures(Table 2) at t0 only. The purpose of administering these outcome
measures in this study was to evaluate the feasibility of using
the app delivery method to assess psychological outcomes ofinterest in the planned efficacy trial and explore the potential
assessment burden to participants. Psychological outcome data
were also used to describe the baseline psychologicalcharacteristics of this feasibility trial sample.

JMIR Form Res 2023 | vol. 7 | e43263 | p. 7
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Ditton et al

Table 2. Summary of outcome measure characteristics and internal consistency (Cronbach α) for this sample.
Outcome measure

Items, N

Measure characteristics

Cronbach α

16•

Validity [62] and reliability have been demonstrated among medical students [63].

.84•

Respondents rate the frequency of each burnout experience using a 7-point Likert
scale.

•

Provides summed total scores for each burnout factor; higher Eb and Cc and lower

Psychological outcomes
Burnout—MBI-GS(S)a[61]

AEd scores reflect a higher frequency of burnout experiences.
Well-being—MHC-SFe
[19]

14

•
•
•

Psychological Flexibility
and Inflexibility—MPFI-

24•

Provides average item scores for 3 well-being factors: EWBf, SWBg, and PWBh.
Higher scores reflect a higher well-being.

•

This scale has demonstrated validity (including convergent, predictive, and discrim- .89

i

SF [54]•••

Depression, anxiety, and21

stress—DASS-21l [66]Validity [19] and reliability (including internal consistency and test-retest reliability) .91[64] have been demonstrated.
Respondents rate the frequency of well-being experiences during the previous monthusing a 6-point Likert scale.

•••inant validity between PFj and PIk scales [55]) and reliability (including internalconsistency [65] and responsiveness to change over time [54]).
Respondents rate the frequency of PF and PI experiences during the past 2 weeks
using a 6-point Likert scale.
Provides global PF and global PI composite scores.
High PF scores reflect higher flexibility, and higher PI scores reflect higher inflexibility.
Validity and reliability have been demonstrated [66,67].
.91
Respondents rate the frequency of depression, anxiety, and stress symptoms experienced during the past week using a 4-point Likert scale.
Provides subscale scores estimating the severity of depression, anxiety, and stresssymptoms [66] and a total score reflecting general negative affectivity or psychological distress [67].

Feasibility outcomesApp usability—SUSm [27] 10

•
•
•
•
•

User Experience Assessment part 1

32

•
•
•

8

•

Participants are asked to provide short-answer responses to open-ended questions
about their experiences of using the app.

Learning experiences assessment

13•

Questions were adapted from Kinnunen et al [70] (items altered to suit learning ob- .90
jectives of this intervention).
Participants rate the degree to which they agree with statements about their learning
experiences using a 5-point Likert scale (1=“strongly disagree” to 5=“strongly
agree”).
The scale provides an average item score.
Higher scores indicate higher perceived skill learning.

••https://formative.jmir.org/2023/1/e43263

RenderX

Respondents rate their level of agreement with each statement about experiences of .93
engaging with the intervention app using a 5-point Likert scale (1=“strongly disagree”
to 5=“strongly agree”).
Provides average user experience score, where higher scores reflect more favorable
intervention experiences.
An additional 2 items (not included in average user experience rating) assessed students’ perceptions of the duration of the intervention (1=“way too short,” 2=“tooshort,” 3=“just right,” 4=“too long,” and 5=“way too long”) and frequency of thereminder emails (1=“way too infrequent,” 2=“too infrequent,” 3=“just right,” 4=“toofrequent,” and 5=“way too frequent”).

User Experience Assessment part 2•

XSL• FO

Validity and reliability have been demonstrated [27].
.81
Respondents rate the degree to which they agree with usability statements about theapp using a 5-point Likert scale (1=“strongly disagree” to 5=“strongly agree”).
Provides a total score ranging from 0 to 100.
Higher scores reflect higher usability (85-100=excellent, 73-84=good, 62-73=low
marginal, and <70=needs to be reviewed and improved) [68].
In accordance with recommendations by other authors, we amended item wording
to suit our intervention context and audience (eg, “system” changed to “App”) [69].

N/An

JMIR Form Res 2023 | vol. 7 | e43263 | p. 8
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
a

Ditton et al

MBI-GS(S): Maslach Burnout Inventory General Survey for Students.

b

E: exhaustion.

c

C: cynicism.

d

AE: academic efficacy.

e

MHC-SF: Mental Health Continuum-Short Form.

f

EWB: emotional well-being.

g

SWB: social well-being.

h

PWB: psychological well-being.

i

MPFI-SF: Multidimensional Psychological Flexibility Index-Short Form.

j

PF: Psychological Flexibility.

k

PI: Psychological Inflexibility.

l

DASS-21: Depression, Anxiety, and Stress Scale-21.

m

SUS: System Usability Scale.

n

N/A: not applicable.

Burnout
Improving burnout outcomes is a core goal of this intervention,and burnout is the primary outcome for the planned efficacy
trial. We assessed burnout using the gold-standard measure, theMaslach Burnout Inventory [61]. Of the range of versions
available, the Maslach Burnout Inventory General Survey forStudents [61] was considered the most appropriate for thiscohort.

Well-being
Improving well-being is a core goal of this intervention, andwell-being is an outcome of interest to the planned efficacy
trial. We assessed this outcome using the short form of the
Mental Health Continuum [19].

Psychological Flexibility and Inflexibility
In the planned efficacy trial, we aim to explore whether
engagement with the intervention app improves medical
students’ Psychological Flexibility and Inflexibility and evaluate
whether changes in these processes mediate burnout andwell-being outcomes [57]. As there is growing evidence that
Psychological Flexibility and Inflexibility are “conceptually
distinct” processes that can exert independent effects on
outcomes and may respond differently to interventions [54,55],
we selected the Multidimensional Psychological Flexibility
Index-Short Form [54], which assesses Psychological Flexibility
and Inflexibility as separate processes rather than as a single
dimension [55]. This will facilitate the evaluation of potential
differential intervention and mediation effects in the future
efficacy study.

Depression, Anxiety, and StressIn addition to burnout and well-being, the planned efficacy trial
aims to explore the potential benefits of the intervention app toother psychological health outcomes relevant to medical
students. As such, the Depression, Anxiety, and Stress Scale-21[66] was administered to evaluate medical students’
psychological health outcomes more broadly.

https://formative.jmir.org/2023/1/e43263

XSL• FO
RenderX

Feasibility Outcome Measures
Behavioral Use Data
App Functionality
The app collected participants’ use data, including whichassessment and intervention components were accessed and inwhich order. This allowed us to observe whether the app
facilitated participants’ progress through the study protocol in
the intended way. We defined the app as functional if theoutcome measures and intervention components were delivered
to all participants at the intended time and in the intended order,
if individualization procedures were implemented as intended,
if at least one participant completed the study protocol in its
entirety, and if the app generated data output required to evaluate
intervention outcomes.
Engagement
Engagement at key points in the study was defined as thenumber of participants who (1) downloaded the app, (2)completed all outcome measures at t0, (3) completed theintroductory module at stage 1, (4) completed all outcome
measures at t1, (5) completed at least one skill training activity
at stage 2, (6) completed all outcome measures at t2, and (7)were still using the app at least 2 weeks after their initial sign-up.
Frequency of App Use
Frequency of app use was defined as the number of log-ins
during the study and the number of skill activities accessed andcompleted.
Time of App Use
The app captured the time of day of each log-in, which we
categorized using the following definitions: early morning (5AM to 9 AM), morning (9 AM to noon), afternoon (noon to 5PM), evening (5 PM to 9 PM), late night (9 PM to midnight),
and after midnight (midnight to 5 AM).

Self-report Measures of Usability and User Experiences
Self-report usability and user experience measures (Table 2)
were administered at t1 (to obtain students’ feedback on theintervention following the stage 1 introductory module) and t2(to obtain feedback following the stage 2 individualized skill
training access period).
JMIR Form Res 2023 | vol. 7 | e43263 | p. 9
(page number not for citation purposes)

JMIR FORMATIVE RESEARCHApp Interface Usability
We assessed medical students’ satisfaction with the usabilityof the app as subjective perceptions of the functionality of adigital psychological intervention can influence engagement[26,71]. The System Usability Scale (SUS) is a valid and reliablemeasure of this outcome [72] and is the most frequently usedsubjective measure of program usability [71].
User Experiences
We used the theoretical framework of acceptability (TFA) [31]to guide the assessment of user experiences with theintervention. The TFA is a comprehensive research-based modelthat recommends evaluating participant-rated acceptability ofhealth care interventions using 7 emotional and cognitive
indicators: affective attitude, perceived effectiveness,intervention coherence, self-efficacy, opportunity costs, burden,
and ethicality [31]. As there were no available standardized
TFA measures, we used the definitions of the 7 indicators to
inform the development of a 2-part User Experience Assessment
for the purpose of this study (see Multimedia Appendix 5 forindicator definitions and assessment questions). The
development of the questions was further guided by research
related to factors that affect the acceptability of psychologicalinterventions among medical students (eg, time [28], privacy[6,47], and accessibility and flexibility [28]), health careinterventions (eg, affective experiences and satisfaction [31]),and digital interventions (eg, self-guided [26], expectation of
benefit [73], quality, and content relevance [26]). Where
appropriate, questions were framed with respect to the outcomestargeted by the intervention (ie, burnout and well-being) toevaluate factors such as the perceived effectiveness andrelevance of this approach among medical students. Part 1 asked
participants to indicate their level of agreement with statements
about their experiences of using the Psychological Flexibility
app. We defined items with an average score of <3.8 as aspectsof the intervention that might require revision before the planned
efficacy trial. This value was selected as it represented anaverage rating of a little below agree. We defined items withan average score of >4 (“agree”) as intervention strengths. Part
2 consisted of 8 short-answer questions, which were included
to collect feedback regarding facilitators of and barriers toengagement as well as other information that we may not haveconsidered.
Learning Experiences
As part of assessing the feasibility of adapting ACTraining intoa digital learning format, we identified key learning goals andasked medical students to rate the degree to which they felt thatthese had been achieved by engaging with the app. A learning
experiences assessment previously used in a similar web-based
intervention study [70] was adapted to align with the focus of
this app (Multimedia Appendix 6 [70]). Questions related to
medical students’ self-reported ability to identify Psychological
Inflexibility processes and apply Psychological Flexibility skills
across each of the core processes.
Skill Activity Feedback
After completing a skill activity during stage 2, participants
were asked to rate whether they liked the activity using asingle-item measure assessed on a binary scale (using a
https://formative.jmir.org/2023/1/e43263

XSL• FO
RenderX

Ditton et al
thumbs-up or thumbs-down icon). This method of evaluatinguser experiences of intervention content has been used inprevious digital intervention studies [74].
Intervention Harms
Participants were invited to report concerns or harms
experienced during the study using contact links provided within
the app and via email communications sent during the trial (eg,daily reminders).

Data Analyses
Quantitative Data
Descriptive analyses (including mean, SD, mode, minimum,and maximum) of quantitative demographic, psychological,
feasibility, and behavioral engagement outcome data were
performed using the Jamovi statistical software (version 2.3.0)[75].

Qualitative Data
Participants’ short-answer responses to the open-ended user
experience questions were evaluated qualitatively. Responses
were independently reviewed by 2 members of the researchteam (ED and BK) and coded thematically according to theirrelevance to usability, accessibility, external factors (eg, time),
behavioral factors (eg, practice), or any of the TFA indicators(affective attitude, perceived effectiveness, intervention
coherence, self-efficacy, opportunity costs, burden, andethicality). Each response was coded as a barrier to or facilitatorof engagement or a positive or negative evaluation of an aspectof the intervention. The coding was then reviewed together todetermine common themes and subthemes in a reiterative
process.

Results
Participant Characteristics at Baseline (t0)
Demographics
Figure 1 shows that 19 medical students enrolled and 11 (58%)
actively engaged in the study and provided user experience
feedback. Of the 11 actively engaged participants, 10 (91%)
identified as female, 10 (91%) were enrolled as domestic
students, and 2 (18%) were Aboriginal individuals. Participants
were aged between 20 and 23 (mean 21, SD 1.2) years. The
average time engaged in the workforce was 4.5 years (SD 2.3),
and medicine was the first career for most (10/11, 91%)
participants. In total, 73% (8/11) of the participants reported
having experienced burnout in the past. Actively engaged
participants rated their physical health as good (mean 3.8, SD
0.4). Diet (mean 3.6, SD 0.5) and self-care (mean 3.5, SD 0.7)
ratings fell between average and good.
Of the participants lost to follow-up, 50% (4/8) identified as
male, reflecting that most of the enrolled male students (4/5,80%) discontinued the study before engaging in or providing
feedback on the intervention. Although the demographic
characteristics of the students lost to follow-up were generally
similar to those of the students who engaged in the intervention,
we noted that reported self-care was lower for students who
discontinued early (mean 2.8, SD 0.7).
JMIR Form Res 2023 | vol. 7 | e43263 | p. 10(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Ditton et al

Psychological Characteristics

Table 3 shows baseline psychological characteristics of theparticipants who actively engaged in the study.

Summary

Table 3. Baseline psychological characteristics of actively engaged participants (N=11).
OutcomeMean (SD)

Range

Emotional exhaustion

16.00 (6.24)

7-24

Cynicism

8.45 (3.50)

5-16

Academic efficacy

24.82 (5.90)

18-32

Emotional

11.36 (2.54)

6-14

Social13.45 (4.06)

8-20

Psychological19.55 (4.57)

14-27

Psychological Flexibility

3.71 (0.67)

2.67-5

Psychological Inflexibility

3.03 (0.47)

2.25-3.92

Depression

8.55 (6.82)

0-20

Anxiety

8.18 (7.12)

0-22

Stress

14.4 (10.23)

0-32

Burnout

Well-being

Burnout
Mean total scores for exhaustion (mean 16.00, SD 6.24) andcynicism (mean 8.45, SD 3.50) for actively engaged students
were comparable with those of a recently published sample ofmedical students who, by subjective self-report, were notexperiencing burnout (exhaustion mean 14.96; cynicism mean
7.59) [63]. Conversely, our sample’s mean total score foracademic efficacy (mean 24.82, SD 5.90) was comparable withthat of a sample of medical students who believed themselves
to be experiencing burnout (mean 24.81) [63].
Well-being
On average, participants’ emotional (mean 11.36, SD 2.54),
social (mean 13.45, SD 4.06), and psychological (mean 19.55,
SD 4.57) well-being scores were slightly lower than those of acomparable published sample of medical students (emotional
well-being mean 12.79; social well-being mean 14.84;
psychological well-being mean 24.42) [76].
Psychological Flexibility and Inflexibility
Multidimensional Psychological Flexibility Index-Short Formscores indicated that, on average, participants had had
experiences consistent with Psychological Flexibility often tovery often during the previous 2 weeks (mean 3.71, SD 0.67),
which was comparable with a general population sample (mean
3.83) [77]. On average, students reported that they had often
had experiences consistent with Psychological Inflexibility
during the previous 2 weeks (mean 3.03, SD 0.47), which wasslightly more frequent than the general population sample (mean2.73) [77].
Depression, Anxiety, and Stress
The average Depression, Anxiety, and Stress Scale-21 scoresindicated that medical students in this trial were experiencing
depression (mean 8.55, SD 6.82), anxiety (mean 8.18, SD 7.12),
https://formative.jmir.org/2023/1/e43263

XSL• FO
RenderX

and stress (mean 10.36, SD 10.23) at levels higher than thegeneral population averages (depression: mean 6.34, SD 6.97;
anxiety: mean 4.7, SD 4.91; stress: mean 10.11, SD 7.91) [66].

Feasibility Outcomes
Behavioral Use Data
App Functionality
The app generally functioned as intended during the trial. Usedata demonstrated that outcome assessments and intervention
components were delivered in the intended order and at theintended time, individualization procedures were implemented
as planned, 16% (3/19) of the participants successfullycompleted the intervention protocol in its entirety, and the app
generated the data output required to evaluate intervention
outcomes. We noted that a system glitch resulted in participants
being sent the daily reminder email up to 4 times each day.
Engagement
Of the 19 participants who were enrolled in the study, 15 (79%)
downloaded the app and created an account, 14 (74%) completed
all t0 outcome measures, 11 (58%) completed stage 1 and thet1 outcome measures, 8 (42%) engaged in stage 2, and 3 (16%)
completed the t2 outcome measures (Figure 1). Of theparticipants who downloaded the app and created an account,
47% (7/15) continued to use the app for at least 2 weeks fromthe date of their initial log-in.
Frequency of App Use
Students who were actively engaged in the study (11/19, 58%)
logged into the app between 2 and 10 times (total=70; mean
6.36, SD 2). Log-in frequency diminished between stage 1(total=41; mean 3.73, SD 1) and stage 2 (total=29; mean 2.64,
SD 2). It took students between 1 and 5 sittings to complete all
the stage 1 activities. Participants who logged in during stage
JMIR Form Res 2023 | vol. 7 | e43263 | p. 11(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
2 (8/11, 73%) accessed a total of 24 skill activities (mean 2.18,
SD 1), of which 16 (67%) were completed (mean 1.45, SD 1).
Time of App Use
Table 4 shows that 56% (39/70) of app use was distributed fairly
evenly throughout the day (between 9 AM and 9 PM) and that

Ditton et al
medical students used the app most frequently between 9 PM
and midnight (21/70, 30%). Although reasonably infrequent(5/70, 7%), medical students occasionally used the app after
midnight.

Table 4. Frequency of app use by time of day (percentage of total log-ins; N=70).
Time of log-in

Log-in frequency, n (%)

Early morning (5 AM to 9 AM)

5 (7)Morning (9 AM to noon)

12 (17)Afternoon (noon to 5 PM)

14 (20)Evening (5 PM to 9 PM)

13 (19)Late night (9 PM to midnight)

21 (30)After midnight (midnight to 6 AM)

5 (7)Self-report Data
Overview
At t1, self-report measures of user experiences were completed
by 100% (11/11) of the participants who completed stage 1. At
t2, these measures were completed by 38% (3/8) of theparticipants who engaged in stage 2. Owing to the low response
rate at t2, we report the minimum and maximum scores for eachoutcome instead of the average scores.
App Interface Usability
The average total SUS score at t1 (mean 85.0, SD 9.3) indicatesexcellent usability [68]. Individual participant SUS scores
ranged between 70 (low marginal) and 96 (excellent), indicatingthat there were substantial differences in participants’satisfaction with the usability of the app interface. Similarly, at
t2, individual usability ratings varied, ranging between 72 and96.
The qualitative feedback provided by participants at t1 offers
insights into the factors that may have influenced this variability.
Many students reported feeling satisfied by the ease of use andnavigation, logical structure and flow of the app interface,provision of simple and clear instructions, and continuation ofaudio play when their device auto-locked. However, app
usability elements that generated dissatisfaction included lackof access to a dashboard or home screen during stage 1,
insufficient feedback regarding how far students had progressed
through the introductory module, having to complete module
components in a fixed order rather than being able to navigate
flexibly through the concepts, and the delivery of reminders via
email rather than in-app push notifications. Although somestudents favored the audio delivery of content, others reported
that having a written version of all content would have improved
accessibility (eg, audio content was a barrier to usability whenin public, and audio content delivery did not match personallearning styles for some).
User Experiences
At t1, the average total user experience ratings ranged between
slightly agree (3.4) and strongly agree (4.8; mean 4.05, SD
https://formative.jmir.org/2023/1/e43263

XSL• FO
RenderX0.34), indicating that participants who engaged in stage 1
generally found the intervention to be acceptable with respect
to the experiences assessed. Similar findings were observed at
t2 (range 3.3-4.6).
The average participant scores for each user experience question
at t1 are presented in Multimedia Appendix 6. We identified
several intervention strengths (average item scores of >4).
Strengths related to the app delivery format and interface
included quality, visual appearance, web-based availability,
ability to access at a personally convenient time, maintenanceof privacy, and did not get in the way of other important life
activities. Strengths related to the ACTraining intervention
content included opportunities for self-reflection, content
relevance, anticipated benefits of continued skill use, anticipatedbenefits to well-being during career, concepts were easy tounderstand, felt confident performing the skill activities while
using the app, felt capable of dealing with any challenging
internal experiences that arose while using the app, alignedwith personal values, and the time and effort invested in thePsychological Flexibility training was worthwhile. Aftercompleting the introductory module, most students (8/11, 73%)
agreed or strongly agreed that they would like to use the app
again in the future (mean 3.91, SD 0.94).
We identified 5 user experience items with average scores below
the 3.8 cutoff at t1, indicating that these intervention elements
might require additional attention before the larger trial. These
items were enjoyed using the app, held interest and attention,
received amount of training needed to achieve the goals thatare important to me, felt confident using the skills in everyday
life, and I believe these skills will help me prevent burnout.
Qualitative feedback provided by 18% (2/11) of the participants
indicated that they did not feel that burnout prevention shouldbe the focus of the intervention as the techniques were “helpful
for many different things.”
Most participants (9/11, 82%) indicated that the 1-hour
introductory module was slightly too long, whereas theremaining 18% (2/11) of participants felt that the duration wasjust right. However, qualitative feedback indicated that someparticipants felt that the content was as concise as it could be
JMIR Form Res 2023 | vol. 7 | e43263 | p. 12(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Ditton et al

without compromising the value of the intervention. At bothtime points, participants indicated that they did not findcompleting the outcome questionnaires to be burdensome.

Discussion

The qualitative user experience feedback at t1 generally reflected
the quantitative findings. Most students who completed stage
1 provided favorable feedback about the Psychological
Flexibility intervention content, describing the app as “great,”“useful,” and “helpful” and suggesting that the training “needs
to be widely available.” One of the reported benefits of the
app-based delivery was that students liked being able to engage
in short sections and pick up where they left off rather thanhaving to complete the entire intervention at once. Students
reported enjoying the skill activities, stating that they “provided
context and demonstrated how to use [the skills],” and somestudents requested “more interactive components and practiceactivities” and more “clinical examples.” T1 qualitative dataalso highlighted the self-efficacy challenges faced by students
when accessing Psychological Flexibility skill training via anapp, including feeling “too overwhelmed” in real-world
situations to use the skills, not recognizing opportunities to use
the skills when they arose, and not knowing which skills toapply in which situations. A participant reported feeling thatthey had learned the Psychological Flexibility “concepts” butnot the “behavior.” These findings align with the fact that stage
1 focused on conceptual learning and participants had only
engaged in minimal skill training at this stage of the intervention.
Students also reported that a key barrier to using the app was
having time available to do so.

This early implementation trial with a small group of Australian
medical students provided preliminary support for the feasibilityof an individualized ACTraining intervention app for well-being
and burnout as well as user experience feedback that may be
used to guide the ongoing development of the app and future
research. In relation to our first study aim, we demonstrated
that the app could successfully administer outcome assessments,
ACTraining intervention components, and Psychological
Flexibility skill individualization procedures. These functional
feasibility data were supported by subjective user experience
feedback provided by medical students who actively engaged
with the app. The interface was, on average, rated as having
excellent usability, and students responded favorably toaccessibility factors related to the app-based delivery (eg,maintenance of privacy and ability to access at any time).
Furthermore, we demonstrated the feasibility of using an app
to deliver ACTraining content and skill training to a sample ofmedical students in a way that they found easy to understandand acceptable.

Qualitative data collected at t2 provided further insights into theperceptions of a small selection of participants regarding thestage 2 individualized skill training. Students liked the app
interface and the short duration of skill activities. However, thehigh frequency of reminders resulting from an app glitch wasreported as unsatisfactory and frustrating. A participantsuggested that reminders should be delivered at the end of theworkday as this would align better with medical students’schedules and reduce burden. Barriers to engagement in skill
training included life commitments, time, forgetting, and thefact that the training had not yet become a priority or a habit.
Learning Experiences
At t1, participants’ learning experience assessment scores ranged
from 2.92 (slightly agree) to 4.1 (agree; mean 3.7, SD 0.51).
The most frequently endorsed rating for 92% (12/13) of thelearning experiences was agree (mode=4). At t2, scores ranged
from 2.62 to 4, with a participant indicating that they did notlearn the skills targeted by the intervention and 18% (2/11) of
the participants agreeing that they had learned the skills.
Skill Activity Feedback
Students “liked” 88% (14/16) of the activities completed during
stage 2.
Intervention Harms
No harms were reported during this trial.

https://formative.jmir.org/2023/1/e43263

XSL• FO
RenderX

Feasibility, Engagement, and User Experiences

More broadly, the feasibility of delivering psychological skill
training via an app is contingent on the degree to which thismethod can facilitate and maintain end-user engagement with
the intervention content. We found that 42% (8/19) of the
medical students who enrolled in the study disengaged withoutcompleting the introductory module, and 50% (4/8) of thosestudents did not proceed as far as downloading or signing up
to the app. These students reported lower average self-careratings than those who completed the introductory module, andfuture research should explore whether existing self-carebehavioral patterns could impede early engagement inpsychological intervention apps. Regarding maintenance ofengagement, the frequency of app log-ins diminished between
stages 1 and 2. Less than half (7/15, 47%) of the students who
downloaded the app were still engaged after 2 weeks. Those
who used the app during stage 2 completed <2 skill activities
on average despite “liking” most activities completed. Although
the observed pattern of disengagement over time is common
for digital and app-based psychological interventions [30,78],
it is incongruent with the fact that students who completed theintroductory module generally reported that they would like to
use the app in the future. These findings reinforce the need fora detailed formal evaluation of end-user experiences to better
understand factors that may affect feasibility [1] and guide app
intervention development, which was the second aim of thisstudy.
Previous research evaluating an app-delivered ACTraining
intervention indicated that disengagement is not necessarily an
unfavorable outcome and might reflect the successfulachievement of an individual’s learning goals [27]. User
experience findings suggest that this was not the case in thissample. Learning experience scores showed that medical
students who completed the introductory module agreed thatthey had learned most of the Psychological Flexibility skills
targeted by the ACTraining app to some extent. However, on
JMIR Form Res 2023 | vol. 7 | e43263 | p. 13(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
average, participants only slightly agreed that they had received
the amount of training needed to achieve the outcomes that were
important to them, and qualitative feedback indicated that theywanted more opportunities for skill practice. Thus, declining
engagement following the introductory module was likely
related to other factors.
Medical students who completed the introductory module
predominantly reported finding the Psychological Flexibility
concepts and skills relevant, helpful, and worthwhile. Students
tended to perceive that the skills had greater benefits andrelevance to well-being than burnout prevention. Given that theintervention was intended to have relevance to both outcomes,
this feedback suggests that content might not have aligned
sufficiently with factors that participants perceived to be relevantto their burnout experiences. A future step in the development
process of this app will involve consultation with medical
student end users regarding which aspects of their academic
and clinical experiences they consider to be most associatedwith burnout risk. Intervention content can then be adjusted todemonstrate how Psychological Flexibility skills might be
implemented to mitigate these risk factors, which could increase
perceived relevance and engagement. Some students astutely
recognized the potential for Psychological Flexibility to offer
benefits beyond burnout prevention and proposed that theintervention shift its focus and messaging to the broader
relevance of these skills. This approach may increase theengagement of medical students interested in well-being andother outcomes. However, removing the burnout prevention
focus could also result in failure to engage students who
self-identify as being at risk as well as missing opportunities totrain important risk-mitigating applications of Psychological
Flexibility skills. Future research might explore whether
engagement is improved by using the app to assess which
outcomes are most important to individual medical students andindividualizing the focus of intervention content based on these
preferences.
Performing self-reflection and skill-learning tasks without
in-person guidance is a potential challenge for individuals
engaging in a psychological intervention using a stand-alone
app [26]. Medical students who actively trialed this intervention
reported both enjoyment and confidence with respect to these
activities while using the app. However, lower levels ofconfidence and self-efficacy were reported with respect toimplementing the skills in real-world situations, with somestudents not knowing when or how to apply specific skills andothers feeling “too overwhelmed” to do so. Previous researchhas demonstrated that perceived self-efficacy can affectengagement in digital health interventions [73]. Although stage
2 of the app was intended to strengthen and generalize skill
acquisition by providing access to targeted training in real-worldcontexts, clearer guidance may have been required regarding
when and how to practice the skills. Future research may also
consider how to use in-app assessment of students’ confidence
and self-efficacy to tailor intervention delivery to their current
skill level and explore whether this enhances engagement.
We identified enjoyment of the intervention and its capacity tohold medical students’ interest and attention as aspects of theapp requiring further attention. Most participants who trialed
https://formative.jmir.org/2023/1/e43263

XSL• FO
RenderX

Ditton et al
the app (9/11, 82%) reported finding the 1-hour introductory
module slightly too long. Given that this was very brief
compared with face-to-face ACTraining interventions [79-81]
and that most students reported wanting further training,
reducing the volume of introductory content could affect future
efficacy and is not an ideal solution. Rather, qualitative user
experience feedback highlighted aspects of the app interface
that may have contributed to this module feeling subjectively
long. Delivery of content in a fixed order was a frustration formany students, who reported finding it difficult to track how
far they had progressed through the module and would havepreferred the autonomy to move between topics using a home
screen. Students indicated that they would have liked moreinteractive components and opportunities to practice skills andself-reflection, suggesting that the introductory content could
have been delivered in a more engaging format. Furthermore,
although many students responded favorably to theaudio-delivered content, others reported that the availability ofa written version would have increased accessibility andengagement. Medical students’ preferences for interacting withapp-based media appeared to affect their experiences ofparticipating in the intervention. When adapting establishedpsychological intervention models to app-based intervention
models, future research may explore whether evaluating andaccommodating medical students’ interface-related preferences
strengthens interest and engagement.
Feedback provided by medical students who actively
participated in the intervention indicated that the short andresumable nature of the app-delivered skill activities facilitatedengagement and that the time invested was worthwhile.
However, time factors and demands related to medical studies
were reported barriers to app use, which is consistent withprevious research evaluating an app-delivered psychologicalintervention for medical residents [22]. Participant feedback
suggested that this might be mitigated by amending the format
and timing of reminders, such as push notifications at the endof the day, when there are fewer competing demands associatedwith medical studies. This approach is supported by the observed
timing of app engagement occurring frequently after 9 PM.
Although this intervention focused on strategies to support
well-being behaviors in the context of time pressures anddemands, future interventions should introduce thesecomponents during the early stages of training, when app
engagement is at its highest.
Medical students in this sample reported depression, anxiety,and stress experiences at levels higher than those observed ingeneral population samples [66], and previous research indicatesthat these factors can affect engagement in digital interventions[73]. Furthermore, this sample reported high baseline
Psychological Inflexibility compared with a general population
sample [77]. Although the ACTraining app was intended to
improve this outcome, if existing inflexible behavioral
repertoires function as barriers to engagement, then steps would
need to be taken to mitigate this during the early stages of theintervention. Future research should explore whether baseline
flexibility and inflexibility predict app engagement.

JMIR Form Res 2023 | vol. 7 | e43263 | p. 14(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Ditton et al

Limitations

Theoretical Contributions and Conclusions

Although this trial had the full support of the universities inwhich it was conducted, the intervention was intentionallydistanced from students’ medical education programs to prevent
perceived participation coercion. Although important from a
research ethics perspective, this meant that the app was delivered
in isolation without external contextual factors that might havefacilitated engagement. For example, delivery as a formal partof medical education training could normalize and legitimize
the value of well-being skill training, which may strengthen
students’ justification for prioritizing and allocating time forengagement [5,6]. There is support for individual skill training
interventions to be formally incorporated into broader well-being
and burnout prevention initiatives within medical education[10], and future research should explore whether this approach
could strengthen engagement in app-based training.

Despite its limitations, this study reports detailed user experience
data that have been largely absent from the literature on digital
psychological interventions for medical students [1] and outlines
a process for the important step of assessing feasibility outcomesas part of app development. The findings support the feasibilityof implementing this app in a larger efficacy trial and provide
user experience feedback that could be used to improve theintervention and strengthen engagement. Intervention
development was driven by research suggesting the relevanceof Psychological Flexibility skills to medical students withrespect to burnout and well-being [6,47-50] and the potential
benefits of app-delivered psychological skill training for thisgroup [1,6,12,26-29]. Although this is early-phase work, thestudy provides preliminary support for the feasibility ofdelivering an ACTraining intervention to medical students using
a stand-alone app and supports the value of further evaluationof this approach in burnout prevention and well-being research.
Psychological Flexibility has recently been described as “thecornerstone of psychological health and resiliency” [82] becauseof its broad well-being benefits. With continued evaluation andrefinement, ACTraining apps offer a promising approach tofacilitating medical students’ development of these importantskills, which could be useful in the context of unavoidable
academic and occupational stressors.

Engagement rates for this trial should be interpreted withcaution. The volume of study-related outcome measures thatwere extraneous to the intervention itself was high and might
have contributed to disengagement, particularly given that time
was reported as a barrier to app engagement even among those
who participated for the full duration of the study. Although
medical students who completed the outcome measures did not
find them to be burdensome, this may not have been the casefor those who disengaged. The user experience findings werebiased toward the opinions of medical students who wereactively engaged in the intervention. Future feasibility studies
should adopt a secondary method of follow-up data collection
to assess the experiences of students who discontinue app use
to facilitate a better understanding of engagement barriers.
The collection of user experience data using a nonstandardized
questionnaire written for the purpose of this study was alimitation. Although the User Experience Assessment wastheory-driven, it is possible that researcher subjectivity might
have biased its development and response interpretation. Given
the lack of quantitative scales based on the TFA [31], we havemade this assessment available (Multimedia Appendix 6) tofacilitate further validation and refinement of feasibility
assessment strategies for psychological app development studies.
The skew toward female participants and small sample size
prevent the generalizability of the observed findings to medical
students more broadly. Furthermore, although subjective
feedback was informative regarding what was important toparticipants while using the app, we were unable to confirm
whether these factors affected actual behavioral engagement
because of the sample size and study design, and further researchis required to explore the proposed relationships. Finally, thelow level of engagement with the stage 2 individualized skill
training component meant that most of the user experience datacollected pertained to the introductory module only.

This study is the first to report detailed feedback from medical
student end users regarding their experience of engaging in anapp-delivered psychological intervention. Although specificfindings may not be generalizable beyond this sample and app,our observations and evaluation process have potential relevanceto researchers in the emerging field of app-delivered
psychological interventions for medical student well-being. We
demonstrated that making an intervention accessible to medical
students via an app does not guarantee that they will access it,even when their self-reported intention is to do so, and thatformal evaluation of user experiences across a range of domains
may facilitate the identification of contributing factors. Ourfindings suggest that it may be useful for intervention developers
to assess medical students’ preferences regarding thefunctionality and usability aspects of app interfaces as well as
individual differences in learning goals, baseline skills, andconfidence. Failure to understand user experiences could resultin disengagement for reasons unrelated to the intervention
model, leading to missed opportunities to fully develop anddeploy potentially valuable psychological skill training [1]. We
recommend that future researchers incorporate formal evaluationof medical students’ subjective user experiences into theintervention development process (including quantitative andqualitative assessments) to better understand and address factorsthat might affect engagement and disengagement.

Acknowledgments
This research project was funded by the Centre for Rehab Innovations and the Priority Research Centre for Stroke and Brain
Injury at the University of Newcastle. The authors wish to acknowledge the Joint Medical Program (University of Newcastle andUniversity of New England) for supporting this research and the medical students who participated in the trial, the Cogniss teamfor their commitment to fine-tuning the app throughout the development process, and Lucy Bailey for providing instrumental
support in the administrative aspects of the study.
https://formative.jmir.org/2023/1/e43263

XSL• FO
RenderX

JMIR Form Res 2023 | vol. 7 | e43263 | p. 15(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Ditton et al

Data Availability
The data sets generated and analyzed during this study are not publicly available for the protection of participant privacy, but
deidentified outcome data sets are available from the corresponding author upon reasonable request.

Authors' Contributions
ED wrote the manuscript, designed the study protocol, and created the app. ED and BK wrote and compiled the intervention
content. NH provided consultation on the study protocol design, data analyses, and app usability. GH led stakeholder engagement
and provided consultation on the translation of various aspects of the protocol and intervention to the cohort context. MN and
RW oversaw the project. All authors reviewed and contributed to the manuscript.

Conflicts of Interest
None declared.

Multimedia Appendix 1
Stage 1 screenshot.
[PNG File , 395 KB-Multimedia Appendix 1]

Multimedia Appendix 2
App home screen.
[PNG File , 371 KB-Multimedia Appendix 2]

Multimedia Appendix 3
App screening questions.
[PNG File , 433 KB-Multimedia Appendix 3]

Multimedia Appendix 4
Stage 2 skill set dashboard.
[PNG File , 119 KB-Multimedia Appendix 4]

Multimedia Appendix 5
Acceptability assessment and mean participant data at t1 (n=11).
[DOCX File , 24 KB-Multimedia Appendix 5]

Multimedia Appendix 6
Learning experiences assessment.
[DOCX File , 15 KB-Multimedia Appendix 6]

References
1.

2.

3.

4.
5.
6.

Yogeswaran V, El Morr C. Effectiveness of online mindfulness interventions on medical students' mental health: a systematic
review. BMC Public Health 2021 Dec 18;21(1):2293 [FREE Full text] [doi: 10.1186/s12889-021-12341-z] [Medline:
34920715]
Rotenstein LS, Ramos MA, Torre M, Segal JB, Peluso MJ, Guille C, et al. Prevalence of depression, depressive symptoms,
and suicidal ideation among medical students: a systematic review and meta-analysis. JAMA 2016 Dec 06;316(21):2214-2236[FREE Full text] [doi: 10.1001/jama.2016.17324] [Medline: 27923088]
Seo C, Corrado M, Fournier K, Bailey T, Haykal KA. Addressing the physician burnout epidemic with resilience curricula
in medical education: a systematic review. BMC Med Educ 2021 Feb 01;21(1):80 [FREE Full text] [doi:10.1186/s12909-021-02495-0] [Medline: 33526019]
Frajerman A, Morvan Y, Krebs MO, Gorwood P, Chaumette B. Burnout in medical students before residency: a systematic
review and meta-analysis. Eur Psychiatry 2019 Jan;55:36-42. [doi: 10.1016/j.eurpsy.2018.08.006] [Medline: 30384110]
Bugaj T, Cranz A, Junne F, Erschens R, Herzog W, Nikendei C. Psychosocial burden in medical students and specificprevention strategies. Ment Health Prevent 2016 Mar;4(1):24-30. [doi: 10.1016/j.mhp.2015.12.003]
Fares J, Al Tabosh H, Saadeddin Z, El Mouhayyar C, Aridi H. Stress, burnout and coping strategies in preclinical medical
students. N Am J Med Sci 2016 Feb;8(2):75-81 [FREE Full text] [doi: 10.4103/1947-2714.177299] [Medline: 27042604]

https://formative.jmir.org/2023/1/e43263

XSL• FO
RenderX

JMIR Form Res 2023 | vol. 7 | e43263 | p. 16(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
7.
8.

9.

10.

11.

12.
13.

14.
15.
16.

17.

18.
19.
20.

21.
22.
23.
24.

25.
26.

27.

28.
29.

Maslach C, Leiter MP. Understanding the burnout experience: recent research and its implications for psychiatry. World
Psychiatry 2016 Jun;15(2):103-111 [FREE Full text] [doi: 10.1002/wps.20311] [Medline: 27265691]
Shanafelt TD, Boone S, Tan L, Dyrbye LN, Sotile W, Satele D, et al. Burnout and satisfaction with work-life balance among
US physicians relative to the general US population. Arch Intern Med 2012 Oct 08;172(18):1377-1385. [doi:10.1001/archinternmed.2012.3199] [Medline: 22911330]
Shanafelt TD, Mungo M, Schmitgen J, Storz KA, Reeves D, Hayes SN, et al. Longitudinal study evaluating the association
between physician burnout and changes in professional work effort. Mayo Clin Proc 2016 Apr;91(4):422-431. [doi:10.1016/j.mayocp.2016.02.001] [Medline: 27046522]
Kemp S, Hu W, Bishop J, Forrest K, Hudson JN, Wilson I, et al. Medical student wellbeing - a consensus statement from
Australia and New Zealand. BMC Med Educ 2019 Mar 04;19(1):69 [FREE Full text] [doi: 10.1186/s12909-019-1505-2][Medline: 30832630]
Walsh AL, Lehmann S, Zabinski J, Truskey M, Purvis T, Gould NF, et al. Interventions to prevent and reduce burnout
among undergraduate and graduate medical education trainees: a systematic review. Acad Psychiatry 2019 Aug;43(4):386-395.
[doi: 10.1007/s40596-019-01023-z] [Medline: 30710229]
Harvey SB, Epstein RM, Glozier N, Petrie K, Strudwick J, Gayed A, et al. Mental illness and suicide among physicians.
Lancet 2021 Sep 04;398(10303):920-930 [FREE Full text] [doi: 10.1016/S0140-6736(21)01596-8] [Medline: 34481571]
Dyrbye LN, Satele D, West CP. Association of characteristics of the learning environment and US medical student burnout,
empathy, and career regret. JAMA Netw Open 2021 Aug 02;4(8):e2119110 [FREE Full text] [doi:10.1001/jamanetworkopen.2021.19110] [Medline: 34369990]
Amanullah S, Ramesh Shankar R. The impact of COVID-19 on physician burnout globally: a review. Healthcare (Basel)
2020 Oct 22;8(4):421 [FREE Full text] [doi: 10.3390/healthcare8040421] [Medline: 33105757]
Ishak W, Nikravesh R, Lederer S, Perry R, Ogunyemi D, Bernstein C. Burnout in medical students: a systematic review.
Clin Teach 2013 Aug;10(4):242-245. [doi: 10.1111/tct.12014] [Medline: 23834570]
Arif NM, Roslan NS, Ismail SB, Nayak RD, Jamian MR, Mohamad Ali Roshidi AS, et al. Prevalence and associated factorsof psychological distress and burnout among medical students: findings from two campuses. Int J Environ Res Public Health2021 Aug 10;18(16):8446 [FREE Full text] [doi: 10.3390/ijerph18168446] [Medline: 34444193]
Wersebe H, Lieb R, Meyer AH, Hofer P, Gloster AT. The link between stress, well-being, and psychological flexibility
during an Acceptance and Commitment Therapy self-help intervention. Int J Clin Health Psychol 2018;18(1):60-68 [FREEFull text] [doi: 10.1016/j.ijchp.2017.09.002] [Medline: 30487911]
Puolakanaho A, Tolvanen A, Kinnunen SM, Lappalainen R. A psychological flexibility -based intervention for burnout:arandomized controlled trial. J Contextual Behav Sci 2020 Jan;15:52-67. [doi: 10.1016/j.jcbs.2019.11.007]
Keyes CL. Mental illness and/or mental health? Investigating axioms of the complete state model of health. J Consult Clin
Psychol 2005 Jun;73(3):539-548 [FREE Full text] [doi: 10.1037/0022-006X.73.3.539] [Medline: 15982151]
Lamers SM, Westerhof GJ, Bohlmeijer ET, ten Klooster PM, Keyes CL. Evaluating the psychometric properties of the
Mental Health Continuum-Short Form (MHC-SF). J Clin Psychol 2011 Jan;67(1):99-110. [doi: 10.1002/jclp.20741][Medline: 20973032]
Lenoir AL, De Troyer C, Demoulin C, Gillain I, Bayot M. Challenges in treating physician burnout: the psychologist's
perspective. Presse Méd 2021 Jun;2:100006. [doi: 10.1016/j.lpmope.2021.100006]
Taylor M, Hageman JR, Brown M. A mindfulness intervention for residents: relevance for pediatricians. Pediatr Ann 2016
Oct 01;45(10):e373-e376. [doi: 10.3928/19382359-20160912-01] [Medline: 27735974]
Wen L, Sweeney TE, Welton L, Trockel M, Katznelson L. Encouraging mindfulness in medical house staff via smartphone
app: a pilot study. Acad Psychiatry 2017 Oct;41(5):646-650. [doi: 10.1007/s40596-017-0768-3] [Medline: 28795335]
Awasthi S, Kaur A, Solanki H, Pamei G, Bhatt M. Smartphone use and the quality of life of medical students in the Kumaun
Region, Uttarakhand. J Family Med Prim Care 2020 Aug;9(8):4252-4258 [FREE Full text] [doi:10.4103/jfmpc.jfmpc_805_20] [Medline: 33110841]
Singh K, Sarkar S, Gaur U, Gupta S, Adams OP, Sa B, et al. Smartphones and educational apps use among medical students
of a smart university campus. Front Commun 2021 Nov 2;6:49102. [doi: 10.3389/fcomm.2021.649102]
Patel S, Akhtar A, Malins S, Wright N, Rowley E, Young E, et al. The acceptability and usability of digital health interventions
for adults with depression, anxiety, and somatoform disorders: qualitative systematic review and meta-synthesis. J MedInternet Res 2020 Jul 06;22(7):e16228 [FREE Full text] [doi: 10.2196/16228] [Medline: 32628116]
Levin ME, Haeger J, Cruz RA. Tailoring acceptance and commitment therapy skill coaching in the moment throughsmartphones: results from a randomized controlled trial. Mindfulness 2018 Jul 21;10(4):689-699. [doi:10.1007/s12671-018-1004-2]
Amanullah S, McNally K, Zelin J, Cole J, Cernovsky Z. Are burnout prevention programs for hospital physicians needed?
Asian J Psychiatr 2017 Apr;26:66-69. [doi: 10.1016/j.ajp.2017.01.009] [Medline: 28483095]
Kalani SD, Azadfallah P, Oreyzi H, Adibi P. Interventions for physician burnout: a systematic review of systematic reviews.
Int J Prev Med 2018 Sep 17;9:81 [FREE Full text] [doi: 10.4103/ijpvm.IJPVM_255_18] [Medline: 30283613]

https://formative.jmir.org/2023/1/e43263

XSL• FO
RenderX

Ditton et al

JMIR Form Res 2023 | vol. 7 | e43263 | p. 17(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
30.

31.

32.

33.
34.
35.

36.
37.

38.

39.

40.
41.
42.

43.
44.

45.
46.

47.
48.
49.

50.

51.

Baumel A, Muench F, Edan S, Kane JM. Objective user engagement with mental health apps: systematic search andpanel-based usage analysis. J Med Internet Res 2019 Sep 25;21(9):e14567 [FREE Full text] [doi: 10.2196/14567] [Medline:
31573916]
Sekhon M, Cartwright M, Francis JJ. Acceptability of healthcare interventions: an overview of reviews and developmentof a theoretical framework. BMC Health Serv Res 2017 Jan 26;17(1):88 [FREE Full text] [doi: 10.1186/s12913-017-2031-8][Medline: 28126032]
Blatch-Jones AJ, Pek W, Kirkpatrick E, Ashton-Key M. Role of feasibility and pilot studies in randomised controlled trials:
a cross-sectional study. BMJ Open 2018 Sep 25;8(9):e022233 [FREE Full text] [doi: 10.1136/bmjopen-2018-022233][Medline: 30257847]
Yogeswaran V, El Morr C. Mental Health for Medical Students, what do we know today? Proc Comput Sci 2022;198:307-310[FREE Full text] [doi: 10.1016/j.procs.2021.12.245]
Thabane L, Ma J, Chu R, Cheng J, Ismaila A, Rios LP, et al. A tutorial on pilot studies: the what, why and how. BMC Med
Res Methodol 2010 Jan 06;10:1 [FREE Full text] [doi: 10.1186/1471-2288-10-1] [Medline: 20053272]
Bell ML, Whitehead AL, Julious SA. Guidance for using pilot studies to inform the design of intervention trials with
continuous outcomes. Clin Epidemiol 2018 Jan 18;10:153-157 [FREE Full text] [doi: 10.2147/CLEP.S146397] [Medline:
29403314]
Onwezen MC, van Veldhoven MJ, Biron M. The role of psychological flexibility in the demands–exhaustion–performance
relationship. Eur J Work Organ Psychol 2014;23(2):163-176. [doi: 10.1080/1359432x.2012.742242]
Lloyd J, Bond FW, Flaxman PE. The value of psychological flexibility: examining psychological mechanisms underpinninga cognitive behavioural therapy intervention for burnout. Work Stress 2013 Apr;27(2):181-199. [doi:10.1080/02678373.2013.782157]
Vilardaga R, Luoma JB, Hayes SC, Pistorello J, Levin ME, Hildebrandt MJ, et al. Burnout among the addiction counselingworkforce: the differential roles of mindfulness and values-based processes and work-site factors. J Subst Abuse Treat
2011 Jun;40(4):323-335 [FREE Full text] [doi: 10.1016/j.jsat.2010.11.015] [Medline: 21257281]
Veage S, Ciarrochi J, Deane FP, Andresen R, Oades LG, Crowe TP. Value congruence, importance and success and in theworkplace: links with well-being and burnout amongst mental health practitioners. J Contextual Behav Sci 2014
Oct;3(4):258-264. [doi: 10.1016/j.jcbs.2014.06.004]
Bond FW, Bunce D. The role of acceptance and job control in mental health, job satisfaction, and work performance. J
Appl Psychol 2003 Dec;88(6):1057-1067. [doi: 10.1037/0021-9010.88.6.1057] [Medline: 14640816]
Gloster AT, Meyer AH, Lieb R. Psychological flexibility as a malleable public health target: evidence from a representative
sample. J Contextual Behav Sci 2017 Apr;6(2):166-171. [doi: 10.1016/j.jcbs.2017.02.003]
Kinnunen SM, Puolakanaho A, Mäkikangas A, Tolvanen A, Lappalainen R. Does a mindfulness-, acceptance-, andvalue-based intervention for burnout have long-term effects on different levels of subjective well-being? Int J Stress Manag
2020 Feb;27(1):82-87. [doi: 10.1037/str0000132]
Hayes SC, Strosahl KD, Wilson KG. Acceptance and Commitment Therapy: The Process and Practice of Mindful Change.
2nd edition. New York, NY, USA: The Guilford Press; 2016.
Stabbe OK, Rolffs JL, Rogge RD. Flexibly and/or inflexibly embracing life: identifying fundamental approaches to life
with latent profile analyses on the dimensions of the Hexaflex model. J Contextual Behav Sci 2019 Apr;12:106-118. [doi:10.1016/j.jcbs.2019.03.003]
Moran DJ. ACT for leadership: using acceptance and commitment training to develop crisis-resilient change managers. Int
J Behav Consult Ther 2010;6(4):341-355. [doi: 10.1037/h0100915]
Ramaci T, Bellini D, Presti G, Santisi G. Psychological flexibility and mindfulness as predictors of individual outcomesin hospital health workers. Front Psychol 2019 Jun 12;10:1302 [FREE Full text] [doi: 10.3389/fpsyg.2019.01302] [Medline:
31249541]
Thompson G, McBride RB, Hosford CC, Halaas G. Resilience among medical students: the role of coping style and social
support. Teach Learn Med 2016;28(2):174-182. [doi: 10.1080/10401334.2016.1146611] [Medline: 27064719]
Kroska EB, Calarge C, O’Hara MW, Deumic E, Dindo L. Burnout and depression in medical students: relations withavoidance and disengagement. J Contextual Behav Sci 2017 Oct;6(4):404-408. [doi: 10.1016/j.jcbs.2017.08.003]
Johns G, Waddington L, Samuel V. Prevalence and predictors of mental health outcomes in UK doctors and final year
medical students during the COVID-19 pandemic. J Affect Disord 2022 Aug 15;311:267-275 [FREE Full text] [doi:10.1016/j.jad.2022.05.024] [Medline: 35569608]
Palladino CL, Ange B, Richardson DS, Casillas R, Decker M, Gillies RA, et al. Measuring psychological flexibility inmedical students and residents: a psychometric analysis. Med Educ Online 2013 Aug 13;18:20932 [FREE Full text] [doi:10.3402/meo.v18i0.20932] [Medline: 23948496]
Szarko AJ, Houmanfar RA, Smith GS, Jacobs NN, Smith BM, Assemi K, et al. Impact of Acceptance and Commitment
Training on psychological flexibility and burnout in medical education. J Contextual Behav Sci 2022 Jan;23:190-199. [doi:10.1016/j.jcbs.2022.02.004]

https://formative.jmir.org/2023/1/e43263

XSL• FO
RenderX

Ditton et al

JMIR Form Res 2023 | vol. 7 | e43263 | p. 18(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
52.

53.

54.

55.

56.

57.

58.
59.

60.
61.
62.
63.

64.

65.

66.
67.

68.
69.
70.

71.
72.
73.

Shahsavari Googhari Z, Hafezi F, Asgari P, Heidari A. The effectiveness of mindfulness-based cognitive therapy andacceptance and commitment therapy on medical science students’ subjective well-being, psychological distress, and emotion
regulation. J Shahrekord Univ Med Sci 2022 Jan 26;24(1):35-41. [doi: 10.34172/jsums.2022.07]
Villatte JL, Vilardaga R, Villatte M, Plumb Vilardaga JC, Atkins DC, Hayes SC. Acceptance and Commitment Therapy
modules: differential impact on treatment processes and outcomes. Behav Res Ther 2016 Feb;77:52-61 [FREE Full text][doi: 10.1016/j.brat.2015.12.001] [Medline: 26716932]
Rolffs JL, Rogge RD, Wilson KG. Disentangling components of flexibility via the hexaflex model: development andvalidation of the multidimensional psychological flexibility inventory (MPFI). Assessment 2018 Jun;25(4):458-482. [doi:10.1177/1073191116645905] [Medline: 27152011]
Rogge RD, Daks JS, Dubler BA, Saint KJ. It's all about the process: examining the convergent validity, conceptual coverage,
unique predictive validity, and clinical utility of ACT process measures. J Contextual Behav Sci 2019 Oct;14:90-102. [doi:10.1016/j.jcbs.2019.10.001]
Reeve A, Tickle A, Moghaddam N. Are acceptance and commitment therapy-based interventions effective for reducing
burnout in direct-care staff? A systematic review and meta-analysis. Ment Health Rev J 2018 Jul 04;23(3):131-155. [doi:10.1108/mhrj-11-2017-0052]
Ditton E, Knott B, Hodyl N, Horton G, Walker FR, Nilsson M. Assessing the efficacy of an individualized psychologicalflexibility skills training intervention app for medical student burnout and well-being: protocol for a randomized controlled
trial. JMIR Res Protoc 2022 Feb 04;11(2):e32992 [FREE Full text] [doi: 10.2196/32992] [Medline: 35119378]
Larmar S, Wiatrowski S, Lewis-Driver S. Acceptance and Commitment Therapy: an overview of techniques and applications.
J Serv Sci Manag 2014 Jun;07(03):216-221. [doi: 10.4236/jssm.2014.73019]
Viskovich S, Pakenham KI. Pilot evaluation of a web-based acceptance and commitment therapy program to promotemental health skills in university students. J Clin Psychol 2018 Dec;74(12):2047-2069. [doi: 10.1002/jclp.22656] [Medline:
29962090]
Firestone J, Cardaciotto L, Levin ME, Goldbacher E, Vernig P, Gambrel LE. A web-based self-guided program to promotevalued-living in college students: a pilot study. J Contextual Behav Sci 2019 Apr;12:29-38. [doi: 10.1016/j.jcbs.2019.01.004]
Schaufeli WB, Martínez IM, Pinto AM, Salanova M, Bakker AB. Burnout and engagement in university students: across-national study. J Cross Cult Psychol 2002 Sep;33(5):464-481. [doi: 10.1177/0022022102033005003]
Maslach C, Jackson SE, Leiter MP. Maslach Burnout Inventory Manual. 3rd edition. Mountain View, CA, USA: CPP
Incorporated; 1996.
Obregon M, Luo J, Shelton J, Blevins T, MacDowell M. Assessment of burnout in medical students using the Maslach
Burnout Inventory-Student Survey: a cross-sectional data analysis. BMC Med Educ 2020 Oct 21;20(1):376 [FREE Fulltext] [doi: 10.1186/s12909-020-02274-3] [Medline: 33087080]
Keyes CL. Brief description of the mental health continuum short form (MHC-SF). The University of North Carolina.
Chapel Hill, NC, USA: The University of North Carolina; 2009. URL: https://peplab.web.unc.edu/wp-content/uploads/
sites/18901/2018/11/MHC-SFoverview.pdf [accessed 2021-01-30]
Seidler D, Stone B, Clark BE, Koran J, Drake CE. Evaluating the factor structure of the Multidimensional Psychological
Flexibility Inventory: an independent replication and extension. J Contextual Behav Sci 2020 Jul;17:23-31. [doi:10.1016/j.jcbs.2020.04.007]
Lovibond SH, Lovibond PF. Manual for the Depression Anxiety Stress Scales. 2nd edition. Sydney, Australia: Psychology
Foundation of Australia; 1995.
Henry JD, Crawford JR. The short-form version of the Depression Anxiety Stress Scales (DASS-21): construct validity
and normative data in a large non-clinical sample. Br J Clin Psychol 2005 Jun;44(Pt 2):227-239. [doi:10.1348/014466505X29657] [Medline: 16004657]
Bangor A, Kortum P, Miller J. Determining what individual SUS scores mean: adding an adjective rating scale. J Usability
Stud 2009 May;4(3):114-123.
Bangor A, Kortum PT, Miller JT. An empirical evaluation of the system usability scale. Int J Human Comput Interact 2008
Jul 30;24(6):574-594. [doi: 10.1080/10447310802205776]
Kinnunen SM, Puolakanaho A, Tolvanen A, Mäkikangas A, Lappalainen R. Does mindfulness-, acceptance-, and value-based
intervention alleviate burnout?—A person-centered approach. Int J Stress Manag 2019 Feb;26(1):89-101. [doi:10.1037/str0000095]
Lewis JR. The system usability scale: past, present, and future. Int J Human Comput Interact 2018 Mar 30;34(7):577-590.
[doi: 10.1080/10447318.2018.1455307]
Brooke J. SUS: a 'quick and dirty' usability scale. In: Jordan PW, Thomas B, McClelland L, Weerdmeester B, editors.
Usability Evaluation in Industry. London, UK: CRC Press; 1996:189-194.
Perski O, Blandford A, West R, Michie S. Conceptualising engagement with digital behaviour change interventions: asystematic review using principles from critical interpretive synthesis. Transl Behav Med 2017 Jun;7(2):254-267 [FREEFull text] [doi: 10.1007/s13142-016-0453-1] [Medline: 27966189]

https://formative.jmir.org/2023/1/e43263

XSL• FO
RenderX

Ditton et al

JMIR Form Res 2023 | vol. 7 | e43263 | p. 19(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
74.

75.
76.
77.

78.

79.
80.
81.

82.

Ditton et al

Prochaska JJ, Vogel EA, Chieng A, Kendra M, Baiocchi M, Pajarito S, et al. A therapeutic relational agent for reducing
problematic substance use (Woebot): development and usability study. J Med Internet Res 2021 Mar 23;23(3):e24850[FREE Full text] [doi: 10.2196/24850] [Medline: 33755028]
The Jamovi Project. Jamovi (Version 2.3). Jamovi. 2022. URL: https://www.jamovi.org/ [accessed 2022-12-08]
Michalec B, Keyes CL. A multidimensional perspective of the mental health of preclinical medical students. Psychol HealthMed 2013;18(1):89-97. [doi: 10.1080/13548506.2012.687825] [Medline: 22676432]
Grégoire S, Gagnon J, Lachance L, Shankland R, Dionne F, Kotsou I, et al. Validation of the english and french versionsof the multidimensional psychological flexibility inventory short form (MPFI-24). J Contextual Behav Sci 2020
Oct;18:99-110. [doi: 10.1016/j.jcbs.2020.06.004]
Wang J, Song B, Shao Y, Zhu J. Effect of online psychological intervention on burnout in medical residents from different
majors: an exploratory study. Front Psychol 2021 May 7;12:632134 [FREE Full text] [doi: 10.3389/fpsyg.2021.632134][Medline: 34025506]
Dindo L. One-day acceptance and commitment training workshops in medical populations. Curr Opin Psychol 2015 Apr
01;2:38-42 [FREE Full text] [doi: 10.1016/j.copsyc.2015.01.018] [Medline: 25793217]
Kroska EB, Roche AI, O'Hara MW. How much is enough in brief Acceptance and Commitment Therapy? A randomized
trial. J Contextual Behav Sci 2020 Jan;15:235-244. [doi: 10.1016/j.jcbs.2020.01.009]
Towey-Swift KD, Lauvrud C, Whittington R. Acceptance and commitment therapy (ACT) for professional staff burnout:a systematic review and narrative synthesis of controlled trials. J Ment Health (forthcoming) 2022 Feb 02:1-13. [doi:10.1080/09638237.2021.2022628] [Medline: 35105253]
Pakenham KI, Landi G, Boccolini G, Furlani A, Grandi S, Tossani E. The moderating roles of psychological flexibility
and inflexibility on the mental health impacts of COVID-19 pandemic and lockdown in Italy. J Contextual Behav Sci 2020
Jul;17:109-118 [FREE Full text] [doi: 10.1016/j.jcbs.2020.07.003] [Medline: 32834969]

Abbreviations
ACTraining: Acceptance and Commitment Training
REDCap: Research Electronic Data Capture
SUS: System Usability Scale
TFA: theoretical framework of acceptability

Edited by A Mavragani; submitted 06.10.22; peer-reviewed by C Jacob, L Grepo; comments to author 07.11.22; revised version
received 22.11.22; accepted 28.11.22; published 10.01.23
Please cite as:
Ditton E, Knott B, Hodyl N, Horton G, Walker FR, Nilsson M
Medical Student Experiences of Engaging in a Psychological Flexibility Skill Training App for Burnout and Well-being: Pilot Feasibility
Study
JMIR Form Res 2023;7:e43263
URL: https://formative.jmir.org/2023/1/e43263
doi: 10.2196/43263
PMID:©Elizabeth Ditton, Brendon Knott, Nicolette Hodyl, Graeme Horton, Frederick Rohan Walker, Michael Nilsson. Originallypublished in JMIR Formative Research (https://formative.jmir.org), 10.01.2023. This is an open-access article distributed under
the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted
use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, isproperly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as wellas this copyright and license information must be included.

https://formative.jmir.org/2023/1/e43263

XSL• FO
RenderX

JMIR Form Res 2023 | vol. 7 | e43263 | p. 20(page number not for citation purposes)