JMIR MEDICAL INFORMATICS

Ologeanu-Taddei

Viewpoint

Assessment of mHealth Interventions: Need for New Studies,
Methods, and Guidelines for Study Designs
Roxana Ologeanu-Taddei, PhD
TBS Business School, Toulouse, France

Corresponding Author:
Roxana Ologeanu-Taddei, PhD
TBS Business School
1, place Alphonse Jourdain
Toulouse, 31068
FrancePhone: 33 5 61 29 48 51
Email: r.ologeanu-taddei@tbs-education.fr

Abstract
This viewpoint argues that the clinical effects of mobile health (mHealth) interventions depends on the acceptance and adoptionof these interventions and their mediators, such as usability of the mHealth software, software performance and features, training
and motivation of patients and health care professionals to participate in the experience, or characteristics of the intervention (eg,personalized feedback).
(JMIR Med Inform 2020;8(11):e21874) doi: 10.2196/21874
KEYWORDS
eHealth; mHealth; usability; management; survey; trust; guidelines; evaluation

BackgroundIn the past years, designs of research studies in medical journals
have been formalized according to the reporting guidelines ofacademic associations, international consortia, and publishers,
which enable publications of clinical trials, systematic reviews,and meta-analyses. The use of mobile health (mHealth)technologies is expected to increase, creating new paths forhealth care delivery. However, there are no specific guidelines
to enable researchers to design and present their studies andresults on this topic, except for the existing guidelines on
randomized clinical trials (RCTs), which can be used for certain
assessments of mHealth interventions.

Measures of Clinical Effects Require
Opening the Black Box of Information
Technology
There is no doubt that RCTs are useful to assess the clinicaloutcomes and effectiveness of mHealth. In this context,
regulatory bodies such as the US Food and Drug Administration(FDA) [1] and European regulations have recently updated therequirements for clinical proofs of mHealth solutions.
Nevertheless, the focus on RCT methods leads to the blackboxing of mHealth interventions [2], which means that the
http://medinform.jmir.org/2020/11/e21874/

XSL• FO
RenderX

technology is considered as a homogeneous device or as apharmaceutical substance. This view misses the main
characteristic of an mHealth intervention (and overall, that ofinformation technology [IT])—the embeddedness of data andclinical processes (as reflected in the guidelines for diagnosis
and personalized monitoring). Therefore, assessing the clinicaleffect of an mHealth intervention should disentangle the effectof a new process of personalized monitoring, the effect of theubiquitous access enabled by mobile devices, and thecomprehension and adoption of clinical guidelines implemented
into the application. In addition, mHealth solutions may differ
from one another because of their different designs of theseprocesses.

Need of Standard Guidelines to Assess
Technology and Mediators of OutcomesMoreover, the clinical effect of mHealth solutions depends on
the acceptance and adoption of these solutions and theirmediators, such as usability of the mHealth software, software
performance and features, training and motivation of patientsand health care professionals to participate in the experience,or characteristics of the intervention (eg, personalized feedback).
For example, a clinical effect such as survival benefits forpatients with cancer who use a surveillance mHealth app
depends on the acceptance and adoption of the app, which canJMIR Med Inform 2020 | vol. 8 | iss. 11 | e21874 | p. 1
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
be influenced by the usability of the app and patients’ priorexperience in using mobile apps, motivation or trust in IT, orother alternative mediators contributing to the main reported
outcome; this influences are often neglected by RCTs.
Settings of mHealth interventions must be carefully describedand assessed in hypothesis-generating studies [3], such as
observational studies and case reports. These studies can identify
specific moderators and mediators that state for whom and underwhat conditions the health intervention works [3]. Moderators
may identify population groups with possible causal mechanisms
or courses of illness. The mHealth mediators identify possible
causal mechanisms, meaning causal links between theintervention and the outcome, through which the intervention
may achieve its effects [3]. As a next step, these moderatorsand mediators should be considered as stratification variables
in forthcoming RCTs focused on hypothesis testing. Otherwise,RCTs are likely to be based on weak assumptions rather thanempirical evidence.

Beyond Effectiveness: Risk Assessment
In addition, RCTs need to be complemented by other clinical
trials and case reports to assess safety risks [4] and unintended
consequences of mHealth. The acknowledgment of these risks
is at the core of the updated regulations for medical devices,which include software and mHealth. However, in most cases,
assessments of mHealth safety risks are conducted separately[5], for example, in feasibility or usability studies, which use
different methods of varying rigor and do not generate
cumulative knowledge. In addition, case reports on the adoption
stages of mHealth solutions should be inspired by engineering
methods (eg, fault tree analysis rather than pharmacovigilance
studies). Moreover, relevant and cumulative knowledge can begathered if publications on the issues of usability and user
acceptance are presented not only in health informatics journals
but also in major medical journals, as these issues cause clinicaleffects. For example, the Journal of the American Medical
Association (JAMA) Network advises authors to use theguidelines of the EQUATOR (Enhancing the QUAlity andTransparency Of health Research) network [6], which include
guidelines related to “economic studies.” Similar initiatives
should be undertaken for studies concerning mHealth. In recentyears, cumulative knowledge has been gathered on the risks
associated with poor usability of health IT [7]. In line with thisliterature, a step forward was taken only for mHealth solutions,which are qualified as medical devices [8]. However, even for
those applications, national and international regulations (ie,CE marking in Europe or FDA regulations) and harmonized
standards (ie, EN 62366 advised for CE marking in Europe)
strengthened the requirements for premarket certifications but
did not standardize a threshold for usability or technicalperformance. We must recognize that recommending theminimum required sample size (eg, 15 users identified by user
profile numbers) makes an improvement to the summative
usability assessment method [9]. It is also necessary to assessuser profiles accurately. These user profiles may be related tohealth care occupations (eg, clinical secretaries, physicians, ornurses), different health departments (eg, infectious diseases orpediatrics), and social or demographic variables. In addition,
http://medinform.jmir.org/2020/11/e21874/XSL• FORenderX

Ologeanu-Taddei
several other characteristics (eg, computer literacy, priorexperience of using mobile apps, and users’ engagement or trustin IT) may mediate or moderate mHealth effects and therefore
could be taken into account to refine user profiles. Further
research is needed on these moderators and mediators to usethem as criteria for construction of user profiles. Although suchresearch can be complex and costly, it is relevant and useful.

Need of Studies in Implementation andAdoption Stages
The implementation of mHealth solutions (beyond pilots) inthe market and their user adoption stages introduce newcontextual and technological factors, such as low technicalperformance, lack of interoperability with existing systems, ormisfit with existing clinical practices. Pilot studies often benefit
from specific resources—both financial and human—and fromhigh motivation of the patients and health care professionals
involved. These factors may be missing in the latter stages ofimplementation and adoption, influencing the outcome
achievement or introducing risks to patient safety. Althoughthese challenges cannot be experimentally controlled, they may
be assessed cautiously in rigorous qualitative and statisticalstudies. In addition, the moderators and mediators of mHealthinterventions, such as engagement levels [10], should beinvestigated more systematically.
We have to mention that the EQUATOR network publishedguidelines for the reporting of mobile phone–based healthinterventions [11]. Formed by the World Health Organization’s
panel of experts, these guidelines are useful to improve thetransparency and harmonization of the reporting of mHealth,enabling comparisons and meta-reviews. These criteria include
usability/content testing and user feedback. Nevertheless, a new
step must be taken toward formulating guidelines on study
designs and methods of the assessment of user feedback on
mHealth interventions.

Moving Forward: Formal Guidelines forStudy Designs on Real-World Usage
The evaluation of moderators and mediators in pilots should be
followed by larger surveys and follow-ups during the wholelife cycle of the mHealth technology [5]. The protocols of thesestudies should be inspired by the rigor of protocols of clinicalinvestigations while considering the relevant factors specific tomHealth. Open trials, observational studies, and case reports
should be conducted to measure mediators beyond a specificclinical setting (the variables and low sample size of which
could introduce serious bias). In addition, anecdotal reports andqualitative studies should use common frameworks, which will
facilitate systematic reviews and afford transferability. The
knowledge generated can thus inform policy decisions.
Moreover, anecdotal reports of suspected adverse reactions
related to the use of mHealth (along with reports of health
IT–related adverse events) should be encouraged and publishedin medical journals, as mHealth can induce specific errors
related to the use of technology. These issues have beenemphasized as crucial for more than 20 years, during which
JMIR Med Inform 2020 | vol. 8 | iss. 11 | e21874 | p. 2
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
numerous studies have shown that bad informatics can havefatal consequences [5]. If the new European regulations onmedical devices (which include mHealth solutions) requirereal-life, postmarket, clinical, and risk assessment follow-upsof these devices, new methods and frameworks should be
elaborated with scientific rigor and inspired by different

Ologeanu-Taddei
academic disciplines, such as engineering and social sciences.
Building on the guidelines for research presentations andknowledge from medical informatics and risk engineering [12],it is time to make rigorous evaluations and formalize guidelines
for research presentations, enabling evidence-based mHealthinterventions.

Conflicts of Interest
None declared.

References
1.

2.

3.
4.
5.
6.
7.
8.
9.
10.

11.

12.

US Food and Drug Administration. Policy for Device Software Functions and Mobile Medical Applications: Guidance forIndustry and Food and Drug Administration staff. 2019. URL: https://www.fda.gov/media/80958/download [accessed2020-08-15]
Free C, Phillips G, Watson L, Galli L, Felix L, Edwards P, et al. The effectiveness of mobile-health technologies to improvehealth care service delivery processes: a systematic review and meta-analysis. PLoS Med 2013;10(1):e1001363 [FREEFull text] [doi: 10.1371/journal.pmed.1001363] [Medline: 23458994]
Kraemer HC, Wilson GT, Fairburn CG, Agras WS. Mediators and moderators of treatment effects in randomized clinical
trials. Arch Gen Psychiatry 2002 Oct;59(10):877-883. [doi: 10.1001/archpsyc.59.10.877] [Medline: 12365874]
Lewis TL, Wyatt JC. mHealth and mobile medical Apps: a framework to assess risk and promote safer use. J Med Internet
Res 2014 Sep 15;16(9):e210 [FREE Full text] [doi: 10.2196/jmir.3133] [Medline: 25223398]
Ammenwerth E, Shaw NT. Bad health informatics can kill--is evaluation the answer? Methods Inf Med 2005;44(1):1-3.
[Medline: 15778787]
EQUATOR Network. URL: https://www.equator-network.org/ [accessed 2020-08-15]
Zapata BC, Fernández-Alemán JL, Idri A, Toval A. Empirical studies on usability of mHealth apps: a systematic literature
review. J Med Syst 2015 Feb;39(2):1. [doi: 10.1007/s10916-014-0182-2] [Medline: 25600193]
Keutzer L, Simonsson US. Medical Device Apps: An Introduction to Regulatory Affairs for Developers. JMIR Mhealth
Uhealth 2020 Jun 26;8(6):e17567 [FREE Full text] [doi: 10.2196/17567] [Medline: 32589154]
Borsci S, Londei A, Federici S. The Bootstrap Discovery Behaviour (BDB): a new outlook on usability evaluation. Cogn
Process 2011 Feb;12(1):23-31. [doi: 10.1007/s10339-010-0376-6] [Medline: 21046191]
Yang Q, Van Stee SK. The Comparative Effectiveness of Mobile Phone Interventions in Improving Health Outcomes:
Meta-Analytic Review. JMIR Mhealth Uhealth 2019 Apr 03;7(4):e11244 [FREE Full text] [doi: 10.2196/11244] [Medline:30942695]
Agarwal S, LeFevre AE, Lee J, L'Engle K, Mehl G, Sinha C, WHO mHealth Technical Evidence Review Group. Guidelines
for reporting of health interventions using mobile phones: mobile health (mHealth) evidence reporting and assessment(mERA) checklist. BMJ 2016 Mar 17;352:i1174. [doi: 10.1136/bmj.i1174] [Medline: 26988021]
Jones R, Mateer J. Indirect risk related failures of Medical Information Systems. 2020 Aug 15 Presented at: 14th IEEE
Conference on Industrial Electronics and Applications (ICIEA). IEEE; 2019; Xi'an, China p. 994-999. [doi:10.1109/iciea.2019.8834057]

Abbreviations
EQUATOR: Enhancing the QUAlity and Transparency Of health Research
FDA: Food and Drug Administration
IT: informational technology
JAMA: Journal of the American Medical Association
mHealth: mobile health
RCT: randomized clinical trial

http://medinform.jmir.org/2020/11/e21874/

XSL• FORenderXJMIR Med Inform 2020 | vol. 8 | iss. 11 | e21874 | p. 3
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Ologeanu-Taddei

Edited by C Lovis; submitted 27.06.20; peer-reviewed by R Marcilly, Q Yang; comments to author 03.08.20; revised version received
05.09.20; accepted 12.09.20; published 18.11.20
Please cite as:
Ologeanu-Taddei R
Assessment of mHealth Interventions: Need for New Studies, Methods, and Guidelines for Study Designs
JMIR Med Inform 2020;8(11):e21874
URL: http://medinform.jmir.org/2020/11/e21874/
doi: 10.2196/21874
PMID: 33206060©Roxana Ologeanu-Taddei. Originally published in JMIR Medical Informatics (http://medinform.jmir.org), 18.11.2020. This is
an open-access article distributed under the terms of the Creative Commons Attribution License(https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium,provided the original work, first published in JMIR Medical Informatics, is properly cited. The complete bibliographic information,a link to the original publication on http://medinform.jmir.org/, as well as this copyright and license information must be included.

http://medinform.jmir.org/2020/11/e21874/

XSL• FORenderXJMIR Med Inform 2020 | vol. 8 | iss. 11 | e21874 | p. 4
(page number not for citation purposes)