## Original Paper

# Quality Assessment of Smartphone Medication Management Apps in France: Systematic Search

Mickael Toïgo<fnref n="1" />, MD; Julie Marc<fnref n="1" />, MSc, MD; Maurice Hayot<fnref n="2" />, MD, PhD; Lionel Moulis<fnref n="3" />,<fnref n="4" />, MD; Francois Carbonnel<fnref n="1" />,<fnref n="5" />, MD, PhD

**Corresponding Author:**

Francois Carbonnel, MD, PhD
Desbrest Institute of Epidemiology and Public Health
Univ Montpellier, INSERM
Campus Santé, IURC, 641 avenue du Doyen Gaston Giraud
Montpellier, 34090
France

Phone: 33 684014834

Email: [Francois.carbonnel@umontpellier.fr](mailto:Francois.carbonnel@umontpellier.fr)

## Abstract

**Background:** Adherence to medication is estimated to be around 50% for chronically ill patients in high-income countries. Improving the effectiveness of adherence interventions could have a far greater impact on population health than any improvement in specific medical treatments. Mobile health (mHealth) is one of the most effective solutions for helping patients improve their medication intake, notably through the use of mobile apps with reminder systems. With more than 327,000 apps available in the mHealth field, it is difficult for health care professionals and patients alike to choose which apps to recommend and use.

**Objective:** We aim to carry out a systematic search of medication management smartphone apps available in France that send reminders to patients and assess their quality using a validated scale.

**Methods:** Mobile apps were identified in October and November 2022 after a systematic keyword search on the 2 main app download platforms: App Store (Apple Inc) and Google Play Store. Inclusion criteria were free availability, date of last update, and availability in French. Next, 2 health care professionals independently evaluated the included apps using the French version of the Mobile App Rating Scale (MARS-F), an objective scoring system validated for assessing the overall quality of apps in the mHealth field. An intraclass correlation coefficient was calculated to determine interrater reliability.

**Results:** In total, 960 apps were identified and 49 were selected (25 from the App Store and 24 from the Google Play Store). Interrater reliability was excellent (intraclass correlation coefficient 0.92; 95% CI 0.87-0.95; *P*<.001). The average MARS-F score was 3.56 (SD 0.49) for apps on the App Store and 3.51 (SD 0.46) for those on the Google Play Store, with 10 apps scoring above 4 out of 5. Further, 2 apps were tested in at least one randomized controlled trial and showed positive results. The 2 apps with the highest ratings were *Mediteo rappel de médicaments* (Mediteo GmbH) and *TOM rappel medicaments, pilule* (Innovation6 GmbH), available on both platforms. Each app's MARS-F score was weakly correlated with user ratings on the App Store and moderately correlated on the Google Play Store.

**Conclusions:** To our knowledge, this is the first study that used a validated scoring system to evaluate medication management apps that send medication reminders. The quality of the apps was heterogeneous, with only 2 having been studied in a randomized controlled trial with positive results. The evaluation of apps in real-life conditions by patients is necessary to determine their acceptability and effectiveness. Certification of apps is also essential to help health care professionals and patients identify validated apps.

(JMIR Mhealth Uhealth 2024;12:e54866) doi: 10.2196/54866

<footnote n="1">¹Department of General Practice, Univ Montpellier, Montpellier, France</footnote>

<footnote n="2">²PhyMedExp, Univ Montpellier, CNRS, INSERM, CHU Montpellier, Montpellier, France</footnote>

<footnote n="3">³Clinical Research and Epidemiology Unit, Department of Public Health, Univ Montpellier, CHU Montpellier, Montpellier, France</footnote>

<footnote n="4">⁴Pathogenesis and Control of Chronic and Emerging Infections, Univ Montpellier, INSERM, EFS, University of Antilles, Montpellier, France</footnote>

<footnote n="5">⁵Desbrest Institute of Epidemiology and Public Health, Univ Montpellier, INSERM, Montpellier, France</footnote>
## KEYWORDS

medication adherence; mobile apps; telemedicine; reminder system; behavioral therapy; mHealth; mobile health; app; apps; applications; smartphone apps; medication; medications; adherence; search; searches; searching; systematic; App Store; Google Play; French; reminder; reminders; MARS; quality; Mobile App Rating Scale; mobile phone

## Introduction

Therapeutic adherence is defined by the World Health Organization as “the extent to which the behaviors of a person required to take medication, follow a diet and/or change lifestyle correspond to the recommendations agreed with a healthcare professional” [1]. It is estimated to be around 50% for people with chronic diseases in high-income countries [1]. The result is an increased risk of complications, hospitalization, and mortality for these patients, as well as consequently higher health care costs [2-5]. Improving the effectiveness of adherence interventions could potentially have a far greater impact on population health than any improvement in specific medical treatments [1].

Mobile health (mHealth) is a promising strategy to optimize therapeutic adherence [6]. mHealth covers medical and public health practices based on mobile devices such as cell phones, patient monitoring systems, personal digital assistants, and other wireless devices [7]. It is dominated by the use of health and wellness smartphone apps, the number of which continues to grow yearly [8]. In 2021, over 327,000 mHealth apps were listed in digital stores [8].

In France, almost 90% of French people aged older than 12 years owned a smartphone in 2022 and 72% of them had downloaded apps [9]. For their part, French general practitioners seem to be in favor of mHealth apps or devices and ready to prescribe these in their practice, but at the same time stress the importance of validating the use of these apps through randomized clinical trials and certification by health care professionals [10].

Several studies have confirmed that the use of a medication management app improves adherence to therapy, notably by sending reminders, even in older patients with no experience of using new technologies [11-14]. However, no study evaluating the quality of medication management apps via a validated score has been published to date.

The main objective of this study was to identify medication management apps to improve patient adherence and assess their quality, using a validated scale. The secondary objective of this study was to identify high-quality medication reminder apps and to provide recommendations to any patient needing to take one or more medications, regardless of pathology.

## Methods

### Overview

This involved a systematic search of smartphone apps with content evaluation, carried out between October 1, 2022, and June 20, 2023. It was reported in line with the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) checklist for reporting a systematic review

(items not pertinent to a systematic search of apps were considered not applicable) [15].

### App Selection

Apps were searched from October 1 to November 30, 2022, on the App Store and Google Play Store. These 2 platforms were used by over 99% of mobile users to download apps in 2022 [16].

The keywords searched on the download platforms were “medication reminder” (“rappel de médicament”), “medication monitoring” (“suivi de médicament”), “treatment reminder” (“rappel de traitement”), “treatment monitoring” (“suivi de traitement”), and “pillbox” (“pilulier”). The selection of search terms was based on existing studies, which were adapted after several prior search attempts on digital download platforms [17]. We also included native health apps, automatically present on iOS and Android phones without the need to download them. Being preinstalled on smartphones, they cannot be identified by our search equations on app stores. However, they are available and easily accessible to users. Android currently has no native health app.

Inclusion criteria were as follows: (1) medication reminder app; (2) availability in the French language; (3) free (apps free for only a trial period were not selected); (4) updated within the last 2 years (since 2022, Google Play Store removes apps without updates for more than 2 years [18], and for the App Store, this period is 3 years for apps with few downloads in the last 12 months [19]); and (5) not specific to a single treatment (eg, the contraceptive pill) or to the treatment of a particular pathology. Inclusion criteria were based on the data available on the apps' presentation page, that is, the title, description, images, and general information about the app. Apps were excluded after download in the event of (1) or (2) unavailability on download platforms at the time of evaluation by one of the evaluators.

The decision to assess exclusively free apps was grounded in the primary target audience of patients encountering diverse and, at times, multiple impediments to consistent medication adherence. The high cost of an app is one of the main reasons why it is not downloaded [20]. In addition, users frequently report hidden costs as the main reason for discontinuation [20]. So, considering the price as a dissuading factor for the adoption of health apps, proposing a paid app to this population appeared inadequate [21].

### Evaluation via the French Version of the Mobile App Rating Scale App

The Mobile App Rating Scale (MARS) was used to evaluate the apps. This score was previously validated for the overall quality assessment of an app in the field of mHealth [22]. The French version of this score, MARS-F, has also been validated for use with French-language apps [23].
MARS-F is a 19-item questionnaire divided into 4 objective evaluation sections (A to D): engagement (5 items: entertainment, interest, personalization, interactivity, and adaptation to target group), functionality (4 items: app performance, ease of use, navigation, and app gesture design), aesthetics (3 items: layout, graphic design, and overall visual appeal), and information (7 items: accuracy of app description, precise app objectives, quality of information, quantity of information, visual information, credibility of information, and scientific evidence; Multimedia Appendix 1). Each item is rated with a Likert scale from 1 (“inadequate”) to 5 (“excellent”). When specific objectives (item 14) were not delineated, and pertinent information (items 15, 16, and 17) or scientific evidence (item 19) was lacking, the respective items were rated as “not applicable.” Consequently, these items were excluded from consideration in the overall scoring process. In total, MARS-F is a score out of 5, corresponding to the average of these 4 sections. Section E, not included in the overall MARS-F calculation, evaluates the apps’ subjective quality via 4 items and is described separately. The MARS-F also includes a preliminary descriptive section on the apps’ characteristics, including users’ ratings on the platforms, the apps’ target objectives, the strategies used, its affiliation (commercial, governmental, and academic), as well as the technical aspect (password protection, possible sharing, and internet access required).

In total, 2 family medicine residents evaluated each app independently. Before starting, they assisted with a training video on the use of MARS-F and trained in real-life conditions by evaluating 3 apps not selected for this study. The scores for each item were then discussed one by one to ensure a similar understanding on the part of the 2 evaluators.

Apps available on the 2 platforms were evaluated independently on iOS and Android. Each app was tested and used for at least 15 minutes. Evaluations took place from February to June 2023.

## Statistical Analysis

App characteristics were described using frequencies and proportions. To determine interrater reliability, a 2-way mixed-effects model intraclass correlation coefficient (ICC) was calculated for the mean of the raters. An ICC below 0.5

was considered poor, moderate between 0.5 and 0.75, good between 0.75 and 0.90, and excellent if above 0.90 [24].

The average of the scores given by the 2 raters or evaluators was used for the final rating of each app. Apps were compared according to their respective MARS-F quality score. The results were presented as mean (SD) and median and quartiles.

The correlation between the average rating of app users on download platforms and the MARS-F obtained was measured by Spearman correlation after a normality test. Regarding the correlation between MARS-F and user ratings, we excluded apps with a limited number of ratings on platforms, as they may not be highly representative of users. The choice of the threshold was determined following a sensitivity analysis, wherein the correlation was computed for various thresholds: five ratings on platforms, 10, 20, 30, 50, and so on. The selected threshold was the lowest one for which both the lower and upper thresholds yielded similar results. The threshold of 20 user ratings was finally selected. The correlation between reviewers’ subjective assessment of the apps via item 23 (“What is your overall star rating for the application?”) and the MARS-F obtained for each app was also calculated by Pearson correlation after a normality test. The correlation was judged as very strong from 1 to 0.9, strong from 0.9 to 0.7, moderate from 0.7 to 0.5, weak from 0.5 to 0.3, and very weak from 0.3 to 0.

All statistical analyses were performed with EasyMedStat (version 3.29; EasyMedStat).

## Ethical Considerations

The Research Ethics Committee of the University of Montpellier approved this research project (UM 2022-006bis; Multimedia Appendix 2). This study was not funded.

# Results

## App Selection

After a keyword search and the addition of native apps, 480 apps were identified on the App Store and 1191 on the Google Play Store. A total of 25 apps meeting the inclusion and exclusion criteria were selected from the App Store and 24 from the Google Play Store (Figure 1).
<figure><img src="image_1.png" /><figcaption>**Figure 1.** Flowchart showing app selection. The numbers under “selection” refer to the inclusion criteria. The sum of the percentages exceeds 100% because a single app may not meet different inclusion criteria.</figcaption></figure>

## Description of the Apps

In total, 51% (n=25) of the apps evaluated were available on the App Store and 49% (n=24) on the Google Play Store, 10 of which were common and available on both platforms. Further, 40 had been rated by users on both platforms (23 on the App Store and 17 on the Google Play Store).

Although all the apps were available free of charge, some had a paid version, as in the case of 15 (60%) App Store apps and 12 (50%) Google Play Store apps. The affiliation of the developers was not always known, but the majority were commercial (24/25, 96% on the App Store; 21/24, 87.5% on the Google Play Store).

Technical aspects according to the download platform are summarized in Table 1.
<table><caption>**Table 1.** Technical aspects of apps according to platform.</caption><thead><tr><th></th><th>App Store (n=25), n (%)</th><th>Google Play (n=24), n (%)</th></tr></thead><tbody><tr><td>Allows sharing</td><td>12 (48)</td><td>10 (42)</td></tr><tr><td>Has an app-linked community</td><td>3 (12)</td><td>1 (4)</td></tr><tr><td>Has the ability to password-protect</td><td>10 (40)</td><td>3 (12.5)</td></tr><tr><td>Requires a login ID</td><td>4 (16)</td><td>2 (8)</td></tr><tr><td>Sends reminders</td><td>25 (100)</td><td>24 (100)</td></tr><tr><td>Needs internet access to work</td><td>10 (40)</td><td>8 (33)</td></tr></tbody></table>

## Evaluation Reliability

MARS-F interrater reliability for all apps was excellent (ICC 0.92; 95% CI 0.87-0.95; *P*<.001).

For each section, interrater reliability was excellent for engagement (ICC 0.92; 95% CI 0.88-0.95; *P*<.001) and for subjective app quality (ICC 0.95; 95% CI 0.92-0.97; *P*<.001). It was good for aesthetics (ICC 0.81; 95% CI 0.70-0.89; *P*<.001)

and the information section (ICC 0.86; 95% CI 0.78-0.92; *P*<.001). Finally, interrater reliability was only moderate for the functionality part (ICC 0.70; 95% CI 0.52-0.82; *P*<.001).

## Quality of Apps Based on MARS-F

The mean MARS-F score was 3.56 (SD 0.49) for iOS apps and 3.51 (SD 0.46) for Android apps (Table 2). The full set of scores by the app is presented in Multimedia Appendix 3.

<table><caption>**Table 2.** Average score, range, and median per section.</caption><thead><tr><th rowspan="2">Variable</th><th colspan="2">iOS (n=25)</th><th colspan="2">Android (n=24)</th></tr><tr><th>Score, mean (SD)</th><th>Score, median (Q1<sup>a</sup>-Q3)</th><th>Score, mean (SD)</th><th>Score, median (Q1-Q3)</th></tr></thead><tbody><tr><td>Section A: engagement</td><td>3.19 (0.616)</td><td>3.3 (2.8-3.6)</td><td>3.15 (0.628)</td><td>3.15 (2.8-3.52)</td></tr><tr><td>Section B: functionality</td><td>4.26 (0.448)</td><td>4.38 (4-4.62)</td><td>4.17 (0.431)</td><td>4.25 (3.84-4.53)</td></tr><tr><td>Section C: aesthetics</td><td>3.5 (0.555)</td><td>3.67 (3.17-4)</td><td>3.38 (0.599)</td><td>3.5 (3-3.83)</td></tr><tr><td>Section D: information</td><td>3.29 (0.708)</td><td>3.3 (2.83-3.8)</td><td>3.33 (0.561)</td><td>3.17 (2.83-3.81)</td></tr><tr><td>MARS-F<sup>b</sup>: global quality</td><td>3.56 (0.485)</td><td>3.6 (3.21-3.98)</td><td>3.51 (0.455)</td><td>3.48 (3.29-3.88)</td></tr><tr><td>Section E: subjective quality</td><td>2.52 (1.12)</td><td>2.38 (1.5-3.62)</td><td>2.55 (1.19)</td><td>2.25 (1.72-3.22)</td></tr></tbody></table>

## General App Features

The iOS apps with the highest MARS-F scores were *TOM rappel medicament*, *Pilule* (MARS-F score: 4.37; Innovation6 GmbH), then *Mediteo rappels de médicaments* from (MARS-F score: 4.34; Mediteo GmbH), followed by *Rappels de médicaments* from (MARS-F at 4.13; smartpatient GmbH). The Android apps with the highest MARS-F scores were *Mediteo rappels de médicaments* (MARS-F score: 4.32), followed by *TOM Rappel medicaments, pilule* (MARS-F score: 4.24) and *Rappel de pilule et medicament* (MARS-F score: 4.09; Medisafe). The MARS-F ranking of apps by platform is available in Multimedia Appendix 4.

In total, 84% (n=21) of the apps available on the App Store had a MARS-F score above 3, with 5 (20%) scoring above 4. On Google Play Store, 79% (n=19) of apps had a MARS-F score higher than 3, including 5 (21%) with a score higher than 4.

The “scientific evidence” item was completed for 4 (16%) apps on the App Store and 2 (8%) apps on the Google Play Store, 2 of which were common to both platforms.

## Correlation Between MARS-F and User Ratings

In total, 23 (92%) apps were rated by users on the App Store, with an average rating of 4.38/5 (range 2.9-5.0) and an average number of ratings of 475 (range 1-3900; Multimedia Appendix 5). On Google Play Store, 17 (71%) apps were rated by users, with an average score of 4.17/5 (range 3.3-4.8) and an average number of ratings of 25,062 (range 24-223,000).

After performing sensitivity analyses, tests looking for a correlation between a given app's MARS-F and average user rating were carried out on apps with at least 20 reviews on the platforms. The correlation was weak on the App Store (ρ=0.46; *P*=.12) and moderate on the Google Play Store (ρ=0.55; *P*=.02).

## Correlation Between MARS-F and Reviewers' Subjective Evaluation

The correlation between MARS-F and item 23 (“What is your overall star rating of the app?”) was considered very strong for iOS apps (ρ=0.93; *P*<.001) and strong for Android apps (ρ=0.88; *P*<.001).

The mean score for this item was 2.90 and 2.98 on iOS and Android, respectively, which is below the respective mean MARS-F.

<footnote><sup>a</sup>Q: quartile.</footnote>

<footnote><sup>b</sup>MARS-F: Mobile App Rating Scale, French version.</footnote>
# Discussion

## Principal Results

The main objective of our study was to identify medication management apps to improve patient adherence. In total, 960 apps were identified: 358 (37.3%) on App Store and 602 (62.7%) on Google Play Store. This imbalance in favor of the Google Play Store has already been observed in several studies [25,26]. However, after selection, the number of apps was balanced 51% (n=25) on the App Store and 49% (n=24) on the Google Play Store, so it seems that the Google Play Store search engine offers more apps for the same keyword. This balanced proportion is consistent with a 2019 medication management app evaluation study in which 328 apps available in English were studied [17]. In this study, 53.4% and 46.6% of apps were retrieved from the Google Play Store and the App Store, respectively [17]. Our study evaluated the content of these apps according to the information available on app download stores without downloading them.

Our study highlighted a cybersecurity issue: only 40% (n=10) of apps identified on iOS had password-protected access, and 4% (n=1) required a login ID. These figures were even lower for Android apps, at 12.5% (n=3) and 8.3% (n=2), respectively. This was also the case for pain management apps, with 39% of apps evaluated allowing password protection and 44% requiring login [27].

The overall quality of the medication reminder apps evaluated is considered acceptable to moderate, with 84% (n=21) of apps on the App Store achieving a MARS-F higher than 3/5 with an average of 3.56 (SD 0.49) and 79% (n=19) on Google Play Store with an average MARS-F of 3.51 (SD 0.46). We have no point of comparison in the absence of any previous French-speaking or English-speaking study evaluating medication reminder apps by a validated score. Nevertheless, these results were expected, as they are consistent with other French studies on health apps for smoking cessation (mean MARS-F of 3.49, SD 0.57) for 14 apps [28], nutrition (mean MARS-F of 3.26, SD 0.43) for 15 apps [29], mental health (mean MARS-F of 3.16, SD 0.43) for 12 apps [30], or oral hygiene (mean MARS-F of 3.20, SD 0.38) for 9 apps [31].

All the other studies reported that the functionality section of the MARS scale had the highest ratings, which was also the case in our study, with an average of 4.26 (SD 0.45) for the App Store and 4.17 (SD 0.43) for the Google Play Store for this section. This shows that most apps are functional, which also coincides with the fact that only 1 app was excluded for malfunction in our study. In contrast, the section with the lowest average score was the engagement section (average of 3.19, SD 0.62, on the App Store and 3.15, SD 0.63, on the Google Play Store), followed by the information section (average of 3.29, SD 0.71, and 3.33, SD 0.56, on the App Store and Google Play Store, respectively), in line with the literature [32-37]. Several other studies have noted lower scores for the information section [25,27,30,31,38-41]. Nevertheless, these 2 sections are regularly cited as being those for which we find lower averages. Further, 1 exception is the study by Salehinejad et al [42], for which the highest-rated section was the information section. This is a

special case, as the evaluation was concerned with COVID-19 management apps, which were probably created for information purposes in the first place.

About the information section, the lack of involvement from health care professionals, universities, or governmental organizations, may explain the observed low scores, thereby lowering the credibility of the apps. This point is assessed in the MARS-F by item 18 ("Does the application come from a legitimate source (specified in the application store description or in the application itself)?"). Often difficult to ascertain [16], most apps were affiliated with a commercial company, with only 14.6% of medication management apps and 15.2% of apps for patients with genitourinary tumors showing involvement by health care professionals [43]. Involvement by health care professionals was sometimes mentioned on the apps' websites, but without explaining their precise role or degree of involvement. The "scientific evidence" item (item 19) was completed for 4 different apps out of the 49 evaluated in our study, 2 of which were common to both the App Store and Google Play Store. These 2 apps are the only ones to have been tested in at least one randomized controlled trial with positive results: smartpatient GmbH or MyTherapy's *Rappels de médicaments* [44] and Medisafe's *Rappel de pilule et medicament* [45]. They are ranked among the best apps on the 2 platforms according to their MARS-F obtained in this study. This lack of scientific validation of health care apps is a recurring theme [25,29,32,33,37].

In our study, user ratings of digital app stores were higher than MARS-F, as expected [27,29]. The absence or low correlation between user rating and MARS score has also been described previously [33,46]. To our knowledge, only the study by Chen et al [47] found a link between the quality of apps assessed by the MARS score and user rating for drug information apps. The average MARS score of apps with higher user star ratings was significantly higher than for apps with lower user star ratings (3.38, SD 0.64, vs 3.05, SD 0.64, *P*<.001) [47]. However, this may be explained by their study design: 3 out of 7 assessors were not health care professionals, which may be a bias.

The correlation between item 23 ("What is your overall star rating for the app?") and the MARS-F was very strong for apps on iOS and strong for apps on Android. These results are consistent with other studies dealing with nutrition and mental health apps, respectively [29,30]. It is interesting to note that the subjective rating of the reviewers correlated with the overall quality of the app assessed by an objective scale, unlike the user rating. It is challenging to predict whether this is due to the evaluators' experience or the fact that they have delved deeper into the evaluation of apps using the MARS score.

## Strengths and Limitations

The main limitation of this study is the mobile app sector itself since this study is a reflection of the supply and quality of apps for a specific period. We note, for example, that 6 apps (3 on the App Store and 3 on the Google Play Store) were excluded because they were no longer available a few months after they were identified and downloaded for evaluation. This difficulty had already been highlighted in a study that aimed to evaluate apps for pregnant women at 2-year intervals, in 2017 and then
in 2019. One of the things that stood out was that the best app in 2017 was no longer available in 2019 at the time of the second evaluation [40]. The presence of new versions following updates is another element that can influence the quality of an app over time. App selection by only one of the evaluators is also a limitation, as it is possible that some eligible apps have not been identified. In terms of evaluation, the apps were assessed after they had been in use for a minimum of 15 minutes, so additional strengths or weaknesses of each app may not have been detected due to this limited duration.

Our choice to evaluate only free apps is also open to criticism. It could be argued that this choice is induced by a bias of the health care system in which the authors of this paper, who are French, operate. France has indeed the lowest share of out-of-pocket health expenditure among all EU countries [48]. Yet, medication nonadherence is a global problem. Costs attributed to “all causes” nonadherence range from US $5271 to US $52,341 [49]. The high cost of a paid app is dissuasive and hidden costs have been demonstrated as one of the main reasons for discontinuation of the use of an app [20,21]. To our knowledge, the superiority of a paid medication management app over a free one has yet to be demonstrated.

Finally, the MARS score was used because its use is simple, validated, and widespread in the evaluation of health care apps. However, it has several limitations. The first is the absence of data security and privacy evaluation criteria, although the presence of a password and login is indicated in the descriptive section. These points are nevertheless a concern for users of health care apps [50]. The second limitation is the absence of a threshold for judging app quality. We have described it here as moderate, as in several other studies, which found similar average scores, but this was not described when this tool was created [51].

To our knowledge, no other study has evaluated medication management apps using a validated scale. The identification method combined with the inclusion and exclusion criteria enabled an exhaustive analysis of free medication reminder apps available to French patients. These were not selected based on user ratings or the number of downloads from digital stores, which made it possible to evaluate apps that are not promoted

on platforms but may nevertheless be relevant to patients. The weak correlation between app quality and user rating found in this study supports this approach. Apps available for iOS and Android devices were evaluated independently on each operating system, as there may be differences in terms of updating or functionality depending on the device used. Finally, the independent testing by 2 evaluators is a strong point, particularly with the observed excellent interevaluator reliability, enabling result confirmation.

## Perspectives

This work is the first step toward facilitating more in-depth studies on top-rated and best-quality apps. App evaluation by patients with the user version of the MARS [52] would be relevant, even though there is currently no validated French version of this scale. The long-term use and benefits of these apps need to be studied in randomized clinical trials, to verify their acceptability and whether or not they improve therapeutic adherence and clinical outcomes in patients undergoing long-term treatment.

The results of this study, added to other studies on app evaluation in the health care field, show the possible ways to improve existing apps and give leads for the creation of new ones. Functionality is paramount, and this point already seems to have been achieved for the majority of apps currently available. The areas that need improvement relate above all to engagement and the information available, of which gamification is 1 avenue to be explored [53]. Data security and privacy protection are also important for patients and should not be neglected.

Currently, France is investing in the use of digital health, particularly for patients with chronic diseases. There is an ongoing project to list more than 50 apps offering exchanges with *Mon Espace Santé* by 2026. Therefore, it is vital to set up certification for existing apps to help doctors in their recommendations [10]. The creation of an app with the help of health care professionals and validated by the Agence du Numérique en Santé would also be a solution that would enable doctors to know which apps to recommend, thus facilitating its use by patients.

## Conflicts of Interest

None declared.

## Multimedia Appendix 1

Mobile App Rating Scale, French version (MARS-F).
[PDF File (Adobe PDF File), 190 KB-Multimedia Appendix 1]

## Multimedia Appendix 2

Ethics committee approval.
[PDF File (Adobe PDF File), 280 KB-Multimedia Appendix 2]

## Multimedia Appendix 3

The full set of scores by app.
[DOCX File, 40 KB-Multimedia Appendix 3]
## Multimedia Appendix 4

Ranking of apps by platform according to the Mobile App Rating Scale, French version (MARS-F).
[DOCX File, 24 KB-Multimedia Appendix 4]

## Multimedia Appendix 5

User ratings and number of evaluation of apps by platform.
[DOCX File, 27 KB-Multimedia Appendix 5]

## Multimedia Appendix 6

PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) checklist.
[DOCX File, 32 KB-Multimedia Appendix 6]

## References

1. Sabaté E. Adherence to Long-Term Therapies: Evidence for Action. World Health Organization. 2003. URL: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6147925/ [accessed 2024-02-28]

2. Xu T, Yu X, Ou S, Liu X, Yuan J, Tan X, et al. Adherence to antihypertensive medications and stroke risk: a dose-response meta-analysis. J Am Heart Assoc. 2017;6(7):e006371. [FREE Full text] [doi: 10.1161/JAHA.117.006371] [Medline: 28743788]

3. Egede LE, Gebregziabher M, Dismuke CE, Lynch CP, Axon RN, Zhao Y, et al. Medication nonadherence in diabetes: longitudinal effects on costs and potential cost savings from improvement. Diabetes Care. 2012;35(12):2533-2539. [FREE Full text] [doi: 10.2337/dc12-0572] [Medline: 22912429]

4. Currie CJ, Peyrot M, Morgan CL, Poole CD, Jenkins-Jones S, Rubin RR, et al. The impact of treatment noncompliance on mortality in people with type 2 diabetes. Diabetes Care. 2012;35(6):1279-1284. [FREE Full text] [doi: 10.2337/dc11-1277] [Medline: 22511257]

5. Hiligsmann M, Cornelissen D, Vrijens B, Abrahamsen B, Al-Daghri N, Biver E, et al. Determinants, consequences and potential solutions to poor adherence to anti-osteoporosis treatment: results of an expert group meeting organized by the European Society for Clinical and Economic Aspects of Osteoporosis, Osteoarthritis and Musculoskeletal Diseases (ESCEO) and the International Osteoporosis Foundation (IOF). Osteoporos Int. 2019;30(11):2155-2165. [FREE Full text] [doi: 10.1007/s00198-019-05104-5] [Medline: 31388696]

6. Pouls BPH, Vriezekolk JE, Bekker CL, Linn AJ, van Onzenoort HAW, Vervloet M, et al. Effect of interactive eHealth interventions on improving medication adherence in adults with long-term medication: systematic review. J Med Internet Res. 2021;23(1):e18901. [FREE Full text] [doi: 10.2196/18901] [Medline: 33416501]

7. Global Observatory for eHealth Series—Volume 3. MHealth: New Horizons for Health Through Mobile Technologies. World Health Organization. 2011. URL: https://iris.who.int/bitstream/handle/10665/44607/9789241564250_eng.pdf?sequence=1 [accessed 2024-02-28]

8. Évaluation des Applications dans le champ de la santé mobile (mHealth)—État des lieux et critères de qualité du contenu médical pour le référencement des services numériques dans l'espace numérique de santé et le bouquet de services des professionnels. La Plaine, Saint-Denis. Haute Autorité de Santé; 2021.

9. Baromètre du numérique. Centre de recherche pour l'étude et l'observation des conditions de vie. 2022. URL: https://www.economie.gouv.fr/files/files/directions_services/cge/barometre-numerique-2022.pdf?v=1684931616 [accessed 2024-01-18]

10. Vecchia CD, Leroy T, Bauquier C, Pannard M, Sarradon-Eck A, Darmon D, et al. Willingness of French general practitioners to prescribe mHealth apps and devices: quantitative study. JMIR mHealth uHealth. 2022;10(2):e28372. [FREE Full text] [doi: 10.2196/28372] [Medline: 35147508]

11. Al-Arkee S, Mason J, Lane DA, Fabritz L, Chua W, Haque MS, et al. Mobile apps to improve medication adherence in cardiovascular disease: systematic review and meta-analysis. J Med Internet Res. 2021;23(5):e24190. [FREE Full text] [doi: 10.2196/24190] [Medline: 34032583]

12. Zhou TT, Wang R, Gu SJ, Xie LL, Zhao QH, Xiao MZ, et al. Effectiveness of mobile medical apps in ensuring medication safety among patients with chronic diseases: systematic review and meta-analysis. JMIR mHealth uHealth. 2022;10(11):e39819. [FREE Full text] [doi: 10.2196/39819] [Medline: 36413386]

13. Fenerty SD, West C, Davis SA, Kaplan SG, Feldman SR. The effect of reminder systems on patients' adherence to treatment. Patient Prefer Adher. 2012;6:127-135. [FREE Full text] [doi: 10.2147/PPA.S26314] [Medline: 22379363]

14. Mira JJ, Navarro I, Botella F, Borrás F, Nuño-Solinís R, Orozco D, et al. A Spanish pillbox app for elderly patients taking multiple medications: randomized controlled trial. J Med Internet Res. 2014;16(4):e99. [FREE Full text] [doi: 10.2196/jmir.3269] [Medline: 24705022]

15. Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. PLoS Med. 2021;18(3):e1003583. [FREE Full text] [doi: 10.1371/journal.pmed.1003583] [Medline: 33780438]
16. Mobile operating system market share France. StatCounter Global Stats. URL: https://gs.statcounter.com/os-market-share/mobile/france [accessed 2024-01-18]

17. Tabi K, Randhawa AS, Choi F, Mithani Z, Albers F, Schnieder M, et al. Mobile apps for medication management: review and analysis. JMIR mHealth uHealth. 2019;7(9):e13608. [FREE Full text] [doi: 10.2196/13608] [Medline: 31512580]

18. Developer program policy. Google Play Console. URL: https://support.google.com/googleplay/android-developer/answer/13837496?hl=en&ref_topic=9877065&sjid=13400681840324225447-EU [accessed 2024-01-18]

19. Clarifying criteria and new timing extension for App Store improvements process. Apple Developer. URL: https://developer.apple.com/news/?id=gi6npkmf [accessed 2024-01-18]

20. Krebs P, Duncan DT. Health app use among US mobile phone owners: a national survey. JMIR mHealth uHealth. 2015;3(4):e101. [FREE Full text] [doi: 10.2196/mhealth.4924] [Medline: 26537656]

21. Folkvord F, Bol N, Stazi G, Peschke L, Lupiáñez-Villanueva F. Preferences in the willingness to download an mHealth app: discrete choice experimental study in Spain, Germany, and the Netherlands. JMIR Form Res. 2023;7:e48335. [FREE Full text] [doi: 10.2196/48335] [Medline: 38145470]

22. Muro-Culebras A, Escriche-Escuder A, Martin-Martin J, Roldán-Jiménez C, De-Torres I, Ruiz-Muñoz M, et al. Tools for evaluating the content, efficacy, and usability of mobile health apps according to the consensus-based standards for the selection of health measurement instruments: systematic review. JMIR mHealth uHealth. 2021;9(12):e15433. [FREE Full text] [doi: 10.2196/15433] [Medline: 34855618]

23. Saliasi I, Martinon P, Darlington E, Smentek C, Tardivo D, Bourgeois D, et al. Promoting health via mHealth applications using a French version of the mobile app rating scale: adaptation and validation study. JMIR mHealth uHealth. 2021;9(8):e30480. [FREE Full text] [doi: 10.2196/30480] [Medline: 34463623]

24. Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med. 2016;15(2):155-163. [FREE Full text] [doi: 10.1016/j.jcm.2016.02.012] [Medline: 27330520]

25. Narrillos-Moraza Á, Gómez-Martínez-Sagrera P, Amor-García MÁ, Escudero-Vilaplana V, Collado-Borrell R, Villanueva-Bueno C, et al. Mobile apps for hematological conditions: review and content analysis using the mobile app rating scale. JMIR mHealth uHealth. 2022;10(2):e32826. [FREE Full text] [doi: 10.2196/32826] [Medline: 35171109]

26. Kanjak J, Likitdee N, Kietpeerakool C, Temtanakitpaisan A. Evaluation of mobile health applications for cervical cancer in the digital marketplace. Obstet Gynecol Sci. 2022;65(3):244-255. [FREE Full text] [doi: 10.5468/ogs.22037] [Medline: 35381627]

27. Salazar A, de Sola H, Failde I, Moral-Munoz JA. Measuring the quality of mobile apps for the management of pain: systematic search and evaluation using the mobile app rating scale. JMIR mHealth uHealth. 2018;6(10):e10718. [FREE Full text] [doi: 10.2196/10718] [Medline: 30361196]

28. Bustamante LA, Ménard CG, Julien S, Romo L. Behavior change techniques in popular mobile apps for smoking cessation in France: content analysis. JMIR mHealth uHealth. 2021;9(5):e26082. [FREE Full text] [doi: 10.2196/26082] [Medline: 33983130]

29. Martinon P, Saliasi I, Bourgeois D, Smentek C, Dussart C, Fraticelli L, et al. Nutrition-related mobile apps in the French App Stores: assessment of functionality and quality. JMIR mHealth uHealth. 2022;10(3):e35879. [FREE Full text] [doi: 10.2196/35879] [Medline: 35285817]

30. Carrouel F, du Sartz de Vigneulles B, Bourgeois D, Kabuth B, Baltenneck N, Nusbaum F, et al. Mental health mobile apps in the French App Store: assessment study of functionality and quality. JMIR mHealth uHealth. 2022;10(10):e41282. [FREE Full text] [doi: 10.2196/41282] [Medline: 36223178]

31. Carrouel F, Bourgeois D, Clément C, Tardivo D, Martinon P, Guiral S, et al. Oral-hygiene-related mobile apps in the French App Stores: assessment of functionality and quality. Int J Environ Res Public Health. 2022;19(12):7293. [FREE Full text] [doi: 10.3390/ijerph19127293] [Medline: 35742541]

32. Gutiérrez NB, Ramallo HR, González MF, Martín LAK. Smartphone apps for patients with hematologic malignancies: systematic review and evaluation of content. JMIR mHealth uHealth. 2022;10(9):e35851. [FREE Full text] [doi: 10.2196/35851] [Medline: 36125860]

33. Paganini S, Terhorst Y, Sander LB, Catic S, Balci S, Küchler AM, et al. Quality of physical activity apps: systematic search in App Stores and content analysis. JMIR mHealth uHealth. 2021;9(6):e22587. [FREE Full text] [doi: 10.2196/22587] [Medline: 34106073]

34. Sharpe JD, Kamara MT. A systematic evaluation of mobile apps to improve the uptake of and adherence to HIV pre-exposure prophylaxis. Sex Health. 2018;15(6):587-594. [doi: 10.1071/SH18120] [Medline: 30347177]

35. Seo S, Cho SI, Yoon W, Lee CM. Classification of smoking cessation apps: quality review and content analysis. JMIR mHealth uHealth. 2022;10(2):e17268. [FREE Full text] [doi: 10.2196/17268] [Medline: 35175213]

36. Kim BY, Sharafoddini A, Tran N, Wen EY, Lee J. Consumer mobile apps for potential drug-drug interaction check: systematic review and content analysis using the Mobile App Rating Scale (MARS). JMIR mHealth uHealth. 2018;6(3):e74. [FREE Full text] [doi: 10.2196/mhealth.8613] [Medline: 29592848]

37. McAleese D, Linardakis M, Papadaki A. Quality and presence of behaviour change techniques in mobile apps for the Mediterranean diet: a content analysis of Android Google Play and Apple App store apps. Nutrients. 2022;14(6):1290. [FREE Full text] [doi: 10.3390/nu14061290] [Medline: 35334947]
38. Mohammadzadeh N, Khenarinezhad S, Ghazanfarisavadkoohi E, Safari MS, Pahlevanynejad S. Evaluation of M-Health applications use in epilepsy: a systematic review. Iran J Public Health. 2021;50(3):459-469. [FREE Full text] [doi: 10.18502/ijph.v50i3.5586] [Medline: 34178793]

39. Gerner M, Vuillerme N, Aubourg T, Messner EM, Terhorst Y, Hörmann V, et al. Review and analysis of German mobile apps for inflammatory bowel disease management using the mobile application rating scale: systematic search in App Stores and content analysis. JMIR mHealth uHealth. 2022;10(5):e31102. [FREE Full text] [doi: 10.2196/31102] [Medline: 35503246]

40. Musgrave LM, Kizirian NV, Homer CSE, Gordon A. Mobile phone apps in Australia for improving pregnancy outcomes: systematic search on App Stores. JMIR mHealth uHealth. 2020;8(11):e22340. [FREE Full text] [doi: 10.2196/22340] [Medline: 33196454]

41. Gupta K, Roy S, Altameem A, Kumar R, Saudagar AKJ, Poonia RC. Usability evaluation and classification of mHealth applications for type 2 diabetes mellitus using MARS and ID3 algorithm. Int J Environ Res Public Health. 2022;19(12):6999. [FREE Full text] [doi: 10.3390/ijerph19126999] [Medline: 35742248]

42. Salehinejad S, Niakan Kalhori SR, Gohari SH, Bahaadinbeigy K, Fatehi F. A review and content analysis of national apps for COVID-19 management using Mobile Application Rating Scale (MARS). Inform Health Soc Care. 2021;46(1):42-55. [doi: 10.1080/17538157.2020.1837838] [Medline: 33164594]

43. Amor-García MÁ, Collado-Borrell R, Escudero-Vilaplana V, Melgarejo-Ortuño A, Herranz-Alonso A, Arija JAA, et al. Assessing apps for patients with genitourinary tumors using the Mobile Application Rating Scale (MARS): systematic search in App Stores and content analysis. JMIR mHealth uHealth. 2020;8(7):e17609. [FREE Full text] [doi: 10.2196/17609] [Medline: 32706737]

44. Nurakysh S, Kurakbayev K, Kosherbaeva L, Tazhiyeva A, Aimakhanova A, Kulkaeva G, et al. Evaluation of the effectiveness of the mobile application on adherence of patients with arterial hypertension. Acta Inform Med. 2022;30(1):18-24. [FREE Full text] [doi: 10.5455/aim.2022.30.18-24] [Medline: 35800910]

45. Morawski K, Ghazinouri R, Krumme A, Lauffenburger JC, Lu Z, Durfee E, et al. Association of a smartphone application with medication adherence and blood pressure control: the MedISAFE-BP randomized clinical trial. JAMA Intern Med. 2018;178(6):802-809. [FREE Full text] [doi: 10.1001/jamainternmed.2018.0447] [Medline: 29710289]

46. Wu X, Xu L, Li PF, Tang TT, Huang C. Multipurpose mobile apps for mental health in Chinese App Stores: content analysis and quality evaluation. JMIR mHealth uHealth. 2022;10(1):e34054. [FREE Full text] [doi: 10.2196/34054] [Medline: 34982717]

47. Chen YC, Liao WW, Su MC, Lin YH. Personalized and self-management: systematic search and evaluation quality factors and user preference of drug reference apps in Taiwan. J Pers Med. 2021;11(8):790. [FREE Full text] [doi: 10.3390/jpm11080790] [Medline: 34442435]

48. State of Health in the EU. France: Country Health Profile 2023. OECD, Paris/European Observatory on Health Systems and Policies. 2023. URL: https://eurohealthobservatory.who.int/docs/librariesprovider3/country-health-profiles/chp2023pdf/chp-france2023.pdf?sfvrsn=a15a53aa_3&download=true [accessed 2024-01-18]

49. Cutler RL, Fernandez-Llimos F, Frommer M, Benrimoj C, Garcia-Cardenas V. Economic impact of medication non-adherence by disease groups: a systematic review. BMJ Open. 2018;8(1):e016982. [FREE Full text] [doi: 10.1136/bmjopen-2017-016982] [Medline: 29358417]

50. Zhou L, Bao J, Watzlaf V, Parmanto B. Barriers to and facilitators of the use of mobile health apps from a security perspective: mixed-methods study. JMIR mHealth uHealth. 2019;7(4):e11223. [FREE Full text] [doi: 10.2196/11223] [Medline: 30990458]

51. Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M. Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR mHealth uHealth. 2015;3(1):e27. [FREE Full text] [doi: 10.2196/mhealth.3422] [Medline: 25760773]

52. Stoyanov SR, Hides L, Kavanagh DJ, Wilson H. Development and validation of the User Version of the Mobile Application Rating Scale (uMARS). JMIR mHealth uHealth. 2016;4(2):e72. [FREE Full text] [doi: 10.2196/mhealth.5849] [Medline: 27287964]

53. Abraham O, LeMay S, Bittner S, Thakur T, Stafford H, Brown R. Investigating serious games that incorporate medication use for patients: systematic literature review. JMIR Serious Games. 2020;8(2):e16096. [FREE Full text] [doi: 10.2196/16096] [Medline: 32347811]

Abbreviations

ICC: intraclass correlation coefficient

MARS: Mobile App Rating Scale

MARS-F: Mobile App Rating Scale, French version

mHealth: mobile health

PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-Analyses
Edited by L Buis; submitted 09.12.23; peer-reviewed by L Bustamante, T Orlemann; comments to author 10.01.24; revised version received 28.01.24; accepted 06.02.24; published 18.03.24

Please cite as:

Toïgo M, Marc J, Hayot M, Moulis L, Carbonnel F

Quality Assessment of Smartphone Medication Management Apps in France: Systematic Search

JMIR Mhealth Uhealth 2024;12:e54866

URL: https://mhealth.jmir.org/2024/1/e54866

doi: 10.2196/54866

PMID:

©Mickael Toïgo, Julie Marc, Maurice Hayot, Lionel Moulis, Francois Carbonnel. Originally published in JMIR mHealth and uHealth (https://mhealth.jmir.org), 18.03.2024. This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mHealth and uHealth, is properly cited. The complete bibliographic information, a link to the original publication on https://mhealth.jmir.org/, as well as this copyright and license information must be included.