JMIR FORMATIVE RESEARCH

Høgsdal et al

Original Paper

Adolescents’ Assessment of Two Mental Health–Promoting Mobile
Apps: Results of Two User Surveys
Helene Høgsdal, MSc; Sabine Kaiser, PhD; Henriette Kyrrestad, PhD
Regional Centre for Child and Youth Mental Health and Child Welfare - North, UiT The Arctic University of Norway, Tromsø, Norway

Corresponding Author:
Helene Høgsdal, MSc
Regional Centre for Child and Youth Mental Health and Child Welfare - NorthUiT The Arctic University of Norway
Campus Tromsø, Breivika
Tromsø
NorwayPhone: 47 77 64 66 19
Email: helene.hogsdal@uit.no

Abstract
Background: The importance of mental health promotion is irrevocable and is especially important at a young age. More mental
health-promoting mobile apps have been developed in the last few years. However, their usability and quality have been rarely
assessed.
Objective: The aim of this study is to investigate how adolescents assess the usability, quality, and potential goal achievementof Opp and NettOpp. Opp is a universal mental health–promoting mobile app aimed at 13- to 19-year-olds, and NettOpp is a
mobile app for children and adolescents between 11 to 16 years of age that have experienced negative incidents online.
Methods: A total of 45 adolescents tested either Opp (n=30) or NettOpp (n=15) for a period of 3 weeks and answered aquestionnaire. The System Usability Scale (SUS) was used to measure the usability of the apps. A SUS score above 70 indicates
acceptable usability. Items from the Mobile Application Rating Scale were adapted for study purposes and used to measure thequality and perceived goal achievement that Opp and NettOpp might have on adolescents’ knowledge, attitudes, and intention
to change behavior. Furthermore, adolescents could answer an open comment question.
Results: Opp had a mean SUS score of 80.37 (SD 9.27), and NettOpp’s mean SUS score was 80.33 (SD 10.30). In the overall
evaluation, Opp and NettOpp were given a mean score of 3.78 (SD 0.42) and 4.20 (SD 0.56), respectively, on a 5-point scale,where 5 was best. Most adolescents who evaluated Opp rated that the app would increase knowledge about mental health andhelp young people deal with stress and difficult emotions or situations. Most adolescents who evaluated NettOpp agreed that theapp would increase awareness and knowledge about cyberbullying, change attitudes toward cyberbullying, and motivate themto address cyberbullying. Some adolescents stated that Opp was difficult to navigate and consisted of too much text. Some of theadolescents that tested NettOpp stated that the app had crashed and that the design was a bit childish.
Conclusions: All in all, this study indicates that Opp and NettOpp have good usability and that adolescents are satisfied withboth apps. It also indicates that the potential goal achievement of the apps, for example, increasing knowledge about mental health(Opp) or cyberbullying (NettOpp) is promising. While there are some comments from the users that are more difficult to solve(eg, Opp is too text-based), some comments helped improve the apps (eg, that the app crashed). Overall, the user evaluation
provided valuable knowledge about how adolescents assess Opp and NettOpp. However, more extensive effectiveness studies
are necessary to measure their actual goal achievement.
(JMIR Form Res 2023;7:e40773) doi: 10.2196/40773
KEYWORDS
mental health app; mobile app; mental health; mental health promotion; cyberbullying; adolescents; user satisfaction; systemusability; app quality; mental health intervention; health promotion; app usability; user experience

https://formative.jmir.org/2023/1/e40773

XSL• FO
RenderX

JMIR Form Res 2023 | vol. 7 | e40773 | p. 1
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Introduction
Background
Adolescence is a crucial time in life to develop healthy habitsthat can improve both physical and mental health in later life[1]. Moreover, evidence suggests that adolescents who are
affected by mental health issues at an early age often experience
negative consequences, such as mental disorders, in later adultlife [2]. This makes it important to work toward ensuringadolescents have good conditions for a healthy upbringing [3].
However, there are many young people who are struggling.
Prevalence rates show that around 10%-20% of the world’s
youth population is affected by mental health problems [4]. It
is therefore important to develop and implement effective
preventive tools for adolescents. In today’s digital world, it has
become increasingly popular to offer young people preventive
interventions through mobile apps to improve health outcomes(ie, mobile health apps) [5].
In general, mental health promotion aims at increasing
resilience, self-esteem, and coping skills [3]. Traditionally, mostof the mental health promotion interventions that exist forchildren and adolescents are conducted in kindergarten or
school, and it is conceivable that mental health promotion
interventions should take place in areas where adolescents spend
most of their time. Yet, there is another place where adolescents
spend much of their spare time, specifically the internet. Firstand foremost, adolescents spend a lot of their time on social
media sites such as YouTube, TikTok, and Instagram [6], butthey also use the internet for information-seeking purposes [7].
Using web-based services to provide adolescents with mental
health interventions may thus have great potential for preventive
health care, as these services can reach a large proportion ofyoungsters, regardless of where they live. The services are also
cost-effective and appeal to young people [8].
Even though young people’s extensive use of social media andonline services enables opportunities for socialization,entertainment, and rapid information retrieval, adolescents may
also experience negative online events, such as mean comments,
exclusion from events and groups, or threats [9]. These events
can have severe consequences for adolescents’ well-being[10,11] and are related to serious mental health problems, suchas depression, anxiety, and suicidal ideation [9,12,13]. For thisreason, there is a need for intervention strategies that can havea reducing effect on negative online events [14] and increaseadolescents’ coping skills so that they can better respond tonegative online events. However, in Norway and in many otherhttps://formative.jmir.org/2023/1/e40773

XSL• FO
RenderX

Høgsdal et al
countries, there is a lack of evidence-based interventions to
support adolescents who are exposed to negative online incidents[15]. Giving young people access to information on how to deal
with negative online events through a mobile app may be asolution that adolescents appreciate.
A quick search on Google Play or the App Store shows that
new apps that aim to improve young people’s mental health areconstantly being developed. However, the apps that are on themarket are of varying quality [16], and there is a broad
agreement that more research is needed into the effect of mental
health apps and the extent to which they can be a useful tool
for mental health promotion and prevention among adolescents[17-19]. It is therefore important that the mental healthinterventions delivered through apps are evidence-based and ofhigh quality. In Norway, the Regional Centre for Child andYouth Mental Health in the North has developed 2 mental
health–promoting mobile apps for adolescents aged 11 to 19
years, called Opp and NettOpp. The apps are educational andinformative self-help tools that can support and help young
people who face various challenges. The aim of this study is to
examine how adolescents evaluated Opp and NettOpp througha user survey.

Opp
Opp (Figure 1) is a universal mental health–promoting mobile
app for adolescents aged between 13 and 19 years. The app wasdeveloped in collaboration with adolescents in the target groupas well as professionals who work with young people and mental
health (ie, nurses, psychologists, and social workers). The
overall aim of Opp is to increase adolescents’ well-being, mental
health, and coping skills so that they can better cope with stress.
The app also aims to reduce mental health problems among
adolescents. The app consists of 2 modules. The first moduleis psychoeducational, with information about mental health andfeelings and details of where adolescents can seek help (eg,parents, teachers, or online resources such as the Red Cross andthe emergency telephone line for children and adolescents). The
second module is a resource module that provides information,
exercises, and techniques for how to better cope with stress anddifficult thoughts and emotions. These exercises include
breathing exercises (visual breathing animation), sound-recorded
sleep hygiene advice, mindfulness and mentalizing exercises,
and exercises inspired by cognitive behavioral therapy(recognizing and challenging negative thoughts). Information
is displayed in the app through text, sound recordings, andanimations.

JMIR Form Res 2023 | vol. 7 | e40773 | p. 2
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Høgsdal et al

Figure 1. Start page of Opp.

NettOpp
NettOpp (Figure 2) is a mobile app designed for adolescents
aged between 11 and 16 years to help them cope with negative
online events. The app was developed in collaboration withadolescents in the target group as well as professionals who
work with young people and bullying (ie, teachers, school healthnurses, community psychologists, and the local antibullying
professional). The overall aims of NettOpp are to reduce distressrelated to negative online incidents (eg, cyberbullying) and tohttps://formative.jmir.org/2023/1/e40773

XSL• FO
RenderX

increase adolescents’ knowledge of and ability to deal withadverse events online. NettOpp consists of 2 modules. The firstmodule is psychoeducational, with information about negative
online incidents (eg, cyberbullying), emotions one may
experience related to adverse online events, and information
about where and how to seek help. The other module is a
resources module, which provides both exercises andstress-related coping techniques. The intervention is describedin Kaiser et al [20].

JMIR Form Res 2023 | vol. 7 | e40773 | p. 3
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Høgsdal et al

Figure 2. Start page of NettOpp.

Objective of This Study
This study examines how adolescents evaluate the usability andquality of Opp and NettOpp, as well as their potential to achieve
their given objectives. The study aims to respond to thefollowing research questions:•

RenderX•

RQ2: How do young people assess the quality of Opp and
NettOpp?
RQ3: How do young people assess Opp and NettOpp’s
potential to achieve their goals?

RQ1: Are young people satisfied with the system usabilityof Opp and NettOpp?

https://formative.jmir.org/2023/1/e40773

XSL• FO•JMIR Form Res 2023 | vol. 7 | e40773 | p. 4
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Methods
Participants and Procedure
A total of 45 adolescents within the age range of 11 to 19 years
were recruited in 2021 to participate in 2 cross-sectional user
surveys. The participants were recruited through the authors’colleagues and acquaintances. A total of 30 adolescents (n=20
girls and n=10 boys) evaluated Opp, and 15 adolescents (n=6girls and n=9 boys) evaluated NettOpp. For Opp, 3 (10%)
adolescents stated that they had not used the app at all and were
therefore excluded from the analyses. Hence, the resultspresented are based on 27 adolescents (n=8 boys and n=19 girls)
evaluating Opp. After a trial period of approximately 3 weeks,
the adolescents who evaluated Opp answered an online
questionnaire in Nettskjema [21], while the adolescents who
evaluated NettOpp answered a paper-based questionnaire.

Measures
System Usability
The usability of the apps was measured using the System
Usability Scale (SUS) [22] with modified SUS statements [23].
The scale consists of 10 items (eg, “I thought the product waseasy to use” and “I found the product unnecessarily complex”).
Responses were rated on a 5-point scale from 0 (strongly
disagree) to 4 (strongly agree). SUS scores were summarized
before being multiplied by 2.5 to transform the SUS score intoa 0 (low perception of usability) to 100 (high perception ofusability) interval scale. SUS scores may be interpreted using
school-grade ratings. The guidelines for interpreting the SUS
scores are D, which ranges from 60 to 70, C from 70 to 80, B
from 80 to 90, and A from 90 to 100 [24]. The SUS wastranslated into Norwegian by the second and third authors. The
adolescents could also comment on the apps’ system usability
in a supplementary open comment field at the end of thequestionnaire.

App Quality
The quality of the apps was assessed with items from the quality
rating section of the Mobile Application Rating Scale (MARS)[25]. Originally, the MARS’ quality rating section consisted of
23 items divided into 4 subscales to assess engagement (section
A), functionality (section B), aesthetics (section C), andinformation (section D), and 1 subscale to assess subjective
quality ratings (section E). To adapt the scale to adolescents,
we only included questions that we anticipated were easy toanswer for young people and that we believed would give us agood indication of important aspects of the quality of the apps.
Hence, we included and adapted 10 items from the original
scale, of which 7 items are from sections A-D (eg, “Howaccurately do the app’s features and components work?”) and3 items are from section E (eg, “What is your overall star ratingof the app?”). All items were rated on a 5-point scale withdifferent response categories from 1 (App is broken and I would
not recommend this app to anyone) to 5 (Perfect/timely response
and one of the best apps I’ve used). The adolescents could alsocomment on the apps’ quality in a supplementary open comment
field at the end of the questionnaire.

https://formative.jmir.org/2023/1/e40773

XSL• FO
RenderX

Høgsdal et al

Potential Goal Achievement
To measure how the adolescents evaluated the apps’ potential
to achieve their given objectives, the app-specific rating section
from the MARS [25] was used for the adolescents who evaluated
NettOpp. The app-specific section in MARS consists of 6 items
that can be used to access the perceived impact of an app on
users’ awareness, knowledge, attitudes, intentions to change,
and the likelihood of actual behavior change. The items were
adjusted to match the target health behavior for the app. Example
items for NettOpp are “NettOpp is likely to increase awarenessof the importance of addressing cyberbullying” or “The use of
NettOpp is likely to encourage seeking further help after being
exposed to cyberbullying.” The adolescents who evaluated Opp
were given more adapted questions (inspired by the app-specificsection in MARS) about each health goal of the app. Example
items from Opp are “Opp is likely to increase well-being andmental health” or “Opp is likely to motivate young people tospeak out or ask for help when they need it.” The adolescents
rated their answers on a 5-point scale from 1 (strongly disagree)
to 5 (strongly agree). The adolescents could also comment on
the apps’ potential goal achievement in a supplementary opencomment field at the end of the questionnaire.

Ethics Approval
The evaluation studies were approved by the Norwegian Centre
for Research Data (reference for Opp: 450518; reference for
NettOpp: 545417). Active parental consent from 1 guardian
was required for the adolescents to participate in the study. The
consent was retrieved in a digital consent form using Nettskjema[21], where the guardian declared that they understood the
purpose of the study and what was required of their child if they
participated. The adolescents’ answers were collectedanonymously. All participants were compensated with a cinema
gift card with a value of NOK 150 (US $15.03). This study did
not need approval from the Regional Committees for Medicaland Health Research Ethics as no health-related information
was collected.

Statistical Analyses
All statistical analyses were performed in SPSS version 27 (IBM
Corp). Descriptive statistics were calculated and included means,standard deviations, and frequency distributions. An independent
sample t test was conducted to investigate the difference in theoverall star evaluation of the apps between Opp and NettOpp
users.

Results
System Usability of Opp and NettOpp
The overall mean SUS scores of Opp and NettOpp were 80.37(SD 9.27) and 80.33 (SD 10.30), respectively. An overview ofadolescents who agreed with statements in the SUS is presented
in Table 1. None of the adolescents who evaluated Opp (0/27,0%) reported that the app was difficult to use; 4% (1/27) statedthat they needed to learn a lot before they could use Opp, and
33% (9/27) thought that they would use the app frequently.
Furthermore, none of the adolescents (0/15, 0%) who evaluated
NettOpp stated that the app was difficult to use, 33% (5/15)
reported that they would use the app frequently, andJMIR Form Res 2023 | vol. 7 | e40773 | p. 5
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
approximately 7% (1/15) stated that they needed to learn a lot

Høgsdal et al
before they could use the app.

Table 1. Percentage of adolescents who answered that they strongly agree or agree with claims about NettOpp and Opp’s system usability.
Opp (n=27), n (%)

NettOpp (n=15), n (%)

I think I would like to use this product frequently

9 (33)

5 (33)

I found the product unnecessarily complex

1 (4)

0 (0)

I thought the product was easy to use

25 (93)13 (87)

I think that I would need the support of a technical person to be able to use this product

0 (0)

0 (0)

I found the various functions in the product were well integrated

22 (82)14 (93)

I thought there was too much inconsistency in this product

0 (0)

0 (0)

I imagine that most people would learn to use this product very quickly

26 (96)15 (100)

I found the product very awkward to use

2 (7)

0 (0)

I felt very confident using the product19 (70)11 (73)

I needed to learn a lot of things before I could get going with this product

1 (4)

1 (7)

The Quality of Opp and NettOpp
Descriptive results from the adolescents’ evaluation of thequality of the apps are presented in Table 2. A large proportionof the adolescents thought that Opp was interesting to use(20/27, 74%), that the app functioned well (26/27, 96%), thatthe visual appeal was good (25/27, 93%), and that the app was
appropriate for the target audience (24/27, 89%). Furthermore,a large proportion of the adolescents thought that NettOpp wasinteresting to use (12/15, 80%), that the app content wasappropriate for the target group (14/15, 93%), and that the visualappeal was good (12/15, 80%). They also stated that they would
recommend NettOpp to anyone who might benefit from it(13/15, 87%). There was a statistically significant difference inthe overall star evaluation between adolescents testing NettOpp(mean 4.20, SD 0.56) and Opp (mean 3.78, SD 0.42; t40=2.75;
P=.01).

Table 2. Descriptive statistics of the quality ratings for the mobile apps NettOpp and Opp.
Opp (n=27)

NettOpp (n=15)

Min-max

Mean (SD)Min-max

Mean (SD)Is the app interesting to use?

3-5

3.78 (0.51)

3-5

4.00 (0.65)

Is the app content appropriate for the target audience?

3-5

4.37 (0.68)

3-5

4.47 (0.64)

How accurately or fast do the app features and components work?

3-5

4.67 (0.55)

1-5

4.13 (1.25)

How easy is it to learn how to use the app?

3-5

4.48 (0.65)

3-5

4.53 (0.64)

Is arrangement and size of buttons or icons (…) on the screen appropriate or zoomable?

2-5

4.37 (0.79)

4-5

4.53 (0.52)

How good does the app look?

2-5

4.15 (0.66)

3-5

4.13 (0.74)

Does the app contain what is described?

3-5

4.26 (0.17)

4-5

4.80 (0.41)

Would you recommend this app to people who might benefit from it?

2-5

3.68 (0.84)

1-5

4.33 (0.74)

How many times do you think you would use this app in the next 12 months?

1-5

3.33 (1.04)

1-5

3.47 (1.25)

What is your overall star rating of the app?

3-4

3.78 (0.42)

3-5

4.20 (0.56)

Potential Goal Achievement of Opp and NettOpp
Tables 3 and 4 provide an overview of the percentages ofadolescents who agreed with statements regarding Opp and
NettOpp’s ability to achieve their goals, respectively. A large
percentage agreed that Opp would increase knowledge about
mental health (25/27, 92%) and help young people deal withstress, difficult emotions, or situations (18/27, 67%).
Furthermore, 52% (14/27) thought Opp would increasewell-being and mental health (11/27, 41% agreed to somehttps://formative.jmir.org/2023/1/e40773

XSL• FO
RenderX

extent), 37% (10/27) thought that Opp would reduce mental
health problems among adolescents (14/27, 52% agreed to someextent), and 33% (9/27) thought Opp would motivate young
people to seek help if they needed it (16/27, 59% agreed to someextent). A large percentage (n=11-15, 73%-100%) of the 15
adolescents who evaluated NettOpp agreed that the app could
increase awareness and knowledge about cyberbullying, attitudes
toward cyberbullying, and motivation to address cyberbullying.
Only 33% (5/15) thought that NettOpp could reducecyberbullying among adolescents.

JMIR Form Res 2023 | vol. 7 | e40773 | p. 6
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Høgsdal et al

Table 3. Proportion of adolescents answering that they strongly agree or agree with statements about Opp’s potential to achieve its goals.
Opp will probably...

Participants (n=27), n (%)

Increase well-being and mental health

14 (52)

Increase knowledge about mental health

25 (92)

Help young people deal with stress, difficult emotions, or situations in a good way

18 (67)

Motivate young people to speak out or ask for help when they need it

9 (33)

Make young people feel better about themselves

14 (52)

Decrease mental health problems

10 (37)

Table 4. Proportion of adolescents who answer that they strongly agree or agree with statements about NettOpp’s potential to achieve its goals.
NettOpp will probably...

Participants (n=15), n (%)

Increase awareness of the importance of addressing cyberbullying

15 (100)

Increase knowledge and understanding of cyberbullying

14 (93)

Change attitudes toward improving reactions of cyberbullying

11 (73)

Increase intentions or motivation to address cyberbullying

13 (87)

Encourage further help seeking after being exposed to cyberbullying

14 (93)

Decrease cyberbullying

5 (33)

User Comments on the Apps’ Usability and Quality
In total, 7 adolescents left a comment about Opp and 5 left acomment about NettOpp in the open comment field (allcomments are translated and presented in Textbox 1). The

https://formative.jmir.org/2023/1/e40773

XSL• FO
RenderX

comments from the young people who tested Opp concerned
design, app functions, and quality. A total of 3 adolescents wholeft a comment about NettOpp stated that the app had crashedor shut down while in use.

JMIR Form Res 2023 | vol. 7 | e40773 | p. 7
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Høgsdal et al

Textbox 1. Supplementary comments from the adolescents in both the Opp and NettOpp samples.
Opp•

I will not use the app again because I currently have nothing that bothers me and I bet it will stay that way, but I will probably use it if I start tostruggle mentally.

•

I really like that the little figure praises you when you finish an exercise.

•

The app is probably best suited for children who are 10-14 years old, for example, if they struggle to talk to their parents. If the app is intended
for all adolescents, the app must be adapted to older adolescents as well.

•

Maybe a little too much text but that is fine.

•

Very nice app! But I think it was difficult to get an overview of the main menu and what the app contained. It seemed more like you pressedsomething, and you also moved on, but it was difficult to come back.

•All the exercises with audio files are very well formulated and informative, but difficult to use. The biggest problem is that you cannot listen to
the audio file while the phone is in sleep mode (the screen is off). It was disturbing for the experience of the exercises when there was aninterruption since the mobile phone automatically switches off and the sound from Opp is cut off. Functions such as listening to sounds such asmusic or podcasts in Spotify, etc is desired – so you can listen to the audio file or videos even if the phone automatically shuts off the screen.

•

There could have been more exercises, for example, relaxation exercises.

•

Feeling empty should have been an alternative, missed it when I was tired and did not really feel anything.

NettOpp•

The app was easy to use. I never needed help from my parents to use the app. The quiz was good. Breathing technique exercise was good. I havetested them several times.

•

The app was very user-friendly. Very good to point out that experiencing cyberbullying is not your fault. The app’s main menu may possibly
feel a bit childish, but I liked the layout.

•

I think it is a very good app with many features that are easy to use. Even though I do not need it, I think those who use it will be happy.

•

A bit too much text and will think that for some it will take away the motivation, and they cannot bear to read everything. Still feels that everythingthat is written is relevant. Technical problem: The app automatically closes several times while in use. I can go into it right afterward and it works
as it should, but after a while is “thrown out” again. I think there is a nice common thread throughout the app, and I like the suggestions thatcome up along the way.

•

The menu button in the left corner can be a little more visible, it was a little difficult to see. Once you have taken the quiz, you will not see results,
you can only choose to return to previous questions or close the page. Had been nice to see results and get a “prize.” The test app crashes all thetime. The lock icon on thought clearing was confusing and made me think that the function was locked and that I could not use it. Very good
design and nice app. Very good idea.

•

Some words were difficult, the app crashes 3 times, PLZ fix it!

•

Try to fix the piece where you are thrown out.

Discussion
Overview
Several health-promoting apps have been developed in recent
years, but only a few of them have been evaluated. Furthermore,studies that evaluate mobile apps are of varying quality [16].
This study is a user survey of 2 mental health-promoting apps,
called Opp and NettOpp. Although a user survey does notsupport conclusions about the effectiveness of the apps in termsof goal achievement, it does provide important information
about how adolescents evaluate the apps’ usability and theiroverall quality. Since online interventions have varying reported
effects on adolescents’ mental health [17], it is also interesting
to know more about what adolescents think a specific app canhelp them with. In this study, young people quality-assured the2 mobile apps NettOpp and Opp, which are intended to promotemental health among adolescents.

https://formative.jmir.org/2023/1/e40773

XSL• FO
RenderX

Principal Results
Overall, the adolescents were satisfied with the system usabilityof both Opp and NettOpp. SUS scores for Opp and NettOpp
corresponded to a B grade on the grade scale and to “excellent”
for the adjective ratings [23]. Usability can be defined as theextent to which “a product can be used by specified users toachieve specified goals with effectiveness, efficiency, andsatisfaction in a specified context of use” [26]. A SUS scoreabove 70 is considered acceptable [23]. Previous research has
found an average SUS rating of 77.3 for the 10 most popular
apps, such as Facebook, YouTube, and iTunes [27], where
mobile apps overall had greater usability than apps adapted toother devices, such as tablets. An examination of each item on
the SUS in this evaluation study showed broad agreement among
the adolescents who evaluated Opp that the app was easy to use
and that most people would learn how to use the app.
Furthermore, the adolescents who evaluated NettOpp agreed
that the app was easy to use, that the functions were wellintegrated, and that most people would learn how to use it. A
smaller proportion of the adolescents who tested the apps
JMIR Form Res 2023 | vol. 7 | e40773 | p. 8
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
thought that they would use Opp and NettOpp regularly
themselves.
There was also overall agreement among both the adolescents
who evaluated Opp and those who evaluated NettOpp that theapps were of good quality. In this study, only some of the items
from MARS were included in the questionnaire, so it will not
be possible to compare the results of this evaluation study withprevious studies using the complete MARS quality ratings[28-30]. However, previous research has shown that adolescentsemphasize the importance of mental health apps being engaging
for young people as well as having a useful and relevant function[31]. In this evaluation study, most of the adolescents statedthat both apps were interesting to use and that the content wasappropriate for the target group. The adolescents also stated thatthe apps’ visual appeal was good. In the overall evaluation of
the apps, NettOpp received a higher ranking than Opp. Although
it is difficult to draw conclusions about the practical implicationsof this finding, we expected that the adolescents would prefer
Opp to NettOpp. NettOpp is more static and has a simpler
design, while Opp is more complex, with more functions,animations, and exercises in addition to a more sophisticateddesign. Kenny et al [31], on the other hand, found thatadolescents appreciate apps that are concise and easy tonavigate. In addition, study participants commented that Opp
was difficult to navigate. Another reason for the higher ranking
of NettOpp might be that this app is specifically targeted atadolescents who might have challenges related to negative
online events, while Opp is a universal mental health–promoting
mobile app for all adolescents. NettOpp might therefore beperceived as more helpful.
As both apps aim to promote the mental health of adolescents,
we sought to assess the adolescents’ opinion of whether theapps would have the ability to achieve their given objectives.
Among the adolescents who evaluated Opp, there was broad
agreement that the app would increase adolescents’ knowledgeof mental health. Previous research has shown that teaching
young people about mental health may be an effective preventive
measure, which can also contribute to reducing the stigma
related to mental disorders [32,33]. Previous studies have also
shown that strengthening adolescents’ knowledge of mental
health may increase help-seeking behavior [34,35]. Motivating
adolescents to seek help when they need it is important because
it can prevent the development of disorders through earlyinterventions. This evaluation study showed that a large
proportion of the adolescents agreed to some extent that Opp
could motivate adolescents to speak out or ask for help when
they needed it. However, a smaller proportion of the sample
thought Opp could reduce mental health problems among
adolescents. This may indicate that the adolescents see mental
health problems as a more complex problem that cannot be
solved, for example, by increasing the adolescents’ knowledge
or motivating them to seek help.
The sample that tested NettOpp thought that the app would
increase awareness and knowledge of negative online incidents.
Previous research has illustrated the importance of increasing
knowledge and awareness about negative online incidents toensure that both adolescents and people who work closely withthe group (eg, teachers) are capable of handling the issue in an
https://formative.jmir.org/2023/1/e40773

XSL• FO
RenderX

Høgsdal et al
effective way [36]. Furthermore, previous research shows thatadolescents who experience negative online incidents are often
reluctant to seek help [37,38]. As adolescents who experience
negative online incidents more often experience mental healthissues, it is important to motivate them to seek help. In thisevaluation study, the adolescents thought NettOpp would
increase help-seeking behavior among adolescents when theyexperienced negative online incidents. There was, however, asmall percentage of the sample who thought NettOpp would
reduce negative online incidents. Previous research supports
the youth’s evaluation regarding the aim of reducing negative
online incidents, as this is a comprehensive problem and simple
intervention is not enough to change it [39]. Reducing
cyberbullying often requires extensive cooperation between
various agencies, such as school workers and parents. However,as cyberbullying can have severe consequences for adolescents’mental health and interventions to prevent psychological distressare greatly needed, NettOpp may be a helpful tool. The
adolescents’ evaluations indicate that NettOpp can give themaccess to information to increase young people’s coping skills
when they face negative online incidents, and this may be aneffective preventive measure.

Limitations
Important limitations to address are related to the sample sizes
and the sample itself that evaluated the apps, as well as to thecross-sectional design of the study. The 2 sample sizes wererelatively small and not representative, so the findings of thisstudy cannot be generalized to all adolescents in Norway.
However, for the purpose of this study, we think that the sample
sizes were sufficient. Some of the adolescents who evaluatedthe apps were also involved in developing the apps. This might
have influenced their assessment in that they might, for example,
have been more positively tuned when answering the questions.
However, since study participation was anonymous, we cannot
examine if it actually influenced the participants. The
participants in the evaluation study were also compensated witha cinema gift card with a value of NOK 150 (US $15.03), which
may have influenced the motivation of the study participants.
Further, the age range of the sample who evaluated the apps
was between 11 and 19 years. NettOpp’s target group is children
and adolescents between the ages of 11 and 16 years, and Opp’s
target group is adolescents aged between 13 and 19 years. Oneof the adolescents left a comment saying that Opp was best
suited for adolescents aged between 10 and 14 years, and anotheradolescent left a comment saying that NettOpp was a bit
childish. These comments may be due to the fact that theevaluators were older than the intended target group. However,
the adolescents’ exact ages were not collected, which is alimitation for both this study and further work with the app.
Another possible explanation is that some young people consider
the design of the apps to be childish because they are moresophisticated than the designers and developers had anticipated.
As the aim of the study was to examine how adolescents assessthe usability and quality of 2 mental health mobile apps, across-sectional design was justified as it allowed for an
impression of a given topic at a given time point. However, across-sectional design does not allow for drawing conclusions
JMIR Form Res 2023 | vol. 7 | e40773 | p. 9
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
about the long-term use of the apps or about their actual
effectiveness.
Another limitation of this evaluation study is related to the apps’
functionality during the trial period. Comments were received
for both Opp and NettOpp. Some adolescents found it difficult
to navigate Opp. Several of the adolescents who evaluated
NettOpp reported that the app crashed while they were using
it. This is a frequent problem reported among app users [40]. It
is therefore possible that these problems affected theadolescents’ experience with the 2 apps. However, because of
the feedback on NettOpp’s functional problems, the app was
updated and fixed.
A small proportion of adolescents who tested Opp thought theapp could reduce mental health problems among adolescents.
Only a few of the adolescents who evaluated NettOpp thought
the app could reduce negative online incidents. It was notcontrolled in either the Opp sample or the NettOpp sample
whether the adolescents had experienced mental health issues

Høgsdal et alor negative online incidents. This was because we were mainlyinterested in mapping app usability and quality before they were
evaluated in wider effectiveness studies.

Conclusions
This user evaluation study has provided helpful and important
knowledge about how adolescents assess the apps’ usability,
quality, and potential to achieve their given objectives. Overall,
the findings are positive and promising. They suggest thatmental health–promoting mobile apps offer a practical, relatively
cheap, and appealing approach to reaching adolescents andimproving their mental health. However, the cross-sectional
design of the study does not allow for drawing any conclusions
about the long-term use of such apps among adolescents. It istherefore necessary to conduct further studies that examine theuse of mental health–promoting mobile apps in the long run.
This study does not provide answers about the effectiveness ofthe 2 apps. It is therefore also necessary to conduct wider
effectiveness studies for both apps in order to draw any
conclusions on their actual goal achievement.

Acknowledgments
We would like to thank the participants in this study. We would also like to thank our dear colleagues: Monica Martinussen,
Frode Adolfsen, Marte Rye, Kyrre Breivik, and Reidar Jakobsen, who contributed to the development of the mobile apps. The
development of the mobile app Opp was funded by a grant from the Dam Foundation through the Norwegian Council of Mental
Health (Ref. 2020HE2-328433). The study was funded by RKBU North at UiT The Arctic University of Norway, with an additional
grant from the Norwegian Directorate of Health. The publication charges for this article have been funded by a grant from thepublication fund of UiT The Arctic University of Norway.

Data Availability
The data sets generated during or analyzed during this study are available from the corresponding author on reasonable request.

Authors' Contributions
All authors provided substantial contributions to the evaluation study. HH wrote the first draft of the manuscript. SK and HK
read, edited, and approved the final manuscript.

Conflicts of Interest
None declared.

References
1.

2.

3.
4.

5.
6.

Guidelines on mental health promotive and preventive interventions for adolescents: helping adolescents thrive. World
Health Organization. URL: https://apps.who.int/iris/bitstream/handle/10665/336864/9789240011854-eng.pdf [accessed2022-07-05]
Kim-Cohen J, Caspi A, Moffitt TE, Harrington H, Milne BJ, Poulton R. Prior juvenile diagnoses in adults with mental
disorder: developmental follow-back of a prospective-longitudinal cohort. Arch Gen Psychiatry 2003 Jul;60(7):709-717.
[doi: 10.1001/archpsyc.60.7.709] [Medline: 12860775]
Barry M, Clarke A, Petersen I, Jenkins R, editors. Implementing Mental Health Promotion. Cham: Springer International
Publishing; 2019.
Kieling C, Baker-Henningham H, Belfer M, Conti G, Ertem I, Omigbodun O, et al. Child and adolescent mental healthworldwide: evidence for action. Lancet 2011;378(9801):1515-1525. [doi: 10.1016/S0140-6736(11)60827-1] [Medline:
22008427]
Powell AC, Landman AB, Bates DW. In search of a few good apps. JAMA 2014 May 14;311(18):1851-1852. [doi:10.1001/jama.2014.2564] [Medline: 24664278]
Children and media 2020: a survey of 9-18-year-olds' digital media habits. The Norwegian Media Authority. 2020 Oct 01.
URL: https://www.medietilsynet.no/globalassets/publikasjoner/barn-og-medier-undersokelser/2020/201015-barn-og-medier-2020-hovedrapport-med-engelsk-summary.pdf [accessed 2022-07-05]

https://formative.jmir.org/2023/1/e40773

XSL• FO
RenderX

JMIR Form Res 2023 | vol. 7 | e40773 | p. 10(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
7.
8.
9.

10.
11.
12.

13.
14.
15.

16.
17.
18.
19.
20.

21.
22.
23.
24.
25.

26.
27.
28.

29.

30.

31.
32.

Kim K, Sin SJ, Tsai T. Individual differences in social media use for information seeking. J Acad Librariansh 2014
Mar;40(2):171-178. [doi: 10.1016/j.acalib.2014.03.001]
Olff M. Mobile mental health: a challenging research agenda. Eur J Psychotraumatol 2015;6:27882 [FREE Full text] [doi:10.3402/ejpt.v6.27882] [Medline: 25994025]
Moore SE, Norman RE, Suetani S, Thomas HJ, Sly PD, Scott JG. Consequences of bullying victimization in childhood
and adolescence: a systematic review and meta-analysis. World J Psychiatry 2017 Mar 22;7(1):60-76 [FREE Full text][doi: 10.5498/wjp.v7.i1.60] [Medline: 28401049]
Giumetti GW, Kowalski RM. Cyberbullying via social media and well-being. Curr Opin Psychol 2022 Jun;45:101314.
[doi: 10.1016/j.copsyc.2022.101314] [Medline: 35313180]
Przybylski AK, Bowes L. Cyberbullying and adolescent well-being in England: a population-based cross-sectional study.
Lancet Child Adolesc Health 2017 Sep;1(1):19-26. [doi: 10.1016/S2352-4642(17)30011-1] [Medline: 30169221]
Kowalski RM, Giumetti GW, Schroeder AN, Lattanner MR. Bullying in the digital age: a critical review and meta-analysis
of cyberbullying research among youth. Psychol Bull 2014 Jul;140(4):1073-1137. [doi: 10.1037/a0035618] [Medline:
24512111]
Kaiser S, Kyrrestad H, Fossum S. Cyberbullying status and mental health in Norwegian adolescents. Scand J Psychol 2020
Oct;61(5):707-713. [doi: 10.1111/sjop.12656] [Medline: 32592170]
Stauffer S, Heath MA, Coyne SM, Ferrin S. High school teachers' perceptions of cyberbullying prevention and intervention
strategies. Psychol. Schs 2012 Mar 01;49(4):352-367. [doi: 10.1002/pits.21603]
Ulriksen R, Knapstad M. Digital mobbing. Kunnskapsoversikt over forskning på effekter av tiltak. Norwegian Institute of
Public Health. 2016. URL: https://www.fhi.no/publ/2016/
digital-mobbing.-kunnskapsoversikt-over-forskning-pa-effekter-av-tiltak/ [accessed 2022-07-05]
Anthes E. Mental health: there's an app for that. Nature 2016 Apr 07;532(7597):20-23. [doi: 10.1038/532020a] [Medline:
27078548]
Clarke AM, Kuosmanen T, Barry MM. A systematic review of online youth mental health promotion and prevention
interventions. J Youth Adolesc 2015 Jan;44(1):90-113. [doi: 10.1007/s10964-014-0165-0] [Medline: 25115460]
Punukollu M, Marques M. Use of mobile apps and technologies in child and adolescent mental health: a systematic review.
Evid Based Ment Health 2019 Nov;22(4):161-166. [doi: 10.1136/ebmental-2019-300093] [Medline: 31358537]
Grist R, Porter J, Stallard P. Mental health mobile apps for preadolescents and adolescents: a systematic review. J MedInternet Res 2017 May 25;19(5):e176 [FREE Full text] [doi: 10.2196/jmir.7332] [Medline: 28546138]
Kaiser S, Martinussen M, Adolfsen F, Breivik K, Kyrrestad H. An app-based intervention for adolescents exposed tocyberbullying in Norway: protocol for a randomized controlled trial. JMIR Res Protoc 2021 Nov 08;10(11):e31789 [FREEFull text] [doi: 10.2196/31789] [Medline: 34747704]
Nettskjema. Universitetet I Oslo. URL: https://www.uio.no/tjenester/it/adm-app/nettskjema/ [accessed 2022-07-05]
Brooke J. SUS-a quick and dirty usability scale. In: Jordan PW, Thomas B, McClelland IL, Weerdmeester B, editors.
Usability Evaluation In Industry. London: CRC Press; 1996.
Bangor A, Kortum PT, Miller JT. An empirical evaluation of the system usability scale. Int J Hum-Comp Int 2008 Jul
30;24(6):574-594. [doi: 10.1080/10447310802205776]
Bangor A, Kortum P, Miller J. Determining what individual SUS scores mean: adding an adjective rating scale. J Usability
Stud 2009;4(3):114-123. [doi: 10.5555/2835587.2835589]
Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M. Mobile app rating scale: a new tool forassessing the quality of health mobile apps. JMIR mHealth uHealth 2015 Mar 11;3(1):e27 [FREE Full text] [doi:10.2196/mhealth.3422] [Medline: 25760773]
Grassi P, Garcia M, Fenton J. Digital identity guidelines. National Institute of Standards and Technology 2017 Jul 01:1-63.
Kortum P, Sorber M. Measuring the usability of mobile applications for phones and tablets. Int J Hum-Comp Int 2015 Jul
31;31(8):518-529. [doi: 10.1080/10447318.2015.1064658]
Kim BY, Sharafoddini A, Tran N, Wen EY, Lee J. Consumer mobile apps for potential drug-drug interaction check:systematic review and content analysis using the mobile app rating scale (MARS). JMIR mHealth uHealth 2018 Mar
28;6(3):e74 [FREE Full text] [doi: 10.2196/mhealth.8613] [Medline: 29592848]
Escriche-Escuder A, De-Torres I, Roldán-Jiménez C, Martín-Martín J, Muro-Culebras A, González-Sánchez M, et al.
Assessment of the quality of mobile applications (apps) for management of low back pain using the mobile app rating scale(MARS). Int J Environ Res Public Health 2020 Dec 09;17(24):9209 [FREE Full text] [doi: 10.3390/ijerph17249209][Medline: 33317134]
Salazar A, de Sola H, Failde I, Moral-Munoz JA. Measuring the quality of mobile apps for the management of pain:
systematic search and evaluation using the mobile app rating scale. JMIR mHealth uHealth 2018 Oct 25;6(10):e10718[FREE Full text] [doi: 10.2196/10718] [Medline: 30361196]
Kenny R, Dooley B, Fitzgerald A. Developing mental health mobile apps: exploring adolescents' perspectives. J Health
Inform 2016 Jun;22(2):265-275 [FREE Full text] [doi: 10.1177/1460458214555041] [Medline: 25385165]
Chandra A, Minkovitz CS. Factors that influence mental health stigma among 8th grade adolescents. J Youth Adolescence
2006 Jul 25;36(6):763-774. [doi: 10.1007/s10964-006-9091-0]

https://formative.jmir.org/2023/1/e40773

XSL• FO
RenderX

Høgsdal et al

JMIR Form Res 2023 | vol. 7 | e40773 | p. 11(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
33.
34.

35.

36.
37.
38.

39.
40.

Høgsdal et al

Naylor PB, Cowie HA, Walters SJ, Talamelli L, Dawkins J. Impact of a mental health teaching programme on adolescents.
Br J Psychiatry 2009 Apr;194(4):365-370. [doi: 10.1192/bjp.bp.108.053058] [Medline: 19336791]
Radez J, Reardon T, Creswell C, Lawrence PJ, Evdoka-Burton G, Waite P. Why do children and adolescents (not) seek
and access professional help for their mental health problems? A systematic review of quantitative and qualitative studies.
Eur Child Adolesc Psychiatry 2021 Feb;30(2):183-211 [FREE Full text] [doi: 10.1007/s00787-019-01469-4] [Medline:
31965309]
Lindow JC, Hughes JL, South C, Minhajuddin A, Gutierrez L, Bannister E, et al. The youth aware of mental healthintervention: impact on help seeking, mental health knowledge, and stigma in U.S. adolescents. J Adolesc Health 2020
Jul;67(1):101-107 [FREE Full text] [doi: 10.1016/j.jadohealth.2020.01.006] [Medline: 32115325]
Wright V, Burnham J, Christopher T, Heather N. Cyberbullying: using virtual scenarios to educate and raise awareness.
JCTE 2009;26(1):35-42. [doi: 10.1080/10402454.2009.10784629]
Gustainiene L, Valiune D. Comparison of attitudes towards help seeking between schoolchildren with and without experienceof cyberbullying. Procedia Soc Behav Sci 2015 Oct;205:391-395. [doi: 10.1016/j.sbspro.2015.09.022]
Spears BA, Taddeo CM, Daly AL, Stretton A, Karklins LT. Cyberbullying, help-seeking and mental health in young
Australians: implications for public health. Int J Public Health 2015 Feb;60(2):219-226. [doi: 10.1007/s00038-014-0642-y][Medline: 25572385]
Sabella RA, Patchin JW, Hinduja S. Cyberbullying myths and realities. Comput Hum Behav 2013 Nov;29(6):2703-2711.
[doi: 10.1016/j.chb.2013.06.040]
Khalid H, Shihab E, Nagappan M, Hassan AE. What do mobile app users complain about? IEEE Softw 2015 May;32(3):70-77.
[doi: 10.1109/MS.2014.50]

Abbreviations
MARS: Mobile Application Rating Scale
SUS: System Usability Scale

Edited by A Mavragani; submitted 05.07.22; peer-reviewed by J Zlamal, P Matthews; comments to author 19.10.22; revised version
received 03.11.22; accepted 03.11.22; published 06.01.23
Please cite as:
Høgsdal H, Kaiser S, Kyrrestad H
Adolescents’ Assessment of Two Mental Health–Promoting Mobile Apps: Results of Two User Surveys
JMIR Form Res 2023;7:e40773
URL: https://formative.jmir.org/2023/1/e40773
doi: 10.2196/40773
PMID:©Helene Høgsdal, Sabine Kaiser, Henriette Kyrrestad. Originally published in JMIR Formative Research(https://formative.jmir.org), 06.01.2023. This is an open-access article distributed under the terms of the Creative Commons
Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction
in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete
bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license
information must be included.

https://formative.jmir.org/2023/1/e40773

XSL• FO
RenderX

JMIR Form Res 2023 | vol. 7 | e40773 | p. 12(page number not for citation purposes)