JMIR FORMATIVE RESEARCH

Metz et al

Original Paper

Evaluating the Impact of a Dutch Sexual Health Intervention for
Adolescents: Think-Aloud and Semistructured Interview Study
Gido Metz1, PhD; Rosa R L C Thielmann1, PhD; Hanneke Roosjen2; Rik Crutzen1, PhD
1

Department of Health Promotion, Care and Public Health Research Institute, Maastricht University, Maastricht, Netherlands

2

Soa Aids Nederland, Amsterdam, Netherlands

Corresponding Author:
Gido Metz, PhD
Department of Health Promotion
Care and Public Health Research Institute
Maastricht University
PO Box 616
Maastricht, 6200 MD
Netherlands
Phone: 31 433882435
Email: g.metz@maastrichtuniversity.nl

Abstract
Background: Engagement with and the potential impact of web-based interventions is often studied by tracking user behavior
with web analytics. These metrics do provide insights into how users behave, but not why they behave as such.
Objective: This paper demonstrates how a mixed methods approach consisting of (1) a theoretical analysis of intended use, (2)a subsequent analysis of actual use, and (3) an exploration of user perceptions can provide insights into engagement with andpotential impact of web-based interventions. This paper focuses on the exploration of user perceptions, using the chlamydia pageof the Dutch sexual health intervention, Sense.info, as a demonstration case. This prevention-focused platform serves as the main
source of sexual and reproductive health information (and care if needed) for young people aged 12-25 years in the Netherlands.
Methods: First, acyclic behavior change diagrams were used to theoretically analyze the intended use of the chlamydia page.
Acyclic behavior change diagrams display how behavior change principles are applied in an intervention and which subbehaviors
and target behaviors are (aimed to be) influenced. This analysis indicated that one of the main aims of the page is to motivate
sexually transmitted infection (STI) testing. Second, the actual use of the chlamydia page was analyzed with the web analytics
tool Matomo. Despite the page’s aim of promoting STI testing, a relatively small percentage (n=4948, 14%) of the 35,347 transfersfrom this page were to the STI testing page. Based on these two phases, preliminary assumptions about use and impact were
formulated. Third, to further explore these assumptions, a study combining the think-aloud method and semistructured interviews
was executed with 15 young individuals aged 16-25 (mean 20, SD 2.5) years. Template analysis was used to analyze interview
transcripts.
Results: Participants found the information on the Sense.info chlamydia page reliable and would visit it mostly for self-diagnosis
purposes if they experienced potential STI symptoms. A perceived facilitator for STI testing was the possibility to learn about
the symptoms and consequences of chlamydia through the page. Barriers included an easily overlooked link to the STI testing
page and the use of language not meeting the needs of participants. Participants offered suggestions for lowering the thresholdfor STI testing.
Conclusions: The mixed methods approach used provided detailed insights into the engagement with and potential impact of
the Sense.info chlamydia page, as well as strategies to further engage end users and increase the potential impact of the page. We
conclude that this approach, which triangulates findings from theoretical analysis with web analytics and a think-aloud study
combined with semistructured interviews, may also have potential for the evaluation of web-based interventions in general.
(JMIR Form Res 2024;8:e48453) doi: 10.2196/48453
KEYWORDS
web-based intervention; eHealth; engagement; potential impact; mixed methods; evaluation; acyclic behavior change diagram;
web analytics; think-aloud method; web-based; user; chlamydia; behavior change
https://formative.jmir.org/2024/1/e48453

XSL• FO
RenderX

JMIR Form Res 2024 | vol. 8 | e48453 | p. 1
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Introduction
Background
The past few decades have witnessed a rise in the development
and use of web-based interventions, and assessing engagement
and the potential impact of such interventions typicallynecessitates using various methods [1]. For example, unobtrusive
tracking of user behavior (eg, web analytics providing use dataon page transfers or time on page) is extensively used as amethod to measure engagement [2]. However, it has been argued
that web analytics should not be used as the sole source of datain evaluating web-based interventions as they show how, but
not why people behave as such [2-4]. This 1D measurement isparticularly insufficient when potential impact is defined as theintervention users’ interaction with the relevant intervention
content and application of it to their own specific context.
The overarching aim of this paper is to demonstrate how a mixed
methods approach consisting of (1) a theoretical analysis ofintended use, (2) a subsequent analysis of actual use, and (3) aMetz et alfurther exploration of user perceptions, can be used to gain
insight into engagement with and potential impact of web-based
interventions. This demonstration takes place in the context ofthe chlamydia page of the Dutch sexual health intervention
website Sense.info (Figure 1) [5]. This prevention-focused
website serves as the entry point for young people aged 12 to
25 years in the Netherlands for information (and care if needed)regarding the whole spectrum of sexual and reproductive health.
Sense.info has a wide reach: in 2021, the website received
4,358,543 visits from 3,917,923 unique visitors. Focusing on
the Netherlands alone, Sense.info garnered 2,392,844 visitsfrom 2,133,572 unique visitors (although it is possible that thesevisits were carried out by users outside this key audience, we
mention here for reference that the Netherlands had 2,711,378
inhabitants between the ages of 12 and 25 years in 2021; [6]).
In addition to the website, Sense.info actively uses TikTok [7]and Instagram [7] and hosts a podcast [8]. These channels serveboth as independent means of engaging with their target
audience and as a way of directing them to the website for more
information.

Figure 1. Screenshots of the chlamydia page of the Dutch sexual health intervention for adolescents (reproduced from Sense.info [5] with permission
from Soa Aids Nederland) in 2020 (left: smartphone view, right: desktop view).

In the following sections, we will outline the mixed methods
approach and apply it to the Sense.info context. The first 2
phases of the mixed methods evaluation of Sense.info havealready been carried out and reported elsewhere [9]. The resultsof these phases are outlined here because they form the basisof the third phase, which explores end user perceptions and isreported on in this paper.

Analysis of Intended Use
The first phase involves a theoretical analysis of how thedevelopers envisioned the use of the intervention and expected
it to impact behavior change (ie, intended use). This analysis
uses acyclic behavior change diagrams (ABCDs) to visualize
the intervention’s active ingredients, underlying assumptions,and causal and structural relationships [10]. An ABCD consistsof chains of seven links that illustrate (1) which behavior change
principles have been applied—(2) taking into account thehttps://formative.jmir.org/2024/1/e48453

XSL• FO
RenderX

parameters for use—in (3) a specific application in anintervention, which (4) subdeterminants and (5) determinants
have been addressed, and (6) which subbehaviors should be
performed to achieve (7) the target behavior.
For the Sense.info chlamydia page, we labeled each element
present on the page using the taxonomy of behavior change
principles corresponding to the Intervention Mapping approach[11,12]. As shown in the ABCD excerpt for the chlamydia pagein Figure 2, several behavior change principles, such as“consciousness raising,” were applied, targeting determinants
such as risk perception and self-efficacy. This ultimately led tothe subbehaviors of “getting tested” and “seeking treatment,”
and to the target behavior “preventing transmission of sexuallytransmitted infections (STIs).” Based on our theoretical analysis,
which highlighted the emphasis on testing on the chlamydia
page, we assumed that the page would be most relevant tosexually experienced young people who may need STI testing.
JMIR Form Res 2024 | vol. 8 | e48453 | p. 2
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Metz et al

Figure 2. Extract of the ABCD for the Sense.info chlamydia page. The entire ABCD is available in the Open Science Framework Repository [13].
The ABCD visualizes which behavior change principles have been applied—taking into account the parameters for use—in a specific application in anintervention, which subdeterminants and determinants have been addressed, and which subbehaviors should be performed to achieve the target behavior.
ABCD: acyclic behavior change diagram; BCP: behavior change principle; STI: sexually transmitted infection.

Analysis of Actual Use
Next, the actual use of the intervention is examined using web
analytics. Specifically, we aim to identify use patterns that either
match our expectations based on the analysis of intended use
or reveal inconsistencies between intended and actual use. Thesefindings will then lead to assumptions about the potential impactof the intervention, which will be further explored in the next
phase.
Matomo, a web analytics tool, was used to study the use patternsof the Sense.info chlamydia page. In this paragraph, we outline
the key data that formed the basis of our assumptions. Additional
data can be found in a previously published paper [9]. Most
visitors reached the page via a Google search (n=54,066, 54.4%),
with queries particularly focusing on symptoms. Other visitors
reached the page through Sense.info pages (n=37,287, 37.5%);
mainly via a referral page with links to different STIs (<Typesof STIs>; n=21,626, 58%), a page with a more generalexplanation of STIs (<What are STIs?>; n=3020, 8.1%), or via
pages about specific STIs (<Gonorrhea>, n=1454, 3.9%;<Candida infection>, n=1342, 3.6%; and <Bacterial vaginosis>,n=1044, 2.8%). Although the chlamydia page is intended tomotivate STI testing, a relatively small percentage (n=4948,
14%) of the 35,347 transfers from <Chlamydia> were to the
STI testing page. The majority (n=14,493, 41%) of thetransitions were to <Types of STIs>, followed by <Gonorrhea>(n=1979, 5.6%), <Genital warts> (n=1413, 4%), and <What are
STIs?> (n=1413, 4%). Moreover, a high bounce rate (ie, thepercentage of sessions in which visitors left Sense.info after
viewing only the chlamydia page; 79%) and a relatively highexit rate (ie, the percentage of sessions that ended on that page;69%) were reported.
These findings led to several preliminary assumptions about
the use and potential impact of the page on STI testing behavior.
Our first assumption was that visitors may have sought
https://formative.jmir.org/2024/1/e48453

XSL• FO
RenderX

information about STIs out of curiosity, without an immediateneed for STI testing. This would not necessarily be a negative
outcome, as our goal was not to increase the transfer percentage
to the STI testing page, but rather to study the potential impactof the page: did visitors in need of an STI test interact with thecontent relevant to them (eg, read information about symptoms,
click on the link transferring them to the STI test page) and did
they apply it (ie, take an STI test)? Visitors may have correctly
concluded that an STI test was unnecessary or irrelevant to theirsituation.
The second assumption was that visitors had a need for an STI
test after exposure to the intervention, but made an appointment
with their general practitioner (GP) or at the Sense consultation
hour (ie, consultation hours at municipal health centersspecifically focused on the young individuals in the Sense.info
target group) in a manner other than clicking on the link to the
STI test page. In this case, the intervention, although undetectedby web analytics, would have had an impact on the subbehaviorof getting tested and ultimately on the target behavior of
preventing STI transmission.
These first 2 assumptions did not necessarily imply that the low
percentage of traffic to the STI test page was problematic. The
third assumption, however, considered the possibility of alimited impact on the page by stating that visitors may have felt
the need to test for STIs, but did not feel capable of getting
tested and therefore did not click on the link to the STI test page.
In such cases, additional efforts to increase self-efficacy might
be needed.

This Study: A Think-Aloud Study to Further Explore
End Users’ Perceptions
The third and current phase of our approach is a think-aloudstudy combined with semistructured interviews. Our first aim
is to shed more light on the assumptions about engagement andimpact from the user’s perspective. The think-aloud method is
JMIR Form Res 2024 | vol. 8 | e48453 | p. 3
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
useful for understanding cognitive processes and emotional
reactions as participants navigate the intervention and view its
content [1,14,15], and the semistructured interviews will allow
participants to elaborate on the aspects mentioned during theexecution of the think-aloud method. While the combination ofa think-aloud method and semistructured interviews has beenshown to be valuable [16,17], we believe that our think-aloudstudy will further benefit from the previous two analyses ofintended and actual use, as these have provided us withassumptions about the potential impact of the intervention thatcan be further explored using the think-aloud method. The open
nature of this method will allow assumptions to be confirmedor refuted, or new explanations to be added for the use of data.
Our second aim is therefore to explore whether the triangulationof data from the analyses of intended use, actual use, and the
exploration of end user perceptions is indeed valuable inassessing engagement and potential impact.

Methods
Ethical Considerations
This study was approved by the ethics review committee of theFaculty of Health, Medicine, and Life Sciences of Maastricht
University (approval FHML-REC/2021/061). Participants signed
informed consent and provided verbal consent reconfirmation
before the start of the think-aloud procedure. Participants were
reimbursed with a €25 (US $30 at a 1.20 conversion rate in June2021) gift card after completion of the think-aloud and interview
procedure. All transcripts have been deidentified. All materials
used in this study—such as the participant information letter,
informed consent form, and interview protocol—can be foundin the Open Science Framework Repository [13]. The
Consolidated Criteria for Reporting Qualitative Research(COREQ) checklist was used in the reporting of this study [18].

Inclusion Criteria
Inclusion criteria were agreed upon with one of the developersof Sense.info, Soa Aids Nederland. Having had sexualinteractions was a first criterion as it was expected that
<Chlamydia> would be mostly visited by and relevant tosexually active people, and would also be most relevant to thisgroup. Following reports on the age at which young people in
the Netherlands start having sex [19], as a second criterion, we
decided to include individuals aged 16-25 years. Third, in
anticipation of the results of a scientific discussion in the STI
prevention field about a possible narrowing of the target
population for (asymptomatic) chlamydia testing and treatment
due to recent findings about the relatively low severity of effects
in most groups and antimicrobial resistance resulting fromovertreatment with antibiotics [20,21], it was decided to include
only individuals having heterosexual intercourse and cisgenderindividuals. As a final criterion, only people currently living in
the Netherlands were included, as people living abroad cannot
use all the services offered by Sense (eg, consultation hours).
Visits to the website from abroad were also not included in our
analysis of use data.

https://formative.jmir.org/2024/1/e48453

XSL• FO
RenderX

Metz et al

Recruitment
Participants were recruited via a banner on Sense.info thatappeared when website visitors spent more than 1 minute on
the same page or when they transferred to another page on thewebsite. The banner read: “Will you help us improve Sense.info?
We are looking for young people to test this website. Reward:€25 gift card.” Clicking on the banner led to a page with general
information about the study and a link to a short intake
questionnaire (created in Formdesk) asking forsociodemographic information (age, being sexually active,
education level, gender identity, sexual orientation, country ofresidence, country of birth, and country of parents’ birth). Only
if the answers met the inclusion criteria, contact details (email
address and phone number) were requested. Contact information
was deleted upon completion of the study. If the inclusion
criteria were not met, individuals were automatically redirected
to a “thank you” page. Due to the sensitive nature of the
inclusion criteria for gender identity and sexual orientation,individuals who did not meet these criteria were notautomatically redirected to this page but received a personal
email stating that they could not participate in this study but
may be able to participate in a future study.
Study recruitment and execution began simultaneously. During
the first weeks of recruitment, all eligible participants who hadthen signed up received an invitation email with a detailed
information letter and a request to select a time slot in a“doodle,” a digital scheduling tool. A reminder was sent if
participants did not respond within 5 working days. Based on
the sociodemographic characteristics of the initial participants,
purposive sampling based on age, education level, and gender
identity was then used to reach as diverse a sample as possible
and increase validity [22]. As a result, some individuals received
both an invitation and a reminder, while others received only
an invitation or no invitation at all. The latter was mostly thecase for individuals who enrolled later in the process and had
similar characteristics to those already recruited.

Procedure
Data collection took place in May and June 2021 until data
saturation was reached, defined as the point at which 3
consecutive interviews did not yield new relevant knowledge[23]. The procedure was carried out through videoconferencing
using a professional Zoom (Zoom Video Communications)account. This approach made it easier to include people fromall regions of the country. A technological advantage was thatparticipants were able to share their screens, which allowed theresearcher to unobtrusively observe their behavior.
Appointments lasted approximately 1 hour. No one other thanthe participant and the researcher was present during theprocedure.
GM (PhD candidate in Health Promotion, identifying as a man,educational background in social psychology and law [MSc,
LLM], and trained and experienced in qualitative researchmethods) conducted the interviews. There was no relationshipestablished with participants prior to study commencement,
other than the email correspondence to explain the research
project by means of an information letter and to set up a timeand date for the think-aloud study. The participants knew that
JMIR Form Res 2024 | vol. 8 | e48453 | p. 4
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
GM was a researcher at the Department of Health Promotion
at Maastricht University and that the research was conductedindependently from the partners at Soa Aids Nederland(developers of Sense.info), which was believed to make it easierfor participants to talk freely about Sense.info. Participants weretold beforehand that Maastricht University and Sense.info had
the mutual goal of evaluating and optimizing Sense.info. No
other characteristics about the interviewer were disclosed to theparticipants.
During the first minutes of the scheduled appointment, theresearcher explained the study and the procedure following theinformation provided in the participant’s letter. Assuming thatthe use of screen-sharing options was more convenient on adesktop or laptop than on a smartphone or tablet, all participants
were asked to use a desktop or a laptop. The page layout mayvary slightly depending on the medium used, but the content
remains the same (Figure 1 [5]). The participant then provided
informed consent via a digital form (using Formdesk), after
which the participant shared their screen and a practicethink-aloud session began. The participant was asked to viewthe home page of the Trimbos Institute website [24] and perform
a search task (find the Trimbos Institute’s phone number) while
thinking aloud. This website dealt with health topics but was
unrelated to the topics on Sense.info (no information about
sexual health in general or chlamydia in particular).
After practicing and making sure the participant understood theprocedure, the think-aloud procedure began. Screen and audio
were recorded using QuickTime (Apple). Participants wereasked to navigate the home page of Sense.info while thinking
aloud. When they indicated that they had seen everything on
the home page, the researcher asked them to search for thechlamydia page while thinking aloud and viewing theinformation as they would if they were alone. It was emphasizedthat they were free to read text, watch clips, and click on links
to different pages (ie, there were no restrictions on what content
they could choose or how long they could use it). If participants
were silent for over 10 seconds, the researcher used the prompt“feel free to think aloud.”
When participants indicated that they had seen enough, thesecond part began: the in-depth interview using an interview
protocol. The first questions were about the participants’ first
impressions of the page. For example, they were asked what
they thought of its appearance and comprehensibility andwhether certain elements should be removed or added. Then,the researcher asked questions related to the determinants thatwere deemed relevant based on the analysis of intended use andremarkable use patterns found in the analysis of actual use. For
example, the analysis of intended use showed that the behavior
change principle “consciousness raising” was used to motivate
treatment seeking for chlamydia by targeting the determinantof risk perception. This insight, combined with transfer rates to
the STI test page, led to the question of whether the information
was able to change participants’ beliefs about the effects ofchlamydia.

AnalysisThink-aloud sessions and semistructured interviews weretranscribed verbatim and then sent to participants forhttps://formative.jmir.org/2024/1/e48453

XSL• FO
RenderX

Metz et alconfirmation of accuracy and approval for further use. Templateanalysis was used to analyze the transcripts because of its
structured and stepwise approach to thematic analysis and its
ability to incorporate both inductive and deductive themes [25].
The software used for the analysis was ATLAS.ti 9 (ATLAS.ti
Scientific Software Development GmbH). First, the coders (GMand RRLCT) read through and familiarized themselves withthe data. They then conducted preliminary coding on a subsetof the data, using both inductively and deductively derived
themes, the latter based on the analyses of intended and actualuse. For example, the ABCD created in the intended use analysis
distinguished different subbehaviors (ie, getting tested, seeking
treatment, using condoms, and notifying partners), for which
we created corresponding codes. Based on the use patterns
observed in the actual use analysis, we were interested in
exploring why people would visit the page. Therefore, wecreated codes inductively, based on the reasons mentioned byparticipants (eg, when experiencing symptoms, when having
had sex without a condom and suspecting that one’s partner
might have chlamydia, when doing a school assignment on
STIs, or when visiting out of interest). The same inductive
approach was followed for the codes for barriers and facilitators
related to the execution of the subbehaviors. The themes werethen organized into clusters (eg, subbehaviors, barriers,
facilitators, reasons for visiting the chlamydia page, usability,
and reliability issues) and an initial coding template was defined.
The initial template was applied to 10% of the data by the twocoders independently, after which the coding was discussed andthe template adjusted accordingly. This version of the template
was then again independently applied to another 10% of thedata set, based on which intercoder agreement was calculatedusing Krippendorff α [26,27] (α=.93). The template was applied
to the entire data set by GM. Upon completion of this analysis,
the results of this analysis were triangulated with the data fromthe intended and actual use analyses. The research team
examined whether the results were consistent with theassumptions based on the intended and actual use analyses orwhether the results suggested the possibility of a new
explanation not covered by the assumptions.

Results
Participants
The digital intake questionnaire was accessible between April
30 and June 18, 2021, and was completed 238 times. A total of
85 submissions did not meet the inclusion criteria, resulting ina group of 153 eligible potential participants.
In total, 79 invitations were sent, of which 2 invitations were
undeliverable due to an incorrect email address. A total of 24
appointments were made. Then, 33 reminders were sent,
resulting in 2 additional appointments. Of the 26 appointments
made, 4 participants canceled, 4 participants were not present
at the appointed time and date, and 3 participants were found
not to meet the age criteria at the beginning of the appointment.
The final sample consisted of 15 participants (mean age 20, SD
2.5 years), 10 of whom identified as female. All participants
were still enrolled in education, except for 2 participants who,
following the International Standard Classification of Education
JMIR Form Res 2024 | vol. 8 | e48453 | p. 5
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Metz et al[28], completed lower secondary education and tertiary
education (university of applied sciences). Two participants
were enrolled in secondary education (1 lower secondary
education and 1 upper secondary education), five participantsin postsecondary nontertiary education, and six participants in
tertiary education (3 university of applied sciences and 3
university). All but one of the participants were born in the
Netherlands. Either one or both parents of 5 participants wereborn elsewhere, for example in Turkey, Morocco, or Surinam(these are also the most common migrant backgrounds in the
Netherlands [29]).

If I read this ... it says that not everyone shows
symptoms, I would get tested then just to be sure you
know, because maybe you don’t have any symptoms
and then ... Yes, better safe than sorry. [P13]Six participants mentioned the possibility of consultation andan STI test at their GP and three participants talked about theoption to do an STI test at the Sense consultation hours. Fourparticipants mentioned partner notification as an action to betaken, and 3 participants mentioned buying and using condoms.

Perceptions About <Chlamydia>

Nine participants stated that learning about the symptoms andconsequences of chlamydia would be a motivator to do an STI
test, which seemed to be related to the perceived severity of thesymptoms and consequences:

Perceived Reliability
In total, 13 participants indicated that they found the information
on <Chlamydia> reliable. Some reasons given were that
Sense.info is used in education and the perception that it isgovernment-funded. Three participants would have liked to see
where the information came from to increase reliability. In
addition, 5 participants explicitly stated that they found theinformation about chlamydia to be complete, but 4 participants
mentioned that they would like to double-check information or
seek additional information, mainly about the symptoms andconsequences of chlamydia.
Maybe add the name of the person who created this,who contributed to the text or something. These areoften people who have a medical degree ... Then it
becomes more reliable. Or that’s how it looks then.
But Sense does look like a good website. And I know
that... at school they always say that it is a goodsource, that you can find very good information there.
[P15]

Reasons to Visit, Perceived Goals of the Page, andExpected Follow-Up Actions
All participants indicated that they would visit the page mainly
if they had symptoms and wanted to investigate what STI theymight have contracted (ie, as an information source for possible
self-diagnosis). Related reasons were having had unsafe sex
and having heard that a sex partner has chlamydia and therefore
wanting to learn more about it and about the chance that they
could have contracted it as well.
I think I would look it [<Chlamydia>] up when I
would get symptoms after unprotected sex. And if I
reached this [table of symptoms], I would have a lookat the symptoms and if I were to recognize some ofthem, I would continue reading and then I would go
see a doctor. [P12]
When asked to elaborate on the goals they thought the page
had, 8 participants mentioned providing information aboutchlamydia, and 6 participants thought the page aimed to referpeople to an STI test. Perceived goals of the page and reasons
to visit the page thus seemed to be consistent.
When asked about the specific actions they would take after
viewing <Chlamydia>, 7 participants would do an STI test:

https://formative.jmir.org/2024/1/e48453

XSL• FO
RenderX

STI Testing: Barriers and Facilitators

It’s a big step for you to get tested, but I think that if
a woman finds out “Oh you can become infertile from
this,” that they will think “Oh shit, let’s get tested
very quickly.” [P07]This could be seen as an indication that providing information
about the symptoms and consequences of chlamydia can
influence knowledge, attitudes, risk perception, and intention
to get tested. Three participants, however, overlooked the link
to <STI test> during the think-aloud phase and stated that they
did not see any information about STI testing on the page. Fiveparticipants thought it would be more convenient to place thislink more prominently. When asked if they knew of otherpathways on Sense.info to find information on STI testing, most
responded in the affirmative.
This could also just be an information page. So only
when I scroll down, I see: oh wait, you can do an STI
test with them ... They do say ‘Sense consultation
hours’ here, but if they made it a little bit clearer, forexample by printing it in bold or whatever. [P03]
There’s the search bar, an FAQ page on the left side,
and a ‘Help & Advice’ page here, so there’s alreadythree places where you can look it up in case you
have a question, or you want an appointment orwhatever. [P10]Participants also suggested options for interaction, such as adigital STI check, that would allow users to report their
symptoms and then receive test recommendations and the option
to schedule an STI test immediately. Two participants expected
to find such a digital STI check by clicking on <STI test>, andthey and four others recommended placing such a check on<Chlamydia>. Two participants stated that the concrete andpersonal advice from the STI check could help people decide
what steps to take:
They could also make it interactive: “Do you havethis symptom, this one, this one, yes,” then go to[sexual health center]. That might be a good addition
for people so that they really know when you haveto… so that you know right away which action is wise
to take. [P09]

JMIR Form Res 2024 | vol. 8 | e48453 | p. 6
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Metz et al

In addition, another participant thought that such an STI checkmight lower the threshold for getting tested since people would
not have to make a phone call:

Discussion

Why is it only possible [to schedule an STI test] byphone? Because if they would... imagine they makesome sort of little questionnaire, and based on thatthey will filter: “okay, these people are eligible to get
tested.” Then they can also call those people or senda message saying “you have been approved, you canschedule a test” ... Because that would lower thethreshold for scheduling a test. [P05]
Some comments were made about the language used in relation
to testing. One participant stated that the word “consultationhour” sounded “scary.” Another participant felt that the
information did not encourage STI testing, but rather made it
feel obligatory:

The overall aim of this study was to demonstrate how a mixed
methods approach using a think-aloud method combined withsemistructured interviews and building on previously conductedanalyses of intended and actual use, can be used to explain
patterns in use data and to assess engagement with and potential
impact of web-based interventions. This was studied in thecontext of the chlamydia page of the Dutch sexual healthintervention Sense.info. Below, we first elaborate on the specificresults concerning <Chlamydia> in relation to the 3 preliminary
assumptions about the use, engagement, and potential impactof this page. After that, we share our methodologicalconsiderations about using think-aloud methods combined withother research methods to gain insight into web-based
interventions in general.

Are you worried, or have you had unprotected sex,
don’t carry it around... Yeah... I might click on it, but
I think this might be a bit scary. Because of the “don’t
carry it around and get tested.” Might feel a bit, ehm,
not threatening, but kind of like “you have to,” youknow ... Yes, I think I would just have a sentence atthe bottom or something like, “if you have any
questions or if you are in doubt about anything, justget in touch with us,” something like that. Because I
think you create a safer space then, like: “it’s okay,
we’re here for you if you’re in doubt or if you thinkit’s scary.” [P07]The presence of a link to a personal story about chlamydia wasevaluated positively by 7 participants, and such stories wereseen as a good way to encourage STI testing by 2 participants.
This may be through the mechanism of social identification.
The young person in the story, who is similar to the reader,
could become a role model by showing how they dealt with thesame situation, which could motivate the reader to do the same.
Then it is a bit recognizable for yourself, that you
realize: oh yes, it is indeed quite serious. Like youoften do, when they put a quote somewhere. That way
you can see it from the perspective of someone else
who might have had the same, yes. So when you readthat quote from somebody who’s has had the samething, you think, maybe I should do it [testing] after
all. [P06]
Another participant also seemed to refer to the importance ofsocial influence when they suggested providing insight into
how many people get chlamydia.
Maybe they can do a subheading that says you don’t
have to feel insecure or anything like that, or that it’sokay, maybe something like that. But I’m thinking
about how that could be designed. Yes, here theydescribe the symptoms and everything, but ... Yeah,
a subheading that it’s okay to get tested or that you’re
not the only one or maybe add a graph about how
many people get it [chlamydia]. So you don’t feelalone. I think the younger you are, the harder it is totake that step. [P14]https://formative.jmir.org/2024/1/e48453

XSL• FO
RenderX

Principal Findings

Our first assumption was that visitors to <Chlamydia> were
just looking for information about chlamydia without having
the need for an STI test. However, participants mainly statedthat they would visit the page with a self-diagnosis motive:
investigating if they could have contracted an STI in response
to showing symptoms or their partner having chlamydia.
Although this finding might not immediately align with therelatively high bounce rates for the chlamydia page, other use
data patterns do seem to show at least initial support for thisself-diagnosis motive. Google search queries leading to<Chlamydia> mainly focused on symptoms, and parts of thevisitors entering <Chlamydia> via other Sense.info pages had
previously visited pages on other STIs. Most transitions from
<Chlamydia> to other Sense pages were to <Types of STIs>,
followed by pages about specific other STIs. These visitors may
indeed have been trying to self-diagnose. Based on thesefindings, we might cautiously reject our first assumption andconclude that in general, visitors to <Chlamydia> have a needfor an STI test.
In light of the above conclusion about the need for STI testing,it is relevant to note that participants generally found theinformation on <Chlamydia> to be reliable, but some indicatedthat they needed more information or wanted to verify
information, for example, about symptoms or consequences ofchlamydia. They stated that they would seek additional
information elsewhere. Klawitter and Hargittai [30] foundsimilar needs in their study about the perceived credibility ofdigital health information and termed it the consistency heuristic:
users cross-reference various sources to determine whether the
information is trustworthy. This finding may be one explanation
for the relatively high bounce and exit rates of <Chlamydia>,although we recognize that several other explanations are
possible. To help visitors find additional information, we
recommend that the chlamydia page include links to reliable
information elsewhere on the internet. In addition, the provisionof information about (the credentials of the people) who
compiled the page content may contribute to the perceived
reliability.
Several participants explicitly mentioned the option of
consultation and STI testing with their GP, in contrast to three
JMIR Form Res 2024 | vol. 8 | e48453 | p. 7
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
participants who mentioned Sense consultation hours. Thisfinding provides initial evidence to support the first part of oursecond assumption, that some visitors to <Chlamydia> feel theneed for an STI test but make an appointment with their GP.
This finding is consistent with data from the National Institutefor Public Health and the Environment (RIVM) on STI testingin the Netherlands: two-thirds of all STI tests take place at the
GP’s office; the rest at the sexual health centers, of which theSense consultation hours are a part [31]. Regarding the second
part of the assumption, that visitors make an appointment at the
Sense consultation hour in a way other than by clicking on thelink to <STI test>, we did indeed observe that participants
overlooked the link but expressed that they knew of other ways
on the website that lead to an appointment at the Sense
consultation hour. Both the participants overlooking the link to
the STI test page and participants getting an STI test at their
GP might be explanations for the relatively high bounce rate
found for the chlamydia page.
The third assumption was that visitors felt the need for an STI
test but did not feel capable of getting tested and therefore did
not click on the link to the page on STI testing. Nine participants
stated that learning about the symptoms of chlamydia would
encourage them to do an STI test. Following the extended
parallel process model [32], we might conclude that the
information on symptoms and consequences seems to be able
to establish a threat high enough to lead to further cognitive
processing. As depicted in the ABCD, this might impact risk
perception of chlamydia and attitudes toward STI testing, which
in turn could lead to STI testing. However, for a threat to lead
to behavior change, there must be sufficient self- andresponse-efficacy-inducing information that leads individuals
to adopt high perceptions of danger control (ie, high threat
appraisal and high efficacy appraisal). Looking again at our
ABCD for <Chlamydia>, we noticed that while the page
provides messages that might influence self-efficacy regarding
chlamydia treatment (“chlamydia is easy to treat with a courseof antibiotics”), no such messages were present regarding STI
testing. We did not find clear evidence of a lack of self-efficacy
or response-efficacy among our participants regarding testing,although some participants mentioned certain barriers indicatingthat the information about testing did not always meet theirneeds (eg, no option for tailored advice, inappropriate locationof the link to <STI test>, and not appealing use of language).
To help overcome the barriers mentioned above, it may be goodto present STI testing as an easier and more natural step. Several
options emerged from the interviews, for example, creating a
digital STI check in which visitors are guided step-by-step inassessing whether they are at risk for an STI and can schedulea test immediately if needed. Moreover, personal stories seemed
to induce motivation to get tested, potentially through the
influence of social identification with the protagonist. Several
theories indeed state that individuals are motivated to act in linewith the behavior of influential others (social cognitive theory[33]) or with the norms of the group they identify with (social
identity theory [34]). Based on these results, we suggest
increasing the use of personal stories about chlamydia and STI
testing, using the behavior change principle “modeling” [34].
This principle has been applied in both web-based and otherhttps://formative.jmir.org/2024/1/e48453

XSL• FO
RenderX

Metz et alinterventions over the years and has shown success in initiating
behavior change through heightened self-efficacy levels [35,36].
A suggestion given by a participant also seemed to touch uponsocial influence and modeling: they suggested providing
information about the prevalence of chlamydia in young people
to show visitors that they are not the only ones (potentially)
having chlamydia. Similarly, the provision of numbers on how
many visitors have done the digital STI test might motivate
other visitors to do so as well.

Practical and Scientific Implications
Specific insights into the engagement with and potential impactof the Sense.info chlamydia page were gained, as well as
strategies to further engage end users and increase the potential
impact of the page. For example, the inclusion of a digital STI
check with tailored advice could be beneficial for those using
the chlamydia page for self-diagnosis. In addition, the inclusionof role model stories may be a promising strategy to encourage
STI testing among individuals who might be reluctant to test
for STIs. These results are also deemed relevant for the
optimization of other STI-related pages on Sense.info or othersexual health-related interventions.
Reflecting on the mixed methods evaluation process used, weconclude that it was valuable in assessing the potential impact
and engagement of Sense.info and that it may also have value
in evaluating other web-based interventions. The think-aloudmethod combined with semistructured interviews revealed
usability issues, as well as psychological issues, for example,
related to STI testing. The usefulness of the think-aloud method
for usability issues was already known, as it is widely used inusability testing [37,38]. This is reflected in the methodology
guidelines of Bonten et al [39], which mention the think-aloudmethod as a suitable method for the “design, development, andusability phase.” In addition to the importance of the think-aloudmethod in these early phases, the conclusion of this study isthat this method also has value in what Bonten et al [39] describe
as the “effectiveness or impact phase.” Since the guidelines also
recognize mixed methods research for the effectiveness or
impact phase, without specifying methods, we recommend thethink-aloud method as a fitting method within such a design.
This facilitates triangulation of data from several methods,
which we see as essential. We [9], as well as others [1,2], argue
that neither of the methods that we used should be used as thesole source of data to draw strong conclusions. For example,
while a study of intended use sheds light on how theory should
be or is applied in an intervention, it does not necessarily reflect
how the intervention is used in daily practice [9]. Actual use
data reveals how the intervention is used, but not why it is usedthat way [1,2]. A think-aloud method can provide several
potential explanations for behavior, and we argue that it is likely
most beneficial when building on results from studies examining
intended and actual use. These prior findings can guideresearchers in determining which aspects of the intervention
require particular attention during the study.

Limitations
A possible limitation of this study is that participants were asked
not to use a smartphone but rather a computer (be it a laptop,
desktop, or a PC with a webcam), as we expected that using
JMIR Form Res 2024 | vol. 8 | e48453 | p. 8
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
screen-sharing options in Zoom would be most convenient onsuch devices. Although most young individuals between theages of 12 and 25 years in the Netherlands have access to amobile phone or smartphone (99.1%) [40], as well as to laptops
or notebooks (96.1%) and PCs or desktops (63.4%), there may
be a difference between access and preferred use. Most visitors
to Sense.info use their smartphones. Therefore, future studies
might consider allowing participants to participate using a deviceof their choice to create the most natural experience possible.
Efforts to engage end users and consider their preferential use
in the evaluation and subsequent optimization phase could leadto more equitable design [41,42].
A think-aloud study in which a participant is being observed
will never be the exact same experience as an unobserved visit,
and we need to take socially desired behavior into account.
Furthermore, there are always multiple explanations possible
for user behavior, and the explanations given by participants
do not have to exclude other explanations. However, social
desirability can never be ruled out in self-report research, andthe think-aloud method is not unique in this respect. In the caseof this study, we did not have the impression that participants
behaved in a socially desirable way; they were open about
several sexual themes including their own sexual health, andcritically appraised the website.
Finally, there are two limitations regarding the recruitment ofparticipants. First, potential bias as a result of recruitment cannot
be ruled out. Participants self-registered for the study, which
may have resulted in a pool of participants who were motivatedand found it easy to talk about the issues at hand. This may haveoverlooked the insights of certain individuals who found it lesseasy to talk about these issues and therefore did not register forthe study. A related issue is that there is no data on thecharacteristics of the real-life users of Sense.info. Whether arepresentative sample of Sense.info users was obtained, istherefore difficult to say. However, purposive sampling basedon age, level of education, and gender was used to ensure thatas diverse a group as possible was created from those who
registered, and to reflect the diversity of the intended target

Metz et alaudience of Sense.info. Second, only individuals having
heterosexual intercourse and cisgender individuals were includedin the study, anticipating the results of a scientific debate aboutnarrowing down the target population for (asymptomatic)chlamydia testing and treatment to these groups. This would
require a differentiation on the Sense.info website in theprovision of chlamydia information based on sexual behavior,
meaning that the current chlamydia page (with its emphasis on
STI testing) would be maintained for cisgender people whohave heterosexual intercourse, and a separate chlamydia pagewould be created for men who have sex with men andtransgender people (with less emphasis on testing). During thedesign phase of the exploration of end user perceptions, thedevelopers of Sense.info and the research team decided to focus
on cisgender people who have heterosexual intercourse because,in the scenario described above, they would remain the groupof interest for the current chlamydia page. After the completionof this study, it appeared that no changes in testing and treatment
policy would be made in the near future, making a distinction
unnecessary. In addition, the Sense.info use data did not allow
analysis of use patterns based on sexual behavior, so it was notpossible to verify whether use patterns could be explained by
one group or the other. Including the entire Sense.info target
group in the think-aloud study, regardless of sexual behavior,would therefore have been more consistent with the goal of
exploring potential explanations for the use data. The views ofindividuals, such as men who have sex with men and transgenderindividuals, will therefore be included in future studies of theSense.info chlamydia page.

Conclusions
Using the Dutch sexual health intervention Sense.info, we
demonstrated how a mixed methods approach consisting of atheoretical analysis of intended use, analysis of use data, and athink-aloud method combined with semistructured interviews
can be used to gain insight into engagement with and potential
impact of web-based interventions. This paper mainly discussed
this approach in the context of Sense.info, but we believe it isvaluable for evaluating web-based interventions in general.

Acknowledgments
We would like to thank all participants for taking part in this study, Leo Koningen (Soa Aids Nederland) for his help during thedata collection phase, and Wessel Zweers and Filippo Zimbile (both Soa Aids Nederland) for providing useful feedback on thismanuscript. This work was supported by ZonMw—the Netherlands Organization for Health Research and Development (project:
555003022). The authors attest that no generative artificial intelligence technology was used within the context of this study,
with the exception of text editing in the final stages of manuscript writing, using DeepL Write.

Data Availability
The data sets generated during or analyzed during this study (ie, interview transcripts) and all materials used (eg, participant letterand interview protocol) are available in the Open Science Framework Repository [13].

Conflicts of Interest
None declared.

References

https://formative.jmir.org/2024/1/e48453

XSL• FO
RenderX

JMIR Form Res 2024 | vol. 8 | e48453 | p. 9
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
1.

2.

3.
4.

5.
6.
7.
8.
9.

10.
11.
12.

13.
14.

15.

16.

17.

18.

19.
20.

21.

22.

23.
24.
25.

Short CE, DeSmet A, Woods C, Williams SL, Maher C, Middelweerd A, et al. Measuring engagement in eHealth andmHealth behavior change interventions: viewpoint of methodologies. J Med Internet Res. Nov 16, 2018;20(11):e292. [FREEFull text] [doi: 10.2196/jmir.9397] [Medline: 30446482]
Perski O, Blandford A, West R, Michie S. Conceptualising engagement with digital behaviour change interventions: asystematic review using principles from critical interpretive synthesis. Transl Behav Med. Jun 2017;7(2):254-267. [FREEFull text] [doi: 10.1007/s13142-016-0453-1] [Medline: 27966189]
Dirican AC, Göktürk M. Psychophysiological measures of human cognitive states applied in human computer interaction.
Procedia Computer Science. 2011;3:1361-1367. [doi: 10.1016/j.procs.2011.01.016]
Michie S, Yardley L, West R, Patrick K, Greaves F. Developing and evaluating digital interventions to promote behavior
change in health and health care: recommendations resulting from an international workshop. J Med Internet Res. Jun 29,
2017;19(6):e232. [FREE Full text] [doi: 10.2196/jmir.7126] [Medline: 28663162]
Sense.info. Chlamydia. URL: https://sense.info/nl/soas/soorten-soas/chlamydia [accessed 2024-08-14]
Youth. Statistics Netherlands. 2021. URL: https://www.cbs.nl/nl-nl/visualisaties/dashboard-bevolking/leeftijd/jongeren[accessed 2024-08-14]
Sense.info. URL: https://www.instagram.com/senseinfo [accessed 2024-08-14]
Sense talk. Sense.info. URL: https://sense.info/nl/ervaringen-van-anderen/podcasts/sense-talk [accessed 2024-08-14]
Metz G, Roosjen H, Zweers W, Crutzen R. Evaluating use of web-based interventions: an example of a Dutch sexual healthintervention. Health Promot Int. Aug 01, 2023;38(4):daab136. [FREE Full text] [doi: 10.1093/heapro/daab190] [Medline:
37596929]
Metz G, Peters G-JY, Crutzen R. Acyclic behavior change diagrams: a tool to report and analyze interventions. Health
Psychol Behav Med. 2022;10(1):1216-1228. [FREE Full text] [doi: 10.1080/21642850.2022.2149930] [Medline: 36518605]
Bartholomew Eldredge LK, Markham CM, Ruiter RAC, Fernández ME, Kok G, Parcel GS. Planning Health Promotion
Programs: An Intervention Mapping Approach. San Francisco, CA. Jossey-Bass; 2016.
Kok G, Gottlieb NH, Peters G-JY, Mullen PD, Parcel GS, Ruiter RAC, et al. A taxonomy of behaviour change methods:
an intervention mapping approach. Health Psychol Rev. Sep 2016;10(3):297-312. [FREE Full text] [doi:10.1080/17437199.2015.1077155] [Medline: 26262912]
OSF Repository. URL: https://doi.org/10.17605/osf.io/k5jfd
Newby K, Crutzen R, Brown K, Bailey J, Saunders J, Szczepura A, et al. An intervention to increase condom use amongusers of chlamydia self-sampling websites (wrapped): intervention mapping and think-aloud study. JMIR Form Res. May
01, 2019;3(2):e11242. [FREE Full text] [doi: 10.2196/11242] [Medline: 31042156]
Yardley L, Morrison L, Bradbury K, Muller I. The person-based approach to intervention development: application todigital health-related behavior change interventions. J Med Internet Res. Jan 30, 2015;17(1):e30. [FREE Full text] [doi:10.2196/jmir.4055] [Medline: 25639757]
Szinay D, Perski O, Jones A, Chadborn T, Brown J, Naughton F. Influences on the uptake of health and well-being apps
and curated app portals: think-aloud and interview study. JMIR Mhealth Uhealth. Apr 27, 2021;9(4):e27173. [FREE Fulltext] [doi: 10.2196/27173] [Medline: 33904827]
Perski O, Blandford A, Ubhi HK, West R, Michie S. Smokers' and drinkers' choice of smartphone applications andexpectations of engagement: a think aloud and interview study. BMC Med Inform Decis Mak. Feb 28, 2017;17(1):25.
[FREE Full text] [doi: 10.1186/s12911-017-0422-8] [Medline: 28241759]
Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist forinterviews and focus groups. Int J Qual Health Care. Dec 2007;19(6):349-357. [doi: 10.1093/intqhc/mzm042] [Medline:
17872937]
De Graaf H, Van den Borne M, Nikkelen S, Twisk D, Meijer S. Sex under the age of 25: sexual health of youth aged 12-25
in the Netherlands in 2017. Eburon. Delft, The Netherlands.; 2017. [FREE Full text]
van Bergen JEAM, Hoenderboom BM, David S, Deug F, Heijne JCM, van Aar F, et al. Where to go to in chlamydia control?
From infection control towards infectious disease control. Sex Transm Infect. Nov 2021;97(7):501-506. [FREE Full text][doi: 10.1136/sextrans-2021-054992] [Medline: 34045364]
Dukers-Muijrers NHTM, Evers YJ, Hoebe CJPA, Wolffs PFG, de Vries HJC, Hoenderboom B, et al. Controversies andevidence on chlamydia testing and treatment in asymptomatic women and men who have sex with men: a narrative review.
BMC Infect Dis. 2022;22(1):255. [FREE Full text] [doi: 10.1186/s12879-022-07171-2] [Medline: 35287617]
Palinkas LA, Horwitz SM, Green CA, Wisdom JP, Duan N, Hoagwood K. Purposeful sampling for qualitative data collection
and analysis in mixed method implementation research. Adm Policy Ment Health. Sep 2015;42(5):533-544. [FREE Fulltext] [doi: 10.1007/s10488-013-0528-y] [Medline: 24193818]
Mason M. Sample size and saturation in PhD studies using qualitative interviews. Forum Qual Soc Res. 2010;11:1-19.
[doi: 10.17169/fqs-11.3.1428]
Trimbos Institute. URL: https://www.trimbos.nl [accessed 2024-08-23]
Brooks J, McCluskey S, Turley E, King N. The utility of template analysis in qualitative psychology research. Qual Res
Psychol. Apr 03, 2015;12(2):202-222. [FREE Full text] [doi: 10.1080/14780887.2014.955224] [Medline: 27499705]

https://formative.jmir.org/2024/1/e48453

XSL• FO
RenderX

Metz et al

JMIR Form Res 2024 | vol. 8 | e48453 | p. 10(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
26.
27.
28.
29.

30.

31.

32.
33.
34.
35.
36.

37.
38.
39.

40.
41.

42.

Metz et al

Hayes AF, Krippendorff K. Answering the call for a standard reliability measure for coding data. Commun Methods Meas.
Apr 2007;1(1):77-89. [doi: 10.1080/19312450709336664]
Krippendorff K. Reliability in content analysis. Human Comm Res. Jul 2004;30(3):411-433. [doi:10.1111/j.1468-2958.2004.tb00738.x]
International Standard Classification of Education: ISCED 2011. Montréal, Quebec. Statistics UIf; 2012:85.
How many people with a migration background live in the Netherlands? Statistics Netherlands. 2022. URL: https://www.
cbs.nl/nl-nl/dossier/dossier-asiel-migratie-en-integratie/hoeveel-mensen-met-een-migratieachtergrond-wonen-in-nederland[accessed 2024-08-14]
Klawitter E, Hargittai E. Shortcuts to well being? Evaluating the credibility of online health information through multiple
complementary heuristics. J Broadcast Electron Media. May 03, 2018;62(2):251-268. [FREE Full text] [doi:10.1080/08838151.2018.1451863]
Staritsky LE, Visser M, Van AF, Op de Coul ELM, Heijne JCM, Van Wees DA. Sexually transmitted infections in the
Netherlands in 2020. National Institute for Public Health and the Environment. 2021. URL: http://hdl.handle.net/10029/
623857 [accessed 2024-08-17]
Witte K. Putting the fear back into fear appeals: the extended parallel process model. Commun Monogr. Dec
1992;59(4):329-349. [doi: 10.1080/03637759209376276]
Bandura A. Social Foundations of Thought and Action: A Social Cognitive Theory. Saddle River, NJ, US. Prentice-Hall,
Inc; 1986.
Tajfel H, Turner JC. An integrative theory of intergroup conflict. In: Worchel S, Austin WG, editors. The Psychology ofIntergroup Relations. Monterey, CA. Brooks-Cole; 1979:33-47.
Kelder SH, Hoelscher D, Perry CL. How individuals, environments, health behaviors interact. In: Glanz K, Rimer BK,
Viswanath K, editors. Health Behavior: Theory, Research, and Practice. San Francisco, CA. Jossey-Bass; 2015:159-182.
Webb TL, Joseph J, Yardley L, Michie S. Using the internet to promote health behavior change: a systematic review andmeta-analysis of the impact of theoretical basis, use of behavior change techniques, and mode of delivery on efficacy. J
Med Internet Res. Feb 17, 2010;12(1):e4. [FREE Full text] [doi: 10.2196/jmir.1376] [Medline: 20164043]
Dumas JS, Redish J. A Practical Guide to Usability Testing. Exeter. Intellect books; 1999:404.
Tomlin WC. UX Optimization: Combining Behavioral UX and Usability Testing Data to Optimize Websites. Berkely, CA.
Apress; 2018.
Bonten TN, Rauwerdink A, Wyatt JC, Kasteleyn MJ, Witkamp L, Riper H, et al. Online guide for electronic health evaluation
approaches: systematic scoping review and concept mapping study. J Med Internet Res. Aug 12, 2020;22(8):e17774. [FREEFull text] [doi: 10.2196/17774] [Medline: 32784173]
Internet; access, use and facilities; 2012-2019. Statistics Netherlands. 2019. URL: https://opendata.cbs.nl/#/CBS/nl/dataset/
83429NED/table?dl=35852 [accessed 2024-08-14]
Goedhart NS, Zuiderent-Jerak T, Woudstra J, Broerse JEW, Betten AW, Dedding C. Persistent inequitable design andimplementation of patient portals for users at the margins. J Am Med Inform Assoc. 2021;28(2):276-283. [FREE Full text][doi: 10.1093/jamia/ocaa273] [Medline: 33463691]
Latulippe K, Hamel C, Giroux D. Social health inequalities and eHealth: a literature review with qualitative synthesis oftheoretical and empirical studies. J Med Internet Res. Apr 27, 2017;19(4):e136. [FREE Full text] [doi: 10.2196/jmir.6731][Medline: 28450271]

Abbreviations
ABCD: acyclic behavior change diagram
COREQ: Consolidated Criteria for Reporting Qualitative Research
GP: general practitioner
STI: sexually transmitted infection

Edited by A Mavragani; submitted 24.04.23; peer-reviewed by M Brown, M Gasmi , D Chen, J Benjamins; comments to author
16.12.23; revised version received 08.05.24; accepted 08.07.24; published 11.09.24
Please cite as:Metz G, Thielmann RRLC, Roosjen H, Crutzen R
Evaluating the Impact of a Dutch Sexual Health Intervention for Adolescents: Think-Aloud and Semistructured Interview StudyJMIR Form Res 2024;8:e48453
URL: https://formative.jmir.org/2024/1/e48453
doi: 10.2196/48453
PMID: 39259573

https://formative.jmir.org/2024/1/e48453

XSL• FO
RenderX

JMIR Form Res 2024 | vol. 8 | e48453 | p. 11(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Metz et al©Gido Metz, Rosa R L C Thielmann, Hanneke Roosjen, Rik Crutzen. Originally published in JMIR Formative Research(https://formative.jmir.org), 11.09.2024. This is an open-access article distributed under the terms of the Creative Commons
Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction
in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete
bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license
information must be included.

https://formative.jmir.org/2024/1/e48453

XSL• FO
RenderX

JMIR Form Res 2024 | vol. 8 | e48453 | p. 12(page number not for citation purposes)