Overview of eRisk at CLEF 2023: Early Risk Prediction
on the Internet (Extended Overview)
Notebook for the eRisk Lab at CLEF 2023
Javier Parapar1,* , Patricia Martín-Rodilla1 , David E. Losada2 and Fabio Crestani3
1

Information Retrieval Lab, Centro de Investigación en Tecnoloxías da Información e as Comunicacións (CITIC),
Universidade da Coruña. Campus de Elviña s/n C.P 15071 A Coruña, Spain
2
Centro Singular de Investigación en Tecnoloxías Intelixentes (CiTIUS),Universidade de Santiago de Compostela. Rúa de
Jenaro de la Fuente Domínguez, C.P 15782, Santiago de Compostela, Spain
3
Faculty of Informatics, Universitá della Svizzera italiana (USI). Campus EST, Via alla Santa 1, 6900 Viganello,
Switzerland

Abstract

This article provides an overview of eRisk 2023, the seventh edition of the CLEF conference’s lab dedicatedto early risk detection. Our lab has been committed to exploring evaluation methodologies, effectivenessmetrics, and other associated processes in the field of early risk detection since its inception. The
applications of early alerting models are wide-ranging and span various domains, including healthand safety. eRisk 2023 encompassed three tasks. The initial task involved ranking sentences based on
their relevance to standardized depression symptoms. The second task concentrated on detecting signs
associated with pathological gambling early. Lastly, the third task required participants to automaticallyestimate an eating disorders questionnaire by analyzing user writings on social media. In this extended
overview, we include additional details about the participants’ proposals and more detailed explanations
about metrics.

Keywords

Early risk, Depression, Pathological gambling, Eating disorders

1. Introduction
The main objective of eRisk is to investigate evaluation methodologies, metrics, and otherpertinent factors about research collection development and identifying signs associated withearly risk detection. The potential of early detection technologies is significant, particularly infields that address safety and health applications. Automated systems can play a crucial rolein issuing early warnings in scenarios involving individuals displaying symptoms of mental
CLEF 2023: Conference and Labs of the Evaluation Forum, September 18–21, 2023, Thessaloniki, Greece
Corresponding author.
$ javier.parapar@udc.es (J. Parapar); patricia.martin.rodilla@udc.es (P. Martín-Rodilla); david.losada@usc.es(D. E. Losada); fabio.crestani@usi.ch (F. Crestani) https://www.dc.fi.udc.es/~parapar (J. Parapar); http://www.incipit.csic.es/gl/persoa/patricia-martin-rodilla(P. Martín-Rodilla); http://tec.citius.usc.es/ir/ (D. E. Losada);
https://search.usi.ch/en/people/4f0dd874bbd63c00938825fae1843200/crestani-fabio (F. Crestani) 0000-0002-5997-8252 (J. Parapar); 0000-0002-1540-883X2 (P. Martín-Rodilla); 0000-0001-8823-7501 (D. E. Losada);
0000-0001-8672-0700 (F. Crestani)*© 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR

Workshop
Proceedings

http://ceur-ws.org
ISSN 1613-0073

CEUR Workshop Proceedings (CEUR-WS.org)

illnesses, infants encountering interactions with sexual abusers, or potential criminals publishingantisocial threats online.
Our lab primarily focuses on psychological issues, specifically depression, self-harm, pathologicalgambling, and eating disorders. Through our work, we have found that the relationshipbetween psychological diseases and language use is intricate, and there is room for enhancing
the effectiveness of automatic language-based screening models. In 2017, we conducted an
exploratory task on early detection of depression, employing innovative evaluation methods
and a test dataset described in [1, 2]. The following year we continued to foster the detection ofearly signs of depression and introduced a new task for detecting early signs of anorexia [3, 4].
In 2019, we further expanded the challenge by focusing on the early identification of anorexia
symptoms, introducing a new challenge on early detection of self-harm, and proposing a third
task centred around estimating a user’s responses to a depression questionnaire based on theirsocial media interactions [5, 6, 7]. In 2020, our efforts continued with the early detection ofself-harm and introducing a task for estimating the severity of depression symptoms [8, 9, 10]. In
2021, our focus shifted to two scenarios: early detection of pathological gambling and self-harmand a task for severity estimation of depression [11, 12, 13]. Lastly, in the previous year, we
presented three tasks: early pathological gambling detection, early detection of depression, andseverity estimation of eating disorders [14, 15, 16].
In 2023, eRisk featured three campaign-style tasks [17, 18]. The first task involved ranking
sentences based on their relevance to each of the 21 symptoms of depression derived fromthe BDI-II questionnaire. Participants in task 1 were provided with a collection of sentences
extracted from publications of social media users. The second task represented the third editionof early risk detection for pathological gambling. The third task marked the second edition of
the eating disorder severity estimation challenge. Detailed descriptions of these tasks can befound in the subsequent sections of this overview article.
A total of 98 teams registered for the lab, out of which we received results from 20 teams, with37 runs for Task 1, 48 runs for Task 2, and 20 runs for Task 3.

2. Task 1: Search for Symptoms of Depression
Task 1 introduced a novel challenge in 2023 involving the production of sentence rankings
based on their relevance to specific symptoms of depression. We instructed participants torank sentences derived from user writings according to their relevance to the 21 standardized
symptoms outlined in the BDI-II Questionnaire [19]. In this context, a sentence was deemed
relevant to a particular symptom if it provided information about the user’s state of thatsymptom. It is important to emphasize that a phrase could be considered relevant even if it
conveyed positive information about the symptom. For instance, "I feel quite happy lately"should be regarded as relevant for symptom 1, "Sadness," in the BDI-II.

2.1. Dataset and format
The corpus provided to the participants was a TREC formatted sentence-tagged dataset (basedon eRisk’s past data). Table 1 reports some statistics of the corpus.

Table 1
Corpus statistics for Task 1: Search for Symptoms of Depression.
Number of users
Number of sentences
Average number of words per sentence

3,107
3,807,115
13.63

1 Q0 sentence-id-121 0001 10 myGroupNameMyMethodName
1 Q0 sentence-id-234 0002 9.5 myGroupNameMyMethodName
1 Q0 sentence-id-345 0003 9 myGroupNameMyMethodName...
21 Q0 sentence-id-456 0998 1.25 myGroupNameMyMethodName
21 Q0 sentence-id-242 0999 1 myGroupNameMyMethodName
21 Q0 sentence-id-347 1000 0.9 myGroupNameMyMethodName

Figure 1: Example of TREC format for a participant’s run for Task 1.
Table 2
Task 1 (Search for Symptoms of Depression): Number of runs from participants.
Team# of submissions

BLUE
Formula-ML
GMU-FAST
Mason-NLP
NailP
OBSER-MENH
RELAI
UMU
UNSL
uOttawa

5
4
2
1
5
5
5
2
3
5

Total

37

Given the corpus of sentences and the description of the symptoms from the BDI questionnaire,
the participants were free to decide on the best strategy to derive queries for representing the
BDI symptoms. Each participating team submitted up to 5 variants (runs). Each run included 21
TREC-style formatted rankings of sentences, as shown in Figure 1.

2.2. Assessment Process
For each symptom, the participants could should submit up to 1000 results sorted by estimatedrelevance. We received 37 runs from 10 participating teams (see Table 2).
The generation of relevance judgments involved the participation of three expert assessors
who annotated a pool of sentences associated with each symptom. We selected the candidate
sentences using a top-k pooling process using the participants’ submissions ( 37 different ranking
methods)

The assessors were provided with explicit instructions regarding the determination of sentence
relevance. They were instructed to consider a sentence relevant if it pertained to the symptom
and provided explicit information about the individual’s state in relation to it. This dual conceptof relevance, encompassing both the topic and the reflection of the user’s state, introduced ahigher complexity level than more conventional relevance assessments. As a result, we developeda robust annotation methodology and formal assessment guidelines to ensure consistency
and accuracy. To create the pool of sentences for assessment, we employed 𝑘 = 50 in thepooling method. Table 3 presents the resulting pool sizes per sentence. It was observed thatcertain sentences had identical text but different IDs, potentially stemming from multiple users
writing the same content. To reduce the assessors’ workload, we automated removing duplicate
sentences. The annotation software, specifically developed to support the annotation processof eRisk 2023, automatically assigned the assessors’ relevance labels to all identical sentences.
Despite implementing these optimizations, the average time spent per sentence by the threeassessors was still over 30 seconds, even for this sentences that are very short documents. These
resulted in more than 210 hours of assessors’ time.
Table 3
Task 1 (Search for Symptoms of Depression): Size of the pool for every BDI ItemBDI Item (#)

original

unique# rels (3/3)# rels (2/3)

Sadness (1)Pessimism (2)Past Failure (3)Loss of Pleasure (4)Guilty Feelings (5)Punishment Feelings (6)Self-Dislike (7)Self-Criticalness (8)Suicidal Thoughts or Wishes (9)Crying (10)Agitation (11)Loss of Interest (12)Indecisiveness (13)Worthlessness (14)Loss of Energy (15)Changes in Sleeping Pattern (16)
Irritability (17)Changes in Appetite (18)Concentration Difficulty (19)Tiredness or Fatigue (20)Loss of Interest in Sex (21)

1110
1150
973
1013
829
1079
1005
1072
953
983
1080
1077
1110
1067
1082
938
1047
984
1024
1033
971

1069
1096
918
948
794
1036
957
1023
923
917
1057
1021
1044
986
1027
904
1008
947
981
994
922

179
104
160
97
83
21
158
76
260
230
69
70
61
71
129
203
94
103
83
123
97

318
325
300
204
143
50
288
174
349
320
154
168
141
144
204
350
155
224
141
221
158

The annotation process involved a team of three assessors with diverse backgrounds and expertise. One assessor was a psychologist, while the other two were computer science researchers—apostdoctoral fellow and a PhD student—specialised in early-risk technologies.
The lab organisers conducted a preparatory session with the assessors to ensure consistency andclarity. During this session, an initial version of the guidelines was discussed, and any doubts orquestions raised by the assessors were thoroughly addressed. Through this collaborative effort,
the final version of the guidelines shown in Figure 2 was developed.
Assume you a r e g i v e n a BDI item , e . g . :1 5 . Loss of Energy− I have a s much e n e r g y a s e v e r .
− I have l e s s e n e r g y than I used t o have .
− I don ’ t have enough e n e r g y t o do v e r y much .
− I don ’ t have enough e n e r g y t o do a n y t h i n g .
The t a s k c o n s i s t s o f a n n o t a t i n g s e n t e n c e s i n t h e c o l l e c t i o n t h a t a r et o p i c a l l y − r e l e v a n t t o t h e i t e m ( r e l a t e d t o t h e q u e s t i o n and / o r t o t h eanswers ) .
Note : A r e l e v a n t s e n t e n c e s h o u l d p r o v i d e some i n f o r m a t i o n a b o u t t h e s t a t e o f
t h e own w r i t e r r e l a t e d t o t h e t o p i c o f t h e BDI i t e m . But i t i s n o t
n e c e s s a r y t h a t t h e e x a c t same words a r e used .
Your j o b i s t o a s s e s s s e n t e n c e s on how t o p i c a l l y − r e l e v a n t t h e y a r e f o r ac o n c r e t e BDI i t e m .
The r e l e v a n c e g r a d e s a r e :1 . R e l e v a n t : A r e l e v a n t s e n t e n c e s h o u l d be t o p i c a l l y − r e l a t e d t o t h e BDI − i t e m( r e g a r d l e s s o f t h e wording ) and , a d d i t i o n a l l y , i t s h o u l d r e f e r t o t h e
s t a t e o f t h e w r i t e r a b o u t t h e BDI − i t e m .
0 . Non− r e l e v a n t : A non − r e l e v a n t s e n t e n c e d o e s n o t a d d r e s s any t o p i c r e l a t e d
t o t h e q u e s t i o n and / o r t h e a n s w e r s o f t h e BDI − i t e m ( o r i t i s r e l a t e d t o
t h e t o p i c b u t d o e s n o t r e p r e s e n t t h e w r i t e r ’ s s t a t e a b o u t t h e BDI − i t e m ) .
F o r example , f o r BDI − i t e m 1 5 , a s e n t e n c e t h a t d o e s n o t t a l k a b o u t t h e
i n d i v i d u a l ’ s l e v e l o f e n e r g y ( r e g a r d l e s s o f t h e wording ) , t h e n i s a non −relevant sentence .
Ex am pl es ( a s s e s s m e n t o f s e n t e n c e s r a n k e d f o r BDI − i t e m number 1 5 ) :I c a n n o t c o n t r o l my e n e r g y t h e s e days : R e l e v a n t
My s i s t e r has no e n e r g y a t a l l : Non− r e l e v a n t s e n t e n c e ( b e c a u s e i t d o e s n o t
r e f e r t o t h e w r i t e r who p o s t e d t h i s s e n t e n c e )The book was a b o u t a h i g h l y e n e r g e t i c man : Non− r e l e v a n t s e n t e n c e ( b e c a u s e i t
d o e s n o t r e f e r t o t h e w r i t e r who p o s t e d t h i s s e n t e n c e )I f e e l more t i r e d than u s u a l : R e l e v a n t
The f o o t b a l l team i s named Top E n e r g y : Non− r e l e v a n t
I am t o t a l l y l o n e l y : Non− r e l e v a n t ( i t d o e s n o t mention e n e r g y )I have j u s t r e c h a r g e d my b a t t e r i e s : R e l e v a n t
I am l o s t : Non− r e l e v a n t
We a d v i s e you t o n o t s t o p t h e a s s e s s m e n t s e s s i o n i n t h e m i d d l e o f one BDI −i t e m ( t h i s h e l p s t o m a i n t a i n c o n s i s t e n c y i n t h e j u d g m e n t s ) . To measuret h e a s s e s s m e n t e f f o r t , we a s k you t o r e c o r d t h e t i m e s p e n t on f u l l ye v a l u a t i n g t h e s e n t e n c e s p r e s e n t e d f o r each BDI − i t e m .

Figure 2: Guidelines for labelling sentences related to depression symptoms (Task 1).

Following these guidelines, a sentence is deemed relevant only if it provides "some information

about the state of the individual related to the topic of the BDI item." This criterion serves as
the foundation for determining the relevance of sentences during the annotation process.
After the initial meeting, the assessors proceeded to label the pools of sentences for the first three
BDI topics (sadness, pessimism, and past failure). Following this phase, we organized anothermeeting to address any additional concerns or questions that arose during the annotation
process. This collaborative session was vital in refining the annotation criteria and ensuringconsistency. The final outcomes of the annotation process, including the number of relevant
sentences per BDI item, are presented in Table 3 (last two columns). We applied two aggregation
criteria for sentence relevance determination: unanimity and majority.
The performance results of the participating systems are presented in Tables 4 (majority-based
qrels) and 5 (unanimity-based qrels). These tables show results for multiple standard ranking
metrics, namely Mean Average Precision (MAP), mean R-Precision, mean Precision at 10, andmean NDCG at 1000. Notably, Formula-ML models based on setence transformers, developedby the NITK Surathkal team, achieved the highest performance rankings across all metrics andrelevance judgment types.

3. Task 2: Early Detection of Pathological Gambling
This task marks the third edition of the challenge, which aims to develop innovative models forthe early identification of pathological gambling risk. Pathological gambling, also known as
ludomania or "gambling addiction," involves an irresistible urge to gamble despite the adverse
consequences. According to the World Health Organization (WHO), the prevalence of adultgambling addiction in 2017 ranged from 0.1% to 6.0% [29]. The objective of this task was toprocess evidence in a sequential manner and detect early indications of compulsive or disorderedgambling as promptly as possible. Participating systems were required to analyze user posts
on social media in the order of their occurrence. Successful outcomes from this task could
potentially be employed for sequential monitoring of user interactions across diverse online
platforms such as blogs, social networks, and other forms of digital media.
The test collection employed for this task followed the same format as the collection describedin the work by Losada and Crestani [30]. This collection comprises writings, including posts andcomments, obtained from a carefully selected group of social media users. Within this dataset,
users are classified into two categories: pathological gamblers and non-pathological gamblers.
For each user, the collection contains a chronological sequence of writings. To facilitate the task
and ensure equitable distribution, we established a dedicated server that systematically provided
user writings to the participating teams. Further details regarding the setup and functioning of
the server can be found on the lab’s official website1 .
This task followed a train-test approach. During the training stage, teams were provided withtraining data, including the complete history of writings for training users. We indicated whichusers explicitly identified themselves as pathological gamblers, allowing participants to tune
their systems using this training data. In 2023, the training data for Task 1 consisted of users
from previous editions of the self-harm task.
1

https://early.irlab.org/server.html

Table 4
Ranking-based evaluation for Task 1 (majority voting)
Team

Run

AP

Formula-ML [20]SentenceTrainsformers_0.25
0.319
Formula-ML
SentenceTrainsformers_0.1
0.308
Formula-ML
result2
0.086
Formula-ML
word2vec_0.1
0.092
OBSER-MENH [21] salida-distilroberta-90-cos
0.294
OBSER-MENH
salida-mpnet-90-cos
0.265
OBSER-MENH
salida-mpnet-21-cos
0.120
OBSER-MENH
salida-distilroberta-21-cos
0.158
OBSER-MENH
salida-mini12-21-cos
0.114
uOttawa [22]USESim
0.160
uOttawa
Glove100Sim
0.017
uOttawa
RobertaSim
0.033
uOttawa
GloveSim
0.011
uOttawa
BertSim
0.084
BLUE [23]SemSearchOnBDI2Queries
0.104
BLUE
SemSearchOnGeneratedQueriesMentalRoberta 0.029
BLUE
SemSearchOnBDI2QueriesMentalRoberta
0.027
BLUE
SemSearchOnGeneratedQueries
0.052
BLUE
SemSearchOnAllQueries
0.065
NailP [24]
T1_M2
0.095
NailP
T1_M4
0.095
NailP
T1_M3
0.073
NailP
T1_M5
0.089
NailP
T1_M1
0.074
RELAI [25]bm25|mpnetbase
0.048
RELAI
BM25
0.016
RELAI
bm25|mpnetbase_simcse
0.030
RELAI
bm25|mpnetqa_simcse
0.027
RELAI
bm25|mpnetqa
0.038
UNSL [26]Prompting-Classifier
0.036
UNSL
Similarity-AVG
0.001
UNSL
Similarity-MAX
0.001
UMU [27]
LexiconMultilingualSentenceTransformer
0.073
UMU
LexiconSentenceTransformer
0.054
GMU-FAST
FAST-DCMN-COS-INJECT
0.001
GMU-FAST
FAST-DCMN-COS-INJECT_FULL
0.001
Mason-NLP [28]MentalBert
0.035

R-PREC P@10 NDCG
0.375
0.359
0.170
0.176
0.359
0.333
0.207
0.249
0.184
0.248
0.052
0.080
0.038
0.150
0.126
0.063
0.044
0.074
0.086
0.146
0.146
0.114
0.140
0.114
0.081
0.061
0.066
0.063
0.075
0.090
0.008
0.011
0.140
0.122
0.002
0.003
0.072

0.861
0.861
0.457
0.500
0.814
0.805
0.471
0.543
0.305
0.600
0.195
0.329
0.162
0.505
0.781
0.367
0.386
0.586
0.629
0.519
0.519
0.471
0.486
0.471
0.538
0.043
0.390
0.376
0.438
0.229
0.010
0.019
0.495
0.362
0.014
0.014
0.286

0.596
0.584
0.277
0.285
0.578
0.550
0.365
0.418
0.329
0.382
0.105
0.150
0.075
0.271
0.211
0.105
0.089
0.139
0.160
0.226
0.221
0.180
0.223
0.189
0.140
0.145
0.114
0.109
0.126
0.180
0.016
0.019
0.222
0.191
0.004
0.005
0.117

During the test stage, participants connected to our server and engaged in an iterative processof receiving user writings and sending their responses. Participants had the discretion to pause
the process and issue an alert at any point in the chronology of user writings. After readingeach user’s writing, teams had to decide between two options: i) issuing an alert about theuser, indicating a predicted sign of gambling risk, or ii) not issuing an alert. Each participant
independently made this choice for every user in the test split. It is important to note that oncean alert was issued, it was considered final, and no further decisions regarding that particular
individual were taken into account. On the other hand, we did not consider the absence ofTable 5
Ranking-based evaluation for Task 1 (unanimity)
Team

Run

MAP R-PREC P@10 NDCG

Formula-ML [20]SentenceTransformers_0.25
0.268
Formula-ML
SentenceTransformers_0.1
0.293
Formula-ML
result2
0.079
Formula-ML
word2vec_0.1
0.085
OBSER-MENH [21] salida-distilroberta-90-cos
0.281
OBSER-MENH
salida-mpnet-90-cos
0.252
OBSER-MENH
salida-distilroberta-21-cos
0.135
OBSER-MENH
salida-mini12-21-cos
0.099
OBSER-MENH
salida-mpnet-21-cos
0.101
uOttawa [22]USESim
0.139
uOttawa
GloveSim
0.008
uOttawa
Glove100Sim
0.011
uOttawa
RobertaSim
0.025
uOttawa
BertSim
0.070
BLUE [23]SemSearchOnBDI2Queries
0.129
BLUE
SemSearchOnAllQueries
0.067
BLUE
SemSearchOnGeneratedQueriesMentalRoberta 0.018
BLUE
SemSearchOnGeneratedQueries
0.052
BLUE
SemSearchOnBDI2QueriesMentalRoberta
0.032
NailP [24]
T1_M2
0.090
NailP
T1_M4
0.090
NailP
T1_M5
0.083
NailP
T1_M1
0.073
NailP
T1_M3
0.073
UMU [27]
LexiconSentenceTransformer
0.044
UMU
LexiconMultilingualSentenceTransformer
0.059
RELAI [25]
BM25
0.012
RELAI
bm25|mpnetbase_simcse
0.026
RELAI
bm25|mpnetqa_simcse
0.023
RELAI
bm25|mpnetqa
0.030
RELAI
bm25|mpnetbase
0.039
UNSL [26]Similarity-MAX
0.001
UNSL
Prompting-Classifier
0.020
UNSL
Similarity-AVG
0.000
GMU-FAST
FAST-DCMN-COS-INJECT_FULL
0.001
GMU-FAST
FAST-DCMN-COS-INJECT
0.001
Mason-NLP [28]MentalBert
0.024

0.360
0.350
0.155
0.163
0.344
0.337
0.216
0.165
0.189
0.232
0.028
0.042
0.068
0.130
0.167
0.105
0.059
0.088
0.058
0.143
0.143
0.139
0.114
0.114
0.110
0.125
0.036
0.059
0.052
0.065
0.069
0.006
0.063
0.005
0.003
0.002
0.054

0.709
0.685
0.357
0.357
0.652
0.643
0.390
0.214
0.319
0.438
0.110
0.110
0.190
0.357
0.643
0.452
0.186
0.381
0.300
0.410
0.410
0.338
0.343
0.343
0.210
0.333
0.019
0.243
0.262
0.290
0.343
0.010
0.090
0.005
0.014
0.010
0.190

0.615
0.611
0.290
0.299
0.604
0.575
0.413
0.329
0.366
0.380
0.063
0.092
0.140
0.260
0.260
0.177
0.085
0.147
0.104
0.229
0.224
0.222
0.192
0.181
0.175
0.209
0.135
0.103
0.097
0.109
0.124
0.012
0.157
0.011
0.006
0.003
0.099

alerts as final, allowing participants to subsequently submit an alert if they detected emerging
signs of risk.
In constructing the ground truth assessments, we employed established approaches that aim
to optimize the utilization of assessors’ time [31, 32]. These methods utilize simulated poolingstrategies, which facilitate the efficient creation of test collections. The key statistics of the testcollection used for Task 2 are presented in Table 6. .
To evaluate the performance of the systems, we used two indicators: the accuracy of the decisions
made and the number of user writings required to reach those decisions. These criteria offer

Table 6
Task 2 (pathological gambling). Main statistics of test collection
Num. subjects
Num. submissions (posts & comments)
Avg num. of submissions per subject
Avg num. of days from first to last submission
Avg num. words per submission

Pathological Gamblers
103
33,719327.33≈ 675
28.9

Control
2071
1,069,152
516.25≈ 878
20.47

valuable insights into the evaluated systems’ effectiveness and efficiency. To support the teststage, the server distributed user writings iteratively and waited for responses from participants.
Importantly, new user data was only provided to a specific participant after the service received
the response from previous evidence from that particular team. The submission period for thetask was open from January 16th, 2023, until April 14th, 2023.

3.1. Decision-based Evaluation
This form of evaluation revolves around the (binary) decisions taken for each user by theparticipating systems. Besides standard classification measures (Precision, Recall and F12 ),we computed 𝐸𝑅𝐷𝐸, the early risk detection error used in previous editions of the lab. A
full description of 𝐸𝑅𝐷𝐸 can be found in [30]. Essentially, 𝐸𝑅𝐷𝐸 is an error measure that
introduces a penalty for late correct alerts (true positives). The penalty grows with the delay inemitting the alert, and the delay is measured here as the number of user posts that had to beprocessed before making the alert.
Since 2019, we complemented the evaluation report with additional decision-based metrics that
try to capture additional aspects of the problem. These metrics try to overcome some limitationsof 𝐸𝑅𝐷𝐸, namely:• the penalty associated to true positives goes quickly to 1. This is due to the functional
form of the cost function (sigmoid).
• a perfect system, which detects the true positive case right after the first round of messages(first chunk), does not get error equal to 0.
• with a method based on releasing data in a chunk-based way (as it was done in 2017 and2018) the contribution of each user to the performance evaluation has a large variance(different for users with few writings per chunk vs users with many writings per chunk).
• 𝐸𝑅𝐷𝐸 is not interpretable.
Some research teams have analysed these issues and proposed alternative ways for evaluation.
Trotzek and colleagues [33] proposed 𝐸𝑅𝐷𝐸𝑜% . This is a variant of ERDE that does not dependon the number of user writings seen before the alert but, instead, it depends on the percentage
of user writings seen before the alert. In this way, user’s contributions to the evaluation arenormalized (currently, all users weight the same). However, there is an important limitation of
2computed with respect to the positive class.

𝐸𝑅𝐷𝐸𝑜% . In real life applications, the overall number of user writings is not known in advance.
Social Media users post contents online and screening tools have to make predictions withthe evidence seen. In practice, you do not know when (and if) a user’s thread of messages is
exhausted. Thus, the performance metric should not depend on knowledge about the total
number of user writings.
Another proposal of an alternative evaluation metric for early risk prediction was done by
Sadeque and colleagues [34]. They proposed 𝐹𝑙𝑎𝑡𝑒𝑛𝑐𝑦 , which fits better with our purposes. Thismeasure is described next.
Imagine a user 𝑢 ∈ 𝑈 and an early risk detection system that iteratively analyzes 𝑢’s writings(e.g. in chronological order, as they appear in Social Media) and, after analyzing 𝑘𝑢 user writings(𝑘𝑢 ≥ 1), takes a binary decision 𝑑𝑢 ∈ {0, 1}, which represents the decision of the system
about the user being a risk case. By 𝑔𝑢 ∈ {0, 1}, we refer to the user’s golden truth label. A
key component of an early risk evaluation should be the delay on detecting true positives (wedo not want systems to detect these cases too late). Therefore, a first and intuitive measure ofdelay can be defined as follows3 :latency𝑇 𝑃= median{𝑘𝑢 : 𝑢 ∈ 𝑈, 𝑑𝑢 = 𝑔𝑢 = 1}

(1)This measure of latency is calculated over the true positives detected by the system and assesses
the system’s delay based on the median number of writings that the system had to process todetect such positive cases. This measure can be included in the experimental report together
with standard measures such as Precision (P), Recall (R) and the F-measure (F):
𝑃

=

𝑅 =
𝐹

=

|𝑢 ∈ 𝑈 : 𝑑𝑢 = 𝑔𝑢 = 1||𝑢 ∈ 𝑈 : 𝑑𝑢 = 1||𝑢 ∈ 𝑈 : 𝑑𝑢 = 𝑔𝑢 = 1||𝑢 ∈ 𝑈 : 𝑔𝑢 = 1|
2·𝑃 ·𝑅
𝑃 +𝑅

(2)
(3)
(4)

Furthermore, Sadeque et al. proposed a measure, 𝐹𝑙𝑎𝑡𝑒𝑛𝑐𝑦 , which combines the effectiveness ofthe decision (estimated with the F measure) and the delay4 in the decision. This is calculated bymultiplying F by a penalty factor based on the median delay. More specifically, each individual(true positive) decision, taken after reading 𝑘𝑢 writings, is assigned the following penalty:
𝑝𝑒𝑛𝑎𝑙𝑡𝑦(𝑘𝑢 ) = −1 +2
1 + exp−𝑝·(𝑘𝑢 −1)(5)where 𝑝 is a parameter that determines how quickly the penalty should increase. In [34], 𝑝 wasset such that the penalty equals 0.5 at the median number of posts of a user5 . Observe that a3

Observe that Sadeque et al (see [34], pg 497) computed the latency for all users such that 𝑔𝑢 = 1. We argue that
latency should be computed only for the true positives. The false negatives (𝑔𝑢 = 1, 𝑑𝑢 = 0) are not detected by
the system and, therefore, they would not generate an alert.
4
Again, we adopt Sadeque et al.’s proposal but we estimate latency only over the true positives.
5
In the evaluation we set 𝑝 to 0.0078, a setting obtained from the eRisk 2017 collection.

0.8
0.4
0.0

penaltydecision right after the first writing has no penalty (i.e. 𝑝𝑒𝑛𝑎𝑙𝑡𝑦(1) = 0). Figure 3 plots how
the latency penalty increases with the number of observed writings.

0

100

200

300

400

number of observed writings

Figure 3: Latency penalty increases with the number of observed writings (𝑘𝑢 )The system’s overall speed factor is computed as:
𝑠𝑝𝑒𝑒𝑑 = (1 − median{𝑝𝑒𝑛𝑎𝑙𝑡𝑦(𝑘𝑢 ) : 𝑢 ∈ 𝑈, 𝑑𝑢 = 𝑔𝑢 = 1})(6)where speed equals 1 for a system whose true positives are detected right at the first writing. A
slow system, which detects true positives after hundreds of writings, will be assigned a speed
score near 0.
Finally, the latency-weighted F score is simply:
𝐹𝑙𝑎𝑡𝑒𝑛𝑐𝑦 = 𝐹 · 𝑠𝑝𝑒𝑒𝑑(7)Since 2019 user’s data were processed by the participants in a post by post basis (i.e. we avoided a
chunk-based release of data). Under these conditions, the evaluation approach has the following
properties:• smooth grow of penalties;• a perfect system gets 𝐹𝑙𝑎𝑡𝑒𝑛𝑐𝑦 = 1 ;• for each user 𝑢 the system can opt to stop at any point 𝑘𝑢 and, therefore, now we do nothave the effect of an imbalanced importance of users;• 𝐹𝑙𝑎𝑡𝑒𝑛𝑐𝑦 is more interpretable than 𝐸𝑅𝐷𝐸.

3.2. Ranking-based Evaluation
In addition to the evaluation discussed above, we employed an alternative form of evaluation
to further assess the systems. After each data release (new user writing), participants wererequired to provide the following information for each user in the collection:• A decision for the user (alert or no alert), which was used to calculate the decision-based
metrics discussed previously.

Table 7
Task 2 (pathological gambling): participating teams, number of runs, number of user writings processedby the team, and lapse of time taken for the entire process.
team
UNSL
ELiRF-UPV
Xabi_EHUOBSER-MENH
RELAI
NLP-UNED-2
NUS-eRisk
BioNLP-IISERB
SINAI
UMU
NLP-UNED

#runs
3
1
5
5
5
5
5
5
5
5
5

#user writingsprocessed

lapse of time(from 1st to last response)

2004
2004
2004
2004
764
2004
2004
61
809
2004
1151

1 day 02:17:36.417
1 day 13:03:54.419
4 days 23:52:41.454
6 days 03:56:44.247
6 days 09:12:00.148
7 days 04:24:33.158
9 days 14:39:26.347
10 days 00:49:40.529
10 days 13:00:02.164
14 days 00:29:30.434
54 days 19:27:42.538• A score representing the user’s level of risk, estimated based on the evidence observed
thus far.
The scores were used to create a ranking of users in descending order of estimated risk. For
each participating system, a ranking was generated at each data release point, simulating acontinuous re-ranking approach based on the observed evidence. In a real-life scenario, thisranking would be presented to an expert user who could make decisions based on the rankings(e.g., by inspecting the top of the rankings). Each ranking can be evaluated using standard
ranking metrics such as P@10 or NDCG. Therefore, we report the performance of the systems
based on the rankings after observing different numbers of writings

3.3. Results
Table 7 presents the participating teams, the number of runs submitted, and the approximate
time duration from the first to the last response. This time-lapse indicates the level of automationand efficiency achieved by each team’s algorithms. While a few submitted runs processed theentire thread of messages (2004), many variants terminated earlier. It is noteworthy that someteams were still submitting results at the deadline. Two teams demonstrated relatively fast
processing times, taking approximately a day to analyze the complete history of user messages.
In contrast, the remaining teams required several days to complete the entire process.
Table 8 reports the decision-based performance achieved by the participating teams. In termsof Precision, 𝐹 1, 𝐸𝑅𝐷𝐸5 , 𝐸𝑅𝐷𝐸50 , and latency-weighted 𝐹 1 the best performing team was
the ELiRF-UPV (run 0). Regarding 𝑙𝑎𝑡𝑒𝑛𝑐𝑦𝑇 𝑃 and 𝑠𝑝𝑒𝑒𝑑 SINAI (runs 0 and 2) are the ones thathaving perfect values obtained the best 𝐹 1. The majority of teams made quick decisions. Overall,these findings indicate that some systems achieved a relatively high level of effectiveness with
only a few user submissions. Social and public health systems may use the best predictive
algorithms to assist expert humans in detecting signs of pathological gambling as early as
possible.

𝐸𝑅𝐷𝐸5

0.854
0.767
0.806
0.883
0.961
0.864
0.913
0.942
0.961
1.000
1.000
1.000
1.000
1.000
0.000
0.971
0.971
0.000
1.000
0.883
0.883
0.922
0.844
0.883
0.951
0.767
0.903
0.971
0.874
0.68
0.592
0.049
0.039
0.000
1.000
1.000
1.000
1.000
0.981
0.388
1.000
1.000
0.311
1.000
0.903
0.845
0.854
0.874
0.728

0.800
0.760
0.798
0.938
0.900
0.877
0.847
0.882
0.846
0.092
0.092
0.092
0.092
0.092
0.000
0.109
0.109
0.000
0.09
0.919
0.914
0.909
0.892
0.819
0.117
0.116
0.113
0.107
0.125
0.787
0.726
0.093
0.075
0.000
0.224
0.206
0.221
0.195
0.168
0.559
0.158
0.092
0.408
0.091
0.108
0.099
0.101
0.103
0.121

0.048
0.048
0.048
0.026
0.030
0.035
0.036
0.033
0.031
0.064
0.063
0.063
0.063
0.063
0.047
0.048
0.048
0.047
0.08
0.034
0.034
0.030
0.038
0.033
0.059
0.068
0.068
0.06
0.065
0.038
0.042
0.045
0.047
0.047
0.029
0.029
0.028
0.03
0.044
0.047
0.039
0.057
0.048
0.053
0.052
0.064
0.056
0.056
0.071

𝑙𝑤𝐹 1

𝐹1

0.752
0.752
0.79
1.000
0.846
0.89
0.79
0.829
0.756
0.048
0.048
0.048
0.048
0.048
0.000
0.058
0.058
0.000
0.047
0.957
0.947
0.896
0.945
0.764
0.062
0.063
0.06
0.057
0.067
0.933
0.938
1.000
1.000
0.000
0.126
0.115
0.124
0.108
0.092
1.000
0.086
0.048
0.593
0.048
0.057
0.053
0.054
0.055
0.066

𝑠𝑝𝑒𝑒𝑑

𝑅

2
0
1
0
0
1
2
3
4
0
1
2
3
4
0
1
2
3
4
1
2
3
0
4
4
0
1
2
3
0
1
3
4
2
3
0
1
2
4
1
0
2
3
4
0
1
2
3
4

𝑙𝑇 𝑃

Run

UNSL [26]UNSL
UNSL
ELiRF-UPV [35]Xabi_EHU [36]Xabi_EHU
Xabi_EHU
Xabi_EHU
Xabi_EHUOBSER-MENH [21]OBSER-MENH
OBSER-MENH
OBSER-MENH
OBSER-MENH
RELAI [25]
RELAI
RELAI
RELAI
RELAI
NLP-UNED-2 [37]NLP-UNED-2
NLP-UNED-2
NLP-UNED-2
NLP-UNED-2
NUS-eRisk
NUS-eRisk
NUS-eRisk
NUS-eRisk
NUS-eRisk
BioNLP-IISERB [38]BioNLP-IISERBBioNLP-IISERBBioNLP-IISERBBioNLP-IISERBSINAI [39]
SINAI
SINAI
SINAI
SINAI
UMU [27]
UMU
UMU
UMU
UMU
NLP-UNED
NLP-UNED
NLP-UNED
NLP-UNED
NLP-UNED

𝐸𝑅𝐷𝐸50Team

𝑃

Table 8
Decision-based evaluation for Task 2.

0.013 14.0 0.949 0.759
0.017 15.0 0.945 0.718
0.014 13.0 0.953 0.761
0.010
4.0 0.988 0.927
0.012
8.0 0.973 0.875
0.017 12.0 0.957 0.839
0.015 13.0 0.953 0.807
0.013 12.0 0.957 0.844
0.013
8.0 0.973 0.823
0.049
3.0 0.992 0.092
0.050
3.0 0.992 0.091
0.050
3.0 0.992 0.091
0.049
3.0 0.992 0.091
0.050
3.0 0.992 0.091
0.047
0.039
1.0 1.000 0.109
0.039
1.0 1.000 0.109
0.047
0.046 11.0 0.961 0.087
0.016 13.0 0.953 0.876
0.016 12.0 0.957 0.875
0.014 10.0 0.964 0.877
0.019 18.0 0.933 0.833
0.010 13.0 0.953 0.781
0.040
6.0 0.981 0.114
0.050 27.0 0.899 0.104
0.043 13.0 0.953 0.107
0.042
4.0 0.988 0.106
0.042 17.0 0.938 0.117
0.037 62.0 0.766 0.603
0.042 62.0 0.766 0.557
0.045
1.0 1.000 0.093
0.046 19.0 0.930 0.070
0.047
0.020
2.0 0.996 0.223
0.021
1.0 1.000 0.206
0.020
2.0 0.996 0.220
0.022
1.0 1.000 0.195
0.027
3.0 0.992 0.166
0.043 94.5 0.651 0.364
0.029
2.0 0.996 0.157
0.044
2.0 0.996 0.091
0.045 80.0 0.701 0.286
0.045
2.0 0.996 0.090
0.052
1.0 1.000 0.108
0.063 141.0 0.502 0.050
0.055
1.0 1.000 0.101
0.056
1.0 1.000 0.103
0.071 142.0 0.500 0.060

Table 9 presents the ranking-based results. Because some teams only processed a few dozens
of user writings, we could only compute their user rankings for the initial rounds. For tie
breaking in the scores for the users, we used the traditional docid criteria (subject name). SINAI(run 1) obtained the best overall values after only one writing. At the other evaluation points,

Table 9
Ranking-based evaluation for Task 2.

1.00 1.00
1.00 1.00
1.00 1.00
1.00 1.00
1.00 1.00
1.00 1.00
1.00 1.00
1.00 1.00
1.00 1.00
1.00 1.00
1.00 1.00
1.00 1.00
1.00 1.00
1.00 1.00
0.30 0.25
0.00 0.00
0.00 0.00
0.00 0.00
0.10 0.06
1.00 1.00
1.00 1.00
1.00 1.00
1.00 1.00
1.00 1.00
0.00 0.00
0.00 0.00
0.00 0.00
0.00 0.00
0.00 0.00
0.40 0.60
0.00 0.00
0.00 0.00
0.00 0.00
0.10 0.10
1.00 1.00
1.00 1.00
1.00 1.00
1.00 1.00
0.80 0.86
1.00 1.00
0.00 0.00
0.00 0.00
0.00 0.00
0.00 0.00
0.00 0.00
0.00 0.00
0.00 0.00
0.00 0.00
0.00 0.00

0.46
0.57
0.55
0.59
0.57
0.56
0.55
0.56
0.58
0.64
0.65
0.65
0.64
0.65
0.08
0.01
0.02
0.01
0.02
0.32
0.40
0.40
0.59
0.57
0.06
0.03
0.04
0.06
0.06
0.14
0.02
0.03
0.05
0.10
0.72
0.73
0.71
0.72
0.53
0.63
0.03
0.00
0.03
0.00
0.03
0.03
0.03
0.03
0.03

ELiRF-UPV was again the best performing.

1.00 1.00
1.00 1.00
1.00 1.00
1.00 1.00
1.00 1.00
0.90 0.94
1.00 1.00
0.90 0.94
1.00 1.00
1.00 1.00
1.00 1.00
1.00 1.00
1.00 1.00
1.00 1.00
0.00 0.00
0.00 0.00
0.00 0.00
0.00 0.00
0.00 0.00
1.00 1.00
1.00 1.00
1.00 1.00
1.00 1.00
1.00 1.00
0.00 0.00
0.00 0.00
0.00 0.00
0.00 0.00
0.00 0.00

0.70
0.78
0.75
0.91
0.50
0.49
0.51
0.51
0.50
0.55
0.56
0.56
0.56
0.56
0.02
0.00
0.01
0.01
0.04
0.83
0.86
0.85
0.92
0.89
0.02
0.02
0.03
0.04
0.03

1.00 1.00
1.00 1.00
1.00 1.00
1.00 1.00
0.80 0.88
0.70 0.76
0.70 0.79
0.80 0.86
0.80 0.86
1.00 1.00
1.00 1.00
1.00 1.00
1.00 1.00
1.00 1.00
0.00 0.00
0.00 0.00
0.00 0.00
0.00 0.00
0.00 0.00
1.00 1.00
1.00 1.00
1.00 1.00
1.00 1.00
1.00 1.00
0.10 0.19
0.10 0.19
0.10 0.10
0.10 0.19
0.10 0.12

0.64
0.67
0.69
0.95
0.41
0.38
0.40
0.41
0.40
0.48
0.49
0.49
0.48
0.49
0.06
0.00
0.02
0.00
0.02
0.90
0.93
0.92
0.95
0.89
0.07
0.06
0.03
0.08
0.05

1.00 1.00
1.00 1.00
1.00 1.00
1.00 1.00
0.90 0.94
0.00 0.00
0.00 0.00
0.00 0.00
0.30 0.31
0.00 0.00
0.00 0.00
0.00 0.00
0.00 0.00
0.00 0.00
0.00 0.00

0.88
0.90
0.87
0.89
0.56
0.01
0.01
0.01
0.12
0.01
0.02
0.02
0.02
0.02
0.02

1.00 1.00
1.00 1.00
1.00 1.00
1.00 1.00
0.70 0.80
0.00 0.00
0.00 0.00
0.00 0.00
0.40 0.36
0.00 0.00
0.00 0.00
0.00 0.00
0.00 0.00
0.00 0.00
0.00 0.00

0.85
0.84
0.86
0.47
0.00
0.00
0.09
0.20
0.09
0.02
0.02
0.02
0.02
0.02

𝑁 𝐷𝐶𝐺@100

𝑁 𝐷𝐶𝐺@101000 writings

𝑃 @10

𝑁 𝐷𝐶𝐺@100

𝑁 𝐷𝐶𝐺@10

500 writings

𝑃 @10

𝑁 𝐷𝐶𝐺@100

𝑁 𝐷𝐶𝐺@10

𝑃 @10

0
1
2
0
0
1
2
3
4
0
1
2
3
4
0
1
2
3
4
0
1
2
3
4
0
1
2
3
4
0
1
2
3
4
0
1
2
3
4
0
1
2
3
4
0
1
2
3
4

100 writings
𝑁 𝐷𝐶𝐺@100

Run

UNSL [26]UNSL
UNSL
ELiRF-UPV [35]Xabi_EHU [36]Xabi_EHU
Xabi_EHU
Xabi_EHU
Xabi_EHUOBSER-MENH [21]OBSER-MENH
OBSER-MENH
OBSER-MENH
OBSER-MENH
RELAI [25]
RELAI
RELAI
RELAI
RELAI
NLP-UNED-2 [37]NLP-UNED-2
NLP-UNED-2
NLP-UNED-2
NLP-UNED-2
NUS-eRisk
NUS-eRisk
NUS-eRisk
NUS-eRisk
NUS-eRisk
BioNLP-IISERB [38]BioNLP-IISERBBioNLP-IISERBBioNLP-IISERBBioNLP-IISERBSINAI [39]
SINAI
SINAI
SINAI
SINAI
UMU [27]
UMU
UMU
UMU
UMU
NLP-UNED
NLP-UNED
NLP-UNED
NLP-UNED
NLP-UNED𝑁 𝐷𝐶𝐺@10

Team

𝑃 @10

1 writing

1.00 1.00
1.00 1.00
1.00 1.00
1.00 1.00
0.90 0.94
0.80 0.88
0.80 0.88
0.90 0.94
0.90 0.94
1.00 1.00
1.00 1.00
1.00 1.00
1.00 1.00
1.00 1.00

0.64
0.70
0.69
0.94
0.41
0.40
0.41
0.42
0.41
0.50
0.50
0.50
0.50
0.50

1.00 1.00
1.00 1.00
1.00 1.00
1.00 1.00
1.00 1.00
0.00 0.00
0.00 0.00
0.00 0.00
0.00 0.00
0.00 0.00

0.90
0.94
0.93
0.93
0.87
0.02
0.01
0.01
0.02
0.02

0.00 0.00
0.00 0.00
0.20 0.16
0.50 0.50
0.20 0.16
0.00 0.00
0.00 0.00
0.00 0.00
0.00 0.00
0.00 0.00

0.00
0.00
0.14
0.23
0.14
0.02
0.02
0.02
0.02
0.02

4. Task 3: Measuring the Severity of Eating Disorders
This task aims to estimate the severity of different symptoms associated with the diagnosisof eating disorders. Participants were provided with a thread of user submissions to analyze.
For each user, we provided the participants with a collection of posts and comments, andparticipants were tasked with estimating the user’s responses to a standardized eating disorderquestionnaire based on the evidence found in the history of posts and comments.
The questionnaire used in this task is derived from the Eating Disorder Examination Questionnaire (EDE-Q)6 , which is a self-reported questionnaire comprising 28 items. It is adapted fromthe semi-structured interview Eating Disorder Examination (EDE)7 [40]. For this task, our focus
was on questions 1-12 and 19-28 from the EDE-Q. This questionnaire is specifically designed toassess various aspects and the severity of features associated with eating disorders. It consistsof four subscales: Restraint, Eating Concern, Shape Concern, and Weight Concern, along with a
global score. An excerpt of the EDE-Q is provided in Table 10.

Table 10: Excerpt of the Eating Disorder Examination Questionarie
Instructions:
The following questions are concerned with the past four weeks (28days) only. Please read each question carefully. Please answer all thequestions. Thank you..
1. Have you been deliberately trying to limit the amount of food you eat
to influence your shape or weight (whether or not you have succeeded)
0. NO DAYS
1. 1-5 DAYS
2. 6-12 DAYS
3. 13-15 DAYS
4. 16-22 DAYS
5. 23-27 DAYS
6. EVERY DAY
2. Have you gone for long periods of time (8 waking hours or more)
without eating anything at all in order to influence your shape or
weight?
0. NO DAYS
1. 1-5 DAYS
2. 6-12 DAYS
3. 13-15 DAYS
4. 16-22 DAYS
6
7

https://www.corc.uk.net/media/1273/ede-q_quesionnaire.pdf
https://www.corc.uk.net/media/1951/ede_170d.pdf

Table 10: Eating Disorder Examination Questionarie (continued)
5. 23-27 DAYS
6. EVERY DAY
3. Have you tried to exclude from your diet any foods that you likein order to influence your shape or weight (whether or not you havesucceeded)?
0. NO DAYS
1. 1-5 DAYS
2. 6-12 DAYS
3. 13-15 DAYS
4. 16-22 DAYS
5. 23-27 DAYS
6. EVERY DAY.
.
.
22. Has your weight influenced how you think about (judge) yourself asa person?
0. NOT AT ALL (0)1. SLIGHTY (1)2. SLIGHTY (2)3. MODERATELY (3)4. MODERATELY (4)5. MARKEDLY (5)6. MARKEDLY (6)23. Has your shape influenced how you think about (judge) yourself asa person?
0. NOT AT ALL (0)1. SLIGHTY (1)2. SLIGHTY (2)3. MODERATELY (3)4. MODERATELY (4)5. MARKEDLY (5)6. MARKEDLY (6)24. How much would it have upset you if you had been asked to weighyourself once a week (no more, or less, often) for the next four weeks?
0. NOT AT ALL (0)1. SLIGHTY (1)2. SLIGHTY (2)Table 10: Eating Disorder Examination Questionarie (continued)
3.
4.
5.
6.

MODERATELY (3)MODERATELY (4)MARKEDLY (5)MARKEDLY (6)The main objective of this task was to investigate the feasibility of automatically estimating theseverity of multiple symptoms associated with eating disorders. The participating algorithms
estimated the user’s response to each question based on their writing history. To evaluate theperformance of the systems, we collected questionnaires completed by users on social media,
along with their corresponding writing history. These user-completed questionnaires served as
the ground truth against which the responses provided by the systems were evaluated.
During the training phase, participants were provided with data from 28 users who participatedin the 2022 edition [14]. This training data included the writing history of the users as well as
their responses to the EDE-Q questions. In the test phase, there were 46 new users for whom theparticipating systems had to generate results. The expected format for submitting the results
followed a specific file format:
username1 answer1 answer2...answer12 answer19...answer28
username2 answer1 answer2...answer12 answer19...answer28.
.
.

Each line has the username and 22 values (no answers from 13 to 18). These values correspondwith the responses to the questions above (the possible values are 0,1,2,3,4,5,6).

4.1. Evaluation Metrics
Evaluation is based on the following effectiveness metrics:• Mean Zero-One Error (𝑀 𝑍𝑂𝐸) between the questionnaire filled by the real user andthe questionnaire filled by the system (i.e. fraction of incorrect predictions).
𝑀 𝑍𝑂𝐸(𝑓, 𝑄) =|{𝑞𝑖 ∈ 𝑄 : 𝑅(𝑞𝑖 ) ̸= 𝑓 (𝑞𝑖 )}|
|𝑄|

(8)

where 𝑓 denotes the classification done by an automatic system, 𝑄 is the set of questionsof each questionnaire, 𝑞𝑖 is the i-th question, 𝑅(𝑞𝑖 ) is the real user’s answer for the i-thquestion and 𝑓 (𝑞𝑖 ) is the predicted answer of the system for the i-th question. Each user
produces a single 𝑀 𝑍𝑂𝐸 score and the reported 𝑀 𝑍𝑂𝐸 is the average over all 𝑀 𝑍𝑂𝐸
values (mean 𝑀 𝑍𝑂𝐸 over all users).
• Mean Absolute Error (𝑀 𝐴𝐸) between the questionnaire filled by the real user and thequestionnaire filled by the system (i.e. average deviation of the predicted response fromthe true response).

∑︀
𝑀 𝐴𝐸(𝑓, 𝑄) =𝑞𝑖 ∈𝑄 |𝑅(𝑞𝑖 ) − 𝑓 (𝑞𝑖 )|

|𝑄|

(9)

Again, each user produces a single 𝑀 𝐴𝐸 score and the reported 𝑀 𝐴𝐸 is the averageover all 𝑀 𝐴𝐸 values (mean 𝑀 𝐴𝐸 over all users).
• Macroaveraged Mean Absolute Error (𝑀 𝐴𝐸𝑚𝑎𝑐𝑟𝑜 ) between the questionnaire filledby the real user and the questionnaire filled by the system (see [41]).
6

1 ∑︁
𝑀 𝐴𝐸𝑚𝑎𝑐𝑟𝑜 (𝑓, 𝑄) =
7
𝑗=0

∑︀

𝑞𝑖 ∈𝑄𝑗 |𝑅(𝑞𝑖 ) − 𝑓 (𝑞𝑖 )|

|𝑄𝑗 |

(10)

where 𝑄𝑗 represents the set of questions whose true answer is 𝑗 (note that 𝑗 goes from 0
to 6 because those are the possible answers to each question). Again, each user produces asingle 𝑀 𝐴𝐸𝑚𝑎𝑐𝑟𝑜 score and the reported 𝑀 𝐴𝐸𝑚𝑎𝑐𝑟𝑜 is the average over all 𝑀 𝐴𝐸𝑚𝑎𝑐𝑟𝑜
values (mean 𝑀 𝐴𝐸𝑚𝑎𝑐𝑟𝑜 over all users).
The following measures are based on aggregated scores obtained from the questionnaires.
Further details about the EDE-Q instruments can be found elsewhere (e.g. see the scoring
section of the questionnaire).
• Restraint Subscale (RS): Given a questionnaire, its restraint score is obtained as themean response to the first five questions. This measure computes the RMSE between the
restraint ED score obtained from the questionnaire filled by the real user and the restraint
ED score obtained from the questionnaire filled by the system.
Each user 𝑢𝑖 is associated with a real subscale ED score (referred to as 𝑅𝑅𝑆 (𝑢𝑖 )) and an
estimated subscale ED score (referred to as 𝑓𝑅𝑆 (𝑢𝑖 )). This metric computes the RMSE
between the real and an estimated subscale ED scores as follows:√︃ ∑︀2𝑢𝑖 ∈𝑈 (𝑅𝑅𝑆 (𝑢𝑖 ) − 𝑓𝑅𝑆 (𝑢𝑖 ))𝑅𝑀 𝑆𝐸(𝑓, 𝑈 ) =
(11)
|𝑈 |
where 𝑈 is the user set.
• Eating Concern Subscale (ECS): Given a questionnaire, its eating concern score is
obtained as the mean response to the following questions (7, 9, 19, 21, 20). This metriccomputes the RMSE (equation 12) between the eating concern ED score obtained fromthe questionnaire filled by the real user and the eating concern ED score obtained fromthe questionnaire filled by the system.
√︃ ∑︀2𝑢𝑖 ∈𝑈 (𝑅𝐸𝐶𝑆 (𝑢𝑖 ) − 𝑓𝐸𝐶𝑆 (𝑢𝑖 ))𝑅𝑀 𝑆𝐸(𝑓, 𝑈 ) =
(12)
|𝑈 |
• Shape Concern Subscale (SCS): Given a questionnaire, its shape concern score is
obtained as the mean response to the following questions (6, 8, 23, 10, 26, 27, 28, 11). This
metric computes the RMSE (equation 13) between the shape concern ED score obtained
from the questionnaire filled by the real user and the shape concern ED score obtained
from the questionnaire filled by the system.

√︃ ∑︀
𝑅𝑀 𝑆𝐸(𝑓, 𝑈 ) =2
𝑢𝑖 ∈𝑈 (𝑅𝑆𝐶𝑆 (𝑢𝑖 ) − 𝑓𝑆𝐶𝑆 (𝑢𝑖 ))

|𝑈 |

(13)

• Weight Concern Subscale (WCS): Given a questionnaire, its weight concern score is
obtained as the mean response to the following questions (22, 24, 8, 25, 12). This metriccomputes the RMSE (equation 14) between the weight concern ED score obtained fromthe questionnaire filled by the real user and the weight concern ED score obtained fromthe questionnaire filled by the system.
√︃ ∑︀2𝑢𝑖 ∈𝑈 (𝑅𝑊 𝐶𝑆 (𝑢𝑖 ) − 𝑓𝑊 𝐶𝑆 (𝑢𝑖 ))𝑅𝑀 𝑆𝐸(𝑓, 𝑈 ) =
(14)
|𝑈 |
• Global ED (GED): To obtain an overall or ‘global’ score, the four subscales scores are
summed and the resulting total divided by the number of subscales (i.e. four) [40]. This
metric computes the RMSE between the real and an estimated global ED scores as follows:√︃ ∑︀2𝑢𝑖 ∈𝑈 (𝑅𝐺𝐸𝐷 (𝑢𝑖 ) − 𝑓𝐺𝐸𝐷 (𝑢𝑖 ))𝑅𝑀 𝑆𝐸(𝑓, 𝑈 ) =
(15)
|𝑈 |

4.2. Results
Table 11 presents the results obtained by the participants in this task. To provide context, thetable includes the performance of three baseline variants in the top block: "all 0s," "all 6s," and"average." The "all 0s" variant represents a strategy where the same response (0) is submitted for
all questions. Similarly, the "all 6s" variant submits response 6 for all questions. The "average"
variant calculates the mean of the responses provided by all participants for each question andsubmits the response that is closest to this mean value (e.g., if the mean response provided by
the participants is 3.7, then this average approach would submit a 4).
The results indicate that the top-performing system in terms of Mean Absolute Error (MAE) was
run 0 by UMU. However, this particular run did not outperform the naive baseline approach ofsubmitting all 0s in terms of Mean Zero-One Error (MZOE). Among the participating systems,
GMU-FAST’ run 3 achieved the best performance in two metrics: 𝑀 𝐴𝐸𝑚𝑎𝑐𝑟𝑜 and Global ED.
For the Eating Concern Subscale, the best-performing system was GMU-FAST’ run 1, while for
the Shape subscale, RiskBusters’ run 4 showed the highest performance.

5. Participating Teams
Table 12 reports the participating teams and the runs that they submitted for each eRisk task.
The next paragraphs give a brief summary on the techniques implemented by each of them.
Further details are available at the CLEF 2023 working notes proceedings for the participants.
BFH-AMI [42]. The BFH-AMI team, affiliated with Applied Machine Intelligence of Switzerland,participated in Task 3 of the eRisk 2023 challenge. The team employed a logistic regression modelthat incorporated user and question embeddings derived from the Large Language Models. To

WCS

SCS

ECS

RS

GED

𝑀 𝐴𝐸𝑚𝑎𝑐𝑟𝑜

MZOE

MAE

Table 11
Task 3 Results. Participating teams and runs with corresponding scores for the metrics.

team

run IDbaselinebaselinebaseline

all0s 2.419 0.674 2.803 3.207 2.138 3.221 3.028 2.682
all6s 3.581 0.834 3.995 3.839 4.814 3.650 3.950 3.318
average 2.091 0.859 1.957 2.391 1.592 2.398 2.162 2.002

BFH-AMI [42]
GMU-FAST
GMU-FAST
GMU-FAST
GMU-FAST
GMU-FAST
RiskBusters [43]
RiskBusters
RiskBusters
RiskBusters
RiskBusters
RiskBusters
RiskBusters
RiskBusters
UMU [27]

0
0
1
2
3
4
0
1
2
3
4
5
6
7
0

2.407
2.529
2.525
2.738
2.671
2.534
2.338
2.352
2.396
2.419
2.346
2.334
2.408
2.347
2.194

0.719
0.902
0.903
0.764
0.833
0.796
0.691
0.699
0.704
0.709
0.705
0.702
0.696
0.696
0.800

2.729 3.169
2.012 2.498
1.992 2.487
2.058 2.708
1.741 1.999
1.879 2.174
1.922 2.294
1.858 2.127
1.861 2.178
1.898 2.251
1.859 2.217
1.854 2.230
1.936 2.365
1.975 2.534
2.027 2.288

2.597
2.585
2.584
2.278
2.740
2.469
1.866
2.025
1.859
1.935
1.862
1.898
2.048
1.911
1.777

2.854 2.923
1.948 1.950
1.924 1.917
2.641 2.295
2.053 2.083
2.136 2.033
2.492 1.999
2.365 2.034
2.484 1.957
2.440 2.037
2.398 1.898
2.381 1.947
2.536 1.985
2.443 2.215
2.412 2.556

2.414
2.221
2.219
2.662
2.401
2.387
2.425
2.466
2.468
2.445
2.395
2.378
2.414
2.494
2.135

address the challenge, the authors employed two different models, namely BERT and GPT-Large,
to generate embeddings for users’ posts and publications. These embeddings were then utilized
as input features for the logistic regression model. The aim was to capture the underlying
patterns and information within the social media writing history of patients that could help
predict their responses to the EDE-Q. The use of logistic regression allowed the team to build apredictive model that could estimate the likelihood of different responses based on the extracted
features. By incorporating user and question embeddings the model could effectively leverage
the semantic information and context within the social media writings.
BLUE [23]. The BLUE team participated in Task 1 of the eRisk 2023 challenge. Their approach
involved using dense retrievers for semantic search, with the aim of retrieving relevant sentencesfrom a collection. They employed two types of queries for the search process. First, they utilized
the descriptions of each symptom found in the BDI-II questionnaire. These symptom descriptions
served as queries for the search. Additionally, they leveraged ChatGPT, a language model, togenerate synthetic queries for each symptom in the BDI-II questionnaire. The team believedthat using ChatGPT to generate synthetic queries would introduce more diversity in expressions
and potentially yield more relevant sentences during the search process. The team conductedfive runs of their approach and employed two transformer-based models for embedding thesocial media posts. These models were the original and generated responses of the BDI-II,
MentalRoBERTa, and a variant of MPNet. Surprisingly, the model that performed semantic

Table 12
eRisk 2023 participants.
team[42] BFH-AMI[23] BLUE[38] BioNLP-IISERB[35] ELiRF-UPV[20] Formula-ML
GMU-FAST[28] Mason-NLP[24] NailP
NLP-UNED[37] NLP-UNED-2
NUS-eRisk[21] OBSER-MENH[25] RELAI[43] RiskBusters[39] SINAI[27] UMU[26] UNSL[22] uOttawa[36] Xabi_EHU

Task 1
#runs
5
4
2
1
5

5
5
2
3
5

Task 2#runs

Task 3
#runs
1

5
1
5
5
5
5
5
5
5
5
3

8
1

5

search using the initial responses from the BDI-II questionnaire as queries achieved better
performance compared to the model using generated queries. Furthermore, it was observedthat the generated synthetic data was too specific for this particular task.
BioNLP-IISERB [38]. BioNLP-IISERB participated in Task 2 of the challenge. The teamsubmitted five different runs, employing various text-mining frameworks and strategies fortext classification and feature engineering. To extract features from the social media posts,
BioNLP-IISERB utilized both traditional and transformer-based approaches. They employed thebag of words model, specifically TF-IDF weighting, as well as transformer-based embeddingssuch as BERT, Longformer, and RoBERTa. For the classification stage, the team explored
several classifiers to determine their effectiveness in the task. They utilized classifiers such as
Adaptive Boosting (AdaBoost), Logistic Regression, Support Vector Machine (SVM), Decision
Tree, Random Forest, and transformer-based classifiers. Among their different approaches,
BioNLP-IISERB found that combining a Support Vector Machine and an Adaptive Boostingclassifier yielded their most successful performance.
ELiRF-UPV [35]. ELiRF-UPV, associated with the Valencian Research Institute for Artificial
Intelligence and the informatics School of Universitat Politècnica de València, participated in
the eRisk 2023 Task 2. Their approach centered around utilizing SVMs, a traditional machinelearning technique. The ELiRF-UPV team employed SVMs with different kernels and regularization parameters in their experiments. They utilized cross-validation techniques to assess
the performance of the SVM classifiers. To represent the text data, they employed a TF-IDF
vectorizer with a maximum limit of 5,000 features. Notably, the ELiRF-UPV team achieved thehighest scores in Precision, F1 score, ERDE5, ERDE50, and latency-weighted F1 in the eRisk
2023 Task 2.
Formula-ML [20]. The Formula-ML team participated in Task 1. To represent the sentences,
Formula-ML utilized various techniques such as word2vec, the TF-IDF model, and sentencetransformers. These techniques allowed them to convert the text data into numerical representations or embeddings. To determine the degree of correlation between the sentences and theanswers from the BDI questionnaire, Formula-ML utilized a soft cosine similarity measure. Thismeasure assessed the relatedness of the topics discussed in the sentences to the symptoms ofdepression. The results obtained by Formula-ML varied depending on the embedding modelused. The models based on SentenceTransformers achieved great performance and even ranked
highly in some evaluation metrics. However, the word2vec-based models did not reach thesame level of performance.
Mason-NLP [28]. The Mason-NLP team from George Mason University participated in Task 1of the CLEF eRisk lab. Their approach incorporated two deep learning models, MentalBERT
and RoBERTa, along with LSTM. The team employed several steps to reduce the number ofsentences before passing them through the MentalBERT model, which computed an embedding
representation for each sentence. The ranking of each symptom was then determined by
calculating the cosine similarity between the embedding representation of each sentence andthe embedding representation of the symptom options. However, the evaluation results of theirapproach did not meet the expectations of the authors. Despite this outcome, they acknowledged
the opportunity for improvement and proposed several avenues for future work. One suggestion
was to fine-tune MentalBERT using BDI training data, which could potentially enhance themodel’s performance.
NailP [24]. The NailP team participated in Task 1. Their approach focused on data preprocessing and calculating similarity between text representations. Initially, the authors performed data pre-processing by selecting publications that contained self-referential content,
specifically highlighting personal pronouns. They then filtered out positive or neutral sentences, focusing only on those with potentially negative sentiment. Sentence embeddings
were calculated using SBERT (Sentence-BERT), a model that generates contextualized sentence
embeddings. Additionally, the team obtained embeddings from the descriptions of the BDI-II
items using the same SBERT model. To rank the sentences, a semantic search was conductedwith different filters to include or exclude negative posts.
NLP-UNED-2 [37]. The NLP-UNED-2, from the Spanish National University of Distance
Education, participated in Task 2 of the eRisk 2023 challenge. Their approach consisted ofdataset relabeling using Approximate Nearest Neighbors (ANN) on vectorial representationsof messages. ANN is a technique that finds approximate matches for a given query in a highdimensional space. By utilizing ANN, the team aimed to improve the labeling of the dataset. For
classification, the UNED team employed neural networks. These neural networks were used toclassify and identify instances of pathological gambling within the social media data.
OBSER-MENH [21]. The OBSER-MENH team participated in both Task 1 and Task 2 ofthe eRisk challenge. For Task 1, their approach involved using SBERT to compute vector
representations of each publication. They then utilized the descriptions of the BDI-II as queries
to rank depressive symptoms. By leveraging SBERT and the BDI-II descriptions, they aimed toassess the relevance of each symptom in the publications. In Task 2, the OBSER-MENH teamfocused on addressing class imbalance and preventing overfitting. They employed an ensemble
approach that combined variants of three models. The ensemble aimed to analyze the different
penalty weights applied to a feedforward neural network (FNN). This approach allowed themto train multiple models with varying penalty weights, helping to find an optimal balance andprevent overfitting.
RELAI [25]. The RELAI team, a collaboration between universities in Quebec and McMaster,participated in both Task 1 and Task 2. For Task 1, the team utilized transformer-based sentence
encoders and employed the Okapi BM25 method to select sentences. The selected sentenceswere then subjected to a similarity task, and the similarity scores were used for ranking, withdifferent strategies employed. While their results did not yield significant improvements, they
did outperform their chosen simple baseline in some cases. In Task 2, the RELAI team employed"lightweight" approaches. One approach involved extracting stylometric and shallow features,such as character and n-gram frequencies and sentence length. These features were then fed
into a multilayer perceptron for classification. Another approach utilized topic modeling of theposts, with the resulting topics used as input to a multilayer perceptron. The final approach
took a probabilistic approach, estimating the proportion of positive users from past editions todetermine the gamma distribution from which they emerge.
RiskBusters [43]. The RiskBuster team from the University of Bucharest participated in Task
3. Their approach involved the use of a transformer-based topic modeling method to measure
the severity of eating disorder symptoms. The team customized the BERTopic framework to
obtain topic distributions at the user level. These topic distributions were then utilized as input
features for downstream classification tasks. By leveraging topic modeling, the team aimed tocapture the underlying themes and severity of eating disorder symptoms within the social media
writings. To improve the quality of embeddings, the RiskBuster team adapted MentalBERT, atransformer-based language model, to the eating disorder domain.
SINAI [39]. The SINAI team, a collaboration between Jaén and Bocconi universities, participatedTask 2. They developed various approaches using XLM-RoBERTa and RoBERTa transformer
models, with their most innovative proposal combining LSTM and RoBERTa architectures. Theirapproach involved encoding each user’s post using RoBERTa to obtain embeddings, which werethen passed through an LSTM. The final prediction was made using a Feed Forward Network.
The team also designed pre-processing steps for the corpus before training. With this approach,they achieved the highest Recall score among all participants for the binary task and had thesecond-highest ERDE value. In terms of ranking-based evaluation, they achieved one of thehighest positions.
UMU [27]. The UMU team introduced different strategies to address the three distinct tasks inthe eRisk 2023 challenge. For Task 1, the team approached it as a question-answering problem
and presented two proposals: one for monolingual text and another for multilingual text. They
employed a pre-trained sentence transformer model to assess the severity of each depressive
symptom within users’ text collections. The team used a depression-domain lexicon approachto select texts closely related to depression and ranked the 21 depression symptoms of the BDI
questionnaire. By treating Task 1 as a question-answering task, they linked the questions of the
BDI questionnaire to their possible answers in users’ writings. This approach allowed themto determine the degree of relationship between each text and the 21 symptoms of depression.
For Task 2, which focused on the early detection of signs of pathological gambling, UMU teampresented five runs that combined decision-making strategies with classification models. In
Task 3, UMU team presented a run based on fine-tuning a pre-trained sentence transformer
model. They processed the dataset and performed emoji feature extraction to enhance themodel’s performance.
UNSL [26]. The UNSL (Universidad Nacional de San Luis) team participated in both Task 1
and Task 2. For Task 1, they submitted three proposals. The first two proposals focused on thesimilarity of contextualized embedding vectors between the posts and the symptoms of the
Beck Depression Inventory. The third proposal utilized a prompting strategy, reformulating
the sentence retrieval task into a masked language problem. They used ChatGPT to generate
synthetic examples related to a specific symptom, and a RoBERTa model was fine-tuned to solve
the masking language problem. The prompting-based strategy yielded better results than thesimilarity-based proposals, providing promising outcomes for information retrieval. For Task 2,
related to the early detection of signs of pathological gambling, the UNSL team proposed threefine-tuned models, followed by a decision policy based on criteria defined by an early detection
framework. One of the models incorporated an extended vocabulary with important words
specific to the domain. The UNSL models demonstrated good performances in decision-based
metrics, ranking-based metrics, and runtime for Task 2.
uOttawa [22]. The uOttawa team, representing the University of Ottawa, participated in Task
1. They employed a combination of word embedding models, including GloVe, DistilBERT,RoBERTa, and the Universal Sentence Encoder (USE). Among the models submitted by theuOttawa team, the Universal Sentence Encoder (USE) consistently performed the best forevery metric, whether using majority voting or unanimity for the qrels. The methods basedon contextual representations, including USE, RoBERTa, and DistilBERT, outperformed themethods based on GloVe embeddings. After empirical analysis, the team hypothesized that thebetter performance of contextual-based methods could be attributed to the removal of pronouns,
which may contain relevant information in the discourse. This removal was not applied to themethods based on GloVe embeddings.
Xabi_EHU [36]. The Xabi_EHU team, representing the University of the Basque Country (UPV/EHU), participated in Task 2. To address the issue of class imbalance, where the PathologicalGamblers group is smaller than the Control group, the team employed a neural network with acustomized loss function. This approach allowed them to adjust penalties for false positives andfalse negatives, providing flexibility to meet specific requirements. By utilizing this customized
loss function, their training approach did not penalize false positives and negatives equally.

6. Conclusions
This paper provides an overview of eRisk 2023, which marked the seventh edition of the lab.
The lab focused on three distinct tasks: symptoms search (Task 1 on depression), early detection(Task 2 on pathological gambling), and severity estimations (Task 3 on eating disorders). In Task
1, participants were presented with a collection of sentences and tasked with ranking them basedon their relevance to each symptom of depression outlined in the BDI-II questionnaire. Task 2
required participants to sequentially analyze social media posts and issue alerts for individuals
displaying signs of gambling risk. In Task 3, participants were provided with the complete user

history and required to automatically estimate the user’s responses to a standardized depression
questionnaire.
A total of 105 runs were submitted by 20 teams for the proposed tasks. While the effectivenessof the solutions varies across tasks, the experimental results highlight the value of extracting
evidence from social media. This suggests the promising potential of automatic or semiautomatic screening tools for detecting at-risk individuals. These findings emphasize thenecessity for the development of benchmarks for text-based risk indicator screening.

Acknowledgments
This work was supported by project PLEC2021-007662 (MCIN/AEI/10.13039/ 501100011033,
Ministerio de Ciencia e Innovación, Agencia Estatal de Investigación, Plan de Recuperación,
Transformación y Resiliencia, Unión Europea-Next Generation EU). The first and second authors thank the financial support supplied by the Xunta de Galicia-Consellería de Cultura,
Educación, Formación Profesional e Universidade (accreditation 2019-2022 ED431G/01, GPC
ED431B 2022/33) and the European Regional Development Fund, which acknowledges the CITIC
Research Center in ICT of the University of A Coruña as a Research Center of the Galician
University System. The third author thanks the financial support supplied by the Xunta de
Galicia-Consellería de Cultura, Educación, Formación Profesional e Universidade (accreditation
2019-2022 ED431G-2019/04, ED431C 2022/19) and the European Regional Development Fund,which acknowledges the CiTIUS-Research Center in Intelligent Technologies of the Universityof Santiago de Compostela as a Research Center of the Galician University System.

References[1] D. E. Losada, F. Crestani, J. Parapar, eRisk 2017: CLEF lab on early risk prediction on
the internet: Experimental foundations, in: G. J. Jones, S. Lawless, J. Gonzalo, L. Kelly,
L. Goeuriot, T. Mandl, L. Cappellato, N. Ferro (Eds.), Experimental IR Meets Multilinguality,
Multimodality, and Interaction, Springer International Publishing, Cham, 2017, pp. 346–
360.
[2] D. E. Losada, F. Crestani, J. Parapar, eRisk 2017: CLEF Lab on Early Risk Prediction on theInternet: Experimental foundations, in: CEUR Proceedings of the Conference and Labs ofthe Evaluation Forum, CLEF 2017, Dublin, Ireland, 2017.
[3] D. E. Losada, F. Crestani, J. Parapar, Overview of eRisk: Early Risk Prediction on theInternet, in: P. Bellot, C. Trabelsi, J. Mothe, F. Murtagh, J. Y. Nie, L. Soulier, E. SanJuan,
L. Cappellato, N. Ferro (Eds.), Experimental IR Meets Multilinguality, Multimodality, andInteraction, Springer International Publishing, Cham, 2018, pp. 343–361.
[4] D. E. Losada, F. Crestani, J. Parapar, Overview of eRisk 2018: Early Risk Prediction on theInternet (extended lab overview), in: CEUR Proceedings of the Conference and Labs of the
Evaluation Forum, CLEF 2018, Avignon, France, 2018.
[5] D. E. Losada, F. Crestani, J. Parapar, Overview of eRisk 2019: Early risk prediction on
the Internet, in: F. Crestani, M. Braschler, J. Savoy, A. Rauber, H. Müller, D. E. Losada,G. Heinatz Bürki, L. Cappellato, N. Ferro (Eds.), Experimental IR Meets Multilinguality,
Multimodality, and Interaction, Springer International Publishing, 2019, pp. 340–357.
[6] D. E. Losada, F. Crestani, J. Parapar, Overview of eRisk at CLEF 2019: Early risk prediction
on the Internet (extended overview), in: CEUR Proceedings of the Conference and Labs ofthe Evaluation Forum, CLEF 2019, Lugano, Switzerland, 2019.
[7] D. E. Losada, F. Crestani, J. Parapar, Early detection of risks on the internet: An exploratory
campaign, in: Advances in Information Retrieval - 41st European Conference on IR
Research, ECIR 2019, Cologne, Germany, April 14-18, 2019, Proceedings, Part II, 2019, pp.
259–266.
[8] D. E. Losada, F. Crestani, J. Parapar, Overview of erisk 2020: Early risk prediction on
the internet, in: Experimental IR Meets Multilinguality, Multimodality, and Interaction 11th International Conference of the CLEF Association, CLEF 2020, Thessaloniki, Greece,
September 22-25, 2020, Proceedings, 2020, pp. 272–287.
[9] D. E. Losada, F. Crestani, J. Parapar, Overview of erisk at CLEF 2020: Early risk prediction
on the internet (extended overview), in: Working Notes of CLEF 2020 - Conference andLabs of the Evaluation Forum, Thessaloniki, Greece, September 22-25, 2020, 2020.
[10] D. E. Losada, F. Crestani, J. Parapar, erisk 2020: Self-harm and depression challenges, in:Advances in Information Retrieval - 42nd European Conference on IR Research, ECIR 2020,
Lisbon, Portugal, April 14-17, 2020, Proceedings, Part II, 2020, pp. 557–563.
[11] J. Parapar, P. Martín-Rodilla, D. E. Losada, F. Crestani, Overview of erisk 2021: Early risk
prediction on the internet, in: Experimental IR Meets Multilinguality, Multimodality, andInteraction - 12th International Conference of the CLEF Association, CLEF 2021, Virtual
Event, September 21-24, 2021, Proceedings, 2021, pp. 324–344.
[12] J. Parapar, P. Martín-Rodilla, D. E. Losada, F. Crestani, Overview of erisk at CLEF 2021:
Early risk prediction on the internet (extended overview), in: Proceedings of the Working
Notes of CLEF 2021 - Conference and Labs of the Evaluation Forum, Bucharest, Romania,
September 21st - to - 24th, 2021, 2021, pp. 864–887.
[13] J. Parapar, P. Martín-Rodilla, D. E. Losada, F. Crestani, erisk 2021: Pathological gambling,
self-harm and depression challenges, in: Advances in Information Retrieval - 43rd European
Conference on IR Research, ECIR 2021, Virtual Event, March 28 - April 1, 2021, Proceedings,
Part II, 2021, pp. 650–656.
[14] J. Parapar, P. Martín-Rodilla, D. E. Losada, F. Crestani, Overview of erisk 2022: Early risk
prediction on the internet, in: Experimental IR Meets Multilinguality, Multimodality, andInteraction - 13th International Conference of the CLEF Association, CLEF 2022, Bologna,
Italy, September 5–8, 2022, 2022, p. 233–256.
[15] J. Parapar, P. Martín-Rodilla, D. E. Losada, F. Crestani, Overview of erisk at CLEF 2022: Early
risk prediction on the internet (extended overview), in: Proceedings of the Working Notes
of CLEF 2022 - Conference and Labs of the Evaluation Forum, Bologna, Italy, September
5–8, 2022, 2022, pp. 821–850.
[16] J. Parapar, P. Martín-Rodilla, D. E. Losada, F. Crestani, erisk 2022: Pathological gambling,
depression, and eating disorder challenges, in: Advances in Information Retrieval - 44th
European Conference on IR Research, ECIR 2022, Stavanger, Norway, April 10-14, 2022,
Proceedings, Part II, 2022, pp. 436–442.
[17] J. Parapar, P. Martín-Rodilla, D. E. Losada, F. Crestani, Overview of erisk 2023: Earlyrisk prediction on the internet, in: Experimental IR Meets Multilinguality, Multimodality,
and Interaction - 14th International Conference of the CLEF Association, CLEF 2023,
Thessaloniki, Greece, September 18–21, 2023, 2023.
[18] J. Parapar, P. Martín-Rodilla, D. E. Losada, F. Crestani, erisk 2023: Depression, pathological gambling, and eating disorder challenges, in: Advances in Information Retrieval 45th European Conference on IR Research, ECIR 2023, Dublin, Ireland, April 2–6, 2023,
Proceedings, Part III, 2023, p. 585–592.
[19] A. T. Beck, C. H. Ward, M. Mendelson, J. Mock, J. Erbaugh, An Inventory for Measuring
Depression, JAMA Psychiatry 4 (1961) 561–571.
[20] N. R, P. Bolimera, Y. Gupta, A. K. M, Exploring depression symptoms through similarity
methods in social media posts, in: Working Notes of CLEF 2023 - Conference and Labs ofthe Evaluation Forum, Thessaloniki, Greece, September 18-21, 2023.
[21] J. Martinez-Romo, L. Araujo, X. Larrayoz, M. Oronoz, A. Pérez, OBSER-MENH at eRisk
2023: Deep learning-based approaches for symptom detection in depression and early
identification of pathological gambling indicators, in: Working Notes of CLEF 2023 Conference and Labs of the Evaluation Forum, Thessaloniki, Greece, September 18-21,
2023.
[22] Y. Wang, D. Inkpen, uOttawa at eRisk 2023: Search for symptoms of depression, in:Working Notes of CLEF 2023 - Conference and Labs of the Evaluation Forum, Thessaloniki,
Greece, September 18-21, 2023.
[23] A.-M. Bucur, Utilizing ChatGPT generated data to retrieve depression symptoms fromsocial media, in: Working Notes of CLEF 2023 - Conference and Labs of the Evaluation
Forum, Thessaloniki, Greece, September 18-21, 2023.
[24] E. Bezerra, L. dos Santos, R. Nascimento, R. P. Lopes, G. Guedes, NailP at eRisk 2023:
Search for symptoms of depression, in: Working Notes of CLEF 2023 - Conference andLabs of the Evaluation Forum, Thessaloniki, Greece, September 18-21, 2023.
[25] D. Maupomé, T. Soulas, F. Rancourt, G. Cantin-Savoie, G. Winterstein, S. Mosser, M.-J.
Meurs, Lightweight methods for early risk detection, in: Working Notes of CLEF 2023
- Conference and Labs of the Evaluation Forum, Thessaloniki, Greece, September 18-21,
2023.
[26] H. Thompson, L. Cagnina, M. Errecalde, Strategies to harness the transformers’ potential:
UNSL at eRisk 2023, in: Working Notes of CLEF 2023 - Conference and Labs of the
Evaluation Forum, Thessaloniki, Greece, September 18-21, 2023.
[27] R. Pan, J. A. G. Díaz, R. Valencia-Garcia, UMUTeam at eRisk@CLEF 2023 shared task:
Transformer models for early detection of pathological gambling, depression, and eating
disorder, in: Working Notes of CLEF 2023 - Conference and Labs of the Evaluation Forum,
Thessaloniki, Greece, September 18-21, 2023.
[28] F. A. Sakib, A. A. Choudhury, Özlem Uzuner, MASON-NLP at eRisk 2023: Deep learningbased detection of depression symptoms from social media texts, in: Working Notes
of CLEF 2023 - Conference and Labs of the Evaluation Forum, Thessaloniki, Greece,
September 18-21, 2023.
[29] M. Abbott, The epidemiology and impact of gambling disorder and other gambling-related
harm, in: WHO Forum on alcohol, drugs and addictive behaviours, Geneva, Switzerland,2017.

[30] D. E. Losada, F. Crestani, A test collection for research on depression and language use, in:Proceedings Conference and Labs of the Evaluation Forum CLEF 2016, Evora, Portugal,
2016.
[31] D. Otero, J. Parapar, Á. Barreiro, Beaver: Efficiently building test collections for novel tasks,
in: Proceedings of the First Joint Conference of the Information Retrieval Communities in
Europe (CIRCLE 2020), Samatan, Gers, France, July 6-9, 2020, 2020.
[32] D. Otero, J. Parapar, Á. Barreiro, The wisdom of the rankers: a cost-effective method
for building pooled test collections without participant systems, in: SAC ’21: The 36th
ACM/SIGAPP Symposium on Applied Computing, Virtual Event, Republic of Korea, March
22-26, 2021, 2021, pp. 672–680.
[33] M. Trotzek, S. Koitka, C. Friedrich, Utilizing neural networks and linguistic metadata
for early detection of depression indications in text sequences, IEEE Transactions on
Knowledge and Data Engineering (2018).
[34] F. Sadeque, D. Xu, S. Bethard, Measuring the latency of depression detection in social
media, in: WSDM, ACM, 2018, pp. 495–503.
[35] A. M. Marco, X. Huang, L.-F. Hurtado, F. Pla, ELiRF-UPV at eRisk 2023: Early detectionof pathological gambling using SVM, in: Working Notes of CLEF 2023 - Conference andLabs of the Evaluation Forum, Thessaloniki, Greece, September 18-21, 2023.
[36] X. Larrayoz, N. Lebeña, A. Casillas, A. Pérez, Representation exploration and deep learning
applied to the early detection of pathological gambling risks, in: Working Notes of CLEF
2023 - Conference and Labs of the Evaluation Forum, Thessaloniki, Greece, September
18-21, 2023.
[37] H. Fabregat, A. Duque, L. Araujo, J. Martinez-Romo, NLP-UNED-2 at eRisk 2023: Detecting
pathological gambling in social media through dataset relabeling and neural networks, in:Working Notes of CLEF 2023 - Conference and Labs of the Evaluation Forum, Thessaloniki,
Greece, September 18-21, 2023.
[38] A. Talha, T. Basu, A natural language processing based risk prediction framework forpathological gambling, in: Working Notes of CLEF 2023 - Conference and Labs of the
Evaluation Forum, Thessaloniki, Greece, September 18-21, 2023.
[39] A. M. Mármol-Romero, F. M. P. del Arco, A. Montejo-Ráez, Nsinai at erisk@clef 2023:
Approaching early detection of gambling with natural language processing, in: Working
Notes of CLEF 2023 - Conference and Labs of the Evaluation Forum, Thessaloniki, Greece,
September 18-21, 2023.
[40] C. G. Fairburn, Z. Cooper, M. O’Connor, Eating disorder examination Edition 17.0D (April,2014).
[41] S. Baccianella, A. Esuli, F. Sebastiani, Evaluation measures for ordinal regression, 2009, pp.
283–287. doi:10.1109/ISDA.2009.230.
[42] G. Merhbene, A. R. Puttick, M. Kurpicz-Briki, BFH-AMI at eRisk@CLEF 2023, in: Working
Notes of CLEF 2023 - Conference and Labs of the Evaluation Forum, Thessaloniki, Greece,
September 18-21, 2023.
[43] D.-N. Grigore, I. Pintilie, Transformer-based topic modeling to measure the severity ofeating disorder symptoms, in: Working Notes of CLEF 2023 - Conference and Labs of the
Evaluation Forum, Thessaloniki, Greece, September 18-21, 2023.