Overview of EXIST 2023 – Learning with
Disagreement for Sexism Identification andCharacterization (Extended Overview)
Laura Plaza1,2,* , Jorge Carrillo-de-Albornoz1,2 , Roser Morante1 , Enrique Amigó1 ,Julio Gonzalo1 , Damiano Spina2 and Paolo Rosso3
1

Universidad Nacional de Educación a Distancia (UNED), 28040 Madrid, Spain
RMIT University, 3000 Melbourne, Australia
3
Universidad Politécnica de Valencia (UPV), 46022 Valencia, Spain
2

Abstract

In recent years, the rapid increase in the dissemination of offensive and discriminatory material aimed at
women through social media platforms has emerged as a significant concern. This trend has had adverse
effects on women’s well-being and their ability to freely express themselves. The EXIST campaign has
been promoting research in online sexism detection and categorization since 2021. The third edition of
EXIST, hosted at the CLEF 2023 conference, consists of three tasks, two of which are the continuationof EXIST 2022 (sexism identification and sexism categorization), and a third and novel one is on source
intention identification. For this edition, new test and training data are provided and the "learningwith disagreement" paradigm is adopted to address disagreements in the labelling process and promote
the development of equitable systems that are able to learn from different perspectives on the sexism
phenomena. 28 teams participated in the three EXIST 2023 tasks, submitting 232 runs. This extended lab
overview describes the tasks, the dataset, the participants’ approaches, the evaluation methodology, andthe results obtained.

Keywords

sexism detection, sexism categorization, learning with disagreement, data bias, EXIST

1. Introduction
Sexism is defined by the Oxford English Dictionary as “prejudice, stereotyping, or discrimination,
typically against women, on the basis of sex”. This phenomenon remains prevalent even incontemporary, developed societies, and among the younger generations who have grown up
in democratic societies. Sexism continues to pose significant challenges for women in various
areas of their lives, such as work, family life, and personal growth, acting as a barrier that
impedes their progress.
Sexism manifests in many different forms and can be categorized according to different
dimensions. Regarding the facet of the women that is attacked, we can find attitudes such as
sexual objectification, stereotyping or patriarchy. Depending on the level of society at which
CLEF 2023: Conference and Labs of the Evaluation Forum, September 18–21, 2023, Thessaloniki, Greece
Corresponding author.
†
These authors contributed equally.
*© 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR

Workshop
Proceedings

http://ceur-ws.org
ISSN 1613-0073

CEUR Workshop Proceedings (CEUR-WS.org)

discrimination occurs, we can find institutional sexism, interpersonal sexism and individual
sexism. Depending on the expression and underlying motivation, sexism can be categorized into
two main types: hostile sexism and benevolent sexism. While hostile sexism involves openly
negative and antagonistic attitudes towards women, benevolent sexism appears positive but
patronizing, involving protective attitudes towards women while still reinforcing restrictive
gender roles. All forms of sexism, despite their differences, contribute to the perpetuation ofgender inequality and restrict the full potential of women.
The persistence of gender inequality and discrimination against women in society is now being
replicated and amplified in the online realm, as highlighted by Azmina et al. [1]. The Internet
not only perpetuates these gender differences but also normalizes sexist attitudes, as notedby Burgos [2]. Specially concerning is the spread of sexist messages through social networks.
Social networks have allowed women to rise their voices to report abuses, discrimination andsexist experiences, but the anonymity that they provide has also facilitated the transmission of
hateful behaviours against women.
This issue is particularly worrying considering that a significant portion of Internet users,
particularly teenagers and social media users, are exposed to this increasing wave of sexism.
Previous studies [3] have shown that media content influences how social realities are perceived, so that the exposure to sexist and even misogynous content may contribute to develop
sexist attitudes or to perceive them as natural or acceptable. Urgent attention is needed tocomprehensively study and engage in a social debate surrounding this issue, leading to concrete
actions, particularly in the realm of education. Moreover, social media platforms are not acting
efficiently to remove or avoid sexist and hateful content.
Up to date, most work dealing with sexism in online media is focused on detecting misogyny
or hatred towards women [4, 5, 6]. However, sexism does not always imply misogyny. Sexism
may sound “friendly” as in (1). Sexism may sound “funny”, as in sexist jokes or humour (2). Or
sexism may sound “offensive” and “hateful”, as in (3). However, even the most subtle forms
of sexism can be as pernicious as the most violent ones and affect women in many facets of
their lives, including domestic and parenting roles, career opportunities, sexual image and life
expectations, to name a few.
(1)

Women must be loved and respected, always treat them like a fragile glass.

(2)

You have to love women... just that... You will never understand them.

(3)

Humiliate, expose and degrade yourself as the fucking bitch you are if you want a real man
to give you attention.

EXIST 2023 (http://nlp.uned.es/exist2023/) at CLEF is the third edition of the EXIST (sEXism
Identification in Social neTworks) challenge that aims at combating sexism on social media.
In 2021 and 2022, the EXIST shared tasks were proposed at the IberLEF forum [7, 8]. These
editions were the first in proposing tasks focusing on identifying and classifying online sexism
in a broad sense, from explicit and/or hostile to other subtle or even benevolent expressions thatinvolve implicit sexist behaviours. In the 2021 and 2022 EXIST editions, more than 50 teams
participated from research institutions and companies from all around the world. While thetwo previous editions only focused on classifying sexist messages according to the facet of thewomen that was being undermined, the 2023 edition tackles an additional task consisting in
determining the intention of the author of a sexist message. Since social networks are usuallyused to report and criticize sexist situations, it is important to distinguish the messages that
aim to undermine women from those that report sexist experiences with the aim of raisingawareness against sexism.
Additionally, the main novelty of the 2023 EXIST edition, and what makes it different toother recent initiatives such as the SemEval-2023 Shared Task 10: “Explainable Detection of
Online Sexism” [9], is the adoption of the “learning with disagreement” (LwD) paradigm [10]for the development of the dataset and for the evaluation of the systems. Our previous work
showed that the perception of sexism is strongly dependent on the demographic and cultural
background of the subject, so that when identifying sexist attitudes and expressions, and evenwhen classifying them in different sexist categories, people sometimes disagree. In the LwD
paradigm, instead of relying on a single “correct” label for each example, the model is trained tohandle and learn from conflicting or diverse annotations. In this way, that different annotators’
perspectives, biases, or interpretations may be taken taken into account by the systems and thelearning process becomes fairer.
In addition to adopting the LwD paradigm, we have made an effort to control bias in the
annotations (see Section 3) and have developed new evaluation metrics that take into account thedisagreement. This will allow us to assess whether including the different views and sensibilitiesof the annotators contributes to the development of more accurate and equitable NLP systems.
In the following sections, we provide comprehensive information about the tasks, the dataset,the evaluation methodology, the results and the different approaches of the systems thatparticipated in the EXIST 2023 Lab. The competition features three distinct tasks: (i) sexism
identification, (ii) source intention classification, and (iii) sexism categorization. A total of
103 teams from 29 different countries registered to participate. Ultimately, we received 232
results from 28 teams, and 24 of them completed the process by submitting the working notes.
Interestingly, a significant number of teams leveraged the diverse labels representing various
demographic groups and provided soft labels as the outputs of their systems. Their results
showcase the effectiveness and advantages of employing the LwD paradigm in our specificdomain: sexism in social networks.

2. Tasks
The two first editions of EXIST focused on detecting sexist messages in two social networks,
Twitter and Gab (https://gab.com/), as well as on categorizing these messages according to thetype of sexist behaviour they enclose. For the 2023 edition, we focus on Twitter only and weaddress an additional task, namely “source intention classification”. Therefore, three tasks areproposed which are described below.

2.1. Sexism Identification
The first task is a binary classification where systems must decide whether or not a given tweet
expresses ideas related to sexism in any of the three forms: it is sexist itself, it describes a sexistsituation in which discrimination towards women occurs, or criticizes a sexist behaviour. The
following statements show examples of sexist and not sexist messages, respectively.
(4)

Sexist: Woman driving, be careful!.

(5)

Not sexist: Just saw a woman wearing a mask outside spank her very tightly leashed dog
and I gotta say I love learning absolutely everything about a stranger in a single instant.

2.2. Source Intention
This task aims to categorize the message according to the intention of the author. We propose
a ternary classification of tweets: (i) direct sexist message, (ii) reported sexist message, and(iii) judgemental message. This distinction will allow us to differentiate sexism that is actually
taking place online from sexism which is being suffered by women in other situations but thatis being reported in social networks with the aim of complaining and fighting against sexism.
The three categories are defined as follows:• Direct sexist message. The intention was to write a message that is sexist by itself orincites to be sexist, as in:(6)

Women shouldn’t code...perhaps be influencer/creator instead...it’s their natural
strength.

• Reported sexist message. The intention is to report and share a sexist situation suffered
by a woman or women in first or third person, as in:(7)

Today, one of my year 1 class pupils could not believe he’d lost a race against a girl.

• Judgemental message. The intention was judgmental, since the tweet describes sexist
situations or behaviours with the aim of condemning them.
(8)

21st century and we are still earning 25% less than men #Idonotrenounce.

2.3. Sexism Categorization
Many facets of a woman’s life may be the focus of sexist attitudes including domestic andparenting roles, career opportunities, sexual image, and life expectations, to name a few. Automatically detecting which of these facets of women are being more frequently attacked insocial networks will facilitate the development of policies to fight against sexism. According tothis, each sexist tweet must be assigned one or more of the following categories:• Ideological and inequality. This category includes all tweets that discredit the feminist
movement in order to devalue, belittle and defame the struggle of women in any aspectof their lives. It also includes messages that reject inequality between men and women,
or present men as victims of gender-based oppression. Some examples of this type ofsexism discourse can be found in the following tweets:(9)#Feminism is a war on men, but it’s also a war on women. It’s a war on female nature,
a war on femininity.

• Role stereotyping and dominance. This category includes messages that express false
ideas about women that suggest they are more suitable or inappropriate for certain tasks.
It also includes any claim that implies that men are somehow superior to women.
(10) Most women no longer have the desire or the knowledge to develop a high quality
character, even if they wanted to.
• Objectification. Objectification and physical stereotyping includes messages where
women are presented as objects apart from their dignity and personal aspects. We also
include messages that assume or describe certain physical qualities that women musthave in order to fulfill traditional gender roles, for example, ideas that suggest that womenshould maintain a standard and ideal of beauty or attacks on a woman’s physique.
(11) No offense but I’ve never seen an attractive african american hooker. Not a single one.
• Sexual violence. This category includes messages where sexual suggestions, requests or
harassment of a sexual nature (rape or sexual assault) are made:(12) I wanna touch your tits..you can’t imagine what I can do on your body.
• Misogyny and non sexual violence: this category includes expressions of hatred andviolence towards women.
(13) Domestic abuse is never okay.... Unless your wife is a bitch.

3. Dataset: Managing Bias in Data and Providing Pre-aggregatedAnnotations
An important problem in NLP that has gained attention in the recent years in parallel to thegrowing protagonism of large language models is bias, both in the data that are used to train
and test systems, and in the way algorithms learn, mainly due to the bias in the data [11]. In
EXIST 2023 we tackle one aspect of this problem, i.e., the data bias that may be introduced bothduring the data selection and during the labeling process. Moreover, we adopt the “learningwith disagreement” paradigm and provide all labels provided by the different annotators toallow systems to learn from conflicting and subjective information.

3.1. Data SamplingTo gather the data, we follow the methodology used in previous EXIST editions [7, 8]. We firstcollected different popular expressions and terms, both in English and Spanish, commonly used
to underestimate the role of women in our society. These expressions have been extracted fromdifferent sources: (a) previous works in the area; (b) Twitter accounts (journalist, teenagers,etc.) or hashtags used to report sexist situations; (c) expressions extracted from The Every Day
Sexism Project (https://everydaysexism.com/); and d) a compendium of feminist dictionaries.
These expressions were later used as seeds to retrieve Twitter data. To mitigate the seed bias,
we have also gathered other common hashtags and expressions less frequently used in sexist

contexts to ensure a balanced distribution between sexist/not sexist expressions. This first setof seeds contains more than 400 expressions.
The set of seeds was then used to extract tweets in English and Spanish (more than 8,000,000
tweets were downloaded). The crawling was performed during the period from the September
1, 2021 till September 30, 2022. 100 tweets were downloaded for each seed per day (no retweets
and promotional tweets were included). To ensure an appropriate balance between seeds, weremoved those with less than 60 tweets. The final set of seeds contains 183 seeds for Spanish
and 163 seeds for English.
To mitigate the terminology and temporal bias, the final sets of tweets were selected asfollows: for each seed, approximately 20 tweets were randomly selected within the period from
1st September 1, February 28, 2022 for the training set, taking into account a representative
temporal distribution between tweets of the same seed. Similarly, 3 tweets per seed wereselected for the development set within the period from 1st to 31st May of 2022, and 6 tweets
per seed within the period from August 1, 2022 to September, 30 2022 were selected for the test
set. Only one tweet per author was included in the final selection to avoid author bias. Finally,
tweets containing less than 5 words were removed. As a result, we have more than 3,200 tweets
per language for the Training set, around 500 per language for the Development set, and nearly
1,000 tweets per language for the Test set.

3.2. Labeling Process
Before starting the annotation process we considered possible sources of label bias [12]. Labelbias may be introduced by socio-demographic differences of the persons that participate in the
annotation process.
The labeling of the data was carried out by crowd-workers. No personally identifiable
information about the crowd-workers was collected. Workers were informed that the tweets
could contain offensive information and were allowed to withdraw voluntarily at any time. Fullconsent was obtained. The workers were selected according to their different demographic
characteristics in order to minimize the label bias. We consider gender (male/female) andage (18–22 y.o./23–45 y.o./+46 y.o). Each tweet was annotated by 6 crowdsourcing annotators
selected through Prolific (https://www.prolific.co/). The Prolific crowdsourcing platform was
specifically selected because of the features it provides to define participant criteria in the
recruiting process – in our case, gender, age, and fluency in the different languages.
Different quality control mechanisms were employed, including small/medium size batches
to ensure that the data is labeled by a significant/diverse amount of annotators, control of thetime employed to perform the task, outlier analysis, and the use of attention mechanisms andground truth data. We communicated frequently with the workers to solve doubts and to correct
errors was kept.
The annotators were provided with annotation guidelines that included a detailed descriptionof the different tasks along with numerous examples, both positive and negative, for all different
categories/labels. The guidelines were developed by two experts in gender issues.

3.3. Guidelines for Annotators
Firstly, the annotators were informed that no identifying information would be obtained fromthem, and they were warned that they would be reading potentially violent messages that could
be distressing. They were informed that they had the option to discontinue the task if they felt
offended.
Secondly, the annotators were informed about the task, which involved reviewing 60 tweets
individually. They were specifically instructed to read each tweet attentively in order to provide
an answer to the following question:
1. Is the tweet sexist, in any form, or does it describe situations in which such discrimination
occurs (i.e., it is sexist itself, describes a sexist situation or criticizes a sexist behaviour)?
a) YES, the tweet is sexist itself, describes a sexist situation or criticizes a sexist behavior.
Examples of tweets in this category are:(14) “It’s less of #adaywithoutwomen and more of a day without feminists, which, tobe quite honest, sounds lovely”.
(15) “I’m sorry but women cannot drive, call me sexist or whatever but it is true”.
(16) “You look like a whore in those pants" - My brother of 13 when he saw me in aleather pant”.
b) NO, the tweet does not prejudice, underestimate or discriminate women. Examplesof tweets in this category are:(17) “Where all the white women at?”.
(18) “The shocking video of a woman at the wheel who miraculously escapes anassassination attempt”(19) “Don’t my arguments convince you? Let’s try to debate. Do you use "feminazi"?
You stay alone”.
IMPORTANT NOTE: Remember that sexism may be expressed in different forms!
It may sound friendly, funny, offensive, hateful or even violent. Sexism comprises
any form of oppression or prejudice against women because of their sex. This
discrimination can be based on different beliefs such as stereotypes, the belief that
men are superior to women or irrational hatred of women (misogyny). Note thatmisogyny is a radical/hate-driven sexism.
If the answer to the previous question was YES, then the two following questions were
showed to the annotators and had to be answered:
2. What do you think the intention of the person that wrote the tweet is?
a) DIRECT: the intention is to write a message that is sexist by itself, as in:(20) “A woman needs love, to fill the fridge, if a man can give this to her in returnfor her services (housework, cooking, etc), I don’t see what else she needs“.
b) REPORTED: the intention of the author is to report or describe a sexist situation
or event suffered by a woman or women in first or third person, as in:(21) “I doze in the subway, I open my eyes feeling something weird: the hand of theman sat next to me on my leg #SquealOnYourPig”.

c) JUDGEMENTAL: the intention of the author is to be judgemental, since the tweet
describes sexist situations or behaviors with the aim to condemning them.
(22) “As usual, the woman was the one quitting her job for the family’s welfare”.
3. What type of sexism does the tweet enclose?
a) Ideological discredit, denying inequality and flipped narrative (IDEOLOGICALINEQUALITY): this category includes all tweets or gabs that discredit the feminist
movement in order to devalue, belittle and defame the struggle of women in allareas. Also included are messages that reject inequality between men and women,
or present men as victims of gender-based oppression. Examples of tweets in thiscategory are:(23) “I think the whole equality thing is getting out of hand. We are different, that’s
how were made!”.
(24) “OK. Whew, that’s good. Get a real degree and forget this poison of victimhood
known as feminism”.
b) Role stereotyping and dominance (STEREOTYPING-DOMINANCE): this category
includes messages that express false ideas about women that suggest they are moresuitable or inappropriate for certain tasks. It also includes any claim that implies
that men are somehow superior to women. Examples of tweets in this category are:(25) “I feel like every time I flirt with a girl they start to imagine all the ways they canutilize me”.
(26) “Woman driving, be careful!”.
c) Objectification and Physical stereotyping (OBJECTIFICATION): it includes messages where women are presented as objects apart from their dignity and personal
aspects. We also include messages that assume or describe certain physical qualities
that women must have to fulfill traditional gender roles, for example, ideas thatsuggest that women should maintain a standard and ideal of beauty or attacks on awoman’s physique. Examples of tweets in this category are:(27) “Most of the times I just want women for sex just because every time I try to
make a relationship out of it they always break my heart”.
(28) “Don’t get married than blame all woman for your poor investment. You shouldof got a hooker but instead you choose to go get a wedding ring.”.
d) Sexual violence (SEXUAL-VIOLENCE): this category includes messages where
sexual suggestions, requests for sexual favors or harassment of a sexual nature (rapeor sexual assault) are made. Examples of tweets in this category are:(29) “fuck that cunt, I would with my fist”.
(30) “Now I understand why woman that get no action are miserable. Semen retention
almost drove me insane”.
e) Misogyny and violence (MISOGYNY-NON-SEXUAL-VIOLENCE): this category
includes expressions of hatred and violence towards women. Examples of tweets inthis category are:

(31) “Some woman are so toxic they don’t even know they are draining everyone
around them in poison. If you lack self awareness you won’t even notice how toxic
you really are”.
(32) “Regardless, if you grab a woman by the throat, you’re a massive cunt”.
IMPORTANT NOTE: Sexism may affect women in many facets of their lives, including domestic and parenting roles, career opportunities, sexual image and life
expectations, to name a few.

3.4. Pre-aggregated Annotations: The Learning with Disagreement Paradigm
As stated in [10], the assumption that natural language expressions have a single and clearly
identifiable interpretation in a given context is a convenient idealization, but far from reality.
To deal with this, Uma et al. [10] have proposed the Learning With Disagreement paradigm(LwD), which consists mainly in letting systems learn from datasets with information about
the annotations from all annotators, in an attempt to gather the diversity of views. In thecase of sexism identification, this is particularly relevant, since the perception of a situation as
sexist or not can be subjective and may depend on the gender, age and cultural background of
the person who is judging it. Following methods proposed for training directly from the datawith disagreements, instead of using an aggregated label [13, 14, 15], the EXIST 2023 dataset
provides multiple annotations per example. The LwD paradigm may also help to mitigate bias
and produce equitable NLP systems. The selection of annotators for the development of the
EXIST 2023 dataset took into account the heterogeneity necessary to avoid gender and age
biases.

3.5. EXIST 2023 Dataset statistics
This section provides a detailed description of the EXIST 2023 dataset, which serves as avaluable resource for analyzing and understanding sexism detection in social media. The dataset
encompasses a wide range of tweets from various countries and includes annotations provided
by a diverse group of annotators. The distribution of data within the EXIST 2023 dataset ispresented in Table 1, highlighting the number of tweets, annotators, and countries represented
in each subset (Training, Development, and Test).
Table 1
Number of tweets, annotators and countries of annotators per partition in the EXIST 2023 dataset
Training
Development
Test

Annotators
725
113
227

Countries
45
23
28

Tweets
6,920
1,038
2,076

As shown in Table 1, the EXIST 2023 dataset was created by large and diverse group of 1,065
annotators, who contributed to the annotation process. This diverse pool of annotators ensuresa wide range of perspectives and enhances the representativeness of the dataset. The dataset

includes annotators from 45 countries in the Training subset, 23 countries in the Development
subset, and 28 countries in the Test subset. This broad representation across countries allows forcross-cultural analysis and provides a comprehensive view of the global understanding sexism
on social media.
In total, the EXIST 2023 dataset consists of 10,034 annotated tweets. The Training subset
contains the majority of the tweets with 6,920 samples, 3660 in Spanish and 3260 in English.
The Dev and Test subsets contain 1,038 and 2,076 tweets, respectively, distributed in 549 in
Spanish and 489 in English for the Development set and 1098 in Spanish and 978 in English for
the Test set.
The final distribution of labels for Task 1 is shown in Figure 1. The label “YES” indicatesthat the annotator considered the tweet to be sexist, while the label “NO” indicates that the
annotator determined that the tweet was non-sexist. The charts in Figure 1 show the percentage
of “YES” and “NO” labels in the three subsets. It can be seen that, in the three subsets, thedistribution among sexist and not sexist tweet is well-balanced, being the “NO” label slightly
more frequent than the “YES” one.

Figure 1: Distribution of labels by subset for Task 1 in the EXIST 2023 dataset

Similarly, the distribution of labels for Task 2 is shown in Figure 2. The DIRECT label indicatesthat the annotator considered the tweet to be inherently sexist in their content. These tweets
displayed explicit or overt sexist language or sentiments. On the other hand, the REPORTED
label indicates that the annotator considered that the tweet was describing specific instances ofsexist events or behaviors experienced by women. These tweets shed light on real-life incidents
and their impact. Lastly, the JUDGEMENTAL label means that the annotator determined thatthe tweet made judgments about sexist acts or situations. These tweets expressed opinions or
evaluations of sexist occurrences.
The charts in Figure 2 show the percentage of “DIRECT” , “REPORTED” and “JUDGEMENTAL”
labels in the three subsets. The distribution is homogeneous across the different datasets, being
the “DIRECT” label the most frequent one (near 50%), followed by the “JUDGEMENTAL” label(around 28%), and finally the “REPORTED” label (around 25%).
Finally, the distribution of labels for Task 3 is shown in Figure 3. Again, the distribution oflabels is quite homogenous across the three subsets, being the “STEREOTYPING-DOMINANCE”
the most frequent label, followed by the “IDEOLOGICAL-INEQUALITY”, “OBJECTIFICATION”,

Figure 2: Distribution of labels by subset for Task 2 in the EXIST 2023 dataset“MISOGYNY-NON-SEXUAL-VIOLENCE”, and finally, “SEXUAL-VIOLENCE”.

Figure 3: Distribution of labels by subset for Task 3 in the EXIST 2023 dataset

In order to provide a comprehensive analysis of the dataset, we present the statistics for the
hard version of the dataset, i.e., that obtained by assigning to each tweet the majority vote among
all annotators’ labels following the methodology described in Section 6. However, as shownin Table 2, this approach results in a significant loss of information in cases where annotators
do not reach a consensus. It is important to note that disagreements among annotators were
expected because, to some extent, they are inherent to subjective tasks such as sexism detection.

4. Lab Setup and Participation
The EXIST 2023 Lab was conducted and overseen via the website http://nlp.uned.es/exist2023/.
Participants had access to task details, dataset information, as well as the registration andparticipation process on this website.
Although 103 teams from 29 different countries registered for participation, the number ofparticipants who finally submitted results were 28, submitting 232 runs. Teams were allowed toTable 2
EXIST 2023 dataset hard labels statistics
ES
Task 1NO
YES
Task 2
DIRECTREPORTEDJUDGEMENTAL
Task 3OBJECTIFICATIONSEXUAL-VIOLENCESTEREOTYPINGDOMINANCEIDEOLOGICALINEQUALITYMISOGYNYNON-SEXUAL-VIOLENCE

Training
EN
Total

Development
ES EN Total

ES

Test
EN Total

1634
1560

1733
1137

3367
2697

229
261

250
194

479
455

491
478

489
349

980
827

749
265
228

545
194
148

1294
459
376

117
40
55

87
35
28

204
75
83

244
78
78

152
45
75

396
123
153

611
359

492
316

1103
675

88
74

95
49

183
123

194
99

150
97

344
196

810

613

1423

136

105

241

257

187

444

632

481

1113

117

95

212

223

161

384

552

304

856

90

68

158

166

109

275

participate in any of the three tasks and submit hard and/or soft outputs. Table 3 summarizes
the participation in the different tasks and evaluation contexts.
Table 3
Runs submitted per task and evaluation scenario# runs# teams

Task 1
Hard Soft
67
54
28

Task 2
Hard Soft
33
25
15

Task 3
Hard Soft
30
23
14

The evaluation campaign started on February 13, 2023 with the release of the training set.
The development set was released on March 27, and the test set was made available on April
10. The participant teams were provided with the official evaluation script. Runs had to besubmitted by May 15. Each team could submit up to three runs per task, that may contain soft
and/or hard outputs.
For a comprehensive description of the systems submitted by the participants, please refer
to Section 5 and to the participant papers. Here we summarize the main approaches. Approximately 90% of the systems submitted utilized large language models, both monolingual andmultilingual. Some teams employed ensembles of multiple language models to enhance theoverall performance. Only two teams utilized deep learning architectures such as BiLSTM,
CNN, and RNN, while two others opted for traditional machine learning methods, including
perceptrons, Naive Bayes, and SVM.

Data augmentation techniques were used by several teams, involving the translation of tweets,the utilization of data from similar tasks (such as previous EXIST editions), and the duplicationof instances within the training set. Additionally, Twitter-specific models and transfer learningtechniques from domains like hate speech, toxicity, and sentiment analysis were also utilized.
Most participants made use of the soft labels and applied the LwD paradigm, rather than
opting for a traditional approach and providing only hard labels as outputs.
For each of the three tasks, the organization also provided different baseline runs:• Majority class: non-informative baseline that classifies all instances as the majority
class.
• Minority class: non-informative baseline that classifies all instances as the minority
class.
• Oracle most voted: hard approach that selects the most voted label following the sameprocedure as the one used to generate the gold hard. Note that this baseline is only
employed in the hard-soft evaluation.

5. Overview of Approaches
In this section we provide a brief description of the approaches adopted by the participants.
More detailed information of each work is provided in the working notes.
• Team AIT_FHSTP [16] utilized embeddings derived from a sentiment analysis and a
toxicity analysis model, which were based on XLM_RoBERTa. These embeddings werethen used to train a random forest model. Additionally, they explored the application of
Principal Component Analysis. They obtained both soft and hard predictions for Tasks 1
and 2, while for Task 3, they only obtained a hard multi-label prediction.
• The AI-UPV team [17] utilized BERT-base-multilingual uncased and XLM-RoBERTa-base.
They obtained both soft and hard predictions for all three tasks by employing ensembles
to combine the probabilities from both models.
• The Alex_P_UPB team [18] utilized the MiniLMv2 model [19], a multilingual model basedon deep self-attention distillation. The selection of the best parameters was performed
manually, as the fine-tuning process involved using all the training data without any
splits. They obtained both soft and hard predictions for all three tasks.
• Team Awakened employed BiLSTM models along with sentence embeddings built using"pysentimiento/bertweet-hate-speech" for English and "cardiffnlp/twitter-xlm-robertabase-hate-spanish" for Spanish. They obtained both soft and hard predictions for all threetasks.
• Team CIC-SDS.KN [20] participated only in the first task. They fine-tuned a multilingual
language model called Bernice specifically for Twitter data. They used the proportionof annotators with a label of YES as the label for training. To obtain hard labels, they
applied a rule where the model’s prediction had to be greater than 0.5 to predict YES, andNO otherwise.

• Team CLassifiers [21] participated in Tasks 1 and 3 using a multilingual approach. They
employed XLMRoBERTa for hard classification in Task 3 and the CardiffNLP Twitter XLMRoBERTa for the hard approach in Task 1. For soft labels, they utilized the CardiffNLP
Twitter XLM-RoBERTa model. In Task 1, they utilized data augmentation with EXIST
2021. For Task 3, they employed a cascade model using the model from Task 1.
• Team CNLP-NITS-PP [22] utilized the GPT-2 model trained exclusively on the provided
training dataset. They generated both soft and hard labels for Task 1, while only hardlabels were provided for Tasks 2 and 3.
• Team DRIM [23] performed fine-tuning on three BERT models separately for Task
1, Task 2, and Task 3. They then froze the models and concatenated them together,incorporating manual features such as tweet length and the number of hashtags. On topof the concatenated representation, they added three dense layers. For generating labels,
they considered the possible distributions, which were finite and had values in multiples
of 1/6 (reflecting the number of annotators). They selected the output that was closest to
the possible distributions, providing both hard and soft labels for all three tasks.
• Team I2C-Huelva-1 [24] explored various approaches for Task 1. They utilized an
ensemble of models including bert-large-uncased, bert-base-uncased, and distilbert. Additionally, they employed a fine-tuned model based on distilbert_sexism from the Hugging
Face library, as well as a fine-tuned model based on bert-large-uncased.
• Team IIIT-Surat [25] exclusively participated in Task 1, utilizing a standard Bi-LSTM
architecture to provide hard predictions.
• Team InsightX [26] solely participated in Task 1, delivering both hard and soft predictions. They employed RoBERTa and BERT models for English and Spanish, along withmonolingual transformers such as BERTTweet for English and distill-BERT for Spanish.
• Team iimasGIL_NLP [27] participated in Tasks 1 and 3 providing both hard and softlabels. For the binary classification task (Task 1), they evaluated many linguistic patterns
combined with bag-of-words and n-gram features as input in classical machine learningalgorithms. For the intention and category of sexist messages categorization (Task 3),
they fine-tuned transformers models in the English and Spanish datasets.
• Team IUExist [28] participated in Task 1, providing both hard and soft predictions. They
fine-tuned and optimized various models from huggingface, including XLM-Roberta
Base, XLM-Roberta Large, TWHIN Base, and TWHIN Large. Additionally, they utilized
extended data from previous EXIST datasets. They trained an XGB Classifier using thepredictions generated by the four mentioned models and evaluated its performance on
the development dataset. They employed multilingual models and data augmentation
techniques with EXIST 2021.
• Team IU-NLP-JeDi [29] participated in Task 1, providing both hard and soft predictions.
They explored various approaches, including SVM with TF-IDF on bigrams and trigrams,
as well as CNN and RNN models.
• Team IU-Percival [30] participated in Task 1, providing both hard and soft predictions.
They employed a traditional machine learning approach using single-layer perceptrons
and utilized 1-, 2-, and 3-grams as features.
• Team JPM-UNED [31] participated in Task 1 and Task 2, providing both hard and softpredictions. They utilized data augmentation techniques by translating tweets from one

language to another and trained different models for each cohort based on the genderand age of the annotators. The best-performing models employed were "PlanTL-GOBES/roberta-base-bne" for Spanish and "distilbert-base-uncased" for English.
• Team KUCST participated in all three tasks, providing only hard labels. They trained a
gradient boosting classifier for Task 1 and Task 2, and employed a separate classifier foreach label in Task 3. Their feature set included Bag of Words, TF-IDF weighted words,
POS tags, n-grams, morphological features, and BERT representations.
• Team M&S NLP [32] participated in Task 1 and Task 2, providing both hard and softlabels. In Task 3, they provided only soft labels. To address imbalanced classes, they
employed text augmentation techniques. Their approach involved the use of multiple
transformers, including BERT, XLM-RoBERTa, and DistilBERT.
• Team Mario [33] participated in all three tasks, providing both hard and soft labels. They
utilized the "open-source GPT-NeoX" model for English and the "BERTIN-GPT-J-6B"
model for Spanish. Their approach involved organizing the LLMs in cascades. One
model was fine-tuned with in-domain training data for all three tasks, while the otherhate-speech boosted model was sequentially fine-tuned on various hate speech datasets,
including an open-sourced hate-speech tweets dataset from the Hugging Face library in
the respective target language (English or Spanish). Finally, both models were fine-tuned
with task-specific training data.
• Team MART participated solely in Task 1, providing hard labels. They employed various
pre-processing techniques, such as TF-IDF vectorization, emoji/abbreviation replacement,
accent removal, tokenization, stop words removal, and lemmatization. In addition, theyutilized a standard BERT model for their approach.
• Team roh-neil [34] participated in the three tasks, providing both hard and soft labels.
They utilized the "xlm-roberta-large-twitter" model from HuggingFace and employed
Optuna for parameter optimization. Additionally, they incorporated data augmentation
techniques by leveraging data from previous editions of EXIST.
• Team shm2023 participated exclusively in Task 1, providing hard labels. They employedthe "bert-lapse" embedded model to embed the tweets, and then utilized a random forest
algorithm to predict the results based on the embedded text.
• Team SINAI [35] participated in the three tasks, providing hard and soft labels. They
utilized the mDeBERTa base model and employed data augmentation techniques by
replicating instances. Additionally, they incorporated categorical information of theannotators, such as gender and age, during the training process. They experimented withdifferent strategies to incorporate this information, such as concatenating the transformer
output and the categorical features, applying a multilayer perceptron on the categoricalfeatures, or using attention-based summation of the transformer outputs and categoricalfeatures, among others.
• Team SMS [36] participated in Tasks 1 and 2, providing hard and soft labels for Task 1,
and hard labels for Task 2. They utilized 200-dimensional GloVe Twitter word embeddings
for English texts and 300-dimensional FastText word embeddings for Spanish texts,
along with an LSTM layer. For the calculation of the soft score, they aimed to replicate
human annotator performance by using different models and aggregating the probabilities

predicted by these models. The ensemble of models included Multinomial Naive Bayes,
Linear Support Vector Classifier, Multi-Layer Perceptron, XGBoost, LSTM using GloVe
embeddings, and LSTM using FastText embeddings. The probabilities predicted by thesemodels were combined to derive the soft score.
• Team Tlatlamiztli [37] participated in Task 1, providing both hard and soft labels.
Their approach involves preprocessing the data using regex, spaCy, spaCymoji, andwordsegment libraries. They further perform translation to Spanish using the Googletranslibrary. They then utilize a RoBERTuito model that has been fine-tuned on the EXIST
2021 dataset. This multilingual model allows for the classification of hate speech in both
English and Spanish.
• Team UMU Team [38] participated in all three tasks. They provided both hard and softlabels for Tasks 1 and 2, and only hard labels for Task 3. Their approach involved using
a combination of different Large Language Models (LLMs) with Label Functions (LFs)and training them on multilingual data. They also experimented with the use of sentence
embeddings extracted from MarIA and mDeBERTa models.
• Team UniBo [39] participated in all three tasks, providing hard labels. They employeda translation approach to convert the Spanish data into English using the Google API.
Additionally, they enriched their feature set by incorporating Emotions retrieved using
EmoBerta, which were added as features to the RobertaHate embeddings.
• Team ZaRa-IU-NLP [40] participated in Task 1 providing only hard labels. They test
two different models: XML-RoBERTa and RoBERTa+ BETO.

6. Evaluation Methodology and Metrics
As in SemEval 2021, we have carried out a "hard evaluation" and a "soft evaluation". In thehard evaluation paradigm, both the system and the gold standard consist of one or more labels
assigned to each instance in the dataset. In contrast, the soft evaluation is intended to measure
the ability of the model to capture disagreements, by considering the probability distributionof labels in the output as a soft label and comparing it with the probability distribution of theannotations.
From the point of view of evaluation metrics, the tasks can be described as follows:• Task 1 (sexism identification): binary classification, monolabel.
• Task 2 (source intention): multiclass hierarchical classification, monolabel. The hierarchyof classes has a first level with two categories, sexist/not sexist, and a second level for thesexist category with three mutually-exclusive subcategories: direct/reported/judgemental.
A suitable evaluation metric must reflect the fact that a confusion between not sexist anda sexist category is more severe than a confusion between two sexist subcategories.
• Task 3 (sexism categorization): multiclass hierarchical classification, multilabel. Again
the first level is a binary distinction between sexist/not sexist, and there is a second
level for the sexist category that includes five subcategories: ideological and inequality,
stereotyping and dominance, objectification, sexual violence, and misogyny and nonsexual violence. These classes are not mutually exclusive: a tweet may belong to several
subcategories at the same time.

The LwD paradigm can be considered in both sides of the evaluation process:• The ground truth. In a "hard" setting, the variability in the human annotations is reduced
by selecting one and only one gold category per instance, the hard label. In a "soft" setting,the gold standard label for one instance is the set of all the human annotations existingfor that instance. Therefore, the evaluation metric incorporates the proportion of humanannotators that have selected each category (soft labels). Note that in Tasks 1 and 2,
which are monolabel problems, the sum of probabilities of each class must be one. But
in Task 3, which is multilabel, each annotator may select more than one category for asingle instance. Therefore, the sum of probabilities of each class may be larger than one.
• The system output. In a "hard", traditional setting, the system predicts one or morecategories for each instance. In a "soft" setting, the system predicts a probability for eachcategory, for each instance. The evaluation score is maximized when the probabilities
predicted match the actual probabilities in a soft ground truth.
In EXIST 2023, for each of the tasks, three types of evaluation have been performed:
1. Soft-soft evaluation. For systems that provide probabilities for each category, we
provide a soft-soft evaluation that compares the probabilities assigned by the systemwith the probabilities assigned by the set of human annotators. The probabilities of
the classes for each instance are calculated according to the distribution of labels andthe number of annotators for that instance. We use a modification of the original ICM
metric (Information Contrast Measure [41]), ICM-Soft (see details below), as the official
evaluation metric in this variant and we also provide results for the normalized version of
ICM-Soft (ICM-Soft Norm). It is important to note that ICM is a measure that quantifies
information, and its upper and lower bounds are +∞ and -∞, respectively. To normalize
the results, we used the gold standard score as the upper bound and minority class baseline(see Section 4) as the lower bound. We also provide results for Cross Entropy.
2. Hard-hard evaluation. For systems that provide a hard, conventional output, we provide
a hard-hard evaluation. To derive the hard labels in the ground truth from the different
annotators’ labels, we use a probabilistic threshold computed for each task. As a result,for Task 1, the class annotated by more than 3 annotators is selected; for Task 2, the classannotated by more than 2 annotators is selected; and for Task 3 (multilabel), the classes
annotated by more than 1 annotator are selected. The instances for which there is no
majority class (i.e., no class receives more probability than the threshold) are removed
from this evaluation scheme. The official metric for this task is the original ICM, as definedby Amigó and Delgado [41]. We also report a normalized version of ICM (ICM Norm) andF1. In Task 1, we use F1 for the positive class. In Tasks 2 and 3, we use the average of F1
for all classes. Note, however, that F1 is not ideal in our experimental setting: although it
can handle multilabel situations, it does not take into account the relationships between
classes. In particular, a confusion between not sexist and any of the sexist subclasses, anda confusion between two of the sexist subclasses, are penalized equally.
3. Hard-soft evaluation. For systems that provide a hard output, we will also provide a hardsoft evaluation comparing the categories assigned by the system with the probabilities

assigned to each category in the ground truth. As in the previous case, we use ICM-Soft
as the official evaluation metric in this variant. In this evaluation, the hard outputs aretransformed into soft outputs by assigning a probability of 1.0 to the selected class and
0.0 to the other classes.
ICM is a similarity function that generalizes Pointwise Mutual Information (PMI), and can beused to evaluate system outputs in classification problems by computing their similarity to theground truth. The general definition of ICM is:
ICM(𝐴, 𝐵) = 𝛼1 𝐼𝐶(𝐴) + 𝛼2 𝐼𝐶(𝐵) − 𝛽𝐼𝐶(𝐴 ∪ 𝐵)
Where 𝐼𝐶(𝐴) is the Information Content of the instance represented by the set of features
A. ICM maps into PMI when all parameters take a value of 1. The general definition of ICM
by [41] is applied to cases where categories have a hierarchical structure and instances may
belong to more than one category. The resulting evaluation metric is proved to be analyticallysuperior to the alternatives in the state of the art. The definition of ICM in this context is:
ICM(𝑠(𝑑), 𝑔(𝑑)) = 2𝐼𝐶(𝑠(𝑑)) + 2𝐼𝐶(𝑔(𝑑)) − 3𝐼𝐶(𝑠(𝑑) ∪ 𝑔(𝑑))Where 𝐼𝐶() stands for Information Content, 𝑠(𝑑) is the set of categories assigned to document𝑑 by system 𝑠, and 𝑔(𝑑) the set of categories assigned to document 𝑑 in the gold standard.
As there is not, to the best of our knowledge, any current metric that fits hierarchical multilabel
classification problems in a learning with disagreement scenario, we have defined an extension
of ICM (ICM-soft) that accepts both soft system outputs and soft ground truth assignments.
ICM-soft works as follows: first, we define the Information Content of a single assignment of acategory 𝑐 with an agreement 𝑣 to a given instance:𝐼({⟨𝑐, 𝑣⟩}) = − log2 (𝑃 ({𝑑 ∈ 𝐷 : 𝑔𝑐 (𝑑) ≥ 𝑣})
Note that the information content of assigning a category 𝑐 with an agreement 𝑣 grows
inversely with the probability of finding an instance that receives category 𝑐 with agreement
equal or larger than 𝑣. To this end, we compute the mean and deviation of the agreement levels
for each class across instances, and applying the cumulative probability over the inferred normal
distribution. Note that, in the case of zero variance, we must consider that the probability forvalues equals or below the mean is 1 (zero IC) and the probability for values above the mean
must be smoothed.
The system output and the gold standards are sets of assignments. Therefore, in order toestimate their information content, we apply a recursive function similar to the one describedby Amigó and Delgado [41].

𝐼𝐶

(︃ 𝑛
⋃︁

)︃
{⟨𝑐𝑖 , 𝑣𝑖 ⟩}= 𝐼𝐶(⟨𝑐1 , 𝑣1 ⟩) + 𝐼𝐶

(︃ 𝑛
⋃︁

𝑖=1

− 𝐼𝐶

)︃
{⟨𝑐𝑖 , 𝑣𝑖 ⟩}

𝑖=2

(︃ 𝑛
⋃︁

)︃
{⟨lca(𝑐1 , 𝑐𝑖 ), 𝑚𝑖𝑛(𝑣1 , 𝑣𝑖 )⟩}

𝑖=2

(33)

where lca(𝑎, 𝑏) is the lowest common ancestor of categories 𝑎 and 𝑏.
Within that definition, ICM accepts multi-class, multi-label, hierarchical and soft classification problems. Hard labels or large soft values assigned to infrequent classes produce highinformation quantity. The class specificity affects both the successes and the failures. The more
the system and the gold standard differ to each other, the more their labelling conjunction isunlikely, the more the joint information quantity IC(S,G) is high, and the more the resultingscore is low.
For example, if the system output and the gold standard are equal then IC(S)=IC(G)=IC(S,G)
and the score is proportional to the gold standard information content (ICM=IC(G)). If the systemdoes not label any class (empty system output) then IC(S)=0 and IC(S,G)=IC(G). Therefore, thescore is negative and proportional to the gold standard information quantity (ICM=-IC(G)).

7. Results
In the next subsections, we report the results of the participants and the baseline systems foreach task.

7.1. Task 1 Sexism Identification
We first report and analyze the results for Task 1, which focuses on sexism identification. Thistask involves a binary classification. As discussed in Section 6, we report three sets of evaluation
results.
7.1.1. Soft-Soft Evaluation
Table 4 presents the results for this evaluation context. 54 runs were submitted. 46 runs
outperformed the non-informative majority class baseline (all instances are labeled as “NO”),
while 51 runs surpassed the non-informative minority class baseline (all instances are labeled
as “YES”).
Looking at the results in Table 4, we observe a notable variation in the performance of theruns, ranging from an ICM-Soft score of 0.903 (equivalent to a 64% ICM-Soft Norm) to -5.6659(lower than the empirically determined lower bound). However, it is worth mentioning that thebest run only reached a 64% ICM-Soft Norm. Once again, this scores are due to the fact that
ICM-Soft highly penalize errors in the minority class, which is really remarkable in a binaryclassification task. Also, this suggests that there is still room for improvement when it comes tocapturing the appropriate distribution that represents real data.
These findings highlight the complexity of modeling the distribution of disagreements in a
subjective task such as sexism identification.

Table 4: Systems’ results for Task 1 in the soft-soft evaluation
Run
Rank ICM-Soft ICM-Soft Norm Cross Entropy
Gold_soft
SINAI_3
CLassifiers_3
CLassifiers_2
CLassifiers_1
CIC-SDS.KN_2
CIC-SDS.KN_3
AI-UPV_2
CIC-SDS.KN_1
IUEXIST_1
Tlatlamiztli_1
UMUTeam_1
JPM_UNED_1
AI-UPV_3
Mario_1
Mario_2
Mario_3
IUEXIST_2
JPM_UNED_2
AIT_FHSTP_3
AIT_FHSTP_1
AI-UPV_1
DRIM_1
UMUTeam_2
SINAI_1
Alex_P_UPB_1
SMS_1
SMS_2
M&S_NLP_1
JPM_UNED_3
M&S_NLP_2
IU-NLP-JeDi_3
InsightX_2
Awakened_2
IU-NLP-JeDi_2
InsightX_1
UMUTeam_3
Awakened_3
IU-Percival_2

0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

3.11820.90300.90270.86980.81720.79600.75550.73430.72000.71150.68790.68180.67790.67720.6696
0.66290.66030.61410.59720.59550.56480.54480.54330.49690.48630.32140.31420.31420.29900.24670.18020.1244−0.0357−0.1496−0.1499−0.2351−0.3460−0.4369−0.44351
0.64210.64210.63680.62830.62480.61830.61490.6126
0.61120.60740.60640.60580.60560.60440.60330.60290.59550.59270.5924
0.58750.58430.58400.57650.57480.54820.54700.54700.54450.53610.52540.51630.49050.47210.47200.45830.44030.42570.4246

Continued on next page

0.54720.79600.97540.98230.96720.77700.76201.36070.7846
1.15371.05380.87070.80231.64001.92471.85361.9503
1.84180.88520.9392
1.14911.55430.89320.81001.57591.17091.66501.66500.84962.2342
0.97241.08780.91220.87060.90970.91450.93180.9282
3.1682

Table 4 – continued from previous page
Run

Rank

ICM-Soft

IU-Percival_1
IU-Percival_3
CNLP-NITS-PP_2
Awakened_1
InsightX_3
iimasGIL_NLP_3
iimasGIL_NLP_2
iimasGIL_NLP_1
Majority_class
M&S_NLP_3
roh-neil_2
roh-neil_1
roh-neil_3
Minority_class
I2C-Huelva-1_3
I2C-Huelva-1_1
I2C-Huelva-1_2
SINAI_2

39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56

−0.4612−0.5491−0.5775−0.5931−0.6356−1.9161−1.9438−2.1678−2.3585−2.4596−2.8848−2.8851−2.8851−3.0717−4.0609−4.1626−4.3175−5.6559

ICM-Soft Norm

Cross Entropy

0.42170.40750.40290.4004
0.39360.18670.18220.14600.1152
0.09890.03020.03010.03010−0.1598−0.1762−0.2013−0.4175

3.17773.25602.01550.96250.9402
1.86191.81512.2546
4.6115
3.16701.54727.30916.76085.35722.57762.44392.90257.3080We next analyze the performance of the top ten systems. As shown in Table 4, the top fivesystems achieve similar results in terms of ICM-Soft, with a difference of less than 2% percentage
points in terms of ICM-Soft Norm. The difference among the top ten systems was also less than
4% percentage points.
Among the top ten systems, the top nine utilized multilingual learning models, while thetenth system used a monolingual Spanish model. The variations between the systems primarily
stemmed from the use of data augmentation techniques, such as in the fourth and tenth ranked
systems, as well as the incorporation of domain-specific Twitter models and ensembles.
7.1.2. Hard-Hard Evaluation
Table 5 presents the results for the hard-hard evaluation. In this scenario, the annotations fromthe six annotators are combined into a single label using the majority vote, resulting in theloss of information about the different perspectives provided by each annotator. Out of the
67 systems submitted for this task, 65 ranked above the majority class baseline (all instances
labeled as “NO”). All systems surpassed the minority class baseline (all instances labeled as“YES”).
Similar to the soft-soft evaluation, the results vary considerably for the ICM-Hard metric,
ranging from 0.6575 (78.50% ICM-Hard Norm) to -0.5335 (2.59%). However, the impact of thisvariation is not as prominent when considering the F1 score, indicating that ICM-Hard penalizessystems that predominantly suggest only one class for all instances more severely. For example,
systems like “shm2023_1” labeled 2063 out of 2076 instances as “NO” , while “M&S_NLP_3”
labeled 1594 out of 2076 instances as “YES”, thus getting very poor results in the ICM metrics.
Furthermore, the comparison between the ICM-Hard and F1 scores, as reflected in the ranking,shows a similar distribution, particularly at the top of the table. A strong correlation between
the two metrics has been observed. However, in contrast to the soft-soft evaluation, the behaviorof the best systems in the hard-hard evaluation aligns more closely with the gold standard. Thishighlights the higher complexity of the soft scenario and the inherent differences between thesoft and hard evaluation contexts

Table 5: Systems’ results for Task 1 in the hard-hard evaluation
Run

Rank

ICM-Hard

ICM-Hard Norm

F1

Gold_hard
Mario_3
Mario_1
Mario_2
roh-neil_1
roh-neil_2
CIC-SDS.KN_2
CIC-SDS.KN_3
SINAI_1
SINAI_2
roh-neil_3
SINAI_3
CLassifiers_2
UniBo_1
UniBo_2
IUEXIST_2
IUEXIST_1
CIC-SDS.KN_1
CLassifiers_3
JPM_UNED_3
AI-UPV_3
CLassifiers_1
AI-UPV_2
AIT_FHSTP_2
UMUTeam_2
I2C-Huelva-1_1
I2C-Huelva-1_2
I2C-Huelva-1_3

0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

0.99480.65750.65400.61200.57950.57950.57150.56470.55840.55430.54740.54400.53900.5381
0.53810.5341
0.53130.53030.5282
0.52230.51190.51130.5106
0.5086
0.50830.50750.50750.50751
0.78500.78280.75600.73530.73530.73020.72590.72190.71920.71490.71270.70950.70890.70890.70640.70460.70400.7026
0.69890.69220.69180.69140.69010.68990.68940.68940.68941
0.81090.80580.80290.78400.78400.77750.77210.78040.77190.77540.77150.77020.7716
0.7716
0.77170.77340.76770.76420.76230.75740.76150.75890.75710.76040.7611
0.7611
0.7611

Continued on next page

Table 5 – continued from previous page
Run

Rank

ICM-Hard

JPM_UNED_1
UMUTeam_1
iimasGIL_NLP_3
Tlatlamiztli_1
MART_1
JPM_UNED_2
AIT_FHSTP_1
AIT_FHSTP_3
iimasGIL_NLP_1
AI-UPV_1
MART_2
iimasGIL_NLP_2
Alex_P_UPB_1
M&S_NLP_1
ZaRa-IU-NLP_2
Awakened_2
M&S_NLP_2
IU-Percival_1
IU-Percival_2
ZaRa-IU-NLP_1
IU-NLP-JeDi_3
InsightX_2
IU-NLP-JeDi_1
IU-Percival_3
SMS_3
InsightX_1
SMS_1
SMS_2
UMUTeam_3
IU-NLP-JeDi_2
Awakened_3
CNLP-NITS-PP_1
CNLP-NITS-PP_2
IIIT SURAT_1
Awakened_1
InsightX_3
shm2023_2
KUCST_2
Majority_class

28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66

0.50570.50530.50240.50130.4937
0.48630.48500.48320.47510.47000.4672
0.4626
0.40210.39750.39140.36230.30570.3024
0.29640.28420.27530.26890.2676
0.26750.24690.24580.23690.23690.18820.18510.14570.10930.10930.10420.0723−0.0403−0.1470−0.3578−0.4413

ICM-Hard Norm

F1

0.68830.68800.68620.68550.6806
0.67590.67510.67390.66880.66550.66370.66080.62220.61930.6154
0.59690.56080.55870.55490.5471
0.54140.53730.53650.53650.52330.52260.51700.51700.48590.48390.45880.43560.43560.43240.41200.34030.27230.13790.08470.75600.76110.75680.75350.75870.75330.75500.75440.74840.74450.74900.74770.73020.72020.73050.72220.68200.6971
0.69810.69550.69090.68830.68720.69070.67170.68090.67870.6787
0.66390.64850.64000.64090.64090.63550.63220.4804
0.46380.40620

Continued on next page

Table 5 – continued from previous page
Run

Rank

ICM-Hard

shm2023_1
M&S_NLP_3
Minority_class

67
68
69

−0.4473−0.5335−0.5742

ICM-Hard Norm

F1

0.08090.025900.0071
0.53120.5698As shown in Table 5, the performance of the top ten runs is more remarkable in the hardcontext evaluation, with a difference of 7% percentage points in terms of ICM-Hard Norm.
This difference is primarily due to the good performance of the top three systems (“Mario”team) which outperform the 4th ranked system by 5%. The “Mario” team utilizes a cascade
model consisting of two fine-tuned monolingual GPT-based models trained on EXIST 2023 dataand on data from other related hate speech tasks. Interestingly, the efficiency of the “Mario”
team’s runs in the soft context is considerably lower, as they are ranked 14th, 15th, and 16th.
Among the other top ten teams, all of them employ multilingual approaches, with some using
Twitter-specific models and none utilizing data augmentation techniques. It is also worth noting
the performance of the “CIS-SDK.KN” team, which achieves similar rankings in both evaluations(5th and 6th in the soft context, and 6th and 7th in hard context).
7.1.3. Hard-Soft Evaluation
The hard-soft evaluation context aims to assess the impact of information loss when using the
majority vote strategy. Although the results of the soft-soft and hard-soft evaluations are notstrictly comparable, they provide insights into the performance of hard systems compared tosoft systems in a real-world context.
The evaluation included 67 systems, 65 outperformed the majority class baseline (all instances
labeled as “NO”), and all surpassed the minority class baseline (all instances labeled as “YES”).
Notably, the loss of information in the hard-soft context was more significant than expected
when compared to the soft-soft context. This had a clear impact on the behaviour of the systems,
as evidenced by the first-ranked system achieving 64.21% in terms of ICM-Soft in the soft-soft
evaluation context compared to the 57.25% achieved in the hard-soft context. This phenomenonis also reflected in the performance of the “EXIST2023_oracle_most_vote” baseline, which
considers the most voted label by the annotators, and achieves a score of 1.1977 in the hard-softcontext and a score of 3.1182 in the soft-soft context.
Also interesting is the difference found between the first system, with a score of 0.4719
in ICM-Soft, and the oracle most voted baseline, where a large difference can be seen. Thisdifference is, in general, very pronounced in all the results obtained by all participants, speciallyin those at the end of the table, in this evaluation context. This fact, again, suggests that theerrors made by the systems in the minority class are strongly penalised by the ICM-Soft metric,
and that these errors are especially identifiable in the hard-soft context.

Table 6: Systems’ results for Task 1 in the hard-soft evaluation
Run
Rank ICM-Soft ICM-Soft Norm
EXIST2023_test_gold_soft
03.1182
1
EXIST2023_oracle_most_voted
1
1.19770.6897
Mario_3
2
0.47190.5725
Mario_1
30.45070.5691
Mario_2
4
0.36340.555
roh-neil_1
50.31110.5465roh-neil_2
60.31110.5465
CIC-SDS.KN_2
7
0.27840.5412SINAI_2
80.26780.5395SINAI_1
9
0.264
0.5389
CIC-SDS.KN_3
100.2606
0.5383
SINAI_3
110.25130.5368roh-neil_3
120.24970.5366
UniBo_1
13
0.2292
0.5333
UniBo_2
140.2292
0.5333IUEXIST_1
15
0.215
0.531
IUEXIST_2
16
0.21090.5303
JPM_UNED_3
170.2041
0.5292
CIC-SDS.KN_1
18
0.197
0.5281
JPM_UNED_1
19
0.16850.5235
CLassifiers_3
200.1652
0.5229UMUTeam_2
210.16140.5223AI-UPV_2
220.15780.5217
UMUTeam_1
230.15780.5217
CLassifiers_2
240.15630.5215
Tlatlamiztli_1
25
0.15120.5207
AI-UPV_3
26
0.14530.5197
AIT_FHSTP_2
27
0.1411
0.519
I2C-Huelva-1_1
28
0.12530.5165I2C-Huelva-1_2
290.12530.5165I2C-Huelva-1_3
300.12530.5165
CLassifiers_1
31
0.11990.5156
JPM_UNED_2
32
0.10750.5136
AIT_FHSTP_1
33
0.10140.5126
iimasGIL_NLP_3
340.1
0.5124
AIT_FHSTP_3
35
0.09320.5113
MART_136
0.08680.5103
iimasGIL_NLP_1
37
0.059
0.5058AI-UPV_1
38
0.04780.504
Continued on next page

Table 6 – continued from previous page
Run
Rank ICM-Soft ICM-Soft Norm
MART_239
0.03330.5016
iimasGIL_NLP_2
40-0.0255
0.4921
Alex_P_UPB_1
41-0.0960.4807
M&S_NLP_1
42-0.14190.4733
ZaRa-IU-NLP_2
43-0.14850.4723
Awakened_2
44-0.25770.4546
M&S_NLP_2
45-0.3266
0.4435IU-Percival_2
46-0.4435
0.4246
ZaRa-IU-NLP_1
47-0.4449
0.4244IU-Percival_1
48-0.46120.4217
InsightX_2
49-0.48130.4185IU-NLP-JeDi_350-0.5071
0.4143IU-NLP-JeDi_151-0.5097
0.4139IU-Percival_3
52-0.5491
0.4075
SMS_3
53-0.5638
0.4052
InsightX_1
54-0.58870.4011
SMS_1
55-0.6017
0.399
SMS_2
56
-0.6017
0.399
IU-NLP-JeDi_2
57-0.70540.3823
UMUTeam_3
58-0.73290.3778
Awakened_3
59-0.8536
0.3583
IIIT SURAT_1
60-0.9432
0.3439CNLP-NITS-PP_1
61-0.9509
0.3426
CNLP-NITS-PP_2
62-0.9509
0.3426
Awakened_1
63-1.13820.3124
InsightX_3
64-1.2069
0.3013
shm2023_2
65-1.61310.2356
KUCST_2
66-1.84310.1985EXIST2023_test_majority_class67-2.3585
0.1152
shm2023_1
68-2.3774
0.1122
M&S_NLP_3
69-2.8050.0431EXIST2023_test_minority_class70-3.07170Regarding the analysis of the top ten systems, the first five positions are exactly the sameas in the hard-hard evaluation context, while the next four represent the same systems but
in different positions. The only system that has changed in the top ten ranking is SINAI_3,
ranked in the 11th position in the hard-hard and ranked 10th in hard-soft. In fact, this behaviour
is observed in nearly all the systems in the ranking, suggesting a high correlation between
both contexts and metrics, with small variations mainly due to the strong weight given toprobabilities in the hard-soft case (classes are assigned with probability 1.0).

7.2. Task 2 Source Intention
The second task is a hierarchical multiclass classification problem where systems must determine
whether a tweet is sexist or not, and categorize the sexist tweets according to the source intention
in: JUDGEMENTAL, REPORTED and DIRECT.
7.2.1. Soft-Soft evaluation
Table 7 presents the results for the soft-soft evaluation of Task 2. The table shows that 25
runs were submitted. Among them, 21 runs achieved better results compared to the majority
class baseline (where all instances are labeled as “NO”). Furthermore, all of the submitted runs
outperformed the minority class baseline (where all instances are labeled as “REPORTED”).
The differences between the scores achieved by the gold standard and the worst-performing
system, the non-informative minority class baseline, are higher compared to Task 1. This can be
attributed to the hierarchical and multiclass nature of Task 2, which allows for a wider range ofvalues in the quantification of information by the ICM-Soft metric. It is also worth noting thecorrelation between the ICM-Soft and Cross-Entropy measures. The results indicate a strong
correlation between the two metrics, but some differences can still be observed. Our initial
analysis suggests that ICM-Soft is better at capturing the hierarchical nature of the task, asevidenced by a preliminary analysis of runs “JPM_UNED_2” and “SINAI_3”. This observation
is further supported by the fact that “JPM_UNED_2” utilizes a cascade model to determine
whether a tweet is sexist or not sexist as a first step. However, further analysis is required toconfirm this observation.

Run

Table 7: Systems’ results for Task 2 in the soft-soft evaluation
Rank ICM-Soft ICM-Soft Norm Cross Entropy

Gold_soft
DRIM_1
AIT_FHSTP_2
JPM_UNED_2
AI-UPV_2
JPM_UNED_3
AI-UPV_3
AIT_FHSTP_3
SINAI_3
UMUTeam_3
UMUTeam_1
Alex_P_UPB_1
Awakened_2
iimasGIL_NLP_3
Mario_2

0
1
2
3
4
5
6
7
8
9
10
11
12
13
14

6.2057−1.3443−1.4350−1.6750−1.6836−1.6888−1.7691−2.1619−2.2900−2.5405−2.5674−3.1765−3.1954−3.5072−3.55091
0.80720.80490.79880.79850.7984
0.7964
0.78630.78310.77670.77600.76040.75990.75200.7509Continued on next page

0.91281.7833
1.64862.55492.16972.55612.54242.08971.67532.22711.81023.20501.66681.88603.0061

Table 7 – continued from previous page
Run

Rank

iimasGIL_NLP_1
iimasGIL_NLP_2
Awakened_1
UMUTeam_2
SINAI_1
Awakened_3
AI-UPV_1
Majority_classroh-neil_1
roh-neil_2
SINAI_2
M&S_NLP_1
Minority_class

15
16
17
18
19
20
21
22
23
24
25
26
27

ICM-Soft−3.5570−3.6387−3.9152−4.0482−4.2437−4.3598−4.3632−5.4460−5.7590−5.7592−10.9851−12.5531−32.9552

ICM-Soft Norm

Cross Entropy

0.75070.74860.74160.73820.73320.73020.73010.70250.69450.69450.56100.521001.90671.91491.77414.04522.37101.85943.01724.62333.75193.28284.62377.56398.8517

The top-performing run achieved a score of -1.3443 in terms of ICM-Soft, which is quite
similar to the scores obtained by the other top nine systems. The tenth system achieved a scoreof -2.5674, resulting in a difference of only 3.1% in terms of ICM-Soft normalized. The firstsystem (“DRIM_1”) proposed an ensemble model consisting of three monolingual models, eachtrained on one of the three EXIST tasks. These models were fine-tuned using the ICM-Soft
measure and optimized with manual features and annotator distribution information to calculate
similarities.
Regarding the other nine systems, most of them utilized multilingual models, with the
exception of “JPM_UNED”. The systems employed a combination of techniques such as data
augmentation, transfer learning, and ensembles, as in the previous task. One notable difference
among the top ten systems is the use of socio-demographic knowledge by the “JPM_UNED” team,
where six models were trained for each population’s cohort to calculate the final distribution ofprobabilities.
Finally, upon comparing the performance of the top ten systems in Task 1 and Task 2 inthe soft-soft context, we find that the “AI-UPV_2” and “SINAI_3” teams consistently excel inaccurately identifying sexism content and its properties in both tasks.
7.2.2. Hard-Hard Evaluation
Table 8 presents the results of the hard-hard evaluation for Task 2, where 33 systems wereassessed against the hard gold standard. 28 runs outperform the majority class baseline (allinstances labeled as “NO”), while all systems demonstrate superior performance comparedto the minority class baseline (all instances labeled as “REPORTED”). Similar to the soft-soft
evaluation context, the discrepancies between the gold standard and the worst-performing
system (minority class baseline) are higher in Task 2 than in Task 1. The correlation between

ICM-Hard and F1 is generally high, with slight variations observed among the top-ranked
systems, and higher variability towards the end of the table. This discrepancy can be attributed
to the inability of F1 to capture the hierarchical nature of the task, unlike ICM-Hard, which
penalizes misclassifications between different levels of the hierarchy more severely.

Table 8: Systems’ results for Task 2 in the hard-hard evaluation
Run

Rank

ICM-Hard

Gold_hard
Mario_2
roh-neil_1
roh-neil_2
UniBo_2
UniBo_1
AIT_FHSTP_1
AI-UPV_2
AI-UPV_3
JPM_UNED_2
JPM_UNED_3
JPM_UNED_1
AIT_FHSTP_3
AIT_FHSTP_2
UMUTeam_1
AI-UPV_1
Awakened_2
UMUTeam_2
SINAI_2
SMS_1
SMS_3
UMUTeam_3
Alex_P_UPB_1
SMS_2
CNLP-NITS-PP_1
SINAI_1
Awakened_1
Awakened_3
KUCST_2
Majority_class
SINAI_3
iimasGIL_NLP_3
iimasGIL_NLP_1

0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

1.53780.48870.38830.38830.27860.24390.22290.19510.18700.18620.18060.16730.16620.14750.14090.12170.0088−0.0453−0.0496−0.0892−0.1226−0.1349−0.1481−0.2571−0.3601−0.5959−0.7515−0.9048−0.9333−0.9504−0.9646−0.9925−1.0631

ICM-Hard Norm

F1

10.77640.75500.75500.73160.72430.71980.71390.71210.71200.71080.70790.70770.70370.70230.6982
0.6741
0.6626
0.66170.65330.64610.64350.64070.61750.59550.54530.51210.47940.47340.4697
0.46670.46080.44571
0.57150.54800.54800.52830.52170.50290.49620.49930.50540.50920.5032
0.49110.47590.50130.48970.46060.44950.49240.36540.35040.43000.42780.3246
0.36630.25620.29100.30870.23830.16030.25440.29100.2505

Continued on next page

Table 8 – continued from previous page
Run

Rank

ICM-Hard

iimasGIL_NLP_2
M&S_NLP_1
Minority_class

33
34
35

−1.0778−2.9687−3.1545

ICM-Hard Norm

F1

0.44260.0396
00.26290.07650.0280Regarding the comparison between the top ten runs in terms of the ICM-Hard, the first system(“Mario_2”) achieves a score of 0.4887 and the tenth system (“JPM_UNED_3”) achieves a scoreof 0.1806 (only a 6.6% point difference). Unlike the previous task and context evaluation, wheremultilingual models were predominant, this task comprises an equal presence of multilingualand monolingual approaches among the top systems. Specifically, the first-ranked system is amonolingual GPT-based model that utilizes transfer learning and a cascade approach to identify
and filter sexist tweets. The other top systems employ similar techniques, including transfer
learning, data augmentation, and Twitter domain-specific models. Once again, the “JPM_UNED”
team stands out as the only top-performing system that leverages the demographic features of
the annotators, training six models, one for each cohort. It is interesting to note the comparison
with Task 1 in the hard-hard context, where the systems from the “roh-neil” and “Mario” teams
are also among the top 5 in the ranking.
7.2.3. Hard-Soft Evaluation
In this evaluation, 35 hard approaches were evaluated against the soft gold standard. Only
2 systems outperformed the majority class baseline, while all systems showed significant
improvement over the minority class baseline. This behaviour is attributed to the fact that
ICM-Soft penalizes errors in the minority classes to a greater extent, as they provide more
informative signals than the majority classes. This observation gains further significance in thiscontext for two reasons: the increasing number of classes and the absolute probabilities assigned
to hard labels (hard labels are interpreted in this context as soft labels with probability 1.0).
Also significant is that the difference between the gold standard and participants’ systems iseven more pronounced than in Task 1. This can be attributed to the multiclass and hierarchicalnature of the task, which presents additional complexities.
Finally, when comparing the results with the soft-soft evaluation, similar trends were observed
as in Task 1. The system that performed the best in the soft-soft evaluation achieved an ICMSoft Norm score of 80.72%, while the top-performing system in the hard-soft evaluation scored71.08%.

Table 9: Systems’ results for Task 2 in the hard-soft evaluation
Run
Rank ICM-Soft ICM-Soft Norm
EXIST2023_test_gold_soft
06.2057
1
EXIST2023_oracle_most_voted
1-2.3974
0.7803
UMUTeam_2
2-5.12
0.7108UMUTeam_3
3-5.3093
0.706
EXIST2023_test_majority_class4-5.4460.7025UMUTeam_1
5-5.53690.7001
SMS_2
6-5.6270.6978
Mario_2
7-5.8157
0.693
SMS_3
8
-5.9579
0.6894
KUCST_2
9-5.9955
0.6884SMS_1
10-6.0774
0.6863
AIT_FHSTP_2
11-6.3494
0.6794roh-neil_1
12-6.522
0.675
roh-neil_2
13-6.522
0.675
AIT_FHSTP_3
14-6.7350.6696
AIT_FHSTP_1
15-6.81430.6675
UniBo_2
16-6.8552
0.6665AI-UPV_2
17-7.0678
0.6611
Awakened_2
18-7.13220.6594
UniBo_1
19-7.2258
0.657
CNLP-NITS-PP_1
20-7.2467
0.6565AI-UPV_3
21-7.2669
0.656
JPM_UNED_2
22-7.33460.6542
JPM_UNED_1
23-7.5078
0.6498
JPM_UNED_3
24-7.5205
0.6495
Awakened_1
25-7.7378
0.6439AI-UPV_1
26-7.76150.6433
Alex_P_UPB_1
27-8.8468
0.6156
iimasGIL_NLP_1
28-10.1498
0.5824
iimasGIL_NLP_2
29-10.6025
0.5708iimasGIL_NLP_3
30-10.7368
0.5674
SINAI_2
31-10.9983
0.5607SINAI_1
32-11.0357
0.5597
Awakened_3
33-11.9799
0.5356SINAI_3
34-15.0466
0.4573
M&S_NLP_1
35-23.8919
0.2314EXIST2023_test_minority_class36-32.9552
0

In the context of the hard-soft evaluation, the discussion of the top ten systems reveals

several interesting findings. Firstly, the best-performing system is based on a monolingual
model for Spanish, highlighting the effectiveness of language-specific approaches in contrast
with previous evaluations. Interestingly, when comparing the results between the first-ranked
system, excluding the majority vote baseline, and the tenth-ranked system, the differences
are minimal. The discrepancy is less than 1 point in the ICM-Soft metric and 4 percentage
points in the normalized metric, suggesting a similar performance. Moreover, the remaining
systems are a mix of monolingual and multilingual models, employing ensemble techniquesand data augmentation. Surprisingly, two teams utilizing deep learning (LSTM), and traditional
ML (gradient boosting), have achieved top 10 results. A preliminary analysis suggests thatthese types of techniques tend to prioritize the majority class, which, when correctly classified,contributes to overall higher accuracy.
These conclusions, however, should be considered preliminary, and further analysis within
this context is necessary to provide a comprehensive comparison between the hard and soft
evaluation contexts.

7.3. Task 3 Sexism Categorization
The third task is the most challenging one since is a hierarchical multiclass and multilabel
classification problem, where systems must determine if a tweet is sexist or not, and categorize
the sexist tweets according to the five categories of sexism defined in Section 2.
7.3.1. Soft-Soft Evaluation
Table 10 displays the results of the soft-soft evaluation for Task 3. A total of 23 runs were
submitted, with 14 runs surpassing the majority class baseline (all instances labeled as “NO”),
and all systems outperforming the minority class baseline (all instances labeled as “SEXUALVIOLENCE”). This highlights the complexity of categorizing sexism in social networks. The
comparison among the system scores and the gold standard further emphasizes this difficulty,
with a notable difference of more than 10 points in ICM-Soft scores: 9.4686 for the gold standard
and -2.3183 for the best system (“AI-UPV_3”). Additionally, the differences between the best andthe worst systems are significantly higher than in any of the previous tasks. However, despite
the complexity, the results of the ICM-Soft Norm metric indicate that the systems are still able
to correctly capture relevant information concerning the different types of sexism.

Table 10: Systems’ results for Task 3 in the soft-soft evaluation
Run
Rank ICM-Soft ICM-Soft Norm
Gold_soft
AI-UPV_3
AI-UPV_2
AI-UPV_1
DRIM_10
1
2
3
4

9.4686−2.3183−2.5616−3.3437−3.6842

Continued on next page1
0.78790.78350.76950.7633

Table 10 – continued from previous page
Run

Rank

Alex_P_UPB_1
CLassifiers_1
roh-neil_1
roh-neil_2
roh-neil_3
SINAI_3
iimasGIL_NLP_3
iimasGIL_NLP_2
iimasGIL_NLP_1
M&S_NLP_1
Majority_class
Mario_1
Mario_2
Mario_3
SINAI_2
CLassifiers_2
Awakened_1
Awakened_2
Awakened_3
SINAI_1
Minority_class

5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

ICM-Soft−4.2139−6.4072−6.6622−6.6622−6.6622−7.1306−7.7704−7.8073−7.8867−8.3574−8.7089−11.4241−11.4241−11.4241−13.5493−14.7828−20.0399−23.6389−25.9233−34.9362−46.1080

ICM-Soft Norm
0.75380.71430.70980.70980.70980.70130.68980.68920.68770.67930.67290.62410.62410.62410.58580.5636
0.46900.40430.36320.20100

The best system in this task utilizes an ensemble approach with two multilingual models
that have been optimized with the number of annotators to calculate probability distributions,achieving an ICM-Soft score of -2.3183. The differences between the top ten systems in this task
are higher compared to previous tasks, with a nearly 9% difference in terms of ICM-Soft Norm
between the 1st and 10th system. In terms of the techniques employed by the top ten systems,
the prevalent architecture is multilingual, often combined with data augmentation techniques,domain-specific models for Twitter, and ensembles. It is noteworthy that the “AI-UPV_2” and“SINAI_3” teams consistently perform well in all three soft-soft evaluation tasks, securing a
position in the top ten of the ranking.
7.3.2. Hard-Hard Evaluation
In the hard-hard evaluation context for the last task, a total of 30 systems were submitted. As
shown in Table 11, 26 systems outperformed the majority class baseline (all instances labeled as“NO”), while all systems achieved better results than the minority class baseline (all instances
labeled as “SEXUAL-VIOLENCE”). Similar to Task 2, the discrepancies between the gold standard
and the worst-performing system (minority class baseline) are higher in Task 3 due to its more
complex nature of being hierarchical, multiclass, and multilabel.

The variation in results among different runs follows a similar distribution to that observedin Task 2, except for the last four systems which obtained substantially lower results due tothat a high number of not sexist instances have been incorrectly assigned to different sexist
subclasses, resulting in a strong penalization by the ICM-Hard metric that considers the classhierarchy.
The correlation between the F1 and the ICM-Hard measure is not as strong as in Task 2. Thiscan be attributed to the fact that the ICM-Hard measure takes into account the hierarchy andpenalizes errors between hierarchy levels more severely.
Finally, comparing the behaviour of the different tasks in a hard-hard context, the efficiencyof the systems in this task, in terms of ICM-Hard Norm, is substantially lower than in previous
tasks, further highlighting the complexity of categorizing sexism.

Table 11: Systems’ results for Task 3 in the hard-hard evaluation
Run

Rank

ICM-Hard

EXIST2023_test_gold_hard
roh-neil_1
roh-neil_2
AIT_FHSTP_1
UniBo_2
UniBo_1
Mario_3
SINAI_2
Mario_2
Mario_1
SINAI_3
AI-UPV_1
Alex_P_UPB_1
AI-UPV_2
SINAI_1
Awakened_2
UMUTeam_3
AI-UPV_3
UMUTeam_1
iimasGIL_NLP_3
Awakened_3
iimasGIL_NLP_1
iimasGIL_NLP_2
CNLP-NITS-PP_1
Awakened_1
roh-neil_3

0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

2.15330.44330.4433
0.23660.22630.17760.17000.14720.12280.08960.0249−0.1862−0.1948−0.2516−0.3020−0.4276−0.5121−0.5788−0.5963−0.6510−0.6731−0.6859−0.7786−0.8412−0.9507−0.9626

Continued on next page

ICM-Hard Norm

F1

10.67630.67630.6372
0.63520.62600.62460.62030.61560.60940.59710.55710.55550.54480.53520.51150.49550.48280.47950.4692
0.46500.46260.44500.43320.41240.4102

1
0.62960.6296
0.58420.59090.58500.53230.58220.51450.50110.50330.4732
0.48170.47570.5306
0.40270.51300.41950.51080.44820.37940.44060.42550.31990.32830.3139

Table 11 – continued from previous page
Run

Rank

ICM-Hard

UMUTeam_2
EXIST2023_test_majority_class
KUCST_2
CLassifiers_2
CLassifiers_1
M&S_NLP_1
EXIST2023_test_minority_class

26
27
28
29
30
31
32

−0.9727−1.5984−1.7934−1.8664−1.8852−2.1587−3.1295

ICM-Hard Norm

F1

0.40830.28980.25290.23910.23550.183800.46300.10690.28890.30470.31260.00170.0288As in the soft-soft evaluation of Task 3, the differences among the top ten systems arehigher than in other tasks, up to 8% points between the 1st (“roh-neil_1”) and 10th (“SINAI_3”)
systems. In this scenario, the best system proposed a multilingual domain specific model trainedon Twitter data, and uses parameter optimization using the Optuna framework. Roughly anequal number of systems utilized monolingual and multilingual approaches. The majority ofthese systems employed a combination of techniques, including data augmentation, domainspecific models, and transfer learning. Interestingly, the only team that exploit the demographic
knowledge is the “SINAI” team. Upon analyzing the performance of the top ten hard approaches
across all tasks, it is evident that the “Mario” and “roh-neil” teams consistently achieve favorable
results, ranking within the top 10.
7.3.3. Hard-Soft Evaluation
Finally, we provide a brief summary of the main findings for the hard-soft evaluation. In thisevaluation, a total of 30 systems were evaluated, and none of them outperformed the majority
class baseline, while all of them improved upon the minority class baseline. As mentioned
in the hard-soft evaluation for Task 2, the ICM-Soft measure heavily penalizes errors in theminority classes, while errors in the majority class are not as heavily penalized. This behaviour
is particularly significant in the hard-soft evaluation due to the automatic assignment of higher
probabilities to the hard labels, and specially significant when the number of classes augmented.
Also, the multi-label nature of the task amplifies the significance of penalizing errors in theminority class even further. In the context of sexism categorization, where multiple labels can bepresent simultaneously, correctly identifying and classifying instances belonging to the minority
classes becomes increasingly challenging. Therefore, the importance of addressing the issue ofpenalization in the context of the minority classes cannot be overstated, as it directly affects theaccuracy and fairness of the system’s predictions. Finally, when comparing the results with thesoft-soft evaluation, we observe that the best performance system in the soft-soft evaluationachieved an ICM-Soft Norm score of 78.79%, while the top-performing system in the hard-soft
evaluation scored 66.52%. This difference is even larger than that found in Tasks 1 and 2, once
again highlighting the complex nature of the task.

Table 12: Systems’ results for Task 3 in the hard-soft evaluation
Run
Rank ICM-Soft ICM-Soft Norm
EXIST2023_test_gold_soft09.4686
1
EXIST2023_oracle_most_voted
1-8.38160.6788EXIST2023_test_majority_class2-8.7089
0.6729
Mario_1
3-9.1398
0.6652
M&S_NLP_1
4-9.5040.6586
Mario_2
5-9.6735
0.6556
Mario_3
6-10.2297
0.6456
AI-UPV_3
7-10.9218
0.6331
CNLP-NITS-PP_1
8-11.3206
0.6259Alex_P_UPB_1
9-11.7145
0.6188AI-UPV_1
10-11.8571
0.6163
Awakened_2
11-11.8918
0.6157SINAI_3
12-12.1399
0.6112roh-neil_1
13-12.195
0.6102roh-neil_2
14-12.195
0.6102AI-UPV_2
15-12.2553
0.6091
AIT_FHSTP_1
16-13.6923
0.5833
UniBo_2
17-14.2263
0.5737
Awakened_3
18-14.3296
0.5718iimasGIL_NLP_2
19-14.3422
0.5716
iimasGIL_NLP_3
20-14.6869
0.5654
iimasGIL_NLP_1
21-14.9609
0.5604
Awakened_1
22-15.2776
0.5547
UniBo_1
23-15.4834
0.551
KUCST_2
24-22.9796
0.4162roh-neil_3
25-24.3698
0.3911SINAI_2
26-27.2984
0.3384UMUTeam_3
27-34.5038
0.2088SINAI_1
28-34.98580.2001
UMUTeam_1
29-35.0505
0.199
CLassifiers_2
30-36.0875
0.1803
UMUTeam_2
31-37.3056
0.1584
CLassifiers_1
32-37.5046
0.1548EXIST2023_test_minority_class33-46.1080

In the hard-soft evaluation, the best-performing system is a monolingual GPT-based model.
The differences between the first-ranked system, excluding the baselines, and the 10th-ranked
system are more noticeable compared to previous tasks, reaching up more than 5 percentage
points in terms of the normalized ICM-Soft metric. The remaining systems in the top 10 primarily

Figure 4: Correspondence between F1 and the ICM metrics across tasks

utilize multilingual systems in combination with ensembles, data augmentation techniques, anddomain-specific models. Unlike what was found in Task 2, more traditional machine learning
systems have not secured positions within the top 10. This observation may be attributed tothe increased complexity of the task, stemming from the larger number of classes and theirhierarchical structure.
Furthermore, it is important to note that these results are preliminary, and a more comprehensive comparative analysis between the soft evaluation context and the hard evaluation context
should be conducted to draw more robust conclusions from the hard-soft evaluation.

7.4. The Information Contrast Model Analysis
In this campaign, ICM is applied over both hard (discrete labels) and soft outputs and gold
standards. Figure 4 shows the correlation between ICM (hard output and hard gold standard)and F1. In these charts, the dashed line represents the score for the empty system output(ICM=-I(G)). According to the graphs, in the hard-hard evaluation, all systems contribute to acertain extent regarding the empty output. Generally, there is a high correlation when ICM is
positive, although in Task 3, there are not enough samples to notice it. The reason is that ICM
penalises failures in infrequent classes (which have more information quantity) to a greater
extent than F1. Particularly in Task 1 and Task 2, an interesting point is that ICM discriminates“returning always the minority class” (red point) from the rest of outputs to a greater extent
than F1.
In Figure 5, we compare the ICM scores when considering the hard output (discrete class)versus soft outputs. The gold standard in both cases is soft. As the graphs show, both scores
differ substantially. In general, considering hard outputs (vertical axis) often leads to a decrease
ICM, especially when considering more than two classes. This means that hard outputs are
missing out on a lot of information when they restrict themselves to outputting one or a few
classes in hard mode. Once again, the dashed line in the ICM on soft outputs represents “doing
nothing” (ICM=-I(G)). Here, it can be observed that as the task becomes more complex, there
are more systems that contributes with useful information (Task 3).

Figure 5: Correspondence of ICM between hard and soft outputs. The figures show that considering
hard outputs substantially decreases the ICM scores, suggesting that relevant information is missed
when discretizing the system outputs

8. Conclusions
The objective of the EXIST challenge is to encourage research on the automated detection
and modeling of sexism in online environments, with a specific focus on social networks.
The EXIST Lab held in 2023 as part of CLEF garnered significant interest, attracting over 100
registered teams for participation. We received a total of 232 submissions. Participants adopted
a wide range of approaches including data augmentation through automatic translation, data
duplication and utilization of past EXIST editions’ data, multilingual language models, Twitterspecific language models, and transfer learning techniques from domains such as hate speech,
toxicity, and sentiment analysis. Fortunately, most participants chose to leverage the multiple
annotations available and embrace the LwD paradigm, rather than opting for a traditional
approach and providing only hard labels as outputs.
In addition to obtaining new insights into the detection and categorization of sexist messages
in social networks, the Lab has also contributed to raise awareness about the importanceof addressing annotator disagreements and label bias by selecting annotators from diverse
population groups. This is particularly true for tasks where subjectivity and moral considerations
come into play, and interpretations may vary across cultures and over time. Moreover, the
EXIST lab has provided a valuable dataset that can be utilized for future experimentation andbenchmarking purposes, further contributing to the advancement of research in this field.
For future editions of EXIST, we plan to extend our study to other languages, communication
channels and media, as well as studying sexism in particular scenarios and population groups.

Acknowledgments
This work has been financed by the European Union (NextGenerationEU funds) through the“Plan de Recuperación, Transformación y Resiliencia”, by the Ministry of Economic Affairs
and Digital Transformation and by the UNED University. It has also been financed by the
Spanish Ministry of Science and Innovation (project FairTransNLP (PID2021-124361OB-C31 and
PID2021-124361OB-C32)) funded by MCIN/AEI/10.13039/501100011033 and by ERDF, EU A way

of making Europe, and by the Australian Research Council (DE200100064 and CE200100005).

References[1] A. Dhrodia, Social media and the silencing effect: Why misogyny online is a humanrights issue, 2017. URL: https://www.newstatesman.com/culture/social-media/2017/11/
social-media-and-silencing-effect-why-misogyny-online-human-rights-issue.
[2] A. Burgos, T. Donoso-Vázquez, E. López, E. Cabañó, Y. Martínez, A. Martín, N. Prado-Soto,
M. Rubio, A. Velasco-Martínez, R. Vila, Violencias de género 2.0, E-litterae, 2014.
[3] G. Gerbner, L. Gross, M. Morgan, N. Signorielli, Living with television: The dynamics ofthe cultivation process, Perspectives on media effects 1986 (1986) 17–40.
[4] M. Anzovino, E. Fersini, P. Rosso, Automatic identification and classification of misogynistic language on twitter, in: M. Silberztein, F. Atigui, E. Kornyshova, E. Métais, F. Meziane(Eds.), Natural Language Processing and Information Systems, Springer International
Publishing, Cham, 2018, pp. 57–64.
[5] E. W. Pamungkas, V. Basile, V. Patti, Misogyny detection in twitter: a multilingual andcross-domain study, Inf. Process. Manag. 57 (2020) 102360.
[6] E. Guest, B. Vidgen, A. Mittos, N. Sastry, G. Tyson, H. Margetts, An expert annotateddataset for the detection of online misogyny, in: Proceedings of the 16th Conference of
the European Chapter of the Association for Computational Linguistics: Main Volume,
Association for Computational Linguistics, Online, 2021, pp. 1336–1350.
[7] F. Rodríguez-Sánchez, J. Carrillo-de Albornoz, L. Plaza, J. Gonzalo, P. Rosso, M. Comet,
T. Donoso, Overview of EXIST 2021: Sexism identification in social networks, Procesamiento del Lenguaje Natural 67 (2021) 195–207.
[8] F. Rodríguez-Sánchez, J. Carrillo-de Albornoz, L. Plaza, A. Mendieta-Aragón, G. MarcoRemón, M. Makeienko, M. Plaza, D. Spina, J. Gonzalo, P. Rosso, Overview of EXIST 2022:
Sexism identification in social networks, Procesamiento del Lenguaje Natural 69 (2022)
229–240.
[9] H. R. Kirk, W. Yin, B. Vidgen, P. Röttger, SemEval-2023 Task 10: Explainable Detection of
Online Sexism, in: Proceedings of the 17th International Workshop on Semantic Evaluation(SemEval), Association for Computational Linguistics, 2023.
[10] A. Uma, T. Fornaciari, A. Dumitrache, T. Miller, J. Chamberlain, B. Plank, E. Simpson,
M. Poesio, SemEval-2021 task 12: Learning with disagreements, in: Proceedings of the
15th International Workshop on Semantic Evaluation (SemEval-2021), Association for
Computational Linguistics, Online, 2021, pp. 338–347.
[11] D. Roselli, J. Matthews, N. Talagala, Managing bias in ai, in: Companion Proceedings of
The 2019 World Wide Web Conference, WWW ’19, Association for Computing Machinery,
New York, NY, USA, 2019, p. 539–544.
[12] D. Hovy, S. Prabhumoye, Five sources of bias in natural language processing, Language
and Linguistics Compass 15 (2021) e12432.
[13] V. S. Sheng, F. Provost, P. G. Ipeirotis, Get another label? improving data quality anddata mining using multiple, noisy labelers, in: Proceedings of the 14th ACM SIGKDD

International Conference on Knowledge Discovery and Data Mining, KDD ’08, Association
for Computing Machinery, New York, NY, USA, 2008, p. 614–622.
[14] F. Rodrigues, F. C. Pereira, Deep learning from crowds, in: Proceedings of the ThirtySecond AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications ofArtificial Intelligence Conference and Eighth AAAI Symposium on Educational Advancesin Artificial Intelligence, AAAI’18/IAAI’18/EAAI’18, AAAI Press, 2018.
[15] J. Peterson, R. Battleday, T. Griffiths, O. Russakovsky, Human uncertainty makes classification more robust, in: 2019 IEEE/CVF International Conference on Computer Vision(ICCV), IEEE Computer Society, Los Alamitos, CA, USA, 2019, pp. 9616–9625.
[16] J. Böck, M. Schütz, N. Q. S. Daria Liakhovets, A. Babic, D. Slijepcevic, M. Zeppelzauer,
A. Schindler, AIT_FHSTP at EXIST 2023 Benchmark: Sexism detection by transfer learning,sentiment and toxicity embeddings and hand-crafted features, in: Working Notes of CLEF
2023 – Conference and Labs of the Evaluation Forum, 2023.
[17] A. F. M. de Paula, G. Rizzi, E. Fersini, D. Spina, AI-UPV at EXIST 2023 – Sexism Characterization Using Large Language Models Under The Learning with Disagreement Regime, in:Working Notes of CLEF 2023 – Conference and Labs of the Evaluation Forum, 2023.
[18] A. Petrescu, Leveraging MiniLMv2 Pipelines for EXIST2023, in: Working Notes of CLEF
2023 – Conference and Labs of the Evaluation Forum, 2023.
[19] W. Wang, F. Wei, L. Dong, H. Bao, N. Yang, M. Zhou, Minilm: Deep self-attention distillation
for task-agnostic compression of pre-trained transformers, in: H. Larochelle, M. Ranzato,
R. Hadsell, M. Balcan, H. Lin (Eds.), Advances in Neural Information Processing Systems,
volume 33, Curran Associates, Inc., 2020, pp. 5776–5788.
[20] J. Angel, S. Aroyehun, A. Gelbukh, Multilingual Sexism Identification using contrastive
learning, in: Working Notes of CLEF 2023 – Conference and Labs of the Evaluation Forum,
2023.
[21] G. Radler, B. I. Ersoy, S. Carpentieri, CLassifiers at EXIST 2023: Detecting Sexism in
Spanish and English Tweets with XLM-T , in: Working Notes of CLEF 2023 – Conferenceand Labs of the Evaluation Forum, 2023.
[22] A. Vetagiri, P. K. Adhikary, P. Pakray, A. Das, Leveraging GPT-2 for Automated Classification of Online Sexist Content, in: Working Notes of CLEF 2023 – Conference and Labs ofthe Evaluation Forum, 2023.
[23] J. Erbani, E. Egyed-Zsigmond, D. Nurbakova, P.-E. Portier, When Multiple Perspectives andan Optimization Process Lead to Better Performance, an Automatic Sexism Identification
on Social Media With Pretrained Transformers in a Soft Label Context, in: Working Notes
of CLEF 2023 – Conference and Labs of the Evaluation Forum, 2023.
[24] V. P. Pablo Cordón, Jacinto Mata, J. L. Domínguez, I2C at CLEF-2023 EXIST task: Leveraging
Ensembling Language Models to Detect Multilingual Sexism in Social Media, in: Working
Notes of CLEF 2023 – Conference and Labs of the Evaluation Forum, 2023.
[25] A. Chaudhary, R. Kumar, Sexism Identification In Social Networks, in: Working Notes of
CLEF 2023 – Conference and Labs of the Evaluation Forum, 2023.
[26] C. Jhakal, K. Singal, M. Suri, D. Chaudhary, I. Gorton, B. Kumar, Detection of Sexism
on Social Media with Multiple Simple Transformers, in: Working Notes of CLEF 2023 –Conference and Labs of the Evaluation Forum, 2023.
[27] A. Sanchez-Urbina, H. Gómez-Adorno, G. Bel-Enguix, V. Rodríguez-Figueroa, A. Monge-

Barrera, iimasGIL_NLP@Exist2023: Unveiling Sexism on Twitter with Fine-tuned Transformers, in: Working Notes of CLEF 2023 – Conference and Labs of the Evaluation Forum,
2023.
[28] Y. Hatekar, M. Abdo, S. Khanna, S. Kübler, IUEXIST: Multilingual Pre-trained Language
Models for Sexism Detection on Twitter in EXIST2023, in: Working Notes of CLEF 2023 –Conference and Labs of the Evaluation Forum, 2023.
[29] M. Buzzell, J. Dickinson, N. Singh, S. Kübler, IU-NLP-JeDi: Investigating Sexism Detection
in English and Spanish, in: Working Notes of CLEF 2023 – Conference and Labs of the
Evaluation Forum, 2023.
[30] E. Gabel, H. Redman, D. Swanson, S. Kübler, IU-Percival: Linear Models for Sexism
Detection, in: Working Notes of CLEF 2023 – Conference and Labs of the Evaluation
Forum, 2023.
[31] J. Pedrosa-Marín, J. C. de Albornoz, L. Plaza, Combining large language models with sociodemographic information for improving Sexism Detection in Social Media, in: Working
Notes of CLEF 2023 – Conference and Labs of the Evaluation Forum, 2023.
[32] H. Mohammadi, A. Giachanou, A. Bagheri, Towards Robust Online Sexism Detection: A
Multi-Model Approach with BERT, XLM-RoBERTa, and DistilBERT for EXIST 2023 Tasks,
in: Working Notes of CLEF 2023 – Conference and Labs of the Evaluation Forum, 2023.
[33] L. Tian, N. Huang, X. Zhang, Efficient multilingual sexism detection via Large Language
Models Cascades, in: Working Notes of CLEF 2023 – Conference and Labs of the Evaluation
Forum, 2023.
[34] R. Koonireddy, N. Adel, ROH_NEIL@EXIST2023: Detecting sexism in tweets using
multilingual language models, in: Working Notes of CLEF 2023 – Conference and Labs ofthe Evaluation Forum, 2023.
[35] M. E. Vallecillo-Rodríguez, F. M. P. del Arco, L. A. Ureña-López, M. T. Martín-Valdivia,
A. Montejo-Ráez, Integrating Annotator Information in Transformer Fine-tuning for
Sexism Detection, in: Working Notes of CLEF 2023 – Conference and Labs of the Evaluation
Forum, 2023.
[36] S. Kelkar, S. Ravi, A. K. Madasamy, LSTM-Attention Architecture for Online Bilingual
Sexism Detection, in: Working Notes of CLEF 2023 – Conference and Labs of the Evaluation
Forum, 2023.
[37] H. Asnani, A. Davis, A. Rajanala, S. Kübler, Tlatlamiztli: Fine-Tuned RoBERTuito for Sexism
Detection, in: Working Notes of CLEF 2023 – Conference and Labs of the Evaluation
Forum, 2023.
[38] J. A. G. Díaz, R. Pan, R. Valencia-Garcia, UMUTeam at EXIST 2023: Sexism identificationand categorisation fine-tuning Multilingual Large Language Models, in: Working Notes of
CLEF 2023 – Conference and Labs of the Evaluation Forum, 2023.
[39] A. Muti, E. Mancini, Enriching Hate-Tuned Transformer-Based Embeddings with Emotions
for the Categorization of Sexism, in: Working Notes of CLEF 2023 – Conference and Labsof the Evaluation Forum, 2023.
[40] R. M. Regulagedda, Z. Leech, S. Kübler, Specialized or Generalized? Sexism Detection
for EXIST 2023 at CLEF, in: Working Notes of CLEF 2023 – Conference and Labs of the
Evaluation Forum, 2023.
[41] E. Amigó, A. Delgado, Evaluating extreme hierarchical multi-label classification, in:Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics,
volume Volume 1: Long Papers, ACL, Dublin, Ireland, 2022, p. 5809–5819.