Combining Transformer Based Language Models with
Socio-demographic Information for Improving Sexism
Detection in Social Media
Notebook for the EXIST Lab at CLEF 2023
Jacobo Pedrosa-Marín1,* , Jorge Carrillo-de-Albornoz1,2 and Laura Plaza1,2
1
2

NLP & IR Group, Universidad Nacional de Educación a Distancia (UNED), 28040, Spain
RMIT University, 3000, Australia

Abstract

Detecting and addressing sexism in social networks is crucial for fostering inclusive and respectful
digital spaces. The third edition of the EXIST competition emphasizes the importance of incorporating
annotator disagreement into the classification process, recognizing the inherent challenges and diversityof perspectives in identifying sexist content. In this paper, we present our participation in the EXIST
2023 campaign where we propose systems for Task 1 (Sexism Identification) and task 2 (Source Intention
Identification), both for hard and soft evaluation contexts. We adopted a primary strategy that involved
data augmentation to enhance the training dataset. By leveraging techniques such as translation and theuse of transformers, we aimed to expand the available data and capture a broader range of linguistic
patterns and expressions related to sexism. Additionally, the EXIST 2023 dataset allows to identify
and exploit annotators characteristics such as gender and age. We have used this socio-demographic
information to train different models that capture each age-gender cohort singularities, and used different
strategies to combine them in a final decision, in the hard approaches, and a probability representation,in the soft approaches. The results achieved suggest that having different models for the different cohorts
improves the efficiency of the classification.

Keywords

Sexism Detection, Sexism Identification Learning with disagreement, Transformer Models, Natural
Language Processing

1. Introduction
In recent years, the rise of social media platforms such as Twitter and Facebook has brought
about a significant transformation in communication and society. These platforms have provided
users with new means to express their ideas, thoughts, and knowledge. These platforms holdtremendous potential for information dissemination, and researchers have extensively examined
their impact in various fields, including politics and medicine. However, the proliferation ofhate speech on these platforms has emerged as a growing concern. The rise of hate speech
CLEF 2023: Conference and Labs of the Evaluation Forum, September 18–21, 2023, Thessaloniki, Greece
Corresponding author.
$ jacobo.pedrosa@lsi.uned.es (J. Pedrosa-Marín); jcalbornoz@lsi.uned.es (J. Carrillo-de-Albornoz);
lplaza@lsi.uned.es (L. Plaza) 0009-0004-1224-3760 (J. Pedrosa-Marín)*© 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR

Workshop
Proceedings

http://ceur-ws.org
ISSN 1613-0073

CEUR Workshop Proceedings (CEUR-WS.org)

poses a significant challenge, demanding careful attention and effective solutions to ensure asafe and inclusive online environment [1].
Detecting and preventing hate speech in social medial can be challenging, especially considering the overwhelming volume of data generated on these platforms every second, which makes
it necessary to employ automated methods and advanced technologies to process and classify
the content efficiently. Over the years, numerous studies and competitions have emerged,
focusing on the analysis and automation of online community management. These initiatives
aim to address various challenges associated with content detection and moderation, including
Anomaly Detection [2], Phishing Detection [3], Toxicity Detection [4] and Sexism Detection[5, 6].
In this paper, we focus on a particular form of harmful content: sexist expressions. Sexism
encompasses actions or attitudes that exhibit prejudice or discrimination towards individuals
based on their gender. It is closely linked to societal beliefs and expectations regarding the roles
that individuals should adhere to, with its repercussions primarily impacting women. Detecting
sexism in online platforms is crucial for creating inclusive and respectful digital spaces. It
enables the identification and moderation of harmful content, promotes gender equality, andhelps to prevent the perpetuation of discriminatory behaviors.
The EXIST challenges serves as an avenue for researchers and practitioners to develop andpresent their approaches and models in tackling the complex task of sexism detection in social
media [7]. The 2021 and 2022 editions of the competition were held in the IberLEF forum 1 andwere the first shared tasks on sexism detection in social networks whose aim was to identify
and classify various forms of sexism, ranging from explicit and hostile expressions to morenuanced or even benevolent behaviors involving implicit sexism. With participation from over
50 teams from research institutions and companies worldwide, the substantial interest shownby the research community underscores the significance of the problem at hand.
The third edition of the EXIST challenge at CLEF builds upon the tasks addressed in previous
years, while facing a new challenge: the identification of the author’s intention behind sexist
messages. However, the main innovation is the adoption of the “learning with disagreements”
paradigm [8] and the participation of annotators of different genders and ages. This approachaims to mitigate the potential “label bias” by incorporating diverse perspectives and sensitivities
from different population groups, ensuring a more comprehensive reflection of viewpoints andrecognizing that annotators’ socio-demographic backgrounds can shape their labeling decisions[9, 10].
This paper presents the participation of the JPM-UNED team in EXIST 2023 at CLEF. Our
approach includes the use of transformers along with different strategies (such as voting andaggregation) to leverage the collective knowledge and the disagreement among the annotators
to derive the most reliable predictions. By considering the socio-demographic variances among
annotators and employing tailored models and strategies, our objective is to enhance theaccuracy and robustness of our classification approach. This approach contributes to a moredeeper understanding of the complexities involved in sexism detection and enhances the overall
effectiveness of our model.
This paper is organized as follows: Section 2 reviews the state of the art in sexism detection
1

https://sites.google.com/view/iberlef2022/homeand learning with disagreements; Section 3 presents this edition of EXIST, including an overview
of the three tasks proposed and the dataset provided; Section 4 presents the systems developed
for participating in the competition; Section 5 discusses the results; and Section 6 summarizes
the main conclusions and discusses potential improvements for future work.

2. Related WorkIn this section, we first examine the importance of sexism detection in social networks, review
previous research in this area, and provide an overview of previous editions of the EXIST
competition. We then briefly discuss the state-of-the-art in Learning With Disagreements(LeWiDi) to consider different approaches for managing tasks of this nature.
The detection of sexism has traditionally been regarded as a distinct form of hate speech [11].
It can be approached through various methods, such as data-driven models that incorporaten-grams and additional features [12], classical machine learning models [13], and deep learningmodels that utilize LSTM and CNN architectures [14]. Additionally, certain studies have exploredthe utilization of offensive lexicons like Hurtlex [15].
However, it is important to recognize that sexism is not always expressed as hate speech. As
highlighted in [7], sexism can take on a "friendly" or even "humorous" tone, such as in the caseof benevolent sexism and sexist jokes [16]. Consequently, novel approaches are necessary todetect the various forms of sexism, ranging from hostile and explicit to subtle and seemingly
benign expressions.
In the following section, we present the EXIST 2021 and 2022 editions, which introduced thechallenging task of detecting sexism in all its nuanced manifestations.

2.1. The EXIST Challenges
In the previous two editions of the EXIST competition, two tasks were proposed. Task 1 focusedon sexism identification, aiming to detect whether a given post contains sexist content or not.
Task 2, on the other hand, focused on sexism categorization, aiming to classify the type ofsexism present in a post into one of the following five classes: (i) ideological and inequality,(ii) stereotyping and dominance, (iii) objectification, (iv) sexual violence y (v) misogyny andnon-sexual violence.
In the 2021 edition of EXIST, the majority of submissions for both tasks relied on transformerbased models for classification. Out of the 23 participating teams, 14 teams utilized BERT, awidely used transformer model, as the foundation of their solutions. Additionally, 10 teams
employed BETO, a version of BERT that is trained on Spanish text. Furthermore, 5 teams
leveraged XLM-R, a multilingual variant of RoBERTa that supports multiple languages, including
Spanish [5]. In the 2022 edition of EXIST, all participating teams utilized transformer-based
solutions. Among these solutions, 8 teams used BERT, 5 teams used BETO, and 4 teams used
RoBERTa. The widespread adoption of transformer-based models in both editions of EXIST
highlights their effectiveness in tackling the challenges associated with sexism detection insocial networks [6].

2.2. Working with Disagreements
Usually, in the fields of Artificial Intelligence (AI) and Natural Language Processing, datasets
are built with instances having a single class or interpretation referred to as the "gold standard".
However, this approach fails to capture the nuances of human behavior, which often involves
disagreement and varying perspectives. Tasks that involve subjectivity or ambiguity inherently
introduce the possibility of biased annotations influenced by the perspectives of individual
annotators. Moreover, socio-demographic factors, such as education, age or gender, have thepotential to influence the annotation process and introduce biases.
An emerging solution that is gaining popularity is to engage multiple annotators, ideally
representing diverse demographic strata and socioeconomic contexts, and retain the labels
provided by each annotator instead of relying solely on a gold standard. This approach allows
the systems to incorporate various perspectives for each instance, enabling them to learn fromdifferent points of view.
Working with a dataset that lacks a unanimous label for each instance offers valuable insights,
but it also requires ways to manage the divergent opinions among annotators. The current stateof the art in learning with disagreement can be categorized into four main categories:• Judgements aggregation: Methods that operate under the assumption that only a single"truth" exists for each instance typically aggregate all crowd annotations into a single label,commonly referred to as the "silver" label. There are multiple approaches to tackle thischallenge, with the most straightforward being the adoption of a "majority vote". However,
one of the widely employed techniques is the utilization of Probabilistic aggregation
methods, which leverage the probabilities assigned to each label by individual annotators[17, 18, 19].
• Filtering hard items: this approach utilizes the disagreement information to filter thedataset by removing instances with significant disagreement. Within this category, [20]proposed two approaches. The first approach involves directly discarding instances thatexhibit disagreement among annotators. The second approach is to train separate models
for each annotator and discard predictions that demonstrate substantial disagreement.
• Learning directly from crowds: This classification approach acknowledges the absenceof a single truth or gold standard and instead focuses on training a classifier directly fromthe crowd, utilizing probabilistic distributions or soft labels. The objective is to capture
the collective knowledge of the annotators and incorporate their diverse perspectives intoa single model. There are various strategies to approach this. For instance, [21] proposea "repeated labeling method" where replicas of each instance are created for each label,enabling multiple annotations per instance. Another interesting method in this category
is presented by [22], which involves adding a crowd layer after the output layer during
training.
• Using both hard labels and information about disagreements: These methods
utilize both gold labels and disagreement to train the models. One approach is to train
with hard labels, while incorporating crowd information as part of the loss function
during the training process. This allows the model to learn from both the ground truthlabels and the disagreements among the crowd annotations, improving its performanceand capturing the collective knowledge of the annotators [23].

As we can see, there are various approaches available to address the challenge of working
with disagreements. For further information on this topic, please refer to the survey conductedby [23]. This survey provides more in-depth insights and details on different methods andtechniques that can be employed to handle disagreement in various tasks or domains.

3. The EXIST 2023 Lab at CLEF 2023
3.1. EXIST 2023 Tasks
The EXIST 2023 edition proposes the following three tasks: (i) sexism detection, (ii) source
intention classification, and (iii) sexism categorization (see [10] for a detailed description). For
each task, participants may provide both hard (a single "gold" label) and soft (a probabilisticlabel) outputs.
• Task 1 - Sexism Detection: The first task is a binary classification task where systems
must decide whether or not a given tweet is sexist.
• Task 2 - Source Intention Classification: This task aims to categorize the message
according to the intention of the author, which provides insights in the role played bysocial networks in the emission and dissemination of sexist messages. In this task, we
propose a ternary classification task: (i) direct sexist message, (ii) reported sexist messageand (iii) judgmental message.
• Task 3 - Sexism Categorization: Each sexist tweet must be categorized in one or moreof the following categories, that reflect the facets of a woman’s life that are the focus of
the sexist message: (i) ideological and inequality, (ii) stereotyping and dominance, (iii)
objectification, (iv) sexual violence and (v) misogyny and non-sexual violence.

3.2. EXIST 2023 Dataset
The EXIST 2023 dataset comprises tweets in both English and Spanish. The training set consistsof over 3,200 tweets per language, while the development set includes 500 tweets per language.
Additionally, the test set contains 1,000 tweets per language. To ensure diverse perspectives andmitigate label bias, each tweet in the dataset has been annotated by six individuals recruited
through the Prolific service 2 . The annotators’ gender (male/female3 ) and age (18-22 years old,
23-45 years old, +46 years old) are taken into account during the labeling process. Consequently,
each tweet is labeled by annotators from a different gender and a different age groups.
For a more comprehensive understanding of the EXIST 2023 dataset, please refer to [10].

4. System description
In this section, we present our systems developed for participating in Task 1 and Task 2 of the
EXIST 2023 competition. Our approach builds upon the methods discussed in the previous
23

https://app.prolific.co
Only male and female genders were consider for availability reasonssection and utilizes Transformer pre-trained models. To establish a baseline, we first examine
the models that have been employed in previous editions of the competition. We then train
and test our models using the EXIST 2021 Dataset for each language, employing the same
configuration. The results for Spanish tweets are presented in Table 1, while Table 2 showcases
the results for English tweets.
Table 1Models review Spanish - baselines
Model

F-Measure

xlm-roberta-base
xlm-roberta-large
bert-base-multilingual-cased
distilbert-base-multilingual-cased
PlanTL-GOB-ES/roberta-base-bne
PlanTL-GOB-ES/roberta-large-bne
bertin-project/bertin-roberta-base-spanish
dccuchile/bert-base-spanish-wwm-cased
CenIA/distillbert-base-spanish-uncased

0.703
0.447
0.701
0.706
0.751
0.741
0.747
0.709
0.713

Table 2Models review English - baselines
Model
xlm-roberta-base
xlm-roberta-large
bert-base-multilingual-cased
distilbert-base-multilingual-casedroberta-large
distilbert-base-uncased
bert-base-uncased

F-Measure
0.652
0.676
0.721
0.701
0.288
0.741
0.733

After analyzing the results of our initial experiments (see Tables 1 and 2, we have selected themodel with the best performance in Spanish, which is PlanTL-GOB-ES/roberta-base-bne, andin English, which is distilbert-base-uncased. Our remaining experiments will be conductedusing these two models as the foundation. Based on these findings, we have decided to utilize
these two models as the base for our future experiments.
As a previous step, we have also created an augmented version of the EXIST 2023 dataset bytranslating tweets from English to Spanish and vice versa. With this expanded dataset in hand,
we have two parallel approaches to enhance the training process and our analysis.
Our initial approach involved fine-tuning the pre-trained models we had selected using the"repeated labeling" technique. This technique involved converting each instance in the datasets
into six duplicated instances, with each instance assigned a unique label representing a single
annotator. We will refer to this approach as "Learning from raw disagreement".
In our second approach, we trained six different models for each language, correspondingto each cohort based on age-gender combinations. We utilized the individual votes of theannotators to train these models. To determine the final label for each instance, we combined
the outputs of each cohort using various methods. Some methods were based on the labels
themselves, while others utilized the probability distributions returned by the models:• Majority vote: This approach determines the label by considering the majority voteamong the six models. In case of a tie, we decided to set the label as "NO" in the firsttask. For the second task, the order of preference for tie-breaking is JUDGMENTAL >REPORTED > DIRECT.
• Gender vote: This approach adds, for each instance, the number of "votes" that each label
receives from each gender’s models. The label that receives more votes is selected. Incase of a tie (two different labels obtain the same number of votes from the two genders’models), then the label selected by the "females" is returned as it has shown better results,
as indicated in table 3.
• Age vote: This approach involves incorporating the "votes" received by each label fromthe models of each age group (18-22, 23-45, and 46 or more) for each instance. In eachage group, we adopt the same decision rule as the previous method. In the case of a tie,
the preferred label for the first task is "NO". For the second task, the order of preference
is JUDGMENTAL > REPORTED > DIRECT.
• Probability distribution mean: This method computes the mean probability for eachlabel and the six models, and the label with the highest mean probability is returned.
• Probability distribution gender: This method computes the most probable label foreach gender by adding the scores of each gender’s model. The label with the highest
probability is selected. Since it is based on probability, it is assumed to be very difficult tohave a tie-breaking situation.
• Probability distribution Age: This method computes the most probable label foreach age range by adding the scores of each range’s model. The label with the highest
probability is selected.
• Probability distribution cohort: This method calculates the probability for each label/cohort by considering the probability outputted by the model. The label with the highest
probability is then selected and returned as the final prediction.
Finally, we employed all of these methods to train models using the augmented version of
the EXIST 2023 train set and assessed their performance on the EXIST 2023 development set.
Table 3 presents the results for Task 1.
Based on the obtained results, we have selected the outputs from the "Learning from raw
disagreements" method as the first run for Task 1. This method has exhibited promising
performance, and we will also employ its results to filter out instances labeled as NO-SEXIST,
as it has demonstrated high accuracy in classifying this label.
As the second run for Task 1, we have submitted the "Gender vote" method. This approachconsolidates the votes of the annotators from each gender to determine the final label for each
instance.
For the third run in Task 1, we employ the outputs from the "Probability distribution (age)"method. This approach considers the probability distributions assigned by each age range
cohort to determine the final label for each instance.

Table 3
TASK 1 evaluation results on the EXIST development dataset
MethodICM
hard-hard

F-measurehard-hard(YES)

F-Measurehard-hard(NO)F-measurehard-hard(macro-F)

ICM
hard-soft

ICM
soft-soft

Learning from raw disagreements
Majority vote
Age vote
Gender vote Male preference
Gender vote Female preference
Probability distribution
Probability distribution (mean)
Probability distribution (age)Probability distribution (gender)
Probability distribution (cohort)

0.58730.5841
0.573
0.58830.59420.575
0.575
0.57850.575
0.58410.80680.80590.8
0.80910.81050.80290.80290.80370.80290.80380.82720.826
0.82460.82550.82790.82330.82330.82470.8233
0.828

0.817
0.816
0.81230.81730.81920.81310.81310.81420.81310.81590.43930.45630.42140.44580.47580.43060.43060.42740.43060.44850.74470.39130.41250.7363-1.5735
0.785
0.8499A similar approach is followed for addressing Task 2. This is a multi-class classification
Task instead of a binary classification one. Initially, tweets labeled as non-sexist (from Task 1)
were eliminated using the approach that achieved the best F-Measure score for the "NO" (nonsexist) class, which involved using the Probability distribution cohort method. Subsequently,
we repeated the previous steps to predict the source intention. The results of this approachevaluated on the development dataset are displayed in Table 4.
Table 4TASK 2 evaluation results on the EXIST development dataset
MethodICM
hard-hard

FMeasure
F-Measurehard-hard
hard-hard(JUDGEMENTAL) (NO)

Learning from raw disagreement
Majority vote
Gender vote
Age vote
Probability distribution
Probability distribution (mean)
Probability distribution (age)Probability distribution (gender)

0.29360.2936
0.28110.31060.29
0.31420.279
0.30210.31780.31780.28120.30150.26670.29740.25880.3099

0.828
0.828
0.828
0.828
0.828
0.828
0.828
0.828

FMeasure
hard-hard(REPORTED)

F-measurehard-hard(DIRECT)F-measurehard-hard(macro-F)

ICM
hard-soft

ICM
soft-soft

0.33940.33940.32930.35090.36810.36590.32940.358

0.59450.59450.59510.60830.59350.59880.59220.59370.51990.51990.50840.52220.5141
0.52250.5021
0.5224-6.8743-6.8743-6.6572-6.7823-6.6209-6.5681-6.4868-6.8162-1.2093-2.2366-1.1792

We have selected the outputs from the "Majority vote" method as the first run for Task 2, the"Probability distribution (mean)" method for the second run, and the "Probability distribution(gender)" method for the third run.

5. Evaluation and results
Table 5 provides a summary of the strategies employed in each of the runs that were ultimately
submitted for both tasks.
For each of the tasks, the organization performed three types of evaluations:• Hard-hard: the hard system output is compared against the hard ground truth.
• Hard-soft: the hard system output is compared against the soft ground truth.
• Soft-soft: the soft system output is compared against the soft ground truth.
For all tasks and evaluation types (hard-hard, hard-soft, and soft-soft), the official metric used
is ICM (Information Contrast Measure) [24]. ICM is a similarity function that extends PointwiseTable 5Strategies employed in the runs submitted for Task 1 and Task 2
Task

Run

MethodTask 1
Task 1
Task 1

1
2
3

Repeated labeling
Gender vote
Probability distribution (cohorts)

Task 2
Task 2
Task 2

1
2
3

Majority vote
Probability distribution (mean)
Probability distribution (gender)

Mutual Information (PMI) and is employed to evaluate system outputs in classification problemsby measuring their similarity to the ground truth categories. An extended version of ICM,
known as ICM-soft, has been specifically developed for the task to accommodate both soft
system outputs and soft ground truth assignments. The results of our three runs for Task 1
and Task 2, evaluated using the three types of evaluation, are presented in Tables 6, 7, 8, 9, 10,
and 11. Each table provides details on various evaluation metrics for both Spanish and English
languages, as well as the combined results for both languages. In each table, the first column
within each column group indicates the ranking position of each run. The first and second rowsof values in each table represent the gold and best results achieved for the respective task ineach evaluation.

5.1. Task 1 - Sexism Identification
Regarding Task 1, as shown in Tables 6, 7, and 8, the third run utilizing the ’probability
distribution (cohorts)’ method outperformed the others in terms of hard metrics for all languages.
However, the first run, based on ’repeated labeling,’ achieved better results in the soft-soft
evaluation. Our best approach secured the 19th position out of 57 participants in the hard-hardevaluation, the 17th position in the hard-soft evaluation, and the 12th position in the soft-soft
evaluation. The higher ranking in the soft evaluations suggests that our approach effectively
captures the different perceptions of sexism among distinct population cohorts.
Table 6
Results for the hard-hard evaluation for Task 1: This table presents the results obtained in the hard-hardevaluation. The three main columns represent the results for different languages: the first column shows
results for all instances, the second column focuses on Spanish instances, and the third column focuses
on English instances. Each column includes the following metrics for each run: ICM, Normalized ICM,
and F-measure. The first row indicates the gold standard result, while the second row represents thebest result. The remaining rows display the results obtained by our three evaluation runs.
RUN
Gold
Best score
JPM_UNED_1
JPM_UNED_2
JPM_UNED_3

Rank
0
1
28
33
19

ICM-Hard
0.99480.65750.5057
0.48630.5223ALL
ICM-Hard Norm
1
0.785
0.68830.67590.6989F1
10.81090.756
0.75330.7623

Rank
1
20
24
14

ICM-Hard
0.99990.69950.514
0.5016
0.545

ES
ICM-Hard Norm
10.80110.67830.6701
0.6988F1
10.83840.77480.77840.7885

Rank
1
30
39
29

ICM-Hard
0.97980.60040.48190.45560.4844EN
ICM-Hard Norm
10.76930.69720.6812
0.6987F1
1
0.776
0.73080.72040.7284

Table 7
Results for the hard-soft evaluation for Task 1: This table presents the results obtained in the hard-soft
evaluation. The three main columns represent the results for different languages: the first column shows
results for all instances, the second column focuses on Spanish instances, and the third column focuses
on English instances. Each column includes the following metrics for each run: ICM-Soft and ICM-Soft
normalized. The first row indicates the gold standard result, while the second row represents the best
result. The remaining rows display the results obtained by our three evaluation runs.
RUN
Gold
Best score
JPM_UNED_1
JPM_UNED_2
JPM_UNED_3

Rank
0
1
19
32
17

ALL
ICM-Soft ICM-Soft Norm
3.1182
1
1.19770.68970.16850.52350.10750.51360.2041
0.5292

Rank
0
1
17
23
13

ES
ICM-Soft ICM-Soft Norm
3.11771
1.34870.6892
0.34850.51350.29270.50370.3927
0.5212

Rank
0
1
31
38
28

EN
ICM-Soft ICM-Soft Norm
3.1141
1
0.96950.6905-0.1243
0.5327-0.1879
0.5235-0.0924
0.5373

Table 8Results for the soft-soft evaluation for Task 1: This table presents the results obtained in the soft-soft
evaluation. The three main columns represent the results for different languages: the first column shows
results for all instances, the second column focuses on Spanish instances, and the third column focuses
on English instances. Each column includes the following metrics for each run: ICM-Soft, ICM-Soft
normalized and Cross entropy. The first row indicates the gold standard result, while the second row
represents the best result. The remaining rows display the results obtained by our three evaluation runs.
RUN
Gold
Best score
JPM_UNED_1
JPM_UNED_2
JPM_UNED_3

Rank
0
1
12
18
29

ICM-Soft
3.11820.903
0.67790.59720.2467ALL
ICM-Soft Norm
10.64210.60580.59270.5361

Cross Entropy
0.54720.796
0.80230.88522.2342

Rank
0
1
18
16
31

ICM-Soft
3.11770.95270.6536
0.66410.1576ES
ICM-Soft Norm
10.6196
0.5671
0.56890.4799Cross Entropy
0.52080.76720.75880.8116
2.1581

Rank
0
1
11
15
24

ICM-Soft
3.11410.8157
0.66320.48530.3506EN
ICM-Soft Norm
10.66830.64630.62070.6012

Cross Entropy
0.577
1.01980.85120.96772.31965.2. Task 2 - Source Intention Identification
For the second task (categorizing the tweets according to the intention of the source), as shownin Tables 9, 10 and 11, the second run utilizing the ’probability distribution (mean)’ method
obtains the best results in this task and achieves the second position in hard-soft evaluationfor Spanish. Over both languages, our best approach secured the 9th position in the hard-hardevaluation, the 22nd position in the hard-soft evaluation, and the 3rd position in the soft-soft
evaluation.
Table 9Results for the hard-hard evaluation for Task 2: This table presents the results obtained in the hard-hardevaluation. The three main columns represent the results for different languages: the first column shows
results for all instances, the second column focuses on Spanish instances, and the third column focuses
on English instances. Each column includes the following metrics for each run: ICM, Normalized ICM,
and F-measure. The first row indicates the gold standard result, while the second row represents thebest result. The remaining rows display the results obtained by our three evaluation runs.
RUN
Gold
Best score
JPM_UNED_1
JPM_UNED_2
JPM_UNED_3

Rank
0
1
11
9
10

ICM-Hard
1.53780.48870.16730.18620.1806ALL
ICM-Hard Norm
10.77640.70790.712
0.7108F1
10.57150.50320.50540.5092

Rank
0
1
10
8
9

ICM-Hard
1.60070.5711
0.19860.23510.2231ES
ICM-Hard Norm
10.77320.69110.69920.6965F1
10.60590.52810.5341
0.5383

Rank
0
1
12
13
11

ICM-Hard
1.44490.36770.10240.09950.1034EN
ICM-Hard Norm
1
0.781
0.727
0.72640.7272F1
10.5224
0.46610.46490.4673

Table 10
Results for the hard-soft evaluation for Task 2: This table presents the results obtained in the hard-soft
evaluation. The three main columns represent the results for different languages: the first column shows
results for all instances, the second column focuses on Spanish instances, and the third column focuses
on English instances. Each column includes the following metrics for each run: ICM-Soft and ICM-Soft
normalized. The first row indicates the gold standard result, while the second row represents the best
result. The remaining rows display the results obtained by our three evaluation runs.
RUN
Gold
Best score
JPM_UNED_1
JPM_UNED_2
JPM_UNED_3

Rank
0
1
23
22
24

ALL
ICM-Soft ICM-Soft Norm
6.20571-2.3974
0.7803-7.5078
0.6498-7.33460.6542-7.5205
0.6495

Rank
0
1
21
19
22

ES
ICM-Soft ICM-Soft Norm
6.24311-1.85020.7684-6.8073
0.6266-6.56220.6336-6.8533
0.6253

Rank
0
1
25
23
24

EN
ICM-Soft ICM-Soft Norm
6.11781-3.2650.7943-8.9034
0.6707-8.8583
0.6717-8.89780.6708

Table 11
Results for the soft-soft evaluation for Task 2: This table presents the results obtained in the soft-soft
evaluation. The three main columns represent the results for different languages: the first column shows
results for all instances, the second column focuses on Spanish instances, and the third column focuses
on English instances. Each column includes the following metrics for each run: ICM-Soft, ICM-Soft
normalized and Cross entropy. The first row indicates the gold standard result, while the second row
represents the best result. The remaining rows display the results obtained by our three evaluation runs.
RUN
Gold
Best score
JPM_UNED_2
JPM_UNED_3

Rank
0
1
3
5

ICM-Soft
6.2057-1.3443-1.675-1.6888

ALL
ICM-Hard Soft1
0.8072
0.79880.7984

Cross Entropy
0.91281.78332.55492.5561

Rank
0
1
2
3

ICM-Soft
6.2431-1.2317-1.4414-1.5006

ES
ICM-Soft Norm
1
0.78610.78010.7785

Cross Entropy
0.8926
1.64152.44722.4511

Rank
0
1
6
4

ICM-Soft
6.1178-1.1471-2.1062-2.0436EN
ICM-Soft Norm
10.84070.81970.8211

Cross Entropy
0.93541.80012.6757
2.674

6. Conclusions
This paper presents the participation of the JPM-UNED team in the Task 1 and the Task 2 of
the EXIST 2023 Lab at CLEF, which focuses on the classification of sexism in social networks
with disagreement. We have investigated different approaches to learning with disagreement,leveraging the current state of the art. Furthermore, we essayed different data augmentation
techniques, such as incorporating translations of tweets between English and Spanish in the
training set.
Among the approaches we explored, some of them proposed different variations of thejudgement aggregation method, which combines the judgments or opinions of the multiple
annotators and models to arrive at the final label. Notably, our best results were obtained in Task
2, where we secured the second position in the soft-soft evaluation for the Spanish language.
This achievement was made possible by employing the Judgement Aggregation approach thatleverages the viewpoints of the six different cohorts.
One limitation of our work stems from the size of the dataset. As some of our approaches
involved splitting the dataset into six cohorts, the resulting training datasets were relatively
small, which presented challenges in effectively training the models.

7. Acknowledgments
This research is funded by FAIRTRANSNLP-DIAGNOSIS: Measuring and quantifying bias and fairness in NLP systems, grant PID2021-124361OB-C32, funded by
MCIN/AEI/10.13039/501100011033 and by ERDF, EU A way of making Europe. This work
has been also funded by the Ministry of Universities and the European Union through theEuropeaNextGenerationUE funds and the “Plan de Recuperación, Transformación y Resiliencia”.

References[1] P. Fortuna, S. Nunes, A survey on automatic detection of hate speech in text, ACM
Computing Surveys (CSUR) 51 (2018) 1–30.
[2] A. R. G. Prasad, V. Seshagiri, L. Ravindranath, et al., A catalytic spectrophotometric method
for the analytical determination of trace amounts of mercury (ii), Chem. Sci. Jour (2010)
1–8.
[3] S. Y. Jeong, Y. S. Koh, G. Dobbie, Phishing detection on twitter streams, in: Trends andApplications in Knowledge Discovery and Data Mining: PAKDD 2016 Workshops, BDM,MLSDA, PACC, WDMBF, Revised Selected Papers 20, Springer, 2016, pp. 141–153.
[4] J. Nogués Graell, Detection of toxicity in social media. a study on semantic orientationand linguistic structure (2022).
[5] F. Rodríguez-Sánchez, J. Carrillo-de Albornoz, L. Plaza, J. Gonzalo, P. Rosso, M. Comet,
T. Donoso, Overview of exist 2021: sexism identification in social networks, Procesamientodel Lenguaje Natural 67 (2021) 195–207.
[6] F. Rodríguez-Sánchez, J. Carrillo-de Albornoz, L. Plaza, A. Mendieta-Aragón, G. MarcoRemón, M. Makeienko, M. Plaza, J. Gonzalo, D. Spina, P. Rosso, Overview of exist 2022:
sexism identification in social networks, Procesamiento del Lenguaje Natural 69 (2022)
229–240.
[7] L. Plaza, J. Carrillo-de Albornoz, R. Morante, E. Amigó, J. Gonzalo, D. Spina, P. Rosso,
Overview of exist 2023: sexism identification in social networks, in: Advances in Information Retrieval: 45th European Conference on Information Retrieval, ECIR 2023, Dublin,
Ireland, April 2–6, 2023, Proceedings, Part III, Springer, 2023, pp. 593–599.
[8] A. Uma, T. Fornaciari, A. Dumitrache, T. Miller, J. Chamberlain, B. Plank, E. Simpson,
M. Poesio, Semeval-2021 task 12: Learning with disagreements, in: Proceedings of the
15th International Workshop on Semantic Evaluation (SemEval-2021), 2021, pp. 338–347.
[9] L. Plaza, J. C. de Albornoz, R. Morante, E. Amigó, J. Gonzalo, D. Spina, P. Rosso, Overview
of exist 2023 – learning with disagreement for sexism identification and characterization(extended overview), in: Working Notes of CLEF 2023 - Conference and Labs of the
Evaluation Forum, 2023.
[10] L. Plaza, J. C. de Albornoz, R. Morante, E. Amigó, J. Gonzalo, D. Spina, P. Rosso, Overview
of exist 2023 – learning with disagreement for sexism identification and characterization,
in: Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings ofthe Fourteenth International Conference of the CLEF Association (CLEF 2023), 2023.
[11] O. Istaiteh, R. Al-Omoush, S. Tedmori, Racist and sexist hate speech detection: Literaturereview, in: 2020 International Conference on Intelligent Data Science Technologies andApplications (IDSTA), IEEE, 2020, pp. 95–99.
[12] H. Abburi, S. Sehgal, H. Maheshwari, V. Varma, Knowledge-based neural frameworkfor sexism detection and classification., in: Proceedings of IberLEF@ SEPLN, 2021, pp.
402–414.
[13] L. Altin, H. Saggion, Automatic detection of sexism in social media with a multilingual
approach, in: Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2021).
CEUR Workshop Proceedings Series, 2021, pp. 415–419.
[14] H. Abburi, P. Parikh, N. Chhaya, V. Varma, Fine-grained multi-label sexism classification
using a semi-supervised multi-level neural approach, Data Sci. Eng. 6 (2021) 359–379.
[15] E. Bassignana, V. Basile, V. Patti, Hurtlex: A multilingual lexicon of words to hurt, in: 5th
Italian Conference on Computational Linguistics, CLiC-it 2018, volume 2253, CEUR-WS,
2018, pp. 1–6.
[16] L. I. Merlo, B. Chulvi, R. Ortega, P. Rosso, When humour hurts: linguistic features to foster
explainability, 2023-03.
[17] A. P. Dawid, A. M. Skene, Maximum likelihood estimation of observer error-rates using
the em algorithm, J. Royal Stat. Soc. Series 28 (1979) 20–28.
[18] F. Rodrigues, F. Pereira, B. Ribeiro, Gaussian process classification and active learningwith multiple annotators, in: Proceedings of ICML, 2014, pp. 433–441.
[19] P. Welinder, P. Perona, Online crowdsourcing: Rating annotators and obtaining costeffective labels, in: IEEE CVPR Workshops, 2010, pp. 25–32.
[20] D. Reidsma, R. Akker, Exploiting ’subjective’ annotations, in: Coling 2008: Proceedings of
the workshop on Human Judgements in Computational Linguistics, 2008, pp. 8–16.
[21] V. S. Sheng, F. Provost, P. G. Ipeirotis, Get another label? improving data quality anddata mining using multiple, noisy labelers, in: Proceedings of the 14th ACM SIGKDD
international conference on Knowledge discovery and data mining, 2008, pp. 614–622.
[22] F. Rodrigues, M. Lourenço, B. Ribeiro, F. C. Pereira, Learning supervised topic models forclassification and regression from crowds, IEEE Transactions on Pattern Analysis andMachine Intelligence 39 (2017) 2409–2422.
[23] A. Uma, T. Fornaciari, D. Hovy, S. Paun, B. Plank, M. Poesio, Learning from disagreement:
A survey, Journal of Artificial Intelligence Research 72 (2021) 1385–1470.
[24] E. Amigo, A. Delgado, Evaluating extreme hierarchical multi-label classification, in:Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics(Volume 1: Long Papers), Dublin, Ireland, 2022, pp. 5809–5819.