JUST_ONE at SemEval-2023 Task 10: Explainable Detection of Online Sexism (EDOS)
Year: 2023

Abstract:
The problem of online sexism, which refers to offensive content targeting women based on their gender or the intersection of their gender with one or more additional identity characteristics, such as race or religion, has become a widespread phenomenon on social media. This can include sexist comments and memes. To address this issue, the SemEval2023 international workshop introduced the "Explainable Detection of Online Sexism Challenge", which aims to explain the classifications given by AI models for detecting sexism. In this paper, we present the contributions of our team, JUST_ONE, to all three sub-tasks of the challenge: subtask A, a binary classification task; subtask B, a four-class classification task; and subtask C, a fine-grained classification task. To accomplish this, we utilized pretrained language models, specifically BERT and RoBERTa from Hugging Face, and a selective ensemble method in task 10 of the SemEval 2023 competition. As a result, our team achieved the following rankings and scores in different tasks: 19th out of 84 with a Macro-F1 score of 0.8538 in task A, 22nd out of 69 with a Macro-F1 score of 0.6417 in task B, and 14th out of 63 with a Macro-F1 score of 0.4774 in task C.

heading: 1 Introduction
text: Social media has facilitated the freedom of expression, but it has also enabled the spread of hate speech against specific communities, such as races, sexes, and ethnicities (Zhang and Luo, 2019; Habash et al., 2022; Faraj and Abdullah, 2021). Sexism is a particularly challenging form of hate speech to address because it can be difficult to identify and significantly impacts women’s lives. Women are often subjected to abusive and discriminatory behaviors based on gender, religion, or other
identity factors (Younus and Qureshi, 2022; Abburi et al., 2021; Frenda et al., 2019). Given the seriousness of the problem, researchers in natural language processing (NLP) have been inspired to develop novel methods for sexism in text detection and classification (Butt et al., 2021).
This paper discusses the JUST_ONE team’s contribution to SemEval 2023-task10. The task aimed to detect and identify various types of sexist content using Gab and Reddit data (Kirk et al., 2023). We used the pre-trained language models: BERT (Devlin et al., 2018), and RoBERTa / unlabeled RoBERTa (Liu et al., 2019) from Hugging face, alongside the ensemble method (selective ensemble), and optimized the models’ hyperparameters for each of the three subtasks (A, B, C). It is worth mentioning that our team ranked 19th out of 84 in task A, with a Macro-F1 score of 0.8538, 22nd out of 69 in task B with a Macro-F1 score of 0.6417, and 14th out of 63 in task C, with a Macro-F1 score of 0.4774.
The rest of the paper is structured as follows: A background of task setup and data description is covered in section 2. Section 3 provides insights into the related work. A system overview is provided in section 4. The experiments are described in section 5; section 6 provides our results. Finally, we conclude this work in section 7.

heading: 2 Background

heading: 2.1 Task Setup
text: In SemEval2023-task 10, there are three hierarchically arranged subtasks (Kirk et al., 2023). Below is the brief definition for each subtask:
• TASK A - Binary Sexism Classification: The system aims to determine if the input text is sexist, with two possible outcomes: sexist or not sexist. An example of input text is: "O come on, there’s no way any men have attracted to her she a pig", which contains a
526
sexist statement, and the corresponding output is sexist.
• TASK B - Category of Sexism: The system aims to categorize sexist content into one of four categories: 1. threats, 2. derogation, 3. animosity, and 4. prejudiced discussions. An example of sexist input text is "O come on, there’s no way any men are attracted to her she is a pig", and the output is class 2, indicating that the text falls into the derogation category because it involves insulting and belittling a woman based on her appearance.
• TASK C - Fine-grained Vector of Sexism: By using fine-grained vectors, the system can capture more detailed information about the category of sexism displayed in a given sexist post, detecting one of eleven Fine-grained Vectors. An example of sexist input text is Ö come on thereś no way any men are attracted to her shes a pig,̈ and the output: 2.3 dehumanizing attacks & overt sexual objectification, which refers to a more detailed category derogation. The phrase s̈heś a pigïs dehumanizing, as it compares the woman to an animal, implying that she is dirty, unattractive, and unworthy of respect or consideration.

heading: 2.2 Data Description
text: The dataset used in this task consists of 20,000 posts labeled in English (Kirk et al., 2023), divided into training, development, and testing sets with proportions of 70%, 10%, and 20%, respectively. Table 1 provides detailed information on these sets, with training data applicable to all subtasks and separate data for each subtask in the development and testing sets. Using this dataset, we created a task-specific dataset, as illustrated in Figure 1. In addition, table 1 provides the definitions for each variable used in the dataset.
tabularx

heading: 3 Related Work
text: The field of text classification has seen significant advancements thanks to machine learning, with a particular focus on identifying instances of sexism in text. Researchers have explored various approaches in this area, including collecting sexist data and employing different techniques to detect the presence of a sexual agenda in a piece of
text. For example, in a recent study by the authors of (Liakhovets et al., 2022), three transformers were used to detect sexism and classify text into multiple categories in response to the 2022 IberLEF EXIST challenge. These included a monolingual T5 model for English, a multilingual BERT, and the XLM-RoBERTa model, which achieved a Macro F1-score of 0.7496 for binary classification and 0.4675 for multi-class classification. The researchers further improved performance by using unsupervised pre-training with additional data and data augmentation.
The authors of (Fersini et al., 2019) developed a unimodal and multimodal classifier for detecting instances of written and visual sexism in social media memes using various machine learning models. The researchers collected a benchmark dataset of 800 memes from various social media platforms and extracted visual and textual features. Using these features, they developed unimodal classifiers for detecting sexism in memes based on visual or textual features and a multimodal classifier created through early and late fusion. The best binary classification performance was achieved using a Support Vector Machine (SVM) classifier with a recall of 0.81 and 0.83 for the neural network classifier.
Shibly and colleagues employed multiple ma-
chine learning algorithms, including Decision Tree, Logistic Regression, Random Forest, Long ShortTerm Memory, and a voting classifier, to automatically detect instances of hate speech targeting women, as detailed in (Shibly et al., 2023). They used the MeToo open-source dataset for their study and found that the voting classifier model achieved the highest F1-measure value of 0.9. In (El Ansari et al., 2020), two different strategies were suggested for creating a thematic training package that integrates manual and machine translations of Arabic materials related to gender inequality and violence against women.
In the SemiEVAL 2020 competition focused on identifying the offensive language in social media across multiple languages (OffensEval 2: Multilingual Offensive Language Identification in Social Media), Das and their team participated and explained their methodology in their publication cited as (Das et al., 2020). The team utilized a checkpoint ensemble-based transformer model composed of Ensembled GPT-2, Ensembled RoBERTa, and Ensembled DistilRoBERTa to address the competition’s three subtasks. Additionally, the attention mask dropout technique was employed to mitigate the issue of poor composition in social media texts. The team’s performance in the English edition of the competition was notable, with Macro F1 scores of 0.909, 0.551, and 0.616 achieved for the three subtasks across five languages. The focus of (Zimmerman et al., 2018) was on hate speech in social media and the concerns around censorship, with a particular emphasis on human rights. The authors proposed new methods to identify and address discrimination while preserving freedom of expression more effectively. They introduced a novel clustering method that utilizes a neural network approach to improve the classification of hate speech. To evaluate its performance, the method was tested on the Twitter hate speech dataset and the common sentiment dataset using a publicly available inclusion model. The results demonstrated an improvement of around 5 points on an F1 scale compared to the original work on a publicly available hate speech assessment dataset. The paper also acknowledges the challenges of reproducing deep learning methods and comparing the results of other studies.

heading: 4 System Overview
text: The study outlines the approach for designing the system, which is tailored for each sub-task based on factors such as the task’s difficulty level and dataset size. Multiple experiments were conducted to ensure the design met the system’s needs. This flexible approach allows for adaptation to the unique requirements of the system. The details of the work on each sub-task are presented separately as follows:

heading: 4.1 Task A
text: We fine-tuned the RoBERTa architecture with specific hyperparameters to identify sexist content through binary classification. The optimal hyperparameters for this model include the following: A learning rate of 2e-5. Four epochs with a batch size of 16. A dropout of 0.2. Two fully connected layers.

heading: 4.2 Task B
text: In this sub-task, we designed a simple selective ensemble approach that depends on the highest probability among two models. The technique combines the predictions of two pre-trained models (RoBERTa and BERT) and selects the one with the highest probability as the final prediction. This approach takes advantage of the strengths of both models, leading to improved performance and reduced overfitting. The selective ensemble was designed to improve the overall F1 score of the system and address the limitations of individual models. We achieved improved results by carefully selecting the models to be combined after diverse experiments. For example, if model 1 generates a prediction with a probability of 0.95 for class 2, and model 2 generates a prediction with a probability of 0.56 for class 1, the final prediction would be class 2 as it has the highest probability among the two models. The approach of the selective ensemble is shown in Figure 2.

heading: 4.3 Task C
text: For this task, we employed the same approach as in task B, which combined the predictions of two pretrained models. However, for this task, we utilized one labeled model (RoBERTa) and one unlabeled model (RoBERTa) from Hugging Face to make predictions. Our experiments utilizing a selective ensemble approach demonstrated that choosing the prediction with the highest probability among the two predictions generated the most optimal outcomes. Additionally, to overcome the issue of class imbalance, we incorporated the widely utilized Focal loss in text classification for all sub-tasks.

heading: 5 Experimental Setup
text: The labeled dataset comprises 10,000 samples from GAB and 10,000 from Reddit, split into 14,000 entries for training, 2,000 for development, and 4,000 for testing. Additionally, 2 million unlabeled data entries augment the training stage. We evaluated multiple pre-trained models by training them on the labeled training set and testing their performance on a validation set with an 80:20 ratio. The models were trained for 3 or 4 epochs, depending on the size of the dataset, and we selected the best model checkpoint based on its Macro F1 score on the development set. We took steps to prevent overfitting during this process. Transfer learning was employed in our approach by reusing fine-tuned embedding parameters. This is done by saving the checkpoint of fine-tuned model embeddings from the earlier stage and utilizing it as the initialization embeddings for the next model stage. The illustration of transfer learning is shown in Figure 3. The methodology employed in this study involved running our code on the Kaggle environment using Python 3.9 and PyTorch. For all experiments, we utilized the implementation by Hugging Face to import pre-trained models. We also used the focal loss to deal with the imbalanced dataset. We performed multiple experiments to fine-tune various pre-trained models, determine which ones were best suited for our sub-tasks, and identify the most appropriate hyperparameters. Table 2 offers a detailed explanation of the particular terms utilized in the experiments. Moreover, Tables 3, 4, and 5 display the outcomes of our fine-tuning and experimentation across three distinct tasks on the development dataset.

heading: 6 Results
text: Our team performed satisfactorily for the three subtasks of SemEval 2023 - Task 10: Explainable Detection of Online Sexism. The RoBERTa large model produced promising results for task A, achieving a Macro-F1 score of 0.8538 and ranking 19 out of 84 teams. For task B, the selective ensemble model resulted in a Macro-F1 score of 0.641 and an order of 22 out of 69, while for task C, it achieved a Macro-F1 score of 0.4774 and a ranking of 14 out of 63. Our participation highlights the importance and urgency of detecting online sexism. We used different approaches for each task to adapt to them. In Task A, we used fine-tuned pre-trained model to get the best performance we reached. While we realized that using the ensemble approach in tasks B and C gave better results than fine-tuning, this is because we assume that ensemble approaches are more effective when there is variance in the data and avoid overfitting when fine-tuning is used with more complex data. Table 6 shows the results we have obtained throughout our contribution.

heading: 7 Conclusion
text: This paper outlines our approach to participating in SemEval 2023 - Task 10: Explainable Detection of Online Sexism (EDOS), a crucial and relevant issue that negatively impacts women. The task attracted numerous teams this year and comprised three subtasks. We utilized several models based on pre-trained RoBERTa and BERT architecture and a selective ensemble approach. For the binary classification subtask (A), we fine-tuned the RoBERTa model, which resulted in a Macro F1 score of 0.8538. Our selective ensemble model achieved a Macro F1 score of 0.6417 for subtask B and 0.4774 for subtask C. In summary, our future work includes investigating alternative ensemble strategies like weighting average and majority voting to improve the performance of our EDOS system in multi-class subtasks (B) and (C). We also plan to explore other pre-training techniques, such
as GPT-3, and to use semi-supervised and unsupervised approaches to address data scarcity by utilizing 2 million unlabeled data.

heading: Future Work
text: In summary, our future work includes investigating alternative ensemble strategies like weighting average and majority voting to improve the performance of our EDOS system in multi-class subtasks (B) and (C). We also plan to explore other pre-training techniques, such
as GPT-3, and to use semi-supervised and unsupervised approaches to address data scarcity by utilizing 2 million unlabeled data