KingsmanTrio at SemEval-2023 Task 10: Analyzing the Effectiveness of Transfer Learning Models for Explainable Online Sexism Detection
Year: 2023

Abstract:
Online social platforms are now propagating sexist content endangering the involvement and inclusion of women on these platforms. Sexism refers to hostility, bigotry, or discrimination based on gender, typically against women. The proliferation of such notions deters women from engaging in social media spontaneously. Hence, detecting sexist content is critical to ensure a safe online platform where women can participate without the fear of being a target of sexism. This paper describes our participation in subtask A of SemEval-2023 Task 10: Explainable Detection of Online Sexism (EDOS). This subtask requires classifying textual content as sexist or not sexist. We incorporate a RoBERTa-based architecture and further finetune the hyperparameters to entail better performance. The procured results depict the competitive performance of our approach among the other participants.

heading: 1 Introduction
text: The free flow of information is the most fascinating feature of social media that has enabled sharing of information, thoughts, and opinions across the world. But this very feature is now used as a tool to incite hatred and threats against individuals or communities. Our society has always exploited women with prejudice, stereotype, and discrimination. The creation and proliferation of social media have provided our society with a more extensive platform to induce such unjust treatment against women. People can post sexist content anonymously using fake and untraceable identities without the fear of accountability. Moreover, the hatred can be spread to the wider community impeding the integration of women into the social network. So it is essential to detect such sexism in social media content so that women can partake and collaborate in social media without being mistreated and threatened. With
All of the authors have equal contributions.
this intention, SemEval-2023 Task 10: Explainable Detection of Online Sexism (EDOS) (Kirk et al., 2023) is introduced. The aim of the task is to detect sexism in the textual content of social media. The task is divided into three hierarchical tasks. We participated in the subtask A - Binary Sexism Detection where a text must be classified as sexist or not-sexist. To explicate this task, two examples are shown in 1.
Here, in the first example, the capability of women in political activities is being doubted just based on their sex. Hence, the first text is considered sexist. On the other hand, the second example depicts violence toward women but doesn’t encourage or support it. So the second text is not sexist.
Detecting sexism in social media has been of great concern for researchers. Numerous works have been conducted on detecting sexism in texts. For instance, (Parikh et al., 2019) employed LSTM and CNN-based architecture along with several word embeddings including ELMo, GloVe, and fastText, and also sentence embedding incorporating BERT, Universal Sentence Encoder (USE), and InferSent. Convolutional filters were explored by (Sharifirad et al., 2019). to extract the most important n-grams in a specific category of sexism (i.e., Indirect sexism, Sexual sexism, and Physical sexism). They filtered out the irrelevant n-grams and clustered the significant ones to enhance the
1916
performance. (Grosz and Conde-Cespedes, 2020) worked on a specialized Twitter dataset that focuses on workplace sexism. They employed GloVe and random word embeddings along with attentionbased LSTMs. Moreover, (Samory et al., 2021) discuss how most of the previous works deal with explicit expressions of sexism. So when faced with implicit sexism and adversarial sample, most of the proposed models don’t work well. Transformer models such as XLM-R and mBERT were also proposed (Schütz et al., 2021). (Chiril et al., 2021) leveraged the problem of detecting gender stereotypes to entail better outcomes in the detection of sexism. On the other hand, (Vaca-Serrano, 2022) employed an ensemble of finetuned DeBERTa, RoBERTa, and BERTweet for detecting sexism in English tweets.
In this paper, we elucidate our findings achieved from our investigation on this task. We propose a RoBERTa-based architecture and further hyperparameter tuning to achieve an enhanced result. The organization of this paper is as follows: we illustrate our proposed methodology in Section 2. Section 3 elaborates on the experimental details. We also incorporate our study on the performance of various approaches and error analysis in Section 4. Finally, we culminate this paper with some
future aspects in Section 5.

heading: 2 Proposed Framework
text: For the aim of detecting explainable online sexism, we described our tested models in this part. Various transfer learning and deep learning models were explored in this aspect.
RoBERTa-Sexism-Classifier: The RoBERTaSexism-Classifier is a well-liked and adaptable architecture that makes use of an enhanced bidirectional encoder mechanism. It can understand more complex intricate relationships between words and phrases because it was trained on a broader and more diverse corpus. We fine-tuned the RoBERTa pre-trained transformer-based architecture with the explainable online sexism dataset to achieve our goal. The overall fine-tuned process is depicted in Figure 1.
An input text is initially tokenized using the RoBERTaTokenizer (Wolf et al., 2019). All of the tokens were padded to the same sequence length. It is then fed to the first transformer layer, and the output of that layer is fed to the second. This procedure continues until the transformer layer 24’s output is obtained. We used the RoBERTaForSequenceClassification (Liu et al., 2019) technique to obtain the final prediction.

heading: 3 Experiments and Evaluations

heading: 3.1 Dataset Description
text: We utilized the dataset proposed on the EDOS task (Kirk et al., 2023) in SemeEval-2023. The dataset has a total of 20000 entries gathered from Gab and Reddit (10000 each). Three annotators label the gathered entries, and the expert reviews any disagreements in entries. The task is broken down into three smaller tasks. In this paper, we only concentrated on Task A. They separated the 20000 entries into three subsets, with the training, validation, and testing sets consisting of 14000, 2000, and 4000 text entries, respectively. The dataset is used for the training model to determine whether or not a post is sexist. Moreover, we discovered that only 24.3 percent of the data in our dataset has sexist and not sexist labels after evaluating the distribution from Table 2, containing sexist and not sexist labels. The Macro F1-Score is the key assessment statistic utilized for this task.

heading: 3.2 Experimental Setup
text: We proposed the RoBERTa-Sexism-Classifier as our final KingsmanTrio system since finetuning its pre-trained embedding layers helps predict better than our other evaluated approaches. Table 3 shows the experimental setup for our best-scoring system.

heading: 3.3 Results Analysis
text: We now evaluate our system’s performance against that of our other participants’ systems. Table 4 shows the performance of the 84 valid submissions with that of the top-performing teams. It shows that our system stands out among the participants’ systems in a competitive manner. It narrowly falls short of PingAnLifeInsurance, the topperforming team, by 3.8%.

heading: 4 Discussion
text: We explored numerous deep learning and transfer learning models in order to estimate and identify the most effective strategy. In Table 5, several model performances are displayed based on the development set.
In Table 5, the analysis of the model’s performance shows that transfer learning models outper-
form deep learning models. Among the other tried approaches in Finetuned BERT+RoBERTa, we consider an ensemble of the labels to be not-sexist if it is predicted by both of the individual model. However, the ensemble does not bring a good outcome though.
Moreover, Models of transfer learning made use of prior knowledge to produce universal features and patterns. Transfer learning models can exceed deep learning models which are trained from scratch on smaller datasets by making use of this general knowledge. Yet, because transfer learning models begin with pre-trained weights, they can converge more quickly during training, enabling the model to pick up knowledge about a given task quickly. Also, we can observe that the RoBERTa model outperforms the BERT model. It might be a result of RoBERTa’s training on a significantly larger corpus of text data than BERT, which likely contained more diverse and high-quality content. As a result, RoBERTa can develop more accurate representations of the language, which will help it perform better on downstream tasks.
We look into the cause of our suggested system’s incorrect predicted labels further. Table 6 provides several instances in this respect.
Our proposed model was able to identify the fact that the first example used derogatory language against women and the use of the word "bitch" is a
form of gender-based harassment towards women, which contributes to a culture of sexism and misogyny. However, in the second example, the text makes a generalization using a double standard meaning that women are not treated equally when it comes to speaking about men versus women. Here, the term “chauvinist asshole" further reinforces the sexist nature of the text. But, because our system was unable to comprehend the significance of this double standard, it made a false forecast. Even though the third example has the gold label "Not Sexist," our methodology predicts "Sexist" due to its rude and demeaning character towards a certain gender. The term "only TWO genders" can be seen as being dismissive and exclusionary since it suggests that non-binary people do not exist. Also violent and unpleasant is the idea that someone is forced to drink entire bottles of coke. To elaborate further, we can say that our method failed to detect 123 out of 486 sexist posts. This may be attributed to the fact that there is an insufficient number of sexist posts in the training dataset.

heading: 5 Conclusion and Future Notion
text: In this paper, we traversed several transformerbased architectures and deep learning models. We further conducted a performance analysis of these models and investigated the mispredictions of the top-performing model. In the future, we intend to incorporate domain-specific features along with other transformer architectures. Moreover, as social media content are not limited to text only, we aspire to detect sexism in other modalities too.

heading: Future Work
text: In the future, we intend to incorporate domain-specific features along with other transformer architectures. Moreover, as social media content are not limited to text only, we aspire to detect sexism in other modalities too