JCT_DM at SemEval-2023 Task 10: Detection of Online Sexism: from Classical Models to Transformers
Year: 2023

Abstract:
This paper presents the experimentation of systems for detecting online sexism relying on classical models, deep learning models, and transformer-based models. The systems aim to provide a comprehensive approach to handling the intricacies of online language, including slang and neologisms. The dataset consists of labeled and unlabeled data from Gab and Reddit, which allows for the development of unsupervised or semi-supervised models. The system utilizes TF-IDF with classical models, bidirectional models with embedding, and pre-trained transformer models. The paper discusses the experimental setup and results, demonstrating the effectiveness of the system in detecting online sexism.

heading: 1 Introduction
text: Sexism is a pervasive problem online, particularly for women who are disproportionately targeted. It can cause harm, make online spaces unwelcoming, and reinforce social inequalities. Automated tools have been developed to identify sexist content at scale, but most current systems only provide generic, high-level classifications without further explanation. This lack of detail makes it difficult for users and moderators to understand and trust the decisions made by these systems.
SemEval 2023 - Task 10 - Explainable Detection of Online Sexism (EDOS) (Kirk et al., 2023) aimed at releasing a dataset to assist in the development of English-language models for sexism detection that are more accurate and explainable. This paper handles the first subtask A in task 10. We experimented with systems relying on classical models, deep-learning models, and transformer-based models to tackle the first task of binary sexism detection on Gab and Reddit. The goal of our system is to provide a comprehensive approach that can handle the intricacies of online language, which often involves slang, colloquialisms, and neologisms.

heading: 2 Related Work
text: Previous competition "sEXism Identification in Social neTworks (EXIST)" (Rodríguez-Sánchez et al., 2021) was conducted during the IberLEF 2021 international forum, which was exclusively focused on the field of Natural Language Processing. The competition aimed to classify multilingual posts into five categories, including ideological and inequality, stereotyping and dominance, objectification, sexual violence, misogyny, and non-sexual violence. The SemEval competition differs and involves the identification of 11 types of sexual posts, indicating a more complex and nuanced classification task.
In addition, Fortuna (Fortuna et al., 2022) addresses the challenges that arise when applying conventional NLP methodologies to hate speech detection. The authors reflect on the inadequacies of frequently used conventions for text classification with respect to hate speech detection and discuss the future of current methodologies for this task. Based on their analysis, they conclude that current models are incapable of detecting hate speech without causing harm to marginalized communities. Therefore, they call for the scientific community to adapt NLP methodologies and center on the impact that used methodologies may have on marginalized communities. The authors believe that by critically reflecting on the potential impacts of methodologies for hate speech detection, the scientific community can identify methodologies that result in more just futures.

heading: 3 Dataset
text: The dataset (Kirk et al., 2023) for the binary sexism detection task consists of three parts: a training set with 14,000 sentences, a development set with 2,000 sentences, and a test set with 2,000 sentences (Table 1). Within the training set, 75% of the sentences are labeled as non-sexist and the remaining
739
25% are labeled as sexist. The purpose of this dataset is to develop English-language models for automated sexism detection, which is an important and growing problem online.
In addition to the labeled data, there are two unlabeled corpora, one from Gab and one from Reddit. These corpora are provided by task organizers as well. Each contains 1,000,000 sentences. These corpora were included to provide additional unlabeled data for the development of unsupervised or semi-supervised models. The availability of this dataset provides a valuable resource for researchers and practitioners to develop more accurate and explainable models for detecting and combatting sexism online.

heading: 4 System Overview
text: Our system incorporates a variety of models and methodologies, including classical models such as Random Forest (Ho, 1995) and Logistic Regression (Grimm and Yarnold, 1995) based on vectors generated by the TF-IDF statistical measure. We also explored deep learning models with bidirectional layers and word embedding created by Word2Vec (Mikolov et al., 2013). Additionally, we explored transformer-based models to further enhance our system’s performance.

heading: 4.1 TF-IDF with Classical Models
text: For the classical models, we utilized the TF-IDF statistical measure, to convert sentences into vectors. We experimented with two approaches, converting sentences based on words and converting them based on characters with multiple values of n-grams. We used the resulting TF-IDF vectors as input for both Random Forest and logistic regression models. We performed this step twice, first by calculating the TF-IDF only on the training dataset, and second by calculating it on the unlabeled dataset.

heading: 4.2 Bidirectional Model with Embedding
text: To obtain vectors with better semantic context, we incorporated pre-trained embedding models such as Google’s Word2Vec, which was trained on the Google News dataset, and GloVe-Twitter (Pennington et al., 2014). The GloVe model was trained
on a smaller dataset, but its corpus is more similar to ours. Additionally, we implemented another word2vec model trained on the unlabeled dataset to obtain a more specific context for our mission.
The embedding matrix size is determined by the number of unique words in the training corpus and the vector’s size. We used the resulting matrix as input to the embedding layers in our bidirectional models. Each row in the matrix represents a unique word in the training dataset. In case the word exists in the embedding model, we initialize the row with its corresponding vector, otherwise, we initialize it with a random vector from a normal distribution (mean and standard deviation similar to those of the embedding vectors).
Our bidirectional model architecture is based on a corresponding problem that involves classifying the emotional sentiment of COVID-19-related text as positive or negative (Arbane et al., 2023). The paper demonstrates the main advantage of replacing a regular LSTM (Hochreiter and Schmidhuber, 1997) with a Bi-LSTM (bidirectional LSTM) (Wang et al., 2017), which allows the system to learn from both directions in the text, unlike systems that use a regular LSTM, which can only learn from left to right.

heading: 4.3 Transformed-Based Models
text: We explored two different pre-trained transformers approaches. The first transformer used a general corpus with a higher volume of data. A smaller corpus, but more closely aligned with our problem, was used in the second approach.
For the first approach, we utilized the RoBERTa (Liu et al., 2019) pre-trained transformer model with its corresponding tokenizer. The RoBERTa model was trained on a large dataset, making it an effective choice for a wide range of tasks. For the second approach, we used the BERTweet (Nguyen et al., 2020) model, which was trained on a smaller dataset of tweets. The model and its corresponding tokenizer were specifically designed for sentiment analysis on Twitter data.

heading: 5 Experimental Setup
text: Our implementation uses Sklearn library (Pedregosa et al., 2011) for the classical models (TFIDF, logistic regression and Random Forest), Gensim library (Rehurek and Sojka, 2011) for the embedding models, Keras library (Chollet et al., 2015) for the Bi-LSTM model and the transformers li-
brary by HuggingFace (Wolf et al., 2020) for the pre-trained transformer models and corresponding tokenizers.

heading: 5.1 Pre-processing
text: The text data of social media is known for its informal nature, with users frequently using slang, abbreviations, and nonstandard grammar. The system incorporates three pre-processing techniques: handling emojis, capital words, and lemmatization. Emojis are graphical symbols used to express emotions and reactions, which can have a substantial impact on the interpretation and meaning of a sentence. Capitalization of words can alter the meaning of a sentence by emphasizing specific words. Finally, lemmatization is employed to identify the base form of a word, which can help to disambiguate the meaning of words with multiple senses or connotations.
During pre-processing, we used the Spacy library (Honnibal and Montani, 2017) and specifically the pre-trained en_core_web_sm model for lemmatization. We also utilized the emoji and emot library for converting emojis to words.

heading: 5.2 Classical Models
text: We experimented with the TF-IDF statistical measure by varying n-gram values for both word and character models. Specifically, we tested the word statistical measure with n-gram ranges of [1, 1], [1, 2], [1, 3], and the character statistical measure with n-gram ranges of [1, 1], [2, 2], [3, 3]. Logistic regression and Random Forest models were trained on the TF-IDF vectors. The logistic regression model was trained with hyperparameters of penalty L2, lbfgs solver, regularization equal to 10, and class weight set to False. The Random Forest model was trained with hyperparameters of 50 n_estimator and 2 max depth. The experiment was conducted twice, once with the TF-IDF being fitted only on the training dataset, and once with the TF-IDF being fitted on the unlabeled dataset.

heading: 5.3 Embedding Models
text: For the Bi-LSTM model, we used GloVe-Twitter200 pre-trained vectors, trained on a corpus of Twitter posts, and thus more similar to our dataset. Additionally, we examined the Word2Vec-GoogleNews-300 pre-trained vectors, which were trained on a larger dataset. We also experimented with training vectors on the unlabelled dataset to obtain
vectors that were from the same corpus and potentially more tuned to our task. The hyperparameters for the pre-trained vectors were a vector dimensionality size of 30, a minimum word count of 7, and a window context size of 5. An architecture for a Bi-LSTM model was developed based on the same layers in the Social Media-based COVID-19 (Arbane et al., 2023) model, with the softmax loss. Additionally, we trained the model twice, once using Focal Loss and once using CrossEntropy.

heading: 5.4 Transformer Models
text: To investigate the performance of state-of-the-art transformer models, we used RoBERTa (RoBERTabase) and BERTweet (vinai/bertweet-base) along with their corresponding tokenizers. We fine-tuned the models using the adamw optimizer, a learning rate of 2e-5, and a batch size of 16. The training was performed on a GPU T4x2 machine for 1.5 epochs. We use the softmax loss function.
The measure for evaluation is the Macro-F1 Score as defined by task organizers. The classical models were trained on Intel(R) Xeon(R) CPU, the embedding model on GPU Tesla T4, and the transformer models on GPU T4x2.

heading: 6 Results
text: The results of various models on the dev dataset, evaluated using the F1 metric score, are presented in Table 2. Our analysis focuses on the classical models, as well as two variants of a BiLSTM model trained with CrossEntropy and Focal Loss, and two transformer models, RoBERTa and BERTweet.
Among the classical models, the Random Forest model failed to learn the underlying patterns, resulting in a poor F1 score of 43%. In contrast, the Logistic Regression model achieved a better F1 score of 73.63% in the char approach and 73.65% in the words approach. Using an unlabeled corpus for training gives a small improvement in the Logistic Regression model’s performance, compared to using only the training corpus. In In general, the TF-IDF model trained on the unlabeled corpus with a word-based approach produced the highest score.
The Bi-LSTM models with GloVe-TwitterCrossEntropy and the Word2Vec model trained on the unlabelled corpus with CrossEntropy achieved a minor improvement, compared to their corresponding models with Focal Loss. In contrast, the
pre-trained Word2vec model achieved a minor improvement in Focal loss compared to the CrossEntropy model. In contrast to the TF-IDF, the leveraging of the unlabeled corpus did not improve performance. The results decreased by 0.636, and 0.429 compared to using the most effective Bi-LSTM models (GloVe-Twitter embedding vectors with and without focal loss, respectively). However, the Bi-LSTM model’s overall performance was lower than that of the Logistic Regression model.
The transformer models, RoBERTa and BERTweet, delivered the best performance among all models. Specifically, BERTweet outperformed the RoBERTa model, resulting in a higher performance improvement of 12.02% compared to the best TF-IDF model and 16.09% compared to the Bi-LSTM model. Hyper-tuning the BERTweet model by limiting the pre-processing to lemmatization resulted in the best score on the dev test with 84.07%, which was submitted and ranked 40th with 83.62% performance on the test set.

heading: 7 Conclusion
text: In this study, we proposed an approach for the Explainable Detection of Online Sexism (EDOS) task, which involved the development of various systems such as classical models, deep learning models, and transformer-based models to address the binary sexism detection challenge. Through our experiments, we found that the use of similar corpora such as Twitter led to improved performance across most models. The vast volume of data on Twitter, coupled with its diverse range of topics, makes it an ideal source for training machine learning models.
Moving forward, potential avenues for future research include the development of the procedure of pre-training BERT on Gab and Reddit as done on Twitter in BERTweet to improve performance. Furthermore, An ensemble of models of all the models developed in this study could lead to even better performance. By leveraging the strengths of each individual model and mitigating their weaknesses, An ensemble of models has the potential to produce more reliable results. As such, we believe that our work contributes to the broader goal of addressing online sexism. It provides a foundation on future studies to build upon.

heading: Future Work
text: Moving forward, potential avenues for future research include the development of the procedure of pre-training BERT on Gab and Reddit as done on Twitter in BERTweet to improve performance. Furthermore, An ensemble of models of all the models developed in this study could lead to even better performance. By leveraging the strengths of each individual model and mitigating their weaknesses, An ensemble of models has the potential to produce more reliable results. As such, we believe that our work contributes to the broader goal of addressing online sexism. It provides a foundation on future studies to build upon