PCJ at SemEval-2023 Task 10: A Ensemble Model Based on Pre-trained Model for Sexism Detection and Classification in English
Year: 2023

Abstract:
This paper describes the system and the resulting model submitted by our team "PCJ" to the SemEval-2023 Task 10 sub-task A contest. In this task, we need to test the English text content in the posts to determine whether there is sexism, which involves emotional text classification. Our submission system utilizes methods based on RoBERTa, SimCSE-RoBERTa pre-training models, and model ensemble to classify and train on datasets provided by the organizers. In the final assessment, our submission achieved a macro average F1 score of 0.8449, ranking 28th out of 84 teams in Task A.

heading: 1 Introduction
text: Sexism is a very serious problem on the Internet nowadays, where unsuspecting people attack women by making statements against them, exposing them to unfair treatment, and eventually causing harmful effects on society. Therefore, tools that can automatically detect sexism are widely deployed (Bordia and Bowman, 2019), which is an important task in the field of NLP (Mai et al.).
Sexism detection is a text classification problem. Text classification is a classical problem in the field of natural language processing, and the main methods can be divided into two categories: machine learning-based methods and deep learning-based methods. For the machine learning-based methods (Agarwal and Mittal), the main idea is artificial feature engineering plus shallow classification model (Li et al., 2020). Text feature engineering is divided into three parts: text preprocessing, feature extraction, and text representation, with the ultimate goal of converting text into a computer-understandable format and encapsulating enough information for classification, and then the classifier uses statistical classification methods, such as SVM, plain Bayesian classification algorithms. This approach overall relies heavily on discrete manual features.
For deep learning-based text classification (Minaee et al., 2021), the main idea is that the text to be classified is represented as a word vector, and then deep learning networks and other variants are used for automatic feature extraction, represented by models such as Transformer (Vaswani et al., 2017).
However, there are still some deficiencies in current sexism detection, which need to be improved, such as the abuse of language detection models, and the misestimation of sexism (Park et al., 2018). Task 10 of SemEval-2023 is the explainable detection of online sexism (Kirk et al., 2023), which aims to address the issue of automated detection of large-scale online sexism. By proposing a hierarchical taxonomy with three tasks for detecting sexist content, it ultimately aims to improve the accuracy and explainability of automated detection tools, and contribute to reducing the harmful effects of online sexism.
In this paper, we describe the details of our research in Subtask A and how it performed on the evaluation data. The specific task is described as performing sexism detection, so that the system must predict whether the post is sexist or not, making a binary classification. The paper is organized as follows. Section 2 briefly describes the relevant background regarding text classification and text preprocessing. Section 3 describes the system architecture used in the experiments to explain the model approach. In Section 4, the experiments and results are presented, and Section 5 presents a summary of the whole paper and areas for future improvements.

heading: 2 Related Work
text: The emergence of pre-trained models (PTMs) has brought the field of natural language processing into a whole new era. Pre-training has been an effective strategy for learning deep neural network parameters, and fine-tuning the downstream tasks based on pre-training models is beneficial for NLP
433
tasks. The development of PTMs goes through two stages from Non-Contextual Embeddings to Contextual Embeddings (Qiu et al.). For NonContextual Embeddings, the representative one is Word2Vec (Mikolov et al., 2013). This type of word embedding usually takes shallow networks for training, and when applied to downstream tasks, the rest of the whole model still needs to be learned from scratch. For Contextual Embedding, the main purpose is to solve the problem of multiple meanings of a word through a pre-trained encoder that can output contextually relevant word vectors, representative ones are OpenAI GPT (Radford et al., 2018), BERT (Devlin et al., 2018).In pre-training models, RoBERTa is an improved version of BERT, which uses a larger training dataset, longer training time, and some other optimization strategies. RoBERTa has achieved better results than BERT in multiple natural language processing tasks (Liu et al., 2019), including text classification tasks. Therefore, it is suitable to be used as the basic pre-training model for this task.
In conclusion, in this competition, we adopt a deep learning-based approach for text classification, using the existing dataset to fine-tune the pretrained models RoBERTa and SimCSE-RoBERTa to obtain the embedding vectors, combining BiLSTM (Wang et al., 2021) and BiGRU (Yu et al., 2021) neural network structures for task classification. Finally, in order to improve the task classification, the model ensemble method is used to fuse the fine-tuned models together (Malla and Alphonse, 2021).

heading: 3 System overview
text: This section describes the models used. We use RoBERTa and SimCSE-RoBERTa pre-trained models as the coding layer for modification, and on top of Roberta-large and sup-Simcse-Roberta-large base models, we then down-join BiLSTM and BiGRU structures to let the models capture more information in combination to contextual semantics. Then, we construct 4 types of text classification models. Finally, the trained models are fused and the final classification results are selected using the voting method. The system architecture used is described below.

heading: 3.1 Model blocks
text: RoBERTa is a more finely tuned version of BERT, proposed by Facebook AI (Liu et al., 2019), which
improves BERT in three aspects. It has two versions, RoBERTa-large and RoBERTa-base. Compared with the latter, the large version has 24-layer Transforms and is more capable of extracting features for this paper, so we choose RoBERTa-large as the base pre-training model. The structural modification based on the RoBERTa model is as follows.
Model 1: The last hidden layer of the RoBERTa model is plugged into the BiLSTM under the final layer, then classified after the Dropout layer, as in Figure 1.
Model 2: First, the output of the RoBERTa model is connected to BiLSTM and then to BiGRU, and then the average pooling and maximum pooling are performed on the output results, and finally, the results are spliced with the pooler-out of RoBERTa model, and then classified after the Dropout layer, as in Figure 2.
Model 3: Each layer of BERT’s structure understands the text differently (Jawahar et al., 2019), and the same is true in RoBERTa. Therefore, based on the original RoBERTa model, only the features of the last four layers are used, and the <CLS> of the last four hidden layers are stitched together, and then after maximum pooling, the dimensionality is reduced and the classification is performed again after the Dropout layer.
SimCSE (Gao et al., 2021) is a simple contrast learning framework that can greatly improve sentence embedding. It can get better embedding of sentences in the task of doing sentence representation. Sup-Simcse-RoBERTa-large model is a RoBERTa model trained with supervised SimCSE.
Model 4: The structure modification method
based on the SimCSE-RoBERTa model is as follows: the result of the last hidden layer obtained from the SimCSE-RoBERTa model is used as input to access the bidirectional LSTM, and then classified after the Dropout layer.

heading: 3.2 Model ensemble
text: In order to improve the classification effect of sexism detection, we use the voting method to obtain the final output from the output of the four models mentioned above. The voting method is an ensemble model approach that follows the principle of majority rule. It can combine each model’s characteristics thus improving the model’s robustness and generalization ability. This is achieved by first training each of the aforementioned models to obtain the best training results, and then having each model predict the same data. The final output is selected based on the majority vote.

heading: 4 Experiments and Results

heading: 4.1 Dataset and Processing
text: The dataset we use is the training data from the English dataset provided by the organizers (Kirk et al., 2023), which comes from Gab and Reddit and is divided into two categories sexist, and not sexist. Its training data consists of 14,000 entries, 3398 of which are classified as sexist. The category classification of the data is highly unbalanced, which greatly impacts the training of the model and the accuracy description.
In the experiment, the original dataset is divided into training set, validation set, and test set in an
8:1:1 ratio. During this process, the stratify parameter is set for stratified sampling to ensure that the proportions of different class samples in each subset are the same as in the original dataset. Then the training and validation sets are preprocessed by using preprocessing first. In order to get a better understanding, or to build a better algorithm, a pair of data sets is needed to remove the noise that may affect the results, including emoji and ’[xxx]’. On the other hand, the test set is not preprocessed, aiming to maintain similarity to the files tested in the final competition, allowing for a more realistic resulting score.

heading: 4.2 Implementation Details
text: In the specific training process, we used the BertAdam (Zhang et al., 2020) optimizer to optimize our model and adjusted the hyperparameters based on the results of the test set. The hyperparameter table using the final model is shown in Table 1.

heading: 4.3 Results
text: For the classification results, the Macro F1 score is used for evaluation, which calculates the precision and recall for each class, averages them and then calculates them according to the F1 score formula, as follows.
Precisionmacro =
∑ Precisioni
n (1)
Recallmacro =
∑ Recalli
n (2)
MacroF1 = 2 ∗ Precisionmacro ∗Recallmacro Precisionmacro +Recallmacro (3)
Where, Precision =
TP
TP + FP (4)
Recall = TP
TP + FN (5)
The final results of the experiments are shown in Table 2. The table shows the Marco F1 scores for each model on the divided test set.The results show that the RoBERTa model can already model text well on the test set. Just adding a BiLSTM structure after RoBERTa cannot improve the classification performance and may even decrease it. Compared with the model of SimCSE-RoBERTa with a BiLSTM structure, SimCSE-RoBERTa can better utilize semantic information to model text through contrastive learning, so a BiLSTM structure can achieve better performance in this task than RoBERTa. Based on RoBERTa-BiLSTM, we added a BiGRU structure and found that it can significantly improve the performance. This is because the BiGRU structure can better capture long-distance dependencies in the text sequence and further extract semantic information, thereby enhancing the model’s classification ability. For the ensemble model, it works better than any single model to improve the classification ability of sex discrimination detection, and it generalizes better. This is reflected in the gap between the results submitted in the development phase of the competition (Kirk et al., 2023) and our own results on the test set.
Table 3 shows the results of the scores based on the validation data provided during the development phase. It can be seen that the scores of the ensemble model do not differ much from the Macro F1 scores on their own divided training set, while the other models differ significantly. The model
scores of RoBERTa alone do not perform very well, while all others have improved, and the ensemble model remains the most effective model.
Table 4 shows the precision and recall of each model on different categories. We can see that there is a significant gap between the precision and recall of all single models in the two categories. In the non-sexism category, both values are above 0.9, indicating that the model can detect this category very well. However, in the sexism category, the values are only around 0.7, indicating that the model’s ability to detect this category is weaker. The reason for this is due to the imbalanced sample distribution, which results in insufficient feature learning for the sexism category. This is also a key focus for future work.

heading: 5 Conclusion
text: This paper describes our binary sexism detection system for subtask A of SemEval-2023 Task 10, including system design, implementation, and evaluation. The final experimental results demonstrate the effectiveness of our pre-trained language ensemble model for sexism detection and classification. Due to the limited time of this event, there are still some improvement strategies that we have not tried. For future improvements, we can start from these aspects: (1) mitigating the impact of imbalanced sample distribution through sampling or loss function adjustments, such as using over-sampling, class weight adjustment; (2) using unlabeled data to perform semi-supervised training so that the system can learn more valid information; (3) using more effective model ensemble strategies, such as XGBoost; (4) using multi-task deep pre-training (Zhang and Yang, 2021).

heading: Future Work
text: For future improvements, we can start from these aspects: (1) mitigating the impact of imbalanced sample distribution through sampling or loss function adjustments, such as using over-sampling, class weight adjustment; (2) using unlabeled data to perform semi-supervised training so that the system can learn more valid information; (3) using more effective model ensemble strategies, such as XGBoost; (4) using multi-task deep pre-training (Zhang and Yang, 2021)