Alexa at SemEval-2023 Task 10: Ensemble Modeling of DeBERTa and BERT Variations for Identifying Sexist Text
Year: 2023

Abstract:
This study presents an ensemble approach for detecting sexist text in the context of the Semeval-2023 task 10. Our approach leverages 18 models, including DeBERTa-v3-base models with different input sequence lengths, a BERT-based model trained on identifying hate speech, and three more models pre-trained on the task’s unlabeled data with varying input lengths. The results of our framework on the development set show an f1-score of 84.92% and on the testing set 84.55%, effectively demonstrating the strength of the ensemble approach in getting accurate results.

heading: 1 Introduction
text: Sexist language has been a persistent issue in various forms of communication, including online communication. The use of sexist language perpetuates gender stereotypes and reinforces the marginalization of certain groups, particularly women. Detecting and addressing instances of sexist language is crucial for promoting gender equality and reducing discrimination.
In recent years, there has been a growing interest in developing methods for automatically detecting sexist language. The Semeval-2023 task 10 provides (Kirk et al., 2023) a platform for researchers to explore and evaluate various approaches to identifying sexist text. This task involves identifying instances of sexist language in various forms of online communication, including social media posts, comments, and reviews.
Previous research has explored different methods for identifying sexist language, including rulebased approaches, machine-learning techniques, and deep-learning models. While these approaches have shown promising results, the challenge of detecting sexist language remains complex due to the subtleties and nuances of language use.
Our approach for identifying the sexist language in the Semeval-2023 task 10 subtask A involves
an ensemble of 18 different models, including DeBERTa-v3-base and pre-trained BERT-based models pre-trained on the task’s unlabeled data. The model’s predictions are combined to improve overall performance. Our method achieved an f1score of 84.92% on the development dataset and 84.55% on the testing dataset, ranking 30th out of 89 participating teams.
Section 2 of this paper introduces an overview of the related work in the field of sexist text identification. Section 3 gives an overview of the details of the dataset we used in this research. Section 4 presents key characteristics of the ensemble approach and provides a detailed description of the models used. Finally, in Section 5, we conclude the paper and discuss future directions for research in this area.

heading: 2 Background
text: Sexism against women on social media has become a growing concern, leading researchers to develop automatic systems that detect sexist text. Online competitions such as the SemEval-2022 Task 5 on "Multimedia automatic misogyny identification" and "sEXism Identification in Social neTworks (EXIST)" at IberLEF 2021 (Fersini et al., 2022; Rodríguez-Sánchez et al., 2021) have helped accelerate this research. The competition aimed to identify sexism in social media content using machine learning, with two sub-tasks: binary classification and fine-grained classification distinguishing between five different types of sexist content.
19 teams participated in the EXIST 2021 benchmark and proposed different approaches to solve the task. The team UMUTeam (García-Díaz et al., 2021) combines linguistic features with state-ofthe-art transformers to achieve an accuracy of 76.47% for binary classification (task 1) and ranks seventh. For the multi-class classification (task 2) they achieve an accuracy of 67.67% ranking third. Another team (Butt et al., 2021), presents
1644
their results on the shared task and emphasizes the importance of pre-processing techniques and data augmentation in overcoming the challenges posed by a multilingual dataset and inconsistencies of social media text. Their work achieves an F1 score of 78.02% for binary classification (task 1) ranking ninth and 49.08% for fine-grained classification (task 2) ranking fifth.
The AIT_FHSTP team (Mina et al., 2021) applies two multilingual transformer models, multilingual BERT and XLM-R, using pre-training with additional data and supervised fine-tuning with augmented data. The best model in their work is the XLM-R with a macro F1-score of 77.52% for binary classification (task 1) ranking 13 and 55.89% for the multi-class classification (task 2) ranking seventh.
(Kalra and Zubiaga, 2021) presents a deep neural network approach to detect sexist text, including BERT and DistilBERT. The best model for binary classification (task 1) uses BERT and a multi-filter CNN model achieving an accuracy of 76.2%. The same model with data augmentation achieves the best performance for multiclass classification with an F1 score of 51.9%
In earlier work, (Sharifirad et al., 2018) explore the use of knowledge graphs to improve the performance of sexist tweet classifiers. The authors propose using ConceptNet and Wikidata to improve sexist tweet classification by two methods: text augmentation and text generation. In the text generation approach, new tweets are generated by replacing words with data from ConceptNet relations, increasing the size of the training set and preserving the label. In the text augmentation approach, words in the tweets are augmented with words from ConceptNet relations and their descriptions from Wikidata, increasing the length of each tweet without changing the number of tweets in each class. The authors find that their approach significantly improved sexist tweet classification across multiple machine learning models and claim it can be applied to other small dataset problems, such as hate speech or abusive language and text classification.

heading: 3 DataSet
text: The Subtask A dataset used in this research consists of 20,000 entries sampled from Gab and Reddit. The training dataset, which constitutes 70% of the overall dataset, contains 14,000 entries, out of which 3,398 are labeled as sexist.
To enable semi-supervised training techniques, the organizers also released two unlabeled datasets consisting of 1 million entries each from Gab and Reddit. The unlabeled datasets were prepared using the same cleaning and preparation procedures as the labeled dataset.
The development data constitutes 10% of the overall dataset and consists of 2,000 entries. The test data, which constitutes 20% of the overall dataset, contains 4,000 entries.
The number of sentences in the training and development datasets is summarized in Table 1:

heading: 4 System overview
text: Our system for identifying sexist language in the Semeval-2023 task 10 involves an ensemble approach that combines multiple transformers-based models (Vaswani et al., 2017), Section 4.2 provides details on these models. Specifically, we used a combination of different variations of DeBERTav3-base (He et al., 2020) and BERT (Devlin et al., 2018) models. The transformer models are finetuned on the provided labeled data and used to classify instances in the development and test sets. The code can be accessed at Code repository.

heading: 4.1 Data Preparation
text: Initially, we attempted to clean the training data by removing punctuations, converting text to lowercase, and removing extra spaces. However, this resulted in a decrease in performance on the development set, so we decided to keep the text as is. Additionally, we also explored the impact of data augmentation on the results. We tried various techniques such as using related data from previous tasks (Rodríguez-Sánchez et al., 2021) to augment the data. However, in general, data augmentation did not lead to a significant improvement in performance. The improvement in results came from ensembling the models rather than data augmentation.

heading: 4.2 Models Used
text: In this study, we explore different models to determine the optimal approach for a particular task. We utilize publicly available models and pre-train them on the unlabeled data provided by the task.
We explore a set of models such as RoBERTa models (Liu et al., 2019), BART models (Lewis et al., 2019), DeBERTa models (He et al., 2020), DistilBERT models (Sanh et al., 2019), and pretrained BERT-based models. Based on the performance of each model on the development set, we eventually ended up choosing only deBERTav3-base and hateBERT (Caselli et al., 2020) and pre-trained them on the unlabeled data provided by the authors.
For both deBERTa-v3-base and hateBERT, we conducted a total of 18 fine-tuning experiments, each employing different combinations of learning rate and max length parameters. We explored various parameter values and ultimately selected those that yielded the highest scores, as shown in Table 2.
In our fine-tuning experiments with deBERTav3-base and hateBERT, the max length parameter proved to be an influential factor in determining the models’ performance. The max length parameter specifies the maximum number of tokens the model processes in each input sequence. Adjusting this parameter allowed us to explore the trade-off between capturing sufficient context for accurate predictions and reducing the computational resources required.
Interestingly, our experiments revealed that sometimes shorter max length values resulted in better performance. This suggests that a more concise input sequence may provide the models with more focused and relevant context, enabling them to make more accurate predictions. On the other hand, longer max length values could introduce additional noise, potentially hindering the models’ performance. We tested various max length values to identify the optimal setting for each model, as demonstrated in Table 2.
To provide a more detailed analysis, Table 2 displays the results of each model when tested individually on the development and test datasets. Several models, such as deBERTa-v3-base, achieve high f1 scores when used alone. Nevertheless, we observed a slight improvement in overall performance on the test set when combining all the predictions using a soft-ensemble approach, see section 4.3.

heading: 4.3 Ensemble Approach
text: We used a soft-ensemble method for combining the predictions of each model to produce the final prediction. The soft-ensemble method has been widely adopted in the field of natural language processing and enables the combination of probabilities, taking into account the confidence of each model’s prediction. We also experiment with the hard ensemble method, which is another commonly used ensemble method. Hard ensemble involves selecting the most frequently predicted class from each model and using that as the final prediction. This method does not take into account the confidence of each model’s prediction, therefore, we compared the performance of both soft-ensemble and hardvoting methods on the dataset to determine which method yielded the best results, see Table 3.
The approach we used involves each model producing a probability score whether it is sexist or not for each instance, and these scores are then combined using a weighted average to produce the final prediction. This method has been shown to be effective in improving the accuracy of predictions compared to using a single model (Risch and Krestel, 2020; Dang et al., 2020; Briskilal and Subalalitha, 2022).
Furthermore, we also explore the ability to eliminate some of the models randomly to increase the overall score. Our approach involves generating a random number and using it to select a corresponding number of models from a list of 18 models. We observed a slight improvement in performance when using this approach, but it takes time to see a slight improvement in the results because it is resource-intensive. Therefore, we did not report or submit the output of this method, but we will keep it for further research opportunities.

heading: 5 Results
text: The obtained results of the individual models and the ensemble approaches are presented in Table 2 and Table 3.
Our results indicate that ensembling multiple models can be an effective approach for identifying sexist text. The combination of different models provides a more robust solution, leveraging the strengths of each individual model. It is possible that adding even more models to the ensemble would lead to further improvements in performance.
During our error analysis, we found that our en-
semble approach had difficulty identifying certain types of sexist sentences. Specifically, we struggled with detecting sexist language that involves immutable gender differences and gender stereotypes, as well as incitement and encouragement of harm. These types of sentences often involve subtle language use and required a deeper understanding of the context and underlying social issues.
Additionally, we found that our approach had difficulty with detecting dehumanizing attacks and overt sexual objectification. These types of language often involved explicit references to the target’s body and required a fine-grained analysis of the language use.

heading: 6 Conclusion
text: In this research, we investigated the use of an ensemble approach for identifying sexist text. Our approach combined multiple transformer models,
including different variations of DeBERTa-v3-base, and BERT. We found that the ensemble approach outperformed the individual models, achieving 84.92% f1-score on the development set of the Semeval task and 84.55% on the testing set.
Our results demonstrate the effectiveness of combining multiple models to identify sexist text. The combination of different models provides a more robust solution, leveraging the strengths of each individual model. Our approach also showed that pre-training some of the models on the unlabeled data provided by the task can be an effective way of incorporating relevant information.
Additionally, we explored the impact of data augmentation on the results. Our findings indicate that data augmentation techniques, such as using online data on the same topic, did not result in a significant improvement in performance. The improvement in results came from ensembling the
models rather than data augmentation. Our conclusion highlights the importance of ensembling multiple models for the task of identifying sexist text. Further research is needed to determine the optimal number of models to include in an ensemble and to improve the performance of the ensemble approach.
We discovered that our system struggles with identifying certain forms of subtle and implicit sexism, which is a common challenge in detecting sexist language. In future work, we plan to explore additional feature engineering techniques and alternative methods for model selection to further improve the performance of our ensemble system.
Overall, this research provides valuable insights into the use of an ensemble approach for identifying sexist text. The findings have important implications for future work in this area and demonstrate the potential for using ensembles to tackle complex NLP tasks.

heading: Future Work
text: In future work, we plan to explore additional feature engineering techniques and alternative methods for model selection to further improve the performance of our ensemble system. Overall, this research provides valuable insights into the use of an ensemble approach for identifying sexist text. The findings have important implications for future work in this area and demonstrate the potential for using ensembles to tackle complex NLP tasks