DUTIR at SemEval-2023 Task 10: Semi-supervised Learning for Sexism Detection in English
Year: 2023

Abstract:
Sexism is an injustice afflicting women and has become a common form of oppression in social media. In recent years, the automatic detection of sexist instances has been utilized to combat this oppression. The Subtask A of SemEval2023 Task 10, Explainable Detection of Online Sexism, aims to detect whether an Englishlanguage post is sexist. In this paper, we describe our system for the competition. The structure of the classification model is based on RoBERTa, and we further pre-train it on the domain corpus. For fine-tuning, we adopt Unsupervised Data Augmentation (UDA), a semisupervised learning approach, to improve the robustness of the system. Specifically, we employ Easy Data Augmentation (EDA) method as the noising operation for consistency training. We train multiple models based on different hyperparameter settings and adopt the majority voting method to predict the labels of test entries. Our proposed system achieves a Macro-F1 score of 0.8352 and a ranking of 41/84 on the leaderboard of Subtask A.

heading: 1 Introduction
text: With the evolution of the internet, people are free to express their opinions on social media. This may lead to the mass dissemination of hateful or abusive messages (Chiril et al., 2020), such as sexist expressions that people use intentionally or unintentionally. Sexism is a complex phenomenon, broadly defined as "prejudice, stereotyping, or discrimination, typically against women, on the basis of sex."1 Sexism takes many forms, including blatant, covert, and subtle sexism (Swim et al., 2004), causing sexism detection to be a challenge for the filtering mechanism of platforms.
The SemEval-2023 Task 10 (Kirk et al., 2023) is an explainable detection task of online sexism, and we participate in Subtask A, the binary classification task, to detect whether an English-language
1Oxford English Dictionary
post is sexist or not. The competition organizers released datasets collected from Gab and Reddit, containing 20,000 labeled entries and 2 million unlabeled entries.
For the sexism detection task, we build a system based on the pre-trained language model RoBERTa. Firstly, we further pre-train RoBERTa on the unlabeled corpus to obtain our domain-specific encoder. Furthermore, we reuse the abundant unlabelled data by Unsupervised Data Augmentation (UDA) (Xie et al., 2020) approach for consistency training, which is a common practice in semi-supervised learning. As for the noising operation, we apply the Easy Data Augmentation (EDA) (Wei and Zou, 2019) method to transform the unlabeled examples into their noised versions. The consistency training can propagate label information from labeled to unlabeled examples to enhance the generalization of the model. To further improve the robustness of our system, we train models based on different hyperparameter settings and perform majority voting on the predicted labels to obtain the final prediction results.
The structure of this paper is as follows: We first take a brief overview of related work in section 2, and then describe our proposed system in section 3. We introduce the details of our experiments in section 4, including the experimental settings, results, and discussions. Finally, a brief conclusion is presented in section 5.

heading: 2 Related work
text: As more and more people suffer or witness sexism on social media, automatically detecting sexist posts can help combat this phenomenon(Abburi et al., 2020). Many studies have been proposed to identify gender-based violence in texts. For misogyny, Anzovino et al. (2018) build a corpus of misogynous tweets labeled from different perspectives and conduct exploratory investigations. Aiming at multi-label fine-grained sexism classification,
892
Abburi et al. (2020) introduce a semi-supervised multi-task neural approach. Inspired by speech acts theory and discourse analysis studies, Chiril et al. (2020) propose a new characterization of sexist content. Besides, some works (Badjatiya et al., 2017; Davidson et al., 2017; Espinosa Anke et al., 2019) regard sexism as a category of hate and detect it using hate speech classification methods.

heading: 3 System Description
text: Given a sentence x = {x1, · · · , xn} consisting of n tokens, the sentence is embedded as a vector by an encoder. We use a sentence-level embedding z to represent the entire sentence and feed it into a linear binary classifier to get the predicted label y. The goal of our system can be formulated as training a model pθ(y|x), where the network parameters are denoted as θ, to predict the ground-truth label y∗ for a given x.
As shown in Figure 1, the general implementation of our system can be described as 3 phases: pretraining, training, and prediction. In pre-training phase, we pre-train the language model on a relevant corpus so that the language model can learn more domain knowledge. Then we adopt the pre-
trained language model as our domain-specific encoder. In training phase, we attach a linear classifier to the encoder and fine-tune the model through semi-supervised training. In prediction phase, we use the majority voting method to predict the final classification labels.

heading: 3.1 Pre-Training
text: As mentioned, a large-scale unlabeled corpus is available to participants, which we denote as D. Since the labeled entries are sampled from this corpus, we consider D to have the same distribution as the labeled dataset Dl. Therefore, the unlabeled data can be used for self-supervised training of the pre-trained model, so that the generated embedding vectors will better match the domain-specific features. In our system, we adopt RoBERTa (Liu et al., 2019) as the pre-trained model, and pre-train it by Masked Language Model (MLM), which enables the model to learn the bidirectional contextual information of the text (Devlin et al., 2019). We then obtain a domain-specific language model denoted as RoBERTaA and serve it as the encoder for our system. Instead of using the entire D, due to hardware and time constraints, we take a tenth of the entries from it and denote the subset as Du.
We merge Du with all the labeled entries as our pre-trained corpus.

heading: 3.2 Training
text: The sentence x is embedded by the domain-specific encoder as s ∈ Rn×d, where d denotes the output hidden dimension size of the encoder. We adopt the last hidden state of [CLS] token as the sentencelevel embedding z and feed it into the linear binary classifier to get the output distribution pθ(y|x).
In this phase, we fine-tune the model by minimizing the final loss L and optimizing the parameters θ based on the hyper-parameter setting β for getting the trained model M . Supervised Learning. The supervised learning is conducted given the labeled data (x, y∗) ∈ Dl with the aim of minimizing the divergence metric between ground-truth label y∗ and the predicted label y ∼ pθ(y|x). The divergence metric is calculated by Cross-Entropy and denoted as CE(y∗, y). With supervised learning, θ is optimized such that pθ(y|x) approaches y∗ for given (x, y∗). Semi-supervised Learning. To enforce the robustness and smoothness of the model, we adopt the Unsupervised Data Augmentation (UDA) approach (Xie et al., 2020) to perform consistency training. To be concrete, given an unlabeled data x ∈ Du and its augmented version x̂, we input the x and x̂ into the model and then obtain their output distributions pθ(y|x) and pθ(y|x̂), respectively. With the minimization of the divergence metric between these two distributions, the model can constrain predictions to be invariant to input noise. We observe that consistency training is performed unsupervised so that the information of a large amount of unlabeled data can be learned. We use the Kullback-Leibler (KL) divergence to measure the above distribution divergence and define it as follows:
KL(x, x̂) = DKL(pθ(y|x)||pθ(y|x̂)) (1)
Therefore, we set the final loss L as follows:
L = N∑ CE(y∗, x1) + αN∑ KL(x2, x̂2) (2)
where (x1, y∗) ∈ Dl, x2 ∈ Du, x̂2 is the augmented version of x2, N is the batch size of supervised data and α is a hyperparameter to control the ratio of unsupervised data in a batch.
In addition, we employ additional training strategies mentioned in Xie et al. (2020), including
confidence-based masking and sharpening predictions, to solve some common training problems. Data Augmentation. We employ Easy Data Augmentation (EDA), a simple but powerful data augmentation method, transforming unlabeled data x into x̂. Details of the EDA augmentation operations are shown in Table 1. Here we list an example of a text and its augmented version:2

heading: 3.3 Prediction
text: In this phase, we employ the models fine-tuned during training phase to predict whether the entry to be identified is sexist by majority voting, a simple ensemble method. Specifically, we optimize the model parameters based on the hyperparameter setting βi, where i ∈ {0, · · · , 5}, and get the corresponding fine-tuned model Mi. Given the entry x, we feed x into these fine-tuned models to obtain the classification result, i.e., sexist or not sexist, and count the number of sexist in these predicted labels. If the number exceeds 2, we mark x as sexist, and vice versa as not sexist. The details of hyperparameter setting are explained in 4.

heading: 4 Experiments

heading: 4.1 Experimental Settings
text: Datasets. The statistics of the labeled datasets are shown in Table 2. The unlabeled dataset Du consists of 200,000 entries sampled from the unlabeled corpus with 2 million, and we adopt it for pre-training and semi-supervised learning. For all the data, we do not perform any other preprocessing because we consider that the competition organizers have already completed the initial clean-up and
2The texts contain offensive speech, and we oppose any use of this kind of speech act.
preparation of the data, and additional processing may lead to loss of information. Evaluation and Model Saving. We evaluate the official metric, macro-F1 score of the model on the development set once every 100 training iteration steps and save the model with the highest current score. The iteration is terminated when macro-F1 stops growing in 20 consecutive evaluations. Hyperparameters. The fixed hyperparameter settings of our system are described below. We adopt the RoBERTa-base as the pre-trained language model and AdamW as the optimizer, both provided by huggingface3. The pre-training for RoBERTabase is conducted for 700 epochs with a batch size of 16. We use BCEWithLogitsLoss and KLDivLoss in Pytorch to compute the supervised and unsupervised losses. Since the labels of Dl are imbalanced, the parameter pos_weight for BCEWithLogitsLoss is set to 4. For confidence-based masking and sharpening predictions, we set the threshold and Softmax temperature to 0.45 and 0.85, respectively. In addition to the fixed hyperparameters, for majority voting, we set the learning rate from 1e-5 to 5e-5 in increments of 1e-5, the batch size of supervised data to 8, 16, 32, or 64, and the ratio α of unsupervised data to from 1 to 6 in increments of 1.
3https://huggingface.co/

heading: 4.2 Results and Analysis
text: Table 3 lists the experimental results of our system on the development set and test set. All models are trained using the same random seed, except for the model with the Vote on the test set, which integrates several models trained based on different hyperparameter settings. We can observe from the results that:
• Our RoBERTaA outperforms RoBERTa-base in both the development and test sets, proving the benefit of domain-specific pre-training. As stated earlier, pre-training the language model on the specific distribution is considered to make the output hidden states better match the domain features.
• The UDA approach improves the performance of the model, especially in the Development set, showing the effectiveness of data augmentation and semi-supervised learning for this task. This motivates us to continue exploring how to make better use of unlabeled data information to enhance the generalization of models in future research.
• The majority voting method has slight improvement on the models on the test set, probably because of the considerable homogeneity among the base models involved in voting (only the hyperparameter settings differ, not the model structure). We should examine models with various structures to make the majority voting more useful.

heading: 5 Conclusion
text: This paper describes our approach in SemEval2023 Task 10 Subtask A to detect sexism in English-language social media posts. Our system employs self-supervised and semi-supervised learning to pre-train and fine-tune the pre-trained language model RoBERTa-base respectively. We demonstrate the effectiveness of our system. There are still considerable areas for improvement in our system, and how to better utilize the unlabeled data is an essential issue for us to explore in the future.

heading: Future Work
text: There are still considerable areas for improvement in our system, and how to better utilize the unlabeled data is an essential issue for us to explore in the future