QCon at SemEval-2023 Task 10: Data Augmentation and Model Ensembling for Detection of Online Sexism
Year: 2023

Abstract:
The web contains an abundance of usergenerated content. While this content is useful for many applications, it poses many challenges due to the presence of offensive, biased, and overall toxic language. In this work, we present a system that identifies and classifies sexist content at different levels of granularity. Using transformer-based models, we explore the value of data augmentation, use of ensemble methods, and leverage in-context learning using foundation models to tackle the task. We evaluate the different components of our system both quantitatively and qualitatively. Our best systems achieve an F1 score of 0.84 for the binary classification task – aiming to identify whether a given content is sexist or not – and 0.64 and 0.47 for the two multi-class tasks that aim to identify the coarse and fine-grained types of sexism present in the given content respectively.

heading: 1 Introduction
text: The web provides tremendous value, in part, because of how easy it is to publish new content and for others to access it. This accessibility and openness, however, comes at a cost. Today’s web with its plethora of user-generated content is rife with toxic, offensive, and obscene language. Therefore, it has become essential to moderate this content at scale. Removing the toxic content is even more critical for technologies that automatically index and surface web-content to end users, as well as generative language models that utilize this content. To address the need for content moderation, the natural language processing (NLP) and machine learning communities have put considerable effort into the problem of automatically detecting toxic language.
In this paper, we describe the system behind our team’s submission to the SemEval-2023 “Explainable Detection of Online Sexism” shared task (Kirk
∗Equal contribution.
et al., 2023). There are three tasks – Task A, B and C. Task A is a binary classification task and tasks B and C are multi-class classification tasks with 4 and 11 classes respectively.
The current dominant approach to detecting toxic language is to fine-tune sequence to sequence transformer (Vaswani et al., 2017) models (Hanu and Unitary team, 2020; Mutanga et al., 2020; Kim et al., 2022; Markov et al., 2022). Our system builds on this thread of research, and extends it by applying methods to ensemble deep learning models (Wenzel et al., 2020; Gontijo-Lopes et al., 2022; Wortsman et al., 2022). We ensemble transformers, where each member of the ensemble uses a different base architecture (i.e. BERT, RoBERTa, etc.), different training data, and different training hyperparameter configurations. In addition, we explore whether using in-context learning and weak-labels from foundation models, such as GPT-3 (Brown et al., 2020), further improve the performance.
Across Tasks A, B, and C, we find that our ensembles of fine-tuned transformers consistently reach higher macro F1 score on the test set than the best individual model that we would have chosen using macro F1 score on the development set. Consistent with the preliminary experiments in (Wortsman et al., 2022), however, the improvements we observe are relatively small. This encouraging result suggests that ensembling fine-tuned transformers has promise, but there is need for additional research into techniques for selecting members of the ensemble, and for computing the aggregated output. For in-context learning using GPT-3 (Brown et al., 2020), we find that the performance is competitive to the fine-tuned models, but it provides mixed results when used as a member of an ensemble.

heading: 2 Related Work
text: Work on automatically detecting abusive language on the web goes back to 2009, when Yin et al. (2009) proposed several features for detecting abu-
1260
sive user-generated content in “Web 2.0” applications. Yin et al. (2009) proposed two particularly interesting features: template patterns that capture common abusive phrases, and contextual features that capture similarities between the language being classified and “nearby” language (e.g. the text from a parent post on a message board). Template patterns of abusive phrases are no longer used as explicit features for abusive language classifiers, but have been used to create model “unit tests” to expose weaknesses of leading systems (Röttger et al., 2021). The issue of whether context is important for determining if a piece of text is offensive is still debated (Pavlopoulos et al., 2020).
Djuric et al. (2015) was the first to use word vector representations in an abusive language classifier, as opposed to discrete representations of words as in a bag of words approach. They used the Le and Mikolov (2014) model, but observed marginal improvements over a simpler bag of words approach (0.8 AUC from 0.79). Nobata et al. (2016) built on Djuric et al. (2015) and further experimented with word and paragraph embeddings combined with several “classical” features, such as character and token n-grams, features derived from dependency parses, and a suite of orthographic features. On the dataset used in Djuric et al. (2015), the approach in Nobata et al. (2016) further improved the AUC to 0.91. Surprisingly, however, the character n-grams alone achieved AUC of 0.90, suggesting that the word and paragraph embeddings offer only marginal improvements to the problem.
Early word embedding approaches (Djuric et al., 2015; Nobata et al., 2016) learned task specific word vectors on relatively small corpora, on the order of hundreds of thousands of texts. This may explain the underwhelming performance observed in these studies. Large “foundation models” have improved this approach. Nikolov and Radivchev (2019) was one of the first to show that fine-tuning BERT for hate speech detection can significantly improve over standard baselines like logistic regression and support vector classifiers. The Nikolov and Radivchev (2019) work showed that BERT improves performance the most for more fine-grained classifications of hate speech, meaning going beyond the basic problem of binary classification of the content. There have since been several proposed extensions of the standard transformer finetuning process. For instance, Tran et al. (2020) shows that learning task-specific vocabularies for
transformers can marginally improve performance. Although fine-tuned transformers have successfully “cracked” many text classification tasks, the problem of hate speech detection remains difficult. Progress on the problem is tricky because the judgement of what is offensive is inherently subjective. This has led to an active thread of ongoing research focused on understanding the roles that data collection, annotation guidelines, and annotator biases play on the quality of hate speech classifiers (see, e.g., (Dixon et al., 2018; Fortuna et al., 2020; Röttger et al., 2022; Garg et al., 2022).
Our system builds on ideas from the emerging literature on methods for creating ensembles of deep, “high-capacity” models (Wenzel et al., 2020; Gontijo-Lopes et al., 2022; Wortsman et al., 2022). The majority of experimental work in the space of model ensemble has been in the area of computer vision. Our paper contributes additional data to the open question of whether ensembling of deep models can successfully solve problems in the area of natural language.

heading: 3 Data Selection
text: For each classification task, our approach involves ensembling different fine-tunings of pre-trained Transformer (Vaswani et al., 2017) based models. These models vary in the selection of: base models, hyper-parameter configurations, and the data used to fine-tune each model. Moreover, they make use of weak-labels generated by prompting GPT-3 (Brown et al., 2020) using in-context learning. In this section we go into the details of the data we used for model training.
For all model trainings, we include the corpus of gold-labeled training data provided for the shared task (Kirk et al., 2023). For tasks A, B, and C, we randomly sample 10% of the gold data, and use it as a validation set to perform hyper-parameter tuning. We choose the best model based on the performance on the corresponding task’s official development set. In addition to the gold data, we leverage a subset of the unlabeled Reddit and Gab data released by the task organizers. We apply silver labeling to this data and augment our original training data with it. To further augment the training dataset, we use two existing public datasets: “EXIST” (Rodríguez-Sanchez et al., 2021) and “Call Me Sexist” (Samory, 2021). Tables 1, 2, and 3 show the statistics of the gold, silver, and public datasets utilized for Tasks A, B and C respectively.

heading: 3.1 Public Datasets
text: For Task A, we include training data samples from the publicly available datasets – (1) “Call Me Sexist” (Samory, 2021) (2) “EXIST” (RodríguezSanchez et al., 2021). We randomly sample 5,000 samples from Call Me Sexist dataset to include in models (G, I, J) and include the entire EXIST corpus for models (L,M ) in our Task A results Table 4.

heading: 3.2 Silver Labeling
text: To label the provided unlabeled data, we use a weighted sum (weights = (0.75, 0.25)) of normalized scores from two weak classifiers – (1) RoBERTa and (2) Sentence Transformer, where a higher score indicates a higher likelihood of samples belonging to the given class. We select a fixed number of highest scored samples for each class according to the task to generate additional silverlabelled training dataset. Additionally, we create an offensive term list for Task A to filter out some samples from unlabelled corpus to get better silverlabelled data.
For Task A, we use both the unlabeled Reddit and Gab datasets, whereas for Tasks B and C we only use samples from the unlabeled Reddit dataset; we experimented with Gab dataset for both tasks but observed lower performance when using it. For Task A, we add 5,000 positive and 5,000 negative samples each from both Reddit and Gab datasets, increasing the training data size by 20,000 samples. For Task C, we added 500 samples for each of the 11 classes only from the Reddit dataset, increasing the training data size by 5,500 samples.
RoBERTa We fine-tune a RoBERTa (large) (Liu et al., 2019) model using only the gold training data provided for the task A and C.
Sentence Transformer We use a sentence transformer model (Reimers and Gurevych, 2020) that maps sentences to a 384-dimensional dense vector space, to generate a similarity score between each labeled training sample and the unlabeled sample. Using these similarity scores, for each unlabeled sample we generate an average class score for all classes for a given task. Based on the task, we select the highest and lowest scored samples to add to the training data.
Offensive Term List For Task A, we utilize a manually-created list of offensive terms. We collect counts of how often terms from this manuallycreated list occur in both the sexist and non-sexist samples in the Task A gold training data, and take the top terms when ranked by their ratio of sexist to non-sexist mentions. Additionally, only terms that appeared more than five times were included for robustness. For gendered slurs, both singular and plural variants of the same nouns were included regardless of the frequency in the gold training data. This offensive term-list was used to select unlabeled training data samples from Reddit and Gab to include as silver labeled training data for Task A. The 5,000 silver labeled samples we include when training Model L in our results Table 4 includes only samples which contained at least one term from this term list.

heading: 4 Models
text: As mentioned earlier, we use an ensemble of Transformer-based models (Vaswani et al., 2017) for the three tasks. To this end, we fine-tuned: BERT (base-uncased) (Devlin et al., 2019), RoBERTa (base and large) (Liu et al., 2019), XLM-RoBERTa (large) (Conneau et al., 2020), ELECTRA (base) (Clark et al., 2020), BERTweet (large) (Nguyen et al., 2020), MiniLM (L12H384) (Wang et al., 2020), ALBERT (Lan et al., 2020).
Additionally, we utilized in-context learning to prompt GPT-3 (Brown et al., 2020) to generate predictions for the development and test set examples for Task A. We follow a similar approach to (Chiu
and Alexander, 2021).1 The prompt format we used was the word "SENTENCE" followed by a randomly selected training set example, then the word "LABEL" followed by either the word "non-sexist" or "sexist", to indicate the label of the preceding example. We included K randomly sampled positive training set examples and K randomly sampled negative training examples for each prompt.2 At the end of the prompt, we included a development or test set example followed by "LABEL" with no following text, prompting for an example completion with either the word "non-sexist" or "sexist" from the model. We used these prompt completions as predicted labels, which are included in our results Tables 4 and 5.

heading: 4.1 Fine-tuning
text: In our approach, we fine-tune models for two ensembles: one ensemble of models for Task A and another ensemble for Task C. For Task B, the finegrained labels from Task C to the coarse-grained labels of Task B. For example, a “1.1 Threats of harm” Task C label would map to the “1. Threats, plans to harm and incitement” Task B label.
For Task A we use a weighted binary cross entropy (BCE) loss function. The loss is equally weighted between the positive and the negative samples present in the training data for each finetuned model.
For multi-class tasks, B and C, we take inspiration from related work on hierarchical modeling tasks using a shared loss function (Wu et al., 2017), and treat both tasks as a single model with two objectives. We capture the hierarchical structure of the coarse and fine-grained labels for Task A and B in this single model’s loss function. We
1The exact model version we used was "text-davinci-003". 2We experimented with K = 5, 10 and 20 and achieved the
best results using 5 examples.
create a composite loss computed of the sum of (1) the standard cross entropy (CE) loss on all 11 Task C outputs, and (2) the cross entropy of the Task C outputs projected onto the 4 outputs for Task B, as shown in equation 1, where YB|C is the true class and ŶB|C is the predicted class. For example, we compute the model’s prediction for “1. Threats, plans to harm and incitement” as the sum of Pr(1.1 Threats of harm) and Pr(1.2 Incitement and encouragement of harm). The key intuition is that the model’s output should be consistent at both the coarse and the fine-grained levels, and that enforcing this consistency will improve the model’s performance in Task C. This loss function introduces an additional hyper-parameter β that weights the Task C loss against the Task B loss in the composite calculation.3
LC = β · CE(YB , ŶB) + CE(YC , ŶC) (1)

heading: 4.2 Ensembling
text: As shown in (Wenzel et al., 2020), ensembling different models with different hyperparameter settings outperforms using a single model. Therefore, we explore the performance of different ensembles of models by measuring the performance of all model combinations in the power set P(S), for the set of models S for each of the tasks. For each task, the predictions for a given ensemble of models is based on the majority vote for each sample. Each ensemble, as well as the individual models, were evaluated on the development set for each task, and the best performing setup was used to label the held out test sets.
3We determined β empirically based on the performance on the development set. Specific values can be found in Appendix A.

heading: 5 Experiments
text: As mentioned earlier, we explore fine-tuning individual models for each task. Tables 4, 6, and 8 report the performance of each individual model, as well as the best ensemble for each of Tasks A, B, and C respectively on the development sets. Tables 5, 7, and 9 report the performance of the best (i.e. submitted) ensemble on the held-out test sets of the three tasks. We report the macro F1 average of all classes for each task – since it is the official metric for all tasks – as well as the per-class F1 score. As a baseline for comparison with our individual models, we include a row with results from always predicting the most frequent class in the training data for a given task – “Majority Baseline” and as a baseline for comparison with our best ensembles, we include results for a simple ensemble of all models trained for each task “ALL”.
Task A: Binary Sexism Detection For Task A, the best performing ensemble includes five finetunings of RoBERTa (Large), and predictions obtained from prompting GPT-3, as described in Section 4.
• Models E and F were trained on the gold training data set. • Model H was trained on both the gold training data set as well as 5,000 silver labeled Reddit and Gab samples. • Model I was trained on the gold training data set, 5,000 silver labeled Reddit and Gab samples (which here were filtered to include only
samples with terms from our curated termlist), and 5,000 samples from the "Call Me Sexist" corpus. • Model L was trained on the gold training data set and the entire EXIST corpus. • Model O are predictions from prompting GPT3.
Tasks B and C For both Task B and Task C we train one set of models A-H on the Task C labels using the loss described in Section 4.1, with the predicted labels for Task C mapped to their corresponding labels for Task B. All the models are trained on gold training data set and silver labeled Reddit samples.
For Task B the best performing ensemble includes models A, B, C, D, E which are fine-tunings of BERT, ELECTRA, RoBERTa (Base), RoBERTa (Large) and, RoBERTa (Large), respectively. For Task C the best performing ensemble includes models B, D, E, F which are fine-tunings of ELECTRA, RoBERTa (Large), RoBERTa (Large), and, XLMRoBERTa respectively. We also include scores from GPT3 (Prompting) as Model I in Table 6. See Appendix A.

heading: 6 Discussion
text: Task A Test set results for Task A show that our submission of the best-performing ensemble on the development set, which has an F1 score of 0.84, is slightly outperformed by an ensemble of all mod-
els we trained for Task A, which has an F1 score of 0.85. Here, the best ensemble on the development set degrades in performance from 0.87 F1 on the development set to 0.84 F1 on the test set, suggesting this ensemble may have overfit to the development set. We also see that the simple average of predictions from all models we trained for Task A gives us a result that does not degrade from the development set to the test set.
When considering our individual Task A models, we see little to no drop in performance (0.0 to 0.2 macro F1 difference) and models B,C, F even improve on this test set. GPT-3 prompting also shows no difference in performance on development and test, although performance is lower than for most of our individually trained Task A models. When comparing with the individual models we trained, we see that our best ensemble on the development set out-performs our best individual model (L) for this task. Submitting the best ensemble on the development set rather than this best individual model leads to an improvement in both precision (from 0.75 to 0.77) and macro average F1 (from 0.83 to 0.84).
Tasks B and C Results for Tasks B and C show a stronger drop in performance when comparing the development set results to the test set results. This trend applies to not just our best performing ensemble on development, but to all individual models we trained. Despite this fact, our submitted ensemble of models which performed best on the development set also achieves the best performance on the test set, compared to both a simple average of all models we trained and compared to individual model performance.
When comparing our performance between Tasks B and C, we see that our approach of training models for Task C using a combined loss for Task B and C does allow us to achieve good results on Task B as well, when mapping the Task C fine-grained predicted labels to Task B coarse-grained labels. For these tasks, we also explored a wider range of models compared to Task A, which added to the diversity of model predictions we could select from for our ensemble.

heading: 6.1 Error Analysis
text: Since the Task C categories are the most finegrained, we investigated for which Task C categories our model under-performed by comparing the ensemble predictions on the development set with the task C development set gold labels. A confusion matrix from the development set is reported in Figure 1. Table 10 show examples of where the data was mislabeled for each. The majority of the development set errors stemmed from the failure of the model to distinguish between:
• Labels 2.2 (Aggressive and emotive attacks) and 3.1 (Casual use of gendered slurs, profanities and insults) • Labels 2.1 (Descriptive attacks) and 2.2 (Aggressive and emotive attacks) • Labels 2.1 (Descriptive attacks) and 3.2 (Immutable gender differences and gender stereotypes)

heading: 2.2 (Aggressive and emotive attacks) vs. 3.1 (Ca-
text: sual use of gendered slurs, profanities and insults) This was the most frequent error category
on the development set for our best ensemble of models. Our model often confused the two labels due to the subtlety of casual versus aggressive use of gendered slurs. Documents with both labels may contain such slurs, but the implications of those slurs can be quite different. For example, in Table 10, the third example shows the speaker’s intention and feeling by directly calling someone a slur. This is different from examples labeled as 3.1 (Casual use of gendered slurs, profanities and insults), where the writers are not intentionally attacking the women but rather casually expressing sexism through their use of slurs. For example, in the fourth example in Table 10 the speaker is not directly attacking women but rather grouping all women into loyal and disloyal, an unfair artificial categorization, which is emphasized with the speaker’s use of a gendered slur. We again could improve our models by including additional silver labeled samples that contain these gendered slurs, and manually placing them into two groups of examples with casual mentions of gendered slurs or direct, intentional use of slurs.
2.1 (Descriptive attacks) vs. 2.2 (Aggressive and emotive attacks) This was the second most frequent error category on the development set for our best ensemble of models. As stated on the shared task website label schema, 2.1 (Descriptive attacks) is for descriptive attacks that disparage women through generalizations, which differs from 2.2 (Aggressive and emotive attacks) where the speaker expresses negative sentiment towards women in the document. Many of these 2.2 (Aggressive and emotive attacks) documents contain the first person pronoun “I”, expressing how the speaker is feeling. A future direction to improve the ability of our model to distinguish a sentence that displays direct negative emotions from the speaker, versus a generalized negative emotion, is to include additional silver labeled training data that contain both “I”-statements, and such attacks on women.
2.1 (Descriptive attacks) vs 3.2 (Immutable gender differences and gender stereotypes) This was the third most frequent error category on the development set for our best ensemble of models. This case is unique in the sense that the majority of cases were documents labeled 3.2 being mislabeled as 2.1 (Descriptive attacks), rather than the opposite. For example, in Table 10, the fifth example
has the phrase “that’s a woman’s line of thinking”. Here, the intention of the speaker is not directly attacking one or more women, but rather expressing an attitude about how women think, which is a gendered stereotype. This statement may not be obviously sexist to all readers. Much of the intention of the sentences in the 2.1 (Descriptive attacks) labeled data (as in our second example in Table 10) is to directly attack women through sexist generalizations. We believe our model was not able to distinguish these two fine grained cases because it could not distinguish if a statement was directly or indirectly sexist.
We see errors for category 3.2 (Immutable gender differences and gender stereotypes) as some of the most challenging for our models to overcome, due to how stereotypes around gender differences may be stated indirectly: these documents may not include any explicit profanity or gendered slurs, or other obvious hallmarks of sexism in the text. Additionally, examples of class 3.2 (Immutable gender differences and gender stereotypes) are less frequent in the gold training and development data. In order to make improvements to our model, we believe collecting additional annotated data using the annotation guidelines written by the shared task organizers is the right direction.

heading: 7 Conclusion
text: In this paper, we described our team’s submission to SemEval-2023’s “Explainable Detection of Online Sexism” shared task, that aims to identify whether or not a given content is sexist (Task A), as well as the broad (Task B) and fine-grained (Task C) class of sexist content present in the given text (e.g., prejudice, animosity, derogation, etc.). We find that utilizing a combination of data augmentation – both
through publicly available datasets as well as silver labeling – and ensembling different transformermodels gives us the best performance. In addition, we also explored using in-context learning using GPT-3, and found that it yields mixed results, helping us achieve better performance for Task A but not for Tasks B or C.

heading: A Appendix
text: A.1 Training Hyper-parameters We chose the Adam optimizer (Kingma and Ba, 2015) with (β1, β2) = (0.9, 0.98) and an initial learning rate of 5e-5, gradient accumulation of 4 and different batch sizes for different base models. For regularization, we use weight decay of 1e-2. To train the model for Task A, we use binary cross-entropy criterion weighted with respect to the proportion of positive and negative samples in the training data set. For Task B and C we use the loss described in Section 4.1 Equation 1.
We train each model for 20 epochs with early stopping and a patience of 3 epochs without improvement on validation set loss.
For Task B and C we introduced the hyperparameter β when combining the loss functions for both
tasks (as explained in Section 4.1). All models we trained for Tasks B and C use β = 0.25 except for model (E) which uses β = 0.1.

heading: Future Work
text: A future direction to improve the ability of our model to distinguish a sentence that displays direct negative emotions from the speaker, versus a generalized negative emotion, is to include additional silver labeled training data that contain both “I”-statements, and such attacks on women. 2. 1 (Descriptive attacks) vs 3. 2 (Immutable gender differences and gender stereotypes) This was the third most frequent error category on the development set for our best ensemble of models. This case is unique in the sense that the majority of cases were documents labeled 3. 2 being mislabeled as 2. 1 (Descriptive attacks), rather than the opposite. For example, in Table 10, the fifth example
has the phrase “that’s a woman’s line of thinking”. Here, the intention of the speaker is not directly attacking one or more women, but rather expressing an attitude about how women think, which is a gendered stereotype. This statement may not be obviously sexist to all readers. Much of the intention of the sentences in the 2. 1 (Descriptive attacks) labeled data (as in our second example in Table 10) is to directly attack women through sexist generalizations. We believe our model was not able to distinguish these two fine grained cases because it could not distinguish if a statement was directly or indirectly sexist. We see errors for category 3. 2 (Immutable gender differences and gender stereotypes) as some of the most challenging for our models to overcome, due to how stereotypes around gender differences may be stated indirectly: these documents may not include any explicit profanity or gendered slurs, or other obvious hallmarks of sexism in the text. Additionally, examples of class 3. 2 (Immutable gender differences and gender stereotypes) are less frequent in the gold training and development data. In order to make improvements to our model, we believe collecting additional annotated data using the annotation guidelines written by the shared task organizers is the right direction