Introduction
Automatic detection of abusive language is an important task since such language in online space can lead to personal trauma, cyber-bullying, hate crime, and discrimination. As more and more people freely express their opinions in social media, the amount of textual contents produced every day grows almost exponentially, rendering it difficult to effectively moderate user content. For this reason, using machine learning and natural language processing (NLP) systems to automatically detect abusive language is useful for many websites or social media services.
Although many works already tackled on training machine learning models to automatically detect abusive language, recent works have raised concerns about the robustness of those systems. BIBREF0 have shown how to easily cause false predictions with adversarial examples in Google's API, and BIBREF1 show that classifiers can have unfair biases toward certain groups of people.
We focus on the fact that the representations of abusive language learned in only supervised learning setting may not be able to generalize well enough for practical use since they tend to overfit to certain words that are neutral but occur frequently in the training samples. To such classifiers, sentences like “You are a good woman” are considered “sexist” probably because of the word “woman.”
This phenomenon, called false positive bias, has been reported by BIBREF1 . They further defined this model bias as unintended, “a model contains unintended bias if it performs better for comments containing some particular identity terms than for comments containing others.”
Such model bias is important but often unmeasurable in the usual experiment settings since the validation/test sets we use for evaluation are already biased. For this reason, we tackle the issue of measuring and mitigating unintended bias. Without achieving certain level of generalization ability, abusive language detection models may not be suitable for real-life situations.
In this work, we address model biases specific to gender identities (gender bias) existing in abusive language datasets by measuring them with a generated unbiased test set and propose three reduction methods: (1) debiased word embedding, (2) gender swap data augmentation, (3) fine-tuning with a larger corpus. Moreover, we compare the effects of different pre-trained word embeddings and model architectures on gender bias.
Related Work
So far, many efforts were put into defining and constructing abusive language datasets from different sources and labeling them through crowd-sourcing or user moderation BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 . Many deep learning approaches have been explored to train a classifier with those datasets to develop an automatic abusive language detection system BIBREF6 , BIBREF7 , BIBREF8 . However, these works do not explicitly address any model bias in their models.
Addressing biases in NLP models/systems have recently started to gain more interest in the research community, not only because fairness in AI is important but also because bias correction can improve the robustness of the models. BIBREF9 is one of the first works to point out the gender stereotypes inside word2vec BIBREF10 and propose an algorithm to correct them. BIBREF11 also propose a method called Word Embedding Association Test (WEAT) to measure model bias inside word embeddings and finds that many of those pretrained embeddings contain problematic bias toward gender or race. BIBREF1 is one of the first works that point out existing “unintended” bias in abusive language detection models. BIBREF12 compare 219 sentiment analysis systems participating in SemEval competition with their proposed dataset, which can be used for evaluating racial and gender bias of those systems. BIBREF13 shows the effectiveness of measuring and correcting gender biases in co-reference resolution tasks. We later show how we extend a few of these works into ours.
Sexist Tweets (st)
This dataset consists of tweets with sexist tweets collected from Twitter by searching for tweets that contain common terms pertaining to sexism such as “feminazi.” The tweets were then annotated by experts based on criteria founded in critical race theory. The original dataset also contained a relatively small number of “racist” label tweets, but we only retain “sexist” samples to focus on gender biases. BIBREF2 , BIBREF3 , the creators of the dataset, describe “sexist” and “racist” languages as specific subsets of abusive language.
Abusive Tweets (abt)
Recently, BIBREF4 has published a large scale crowdsourced abusive tweet dataset with 60K tweets. Their work incrementally and iteratively investigated methods such as boosted sampling and exploratory rounds, to effectively annotate tweets through crowdsourcing. Through such systematic processes, they identify the most relevant label set in identifying abusive behaviors in Twitter as INLINEFORM0 resulting in 11% as 'Abusive,' 7.5% as 'Hateful', 22.5% as 'Spam', and 59% as 'None'. We transform this dataset for a binary classification problem by concatenating 'None'/'Spam' together, and 'Abusive'/'Hateful' together.
Methodology
Gender bias cannot be measured when evaluated on the original dataset as the test sets will follow the same biased distribution, so normal evaluation set will not suffice. Therefore, we generate a separate unbiased test set for each gender, male and female, using the identity term template method proposed in BIBREF1 .
The intuition of this template method is that given a pair of sentences with only the identity terms different (ex. “He is happy” & “She is happy”), the model should be able to generalize well and output same prediction for abusive language. This kind of evaluation has also been performed in SemEval 2018: Task 1 Affect In Tweets BIBREF12 to measure the gender and race bias among the competing systems for sentiment/emotion analysis.
Using the released code of BIBREF1 , we generated 1,152 samples (576 pairs) by filling the templates with common gender identity pairs (ex. male/female, man/woman, etc.). We created templates (Table TABREF6 ) that contained both neutral and offensive nouns and adjectives inside the vocabulary (See Table TABREF7 ) to retain balance in neutral and abusive samples.
For the evaluation metric, we use 1) AUC scores on the original test set (Orig. AUC), 2) AUC scores on the unbiased generated test set (Gen. AUC), and 3) the false positive/negative equality differences proposed in BIBREF1 which aggregates the difference between the overall false positive/negative rate and gender-specific false positive/negative rate. False Positive Equality Difference (FPED) and False Negative Equality Difference (FNED) are defined as below, where INLINEFORM0 . INLINEFORM1
Since the classifiers output probabilities, equal error rate thresholds are used for prediction decision.
While the two AUC scores show the performances of the models in terms of accuracy, the equality difference scores show them in terms of fairness, which we believe is another dimension for evaluating the model's generalization ability.
Debiased Word Embeddings (DE) BIBREF9 proposed an algorithm to correct word embeddings by removing gender stereotypical information. All the other experiments used pretrained word2vec to initialized the embedding layer but we substitute the pretrained word2vec with their published embeddings to verify their effectiveness in our task.
Gender Swap (GS) We augment the training data by identifying male entities and swapping them with equivalent female entities and vice-versa. This simple method removes correlation between gender and classification decision and has proven to be effective for correcting gender biases in co-reference resolution task BIBREF13 .
Bias fine-tuning (FT) We propose a method to use transfer learning from a less biased corpus to reduce the bias. A model is initially trained with a larger, less-biased source corpus with a same or similar task, and fine-tuned with a target corpus with a larger bias. This method is inspired by the fact that model bias mainly rises from the imbalance of labels and the limited size of data samples. Training the model with a larger and less biased dataset may regularize and prevent the model from over-fitting to the small, biased dataset.
Experimental Setup
We first measure gender biases in st and abt datasets. We explore three neural models used in previous works on abusive language classification: Convolutional Neural Network (CNN) BIBREF7 , Gated Recurrent Unit (GRU) BIBREF14 , and Bidirectional GRU with self-attention ( INLINEFORM0 -GRU) BIBREF8 , but with a simpler mechanism used in BIBREF15 . Hyperparameters are found using the validation set by finding the best performing ones in terms of original AUC scores. These are the used hyperparameters:
CNN: Convolution layers with 3 filters with the size of [3,4,5], feature map size=100, Embedding Size=300, Max-pooling, Dropout=0.5
GRU: hidden dimension=512, Maximum Sequence Length=100, Embedding Size=300, Dropout=0.3
INLINEFORM0 -GRU: hidden dimension=256 (bidirectional, so 512 in total), Maximum Sequence Length=100, Attention Size=512, Embedding Size=300, Dropout=0.3
We also compare different pre-trained embeddings, word2vec BIBREF10 trained on Google News corpus, FastText BIBREF16 ) trained on Wikipedia corpus, and randomly initialized embeddings (random) to analyze their effects on the biases. Experiments were run 10 times and averaged.
Debiased word2vec BIBREF9 is compared with the original word2vec BIBREF10 for evaluation. For gender swapping data augmentation, we use pairs identified through crowd-sourcing by BIBREF13 .
After identifying the degree of gender bias of each dataset, we select a source with less bias and a target with more bias. Vocabulary is extracted from training split of both sets. The model is first trained by the source dataset. We then remove final softmax layer and attach a new one initialized for training the target. The target is trained with a slower learning rate. Early stopping is decided by the valid set of the respective dataset.
Based on this criterion and results from Section SECREF13 , we choose the abt dataset as source and st dataset as target for bias fine-tuning experiments.
Results & Discussions
Tables TABREF12 and TABREF14 show the bias measurement experiment results for st and abt, respectively. As expected, pre-trained embeddings improved task performance. The score on the unbiased generated test set (Gen. ROC) also improved since word embeddings can provide prior knowledge of words.
However, the equality difference scores tended to be larger when pre-trained embeddings were used, especially in the st dataset. This confirms the result of BIBREF9 . In all experiments, direction of the gender bias was towards female identity words. We can infer that this is due to the more frequent appearances of female identities in “sexist” tweets and lack of negative samples, similar to the reports of BIBREF1 . This is problematic since not many NLP datasets are large enough to reflect the true data distribution, more prominent in tasks like abusive language where data collection and annotation are difficult.
On the other hand, abt dataset showed significantly better results on the two equality difference scores, of at most 0.04. Performance in the generated test set was better because the models successfully classify abusive samples regardless of the gender identity terms used. Hence, we can assume that abt dataset is less gender-biased than the st dataset, presumably due to its larger size, balance in classes, and systematic collection method.
Interestingly, the architecture of the models also influenced the biases. Models that “attend” to certain words, such as CNN's max-pooling or INLINEFORM0 -GRU's self-attention, tended to result in higher false positive equality difference scores in st dataset. These models show effectiveness in catching not only the discriminative features for classification, but also the “unintended” ones causing the model biases.
Reducing Gender Biases
We experiment and discuss various methods to reduce gender biases identified in Section SECREF13 .
Results & Discussion
Table TABREF16 shows the results of experiments using the three methods proposed. The first rows are the baselines without any method applied. We can see from the second rows of each section that debiased word embeddings alone do not effectively correct the bias of the whole system that well, while gender swapping significantly reduced both the equality difference scores. Meanwhile, fine-tuning bias with a larger, less biased source dataset helped to decrease the equality difference scores and greatly improve the AUC scores from the generated unbiased test set. The latter improvement shows that the model significantly reduced errors on the unbiased set in general.
To our surprise, the most effective method was applying both debiased embedding and gender swap to GRU, which reduced the equality differences by 98% & 89% while losing only 1.5% of the original performance. We assume that this may be related to the influence of “attending” model architectures on biases as discussed in Section SECREF13 . On the other hand, using the three methods together improved both generated unbiased set performance and equality differences, but had the largest decrease in the original performance.
All methods involved some performance loss when gender biases were reduced. Especially, fine-tuning had the largest decrease in original test set performance. This could be attributed to the difference in the source and target tasks (abusive & sexist). However, the decrease was marginal (less than 4%), while the drop in bias was significant. We assume the performance loss happens because mitigation methods modify the data or the model in a way that sometimes deters the models from discriminating important “unbiased” features.
Conclusion & Future Work
We discussed model biases, especially toward gender identity terms, in abusive language detection. We found out that pre-trained word embeddings, model architecture, and different datasets all can have influence. Also, we found our proposed methods can reduce gender biases up to 90-98%, improving the robustness of the models.
As shown in Section SECREF13 , some classification performance drop happens when mitigation methods. We believe that a meaningful extension of our work can be developing bias mitigation methods that maintain (or even increase) the classification performance and reduce the bias at the same time. Some previous works BIBREF17 , BIBREF18 employ adversarial training methods to make the classifiers unbiased toward certain variables. However, those works do not deal with natural language where features like gender and race are latent variables inside the language. Although those approaches are not directly comparable to our methods, it would be interesting to explore adversarial training to tackle this problem in the future.
Although our work is preliminary, we hope that our work can further develop the discussion of evaluating NLP systems in different directions, not merely focusing on performance metrics like accuracy or AUC. The idea of improving models by measuring and correcting gender bias is still unfamiliar but we argue that they can be crucial in building systems that are not only ethical but also practical. Although this work focuses on gender terms, the methods we proposed can easily be extended to other identity problems like racial and to different tasks like sentiment analysis by following similar steps, and we hope to work on this in the future.
Acknowledgments
This work is partially funded by ITS/319/16FP of Innovation Technology Commission, HKUST, and 16248016 of Hong Kong Research Grants Council.

Question: What other scenarios can the bias mitigation methods be applied to?
Answer: sentiment analysis , other identity problems like racial

other identity problems like racial, sentiment analysis

developing bias mitigation methods that maintain (or even increase) the classification performance and reduce the bias at the same time

Question: Are the three bias mitigation methods combined in any model?
Answer: Yes

Debiased Word Embeddings, Gender Swap, Bias fine-tuning

Question: Which of the three bias mitigation methods is most effective?
Answer: Gender Swap

most effective method was applying both debiased embedding and gender swap

Question: What model architectures are used?
Answer: Convolutional Neural Network, Gated Recurrent Unit, Bidirectional GRU with self-attention

Convolutional Neural Network, Gated Recurrent Unit, Bidirectional GRU with self-attention

Convolutional Neural Network (CNN), Gated Recurrent Unit (GRU), Bidirectional GRU with self-attention ( INLINEFORM0 -GRU)

Question: What pre-trained word embeddings are used?
Answer: word2vec, FastText, randomly initialized embeddings (random)

word2vec train on Google News corpus; FastText train on Wikipedia corpus; randomly initialized embeddings

word2vec BIBREF10 trained on Google News corpus, FastText BIBREF16 ) trained on Wikipedia corpus,

Question: What metrics are used to measure gender biases?
Answer: False Positive Equality Difference, False Negative Equality Difference

AUC scores on the original test set , AUC scores on the unbiased generated test set, the false positive/negative equality differences proposed in BIBREF1 which aggregates the difference between the overall false positive/negative rate and gender-specific false positive/negative rate

AUC scores on the original test set (Orig. AUC),  AUC scores on the unbiased generated test set (Gen. AUC), false positive/negative equality differences