Introduction
Recent work has shown evidence of substantial bias in machine learning systems, which is typically a result of bias in the training data. This includes both supervised BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 and unsupervised natural language processing systems BIBREF4 , BIBREF5 , BIBREF6 . Machine learning models are currently being deployed in the field to detect hate speech and abusive language on social media platforms including Facebook, Instagram, and Youtube. The aim of these models is to identify abusive language that directly targets certain individuals or groups, particularly people belonging to protected categories BIBREF7 . Bias may reduce the accuracy of these models, and at worst, will mean that the models actively discriminate against the same groups they are designed to protect.
Our study focuses on racial bias in hate speech and abusive language detection datasets BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12 , all of which use data collected from Twitter. We train classifiers using each of the datasets and use a corpus of tweets with demographic information to compare how each classifier performs on tweets written in African-American English (AAE) versus Standard American English (SAE) BIBREF13 . We use bootstrap sampling BIBREF14 to estimate the proportion of tweets in each group that each classifier assigns to each class. We find evidence of systematic racial biases across all of the classifiers, with AAE tweets predicted as belonging to negative classes like hate speech or harassment significantly more frequently than SAE tweets. In most cases the bias decreases in magnitude when we condition on particular keywords which may indicate membership in negative classes, yet it still persists. We expect that these biases will result in racial discrimination if classifiers trained on any of these datasets are deployed in the field.
Related works
Scholars and practitioners have recently been devoting more attention to bias in machine learning models, particularly as these models are becoming involved in more and more consequential decisions BIBREF15 . Bias often derives from the data used to train these models. For example, BIBREF16 show how facial recognition technologies perform worse for darker-skinned people, particularly darker-skinned women, due to the disproportionate presence of white, male faces in the training data. Natural language processing systems also inherit biases from the data they were trained on. For example, in unsupervised learning, word embeddings often contain biases BIBREF4 , BIBREF5 , BIBREF6 which persist even after attempts to remove them BIBREF17 . There are many examples of bias in supervised learning contexts: YouTube's captioning models make more errors when transcribing women BIBREF1 , AAE is more likely to be misclassified as non-English by widely used language classifiers BIBREF0 , numerous gender and racial biases exist in sentiment classification systems BIBREF2 , and errors in both co-reference resolution systems and occupational classification models reflect gendered occupational patterns BIBREF18 , BIBREF3 .
While hate speech and abusive language detection has become an important area for natural language processing research BIBREF19 , BIBREF7 , BIBREF20 , there has been little work addressing the potential for these systems to be biased. The danger posed by bias in such systems is, however, particularly acute, since it could result in negative impacts on the same populations the systems are designed to protect. For example, if we mistakenly consider speech by a targeted minority group as abusive we might unfairly penalize the victim, but if we fail to identify abuse against them we will be unable to take action against the perpetrator. Although no model can perfectly avoid such problems, we should be particularly concerned about the potential for such models to be systematically biased against certain social groups, particularly protected classes.
A number of studies have shown that false positive cases of hate speech are associated with the presence of terms related to race, gender, and sexuality BIBREF21 , BIBREF22 , BIBREF10 . While not directly measuring bias, prior work has explored how annotation schemes BIBREF10 and the identity of the annotators BIBREF8 might be manipulated to help to avoid bias. BIBREF23 directly measured biases in the Google Perspective API classifier, trained on data from Wikipedia talk comments, finding that it tended to give high toxicity scores to innocuous statements like “I am a gay man”. They called this “false positive bias”, caused by the model overgeneralizing from the training data, in this case from examples where “gay” was used pejoratively. They find that a number of such “identity terms” are disproportionately represented in the examples labeled as toxic. BIBREF24 build upon this study, using templates to study gender differences in performance across two hate speech and abusive language detection datasets. They find that classifiers trained on these data tend to perform worse when female identity terms used, indicating gender bias in performance. We build upon this work by auditing a series of abusive language and hate speech detection datasets for racial biases. We evaluate how classification models trained on these datasets perform in the field, comparing their predictions for tweets written in language used by whites or African-Americans.
Hate speech and abusive language datasets
We focus on Twitter, the most widely used data source in abusive language research. We use all available datasets where tweets are labeled as various types of abuse and are written in English. We now briefly describe each of these datasets in chronological order.
BIBREF9 collected 130k tweets containing one of seventeen different terms or phrases they considered to be hateful. They then annotated a sample of these tweets themselves, using guidelines inspired by critical race theory. These annotators were then reviewed by “a 25 year old woman studying gender studies and a nonactivist feminist” to check for bias. This dataset consists of 16,849 tweets labeled as either racism, sexism, or neither. Most of the tweets categorized as sexist relate to debates over an Australian TV show and most of those considered as racist are anti-Muslim.
To account for potential bias in the previous dataset, BIBREF8 relabeled 2876 tweets in the dataset, along with a new sample from the tweets originally collected. The tweets were annotated by “feminist and anti-racism activists”, based upon the assumption that they are domain-experts. A fourth category, racism and sexism was also added to account for the presence of tweets which exhibit both types of abuse. The dataset contains 6,909 tweets.
BIBREF10 collected tweets containing terms from the Hatebase, a crowdsourced hate speech lexicon, then had a sample coded by crowdworkers located in the United States. To avoid false positives that occurred in prior work which considered all uses of particular terms as hate speech, crowdworkers were instructed not to make their decisions based upon any words or phrases in particular, no matter how offensive, but on the overall tweet and the inferred context. The dataset consists of 24,783 tweets annotated as hate speech, offensive language, or neither.
BIBREF11 selected tweets using ten keywords and phrases related to anti-black racism, Islamophobia, homophobia, anti-semitism, and sexism. The authors developed a coding scheme to distinguish between potentially offensive content and serious harassment, such as threats or hate speech. After an initial round of coding, where tweets were assigned to a number of different categories, they simplified their analysis to include a binary harassment or non-harassment label for each tweet. The dataset consists of 20,360 tweets, each hand-labeled by the authors.
BIBREF12 constructed a dataset intended to better approximate a real-world setting where abuse is relatively rare. They began with a random sample of tweets then augmented it by adding tweets containing one or more terms from the Hatebase lexicon and that had negative sentiment. They criticized prior work for defining labels in an ad hoc manner. To develop a more comprehensive annotation scheme they initially labeled a sample of tweets, allowing each tweet to belong to multiple classes. After analyzing the overlap between different classes they settled on a coding scheme with four distinct classes: abusive, hateful, spam, and normal. We use a dataset they published containing 91,951 tweets coded into these categories by crowdworkers.
Training classifiers
For each dataset we train a classifier to predict the class of unseen tweets. We use regularized logistic regression with bag-of-words features, a commonly used approach in the field. While we expect that we could improve predictive performance by using more sophisticated classifiers, we expect that any bias is likely a function of the training data itself rather than the classifier. Moreover, although features like word embeddings can work well for this task BIBREF25 we wanted to avoid inducing any bias in our models by using pre-trained embeddings BIBREF24 .
We pre-process each tweet by removing excess white-space and replacing URLs and mentions with placeholders. We then tokenize them, stem each token, and construct n-grams with a maximum length of three. Next we transform each dataset into a TF-IDF matrix, with a maximum of 10,000 features. We use 80% of each dataset to train models and hold out the remainder for validation. Each model is trained using stratified 5-fold cross-validation. We conduct a grid-search over different regularization strength parameters to identify the best performing model. Finally, for each dataset we identify the model with the best average F1 score and retrain it using all of the training data. The performance of these models on the 20% held-out validation data is reported in Table 1 . Overall we see varying performance across the classifiers, with some performing much better out-of-sample than others. In particular, we see that hate speech and harassment are particularly difficult to detect. Since we are primarily interested in within classifier, between corpora performance, any variation between classifiers should not impact our results.
Race dataset
We use a dataset of tweets labeled by race from BIBREF13 to measure racial biases in these classifiers. They collected geo-located tweets in the U.S. and matched them with demographic data from the Census on the population of non-Hispanic whites, non-Hispanic blacks, Hispanics, and Asians in the block group where the tweets originated. They then identified words associated with particular demographics and trained a probabilistic mixed-membership language model. This model learns demographically-aligned language models for each of the four demographic categories and is used to calculate the posterior proportion of language from each category in each tweet. Their validation analyses indicate that tweets with a high posterior proportion of non-Hispanic black language exhibit lexical, phonological, and syntactic variation consistent with prior research on AAE. Their publicly-available dataset contains 59.2 million tweets.
We define a user as likely non-Hispanic black if the average posterior proportion across all of their tweets for the non-Hispanic black language model is $\ge 0.80$ (and $\le 0.10$ Hispanic and Asian combined) and as non-Hispanic white using the same formula but for the white language model. This allows us to restrict our analysis to tweets written by users who predominantly use one of the language models. Due to space constraints we discard users who predominantly use either the Hispanic or the Asian language model. This results in a set of 1.1m tweets written by people who generally use non-Hispanic black language and 14.5m tweets written by users who tend to use non-Hispanic white language. Following BIBREF0 , we call these datasets black-aligned and white-aligned tweets, reflecting the fact that they contain language associated with either demographic category but which may not all be produced by members of these categories. We now describe how we use these data in our experiments.

Question: Any other bias may be detected?
Answer: Unanswerable