SlowGuess/NAACL_2022_OCR huggingface dataset:
Alan Ramponi  
Fondazione Bruno Kessler  
Trento, Italy  
alramponi@fbk.eu
Sara Tonelli  
Fondazione Bruno Kessler  
Trento, Italy  
satonelli@fbk.eu
A Warning: this paper contains content that may be offensive or upsetting.
Avoiding to rely on dataset artifacts to predict hate speech is at the cornerstone of robust and fair hate speech detection. In this paper we critically analyze lexical biases in hate speech detection via a cross-platform study, disentangling various types of spurious and authentic artifacts and analyzing their impact on out-of-distribution fairness and robustness. We experiment with existing approaches and propose simple yet surprisingly effective data-centric baselines. Our results on English data across four platforms show that distinct spurious artifacts require different treatments to ultimately attain both robustness and fairness in hate speech detection. To encourage research in this direction, we release all baseline models and the code to compute artifacts, pointing it out as a complementary and necessary addition to the data statements practice.<sup>1</sup>
Hate speech in online social communities is a serious and pervasive concern, which requires fair and robust automated approaches to be tackled at scale. However, despite the great progress in natural language processing for detecting hate speech, current models have shown to be brittle when applied to real-world data, exhibiting limited out-of-distribution (OOD) robustness (Vidgen et al., 2019) and perpetuating and amplifying harmful social biases (Röttger et al., 2021). Noticeably, hate speech detection systems are typically trained on data from limited language varieties such as individual platforms, which inevitably exhibit differences in writing norms, language use, and hate targets, hampering generalization (Vidgen and Derczynski, 2020).
One of the main reasons for limited robustness and fairness of mainstream hate speech detection
fair	robust
what have jews done to you?	×
RT [user]: I'm mad at this	×
All black people literally go there	×	×
Table 1: Posts wrongly labeled as hateful by a fine-tuned BERT classifier due to the presence of spurious lexical artifacts (identity and non identity-related) and their negative impact  $(X)$  on fairness and robustness.
systems is largely ascribable to spurious statistical correlations between surface lexical items and labels in training data, which models exploit to derive predictions. These biases are commonly referred to as lexical dataset artifacts, and have recently attracted attention in the NLP community, particularly in natural language inference (NLI) studies (Belinkov et al., 2019; Gururangan et al., 2018; Poliak et al., 2018, inter alia). Efforts to tackle the issue in hate speech detection are instead rather scattered, and mainly focus on fairness using datasets from few platforms (Zhou et al., 2021; Kennedy et al., 2020b, inter alia), leaving the study on OOD robustness largely unexplored. We instead argue that fairness and robustness are strongly intertwined aspects (Table 1), and thus should be studied jointly, with the goal to understand to what extent these two dimensions are related.
Previous work has shown that state-of-the-art models overly rely on identity words (e.g., "jews", "gay") to predict hateful content (Zhou et al., 2021; Kennedy et al., 2020b, inter alia), further demoting voices of people from already marginalized groups (Bender et al., 2021). However, non identity-related lexical items – such as "sport", "announcer", and "football" in Waseem and Hovy (2016) – are also often spuriously associated with hate speech due to a biased data collection process (Wiegand et al., 2019), undermining OOD robustness. Despite the recent trend in minimizing
1Code and resources are available at https://github. com/dhfbk/hate-speech-artifacts.
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3027 - 3040 July 10-15, 2022 ©2022 Association for Computational Linguistics
topic bias in data sampling, we show that some spurious lexical artifacts still remain highly-predictive on certain distributions even if data has been sampled in a more attentive fashion (e.g., artifacts that are potentially data- or platform-specific – Figure 1, highlighted in gray).
We argue that disentangling artifacts into fine-grained categories by means of a cross-platform analysis may be beneficial to drive a broader debiasing of current hate speech models, ultimately improving both fairness and robustness to out-of-distribution data. To this purpose, we critically analyze artifacts in hate speech detection across multiple platforms and propose simple yet effective data-centric baselines exploiting spurious lexical items. We show that although we achieve substantial improvements in OOD fairness by exploiting spurious identity-related artifacts, this comes at the cost of robustness. This confirms that fairness and robustness are strictly interrelated dimensions that should be studied together in future research.
Contributions To the best of our knowledge, we are the first to (i) conduct a thorough investigation of lexical artifacts across online platforms; (ii) disentangle artifacts into fine-grained categories; and (iii) propose a viable data-centric approach based on masking that consistently improves fairness over all baselines across all platforms. To foster future research on the topic, we also release (iv) code to reproduce all experiments, and (v) disaggregated lexical artifact annotations, more broadly (vi) suggesting the inclusion of dataset artifacts in data statements (Bender and Friedman, 2018), which can be easily revealed using our codebase.
2 Lexical Artifacts are not all the Same
We conceptualize dataset artifacts at the lexical level as emergent correlations between tokens and labels in input data, consistently to lexical annotation artifacts in NLI (Gururangan et al., 2018). As such, given a target class  $c$ , we formally define lexical artifacts  $\mathcal{L}_c$  as the set of highly-discriminating $^2$  tokens for  $c$ , which comprise authentic artifacts  $\mathcal{A}_c$  – items that potentially carry useful information for the class at hand – and spurious artifacts  $S_c$  – items that are spuriously (or undesirably) associated to the target class – such that  $\mathcal{L}_c = \mathcal{A}_c \cup S_c$ . In the context of hate speech detection, we consider
Figure 1: Illustration of lexical artifacts in hate speech detection, including relevant examples.  $\mathcal{A}_I$ : authentic artifacts, identity-related;  $\mathcal{A}_{-I}$ : authentic artifacts, non identity-related;  $S_I$ : spurious artifacts, identity-related;  $S_{-I}$ : spurious artifacts, non identity-related. Items highlighted in gray are potentially platform-specific.
the hateful class as  $c$  unless otherwise specified and simplify the notation (i.e., from “ $\cdot_c$ ” to “ $\cdot$ ”).
We build our definitions on top of the categories of lexical biases by Zhou et al. (2021), which originally identify three bias groups:  $i$  ) minority identity mentions which are not offensive, ii) minority identity mentions which are potentially offensive, and iii) non-identity mentions which are possibly offensive. We enrich this categorization by introducing a high-level separation into spurious (i.e., group  $i$  in Zhou et al. (2021)) and authentic artifacts (i.e., group ii) and iii)), and including an additional spurious, non identity-related category (Section 2.2).
Indeed, given the broad nature of authentic and spurious artifacts, we further categorize them in Section 2.1 and 2.2 (see Figure 1 for an overview).
2.1 Authentic artifacts
We define authentic lexical artifacts  $\mathcal{A}$  as the subset of highly-discriminating tokens which potentially convey harmfulness, profanity, or are otherwise frequently associated with hateful contexts. Intuitively,  $\mathcal{A}$  is the set of artifacts which is likely to be informative to detect hate speech across distributions. Authentic artifacts enclose minority identity-related artifacts  $\mathcal{A}_I$  and non-identity artifacts  $\mathcal{A}_{\neg I}$ .
Identity-related  $(\mathcal{A}_I)$  Potentially offensive or stereotyping terms towards minority identities (e.g., "n*gro", "f*ggot", "k*ke", "wh*re"), as well as reclaimed slurs (e.g., "n*gga") (Figure 1, top left).
Non-identity related  $(\mathcal{A}_{\neg I})$  Swear words and profanities (e.g., "f*ck", "sh*t") as well as broad
2Highly-discriminating tokens can be computed and filtered using information theory and statistics measures.
terms typically associated with hateful contexts (e.g., "kill", "idiots") (Figure 1, bottom right).
2.2 Spurious artifacts
Spurious lexical artifacts  $S$  broadly enclose all tokens which we do not expect to be predictive for the target class at hand. As such, we postulate that those artifacts are a main reason for insufficient robustness and fairness of current hate speech detectors, and thus may play a positive role in lexical debiasing. We specifically focus on these artifacts in our experiments. As for authentic artifacts, spurious items can be grouped into minority identity-related artifacts  $S_I$  and non-identity artifacts  $S_{\neg I}$ , the latter being currently disregarded in research investigating fairness only (Zhou et al., 2021).
Identity-related  $(S_I)$  Terms describing a social minority, which are typically associated to hate speech due to their frequency on offensive statements on online fora (e.g., "muslim", "woman", "Islam", "nigerian", "LGBT") (Figure 1, top right).
Non-identity related  $(S_{\neg I})$  All non-identity tokens which are unexpectedly associated to hate speech, e.g., due to platform-specificity, bias in collection timeframe, etc. (e.g., "people", "RT", "streets", "Trump", "yeah") (Figure 1, bottom left).
In this work we focus on hate speech, i.e., messages whose content spreads hatred or incites violence, or threatens people's freedom, dignity and safety, and whose target is a protected group, or an individual targeted for belonging to such a group and not for his/her individual characteristics (Poletto et al., 2021). Hate speech typically encompasses serious cases of offense with severe moral and legal implications, i.e., those cases that are of primary importance for content moderation.
We collect hate speech corpora that meet the following criteria: (i) they minimize topic and author biases in data collection (Wiegand et al., 2019), using alternatives to keyword and user searches such as pure or boosted random sampling, (ii) they pertain to different social media platforms, and (iii) they follow similar annotation guidelines, where hate speech is clearly defined and separated from other types of offensive language. For each corpus we create hateful and non-hateful examples. All datasets follow consistent preprocessing, dedduplication, and anonymization (Appendices A.1 and A.2).
REDDIT ( ) We use the recently introduced Reddit dataset (v1.1) by Vidgen et al. (2021) which preserves a variety of grammar, topic, and style features due to a community-based sampling approach. The corpus contains 27,494 entries annotated following a hate speech taxonomy comprising abusive (identity-directed, affiliation-directed, person-directed) and non-abusive labels (non-hateful slurs, counter speech, and neutral). We follow the widely accepted definition of hate speech as "abuse targeting a protected group or its members for being a part of that group"3 (Röttger et al., 2021; Banko et al., 2020; Vidgen et al., 2019, inter alia) to create the hateful label from identity-directed examples, and the non-hateful label from the remaining examples. For the purpose of this study, we discard instances marked as requiring previous content to be interpreted.4 The final dataset after preprocessing consists of 1,688 hateful and 19,888 non-hateful examples, for a total of 21,576 unique instances.
Twitter  $(\mathcal{V})$  We select a widely used hate speech dataset which has been collected following a bootstrap random sampling approach (Founta et al., 2018). The dataset consists of 99,996 tweets annotated as hateful, abusive, spam, and normal. Similarly to previous work, we discard the spam category (Zhou et al., 2021), forming the hateful class following the original classification provided by the authors. This led to 3,937 hateful and 70,554 non-hateful examples, for a total of 74,491 tweets.
GAB ( ) We use the GAB hate corpus by Kennedy et al. (2020a), whose data has been sampled purely randomly due to the frequency of hate speech of the "free speech-preserving" (Zannettou et al., 2018) GAB social network. The corpus (v.2021-03-03) consists of 27,546 posts annotated with (assault on) human dignity, call for violence, and vulgarity/offensive labels. Similarly to previous work, we take the union of human dignity and call for violence labels for the hateful class (Kennedy et al., 2020b), whereas we create the non-hateful class from the remaining examples. We also leverage target annotations and consider messages towards ideology/political groups as nonhateful, to ensure consistency among datasets. As
<sup>3</sup>Groups based on age, disability, familial status, gender identity, national/ethnic origins, pregnancy, race, religion, sex or sexual orientation, as defined in Röttger et al. (2021), which in turn reflects the US 1964 Civil Rights Act, the EU's Charter of Fundamental Rights, and the UK's 2010 Equality Act.
4We leave the investigation of lexical artifacts in context-aware hate speech detection for future work.
a result, the final dataset is made up of 27,014 messages: 1,785 hateful and 24,829 non-hateful.
STORMFRONT (♂) We use the dataset pertaining to a white supremacist web forum collected by de Gibert et al. (2018) following a random sampling procedure. It consists of a total of 10,944 messages with annotations for hate, no-hate, relation, and skip labels. We remove relation and skip examples, since they require previous context, or they represent spam / content written in other languages, respectively. We then use the hate examples for the hateful class, and the no-hate instances for the non-hateful class. This led to a total of 10,448 examples, 1,192 hateful and 9,256 normal.
4 Disentangling Lexical Artifacts
In order to disentangle lexical artifacts, we first compute the correlation between each token and the hateful class for each dataset (Section 4.1), then assessing the cross-distribution indicativeness of each token (Section 4.2). For segmenting texts into tokens, we rely on the training portion of each dataset only (Section 5) and employ the WordPiece (Schuster and Nakajima, 2012) subword tokenizer as used in BERT (Devlin et al., 2019). Finally, we perform lexical artifacts annotation (Section 4.3) following the categories defined in Section 2.
4.1 In-distribution artifacts
We follow Gururangan et al. (2018) and employ pointwise mutual information (PMI; Fano, 1961) to compute the discriminativeness of each token to the target class.<sup>6</sup> Since lexical artifacts are meant to be used for downstream debiasing, we argue that tokens should be consistent with inputs to the end model. As a result, we use tokens as given by the WordPiece subword tokenizer, the same tokenizer used by models employed in our experiments (Section 5). Formally, given a token  $t$  and a class  $c$ , the PMI is defined as follows:
$$
\operatorname {P M I} (t, c) = \log \frac {p (t , c)}{p (t | \cdot) p (\cdot | c)} \tag {1}
$$
We further apply reweighting to emphasize highly-discriminative token-class correlations, and normalize  $\leq 0$  values to zero since negative PMI
Rank	#	#	#	#	Avg.
1	##tar	##gga	white	n*gro	##s
2	##ded	hate	jews	white	white
3	##s	rt	##gger	black	black
4	fa	##s	##s	##s	jews
5	b*tch	用户的	jew	jews	hate
6	##gg	idiot	islam	whites	##es
7	gay	trump	muslim	blacks	women
8	women	ass	whites	jew	people
9	##ds	idiots	##gg	race	##tar
10	f*cking	people	women	##es	jew
Table 2: Top 10 most informative tokens for the hateful class according to PMI, divided per platform dataset (left), and after cross-distribution computation (right).
scores are known to be unreliable on relatively small corpora (Jurafsky and Martin, 2021, Ch. 6).
Discussion The top 10 tokens on each platform that are more associated with the hateful class are presented in Table 2 (left). All platforms exhibit a variety of lexical artifact types (cf. Section 2); however, we observe clear divergences across distributions. While artifacts in Stormfront data are mainly related to race, on Gab the focus is more on religion. Reddit and Twitter conversations are instead more varied, with higher occurrence of spurious, non-identity artifacts (e.g., "RT", "people").
4.2 Cross-distribution artifacts
When datasets from multiple platforms are available, we hypothesize that leveraging individual scores makes possible to better identify artifacts. Given  $\mathrm{PMI}(t,c)^d$  the score of a token  $t$  for the hateful class  $c$  on a given distribution  $d\in D$  (e.g., platform), we normalize it in [0,1] by applying a min-max normalization function to enable cross-platform score comparability - obtaining  $\mathrm{PMI}(t,c)_{[0,1]}^d$  - further applying a  $\log_2$  transformation to mitigate the skewness of the original PMI distribution. As a result, the final cross-distribution score  $S(t,c)$  for each token is given by the average of the corresponding individual scores:
$$
S (t, c) = \frac {1}{| D |} \sum_ {d = 1} ^ {D} \log_ {2} \left(\operatorname {P M I} (t, c) _ {[ 0, 1 ]} ^ {d}\right) \tag {2}
$$
We then sort tokens by descending score, highlighting lexical artifacts that are highly discriminating for the hateful class across distributions. Table 2 (right) shows the top 10 tokens after the cross-platform computation is carried out.
In preliminary experiments we found similar results when using the byte-level BPE tokenizer (Sennrich et al., 2016) as used in RoBERTa (Liu et al., 2019).
6 A comparative assessment of different metrics for computing token-class correlations is out-of-scope in this study and will be investigated in future work.
Discussion As shown in Table 2 (right), cross-platform importance of tokens for the hateful class demotes scores (and thus, ranks) of lexical artifacts which are likely to be more indicative on some platforms only (e.g., "RT"), while consolidating the informativeness of cross-platform items (e.g., "jews", "hate", "#s", "#es", "people"). This confirms our hypothesis that encompassing multiple platforms is beneficial for capturing lexical items that are likely to be predictive across distributions.
4.3 Artifacts annotation
In order to disentangle lexical artifacts for further debiasing, we select the  $k$  most predictive tokens given by the cross-distribution rank of discriminativeness (Section 4.2) to be manually annotated. $^{8}$  In our experiments, we set  $k = 200$  as it matches the subset of tokens which are highly informative  $(\geq 0.33)$ . All  $k$  tokens have been labeled as potentially hateful and/or related to minority identities by two annotators - male and female, fluent in English - with background in linguistics and NLP, and past experience in hate speech activities with NGOs. Each annotator was provided with five examples of tokens in context for enabling more informed annotation decisions, represented by randomly sampled posts from the four platforms included in this study.
After annotation, the two annotators were involved in an adjudication session in order to discuss the cases of disagreement, followed by correction wherever possible. We calculate the inter-annotator agreement (IAA) score before and after adjudication using Cohen's kappa (Cohen, 1960). We obtain  $\kappa = 0.6887$  before and  $\kappa = 0.8311$  after the adjudication session, which is high agreement.
Discussion Although some cases of disagreement were easily resolvable (e.g., annotation errors), we found tokens which are difficult to discern due to ambiguity – mostly subwords – or due to real disagreement in the interpretation of the terms. This is in line with existing works showing that disagreement in toxicity annotation is inherent to the task and cannot always be solved through majority voting or adjudication (Aroyo et al., 2019;
Figure 2: Cumulative Cohen's kappa  $(\kappa)$  scores for the full annotation of lexical artifacts (top), and for decisions on potentially hateful or identity-related artifacts only (middle and bottom, respectively), ordered by informativeness according to cross-distribution scores.
Leonardelli et al., 2021). Interestingly, this disagreement follows the trend of cross-distribution rank of artifacts (Figure 2). We decided to leave the analysis of annotators' disagreement for future work, and we release these cases as disaggregated labels. In Table 3 we show the most informative artifacts by type, whereas the full list of spurious artifacts used in the experiments is in Appendix B.
We investigate the impact of spurious lexical artifacts on fairness and robustness in hate speech detection. Similarly to previous studies (Kennedy et al., 2020a; Röttger et al., 2021), we cast hate speech detection as a binary classification problem, where the two classes to be predicted are hateful and non-hateful, as defined in Section 3. We carry out in-distribution and OOD experiments, namely training and testing all models on the same or different platform data, respectively. We evaluate performance of models using macro  $F_{1}$  score, whereas for fairness we use false positive rate (FPR) on test instances containing  $S_{I}$  mentions, consistently to
7We found "##s" and "##es" tokens typically correspond to plural suffixes of out-of-vocabulary words.
The main advantage of token-level annotation compared to word-level annotation is that it allows to discern generic subwords from hateful or identity-related ones – e.g., “homophobia”  $\mapsto$  {"homo", "#phobia"} - without losing important information when doing removal or masking (Section 5).
<sup>9</sup>We leave the investigation of larger thresholds for future work due to space and annotation constraints.
Authentic artifacts	Spurious artifacts
AI	A¬I	SI	S¬I
retar*s	hate	white	_s
b*tch	dumb	black	_es
n*gro	stupid	jews	people
f*ggot	disgusting	women	country
n*gger	kill	jew	_ing
n*gga	racist	whites	anti
r*tarded	filthy	blacks	illegal
fa*s	evil	muslim	bunch
f*gs	ass	gay	_t
f*ggot	rape	m Muslims	kids
Table 3: Top 10 tokens per artifact type after annotation. Gray letters indicate the most common token prefixes/suffixes which occur with the lexical items. If a variety of them is available, we indicate it with “_”.
previous work (Zhou et al., 2021).
We outline the experimental setup in Section 5.1, whereas data-centric baselines are presented in Section 5.2. Lastly, we present results and a thorough discussion in Section 5.3.
For all our experiments, we employ the uncased BERT-base model (Devlin et al., 2019) as implemented in the MaChAmp v0.2 toolkit (van der Goot et al., 2021), since it has been shown to achieve state-of-the-art performance in hate speech detection (Tran et al., 2020). We use default hyperparameters, and perform a grid search to determine the number of epochs, the learning rate, and the batch size, using the search space suggested by Devlin et al. (2019) - i.e., [2, 3, 4] for epochs, [2e-05, 3e-05, 5e-05] for learning rate, and [16, 32] for batch size. We use stratified  $80\%$  train,  $10\%$  development,  $10\%$  test splits for each dataset, selecting the best model based on the average macro  $F_{1}$  score on the development test across all platforms. During fine-tuning, we emphasize the minority hateful class using a cross-entropy loss with balanced class weights. The final hyperparameters are: 4 for epochs, 2e-05 for learning rate, and 16 for batch size. All experiments have been run on a NVIDIA Tesla V100-SXM2 GPU, with a training time ranging from 10 to 40 minutes each. The number of trainable parameter for all models are  $\approx 110\mathrm{M}$ .
We investigate the impact of spurious identity-related and non identity-related lexical artifacts on the robustness and fairness of hate speech detection by employing the following data-centric baselines.