SlowGuess/NAACL_2022_OCR huggingface dataset:
VANILLA We fine-tune the BERT-base model on each corpus, so that the proposed baselines can be directly compared to a commonly employed setup.
FILTERING Swayamdipta et al. (2020) have shown that the most ambiguous training data instances promote OOD generalization while preserving in-distribution performance. We thus leverage the VANILLA model's training dynamics to filter training data to contain the  $33\%$  most ambiguous instances only, in line with the subset size in Swayamdipta et al. (2020). Intuitively, those are instances whose class probabilities fluctuate frequently across training epochs. We then fine-tune the BERT-base model on the resulting subset. This setup is similar in spirit to the one employed in Zhou et al. (2021); however, we assess it on data from multiple platforms, also removing duplicate instances (Appendix A.2) which may potentially confound the debiasing results.
REMOVAL Prior to fine-tuning, we naively remove any occurrence of spurious lexical artifacts from training and development data. This matches previously employed baselines for assessing fairness in hate speech detection (Kennedy et al., 2020b). However, since we are also interested in OOD robustness, we experiment with two removal variants: one for  $S_I$  and one for  $S_{\neg I}$  artifacts. We hypothesize that removing  $S_I$  tokens potentially improves fairness, whereas removing  $S_{\neg I}$  tokens mostly contributes to OOD robustness.
MASKING We propose a novel data-centric debi- asing alternative based on token masking. Instead of removing spurious artifacts altogether, we reserve a special token in the vocabulary of the model that we use as replacement for spurious artifacts. We then fine-tune the model on the masked data. Intuitively, this way we encourage the model to blend all artifacts to a single contextualized representation that will never appear during testing, also avoiding to redistribute the informativeness of spurious lexical items to surrounding tokens. As for REMOVAL, we experiment with  $S_{\neg I}$  and  $S_I$  masking variants.
5.3 Results and discussion
In Table 4 we report the results for all baselines along the in-distribution and out-of-distribution dimensions from the lens of fairness and robustness.
For fair comparison, we also provide results with less  $(50\%)$  and more  $(25\%)$  aggressive thresholds in Appendix C.1.
In-distribution	Out-of-distribution
→♂	→♀	→♂	→♀
F1↑	FPR↓	F1↑	FPR↓	F1↑	FPR↓	F1↑	FPR↓	F1↑	FPR↓
VANILLA	75.830.3	11.260.9	59.260.7	9.601.3	68.240.4	19.801.4	69.580.3	16.362.0
FILTERING	72.791.0	14.574.4	58.950.3	12.054.4	65.571.3	19.196.6	67.681.9	19.672.5
REMOVAL (S-I)	74.960.8	10.391.1	59.280.3	11.090.5	68.560.3	19.291.9	67.651.2	15.321.8
REMOVAL (SI)	74.960.8	9.520.9	58.990.4	8.931.7	66.310.6	13.131.1	63.001.5	14.493.6
MASKING (S-I)	74.761.1	10.820.7	59.140.2	13.472.5	67.620.6	19.292.3	66.370.9	17.813.6
MASKING (SI)	76.410.6	7.502.0	58.830.4	8.262.3	65.660.8	10.101.5	63.481.6	6.631.9
VANILLA	68.830.4	11.011.5	60.611.2	29.874.1	65.950.6	41.724.8	67.680.6	40.175.3
FILTERING	68.460.3	14.661.1	61.160.2	38.961.5	63.660.7	52.532.1	65.970.7	53.001.4
REMOVAL (S-I)	67.891.0	13.761.4	60.301.9	37.524.3	65.350.6	48.795.6	65.831.4	54.246.2
REMOVAL (SI)	67.650.4	6.991.1	58.770.4	17.462.5	65.710.3	27.985.2	66.951.7	21.743.9
MASKING (S-I)	68.500.5	9.601.4	61.171.2	29.442.3	66.710.9	40.612.8	67.111.0	36.857.4
MASKING (SI)	66.720.7	5.360.4	57.721.2	12.552.0	65.070.6	24.143.2	63.133.1	11.593.1
VANILLA	71.290.6	31.414.6	64.510.7	29.155.8	61.121.4	10.042.0	67.660.9	37.686.5
FILTERING	71.130.1	27.477.1	64.310.7	23.675.6	61.090.6	9.904.4	68.151.2	31.886.2
REMOVAL (S-I)	71.040.3	30.003.4	64.191.1	27.714.3	61.581.1	10.492.3	68.440.8	34.992.2
REMOVAL (SI)	69.780.5	21.525.4	65.080.4	21.936.3	60.541.3	6.991.4	66.521.4	24.227.8
MASKING (S-I)	71.060.4	27.884.7	64.460.7	25.972.8	61.910.6	8.631.6	68.860.6	33.334.7
MASKING (SI)	69.720.8	13.640.9	65.550.8	15.012.2	60.171.5	3.200.6	66.642.5	13.041.9
VANILLA	78.330.9	15.732.2	60.200.5	17.893.2	58.220.8	5.580.4	64.760.6	25.562.0
FILTERING	73.423.1	17.391.6	58.381.1	18.331.3	57.251.6	6.851.8	62.010.8	25.453.2
REMOVAL (S-I)	76.771.0	17.811.3	61.321.6	17.171.5	59.072.0	6.180.8	65.260.4	24.751.4
REMOVAL (SI)	75.621.1	15.323.9	58.580.7	20.063.5	58.950.4	7.291.8	61.960.6	22.122.6
MASKING (S-I)	77.011.0	17.810.7	60.000.7	19.054.2	58.990.8	6.180.3	64.440.2	27.272.0
MASKING (SI)	76.390.6	9.941.6	57.811.2	14.431.1	57.331.5	4.320.6	62.970.4	18.281.7
Table 4: In-distribution and out-of-distribution results ( $F_{1}$  for accuracy and FPR for fairness). Out-of-distribution results are on  $\rightarrow$  REDDIT,  $\rightarrow$  TWITTER,  $\rightarrow$  GAB, and  $\rightarrow$  STORMFRONT. Scores are averages of 3 runs with different seeds, whereas subscripts indicate standard deviation.  $\uparrow$ : greater the better;  $\downarrow$ : lower the better.
Since we argue that in-distribution performance is not a reliable measure for the performance of a hate speech detection system in the wild, due to space constraints we here focus on the more realistic yet more challenging out-of-distribution setup.
Filtering is not a one-size-fits-all solution Despite the improvements in OOD generalization on commonsense reasoning, question answering, and NLI tasks (Swayamdipta et al., 2020), training on ambiguous instances collected from training dynamics is not as effective in hate speech detection. Instead, our results show that FILTERING leads to mixed results for OOD fairness compared to the VANILLA baseline. This is consistent with results on Twitter data (Zhou et al., 2021), and we further confirm it is the case also across platforms. Importantly, we also notice that FILTERING has a detrimental effect on OOD robustness, except for two cases only (i.e.,  $\rightarrow$  and  $\rightarrow$ ). This indicates that hate speech detection is a nuanced task requiring more targeted approaches than automated data filtering.
Removing  $S_I$  is not as strong as it has been previously thought. Removing identity terms from data altogether is a commonly used baseline for testing downstream fairness (e.g., Kennedy et al., 2020b). Indeed, our results confirm that  $\operatorname{REMOVAL}(S_I)$  consistently reduces the FPR on test instances containing  $S_I$  mentions compared to the VANILLA baseline - with the only exception of  $\rightarrow \rightarrow$ . However, it only improves OOD robustness on  $\rightarrow \rightarrow$  and  $\rightarrow \rightarrow$ . Moreover, it consistently scores lower than  $\operatorname{MASKING}(S_I)$  on fairness, as discussed below. This raises the question of whether  $\operatorname{REMOVAL}(S_I)$  should continue to be used as fairness baseline in future studies.
Masking  $S_I$  improves fairness When masking  $S_I$ , we notice a consistent improvement in fairness over all approaches, both in-distribution and out-of-distribution, on all platforms. Reduction in FPR over the VANILLA baseline is as large as  $3 \times$ , as results for  $\{\mathcal{V}; \mathcal{G}\} \to \mathcal{G}$  and  $\mathcal{G} \to \mathcal{V}$  show. Most of the remaining train-test pairs show a  $2 \times$  improvement in FPR, also compared to the common REMOVAL  $(S_I)$  baseline. We hypothesize the improved fairness performance with respect to RE
<sup>11</sup>We notice this holds true also when employing less/more aggressive filtering thresholds, as shown in Appendix C.1.
MOVAL  $(S_I)$  is due to the way contextualized representations are formed during training, as discussed in Section 5.2. Despite being surprisingly simple, we envision  $\mathsf{MASKING}(S_I)$  as a strong baseline for future work on fairness in hate speech detection.
$\mathcal{S}_{\neg I}$  artifacts are not as useful as  $\mathcal{S}_I$  We observe that methods exploiting  $\mathcal{S}_{\neg I}$  artifacts lead to mixed results. This suggests that while a substantial FPR reduction can be achieved exploiting  $\mathcal{S}_I$  artifacts, robustness calls for more complex debiasing strategies to transfer well across distributions.
Fairness comes at the cost of robustness. Overall, we observe an important trade-off between fairness and robustness. Data-centric approaches that achieve a consistently high level of fairness – namely,  $\mathsf{MASKING}(\mathcal{S}_I)$  and  $\mathsf{REMOVAL}(\mathcal{S}_I)$  – typically show a decrease in in-distribution and out-of-distribution performance – with the exception of  $\bullet \rightarrow \bullet$  and  $\bullet \rightarrow \bullet$  for  $\mathsf{MASKING}(\mathcal{S}_I)$ , and  $\bullet \rightarrow \bullet$  and  $\bullet \rightarrow \bullet$  for  $\mathsf{REMOVAL}(\mathcal{S}_I)$ . On one hand, this suggests that spurious, identity-related lexical artifacts do play an important role in performance across distributions. On the other hand, we believe this reflects the real performance of a prototypical model that is substantially fairer, and thus to which future work in hate speech detection should be compared to. We argue that  $\mathsf{MASKING}(\mathcal{S}_I)$  represents a starting point to achieve both fairness and OOD robustness, the latter requiring more complex, model-centric debiasing approaches. A summary of the results over all corpus pairs for each method is presented in Appendix C.2.
6 Towards Artifacts Documentation
The practice of data statements (Bender and Friedman, 2018) has been recently adopted by the NLP community as a way to include relevant information about the creators, the methodology and possible biases when a dataset is released. This should in turn have a positive impact on systems trained on such data, contributing to a better evaluation of models' generalization and fairness. We propose that an artifacts statement should be added to this documentation as a way to contribute to diagnosis (and thus mitigation) of pre-existing bias, which is also one of the goals of data statements.
In particular, we propose a template for lexical artifacts documentation and publicly release code to easily compute ranked correlations between tokens and target classes of interest for a given annotated
corpus. To ensure the process of documenting lexical artifacts will be as smooth as possible – and thus allows widespread adoption of artifacts statement in the future – our code automatically generates outputs in different formats, from raw text to LaTeX code for seamless inclusion in publications.
We present the artifacts statement template below, and provide a full example in Appendix D.
1) TOP LEXICAL ARTIFACTS. Which are the  $k$  most informative tokens in the corpus for the class(es) of interest? This can be a ranked list of  $(k\geq 10)$  tokens in plain text or in a tabular format, optionally along with associated scores. If there are multiple classes of interest, top  $k$  lexical artifacts for each class should be included.
II) CLASS DEFINITIONS. Different definitions for the same class may exist across datasets. This impacts the annotation, which in turn has an effect on resulting lexical artifacts. An explicit definition of the target class(es) for which the top lexical artifacts are computed should be provided here.
III) METHODS AND RESOURCES. The method used to compute the correlation between tokens and class(es) (e.g., PMI, interpretability approaches) in the annotated corpus should be reported here, possibly with a link to code. If preprocessing and dedduplication have been performed, they should be clearly reported. Resources such as full lists of lexical artifacts can be additionally included.
The problem of models' generalizability related to hate speech detection has been extensively discussed in recent works (Vidgen and Derczynski, 2020; Yin and Zubiaga, 2021; Wich et al., 2021). Indeed, it has been shown that state-of-the-art performance on this task overestimates the capability of models to yield the same results over time (Florio et al., 2020) or across different domains (Wiegand et al., 2019). Possible mitigation strategies include domain adaptation techniques (Ramponi and Plank, 2020), augmenting smaller datasets with a larger dataset from a different domain (Karan and Snajder, 2018), the use of a domain lexicon to transfer knowledge across domains (Pamungkas and Patti, 2019) and the fine-tuning of HateBERT (Caselli et al., 2021) on the target corpus (Bose et al., 2021), among others.
Concerning bias and fairness, several works have pointed out the presence of bias in hate and abusive language datasets (Wiegand et al., 2019; Sap
et al., 2019, 2020). This issue has been addressed in different ways, including functional tests for hate speech detection models (Röttger et al., 2021; Manerba and Tonelli, 2021) and post-hoc explanations to measure models' bias towards identity terms (Kennedy et al., 2020b). As regards bias mitigation, the task has been addressed through a number of approaches, e.g., via adversarial feature learning (Vaidya et al., 2020), by using debiased word embeddings and gender swap data augmentation (Park et al., 2018) or by adding non-toxic examples to better balance the data (Dixon et al., 2018). The work probably most related to ours is Zhou et al. (2021), which presents an analysis of lexical and dialectal biases in the dataset by Founta et al. (2018). The authors propose lexical bias categories which we extend in this work (see Section 2). However, they focus only on one dataset and on indomain bias reduction. Moreover, they start from a list of "bad words", whereas we compute it from data. To our knowledge, this is the first work advocating for a joint view on fairness and robustness, both identified as critical aspects related to the classification of hate speech (Wich et al., 2021).
Our work is a step forward towards a better understanding of the bias that can be encoded in hate speech detection corpora (Blodgett et al., 2020). However, we are aware of some limitations. First, all findings in this work are related to hate speech datasets in English. With the increasing availability of hate speech data in languages other than English, we aim to investigate our methods on other languages too. Second, annotated data from multiple platforms may not be available for some languages, and this can limit the cross-distribution computation of artifacts. Lastly, we acknowledge spurious statistical correlations may go beyond the token level. We believe our study is a first step towards contextual debiasing from spurious lexical artifacts, and thus can be of inspiration for future studies.
This paper investigates the impact of lexical artifacts on out-of-distribution fairness and robustness in hate speech detection, raising awareness on the interplay between the two dimensions that should be studied together in future work. We propose a fine-grained categorization of lexical artifacts and simple yet effective data-centric baselines, show
ing that while robustness calls for model-centric approaches, masking spurious identity artifacts is a viable approach that we argue should be used as strong baseline for fairness assessment in future research. In future work we aim to investigate the role of dialectal biases and non-lexical artifacts, extending the study on languages other than English. We release all baseline models, resources, and the code to compute lexical artifacts, broadly suggesting the inclusion of "artifacts statement" as a way to document potential lexical biases when a dataset is released, to provide a complementary view to data statements (Bender and Friedman, 2018).
The annotation task described in Section 4.3 was carried out by two researchers regularly employed at Fondazione Bruno Kessler as part of their work.
Overall, we do not foresee any specific ethical concern related to this work. On the contrary, our goal is to propose artifacts statement as a desirable practice for documenting potential biases in newly released datasets, and improve current debi-aising methods by distinguishing among different types of lexical artifacts. However, the (finite set of) identity-related and offensive tokens considered in this work are all in English and centered around Western cultural context. We leave the evaluation of our methodology to assess whether there are language- or more broadly culture-dependent changes for future work, following recent work on biases in geo-cultural contexts (Ghosh et al., 2021).
Part of this work was funded by the PROTECTOR European project (ISFP-2020-AG-PROTECT101034216-PROTECTOR). This research was also supported by the KID ACTIONS REC-AG project (n. 101005518) on "Kick-off preventIng and responDing to children and AdolesCenT cyberbullyIng through innovative mOnitoring and educatioNal technologieS".
Lora Aroyo, Lucas Dixon, Nithum Thain, Olivia Redfield, and Rachel Rosen. 2019. Crowdsourcing subjective tasks: The case study of understanding toxicity in online discussions. In *Companion Proceedings of The 2019 World Wide Web Conference*, WWW '19, page 1100-1105, New York, NY, USA. Association for Computing Machinery.
Michele Banko, Brendon MacKeen, and Laurie Ray. 2020. A unified taxonomy of harmful content. In Proceedings of the Fourth Workshop on Online Abuse and Harms, pages 125-137, Online. Association for Computational Linguistics.
Yonatan Belinkov, Adam Poliak, Stuart Shieber, Benjamin Van Durme, and Alexander Rush. 2019. Don't take the premise for granted: Mitigating artifacts in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 877-891, Florence, Italy. Association for Computational Linguistics.
Emily M. Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587-604.
Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610-623.
Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of "bias" in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454-5476, Online. Association for Computational Linguistics.
Tulika Bose, Irina Illina, and Dominique Fohr. 2021. Unsupervised domain adaptation in cross-corpora abusive language detection. In Proceedings of the Ninth International Workshop on Natural Language Processing for Social Media, pages 113-122, Online. Association for Computational Linguistics.
Tommaso Caselli, Valerio Basile, Jelena Mitrovic, and Michael Granitzer. 2021. HateBERT: Retraining BERT for abusive language detection in English. In Proceedings of the 5th Workshop on Online Abuse and Harms (WOAH 2021), pages 17-25, Online. Association for Computational Linguistics.
Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37-46.
Ona de Gibert, Naiara Perez, Aitor Garcia-Pablos, and Montse Cuadros. 2018. Hate speech dataset from a white supremacy forum. In Proceedings of the 2nd Workshop on Abusive Language Online (ALW2), pages 11-20, Brussels, Belgium. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference
of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2018. Measuring and mitigating unintended bias in text classification. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, AIES '18, page 67-73, New York, NY, USA. Association for Computing Machinery.
Robert M Fano. 1961. Transmission of information: A statistical theory of communications. American Journal of Physics, 29:793-794.
Komal Florio, Valerio Basile, Marco Polignano, Pierpaolo Basile, and Viviana Patti. 2020. Time of your hate: The challenge of time in hate speech detection on social media. Applied Sciences, 10(12).
Antigoni Maria Founta, Constantinos Djouvas, Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Gianluca Stringhini, Athena Vakali, Michael Sirivianos, and Nicolas Kourtellis. 2018. Large scale crowdsourcing and characterization of twitter abusive behavior. In Twelfth International AAAI Conference on Web and Social Media.
Sayan Ghosh, Dylan Baker, David Jurgens, and Vinodkumar Prabhakaran. 2021. Detecting cross-geographic biases in toxicity modeling on social media. In Proceedings of the Seventh Workshop on Noisy User-generated Text (W-NUT 2021), pages 313–328, Online. Association for Computational Linguistics.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107-112, New Orleans, Louisiana. Association for Computational Linguistics.
Daniel Jurafsky and James H. Martin. 2021. Speech and Language Processing, 3rd edition. Prentice Hall.
Mladen Karan and Jan Šnajder. 2018. Cross-domain detection of abusive language online. In Proceedings of the 2nd Workshop on Abusive Language Online (ALW2), pages 132-137, Brussels, Belgium. Association for Computational Linguistics.
Brendan Kennedy, Mohammad Atari, Aida M. Davani, Leigh Yeh, Ali Omrani, Yehsong Kim, Kris Coombs, Shreya Havaldar, Gwenyth Portillo-Wightman, Elaine Gonzalez, Joe Hoover, Aida Azatian, Gabriel Cardenas, Alyzeh Hussain, Austin Lara, Adam Omary, Christina Park, Xin Wang, Clarisa Wijaya, Yong Zhang, Beth Meyerowitz, and