SlowGuess/NAACL_2022_OCR huggingface dataset:
Morteza Dehghani. 2020a. The Gab hate corpus: A collection of 27k posts annotated for hate speech.
Brendan Kennedy, Xisen Jin, Aida Mostafazadeh Davani, Morteza Dehghani, and Xiang Ren. 2020b. Contextualizing hate speech classifiers with posthoc explanation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5435-5442, Online. Association for Computational Linguistics.
Elisa Leonardelli, Stefano Menini, Alessio Palmero Aprosio, Marco Guerini, and Sara Tonelli. 2021. Agreeing to disagree: Annotating offensive language datasets with annotators' disagreement. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10528-10539, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Marta Marchiori Manerba and Sara Tonelli. 2021. Fine-grained fairness analysis of abusive language detection systems with CheckList. In Proceedings of the 5th Workshop on Online Abuse and Harms (WOAH 2021), pages 81-91, Online. Association for Computational Linguistics.
Endang Wahyu Pamungkas and Viviana Patti. 2019. Cross-domain and cross-lingual abusive language detection: A hybrid approach with deep learning and a multilingual lexicon. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pages 363-370, Florence, Italy. Association for Computational Linguistics.
Ji Ho Park, Jamin Shin, and Pascale Fung. 2018. Reducing gender bias in abusive language detection. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2799-2804, Brussels, Belgium. Association for Computational Linguistics.
Fabio Poletto, Valerio Basile, Manuela Sanguinetti, Cristina Bosco, and Viviana Patti. 2021. Resources and benchmark corpora for hate speech detection: a systematic review. Lang. Resour. Evaluation, 55:477-523.
Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in natural language inference. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 180-191, New Orleans, Louisiana. Association for Computational Linguistics.
Alan Ramponi and Barbara Plank. 2020. Neural unsupervised domain adaptation in NLP—A survey. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6838-6855, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Paul Röttger, Bertie Vidgen, Dong Nguyen, Zeerak Waseem, Helen Margetts, and Janet Pierrehumbert. 2021. HateCheck: Functional tests for hate speech detection models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 41-58, Online. Association for Computational Linguistics.
Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019. The risk of racial bias in hate speech detection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1668-1678, Florence, Italy. Association for Computational Linguistics.
Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A. Smith, and Yejin Choi. 2020. Social bias frames: Reasoning about social and power implications of language. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5477-5490, Online. Association for Computational Linguistics.
Mike Schuster and Kaisuke Nakajima. 2012. Japanese and korean voice search. In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5149-5152. IEEE.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725, Berlin, Germany. Association for Computational Linguistics.
Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. 2020. Dataset cartography: Mapping and diagnosing datasets with training dynamics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9275-9293, Online. Association for Computational Linguistics.
Thanh Tran, Yifan Hu, Changwei Hu, Kevin Yen, Fei Tan, Kyumin Lee, and Se Rim Park. 2020. HABERTOR: An efficient and effective deep hatespeech detector. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7486-7502, Online. Association for Computational Linguistics.
Ameya Vaidya, Feng Mai, and Yue Ning. 2020. Empirical analysis of multi-task learning for reducing
identity bias in toxic comment detection. Proceedings of the International AAAI Conference on Web and Social Media, 14(1):683-693.
Rob van der Goot, Ahmet Üstün, Alan Ramponi, Ibrahim Sharaf, and Barbara Plank. 2021. Massive choice, ample tasks (MaChAmp): A toolkit for multi-task learning in NLP. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 176-197, Online. Association for Computational Linguistics.
Bertie Vidgen and Leon Derczynski. 2020. Directions in abusive language training data, a systematic review: Garbage in, garbage out. *PloS one*, 15(12):e0243300.
Bertie Vidgen, Alex Harris, Dong Nguyen, Rebekah Tromble, Scott Hale, and Helen Margetts. 2019. Challenges and frontiers in abusive content detection. In Proceedings of the Third Workshop on Abusive Language Online, pages 80-93, Florence, Italy. Association for Computational Linguistics.
Bertie Vidgen, Dong Nguyen, Helen Margetts, Patricia Rossini, and Rebekah Tromble. 2021. Introducing CAD: the contextual abuse dataset. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2289-2303, Online. Association for Computational Linguistics.
Zeerak Waseem and Dirk Hovy. 2016. Hateful symbols or hateful people? predictive features for hate speech detection on Twitter. In Proceedings of the NAACL Student Research Workshop, pages 88-93, San Diego, California. Association for Computational Linguistics.
M. Wich, T. Eder, H. Al Kuwatly, and G. Groh. 2021. Resources and benchmark corpora for hate speech detection: a systematic review. AI Ethics, 19:1-23.
Michael Wiegand, Josef Ruppenhofer, and Thomas Kleinbauer. 2019. Detection of Abusive Language: the Problem of Biased Datasets. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 602-608, Minneapolis, Minnesota. Association for Computational Linguistics.
Wenjie Yin and Arkaitz Zubiaga. 2021. Towards generalisable hate speech detection: a review on obstacles and solutions. PeerJ Computer Science, 7:e598.
Savvas Zannettou, Barry Bradlyn, Emiliano De Cristofaro, Haewoon Kwak, Michael Sirivianos, Gianluca Stringini, and Jeremy Blackburn. 2018. What is gab: A bastion of free speech or an alt-right echo chamber. In Companion Proceedings of the The Web Conference 2018, pages 1007-1014.
Xuhui Zhou, Maarten Sap, Swabha Swayamdipta, Yejin Choi, and Noah Smith. 2021. Challenges in automated debiasing for toxic language detection. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3143-3155, Online. Association for Computational Linguistics.
A Data: Additional Details
A.1 Preprocessing and anonymization
We preprocess texts across platforms in a consistent way by anonymizing user mentions, URLs, and email addresses with [USER], [URL], and [EMAIL] placeholders, respectively. We segment hashtags into constituent words using the wordsegment package, $^{12}$  remove newlines, unescape HTML tags, and lowercase the texts.
A.2 Dedduplication
We found many duplicates in the data for all platforms. We argue that retaining duplicates as done in most previous work could severely affect the reliability of any bias analysis (and debiasing method) and its subsequent conclusions. Specifically, duplicates can (i) skew the distribution of actual artifacts in the data, overamplifying some lexical items and demoting others, and (ii) result in unfair evaluations due to identical examples falling in multiple instances of the training, development and test sets, also potentially leading to overfitting.[13]
Following this intuition, we thus remove duplicate instances after preprocessing. $^{14}$  Specifically, we removed 485 duplicate instances from Vidgen et al. (2021), 10,911 duplicate instances from Founta et al. (2018), 521 from Kennedy et al. (2020a), and 255 from de Gibert et al. (2018). Moreover, for the purpose of this work we remove duplicates whose single instances exhibit opposing labels, leaving the exploration and exploitation of annotator disagreement for future work.
B List of Spurious Artifacts
In the following, we provide the list of all spurious lexical artifacts annotated as  $S_I$  and  $S_{\neg I}$  as described in Section 4.3. All these are the ones that exhibit full agreement. In our shared repository we also release all artifacts that exhibit disagreement even after adjudication, in order to encourage future work on this direction.
Identity-related  $(S_I)$  "white", black, "jews", "women", "jew", "whites", "blacks", " muslim", "gay", " muslims", "islam", "woman", "jewish", "islamic", "immigrants", "mexican", "asian", "homosexual", "americans", "lesbian", "homo", "females", "america", "brown", "israel", "arabs", "zionist", "trans", "lgbt", "girl", "hispanic", "refugees", "male", "african", "africa", "girls", "indians", "queer", "# grate", "guy".
Non identity-related  $(S_{\neg I})$  “##s”,“##es”,“people”,“country”,“##ing”,“anti”,“illegal”,“bunch”,“##t”,“kids”,“culture”,“brain”,“#lly”,“#bt”,“#d”,“sex”,“ho”,“#nt”,“countries”,“#ic”,“#ers”,“liberal”,“reason”,“#y”,“human”,“genocide”,“#ed”,“#ists”,“wrong”,“lives”,“bad”,“god”,“#oic”,“lying”,“#ard”,“racism”,“#e”,“#oid”,“#w”,“yeah”,“millions”,“soi ciety”,“#g”,“leftist”,“crime”,“sp”,“des”,“#ist”,“#ry”,“mouth”,“#ards”,“#rs”,“#ize”,“burn”,“murdered”,“worship”,“#en ing”,“#ism”,“living”,“#fa”,“coming”,“call ing”,“streets”,“#ting”,“force”,“mis”,“#ss", “blame”,“typical”,“#pe”,“baby”,“death”,“talking”,“#gen”,“belong”,“respect”，"di”，"#yp", "sexual”，"less","mad”，"war".
C Experiments: Additional Results
C.1 Filtering with different thresholds
In Table 5 we present results for the FILTERING baseline using different sampling thresholds. Specifically, in addition to using the  $33\%$  (1/3) most ambiguous training data instances as in Swayamdipta et al. (2020), we provide full results using more aggressive (i.e.,  $25\%$ , 1/4) and less aggressive (i.e.,  $50\%$ , 1/2) filtering thresholds. We notice mixed results that make hard to determine which is the best threshold across platforms. FILTERING  $(25\%)$  improves OOD robustness on  $\mathbf{S}_I \rightarrow \mathbf{S}_I$ , and FILTERING  $(50\%)$  provides best overall in-domain performance on  $\mathbf{S}_I$ . However, MASKING  $(S_I)$  outperforms all FILTERING approaches according to the FPR metric.
C.2 Average results over all corpus pairs
We provide a summary of the results for all methods in Table 6, where we report average scores over all corpus pairs (refer to Table 4 for full results). On average,  $\mathsf{MASKING}(\mathcal{S}_I)$  improvement in FPR over the VANILLA baseline is as large as  $2\times$ , both in-distribution and out-of-distribution. This comes at
$^{12}$ https://github.com/grantjenks/python-wordsegment
13Among duplicates, we found some tweets with more than 100 duplicate instances in Founta et al. (2018).
14 Most work do not explicitly mention if dedduplication is carried before or after preprocessing texts. We believe this is an important detail to foster reproducibility – we found many examples with the same text but different URLs, unveiling possibly bot-generated messages we removed this way.
In-distribution	Out-of-distribution
F1↑	FPR↓	→♂	→♀	→♂	→♀	→♂	→♀	→♂	→♀
F1↑	FPR↓	F1↑	FPR↓	F1↑	FPR↓	F1↑	FPR↓
FILTERING (25%)	72.350.7	13.421.9	58.250.6	11.982.8	65.140.9	18.183.2	67.842.3	20.089.9
FILTERING (33%)	72.791.0	14.574.4	58.950.3	12.054.4	65.571.3	19.196.6	67.681.9	19.672.5
FILTERING (50%)	74.870.9	11.262.6	59.200.4	9.822.5	66.340.3	19.392.8	69.542.4	20.504.5
FILTERING (25%)	68.230.6	15.772.3	60.730.3	40.981.0	62.890.7	53.642.6	65.660.6	59.836.6
FILTERING (33%)	68.460.3	14.661.1	61.160.2	38.961.5	63.660.7	52.532.1	65.970.7	53.001.4
FILTERING (50%)	68.770.5	12.051.6	61.110.4	35.642.5	65.311.2	46.873.8	67.120.5	48.035.6
FILTERING (25%)	71.160.6	30.512.6	65.920.1	26.701.6	61.600.8	11.311.6	68.700.8	34.582.9
FILTERING (33%)	71.130.1	27.477.1	64.310.7	23.675.6	61.090.6	9.904.4	68.151.2	31.886.2
FILTERING (50%)	71.481.0	27.475.4	64.111.1	24.396.1	61.810.5	8.782.5	67.910.9	29.198.8
FILTERING (25%)	72.942.2	14.702.2	59.040.7	14.571.6	57.122.1	6.771.4	62.430.2	22.734.2
FILTERING (33%)	73.423.1	17.391.6	58.381.1	18.331.3	57.251.6	6.851.8	62.010.8	25.453.2
FILTERING (50%)	76.421.1	13.871.4	59.501.0	14.862.9	58.011.3	5.210.3	63.420.4	22.532.0
Table 5: Additional results for the FILTERING baseline using different sampling thresholds (25%, 33%, 50%).
In-distr.	Out-of-distr.
F1↑	FPR↓	F1↑	FPR↓
VANILLA	73.57	17.35	63.98	23.62
FILTERING	71.45	18.52	62.85	25.96
REMOVAL (S→I)	72.67	17.99	63.90	25.63
REMOVAL (SI)	72.00	13.34	62.61	17.20
MASKING (S→I)	72.83	16.53	63.90	23.16
MASKING (SI)	72.31	9.11	62.03	11.80
Table 6: Average in-distribution and OOD results over all corpus pairs for each method.
the cost of a minimal in-distribution and OOD drop in macro  $F_{1}$  (i.e., -1.26 and -1.95, respectively).
D Lexical Artifacts Statement Example
An example of lexical artifacts statement for the Reddit dataset (Vidgen et al., 2021) used in this study is presented in the following.
1) TOP LEXICAL ARTIFACTS. We present the top  $k = 10$  most informative tokens for the hateful class along with their scores in Table 7.
Rank	Token	Score	Rank	Token	Score
1	##tar	1.00	6	##gg	0.80
2	##ded	0.91	7	gay	0.79
3	##s	0.86	8	women	0.76
4	fa	0.85	9	##ds	0.74
5	b*tch	0.83	10	f*cking	0.74
Table 7: Top 10 most informative tokens for the hateful class on the Reddit dataset according to PMI.
II) CLASS DEFINITIONS. The hateful class is represented by originally identity-directed labeled
examples in CAD (Vidgen et al., 2021), and is defined as "Content which contains a negative statement made against an identity. An 'identity' is a social category that relates to a fundamental aspect of individuals' community, socio-demographics, position or self-representation [...] It includes but is not limited to Religion, Race, Ethnicity, Gender, Sexuality, Nationality, Disability/Ableness and Class." (Vidgen et al., 2021).
III) METHODS AND RESOURCES. In order to compute the correlation between tokens to the hateful class we employ PMI as implemented in [this work] (code: https://github.com/ dhfbk/hate-speech-artifacts). Input texts have been preprocessed by anonymizing user mentions, URLs, and email addresses with [USER], [URL], and [EMAIL] placeholders. Hashtags have been segmented using wordsegment, $^{15}$  and we remove newlines, unescape HTML tags, and lowercase texts. Duplicate instances have been removed after preprocessing.
The full list of lexical artifacts along with associated scores is available at https://github.com/dhfbk/hate-speech-artifacts.
15 https://github.com/grantjenks/ python-Wordsegment