SlowGuess/NAACL_2022_OCR huggingface dataset:
American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4110-4124, Online. Association for Computational Linguistics.
J. Richard Landis and Gary G. Koch. 1977. The Measurement of Observer Agreement for Categorical Data. Biometrics, 33(1):159-174.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv:1907.11692 [cs].
Nikola Ljubesic and Darja Fiser. 2016. A global analysis of emoji usage. In Proceedings of the 10th Web as Corpus Workshop, pages 82-89, Stroudsburg, PA, USA. Association for Computational Linguistics.
Zhiyi Ma, Kawin Ethayarajh, Tristan Thrush, Somya Jain, Ledell Wu, Robin Jia, Christopher Potts, Adina Williams, and Douwe Kiela. 2021. Dynaboard: An Evaluation-As-A-Service Platform for Holistic NextGeneration Benchmarking. In Advances in Neural Information Processing Systems, volume 34, pages 10351-10367. Curran Associates, Inc.
Dat Quoc Nguyen, Thanh Vu, and Anh Tuan Nguyen. 2020. BERTweet: A pre-trained language model for English Tweets. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 9-14, Online. Association for Computational Linguistics.
Christopher Potts, Zhengxuan Wu, Atticus Geiger, and Douwe Kiela. 2021. DynaSent: A Dynamic Benchmark for Sentiment Analysis. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2388-2404, Online. Association for Computational Linguistics.
Justas Randolph. 2005. Free-marginal multirater kappa: An alternative to fleiss. In Joensuu Learning and Instruction Symposium, page 20.
Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond Accuracy: Behavioral Testing of NLP Models with CheckList. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4902-4912, Online. Association for Computational Linguistics.
David Rodrigues, Marília Prada, Rui Gaspar, Margarida V. Garrido, and Diniz Lopes. 2018. Lisbon Emoji and Emoticon Database (LEED): Norms for emoji and emoticons in seven evaluative dimensions. Behavior Research Methods, 50(1):392-405.
Paul Röttger, Bertie Vidgen, Dirk Hovy, and Janet B. Pierrehumbert. 2022. Two Contrasting Data Annotation Paradigms for Subjective NLP Tasks. arXiv:2112.07475 [cs].
Paul Röttger, Bertie Vidgen, Dong Nguyen, Zeerak Waseem, Helen Margetts, and Janet Pierrehumbert. 2021. HateCheck: Functional Tests for Hate Speech Detection Models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 41-58, Online. Association for Computational Linguistics.
Niloofar Safi Samghabadi, Afsheen Hatami, Mahsa Shafaei, Sudipta Kar, and Thamar Solorio. 2019. Attending the emotions to detect online abusive language.
Leandro Silva, Mainack Mondal, Denzil Correa, Fabricio Benevenuto, and Ingmar Weber. 2016. Analyzing the targets of hate in online social media. arXiv.
Zeerak Talat, Thomas Davidson, Dana Warmsley, and Ingmar Weber. 2017. Understanding Abuse: A Typology of Abusive Language Detection Subtasks. In Proceedings of the First Workshop on Abusive Language Online, pages 78-84, Vancouver, BC, Canada. Association for Computational Linguistics.
United Nations. 2019. UN strategy and plan of action on hate speech. Technical report.
Bertie Vidgen, Alex Harris, Dong Nguyen, Rebekah Tromble, Scott Hale, and Helen Margetts. 2019. Challenges and frontiers in abusive content detection. In Proceedings of the Third Workshop on Abusive Language Online, pages 80-93, Florence, Italy. Association for Computational Linguistics.
Bertie Vidgen, Dong Nguyen, Helen Margetts, Patricia Rossini, and Rebekah Tromble. 2021a. Introducing CAD: The Contextual Abuse Dataset. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2289-2303, Online. Association for Computational Linguistics.
Bertie Vidgen, Tristan Thrush, Zeerak Waseem, and Douwe Kiela. 2021b. Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1667-1682, Online. Association for Computational Linguistics.
Michael Wiegand and Josef Ruppenhofer. 2021. Exploiting Emojis for Abusive Language Detection. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 369-380, Online. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen,
Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. HuggingFace's Transformers: State-of-the-art Natural Language Processing. arXiv:1910.03771 [cs].
Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019. SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval). In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 75–86, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
Xiang Zhou, Yixin Nie, Hao Tan, and Mohit Bansal. 2020. The Curse of Performance Instability in Analysis Datasets: Consequences, Source, and Suggestions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8215-8228, Online. Association for Computational Linguistics.
Yukun Zhu, Ryan Kiros, Richard Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books. arXiv:1506.06724 [cs].
A Ethical Considerations
Misuse: We release two datasets of challenging emoji examples on which commercial solutions and state-of-the-art transformer models have been proven to fail. Malicious actors could take inspiration for bypassing current detection systems on internet platforms, or in principal train a generative hate speech model. This concern is also raised by Vidgen et al. (2021b), but the conclusion is reached that such risk of misuse is small and outweighed by the considerable scientific and social benefits.
Harm Statement: Following the advice of Derczynski et al. (2022), we describe the risks to wellbeing during the production and publication of this research and the steps taken to mitigate them. There is a risk of harm to data subjects i.e., the targets of hate, from reinforcing hateful, dehumanizing or derogatory statements, and to readers viewing this content in the paper. To mitigate these harms, we include a content warning directly after the abstract, at least a page before any harmful content is displayed and colored in red for maximum visibility. We also include a section-specific content warning before §2.2, which includes a number of hateful examples. Hateful examples in §2.2 are consistently formatted and visually distanced with gray text. All other examples (including Tab. 1, Tab. 4) are presented with placeholders for the [IDENTITY]. Hateful slurs are starred out with an asterisk where possible (e.g., Tab. 5) but we cannot star out emoji without hindering interpretation. There is also a risk to researchers and annotators from labeling and viewing harmful content. As authors, we oppose the use of hateful language. We follow protocols for protecting annotator wellbeing, including briefing sessions, regular check-ins and provision of mental health support.
B Data Statement for HATEMOJICHECK
We provide a data statement (Bender and Friedman, 2018) to document the generation and provenance of HATEMOJICHECK.
B.1 Curation Rationale
To construct HATEMOJICHECK, we hand-crafted 3,930 short-form English-language texts using a template-based method for group identities and slurs. Each test case exemplifies one functionality and is associated with a binary gold standard label (hate versus not hate). All 3,930 cases were labeled
by a trained team of three annotators, who could also flag examples that were unrealistic. Any test cases with multiple disagreements or flags were replaced with alternative templates and re-issued for annotation to improve the quality of examples in the final set of test cases. The purpose of HATEMOJICHECK is to evaluate the performance of black-box models against varied constructions of emoji-based hate.
B.2 Language Variety
The test cases are in English. This choice was motivated by the researchers' and annotators' expertise, and to maximize HATEMOJICHECK's applicability to previous hate speech detection studies, which are predominantly conducted on English-language data. We discuss the limitations of restricting HATEMOJICHECK to one language and suggest that future work should prioritize expanding the test suite to other languages.
B.3 Speaker Demographics
All test cases were hand-crafted by the lead author, who is a native English-speaking researcher at a UK university with extensive subject matter expertise in online harms.
B.4 Annotator Demographics
We recruited a team of three annotators who worked for two weeks in May 2021 and were paid £16/hour. All annotators were female and between 30-39 years old. One had an undergraduate degree, one a taught graduate degree and one a postgraduate research degree. There were three nationalities: Argentinian, British and Iraqi, two ethnicities: White and Arab, and three religious affiliations: Catholic, Muslim and None. One annotator was a native English speaker and the others were non-native but fluent. All annotators used emoji and social media more than once per day. All annotators had seen others targeted by abuse online, and one had been targeted personally.
B.5 Speech Situation
The modality of all test cases is written text embedded with emoji. The empirical investigation using Twitter's streaming API was conducted between 1st-14th April 2021. The first set of test cases was created between 26th April-7th May 2021. The first round of annotation ran between 7th-14th May 2021. The second round of cases was created and re-issued between 14th-21st May 2021.
B.6 Text Characteristics
The genre of texts is hateful and non-hateful statements using emoji constructions. Renderings of emoji vary by operating system and browser providers. The renderings in this paper are from WhatsApp. The composition of the dataset by labels (hate versus not hate) and by set (originals versus perturbations) is described in Tab. 3 of the main paper.  $68\%$  of the test cases are hateful. HATEMOJICHECK has 644 cases for Muslims, 608 cases each for gay people, disabled people and women, 582 cases for Black people and 566 cases for trans people. identity perturbations switch a protected identity for a non-protected identity so 314 cases have no protected identity tag.
C Constructing HATEMOJICHECK
C.1 List of Hateful Slurs in HATEMOJICHECK
The slurs used in HATEMOJICHECK are selected in the same way as for HATECHECK (Röttger et al., 2021). For each of the six identities in HATEMOJICHECK, the most common slurs are taken from hatebase.org, an online crowd-sourced repository of hateful terms and slurs. The top three most common slurs logged by users of hatebase are used, unless the 3rd ranked slur is significantly less common than the 2nd ranked. Each identity thus has two or three associated slurs (Tab. 5).
C.2 Defining Perturbations
Each template has three perturbations but the mapping of original test cases to perturbations is not one-to-one. Instead, there is a set of original templates contrasted by three sets of perturbation templates. The perturbation templates have fewer degrees of freedom than the original templates so test cases generated from them are fewer in number. Within each perturbed template we use the same emoji for substitutions to ensure consistency, but across templates, we use a range of emoji to ensure linguistic variety.
The perturbed templates are constructed as follows. (1) Identity perturbations: the protected identity (which could be an emoji or a word depending on the functionality) in each original template is substituted with one non-protected entity (which could be an emoji or a word). (2) Polarity perturbations: the negative term (which could be a word or an emoji) in each original template is substituted for one positive term (which could be a word or
Table 5: Hateful slurs in HATEMOJICHECK
Protected Group	Slurs
Women	b*tch, sl*t, wh*re
Trans people	tr*nny, sh*male
Gay people	f*ggot, f*g, q*eer
Black people	n*gger, c*on
Disabled people	r*tard, cr*pple, m*ng
Muslims	m*zzie, J*hadi, camelf*cker
an emoji). (3) No emoji perturbations: all emoji elements of the original template are replaced by equivalent text where one word is used to cover all versions e.g. and are both substituted for the verb 'harm'. For F5 and F6, the appended emoji is removed not replaced.
D Data Statement for HATEMOJIBUILD
We provide a data statement (Bender and Friedman, 2018) to document the generation and provenance of HATEMOJIBUILD.
D.1 Curation Rationale
We use an online interface designed for dynamic dataset generation and model benchmarking (Dynabench) to collect synthetic adversarial examples in three successive rounds, running between 24th May-11th June. Each round contains  $\sim 2,000$  entries, where each original entry inputted to the interface is paired with an offline perturbation. Data was synthetically-generated by a team of trained annotators, i.e., not sampled from social media.
D.2 Language Variety
All entries are in English. Language choice was dictated by the expertise of researchers and annotators. Furthermore, English is used for a wide number of benchmark hate speech datasets (Davidson et al., 2017; Founta et al., 2018) and was also used in the adversarial dataset for textual hate speech (Vidgen et al., 2021b). The method be could be adapted for other languages in future work.
D.3 Speaker Demographics
All entries are synthetically-created by annotators so the speaker demographics match the annotator demographics.
D.4 Annotator Demographics
Ten annotators were recruited to work for three weeks, and paid £16/hour. An expert annotator was recruited for quality control purposes and paid £20/hour. In total, there were 11 annotators. All
annotators received a training session prior to data collection and had previous experience working on hate speech projects. A daily 'stand-up' meeting was held every morning to communicate feedback and update guidelines as rounds progressed. Annotators were able to contact the research team at any point using a messaging platform. Of 11 annotators, 8 were between 18–29 years old and 3 between 30–39 years old. The completed education level was high school for 3 annotators, undergraduate degree for 1 annotator, taught graduate degree for 4 annotators and post-graduate research degree for 3 annotators. 6 annotators were female, and 5 were male. Annotators came from a variety of nationalities, with 7 British, as well as Jordanian, Irish, Polish and Spanish. 7 annotators identified as ethnically White and the remaining annotators came from various ethnicities including Turkish, Middle Eastern, and Mixed White and South Asian. 4 annotators were Muslim, and others identified as Atheist or as having no religious affiliation. 9 annotators were native English speakers and 2 were non-native but fluent. The majority of annotators (9) used emoji and social media more than once per day. 10 annotators had seen others targeted by abuse online, and 7 had been personally targeted.
D.5 Speech Situation
Entries were created from 24th May-11th June 2021. Their modality is short-form written texts embedded with emoji. Entries are synthetically-generated but annotators were trained on real-world examples of emoji-based hate from Twitter.
D.6 Text Characteristics
The genre of texts is hateful and non-hateful statements using emoji constructions. Annotators inputted emoji into the platform using a custom emoji selector.[10] The composition of the final dataset is described in Tab. 6.  $50\%$  of the 5,912 test cases are hateful.  $50\%$  of the entries in the dataset are original content and  $50\%$  are perturbations.
E Constructing HATEMOJIBUILD
Types of Hate We adopt the same categorization used by Vidgen et al. (2021b, p.3).<sup>11</sup> There are four types of hate. Derogation: Language which explicitly derogates, demonizes, demeans or insults
Table 6: Summary statistics across three rounds of data from HATEMOJIBUILD.
R5	R6	R7
n	1994	1966	1952
Split, n (%)	Train	1595 (80.0)	1572 (80.0)	1561 (80.0)
Dev	199 (10.0)	197 (10.0)	195 (10.0)
Test	200 (10.0)	197 (10.0)	196 (10.0)
Label, n (%)	Hate	1006 (50.5)	983 (50.0)	976 (50.0)
Not hate	988 (49.5)	983 (50.0)	976 (50.0)
Set, n (%)	Original	997 (50.0)	983 (50.0)	976 (50.0)
Perturbation	997 (50.0)	983 (50.0)	976 (50.0)
Type, n (%)	None	988 (49.5)	983 (50.0)	976 (50.0)
Derogation	718 (36.0)	649 (33.0)	594 (30.4)
Animosity	74 (3.7)	219 (11.1)	275 (14.1)
Threatening	101 (5.1)	50 (2.5)	52 (2.7)
Dehumanizing	113 (5.7)	65 (3.3)	55 (2.8)
# Emoji, μ (σ)	1.7 (2.2)	1.7 (1.0)	1.6 (1.1)
a group. Animosity: Expressions of abuse through implicit statements or mockery, where a logical step must be taken between the sentence and its intended negativity. Threatening language: Statements of intent to take action against a group, with the potential to inflict serious or imminent harm on its members. Dehumanizing language: Comparing groups to insects, animals, germs or trash.
Targets of Hate Annotators were provided with a non-exhaustive list of high-priority identities to focus on which included categorizations by gender identity (e.g., women, trans), sexual orientation (e.g., gay), ethnicity (e.g., Hispanic people), religion (e.g., Sikh), nationality (e.g., Polish), disability and class, alongside intersections (e.g., Muslim women). Hate directed towards majority groups (e.g., men, white people and heterosexuals) is outside the remit of this work. The explicit decision not to focus on issues such as 'reverse racism' (Bax, 2018) is made due to the complex debate on its inclusion in hate speech definitions.
Composition Tab. 7 reports the inter-annotator agreement and number of final entries per round. Each round has approximately the same number of entries with slightly fewer in R7 due to more quality control issues. Labels are equally distributed in each round (see Tab. 6). For 75 pairs of originals and perturbations, the perturbation unsuccessfully flipped the label given by majority agreement between three annotators. All other pairs have opposite labels. Derogation is always the most
10https://hatemoji.stackblitz.io/
<sup>11</sup>One of their categories "support for hateful entities" is excluded because it introduced confusion and ambiguity.
Table 7: Quality control checks per round of data collection, showing Randolph-kappa score across three annotators per entry  $R(S)$ , entries reviewed by an expert annotator, and dropped entries.
Input Entries R(S) Reviewed Dropped Final Entries
R5	Original	1020	0.903	273	6	1014
Perturbation	1014	0.938	131	17	997
R6	Original	1008	0.902	189	8	1000
Perturbation	1000	0.911	167	17	983
R7	Original	993	0.913	169	12	981
Perturbation	981	0.928	168	5	976
commonly inputted form of hate. From R5 to R7, there is a rise in animosity entries paired with a decline in threatening and dehumanizing language entries. Annotators were given substantial freedom in the targets of hate resulting in 54 unique targets, and 126 unique intersections of these. The entries from R5-R7 contain 1,082 unique emoji out of 3,521 defined in the Unicode Standard as of September 2020. The mode of emoji per entries is 1 and mean is approximately 1.5 in each round. The frequency of targets and emoji follow a long-tailed distribution, similar to a Zipf curve. These distributions match those found online for targets (Silva et al., 2016) and for emoji (Felbo et al., 2017; Cappallo et al., 2019; Bick, 2020).
In the first round, annotators commonly employed a strategy of substituting identities for emoji so three identity emojis were frequently used (♂, ♂, ♂). Using emoji character substitutions in slurs was also a successful strategy (A: ⌊), as was substituting the entire slur for an emoji homonym (f*g): ⌊). In the final round, the model was wise to such strategies, and so annotators changed their adversarial techniques to irony, satire or mockery, as reflected in the top emojis (♂, ⌊, ⌊, ⌊). For a qualitative understanding of annotators' experiences, we conducted a post-study survey.12
F Target Models
In each round we assessed two candidate model architectures. The first is an uncased DeBERTa base model (134M params) with a sequence classification head (He et al., 2021). DeBERTa has been shown to improve on BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) models by incorporating a disentangled attention mechanism and enhanced mask decoder. However, emoji are likely relatively sparse in DeBERTa's pre-training
material which includes English Wikipedia, Book Corpus (Zhu et al., 2015) and a subset of Common-Crawl. DeBERTa uses the BPE-encoding method so the tokenizer can encode emoji as unique tokens. The second candidate is an uncased BERTweet model (135M params) with a sequence classification head where the RoBERTa training procedure was repeated on 850M English Tweets using a custom vocabulary (Nguyen et al., 2020). BERTweet tokenizes emoji by first translating them to text representations using the emoji package. These word representations are derived from the Unicode Standard, so for example  $\bullet$  becomes :crying_face::.
Upsampling ratios are evaluated for each new round of training data with multipliers of 1, 5, 10, 100. For the R1-4 data, we carry forward the upsampling from Vidgen et al. (2021b): R1 is upsampled five times, R2 is upsampled 100 times, R3 is upsampled once, and R4 is upsampled once. Combining the model architectures and upsampling ratios gives 8 candidate models for each round's target model. All models were implemented using the transformers library (Wolf et al., 2020). All models were trained for 3 epochs with early stopping based on the dev set loss, a learning rate of  $2e - 5$  and a weighted Adam optimizer. Other hyperparameters were set to HuggingFace defaults. We train and evaluate each model once and report results for this single run. Training took approximately 7 hours for each BERTweet model and 15 hours for each DeBERTa model using 8 NVIDIA Tesla V100 GPU on the JADE2 supercomputer.
The best target model for each round is selected by weighted accuracy between all prior rounds and the current round. For R6-T, a DeBERTa model with 100x upsampling on R5 performs best. Given the dearth of emoji in R1-R4, it is unsurprising a large upsample improves performance on emoji-based hate. For R7-T, DeBERTa with one upsample on R6 performs best. For R8-T, DeBERTa with five upsamples on R7 performs best. In all rounds, DeBERTa significantly outperforms BERTweet, while the upsampling ratio less substantially affects performance. This suggests first converting emoji to their text representations does not substantially aid performance, though there may be other differences between the models driving results. We upload models to Dynalab for model-in-the-loop evaluation and data collection (Ma et al., 2021).
12This survey is on our Github.