SlowGuess/NAACL_2022_OCR huggingface dataset:
For F1 Verb Swap, P-IA and R5-T perform well on original statements, but then perform poorly on the polarity perturbations (0.20 and 0.42 accuracy, respectively). Our models have much stronger performance on polarity perturbations (between 0.73 and 0.77), and comparably high performance on the other sets of test cases. For F2 Identity Swap, P-IA and R5-T perform very poorly on original statements (0.14 and 0.33 accuracy, respectively) but then perform well on the polarity perturbations and no emoji perturbations. This vulnerability is carried forward to performance on F4 Double Swap. R5-T only achieves 0.03 and P-IA makes zero correct
predictions. In contrast, our models achieve accuracy of 0.83 on F2 and accuracies between 0.70 and 0.79 on F4. For F3 Descriptor Swap, our models improve over P-IA and R5-T on the original statements. The relative improvement is particularly large for the polarity perturbations (0.25 for P-IA compared with 0.93 for R8-T). For F5 Append, P-IA and R5-T perform moderately with accuracies of 0.73 and 0.69. Our models perform far better, with accuracies from 0.87 to 0.99. However, they do not show a substantial increase in performance for the no emoji perturbations in F5.3. For F6 Positive Confounder, our models perform worse than P-IA and R5-T on the original statements, but then have far more consistent performance across the perturbations. For instance, R8-T achieves accuracies between 0.89 to 0.93 across all sets of test cases in this functionality, compared with a range of 0.44 to 0.93 for P-IA. P-IA and R5-T perform well on original statements and poorly on perturbations precisely because they ignore the effect of the emoji confounder. F7 Emoji Leetspeak is a successful adversarial strategy. R5-T achieves 0.85 accuracy on the original statements but just 0.67 on the identity perturbations, while P-IA does even worse, with only 0.59 on the original statements and 0.57 on the polarity perturbations. Our models perform better on the original statements, but still struggle with identity perturbations, showing how challenging this functionality is. However, non-hateful leetspeak constructions are likely less prevalent on social media than slur-based leetspeak.
Overall, all models trained on HATEMOJIBUILD perform well on emoji-based hate, but there is no clear 'best' model among them. After the large improvement from R5-T to R6-T, subsequent models produce minimal performance gains across the functionalities. This suggests the gains, as measured on HATEMOJICHECK's intentionally short and simple statements, are quickly saturated.
4.2 Model Performance by Target Group
HATEMOJICHECK contains target group labels for hateful and non-hateful entries, allowing models to be compared by their subgroup fairness. However, defining fairness is a non-trivial task, and there is substantial debate in the machine learning community as to which metrics are most appropriate. We consider six metrics calculated across subgroups: accuracy, precision, recall, false positive rate, false
Table 4: Accuracy across HATEMOJICHECK's seven functionalities and their associated perturbation sets.
Functionality	Label	Example Test Case Template	n	Pre-Emoji	Post-Emoji
P-IA	R5-T	R6-T	R7-T
F1	Verb Swap	hate	I want to / all [IDENTITY]	300	0.94	0.85	0.84	0.79
F1.1	Identity perturbation	not hate	I want to / all bigots	50	0.94	1.00	1.00	0.98
F1.2	Polarity perturbation	not hate	I want to / all [IDENTITY]	60	0.20	0.42	0.75	0.77
F1.3	No emoji perturbation	hate	I want to kill all [IDENTITY]	60	1.00	0.95	0.97	0.98
emoji diff:	-0.06	-0.10	-0.12	-0.18
F2	Identity Swap	hate	I hate [IDENTITY EMOJI]	120	0.14	0.33	0.83	0.83
F2.1	Identity perturbation	not hate	I hate	20	0.90	0.70	1.00	0.85
F2.2	Polarity perturbation	not hate	I love [IDENTITY EMOJI]	120	1.00	0.98	0.78	0.88
F2.3	No emoji perturbation	hate	I hate [IDENTITY]	120	0.98	0.98	0.97	1.00
emoji diff:	-0.84	-0.66	-0.13	-0.17
F3	Descriptor Swap	hate	[IDENTITY] are nothing but	260	0.92	0.83	0.99	0.99
F3.1	Identity perturbation	not hate	Bullies are nothing but	40	1.00	1.00	1.00	1.00
F3.2	Polarity perturbation	not hate	[IDENTITY] are nothing but	60	0.25	0.48	0.78	0.82
F3.3	No emoji perturbation	hate	[IDENTITY] are nothing but vermin	60	0.98	1.00	1.00	1.00
emoji diff:	-0.06	-0.17	-0.01	-0.01
F4	Double Swap	hate	I will [IDENTITY EMOJI]	288	0.00	0.03	0.79	0.70
F4.1	Identity perturbation	not hate	I will	46	1.00	1.00	0.91	0.98
F4.2	Polarity perturbation	not hate	I will / [IDENTITY EMOJI]	60	1.00	0.98	0.92	0.85
F4.3	No emoji perturbation	hate	I will hurt [IDENTITY]	60	0.97	1.00	1.00	1.00
emoji diff:	-0.96	-0.97	-0.21	-0.30
F5	Append	hate	My new manager is [IDENTITY]	288	0.73	0.69	0.99	0.89
F5.1	Identity perturbation	not hate	My new manager is a cat-lover	48	1.00	1.00	1.00	1.00
F5.2	Polarity perturbation	not hate	My new manager is [IDENTITY]	60	0.47	0.55	0.85	0.78
F5.3	No emoji perturbation	not hate	My new manager is [IDENTITY]	60	0.35	0.48	0.45	0.43
emoji diff:	†	†	†	†
F6	Positive Confounder	hate	I find all [IDENTITY] disgusting	440	0.93	1.00	0.85	0.90
F6.1	Identity perturbation	not hate	I find all reptiles disgusting	65	0.75	0.92	0.92	0.92
F6.2	Polarity perturbation	not hate	I find all [IDENTITY] inspiring	112	0.44	0.88	0.96	0.91
F6.3	No emoji perturbation	hate	I find all [IDENTITY] disgusting	88	0.93	0.99	0.95	1.00
emoji diff:	0.00	0.01	-0.10	-0.10
F7	Emoji Leetspeak	hate	I hate [SLUR_EMOJI]	430	0.59	0.85	0.91	0.83
F7.1	Identity perturbation	not hate	I hate dōgs	45	0.82	0.67	0.62	0.64
F7.2	Polarity perturbation	not hate	I hate people who say [SLUR_EMOJI]	430	0.57	0.79	0.79	0.86
F7.3	No emoji perturbation	hate	I hate [SLUR]	140	0.61	0.99	0.99	0.99
emoji diff:	-0.03	-0.14	-0.09	-0.11
Notes:  $\dagger$  The emoji difference is not defined because the no emoji perturbation is non-hateful, i.e., the opposite label to the original test case, so accuracy over these sets cannot be fairly compared.
Figure 2: Subgroup fairness metrics by model. Abbreviations respectively refer to Muslims, Black people, Disabled people, Gay people, Trans people and Women.
negative rate and selection rate. $^{9}$  Fig. 2 shows subgroup fairness metrics compared across three models: P-IA, R5-T and R8-T. It is concerning that P-IA has such differential performance, especially regarding the unbalanced false positive and negative rates for women and disabled people, as both of these error types are societally harmful. R8-T has more balanced accuracy, precision, recall and selection rate across subgroups, driven by stable performance in false negatives and positives. In Appendix H we provide two further between-group fairness metrics: demographic parity ratio (Agarwal et al., 2019), and equalized odds ratio (Hardt et al., 2016; Agarwal et al., 2018). There is also substantial improvement in these metrics from adversarial training schemes relative to commercial solutions or statically-trained models.
HATEMOJICHECK reveals critical model weaknesses in detecting emoji-based hate. Existing commercial and academic models perform poorly at identifying hate where the identity term has been replaced with an emoji representation (F2 and F4), even though they perform well at identifying the equivalent textual statements. These models have better performance on Verb Swap (F1) but then struggle with the polarity perturbations (F1.2). This suggests that the models are overfitting to the identity term and ignore the sentiment from the emoji, leading to false positive predictions. Our newly trained models substantially improve performance on original hateful statements from F2, F4 and F5, indicating they have a better semantic grasp of emoji substitutions and appends. They also make large performance gains on the polarity perturbations in F1.2, F3.2, F5.2 and F6.2, suggesting they better incorporate information on how different emoji condition the likelihood of harmfulness. Despite improving on existing models, our models still perform relatively more poorly on F4, as well as the F1.2, F5.3 and F7.1 perturbations. These weaknesses could potentially be addressed in future work through additional data generation.
Training on just one round of adversarial data yields the biggest improvement on HATEMOJI-CHECK. Thereafter, performance plateaus. This aligns with the sharp increase and then fall in MER
across emoji-based rounds of data generation (in Fig. 1). On the adversarial test sets, R8-T only marginally outperforms R6-T and R7-T on all R1-7 test sets. It slightly underperforms on the R5-7 test sets from HATEMOJIBUILD. This suggests that while training on a relatively small number of entries can substantially improve performance, the gains quickly saturate. Future work could investigate performance differences in more detail using the labels for type and target of hate that we provide for all three rounds of data.
Due to practical constraints, HATEMOJICHECK has several limitations, which could be addressed in future work. First, it includes relatively simplistic statements, and so offers negative predictive power: high performance only indicates the absence of specific weaknesses (Gardner et al., 2020; Röttger et al., 2021). Second, it is inspired by real-world hate but synthetically generated. Future work could evaluate against more complex and diverse forms of emoji-based hate. Third, it is limited in scope. It only considers short English-language statements with one binary label. Only six identities are included, none of which are intersectional, and the set of emoji covers a fraction of the full Unicode Standard. In-scope identities and emoji usage are culturally-dependent (Barbieri et al., 2016; Ljubesić and Fiser, 2016) so future work could assess the generalizability of HATEMOJICHECK to other cultures and languages. These limitations do not diminish HATEMOJICHECK's utility as a tool for effectively identifying model weaknesses to emoji-based hate. By publicly releasing the test suite, practitioners and academics can scrutinize their models prior to deployment or publication.
Adversarial data generation is a powerful technique for creating diverse, complex and informative datasets. However, it also introduces challenges. First, the entries are 'synthetic' rather than sampled from 'real-world' examples. Substantial training and time are required to ensure that annotators understand real online hate and can imitate it. Second, annotators can exhaust their creativity and start producing unrealistic, simplistic or non-adversarial examples. Third, because of the need for training, supervision and support, only a small pool of annotators is feasible, which can introduce additional idiosyncratic biases. Given these issues, quality control is a key consideration throughout the data generation process. Encouragingly, we find carefully-curated and adversarially-generated
Selection rate is  $\mathbb{P}[h(x)|A = a]$ , i.e., the proportion of the population in subgroup  $a$  who have 1 (hate) as the given class label. The 'true' ratio of class labels is shown in Fig. 2
training datasets can significantly improve performance on emoji-based hate as a particular type of challenging content, and that this approach is effective with relatively few training examples. Thus, substantial model improvements can be realized with minimal financial and computational cost.
Emoji-based hate has received limited attention in prior work on hate and abusive language detection. Studies that attend to emoji often do so as a potential input feature to aid classifier performance. For example, Samghabadi et al. (2019) improve offensive language classification with an 'emotion-aware' mechanism built on emoji embeddings. Ibrohim et al. (2019) find that adding emoji features to Continuous Bag of Words and word unigram models marginally improves performance for abusive language detection on Indonesian Twitter. Bick (2020) identifies examples of subtle and non-direct hate speech in German-Danish Twitter conveyed through 'winking' or 'skeptical' emoji that flag irony or non-literal meaning in their accompanying text. Corazza et al. (2020) train an emoji-based Masked Language Model (MLM) for zero-shot abuse detection. They show this method improves performance on classifying abuse in German, Italian and Spanish tweets compared to an MLM which does not attend to emoji. Wiegand and Ruppenhofer (2021) use abusive emoji as a proxy for learning a lexicon of abusive words. Their findings indicate that emoji can disambiguate abusive and profane usages of words such as f\*ck and b\*tch. By contrast, our work focuses on emoji-based hate as a challenge for hate detection models. With HATEMOJICHECK, we enable a systematic evaluation of how well models handle different types of emoji-based hate. Rather than adjusting the model architecture, we account for emoji-based hate in our iterative data generation process and show that models trained on such data perform better on emoji-based hate, while retaining strong performance on text-only hate.
As a suite of functional tests for evaluation, HATEMOJICHECK directly builds on previous work by Ribeiro et al. (2020) and Röttger et al. (2021). Ribeiro et al. (2020) introduced functional tests as a framework for NLP model evaluation with CHECKLIST, showing that their approach can identify granular model strengths and weaknesses that are obscured by high-level metrics like accuracy
and F1-score. Röttger et al. (2021) adapted this framework to hate detection with HATECHECK, which covers 29 model functionalities motivated by interviews with civil society stakeholders and a review of previous hate speech literature. Like HATECHECK, we pair hateful test cases with contrasting perturbations that are particularly challenging to models relying on overly simplistic decision rules and thus reveal granular decision boundaries. HATECHECK did not consider emoji-based hate, which is the main focus of our work.
Our approach to training better models for emoji-based hate builds directly on work by Vidgen et al. (2021b), who apply iterative human-and-model-in-the-loop training to hate detection models. Like us, they used the Dynabench interface (Kiela et al., 2021) to implement their training system, which has also been used to improve model performance for other tasks such as reading comprehension (Bartolo et al., 2020) and sentiment analysis (Potts et al., 2021). Earlier work by Dinan et al. (2019) introduced a similar 'build it, break it, fix it' system of repeated interactions between a hate classifier and crowdworkers to develop safe-by-design chatbots. Unlike previous work, we focus data generation on a particular type of hateful content, emoji-based hate, and show that the training scheme can address specific model weaknesses on such content without sacrificing performance on text-only hate.
Online hate is a pervasive, harmful phenomenon, and hate detection models are a crucial tool for tackling it at scale. We showed that emoji pose a particular challenge for such models, and presented HATEMOJICHECK, a first-of-its-kind evaluation suite for emoji-based hate. It covers seven functionalities with 3,930 test cases. Using this test suite, we exposed clear weaknesses in the performance of commercial and academic models on emoji-based hate. To address these weaknesses, we created the HATEMOJIBUILD dataset using an innovative human-and-model-in-the-loop approach. We showed that models trained on this adversarial data are substantially better at detecting emoji-based hate, while retaining strong performance on text-only hate. Our approach of first identifying granular model weaknesses, and then generating an adversarial dataset to address them presents a promising direction for building models to detect other diverse and emerging forms of online harm.
We are thankful for support that the Oxford authors received to facilitate annotation and computational resources from the Volkswagen Foundation, Meedan, Keble College, the Oxford Internet Institute, Rewire and The Alan Turing Institute. We owe a debt of gratitude to all our annotators, to Dynabench and to our anonymous reviewers. We are also grateful to Zeerak Talat and Douwe Kiela for their helpful advice on this research, as well as support from Devin Gaffney and Darius Kazemi. Hannah Rose Kirk was supported by the Economic and Social Research Council grant ES/P000649/1. Paul Röttger was supported by the German Academic Scholarship Foundation.
Alekh Agarwal, Alina Beygelzimer, Miroslav Dudík, John Langford, and Hanna Wallach. 2018. A Reductions Approach to Fair Classification. arXiv:1803.02453 [cs].
Alekh Agarwal, Miroslav Dudik, and Zhiwei Steven Wu. 2019. Fair Regression: Quantitative Definitions and Reduction-Based Algorithms. In Proceedings of the 36th International Conference on Machine Learning, pages 120-129. PMLR.
Francesco Barbieri, German Kruszewski, Francesco Ronzano, and Horacio Saggion. 2016. How Cosmopolitan Are Emojis? Exploring Emojis Usage and Meaning over Different Languages with Distributional Semantics. In Proceedings of the 24th ACM International Conference on Multimedia, MM '16, pages 531-535, New York, NY, USA. Association for Computing Machinery.
Max Bartolo, Alastair Roberts, Johannes Welbl, Sebastian Riedel, and Pontus Stenetorp. 2020. Beat the AI: Investigating adversarial human annotation for reading comprehension. Transactions of the Association for Computational Linguistics, 8:662-678.
Anna Bax. 2018. "The C-Word" Meets "the N-Word": The Slur-Once-Removed and the Discursive Construction of "Reverse Racism". Journal of Linguistic Anthropology, 28(2):114-136.
Boris Beizer. 1995. Black-Box Testing: Techniques for Functional Testing of Software and Systems. John Wiley & Sons, Inc., New York, NY, USA.
Emily M. Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587-604.
Eckhard Bick. 2020. Annotating emoticons and emojis in a german-danish social media corpus for hate speech research. RASK, 52.
Brandwatch. 2018. The Emoji Report.
Spencer Cappallo, Stacey Svetlichnaya, Pierre Garrigues, Thomas Mensink, and Cees G. M. Snoek. 2019. New Modality: Emoji Challenges in Prediction, Anticipation, and Retrieval. IEEE Transactions on Multimedia, 21(2):402-415.
Michele Corazza, Stefano Menini, Elena Cabrio, Sara Tonelli, and Serena Villata. 2020. Hybrid Emoji-Based Masked Language Models for Zero-Shot Abusive Language Detection. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 943-949, Online. Association for Computational Linguistics.
Juliet M. Corbin and Anselm Strauss. 1990. Grounded theory research: Procedures, canons, and evaluative criteria. Qualitative Sociology, 13(1):3-21.
Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated Hate Speech Detection and the Problem of Offensive Language. Proceedings of the International AAAI Conference on Web and Social Media, 11(1):512-515.
Leon Derczynski, Hannah Rose Kirk, Abeba Birhane, and Bertie Vidgen. 2022. Handling and Presenting Harmful Text. arXiv:2204.14256 [cs].
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Emily Dinan, Samuel Humeau, Bharath Chintagunta, and Jason Weston. 2019. Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4537-4546, Hong Kong, China. Association for Computational Linguistics.
Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2018. Measuring and Mitigating Unintended Bias in Text Classification. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, AIES '18, pages 67-73, New York, NY, USA. Association for Computing Machinery.
Bjarke Felbo, Alan Mislove, Anders Søgaard, Iyad Rahwan, and Sune Lehmann. 2017. Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm.
In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1615-1625, Copenhagen, Denmark. Association for Computational Linguistics.
Antigoni-Maria Founta, Constantinos Djouvas, Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Gianluca Stringhini, Athena Vakali, Michael Sirivianos, and Nicolas Kourtellis. 2018. Large Scale Crowdsourcing and Characterization of Twitter Abusive Behavior. In Twelfth International AAAI Conference on Web and Social Media.
Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hannaneh Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, and Ben Zhou. 2020. Evaluating Models' Local Decision Boundaries via Contrast Sets. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1307-1323, Online. Association for Computational Linguistics.
Katharine Gelber and Luke McNamara. 2016. Evidencing the harms of hate speech. Social Identities, 22(3):324-341.
Tarleton Gillespie. 2020. Content moderation, AI, and the question of scale. *Big Data and Society*, 7(2).
Moritz Hardt, Eric Price, and Nati Srebro. 2016. Equality of Opportunity in Supervised Learning. In Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. DeBERTa: Decoding-enhanced BERT with Disentangled Attention. arXiv:2006.03654 [cs].
Muhammad Okky Ibrohim, Muhammad Akbar Setiadi, and Indra Budi. 2019. Identification of hate speech and abusive language on Indonesian Twitter using the Word2vec, part of speech and emoji features. In Proceedings of the International Conference on Advanced Information Science and System, AISS '19, pages 1-5, New York, NY, USA. Association for Computing Machinery.
Alastair Jamieson. 2020. Racist comments after Euro 2020: Saka, Sancho and Rashford racially abused online after England defeat.
Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. 2021. Dynabench: Rethinking Benchmarking in NLP. In Proceedings of the 2021 Conference of the North