SlowGuess/NAACL_2022_OCR huggingface dataset:
HATEMOJI: A Test Suite and Adversarily-Generated Dataset for Benchmarking and Detecting Emoji-Based Hate
Hannah Rose Kirk $^{1,2\ddagger}$ , Bertie Vidgen $^{1,2}$ , Paul Röttger $^{1,2}$ , Tristan Thrush $^{3*}$ , Scott A. Hale $^{1,2,4}$
<sup>1</sup>University of Oxford, <sup>2</sup>The Alan Turing Institute, <sup>3</sup>Hugging Face, <sup>4</sup>Meedan
$\ddagger$ hannah.kirk@oii.ox.ac.uk
Detecting online hate is a complex task, and low-performing models have harmful consequences when used for sensitive applications such as content moderation. Emoji-based hate is an emerging challenge for automated detection. We present HATEMOJICHECK, a test suite of 3,930 short-form statements that allows us to evaluate performance on hateful language expressed with emoji. Using the test suite, we expose weaknesses in existing hate detection models. To address these weaknesses, we create the HATEMOJIBUILD dataset using a human-and-model-in-the-loop approach. Models built with these 5,912 adversarial examples perform substantially better at detecting emoji-based hate, while retaining strong performance on text-only hate. Both HATEMOJICHECK and HATEMOJIBUILD are made publicly available. $^{1}$
Content Warning This article contains examples of hateful language from HATEMOJICHECK to illustrate its composition. Examples are quoted verbatim, except for slurs and profanity in text, for which the first vowel is replaced with an asterisk. The authors oppose the use of hateful language.
Online hate harms its targets, disrupts online communities, pollutes civic discourse and reinforces social power imbalances (Gelber and McNamara, 2016). The sheer scale of hateful content online has led to widespread use of automated detection systems to find, monitor and stop it (Vidgen et al., 2019; Gillespie, 2020). However, hateful content is complex and diverse, which makes it challenging to detection systems. One particular challenge is the use of emoji for expressing hate. Emoji are pictorial representations that can be embedded in text,
allowing complex emotions, actions and intentions to be displayed concisely (Rodrigues et al., 2018). Over  $95\%$  of internet users have used an emoji and 10 million are sent every day (Brandwatch, 2018). Following England's defeat in the Euro 2020 football final, there was widespread racist use of emoji such as  $\odot$ ,  $\diamond$  and  $\odot$  (Jamieson, 2020). This paper focuses on emoji-based hate, answering two research questions:
RQ1 What are the weaknesses of current detection systems for hate expressed with emoji?
RQ2 To what extent can human-and-model-in-the-loop training improve the performance of detection systems for emoji-based hate?
To answer RQ1, we present HATEMOJICHECK, a suite of functional tests for emoji-based hate. We provide 3,930 test cases for seven functionalities, covering six identities. 2,126 original test cases are matched with three types of challenging perturbations to enable accurate evaluation of model decision boundaries (Gardner et al., 2020). We use HATEMOJICHECK to assess Google Jigsaw's Perspective API as well as models trained on academic datasets, exposing critical model weaknesses.
To answer RQ2 and address the model weaknesses identified by HATEMOJICHECK, we implement a human-and-model-in-the-loop dynamic training scheme. We build on the work of Vidgen et al. (2021b), who used this approach for textual hate. Our work begins where their study ends. We conduct three rounds of adversarial data generation focused explicitly on emoji-based hate, tasking annotators to generate sentences that the model-in-the-loop misclassifies. This process yields a dataset of 5,912 entries, half of which are challenging contrasts. We call this dataset HATEMOJIBUILD. The dataset is evenly split between hate and non-hate, and each hateful entry has labels for the type and target. Between each round, the model-in-the-loop
*TT's work was done at Facebook AI Research.
<sup>1</sup>See our Github Repository. HATEMOJICHECK, HATEMOJIBUILD and the final HATEMOJI Model are also available on HuggingFace.
Human Language Technologies, pages 1352 - 1368
is re-trained so that annotators are trying to trick a progressively stronger and more 'emoji-aware' model. Relative to existing commercial and academic models, our models improve performance on the detection of emoji-based hate, without sacrificing performance on text-only hate.
We make several contributions: (1) we construct HATEMOJICHECK, which tests key types of emoji-based hate as separate functionalities, (2) we evaluate the performance of existing academic and commercial models at detecting emoji-based hate, (3) we present HATEMOJIBUILD, a labeled emoji-based hate speech dataset that is adversarially-generated for model training and (4) we train models that can accurately detect emoji-based hate. These data-centric contributions demonstrate the benefits of systematic and granular evaluation, and the need to diversify how hate detection systems are trained. We make both HATEMOJICHECK and HATEMOJIBUILD publicly available.
Definition of Hate We use the United Nations definition of hate speech: "any kind of communication in speech, writing or behavior, that attacks or uses pejorative or discriminatory language with reference to a person or a group on the basis of who they are, in other words, based on their religion, ethnicity, nationality, race, color, descent, gender or other identity factor" (United Nations, 2019, p.2).<sup>2</sup>
2 HATEMOJICHECK: Functional Tests for Emoji-Based Hate
The concept of functional testing, an expected input-output behavior (Beizer, 1995), has been adapted from software engineering for NLP tasks (Ribeiro et al., 2020), including hate speech detection (Röttger et al., 2021). These diagnostic tests identify model vulnerabilities to intentionally simple and non-ambiguous constructions of hate so comprise a minimal performance standard.
2.1 Identifying Functionalities
A functionality describes the ability of a model to provide a correct classification when presented with hateful or non-hateful content (Röttger et al., 2021). Each functionality has a set of corresponding test cases that share one gold-standard label. We select the functionalities in HATEMOJICHECK to be (1) realistic: they capture real-world uses of
emoji-based hate, (2) unique: each covers a distinct aspect of emoji-based hate, without overlaps between functionalities, and (3) unambiguous: they have clear gold-standard labels. The functionalities are motivated from two perspectives:
Previous Literature We identify distinct uses of emoji in online communications, particularly abuse. This includes appending emoji to the end of otherwise neutral statements to reveal emotive intent (Wiegand and Ruppenhofer, 2021), using emoji as a 'pictorial slur' to attack or dehumanize groups (Bick, 2020; Wiegand and Ruppenhofer, 2021) and substituting emoji for identity terms or threatening verbs (Samghabadi et al., 2019; Bick, 2020; Wiegand and Ruppenhofer, 2021). We also review a range of literature on online hate, identifying distinct types of hateful language (Talat et al., 2017; Zampieri et al., 2019; Vidgen et al., 2021a).
Empirical Investigation Based on the literature, we define a list of potentially hateful emoji and words, and use Twitter's Streaming API to search for the Cartesian products of emoji-emoji and emoji-word pairs over a two week period. To identify different forms of emoji-based hate, we apply a grounded theory approach (Corbin and Strauss, 1990) on a sample of 3,295 tweets, splitting out distinctive categories, and recursively selecting subcategories until all key parts of the data are captured and the framework is 'saturated'.
2.2 Functionalities in HATEMOJICHECK
This section presents verbatim examples of emoji-based hate (in gray text). For masked versions, see Tab. 1 and Tab. 4. HATEMOJICHECK has seven functionalities (see Tab. 1). F1 Verb Swap tests threatening hate where the verb is replaced with the equivalent emoji:  $\text{、}$ $\text{、}$ $\text{、}$ $\text{、}$ $\text{、}$  . It includes both direct threats and normative statements of threat. F2 Identity Swap tests derogatory hate where the identity term is replaced with an emoji representation: woman  $(\bullet)$  trans people  $(\bullet)$  gay people  $(\bullet)$  disabled people  $(\bullet)$  Black people  $(\bullet)$  and Muslims  $(\bullet)$ .3 F3 Descriptor Swap tests hate where the negative descriptor is replaced with the equivalent emoji. It includes both dehumanizing language and expressions of disgust. For dehumanizing language, five descriptors are used:  $\text{、}$ $\text{、}$ $\text{、}$ $\text{、}$  . Based on our Twitter analysis, target-specific descriptors are
${}^{3}$  was commonly used to denote Muslims on Twitter,despite its Unicode description being 'person wearing a urban'.
2We recognize the presence of annotator bias in cultural interpretations of this definition, and provide data statements for both datasets in the appendix.
Table 1: Summary of the seven functionalities in HATEMOJICHECK.
Functionality (Emoji Used)	Templates	Emoji	Identities	Cases
F1 Verb Swap: Swap threatening verb (√, ⌊, ⌊, ⌒)	10	5	6	300
F2 Identity Swap: Swap protected identity ( ⌊, ⌒, ⌒, ⌒, ⌒)	20	1	6	120
F3 Descriptor Swap: Swap descriptor noun/emotion ( ⌊, ⌒, ⌒, ⌒, ⌒, ⌒, ⌒, ⌒ / ⌒, ⌒, ⌒)	10	3-7	6	260
F4 Double Swap: Combine two of the above swaps	10	3-7	6	288
F5 Append: Append negative emoji to neutral text ( ⌊, ⌒, ⌒, ⌒, ⌒)	10	3-5	6	288
F6 Positive Confounder: Append positive emoji to hateful text (♥, ⌒, ⌒, ⌒, ⌒)	13	5	6	440
F7 Emoji Leetspeak: Substitute for character or word-piece ( ⌐, ⌐, ⌐, ⌐, ⌐, ⌐)	15	3-4	6 (+16 slurs)	430
Totals of Unique Templates, Emoji, Identity and Test Cases	88	36	6	2,126
used for Black people: (, ) and Muslims: (). For expressions of disgust, three descriptors are used: , , . F4 Double Swap tests a combination of F1 with either F2 or F3, where emoji are swapped for both the identity and the verb, or both the identity and the negative descriptor. Some templates contain only emoji e.g., “/”, while others have some text e.g., “make me”. F5 Append tests statements which would be non-hateful on their own but become hateful when appended with a negative emoji such as , , . For instance, “My new manager is a woman” is neutral but “My new manager is a woman ” is hateful. F6 Positive Confounder tests clearly hateful statements appended with a positive emoji such as , , . For instance, “Kill all women ” or “All women deserve to die”. F7 Emoji Leetspeak tests statements where parts of the hateful language have been obfuscated by replacing characters with emoji. The character substitution can occur in the term expressing negative sentiment, e.g., “I hate gays”, the identity, e.g., “I hate g ys” or the slur, e.g., “I hate f gs”. There are three emoji character substitutions: (1) a numeric substitution which mimics leetspeak (a: 4, i: 1, e: 3, o: 0), (2) a star mimicking * for vowels in slurs and profanities (vowel: ), and (3) a heart mimicking starring out vowels but in a more adversarial manner (vowel: ). To reflect specific cases on Twitter of anti-Muslim hate, two additional slurs are included where a word-piece is substituted with an emoji (f*cker, f*cker).
2.3 Test Cases in HATEMOJICHECK
HATEMOJICHECK is strongly inspired by empirical examples of emoji-based hate but we want it to comprise a clear, minimum performance standard. Thus, for each functionality we hand-craft test cases, i.e., short English-language text documents that clearly correspond to just one gold-standard label. The test cases are varied in their vocabulary and syntax to ensure they assess model robustness
to diverse language (Zhou et al., 2020). To generate test cases at scale and ensure balanced coverage across targeted groups, we use a template-based approach (Dixon et al., 2018; Ribeiro et al., 2020; Röttger et al., 2021). First, each template is given an identity placeholder e.g., "I hate [IDENTITY]" or a slur placeholder e.g., "You are just a [SLUR] to me". Six protected identities populate these placeholders: women, trans people, gay people, Black people, disabled people and Muslims. For templates with slurs, we use the 16 slurs compiled by Röttger et al. (2021) (see Appendix C.1). Second, each template is given an emoji placeholder that is populated by emoji that share semantic and syntactic properties, so that the test suite can be scaled across different emoji. For example, the template "I will [EMOJI] all [IDENTITY]" could be populated with {'', 'trans people'}. HATEMOJICHECK comprises 3,930 entries. There are 2,126 original test cases and 1,804 perturbations (see Tab. 1).
2.3.1 Perturbations
Each set of test cases is matched to a set of perturbations which are constructed through minimal edits to the original hateful template. Test cases generated from these perturbed templates share vocabulary and syntax with the original test cases, enabling fine-grained assessment of model weaknesses (Gardner et al., 2020). We provide three types of perturbations. For identity perturbations, the identity targeted by the original hateful statement is substituted for a non-protected entity, making it non-hateful. Possible identity perturbations include non-protected groups (e.g., "I hate accountants"), non-human entities (e.g., "I hate spiders"), and inanimate objects (e.g., "I hate pizza"). For polarity perturbations, the negative sentiment of the original hateful statement is reversed to be positive, making it non-hateful, without changing the target of the statement. Possible polarity per
4See Appendix C.2 for details on perturbations.
turbations include changing threatening language to supportive language, e.g., "kill" to "respect", changing expressions of hate to expressions of love, e.g., "I hate..." to "I love..." or replacing hateful use of slurs with counterspeech against the slurs, e.g., "[IDENTITY] are [SLUR]" to "[IDENTITY] should never be called [SLUR]". For no emoji perturbations, emoji are removed or replaced with their equivalent text to preserve the semantic expression (e.g., "♥" becomes "shoot" and "♥" becomes "love"). For most functionalities this perturbation preserves the original label of the test case, e.g., "[IDENTITY] makes me ⿱" is hateful, and its perturbation "[IDENTITY] makes me sick" is still hateful. However, for F5 Append the label changes because when the negative emoji appended to the neutral statement is removed, the part of the statement that remains is non-hateful.
2.4 Validating Test Cases
To validate the gold-standard labels assigned to each test case, we recruited three annotators with prior experience on hate speech projects. Annotators were given extensive and prescriptive guidelines (Röttger et al., 2022), as well as test tasks and training sessions, which included examining real-world examples of emoji-based hate from Twitter. We followed guidance for protecting annotator well-being (Vidgen et al., 2019). There were two iterative rounds of annotation. In the first round, each annotator labeled all 3,930 test cases as hateful or non-hateful, and had the option to flag unrealistic entries. Test cases with any disagreement or unrealistic flags were reviewed by the study authors  $(n = 289)$ . One-on-one interviews were conducted with annotators to identify dataset issues versus annotator error. From 289 test cases, 119 were identified as ambiguous or unrealistic, replaced with alternatives and re-issued to annotators for labeling. No further issues were raised. We measured inter-annotator agreement using Randolph's Kappa (Randolph, 2005), obtaining a value of 0.85 for the final set of test cases, which indicates "almost perfect agreement" (Landis and Koch, 1977).
3 Building Better Models with HATEMOJIBUILD
As reported in §4, we find existing models perform poorly on emoji-based hate as measured with HATEMOJICHECK. We address those failings by
implementing a human-and-model-in-the-loop approach using the Dynabench interface in order to train a model that better detects emoji-based hate.
Dataset Generation We implemented three successive rounds of data generation and model retraining to create the HATEMOJIBUILD dataset. In each round we tasked a team of 10 trained annotators with entering content that the model-in-the-loop would misclassify.<sup>7</sup> We refer to this model as the target model. Annotators were instructed to generate linguistically diverse entries while ensuring each entry was (1) realistic, (2) clearly hateful or non-hateful and (3) contained at least one emoji. Each entry was first given a binary label of hateful or non-hateful, and hateful content was assigned secondary labels for the type and target of hate (Zampieri et al., 2019; Vidgen et al., 2021a,b). Each entry was validated by two additional annotators, and an expert resolved disagreements. After validation, annotators created a perturbation for each entry that flipped the label. To maximize similarity between originals and perturbations, annotators could either make an emoji substitution while fixing the text or fix the emoji and minimally change the surrounding text. Including a hateful and non-hateful version of each sentence in the dataset prevents overfitting to certain emoji or identity references. Each perturbation received two additional label annotations, and disagreements were resolved by the expert. This weekly cadence of annotator tasks was repeated in three consecutive weeks. The dataset composition and inter-annotator agreement is described in Appendix E.
Model Implementation Our work follows directly from Vidgen et al. (2021b), who collect four rounds (R1-4) of dynamically-generated textual data as well as 468,928 entries compiled from 11 English-language hate speech datasets (R0). Our rounds of data collection are referred to as R5-7. At the end of each round, the data for that round is assigned a 80/10/10 train/dev/test split. The test split is never used for any future training. The train split is upsampled to improve performance with multipliers of 1, 5, 10, 100, with the optimum ratio taken forward to subsequent rounds. The target
$^{6}$ Dynabench is an open-source platform which supports dynamic dataset generation and model benchmarking for a variety of NLP tasks (Kiela et al., 2021).
There was no overlap in annotators from the previous HATEMOJICHECK task; demographics are in Appendix D.
See annotator demographics in Appendix B.
model is then re-trained on the training data from all prior rounds as well as the current round. For R5 data collection, the target model (R5-T) is the DeBERTa model released by Ma et al. (2021), trained on R0-R4 from Vidgen et al. (2021b). This model performs well on text-only hate, but has seen limited emoji in training. For subsequent target models, we evaluate two architectures: DeBERTA (He et al., 2021) and BERTweet (Nguyen et al., 2020).
We select the best models and upsampling ratios at the end of each round. The criteria to select the best model must satisfy two requirements: (1) the target model must still perform well on prior rounds of text-based examples because emoji-based hate represents just one construction of textual hate, and (2) the target model must successfully handle the most recent round data because iterative data collection produces more challenging examples in successive rounds. To balance these requirements, we use a weighted accuracy metric with  $50\%$  weight on the test sets from all prior rounds and  $50\%$  weight on the current round test set, which enforces a recency-based discount factor (Kiela et al., 2021). This ensures we can assess performance against the latest (emoji-specific and most-adversarial) round without overfitting and reducing performance on the previous test sets.
Figure 1: MER for R1-4 (text) and R5-R7 (emoji).
Model Error Rate For each emoji-based round of data generation (R5-7) as well as prior text-only rounds from Vidgen et al. (2021b), we calculate model error rate (MER) as the proportion of annotators' original entries that fool the model (Fig. 1). Between R4 and R5, MER increases by 35 percentage points (pp) to over  $60\%$ , higher than the R1 MER. From R5 to R6, once the model has been trained on emoji-based hate, there is a steep reduction in MER of 24pp. From R6 to R7, there is a smaller reduction in MER of 1pp. Overall, the
model is easily tricked by emoji content at first but then becomes much harder to trick after our first round of re-training.
Performance on Adversarial Test Sets To evaluate model performance across rounds, we calculate F1-score and accuracy for each test set (Tab. 2). The baseline R5-T model is the highest performing model on the text-only test set (R1-4), with an F1 of 0.85, but only scores 0.49 on the newly generated emoji-containing test sets from HATEMOJIBUILD (R5-7). R6-T, R7-T and R8-T perform better on R5-7, with F1 between 0.76 and 0.77. They perform similarly well on the R1-4 test set, with F1 of 0.84, suggesting no trade-off for performance on text-only hate speech. The best performing model across all R1-7 test sets is R8-T, with an F1 of 0.83. The greatest performance gain in the adversarily-trained models is from R5-T to R6-T, with an increase in F1 of 0.28 on the emoji test set. This is achieved with only 2,000 emoji-containing examples. By comparison, R7-T and R8-T yield very small improvements.
Table 2: Performance of target models on emoji test sets from HATEMOJIBUILD, text sets from Vidgen et al. (2021b) and the combination of these test sets.
Emoji Test Set (R5-R7)
n = 593	Text Test Set (R1-R4)
n = 4119	All Test Sets (R1-R7)
n = 4712
Acc	F1	Acc	F1	Acc	F1
R5-T	0.585	0.490	0.828	0.847	0.786	0.801
R6-T	0.757	0.769	0.823	0.837	0.813	0.825
R7-T	0.759	0.762	0.824	0.842	0.813	0.829
R8-T	0.744	0.755	0.827	0.844	0.814	0.829
4 Evaluating Models with HATEMOJICHECK
We present results for two existing models as baselines, with additional baselines shown in Appendix G. The first baseline is Google Jigsaw's Perspective API, a widely-used commercial tool for content moderation. We use Perspective's "Identity Attack" attribute, which is defined as "negative or hateful comments targeting someone because of their identity" and thus closely matches our definition of hate. The returned score is converted to a binary label with a  $50\%$  cutoff. We refer to this model as P-IA. The second baseline is the R5-T model from Ma et al. (2021), introduced above. To compare model performance on HATEMOJICHECK, we use accuracy because three sets of test cases (originals, polarity perturbations and identity perturba
8For further details of model training, selection and emoji encodings see Appendix F.
Table 3: Aggregate accuracy for models evaluated on HATEMOJICHECK.
n	Pre-Emoji	Post-Emoji
P-IA	R5-T	R6-T	R7-T	R8-T
Overall	3930	0.689	0.779	0.879	0.867	0.871
Label
Hate	2654	0.706	0.770	0.905	0.876	0.896
Not hate	1276	0.653	0.799	0.825	0.849	0.820
Set
Original	2126	0.664	0.717	0.887	0.850	0.877
Identity p.	314	0.908	0.917	0.917	0.917	0.876
Polarity p.	902	0.584	0.778	0.818	0.853	0.830
No emoji p.	588	0.823	0.934	0.925	0.925	0.910
emoji diff	-0.159	-0.217	-0.038	-0.075	-0.033
tions) have one class label, making F1-score incompatible. To measure emoji-specific weaknesses, we also calculate emoji difference, the difference between averaged model accuracy on the original emoji test cases compared with averaged accuracy on the no emoji perturbations.
On HATEMOJICHECK as a whole, our newly trained models R6-T, R7-T and R8-T perform best, with overall accuracy from 0.87 to 0.88 (Tab. 3). They substantially outperform P-IA, with an accuracy of 0.69, and R5-T, with an accuracy of 0.78. Our newly trained models have the smallest emoji difference, between 0.03 and 0.08. In contrast, P-IA and R5-T have emoji differences of 0.16 and 0.22, respectively. Comparing the three models trained on HATEMOJIBUILD, the first round of adversarial data generation yields the largest relative improvement, and in many ways R6-T is at least as good a model, if not better, than R8-T.
4.1 Model Performance by Functionality
Our models trained on HATEMOJIBUILD perform better than the two baseline models on nearly every functionality (Tab. 4). They also perform far more consistently across all perturbation types.