SlowGuess/NAACL_2022_OCR huggingface dataset:
Table 8: Performance of models on emoji, text and all adversarial test sets, alongside benchmark evaluation sets: HATEMOJICHECK and HATECHECK.
Emoji Test Sets	Text Test Sets	All Rounds
R5-R7n = 593	HMOJICheckn = 3930	R1-R4n = 4119	HATECHECKn = 3728	R1-R7n = 4712
Acc	F1	Acc	F1	Acc	F1	Acc	F1	Acc	F1
P-IA	0.508	0.394	0.689	0.754	0.679	0.720	0.765	0.839	0.658	0.689
P-TX	0.523	0.448	0.650	0.711	0.602	0.659	0.720	0.813	0.592	0.639
B-D	0.489	0.270	0.578	0.636	0.589	0.607	0.632	0.738	0.591	0.586
B-F	0.496	0.322	0.552	0.605	0.562	0.562	0.602	0.694	0.557	0.532
R5-T	0.585	0.490	0.779	0.825	0.828	0.847	0.956	0.968	0.786	0.801
R6-T	0.757	0.769	0.879	0.910	0.823	0.837	0.961	0.971	0.813	0.825
R7-T	0.759	0.762	0.867	0.899	0.824	0.842	0.955	0.967	0.813	0.829
R8-T	0.744	0.755	0.871	0.904	0.827	0.844	0.966	0.975	0.814	0.829
G Robustness Analysis of Baselines
In addition to the models analyzed in the main paper, we evaluate three further models. The first is the 'toxicity' attribute returned by Perspective API (P-TX). We test this model because toxicity ratings are the most popular attributes. $^{13}$  The second and third models are two uncased BERT models (Devlin et al., 2019) trained on publicly-available academic datasets. B-D is trained on the Davidson et al. (2017) dataset of 24,783 tweets, labeled as hateful, offensive and neither. B-F is trained on the Founta et al. (2018) dataset of 99,996 tweets, labeled as hateful, abusive, spam and normal. Any labels besides hateful are binarized into a single non-hateful label. Two factors motivated the testing of these models. (1) There is a lack of emoji in the training data: the Davidson et al. dataset has  $5.8\%$  hateful cases, but only  $7.4\%$  of these contain emoji, and the Founta et al. dataset has  $5.0\%$  hateful cases, of which  $14.7\%$  contain emoji. (2) Despite BERT being a commonly-used architecture for hate speech detection, it encodes emoji as <UNK> tokens by default. We compare performance of the full set of pre-emoji models including those analyzed in the main paper (P-IA, P-TX, B-D, B-F, R5-T) versus our 'emoji-aware' models from each round of data collection (R6-T, R7-T, R8-T). Tab. 8 shows performance against the adversarially produced datasets in the emoji rounds of HATEMOJIBUILD and text rounds from Vidgen et al. (2021b), alongside two benchmark evaluation sets HATEMOJICHECK and HATECHECK (Röttger et al., 2021). P-TX has comparable performance to P-IA. B-D and B-F perform poorly on HATEMOJICHECK ( $F1 = 0.64$ , 0.61), and even more poorly on the adversarial test sets of emoji ( $F1 = 0.27$ , 0.32).
H Fairness Considerations
As well as the six metrics discussed in the main paper, Tab. 9 shows two further fairness metrics: (1) the demographic parity ratio (Agarwal et al., 2019), which is equal to one when the selection rate is balanced across subgroups, and (2) the equalized odds ratio (Hardt et al., 2016; Agarwal et al., 2018), which is equal to one when the true positive, true negative, false positive, and false negative rates are balanced across subgroups. The academic BERT models perform worst (0.14 to 0.28 across the metrics), followed by the Perspective models (0.41 to 0.67). The most fair model is R5-T with Demographic Parity Ratio of 0.90 and Equalized Odds Ratio of 0.77. However, the overall performance of the emoji-aware models is higher (see Fig. 2): for every subgroup, R8-T has higher accuracy, precision and recall, and lower false positive and negative rates. Thus, the marginally worse performance by between-group fairness metrics is paired with a better ability to protect all these subgroups against emoji-based hate.
Table 9: Between-group fairness metrics by model.
Demographic Parity Ratio	Equalized Odds Ratio
P-IA	0.668	0.430
P-TX	0.611	0.405
B-D	0.174	0.138
B-F	0.276	0.257
R5-T	0.898	0.767
R6-T	0.818	0.711
R7-T	0.848	0.667
R8-T	0.832	0.600
<sup>13</sup>https://support.perspectiveapi.com/s/about-the-api-faqs