Venue: NAACL 2025

# What the #?*!: Disentangling Hate Across Target Identities

Yiping Jin\({}^{1}\), Leo Wanner\({}^{2,1,3}\), Aneesh Moideen Koya\({}^{4}\)

\({}^{1}\)NLP Group, Pompeu Fabra University, Barcelona, Spain

\({}^{2}\)Catalan Institute for Research and Advanced Studies

\({}^{3}\)Barcelona Supercomputing Center

\({}^{4}\)Knorex, 1159 Sonora Court, Suite 122, Sunnyvale, CA 94086, USA

{yiping.jin, leo.wanner}@upf.edu

###### Abstract

Hate speech (HS) classifiers do not perform equally well in detecting hateful expressions towards different target identities. They also demonstrate systematic biases in predicted hatefulness scores. Tapping on two recently proposed functionality test datasets for HS detection, we quantitatively analyze the impact of different factors on HS prediction. Experiments on popular industrial and academic models demonstrate that HS detectors assign a higher haefulness score merely based on the mention of specific target identities. Besides, models often confuse hatefulness and the polarity of emotions. This result is worrisome as the effort to build HS detectors might harm the vulnerable identity groups we wish to protect: posts expressing anger or disapprovind of hate expressions might be flagged as hateful themselves. We also carry out a study inspired by social psychology theory, which reveals that the accuracy of hatefulness prediction correlates strongly with the intensity of the stereotype.1

Footnote 1: The source code is available at [https://github.com/VipingMUS/disentangle-hate](https://github.com/VipingMUS/disentangle-hate).

**Content Warning:**_This document discusses examples of harmful content (hate, abuse, and negative stereotypes). The authors do not support the use of harmful language._

## 1 Introduction

The surge of interest in combating online hate led to increased efforts in creation of benchmark datasets and organization of shared tasks, and, as a consequence, rapid development of hate speech (HS) detection models Caselli et al. (2020); Poletto et al. (2021). However, state-of-the-art HS detectors do not perform equally well across different datasets Fortuna et al. (2021) and different target identities Ludwig et al. (2022). These performance discrepancies have been attributed to diverging dataset annotations Fortuna et al. (2020), out-of-domain distribution Jin et al. (2023), spurious correlation between specific target identities and the labels Ramponi and Tonelli (2022), and specific topical focuses Bourgeade et al. (2023).

Unfortunately, none of these diagnoses resulted so far in a revision of the datasets or models, and, as a matter of fact, this is not surprising. Instead of treating HS detectors as black boxes and HS datasets as given commandments, the research community should understand what the datasets entail and how the models behave under different circumstances. Such insights can help us make practical

Figure 1: Overview of our approach. We analyze target identity mentions’ impact on hatefulness prediction in a minimal set experiment on the HateCheck dataset. We then extract fine-grained emotions and stereotypes from examples in GPT-HateCheck dataset to analyze the distributional difference among target identities and its impact on the classifiers’ performance.



progress in building more robust and fair classifiers (Chen et al., 2024) beyond pushing a single metric such as accuracy or F\({}_{1}\) score.

Some work has already been done in this direction. To provide more diagnostic insights, Rottger et al. (2021) introduced HateCheck, a comprehensive suite of functional tests that covers 29 model "functionalities" across seven target identities. Each functionality tests the models' behavior on a specific kind of hateful or contrastive non-hateful content (e.g., "denouncements of hate."). To generate examples at scale, handcrafted templates (Ribeiro et al., 2020) for each functionality (e.g., "[Identity] belong in a zoo.") have been used. Jin et al. (2024) further improved HateCheck by substituting its simplistic and boilerplate examples by a new dataset GPTHateCheck, with LLMs-generated test cases. They demonstrate that the new dataset has better lexical diversity and naturalness than HateCheck.

In this work, we aim to disentangle the impact of different factors on HS prediction by tapping on the two aforementioned functionality test datasets. Specifically, we study the difference in predicted scores among _minimal sets_ from HateCheck, where the only variable is the mention of the target identity (Section 3.1). We then identify fine-grained emotions (Section 3.2) and stereotypes (Section 3.3) from GPT-HateCheck and analyze their impact on classifiers' performance; Figure 1 illustrates the analyses we conduct. The experiments reveal that HS detection models possess a systematic bias based on specific target identity mentions. Models predict accurately in case of intense stereotypes but struggle when the stereotype is mild. What is more concerning is that models tend to misclassify non-hateful posts expressing negative emotions as hateful, such as counterspeech or posts expressing sadness towards HS. Our contributions are threefold:

* We quantitatively measure the impact of different factors on HS prediction.
* We conduct emotion and stereotype analyses of the recently-introduced GPT-HateCheck dataset. To the best of our knowledge, it is the first systematic analysis of the impact of emotions and stereotypes on HS prediction.
* We highlight critical model weaknesses, such as the confusion of hate with negative emotions. These findings shed light on new directions for improving the robustness and fairness of HS detectors.

## 2 Related Work

### Hate Speech Detection Datasets

Early work in hate speech (HS) detection focused on specific phenomena such as "racism", "sexism' or "xenophobia" (Waseem and Hovy, 2016; Basile et al., 2019) or treated it as coarse-grained classification without explicitly stating the target identities involved (Davidson et al., 2017; Founta et al., 2018). However, ignoring the target identity and the difference among related concepts such as "abusive", "offensive", and "toxic" may cause HS detectors to learn frequently occurring patterns in a particular context and harm generalizability (Vidgen and Derczynski, 2020; Fortuna et al., 2020). Therefore, more recent datasets often provide additional contextual information.

Zampieri et al. (2019) introduced the OLID dataset, where each offensive message is assigned a target \(\in\) {"individual", "group", "other"}. Caselli et al. (2020) augmented the OLID dataset by adding new annotation dimensions like "abusiveness" and "explicitness". Ousidoum et al. (2019) labeled five attributes for each post: directness (2)2, hostility (6), target attribute (6), target group (5), and sentiment of the annotator (7). Similarly, Mathew et al. (2021) provide rich annotation, including 18 fine-grained target groups related to race, religion, gender, sexual orientation, and rationale text spans on which the labeling decision is based.

Footnote 2: Indicates the number of unique values for each attribute.

Due to data rarity, most HS detection datasets are collected using keywords, favoring explicit HS expressions (Poletto et al., 2021; Yin and Zubiaga, 2021; Rahman et al., 2021), which may also cause models trained on such datasets to be over-reliant on a specific set of keywords. To prevent an overestimation of generalizable model performance, Rottger et al. (2021) introduced HateCheck, a suite of functional tests for HS detection models. They developed 29 functionalities representing challenges in tackling online hate through interviews with NGO workers. Then, they crafted test cases for each functionality consisting of short sentences with unambiguous labels. Templates such as "[Identity] are disgusting." are utilized to generate test cases at scale by replacing the special token "[Identity]" with a specific target identity.

 

Most recently, Jin et al. (2024) introduced GPT-HateCheck, which follows the list of HateCheck functionalities but generates examples with OpenAI's GPT-3.5 Turbo model3 instead relying on templates. They demonstrated that the new dataset has higher lexical diversity and is more realistic than the template-based HateCheck counterpart. While the new dataset is of great utility to test models' performance in a more realistic setting, Hatcheck has the advantage of allowing minimal pair analysis, which is commonly used in linguistic studies to understand models' behavior Warstadt et al. (2020). Furthermore, although Jin et al. (2024) claimed that GPT-HateCheck covers distinct HS aspects associated with different target identities and provided promising qualitative examples, a quantitative analysis on the distribution of the aspects is missing. We tap on both datasets' strengths. Firstly, we conduct a minimum set analysis by differentiating models' prediction across different target identities with templates in the HateCheck dataset. Then, we analyze fine-grained emotions and stereotypes in the more realistic and diverse GPT-HateCheck dataset and correlate them with HS prediction.

Footnote 3: [https://platform.openai.com/docs/models/gpt-3-5-turbo](https://platform.openai.com/docs/models/gpt-3-5-turbo)

### Bias Analysis and Mitigation

HS classifiers can absorb unintentional bias across different stages of model development, such as data sampling, annotation, and model learning Fortuna et al. (2022). Classifiers also often have a superficial understanding of language and are heavily affected by _spurious correlations_. Wiegand et al. (2019) found that many top words strongly correlated with the hateful category are non-offensive topical words like "football" or "commutator". They argued that it is due to the narrow sampling strategy used to create the dataset.

Park et al. (2018) observed that HS detectors are biased towards _gender_ identities. For example, "You are a good woman" was classified as "sexist". They proposed mitigation approaches including debiased word embeddings, gender swap data augmentation, and fine-tuning with a larger corpus to reduce the inequality measure. On the other hand, studies on _dialectal/racial_ bias Davidson et al. (2019); Sap et al. (2019); Mozafari et al. (2020) revealed that African American English (AAE) is much more likely to be predicted as offensive. Furthermore, Maronikolakis et al. (2022) studied the intersection of gender and racial attributes and showed that the bias could be amplified for certain attribute combinations (e.g., masculine and AAE).

Zhou et al. (2021) introduced ToxDect-roberta, focusing on mitigating lexical (e.g., swear words, identity mentions) and dialectbial bias towards AAE. They explored debiased training Clark et al. (2019) and data filtering Le Bras et al. (2020); Swayamdipta et al. (2020) but obtained limited success. However, translating AAE to white-aligned English (WAE) automatically with GPT-3 and relabeling toxic AAE tweets whose WAE translation is predicted as non-toxic yields greater improvement for dialectAdebiasing.

Fraser et al. (2021) proposed an interpretation of stereotypes towards different target identities based on the Stereotype Content Model (SCM) Fiske et al. (2002), which captures stereotypes along two primary dimensions: warmth and competence. Our stereotype analysis is inspired by Fraser et al. (2021). However, their work employed static word embedding models to study stereotypes expressed through unigram words. In contrast, we analyze stereotypes in natural language sentences by assigning scores along the "warmth" and "competence" dimensions with an NLI model He et al. (2021).

## 3 Methodology

DatasetsWe use the HateCheckRottger et al. (2021) and GPT-HateCheckJin et al. (2024) datasets to conduct our analyses, as these datasets provide additional diagnostic insights. Both datasets cover the same seven target identities and 24 functionalities (GPT-HateCheck omitted the five functionalities related to spelling variations in HateCheck). Table 1 displays the number of documents for each target identity in both datasets, and Appendix A shows examples from the two datasets for each functionality.

ModelsBelow, we detail the models we experimented with: HateBERT, ToxDect-roberta, Perspective API, and Llama Guard 3. HateBERT Caselli et al. (2021) and ToxDect-roberta Zhou et al. (2021) are open-source models, while Perspective API is an industry-standard API developed by Jigsaw and Google's Counter Abuse Technology team to combat online toxicity and harassment.4Llama Guard 3 Inan et al. (2023) is a recent LLM safeguard model based on Meta's Llama 3 Dubey et al. (2024).

 

* **HateBERT:5** A pre-trained BERT model further trained with over 1 million posts from banned Reddit communities. We use the best-performing model variant fine-tuned on OffensEval dataset Caselli et al. (2020). Footnote 5: The model checkpoint: [https://osf.io/tbd58/](https://osf.io/tbd58/).
* **ToxDet-roberta:6** A toxicity detector based on Roberta-large model, aiming to reduce lexical and dialectal biases via automatic data correction. The model was trained using original and synthetically labeled examples from the Founta dataset Founta et al. (2018). Footnote 7: [https://huggingface.co/Cruen-llama/Llama-Guard-3-1B](https://huggingface.co/Cruen-llama/Llama-Guard-3-1B) and [https://huggingface.co/meta-llama/Llama-Guard-3-8B](https://huggingface.co/meta-llama/Llama-Guard-3-8B).
* **Perspective API:** A Google API that uses machine learning models to identify abusive comments. Following Rottger et al. (2022), we use the "identity attack" model instead of the standard "toxicity" model because it aligns more closely with the definition of hate speech adopted in HateCheck. The two models are compared in Appendix G.
* Hate" as other categories are beyond the scope of hate speech (e.g., election, intellectual property). We use the probability of the first token ("safe"/"unsafe") as the prediction score as recommended in the model card and 0.5 threshold to obtain the predicted label. We experiment with two model sizes: 1B and 8B7. Footnote 7: [https://huggingface.co/meta-llama/Llama-Guard-3-1B](https://huggingface.co/meta-llama/Llama-Guard-3-1B) and [https://huggingface.co/meta-llama/Llama-Guard-3-8B](https://huggingface.co/meta-llama/Llama-Guard-3-8B).

### Disentangling Target Identity Mentions

We use examples from HateCheck for a minimal set analysis to measure the models' bias towards target identity mentions. Previous work either compares the subset accuracy of posts from different populations Davidson et al. (2019); Zhou et al. (2021) or performs perturbation to create adversarial examples Fortuna et al. (2022). The first approach's drawback is that subsets differ not only in identity mentions but also in topics, styles, and vocabulary preference, among other factors, such that it is not possible to isolate the impact of identity mentions. The limitation of the second approach is that ad hoc perturbation often focuses on particular examples instead of the big picture. Using examples from HateCheck yields two advantages: 1) It covers a wide variety of scenarios of hateful and non-hateful posts, and 2) Five annotators manually validated each example to ensure its validity and consistency with the gold-standard label.

A large share of examples in HateCheck is generated using templates. We select those templates in the dataset that instantiate exactly seven examples (one for each of the considered target identities). We exclude examples with no target identity, such as "abusing objects". We also exclude templates that use target-specific slur words (e.g., "n*gger", "b*tch") because they may entail different degrees of hatefulness. This leaves us with a total of 333 templates and 333\(\times\)7=2,331 examples. Since identity mentions are neutral, we assume that the hatefulness prediction for exam

\begin{table}
\begin{tabular}{l c c} \hline \hline
**Target** & **HateCheck** & **GPT-HateCh.** \\ \hline Women & 509 & 606 \\ Trans ppl. & 463 & 611 \\ Gay ppl. & 551 & 646 \\ Black ppl. & 482 & 741 \\ Disabled ppl. & 484 & 644 \\ Muslims & 484 & 663 \\ Immigrants & 463 & 684 \\ \hline \end{tabular}
\end{table}
Table 1: Number of examples for each target identity in HateCheck and GPT-HateCheck. We omit the functionalities without targeting identity, such as abusing objects or non-protected groups.

Figure 2: Normalized hatefulness predictions of models across target identities.

 

ples instantiated from the same template should not alter depending on the target identity, i.e., the predicted hatefulness score for "I hate _immigrants_" and "I hate _trans_" should be equal regardless of the absolute predicted score. Based on this assumption, we predict hatefulness scores for all 2,331 examples. Then, we calculate the normalized prediction for each example as the predicted score minus the median score of all examples instantiated from its corresponding template. Finally, we calculate the mean of the normalized predictions for each target identity and present different models' results in Figure 2.

While the models show different degrees of bias towards identity mentions, the bias orientation is often the same: All models have a positive bias (predicting as more hateful) towards gaps, black people, and Muslims and a negative bias towards women and disabled people. Surprisingly, ToxDectroberta, which is trained explicitly to mitigate bias, possesses the largest bias towards identity mentions, reaching as high as +33.9% for black people. Comparing Llama Guard 3-1B and -8B, we observe that the larger LLM can better handle identity bias.

We now focus on the impact of the identity mention bias on models' classification performance. For this experiment, we use GPT-HateCheck because its examples are more realistic. We report each model's per-target-identity P/R/F\({}_{1}\) scores for the hateful category in Table 2.

Perspective API performs consistently best among non-LLM baselines. ToxDect-roberta performs the worst, primarily due to its poor recall for the categories "women", "trans", "disabled people", and "immigrants". We hypothesize that these target identities are not well represented in the model's training dataset due to the significant performance discrepancy among different target identities8. The Llama Guard 3 models obtained better recall scores than other baselines, showing LLMs' capability to catch more nuanced hateful expressions. While the larger 8B model performs better, it requires much more computation and consumes 30GB vRAM for inference only, which cannot fit into a current desktop GPU.

Footnote 8: Founta et al. (2018) provided no information regarding the distribution of the target identities.

Debiasing could potentially reduce the impact of the identity mention bias. However, an in-depth comparison of debiasing methods is beyond the scope of this paper. Therefore, we merely apply a naive debiasing method by subtracting the prediction by the model's target-identity bias.9 Target identities with strong negative bias in the minimum set experiment, such as "women" and "disabled people", also have a much lower recall for the "hateful" category compared to other target identities. Subtracting the negative bias helped HateBERT and Perspective API improve the recall for these categories by a large margin with a much smaller sacrifice in precision. However, debiasing has little effect on ToxDect-roberta and Llama Guard 3 models because their predicted scores concentrate near 0 or 1 and are poorly calibrated, as shown in Appendix C.

Footnote 9: For example, if model A predicts a hatefulness score of 0.65 for an input related to Muslims, and it has a positive bias of 0.13 towards Muslims from the previous minimal set experiment, the debiased prediction will be 0.65\(-\)0.13=0.52.

### Disentangling Emotions

Hateful and non-hateful posts entail distinct emotions, which may affect the accuracy of HS detectors. We want to study whether emotions are uniformly associated with different target identities

\begin{table}
\begin{tabular}{l|c c c c c c c c} \hline
**Model** & **Women** & **Trans** & **Gays** & **Black** & **Disabled** & **Muslims** & **Immiger** & **Avg** \\ \hline HateBERT & **7.77**/.59/.67 & **86.7**/.78/.82 & **87.87**/.87 & 7.98/**.86/.83 & 8.2/61.7/.70 & **84.85**/.85 & **86.76**/**.80 & 8.3/76.79 \\ \(+\)Debias & 7.75**/**65**/.69 & 8.58**/**.82**/.83 & **88**/.86/.87 & **8.1**/.79/.80 & 8.2/**73**/.77 & **85**/.81/.83 & 86.73/.79 & 8.3/**77**/.80** \\ \hline ToxDect & 7.12/.57/.37 & 87.35/.49 & 83.81/.82 & 7.09/.96/.81 & 8.2/.23/.36 & 84.97/.90 & 9.5/.36/.52 & 8.2/.56/.61 \\ \(+\)Debias & 7.11**/**26**/.38 & 87.35/.49 & 83.80/.82 & **7.2**/.96/.82 & 8.2/.23/.36 & 84.97/.90 & 9.5/.36/.52 & 8.2/.56/.61 \\ \hline Perspective & **.98**/.62/.76 & 99.87**/**.91 & 90**/**.95**/.93 & 84.97**/**.90 & **9.8**/.57/.71 & 96.95**/.95 & **9.99**/.58/.73 & 9.5/.78/.84 \\ \(+\)Debias & 9.57**/**8**/.86 & 99.97/.82/.90 & **99**/.86/.92 & **89**/.84/.87 & 96.75**/**.84** & **97**/.94/**.96 & 97.69**/**.81** & **96**/**.81**/.88** \\ \hline Llama3-lb & 9.97**/.98/.89 & 97.83/.90 & 93.97/.95 & 88.94/.91 & 9.77/.76/.85 & 91.95/.93 & 99.81/.89 & 95.86/.90 \\ \(+\)Debias & 9.99**/**80**/.89 & 97.83/.90 & 93.97/.95 & 88.84/.91 & 9.77/**.86** & **92**/.95/.93 & 99.82/.89 & 95.87/.90 \\ \hline Llama3-sb & 1.0/.83/.90 & 1.0/.95/.97 & 99.99/.99 & .88/.98/.93 & .99/.93/.96 & 1.0/.99/.99 & 1.0/.79/.88 & 98.92/.95 \\ \(+\)Debias & 1.0/**84**/**.92 & 1.0/.95/.97 & 99.99/.99 & .88/.98/.93 & .99/.93/.96 & 1.0/.99/.99 & 1.0/.80/.89 & 98.93/.95 \\ \hline \end{tabular}
\end{table}
Table 2: Per target identity P/R/F\({}_{1}\) scores of each model with and without debiasing. We highlight the best score for each model in **bold**.

 

or some emotions are more prominent for certain target identities.

We prompt GPT-4 (Achiam et al., 2023)10 to identify fine-grained emotions from posts in GPT-HateCheck using the taxonomy proposed by Demszky et al. (2020), which contains 27 distinct emotions. We provide the full prompt in Appendix F. Figure 3 presents the detected emotions ranked by frequency. 4313 out of 4438 messages have emotions detected in them (97.2%). Hateful posts focus primarily on four emotions: disgust, disapproval, anger, and fear, while non-hateful posts demonstrate a much broader range of emotions, both positive and negative ones.

Footnote 10: gpt-4o-2024-05-13 model checkpoint.

Then, we analyze the distribution of target identities for each detected emotion and present the result in Figure 4. It is manifest that the emotions expressed towards each target identity have a unique composition. In hateful examples, the dominant emotions expressed towards Muslims and immigrants are "anger" and "fear", while the most prominent emotion towards black and disabled people is "disgust". For non-hateful examples, "love" stands out for gaps, "sadness" for black people, and "pride" and "approval" for trans. In addition, we analyze the correlation between functionalities and emotions in Appendix D.

Tabel 3 presents the fine-grained emotion level accuracy of each model for emotions with at least ten occurrences. The emotions with which models struggle the most are "annoyance", "disapproval", "sadness", and "fear".

We further group the fine-grained emotions into positive (1), negative (-1), and ambiguous (0), based on Demszky et al. (2020)'s taxonomy and present the models' classification accuracy in the presence of emotions with different polarities in Table 4. The result is revelatory: All models can relatively accurately identify hateful posts with negative emotions and non-hateful posts with positive emotions. However, the accuracy degrades drastically for non-hateful posts with negative emotions, especially for HateBERT and ToxDect-roberta.11 This result is alarming since it suggests that HS detectors are entangled with emotion polarity, and some safe posts with negative emotions, such as counter-speech expressing disapproval or sadness, are likely marked as hateful, potentially silencing the voices of vulnerable groups.

Figure 3: Frequent emotions detection in GPT-HateCheck dataset with at least ten occurrences.

 



### Disentangling Stereotypes

Jin et al. (2024) motivated the use of LLMs with the generation of test cases that account for distinct stereotypes associated with different target identities (e.g., criminality for immigrants and sexuality for trans). However, they did not analyze which stereotypes are covered in their dataset and whether a distinction exists among target identities. We present an in-depth analysis of the stereotypes/counter-stereotypes in GPTHateCheck by 1) Interpreting stereotypes based on an established social psychology theory, 2) Analyzing the correlation between stereotypes and HS prediction accuracy, and 3) Extracting and qualitatively analyzing stereotypes/counter-stereotypes.

Stereotypes InterpretationFiske et al. (2002; 2007) proposed the Stereotype Content Model, which uses the universal dimensions "warmth" and "competence", to describe social perceptions and stereotypes. The model maps each stereotype onto interpretable semantic axes "warmth" vs. "coldness" and "competence" vs. "incompetence". We use a state-of-the-art NLI model He et al. (2021)12 to assign "warmth" and "competence" scores to each example in the GPT-HateCheck dataset. Inspired by Mathew et al. (2020), we derive the scores via semantic differentials of two opposite concepts (e.g., "warmth" and "coldness"). Specifically, we test four hypotheses for each example:

Footnote 12: [https://huggingface.co/cross-encoder/nli-deberta-v3-large](https://huggingface.co/cross-encoder/nli-deberta-v3-large).

* \(\mathbb{H}_{1}^{+}\): This message expresses _warmth_ towards [target identity].
* \(\mathbb{H}_{1}^{-}\): This message expresses _coldness_ towards [target identity].
* \(\mathbb{H}_{2}^{+}\): This message expresses that [target identity] are _competent_.
* \(\mathbb{H}_{2}^{-}\): This message expresses that [target identity] are _incompetent_.

The NLI model returns logit scores for the three classes: "entail", "contradict", and "neutral". We first take the softmax over the three classes and derive the score for "warmth" as:

\[\mathbb{S}_{warmth}=\mathcal{P}_{entail}(\mathbb{H}_{1}^{+})+ \mathcal{P}_{ contradict}(\mathbb{H}_{1}^{-})\\ -\mathcal{P}_{ contradict}(\mathbb{H}_{1}^{+})-\mathcal{P}_{ entail}(\mathbb{H}_{1}^{-}) \tag{1}\]

We derive \(\mathbb{S}_{competence}\) similarly by replacing \(\mathbb{H}_{1}^{*}\) with \(\mathbb{H}_{2}^{*}\) in Equation 1. Due to the softmax operation, \(\mathbb{S}_{warmth}\) and \(\mathbb{S}_{competence}\) are both bounded in the range of [-2, 2].

Figure 5 plots the kernel density estimate (KDE) in the warmth-competence semantic space.13

Figure 4: Distribution of target identities for each detected emotion.



While different target identity distributions overlap substantially, we can observe some patterns:

* Non-hateful: Many examples related to women have high "competence" scores, highlighting a typical counter-speech pattern. Meanwhile, examples related to gaps tend to have a high "warmth" score.
* Hateful: Some examples related to women and disabled people receive very low "competence" scores but comparatively higher "warrmth" scores, compared to other hateful examples (the lower right corner).

The mean "warmth" and "competence" scores for each target identity are presented in Table 5. We can observe a clear push-back pattern: The higher the "coldness" or "incompetence" scores are for hateful stereotypes towards a target identity, the stronger the counter-stereotypes are in the opposite directions; consider, for illustration, the "warmth" dimension for gays and the "competence" dimension for women.

Correlation with Hate PredictionWe also investigate whether there is a correlation between the location in the warmth-competence semantic space and the HS detectors' accuracy. To this end, we apply the \(k\)-means algorithm to cluster the data points into 10 clusters. For each cluster, we compute its centroid's Euclidean distance to the origin and Perspective API's accuracy. We plot the correlation between these two factors in Figure 6. The figure reveals a clear trend that the farther from the origin (the more intense the example in the "warmth" or "competence" dimension), the more accurate the classifier is. The model seems to struggle particularly when both "warmth" and "competence" scores have a low magnitude. Other baselines demonstrate a similar trend (cf. Figure 11).

Figure 5: Kernel density estimate (KDE) in the warmth-competence semantic space of various target identities.

Figure 6: The cluster’s distance to the origin has a positive correlation with the hatefulness prediction accuracy.

 



#### Stereotype Phrase Analysis

Finally, we instructed GPT-4o to extract stereotypes/counter-stereotypes from hateful and non-hateful posts, similarly to Section 3.2. However, instead of categorizing into a fixed taxonomy, we prompt the model to identify the minimum text span supporting the prediction. In some cases, the model should slightly modify the span to match the original meaning. For example, the expected counter-stereotype for the input "No immigrant should be denied access to healthcare" is "should _not_ be denied access to healthcare". We present the full prompt in Appendix F and examples of extracted stereotypes/counter-stereotypes in Table 6.

Appendix E presents an expanded list of extracted stereotypes/counter-stereotypes associated with each target identity. Some stereotypes are shared across target identities, such as "drain on the economy" for disabled people and immigrants and "attention seeking" for trans and women. Meanwhile, some stereotypes are associated with a unique target identity, such as "terrorists" for Muslims and "crime and violence" for black people. On the other hand, many counterarguments are broader, such as calling for respect, acceptance, and treatment with dignity.

## 4 Conclusions and Future Work

We presented a comprehensive analysis of various factors that influence the behavior and accuracy of HS detectors. Empirical results revealed that popular industrial and academic HS classifiers are still prone to bias due to specific mentions of the target identity. They often confuse hatefulness and the polarity of the expressed emotions, and the stereotype intensity strongly impacts the classifiers' accuracy. While the result may seem pessimistic, our work opens up new venues for the NLP community to improve the robustness of HS detectors further and mitigate various biases. In future work, we plan to apply our method to more datasets and models and introduce an open-source evaluation benchmark to facilitate the future development of HS detectors.

### Limitations

We conduct experiments on two functionality test datasets: HateCheck(Rottger et al., 2021) and GPT-HateCheck(Jin et al., 2024). These datasets provide rich metadata such as the target identity and the type of hate expressions (functionality). The messages in these datasets were composed by crowd-source workers or LLMs. We chose not to use HS detection datasets sampled from social media platforms because 1) they usually do not provide fine-grained target identity information and 2) they do not provide detailed information on data sampling Fortuna et al. (2022). Sampling examples for different target identities from different domains (e.g., subreddits) or using different keywords might introduce compounding factors and obscure the conclusions. Nevertheless, we demonstrate the utility of our framework by presenting preliminary experimental results on a multi-source social media dataset in Appendix B.

The main contribution of our paper is the analysis of the impact of various factors in HS detection. The related problem of the analysis of bias mitigation methods was not in the focus of our work. While there exists an array of excellent surveys on bias mitigation methods Meade et al. (2022); Kumar et al. (2023); Gallegoos et al. (2024), including a comprehensive evaluation of bias mitigation methods would take up too much space and prevent us going into depth in the analysis. As we demonstrated in Section 3.1 and Appendix C, the naive debiasing method we use only helps when models predict well-calibrated probability-like scores. We claim neither the effectiveness nor the novelty of this method.

Furthermore, we used LLMs to detect emotions and stereotype phrases and a pre-trained NLI model to score the two stereotype dimensions. This helped us develop a prototype and validate our hypotheses rapidly. Although we performed some

\begin{table}
\begin{tabular}{p{85.4pt}|p{79.7pt}} \hline
**Example** & **Stereotype** \\ \hline I can’t stand gays and their disguesting lifestyle! & disgusting lifestyle \\ \hline No black person should face systemic racism. & should not face systemic racism \\ \hline It is wrong to discriminate against Muslims. & is wrong to discriminate \\ \hline Muslims are advocates for social justice and work towards creating a fair and equitable society for all. & advocates for social justice \\ \hline Immigrants can bring new perspectives and ideas to their host communities. & bring new perspectives and ideas \\ \hline \end{tabular}
\end{table}
Table 6: Extracted stereotypes/counter-stereotypes from samples in GPT-HateCheck.



prompt engineering and exploration, the accuracy was not perfect. If time and resources allow, hiring domain experts to relabel the examples would yield a more reliable result.

Lastly, stereotypes and emotions towards target identities strongly depend on the cultural context. The examples in GPT-HateCheck are written by LLMs, which align best with views of Western, educated, white, and younger population (Santy et al., 2023). Studying how the findings might alter under distinct socio-demographic backgrounds would be an exciting extension of this work.

## Acknowledgement

This work has been partially funded by the European Commission under contract numbers HE101070278 and ISF-101080090. We are grateful for the insightful discussion with Paul Rottger and his suggestion for conducting minimal pair analysis for target identities. We thank the anonymous reviewers for the careful reading and constructive feedback so that we could improve the manuscript. Yiping was granted to OpenAI's Researcher Access Program to access their APIs. Last but not least, We also received help from Perspective API team to increase our quota.

## References

* Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. Cited by: SS5.
* D. Dasikoglu, C. Bosco, E. Fersini, D. Nozza, V. Patti, P. Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti (2019)SemEval-2019 task 5: multilingual detection of hate speech against immigrants and women in Twitter. In Proceedings of the 13th International Workshop on Semantic Evaluation, Minneapolis, Minnesota, USA, pp. 54-63. External Links: Link Cited by: SS1, SS2.
* T. Bourgeade, P. Chiril, Farah Benamara, and Veronique Moriceau (2023)What did you learn to hate? a topic-oriented analysis of generalization in hate speech detection. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, Dubrovnik, Croatia, pp. 3495-3508. External Links: Link Cited by: SS1, SS2.
* T. Caselli, V. Basile, J. Mitrovic, and M. Granitzer (2021)HateBERT: retraining BERT for abusive language detection in English. In Proceedings of the 5th Workshop on Online Abuse and Harms (WOAH 2021), Online, Asia, pp. 17-25. External Links: Link Cited by: SS1, SS2.
* T. Chen, D. Wang, X. Liang, M. Risius, G. Demartini, and H. Yin (2024)Hate speech detection with generalizable target-aware fairness. arXiv preprint arXiv:2406.00046. Cited by: SS1, SS2.
* C. Clark, M. Yatskar, and L. Zettlemoyer (2019)Don't take the easy way out: ensemble based methods for avoiding known dataset biases. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, pp. 4069-4082. External Links: Link Cited by: SS1, SS2.
* T. Davidson, D. Bhattacharya, and I. Weber (2019)Racial bias in hate speech and abusive language detection datasets. In Proceedings of the Third Workshop on Abusive Language Online, Florence, Italy, pp. 25-35. External Links: Link Cited by: SS1, SS2.
* T. Davidson, D. Warmsley, M. Macy, and Ingmar Weber (2017)Automated hate speech detection and the problem of offensive language. In Proceedings of the international AAAI conference on web and social media, Vol. 11, pp. 512-515. External Links: Link Cited by: SS1, SS2.
* D. Dzaszyk, D. Movshovitz-Attias, J. Ko, A. Cowen, G. Nemade, and S. Ravi (2020)GoEmotions: a dataset of fine-grained emotions. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 4040-4054. External Links: Link Cited by: SS1, SS2.
* A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Lateman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al. (2024)The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Cited by: SS1, SS2.
* S. T. Fiske, A. J. Cuddy, and P. Glick (2007)Universal dimensions of social cognition: warmth and competence. Trends in cognitive sciences11 (2), pp. 77-83. External Links: Document, ISSN 0037-64X, Link Cited by: SS1, SS2.
* S. T. Fiske, J. Cuddy Amy, and Xu Jun Glick Peter (2002)A model of (often mixed) stereotype content: competence and warmth respectively follow from perceived status and competition. Journal of Personality and Social Psychology82 (6), pp. 878-902.

 

Paula Fortuna, Monica Dominguez, Leo Wanner, and Zeerak Talat. 2022. Directions for nlp practices applied to online hate speech detection. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 11794-11805, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Fortuna et al. (2020) Paula Fortuna, Juan Soler, and Leo Wanner. 2020. Toxic, hateful, offensive or abusive? what are we really classifying? an empirical analysis of hate speech datasets. In Proceedings of the Twelfth Language Resources and Evaluation Conference, Pages 6786-6794, Marseille, France. European Language Resources Association.
* Fortuna et al. (2021) Paula Fortuna, Juan Soler-Company, and Leo Wanner. 2021. How well do hate speech, toxicity, abusive and offensive language classification models generalize across datasets? Information Processing & Management, 58(3):102524.
* Founta et al. (2018) Antigoni Founta, Constantinos Djouvas, Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Gianluca Stringhini, Athena Vakali, Michael Sirivianos, and Nicolas Kourtellis. 2018. Large scale crowdsourcing and characterization of twitter abusive behavior. In Proceedings of the international AAAI conference on web and social media, Vol. 12.
* Fraser et al. (2021) Kathleen C. Fraser, Isar Nejadgholi, and Svetlana Kiritchenko. 2021. Understanding and countering stereotypes: A computational approach to the stereo-type content model. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online. Association for Computational Linguistics.
* Gallegos et al. (2024)Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Derronnocourt, Tong Yu, Ruiyi Zhang, and Nesreen K. Ahmed. 2024. Bias and fairness in large language models: A survey. Computational Linguistics, 50(3):1097-1179.
* He et al. (2021) Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. Deberta: Decoding-enhanced bert with disentangled attention. In International Conference on Learning Representations.
* Inan et al. (2023) Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testugine, et al. 2023. LLama guard: Llm-based input-output safeguard for human-ai conversations. arXiv preprint arXiv:2312.06674.
* Jin et al. (2023)Yiping Jin, Leo Wanner, Vishakhha Kadam, and Alexander Shvets. 2023. Towards weakly-supervised hate speech classification across datasets. In The 7th Workshop on Online Abuse and Harms (WOAH), Toronto, Canada. Association for Computational Linguistics.
* Jin et al. (2024) Yiping Jin, Leo Wanner, and Alexander Shvets. 2024. GPT-HateCheck: Can LLMs write better functional tests for hate speech detection? In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), Torino, Italia. ELRA and ICCL.
* Kumar et al. (2023) Sachin Kumar, Vidhisha Balachandran, Lucilhe Njoo, Antoniosa Anastagopoulos, and Yulia Tsvetkov. 2023. Language generation models can cause harm: So what can we do about it? an actionable survey. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 3299-3321, Dubrovnik, Croatia. Association for Computational Linguistics.
* Bras et al. (2020) Ronan Le Bras, Swabha Swayamdipta, Chandra Bhagavatula, Rowan Zellers, Matthew Peters, Ashish Sabharwal, and Yejin Choi. 2020. Adversarial filters of dataset biases. In International conference on machine learning, pp. 1078-1088. PMLr.
* Ludwig et al. (2022) Florian Ludwig, Klara Dolos, Torsten Zesch, and Eleanor Hobley. 2022. Improving generalization of hate speech detection systems to novel target groups via domain adaptation. In Proceedings of the Sixth Workshop on Online Abuse and Harms (WOAH), Seattle, Washington (Hybrid). Association for Computational Linguistics.
* Maronikolakis et al. (2022) Antonis Maronikolakis, Philip Baader, and Hinrich Schutze. 2022. Analyzing hate speech data along racial, gender and intersectional axes. In Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP), Seattle, Washington. Association for Computational Linguistics.
* Mathew et al. (2021) Binny Mathew, Punyajoy Saha, Seid Muhie Yimam, Chris Biemann, Pawan Goyal, and Animeesh Mukherjee. 2021. Hatexplain: A benchmark dataset for explainable hate speech detection. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 14867-14875.
* Mathew et al. (2020)Binny Mathew, Sandipan Sikdar, Florian Lemmerich, and Markus Strohmaier. 2020. The polar framework: Polar opposites enable interpretability of pre-trained word embeddings. In Proceedings of The Web Conference 2020, pages 1548-1558.
* Meade et al. (2022) Nicholas Meade, Elinor Poole-Dayan, and Siva Reddy. 2022. An empirical survey of the effectiveness of debiasing techniques for pre-trained language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Dublin, Ireland. Association for Computational Linguistics.
* Mozafari et al. (2020)Marzieh Mozafari, Reza Farahbakhsh, and Noel Crespi. 2020. Hate speech detection and racial bias mitigation in social media based on bert model. PloS one, 15(8):e0237861 

Nedjma Ousidhoum, Zizheng Lin, Hongming Zhang, Yangqiu Song, and Dit-Yan Yeung. 2019. Multilingual and multi-aspect hate speech analysis. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 4675-4684, Hong Kong, China. Association for Computational Linguistics.
* Park et al. [2018] Ji Ho Park, Jamin Shin, and Pascale Fung. 2018. Reducing gender bias in abusive language detection. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 2799-2804, Brussels, Belgium. Association for Computational Linguistics.
* Poletto et al. [2021] Fabio Poletto, Valerio Basile, Manuela Sanguinetti, Cristina Bosco, and Viviana Patti. 2021. Resources and benchmark corpora for hate speech detection: a systematic review. _Language Resources and Evaluation_, 55(2):477-523.
* Rahman et al. [2021] Md Mustafizur Rahman, Dinesh Balakrishnan, Dhiraju Murthy, Mucahid Kutlu, and Matthew Lease. 2021. An information retrieval approach to building datasets for hate speech detection. In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_.
* Ramponi and Tonelli [2022] Alan Ramponi and Sara Tonelli. 2022. Features or spurious artifacts? data-centric baselines for fair and robust hate speech detection. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 3027-3040, Seattle, United States. Association for Computational Linguistics.
* Ribeiro et al. [2020] Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behavioral testing of NLP models with CheckList. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 4902-4912, Online. Association for Computational Linguistics.
* Rottger et al. [2022] Paul Rottger, Haitham Seelawi, Debora Nozza, Zeerak Talat, and Bertie Vidgen. 2022. Multilingual Hate-Check: Functional tests for multilingual hate speech detection models. In _Proceedings of the Sixth Workshop on Online Abuse and Harms (WOAH)_, pages 154-169, Seattle, Washington (Hybrid). Association for Computational Linguistics.
* Rottger et al. [2021] Paul Rottger, Bertie Vidgen, Dong Nguyen, Zeerak Waseem, Helen Margetts, and Janet Pierrehumbert. 2021. HateCheck: Functional tests for hate speech detection models. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 41-58, Online. Association for Computational Linguistics.
* Santy et al. [2023] Sebastian Santy, Jenny Liang, Ronan Le Bras, Katharina Reinecke, and Maarten Sap. 2023. NLPositionality: Characterizing design biases of datasets and models. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 9080-9102, Toronto, Canada. Association for Computational Linguistics.
* Sap et al. [2019] Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019. The risk of racial bias in hate speech detection. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 1668-1678, Florence, Italy. Association for Computational Linguistics.
* Sap et al. [2020] Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Juraffsky, Noah A. Smith, and Yejin Choi. 2020. Social bias frames: Reasoning about social and power implications of language. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 5477-5490, Online. Association for Computational Linguistics.
* Suwayamidipta et al. [2020] Swabha Suwayamidipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. 2020. Dataset cartography: Mapping and diagnosing datasets with training dynamics. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 9275-9293, Online. Association for Computational Linguistics.
* Vidgen and Perczynski [2020] Bertie Vidgen and Leon Perczynski. 2020. Directions in abusive language training data, a systematic review: Garbage in, garbage out. _Plos one_, 15(12):e0243300.
* Warstadt et al. [2020] Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. BLiMP: The benchmark of linguistic minimal pairs for English. _Transactions of the Association for Computational Linguistics_, 8:377-392.
* Waseem and Hovy [2016] Zeerak Waseem and Dirk Hovy. 2016. Hateful symbols or hateful people? predictive features for hate speech detection on Twitter. In _Proceedings of the NAACL Student Research Workshop_, pages 88-93, San Diego, California. Association for Computational Linguistics.
* Waskom [2021] Michael L. Waskom. 2021. seaborn: statistical data visualization. _Journal of Open Source Software_, 6(60):3021.
* Wiegand et al. [2019] Michael Wiegand, Josef Ruppenhofer, and Thomas Kleinbauer. 2019. Detection of Abusive Language: the Problem of Biased Datasets. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 602-608, Minneapolisata. Association for Computational Linguistics.
* Yin and Zubiaga [2021] Wenjie Yin and Arkaitz Zubiaga. 2021. Towards generalisable hate speech detection: a review on obstacles and solutions. _PeerJ Computer Science_, 7:e598 

* Zampieri et al. (2019) Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019. SemEval-2019 task 6: Identifying and categorizing offensive language in social media (OffensEval). In _Proceedings of the 13th International Workshop on Semantic Evaluation_, pages 75-86, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
* Zhou et al. (2021) Xuhui Zhou, Maarten Sap, Swabha Swayamdipta, Yejin Choi, and Noah Smith. 2021. Challenges in automated debiasing for toxic language detection. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_, pages 3143-3155, Online. Association for Computational Linguistics.

## Appendix A Examples from Datasets

Table 7 presents examples from HateCheck and GPT-HateCheck.

## Appendix B Experiments on SBIC Dataset

We conduct a preliminary experiment on the Social Bias Inference Corpus (SBIC) (Sap et al., 2020) to demonstrate the utility of our framework on real-world social media data. The dataset consists of examples from Twitter, Reddit, and various hate sites. We randomly sampled 200 examples from the following each of the six target identities in the dataset: "Asians", "Black people", "Gays", "Jewish", "Muslims", "Women".

Out of the 200\(\times\)6=1,200 messages, 705 have detected emotions. Table 8 shows the most frequent emotions broken down by target identities. The emotions most associated with black people are "disgust" and "disapproval", while "anger" stands out towards Muslims. This result is consistent with the GPT-HateCheck dataset (cf. Figure 4).

Figure 7 depicts each target identity's mean "warmth" and "competence" scores. Notably, messages targeting Jewish people have the lowest "warmth" and highest "competence" scores, revealing a strong antisemitism sentiment in the samples. Meanwhile, black people and women receive the lowest "competence" scores, consistent with the result on the GPT-HateCheck dataset (cf. Table 5). Asians have the highest "warmth" score, showing that stereotypes against this group are less toxic compared to other target minorities.

## Appendix C Reliability Analysis of HS Classifiers

Debiasing contributes to different extents to different models, as shown in Table 2. We try to uncover the cause by analyzing the raw model predicted scores.

Figure 7(a) shows the distribution of predicted hateful probabilities from all models. The predicted scores of Perspective API and HateBERT are more evenly distributed, while ToxDect-roberta and LLama Guard 3 models predict almost exclusively near 0 or 1. Furthermore, Figure 7(b) shows that their predicted scores are much worse calibrated than the other two models. It explains why subtracting the bias (equivalent to adjusting the classification threshold) from the three models' predictions would contribute much less.

## Appendix D Correlation Between Functionality and Emotion

Figure 9 presents the heat map of detected emotions across functionalities. Positive statements about protected identities (F19) predominantly express admiration. Direct thread (F5) is often expressed through anger, while implicit derogation (F4) often demonstrates disapproval.

## Appendix E Details for Stereotype Analysis

Figure 10 presents the scatter plot of "warrmth" and "competence" scores assigned by the NLI model. The data points are distributed in a grid-like pattern because most "entail" and "contradict" scores are close to 0 or 1 after the softmax operation.

Table 9 shows sample messages with different "warmth" and "competence" scores assigned by the NLI model.

Figure 7: Each target identity’s mean “warmth” and “competence” scores for sample messages in SBIC dataset.

 

Figure 8: Analysis of model prediction scores.

 

Figure 10: Scatter plot in the warmth-competence semantic space of various target identities for examples in GPT-HateCheck dataset.

Figure 9: Heat map of detected emotions for each functionality in GPT-HateCheck dataset. Red color denotes hateful functionalities, and green color denotes non-hateful functionalities.

 

We use the Seaborn library (Waskom, 2021)14 to produce the kernel density estimate (KDE) plot in Figure 5. We set the parameters levels=4 (reduces the number of contour levels) and bw_adjust=1.5 (increases smoothing) to unclutter the figures and make them more legible.

Footnote 14: [https://seaborn.pydata.org/generated/seaborn.kdeplot.html](https://seaborn.pydata.org/generated/seaborn.kdeplot.html).

Figure 11 shows the correlation between the cluster centroid's distance to the origin and the average prediction accuracy of HateBERT, ToxDectroberta, Llama Guard 3-1B and -8B. The result is consistent with the experiment on Perspective API (Figure 6).

Table 10 overviews the most frequent stereotype/counter-stereotype phrases extracted from GPT-HateCheck dataset.

## Appendix F Prompts for GPT-4o

Table 11 shows the full prompt used for emotion detection (Section 3.2), and Table 12 shows the full prompt used for stereotype detection (Section 3.3).

## Appendix G Comparison of Perspective API Models

Perspective API can return six related attributes: "toxicity", "severe toxicity", "identity attack", "insult", "profanity", and "threat"15. Rottger et al. (2022) remarked that the default "toxicity" model covers a much broader concept than hate speech, which also includes abuse and profanity. On the other hand, the "identity attack" model aligns with the definition of hate speech in HateCheck(Rottger et al., 2021) and GPT-HateCheck(Jin et al., 2024). The official definitions of the two attributes are as follows:

Footnote 15: [https://developers.perspectiveapi.com/s/about-the-api-attributes-and-linguages](https://developers.perspectiveapi.com/s/about-the-api-attributes-and-linguages)

Toxicity:A rude, disrespectful, or unreasonable comment that is likely to make people leave a discussion.

Identity attack:Negative or hateful comments targeting someone because of their identity.

We validate Rottger et al. (2022)'s observation by comparing the accuracy of the two attribute models on GPT-HateCheck (Table 13). Indeed, the "identity attack" model achieved better accuracy on all target identities, demonstrating that it aligns well with the definition of hate speech.

Additionally, we plot the predicted "toxicity" and "identity attack" scores in Figure 12. We can observe that the functionalities with which the "toxicity" and "identity attack" models disagree the most (the lower right corner) are contrastive non-hateful content such as "non-hateful use of profanity", "abuse targeted at individuals", and "relaimed slurs". These examples receive a high "toxicity" score and a low "identity attack" score.

 

Figure 11: Correlation between distance to the origin of the warmth-competence space and models’ accuracy.

 



## 6 Conclusion

Figure 12: Predicted “toxicity” and “identity attack” for examples in GPT-HateCheck. Different colors denote different functionalities. \(\bullet\) denotes cases where two attribute models’ predictions disagree using a 0.5 threshold. and \(\times\) vice versa.









\begin{table}
\begin{tabular}{l|c c c c c c|c} \hline
**Emotion** & **Asians** & **Black** & **Gays** & **Jewish** & **Muslims** & **Women** & **Total** \\ \hline Disgust & 40 & **88** & 31 & 68 & 39 & 19 & 285 \\ Disapproval & **49** & 44 & 34 & 22 & 34 & 33 & 216 \\ Anger & 8 & 19 & 14 & 10 & **25** & 6 & 82 \\ Approval & 14 & **21** & 17 & 8 & 11 & 10 & 81 \\ Annoyance & 2 & 2 & 1 & 0 & 2 & **4** & 11 \\ Disappointment & **5** & 3 & 0 & 0 & 0 & 0 & 8 \\ \hline \end{tabular}
\end{table}
Table 8: Most frequent detected emotions from samples in the SBIC dataset. For each emotion, we highlight the target identity with the most examples in **bold**.

\begin{table}
\begin{tabular}{p{113.8pt}|c} \hline
**Example** & **Warm** & **Comp.** \\ \hline F*ck immigrants and their stupid cultures. This is America, not a melting pot. & -2.00 & -1.99 \\ \hline It’s time to take action against immigrants and ensure they never set foot in our country again. & -1.93 & -1.00 \\ \hline It’s a waste of time to educate women, they ’ll never be as smart as men. & -1.00 & -2.00 \\ \hline Are black people naturally more violent and aggressive? & -1.00 & -0.99 \\ \hline Black people have a natural inclination towards violence and aggression. & -1.00 & -0.46 \\ \hline Women are so emotional and irrational, it’s no wonder they can’t make good decisions. & 0.00 & -1.00 \\ \hline Being a retard is just one aspect of my incredible and diverse personality. & 1.00 & 1.00 \\ \hline No woman should be denied education or employment. & 1.00 & 1.00 \\ \hline Muslims contribute greatly to society through their diverse talents and skills. & 1.00 & 1.99 \\ \hline Being a trans woman is a source of strength and empowerment for me. & 1.85 & 1.00 \\ \hline Cheers to the muzzie community for their resilience and strength in the face of adversity. & 2.00 & 1.01 \\ \hline Sending love and empowerment to all the incredible bitches in my life. & 2.00 & 2.00 \\ \hline \end{tabular}
\end{table}
Table 9: Samples in GPT-HateCheck with different “warmth” and “competence” scores.





\begin{table}
\begin{tabular}{l l p{142.3pt}} \hline
**Target Group** & **Hate** & **Non-Hate** \\ \hline Women & just objects for men’s pleasure (8), nothing more than objects (7), terrible drivers (6), attention-seeking drama queens (5), needy and clingly (5), so weak and fragile (4), too emotional and irrational (4), so manipulative (4), only good for their looks (3) & Sending love (3), made significant contributions (3), beautiful inside and out (2), keep shining (2), resilient (2), are not inferior to men (2), have the ability to excel (2), are not meant to serve men (2) & should not be denied the right (11), should not be denied access (4), showling new ideas and perspectives (6), appreciate the diversity (2), bring unique skills and talents (2), are not a threat (2) \\ \end{tabular}
\end{table}
Table 10: Most frequent extracted stereotypes/counter-stereotypes for each target identity. We display the frequency in the bracket.



* [15] M. C. Bader, A. K. Goyal, and R. J. Brox. The role of the motion in a multi-person system. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 2378-2390, 2016.
* [16] D. P. Kingma and S. Welling. Adam: A method for stochastic optimization. _arXiv preprint arXiv:1412.6110_, 2014.
* [17] L. Chen, Y. Bengio, and Z. Xing. Learning deep neural networks with applications to image recognition. In _Advances in Neural Information Processing Systems (NIPS'09)_, pages 1319-1320, 2009.
* [18] X. Deng, W. Dong, and H. Li. Deep residual learning for image recognition. In _IEEE International Conference on Acoustics and Speech (ICASSP)_, pages 1319-1323, 2016.
* [19] T. Darrell, N. Ramanan, and E. Shelhamer. Imagenet classification via deep convolutional nets. In _International conference on computer vision and pattern recognition_, volume 1, pages 103-113, 2016.
* [20] I. Goodfellow, I. Sutskever, and J. Warde. Generative adversarial nets. In _Advances in neural information processing systems_, pages 1339-1346, 2016.
* [21] V. Khoslavi, A. Graves, A. Mohamed, and A. Rabinovich. Very accurate representations of tiny images. In _Advances in neural information processing systems_, pages 1339-1347, 2016.
* [22] A. Karpathy, S. Hochreiter, and J. Schmidhuber. Denschap: Going deeper into rectifiers. In _Advances in neural information processing systems_, pages 1339-1347, 2016.
* [23] A. Karpathy, S. Hochreiter, and J. Schmidhuber. Long short-term memory hierarchies. In _Advances in neural information processing systems_, pages 1339-1347, 2016.

[MISSING_PAGE_POST]

 



\begin{table}
\begin{tabular}{l l l l l l l l} \hline
**Model** & **Women** & **Trans** & **Gays** & **Black** & **Disabled** & **Muslims** & **Immigr.** \\ \hline Toxicity & .646 & .772 & .864 & .748 & .697 & .892 & .559 \\ Identity Attack & **.731** & **.897** & **.895** & **.879** & **.707** & **.936** & **.689** \\ \hline \end{tabular}
\end{table}
Table 13: Per target identity accuracy scores of Perspective API’s different attribute models on GPT-HateCheck.