RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Bias Analysis and Mitigation
HS classifiers can absorb unintentional bias across different stages of model development, such as data sampling, annotation, and model learning (Fortuna et al., 2022 ) . Classifiers also often have a superficial understanding of language and are heavily affected by spurious correlations . Wiegand et al. ( 2019 ) found that many top words strongly correlated with the hateful category are non-offensive topical words like “football” or “commentator”. They argued that it is due to the narrow sampling strategy used to create the dataset.

Park et al. ( 2018 ) observed that HS detectors are biased towards gender identities. For example, “You are a good woman” was classified as “sexist”. They proposed mitigation approaches including debiased word embeddings, gender swap data augmentation, and fine-tuning with a larger corpus to reduce the inequality measure. On the other hand, studies on dialectal/racial bias (Davidson et al., 2019 ; Sap et al., 2019 ; Mozafari et al., 2020 ) revealed that African American English (AAE) is much more likely to be predicted as offensive. Furthermore, Maronikolakis et al. ( 2022 ) studied the intersection of gender and racial attributes and showed that the bias could be amplified for certain attribute combinations (e.g., masculine and AAE).

Zhou et al. ( 2021 ) introduced ToxDect-roberta, focusing on mitigating lexical (e.g., swear words, identity mentions) and dialectal bias towards AAE. They explored debiased training (Clark et al., 2019 ) and data filtering Le Bras et al. ( 2020 ); Swayamdipta et al. ( 2020 ) but obtained limited success. However, translating AAE to white-aligned English (WAE) automatically with GPT-3 and relabeling toxic AAE tweets whose WAE translation is predicted as non-toxic yields greater improvement for dialectal debiasing.

Fraser et al. ( 2021 ) proposed an interpretation of stereotypes towards different target identities based on the Stereotype Content Model (SCM) (Fiske et al., 2002 ) , which captures stereotypes along two primary dimensions: warmth and competence. Our stereotype analysis is inspired by Fraser et al. ( 2021 ) . However, their work employed static word embedding models to study stereotypes expressed through unigram words. In contrast, we analyze stereotypes in natural language sentences by assigning scores along the “warmth” and “competence” dimensions with an NLI model (He et al., 2021 ) .
Datasets
We use the HateCheck (Röttger et al., 2021 ) and GPT-HateCheck (Jin et al., 2024 ) datasets to conduct our analyses, as these datasets provide additional diagnostic insights. Both datasets cover the same seven target identities and 24 functionalities ( GPT-HateCheck omitted the five functionalities related to spelling variations in HateCheck ). Table 1 displays the number of documents for each target identity in both datasets, and Appendix A shows examples from the two datasets for each functionality.
Models
Below, we detail the models we experimented with: HateBERT, ToxDect-roberta, Perspective API, and Llama Guard 3. HateBERT (Caselli et al., 2021 ) and ToxDect-roberta (Zhou et al., 2021 ) are open-source models, while Perspective API is an industry-standard API developed by Jigsaw and Google’s Counter Abuse Technology team to combat online toxicity and harassment. 4 4 4 https://www.perspectiveapi.com/ . Llama Guard 3 (Inan et al., 2023 ) is a recent LLM safeguard model based on Meta’s Llama 3 (Dubey et al., 2024 ) .

HateBERT: 5 5 5 The model checkpoint: https://osf.io/tbd58/ . A pre-trained BERT model further trained with over 1 million posts from banned Reddit communities. We use the best-performing model variant fine-tuned on OffensEval dataset (Caselli et al., 2020 ) .

ToxDect-roberta: 6 6 6 https://huggingface.co/Xuhui/ToxDect-roberta-large . A toxicity detector based on Roberta-large model, aiming to reduce lexical and dialectal biases via automatic data correction. The model was trained using original and synthetically labeled examples from the Founta dataset (Founta et al., 2018 ) .

Perspective API: A Google API that uses machine learning models to identify abusive comments.
Following Röttger et al. ( 2022 ) ,
we use the “identity attack” model instead of the standard “toxicity” model because it aligns more closely with the definition of hate speech adopted in HateCheck . The two models are compared in Appendix G .

Llama Guard 3: A Llama-3.1 model fine-tuned for content safety classification. We use the pre-built category “S10 - Hate” as other categories are beyond the scope of hate speech (e.g., election, intellectual property). We use the probability of the first token (“safe”/“unsafe”) as the prediction score as recommended in the model card and 0.5 threshold to obtain the predicted label. We experiment with two model sizes: 1B and 8B 7 7 7 https://huggingface.co/meta-llama/Llama-Guard-3-1B and https://huggingface.co/meta-llama/Llama-Guard-3-8B . .
Disentangling Emotions
Hateful and non-hateful posts entail distinct emotions, which may affect the accuracy of HS detectors. We want to study whether emotions are uniformly associated with different target identities or some emotions are more prominent for certain target identities.

We prompt GPT-4 (Achiam et al., 2023 ) 10 10 10 gpt-4o-2024-05-13 model checkpoint. to identify fine-grained emotions from posts in GPT-HateCheck using the taxonomy proposed by Demszky et al. ( 2020 ) , which contains 27 distinct emotions. We provide the full prompt in Appendix F . Figure 3 presents the detected emotions ranked by frequency. 4313 out of 4438 messages have emotions detected in them (97.2%). Hateful posts focus primarily on four emotions: disgust, disapproval, anger, and fear, while non-hateful posts demonstrate a much broader range of emotions, both positive and negative ones.

Then, we analyze the distribution of target identities for each detected emotion and present the result in Figure 4 . It is manifest that the emotions expressed towards each target identity have a unique composition. In hateful examples, the dominant emotions expressed towards Muslims and immigrants are “anger” and “fear”, while the most prominent emotion towards black and disabled people is “disgust”. For non-hateful examples, “love” stands out for gays, “sadness” for black people, and “pride” and “approval” for trans. In addition, we analyze the correlation between functionalities and emotions in Appendix D .

Tabel 3 presents the fine-grained emotion level accuracy of each model for emotions with at least ten occurrences. The emotions with which models struggle the most are “annoyance”, “disapproval”, “sadness”, and “fear”.

We further group the fine-grained emotions into positive (1), negative (-1), and ambiguous (0), based on Demszky et al. ( 2020 ) ’s taxonomy and present the models’ classification accuracy in the presence of emotions with different polarities in Table 4 . The result is revelatory: All models can relatively accurately identify hateful posts with negative emotions and non-hateful posts with positive emotions. However, the accuracy degrades drastically for non-hateful posts with negative emotions, especially for HateBERT and ToxDect-roberta. 11 11 11 We observe a similar trend for hateful posts with positive or ambiguous emotions, although the number of such cases is much smaller. This result is alarming since it suggests that HS detectors are entangled with emotion polarity, and some safe posts with negative emotions, such as counter-speech expressing disapproval or sadness, are likely marked as hateful, potentially silencing the voices of vulnerable groups.
Disentangling Stereotypes
Jin et al. ( 2024 ) motivated the use of LLMs with the generation of test cases that account for distinct stereotypes associated with different target identities (e.g., criminality for immigrants and sexuality for trans). However, they did not analyze which stereotypes are covered in their dataset and whether a distinction exists among target identities.
We present an in-depth analysis of the stereotypes/counter-stereotypes in GPT-HateCheck by 1) Interpreting stereotypes based on an established social psychology theory, 2) Analyzing the correlation between stereotypes and HS prediction accuracy, and 3) Extracting and qualitatively analyzing stereotypes/counter-stereotypes.
Stereotypes Interpretation
Fiske et al. ( 2002 ; 2007 ) proposed the Stereotype Content Model, which uses the universal dimensions “warmth” and “competence”, to describe social perceptions and stereotypes. The model maps each stereotype onto interpretable semantic axes “warmth” vs. “coldness” and “competence” vs. “incompetence”.
We use a state-of-the-art NLI model (He et al., 2021 ) 12 12 12 https://huggingface.co/cross-encoder/nli-deberta-v3-large . to assign “warmth” and “competence” scores to each example in the GPT-HateCheck dataset. Inspired by Mathew et al. ( 2020 ) , we derive the scores via semantic differentials of two opposite concepts (e.g., “warmth” and “coldness”). Specifically, we test four hypotheses for each example:

\mathbb{H}_{1}^{+} : This message expresses warmth towards {target_identity} .

\mathbb{H}_{1}^{-} : This message expresses coldness towards {target_identity} .

\mathbb{H}_{2}^{+} : This message expresses that {target_identity} are competent .

\mathbb{H}_{2}^{-} : This message expresses that {target_identity} are incompetent .

The NLI model returns logit scores for the three classes: “entail”, “contradict”, and “neutral”. We first take the softmax over the three classes and derive the score for “warmth” as:

\mathbb{S}_{warmth}=\mathcal{P}_{entail}(\mathbb{H}_{1}^{+})+\mathcal{P}_{contradict}(\mathbb{H}_{1}^{-})\ -\mathcal{P}_{contradict}(\mathbb{H}_{1}^{+})-\mathcal{P}_{entail}(\mathbb{H}_{1}^{-}) (1)

We derive \mathbb{S}_{competence} similarly by replacing \mathbb{H}_{1}^{*} with \mathbb{H}_{2}^{*} in Equation 1 . Due to the softmax operation, \mathbb{S}_{warmth} and \mathbb{S}_{competence} are both bounded in the range of [-2, 2].

Figure 5 plots the kernel density estimate (KDE) in the warmth-competence semantic space. 13 13 13 We present the scatter plot and examples with different warmth-competence scores in Appendix E . While different target identity distributions overlap substantially, we can observe some patterns:

Non-hateful: Many examples related to women have high “competence” scores, highlighting a typical counter-speech pattern. Meanwhile, examples related to gays tend to have a high “warmth” score.

Hateful: Some examples related to women and disabled people receive very low “competence” scores but comparatively higher “warmth” scores, compared to other hateful examples (the lower right corner).

The mean “warmth” and “competence” scores for each target identity are presented in Table 5 . We can observe a clear push-back pattern: The higher the “coldness” or “incompetence” scores are for hateful stereotypes towards a target identity, the stronger the counter-stereotypes are in the opposite directions; consider, for illustration, the “warmth” dimension for gays and the “competence” dimension for women.
Correlation with Hate Prediction
We also investigate whether there is a correlation between the location in the warmth-competence semantic space and the HS detectors’ accuracy. To this end, we apply the k -means algorithm to cluster the data points into 10 clusters. For each cluster, we compute its centroid’s Euclidean distance to the origin and Perspective API’s accuracy. We plot the correlation between these two factors in Figure 6 . The figure reveals a clear trend that the farther from the origin (the more intense the example in the “warmth” or “competence” dimension), the more accurate the classifier is. The model seems to struggle particularly when both “warmth” and “competence” scores have a low magnitude. Other baselines demonstrate a similar trend (cf. Figure 11 ).
Stereotype Phrase Analysis
Finally, we instructed GPT-4o to extract stereotypes/counter-stereotypes from hateful and non-hateful posts, similarly to Section 3.2 . However, instead of categorizing into a fixed taxonomy, we prompt the model to identify the minimum text span supporting the prediction. In some cases, the model should slightly modify the span to match the original meaning. For example, the expected counter-stereotype for the input “No immigrant should be denied access to healthcare” is “should not be denied access to healthcare”. We present the full prompt in Appendix F and examples of extracted stereotypes/counter-stereotypes in Table 6 .

Appendix E presents an expanded list of extracted stereotypes/counter-stereotypes associated with each target identity. Some stereotypes are shared across target identities, such as “drain on the economy” for disabled people and immigrants and “attention seeking” for trans and women. Meanwhile, some stereotypes are associated with a unique target identity, such as “terrorists” for Muslims and “crime and violence” for black people. On the other hand, many counterarguments are broader, such as calling for respect, acceptance, and treatment with dignity.
Conclusions and Future Work
We presented a comprehensive analysis of various factors that influence the behavior and accuracy of HS detectors. Empirical results revealed that popular industrial and academic HS classifiers are still prone to bias due to specific mentions of the target identity. They often confuse hatefulness and the polarity of the expressed emotions, and the stereotype intensity strongly impacts the classifiers’ accuracy.
While the result may seem pessimistic, our work opens up new venues for the NLP community to improve the robustness of HS detectors further and mitigate various biases. In future work, we plan to apply our method to more datasets and models and introduce an open-source evaluation benchmark to facilitate the future development of HS detectors.
Limitations
We conduct experiments on two functionality test datasets: HateCheck (Röttger et al., 2021 ) and GPT-HateCheck (Jin et al., 2024 ) . These datasets provide rich metadata such as the target identity and the type of hate expressions (functionality). The messages in these datasets were composed by crowd-source workers or LLMs. We chose not to use HS detection datasets sampled from social media platforms because 1) they usually do not provide fine-grained target identity information and 2) they do not provide detailed information on data sampling (Fortuna et al., 2022 ) . Sampling examples for different target identities from different domains (e.g., subreddits) or using different keywords might introduce compounding factors and obscure the conclusions. Nevertheless, we demonstrate the utility of our framework by presenting preliminary experimental results on a multi-source social media dataset in Appendix B .

The main contribution of our paper is the analysis of the impact of various factors in HS detection.
The related problem of the analysis of bias mitigation methods was not in the focus of our work. While there exists an array of excellent surveys on bias mitigation methods (Meade et al., 2022 ; Kumar et al., 2023 ; Gallegos et al., 2024 ) , including a comprehensive evaluation of bias mitigation methods would take up too much space and prevent us going into depth in the analysis. As we demonstrated in Section 3.1 and Appendix C , the naïve debiasing method we use only helps when models predict well-calibrated probability-like scores. We claim neither the effectiveness nor the novelty of this method.

Furthermore, we used LLMs to detect emotions and stereotype phrases and a pre-trained NLI model to score the two stereotype dimensions. This helped us develop a prototype and validate our hypotheses rapidly. Although we performed some prompt engineering and exploration, the accuracy was not perfect. If time and resources allow, hiring domain experts to relabel the examples would yield a more reliable result.

Lastly, stereotypes and emotions towards target identities strongly depend on the cultural context. The examples in GPT-HateCheck are written by LLMs, which align best with views of Western, educated, white, and younger population (Santy et al., 2023 ) . Studying how the findings might alter under distinct socio-demographic backgrounds would be an exciting extension of this work.
Acknowledgement
This work has been partially funded by the European Commission under contract numbers HE101070278 and ISF-101080090. We are grateful for the insightful discussion with Paul Röttger and his suggestion for conducting minimal pair analysis for target identities. We thank the anonymous reviewers for the careful reading and constructive feedback so that we could improve the manuscript. Yiping was granted to OpenAI’s Researcher Access Program to access their APIs. Last but not least, We also received help from Perspective API team to increase our quota.
Toxicity:
A rude, disrespectful, or unreasonable comment that is likely to make people leave a discussion.
Identity attack:
Negative or hateful comments targeting someone because of their identity.

We validate Röttger et al. ( 2022 ) ’s observation by comparing the accuracy of the two attribute models on GPT-HateCheck (Table 13 ). Indeed, the “identity attack” model achieved better accuracy on all target identities, demonstrating that it aligns well with the definition of hate speech.

Additionally, we plot the predicted “toxicity” and “identity attack” scores in Figure 12 . We can observe that the functionalities with which the “toxicity” and “identity attack” models disagree the most (the lower right corner) are contrastive non-hateful content such as “non-hateful use of profanity”, “abuse targeted at individuals”, and “reclaimed slurs”. These examples receive a high “toxicity” score and a low “identity attack” score.
Figure 1: Overview of our approach. We analyze target identity mentions’ impact on hatefulness prediction in a minimal set experiment on the HateCheck dataset. We then extract fine-grained emotions and stereotypes from examples in GPT-HateCheck dataset to analyze the distributional difference among target identities and its impact on the classifiers’ performance.; Overview of our approach. We analyze target identity mentions’ impact on hatefulness prediction in a minimal set experiment on the HateCheck dataset. We then extract fine-grained emotions and stereotypes from examples in GPT-HateCheck dataset to analyze the distributional difference among target identities and its impact on the classifiers’ performance.
Figure 2: Normalized hatefulness predictions of models across target identities.; Normalized hatefulness predictions of models across target identities.
Figure 3: Frequent emotions detection in GPT-HateCheck dataset with at least ten occurrences.; Frequent emotions detection in GPT-HateCheck dataset with at least ten occurrences.
(a) Hateful examples.; (b) Non-hateful examples.; Distribution of target identities for each detected emotion.
(a) Non-Hateful examples.; (b) Hateful examples.; Kernel density estimate (KDE) in the warmth-competence semantic space of various target identities.
Figure 6: The cluster’s distance to the origin has a positive correlation with the hatefulness prediction accuracy.; The cluster’s distance to the origin has a positive correlation with the hatefulness prediction accuracy.
Figure 7: Each target identity’s mean “warmth” and “competence” scores for sample messages in SBIC dataset.; Each target identity’s mean “warmth” and “competence” scores for sample messages in SBIC dataset.
(a) Distribution of models’ predicted scores, separated by the ground-truth label.; (b) Models’ reliability diagrams plotting the true frequency of the positive label against its predicted probability for binned predictions ( n=20 ). The closer the dots are to the diagonal line, the more well-calibrated/reliable the predicted scores are.; Analysis of model prediction scores.
Figure 9: Heat map of detected emotions for each functionality in GPT-HateCheck dataset. Red color denotes hateful functionalities, and green color denotes non-hateful functionalities.; Heat map of detected emotions for each functionality in GPT-HateCheck dataset. Red color denotes hateful functionalities, and green color denotes non-hateful functionalities.
Figure 10: Scatter plot in the warmth-competence semantic space of various target identities for examples in GPT-HateCheck dataset.; Scatter plot in the warmth-competence semantic space of various target identities for examples in GPT-HateCheck dataset.
(a) HateBERT; (b) ToxDect-roberta; (c) Llama Guard 3 1B; (d) Llama Guard 3 8B; Correlation between distance to the origin of the warmth-competence space and models’ accuracy.
Figure 12: Predicted “toxicity” and “identity attack” for examples in GPT-HateCheck . Different colors denote different functionalities. \bullet denotes cases where two attribute models’ predictions disagree using a 0.5 threshold. and \times vice versa.; Predicted “toxicity” and “identity attack” for examples in GPT-HateCheck . Different colors denote different functionalities. \bullet denotes cases where two attribute models’ predictions disagree using a 0.5 threshold. and \times vice versa.
Number of examples for each target identity in HateCheck and GPT-HateCheck . We omit the functionalities without targeting identity, such as abusing objects or non-protected groups.
Per target identity P/R/F 1 scores of each model with and without debiasing. We highlight the best score for each model in bold .
Classification accuracy of HateBERT, ToxDect-roberta, Perspective API, and Llama Guard 3 1/8B on GPT-HateCheck grouped by the detected emotions. We highlight the three emotions with the lowest accuracy for each model in red .
Classification accuracy of HateBERT, ToxDect-roberta, Perspective API, and Llama Guard 3 1/8B on GPT-HateCheck grouped by the hatefulness label (hate) and the polarity of the detected emotions (emo). We highlight the “positive” labels in green (“non-hateful” and positive emotions) and “negative” labels in red . The ambiguous emotions are highlighted in yellow .
The mean “warmth” and “competence” scores for hateful (H) and non-hateful (N/H) examples. We highlight the scores with the highest magnitude in bold .
Extracted stereotypes/counter-stereotypes from samples in GPT-HateCheck .
Examples from HateCheck and GPT-HateCheck . The table is reproduced from Jin et al. ( 2024 ) .
Most frequent detected emotions from samples in the SBIC dataset. For each emotion, we highlight the target identity with the most examples in bold .
Samples in GPT-HateCheck with different “warmth” and “competence” scores.
Most frequent extracted stereotypes/counter-stereotypes for each target identity. We display the frequency in the bracket.
Prompts for emotion detection.
Prompts for stereotype detection.
Per target identity accuracy scores of Perspective API’s different attribute models on GPT-HateCheck .