Hate Speech Detection Datasets
Early work in hate speech (HS) detection focused on specific phenomena such as “racism”, “sexism” or “xenophobia” (Waseem and Hovy, 2016 ; Basile et al., 2019 ) or treated it as coarse-grained classification without explicitly stating the target identities involved (Davidson et al., 2017 ; Founta et al., 2018 ) . However, ignoring the target identity and the difference among related concepts such as “abusive”, “offensive”, and “toxic” may cause HS detectors to learn frequently occurring patterns in a particular context and harm generalizability (Vidgen and Derczynski, 2020 ; Fortuna et al., 2020 ) . Therefore, more recent datasets often provide additional contextual information.

Zampieri et al. ( 2019 ) introduced the OLID dataset, where each offensive message is assigned a target \in {“individual”, “group”, “other”}. Caselli et al. ( 2020 ) augmented the OLID dataset by adding new annotation dimensions like “abusiveness” and “explicitness”. Ousidhoum et al. ( 2019 ) labeled five attributes for each post: directness (2) 2 2 2 Indicates the number of unique values for each attribute. , hostility (6), target attribute (6), target group (5), and sentiment of the annotator (7). Similarly, Mathew et al. ( 2021 ) provide rich annotation, including 18 fine-grained target groups related to race, religion, gender, sexual orientation, and rationale text spans on which the labeling decision is based.

Due to data rarity, most HS detection datasets are collected using keywords, favoring explicit HS expressions (Poletto et al., 2021 ; Yin and Zubiaga, 2021 ; Rahman et al., 2021 ) , which may also cause models trained on such datasets to be over-reliant on a specific set of keywords.
To prevent an overestimation of generalizable model performance, Röttger et al. ( 2021 ) introduced HateCheck , a suite of functional tests for HS detection models. They developed 29 functionalities representing challenges in tackling online hate through interviews with NGO workers. Then, they crafted test cases for each functionality consisting of short sentences with unambiguous labels.
Templates such as “ [identity] are disgusting.” are utilized to generate test cases at scale by replacing the special token “ [identity] ” with a specific target identity.

Most recently, Jin et al. ( 2024 ) introduced GPT-HateCheck , which follows the list of HateCheck functionalities but generates examples with OpenAI’s GPT-3.5 Turbo model 3 3 3 https://platform.openai.com/docs/models/gpt-3-5-turbo instead relying on templates. They demonstrated that the new dataset has higher lexical diversity and is more realistic than the template-based HateCheck counterpart. While the new dataset is of great utility to test models’ performance in a more realistic setting, HateCheck has the advantage of allowing minimal pair analysis, which is commonly used in linguistic studies to understand models’ behavior (Warstadt et al., 2020 ) . Furthermore, although Jin et al. ( 2024 ) claimed that GPT-HateCheck covers distinct HS aspects associated with different target identities and provided promising qualitative examples, a quantitative analysis on the distribution of the aspects is missing.
We tap on both datasets’ strengths. Firstly, we conduct a minimum set analysis by differentiating models’ prediction across different target identities with templates in the HateCheck dataset. Then, we analyze fine-grained emotions and stereotypes in the more realistic and diverse GPT-HateCheck dataset and correlate them with HS prediction.