Disentangling Target Identity Mentions
We use examples from HateCheck for a minimal set analysis to measure the models’ bias towards target identity mentions. Previous work either compares the subset accuracy of posts from different populations (Davidson et al., 2019 ; Zhou et al., 2021 ) or performs perturbation to create adversarial examples (Fortuna et al., 2022 ) . The first approach’s drawback is that subsets differ not only in identity mentions but also in topics, styles, and vocabulary preference, among other factors, such that it is not possible to isolate the impact of identity mentions. The limitation of the second approach is that ad hoc perturbation often focuses on particular examples instead of the big picture. Using examples from HateCheck yields two advantages: 1) It covers a wide variety of scenarios of hateful and non-hateful posts, and 2) Five annotators manually validated each example to ensure its validity and consistency with the gold-standard label.

A large share of examples in HateCheck is generated using templates. We select those templates in the dataset that instantiate exactly seven examples (one for each of the considered target identities). We exclude examples with no target identity, such as “abusing objects”. We also exclude templates that use target-specific slur words (e.g., “n*gger”, “b*tch”) because they may entail different degrees of hatefulness. This leaves us with a total of 333 templates and 333 \times 7=2,331 examples. Since identity mentions are neutral, we assume that the hatefulness prediction for examples instantiated from the same template should not alter depending on the target identity, i.e., the predicted hatefulness score for “I hate immigrants ” and “I hate trans ” should be equal regardless of the absolute predicted score. Based on this assumption, we predict hatefulness scores for all 2,331 examples. Then, we calculate the normalized prediction for each example as the predicted score minus the median score of all examples instantiated from its corresponding template. Finally, we calculate the mean of the normalized predictions for each target identity and present different models’ results in Figure 2 .

While the models show different degrees of bias towards identity mentions, the bias orientation is often the same: All models have a positive bias (predicting as more hateful) towards gays, black people, and Muslims and a negative bias towards women and disabled people. Surprisingly, ToxDect-roberta, which is trained explicitly to mitigate bias, possesses the largest bias towards identity mentions, reaching as high as +33.9% for black people. Comparing Llama Guard 3-1B and -8B, we observe that the larger LLM can better handle identity bias.

We now focus on the impact of the identity mention bias on models’ classification performance. For this experiment, we use GPT-HateCheck because its examples are more realistic. We report each model’s per-target-identity P/R/F 1 scores for the hateful category in Table 2 .

Perspective API performs consistently best among non-LLM baselines. ToxDect-roberta performs the worst, primarily due to its poor recall for the categories “women”, “trans”, “disabled people”, and “immigrants”. We hypothesize that these target identities are not well represented in the model’s training dataset due to the significant performance discrepancy among different target identities 8 8 8 Founta et al. ( 2018 ) provided no information regarding the distribution of the target identities. . The Llama Guard 3 models obtained better recall scores than other baselines, showing LLMs’ capability to catch more nuanced hateful expressions. While the larger 8B model performs better, it requires much more computation and consumes 30GB vRAM for inference only, which cannot fit into a current desktop GPU.

Debiasing could potentially reduce the impact of the identity mention bias. However, an in-depth comparison of debiasing methods is beyond the scope of this paper. Therefore, we merely apply a naïve debiasing method by subtracting the prediction by the model’s target-identity bias. 9 9 9 For example, if model A predicts a hatefulness score of 0.65 for an input related to Muslims, and it has a positive bias of 0.13 towards Muslims from the previous minimal set experiment, the debiased prediction will be 0.65 - 0.13=0.52. Target identities with strong negative bias in the minimum set experiment, such as “women” and “disabled people”, also have a much lower recall for the “hateful” category compared to other target identities. Subtracting the negative bias helped HateBERT and Perspective API improve the recall for these categories by a large margin with a much smaller sacrifice in precision. However, debiasing has little effect on ToxDect-roberta and Llama Guard 3 models because their predicted scores concentrate near 0 or 1 and are poorly calibrated, as shown in Appendix C .