Hierarchical Adversarial Correction to Mitigate Identity Term Bias in Toxicity Detection (2024)

Abstract: Corpora that are the fundament for toxicity detection contain such expressions typically directed against a target individual or group, e.g., people of a specific gender or ethnicity. Prior work has shown that the target identity mention can constitute a confounding variable. As an example, a model might learn that Christians are always mentioned in the context of hate speech. This misguided focus can lead to a limited generalization to newly emerging targets that are not found in the training data. In this paper, we hypothesize and subsequently show that this issue can be mitigated by considering targets on different levels of specificity. We distinguish levels of (1) the existence of a target, (2) a class (e.g., that the target is a religious group), or (3) a specific target group (e.g., Christians or Muslims). We define a target label hierarchy based on these three levels and then exploit this hierarchy in an adversarial correction for the lowest level (i.e. (3)) while maintaining some basic target features. This approach does not lower the toxicity detection performance but increases the generalization to targets not being available at training time.

heading: 1 Introduction
text: The EU Code of conduct on countering illegal hate speech online relies on the definition of hate speech as “all conduct publicly inciting to violence or hatred directed against a group of persons or a member of such a group defined by reference to race, colour, religion, descent or national or ethnic origin.”1,2 This definition points out the role of the target in hate speech, which is one form of toxicity in text, next to other offensive language (Leite et al., 2020). Targets as a constituting element already
1This paper contains some examples of toxicity. This is strictly for the purpose of explaining subtleties of the phenomenon that are important for this research. Please be aware that this content could be offensive and cause you distress.
2https://ec.europa.eu/newsroom/just/document. cfm?doc_id=42985
received some attention in previous work (Silva et al., 2016; Lemmens et al., 2021, i.a.).
Hate speech expressions vary a lot, from explicit formulations to more implicit, and sometimes even intentionally cryptic references, to bypass automatic filters. This is an issue, because data collection procedures can never be entirely fair – they suffer from being focused on specific time frames, topics, and therefore also targets (Dixon et al., 2018). The working hypothesis in our paper follows Waseem and Hovy (2016), Talat et al. (2018) and Davidson et al. (2019) who have shown that models learn regularly occurring target terms as features of toxicity, because corpora developed for annotation and training might mention potential targets predominantly in a toxic context. For toxicity directed against less frequently mentioned targets or where identity terms are not explicitely mentioned (e.g., Examples #8 and #9 in Table 1), a biased model is more apt to not detect toxicity.
We aim at improving on this situation and propose to perform adversarial correction of toxicity classifiers with regard to target identities. This leads to a challenge: How specific should the target mention that we correct for be? Correcting for specific targets might lead to a sparsity problem while correcting for the occurrence in a binary fashion might not provide sufficiently specific information
35
to the adversary. Further, the mere occurrence of a target might provide valuable information to the toxicity classifier, without confounding it. A novelty in our method is therefore our formulation of the gradient update to consider various hierarchical levels of specificity of target identities.
We assume in our experiments access to a corpus annotated on the text/instance level for toxicity (Tox) and for concrete classes of target groups (a requirement that is fulfilled by the CivilComments dataset by Borkan et al., 2019) and infer hierarchical labels from these annotations: binary (Occurrence O; identity mentioned or not), the mention of specific groups (Class C; e.g., religions, sexual orientations, ethnicities) or concrete instances of these groups (Identity I; e.g., atheist, buddhist; heterosexual, bisexual; black, asian, white). Figure 1 shows an example of a toxic text with such hierarchical annotations. Our desideratum is to correct for concrete group mentions and particular groups, such that a toxicity classifier works well also for texts that mention new identities (for instance, a not commonly targeted religion).
The contribution of this paper is therefore to answer the following research questions:
1. Does jointly learning binary target occurrence detection with toxicity detection improve the latter? (No.)
2. Does the performance of a toxicity classifier decrease if the underlying encoder is optimized to not being able to represent specific target groups or identities while maintaining target occurrence features? (No.)
3. Does adversarial correction of specific target identities lead to better generalization? (Yes.)
4. Does such correction lead to a more reasonable decision by the model? Do debiased models rely on concepts which are more meaningful for toxicity detection? (Yes.)

heading: 2 Related Work

heading: 2.1 Toxicity Detection
text: Most previous work focused on toxicity detection as binary classification (Nobata et al., 2016; Golbeck et al., 2017; Gao and Huang, 2017, i.a.) with a large set of shared tasks on the topic (Bosco et al., 2018; Wiegand et al., 2018; Zampieri et al., 2019b; Basile et al., 2019; Struß et al., 2019; Mandl et al., 2020, 2021). Schmidt and Wiegand (2017) provide a general overview of approaches to detection.
Various studies recognized the importance of fine-grained aspects of hate speech. Struß et al. (2019) propose a classification of offensive posts into subcategories of explicit and implicit aversions. Davidson et al. (2017) separate hate speech from instances of untargeted offensive language. They highlight that cases where explicit features are absent are hard to distinguish. Sachdeva et al. (2022) investigate mentions of identity groups as targets of hate speech. They find that the target detection performance suffers for cases of rarely represented identity groups. Plaza-del-Arco et al. (2021) train a model jointly for hate speech and targets.
There is a set of corpora annotated for concepts from the realm of toxicity and targets. Davidson et al. (2019) provide data annotated for hate speech and rely on Waseem and Hovy (2016) for the subcategories of sexism and racism. The Gab Hate corpus by Kennedy et al. (2022) considers hate speech and target identity groups, however does not contain fine-grained identity term labels.
In our experiments, we use the CivilComments dataset by Borkan et al. (2019). This dataset is annotated for toxicity and 24 categories of identity terms, which can be used to measure unintended biases. Koh et al. (2021) use a subset of these data to investigate shifts regarding different distributions of categories such as identity terms. They show that standard training yields substantially lower out-ofdistribution than in-distribution performance. This motivates the use of debiasing as a possible method to improve out-of-distribution performance.

heading: 2.2 Debiasing Approaches
text: Debiasing methods that either modify the training data or the training process have been applied to hate speech detection. Talat et al. (2018) highlight the issue of social biases in datasets when they are used to train detection systems which is taken up with a classifier-centric consideration by Davidson et al. (2019). Sap et al. (2019) show that annotation bias further aggravates the issue. Such biases were also found in abusive language data (Dixon et al., 2018; Wiegand et al., 2019). Biases in the data carry over to a trained model (Dixon et al., 2018). Social stereotypes against marginalized groups have been shown to be echoed in hate speech classifiers (Thylstrup and Talat, 2020; Davani et al., 2023; Gehman et al., 2020; Sap et al., 2020). To facilitate the testing of models, Röttger et al. (2021) developed the HateCheck corpus covering a range of identity terms.
Bias mitigation techniques may be applied to alter the training data directly, by masking potentially confounding tokens. These tokens have been recognized based on attention mechanisms, entity detection, and keyword recognition (Wiegand et al., 2018; Dayanik and Padó, 2020; Kumar et al., 2019). Ramponi and Tonelli (2022) detect tokens to be masked via pointwise mutual information (PMI). Furthermore, Badjatiya et al. (2019) suggest to identify tokens to be masked based on their partof-speech. Xue et al. (2023) propose a different approach than masking, namely balancing the spurious attributes across all classes.
Rather than changing the input, the training process can also be manipulated directly. Vaidya et al. (2020) suggest a classification model for toxicity detection that jointly detects identity terms. This is in contrast to our work, which aims at correcting for the target mentions’ influence instead of exploiting it. The authors show that their approach improves classification performance for comments related to some identities, however, they do not evaluate the generalization capability of the resulting model. Further, Kennedy et al. (2020) use a regularization technique that learns to contextualize mentions of identity terms and is thus less reliant on high-frequency words in unbalanced data.
In the last years adversarial correction for debiasing received some attention. It is used to “unlearn” properties of confounding concepts in the encoder of the model (Ganin et al., 2016). This approach of gradient reversal has been tested with several applications, including satire detection (correction for publication source, McHardy et al., 2019), gender identification (correcting for text topic, Dayanik and Padó, 2021) and also hate speech (language variety detection, Xia et al., 2020).

heading: 3 Methods
text: Overview. In order to avoid co-learning identity term bias in dataset-based learning of hate speech detection, our approach is to exploit the hierarchical properties of identities. The basic structure of the network used in our experiments is displayed in Figure 2. It consists of a shared encoder and four classifiers (grey boxes) which are all aggregated in parallel. The main classifier is the toxicity detector. The hierarchical dependencies of the three identity term detectors arise from the labels. On the highest level we consider identities as a binary label (Occurrence: O) which is positive if there is at least one
identity annotated for a message. The intermediate level classifier (Class: C) categorizes identities into five groups. Each of the five categories corresponds to a binary label which is positive if at least one identity from the respective category is annotated. The most fine-grained classifier considers 24 different identity labels on the lowest level (Identity: I) with binary values for each of those. Each value is positive if the respective identity is annotated for the message. Each of the mentioned classifiers consists of a single linear layer with the number of neurons corresponding to the respective number of classes. The identity classifiers for Class and Identity are both multi-label classifiers, as several of their classes can occur together in a message. With this model we can exploit the hierarchical structure of the unwanted confounding variable in parallel (potentially partially adversarial) classifiers.3
Adversarial Correction. Following Ganin et al. (2016), the gradient updates for the adversary are reversed and weighted for a correction of the encoder, while the parameters of the classifiers are updated without modification. However, we allow for multiple adversaries and apply an individual weight for each level in the identity hierarchy, namely λ1 for the binary occurrence detection, λ2 for the class of the identity term, and λ3 for the identity term on the lowest level. Hence, our parameter update for the parameters in the encoder θe is
θe := θe−η (∂JTox ∂θe −λ1 ∂JO ∂θe −λ2 ∂JC ∂θe −λ3 ∂JI ∂θe ) ,
3We use BERT (Devlin et al., 2019) as the intitialization of our encoder (concretely bert-base-uncased) and apply a dropout (probability 0.2).
where η is the learning rate. JTox is the loss function for the toxicity classifier and JO, JC, and JI are the binary cross entropy loss functions (including a sigmoid function) for each layer in the identity hierarchy, respectively. Hence, λi > 0 corresponds to adversarial learning and λi < 0 to joint learning of the encoder. The parameter updates for the classifiers (grey boxes in Figure 2) are θTox := θTox − η ∂JTox∂θTox for the Toxicity categorization, θO := θO − η ∂JO∂θO for the Occurrence categorization, θC := θC − η ∂JC∂θC for the Class categorization, and θI := θI − η ∂JI∂θI for the Identity detection. The optimizer minimizes the overall loss J = JTox + JO + JC + JI.

heading: 4 Experimental Setting
text: In the following, we explain the data that we use (§4.1) and the experimental setting (§4.2).4

heading: 4.1 Data
text: We use the CivilComments dataset (Borkan et al., 2019), the largest corpus in English annotated for both toxicity and identity terms with approximately 450,000 instances. We infer the hierarchical annotations from the 24 identity labels (see Table 1).
In these data, instances consist of individual posts as short text messages (the average instance length in the development data is 78 tokens) with all annotations on instance level. We transform the fractions of annotators that agree on a label to binary values by majority vote (following Xiang
4Our code to replicate the experiments can be accessed via https://www.uni-bamberg.de/en/nlproc/resources/ hierarchical-detox/
et al., 2021; Faal et al., 2021; Baldini et al., 2022; Lobo et al., 2022). From the 24 fine-grained annotated classes (I), we infer five more coarse-grained categories (C):
1. Gender: male, female, transgender, other gender 2. Sexual orientation: heterosexual, homosexual gay
or lesbian, bisexual, other sexual orientation
3. Religion: christian, jewish, muslim, hindu, buddhist, atheist, other religion
4. Race or ethnicity: black, white, asian, latino, other race or ethnicity
5. Disability: physical disability, intellectual or learning disability, psychiatric or mental illness, other disability
This leads to a hierarchical multi-label annotation for identities. Our goal is to mitigate the bias towards frequently mentioned identity terms during training in order to improve generalization for other cases: namely, to correctly detect toxicity in cases where no explicit target identity is mentioned (e.g., as in Examples #8 and #9), and to not detect toxicity based solely on the presence of specific target mentions (e.g., as in Examples #10 and #11). In this dataset, toxic instances contain identity terms in 61% of cases, but only 40% of non-toxic instances do (see Appendix B for more details).
For the Jigsaw Unintended Bias in Toxicity Classification challenge on Kaggle5 this dataset was split into a development set with 405,130 instances and two test sets with a total of 42,870 instances. For our experiments we randomly split this development set into training (100k instances), validation
5https://www.kaggle.com/c/jigsaw-unintended-b ias-in-toxicity-classification
for early stopping during training (50k) and hyperparameter optimization (≈255k). Further details are given in Appendix A. For evaluation, we use the official test sets combined (≈43k instances).

heading: 4.2 Model Configurations
text: We train different model configurations in order to answer our research questions (cf. Section 1). The main goal of these experiments is to assess whether the performance of a toxicity classifier decreases if the underlying encoder is optimized to not being able to represent specific target groups or identities while maintaining target occurrence features. Therefore, we configure a baseline model to compare its toxicity detection performance with debiased models. Additionally, we evaluate the performance of the different models in recognizing identity terms. This serves to identify whether, and to what extent, the different toxicity detection models pay attention to target identities. We explore different combinations of joint multi-task learning with target occurrence and adversarial correction of specific target identities to determine which effect the different levels of the identity term hierarchy x ∈ (1, 2, 3) have on the toxicity classifier. Baseline. We train a model purely for toxicity detection. In this TOX setup, the Occurrence/Class/Identity classifiers are also trained, but the encoder is not optimized via backpropagation with this information (λ1 = λ2 = λ3 = 0). Here the encoder is only trained by the toxicity detector. This setup serves the purpose of investigating whether the uncontrolled and unguided toxicity classifier relies on features which contain information regarding identity term mentions.
Debiased Baseline. In order to compare our correction method to an established bias mitigation method, we adopt the debiasing approach by Ramponi and Tonelli (2022). We refer to this model as RT (2022). It cannot rely on features of identity terms as it automatically masks the tokens most strongly associated with each identity term label. Following Ramponi and Tonelli (2022) we use normalized PMI scores to automatically extract such spurious artifacts. While Ramponi and Tonelli (2022) manually annotate the top 200 entries, we automate this process by filtering all tokens with normalized PMI values > .6. This cutoff value was chosen based on the identity term Muslim where we find the tokens muslim, muslims, islam and islamic with values > .80 but
also mosque (.65), quran (.63) and mosques (.62) amongst other tokens which are not as obviously connected: world (.71), religious (.71) or europe (.70). This approach filters a total of 751 word types for all identities. Besides operating on partially masked text, this baseline follows the configuration of the TOX setup mentioned above.
MTL (multi-task learning). Our data analysis has shown (see Section 4.1) that there is a correlation between toxicity and identity terms. We now want to test whether this carries over to the model level (cf. RQ1 in Section 1). Thus, we use MTL to guide the encoder to explicitly learn features for both toxicity detection and target occurrence in a joint setup (Model Tox+O,C,I with λ1 = −1, λ2 = λ3 = 0). To create an upper bound for the identity term detection performance on all three levels we train a model where all classifiers are combined jointly (Model Tox+O+C+I, λx = −1).
Adversarial. In order to assess the importance of target occurrence features for the detection of toxicity, we train a model for comparison in which we instruct the encoder to unlearn precisely these features. In this model identity occurrence is used as an adversary (Model Tox−O, λ1 ∈ {0.10, 0.25, 0.50, 1.00}). Additionally, as a starting point to debias the model for identities, we train a model where we use an adversary on the lowest level of the identity hierarchy (Model Tox−I, λ3 ∈ {0.10, 0.25, 0.50, 1.00}).
MTL&Adversarial. Based on the intuition that we want to guide the toxicity detector with features from mentioned targets while debiasing for identities, we combine parameterizations for multiple levels of the hierarchy. In addition to the joint toxicity and target occurrence classifier (λ1 = −1), we now debias the model for specific identity terms to understand whether this has a negative effect on the performance (cf. RQ2 in Section 1). We include an adversary via a gradient reversal layer on the lowest level of the identity term detection (λ3 ∈ {0.10, 0.25, 0.50, 1.00}) and, thus, deprive the model of the ability to distinguish between different identity terms (e.g., which specific religion is mentioned). This serves to unlearn identity term features in the encoder and to determine whether this increases the generalization ability of the model. In order to evaluate the role of the intermediate level, we include the classifier for the identity class jointly (λ1 = −1, resulting
in Model Tox+O+C−I) or as another adversary (λ1 ∈ {0.10, 0.25, 0.50, 1.00}, resulting in Model Tox+O−C−I). We hypothesize that correcting for both class and identity might lead to a more comprehensive mitigation of the identity term bias than the experimental design with only one adversary.

heading: 5 Results
text: We will now discuss the results obtained with the setting described in the previous sections. Table 2 depicts the results for the best-performing models based on the parameter λ. Further results can be found in Appendix C. Table 2 shows F1 values for different combinations of toxicity detection and identity detection on the three levels of our hierarchy. On top, we see the baseline that only optimizes the encoder with the toxicity information followed by the debiased baseline RT (2022).
We observe that RT (2022) shows a lower performance at identity classification than the baseline TOX (e.g., F1C drops from .25 to .13). Therefore, the toxicity classifier in RT (2022) learns less identity-specific features, i.e., the model is successful in reducing bias. Conversely, this also means that the baseline TOX model automatically learns identity features without being guided to do so, i.e., it in fact contains a bias. However, the results also show that debiasing following RT (2022) does lead to a drop in toxicity detection performance (F1Tox drops from .64 to .55).
RQ1: Does jointly learning binary target occurrence detection with toxicity detection im-
prove the latter? To measure if target mentions are important for toxicity detection, we now focus on specific models. We compare the performance of the baseline model (TOX) to the model which is also informed with the identity occurrence classifier (Model Tox+O,C,I) and to the model which uses a identity occurrence adversary (Model Tox−O). The results show (cf. Table 2) the F1Tox scores for all of these models on the same level. Therefore, while targets are a constituent variable of the concept of hate speech, we cannot infer from this evaluation that they are also an essential feature for toxicity detection. The toxicity classifier manages to maintain its performance level, even if we instruct the encoder to learn identity-occurrence features or, conversely, to unlearn exactly those features by adversarial correction. However, in further evaluations (cf. RQ3 below), we will see that unlearning identity occurrence features does not have a positive effect on the generalization ability of the model, which could be due to the fact that they are important for learning toxicity detection after all.
RQ2: Does the performance of a toxicity classifier decrease if the underlying encoder is optimized to not being able to represent specific target groups or identities while maintaining target occurrence features? We first evaluate overall toxicity detection performance and then address the details of identity detection performance to identify specific differences between models. We obtain the results by comparing the performance of the models corrected for identities (adversaries are marked
by −) to the baseline model (TOX). Overall the F1 score for toxicity detection (see column F1Tox in Table 2) is fairly constant in the range of .63 to .64. This shows that the toxicity detection does not suffer from the adversarial correction for identities. In contrast, the differently debiased model RT (2022) (which also has been debiased, however, by masking identity-specific tokens) shows a substantial performance drop (F1Tox .55). The test dataset used for this entire evaluation was sampled from the same source as the training dataset and is therefore also biased towards the same identities. Therefore, we presume that this evaluation is unable to demonstrate a positive effect of debiasing on toxicity detection performance. Further evaluations below under RQ3 and RQ4 show the performance gain for toxicity detection.
We now want to understand how the capability of the encoder to represent identity terms changes at different levels of the hierarchy. We see this from the performance scores in Table 2: columns F1O, F1C and F1I (for Occurrence, Class, and Identity). As expected, the identity classifiers on each of the three levels in the MTL model (Model Tox+O+C+I) outperform the models where the particular level is used as an adversary. When we use an adversary for identity detection (Model Tox−I) the performance at identity detection drops (from .07 for Model TOX to .05), i.e., the model loses some of its ability to represent identities. In settings where we emphasize learning of identity occurrence features (models with +O), the encoder also represents more identity features overall, e.g. F1I rises from .07 for Model TOX to .10 for Model Tox+O,C,I. In a model that additionally learns identity occurrence jointly, we still see the effect of the adversary on F1I. It drops from .10 for Model Tox+O,C,I to .08 for Model Tox+O−C−I. Analogously, this can also be observed for the models which additionally use the identity class classifier in a joint MTL setting (F1I drops from .38 for Model Tox+O+C+I to .24 for Model Tox+O+C−I). Thus, we conclude that adversarial correction has the desired effect of depriving the models of the ability to perform the task of identity term identification on the lowest level while maintaining target occurrence features. In addition, the procedure does not harm toxicity detection.
Finally, we investigate the role of the intermediate level in this setting. Comparing Model Tox+O+C−I to Model Tox+O−C−I shows that
using the intermediate level as additional adversary also has an effect on the lowest level as F1I drops from .24 to .08. Thus, we conclude that this further reinforces unlearning features for the lowest level and leads to a more comprehensive correction.
RQ3: Does adversarial correction of specific target identities lead to better generalization? We have now seen that the model debiased for identities on the lowest level of the hierarchy does perform as well at toxicity detection as the one that is not corrected. The performance scores for the identity term detection suggest that the encoder can no longer represent the identities to the same extent. This should enable an improved generalization across domains. We analyze this in two settings, firstly with an evaluation on target identity terms which have not been considered during training, and secondly with other datasets that have not been used during model development and training.
Regarding the first setup, we train the baseline and corrected models for the best configurations on data which has been filtered for all identities belonging to the religion class.6 Table 3 shows the evaluation of these models for toxicity detection on different fractions of the test set. In the first column we repeat the results of the models from the first experiment, which were trained on the full dataset. The last three columns show the models which were trained on non-religion data (training data: NR). Here we see that all corrected models show a drop in performance on in-domain test data
6We chose the religion class since it comprises the largest number of identities (7 out of the 24) and accounts for a substantial number of instances (7,514) in the test data. For the model RT (2022) we repeat the process of identifying the tokens that are masked on the basis of the filtered dataset.
compared to the baseline (second to last column, test data: NR). However, our corrected models show an improved performance on out-of-domain test data (last column, test data: R) in comparison to the baseline. Only the model corrected for Occurrence (Tox−O) does not show an improvement. This confirms our intuition that the correction for general target terms is not the best choice since it also includes features which are beneficial for toxicity detection. We conclude that our correction does lead to a better toxicity prediction generalization.
The second generalization evaluation is performed with out-of-distribution performance evaluations. We show the datasets that we use and the detailed results for the out-of-distribution hate speech and toxicity detection performance in Appendix D. We observe that all models show on average a similar performance on out-of-distribution data (RT (2022) being an exception). Thus, we conclude that all corrected models show similar cross-corpus performance compared to the baseline models. We assume that the corpora used represent different domains and only share the targets of hate speech, our identity terms, to a limited extent.
RQ4: Does such correction lead to a more reasonable decision by the model? Do debiased models rely on concepts which are more meaningful for toxicity detection? To understand if the corrected model relies more on concepts that do not correspond to identities – potential targets of offensive language –, we analyze the change in toxicity detection performance for specific target terms. We subdivide the test dataset into subsets mentioning specific identities and evaluate the toxicity detection performance of the different models. The detailed evaluation is given in Appendix E. We observe that the performance of the corrected models for detecting toxicity mentioning the most frequent target terms is comparable to the baseline. However, for subsets with less frequent identities, adversarial correction improves toxicity detection in two-thirds of all cases. Thus, we conclude that the corrected models rely less on identities as features and learn other, more meaningful concepts.
We further visualize this effect on selected examples using LIME (Ribeiro et al., 2016) to calculate local explanations on the words of an instance that are most important. Figure 3 displays such explanations for five selected examples from the test dataset where the debiased model Tox+O−C−I corrects errors of the baseline model. Examples
1 to 3 are all non-toxic instances which contain identity terms. The biased baseline TOX focuses only on these terms (e.g. “black”, “gay”, “transgender”, “white” and “male”) and incorrectly predicts toxicity. The debiased model corrects the error and correctly predicts the instances as non-toxic. It achieves this by also taking into consideration other tokens (most of them are marked with an orange background color) in which it finds no decisive features of toxicity. Examples 4 and 5 in Figure 3 show toxic instances with target mentions. Here both models manage to classify the instances as toxic. However, the biased baseline TOX bases its
decision mostly on the presence of identity terms (e.g. “Islamic”, “Moslems” and “white”). In contrast, the debiased model does not rely as much on the mentioned identity terms which leads to a more reasonable decision with higher weights on words such as “terrorists” and “crazy”.
We additionally investigate instances of toxicity with targets which are not included in the set annotated on the training data (such as “Germans” and “researchers”). We display LIME explanations in Figure 4 for predictions of examples from the test data which we modified to include such new target terms. These examples visualize cases of the improved generalization capability of the corrected model. Example 1 is incorrectly predicted as non-toxic by the biased baseline TOX model. The debiased model corrects the error as it is able to rely more on the new target. For Example 2, both models correctly predict the toxicity, however, the corrected model again assigns a higher weight on the new target. While this is a small-scale analysis based on a few examples, it suggests that there are cases where the corrected models use more meaningful features for toxicity detection.

heading: 6 Conclusion
text: We have shown that hierarchical adversarial correction for target identities leads to a toxicity classifier with an improved robustness. The corrected models show the same performance at toxicity detection as the biased baseline model. We presented a method to apply adversarial correction for the lowest level of hierarchical information regarding identity term
mentions. Our results have demonstrated that it is possible to simultaneously maintain basic target occurrence features. However, target occurrence has not been shown to be as important for the detection of toxicity as the related concept of hate speech would suggest. This motivates future work to divise toxicity into more fine-grained concepts such as hate speech, offensive language and profanity, in the delineation of which target occurrence features presumably play a more decisive role.
Furthermore, when debiasing for individual identity terms, our experiments with the different hierarchical levels of specificity of the confounding variable have shown that it is more beneficial to additionally correct for classes of identities. It follows that a coarser grouping of identity categories must also be considered when defining the label set for annotation in order to achieve a more comprehensive correction during training.
Overall, our correction has shown to lead to a more reasonable decision by the model as it does not exclusively rely on identity features for toxicity detection and shows better generalization capabilities. This affects real-world applications of such models in that these models are required to be demonstrably debiased and treat individual identities fairly. Additionally, this motivates that a full evaluation of model performance must test the generalization ability of such models on further datasets where different identities are mentioned, as in-distribution biases do not show up in standard evaluations with a single test dataset.
Our research opens a set of important follow-up questions. In particular, whether further fine-tuning of the training process can lead to an improved overall toxicity detection with adversarial correction. This might be achieved, e.g., by testing different individual learning rates for optimizing the classifiers, the adversary and the encoder separately or by using multiple adversaries for latent variables as presented by Kumar et al. (2019). Also, since target detection might play a more significant role to distinguish hate speech from offensive language, an evaluation of our correction approach on such data would be an important next step to fight online toxicity.

heading: Acknowledgements
text: This work has been partially supported by the CEAT project (KL 2869/1-2), funded by the German Research Foundation (DFG).
Limitations
We only ran all model configurations once due to limited time. The implementation contains randomized steps (initialization of weights, shuffling of training instances). Thus, the reported performance scores might not be entirely robust. However, our reported conclusions are based on substantial differences in performance of the different models.
Our implementation of the debiased baseline methods by Ramponi and Tonelli (2022) does only partially follow their suggested approach. While we do not consider manual annotation of top-n lists and use a fix threshold PMI value, choosing a top-n cut-off might be a more justified choice. Furthermore, Ramponi and Tonelli (2022) suggest multiple different approaches to deal with the identified spurious artifacts while we only use the removal method for comparison as a baseline.
In the experiment with filtering training data for specific identity classes, we focused on the evaluation of a setting where we filtered the religious identities. We chose the religion class since it comprises the largest number of identities (7 of the 24) and accounts for a substantial number of instances in the test data (7,514) which we can evaluate separately. For full expressiveness of the results, experiments where identities from other classes are filtered, should also be conducted. However, we presume that statistical evidence for the performance of less frequent classes (e.g. there are only 544 test instances for the disability class) might be limited.
Ethical Considerations
Potential risks. We mention examples of toxicity and hate speech which might offend readers of this paper. They are taken from empirically collected datasets and do not portray our own opinions. However, we believe that it is inevitable to investigate concrete instances when discussing detection approaches.
Reproducibility. We use datasets with annotations for toxicity and hate speech. All of these datasets are freely available for research use. We use these data for their intended use, to develop detection systems. Since we research toxicity and mentions of identity terms, the datasets have not been filtered or anonymized for such attributes.
We publish our program code for maximum transparency. The described models and predic-
tions of labels can be reproduced with this code. For training we randomly split the dataset into specific portions. As these are quite large, we believe that they are representative for the entire corpus and that the same experiments with different partitions lead to the same conclusions. Additionally, we provide a script to reproduce the random split used in our experiments to benefit future research.
We report relevant information for the used artifacts and refer to the original publications for further documentation. We describe the structure and size of the models we create. We believe that these descriptions make our approach reproducible.

heading: A Hyperparameter Optimization
text: Datasets. To limit training time, we select 100k instances as training data and an additional 50k instances as validation data to determine a suitable point for early stopping. The remaining approximately 255k instances are used for hyperparameter optimization. Optimizing all models with the described setup takes about a month on a Nvidia Quadro RTX 8000 GPU. We do not expect using more than 100k training instances to change the results of our experiments regarding the comparison of the debiasing methods. To ensure that
this is indeed the case, we trained another TOX baseline model with a larger subset for training by splitting the dev set as follows: 80% training, 10% validation-1, 10% validation-2. The resulting model (being trained on more than three times as many instances) achieves a slightly improved performance by 1.5 percentage points (F1Tox = .66) on the same test data. Hence, the chosen split does not have an impact on our conclusions.
For evaluation, we use the combined public and private test datasets from the Jigsaw Unintended Bias in Toxicity Classification challenge which does allow a straightforward comparison with past and future work.
We constrain input text instances to a maximum length of 236 tokens. This value corresponds to the 99th percentile of instance lengths in the development set. Thus, only 1% of the instances are truncated.
To deal with the skewed class distribution, we use class weights based on the inverse class frequency in the training data for all attributes in each loss.
Early stopping configuration. While all our setups operate with the same model, we monitor only relevant performance measures for each setup. Early stopping for the TOX setup monitors only the performance of the Tox classifier. In the joint setup, early stopping is based on toxicity and any active identity term classifiers in combination (all classifiers weighted equally). In the adversarial setups, early stopping is determined by monitoring the sum of the Tox classifier performance (or all MTL classifiers) and the negated adversary’s performance (weighted by 0.1).
We use early stopping with patience = 3 and reload the best model if the maximum of 10 epochs is reached.
Training process metadata. On the mentioned data, our model trains for approximately 32 minutes per epoch on a single GPU (Nvidia Quadro RTX 8000). Each model has approximately 109 million trainable parameters.
Learning rate optimization. We run each experiment with different learning rates lr ∈ {5 · 10−6, 7.5 ·10−6, 1 ·10−5, 2.5 ·10−5, 5 ·10−5}. For optimization we select the best lr value for each model according to its performance on the portion of the dataset which has not been used during training (255k instances). As performance mea-
sure we calculate the toxicity F1 score, possibly (if the model uses joint MTL) add F1 scores for joint MTL identity term classifiers and possibly (if the model uses an adversary) subtract the F1 score of the adversarial task. Since our main goal is to optimize the toxicity detection performance, we multiply the F1 scores of the identity classifiers by a reduced weight of 0.1 in this measure.

heading: B CivilComments Data
text: Table 4 shows the distribution of binary toxicity and identity term annotations in the development set from the CivilComments dataset (Borkan et al., 2019). This suggests a correlation between toxicity and mentions of identity terms, as toxic instances contain identity terms in 61% of instances, but only 40% of non-toxic instances.

heading: C Full Results
text: The performance of the models with optimized learning rate (cf. Appendix A) on the test dataset is displayed in Table 5. In addition to the performance scores shown in Table 2, this table provides the results for several models for each setup with different λx values as well as some further setups with other combinations of the hierarchical identity classifiers. Table 2 only shows the best-performing corrected models based on the in-distribution test set performance with high F1Tox and low respective identity detection F1 for each setting (see underlined values in Table 5). The additional setups included in Table 5 which are not directly related to our research questions are briefly motivated in the following.
We additionally test a setting where we are correcting for both Identity and Class (λ2, λ3 ∈ {0.10, 0.25, 0.50, 1.00}, Model Tox−C−I). This setting is based on the assumption that a combined correction for both C and I could capture the more
general set of features on the one hand, which are also sufficiently specific properties of identities on the other. We hypothesize that this setting will lead to a more comprehensive mitigation of the identity term bias than the experimental design with single adversaries.
Further, we test additional setups with all three levels, where we explore combinations incorporating the Occurrence classifier as adversary (λ1 ∈ {0.10, 0.25, 0.50, 1.00}, Model Tox−O−C−I). Here we test whether the additional correction for O does not contribute to a broader mitigation of the target identity term bias that we are aiming for and might harm the overall toxicity detection performance.
Additionally, we explore the configuration where we only correct for identity features while jointly promoting Occurrence and Class information (Model Tox+O+C−I). The idea behind this is that we may want to let the model represent features of the target occurrence as well as some distinguishing features of identity classes. There could be substantial differences in the type of toxicity that targets certain groups compared to other types of toxicity that target other groups. Therefore, we want to enable co-learning of such properties of identities in this setup. Here, we also test higher weights (λ3 ∈ {0.10, 0.25, 0.50, 1.00, 2.00, 3.00}) to empower the Identity adversary to possibly out-
weigh the three joint classifiers which presumably induce identity bias. A more powerful adversary on the lowest level might be sucessful at unlearning specific features of identities which constitute the target bias and result in an improved generalization ability of the trained model. However, by including the classifiers O and C jointly with the toxicity classifier, this model could still retain the ability to learn more general categories of targets of toxic statements.

heading: D Cross-Corpus Evaluation
text: We evaluate the performance of different models trained on the CivilComments dataset to predict hate speech on other datasets. We show the datasets that we use in Table 6. We selected publicly available datasets covering general types of hate speech and toxicity as well as datasets with a focus on specific subtypes, such as hate speech directed towards specific targets. In cases where the original authors declare a specific portion of the data as a test subset, we only use this portion in our evaluation. Otherwise we evaluate on the entire dataset.
The results for the out-of-distribution hate speech and toxicity detection performance are displayed in Table 7. The performance of all models on the ‘gk’ and ‘as’ datasets is rather low (F1 ≤ .30), presumably because these corpora are focused
on special cases of hate speech (towards specific individuals or particular ethnicities).

heading: E Evaluation of Identity-Specific Subsets
text: Table 8 shows the F1 scores of the baseline TOX model in comparison to the debiased baseline RT (2022) and our corrected models for these different portions of the test dataset. We additionally provide the number of instances which are considered in each subset. We discard all identity labels with less than 100 instances in the test dataset in this evaluation as there is presumably not enough statistical evidence for such categories.

heading: Future Work
text: For evaluation, we use the combined public and private test datasets from the Jigsaw Unintended Bias in Toxicity Classification challenge which does allow a straightforward comparison with past and future work. We constrain input text instances to a maximum length of 236 tokens. This value corresponds to the 99th percentile of instance lengths in the development set. Thus, only 1% of the instances are truncated. To deal with the skewed class distribution, we use class weights based on the inverse class frequency in the training data for all attributes in each loss. Early stopping configuration. While all our setups operate with the same model, we monitor only relevant performance measures for each setup. Early stopping for the TOX setup monitors only the performance of the Tox classifier. In the joint setup, early stopping is based on toxicity and any active identity term classifiers in combination (all classifiers weighted equally). In the adversarial setups, early stopping is determined by monitoring the sum of the Tox classifier performance (or all MTL classifiers) and the negated adversary’s performance (weighted by 0. 1). We use early stopping with patience = 3 and reload the best model if the maximum of 10 epochs is reached. Training process metadata. On the mentioned data, our model trains for approximately 32 minutes per epoch on a single GPU (Nvidia Quadro RTX 8000). Each model has approximately 109 million trainable parameters. Learning rate optimization. We run each experiment with different learning rates lr ∈ {5 · 10−6, 7. 5 ·10−6, 1 ·10−5, 2. 5 ·10−5, 5 ·10−5}. For optimization we select the best lr value for each model according to its performance on the portion of the dataset which has not been used during training (255k instances). As performance mea-
sure we calculate the toxicity F1 score, possibly (if the model uses joint MTL) add F1 scores for joint MTL identity term classifiers and possibly (if the model uses an adversary) subtract the F1 score of the adversarial task. Since our main goal is to optimize the toxicity detection performance, we multiply the F1 scores of the identity classifiers by a reduced weight of 0. 1 in this measure