Benchmarking Offensive and Abusive Language in Dutch Tweets
Year: 2023

Abstract:
We present an extensive evaluation of different fine-tuned models to detect instances of offensive and abusive language in Dutch across three benchmarks: a standard held-out test, a taskagnostic functional benchmark, and a dynamic test set. We also investigate the use of data cartography to identify high quality training data. Our results show a relatively good quality of the manually annotated data used to train the models while highlighting some critical weakness. We have also found a good portability of trained models along the same language phenomena. As for the data cartography, we have found a positive impact only on the functional benchmark and when selecting data per annotated dimension rather than using the entire training material.

text: The 7th Workshop on Online Abuse and Harms (WOAH), pages 69–84 July 13, 2023 c©2023 Association for Computational Linguistics

heading: 1 Introduction
text: Being able to correctly detect instances of offensive and abusive language plays a pivotal role in creating safer and more inclusive environments, especially on Social Media platforms. Since current methods for these phenomena are based on supervised techniques, a pending issue is represented by the quality of the data used to train the corresponding systems. Standard evaluation methods based on held-out test sets only provide a partial picture of the actual robustness of fine-tuned models while being silent about potential annotators’ bias, topic and author biases (Wiegand et al., 2019). Recent work has show that held-out tests may result in overly optimistic performance estimates which do not translate into real-world performance (Gorman and Bedrick, 2019; Søgaard et al., 2021). To get a realistic performance estimate, models should be evaluated on out-of-corpus data, i.e. a different data distribution but within the same language variety (Ramponi and Plank, 2020), or even on a held-out test set from a different but related domain. Out-of-corpus evaluation requires the devel-
opment of multiple datasets which can be expensive, time consuming, and, in the case of less- or poor-resources languages, unfeasible.
A complementary solution is the use of functional tests, i.e., sets of systematically generated test cases aiming at evaluating in a task-agnostic methodology trained models (Ribeiro et al., 2020; Lent et al., 2021; Sai et al., 2021; Röttger et al., 2021; Manerba and Tonelli, 2021). Functional testing enables more targeted insights and diagnostics on multiple levels. For instance, the systematic categorisation as hateful of messages containing a protected identity term (e.g., “gay”, “trans”, among others) of a system trained to detect hate speech against LGBTQIA+ people is an indicator of the weakness of the model(s) as well as of biases in the training data.
Although limited in terms of number of datasets and annotated phenomena, Dutch covers a peculiar position in the language resource panorama: it has a comprehensively annotated corpus for offensive and abusive language whose standard heldout test set does not present any overlap with the training set; it includes a dynamic benchmark for offensive language, OP-NL (Theodoridis and Caselli, 2022); and it presents a functional benchmark, HATECHEK-NL, that extends MULTILINGUAL HATECHEKCK (Röttger et al., 2022). This puts us is an optimal position to conduct an extensive benchmarking of different models for offensive and abusive language in Dutch and reflect on the potential shortcomings of the Dutch Abusive Language Corpus v2.0 (DALC-V2.0) (Ruitenbeek et al., 2022). In addition to this, we apply data cartography (Swayamdipta et al., 2020) to carve out different subsets of training materials to investigate whether this method is valid on DALC-V2.0 to identify robust and good quality training data.
Our contributions Our major contributions are the followings: (i) we present and discuss our ex-
69
tensions of HATECHEK-NL (Section 2); (ii) we apply data cartography (Swayamdipta et al., 2020) to DALC-V2.0 to investigate whether we can identify robust subsets of training data (Section 3); (iii) we conduct an extensive evaluation of different systems based on a monolingual pre-trained language model, namely BERTje (de Vries et al., 2019), against multiple test sets (Section 4).1

heading: 2 Data
text: In this section, we present the data we use to fine-tune and evaluate the models based on BERTje (de Vries et al., 2019).
DALC-V2.0 DALC-V2.0 contains 11,292 messages from Twitter in Dutch, covering a time period between November 2015 and August 2020. Messages have been annotated using a multi-layer annotation scheme compliant with Waseem et al. (2017) for two dimensions: offensive and abusive language. Offensive language in DALC-V2.0 is the same as in Zampieri et al. (2019), i.e., messages “containing any form of non-acceptable language (profanity) or a targeted offence, which can be veiled or direct”. Abusive language corresponds to “impolite, harsh, or hurtful language (that may contain profanities or vulgar language) that result in a debasement, harassment, threat, or aggression of an individual or a (social) group, but not necessarily of an entity, an institution, an organisations, or a concept.” (Caselli et al., 2021, 56–57). Each dimension is further annotated along two layers: explicitness and target. The explicitness layer is used to annotate whether a message is belonging to the positive category or not. In the former case, the values explicit (EXP) and implicit (IMP) are used to distinguish the way the positive category is realised. The target layer is used to annotate towards who or what the offence, or abuse, is directed to. Target layers inherit values from Zampieri et al. (2019), namely individual (IND), group (GRP), other (OTH).
Here we focus only on the explicitness layer, considering each dimension separately and jointly. In particular, when addressing each dimension separately, we frame the task as a binary classification by collapsing the explicit and implicit labels either into OFF and ABU for the offensive and abusive dimension, respectively. When working on both
1All code, data, and trained models are available via https://github.com/tommasoc80/DALC
dimensions jointly, we face a multi-class classification where systems must distinguish between two positive classes (OFF and ABU) and one negative (NOT). Table 1 illustrates the distribution of the data for the dimensions in analysis across the Train/Dev and standard held-out test splits.
Labels are skewed towards the negative class as in previous work (Basile et al., 2019; Davidson et al., 2017; Zampieri et al., 2019, 2020). When considering each dimension separately, the offensive dimension is larger than the abusive one (approx 33% of the total vs. ≈ 19%, respectively). In the joint setting, the OFF messages drop to ≈ 15%. This reflects the definitions of offensive and abusive language and how the two phenomena interact: abusive language is more specific and subject to a stricter set of criteria for its identification (e.g., a target must always be present), resulting in a “specialized instance” of offensive language (Poletto et al., 2020). In other words, while every abusive message is also offensive, the contrary does not hold. In their analysis of the corpus, the authors do not report evidence of any specific topic bias and they state that train and test splits have no overlap (Caselli et al., 2021; Ruitenbeek et al., 2022).
HATECHEK-NL HATECHEK-NL extends MULTILINGUAL HATECHEKCK (MHC) (Röttger et al., 2022). MHC defines hate speech as “abuse that is targeted at a protected group or at its members for being a part of that group.” (Röttger et al., 2022, 155). This definition is more specific than the language phenomena in DALC-V2.0, although it is compatible. MHC has 27 common functionalities for 10 languages, including Dutch, 18 specific for expressions of hate and nine non-hateful to contrast the hateful cases. Each test is realised by a short text uniquely identifying a gold label (e.g.,
hateful vs. non-hateful). To massively generate tests, MHC makes use of templates (Ribeiro et al., 2020). We have extended the functionalities in MHC with two extra tests to include the use of reclaimed slurs and profanities in a non hateful way (F8, F9). These two functional tests are present in the original English HATECHECK (Röttger et al., 2021) but they were excluded from MHC to maintain a more homogeneous distribution of functional tests across all languages. Röttger et al. (2022) observe that these functionalities have no direct equivalents in most of the languages in MHC, but this is not the case for Dutch. For the functionality F8 (non-hateful homonyms of slurs), we have identified four slurs that are each aimed at one of the target identities and have a non-hateful homonym. For instance, the term “f*****r” is used to refer to gay men or as a verb meaning flickering of a light, to fall or to drop something. Reclaimed slurs (F9) have been partially translated from English, excluding terms such as “n****r” and “b***h” for which we have not found evidence of their use in Dutch nor have we identified corresponding terms.
HATECHEK-NL contains 3,835 functional tests across the 29 functionalities. A total of 2,640 (68.83%) tests are hateful and 1,195 (31.16%) are non-hateful, a distribution in line with the original HATECHECK. An overview of all the functionalities in HATECHEK-NL is in Table A.1 in Appendix A. On the basis of the annotated dimensions in DALC-V2.0, we expect that models trained on offensive language may overgeneralise the identification of hateful messages, also for challenging non-hateful cases (e.g., F8, F9). On the other hand, we expect models trained on abusive language (both in isolation and jointly) to perform better, although the emphasis on “protected group and its members” in HATECHEK-NL may present an extra challenge since no specific protected group is part of DALC-V2.0.
OP-NL Offend the Politicians Benchmark (OPNL) is a dynamic test set composed by 1,500 tweets collected in March 2021 containing at least one mention of a Dutch politician from the Tweede Kamer (i.e., the Dutch House of Representatives). The messages have been annotated for offensive language using the same definition of DALC-V2.0, making OP-NL perfectly compatible and suitable as a dynamic benchmark. The labels in OP-NL are distributed as follows: 961 messages (64%) are not offensive (NOT) and 539 (36%) are offen-
sive (OFF). The ratio between non-offensive and offensive messages is 1.78 : 1, very close to the label distribution in DALC-V2.0. In this case, we expect offensive language models (in isolation or jointly with abusive language) to obtain good performances, i.e., in-line with those on DALCV2.0 for offensive language. On the contrary, models trained for abusive language are expected to struggle, mainly on the recall for the positive class.

heading: 3 Experiment settings
text: We have designed three sets of experiments for each annotated dimension to fine-tune a monolingual pre-trained language model for Dutch, BERTje, with varying training splits. All fine-tuned models are evaluated both on the official DALCV2.0 held-out test set, HATECHEK-NL, and OPNL. All pre-processing steps and fine-tuning (hyper)parameters are detailed in Appendix B for replicability.
The first block of experiment has a standard setting: for each annotated dimension (in isolation or jointly) we fine-tuned BERTje using all available training data in DALC-V2.0. We will refer to these models as standard (std).
For the second block, we use data cartography (Swayamdipta et al., 2020). The cartography approach uses a model’s confidence in the true class and the variability of this confidence across multiple training epochs (i.e., training dynamics) to identify a subset of training instances that qualify as more reliable and informative. In this way, it is possible to train a model using less data and still achieve state-of-the-art results, if not better. When plotting statistics from the training dynamics into a map, they result into a spectrum of data points: some easy (high-confidence, low variability), some hard (low-confidence, low variability), and some ambiguous (mid-range confidence, high variability). Previous work (Swayamdipta et al., 2020; Bhargava et al., 2021) has shown that, in classification tasks, the use of ambiguous data points at training time results in better models than those obtained when using the entire training split. Our goal is to test the validity of this method on DALCV2.0, a smaller dataset than those where data cartography has been successfully applied.
To identify the ambiguous data points, we have used the training dynamics from the fine-tuned models from each classification task from DALCV2.0. Given its skewed distribution and size, we
have investigated two methods to select the ambiguous data: the first (amb-dim) follows the approach in Swayamdipta et al. (2020) by retaining 1/3 of the original training data (i.e., 2,272 examples) corresponding to the top ambiguous cases per annotated dimension (separately and jointly). The second (amb-class) independently retains the top ambiguous examples for each class. In particular, we have carved three training splits of 2,272 examples where the distribution of instances per class is perfectly balanced (50-50 for binary settings, and 1/3 each for the multi-class setting). As the figures in Table 2 show, the class distribution is less skewed when compared to the original DALC-V2.0 training. For the abusive dimension, the distribution of the labels is perfectly balanced also when using the amb-dim method. The variability, across all data selection methods, is not particularly high. However, we observe a systematic difference between
the values of the amb-dim and the amb-class data, with the latter being always lower of ≈ 0.1 points. Although in both cases the selected data instances qualifies as “ambiguous”, the relatively low variability questions their efficacy as more robust training instances.
Figures 1a, 1b, and 1c illustrate the data maps of the training examples for the offensive and abusive dimension, separately and jointly. We can observe a consistent overlap between the easy and the ambiguous cases which questions the use of the ambiguous instances as effective training material from DALC-V2.0. At the same time, we observe that the hard examples are limited and well clus-
tered for each dimension separately (Figures 1a and 1b), while this does not hold in the joint case (Figure 1c). In this case, the overlap between the hard and the ambiguous instances is larger, indicating, on one side, that the classification task is more challenging and, on the other side, that the distinction among the three classes is less clear than it seems.
The last set of training data has the same size of the ambiguous data (2,272 instances) but it is randomly extracted from the original training set (rand). It is a control to better asses the effectiveness of the data cartography on DALC-V2.0. Random splits have been sampled three times with different seeds and no substitution. Table 3 illustrates their distribution. In this case, the data are skewed towards the negative class and their variability is consistently lower than that of the ambiguous ones, suggesting that the corresponding fine-tuned models should obtain worst results.

heading: 4 Results
text: For the analysis of the results we first focus on DALC-V2.0, and subsequently on HATECHEKNL and OP-NL. All fine-tuned models are compared against a baseline. For DALC-V2.0 and OPNL, we use a dummy classifier that always assigns the most frequent class, i.e., NOT; for HATECHEKNL, we use a random classifier (balanced for the hateful and non-hateful class distribution). The random classifier for HATECHEK-NL represents a more realistic baseline than a majority label classifier given the nature of the benchmark. Detailed results for each dataset are illustrated in Appendix C.
DALC-V2.0 Table 4 summarises the results on DALC-V2.0. All models largely outperform the baselines. When compared to previous work based on data cartography (Swayamdipta et al., 2020; Bhargava et al., 2021), we cannot find the same trends. Across all annotated dimensions and classification tasks (binary vs. multi-class), the use of the full training set (std) returns the best results, with a macro-F1 of 79.93 for offensive language, 72.33 for abusive language, and 58.90 for the two dimensions in conjunction. The identification of offensive and abusive language separately clearly returns better results than when the two dimensions are predicted jointly. This confirms the observations from the data maps (Figure 1c). In this latter case, the system mostly struggles to distinguish between the two positive classes. As it appears from the analysis of the predictions using a confusion
matrix, for the abusive class the largest number of errors are messages classified as OFF (125 out of 463 instances), while for the offensive class most of the messages are wrongly classified either as ABU (137 out 404 instances) or as NOT (159 out 404 instances).
DALC Train split Offensive Abusive Off. & Abu.
The use of random subsets for training (rdm) is unexpectedly competitive when compared to the std split and both ambiguous subsets from the data maps. A better impact of selecting ambiguous data per class (amb-class) to generate balanced training sets is evident for all dimensions. A further unexpected behaviour is the better performances of low variability training sets (i.e., amb-class and rdm). While the results of the amb-class set may suggest a different way of selecting robust sub-samples using data maps, the rdm blocks question the validity of data maps with small datasets.
When narrowing down the analysis to the differences between the reduced training data, we identify a peculiar behaviour of the data map splits. In particular, amb-dim and amb-class tend to overgeneralise the positive classes, with higher recall values at the cost of precision. Given the distribution of the labels (see Table 2), it is difficult to explain this behaviour in terms of class imbalance. On the other hand, this effect appears to be directly related to the use of the data maps. The impression is that the selected training data for the positive classes are too “ambiguous” for the system resulting in overgeneralisations to the detriment (mainly) of the negative class. Support in this direction comes from the results of the rdm splits where precision and recall are more balanced.
HATECHEK-NL Table 5 reports the performances of the trained models on HATECHEK-NL.
At evaluation time, for the joint model we have considered valid only the predictions for the ABU class, with the OFF labels as non-hateful messages.
In general, all fine-tune models outperform the baseline with the exceptions of the models finetuned on the rdm training data for abusive language and for offensive and abusive language jointly.
Models fine-tuned on offensive language obtain a better global accuracy. The sole deviation is represented by the model fine-tuned using the amb-dim data (59.35). This is mainly due to an overgeneralisation of the positive class in each functional test due to the broader and encompassing definition of offensive language. Being HATECHEKNLunbalanced for the hateful labels, this gives the false impression of dealing with better models. To put things in perspective, consider that the average accuracy based on the majority label (i.e., all hateful) would be 68.83% - a score that no finetuned model can beat. Furthermore, these models fail the majority of the non-hateful functional tests, as we have predicted: in this cases, the accuracy ranges from 28.77% for amb-class to 52.57% for rdm, with only the model fine-tuned on rdm being above 50% (see also Table C.1). In particular, for the most challenging non-hateful tests, such as F9 (reclaimed slurs), F11 (not hateful use of profanities), F21 (quotation of hate speech to counteract hate speech), F23–24 (non hateful messages with individual or group targets), the accuracy is consistently below 50% across all training splits. At the same time, this is an indirect positive feedback on the quality of the annotation for offensive language in DALC-V2.0: the non-hateful tests may contain language and expressions that can be perceived as offensive, and thus are flagged by the models. This is particular evident with the results for F11 where accuracy ranges between 15% and 33.67% since the presence of a profanity is flagged as offensive.
As for the use of abusive language as training,
models have a more balanced behaviour between the hateful and the non-hateful cases. In particular, across all non-hateful tests, accuracy ranges from 36.29% for amb-dim to 65.72% for rdm, with one extra model, std, being above 50% (see Table C.2). For the challenging non-hateful tests, there is only one case where the performance is consistently below 50% across all training splits, namely F16 (hate expressed via a question). For all the other non-hateful tests, the behaviour of the models is more varied with at least one or two models achieving results above 50%. To make a direct comparison with the offensive training splits, on F9 and F11 only two out four models are below 50% (amb-dim, and amb-class), while on F21 and F23–24, three out of four are below 50% (std, amb-dim, and amb-class). In addition, the accuracy of these models is consistently higher when compared to their counterparts fine-tuned using offensive language. Again, this provides an indirect feedback on the quality of the annotated data and the compatibility of the definition of abusive language in DALC-V2.0 with that of hate speech in HATECHEK-NL. The results for std and rdm on F9–F11 are particularly relevant. These functional tests are very useful to assess the generalisation functionalities of fine-tune models to distinguish between abusive/hateful content and the mere presence of slurs or swear words. Although half of the models achieve a score which is higher than 50%, there is still room for improvement: the best results for F9 is only 66.70% (with std) and that for F11 is 62.67 (with rdm).
When focusing on the joint models, the picture that emerges is more complex than it seems at a first look. First, the joint models have a lower overall accuracy. Yet, these are the models that achieve the best results for all non-hateful tests, with the accuracy ranging between 47.77% for amb-class to 76.50% for rdm, and with only one
model, amb-dim below 50% (see also Table C.3). While struggling on the positive classes - in a way that is similar to models fine-tuned on abusive language only - the pattern on the non-hateful tests indicates that the presence of an extra dimension (i.e., offensive language) seems to improve the overall precision. Although the behaviour on the DALCV2.0 held-out test may suggest that this could be due by chance rather than robustness, the performance on the challenging functionalities F9–F11 cautiously indicates the contrary. Indeed, this is the only case where only one fine-tuned model has performance below 50% (amb-class for both tests). For F11, the best accuracy (70.00% - amb-dim) is better than that of the models trained on abusive language only. Further improvements can be seen for F21 with two models above 50% (amb-dim and rdm), and F24, with three models (std, amb-dim and rdm). At the same time, issues persist on other functionalities. In particular, for F23 we observe a downgrade of the accuracy when compared to the abusive language models, and for F16, where all models are well below the 50% threshold.
A notable difference, when compared to DALCV2.0, concerns the behaviour of the data maps training splits. With the sole exception of the amb-dim from the offensive dimension, in all the other cases they help to achieve better results when compared to the use of the full training set as well as the use of random training splits. In particular, the selection of ambiguous data per dimension (amb-dim) consistently outperforms all other settings, a trend already observed for DALC-V2.0. Although for the abusive dimension we observe a better results for the amb-dim setting, the difference is not statistically significant.
Focusing on the best models, the use of offensive data allows the model to achieve 85.50% accuracy on all hateful tests on average, while it only obtains 76.88% with abusive data and 72.64% for the joint model. In only two functionalities, namely F5 (direct threat) and F7 (hateful slurs), the use of abusive language obtains better results. As for the joint model, the best results are mainly on the non-hateful functionalities, namely F19 (use of protected group identifiers in a positive statement), F20 (denouncement of hate via quote) and F22 (abuse at objects). The only hateful functionality where it obtains the best score is F26 (change of hateful term by eliminating characters).
Finally, it is clear that the annotations in DALC-
V2.0, and consequently the fine-tuned models, have limits that emerge with HATECHEK-NL while being hidden by looking at their performances of the respective DALC-V2.0 test sets. Even the use of abusive language data, which are the most similar to hate speech to fine-tune models, does not allow to properly pass all the tests. From the analysis of the results of every single functional test, it appears evident that very good results are obtained on the easy cases: as soon the expressions of hate become more subtle or fine-grained, models finetuned on DALC-V2.0, regardless of the training split and annotated dimension used, fail.
OP-NL Results for OP-NL are also reported on Table 5. Differently from HATECHEK-NL, we have converted the prediction for the ABU class of the joint model into offensive labels.
Like in the previous cases, all fine-tuned models outperform the baselines. The use of the full training data (std) results in the best scores only for the offensive and the joint models, while the model fine-tuned on abusive language only underperforms. This is actually a positive result: abusive language is more specific than its offensive counterpart, and the lower results further confirm the quality of the annotated data for each language phenomenon in DALC-V2.0. On the other hand, the results for the joint model are quite disappointing. Although competitive with the offensive dimension model, the results are ≈ 2 points lower. By looking at the distribution of the errors, we observe that the biggest sources of errors are offensive messages misclassified as NOT, a behaviour in-line with what we have observed when the same model is evaluated against the DALC-V2.0 held-out test set.
Similarly to the other evaluation settings, the amd-class data maps for the offensive and abusive models in isolation obtain competitive results when compared to the std models. When using the abusive language dimension as training material, the model fine-tuned with amd-class achieves the best macro F1 (68.75). Only for the joint model, we observe better results for the rdm splits. Lastly, the only model which across all training splits overgeneralises the positive class is the joint model. On the basis of the errors observed in DALC-V2.0 for this model, it appears that the overgeneralisation is a consequence of the conversion process of the labels for offensiveness to make the predictions compatible with OP-NL.

heading: 5 Discussion
text: Concerning data maps, we observe inconsistent behaviours of the fine-tuned models: on DALCV2.0, they are unsuccessful while they achieve either the best performances or very competitive results on HATECHEK-NL and OP-NL. By analysing the variability per class across amb-dim, ambclass, and rdm, we can see that amb-dim is the data split that contains core ambiguous cases for all classes, separately and jointly. The ambiguity for the positive class remain relatively high also in amb-class, but we observe a drop in the values for the NOT class (0.096 for offensive language, 0.062 for abusive language, and 0.095 when the two dimensions jointly). This means that in the negative class we mainly have easy examples and relatively ambiguous cases for the positive classes. A similar distribution can be observed for the variability for all rdm splits, where the variability for the negative class is substantially lower than that of the positive classes. When compared to our expectations on the behaviour of the models based on the ambiguous and the random splits, these observations help to explain the results of these models. Overall, the use of ambiguous examples only on the positive class(es) forces models to pay more attention towards the challenging cases and “disregard” the contributions of the easy ones. This confirms our explanation for the overgeneralisation of the positive class(es). As for the randomly extracted data (rdm), it appears that their better performances on DALC-V2.0 is an effect of the distribution of the training instances closer to those in the held-out test data. As for the amb-dim, there is a consistent pattern of underperformance across all test data. Rather than issues in the variability scores, i.e., not very “strong” ambiguous cases, it appears that the culprit for the low results should be found in the size of the original DALC-V2.0 training data which makes it difficult to identify good ambiguous cases with respect to the easy (or hard) ones. A similar pattern has been identified by Richburg and Carpuat (2022) when applying data cartography to low- and very-low Machine Translation settings. Furthermore, across all the test sets, we found that only for HATECHEKNL the use of ambiguous training instances leads to improved out-of-domain performance as reported by Swayamdipta et al. (2020).
When comparing the results of our models against the English HATECHECK for a BERT model fine-tuned on Davidson et al. (2017), the
core set of non-hateful functional tests (i.e., F9, F20–21, F23–24) are consistently failed in both languages. Things are quite different for MHC. In this case, the tested model is fine-tuned by concatenating three datasets whose definitions of hate speech perfectly matches the one adopted in MHC. While for F9 results are excellent, the model still struggles for F20–21, F23–242

heading: 6 Conclusions and Future Directions
text: In this paper we have presented an extensive benchmarking of models fine-tuned with DALCV2.0 across three test portions: an internal held-out test, a functional benchmark, HATECHEK-NL, and a dynamic test, OP-NL. Our experiments have investigated the reliability of DALC-V2.0 as a training set for three classification tasks: offensive and abusive language detection in isolation and jointly. Overall, addressing each task in isolation results in better performances than when running a joint experiment. The challenge here lies both in the strict connections between the two language phenomena in analysis and in the limited training data. When the fine-tuned models are applied on the out-ofcorpus test sets, we observe a good performance on OP-NL and less satisfying results on HATECHEKNL. The compatibility of the annotated phenomena in the training data actually plays a major role on this behaviour and it indicates that the quality of the annotated data in DALC-V2.0 contributes to develop robust models.
We have further investigated the effectiveness of the use of data cartography to identify more informative subsets of training materials. Unlike previous work, we observe a limited beneficial effects of this data selection method with DALC-V2.0. While the size of the dataset appears limited for an effective application of this method, we have found that selecting training subsets on the basis of the training dynamics of each annotated dimension results in better systems than when using training dynamics of the whole training split.
The results on HATECHEK-NL clearly identify limitations of the use of DALC-V2.0 to detect hate speech. While its abusive dimension can be considered a good proxy, all fine-tuned models systematically fails on core non-hateful functional tests, indicating limitations in the annotated data.
Future work will focus on extending DALCV2.0 with multiple hate speech datasets and further
2These correspond to F18–19, F21–22 in MHC.
validate the functionalities of HATECHEK-NL.
Ethical statement
Limitations HATECHECK-NL is based on MHC and it inherits its limits. However, as we have discussed in Section 2, we failed to fully implement some functional tests (e.g., reappropriation of slurs) because we were not able to find evidence during our research. To address these limitations, we plan to conduct focused interviews with Dutch organizations such as The Black Archives3.
Intended use HATECHEK-NLis a diagnostic tool for hate speech against specific protected groups. We have shown its functionalities and its impact on the evaluation of models trained both on a different language phenomenon, e.g., offensive language, and on related and comparable one, e.g., abusive language. The results have shown critical weaknesses mainly on the non-hateful tests rather than showing the strengths of the systems/models on the hateful examples. Similarly, OP-NL is a dynamic test for offensive language whose use is to help assessing the robustness and portability of models trained for offensive language detection.
Goodness of data DALC-V2.0 is the only publicly available resource for investigating the behavior of models on offensive and abusive language phenomena in Dutch. None of the annotated dimensions in DALC-V2.0 explicitly address hate speech as we discussed in Section 2. The results of the fine-tuned models on HATECHEK-NL for the abusive language dimension indicate a compatibility between abusive language in DALC-V2.0 and hate speech. The use of offensive training data on HATECHEK-NL better highlights the limitations of the data, especially as pointed out by the systematic failure on the functions F23–24. At the same time, the results on OP-NL for offensive language show a relatively good portability of the models for this language phenomenon.

heading: A HateCheck-NL: List of Functional
text: Tests

heading: B Replicability: Preprocessing and Hyperparameters
text: Preprocessing All experiments have been conducted with common pre-processing steps, namely:
• lowercasing of all words
• all users’ mentions have been substituted with a placeholder (MENTION);
• all URLs have been substituted with a with a placeholder (URL);
• all ordinal numbers have been replaced with a placeholder (NUMBER);
• emojis have been replaced with text (e.g. → :cat_face_joy:) using Python
emoji package;
• hashtag symbol has been removed from hasthtags (e.g. #kadiricinadalet → kadiricinadalet);
• extra blank spaces have been replaced with a single space;
• extra blank new lines have been removed.
Models’ hyperparameters All hyperparamters used for the experiments are reported in Table B.1.

heading: C Detailed Results

heading: Future Work
text: Future work will focus on extending DALCV2. 0 with multiple hate speech datasets and further
2These correspond to F18–19, F21–22 in MHC. validate the functionalities of HATECHEK-NL