Unmasking the Hidden Meaning: Bridging Implicit and Explicit Hate Speech Embedding Representations
Authors: Nicolás Benjamín Ocampo
Elena Cabrio
Serena Villata
Venue: EMNLP 2023 Findings (2023)
Keywords: Hate Speech Detection
Implicit Hate Embeddings

Abstract: Research on automatic hate speech (HS) detection has mainly focused on identifying explicit forms of hateful expressions on user-generated content. Recently, a few works have started to investigate methods to address more implicit and subtle abusive content. However, despite these efforts, automated systems still struggle to correctly recognize implicit and more veiled forms of HS. As these systems heavily rely on proper textual representations for classification, it is crucial to investigate the differences in embedding implicit and explicit messages. Our contribution to address this challenging task is fourfold. First, we present a comparative analysis of transformer-based models, evaluating their performance across five datasets containing implicit HS messages. Second, we examine the embedding representations of implicit messages across different targets, gaining insight into how veiled cases are encoded. Third, we compare and link explicit and implicit hateful messages across these datasets through their targets, enforcing the relation between explicitness and implicitness and obtaining more meaningful embedding representations. Lastly, we show how these newer representation maintains high performance on HS labels, while improving classification in borderline cases.

Unmasking the Hidden Meaning: Bridging Implicit and Explicit Hate Speech Embedding Representations
Nicolas Ocampo and Elena Cabrio and Serena Villata
Universite Côte d' Azur, CNRS, Inria, I3S, France
(nicolas-benjaml) ocampo, elena. cabrio, serena. wwtrrren.vvillaaaaaaaaa villlata)eumvv-ootedauuu fr
Abstract
coded, ambiguous or indirect language that does not immediately denote hate, but still disparages a person or a group based on protected charac- teristics such as race, gender, cultural identity, or religion (e.g.. "I think it is a bit late to think to look after the safety and the future of white peo- ple in South-Africa" the White Supremacy Forum Dataset (de Gibert et al., 2018)). The performance of current HS systems heavily relies on how coded language is represented and how well classifiers can capture the underlying semantic meaning messages through embeddings. Hence, obtaining better text representation becomes crucial in effec- tively identifying implicit HS messages.
Research automatic hate speech (HS) detec- tion has mainly focused on identifying explicit forms of hateful expressions on user-generated content. Recently, few works have started to investigate methods to address more implicit subtle abusive content. However, despite these efforts, automated systems still struggle to correctly recognize implicit and more veiled forms HS. these systems heavily rely on proper textual representations for classification, it is crucial to investigate the differences in em- bedding implicit and explicit messages. Our contribution to address this challenging task is fourfold. First, we present a comparative anal- ysis of transformer-based models, evaluating their performance across five datasets contain- ing implicit HS messages. Second, we examine the embedding representations of implicit mes- sages across different targets, gaining insight into how veiled cases are encoded. Third, we compare and link explicit and implicit hateful messages across these datasets through their targets, enforeing the relation between explicit- ness and implicitness and obtaining more mean- ingful embedding representations. Lastly, we show how these newer representation maintains high performance on HS labels, while improv- ing classification borderline cases.
In this direction, the goal of our work is to bridge the gap between explicit and implicit messages, aiming to enhance the embedding representations of SOTA models. Our contribution is fourfold: i) We analyze the embedding representations of five benchmark datasets with veiled hateful content, examining the levels of explicitness and implicit- ness, through cross-evaluation using state-of-the- art transformer models. ii) We examine the embed- ding representations of implicit messages across different target groups. Through this analysis, we gain insights into how implicit HS messages are en- coded based on their target groups iii We propose a novel approach to link explicit and implicit HS messages in the representation space. iv) We illus- trate that the newer representation space preserves strong efficacy for HS labels, while also refining classification in borderline instances. Using con- trastive learning techniques (Gunel al., 2020; Rethmeier and Augenstein, 2021; Kim et al., 2022; Tian al., 2020), we aim to push explicit and im- plicit messages effectively enforcing the uncovered relation between these two notions and thereby ob- taining more meaningful representations than those obtained through fine-tuning learning methods.
1 Introduction
The proliferation of hate speech (HS) on social media platforms has become pressing concern online social communities. While significant progress has been made in the development of HS detection methods, current SOTA models focus on detecting explicit HS, leaving implicit hate cases undetected (EISherief et al., 2021; Ocampo et al., 2023). This issue is aggravated by the sheer volume of implicit hate speech content being spread across various online platforms, necessitating automated approaches to detect them effectively.
Implicit HS detection poses unique challenges compared to its explicit counterpart: it contains
'The accompanying software can he found at: https:// tiub. com/ com/bbenjaminocampo/ooo / wwwbbrrdddiggiii siannnii_hh__ he_hs_______ ie_hs_embs embs.

---

NOTE:
This paper contains examples of language which may be offensive some readers. They do not represent the views of the authors.
both explicit and implicit hate. In particular, we target the following rescarch questions (RQ):
RQ1: How do the models' embeddings capture the HS classes? Are explicit and implicit hateful mes- suges encoded differently across different datasets? What is the extent of this variation?
2 Related Work
HS detection has been extensively studied by the research community providing multiple resources, such as lexicons (Wiegand et al., 2018; Bassig- nana al., 2018), datasets (Zampieri et al., 2019; Basile et al., 2019; Davidson et al., 2017; Founta et al., 2018), and supervised methods (Park and Fung. 2017: Gambäck and Sikdar, 2017; Wang et al., 2020; Lee et al., 2019) (for a survey, see (Poletto et al., 2021)). These studies provide solid starting point to examine the problem of abu- sive language, especially in social media messages. Lately, there has been growing interest in the detec- tion of implicit HS, which provides additional chal- lenges. Datasets specifically designed for implicit HS (Ocampo et al., 2023; Hartvigsen et al., 2022; ElSherief et al., 2021; Vidgen et al., 2021; Sap et al., 2020), more solid veiled detectors (Han and Tsvetkov, 2020), guided augmentation strategies (Nejadgholi et al., 2022; Roychowdhury and Gupta, 2023), and theoretical analysis (Jurgens et al., 2019; Waseem et al., 2017; Wiegand et al., 2021) have been recently proposed to advance in this direction. However, little attention has been dedicated to effectively represent implicit messages through embeddings on these benchmarks. Embeddings play a crucial role in the performance of classi- fiers (Pavlopoulos et al., 2017; Kshirsagar et al., 2018; Ocampo al., 2023), yet their application to capture the implicit nature of HS has been under- investigated. In this direction, (Kim al., 2022) tackles cross-dataset underperforming issues on HS classifiers and proposes a contrastive learning method that encodes a hateful post and its corre- sponding implication close in representation space, closely depending the annotated implications and without contra positioning expliciness with implicitness. (Bourgeade et al., 2023) captures topic-generic and topic-specific knowledge when trained on different data to improve generalization.
RQ2: Does grouping the test sets by target result in similar encoding patterns for explicit HS and distinct encoding patterns for implicit HS in the embeddings? RQ2 builds upon the analysis con- ducted in RQ1, but with a focus on target groups. RQ3: Can link and bring explicit and implicit embedding representations closer together within the learned embedding space through their target groups?
RQ4: How do these newer embedding representa- tions capture HS classes in comparison with RQ1?
3.2 Datasets
We carried out our analysis on the following stan- dard datasets, containing implicit HS messages: Im- plicit Subtle Hate (ISHate) (Ocampo et al., 2023), Social Bias Inference (SBIC) (Sap et al., 2020), Implicit Hate Corpus (THC) (EISherief al., 2021), Dynahate (DYNA) (Vidgen et al., 2021), and Toxi- gen (TOX) (Hartvigsen et al., 2022). We ensured that the defimitions of HS were consistent across the datasets. Specifically, for the SBIC dataset, mes- sages are considered as HS if labeled offensive and target specific group. As for the explicit- implicit HS labeling across all datasets, the pro- vided implicit labels are used for IHC and ISHate datasets. For SBIC, DYNA, and TOX, we com- puted the percentage of HS implicit messages as the ones where none of the words of the Google pro- fanity words resource was present?. The datasets were divided into train, dev, and test sets. Existing dataset splits were retained, while datasets with- out predefined splits were divided using a stratified splitting method with 60% train, 20% dev, and 20% test ratio. Table Appendix shows the per- centage of implicit/explicit instances per dataset.
3.3 Experimental Settings
Concerning our research questions (Section 3.1), to answer to RQ1 we performed fine-tuning on two state-of-the-an models commonly used for HS de- tection: BERT and HateBERT (Caselli et al., 2021). Both models were fine-tuned on each dataset using
3 Implicit and Explicit HS Embeddings
3.1 Research Questions
We will focus on the behavior of SOTA models in cross-evaluation settings, specifically on datasets containing implicit hate. The study explores the models' behavior on different HS classes, including
1 of swear words banned by Google: https:// gi tht ctw Ctm///obertJGGrrtel//oogggo

---

two-label classification approach, distinguishing between non-HS and HS messages. To ensure ro- bustness and account for randomness the train- ing process, we repeated the fine-tuning procedure five times, each time employing a different random seed. This allowed us to evaluate the performance of the models consistency. To assess the perfor- mance of the models, we cross-evaluate the bench- marks calculating the average F1-score across all fine-tuning runs. Additionally, we calculated the standard deviation to quantify the variability in performance observed across the different runs. Fi- nally, we calculated the embeddings of these mod- els using TSNE highlighting how explicit and im- plicit messages were encoded. We used the base versions size of these models with batch size of 32, weight decay of 0.01, 4 epochs, and learning rate of 2c-5. As for TSNE, we use perplexity of 30, and 1000 maximum iterations for convergence.
The contrastive loss is defined follows:
loss_cont mean ((1
(1)
+ (0aa((0,m
=))²
Where / represents the label pair (1 for positive pairs, 0 for negative pairs), is the cosine simi- larity between paired messages, and m is the mar- gin entropy loss is:
For classification, the cross-
N-1
loss_clf =
Σ i=
(gdlg((;))
+ (1 9:)log(1 -pi))
Where g is the gold label (labels of the dataset which the model fine-tuned) and p is the predic- tion. The final loss is:
total loss = loss_cont loss_clf
(3)
For RQ2, we grouped the embeddings per target the plots. To ensure consistency across datasets, we standardized the target names, addressing la- bel variations, e.g., we resolved differences like "asian" " and "asian people" by using a unified la- bel. Moreover, when message targeted multiple offensive groups (e.g. Asians and Migrants), selected the label corresponding to the predomi- nant target (among MUSLIMS, WOMEN, JEWS, LGBTQ+, BLACK PEOPLE, WHITE PEOPLE, IMMIGRANTS, ASIAN, and DISEASE).
By combining them, we optimize both the model's understanding of embeddings and classification. For RQ4, we fine-tuned both BERT and Hate- BERT using our enhanced embeddings (same set- tings of our initial RQs). Additionally, to gain more targeted diagnostic insights, models' accuracy was evaluated on three categories defined on the SBIC dataset (Non-HS, Explicit HS, and Implicit HS), and the HateCheck dataset (Röttger et al., 2021), a suite of functional tests for HS detection models.
3.4 Obtained Results and Discussion
For RQ3, we aim to validate the potential link- age between explicit and implicit messages through their target groups. To achieve this, we employ con- trastive learning techniques on the pre-trained and fine-tuned models. Contrastive learning involves defining pairs of positive and negative samples and training the model using modified loss function. our experimental settings, designate pairs of implicit and explicit messages with the same get as positive samples. For each implicit ones, randomly selected explicit message with the same target is paired. In cases where they are unavail- able or when the implicit instance lacks a target label, we randomly assign any explicit message. Negative samples consist of pairs of HS and Non- HS instances. For every Non-HS instance, one HS instance is randomly selected. Using contrastive learning facilitates the training process by pushing positive pairs closer together while pushing nega- tive pairs further apart within the embedding space.
ImlioR, Jayer Explicll HS Implictt
Non HS
-10
-20
-50
Figure 1: RQI: TSNE embeddings of the SBIC test using HateBERT fine-tuned on DYNA.
Regarding RQ1, Table Ia shows that training and evaluating HateBERT (BERT results can be found the Appendix) on the same dataset yields

---

Test
IHC
SBIC
DYУN
ISHate
TOX
Train
IHC
0,7618 # 0,0027
0,6824 # 0,0129
0,5679 = 0,0522
0,6959 = 0,0238
0,5896 + 0,0119
SBIC
0,6385 0,0310
0,8632 # 0,0099
0,6355 0,0399
0,7522 = 0,0153
0,6998 = 0,00086
DYNA
0,6717 0,0310
0,7473 0,0357
0,7860
0,0111
0,7602 0,0025
0,7526 0,0075
ISHate
0,6188 = 0,0052
0,7209 * 0,0400
0,6190 = 0,0064
0,8684 x 0,0035
0,6034 = 0,0045
TOX
0,5063 = 0,0140
0,5428 = 0,0133
0,4952 = 0,0197
0,5900 = 0,0195
0,7650 = 0,0102
(a) Cross-evaluation results with HatcBERT.
Test
IHC
SBIC
DYNA
ISHate
TOX
Train
IHC
0,7433 = 0,0107
0,6618 * 0,0173
0,5415 + 0,0092
0,6678 = 0,0058
0,5843 + 0,0153
SBIC
0,6593 0,0095
0,8620 0,0064
0,6591 0,0069
0,7583 0,0086
0,6742 = 0,0208
DYNA
0,6520 0,0102
0,7323 0,0116
0,7831 0,0030
0,7566 0,0135
0,7147 = 0,0075
ISHate
0,6253 # 0,0060
0,6753 0,0442
0,6165 + 0,0116
0,8394 = 0,0064
0,5973 # 0,0210
TOX
0,5354 0,0210
0,5637 0,0346
0,5202 0,0098
0,6180 0,0256
0,7610 + 0.0103
(b) Cross-cvaluation results with Contrustive HateBERT. Bold values indicate improvements compared Table la.
Table
HateBERT and Contrastive HateBERT cross-evaluation results with five different run seeds.
plicit and implicit text representations per target group highlighting how, in general, embeddings of explicit and implicit messages tend be linked by their target groups in representation spaces. Fi- nally, as for RQ3, Figure demonstrates that the embedding representations of explicit and implicit instances starts to overlap across all datasets when using HateBERT trained on DYNA. Additionally, Figure 4 highlights that by leveraging the targets of HS using contrastive learning, explicit and implicit messages exhibit similar representation.
senitized_
IGBTO
BLACK PEOPLE
ASINN
WOMEN
DISEASE
IMMIGRANTS
MUSLIMS
implicit layer
Explicit
Implicit
-20
80
20
20 06
Figure 2: RQ2: TSNE embeddings the SBIC test set using HateBERT fine-tuned on DYNA based their target groups. Non-HS are excluded from the plot.
Imalict
Implicit HS Non-HS
better results overall, could be expected. How- ever, even in cross-evaluation scenarios, reasonable performances are observed. Notably, among the most generalizable models, HateBERT trained on DYNA exhibits better generalization. We therefore selected HateBERT trained on DYNA as the best configuration and we plot the embeddings for the test sets of all the datasets, applying the TSNE algo- rithm. Figure I shows how explicit HS and non-HS messages encoded with clear separation, result- ing in a noticeable distance between them. On the other hand, implicit HS instances tend be inter- twined with both non-HS HS and explicit HS messages. This pattern holds true across all 5 datasets. As for the results for RQ2, Figure shows ex-
20
-40
Figure 3: RQ3: TSNE embeddings of the SBIC test set using HateBERT fine-tuned on DYNA and linking explicit and implicit instances.
As for RQ4, Table 1b illustrates that the novel representation enhances the Fl-score for certain datasets, such SBIC, TOX. and ISHate. Conversely, for other datasets like IHC and DYNA, the performance remains comparable to that of the non-contrastive approach. Table shows higher capability the contrastive
'Due to space constraints, we show only the plots of the TSNE embeddings of the SBIC test set using HateBERT fine- tuned on DYNA. The plots showing the emheddings for all the other test sets can he found in the Appendix.

---

HatcBERT
Contrastive
HamBERT
Contrastive
HatcBRRT
Contrastive
HIcBERT
Contrastive ISHate
HaRBERT
Contrastive
Cue
IHC
IHC
SBIC
SBIC
DYNA
DYNA
ISHate
TOX
TOX
quose_ah
486 1082
.499 .085
276 .213
.422
.114
857 .048
962 3: .027
.779 4: .079
.54 1: .160
0:4:0
.000 4: .009
counter, ref_ nh
.766 + 044
.617 + .086
.240 = -168
.393
.144
882 + .018
.02 + 0:37
.387 + 0138
.511 + .187
.17 +: .043
.295 + .117
ideut pos_
.41 .020
.446 1136
.301 -776
.36 0.38
849
.021
767 + .099
.272 + 002
350
470 +- .126
.75 + .133
negale, neg. oh
.502- + 054
.517 + .134
.120 060
.161
1104
448 + 037
488 + 044
164 + 030
2211 + 106
187 + 006
.194 + .077
profanity oh
.796 033
.774 076
922
899
.994
-005
1+:0
1+00
992
004
.9%6 0005
202 101
456 + .105
stor homosym_ oh
.353 + 061
.413 + 1964
393
= 162
.533
199
813 + 030
787 + 0773
680 + 056
.27 + 101
320 + 051
.473 + 0064
stor seclainedd_....
.21 0445
0964
196
.117
018
044
070
.02 .13
272 031
346 .967
target, targe_groupp group_th nh
.210 4: 0.36
.200
4: 022
623 328
810 .07
968 0
.971 1: 002
.48 4: 022
.61 1: 066
0
.006 1. .014
target
.538 049
.572
104
.28
782
033
809 061
003 .007
007
obggt_
.637 042
.622
d: 105
923
086
966
.037
969 +: .011
.985 +: .015
735 4: 011
.757 .073
000 4: .008
.034 #: .035
Table 2: Comparative accuracy performance of HateBERT vs Contrastive HateBERT trained in each dataset and evaluated across various test cases on HateCheck.
Train
Explicit
Implicit
Non-HS
highlighting their nearly indistinguishable impact on the model's s aggregate performance. Also, the importance of the Non-HS category is underscored. varying with different training datasets, yet remaining a critical component.
THC
0.8832
0.8842
0.4659
SBIC
0.8971
0.8440
0,8568
DYNA
0.6071
0.6665
0,8314
ISHate
0,5515
0.4676
0,8768
TOX
0.7938
0.8617
0,3439
Table 3: Contrastive HateBERT avg accuracy across Explicit, Implicit, and Non-HS (SBIC test set).
Hence, our experiments emphasize the impor- tance of studying implicit representations, as classi- cal training strategies cannot encode them properly (RQ1). We showed that implicit and explicit mes- sages share a connection conveying similar mes- sages to the same target (RQ2) and how contrastive learning effectively forces that property by bridg- ing explicit and implicit instances through their targets (RQ3), thereby obtaining more meaningful representations that the ones obtained through fine- tuning. Finally, we reduced biases in non-hateful implicit cases often misclassified due to trigger words or nuanced content. Our enhanced method maintains high performance on HS labels while im- proving classification in borderline cases, proving its robusiness and precision (RQ4).
sanitzzdd target
.**
DISEASE
BLACK REOPLE REOPLE
WOMEN
MUSLIMS
JEWS
LGBTQ+
ASIAN
0
IMMIGRANTS
WHITE PEOPLE
implicit layer
-20
... Expliitt HS
Implicit
-40
Figure 4: RQ3: TSNE embeddings of the SBIC test set using HateBERT fine-tuned on DYNA based on their target groups Non-HS are excluded from the plot.
4 Conclusions
Our contribution in this study is fourfold: i) We studied how models' embeddings capture HS w.r.t. explicitness and implicitness, ii) We showed how explicit and implicit HS messages result in similar encodings if grouped by their protected target, ii) We analyzed a contrastive learning method to foree this property when representing implicit text. We prove our research hypothesis on 5 HS benchmarks, moving a step forward in bridging the gap between explicitness and implicitness, and iv) We show how the newer representation space maintains high per- formance on HS labels while improving classifi- cation in borderline cases. In future work, we'll refine contrastive learning, delving into contextual pairing based on other semantic dependencies be- tween explicit and implicit cues, aiming to sharpen nuanced hate speech detection.
HateBERT in accurately classifying challenging Non-HS messages across all five datasets. A significant reduction in false positives is also observed in HateCheck categories such as quoted announcements (counter_quote_ng)) direct references (counter_ref_ni) pos- itive identifiers (ident_pos_nh), negated hateful remarks (negate_neg_nh)) non- hateful profanity (profanity_nh), reclaimed slurs (slur_reclaimed. (slur_homonym_nh), well as targeted abuse directed at individuals (target_indiv objects (target_obj_nh), and non-protected groups (target_group_nh) Additionally, Table 3 indicates that both Explicit and Implicit cat- egories exhibit similarly high accuracy levels,
homonym slurs

---

Limitations
search Agency (ANR) with the reference number ANR- 19-P3IA-0002.
In this study, are aware of some key issues, one of which pertains to the selection of positive and negative samples in contrastive learning. The effec- tiveness of the algorithm heavily relies on the care- ful selection of these pairs. While our investigation demonstrates that explicit and implicit messages exhibit a relationship through their target groups across five distinct datasets, it is important to ac- knowledge that this assumption may not always hold true. Additionally, ensuring clear separation between non-hateful and HS instances can be chal- lenging due to the heterogeneity of each category. Moreover, the efficacy of our approach is con- tingent upon the availability and alignment of tar- get information across the datasets. While target information is commonly provided in benchmark datasets, different datasets may address various pro- tected characteristics. Our approach assumes that there is some degree of overlap terms of target groups among the selected datasets.
References
Valerio Basile, Cristina Bosco, Elisabetta Fersini, Dehora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti. 2019. SemEval-2019 task Multilingual detection of hate speech against immigrants and women in Twitter. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 54-63, Min- neapolis, Minnesota, USA. Association for Compu- tational Linguistics.
Elisa Bassignana, Valerio Basile, and Viviana Patti, 2018. Hurtlex: A Multilingual Lexicon of Words to Hurt. In Elena Cabrio, Alessandro Mazzei, and Fabio Tamburini, editors, Proceedings of the Fifth Italian Conference on Computational Linguistics CLiC-it 2018, pages 51 31-56. Accademia University Press.
Tom Bourgeade, Patricia Chiril, Farah Benamara, and Véronique Moriceau. 2023. What did you learn to hate? topic-oriented analysis of generalization in hate speech detection. In Proceedings of the 17th Conference of the European Chapter of the Asso- ciation for Computational Linguistics, pages 3495- 3508, Dubrovnik, Croatia. Association for Computa- tional Linguistics.
Furthermore, the selection of pairs when linking explicit and implicit messages can vary in terms of the number of combinations. However, it is important to note that the number of pairs in- creases, the training requirements tend grow sig- nificantly, resulting in slower training processes, This trade-off between the number of pairs and training efficiency should be carefully considered when implementing the approach.
Tommaso Caselli, Valerio Basile, Jelena Mitrović, and Michael Granitzer. 2021. HateBERT: Retraining BERT for abusive language detection in English. In Proceedings of the 5th Workshop on Online Abuse and Harms (WOAH 2021), pages 17-25, Online. As- sociation for Computational Linguistics
Ethics Statement
Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated Hate Speech De- tection and the Problem of Offensive Language. Pro- ceedings of the International AAAI Conference on Weh and Social Media, 11(1):112-515 Number: I.
This paper uses collection of HS examples ex- tracted from linguistic resources commonly em- ployed for HS detection, ensuring their indepen- dence from the authors' personal opinions. The datasets used in this study have been meticulously handled to address privacy concerns associated with user data. While we acknowledge the poten- tial for misuse, we firmly believe that developing robust HS classifiers essential in combating the proliferation of harmful content. this regard, our work represents a significant contribution towards this objective and encourages further exploration and investigation within the scientific community.
Ona de Gibert, Naiara Perez, Aitor García-Pablos, and Montse Cuadros. 2018. Hate speech dataset from white supremacy forum. In Proceedings of the 2nd Workshop on Abusive Language Online (ALW2), pages 1 11-20, Brussels, Belgium. Association for Computational Linguistics.
Mai ElSherief, Calcb Ziems, David Muchlinski, Vaish- navi Anupindi, Jordyn Seybolt, Munmun De Choud- hury, and Diyi Yang. 2021. Latent hatred: A bench- mark for understanding implicit hate speech. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 345-363, Online and Punta Cana, Dominican Republic. Asso- ciation for Computational Linguistics.
Acknowledgements
This work has been supported by the French gov- emmment, through the 31A Côte d' Azur Investments in the Future project managed by the National Re-
Antigoni Founta, Constantinos Djouvas, Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Gi- anluca Stringhini, Athena Vakali, Michael Sirivianos,

---

and Nicolas Kourtellis. 2018. Large Scale Crowd- sourcing and Characterization of Twitter Abusive Behavior. Proceedings of the International AAAI Conference on Web and Social Media, 12(1). Num- ber: 1.
Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 5517-5529. Dublin, Ireland. Association for Computational Lin- guistics.
Nicolas Ocampo, Ekaterina Sviridova, Elena Cabrio, and Serena Villata. 2023. An in-depth analysis of implicit and subtle hate speech messages. In Proceed- ings of the 17th Conference of the European Chap- ter of the Association for Computational Linguisties. pages 1997-2013, Dubrovnik, Croatia. Association for Computational Linguistics.
Bjöm Gambäck and Utpal Kumar Sikdar. 2017. Using convolutional neural networks to classify hate-speech. In Proceedings of the First Workshop on Abusive Lan- guage Online, pages 85-90, Vancouver, BC, Canada. Association for Computational Linguisties.
Beliz Gunel, Jingfei Du, Alexis Conneau, and Ves Stoyanov. 2020. Supervised contrastive learning for pre-trained language model fine-tuning. CoRR, abs/2011 1.04433...
Ho Park and Pascale Fung. 2017. One-step and two- step classification for abusive language detection on Twitter. In Proceedings of the First Workshop Abusive Language Online, pages 41-45, Vancouver, BC, Canada. Association for Computational Linguis- tics.
Xiaochuang Han and Yulia Tsvetkov. 2020. Fortify- ing toxic speech detectors against veiled toxicity. In Proceedings of the 2020 Conference Empirical Methods Natural Language Processing (EMNLP), pages 7732-7739, Online. Association for Computa- tional Linguisties.
John Pavlopoulos, Prodromos Malakasiotis, Juli Baka- gianni, and Ion Androutsopoulos 2017. Improved abusive comment moderation with user embeddings. In Proceedings of the 2017 EMNLP Workshop: Nat- ural Language Processing meets Journalism, pages 51-55, Copenhagen, Denmark. Association for Com- putational Linguistics.
Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap. Dipankar Ray, and Ece Kamar. 2022. ToxiGen: large-scale maacine--gneratte dataset for adversarial and implicit hate speech detection. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pages 3309-33222 Dublin, Ireland. Association for Computational Linguistics.
Fabio Poletto, Valerio Basile, Manuela Sanguinetti, Cristina Bosco, and Viviana Patti. 2021. Resources and benchmark corpora for hate speech detection; a systematic review. Language Resources and Evalua- tion, 55(2):477-523.
David Jurgens, Libby Hemphill, and Eshwar Chan- drasekharan. 2019. A just and comprehensive strat- egy for using NLP address online abuse. In Pro- ceedings of the 57th Annual Meeting of the Asso- ciationfor Computational Linguistics. pages 3658- 3666, Florence, Italy. Association for Computational Linguistics.
Nils Rethmeier and Isabelle Augenstein. 2021. A primer on contrastive pretraining in language processing: Methods, lessons learned and perspectives. CoRR, abs/2102.12982.
Youngwook Kim, Shinwoo Park, and Yo-Sub Han. 2022. Generalizable implicit hate speech detection using contrastive learning. In Proceedings of the 29th Inter- national Conference on Computational Linguistics, pages 6667-6679, Gyeongju. Republic of Korea. In- ternational Committee on Computational Linguistics.
Paul Röttger, Bertie Vidgen, Dong Nguyen, Zeerak Waseem, Helen Margetts, and Janet Pierrehumbert. 2021. HateCheck: Functional tests for hate speech detection models. nual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 41-58, Online. Association for Computational Linguistics.
Proceedings of the 59th An-
Rohan Kshirsagar, Tyrus Cukuvac, Kathy McKcown, and Susan McGregor. 2018 Predictive embeddings for hate speech detection on Twitter. In Proceedings of the 2nd Workshop on Abusive Language Online (ALW2). pages 32, Brussels, Belgium. Associa- tion for Computational Linguistics.
Sumegh Roychowdhury and Vikram Gupta. 2023. Data- efficient methods for improving hate speech detec- tion. In Findings of the Association for Compu- tational Linguistics: FACI. 2023, pages 125-132, Dubrovnik, Croatia. Association for Computational Linguistics.
Ju-Hyoung Lee, Jun-U Park, GoongWWon Cha, and Yo- Sub Han. 2019. Detecting context abusiveness using hierarchical deep learning. In Proceedings of the Second Workshop on Natural Language Processing for Internet Freedom: Censorship, Disinformation. and Propaganda, pages 10,19, Hong Kong, China. Association for Computational Linguisties.
Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Juraf- sky, Noah A. Smith, and Yejin Choi. 2020. Social bias frames: Reasoning about social and power im- plications of language. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5477-5490 Online, Association for Computational Linguistics.
Isar Nejadgholi, Kathleen Fraser, and Svetlana Kir- itchenko. 2022. Improving generalizability in im- plicitly abusive language detection with concept ac- tivation vectors. In Proceedings of the 60th Annual

---

Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan. Cordelia Schmid, and Phillip Isola. 2020. What makes for good views for contrastive learning. CoRR, 0iss20055.1223.
Bertie Vidgen, Tristan Thrush, Zeerak Waseem, and Douwe Kiela. 2021. Learning from the worst: Dy- namically generated datasets to improve online hate detection. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume 1: Long Papers), pages 1667-1682, Online. Association for Computa- tional Linguistics.
Kunze Wang. Dong Lu,, Caren Han, Siqu Long, and Josiah Poon. 2020. Detect all abuse! toward univer- sal abusive language detection models. In Proceed- ings of the 28th International Conference putational Linguisties, pages 6366 6376, Barcelona, Spain (Online). International Committee on Compu- tational Linguistics.
Com-
Zeerak Waseem, Thomas Davidson, Dana Warmsley, and Ingmar Weber. 2017. Understanding abuse: typology of abusive language detection subtasks. In Proceedings of the First Workshop Abusive Lan- guage Online, pages 78-84, Vancouver, BC, Canada. Association for Computational Linguistics.
Michael Wiegand, Josef Ruppenhofer, and Elisabeth Eder. 2021. Implicitly abusive language what does it actually look like and why are we not getting there? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 576-587, Online. Association for Computa- tional Linguisties.
Michael Wiegand, Josef Ruppenhofer, Anna Schmidt, and Clayton Greenberg. 2018. Inducing a lexicon of abusive words a feature-based approach. In Pro- ceedings of the 2018 Conference of the North Amer- ican Chupter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). pages (046-1056 New Orleans, Louisiana. Association Computational Linguis- tics.
Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019. SemfEvl--2019 task 6: Identifying and cat- egorizing offensive language in social media (Of- fensEval). In Proceedings the 13th International Workshop on Semantic Evaluation, pages 75-86, Min- neapolis, Minnesota, USA. Association for Compu- tational Linguistics.

---

A Datasets Statistics and Details
Table displays statistics related to the datasets used in our study, including IHC, SBIC, DYNA, TOX, and ISHate. The table presents the percent- age of implicit and explicit instances per dataset, along with their distribution across set partitions, and target groups distribution.
B Evaluation results with BERT
In this section we show the evaluation of the BERT and Contrastive BERT models for RQI and RQ4 specified in sections 3.1 and 3.3.
Table 5a demonstrates BERT's efficacy in cross- evaluation contexts, mirroring the results seen with HateBERT, Among the models, SBIC stands out, displaying superior generalization capabilities. Conversely, Table 5b illustrates that while models like SBIC, IHC, and TOX reap advantages from contrastive learning, others experience slight dip in performance, though maintaining an overall high-quality output.
Moving to Table it's evident that seg- ment of the enhancement attributed the precise categorization of challenging Non-HS messages prevalent across all five datasets. This precision underscores a more conservative and meticulous approach in classifying message as Hateful.
Finally, Table 7 highlights BERT's consistent performance, boasting high accuracy in handling Non-HS instances for each dataset. This is achieved without compromising the emphasis on discerning between Explicit and Implicit labels, thereby ensuring that the model maintains a bal- anced focus on varied content nuances.
TSNE embeddings for RQ1, RQ2, and RQ3 in all datasets
This section presents the TSNE results for each research questions RQ1, RQ2, and RQ3, illustrated in Figures 5, 6, 7, and These visualizations are generated from the embeddings captured by Hate- BERT, specifically trained on the DYNA dataset and evaluated on all datasets.

---

Datasets
Source
Size
% Implicit
% Explicit
% Hate Class
IHC
Twitter
21480
86,7017
13.2983
38,1238
SBIC
Social Media
144649
58,9586
41,0414
38,9667
DYNA
Human-Machine Machine Adv.
41144
58,0654
41,9346
53,8961
HSHete
Media
71,5686
28,4314
66,3313
TOX
GPT-3
9866
42,381
57,619
48,956
Table 4: Comparing toxic language datasets, % Hate Class, % Implicit, and % Explicit are the percent labeled hate, implicit hate, and explicit hate, respectively.
Train
Test
IHC
SBIC
DYNA
ISHate
TOX
IHC
0,7625 0,0063
0,6891 0,0072
0,5511 0.0077
0,6824 0,0068
0,6074 0,0329
SBIC
0,6603 0,0301
0,8568 0,0092
0,6500 0,0310
0,7581 0,0167
0,6939 0,0086
DYNA
0,6660 0.0046
0,7412 - 0.0098
0,7831 0,0027
0,7515 + 0,0039
0,7501 + 0,0036
ISHate
0,6214 0,0040
0,7480 0,0058
0,6279
0,0056
0,8635
0,0029
0,6012 + 0,0090
TOX
0,5455 0,0091
0,5855 0,0312
0,5193 0,0204
116140=000277
0,7824 + 0,0094
(a) Cross-cvaluation results with BERT.
Train
Test
IHC
SBIC
DYNA
ISHate
TOX
IHC
0.7418 0.0066
0.6877 0.0157
0.5463 0.0183
0.6748 0.0117
0,6141 + 0.0100
SBIC
0.6630 0.0049
0.8602 0.0034
0.6390
0.0056
0.7493 0.0093
0.6574 0.0145
DYNA
0.6431 0.0074
0.6973
0.0241
0.6979
0.0080
0.7414 0.0124
0,7001 0.0145
ISHate
0.6202 0.0156
0.6808 0.0398
0.6233
0.0078
0.8350 = 0.0097
0.5797 = 0.0206
TOX
0.5501 0.0396
0.5741 0,0410
0.5611 0.0264
0.6086 0.0349
0.7645 0.0103
(b) Cross-evaluation results with Contrastive BERT. Bold values indicate improvements compared Table
Table 5: BERT and Contrastive BERT cross-evaluation results with five different run seeds.
Contrutive
BERT
Contrassive IStlate
BERT
Contrastive
Test Case
IHC
BBC
SBIC
SBIC
Dynallate
Dynallate
ISt
ToxiGen
TesiGen
colantor quote
297 .99
.410 .770
282 .143
.095
843 007
319 044
.34 .000
007 001
1121
counter ref oh
329 + 055
.67 .149
201
493
264 .733
848 + 033
908 + .18
410 + 033
555 .17
380 + 152
359 + 193
ident pos nogatc_neg nh
.40 039
.406 00X
233
.36 1665
846 044
808 137
317 + 060
.54 044
556 130
.24 +: 1944
346 =: 0154
.427 113
D48
833
.128 =: 007
406 + 055
.92 2: .662
093 +: 022
.194 2: .99
218 4: 146
.262 2: 176
profanity nhh
810 0054
.822 :: .52
1966
115
1
780 1: 203
688 1: 168
slur_ hur_hhoooyym, nh
.513
.045
.500 120
.20
890
513 2 .12
860 1 003
.875 1. 015
.80
1: 038
.53 1: 0138
667 1. 100
667 1 113
slue ret__reclamedd... pb
.165
1: .060
.215 1: 894
.38
113
.411 081
815 1: .045
.815 1: 003
.64
1: 064
.790
0049
430 1: 050
420 1. 156
carget_grou__nt
.684
.031
.716 849
732
267
7771
.77
987
.013
.99 1. .016
.416
033
.599
.00
.01
036
.844 119
langet_indiv_ni _nh
.557
.063
.498
135
797
351
938
.38
1 110
1 1.0
.65
028
.188
.077
.12 1. 127
074 1. 073
langeL_ok Lobi_objjun
.677
.067
7711
111
988
828
978
.088
110
110
775
0331
853
005
228 1822
218 .175
Table 6: Comparative accuracy performance BERT vs Contrastive BERT trained in each dataset and evaluated across various test cases on HateCheck.
Train
Explicit
Implicit
Non-HS
IHC
0.8754
0.8678
0.5206
SBIC
0.8920
0.8348
0.8607
DYNA 0.5796
0.6113
0.8068
ISHate 0.5542
0,4387
0.9067
TOX 0.6892
0.7727
0.4408
Table 7: Contrastive BERT avg accuracy across Explicit, Implicit, and Non-HS (SBIC test set).

---

O00I111 19915
MEDKES
(a) DYNA IHC
(b) DYNA - SBIC
(c) DYNA - DYNA
DEEITIC SEVIO
DOVID
(d) DYNA > ISHate
(e) DYNA -> TOX
Figure 5: RQ1: TSNE embeddings of the test sets for all the datasets using HateBERT fine-tuned on DYNA.
GET DEL DNORT
AGAN
PROCE
MUSLPO
OSEASE
DAETS
(a) DYNA
IHC
(b) DYNA
SBIC
(c) DYNA
DYNA
GISTES SNER
WOMEN
DWS
LGITON
DECACE
CCEAAE
YORL
(d) DYNA ISHate
(e) DYNA TOX
Figure 6: RQ2: TSNE embeddings of the test sets for all the datasets using HateBERT fine-tuned on DYNA based their target groups Non-hateful instances excluded from these plots.

---

MURAN MeeRs
SALL
(a) DYNA IHC
(b) DYNA
SBIC
(c) DYNA - DYNA
DHOR HS S0s
Ore NasiE
(d) DYNA + ISHate
(c) DYNA + TOX
Figure 7: RQ3: TSNE embeddings of the test sets for all the datasets using HateBERT fine-tuned on DYNA and linking explicit and implicit instances
SIP MARE SIORT
MRS WE LIENT
ELACK BIDEINATS
DOMM
CREAAE
ANITS
BLACK PEPLL
FipACE
(a) DYNA
IHC
(b) DYNA
SBIC
(c) DYNA
DYNA
SITSINE SHEER
WOMEN
YEONL JON
PESCL
KSAU
(d) DYNA ISHate
(e) DYNA TOX
Figure RQ3: TSNE embeddings test sets for all the datasets using HateBERT fine-tuned on DYNA based their target groups. Non-hateful instances excluded from these plots.


Reviews:
[{"review_id": "kOf7NhxEwx", "reviewer": "Reviewer_TtZW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments.\n\n**1)** Regarding the paper's novelty, our work i) examines the variability of explicit and implicit hate speech (HS) across different target groups using five benchmark datasets, ii) we propose a method that bridges explicit and implicit HS in the representation space through contrastive learning techniques. We analyze those representations both before and after applying our technique. To strengthen the conclusions of our work, we conducted a supplemental experiment fine-tuning both BERT and HateBERT using our enhanced embeddings, adhering to the settings of our initial RQs by using multiple seeds and averaging accuracy on HateCheck (see table below). We show that the enhanced models **(CONT columns)** are better at correctly classifying challenging Non-HS messages across the five datasets. They also notably reduce false positives in HateCheck categories like quoted announcements (`counter_quote_nh`), direct references (`counter_ref_nh`), positive identifiers (`ident_pos_nh`), negated hateful remarks (`negate_neg_nh`), non-hateful profanity (`profanity_nh`), reclaimed slurs (`slur_reclaimed_nh`), homonym slurs (`slur_homonym_nh`) and targeted abuse at individuals (`target_indiv_nh`), objects (`target_obj_nh`), and other non-protected groups (`target_group_nh `). As a result, we reduce biases in non-hateful implicit cases often misclassified as Explicit HS due to trigger words or nuanced content. Our enhanced method maintains high performance on HS labels while improving classification in borderline cases, affirming the robustness and precision of our approach. \n\nWe will include this new table as well as results on the cross-evaluation for BERT and HateBERT for Explicit, Implicit, and Non-HS categories in the camera-ready version if the paper gets accepted.\n\n| Test Case\t\t | HateBERT IHC | CONT IHC | HateBERT SBIC | CONT SBIC | HateBERT DynaHate | CONT DynaHate | HateBERT ISHate | CONT ISHate | HateBERT ToxiGen | CONT ToxiGen |\n|-------------------|--------------|----------|---------------|-----------|--------------------|---------------|-----------------|-------------|------------------|--------------|\n| counter_quote_nh  | 0,486        | 0,499    | 0,276         | 0,422     | 0,857              | 0,962         | 0,379           | 0,524        | 0,000            | 0,010         |\n| counter_ref_nh    | 0,576        | 0,617    | 0,240         | 0,393     | 0,882              | 0,902         | 0,387           | 0,521        | 0,177            | 0,295         |\n| ident_pos_nh \t | 0,441        | 0,446    | 0,301         | 0,361     | 0,849              | 0,767         | 0,272           | 0,350        | 0,470            | 0,575         |\n| negate_neg_nh     | 0,502        | 0,517    | 0,120         | 0,161     | 0,448              | 0,486         | 0,164           | 0,211        | 0,087            | 0,194         |\n| profanity_nh      | 0,796        | 0,774    | 0,922         | 0,994     | 1,000              | 1,000         | 0,992           | 0,996        | 0,292            | 0,456         |\n| slur_homonym_nh   | 0,353        | 0,413    | 0,393         | 0,513     | 0,813              | 0,787         | 0,680           | 0,827        | 0,320            | 0,473         |\n| slur_reclaimed_nh | 0,217        | 0,277    | 0,472         | 0,711     | 0,891              | 0,879         | 0,741           | 0,802        | 0,272            | 0,346         |\n| target_group_nh   | 0,710        | 0,700    | 0,623         | 0,810     | 0,968              | 0,971         | 0,448           | 0,561        | 0,000            | 0,006         |\n| target_indiv_nh   | 0,538        | 0,572    | 0,655         | 0,951     | 1,000              | 1,000         | 0,782           | 0,809        | 0,003            | 0,003         |\n| target_obj_nh     | 0,637        | 0,622    | 0,923         | 0,966     | 0,969              | 0,985         | 0,735           | 0,757        | 0,006            | 0,034         |\n\n\n**2)** Regarding RQ3 and the novel architecture. We modify the training optimizer to calculate cosine similarities between positive and negative pairs of embeddings corresponding to the CLS tokens in language models. Positive pairs are drawn together, while negative ones are pushed apart. These enhanced embeddings then go through a classification layer. Our training then combines cross-entropy loss with contrastive loss. The cross-entropy loss uses the gold labels from the datasets instead of the sample pairs. Hence, it is not merely a pipeline but a novel, integrated model architecture that influences learning, representation, and classification.\n\n**3)** Figure 1 illustrates the baseline results *without* our technique, which is why the pink and blue dots are substantially mixed. This shows the need for our novel, integrated approach, which aims to better segregate these categories in the representation space (see table above).\n\n**4)** Running experiments across varied datasets aimed to show how our experiments consistently support our claims across disparate contexts, topics, and social media. This suggests that our method is robust, has broader applicability, and can be effectively generalized.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-structured, and the figures included are illustrative.\nBesides, the paper conducts extensive experiments on five datasets, and demonstrates an adequate understanding of hate speech detection.", "reasons_to_reject": "Although hate speech detection has been a topic of concern for some time, this paper does not significantly contribute to the field. There is little novelty in either methodology or conclusion. Specifically, the paper primarily utilizes existing visualization tools (TSNE) and pre-trained models (BERT and HateBERT). The proposed method for RQ3 resembles a pipeline rather than a novel model architecture. Furthermore, the findings are limited, with no further analysis of linguistic or causal mechanisms, providing little implications for research, practice, or society. Therefore, the paper seems more like a visualized experiment report rather than a linguistic analysis or modeling work.\nMoreover, the conclusions drawn from the experimental results are weak, with some lacking supporting evidence. For instance, in section 3.4, the paper claims that the embeddings of explicit hate speech and non-hate speech are clearly separated, but figure 1 indicates that pink and blue dots actually mix to a large extent. Similarly, in figure 4, while the embeddings of explicit and implicit hate speech are pulled closer as expected, the distance between the embeddings of hate speech and non-hate speech remains unclear. This raises doubts about the effectiveness of the proposed method in practical hate speech detection scenarios.\nWhile some of the research questions are intriguing, the analysis presented in the paper is not sufficiently thorough. For instance, regarding the target group of hate speech, it is suggested to conduct quantitative research on the explicitness of such speech and further analyze the similarities and differences among them. \nAdditionally, using different datasets with varying standards for implicit hate speech labeling (or even hate speech labeling) may lead to inconsistencies, potentially causing reduced accuracy and reliability in the results.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper discusses the problem of hate speech from a computational perspective, which is motivated by practical scenarios where hate speech on social media often appears in implicit forms and automated detection is necessary. In exploring the impact of explicitness and target groups on hate speech detection, the paper presents visualizations of hate speech embeddings and proposes new ones using contrastive learning. Through these visualizations, the paper reveals the learned representations of hate speech across different groups and levels of explicitness, while using contrastive learning to enhance the representations of implicit hate speech.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "When using an abbreviation for the first time in the paper, it is recommended to provide its full name as well."}, "extra_text": {}}, {"review_id": "VZaHuNYQD4", "reviewer": "Reviewer_UB9Q", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments.\n\n**1)** In response to RQ1 appearing redundant, our aim was not just to show that performance varies in a cross-data setting but to quantify the extent of this variation. Thanks for the suggested reference. We will cite it in the paper. Following that, and to strengthen the conclusions of our work, we conducted a supplemental experiment fine-tuning both BERT and HateBERT using our enhanced embeddings, adhering to the settings of our initial RQs by using multiple seeds and averaging accuracy on HateCheck (see table below). We show that the enhanced models **(CONT columns)** are better at correctly classifying challenging Non-HS messages across the five datasets. They also notably reduce false positives in HateCheck categories like quoted announcements (`counter_quote_nh`), direct references (`counter_ref_nh`), positive identifiers (`ident_pos_nh`), negated hateful remarks (`negate_neg_nh`), non-hateful profanity (`profanity_nh`), reclaimed slurs (`slur_reclaimed_nh`), homonym slurs (`slur_homonym_nh`) and targeted abuse at individuals (`target_indiv_nh`), objects (`target_obj_nh`), and other non-protected groups (`target_group_nh `). As a result, we reduce biases in non-hateful implicit cases often misclassified as Explicit HS due to trigger words or nuanced content. Our enhanced method maintains high performance on HS labels while improving classification in borderline cases, affirming the robustness and precision of our approach. \n| Test Case   \t  | HateBERT IHC | CONT IHC | HateBERT SBIC | CONT SBIC | HateBERT DynaHate | CONT DynaHate | HateBERT ISHate | CONT ISHate | HateBERT ToxiGen | CONT ToxiGen |\n|-------------------|--------------|----------|---------------|-----------|--------------------|---------------|-----------------|-------------|------------------|--------------|\n| counter_quote_nh  | 0,486    \t| 0,499\t| 0,276     \t| 0,422 \t| 0,857          \t| 0,962     \t| 0,379       \t| 0,524    \t| 0,000        \t| 0,010     \t|\n| counter_ref_nh\t| 0,576    \t| 0,617\t| 0,240     \t| 0,393 \t| 0,882          \t| 0,902     \t| 0,387       \t| 0,521    \t| 0,177        \t| 0,295     \t|\n| ident_pos_nh      | 0,441    \t| 0,446\t| 0,301     \t| 0,361 \t| 0,849          \t| 0,767     \t| 0,272       \t| 0,350    \t| 0,470        \t| 0,575     \t|\n| negate_neg_nh \t| 0,502    \t| 0,517\t| 0,120     \t| 0,161 \t| 0,448          \t| 0,486     \t| 0,164       \t| 0,211    \t| 0,087        \t| 0,194     \t|\n| profanity_nh  \t| 0,796    \t| 0,774\t| 0,922     \t| 0,994 \t| 1,000          \t| 1,000     \t| 0,992       \t| 0,996    \t| 0,292        \t| 0,456     \t|\n| slur_homonym_nh   | 0,353    \t| 0,413\t| 0,393     \t| 0,513 \t| 0,813          \t| 0,787     \t| 0,680       \t| 0,827    \t| 0,320        \t| 0,473     \t|\n| slur_reclaimed_nh | 0,217    \t| 0,277\t| 0,472     \t| 0,711 \t| 0,891          \t| 0,879     \t| 0,741       \t| 0,802    \t| 0,272        \t| 0,346     \t|\n| target_group_nh   | 0,710    \t| 0,700\t| 0,623     \t| 0,810 \t| 0,968          \t| 0,971     \t| 0,448       \t| 0,561    \t| 0,000        \t| 0,006     \t|\n| target_indiv_nh   | 0,538    \t| 0,572\t| 0,655     \t| 0,951 \t| 1,000          \t| 1,000     \t| 0,782       \t| 0,809    \t| 0,003        \t| 0,003     \t|\n| target_obj_nh \t| 0,637    \t| 0,622\t| 0,923     \t| 0,966 \t| 0,969          \t| 0,985     \t| 0,735       \t| 0,757    \t| 0,006        \t| 0,034     \t|\n\nAdditionally, we evaluated the influence of both explicit and implicit forms of hate speech on the detection capabilities of our enhanced HateBERT model. The table below provides a snapshot of the model's average accuracy across three pivotal categories defined on the DynaHate test set: Explicit, Implicit, and Non-HS. These results are calculated as the mean across five model instances for each training dataset. Model instances are initiated with a unique random seed.\n\n| Train \t| Explicit | Implicit | Non-HS  |\n|-----------|----------|----------|---------|\n| IHC   \t| 0,6855   | 0,8184   | 0,3447  |\n| SBIC  \t| 0,6566   | 0,6592   | 0,6638  |\n| DynaHate  | 0,8187   | 0,7186   | 0,8117  |\n| ISHate\t| 0,7326   | 0,5870   | 0,5878  |\n| ToxiGen   | 0,8353   | 0,8061   | 0,2708  |\n\nThe data illustrates that Explicit and Implicit categories demonstrate comparably high levels of accuracy in hate speech detection. This underscores their near-equivalent influence on the model's overall performance. Equally important is the significance of the Non-HS category, which varies across different training data sets but remains an essential factor for comprehensive hate speech detection.\n\nWe will include these new tables as well as results on the cross-evaluation for BERT and HateBERT for Explicit, Implicit, and Non-HS categories in the camera-ready version if the paper gets accepted.\n\n**2)** In response to the comment regarding the use of contrastive learning to bring implicit and explicit hate embeddings closer, we've taken it a step further. We've innovated our training optimizer to compute cosine similarities between positive and negative pairs of embeddings, which are based on the CLS tokens in language models. Positive pairs are drawn closer in the representation space, while negative pairs are pushed apart. These refined embeddings are then channeled to a classification layer. Our approach highlights the unique combination of cross-entropy loss with contrastive loss. This dual optimization strategy allows for effective training of downstream classification tasks, integrating both contrastive learning and cross-entropy in a seamless manner. The effectiveness of this novel architecture is evidenced by the results highlighted in **1)**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The authors uncover an important insight that explicit and implicit hate speech embeddings are anchored around target groups.\n2) They also show that using a contrastive learning strategy the above mentioned embeddings can be brought even closer to each other.", "reasons_to_reject": "1) The RQ1 mentioned in the paper seems redundant. This adds no extra information for the audience. It is expected the performance will vary across multiple HS datasets  when evaluated in cross-data setting. Another interesting point to analyse would've been how % of explicit hate information in the dataset affects implicit hate speech detection performance and vice-versa and it's corresponding effect on RQ2 & RQ3 t-sne plots. (Reference - https://aclanthology.org/2023.findings-eacl.9/)\n\n2) Again it is only obvious / intuitive that employing a contrastive learning strategy would bring together the implicit and explicit hate embeddings. What would be interesting to understand is that how can these correlations be leveraged to improve the downstream classification performance ?\n\nIn it's current form the paper lacks enough significant learnings / contribution to be accepted. Incorporating the above mentioned suggestions should be sufficient ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper studies the relation between implicit and explicit hate-speech detection. They study the learned embedding distribution for both cases as well showing they are pivoted around target groups visualised using t-sne plots. They employ an contrastive learning strategy to bring together the the explicit and implicit embeddings based on target groups.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Reference - https://aclanthology.org/2023.findings-eacl.9/ (already mentioned in review above)"}, "extra_text": {}}, {"review_id": "gGxUhhMNj0", "reviewer": "Reviewer_QeTe", "summary": "", "questions": "1) Line 212: How is the loss function modified for contrastive learning? A formula may help.\n\n2) Table 1: What do the numbers in bold face indicate?\n\n3) While the visualisation after contrastive learning is interesting, it is not clear if it results in an improvement in hate speech detection. Do the authors have results around that?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments.\n\n**1)** RQ1 investigates how machine learning models capture HS classes and whether explicit and implicit hateful messages are encoded differently across datasets. To answer this, we combine cross-evaluation with t-SNE analysis, focusing on HS, Non-HS, Explicit HS, and Implicit HS categories.\nThe cross-evaluation, which involves training models on multiple HS datasets, provides us insights into the models' capabilities to recognize HS across various contexts and environments. This establishes a baseline understanding of the models' performance. The t-SNE plots visually demonstrate the impact of implicit and explicit factors on these embeddings. Together, these analyses offer a comprehensive view, satisfying the queries raised in RQ1.\n\n**2)** Regarding the paper's novelty, our work i) examines the variability of explicit and implicit hate speech (HS) across different target groups using five benchmark datasets, ii) we propose a method that bridges explicit and implicit HS in the representation space through contrastive learning techniques. We analyze those representations both before and after applying our technique. To strengthen the conclusions of our work, we conducted a supplemental experiment fine-tuning both BERT and HateBERT using our enhanced embeddings, adhering to the settings of our initial RQs by using multiple seeds and averaging accuracy on HateCheck (see table below). We show that the enhanced models **(CONT columns)** are better at correctly classifying challenging Non-HS messages across the five datasets. They also notably reduce false positives in HateCheck categories like quoted announcements (`counter_quote_nh`), direct references (`counter_ref_nh`), positive identifiers (`ident_pos_nh`), negated hateful remarks (`negate_neg_nh`), non-hateful profanity (`profanity_nh`), reclaimed slurs (`slur_reclaimed_nh`), homonym slurs (`slur_homonym_nh`) and targeted abuse at individuals (`target_indiv_nh`), objects (`target_obj_nh`), and other non-protected groups (`target_group_nh `). As a result, we reduce biases in non-hateful implicit cases often misclassified as Explicit HS due to trigger words or nuanced content. Our enhanced method maintains high performance on HS labels while improving classification in borderline cases, affirming the robustness and precision of our approach.\n\nWe will include this new table as well as results on the cross-evaluation for BERT and HateBERT for Explicit, Implicit, and Non-HS categories in the camera-ready version if the paper gets accepted.\n\n| Test Case   \t  | HateBERT IHC | CONT IHC | HateBERT SBIC | CONT SBIC | HateBERT DynaHate | CONT DynaHate | HateBERT ISHate | CONT ISHate | HateBERT ToxiGen | CONT ToxiGen |\n|-------------------|--------------|----------|---------------|-----------|--------------------|---------------|-----------------|-------------|------------------|--------------|\n| counter_quote_nh  | 0,486    \t| 0,499\t| 0,276     \t| 0,422 \t| 0,857          \t| 0,962     \t| 0,379       \t| 0,524    \t| 0,000        \t| 0,010     \t|\n| counter_ref_nh\t| 0,576    \t| 0,617\t| 0,240     \t| 0,393 \t| 0,882          \t| 0,902     \t| 0,387       \t| 0,521    \t| 0,177        \t| 0,295     \t|\n| ident_pos_nh      | 0,441    \t| 0,446\t| 0,301     \t| 0,361 \t| 0,849          \t| 0,767     \t| 0,272       \t| 0,350    \t| 0,470        \t| 0,575     \t|\n| negate_neg_nh \t| 0,502    \t| 0,517\t| 0,120     \t| 0,161 \t| 0,448          \t| 0,486     \t| 0,164       \t| 0,211    \t| 0,087        \t| 0,194     \t|\n| profanity_nh  \t| 0,796    \t| 0,774\t| 0,922     \t| 0,994 \t| 1,000          \t| 1,000     \t| 0,992       \t| 0,996    \t| 0,292        \t| 0,456     \t|\n| slur_homonym_nh   | 0,353    \t| 0,413\t| 0,393     \t| 0,513 \t| 0,813          \t| 0,787     \t| 0,680       \t| 0,827    \t| 0,320        \t| 0,473     \t|\n| slur_reclaimed_nh | 0,217    \t| 0,277\t| 0,472     \t| 0,711 \t| 0,891          \t| 0,879     \t| 0,741       \t| 0,802    \t| 0,272        \t| 0,346     \t|\n| target_group_nh   | 0,710    \t| 0,700\t| 0,623     \t| 0,810 \t| 0,968          \t| 0,971     \t| 0,448       \t| 0,561    \t| 0,000        \t| 0,006     \t|\n| target_indiv_nh   | 0,538    \t| 0,572\t| 0,655     \t| 0,951 \t| 1,000          \t| 1,000     \t| 0,782       \t| 0,809    \t| 0,003        \t| 0,003     \t|\n| target_obj_nh \t| 0,637    \t| 0,622\t| 0,923     \t| 0,966 \t| 0,969          \t| 0,985     \t| 0,735       \t| 0,757    \t| 0,006        \t| 0,034     \t|\n\n**Q1)** The contrastive loss operates on the embeddings corresponding to the CLS tokens in language models. It is defined as follows:\n\n```python\nloss_cont = mean((1 - label_pair) * (similarity ** 2) + label_pair * (max(0, margin - similarity) ** 2))\n```\n\nHere, `label_pair` is set to 1 for positive pairs and 0 for negative pairs. `similarity` refers to the cosine similarity calculated between a pair of messages, while `margin` serves as a hyperparameter that can be adjusted for optimization.\nFor classification, we use the cross-entropy loss:\n\n```python\nloss_clf = -sum(gold_label[i] * math.log(pred[i]) + (1 - gold_label[i]) * math.log(1 - pred[i]) for i in range(len(gold_label)))\n```\n\nIn this case, `gold_label` corresponds to the labels of the dataset on which the model is fine-tuned.\nThe final loss function used for optimization during training is the sum of these two losses:\n\n```python\nTotal_loss = loss_cont + loss_clf\n```\n\nBy combining these losses, we optimize both the model's understanding of the embedding space and its classification performance.\n\n**Q2)** The table displays for each column, i.e. for each dataset  the top two scores. As could be expected, the highest score is obtained when training and testing on the same dataset, serving as a best-case benchmark. The second-highest score reveals the next most effective model.\n\n**Q3)** It is answered in points 1) and 2).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The paper is well-written, and lays out the research questions systematically (however, there is a question about RQ1). \n\n2) The problem of implicit hate messages is handled using contrastive learning. This seems to be an interesting use of contrastive learning.\n", "reasons_to_reject": "1) It is not clear how the cross-evaluation results address RQ1.\n\n2) It is not clear if contrastive learning improves hate speech classification - or generalisation across the datasets. The paper does not present results to suggest so.", "questions_to_authors": "1) Line 212: How is the loss function modified for contrastive learning? A formula may help.\n\n2) Table 1: What do the numbers in bold face indicate?\n\n3) While the visualisation after contrastive learning is interesting, it is not clear if it results in an improvement in hate speech detection. Do the authors have results around that?", "paper_topic_and_main_contributions": "The paper describes how implicit and explicit hate expressions affect hate speech detection. A cross-validation evaluation on different hate speech datasets shows that the models do not necessarily generalise across the datasets. Contrastive learning is employed to link implicit and explicit hate instances. Visualisations showing embedding representations show that contrastive learning may work well.\n\nContribution: The idea of using contrastive learning to link explicit and implicit hate is an interesting idea for hate speech detection.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1) Line 133: by target results -> result*\n2) Line 151: It is not clear why the definition for SBIC dataset is mentioned but not for others (where the paper only says the definitions are consistent)."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes the use of a contrastive learning based strategy to boost performance on hate speech detection with a focus on both implicit and explicit instances. Experiments show the overall potential of the proposed solution to improve performance over other baselines.

The experiments are overall sound and the problem tackled is an important and challenging one, with one of the weaker points being that the proposed solution is somewhat incremental. Another aspect that would have been good to see is some more depth in the experimentation, possibly with ablation experiments to demonstrate where the improvement is coming from, e.g. whether it is from the use of contrastive learning.

While this is somewhat understandable from a short paper, I appreciate the extensive rebuttal with new results addressing the above limitations and I would appreciate the authors to incorporate these in a further revision.
Decision: Accept-Findings