SHAREDCON: Implicit Hate Speech Detection using Shared Semantics (2024)

Abstract: The ever-growing presence of hate speech on social network services and other online platforms not only fuels online harassment but also presents a growing challenge for hate speech detection. As this task is akin to binary classification, one of the promising approaches for hate speech detection is the utilization of contrastive learning. Recent studies suggest that classifying hateful posts in just a binary manner may not adequately address the nuanced task of detecting implicit hate speech. This challenge is largely due to the subtle nature and context dependency of such pejorative remarks. Previous studies proposed a modified contrastive learning approach equipped with additional aids such as human-written implications or machine-generated augmented data for better implicit hate speech detection. While this approach can potentially enhance the overall performance by its additional data in general, it runs the risk of overfitting as well as heightened cost and time to obtain. These drawbacks serve as motivation for us to design a methodology that is not dependent on humanwritten or machine-generated augmented data for training. We propose a straightforward, yet effective, clustering-based contrastive learning approach that leverages the shared semantics among the data.1

heading: 1 Introduction
text: Warning: this paper contains content that may be offensive or upsetting.
Detecting hate speech is more challenging compared to other tasks due to its varying perception and definition among individuals, which is reflected in the inconsistencies between datasets (Luo et al., 2023). Implicit hate speech, in particular, often lacks explicit cues such as swear words or insults,
1Our code is available at https://github.com/hsannn/ sharedcon.
making its detection even more challenging. Previous studies suggested models for identifying implicit hate speech (Sridhar and Yang, 2022; Lin, 2022). While these approaches exhibit promising performance for the in-dataset scenario, the performance is significantly degraded for the crossdataset scenario (Wiegand et al., 2019) because of their inconsistent nature. Recent studies have explored methods such as data augmentation (DA) to generate positive pairs (Kim et al., 2022), or creating implicit meanings (implication) of hateful posts manually by human (ElSherief et al., 2021). Especially, Kim et al. (2022) showed the importance of the model’s generalization at implicit hate speech detection and proposed a contrastive learning approach that utilizes common implications in the datasets. Although their approach shows good performance for in-dataset, using implications of in-dataset makes it difficult for them to perform well in cross-dataset scenarios. In other words, they
10444
are hard to generalize the model in cross-dataset. In the realm of implicit hate speech detection, the SOTA performance has been achieved by contrastive learning (CL) with humman-written implications or machine-generated augmented texts. However, there are some limitations to these approaches: (1) It requires experts to manually generate implications, which is expensive. For example, for an implicit hate speech dataset like IHC (ElSherief et al., 2021), the total annotation cost was $15k and it took a considerable amount of time to complete the dataset construction due to the complexity and scale of the work involved in developing a resource for understanding implicit hate speech. This limitation significantly restricts the scalability of this approach when applied to a wide range of datasets or different domains (Sheth et al., 2023). Furthermore, (2) it is hard to say that DA approach uses the correct positive pairs. The DA strategy merely employs post-augmentation as a positive pair, which lacks semantic consistency across posts. Figure 2 shows the comparison between the existing CL approaches and our methodology.
In order to address implication costs and enhance the existing approaches, we propose SharedCon, a simple yet effective technique—leveraging shared semantics from the training data itself without additional data construction steps or extra modules. We extract shared semantics of training datasets by grouping posts and consider the centroid post of each group as the representative semantics shared
by the posts. SharedCon is evaluated on three hate speech benchmark datasets in both in-dataset and cross-dataset settings. Experimental results demonstrate that SharedCon outperforms our baselines with a 0.43%p average improvement in the indataset setting and 1.43%p average improvement in the cross-dataset setting. Furthermore, we compare the performance of SharedCon with current SOTA approach (ImpCon). The results show that SharedCon can deliver comparative (even outperform for some cases) performance, against the current SOTA approach that requires a human-laborintensive step. Our empirical analysis displays that utilizing shared semantics in the quality-assured sentence embedding space can work as effective as manually produced implications. While the representation of shared semantics works similarly to the implication in Kim et al. (2022), our process provides the flexibility to train the model on various datasets, even those without human-annotated implications. The approach is simple yet effective without extra modules; we cluster the training data and select one post as a core sample of shared semantics, then the selected post plays the role of the anchor in the CL step.
Our main contributions are
• We propose a straightforward and effective technique—leveraging shared semantics from the training data without the need for humanannotated implications.
• We verify the efficacy of shared semantics by enhancing performance across both in-dataset and cross-dataset scenarios.
• We achieve performance that is comparable to and in some cases, even outperforms the current SOTA model, all without the need for human labor-intensive steps.

heading: 2 Contrastive Learning with Shared Semantics
text: While the idea of utilizing shared semantics in the fine-tuning process is promising for implicit hate speech detection, one major drawback is that it requires a dataset with implications, which are often written by humans manually. Therefore, we study an effective way of finding shared semantics without human involvement and improving the cross-dataset performance.

heading: 2.1 Shared Semantics
text: We use the term shared semantics to describe the approach for finding shared semantics without human involvement, which we assume that instances of real-world data exhibit inherent shared characteristics. Figure 1 illustrates an example of shared semantics extracted from the IHC (ElSherief et al., 2021) dataset; we observe that shared semantics effectively captures the latent meaning of hate posts, which corresponds to implication, a human-generated label representing the implicit meaning of a post.
As depicted in Figure 3, we first extract sentence embeddings of the training data using the existing sentence embedding models. Let us denote a post in the train set and its corresponding label pairs as {xt, yt}t∈T , where T is a total size of a train set. Then, a sentence embedding model constructs high-quality embeddings of xt. We use the K-means algorithm to cluster the embeddings with the same label. Let Ck be clusters created using Kmeans and µk be a centroid of a cluster Ck, where k ∈ [1, 2K].2 K is a manually adjustable parameter that represents the number of clusters. Each cluster formed in this way is expected to share a common semantic meaning. From this perspective, we consider the centroid posts as the closest sentence from the corresponding cluster centers and define our shared semantics Sk to be
2We use K clusters for each of two labels (hate or not).
Sk = argmin xc∈Ck
||xc − µk||, (1)
where xc is a post in a cluster Ck. We then train the model to bring posts within the same cluster closer together and to push posts from different clusters apart.

heading: 2.2 SHAREDCON
text: We propose SharedCon that adjusts supervised contrastive learning loss in Eq. (2) by Khosla et al. (2020) for implicit hate speech detection. We denote a pair of input post and its corresponding label as {xi, yi}i∈I , where I = {1, . . . , B} is a set of indices in a mini-batch and B is the batch size. As a positive sample xp for an input post xi, we train the model to bring xi and xp closer together as a positive pair and to push xi and xa further apart as a negative pair:
LSCL = ∑
i=1
−1 |Pi|
∑ p∈Pi log ehi·hp/τ∑ a∈I\{i} e hi·ha/τ ,
(2) where hi, hp and ha are the representations from an encoder for the inputs xi, xp and xa, respectively. Pi is the set of indices of positive samples and τ is a temperature parameter which is specified in Section 3.3.
A mini-batch is augmented with shared semantics in Eq. (1) per input. Having a mini-batch of B original samples, we set the location of the corresponding shared semantics of the ith (i ≤ B) original sample xi as i+B. This ensures that each input post in a mini-batch is paired with its positive counterpart, resulting in a total of 2B samples for each mini-batch.
We modifiy Eq. (2) and propose our SharedCon loss as follows:
LSharedCon =
− 2B∑
i=1
∑ p∈SHA(xi) log e hi·hp/τ∑ a∈I\{i} e hi·ha/τ
|SHA(xi)| .
(3)
This supervised contrastive learning method aims to shorten the distance between training samples and their corresponding shared semantics. Step 3 and step 4 in Figure 3 illustrate this process. Here, we consider the scenario where posts from the same cluster appear in a mini-batch. Every post from the same cluster as an input post is
also considered positive. In other words, we allow multiple positives within a mini-batch when members of the same cluster are included.
Suppose that a module SHA(·) constructs our positive set by assigning shared semantics. Then, we arrange the positive sample of xi as xp, where p ∈ SHA(xi). For i ≤ B, SHA(xi) represents the set of shared semantics of the input xi, but for i > B, SHA(xi) becomes the set of original input posts that come from the same cluster as xi. Note that we set the location of the original input posts to be in [1, B] and the location of each input’s corresponding positive pair—shared semantics—to be in [B + 1, 2B]. We construct our positive set SHA(xi) as follows:
SHA(xi) =   {p | C(xi) = C(xp), p ∈ [B + 1, 2B]}, for i ≤ B {p | C(xi) = C(xp), p ∈ [1, B]}, for i > B,
where C(xi) represents the cluster number of xi. SHA(xi) serves the role of allowing members within the same cluster to function as positive samples. During training, the model learns the implication of hate speech posts through the SharedCon loss in Eq. (3).
The cross-entropy loss is a typical loss function
used in hate speech detection and is defined as follows:
LCE = − 1
B
B∑
i=1
[yi log ŷi +(1− yi) log (1− ŷi)],
(4) where ŷi is the model prediction for the xi and yi is the ground truth label of xi.
Since the two losses optimize different aspects, it is necessary to combine them. The contrastive learning loss in Eq. (3) facilitates learning representations that move closer to the centroid as training progresses for better generalization, while the cross-entropy loss in Eq. (4) is used for binary classification itself.
The overall objective is
LOverall = λLCE + (1− λ)LSharedCon,
where λ is a scaling parameter that controls the balance between the cross-entropy loss and the contrastive learning loss.

heading: 3 Experimental Results

heading: 3.1 Datasets
text: We conduct multiple experiments utilizing three hate speech datasets reported in Table 1. Note that IHC and SBIC have implications but DynaHate does not.

heading: 3.2 Baselines
text: We consider the following three baselines for comparision.
CE We use the cross-entropy loss (CE) to finetune the model, which is a fundamental method for classifying hate speech.
AugCon utilizes augmented inputs as positive samples for inputs in the process of contrastive learning. While AugCon can be done without any additional need for human annotations (except for inputs and their labels), AugCon showed improved performance on some limited settings.
ImpCon utilizes shared implications as positive samples for hate speeches in the process of contrastive learning. While ImpCon offered consistently superior performance compared to other methods, it necessitates additional human annotation for the implication.
We design a contrastive learning loss that has strengths of both approaches; given that utilizing shared semantics among inputs was effective, we aim to utilize shared semantics for achieving generalization ability (= ImpCon strength) without additional human involvements (= AugCon strength).

heading: 3.3 Implementation Details
text: For the hyperparameter optimization, we select the learning rate from {5e-6, 1e-5, 2e-5, 3e-5, 5e-5} and the temperature parameter τ from {0.1, 0.3, 0.5}. We also specify λ as a scaling factor from {0.25, 0.5, 0.75} and choose the number K of clusters from {10, 25, 50, 75, 100, 125, 150, 200}. The model archiving the highest validation F1 score is selected as the best-performing one. All experiments are executed using 3 different random seeds and we report the average score of macro-F1 for the results. Unless otherwise stated, we use the SimCSE’s embeddings and the model is trained with the IHC dataset. We perform training on models for a span of 6 epochs employing an NVIDIA RTX 4090 GPU.

heading: 3.4 Detection Performance
text: We verify the effectiveness of our method by evaluating the performance on both in-dataset and cross-dataset scenarios. We train BERT (Devlin et al., 2019) on each of the three datasets and evaluate them on those three respectively. Table 2 demonstrates the outcomes of in-dataset evaluations, with respect to sentence embedding models: SimCSE (Gao et al., 2021). Table 3 presents the evaluation results for cross-dataset scenarios, utilizing SimCSE embeddings as well.
In the experiments of the in-dataset scenario, we observe a slight improvement of approximately 0.43%p on average over the baselines in Section 3.2. Furthermore, in the cross-dataset setting, particularly for models trained on IHC, there was a significant increase of up to 4.5%p in SBIC evaluation scenario. The noteworthy thing is that the DynaHate dataset does not have implications; our approach selects appropriate representatives of shared semantics from the training dataset and improves the performance.
These results demonsrate the effectiveness of our
approach even when compared to the utilization of human-annotated implications. Overall, the experimental results show that SharedCon shows competitive or superior performance compared to the existing SOTA without using any human-written implications.

heading: 4 Analysis
text: Our discussion focuses on validating the hypothesis that shared semantics should be identified from the centroid of each cluster, based on a twofold additional scenarios. 1) In Section 4.1, we randomly select a post and allocate it as an anchor; and 2) in Section 4.2, we choose one sentence among the top three or five sentences that are most proximal (in terms of the Euclidean distance) to the centroid of the cluster as an anchor.

heading: 4.1 Random Anchors
text: We examine where shared semantics are located in the embedding space by altering the method of selecting anchor posts. We replace the closest post from the centroid of a cluster in Eq. (1) with S∗ = RANDOM(xt) such that RANDOM(·) selects a random post, where t ∈ T and T is the count of posts within a train set. When a post S∗ from a random position is chosen as an anchor for contrastive learning framework, the performance decreases compared to selecting a centroid sentence as an anchor. Figure 4 (Left) demonstrates the superior performance of SharedCon compared to the random-anchor-selection approach. The gray graph represents macro-F1 score of SharedCon and the violet graph represents the score for random selection. K represents the number of clusters formed in the train set for utilizing the sentence embeddings generated via SimCSE (Gao et al., 2021). As K changes, both selection strategies show minor fluctuations in macro-F1 score, but SharedCon consistently achieves good performance. We observed
that the in-dataset score declined by 1.2%p and the cross-dataset scores decreased by 4.5%p on average in random-anchor experiment setting. These experimental results confirm that shared semantics are located at the center of clusters in a dataset.

heading: 4.2 Centroid-proximate Anchors
text: We conducted additional experiments, applying variations to the previously established positive pairs, to further investigate the methodology for extracting shared semantics. Here, we select the shared semantics in two ways: randomly selecting the shared semantics 1) from the three posts nearest to the centroid of the cluster and 2) from the five posts nearest to the cluster’s centroid. Therefore, we modify Eq. (1) and have
Sk = argminm xc∈Ck
||xc − µk||
S∗ = RANDOM(Sk),
where argminm ||xc − µk|| denotes the set of topm posts that minimize the distance of the post xc
from the cluster centroid µk. We set m ∈ {3, 5}. Figure 4 (Right) depicts two results of changing the anchor for contrastive learning from the centroid post among the top-3 and top-5 closest posts to the centroid. In the embedding space of SimCSE, we trained the model with the IHC dataset. Both performances of 3-anchors’ (cp-3) and 5-anchors’ (cp-5) are comparable to that of SharedCon (cp1), or occasionally surpass it. This suggests that choosing centroid-around posts might play a role as noise and contribute to the effectiveness of adversarial training. We hypothesize the presence of shared semantics in the center of clusters, but consider the exact location to be an open question for precise positioning. We expect the exploration of alternative methods for selecting representative points beyond the cluster centroids will be reserved for future work. This aspect remains an avenue for further investigation and enhancement in our research. Hence, it can be observed that the centroid sentence performs most effectively in embodying the role of shared semantics.

heading: 4.3 Generalizability
text: While our approach marginally outperforms indataset results compared to our baselines, as Kim et al. (2022) tackled, the evaluation of cross-dataset is important to show the generalization ability of a model. Therefore, we conduct a cross-dataset evaluation for the models trained on the three hate speech benchmark datasets: IHC, SBIC, DynaHate (ElSherief et al., 2021; Kim et al., 2023; Vidgen et al., 2021) to check generalizability of our proposed model. Table 3 compares crossdataset evaluation results of AugCon, ImpCon and SharedCon in SimCSE (Gao et al., 2021) embedding space. We observe that IHC-trained model achieved the highest performance while evaluated in SBIC dataset. We speculate that the reason for the high performance could be attributed to the potential presence of label errors within the dataset. After suspecting potential errors within the IHC dataset, we identified certain noisy implication annotations as shown in Table 4. These labels contribute to confusion in the model, resulting in a decrease in its performance. In other words, it is better to find shared semantics among the given training dataset directly instead of using human-written implications or machine-generated augmented texts.

heading: 4.4 Representation by Sentence Embedding Models
text: We assume that the space of the sentence embedding model already effectively reflects semantic similarity. In order to validate our assumption, we also implemented an extra experiment by changing the sentence embedding model. Figure 5 shows that SharedCon consistently demonstrates strong performance regardless of variations in the sentence embedding model.
Sentence embedding model We leverage three different sentence embedding models: SBERT (Reimers and Gurevych, 2019), SimCSE (Gao et al., 2021) and AnglE (Li and Li, 2023)
• SBERT is a modified BERT using siamese / triplet structures, excels in semantic similarity and transfer learning.
• SimCSE enhanced sentence embeddings through contrastive learning, improving semantic understanding in textual tasks via unsupervised and supervised methods.
• AnglE addressed gradient issues in cosinebased methods across diverse text types and outperformed prior models in semantic similarity tasks.
By assuming that the space represented by a sentence embedding model embodies a comprehensive semantic structure, we discerned shared semantics within this framework. We aim to empirically substantiate the validity of high-quality representation space. Figure 5 shows the results for SBERT, SimCSE and AnglE consistently exhibited favorable performance without significant deviations. In this section, we train the model with the IHC dataset and present the average results from five random
seeds. The consistency in the performance of these three models demonstrates the embedding invariance of our approach. This indicates that SharedCon can achieve fair performance regardless of which model is used.
The effectiveness of our methodology lies in extracting implications within a high-quality representation (embedding) space. In other words, the sentence embedding model provides a reasonably good representation space for finding shared semantics and our approach of identifying positive pairs on this basis makes sense.

heading: 4.5 Error Analysis
text: We analyze the incorrect predictions of our approach and ImpCon for the IHC dataset. SharedCon had 355 false positives and 311 false negatives while ImpCon had 365 and 335 respectively. As depicted in the confusion matrices in Figure 6, SharedCon has 0.64%p less false negatives and 0.27%p less false positives compared to ImpCon. Table 5 shows some examples of errors both ImpCon and SharedCon made. For SharedCon, out
of a total of 666 incorrections in both false positive and false negative, we observed that there are 22 broken sentence posts for both in false positive and false negative. That is, both methodologies encounter difficulties with the broken sentences, while ShareCon is comparatively robust than ImpCon with grammatically correct sentences. ImpCon is more vulnerable to broken sentences than SharedCon; clearly, broken sentences impact the model’s prediction errors. Since broken sentences and obfusticated posts are commonly observed in recent social media platforms, effectively detecting these aspects is an important avenue for future research. We provide samples of broken sentences that both methodologies struggle with at Table 5.
Regarding the misclassification by SharedCon, we point out errors within the dataset itself. We often observe errors in human-written implications in the dataset; Table 4 shows some examples that capture subtly incorrect implications. These examples illustrate instances where responses are skewed towards negative connotations or mislabeled altogether, irrespective of context. For a very basic analysis to identify the simple label error ratio, we randomly sampled 100 posts from the train set of IHC and found that approximately 27% of them were misclassified. Out of these misclassifications, about 7 posts were ambiguous and the remaining 20 posts were incorrectly classified. We assume that these incorrect implications can cause further performance degradation.

heading: 5 Related Work
text: Hate Speech Detection There are several datasets on explicit hate speech with different purposes (Warner and Hirschberg, 2012; Davidson et al., 2017; Founta et al., 2018; Basile et al., 2019). Davidson et al. (2017) addressed distinguishing hate speech from offensive language by using a crowd-sourced hate speech lexicon. Implicit hate speech, on the other hand, characterized by its subtle linguistic expressions, presents a more complex obstacle for detection compared to its explicit hate speech (Ocampo et al., 2023). Recently, researchers have started considering implicit hatred or abusiveness. Han and Tsvetkov (2020) introduced a novel generic framework, MACAS, for online abusive language detection across different domains. Wiegand et al. (2021) addressed the challenges in detecting implicitly abusive language, highlighting the limitations of existing datasets and
proposed a divide-and-conquer strategy to categorize subtypes of implicit abuse. In order to identify implicit hate speech ElSherief et al. (2021) provided a benchmark dataset to understand implicit hate speech beyond explicit forms. Leveraging the implicit hate speech dataset, Kim et al. (2022) introduced ImpCon, a contrastive learning method that addressed the challenge of model generalization in implicit hate speech detection. In recent years, there have been some research proposing model training strategy (Jafari et al., 2023) for implicit hate speech detection, as well as suggesting new architectures for models (Ghosh et al., 2023).
Contrastive learning Recently, contrastive learning has become a prominent tool in unsupervised representation learning in various domains. SimCLR (Chen et al., 2020) randomly selects two augmented examples from a mini-batch as a positive pair, while the other augmented examples within the mini-batch are treated as negative pairs. Khosla et al. (2020) proposed to select positive pairs and negative pairs by considering anchor points. In the natural language processing field, SimCSE (Gao et al., 2021) selected positive pairs by using annotated pairs from NLI datasets, where entailment pairs are considered as positives and contradiction pairs used as hard negatives. Gunel et al. (2021) suggested using contrastive learning as a fine-tuning tool to capture similarities between examples from the same classes and contrast them with examples from other classes to improve generalization. Suresh and Ong (2021) performed well on fine-grained text classification tasks by adaptively weighting the relationships between classes to distinguish between more difficult samples. Recently, Lu et al. (2023) proposed dual contrastive learning approach for detecting hate speech. We also used contrastive learning to fine-tune model performance.
Clustering for contrastive learning Yang et al. (2023) performs cluster-level contrastive learning to incorporate measurable emotion prototypes. Gao et al. (2022) proposed a weakly supervised contrastive learning method that allows us to consider multiple positives and multiple negatives. These are prototype-based clustering methods that avoid semantically related events being pulled apart. We suggest selecting multiple positives by clustering to use contrastive learning in the fine-tuning process for shared semantics without human-written implications.

heading: 6 Conclusions
text: We have enhanced the existing implicit hate speech detection approach, which previously relied on human-generated implications. This improvement involves clustering sentence embeddings and utilizing the centroid sentence of each cluster as positive examples during the contrastive learning process. The experimental results have demonstrated that posts in the dataset have shared semantics and our approach of using them as positive samples for contrastive learning works well. Particularly, our methodology has shown competitive or superior performance improvement of fine-tuned models in cross-dataset experiments. To sum up, our approach has enhanced model generalization without human-written implications, providing a better solution for implicit hate speech detection in terms of cost.
Limitations
While our approach reduces possible errors of wrong implications written by humans, it might be possible that a centroid sentence itself might be labeled wrong in the first place. However, it is less likely for a centroid sentence to have a wrong label since it has similar semantics compared with the
other sentences in the same cluster. Another issue would be a selection process of shared-semantic sentences. Certainly, there could be a better way, but our approach—selecting a centroid sentence in a cluster—constantly shows good and stable performance compared with several other heuristics.
Ethical Consideration
Generalization Ability Enhancement Our work focuses on enhancing the generalization ability to train a hate speech detector that can consistently adapt to new forms of hate speech. Specifically, by training a model with improved performance, we aim to equip it with the capability to respond to emerging manifestations of hate speech even on the same dataset. This emphasis on enhanced generalization entails training a model that demonstrates superior performance when presented with the same dataset, enabling it to effectively address novel expressions of hate speech.
Annotator Exposure Mitigation In the past, state-of-the-art (SOTA) models exposed annotators to the risk of hate speech exposure during the process of adding implications as comments to secure such capabilities. However, our methodology has achieved comparable performance without such risks.
Risks and Potential Misuse The enhancement of these capabilities represents a positive development in the field of hate speech detection; however, it also inherently carries risks. If the improved generalization ability is exploited, it could be utilized to generate new forms of hate speech or conceal hateful behaviors. It is crucial to be vigilant about these potential risks and to carefully evaluate the use of the model and its outcomes.

heading: Acknowledgements
text: This research was supported by the NRF grant (RS2023-00208094) and the AI Graduate School Program at Yonsei University (No. RS-2020-II201361) funded by the Korean government (MSIT). Han is a corresponding author.

heading: Future Work
text: Since broken sentences and obfusticated posts are commonly observed in recent social media platforms, effectively detecting these aspects is an important avenue for future research. We provide samples of broken sentences that both methodologies struggle with at Table 5. Regarding the misclassification by SharedCon, we point out errors within the dataset itself. We often observe errors in human-written implications in the dataset; Table 4 shows some examples that capture subtly incorrect implications. These examples illustrate instances where responses are skewed towards negative connotations or mislabeled altogether, irrespective of context. For a very basic analysis to identify the simple label error ratio, we randomly sampled 100 posts from the train set of IHC and found that approximately 27% of them were misclassified. Out of these misclassifications, about 7 posts were ambiguous and the remaining 20 posts were incorrectly classified. We assume that these incorrect implications can cause further performance degradation