Improving Hateful Meme Detection through Retrieval-Guided Contrastive Learning (2024)

Abstract: Hateful memes have emerged as a significant concern on the Internet. Detecting hateful memes requires the system to jointly understand the visual and textual modalities. Our investigation reveals that the embedding space of existing CLIP-based systems lacks sensitivity to subtle differences in memes that are vital for correct hatefulness classification. We propose constructing a hatefulness-aware embedding space through retrieval-guided contrastive training. Our approach achieves state-of-theart performance on the HatefulMemes dataset with an AUROC of 87.0, outperforming much larger fine-tuned large multimodal models. We demonstrate a retrieval-based hateful memes detection system, which is capable of identifying hatefulness based on data unseen in training. This allows developers to update the hateful memes detection system by simply adding new examples without retraining — a desirable feature for real services in the constantly evolving landscape of hateful memes on the Internet. This paper contains content for demonstration purposes that may be disturbing for some readers.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5333–5347 August 11-16, 2024 ©2024 Association for Computational Linguistics
This paper contains content for demonstration purposes that may be disturbing for some readers.

heading: 1 Introduction
text: The growth of social media has been accompanied by a surge in hateful content. Hateful memes,
which consist of images accompanied by texts, are becoming a prominent form of online hate speech. This material can perpetuate stereotypes, incite discrimination, and even catalyse real-world violence. To provide users the option of not seeing it, hateful memes detection systems have garnered significant interest in the research community (Kiela et al., 2021; Suryawanshi et al., 2020b,a; Pramanick et al., 2021a; Liu et al., 2022; Hossain et al., 2022; Prakash et al., 2023; Sahin et al., 2023).
Correctly detecting hateful memes remains difficult. Previous literature has identified a prominent challenge in classifying "confounder memes", in which subtle differences in either image or text may lead to a completely different meaning (Kiela et al., 2021). As shown in Figure 1, the top left and top middle memes share the same caption. However, one of them is hateful and the other benign depending on the accompanying images. Confounder memes resemble real memes on the Internet, where the combined message of images and texts contribute to their hateful nature. Even state-of-the-art models, such as HateCLIPper (Kumar and Nandakumar, 2022), exhibit limited sensitivity to nuanced hateful memes.
We find that a key factor contributing to misclassification is that confounder memes are located in close proximity in the embedding space due to the similarity of text or image content. For instance, HateCLIPper’s embedding of the confounder meme in Figure 1 has a high cosine similarity score with the left anchor meme even though they have opposite meanings. This poses challenges for the classifier to distinguish harmful and benign memes.
We propose “Retrieval-Guided Contrastive Learning” (RGCL) to learn hatefulness-aware vision and language joint representations. We align the embeddings of same-class examples that are semantically similar with pseudo-gold positive examples and separate the embeddings of opposite-class
5333
examples with hard negative examples. We dynamically retrieve these examples during training and train with a contrastive objective in addition to cross-entropy loss. RGCL achieves higher performance than state-of-the-art large multimodal systems on the HatefulMemes dataset with far fewer model parameters. We demonstrate that the RGCL embedding space enables the use of K-nearestneighbor majority voting classifier. The encoder trained on HarMeme (Pramanick et al., 2021a) can be applied to HatefulMemes (Kiela et al., 2021) without additional training while maintaining high AUC and accuracy using the KNN majority voting classifier, even outperforming large multi-modal models under similar settings. This allows efficient transfer and update of hateful memes detection systems to handle the fast-evolving landscape of hateful memes in real-life applications. Our contributions are:
1. We propose RGCL for hateful memes detection which learns a hatefulness-aware embedding space via an auxiliary contrastive objective with dynamically retrieved examples. We propose to leverage novel pseudo-gold positive examples to improve the quality of positive examples.
2. Our proposed approach achieves state-of-theart performance on HatefulMemes and the HarMeme. We show RGCL’s capability across various domains of meme classification tasks on MultiOFF, Harm-P and Memotion7K.
3. Our retrieval-based KNN majority voting classifier facilitates straightforward updates and extensions of hateful meme detection systems across various domains without retraining. With RGCL training, the retrieval-based classifier demonstrates strong cross-dataset generalizability, making it suitable for real services in the dynamic environment of online hateful memes.

heading: 2 Related Work
text: Hateful Meme Detection Systems in previous work can be categorized into three types: Object Detector (OD)-based vision and language models, CLIP (Radford et al., 2021) encoder-based systems, and Large Multimodal Models (LMM).
OD-based models such as VisualBERT (Li et al., 2019), OSCAR (Li et al., 2020), and UNITER
(Chen et al., 2020) use Faster R-CNN (Ren et al., 2015) based object detectors (Anderson et al., 2018; Zhang et al., 2021) as the vision model. The use of such object detectors results in high inference latency (Kim et al., 2021).
CLIP-based systems have gained popularity for detecting hateful memes due to their simpler endto-end architecture. HateCLIPper (Kumar and Nandakumar, 2022) explored different types of modality interaction for CLIP vision and language representations to address challenging hateful memes. In this paper, we show that such CLIP-based models can achieve better performance with our proposed retrieval-guided contrastive learning.
LMMs like Flamingo (Alayrac et al., 2022) and LENS (Berrios et al., 2023) have demonstrated their effectiveness in detecting hateful memes. Flamingo 80B achieves a state-of-the-art AUROC of 86.6, outperforming previous CLIP-based systems although requiring an expensive fine-tuning process.
Contrastive Learning is widely used in vision tasks (Schroff et al., 2015; Song et al., 2016; Harwood et al., 2017; Suh et al., 2019) and retrieval tasks , however, its application to multimodally pretrained encoders for hateful memes has not been well-explored. Lippe et al. (2020) incorporated negative examples in contrastive learning for detecting hateful memes. However, due to the low quality of randomly sampled negative examples, they observed a degradation in performance. In contrast, our paper shows that by incorporating dynamically sampled positive and negative examples, the system is capable of learning a hatefulnessaware vision and language joint representation.
Sparse retrieval methods, such as BM-25 (Robertson and Zaragoza, 2009) have been used in contrastive learning to obtain collections of hard triplets (Karpukhin et al., 2020; Schroff et al., 2015; Khattab and Zaharia, 2020; Nguyen et al., 2023). In contrast, dense retrieval, which is based on vector similarity scores, has been widely adopted for various passage retrieval tasks (Karpukhin et al., 2020; Santhanam et al., 2022; Diaz et al., 2021; Herzig et al., 2021; Lin et al., 2023, 2024). Our method leverages dense retrieval to dynamically select both hard negative and pseudo-gold positive examples.
example i. Additionally, the VL Encoder encodes the training memes into a retrieval database G. 2 During training, pseudo-gold and hard negative examples are obtained using the Faiss nearest neighbour search. During inference, K nearest neighbours are obtained using the same querying process to perform the KNN-based inference. 3 During training, we optimise the joint loss function L. 4 For inference, we use conventional logistic classifier and our proposed retrieval-based KNN majority voting. For a test meme j, we denote the prediction from logistic regression and KNN classifier as ŷj and ŷ′j , respectively.

heading: 3 RGCL Methodology
text: In each training example {(Ii, Ti, yi)}Ni=1, Ii ∈ RC×H×W is the image portion of the meme in pixels; Ti is the caption overlaid on the meme; yi ∈ {0, 1} is the meme label, where 0 stands for benign, 1 for hateful.
We leverage a Vision-Language (VL) encoder to extract image-text joint representations from the image and the overlaid caption:
gi = F(Ii, Ti) (1) We encode the training set with our VL encoder to obtain the encoded retrieval vector database G:
G = {(gi, yi)}Ni=1 (2) We index this retrieval database with Faiss (Johnson et al., 2019) to perform training and retrievalbased KNN classification.
As shown in Figure 2, the VL encoder comprises a frozen CLIP encoder followed by a trainable multilayer perceptron (MLP). The frozen CLIP encoder encodes the text and image into embeddings that are then fused into a joint VL embedding before feeding into the MLP.
We use HateCLIPper (Kumar and Nandakumar, 2022) as our frozen CLIP encoder. The model architecture is detailed in Appendix C. In Sec.4.4, we compare different choices of the frozen CLIP encoder to demonstrate that our approach does not depend on any particular base model.

heading: 3.1 Retrieval Guided Contrastive Learning
text: For each meme in the training set (the “anchor meme”), we dynamically obtain three types of con-
trastive learning examples: (1) pseudo-gold positive; (2) hard negative; (3) in-batch negative to train our proposed retrieval-guided contrastive loss.
(1) Pseudo-gold positive examples are samelabel samples in the training set that have high similarity scores under the embedding space. Incorporating these examples pulls same-label memes with similar semantic meanings closer in the embedding space.
(2) Hard negative examples (Schroff et al., 2015) are opposite-label samples in the training set that have high similarity scores under the embedding space. These examples are often confounders of the anchor memes. By incorporating hard negative examples, we enhance the embedding space’s ability to distinguish between confounder memes.
(3) For a training sample i, the set of in-batch negative examples (Yih et al., 2011; Henderson et al., 2017) are the examples in the same batch that have a different label as the sample i. In-batch negative examples introduce diverse gradient signals in the training and this causes the randomly selected in-batch negative memes to be pushed apart in the embedding space.
Next, we describe how we obtain these examples to train the system with Retrieval-Guided Contrastive Loss.

heading: 3.1.1 Finding pseudo-gold positive examples and hard negative examples
text: For a training sample i, we obtain the pseudo-gold positive example and hard negative example from the training set with Faiss nearest neighbour search (Johnson et al., 2019) which computes the similar-
ity scores between sample i’th embedding vector gi and any target embedding vector gj ∈ G. The encoded retrieval vector database G is updated after each epoch.
We denote the pseudo-gold positive example’s embedding vector:
g+i = argmax gj∈G/gi,yi=yj sim(gi,gj), (3)
similarly for the hard negative example’s embedding vector:
g−i = argmax gj∈G,yi ̸=yj sim(gi,gj). (4)
We use cosine similarity for similarity measures. We denote the embedding vectors for the inbatch negative examples as {g−i,1,g−i,2, ...,g−i,n−}. We concatenate the hard negative example with the in-batch negative examples to form the set of negative examples G−i = {g−i ,g−i,1,g−i,2, ...,g−i,n−}

heading: 3.1.2 RGCL training and inference
text: Following previous work (Kumar and Nandakumar, 2022; Kiela et al., 2021; Pramanick et al., 2021b), we use logistic regression to perform memes classification as shown in Figure 2. We denote the output from the logistic regression as ŷj for sample j.
To train the logistic classifier and the MLP within the VL Encoder, we optimize a joint loss function. The loss function consists of our proposed Retrieval-Guided Contrastive Learning Loss (RGCLL) and the conventional cross-entropy (CE) loss for logistic regression:
Li = LRGCLLi + LCEi = LRGCLLi + (yi log ŷi + (1− yi) log(1− ŷi)),
(5)
where the RGCLL is computed as:
LRGCLLi = L(gi,g+i ,G−i )
= − log e sim(gi,g+i )
esim(gi,g + i ) + ∑ g∈G−i esim(gi,g) .
(6)
In Appendix G, we compare different similarity metrics and loss functions.

heading: 3.2 Retrieval-based KNN classifier
text: In addition to logistic classifier, we introduce a retrieval-based KNN majority voting classifier
which relies on the inherent discrimination capability of the trained joint embedding space. Only when the trained embedding space successfully splits hateful and benign examples will majority voting achieve reasonable performance. The KNN classifier is suitable for real services in the constantly evolving landscape of online hateful memes as the the retrieval database can be extended without the need to retrain the system. In Section 4.2, we show that our proposed KNN classifier generalizes well to unseen data without additional training.
For a test meme t, we retrieve K memes located in close proximity within the embedding space from the retrieval vector database G (see Eq. 2). We keep a record of the retrieved memes’ labels yk and similarity scores sk = sim(gk, gt) with the test meme t, where gt is the embedding vector of the test meme t. We perform similarity-weighted majority voting to obtain the prediction:
ŷ′t = σ( K∑
k=1
ȳk · sk), (7)
where σ(·) is the sigmoid function and
ȳk := { 1 if yk = 1 −1 if yk = 0 . (8)
We conduct experiments in Sec. 4.2 to show that applying RGCL leads to much better performance with retrieval-based KNN inference than using only the cross-entropy loss.

heading: 4 RGCL experiments
text: We primarily evaluate the performance of RGCL on the HatefulMemes dataset (Kiela et al., 2021) and the HarMeme dataset (Pramanick et al., 2021a). The HarMeme dataset consists of COVID-19related harmful memes collected from Twitter. In Section 4.7, we evaluate three additional datasets to show the generalizability of RGCL beyond hateful meme classification. The dataset statistics are shown in Appendix D.
To make a fair comparison, we adopt the evaluation metrics used in previous literature (Kumar and Nandakumar, 2022; Cao et al., 2022; Kiela et al., 2021) for HatefulMemes and HarMeme: Area Under the Receiver Operating Characteristic Curve (AUC) and Accuracy (Acc).
The experiment setup, including the statistical significance tests, and hyperparameter settings are detailed in Appendices A and B.

heading: 4.1 Comparing RGCL with baseline systems
text: Table 1 presents the experimental results with logistic regression. RGCL is compared to a range of baseline models including OD-based models, LMMs, and CLIP-based systems. On the HatefulMemes dataset, RGCL obtains an AUC of 87.0% and an accuracy of 78.8%, outperforming all baseline systems, including the 200 times larger Flamingo-80B. OD-based models ERNIE-Vil (Yu et al., 2021), UNITER (Chen et al., 2020) and OSCAR (Li et al., 2020) performs similarly with AUC scores of around 79%. LMMs Flamingo-80B (Alayrac et al., 2022) is the previous state-of-the-art model for HatefulMemes, with an AUC of 86.6%. We also fine-tune LLaVA (Liu et al., 2023) with the procedure in Appendix E. LLaVA achieves 77.3% accuracy and 85.3% AUC, performing worse than the much larger Flamingo, but better than OD-based models. CLIP-based systems PromptHate (Cao et al., 2022) and HateCLIPper (Kumar and Nandakumar, 2022), built on top of CLIP (Radford et al., 2021), outperform both the original CLIP and OD-based models. HateCLIPper achieves an AUC of 85.5%, surpassing the original CLIP (79.8% AUC) but falling short of Flamingo80B (86.6% AUC). Our system, utilising HateCLIPper’s modelling, improves over HateCLIPper by nearly 3% in accuracy, reaching 78.8%. For the AUC score, our system achieves 87.0%, surpassing the previous state-of-the-art Flamingo-80B.
For HarMeme, RGCL obtained an accuracy of 87%, outperforming HateCLIPper with an accuracy of 84.8%, PromptHate with an accuracy of 84.5% and LLaVA with an accuracy of 83.3%. Our system’s state-of-the-art performance on the HarMeme dataset further emphasises RGCL’s robustness and generalisation capacity to different types of hateful memes.
4.2 Performance with retrieval-based KNN classifier
Online hate speech is constantly evolving, and it is not practical to keep retraining the detection system. We demonstrate that our system can effectively transfer to the unseen domain of hateful memes without retraining.
We train HateCLIPper with and without RGCL using the HarMeme dataset and evaluate on the
HatefulMemes HarMeme Model AUC Acc. AUC Acc.
HatefulMemes dataset. We report the performance of the KNN classifier when using the HarMeme and HatefulMemes dataset as the retrieval database in Table 2 (II) and (III) respectively. We only use the training set as the retrieval database to avoid label leaking.
We compare our method with state-of-the-art LMMs, including Flamingo (Alayrac et al., 2022), Lens (Berrios et al., 2023), Instruct-BLIP (Ouyang et al., 2022) and LLaVA (Liu et al., 2023) as shown in Table 2 (I). We report the zero-shot performance of these LMMs to replicate the scenario when the model predicts the unseen domain of hateful memes. To ensure a fair comparison, we report the performance of LLaVA fine-tuned on the HarMeme to align with RGCL’s setting in Table 2 (II) and (III).
Lastly, we also report the performance of our methods when trained and evaluated on HatefulMemes in Table 2 (IV).
(I) We report LMMs with diverse backbone language models, ranging from Flan-T5 (Chung et al., 2022) and the more recent Vicuna (Chiang et al., 2023). Among these models, Lens with Flan-T5XXL 11B performs the best, achieving an AUC of 59.4%. When LLaVA is fine-tuned on the HarMeme dataset and evaluated on the HatefulMemes dataset, its performance does not improve beyond its zero-shot performance. Its accuracy drops from 54.8% in zero-shot to 54.3% in fine-
1Since Flamingo is not open-sourced, we are unable to obtain accuracy.
2Reproduced with HateCLIPper’s code base.
tuned. These findings indicate that the fine-tuned LLaVA struggles to generalise effectively to diverse domains of hateful memes.
(II) When using the HarMeme as the retrieval database, our system achieves an AUC of 60.0%, surpassing both the baseline HateCLIPper’s AUC of 55.8% and the best LMM’s zero-shot AUC score.
Additionally, we provide the results of using logistic regression (LR) as an alternative to the KNN classifier, both with and without RGCL training, when systems trained on HarMeme are tested on HatefulMemes. The performance of logistic regression consistently falls short of the KNN classifier. Logistic regression with RGCL training achieves an AUC of 59.4%, outperforming the HateCLIPper’s baseline by 7%. Note that the logistic regression does not the retrieval of examples.
(III) When using HatefulMemes as the retrieval database, the HateCLIPper’s performance degrades, suggesting its embedding space lacks generalizing capability to different domains of hateful memes. RGCL boosts the AUC to 66.6%, outperforming the baseline by a large margin of 12.2%. RGCL achieves an accuracy of 59.9%, surpassing the baseline by 9.6%. RGCL’s AUC and accuracy score also surpass the zero-shot LMMs.
(IV) When our system is trained and evaluated on the HatefulMemes dataset (the same system from Table 1), the KNN classifier obtains 86.7%
AUC and 78.3% accuracy. These scores also surpass all baseline systems including fine-tuned LMMs in Table 1.

heading: 4.3 Effects of incorporating pseudo-gold positive and hard negative examples
text: In Table 3, we report a comparative analysis by examining performance when specific examples are excluded during the training process.
When we omit the pseudo-gold positive examples, only in-batch positive examples are incorporated during the training. This results in an accuracy degradation of 1.5%. Hard positive examples, same-label samples with high similarity scores, are commonly used in contrastive learning literature. In our case, when incorporating hard positive examples rather than pseudo-gold positive examples, the training becomes unstable and results in divergence.
When the hard negative examples are excluded, leaving only in-batch negative samples, the performance degrades 1.7% for accuracy. When removing both types of examples, there is more performance degradation. Both the pseudo-gold positive examples and the hard negative examples are needed for accurately classifying hateful memes.
When excluding the in-batch negative examples, training becomes unstable and fail to converge, which is consistent with previous findings in (Henderson et al., 2017).

heading: 4.4 Effects of different VL Encoder
text: We ablate the performance when incorporating RGCL on various VL encoders. As shown in Table 4, we experiment with various encoders in the CLIP family: the original CLIP (Radford et al., 2021), OPENCLIP (Ilharco et al., 2021; Schuhmann et al., 2022; Cherti et al., 2023), and AltCLIP (Chen et al., 2022). Our method boosts the performance of all these variants of CLIP by around 3%.
To verify that our method does not depend on the CLIP architecture, we carry out experiments with ALIGN3 (Jia et al., 2021). As shown in Table 4, RGCL enhances the AUC score by a margin of 4.4% over the baseline ALIGN model.

heading: 4.5 Effects of dense/sparse retrieval
text: We compare the commonly used sparse retrieval to our proposed dynamic dense retrieval for obtaining contrastive learning examples. We detail our approach for sparse retrieval in Appendix H.
As shown in Table 5, using a variable number of objects in object detection performs the best in sparse retrieval. However, the accuracy degrades by 0.7% compared to the dense retrieval baseline. When using a fixed number of objects in object detection, the performance degrades even more. Our proposed dynamic dense retrieval obtains better performance than the commonly used sparse retrieval methods.

heading: 4.6 Effects of Retrieval-Guided Contrastive Learning Loss
text: As shown in Eq. 5, the mixing ratio between RGCLL and the CE loss is 1:1 by default. In Table 6, we compare the different mixing ratios be-
3ALIGN only open-sourced the base model which is less capable than the larger CLIP-based models.
tween the two loss functions. We observe a significant performance improvement whenever RGCLL is included. For simplicity, we maintain a 1:1 mixing ratio. Notably, in the absence of cross-entropy loss, we identified several examples where models with RGCL fail but models without RGCL succeed. Conversely, inclusion of cross-entropy loss eliminates such discrepancies.

heading: 4.7 Effects of RGCL on different Meme Classification tasks
text: To demonstrate RGCL’s versatility beyond hateful meme classification, we assess its efficacy on three additional datasets: MultiOFF (Suryawanshi et al., 2020a), Harm-P (Pramanick et al., 2021b), and Memotion7K (Sharma et al., 2020). These datasets originally used the F1 score as their evaluation metric; we also include Accuracy. We train a separate model for each of the datasets following the procedures detailed in Appendices A and B. Table 7 shows the results for CLIP with and without RGCL training on these three datasets4.
MultiOFF contains memes related to the 2016 U.S. presidential election sourced from social media sites, such as Twitter and Instagram. The memes are labeled as non-offensive and offensive. MultiOFF is a relatively small dataset, containing less than 500 training examples. RGCL outperforms the baseline by a significant margin of 4.7% in accuracy. RGCL yields consistent gain even with relatively small datasets.
Harm-P contains harmful and harmless memes on US politics sourced from social media sites. RGCL shows more than 2% gain in both accuracy and F1 score over the baseline system.
Memotion7K, designed for multi-task meme emotion analysis, includes annotations for humor, sarcasm, offensiveness, and motivation. RGCL
4Since CLIP surpasses almost all other prior published systems on these datasets, we do not include prior results in the comparison.
shows improvement over baseline across all four emotion classification tasks with an average gain of more than 3% on both accuracy and F1 scores. These results highlight RGCL’s capability for improving emotion detection.
w/o RGCL w/ RGCL Dataset Acc. F1 Acc. F1

heading: 5 Case Analysis
text: We now analyze how RGCL improves relative to baseline systems on confounding memes.

heading: 5.1 Quantitative analysis
text: From the 500 validation samples of HatefulMemes, we annotated 101 examples and picked 24 confounder memes. On this confounder subset, HateCLIPper without RGCL obtains an accuracy of 66.7%, while RGCL significantly boosts the accuracy to 83.3%. These results show that RGCL improves the classification of challenging confounder memes, which exhibit differences in either the image or text.
Next, we analyze how RGCL improves the classification through examples of confounder memes from the subset.

heading: 5.2 Qualitative analysis
text: In Table 8, we demonstrate how RGCL addresses the classification errors associated with confounder memes. Our approach significantly reduces the similarity scores between anchor memes and confounder memes. This shows that RGCL effectively learns a hatefulness-aware embedding space, placing the meme within the embedding space with a comprehensive hateful understanding derived from both vision and language components. By aligning semantically similar memes closer and pushing apart dissimilar ones in the embedding space, RGCL enhances classification accuracy.

heading: 6 Conclusion
text: We introduce Retrieval-Guided Contrastive Learning to enhance any VL encoder in addressing challenges in distinguishing confounding memes. Our approach uses novel auxiliary loss with dynamically retrieved examples and significantly improves contextual understanding. Achieving an AUC score of 87.0% on the HatefulMemes dataset, our system outperforms prior state-of-the-art models. Our approach also transfers to different tasks, emphasizing its usefulness across diverse meme domains.
Limitation
Hate speech can be defined by different terminologies, such as online harassment, online aggression, cyberbullying, or harmful speech. United Nations Strategy and Plan of Action on Hate Speech stated that the definition of hateful could be controversial and disputed (Nderitu, 2020). Additionally, according to the UK’s Online Harms White Paper, harms could be insufficiently defined (Woodhouse, 2022). We use the definition of hate speech from the two datasets: HatefulMemes (Kiela et al., 2021) and HarMeme (Pramanick et al., 2021a). These datasets adopt Facebook’s definition of hate speech 5 to strike a balance between reducing harm and preserving freedom of speech. Tackling the complex issue of how to define hate speech will require a cooperative effort by stakeholders, including governmental policy makers, academic scholars, the United Nations Human Rights Council, and social media companies. We align our research with the ongoing process of defining the hate speech problem and will continue to integrate new datasets, as they become available.
In examining the error cases of our system, we find that the system is unable to recognize subtle facial expressions. This can be improved by using a more powerful vision encoder to enhance image understanding. We leave this to future work.
Ethical Statement
Reproducibility. We present the detailed experiment setups and hyperparameter settings in Appendices A and B. The source code will be released upon publication.
Usage of Datasets. The HatefulMemes, HarMeme, MultiOFF, and Harm-P datasets were
5https://transparency.fb.com/en-gb/policies/communitystandards/hate-speech/
(a)
Anchor memes Image confounders Text confounders Ground truth labels Hateful Benign Benign
curated and designed to help fight online hate speech for research purposes only. Throughout the research, we strictly follow the terms of use set by their authors.
Societal benefits. Hate speech detection systems like RGCL contribute significantly to reducing online hate speech, promoting safer digital environments, and aiding in protecting human content moderators. These positive impacts, we believe, are substantial and crucial in the broader context of online communication and safety.
Intended use. We intend to enforce strict access controls upon the model release. The model will only be shared with researchers after signing the terms of use. We will clearly state that the system is intended for the detection and prevention of hateful speech. We will specify that it should not be used for any purposes that promote, condone, or encourage hate speech or harmful content.
Implementation consideration. Because our system is based on retrieving examples, multiple retrieval sets reflect different cultural sensitivities that can be applied in reality. Our architecture is well suited to addressing the problem of cultural differences or subjective topics without retraining. However, the annotation of datasets in handling cultural differences or subjective topics needs to be take into consideration before any deployment of systems. The factors need to be considered includes the data curation guidelines, bias of the annotators, and the limited definition of hate speech.
Misuse Potential. Our proposed system does not induce biases. However, training the system on HatefulMemes or HarMeme may cause unintentional biases towards certain individuals, groups, and entities (Pramanick et al., 2021b). To counteract potential unfair moderation stemming from dataset-induced biases, incorporating human moderation is necessary.
Environmental Impact Training large-scale Transformer-based models requires a lot of computations on GPUs/TPUs, which contributes to global warming. However, this is a bit less of an issue for our system, since we only fine-tune small components of vision-language models. Our system can be trained under 30 minutes on a single GPU. The fine-tining takes far less time compared to LMMs. Moreover, as our model is relatively small, the inference cost is much less compared to LMMs.

heading: Acknowledgments
text: Jingbiao Mei is supported by Cambridge Commonwealth, European and International Trust for the undertaking of the PhD in Engineering at the University of Cambridge.
Jinghong Chen is supported by the Warwick Postgraduate Studentship from Christ’s College and the Huawei Hisilicon Studentship for the undertaking of the PhD in Engineering at the University of Cambridge.
Weizhe Lin is supported by a Research Studentship funded by Toyota Motor Europe (RG92562(24020)) for the undertaking of the PhD in Engineering at the University of Cambridge.
Prof. Bill Byrne holds concurrent appointments as a Professor of Information Engineering at Cambridge University and as an Amazon Scholar. This publication describes work performed at Cambridge University and is not associated with Amazon.
We would also like to thank all the reviewers for their knowledgeable reviews.

heading: A Experiment Setup
text: A work station equipped with NVIDIA RTX 3090 and AMD 5900X was used for the experiments. PyTorch 2.0.1, CUDA 11.8, and Python 3.10.12 were used for implementing the experiments. HuggingFace transformer library (Wolf et al., 2019) was used for implementing the pretrained CLIP encoder (Radford et al., 2021). Faiss (Johnson et al., 2019) vector similarity search library with version faiss-gpu 1.7.2 was used to perform dense retrieval. Sparse retrieval was performed with rank-bm25 0.2.2 6. All the reported metrics were computed by TorchMetrics 1.0.1. For LLaVA (Liu et al., 2023), we fine-tuned the model on a system with 4 A100-80GB. The runtime was 4 hours on the HatefulMemes and 3 hours on the HarMeme. The details for fine-tuniung is covered in Appendix E. All the metrics were reported based on the mean of three runs with different seeds. Due to the limited space in Table 1, we provide more details for our main results here. HateCLIPper with RGCL obtained an accuracy of 78.77± 0.25 and an AUC of 86.95± 0.21 on HatefulMemes.

heading: B Hyperparameter
text: The default hyperparameter for all the models are shown in Table 9. The modelling hyperparameter is based on HateCLIPper’s setting (Kumar and
6https://github.com/dorianbrown/rank_bm25
Nandakumar, 2022) for a fair comparison. For vision and language modality fusion, we perform element-wise product between the vision embeddings and language embeddings. This is known as align-fusion in HateCLIPper (Kumar and Nandakumar, 2022). The hyperparameters associated with retrieval-guided contrastive learning are manually tuned with respect to the evaluation metric on the development set. With this configuration of hyperparameter, the number of trainable parameters is about 5 million and training takes around 30 minutes.

heading: C HateCLIPper’s Architecture
text: For the ith image and text pair (Ii, Ti), HateCLIPper obtains the feature embeddings fI and fT 7 with the pretrained CLIP vision and language encoders. To facilitate the learning of task-specific features, distinct trainable projection layers are employed after the extracted feature vectors to obtain projected features f ′I and f ′ T . f ′ I and f ′ T are vectors of dimension n, which is a hyperparameter to tune. These trainable projection layers consist of a feedforward layer followed by a dropout layer. These feature vectors undergo explicit cross-modal interaction via Hadamard product i.e., element-wise multiplication. This fusion process is referred to "align-fusion" within the HateCLIPper framework. After the align-fusion, a series of Pre-Output layers are employed, comprising multiple feedfor-
7Dropped subscript i for simplicity
ward layers incorporating activation functions and dropout layers. These layers are applied to the image and text representation f ′I and f ′ T to obtain the final embedding vector gi. The number of PreOutput layers is a hyperparameter to tune. We shorthand this process of obtaining the joint embedding vector with F(·, ·) for simplification as denoted in Eq. 1.

heading: D Dataset details and statistics
text: Table 10 shows the data split for the HatefulMemes and HarMeme datasets. Note that HarMeme is first introduced in Pramanick et al. 2021a, however, in Pramanick et al. 2021b, HarMeme had been renamed to Harm-C. Following the notation of previous works (Cao et al., 2022), we use its original name HarMeme in this paper. The memes in HarMeme are labeled with three classes: very harmful, partially harmful, and harmless. Following previous work (Cao et al., 2022; Pramanick et al., 2021b), we combine the very harmful and partially harmful memes into hateful memes and regard harmless memes as benign memes.
In addition to hateful memes classification, we also evaluate the MultiOFF, Harm-P and Memotion7K datasets. Table 11 shows the dataset statistics.
To access the Facebook HatefulMemes dataset, one must follow the license from Face-
book8.HarMeme and Harm-P is distributed for research purpose only, without a license for commercial use. MultiOFF is licensed under CC-BY-NC. Memotion7K has no specific license mentioned.

heading: E LLaVA experiments
text: For fine-tuning LLaVA (Liu et al., 2023), we follow the original hyperparameters setting9 for finetuning on downstream tasks. For the prompt format, we follow InstructBLIP (Dai et al., 2023). For computing the AUC and accuracy metrics, we also follow InstructBLIP’s procedure.

heading: F Ablation study on numbers of retrieved examples
text: We experiment with using more than one hard negative and pseudo-gold positive gold examples in training.
The inclusion of more than one example for both types of examples causes the performance to degrade. This phenomenon aligns with recent findings in the literature, as Karpukhin et al. (2020) reported that the incorporation of multiple hard negative examples does not necessarily enhance performance in passage retrieval.

heading: G Ablation study on loss function and similarity metrics
text: Inner product (IP) and Euclidean L2 distance are also commonly used as similarity measures. Since Euclidean distance (L2) is a distance metric, we take its negative to serve as a measure of similarity. We tested these alternatives and found cosine similarity performs slightly better as shown in Table 13.
Additionally, another popular loss function for ranking is triplet loss (Chechik et al., 2009; Schroff et al., 2015) which compares a positive example
8https://hatefulmemeschallenge.com/#download 9https://github.com/haotian-liu/LLaVA
with a negative example for an anchor meme. Our results in Table 13 suggest that using triplet loss performs comparably to the default NLL loss.

heading: H Sparse retrieval
text: We use VinVL object detector (Zhang et al., 2021) to obtain the region-of-interest object prediction and its corresponding attributes.
After obtaining these text-based image features, we concatenate these text with the overlaid caption from the meme to perform the sparse retrieval. We use BM-25 (Robertson and Zaragoza, 2009) to perform sparse retrieval. For variable number of object predictions, we set a region-of-interest bounding box detection threshold of 0.2, a minimum of 10 bounding boxes, and a maximum of 100 bounding boxes, consistent with the default settings of the VinVL.

heading: Future Work
text: We leave this to future work