SlowGuess/NAACL_2022_OCR huggingface dataset:
Sean MacAvaney, Hao-Ren Yao, Eugene Yang, Katina Russell, Nazli Goharian, and Ophir Frieder. 2019. Hate speech detection: Challenges and solutions. PLOS ONE, 14(8):1-16.
Alexandros Mittos, Savvas Zannettou, Jeremy Blackburn, and Emiliano De Cristofaro. 2020. "And We Will Fight for Our Race!" A Measurement Study of Genetic Testing Conversations on Reddit and 4chan. In Proceedings of the Fourteenth International AAAI Conference on Web and Social Media, ICWSM '20, pages 452-463, Atlanta, Georgia, USA.
Van-Hoang Nguyen, Kazunari Sugiyama, Preslav Nakov, and Min-Yen Kan. 2020. FANG: Leveraging social context for fake news detection using graph representation. In Proceedings of the 29th ACM International Conference on Information and Knowledge Management, CIKM '20, pages 1165-1174.
Benet Oriol, Cristian Canton-Ferrer, and Xavier Giroi Nieto. 2019. Hate Speech in Pixels: Detection of Offensive Memes towards Automatic Moderation. In Proceedings of the NeurIPS 2019 Workshop on AI for Social Good, Vancouver, Canada.
Hamed Pirsiavash, Deva Ramanan, and Charless Fowlkes. 2009. Bilinear classifiers for visual recognition. In Advances in Neural Information Processing Systems: Proceedings of the International Conference on Neural Information Processing Systems, volume 22, pages 1482-1490, Vancouver British Columbia Canada. Curran Associates, Inc.
Shraman Pramanick, Dimitar Dimitrov, Rituparna Mukherjee, Shivam Sharma, Md. Shad Akhtar, Preslav Nakov, and Tanmoy Chakraborty. 2021a. Detecting Harmful Memes and Their Targets. In Findings of the Association for Computational Linguistics, ACL-IJCNLP '21, pages 2783-2796, Bangkok, Thailand.
Shraman Pramanick, Shivam Sharma, Dimitar Dimitrov, Md. Shad Akhtar, Preslav Nakov, and Tanmoy Chakraborty. 2021b. MOMENTA: A Multimodal Framework for Detecting Harmful Memes and Their Targets. In Findings of the Association for Computational Linguistics: EMNLP 2021, EMNLP 21, pages 4439-4455, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning Transferable Visual Models From Natural Language Supervision. In Proceedings of the 38th International Conference on Machine Learning, ICML '21, pages 8748-8763.
Kunal Relia, Zhengyi Li, Stephanie H. Cook, and Rumi Chunara. 2019. Race, Ethnicity and National Origin-Based Discrimination in Social Media and Hate Crimes across 100 U.S. Cities. Proceedings of the International AAAI Conference on Web and Social Media, 13(01):417-427.
Barbara Gomes Ribeiro, Manoel Horta Ribeiro, Virgílio A. F. Almeida, and Wagner Meira Jr. 2021. Follow the Money: Analyzing @slpng_giants_ct's Strategy to Combat Misinformation. CoRR, abs/2105.07523.
Vlad Sandulescu. 2020. Detecting Hateful Memes Using a Multimodal Deep Ensemble. arXiv:2012.13235.
Morgan Klaus Scheuerman, Jialun Aaron Jiang, Casey Fiesler, and Jed R. Brubaker. 2021. A Framework of Severity for Harmful Content Online. Proceedings ACM Hum.-Comput. Interact., 5(CSCW2).
Shaden Shaar, Firoj Alam, Giovanni Da San Martino, and Preslav Nakov. 2022. The role of context in detecting previously fact-checked claims. In *Findings of the Association for Computational Linguistics: NAACL-HLT* 2022, NAACL-HLT '22, Seattle, Washington, USA.
Lanyu Shang, Christina Youn, Yuheng Zha, Yang Zhang, and Dong Wang. 2021a. KnowMeme: A Knowledge-enriched Graph Neural Network Solution to Offensive Meme Detection. In Proceedings of the 2021 IEEE 17th International Conference on eScience, eScience '21, pages 186-195.
Lanyu Shang, Yang Zhang, Yuheng Zha, Yingxi Chen, Christina Youn, and Dong Wang. 2021b. AOMD: An Analogy-aware Approach to Offensive Meme Detection on Social Media. Information Processing & Management, 58(5):102664.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL '18, pages 2556-2565, Melbourne, Australia.
Shivam Sharma, Firoj Alam, Md. Shad Akhtar, Dimitar Dimitrov, Giovanni Da San Martino, Hamed Firooz, Alon Halevy, Fabrizio Silvestri, Preslav Nakov, and Tanmoy Chakraborty. 2022. Detecting and understanding harmful memes: A survey. In Proceedings of the 31st International Joint Conference on Artificial Intelligence, IJCAI-ECAI '22, Vienna, Austria.
Kai Shu, Amy Sliva, Suhang Wang, Jiliang Tang, and Huan Liu. 2017. Fake News Detection on Social Media: A Data Mining Perspective. SIGKDD Explor. Newsl., 19(1):22-36.
Karen Simonyan and Andrew Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. In Proceedings of the International Conference on Learning Representations, ICLR '15, San Diego, California, USA.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019. Energy and Policy Considerations for Deep Learning in NLP. In Proceedings of the 57th
Annual Meeting of the Association for Computational Linguistics, ACL '19, pages 3645-3650, Florence, Italy.
Shardul Suryawanshi, Bharathi Raja Chakravarthi, Michael Arcan, and Paul Buitelaar. 2020. Multimodal Meme Dataset (MultiOFF) for Identifying Offensive Content in Image and Text. In Proceedings of the Second Workshop on Trolling, Aggression and Cyberbullying, pages 32-41, Marseille, France. European Language Resources Assoc. (ELRA).
Yuping Wang, Fatemeh Tahmasbi, Jeremy Blackburn, Barry Bradlyn, Emiliano De Cristofaro, David Magerman, Savvas Zannettou, and Gianluca Stringhini. 2021. Understanding the Use of Fauxtography on Social Media. Proceedings of the International AAAI Conference on Web and Social Media, 15(1):776-786.
Liang Wu and Huan Liu. 2018. Tracing Fake-News Footprints: Characterizing Social Media Messages by How They Propagate. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, WSDM '18, pages 637-645, Marina Del Rey, CA, USA. Association for Computing Machinery.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. XLNet: Generalized Autoregressive Pretraining for Language Understanding. In Advances in Neural Information Processing Systems, volume 32 of NIPS '19, Vancouver, BC, Canada.
Marcos Zampieri, Preslav Nakov, Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Hamdy Mubarak, Leon Derczynski, Zeses Pitenis, and Căgri Öltekin. 2020. SemEval-2020 task 12: Multilingual offensive language identification in social media (OffensEval 2020). In Proceedings of the Fourteenth Workshop on Semantic Evaluation, SemEval '20, pages 1425–1447, Barcelona (online). International Committee for Computational Linguistics.
Ziqi Zhang and Lei Luo. 2019. Hate Speech Detection: A Solved Problem? The Challenging Case of Long Tail on Twitter. Semantic Web, 10(5):925-945.
Xiayu Zhong. 2020. Classification of Multimodal Hate Speech - The Winning Solution of Hateful Memes Challenge. arXiv e-prints.
Kaimin Zhou, Chang Shu, Binyang Li, and Jey Han Lau. 2019. Early Rumour Detection. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT '19, pages 1614-1623, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
Yi Zhou, Zhenhao Chen, and Huiyuan Yang. 2021. Multimodal Learning for Hateful Memes Detection. In Proceedings of the 2021 IEEE International Conference on Multimedia & Expo Workshops, ICMEW '21, pages 1-6, Shenzhen, China.
A Implementation Details and Hyper-parameter Values
We trained all our models using PyTorch on NVIDIA Tesla V100 GPU, with 32 GB dedicated memory, CUDA-11.2 and cuDNN-8.1.1 installed. For the unimodal models, we imported all the pre-trained weights from the TORCHVISION MODELS<sup>8</sup>, a sub-package of the PyTorch framework. We initialized the remaining weights randomly using a zero-mean Gaussian distribution with a standard deviation of 0.02. We train DISARM in a setup considering only harmful class data from Harm-P (Pramanick et al., 2021b). We extended it by manually annotating for harmful targets, followed by including non-harmful examples using automated entity extraction (textual and visual) strategies for training/validation splits and manual annotation (for both harmful and non-harmful) for the test split.
When training our models and exploring various values for the different model hyperparameters, we experimented with using the Adam optimizer (Kingma and Ba, 2015) with a learning rate of  $1e^{-4}$ , a weight decay of  $1e^{-5}$ , and a Binary Cross-Entropy (BCE) loss as the objective function. We extensively fine-tuned our experimental setups based upon different architectural requirements to select the best hyper-parameter values. We also used early stopping for saving the best intermediate checkpoints. Table 4 gives more detail about the hyper-parameters we used for training. On average, it took approximately 2.5 hours to train a multi-modal neural model.
BS	#Epochs	LR	V-Enc	T-Enc	#Param
UM	GRU	32	25	0.0001	-	bert	2M
XLNet	16	20	0.0001	-	xlnet	116M
VGG16	32	25	0.0001	VGG16	-	117M
ViT	16	20	0.0001	vit	-	86M
MM	MMFT	16	20	0.001	ResNet-152	bert	170M
MMBT	16	20	0.001	ResNet-152	bert	169M
ViBERT*	16	10	0.001	Faster RCNN	bert	112M
V-BERT*	16	10	0.001	Faster RCNN	bert	247M
DISARM	16	30	0.0001	vit	bert	111M
Table 4: Hyperparameters summary. [BS→Batch Size; LR→Learning Rate; V/T-Enc→Vision/TextEncoder; vit→vit-base-patch16-224-in21k; bert::bert-base-uncased; x1net→x1net-base-uncased].
In this section, we present some ablation studies for sub-modules of DISARM based on CE, EH, CT, and CI, examined in isolation and in combinations, and finally for DISARM using CMM.
B.1 Test Set A
As observed in the comparisons made with the other baseline systems for the Test Set A in Table 2, the overall range of the F1 scores is relatively higher with the lowest value being 0.66 for XLNet (text-only) model. The results for unimodal systems, as can be observed in Table 5, is satisfactory with values of 0.74, 0.73, and 0.77 for CE EH, and CI unimodal systems, respectively. For multimodal systems, we can observe distinct lead for the MMLRBP-based fusion strategy, for both CE and EH systems over the concatenation-based approach, except for EH's recall drop by  $7\%$ . Finally DISARM yields the best overall F1 score of 0.78.
B.2 Test Set B
With context not having any harmfulness cues for a given meme when considered in isolation, the unimodal CE module performs the worst with 0.48 F1 score, and 0.07 recall for the harmful class, in the open-ended setting of Test Set B. In contrast, EH yields an impressive F1 score of 0.55, and a harmful recall of 0.41. This relative gain of  $7\%$  in terms of F1 score could be due to the presence of explicit harmfulness cues. The complementary effect of considering contextual information can be inferred from the joint modeling of CE and EH, to obtained CT, that enhances the F1 score and the harmful recall by  $2\%$  and  $3\%$ , respectively (see Table 5). Unimodal assessment of CI performs moderately with an F1 score of 0.51, but with a poor harmful recall of 0.15. MMLRBP, towards joint-modeling of CE and CI yields a significant boost in the harmful recall to 0.52 (see Table 5). On the other hand, MMLRBP-based fusion of EH and CI yields 0.54 F1 score, which is  $1\%$  below that for the unimodal EH system. This emphasizes the importance of accurately modeling the embedded harmfulness, besides augmenting with additional context. A complementary impact of CE, EH, and CI is observed for DISARM with a balanced F1 score of 0.6 and a competitive harmful recall value of 0.69.
http://pytorch.org/docs/stable/ torchvision/models.html
Approach	Test Set A	Test Set B	Test Set C
F1	Not-harmful	Harmful	F1	Not-harmful	Harmful	F1	Not-harmful	Harmful
P	R	P	R	P	R	P	R	P	R	P	R
CE	0.7411	0.71	0.78	0.77	0.71	0.4847	0.78	0.95	0.29	0.07	0.4829	0.83	0.93	0.17	0.06
EH	0.7250	0.75	0.66	0.71	0.79	0.5544	0.81	0.72	0.3	0.41	0.5658	0.88	0.68	0.27	0.56
CI	0.7729	0.74	0.82	0.81	0.73	0.5174	0.79	0.89	0.29	0.15	0.5314	0.84	0.87	0.23	0.19
CE + EH	0.7406	0.71	0.78	0.78	0.7	0.5775	0.82	0.74	0.33	0.44	0.5840	0.89	0.7	0.29	0.57
CE + CI (concat)	0.7361	0.71	0.77	0.77	0.7	0.4230	0.74	0.51	0.18	0.37	0.4125	0.82	0.43	0.17	0.56
CE + CI (MMLRBP)	0.7790	0.74	0.84	0.83	0.72	0.5079	0.81	0.57	0.26	0.52	0.4857	0.88	0.5	0.22	0.69
EH + CI (concat)	0.6609	0.67	0.6	0.66	0.72	0.4964	0.78	0.65	0.23	0.37	0.4421	0.81	0.57	0.15	0.38
EH + CI (MMLRBP)	0.7260	0.74	0.67	0.72	0.78	0.5470	0.8	0.74	0.29	0.37	0.4836	0.83	0.68	0.17	0.31
DISARM	0.7845	0.74	0.81	0.74	0.86	0.6498	0.83	0.79	0.38	0.69	0.6412	0.86	0.76	0.36	0.7
Table 5: Ablation results for DISARM and its variants for Test Sets A, B, and C.
B.3 Test Set C
As observed in the previous scenario, the unimodal models for CE yield a low F1 score of 0.48 and the worst harmful recall value of 0.06. Much better performance is observed for unimodal setups including EH, and its joint modelling with CE with improved F1 scores of 0.56 and 0.58, respectively, along with the harmful recall score of 0.56 and 0.57, respectively. CI based unimodal evaluation again yields a moderate F1 score of 0.53 (see Table 5), along with a poor harmful recall of 0.19, which shows its inadequacy to model harmful targeting on its own. For multimodal setups, the joint modelling of CE and CI benefits from MMLRBP based fusion, yielding a gain of  $7\%$  and  $13\%$  in terms of F1 score and harmful recall, respectively. This confirms the importance of contextual multimodal semantic alignment. Correspondingly, joint multimodal modelling of EH and CI regresses the unimodal affinity within the EH. Finally, DISARM outperforms all other systems in this category with the best F1 score of 0.64, with a decent harmful recall score of 0.7.
The experimental results here are for comparison and analysis of the optimal set of design and baseline choices. We should note that we performed extensive experiments as part of our preliminary investigation, with different contextual modelling strategies, attention mechanisms, modelling choices, etc., to reach a conclusive architectural configuration that show promise for addressing the task of target detection in harmful memes.
C Error Analysis
It is evident from the results shown in Tables 2 and 3 that DISARM still has shortcomings. Examples like the one shown in Fig. 6 are seemingly harmless, both textually and visually, but imply serious harm to a person of color in an implicit way.
(c) V-AT-DISARM
(d) V-AT-ViLBERT
Target Candidate  $\rightarrow$  person of color Context  $\rightarrow$  During the evening of the VP debates, Joe Biden settled down on his soft couch with a glass of warm milk to watch this.
Figure 6: Comparison of attention maps for miclassification between DISARM [(a), (b) & (c)] and ViLBERT [(d)] using BertViz and Grad-CAM.
This kind of complexity can be challenging to model without providing additional context about the meme like people of colour face racial discrimination all over the world. This is also analogous to a fundamental challenge associated with detecting implicit hate (MacAvaney et al., 2019). In this particular example, despite modelling contextual information explicitly in DISARM, it misclassifies this meme anyway.
(a) Harmful analogy
(b) Sensitive visuals
(c) Political grounds
Figure 7: Examples of memes depicting different types (a)-(e) of harmful targeting.
(d) Religious grounds
(e) International threat
Even though the context obtained for this meme pertains to its content (see Fig. 6), it does not relate to global racial prejudice, which is key to ascertaining it as a harmfully targeting meme. Moreover, besides context, visuals and the message embedded within the meme do not convey definite harm when considered in isolation. This error can be inferred clearly from the embedded-harmfulness, contextualised-visuals, and the visuals being attended by DISARM as depicted in Fig. 6a, Fig. 6b, and Fig. 6c, respectively. On the other hand, as shown in the visual attention plot for ViLBERT in Fig. 6d, the field of view that is being attended encompasses the visuals of Kamala Harris, who is the person of colour that i sbeing primarily targeted by the meme. Besides the distinct attention on the primary target-candidate within the meme, ViLBERT could have leveraged the pre-training it received from Conceptual Captions (CC) (Sharma et al., 2018), a dataset known for its diverse coverage of complex textual descriptions. This essentially highlights the importance of making use of multimodal pre-training using the dataset that is not as generic as MS COCO (Lin et al., 2014), but facilitates modelling of the complex real-world multimodal information, especially for tasks related to memes.
D Annotation Guidelines
Before discussing some details about the annotation process, revisiting the definition of harmful memes would set the pretext towards consideration of harmful targeting and non-harmful referencing. According to Pramanick et al. (2021b), a harm can be expressed as an abuse, an offence, a disrespect, an insult, or an insinuation of a targeted entity or any socio-cultural or political ideology, belief, principle, or doctrine associated with that entity. The harm can also be in the form of a more subtle attack such as mocking or ridiculing a person or an idea.
Another common understanding[9,10,11] about the harmful content is that it could be anything online that causes distress. It is an extremely subjective phenomenon, wherein what maybe be harmful to some might not be considered an issue by others. This makes it significantly challenging to characterize and hence to study it via the computational lens.
Based on a survey of 52 participants, Scheuerman et al. (2021) defines online harm to be any violating content that results in any (or a combination) of the following four categories:  $(i)$  physical harm,  $(ii)$  emotional harm,  $(iii)$  relational harm, and  $(iv)$  financial harm. With this in mind, we define two types of referencing that we have investigated in our work within the context of internet memes:  $(i)$  harmful and  $(ii)$  non-harmful.
D.1 Reference Types
Harmful. The understanding about harmful referencing (targeting) in memes, can be sourced back to the definition of harmful memes by Pramanick et al. (2021b), wherein a social entity is subjected to some form of ill-treatment such as mental abuse, psycho-physiological injury, proprietary damage, emotional disturbance, or public image damage, based on their background (bias, social background, educational background, etc.) by a meme author.
Not-harmful. Non-harmful referencing in memes is any benign mention (or depiction) of a social entity via humour, limerick, harmless pun or any content that does not cause distress. Any reference that is not harmful falls under this category.
$^{9}$ https://reportharmfulcontent.com/advice/other/further-advice/harmful-content-online-an-explainer
10https://swgfl.org.uk/services/ report-harmful-content
11https://saferinternet.org.uk/ report-harmful-content
Harmful meme	Not-harmful meme
Individual	Organization	Community	Individual	Organization	Community
joe biden (333)	democratic party (184)	mexicans (11)	donald trump (106)	green party (189)	trump supporters (86)
donald trump (285)	republican party (130)	black (7)	republican voter (102)	biden camp (162)	white (50)
barack obama (142)	libertarian party (44)	muslim (7)	barack obama (94)	communist party (114)	african american (47)
hillary clinton (35)	cnn (6)	islam (6)	joe biden (47)	america (64)	democrat officials (45)
mike pence (13)	government (5)	russian (5)	alexandria ocasio cortez (44)	trump administration (52)	republican (44)
Table 6: The top-5 most frequently referenced entities in each harmfulness class and their target categories. The total frequency for each word is shown in parentheses.
D.2 Characteristics of Harmful Targeting
There are several factors that collectively facilitate the characterisation of harmful targeting in memes. Here are some:
1. A prominent way of harmfully targeting an entity in a meme is by leveraging sarcastically harmful analogies, framed via either textual or visual instruments (see Fig. 7a).
2. There could be multiple entities being harmfully targeted within a meme as depicted in Fig. 2. Hence, annotators were asked to provide all such targets as harmful, with no exceptions.
3. A harmful targeting within a meme could have visual depictions that are either gory, violent, graphically sensitive, or pornographic (see Fig. 7b).
4. Any meme that insinuates an entity on either social, political, professional, religious grounds, can cause harm (see Fig. 7c and 7d).
5. Any meme that implies an explicit/implicit threat to an individual, a community, a national or an international entity is harmful (see Fig. 7d and 7e).
6. Whenever there is any ambiguity regarding the harmfulness of any reference being made, we requested the annotators to proceed following the best of their understanding.
E Ext-Harm-P Characteristics
Below, we perform some analysis of the lexical content of the length of the meme text.
E.1 Lexical Analysis
Interestingly, a significant number of memes are disseminated making references to popular individuals such as Joe Biden, Donald Trump, etc., as can be observed for individual sub-categories (for both harmful and non-harmful memes) in Table 6.
We can see in Table 6 that for harmful-organization, the top-5 harmfully targeted organizations include the top-2 leading political organizations in the USA (the Democratic Party and the Republican Party), which are of significant political relevance, followed by the Libertarian Party, a media outlet (CNN), and finally the generic government. At the same time, non-harmfully referenced organizations includes the Biden camp and the Trump administration, which are mostly leveraged for harmfully targeting (or otherwise) the associated public figure. Finally, communities such as Mexicans, Black, Muslim, Islam, and Russian are often immensely prejudiced against online, and thus also in our meme dataset. At the same time, non-harmfully targeted communities such as the Trump supporters and the African Americans are not targeted as often as the aforementioned ones, as we can see in Table 6.
The above analysis of the lexical content of the memes in our datasets largely emphasizes the inherent bias that multimodal content such as memes can exhibit, which in turn can have direct influence on the efficacy of machine/deep learning-based systems for detecting the entities targeted by harmful memes. The reasons for this bias are mostly linked to societal behaviour at the organic level, and the limitations posed by current techniques to process such data. The mutual exclusion for harmful vs. non-harmful categories for community shows the inherent bias that could pose a challenge, even for the best multi-modal deep neural systems. The high pervasiveness of a few prominent keywords could effectively lead to increasing bias towards them for specific cases. At the same time, the significant overlap observed in Table 6 for the enlisted entities, between harmful and not-harmful individuals, highlights the need for sophisticated multi-modal systems that can effectively reason towards making a complex decision like detecting harmful targeting within memes, rather than exploit the biases towards certain entities in the training data.
(a) Trump
(d) Biden
Figure 8: Distributions of the OCR's length for the memes of top-5 harmful references: harmful (Blue) and non-harmful (Orange). The depiction is for Individual: (a) and (d); Organization: (b) and (e); and Community: (c) and (f).
(b) Republican Party
(e) Democratic Party
(c) Mexican
(f) Black
E.2 Meme-Message Length Analysis