SlowGuess/NAACL_2022_OCR huggingface dataset:
- Unimodal Systems: VGG16, VIT: For the unimodal (image-only) systems, we use two well-known models: VGG16 (Simonyan and Zisserman, 2015) and VIT (Vision Transformers) that emulate a Transformer-based application jointly over textual tokens and image patches (Dosovitskiy et al., 2021). GRU, XLNet: For the unimodal (text-only) systems, we use GRU (Cho et al., 2014), which adaptively captures temporal dependencies, and XLNet (Yang et al., 2019), which implements a generalized auto-regressive pre-training strategy.
- Multimodal Systems:  $\triangleright$  MMF Transformer: This is a multimodal Transformer model that uses visual and language tokens with self-attention.6  $\triangleright$  MMBT: Multimodal Bittransformer (Kiela et al., 2019) captures the intramodal and the inter-modal dynamics.  $\triangleright$  ViLBERT CC: Vision and Language BERT (Lu et al., 2019), pre-trained on CC (Sharma et al., 2018), is a strong model with task-agnostic joint representation.  $\triangleright$  Visual BERT COCO: Visual BERT (Li et al., 2019), pre-trained on the MS COCO dataset (Lin et al., 2014).
http://mmf.sh/docs/notes/model_zoo
4github.com/facebookresearch/mmf
Additional details along with the values of the hyperparameters are given in Appendix A.
System	Modality	Approach	Test Set A	Test Set B
Acc	Prec	Rec	F1	Not-harmful	Harmful	Acc	Prec	Rec	F1	Not-harmful	Harmful
P	R	P	R	P	R	P	R
Baselines	Unimodal	XLNet Text-only	0.6765	0.69	0.67	0.6663	0.73	0.52	0.65	0.82	0.5041	0.425	0.405	0.4060	0.72	0.59	0.13	0.22
VGG Image-only	0.7451	0.75	0.745	0.7438	0.71	0.81	0.79	0.68	0.5455	0.42	0.405	0.4101	0.73	0.66	0.11	0.15
GRU Text-only	0.7484	0.745	0.75	0.7473	0.73	0.76	0.76	0.74	0.5455	0.43	0.42	0.4210	0.73	0.65	0.13	0.19
ViT Image only	0.7647	0.765	0.765	0.7642	0.74	0.79	0.79	0.74	0.5207	0.525	0.535	0.4843	0.8	0.51	0.25	0.56
Multimodal	ViLBERT CC	0.6895	0.69	0.685	0.6835	0.71	0.6	0.67	0.77	0.438	0.535	0.53	0.4302	0.82	0.35	0.25	0.71
MM Transformer	0.6993	0.71	0.695	0.6926	0.75	0.57	0.67	0.82	0.7769	0.53	0.575	0.5032	0.78	0.51	0.28	0.64
VisualBERT	0.7026	0.725	0.69	0.6918	0.78	0.54	0.67	0.84	0.5537	0.545	0.565	0.5108	0.82	0.54	0.27	0.59
VisualBERT - COCO	0.7059	0.71	0.7	0.7014	0.73	0.62	0.69	0.78	0.5785	0.53	0.545	0.5147	0.8	0.61	0.26	0.48
MMBT	0.7157	0.72	0.71	0.7121	0.74	0.64	0.7	0.78	0.6116	0.54	0.55	0.5310	0.81	0.66	0.27	0.44
ViLBERT	0.7516	0.755	0.75	0.7495	0.78	0.68	0.73	0.82	0.6612	0.58	0.595	0.5782	0.83	0.71	0.33	0.48
Prop. system & variants	CE + CI (concat)	0.7353	0.74	0.735	0.7361	0.71	0.77	0.77	0.7	0.4793	0.46	0.44	0.4230	0.74	0.51	0.18	0.37
CE + CI (MMLRBP)	0.781	0.785	0.78	0.7790	0.74	0.84	0.83	0.72	0.562	0.535	0.545	0.5079	0.81	0.57	0.26	0.52
EH + CI (concat)	0.6634	0.665	0.66	0.6609	0.67	0.6	0.66	0.72	0.5868	0.505	0.51	0.4964	0.78	0.65	0.23	0.37
EH + CI (MMLRBP)	0.7255	0.73	0.725	0.7260	0.74	0.67	0.72	0.78	0.6612	0.545	0.555	0.5470	0.8	0.74	0.29	0.37
DISARM	0.781	0.74	0.835	0.7845	0.74	0.81	0.74	0.86	0.74	0.605	0.74	0.6498	0.83	0.79	0.38	0.69
Δ(DISARM-ViLBERT)×100(%)	↑ 2.94%	↓ 1.5%	↑ 8%	↑ 3.5%	↓ 4%	↑ 13%	↑ 1%	↑ 4%	↑ 7.88%	↑ 2.5%	↑ 14.5%	↑ 7.16%	-	↑ 8%	↑ 5%	↑ 21%
Table 2: Performance comparison of unimodal and multimodal models vs. DISARM (and its variants) on Test Sets A and B.
Sys	Approach	Acc	Prec	Rec	F1	Not-harmful	Harmful
P	R	P	R
Baselines	Unimodal	GRU Text-only	0.478	0.45	0.41	0.394	0.78	0.51	0.12	0.31
ViLBERT	0.532	0.435	0.4	0.403	0.78	0.61	0.09	0.19
XLNet Text-only	0.445	0.51	0.515	0.415	0.84	0.41	0.18	0.62
VGG Image-only	0.532	0.45	0.42	0.414	0.79	0.59	0.11	0.25
Multimodal	ViBERT CC	0.358	0.53	0.49	0.350	0.87	0.26	0.19	0.72
VisualBERT	0.478	0.535	0.56	0.442	0.87	0.43	0.2	0.69
MM Transformer	0.510	0.505	0.505	0.448	0.83	0.51	0.18	0.5
ViBERT	0.608	0.525	0.54	0.505	0.84	0.64	0.21	0.44
VisualBERT - COCO	0.771	0.525	0.515	0.511	0.83	0.91	0.22	0.12
MMBT	0.587	0.55	0.575	0.514	0.87	0.59	0.23	0.56
Prop system & variants	CE + CI (concat)	0.456	0.495	0.495	0.412	0.82	0.43	0.17	0.17	0.56
CE + CI (MMLRBPT)	0.532	0.55	0.595	0.485	0.88	0.5	0.22	0.69
EH + CI (concat)	0.532	0.48	0.475	0.442	0.81	0.57	0.15	0.38
EH + CI (MMLRBPT)	0.619	0.5	0.495	0.483	0.83	0.68	0.17	0.31
DISARM	0.739	0.61	0.73	0.641	0.86	0.76	0.36	0.7
Δ(DISARM - MMBRT)×100 (%)	↑15.21%	↑6%	↑15.8%	↑12.66%	↓1%	↑17%	↑13%	14%
Table 3: Performance comparison of unimodal and multimodal models vs. DISARM (and its variants) on Test Set C.
Experimental Results. We compare the performance of several unimodal and multimodal systems (pre-trained or trained from scratch) vs. DISARM and its variants. All systems are evaluated using the 3-way testing strategy described above. We then perform ablation studies on representations that use the contextualized-entity, its fusion with embedded-harmfulness resulting into contextualized-text, and the final fusion with contextualized-image yielding the contextualized-multimodal modules of DISARM (see Appendix B for a detailed ablation study). This is followed by interpretability analysis. Finally, we discuss the limitations of DISARM by performing error analysis (details in Appendix C).
All Entities Seen During Training: In our unimodal text-only baseline experiments, the GRU-based system yields a relatively lower harmful recall of 0.74 compared to XLNet's 0.82, but a better overall F1 score of 0.75 vs. 0.67 for XLNet, as shown in Table 2. The lower harmful precision of 0.65 and the not-harmful recall of 0.52 contribute to the lower F1 score for XLNet.
Among the image-only unimodal systems, VGG performs better with a non-harmful recall of 0.81, but its poor performance for detecting harmful memes yields a lower harmful recall of 0.68. At the same time, VIT has a relatively better harmful recall of 0.74. Overall, the unimodal results (see Table 2) indicate the efficacy of self-attention over convolution for images and RNN (GRU) sequence modeling for text.
Multimodally pre-trained models such as VisualBERT and ViLBERT yield moderate F1 scores of 0.70 and 0.68, and harmful recall of 0.78 and 0.77, respectively (see Table 2). Fresh training facilitates more meaningful results in favour of non-harmful precision of 0.78 for both models, and harmful recall of 0.84 and 0.82 for VisualBERT and ViLBERT, respectively. Overall, ViLBERT yields the most balanced performance of 0.75 in terms of F1 score. It can be inferred from these results (see Table 2) that multimodal pre-training leverages domain relevance.
We can see in Table 2 that multimodal low-rank bi-linear pooling distinctly enhances the performance in terms of F1 score. The improvements can be attributed to the fusion of the CE and EH representations, respectively, with CI, instead of a simple concatenation. This is more prominent for CE with an F1 score of 0.78, which shows the importance of modeling the background context. Finally, DISARM yields a balanced F1 score of 0.78, with a reasonable precision of 0.74 for non-harmful category, and the best recall of 0.86 for the harmful category.
All Entities Unseen as Harmful Targets During Training: With Test Set B, the evaluation is slightly more challenging in terms of the entities to be assessed, as these were never seen at training time as harmful.
<sup>7</sup>We use the abbreviations CE, CT, CI, CMM, EH, and MMLRBP for the contextualized representations of the entity, the text, the image, the multimodal representation, the embedded-harmfulness, and the multimodal low-rank bilinear pooling, respectively.
(a) L-AT
(b) MM-AT-CLIP
(c) V-AT-DISARM
Figure 5: Comparison of the attention-maps for DISARM [(a), (b) & (c)] and ViLBERT [(d)] using BertViz and Grad-CAM.
(d) V-AT-ViLBERT
Target Candidate  $\rightarrow$  democratic party
Context→Politics tears families apart during bruising political season, when many Americans drop friends and family members who have different political views.
Unimodal systems perform poorly on the harmful class, with the exception of XLNet (see Table 2), where the harmful class recall as 0.56. For the multimodal baselines, systems pre-trained using COCO (VisualBERT) and CC (ViLBERT) yield a moderate recall of 0.64 and 0.71 for the harmful class in contrast to what we saw for Test Set A in Table 2. This could be due to additional common-sense reasoning helping such systems, on a test set that is more open-ended compared to Test Set A. Their non-pre-trained versions along with the MM Transformer and MMBT achieve better F1 scores, but with low harmful recall.
Multimodal fusion using MMLRBP improves the harmful class recall for CE to 0.52, but yields lower values of 0.37 for EH fusion with CI (see Table 2). This reconfirms the utility of the context. In comparison, DISARM yields a balanced F1 score of 0.65 with the best precision of 0.83 and 0.38, along with decent recall of 0.79 and 0.69 for non-harmful and harmful memes, respectively.
All Entities Unseen During Training: The results decline in this scenario (similarly to Test Set B), except for the harmful class recall of 0.62 for XLNet, as shown in Table 3. In the current scenario (Test Set C), none of the entities being assessed at testing is seen during the training phase. For multimodal baselines, we see a similar trend for VisualBERT (COCO) and ViLBERT (CC), with the harmful class recall of 0.72 for ViLBERT (CC) being significantly better than the 0.12 for VisualBERT (COCO). This again emphasizes the need for the affinity between the pre-training dataset and the downstream task at hand. In general, the precision for the harmful class is very low.
We observe (see Table 3) sizable boost for the harmful class recall for MMLRBP-based multimodal fusion of CI with CE  $(0.69\%)$ , against a decrease with EH  $(0.31\%)$ . In comparison, DISARM yields a low, yet the best harmful precision of 0.36, and a moderate recall of 0.70 (see Table 3). Moreover, besides yielding reasonable precision and recall of 0.86 and 0.76 for the non-harmful class, DISARM achieves better average precision, recall, and F1 scores of 0.61, 0.73, and 0.64, respectively.
Generalizability of DISARM. The generalizability of DISARM follows from its characteristic modelling and context-based fusion. DISARM demonstrates an ability to detect harmful targeting for a diverse set of entities. Specifically, the three-way testing setup inherently captures the efficacy with which DISARM can detect unseen harmful targets. The prediction for entities completely unseen on training yields better results (see Tables 2 and 3), and suggests possibly induced bias in the former scenario. Moreover, it is a direct consequence of the fact that we were able to incorporate only a limited set of the 246 potential targets. Overall, we argue that DISARM generalizes well for unseen entities with 0.65 and 0.64 macroF1 scores, as compared to ViLBERT's 0.58 and MMBT's 0.51, for Test Sets B and C, respectively.
Comparative Diagnosis. Despite the marginally better harmful recall for ViLBERT (CC) on Test Set B (see Table 2) and Test Set C (see Table 3), the overall balanced performance of DISARM appears to be reasonably justified based on the comparative interpretability analysis between the attention maps for the two systems.
Fig. 5 shows the attention maps for an example meme. It depicts a meme that is correctly predicted to harmfully target the Democratic Party by DISARM and incorrectly by ViLBERT. As visualised in Fig. 5a, the harmfully-inclined word killing effectively attends not only to baby, but also to Democrats and racist. The relevance is depicted via different color schemes and intensities, respectively. Interestingly, killing also attends to the Democratic Party, both as part of the OCR-extracted text and the target-candidate, jointly encoded by BERT. The multimodal attention leveraged by DISARM is depicted (via the CLIP encoder) in Fig. 5b, demonstrating the utility of contextualised attention over the male figure that represents an attack on the Democratic Party. Also, DISARM has a relatively focused field of vision, as shown in Fig. 5c, as compared to a relatively scattered one for ViLBERT (see Fig. 5d). This suggest a better multimodal modelling capacity for DISARM as compared to ViLBERT.
We introduced the novel task of detecting the targeted entities within harmful memes and we highlighted the inherent challenges involved. Towards addressing this open-ended task, we extended Harm-P with target entities for each harmful meme. We then proposed a novel multimodal deep neural framework, called DISARM, which uses an adaptation of multimodal low-rank bi-linear pooling-based fusion strategy at different levels of representation abstraction. We showed that DISARM outperforms various uni/multi-modal baselines in three different scenarios by  $4\%$ ,  $7\%$ , and  $13\%$  increments in terms of macro-F1 score, respectively. Moreover, DISARM achieved a relative error rate reduction of  $9\%$  over the best baseline. We further emphasized the utility of different components of DISARM through ablation studies. We also elaborated on the generalizability of DISARM, thus confirming its modelling superiority over ViLBERT via interpretability analysis. We finally analysed the shortcomings in DISARM that lead to incorrect harmful target predictions.
In the present work, we made an attempt to elicit some inherent challenges pertaining to the task at hand: augmenting the relevant context, effectively fusing multiple modalities, and pretraining. Yet, we also leave a lot of space for future research for this novel task formulation.
Reproducibility. We present detailed hyperparameter configurations in Appendix A and Table 4. The source code, and the dataset ExtHarm-P are available at https://github.com/LCS2-IIITD/DISARM
User Privacy. The information depicted/used does not include any personal information. Copyright aspects are attributed to the dataset source.
Annotation. The annotation was conducted by NLP experts or linguists in India, who were fairly treated and were duly compensated. We conducted several discussion sessions to make sure all annotators could understand the distinction between harmful vs. non-harmful referencing.
Biases. Any biases found in the dataset are unintentional, and we do not intend to cause harm to any group or individual. We acknowledge that detecting harmfulness can be subjective, and thus it is inevitable that there would be biases in our gold-labelled data or in the label distribution. This is addressed by working on a dataset that is created using general keywords about US Politics, and also by following a well-defined schema, which sets explicit definitions for annotation.
Misuse Potential. Our dataset can be potentially used for ill-intended purposes, such as biased targeting of individuals/communities/organizations, etc. that may or may not be related to demographics and other information within the text. Intervention with human moderation would be required to ensure that this does not occur.
Intended Use. We make use of the existing dataset in our work in line with the intended usage prescribed by its creators and solely for research purposes. This applies in its entirety to its further usage as well. We commit to releasing our dataset aiming to encourage research in studying harmful targeting in memes on the web. We distribute the dataset for research purposes only, without a license for commercial use. We believe that it represents a useful resource when used appropriately.
Environmental Impact. Finally, large-scale models require a lot of computations, which contribute to global warming (Strubell et al., 2019). However, in our case, we do not train such models from scratch; rather, we fine-tune them on a relatively small dataset.
The work was partially supported by a Wipro research grant, Ramanujan Fellowship, theInfosys Centre for AI, IIIT Delhi, and ihub-Anubhuti-iitd Foundation, set up under the NM-ICPS scheme of the Department of Science and Technology, India. It is also part of the Tanbih mega-project, developed at the Qatar Computing Research Institute, HBKU, which aims to limit the impact of "fake news," propaganda, and media bias by making users aware of what they are reading, thus promoting media literacy and critical thinking.
Firoj Alam, Stefano Cresci, Tanmoy Chakraborty, Fabrizio Silvestri, Dimiter Dimitrov, Giovanni Da San Martino, Shaden Shaar, Hamed Firooz, and Preslav Nakov. 2021. A Survey on Multimodal Disinformation Detection. arXiv 2103.12541.
Monther Aldwairi and Ali Alwahedi. 2018. Detecting Fake News in Social Media Networks. Procedia Computer Science, 141:215-222.
Justin Cheng, Michael Bernstein, Cristian Danescu-Niculescu-Mizil, and Jure Leskovec. 2017. Anyone Can Become a Troll: Causes of Trolling Behavior in Online Discussions. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing, CSCW '17, pages 1217-1230, Portland, Oregon, USA. Association for Computing Machinery.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP '14, pages 1724-1734, Doha, Qatar. Association for Computational Linguistics.
Christine Cook, Juliette Schaafsma, and Marjolijn Antheunis. 2018. Under the bridge: An in-depth examination of online trolling in the gaming context. New Media & Society, 20(9):3323-3340. PMID: 30581367.
Abhishek Das, Japsimar Singh Wahi, and Siyao Li. 2020. Detecting Hate Speech in Multi-modal Memes. arXiv/2012.14891.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language
Technologies, NAACL-HLT '19, pages 4171-4186, Minneapolis, Minnesota, USA.
Dimitar Dimitrov, Bishr Bin Ali, Shaden Shaar, Firoj Alam, Fabrizio Silvestri, Hamed Firooz, Preslav Nakov, and Giovanni Da San Martino. 2021. Detecting propaganda techniques in memes. In Proceedings of the Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACLIJCNLP '21, pages 6603-6617.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An Image is Worth 16x16 Words: Transformers for Image Recognition at scale. In Proceedings of the 9th International Conference on Learning Representations, ICLR '21, Vienna, Austria.
Rafael Ferreira, Rafael Dueire Lins, Steven J. Simske, Fred Freitas, and Marcelo Riss. 2016. Assessing sentence similarity through lexical, syntactic and semantic analysis. Computer Speech & Language, 39:1-28.
Raul Gomez, Jaume Gibert, Lluis Gomez, and Dimosthenis Karatzas. 2020. Exploring Hate Speech Detection in Multimodal Publications. In Proceedings of the 2020 IEEE Winter Conference on Applications of Computer Vision, WACV '20, pages 99-1467, Snowmass Village, CO, USA.
Eduardo Graells-Garrido, Ricardo Baeza-Yates, and Mounia Lalmas. 2020. Every Colour You Are: Stance Prediction and Turnaround in Controversial Issues. In Proceedings of the 12th ACM Conference on Web Science, WebSci '20, pages 174-183, Southampton, UK. ACM.
Momchil Hardalov, Arnav Arora, Preslav Nakov, and Isabelle Augenstein. 2022. A survey on stance detection for mis- and disinformation identification. In Findings of NAACL 2022, Seattle, Washington, USA.
Kimmo Karkkainen and Jungseock Joo. 2021. *Fairface: Face Attribute Dataset for Balanced Race*, Gender, and Age for Bias Measurement and Mitigation. In *Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision*, WACV '21, pages 1548-1558.
Douwe Kiela, Suvrat Bhooshan, Hamed Firooz, Ethan Perez, and Davide Testuggine. 2019. Supervised Multimodal Bittransformers for Classifying Images and Text. In Proceedings of the NeurIPS Workshop on Visually Grounded Interaction and Language, ViGIL '19, Vancouver, Canada.
Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and
Davide Testuggine. 2020. The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes. In Proceedings of the 34th International Conference on Neural Information Processing Systems, volume 33 of NeurIPS '20.
Jin-Hwa Kim, Kyoung Woon On, Woosang Lim, Jeonghee Kim, Jung-Woo Ha, and Byoung-Tak Zhang. 2017. Hadamard Product for Low-rank Bilinear Pooling. In Proceedings of the 5th International Conference on Learning Representations, ICLR '17, Toulouse, France.
Seunghyun Kim, Afsaneh Razi, Gianluca Stringhini, Pamela J. Wisniewski, and Munmun De Choudhury. 2021. A Human-Centered Systematic Literature Review of Cyberbullying Detection Algorithms. Proceedings ACM Hum. Comput. Interact., 5(CSCW2):1-34.
Diederik P Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In Proceedings of the 3rd International Conference on Learning Representations, ICLR '15, San Diego, California, USA.
Robin Kowalski, Gary Giumetti, Amber Schroeder, and Micah Lattanner. 2014. Bullying in the Digital Age: A Critical Review and Meta-Analysis of Cyberbullying Research Among Youth. Psychological bulletin, 140.
Sumeet Kumar and Kathleen Carley. 2019. Tree LSTMs with Convolution Units to Predict Stance and Rumor Veracity in Social Media Conversations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, ACL '19, pages 5047-5058, Florence, Italy. Association for Computational Linguistics.
Lianian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019. VisualBERT: A Simple and Performant Baseline for Vision and Language. arXiv:1908.03557.
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dolar, and C Lawrence Zitnick. 2014. Microsoft COCO: Common Objects in Context. In Proceedings of the European Conference on Computer Vision, ECCV '14, pages 740-755, Zurich, Switzerland.
Phillip Lippe, Nithin Holla, Shantanu Chandra, Santhosh Rajamanickam, Georgios Antoniou, Ekaterina Shutova, and Helen Yannakoudakis. 2020. A Multimodal Framework for the Detection of Hateful Memes. arXiv:2012.12871.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks. In Proceedings of the Conference on Neural Information Processing Systems, NeurIPS '19, pages 13-23, Vancouver, Canada.