SlowGuess/NAACL_2022_OCR huggingface dataset:
4 Experimental Setup and Evaluation
In this section, we describe the features, baseline models, and the comprehensive experiments using our novel dataset. We test a variety of features and recent multimodal state-of-the-art models.
4.1 Features
Pre-processing: For images, we use the standard pre-processing of resizing and normalizing an image, whereas text is cleaned and normalized according to Cheema et al. (2020a) using the Ekphrasing (Baziotis et al., 2017) tool. Besides digits and alphabets, we also keep punctuation to reflect the syntax and style of a written claim.
Image Features: For image encoding, we use a ResNet-152 (He et al., 2016) model trained on ImageNet (Russakovsky et al., 2015) and extract the 2048-dimensional feature vector from the last pooling layer.
Text Features: For encoding tweet and OCR text, we test BERT (Devlin et al., 2019) uncased models to extract contextual word embeddings. For classification using Support Vector Machine (SVM, Cortes and Vapnik, 1995)), we employ a pooling strategy by adding the last four layers' outputs and then average them to obtain the final 768-dimensional vector.
Multimodal Features: We use the following two pre-trained image-text representation learning architectures to extract multimodal features.
The ALBEF (ALign BBefore Fuse) embedding (Li et al., 2021) results from a recent multimodal state-of-the-art model for vision-language downstream tasks. It is trained on a combination of several image captioning datasets ( $\sim$ 14 million image-text pairs) and uses BERT and a visual transformer (Dosovitskiy et al., 2021) for text and image encoding, respectively. It produces a multimodal embedding of 768 dimensions.
The CLIP (Contrastive Language-Image Pretraining) model (Radford et al., 2021) is trained without any supervision on 400 million image-text pairs.
$^{2}$ Source code is available at: https://github.com/TIBHannover/MM_Clauses  
Dataset (Tweet IDs) and labels are available at: https://data.uni-hannover.de/dataset/mm_claims  
For complete labeled data access (Images and Tweets), please contact at gullal.cheema@tib.eu or gullalcheema@gmail.com
We evaluate several image encoder backbones including ResNet and vision transformer (Dosovitskiy et al., 2021). The CLIP model outputs two embeddings of same size, i.e., the image  $(CLIP_{I})$  and the text  $(CLIP_{T})$  embedding, while  $CLIP_{I\oplus T}$  denotes the concatenation of two embeddings.
4.2 Training Baselines
In the following, we describe training details, hyper-parameters, input combinations, and different baseline models' details.
4.2.1 SVM
To obtain unimodal and multimodal embeddings for our experiments, we first use PCA (Principal Component Analysis) to reduce the feature size and train a SVM model with the  $RBF$  kernel. We perform grid search over PCA energy (\%) conservation, regularization parameter  $C$  and  $RBF$  kernel's gamma. The parameter range for PCA varies from 100% (original features) to 95% with decrements of 1. The parameter range for  $C$  and gamma vary between -1 to 1 on a log-scale with 15 steps. For multimodal experiments, image and text embeddings are concatenated before passing them to PCA and SVM. We normalize the final embedding so that  $l2$  norm of the vector is 1.
4.2.2 BERT and ALBEF Fine-tuning (FT)
We experiment with fine-tuning the last few layers of unimodal and multimodal transformer models to get a strong multimodal baseline and see whether introducing cross-modal interactions improves claim detection performance. We fine-tune the last layers of both the models and report the best ones in Table 3. Additional experimental results on fine-tuned layers are provided in Appendix A.2.5. For fine-tuning, we limit the tweet text to the maximum number of tokens (91) seen in a tweet in the training data and pad the shorter tweets with zeros. Hyper-parameter details for fine-tuning are provided in the Appendix A.1.
4.2.3 Models with OCR Text
To incorporate OCR text embeddings into our models, we experiment with two strategies for embedding generation and one strategy to fine-tune models. To obtain an embedding for SVM models, we experimented with concatenating the OCR embedding to image and tweet text embeddings as well as adding the OCR embedding directly to tweet text embedding. To fine-tune the models, we concate
nate the OCR text to tweet text and limit the OCR text to 128 tokens.
4.2.4 State-of-the-Art Baselines
We compare our models with two state-of-the-art approaches for multimodal fake news detection.
MVAE (Khattar et al., 2019) is a multimodal variational auto-encoder model that uses a multi-task loss to minimize the reconstruction error of individual modalities and task-specific cross-entropy loss for classification. We use the publicly available source code and hyper-parameters for our task.
SpotFake (Singhal et al., 2019) is a model built as a shallow multimodal neural network on top of VGG-19 image and BERT text embeddings using a cross-entropy loss. We re-implement the model in PyTorch and use the hyper-parameter settings given in the paper.
We report accuracy (Acc) and Macro-F1 (F1) for binary (BCD) and tertiary claim detection (TCD) in Table 3. We also present the fraction (in %) of visually-relevant and visually-irrelevant (textual only) claims retrieved by each model in Table 4. Please note that in Table 3 and Table 6, BCD results are shown for only one split  $(T_{C} \rightarrow E_{C})$ , because there are no conflicts in the labels for binary claim classification. Although we do not train the models specifically to detect visual claim labels, we analyze the fraction of retrieved samples in order to evaluate the bias of binary classification models towards a modality.
4.3.1 Impact of Annotation Disagreements
As mentioned in Section 3, we observed disagreements in the annotated data that reflect the real-world difficulty and subjectivity of the problem. Therefore, we analyze the effect of keeping  $(T_{C}, E_{C})$  and removing  $(T, E)$  conflicting examples in training and evaluation data splits (Table 3, 6). The findings are as follows: 1) multimodal models are more sensitive to the conflict resolution strategy as most have lower accuracy when trained on  $T_{C}$  but relatively better F1 score. On the contrary, visual and textual models perform better on both metrics with training on  $T_{C}$ , 2) overall, training on  $T_{C}$  with conflict resolution is a better strategy with a higher F1 score, i.e., better on claim and check-worthiness (fewer samples) detection; and 3) when comparing all the cross-split experiments in Table 3 and Table 6, multimodal models perform the best in case
Task →	BCD	TCD
Data Splits →	TC→EC	TC→EC	TC→EC
Models ↓	Acc	F1	Acc	F1	Acc	F1
Random	50.7	50.2	33.3	30.6	33.3	30.6
Majority	62.7	38.5	56.2	35.9	56.2	35.9
ImageNet	63.1	62.6	58.3	42.9	58.5	43.9
CLIP1	70.0	69.8	64.1	50.5	62.4	48.7
BERT	80.5	79.9	71.9	54.1	69.6	59.8
←FT	80.9	80.1	72.5	54.5	75.4	64.6
CLIPT	75.6	74.7	70.6	53.4	67.4	54.5
BERT ⊕ ImageNet	81.4	80.9	72.7	57.6	71.6	56.9
← ⊕ OCR	80.9	80.4	72.8	58.2	71.9	58.6
CLIPI ⊕ T	77.8	77.4	71.6	52.9	68.4	54.6
CLIPI ⊕ BERT	80.3	79.7	72.7	57.9	69.4	59.7
ALBEF	76.9	76.5	71.5	56.1	65.6	57.3
←FT	80.2	79.7	74.5	60.7	72.5	61.0
← ⊕ OCR ⊕ FT	81.4	81.1	72.7	58.2	73.0	60.8
MVAE	64.1	62.9	60.0	41.2	59.7	44.8
SpotFake	71.8	71.4	67.0	49.5	66.3	52.2
of "without conflicts"  $T$  and  $E$  splits. The latter two observations also apply to retrieval of visually-relevant and visually-irrelevant claims in Table 4 and Table 7.
Table 3: Accuracy (Acc) and Macro-F1 (F1) for binary (BCD) and tertiary claim detection (TCD) in percent [%]. As described in Section 3.5, we use the training split  $(T)$  with resolved (index  $C$ ) and without (no index) conflicts, and evaluation (test) split  $(E_C)$  with conflicts. This evaluation split reflects the real-world scenario for the subjective task of tertiary claim classification (TCD). Unless FT (fine-tuning) is written, all models (except MVAE and SpotFake) are SVM models trained on extracted features.
Data Splits →	T → EC	TC → EC
Models ↓	V (111)	T (145)	V (111)	T (145)
ImageNet	35.1	39.3	61.3	57.9
CLIPI	70.3	67.6	76.6	73.8
BERT←FT	49.6	76.6	57.7	82.1
52.3	75.9	55.9	82.8
CLIPT	46.9	73.1	54.9	73.1
BERT ⊕ ImageNet← ⊕ OCR	57.7	66.2	71.2	77.9
65.8	75.9	71.2	79.3
CLIPI ⊕ T	65.8	66.9	72.9	75.2
CLIPI ⊕ BERT	57.7	72.4	57.7	82.8
ALBEF← FT	61.2	75.2	63.9	77.9
62.2	77.2	70.3	78.6
← ⊕ OCR ⊕ FT	71.2	79.3	75.7	82.1
Table 4: Visually-relevant (V) and visually-irrelevant (text-only) (T) claim detection evaluation. The number of test samples is reported in brackets and the fraction, how many of them were retrieved, is given in percent  $[\%]$ . The underlying models are trained for binary claim detection (BCD). The labels for visual relevance are only used for retrieval evaluation.
4.3.2 Results for Unimodal Models
For image-based models,  $CLIP_{I}$  performs (70.0, 69.8) considerably better than ResNet-152's ImageNet (63.1, 62.6) features in terms of both accuracy and F1 metrics (Table 3, block 2). This result is compliant to previous work (Kirk et al., 2021) where the task has a variety of information and text in images. It is further exaggerated and clearly observable in Table 4 where fraction of visually-relevant claims retrieved using  $CLIP_{I}$  (70.3) is higher and comparable to fine-tuned ALBEF  $\oplus$  OCR (71.2).
For text-based models, fine-tuning (FT) BERT gives the best performance, better than any other unimodal model. This result indicates that the problem is inherently a text-dominant task. The model also retrieves the most visually-irrelevant claims when trained on  $T_{C}$ . It should be noted that textual models can still identify visually-relevant claims since they can have a claim or certain cues in the tweet text that refer to the image. Finally, the  $CLIP_{T}$  features perform considerably worse than BERT features, possibly because CLIP is limited to short text (75 tokens) and is not trained like vanilla BERT on a large text corpus.
4.3.3 Results for Multimodal Models
For multimodal models, the combination of BERT and ResNet-152 features performs slightly better  $(0.5 - 1\%)$  on two metrics in Table 3 on full dataset in binary task and with  $T$  split training in case of tertiary. Although this gain is not impressive, the benefit of combining two modalities is more obvious in identifying visually-relevant claims  $(>10\%)$  in Table 4, which comes at the cost of a lower fraction of visually-irrelevant claims. Similarly with CLIP, the combination of image and text features  $\mathrm{(CLIP_{I\oplus T})}$  improves the overall accuracy from  $\mathrm{CLIP}_I$  or  $\mathrm{CLIP}_T$ . However, we do not see the same result for identifying visually-relevant claims  $(< 4 - 5\%)$ . We also experiment with the combination of BERT features with CLIP's image features, which improves the overall accuracy further but indicates that the model relies strongly on text (65.8 vs. 57.7 visual retrieval %) rather than the combination. The stronger reliance on text is possibly not a trait of the model alone, but could be also caused by an incompatibility of BERT and  $\mathrm{CLIP}_I$  features.
Finally, we achieve the best performance (by  $1 - 4\%$ ) on binary and tertiary (when trained on  $T$ ) claim detection by fine-tuning the ALBEF with and without OCR, respectively (Table 3, block 3,
Figure 2: Qualitative examples where our best multimodal model classifies correctly and unimodal models do not.  $F$  - false classification,  $T$  - true classification.
last row). While the benefit of using OCR text in SVM models is not optimal and not considerably helpful, OCR addition to ALBEF retrieves the maximum number of visually-relevant claims (71.2%) without losing much on visually-irrelevant claims (79.3%) when trained on  $T$  (Table 4, block 2, last row). These results point towards a major challenge of combining multiple modalities and retaining intra-modal information (and influence) for the task at hand. As noted in section 4.3.1, an interesting result is that ALBEF in particular is less robust to resolved conflicts (split  $T_{C}$ ) in the data when compared to just using BERT. On closer inspection, these conflicts are mostly caused by the image relevance to the text. The gap is further exaggerated in Table 6, where ALBEF performs much better than BERT, when conflict examples are removed from both training and evaluation. Figure 2 shows a few examples where our best multimodal model correctly classifies, whereas unimodal models based on either image or text do not. All the samples in the figure have images that have some connection to the tweet text. The image in Figure 2b has a connection to one of the words or phrases (e.g., washing your hands) in the tweet text but is not relevant for the claim itself. Figure 2a includes an image with the claim itself and a very generic scene in the background. Both image and text in Figure 2c and Figure 2d are relevant, and the image acts as evidence and additional information. In all these examples, a rich set of information extraction and complex cross-modal learning is required to identify claims in multimodal tweets. When comparing results of recent state-of-the-art architectures for fake news detection, SpotFake (Singhal et al., 2019) does considerably better than MVAE (Khattar et al., 2019) but worse than any of our baseline models.
In this paper, we have presented a novel MM-Claims dataset to foster research on multimodal claim analysis. The dataset has been curated from Twitter data and contains more than 3000 manually annotated tweets for three tasks related to claim detection across three topics, COVID-19, Climate Change, and Technology. We have evaluated several baseline approaches and compared them against two state-of-the-art fake news detection approaches. Our experimental results suggest that the fine-tuning of pre-trained multimodal and unimodal architectures such as ALBEF and BERT yield the best performance. We also observed that the overlaid text in images is important in information dissemination, particularly for claim detection. To this end, we evaluated a couple of strategies to incorporate OCR text into our models, which yielded a much better trade-off between identifying visually-relevant and visually-irrelevant (text-only) claims.
In the future, we will explore other and novel architectures for multimodal representation learning and other information extraction techniques to incorporate individual modalities better. We also plan to investigate fine-grained overlaps of concepts and meaning in image and text, and expand the dataset to COVID-19 related sub-topics and specific climate change events.
This work was funded by European Union's Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement no 812997 (CLEOPATRA project), and by the German Federal Ministry of Education and Research (BMBF, FakeNarratives project, no. 16KIS1517).
Oluwaseun Ajao, Deepayan Bhowmik, and Shahrzad Zargari. 2018. Fake news identification on twitter with hybrid cnn and rnn models. In Proceedings of the 9th international conference on social media and society, pages 226-230.
Firoj Alam, Shaden Shaar, Alex Nikolov, Hamdy Mubarak, Giovanni Da San Martino, Ahmed Abdelali, Fahim Dalvi, Nadir Durrani, Hassan Sajjad, Kareem Darwish, and Preslav Nakov. 2020. Fighting the COVID-19 infodemic: Modeling the perspective of journalists, fact-checkers, social media platforms, policy makers, and the society. CoRR, abs/2005.00033.
Juan M. Banda, Ramya Tekumalla, Guanyu Wang, Jingyuan Yu, Tuo Liu, Yuning Ding, and Gerardo Chowell. 2020. A large-scale COVID-19 twitter chatter dataset for open scientific research - an international collaboration. CoRR, abs/2004.03688.
Alberto Barrón-Cedeno, Tamer Elsayed, Preslav Nakov, Giovanni Da San Martino, Maram Hasanain, Reem Suwaileh, Fatima Haouari, Nikolay Babulkov, Bayan Hamdan, Alex Nikolov, et al. 2020. Overview of checkthat! 2020: Automatic identification and verification of claims in social media. In International Conference of the Cross-Language Evaluation Forum for European Languages, pages 215-236. Springer.
Christos Baziotis, Nikos Pelekis, and Christos Doulkeridis. 2017. Datastories at semeval-2017 task 4: Deep LSTM with attention for message-level and topic-based sentiment analysis. In Proceedings of the 10th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2016, San Diego, CA, USA, June 16-17, 2016, pages 747-754. The Association for Computer Linguistics.
Christina Boididou, Symeon Papadopoulos, Duc-Tien Dang-Nguyen, Giulia Boato, Michael Riegler, Stuart E. Middleton, Andreas Petlund, and Yiannis Kompatsiaris. 2016. Verifying multimedia use at mediaeval 2016. In Working Notes Proceedings of the MediaEval 2016 Workshop, Hilversum, The Netherlands, October 20-21, 2016, volume 1739 of CEUR Workshop Proceedings. CEUR-WS.org.
Tuhin Chakrabarty, Christopher Hidey, and Kathy McKeown. 2019. IMHO fine-tuning improves claim detection. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 558-563. Association for Computational Linguistics.
Gullal S. Cheema, Sherzod Hakimov, and Ralph Ewerth. 2020a. Check_square at checkthat! 2020 claim detection in social media via fusion of transformer and syntactic features. In Working Notes of CLEF 2020 - Conference and Labs of the Evaluation Forum, Thessaloniki, Greece, volume 2696 of CEUR Workshop Proceedings. CEUR-WS.org.
Gullal S Cheema, Sherzod Hakimov, and Ralph Ewerth. 2020b. Tib's visual analytics group at mediaeval'20: Detecting fake news on corona virus and  $5\mathrm{g}$  conspiracy. MediaEval 2020 Workshop.
Gullal S. Cheema, Sherzod Hakimov, Eric Muller-Budack, and Ralph Ewerth. 2021. On the role of images for analyzing claims in social media. In Proceedings of the 2nd International Workshop on Crosslingual Event-centric Open Analytics co-located with the 30th The Web Conference (WWW 2021), Ljubljana, Slovenia, April 12, 2021 (online event due to COVID-19 outbreak), volume 2829 of CEUR Workshop Proceedings, pages 32-46. CEUR-WS.org.
Matteo Cinelli, Walter Quattrociocchi, Alessandro Galeazzi, Carlo Michele Valensise, Emanuele Brugnoli, Ana Lucia Schmidt, Paola Zola, Fabiana Zollo, and Antonio Scala. 2020. The COVID-19 social media infodemic. Scientific Reports, 10(1):1-10.
Corinna Cortes and Vladimir Vapnik. 1995. Support-vector networks. Mach. Learn., 20(3):273-297.
Johannes Daxenberger, Steffen Eger, Ivan Habernal, Christian Stab, and Iryna Gurevych. 2017. What is the essence of a claim? cross-domain claim identification. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2055-2066. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, pages 4171-4186.
DGC. 2020. Un tackles 'infodemic' of misinformation and cybercrime in Covid-19 crisis.
Dimitar Dimitrov, Erdal Baran, Pavlos Fafalias, Ran Yu, Xiaofei Zhu, Matthaus Zloch, and Stefan Dietze. 2020. Tweetscov19 - A knowledge base of semantically annotated tweets about the COVID-19 pandemic. In CIKM '20: The 29th ACM International Conference on Information and Knowledge Management, Virtual Event, Ireland, October 19-23, 2020, pages 2991-2998. ACM.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Pavlos Fafalios, Vasileios Iosifidis, Eirini Ntoutsi, and Stefan Dietze. 2018. Tweetskb: A public and large-scale RDF corpus of annotated tweets. In The Semantic Web - 15th International Conference, ESWC 2018,
Heraklion, Crete, Greece, June 3-7, 2018, Proceedings, volume 10843 of Lecture Notes in Computer Science, pages 177-190. Springer.
Fayez. 2021. A simple, pillow-friendly, wrapper around the tesseract-ocr api for optical character recognition (ocr).
Anastasia Giachanou, Guobiao Zhang, and Paolo Rosso. 2020. Multimodal multi-image fake news detection. In 2020 IEEE 7th International Conference on Data Science and Advanced Analytics (DSAA), pages 647-654. IEEE.
Shreya Gupta, Parantak Singh, Megha Sundriyal, Md Shad Akhtar, and Tanmoy Chakraborty. 2021. Lesa: Linguistic encapsulation and semantic amalgamation based generalised claim detection from online content. arXiv preprint arXiv:2101.11891.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, pages 770-778. IEEE Computer Society.
Neslihan Iskender, Robin Schaefer, Tim Polzehl, and Sebastian Möller. 2021. Argument mining in tweets: Comparing crowd and expert annotations for automated claim and evidence detection. In *Natural Language Processing and Information Systems - 26th International Conference on Applications of Natural Language to Information Systems*, NLDB 2021, Saarbrücken, Germany, June 23-25, 2021, Proceedings, volume 12801 of Lecture Notes in Computer Science, pages 275-288. Springer.
Zhiwei Jin, Juan Cao, Han Guo, Yongdong Zhang, and Jiebo Luo. 2017. Multimodal fusion with recurrent neural networks for rumor detection on microblogs. In Proceedings of the 25th ACM international conference on Multimedia, pages 795-816.
Sarthak Jindal, Raghav Sood, Richa Singh, Mayank Vatsa, and Tanmoy Chakraborty. 2020. Newsbag: A benchmark multimodal dataset for fake news detection. In Proceedings of the Workshop on Artificial Intelligence Safety, co-located with 34th AAAI Conference on Artificial Intelligence, SafeAI@AAAI 2020, New York City, NY, USA, February 7, 2020, volume 2560 of CEUR Workshop Proceedings, pages 138-145. CEUR-WS.org.
Dhruv Khattar, Jaipal Singh Goud, Manish Gupta, and Vasudeva Varma. 2019. MVAE: multimodal variational autoencoder for fake news detection. In *The World Wide Web Conference*, WWW 2019, San Francisco, CA, USA, pages 2915–2921. ACM.
Hannah Rose Kirk, Yennie Jun, Paulius Rauba, Gal Wachtel, Ruining Li, Xingjian Bai, Noah Broestl, Martin Doff-Sotta, Aleksandar Shtedritski, and Yuki Markus Asano. 2021. Memes in the wild: Assessing the generalizability of the hateful memes challenge dataset. CoRR, abs/2107.04313.