SlowGuess/NAACL_2022_OCR huggingface dataset:
Klaus Krippendorff. 2011. Computing krippendorff's alpha-reliability.
Rabindra Lamsal. 2020. Coronavirus (covid-19) tweets dataset.
Ran Levy, Yonatan Bilu, Daniel Hershcovich, Ehud Aharoni, and Noam Slonim. 2014. Context dependent claim detection. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 1489-1500. Dublin City University and Association for Computational Linguistics.
Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Deepak Gotmare, Shafiq R. Joty, Caiming Xiong, and Steven C. H. Hoi. 2021. Align before fuse: Vision and language representation learning with momentum distillation. CoRR, abs/2107.07651.
Marco Lippi and Paolo Torroni. 2015. Context-independent claim detection for argument mining. In Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, IJCAI 2015, Buenos Aires, Argentina, pages 185-191. AAAI Press.
Justin Littman and Laura Wrubel. 2019. Climate Change Tweets Ids.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Kai Nakamura, Sharon Levy, and William Yang Wang. 2020. Fakeddit: A new multimodal benchmark dataset for fine-grained fake news detection. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 6149-6157. European Language Resources Association.
Preslav Nakov, Giovanni Da San Martino, Tamer Elsayed, Alberto Barrón-Cedeno, Rubén Miguez, Shaden Shaar, Firoj Alam, Fatima Haouari, Maram Hasanain, Nikolay Babulkov, Alex Nikolov, Gautam Kishore Shahi, Julia Maria Struß, and Thomas Mandl. 2021. The CLEF-2021 checkthat! lab on detecting check-worthy claims, previously fact-checked claims, and fake news. In Advances in Information Retrieval - 43rd European Conference on IR Research, ECIR 2021, Virtual Event, March 28 - April 1, 2021, Proceedings, Part II, volume 12657 of Lecture Notes in Computer Science, pages 639-649. Springer.
Letitia Parcalabescu, Nils Trost, and Anette Frank. 2021. What is multimodality? In Proceedings of the 1st Workshop on Multimodal Semantic Representations (MMSR).
Gordon Pennycook, Jonathon McPhetres, Yunhao Zhang, Jackson G Lu, and David G Rand. 2020. Fighting COVID-19 misinformation on social media: Experimental evidence for a scalable accuracy-nudge intervention. Psychological Science, 31(7):770-780.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 8748-8763. PMLR.
Sara Rosenthal and Kathleen R. McKeown. 2012. Detecting opinionated claims in online discussions. In Sixth IEEE International Conference on Semantic Computing, ICSC 2012, Palermo, Italy, September 19-21, 2012, pages 30-37. IEEE Computer Society.
Alessandro Rovetta and Akshaya Srikanth Bhagavathula. 2020. Covid-19-related web search behaviors and infodemic attitudes in italy: Infodemiological study. *JMIR public health and surveillance*, 6(2):e19374.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. 2015. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-252.
Celia Andreu Sánchez and Miguel Angel Martín Pascual. 2020. Fake images of the sars-cov-2 coronavirus in the communication of information at the beginning of the first Covid-19 pandemic. *El profes-sional de la información*, 29(3):6.
Shivangi Singhal, Rajiv Ratn Shah, Tanmoy Chakraborty, Ponnurangam Kumaraguru, and Shin'ichi Satoh. 2019. Spotfake: A multi-modal framework for fake news detection. In Fifth IEEE International Conference on Multimedia Big Data, BigMM 2019, Singapore, September 11-13, 2019, pages 39-47. IEEE.
Yaqing Wang, Fenglong Ma, Zhiwei Jin, Ye Yuan, Guangxu Xun, Kishlay Jha, Lu Su, and Jing Gao. 2018. EANN: event adversarial neural networks for multi-modal fake news detection. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2018, London, UK, pages 849-857. ACM.
Youze Wang, Shengsheng Qian, Jun Hu, Quan Fang, and Changsheng Xu. 2020. Fake news detection via knowledge-driven multimodal graph convolutional networks. In Proceedings of the 2020 International Conference on Multimedia Retrieval, pages 540-547.
Amelie Wuhrl and Roman Klinger. 2021. Claim detection in biomedical twitter posts. In Proceedings of the 20th Workshop on Biomedical Language Processing, BioNLP@NAACL-HLT 2021, Online, June 11, 2021, pages 131-142. Association for Computational Linguistics.
Dimitrina Zlatkova, Preslav Nakov, and Ivan Koychev. 2019. Fact-checking meets faustography: Verifying claims about images. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2099-2108. Association for Computational Linguistics.
In the following we include additional hyperparameter (A.1) details and experimental results (A.2), additional dataset and annotation process details (A.3), and some annotated tweets for multimodal claim detection (A.4).
A.1 Other Hyper-parameter Details
For fine-tuning  $BERT$  and  $ALBEF$ , we use a batch size of 16 and 8 (size constraints), respectively. We train the models for five epochs and use the best performing model (in terms of accuracy on the validation set) for evaluation. For  $BERT$ , a dropout with the ratio of 0.2 is applied before the classification head. Further, we use AdamW (Loshchilov and Hutter, 2019) as the optimizer with a learning rate of  $3e - 5$  and a linear warmup schedule. The learning rate is first linearly increased from 0 to  $3e - 5$  for iterations in the first epoch and then linearly decreased to 0 for the rest of the iterations in 4 epochs. For  $ALBEF$ , we use the recommended fine-tuning hyper-parameters and settings from the publicly available code.
A.2 Additional Experimental Results
A.2.1 CLIP Variants
We experiment with CLIP's three variants that use different visual encoder backbones, ResNet-50 (RN50), ResNet-50x4 (RN504) and a vision transformer (ViT-B/16) (Dosovitskiy et al., 2021) with BERT as textual encoder backbone. We select the models for textual and multimodal SVM experiments based on the performance (higher accuracy) using features from the visual encoders. Table 5 shows different visual encoders' features (with SVM) performance on binary and tertiary claim detection.
It should be noted that just like ALBEF, CLIP models can be fine-tuned with image-text tweet pairs for binary and tertiary tasks. However, when we experimented with fine-tuning the last few layers of CLIP with a classification head on top, it always performed worse than using extracted features for classification with SVM. This phenomenon is probably because of our relatively smaller sized labeled dataset, which is not enough for fine-tuning CLIP for the task.
Task →	BCD	TCD
Data Splits →	TC → EC	T → EC	TC → EC
Models ↓	Acc	F1	Acc	F1	Acc	F1
RN50	66.3	65.7	64.1	50.6	62.4	48.7
RN50x4	70.0	69.9	61.5	51.5	61.4	48.5
ViT-B/16	68.6	68.4	64.3	49.8	59.7	48.3
Table 5: CLIP's different visual encoder backbones features' performance evaluation. Accuracy (Acc) and Macro-F1 (F1) for binary (BCD) and tertiary claim detection (TCD) in percent [%]. As described in Section 3.5, we use the Training Split  $(T)$  and Evaluation (Testing) Split  $(E)$  with resolved (index  $C$ ) and without (no index) conflicts.
A.2.2 Results for "without conflicts"  $(E)$  Evaluation Split
In Section 4, we show results for tertiary claim detection (TCD) on evaluation splits "with resolved conflicts"  $(E_C)$  by training on  $T$  and  $T_{C}$ . Here in Table 6, we show the evaluation on "without conflicts" evaluation split  $(E)$ . As with evaluation on  $E_{C}$ , multimodal models are more sensitive to training on  $T_{C}$  where conflict resolution strategy causes the accuracy to drop for all models. However, CLIP and ALBEF models, in this case, have higher F1-score (as well as accuracy) when trained on  $T$ . Even with less training data, the models perform better and best among all evaluated multimodal models. In the case of training on  $T_{C}$ , BERT performs the best, which is closely followed by ALBEF with OCR text.
As described in section 4.3.1, the evaluation of retrieved visually-relevant and visually-irrelevant claims on  $E$  follows the evaluation on  $E_{C}$ . Even though  $CLIP_{I}$  and fine-tuned BERT retrieves the most amount of two types of claims, all models do better when trained on  $T_{C}$  than on  $T$ .
Overall, for a realistic scenario, training on  $T_{C}$  gives the best performance trade-off between Acc, F1 and retrieved claims for multimodal models.
A.2.3 Confusion Matrix
Following the results on  $E_{C}$  in section 4 for binary and tertiary tasks, we show normalized (by row) confusion matrices based on predictions from the ALBEF  $\oplus$  OCR  $\oplus$  FT model. Figure 3a is the confusion matrix on  $E_{C}$  for binary claim detection (BCD). Whereas, Figure 3b shows the matrices on  $E_{C}$  with training on  $T_{C}$  (b.1) and  $T$  (b.2). Although the not-claim's true positives remain the same, confusion for the not-check-worthy and check-worthy class is less severe when trained on  $T_{C}$ .
Task →	TCD
Data Splits →	T → E	TC → E
Models ↓	Acc	F1	Acc	F1
Random	33.7	28.2	33.7	28.2
Majority	62.7	38.5	62.7	38.5
ImageNet	62.5	40.9	62.5	42.1
CLIP1	68.9	50.2	67.2	48.7
BERT	77.9	52.9	72.8	56.9
← FT	78.3	51.2	79.2	61.4
CLIPT	77.3	54.4	71.6	52.3
BERT ⊕ ImageNet	77.5	56.0	77.0	56.9
← ⊕ OCR	77.7	55.0	76.6	55.8
CLIPI ⊕ T	77.5	56.4	73.0	52.6
CLIPI ⊕ BERT	77.9	53.3	72.6	56.8
ALBEF	76.6	55.0	67.6	52.7
← FT	80.0	63.3	76.8	59.7
← ⊕ OCR ⊕ FT	78.7	63.5	77.5	59.9
MVAE	64.8	40.7	62.9	43.2
SpotFake	72.8	49.7	70.7	50.4
Table 6: Accuracy (Acc) and Macro-F1 (F1) for tertiary claim detection (TCD) in percent [%]. As described in Section 3.5, we use the Training Split  $(T)$  and Evaluation (Testing) Split  $(E)$  with resolved (index  $C$ ) and without (no index) conflicts. Additional results on evaluation split without conflicts  $(E)$ . Unless FT (fine-tuning) is written, all models (except MVAE and SpotFake) are SVM models trained on extracted features.
Data Splits →	T → E	TC → E
Models ↓	V (76)	T (120)	V (76)	T (120)
ImageNet	39.8	39.2	67.1	58.3
CLIPI	72.4	69.2	78.9	76.7
BERT←FT	52.6	80.0	61.8	85.0
53.9	79.2	57.9	85.8
CLIPT	51.3	76.7	60.5	76.7
BERT ⊕ ImageNet← ⊕ OCR	63.2	68.3	75.0	80.8
69.7	78.3	75.0	81.7
CLIPI ⊕ T	68.4	70.0	76.3	78.3
CLIPI ⊕ BERT	60.5	75.0	60.5	85.0
ALBEF← FT	63.2	77.5	65.8	80.8
65.8	79.2	75.0	80.8
← ⊕ OCR ⊕ FT	76.3	82.5	77.6	85.0
Table 7: Visually-relevant (V) and visually-irrelevant (T) claim detection evaluation. The amount of test samples is reported in brackets and the fraction, how many of them were retrieved, is given in percent  $[\%]$ . Additional results on evaluation split without conflicts  $(E)$ . The underlying models are trained for binary claim detection (BCD). The labels for visual relevance are only used for retrieval evaluation.
A.2.4 Ablation on OCR length
The amount of text that can be detected from an image varies, as it can be seen in Figure 8. As a consequence, we experimented with the length of OCR text in terms of the number of words for both binary and tertiary claim detection with ALBEF. We observe (see Figure 5) that 128 words give
a) Binary Claim Detection
b.1) With Conflicts  $(\mathsf{T}_{\mathbb{C}}\to \mathsf{E}_{\mathbb{C}})$
b) Tertiary Claim Detection
Figure 3: Normalized (by row) Confusion Matrices for the Binary and Tertiary Claim Classification Tasks. NC: Not-Claim, NCW: Not-check-worthy-Claim, C: Claim, CW: check-worthy-Claim
b.2) Without Conflicts  $(\mathrm{T}\rightarrow \mathrm{E}_{\mathrm{C}})$
comparable or better performance than any less number of words in OCR text across tasks and number of layers fine-tuned. We chose 128 words instead of 64 because the model with 128 words showed a balanced performance for binary, tertiary and retrieved claims. Models with 64 or greater than 128 words had a lower performance for either visually-relevant or irrelevant retrieved claims.
Figure 4: Ablation experiment on number of layers fin-tuned in BERT and ALBEF
A.2.5 Ablation on number of layers trained
We ran ablation experiments to see the effect of training the last few layers of BERT and ALBEF  $\oplus$  OCR. We experiment with fine-tuning the last six, four, two layers and only the last layer of each model. The results are shown in Figure 4. Overall, fine-tuning the last two and four layers of BERT and ALBEF respectively gives the best
results. Therefore, all the fine-tuning results for BERT, ALBEF and ALBEF  $\oplus$  OCR are based on the above observation. For fine-tuning six or more layers, the unlabeled dataset can be incorporated in the future as a pre-training step followed by task-specific training.
Figure 5: Ablation experiment on OCR text length (number of words) in ALBEF
A.3 Additional Dataset and Annotation Details
A.3.1 Claim Definition
Factually-verifiable Claims: should ideally have some of the following information (extended from Barrón-Cedeno et al. (2020)):
- reference to who, where, when, what, etc
- a definition, procedure, law or a process
- numbers or quantities in the tweet, e.g. sums of money, number of cases or deaths
- verifiable predictions
- refers to people, events, (event) locations
- refers to images and videos in the tweet
- personal opinions with claims that have factually-verifiable information
Check-worthy Claims: We follow a similar definition as Barrón-Cedeno et al. (2020), where claims are check-worthy if the information has some of the following properties:
- Harmful: if the statement attacks a person, organization, country, group, race, community, etc. The intention of such statements can be to spread rumours about an individual or a group, which should be checked by a professional or flagged and prioritized for further checking.
Figure 6: Class distributions in the annotated dataset ("with resolved conflicts") across different topics
- Urgent or breaking news: such statements are news-like where the claim is about prominent people (public personality like politicians, celebrities), organizations, countries and events (like disease outbreaks, forest fires, stock market crash).
- Up-to-date: such claims often refer to official documents and contain parts of clauses in climate agreements or articles in a constitution. This information is vital for checking, as many people consume social media as means of news, information and believe it to be true.
A.3.2 Filtering Strategies
The following Table 8 shows number of samples after each filtering step. The duplicate removal is performed across all the data irrespective of the topic in order to avoid duplicates that might fall into more than one topic.
Filtering Strategy	COVID	Climate	Tech.
No Filter	214 715	28 374	417 403
Empty text	214 715	28 374	417 403
Duplicate removal	28 522	11 333	383 043
Tweets with no image	28 522	11 333	383 043
Text not in English	28 148	11 274	377 532
Image size (200x200)	27 572	10 895	369 735
Hashtags > 6	26 786	10 013	287 242
Top-300.Hashtags	17 771	4874	62 887
Table 8: Data corpus statistics after applying different filtering strategies (in order).
A.3.3 Class Distributions Across Topics
In Figure 6, we provided the topic and class distributions in the labeled dataset.
A.3.4 Split-wise Statistics
The following Table 9 shows split-wise distribution of topics and labels in data. Numbers in red and black are for "with resolved conflicts" and "without conflicts" splits, respectively.
Types of La-bels	COVID	Climate	Tech
Not Claims	306/34/73	449/38/120	617/81/136
306/34/73	449/38/120	617/81/136
Claims	545/64/123	351/35/70	265/30/63
478/58/104	251/24/48	198/21/44
Not check-worthy	77/8/16	238/27/23	155/24/24
25/4/3	141/16/5	97/9/8
check-worthy	468/56/107	113/8/47	110/6/39
453/54/101	110/8/43	101/12/36
Not Visual	302/31/78	112/8/33	125/15/34
285/30/70	91/10/21	104/10/29
Visual	243/33/45	239/27/37	140/15/29
193/28/34	160/14/27	94/11/15
Total	851/98/196	800/73/190	882/111/199
784/92/177	700/62/168	815/102/180
Table 9: Labeled data characteristics in terms of type of labels and topic. Shown as Training/Validation/Evaluation splits. Second and third blocks are claims which are check-worthy (and not) and visual claims (and not) respectively. Red - "with resolved conflicts" and black - "without conflicts"
A.3.5 Relevant Hashtags
Although we crawl tweets from topic-based corpora, we further filter tweets by manually marking top 300 hashtags (sorted by occurrence) relevant to the topic. Figure 7 shows top-20 relevant hashtags for each topic.
A.3.6 Annotation Tool
Figure 7d shows the annotation screen with the image-text pair, claim questions and a text box for feedback on difficult and missing image tweets.
A.4 Annotated Samples from the MM-Claims Dataset
We included multiple annotated samples corresponding to visually-relevant claim (see Figure 8) and not a claim (see Figure 9) classes.
(a) COVID-19
(d) Graphical User Interface that is used to annotate image-text tweets
Figure 7: Top-20 manually selected hashtags for topic relevance filtering strategy.
(b) Climate Change
(c) Technology
Annotation GUI
Hi, gulla! You have annotated 0 / 120 documents. Logout
Document Selection
1033990256494829568: Apple buys rights to series based on New York Time
Apple buys rights to series based on New York Times climate change article, "Losing Earth"
https://t.co/aXmmlnOpiX via @nytimes https://t.co/xabna1rEQn
Document Annotation
Does the tweet-image pair contain a factually verifiable claim?
Does the claim contain harmful, urgent, up-to-date or news-worthy information?
Does the image contain information about the claim or the claim itself (in the overlay text)?
Feedback about the image-text pair?
5G-Heat waves artificially created by electromagnetic radiation-HAARP. 5G is a proven military weapon
Climate change has already hit home prices, led by Jersey Shore...
Source:First Street Foundation
Little fact about #coronavirus. I don't know how much it has affected your country but please be careful ...
Figure 8: Additional examples for visually relevant claims for the topics COVID-19 (bottom row), Climate Change (middle row), and Technology (top row).
Centenarians and supercentenarians have delayed vascular aging. As long as our brain doesn't melt, it seems prudent to maintain...
The Sunniest Climate Change Story YOU HAVE EVER READ
In 2014, the world's economy grew without carbon emissions also growing, something that had never happened before.
HERE'S HOW WE GOT THERE:
China coronavirus: tensions high as thousands queue in Hong Kong desperate for masks, many leaving empty-handed.
Figure 9: Additional examples that are not-claims for the topics COVID-19 (top row), Climate Change (bottom row), and Technology (middle row).
Powell just said "coronavirus."
At $99, Nvidia's Jetson Nano minicomputer seeks to bring robotics to the masses - Digital Trends ...
Has anyone heard about the coronavirus in Africa?
Journey of a Thousand Miles Begins with a Single Step! Basic Training in Canada! We've got great news! ...
Saami Culture Must Be Secured Through Sustainable Management in the Arctic