SlowGuess/NAACL_2022_OCR huggingface dataset:
Gullal S. Cheema<sup>1</sup>, Sherzod Hakimov<sup>1,3</sup>, Abdul Sittar<sup>2</sup>,
Eric Müller-Budack<sup>1,3</sup>, Christian Otto<sup>3</sup>, and Ralph Ewerth<sup>1,3</sup>
$^{1}$ TIB – Leibniz Information Centre for Science and Technology, Hannover, Germany
$^{2}$ Jozef Stefan Institute, Ljubljana, Slovenia
$^{3}$ L3S Research Center, Leibniz University Hannover, Germany
{gullal.cheema, sherzod.hakimov, eric.mueller}@tib.eu
{christian.otto, ralph ewerth}@tib.eu
abdul.sittar@ijs.si
In recent years, the problem of misinformation on the web has become widespread across languages, countries, and various social media platforms. Although there has been much work on automated fake news detection, the role of images and their variety are not well explored. In this paper, we investigate the roles of image and text at an earlier stage of the fake news detection pipeline, called claim detection. For this purpose, we introduce a novel dataset, MM-Claims, which consists of tweets and corresponding images over three topics: COVID-19, Climate Change and broadly Technology. The dataset contains roughly 86000 tweets, out of which 3400 are labeled manually by multiple annotators for the training and evaluation of multimodal models. We describe the dataset in detail, evaluate strong unimodal and multimodal baselines, and analyze the potential and drawbacks of current models.
The importance of combating misinformation was once again illustrated by the coronavirus pandemic, which came along with a lot of "potentially lethal" misinformation. At the beginning of the COVID-19 pandemic, the United Nations (UN) (DGC, 2020) started even using the term "infodemic" for this phenomenon of misinformation and called for proper dissemination of reliable facts. However, tackling misinformation online and specifically on social media platforms is challenging due to the variety of information, volume, and speed of streaming data. As a consequence, several studies have explored different aspects of COVID-19 misinformation online including sharing patterns (Pennycook et al., 2020), platform-dependent engagement patterns (Cinelli et al., 2020), web search behaviors (Rovetta and Bhagavathula, 2020), and fake images (Sánchez and Pascual, 2020).
We are primarily interested in claims on social media from a multimodal perspective (Figure 1).
Breathtaking Photos Capture Loss and Hope in the Age of Climate Change
a) Not a claim
Worst yet to come? Experts say, 'Kerala rains match climate change forecasts'
b) Claim but not checkworthy
c) Checkworthy claim
CDC tells travelers to avoid China in expanded travel warning as coronavirus spreads
The world remains far off course to meet the Paris climate goals of  $2^{\circ}\mathrm{C}$  warming and striving to reach a rise of just  $1.5^{\circ}\mathrm{C}$
Figure 1: Examples for each of the four classes in the MM-Claims dataset: a) not a claim (both image and text together abstractly represent effects of climate change), b) claim but not checkworthy (claim in text, but lacks details like to which experts is referred to, while image is relevant), c) checkworthy but not visually relevant (claim in text targets CDC and China but the image is a stock photograph), and d) checkworthy and visually relevant (claim in text and in image with important details in both).
Claim detection can be seen as an initial step in fighting misinformation and as a precursor to prioritize potentially false information for fact-checking. Traditionally, claim detection is studied from a linguistic standpoint where both syntax (Rosenthal and McKeown, 2012) and semantics (Levy et al., 2014) of the language matter to detect a claim accurately. However, claims or fake news on social media are not bound to just one modality and become a complex problem with additional modalities like images and videos. While it is clear that a claim in the text is denoted in verbal form, it can also be part of the visual content or as overlaid text in the
Findings of the Association for Computational Linguistics: NAACL 2022, pages 962 - 979
image. Even though much effort has been spent on the curation of datasets (Boididou et al., 2016; Nakamura et al., 2020; Jindal et al., 2020) and the development of computational models for multimodal fake news detection on social media (Ajao et al., 2018; Wang et al., 2018; Khattar et al., 2019; Singhal et al., 2019), hardly any research has focused on multimodal claims (Zlatkova et al., 2019; Cheema et al., 2020b).
In this paper, we extend the definitions of claims and check-worthiness from previous work (Barrón-Cedeno et al., 2020; Nakov et al., 2021) to multimodal claim detection and introduce a novel dataset called Multimodal Claims (MM-Claims) curated from Twitter to tackle this critical problem. While previous work has focused on factually-verifiable check-worthy (Barrón-Cedeno et al., 2020; Alam et al., 2020) or general claims (i.e., not necessarily factually-verifiable, e.g., (Gupta et al., 2021)) on a single topic, we focus on three different topics, namely COVID-19, Climate Change and Technology. As shown in Figure 1, MM-Claims aims to differentiate between tweets without claims (Figure 1a) as well as tweets with claims of different types: claim but not check-worthy (Figure 1b), check-worthy claim (Figure 1c), and check-worthy visually relevant claim (Figure 1d). Our contributions can be summarized as follows:
- a novel dataset for multimodal claim detection in social media with more than 3000 manually annotated and roughly 82000 unlabeled image-text tweets is introduced;
- we present details about the dataset and the annotation process, class definitions, dataset characteristics, and inter-coder agreement;
- we provide a detailed experimental evaluation of strong unimodal and multimodal models highlighting the difficulty of the task as well as the role of image and text content.
The remainder of the paper is structured as follows. Section 2 describes the related work on unimodal and multimodal approaches for claim detection. The proposed dataset and the annotation guidelines are presented in Section 3. We discuss the experimental results of the compared models in Section 4, while Section 5 concludes the paper and outlines areas of future work.
2.1 Text-based Approaches
Before research on claim detection targeted social media, pioneering work by Rosenthal and McKeown (2012) focused on claims in Wikipedia discussion forums. They used lexical and syntactic features in addition to sentiment and other statistical features over text. Since then, researchers have proposed context-dependent (Levy et al., 2014), context-independent (Lippi and Torroni, 2015), cross-domain (Daxenberger et al., 2017), and in-domain approaches for claim detection. Recently, transformer-based models (Chakrabarty et al., 2019) have replaced structure-based claim detection approaches due to their success in several Natural Language Processing (NLP) downstream tasks. A series of workshops (Barrón-Cedeno et al., 2020; Nakov et al., 2021) focused on claim detection and verification on Twitter and organized challenges with several sub-tasks on text-based claim detection around the topic of COVID-19 in multiple languages. Gupta et al. (2021) addressed the limitations of current methods in cross-domain claim detection by proposing a new dataset of about  $\sim 10000$  claims on COVID-19. They also proposed a model that combines transformer features with learnable syntactic feature embeddings. Another dataset introduced by Iskender et al. (2021) includes tweets in German about climate change for claim and evidence detection. Wuhrl and Klinger (2021) created a dataset for biomedical Twitter claims related to COVID-19, measles, cystic fibrosis and depression. One common theme and challenge among all the datasets is the variety of claims where some types of claims (like implicit) are harder to detect than explicit ones where a typical claim structure is present. Table 1 shows a comparison of existing social media based claim datasets, with number of samples, modalities, data sources, language, topic, and type of tasks.
2.2 Multimodal Approaches
From the multimodal perspective, very few works have analyzed the role of images in the context of claims. Zlatkova et al. (2019) introduced a dataset that consists of claims and is created from the idea of investigating questionable or outright false images which supplement fake news or claims. The authors used reverse image search and several image metadata features such as tags from Google Vision API, URL domains and categories, relia
Datasets	#Samples	Modality	Data source	Language	Topic	Task(s)
Zlatkova et al. (2019)*	1233	Image, Text	Snomes, Reuters	English	Multi-topic	True vs False
Nakov et al. (2021)	18, 014†	Text	Twitter	Multi	Multi-topic†	check-worthiness estimation
Gupta et al. (2021)	9981	Text	Twitter	English	COVID-19	claim detection
Iskender et al. (2021)	300 pairs	Text	Twitter	German	Climate change	claim, evidence detection
Wührl and Klinger (2021)	1200	Text	Twitter	English	Biomedical & COVID-19	claim & claim type detection
MM-Claims (Ours)	3400	Image, Text	Twitter	English	COVID-19, Climate Change, Technology	claim, check-worthiness, visual relevance
Table 1: Comparison of social media based claim datasets. *Zlatkova et al. (2019) is a mix of actual news photographs (from Reuters) and possibly fake images (from Snopes), which went viral on social media sites like Reddit.† 1312 samples are in English and only on the topic of COVID-19.
bility of the image source, etc. Similarly, Wang et al. (2020) performed a large-scale study by analyzing manipulated or misleading images in news discussions on forums like Reddit, 4chan and Twitter. For claim detection, Cheema et al. (2021) extended the text-based claim detection datasets of Barrón-Cedeno et al. (2020) and Gupta et al. (2021) with images to evaluate multimodal detection approaches. Although previous work has provided multimodal datasets on claims, they are either on veracity (true or false) of claims or labeled only text-based for a single topic (COVID-19). In terms of multimodal models for image-text data, most previous work is in the related area of multimodal fake news, where several benchmark datasets and models exist for fake news detection (Nakamura et al., 2020; Boididou et al., 2016; Jindal et al., 2020). In an early work, Jin et al. (2017) explored rumor detection on Twitter using text, social context (emoticons, URLs, hashtags), and the image by learning a joint representation in a deep recurrent neural network. Since then, several improvements have been proposed, such as multi-task learning with an event discriminator (Wang et al., 2018), multimodal variational autoencoder (Khattar et al., 2019) and multimodal transfer learning using transformers for text and image (Giachanou et al., 2020; Singhal et al., 2019).
3 MM-Claims Tasks and Dataset
This section describes the problem of multimodal claim detection (Section 3.1), the data collection (Section 3.2), the guidelines for annotating multimodal claims (Section 3.3), and the annotation process (Section 3.4) to obtain the new dataset.
Given a tweet with a corresponding image, the task is to identify important factually-verifiable or check-worthy claims. In contrast to related work, we introduce a novel dataset for claim detection that is labeled based on both the tweet and the corresponding image, making the task truly multimodal. Our scope of claims is motivated by Alam et al. (2020) and Gupta et al. (2021), which have provided detailed annotation guidelines. We restrict our dataset to factually-verifiable claims (as in Alam et al. (2020)) since these are often the claims that need to be prioritized for fact-checking or verification to limit the spread of misinformation. On the other hand, we also include claims that are personal opinions, comments, or claims existing at sub-sentence or sub-clause level (as in Gupta et al. (2021)), with the condition that they are factually-verifiable. Subsequently, we extend the definition of claims to images along with factually-verifiable and check-worthy claims.
In previous work on claim detection in tweets, most of the publicly available English language datasets (Alam et al., 2020; Barrón-Cedeno et al., 2020; Gupta et al., 2021; Nakov et al., 2021) are text-based and on a single topic such as COVID-19, or U.S. 2016 Elections. To make the problem interesting and broader, we have collected tweets on three topics, COVID-19, Climate Change and broadly Technology, that might be of interest to a wider research community. Next, we describe the steps for crawling and preprocessing the data.
3.2.1 Data Crawling
We have used an existing collection of tweet IDs, where some are topic-specific Twitter dumps, and extracted tweet text and the corresponding image to create a novel multimodal dataset.
COVID-19: We combined tweets from three Twitter resources (Banda et al., 2020; Dimitrov et al., 2020; Lamsal, 2020) that were posted between October 2019 and April 2020. In our dataset, we use tweets in the period from March - April 2020.
Climate Change: We used a Twitter resource (Littman and Wrubel, 2019) that contains tweet IDs related to climate change from September 2017 to May 2019. The tweets were originally crawled based on hashtags like climatechange, climatechangeisreal, actonclimate, globalwarming, climatedeniers, climatechangeisfalse, etc.
Technology: For the broad topic of Technology, we used the TweetsKB (Fafalias et al., 2018) corpus. To avoid the extraction of all the tweets from 2019 to 2020 irrespective of the topic, we followed a two-step process to find tweets remotely related to technology. The corpus is available in form of RDF (Resource Description Framework) triples with attributes like tweet ID, hashtags, entities and emotion labels, but without tweet text or media content details. First, we selected tweet IDs based on hashtags and entities, and only kept those that contain keywords like technology, cryptocurrency, cybersecurity, machine learning, nano technology, artificial intelligence, IOT, 5G, robotics, blockchain, etc. The second step of filtering tweets based on a selected set of hashtags for each topic is described in the next subsection.
From the above resources, we collected 214715, 28374 and 417403 tweets for the topics COVID-19, Climate Change and Technology, respectively.
3.2.2 Data Filtering
We perform a number of filtering steps to remove inconsistent samples: 1) tweets that are not in English or without any text, 2) duplicated tweets based on tweet IDs, processed text and retweets, 3) tweets with corrupted or no images, 4) tweets with images of less than  $200 \times 200$  pixels resolution, 5) tweets that have more than six hashtags, and finally, 6) we make a list of the top 300 hashtags in each topic based on count and manually select those related to the selected topics. We only keep those tweets where all hashtags are in the list of top selected hashtags. The hashtags are manually marked because some top hashtags are not relevant to the
main topic of interest. The statistics of tweets after each filtering step are provided in the Appendix (see Table 8). In summary, we end up with 17 771, 4874, and 62 887 tweets with images for COVID-19, Climate Change and Technology, respectively.
In this section, we provide definitions for all investigated claim aspects, the questions asked to annotators, and the cues and explanations for the annotation questions. We define a claim as state or assert that something is the case, typically without providing evidence or proof using the definition in the Oxford dictionary (like Gupta et al. (2021)).
The definition of a factually-verifiable claim is restricted to claims that can possibly be verified using external sources. These external sources can be reliable websites, books, scientific reports, scientific publications, credible fact-checked news reports, reports from credible organizations like World Health Organization or United Nations. Although we did not provide external links of reliable sources for the content in the tweet, we highlighted named entities that pop-up with the text and image description. External sources are not important at this stage because we are only interested in marking claims, which have possibly incorrect details and information. A list of identifiable cues (extended from Barrón-Cedeno et al. (2020)) for factually-verifiable claims is provided in the Appendix A.3.1.
To define check-worthiness, we follow Barron-Cedeno et al. (2020) and identify claims as checkworthy if the information in the tweet is, 1) harmful (attacks a person, organization, country, group, race, community, etc), or 2) urgent or breaking news (news-like statements about prominent people, organizations, countries and events), or 3) up-to-date (referring to recent official document with facts, definitions and figures). A detailed description of these cases is provided in the Appendix A.3.1. Given these key points, the answer to whether the claim is check-worthy is subjective since it depends on the person's (annotator's) background and knowledge.
Annotation Questions: Based on the definitions above, we decided on the following annotation questions in order to identify factually-verifiable claims in multimodal data.
- Q1: Does the image-text pair contain a factually-verifiable claim? - Yes / No
• Q2: If “Yes” to Q1, Does the claim contain harmful, up-to-date, urgent or breaking-news information? - Yes / No
- Q3: If “Yes” to Q1, Does the image contain information about the claim or the claim itself (in the overlaid text)? - Yes / No
Question 3 (Q3) intends to identify whether the visual content contributes to a tweet having factually-verifiable claims. The question is answered "Yes" if one of the following cases hold true: 1) there exists a piece of evidence (e.g. an event, action, situation or a person's identity) or illustration of certain aspects in the claim text, or 2) the image contains overlay text that itself contains a claim in a text form. Please note that we asked the annotators to label tweets with respect to the time they were posted. During our annotation dry runs we observed that there were several false annotations for the tweets where the claims were false but already well known facts. This aspect intends to ignore the veracity of claims since some of the claims become facts over time. In addition, we ignore tweets that are questions and label them as not claims unless the corresponding image consists of a response to the question and is a factually-verifiable claim.
3.4 Annotation Process
Each annotator was asked to answer these questions by looking at both image and text in a given tweet. We distribute the data among nine external and four expert internal annotators for the annotation of training and evaluation splits, respectively. The nine annotators are graduate students with engineering or linguistics background. These annotators were paid 10 Euro per hour for their participation. The four expert annotators are doctoral and postdoctoral researchers of our group with a research focus on computer vision and multimodal analytics. Each annotator was shown a tweet text with its corresponding image and asked to answer the questions presented in Section 3.3. Exactly three annotators labeled each sample, and we used a majority vote to obtain the final label.
3.4.1 Claim Categories
We selected a total of 3400 tweets for manual annotation of training (annotated by external annotators) and evaluation (annotated by internal experts) splits. Each split contains an equal number of samples for the topics: COVID-19, Climate Change,
and Technology. Labels for three types of claim<sup>1</sup> annotations are derived:
- binary claim classes: not a claim, and claim
- tertiary claim classes: not a claim, claim but not check-worthy, and check-worthy claim
- visual claim classes: not a claim, visually-irrelevant claim, and visually-relevant claim
3.4.2 Annotator Training
The annotators were trained with detailed annotation guidelines, which included the definitions given in Section 3.3 and multiple examples. To ensure the quality, we performed two dry runs using a set of samples (30-40) to annotate. Afterwards, the annotations were discussed to check agreements among annotators and the guidelines were refined based on the feedback.
3.4.3 Inter-Annotator Agreement
We measured the agreements between two groups of annotators using Krippendorff's alpha (Krippendorff, 2011). The agreements were computed for the three types of annotations described in the previous section. For the training dataset group, we observe 0.53, 0.39, and 0.42 as agreement scores for the binary, tertiary, and visual claims, respectively. For the test dataset group, we observe the following agreement scores: 0.57, 0.47, and 0.52 for three classifications, respectively. The moderate agreement scores suggest that the problem of identifying check-worthy claims is partially a subjective task for both non-experts and experts.
3.4.4 Conflict Resolution Strategy
While a majority is always possible for the binary claim classification that allows us to derive unambiguous labels, entirely different labels could be chosen for the tertiary and visually-relevant claim classification task since the annotators assign three possible classes. Consequently, it is not possible to derive a label with majority voting when each annotator selects a different option. In such cases, we resolve the conflict by prioritizing the claim but not check-worthy class since check-worthiness is a stricter constraint and chosen by only one annotator, while two annotators agreed it is a claim. First row in Table 2 shows this case when two annotators indicated that the given sample is a claim (A-2 → Q1-Yes, A-3 → Q1-Yes). For visual claims, we select a visually-relevant claim since it is possible
Here claim is a factually-verifiable claim not any claim
A-1	A-2	A-3	Derived Class
Q1-No	Q1-Yes	Q1-Yes	Claim but not check-worthy
Q2-No	Q2-Yes
Q1-No	Q1-Yes	Q1-Yes	Visually relevant claim
Q3-No	Q3-Yes
Table 2: Conflict resolution strategies to derive class labels where a majority vote can not be reached among three annotators (A) for check-worthiness and visual relevance tasks.
that image and text are related, even when one annotator marked "no" to the claim question. See row two in the table, where one annotator marked "no" to the claim question (A-1  $\rightarrow$  Q1-No), but at least one annotator indicated that the sample is a visually-relevant claim (A-3  $\rightarrow$  Q3-Yes).
3.5 The MM-Claims Dataset
As a result of the annotation process, the Multimodal Claims (MM-Claims) dataset consists of 2815  $(T_{C}$  (training)) and 585  $(E_{C}$  (evaluation)) samples ( $C$  in the subscript stands for "with resolved conflicts"). However, as discussed above, there are conflicting examples for the tertiary and visual claim labels. To train and evaluate our models on unambiguous examples, we derive a subset of Multimodal Claim (MM-Claims) dataset that contains 2555 ( $T$ ) and 525 ( $E$ ) samples "without conflicts" where a majority vote can be taken. We divided the training set  $(T_{C}, T)$  in each case further into training and validation in a 90:10 split for hyper-parameter tuning.
We noticed that one-third of the images in the dataset contains a considerable amount of overlaid text (five or more words). As suggested by previous work (Cheema et al., 2021; Parcalabescu et al., 2021; Kirk et al., 2021), overlaid text in images should be considered in addition to tweet text and other image content. Specifically, the images with overlaid text not only act as related information to the tweet text but are sometimes the central message of the tweet. We used Tesseract-OCR (Fayez, 2021) to select images that contain five or more words in their overlay text. In an internal pre-test with 100 images, we observed that Tesseract-OCR produced more random (and incorrect) text from
images than Google Vision API. To reduce the incorrect text, we ran Google Vision API on the selected images (avoiding unnecessary costs) in the second step that resulted in a better quality OCR detected text. Besides the labeled dataset, we will also provide the images, tweet text, and the overlay text (extracted using OCR methods as described above) of the unlabeled portion of the dataset.