SlowGuess/NAACL_2022_OCR huggingface dataset:
Giscard Biamby *
Grace Luo *
{gbiamby, graceluo, trevordarrell, anna.rohrbach}@berkeley.edu
Detecting out-of-context media, such as "mis-captioned" images on Twitter, is a relevant problem, especially in domains of high public significance. In this work we aim to develop defenses against such misinformation for the topics of Climate Change, COVID-19, and Military Vehicles. We first present a large-scale multimodal dataset with over 884k tweets relevant to these topics. Next, we propose a detection method, based on the state-of-the-art CLIP model, that leverages automatically generated hard image-text mismatches. While this approach works well on our automatically constructed out-of-context tweets, we aim to validate its usefulness on data representative of the real world. Thus, we test it on a set of human-generated fakes created by mimicking in-the-wild misinformation. We achieve an  $11\%$  detection improvement in a high precision regime over a strong baseline. Finally, we share insights about our best model design and analyze the challenges of this emerging threat.
Out-of-context images are a popular form of misinformation where an image is miscaptioned to support a false claim (Fazio, 2020). Such image repurposing is extremely cheap yet can be as damaging as more sophisticated fake media. In this work we focus on domains important for society and national security, where implications of inexpensive yet effective misinformation can be immense.
Specifically, we analyze multimodal Twitter posts that are of significant public interest, related to topics of COVID-19, Climate Change and Military Vehicles. Our goal is to learn to categorize such image-text posts as pristine or falsified (out-of-context) by means of detecting semantic inconsistencies between images and text. To that end, we first collect a large-scale dataset of multimodal tweets, Twitter-COMMs, with
over 884k tweets. In our approach, we fuse input image and text embeddings generated by CLIP (Radford et al., 2021) via an elementwise product, and train a classifier to distinguish real tweets from automatically constructed random and hard mismatches. To validate this approach and demonstrate the usefulness of the Twitter-COMMs dataset, we report results on human-generated test data, created to mimic real-world misinformation. We discuss the results and model ablations, and provide additional insights into the challenges of this task. Our dataset is publicly available at: https://github.com/GiscardBiamby/Twitter-COMMs.
There exist a number of large-scale Twitter datasets concentrated on topics such as COVID-19 (Banda et al., 2021) or Climate Change (Littman and Wrubel, 2019). However, it remains difficult to collect labeled misinformation. Researchers have collected COVID-19 misconceptions on social media via manual annotation (Hossain et al., 2020) or by linking to fact checking articles (Patwa et al., 2021). Not only are these datasets small (a few thousand samples), but they focus on false claims rather than multimodal inconsistency. Here, we curate social media posts that are topical and multimodal, and we demonstrate an application to misinformation detection of human-generated fakes.
Recent work has developed approaches for multimodal fact checking, e.g., Jaiswal et al. (2017) and Müller-Budack et al. (2020), who query an external knowledge base. Similar to Luo et al. (2021) in the news domain, we use a large pretrained model that does not require an external reference set.
3 Twitter-COMMs Dataset
Here, we describe the data collection strategies behind Twitter-COMMs, which consists of mul
Human Language Technologies, pages 1530 - 1549
Table 1: Twitter-COMMs breakdown. "Collected" denotes all unique samples collected via the Twitter API. "Pristine" and "Falsified" denote all samples in our automatically generated Training set. To ensure the balanced Training set, we "repeat" Pristine samples such that there is an equal number of Pristine and Falsified samples.
Topic / Samples	Collected	Pristine	Falsified
Random	Hard
Climate Change	212,665	298,809	84,432	214,377
COVID-19	569,982	736,539	162,410	574,129
Military Vehicles	101,684	139,213	35,376	103,837
Cross Topic	-	59,735	59,735	-
Total	884,331	2,468,592
timodal tweets covering the topics of COVID-19, Climate Change, and Military Vehicles.
Data Collection: We collected data using Twitter API v2 $^{1}$  in three stages for COVID-19 and Climate Change, and two stages for Military Vehicles, refining the filters at each stage to acquire more relevant tweets. COVID-19 and Climate Change stages progressed from simple high level keywords towards more specific ones in stage two and tweets authored by news organizations in the final stage. For Military Vehicles the first stage used high level search terms such as "military", "aircraft", "tank", which resulted in noisy data, so the second stage used a large number of highly specific terms related to vehicle models. Full details can be found in Appendix A.1. We employed the following global filters for all topics: (1) language=English, (2) has at least one image, and (3) not a retweet.
In total, we have collected 884,331 tweets, each having at least one image (composed of  $24\%$  Climate Change,  $64.5\%$  COVID-19, and  $11.5\%$  Military Vehicles tweets), see Table 1. Tweets for Climate Change and Military Vehicles were collected starting from June 2016 and for COVID-19 starting from February 2020, all ending in September 2021.
Falsified Samples: In addition to the pristine samples, we automatically generate falsified samples where there is some inconsistency between image and text. We create random negatives (denoted as "Random") by selecting an image for a given caption at random. We also create hard negatives (denoted as "Hard") by retrieving the image of the sample with the greatest textual similarity for a given caption (following the "Semantics / CLIP
Text-Text" split from Luo et al. (2021)). We mainly generate mismatches within each topic (COVID-19, Climate Change, Military Vehicles), except for a small set of random mismatches across topics (denoted as "Cross Topic"). Our dataset is balanced with respect to labels, where half of the samples are pristine and half are falsified. Table 1 presents summary statistics for the training samples. We detail our development set and other data used for evaluation in the next section.
Qualitative Analysis: We present random examples from our training set in Figure 1. Overall, we see that the collected Twitter samples tend to be "on topic" and the amount of noise is low. Hard negatives are often visually grounded, while random negatives contain image/text pairs that are only weakly related, since they pertain to the same topic. The Climate Change hard negative depicts an image of flooded homes to represent "droughts, fires and floods" while the random negative depicts an image of cars relevant to climate but inconsistent with "polar bears". The COVID-19 hard negative uses an image of a Nigerian spokesman to depict news pertaining to "ECOWAS²" while the random one uses a stock photo of lab testing to represent Covid. These entity-level, rather than topic-level, alignments more closely resemble real-world out-of-context images that often reference and misrepresent visually depicted entities. Note the diversity of images and text in our training set, where there exist both natural images and info-graphics, and language varies from organizational announcements and news headlines to personal opinions.
Next, we discuss the data used for evaluation, present our approach and ablate various design choices, report results on our evaluation sets, and provide additional analysis of the task difficulty.
4.1 Evaluation Sets
We report results on three evaluation sets. (a) We validate our approach on samples synthetically generated using the same procedure as our training set (denoted Dev), where all topics and falsification methods are equally represented (i.e., the ratio of random vs. hard negatives is 50-50). We also evaluate on human-curated samples from the
<sup>1</sup>https://developer/twitter.com/en/docs/twitter-api/getting-started/about-twitter-api
2Economic Community of West African States
Did you miss the NPA2014_2020
#SMART renew #EUGreenWeek event
The Role of #Geothermal #Energy in the
Net Zero Transition? Catch up with the
session recording #climateaction
#climatechange #cleanenergy
#energytransition #renewables
#renewablenergy
smartrenew
The Role of Geothermal Energy
In the Net Zero Transition
Nick Timmons Letterkenny Institute of Technology nick.timmons@
Melinhard Ellasen Faroese Energy Department
Asmo Huusko Geological Survey of Finland
Annu Martinkaupi Geological Survey of Finland  
Bepar Asmundsson Energy Fund Iceland
Rajal Ramonasson Energy and Iceland Sigurdur Frrideleifsson Energy Agency Iceland
Military Vehicles
Leopard-2A5s of the 2nd Battalion, 1st Warsaw Armoured Brigade of the Polish Army during the evaluation training. #PolishArmy #Leopard2
Second-biggest single-day spike takes #Covid19 cases in India to over 1.1 lakh Read more:
Polar bears hurt by climate change are more likely to turn to a new food source — humans
CASCAdapt #ClimateModels
Droughts, fires and floods: How climate change will impact Europe The south of the Continent will be harder hit than the north, driving a wedge into one of the European Union's deepest fault lines politico PriestleyCentre piersforster
Pentagon Testing Chief Slams 'High Risk' Software Updates for F-35s: Program for Lockheed Martin's F-35 Lightning II stealth aircraft falling further behind schedule #WorldatWar #f35 #fighter #et #Lockheed
Great to welcome Gen Vote, CENTCOM Commander, to the RAF's Middle East HQ where he toured the RAF's latest transport aircraft, the #A400M, received an on the ground update and recognised #83EAW and #901EAW personnel for their efforts in supporting operations. RoyalAirForce
Figure 1: Twitter-COMMs examples of Pristine and Falsified (Random / Hard) samples by topic.
amitmalmviya according to world bank, in 2019 India's natural death rate was 7 in 1000. So it's possible that bodies were floating. During Covid, death rate cannot be less than natural death rate? Can it? Why are your ministers writing to Yogi on mismanagement of 02.
COVID test fee fixed at $50 for all ECOWAS nationals, says FG | TheCable #COVID19
DARPA Semantic Forensics (SemaFor) Program<sup>3</sup> derived from (b) news images and captions (denoted hNews) and (c) Twitter (denoted hTwitter). To generate this data, humans manually introduced inconsistencies to pristine image-caption pairs.<sup>4</sup> While hNews/hTwitter data is not real misinformation, it is in-the-wild w.r.t. our synthetic training data and much more representative of real-world human-generated misinformation. All three evaluation sets contain a mixture of samples relevant to the topics of COVID-19, Climate Change, and Military Vehicles (Figure 2). Table 2 provides the number of samples in each set. While the hNews
set is available to us, the hTwitter set is hidden.
Table 2: Evaluation samples breakdown.
Domain	Pristine	Falsified	Total
Dev	Social Media	13,276	13,276	26,552
hNews	News	1,112	256	1,368
hTwitter	Social Media	114	122	236
4.2 Approach and Design Choices
For our approach we fine-tune CLIP (Radford et al., 2021), a large pretrained multimodal model that maps images and text into a joint embedding space via contrastive learning. Our model generates CLIP embeddings using the RN50x16 backbone, multiplies the image and text embeddings, and passes the result to a classifier that scores the pair as pristine or falsified. We use a learning rate of 5e-08 for CLIP and 5e-05 for the classifier and
$^{3}$ Dedicated to defense against misinformation and falsified media: https://www.darpa.mil/program/semantic-forensics
<sup>4</sup>We thank PAR Tech, Syracuse University, and the University of Colorado, Denver for creating the evaluation data.
Day
The Louvre moves its treasures to a storage facility designed to stand up to global warming impact as climate change brings more floods to Paris via TRF Climate.
hNews
Tensions on the border intensified as North Korea threatened to send its army into the demilitarised border zone. Photo: AFP
hTwitter
Figure 2: Examples of the falsified samples from the evaluation sets. Dev example is our automatically constructed hard negative sample. hNews and hTwitter samples are manually curated. Note, for hNews/hTwitter we do not show the actual samples but create similar examples for illustrative purpose, as the data is not yet publicly available.
He is the Head of Anaheim Regional Medical Center, Mahatma Gandhi. Since the start of the pandemic, he has been underreporting hospital deaths!
train for 16 epochs. For our baseline CLIP Zero Shot model, we generate CLIP embeddings of the shelf and compute a dot product, which is used to score the pair. For more details, see the Appendix.
We report metrics for varying thresholds over the predicted scores; in most tables we report balanced classification accuracy at equal error rate (Acc @ EER). We also report falsified class accuracy at two thresholds (pD @ 0.1 FAR and pD @ EER).
Multimodal Fusion: First, we compare different multimodal fusion techniques, see Table 3. We try three fusion methods: concatenating the CLIP image and text embeddings (Concat), concatenating the embeddings and their dot product (Concat + Dot), and multiplying the embeddings elementwise (Multiply). Inspired by how CLIP was trained to maximize the dot product of normalized image-text pairs, Concat + Dot and Multiply incentivize the classifier to stay faithful to the pre-initialized joint embedding space. These architecture choices yield on average a  $7\%$  performance improvement over simple concatenation. For future experiments we choose the Multiply method to minimize trainable parameters and maintain a simple approach.
Table 3: Balanced binary classification accuracy at EER by fusion method, Dev set.
Climate Change	COVID-19	Military Vehicles
Random	Hard	Random	Hard	Random	Hard
Concat	0.8712	0.6810	0.8797	0.6882	0.9111	0.6775
Concat+Dot	0.9305	0.8038	0.9191	0.7848	0.9485	0.7472
Multiply	0.9344	0.7968	0.9247	0.7807	0.9440	0.7467
Percentage of Hard Negatives: Next, we analyze the importance of using hard negatives in our training data. Specifically, we measure the impact of different percentages of hard negative samples, where the rest are random negatives. Table 4
presents the results. More hard negatives in training generally improves the performance on hard negatives in our development set, but there is also a trade-off in performance on random negatives. Given that we care about samples that more closely mimic challenging real-world misinformation but also want to avoid degrading performance on easy samples, we opt for a ratio of  $75\%$  hard and  $25\%$  random negatives for future experiments.
Table 4: Balanced binary classification accuracy at EER by percentage of hard negatives, Dev set.
Climate Change	COVID-19	Military Vehicles
Random	Hard	Random	Hard	Random	Hard
0%	0.9352	0.7714	0.9188	0.7600	0.9405	0.7236
50%	0.9344	0.7968	0.9247	0.7807	0.9440	0.7467
75%	0.9356	0.7979	0.9241	0.7809	0.9410	0.7470
100%	0.9311	0.8004	0.9227	0.7834	0.9425	0.7457
Results on hNews, hTwitter Sets: Our final model was directly fine-tuned on the entire training set of over 2M training samples, with a ratio of  $75\%$  hard and  $25\%$  random negatives. We report results in Table 5, comparing to CLIP Zero Shot. We improve by  $11\%$  in pD @ 0.1FAR, meaning that our method is able to detect more falsified samples with minimal false alarms. At equal error rate we improve by  $5\%$  in both detection and accuracy. We emphasize that the hTwitter data is unseen to us. Next, we analyze the performance of our final model w.r.t. several characteristics on our Dev set.
OCR Coverage: Given that text present in images can often be used to corroborate captions, we break down model performance by the amount of text detected by an English OCR model<sup>5</sup>. In
<sup>5</sup>https://github.com/JaidedAI/EasyOCR
Table 5: Balanced binary classification accuracy at varying thresholds on Dev, hNews and hTwitter sets. We report based on Probability of Detection (pD), False Alarm Rate (FAR), and Equal Error Rate (EER).
pD @ 0.1 FAR	pD @ EER	Acc @ EER
Dev	Zero Shot	0.7396	0.8287	0.8286
Ours	0.8044	0.8546	0.8546
hNews	Zero Shot	0.2852	0.6133	0.6133
Ours	0.4219	0.6836	0.6840
hTwitter	Zero Shot	0.7623	0.8279	0.8306
Ours	0.8771	0.8771	0.8771
Table 6 (top), we report results broken down by the  $\%$  of the image covered by text (the area of the union of text detections divided by the image size). Each bucket roughly corresponds to natural images, natural images with scene text, graphics, and screenshots of text. The presence of any text yields more than a  $6\%$  improvement for pD @ 0.1FAR and performance peaks at  $10 - 50\%$  coverage.
Table 6: Balanced binary classification accuracy at varying thresholds on Dev set broken down by: % of image covered by text (top), various text-image relationships (middle) and within- vs. cross-cluster status of the hard falsifications (bottom). The latter results are obtained on the subset of hard falsified samples and their corresponding pristine samples.
pD @ 0.1 FAR	pD @ EER	Acc @ EER
OCR Coverage
=0%	0.7588	0.8329	0.8329
0-10%	0.8192	0.8575	0.8575
10-50%	0.8367	0.8709	0.8710
>50%	0.8412	0.8588	0.8588
Text-Image Relationship
Image does not add	0.7908	0.8471	0.8470
Image adds	0.8308	0.8675	0.8674
Text not represented	0.7696	0.8401	0.8401
Text represented	0.8518	0.8745	0.8745
Tweet Text Clustering
Climate Change
Cross-cluster	0.7214	0.8268	0.8268
Within-cluster	0.6571	0.8055	0.8055
COVID-19
Cross-cluster	0.6837	0.8099	0.8103
Within-cluster	0.6013	0.7758	0.7753
Military Vehicles
Cross-cluster	0.7826	0.8634	0.8618
Within-cluster	0.6000	0.7539	0.7545
Text-Image Relationship: Within social media, there exist more complex interactions than the direct relationships seen in formats like image alt-text. As such, we trained a CLIP model on the dataset presented by (Vempala and Preotjiuc-Pietro, 2019) to characterize these relationships: classifying if the image content adds additional meaning (image adds / does not add) or if there is semantic overlap between the text and image (text represented / not
represented). As observed in Table 6 (middle), for samples with text represented model performance improves by  $8\%$  and for samples where image adds performance improves by  $4\%$  for detection in a high precision regime (pD @ 0.1FAR). Although the text-image relationship model has somewhat noisy classifications for the text task, the text represented class generally contains samples with a shared entity between image and text, which would make fine-grained misinformation detection easier. The image adds class mostly contains graphics, likely due to training data bias, which aligns with the OCR coverage experiments above.
Tweet Text Clustering: Finally, we analyze the sub-topics obtained as a result of clustering Tweets within each topic<sup>7</sup>. This allows us to tease out clusters, e.g., vaccination for COVID-19, floods for Climate Change or drones for Military Vehicles. Recall that our model performs the best on Climate Change and the worst on the Military Vehicles (Table 4). Possible factors include the smaller amount of training data and visual similarity of different vehicle types. We also observe that among the hard negatives for Military Vehicles, only  $39\%$  are cross-cluster (while Climate Change and COVID-19 have  $51\%$  and  $58\%$  respectively), indicating the Military Vehicles set contains a larger proportion of harder fakes. These factors may explain the larger difference between cross/within cluster performance for this topic (Table 6, bottom).
In this work we tackle a real-world challenge of detecting out-of-context image-text tweets on COVID-19, Climate Change, Military Vehicles topics. To approach it, we collect Twitter-COMMs, a large-scale topical dataset with multimodal tweets, and construct corresponding hard mismatches. We design our approach based on the CLIP model with several important design choices, e.g. multiplying the embeddings for multimodal fusion and increasing the percentage of hard negatives in our training data. This approach substantially improves over a powerful baseline, an off-the-shelf CLIP model, when evaluated on human-curated in-the-wild mismatches. We hope our work and insights will benefit multimedia forensics practitioners.
Our model achieves  $86\%$  and  $62\%$  on the image and text binary classification tasks respectively, which is  $5\%$  and  $4\%$  higher than the best models presented in the original paper.
See Appendix A.3.4 for details.
Here, we discuss ethical considerations regarding our work. Image repurposing is a prominent societal issue that lacks sufficient training data in general, and in particular for content on social media platforms such as Twitter. Even more, our work aims to be proactive in studying the threat of out-of-context media and proposes an approach for detecting such misinformation. By presenting a dataset, a detection approach, and several key observations about falsified out-of-context Tweets, we hope that our work serves as a net benefit for society.
How was the data collected? We collected data using the Twitter Search API v2. Our methodology is described in detail in Appendix A.1.
What are the intellectual property rights? Twitter owns the intellectual property for the Tweets in our Twitter-COMMs dataset. We adhere to the restrictions they place on Tweets downloaded via their API, namely that we may not share the content downloaded from the API, but we have released the Tweet ID's — which others can use to download the Tweets and images from the API.
How did we address participant privacy rights? N/A
Were annotators treated fairly? Did we require review from a review board? N/A
Which populations do we expect our dataset to work for? Our dataset is specific to social media posts from Twitter that are written in English; it will primarily be useful for audiences from English speaking countries, such as the US, UK, Australia, and India. The biases inherent to the short text style (280 characters or less) and of Tweets with images will be useful for those interested in researching multimodal misinformation on Twitter.
What is the generalizability of our claims? Our results apply primarily to Tweets on our three topics of interest (COVID-19, Climate Change, Military Vehicles) written in English and having at least one attached image.
How did we ensure dataset quality? Our data collection methodology is described in detail in Appendix A.1. To address data quality for Military Vehicles we created an image classifier to filter out tweets that did not have images of military vehicles or aircraft (Appendix A.1.2). Additionally, the sub-topic clustering we performed (Section 4.3, Appendix A.3.4) reveals that most of the text falls into clusters that are related to the three main topics. We also provide some statistics for tweets with
possibly sensitive content as flagged by Twitter in Table 14 (Appendix).
What is the climate impact? Our final model used 8 days of training on 10 GPUs. Additional experiments such as the investigation of text image relationships used 4 days on a single GPU, and tweet text clustering used 10 hours on a single GPU. The GPU used for all experiments were GeForce 2080 RTX Ti's. In total we used 2,026 GPU hours, and total emissions are estimated to be  $218.81\mathrm{kgCO_2eq}$  of which 0 percents were directly offset. Estimations were conducted using the MachineLearning Impact calculator presented in (Lacoste et al., 2019).