Venue: NAACL 2022

# Twitter-COMMs: Detecting Climate, COVID, and Military Multimodal Misinformation

Giscard Biamby 1

Grace Luo 1

Trevor Darrell   Anna Rohrbach

University of California, Berkeley

{gbiamby, graceluo, trevordarrell, anna.rohrbach}@berkeley.edu

Footnote 1: Denotes equal contribution.

###### Abstract

Detecting out-of-context media, such as "miscaptioned" images on Twitter, is a relevant problem, especially in domains of high public significance. In this work we aim to develop defenses against such misinformation for the topics of Climate Change, COVID-19, and Military Vehicles. We first present a large-scale _multimodal_ dataset with over 884k tweets relevant to these topics. Next, we propose a detection method, based on the state-of-the-art CLIP model, that leverages automatically generated hard image-text mismatches. While this approach works well on our automatically constructed out-of-context tweets, we aim to validate its usefulness on data representative of the real world. Thus, we test it on a set of human-generated fakes created by mimicking in-the-wild misinformation. We achieve an \(11\%\) detection improvement in a high precision regime over a strong baseline. Finally, we share insights about our best model design and analyze the challenges of this emerging threat.

+
Footnote †: July 10-15, 2022 ©2022 Association for Computational Linguistics

## 1 Introduction

Out-of-context images are a popular form of misinformation where an image is miscaptioned to support a false claim Fazio (2020). Such image repurposing is extremely cheap yet can be as damaging as more sophisticated fake media. In this work we focus on domains important for society and national security, where implications of inexpensive yet effective misinformation can be immense.

Specifically, we analyze multimodal Twitter posts that are of significant public interest, related to topics of COVID-19, Climate Change and Military Vehicles. Our goal is to learn to categorize such image-text posts as pristine or falsified (out-of-context) by means of detecting semantic inconsistencies between images and text. To that end, we first collect a large-scale dataset of _multimodal_ tweets, **Twitter-COMMs**, with over 884k tweets. In our approach, we fuse input image and text embeddings generated by CLIP Radford et al. (2021) via an elementwise product, and train a classifier to distinguish real tweets from automatically constructed random and hard mismatches. To validate this approach and demonstrate the usefulness of the Twitter-COMMs dataset, we report results on human-generated test data, created to mimic real-world misinformation. We discuss the results and model ablations, and provide additional insights into the challenges of this task. Our dataset is publicly available at: [https://github.com/GiscardBiamby/Twitter-COMMs](https://github.com/GiscardBiamby/Twitter-COMMs).

## 2 Related Work

There exist a number of large-scale Twitter datasets concentrated on topics such as COVID-19 Banda et al. (2021) or Climate Change Littman and Wrubel (2019). However, it remains difficult to collect labeled misinformation. Researchers have collected COVID-19 misconceptions on social media via manual annotation Hossain et al. (2020) or by linking to fact checking articles Patwa et al. (2021). Not only are these datasets small (a few thousand samples), but they focus on false claims rather than multimodal inconsistency. Here, we curate social media posts that are topical and multi-modal, and we demonstrate an application to misinformation detection of human-generated fakes.

Recent work has developed approaches for multimodal fact checking, e.g., Jaiswal et al. (2017) and Muller-Budack et al. (2020), who query an external knowledge base. Similar to Luo et al. (2021) in the news domain, we use a large pretrained model that does not require an external reference set.

## 3 Twitter-COMMs Dataset

Here, we describe the data collection strategies behind **Twitter-COMMs**, which consists of mul

timodal tweets covering the topics of COVID-19, Climate Change, and Military Vehicles.

**Data Collection:** We collected data using Twitter API v21 in three stages for COVID-19 and Climate Change, and two stages for Military Vehicles, refining the filters at each stage to acquire more relevant tweets. COVID-19 and Climate Change stages progressed from simple high level keywords towards more specific ones in stage two and tweets authored by news organizations in the final stage. For Military Vehicles the first stage used high level search terms such as "military", "aircraft", "tank", which resulted in noisy data, so the second stage used a large number of highly specific terms related to vehicle models. Full details can be found in Appendix A.1. We employed the following global filters for all topics: (1) language=English, (2) has at least one image, and (3) not a retweet.

Footnote 1: [https://developer.twitter.com/en/docs/twitter-api/getting-started/about-twitter-api](https://developer.twitter.com/en/docs/twitter-api/getting-started/about-twitter-api)

In total, we have collected \(884,331\) tweets, each having at least one image (composed of 24% Climate Change, 64.5% COVID-19, and 11.5% Military Vehicles tweets), see Table 1. Tweets for Climate Change and Military Vehicles were collected starting from June 2016 and for COVID-19 starting from February 2020, all ending in September 2021.

**Falsified Samples:** In addition to the pristine samples, we automatically generate falsified samples where there is some inconsistency between image and text. We create random negatives (denoted as "Random") by selecting an image for a given caption at random. We also create hard negatives (denoted as "Hard") by retrieving the image of the sample with the greatest textual similarity for a given caption (following the "Semantics / CLIP Text-Text" split from Luo et al. (2021)). We mainly generate mismatches _within_ each topic (COVID-19, Climate Change, Military Vehicles), except for a small set of random mismatches _across_ topics (denoted as "Cross Topic"). Our dataset is balanced with respect to labels, where half of the samples are pristine and half are falsified. Table 1 presents summary statistics for the training samples. We detail our development set and other data used for evaluation in the next section.

**Qualitative Analysis:** We present random examples from our training set in Figure 1. Overall, we see that the collected Twitter samples tend to be "on topic" and the amount of noise is low. Hard negatives are often visually grounded, while random negatives contain image/text pairs that are only weakly related, since they pertain to the same topic. The Climate Change hard negative depicts an image of flooded homes to represent "droughts, fires and floods" while the random negative depicts an image of cars relevant to climate but inconsistent with "polar bears". The COVID-19 hard negative uses an image of a Nigerian spokesman to depict news pertaining to "ECOWAS2" while the random one uses a stock photo of lab testing to represent Covid. These entity-level, rather than topic-level, alignments more closely resemble real-world out-of-context images that often reference and misrepresent visually depicted entities. Note the diversity of images and text in our training set, where there exist both natural images and info-graphics, and language varies from organizational announcements and news headlines to personal opinions.

Footnote 2: Economic Community of West African States

## 4 Experiments

Next, we discuss the data used for evaluation, present our approach and ablate various design choices, report results on our evaluation sets, and provide additional analysis of the task difficulty.

### Evaluation Sets

We report results on three evaluation sets. (a) We validate our approach on samples synthetically generated using the same procedure as our training set (denoted Dev), where all topics and falsification methods are equally represented (i.e., the ratio of random vs. hard negatives is 50-50). We also evaluate on _human-curated_ samples from the

\begin{table}
\begin{tabular}{l|c|r r c} \hline \hline Topic / Samples & Collected & Pristine & \multicolumn{2}{c}{Falsefied} \\  & & & Random & Hard \\ \hline Climate Change & 212,665 & 298,809 & 84,432 & 214,377 \\ COVID-19 & 569,982 & 736,539 & 162,410 & 574,129 \\ Military Vehicles & 101,684 & 139,213 & 35,376 & 103,837 \\ Cross Topic & - & 59,735 & 59,735 & - \\ \hline Total & 884,331 & \multicolumn{2}{c}{2,468,592} \\ \hline \hline \end{tabular}
\end{table}
Table 1: **Twitter-COMMs breakdown. “Collected” denotes all unique samples collected via the Twitter API. “Pristine” and “Falsified“ denote all samples in our automatically generated Training set. To ensure the balanced Training set, we “repeat” Pristine samples such that there is an equal number of Pristine and Falsified samples.** 

DARPA Semantic Forensics (SemaFor) Program3 derived from (b) news images and captions (denoted hNews) and (c) Twitter (denoted hTWitter). To generate this data, humans manually introduced inconsistencies to pristine image-caption pairs.4 While hNews/hTwitter data is not _real_ misinformation, it is _in-the-wild_ w.r.t. our synthetic training data and much more representative of real-world human-generated misinformation. All three evaluation sets contain a mixture of samples relevant to the topics of COVID-19, Climate Change, and Military Vehicles (Figure 2). Table 2 provides the number of samples in each set.

Footnote 3: Dedicated to defense against misinformation and falsified media: [https://www.darpa.mil/program/semantic-forensics](https://www.darpa.mil/program/semantic-forensics)

Footnote 4: We thank PAR Tech, Syracuse University, and the University of Colorado, Denver for creating the evaluation data.

### Approach and Design Choices

For our approach we fine-tune CLIP Radford et al. (2021), a large pretrained multimodal model that maps images and text into a joint embedding space via contrastive learning. Our model generates CLIP embeddings using the RN50x16 backbone, multiplies the image and text embeddings, and passes the result to a classifier that scores the pair as pristine or falsified. We use a learning rate of 5e-08 for CLIP and 5e-05 for the classifier and

\begin{table}
\begin{tabular}{l l|c c c} \hline \hline  & Domain & Pristine & Falsified & Total \\ \hline Dev & Social Media & 13,276 & 13,276 & 26,552 \\ hNews & News & 1,112 & 256 & 1,368 \\ hTwitter & Social Media & 114 & 122 & 236 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Evaluation samples breakdown.

Figure 1: **Twitter-COMMs** examples of Pristine and Falsified (Random / Hard) samples by topic.

 

train for 16 epochs. For our baseline CLIP Zero Shot model, we generate CLIP embeddings of-the-shelf and compute a dot product, which is used to score the pair. For more details, see the Appendix.

We report metrics for varying thresholds over the predicted scores; in most tables we report balanced classification accuracy at equal error rate (Acc @ EER). We also report falsified class accuracy at two thresholds (pD @ 0.1 FAR and pD @ EER).

**Multimodal Fusion:** First, we compare different multimodal fusion techniques, see Table 3. We try three fusion methods: concatenating the CLIP image and text embeddings (Concat), concatenating the embeddings and their dot product (Concat + Dot), and multiplying the embeddings elementwise (Multiply). Inspired by how CLIP was trained to maximize the dot product of normalized image-text pairs, Concat + Dot and Multiply incentivize the classifier to stay faithful to the pre-initialized joint embedding space. These architecture choices yield on average a 7% performance improvement over simple concatenation. For future experiments we choose the Multiply method to minimize trainable parameters and maintain a simple approach.

**Percentage of Hard Negatives:** Next, we analyze the importance of using hard negatives in our training data. Specifically, we measure the impact of different percentages of hard negative samples, where the rest are random negatives. Table 4 presents the results. More hard negatives in training generally improves the performance on hard negatives in our development set, but there is also a trade-off in performance on random negatives. Given that we care about samples that more closely mimic challenging real-world misinformation but also want to avoid degrading performance on easy samples, we opt for a ratio of 75% hard and 25% random negatives for future experiments.

### Results and Analysis

**Results on hNews, hTwitter Sets:** Our final model was directly fine-tuned on the entire training set of over 2M training samples, with a ratio of 75% hard and 25% random negatives. We report results in Table 5, comparing to CLIP Zero Shot. We improve by 11% in pD @ 0.1FAR, meaning that our method is able to detect more falsified samples with minimal false alarms. At equal error rate we improve by 5% in both detection and accuracy. We emphasize that the hTwitter data is unseen to us.

Next, we analyze the performance of our final model w.r.t. several characteristics on our Dev set.

**OCR Coverage:** Given that text present in images can often be used to corroborate captions, we break down model performance by the amount of text detected by an English OCR model5. In

\begin{table}
\begin{tabular}{l|c c|c c|c c} \hline \hline  & \multicolumn{2}{c|}{Climate Change} & \multicolumn{2}{c|}{COVID-19} & \multicolumn{2}{c}{Military Vehicles} \\  & Random & Hard & Random & Hard & Random & Hard \\ \hline
0\% & 0.9352 & 0.7714 & 0.9188 & 0.7600 & 0.9405 & 0.7236 \\
50\% & 0.9344 & 0.7968 & **0.9247** & 0.7807 & **0.9440** & 0.7467 \\
75\% & **0.9356** & 0.9797 & 0.9241 & 0.7809 & 0.9410 & **0.7470** \\
100V & 0.9311 & **0.8004** & 0.9227 & **0.7834** & 0.9425 & 0.7457 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Balanced binary classification accuracy at EER by percentage of hard negatives, Dev set.

Figure 2: Examples of the falsified samples from the evaluation sets. Dev example is our automatically constructed hard negative sample. hNews and hTwitter samples are manually curated. Note, for hNews/hTwitter we do not show the actual samples but create similar examples for illustrative purpose, as the data is not yet publicly available.

 

Table 6 (top), we report results broken down by the % of the image covered by text (the area of the union of text detections divided by the image size). Each bucket roughly corresponds to natural images, natural images with scene text, graphics, and screenshots of text. The presence of any text yields more than a 6% improvement for pD @ 0.1FAR and performance peaks at 10-50% coverage.

**Text-Image Relationship:** Within social media, there exist more complex interactions than the direct relationships seen in formats like image alt-text. As such, we trained a CLIP model on the dataset presented by Vempala and Preotiuc-Pietro (2019) to characterize these relationships: classifying if the image content adds additional meaning (image adds / does not add) or if there is semantic overlap between the text and image (text represented / not represented).6 As observed in Table 6 (middle), for samples with _text represented_ model performance improves by 8% and for samples where _image adds_ performance improves by 4% for detection in a high precision regime (pD @ 0.1FAR). Although the text-image relationship model has somewhat noisy classifications for the text task, the _text represented_ class generally contains samples with a shared entity between image and text, which would make fine-grained misinformation detection easier. The _image adds_ class mostly contains infographics, likely due to training data bias, which aligns with the OCR coverage experiments above.

Footnote 6: Our model achieves 86% and 62% on the image and text binary classification tasks respectively, which is 5% and 4% higher than the best models presented in the original paper.

**Tweet Text Clustering**: Finally, we analyze the sub-topics obtained as a result of clustering Tweets within each topic7. This allows us to tease out clusters, e.g., _vaccination_ for COVID-19, _floods_ for Climate Change or _drones_ for Military Vehicles. Recall that our model performs the best on Climate Change and the worst on the Military Vehicles (Table 4). Possible factors include the smaller amount of training data and visual similarity of different vehicle types. We also observe that among the hard negatives for Military Vehicles, only 39% are cross-cluster (while Climate Change and COVID-19 have 51% and 58% respectively), indicating the Military Vehicles set contains a larger proportion of harder fakes. These factors may explain the larger difference between cross/within cluster performance for this topic (Table 6, bottom).

Footnote 7: See Appendix A.3.4 for details.

## 5 Conclusion

In this work we tackle a real-world challenge of detecting out-of-context image-text tweets on COVID-19, Climate Change, Military Vehicles topics. To approach it, we collect **Twitter-COMMs**, a large-scale topical dataset with _multimodal_ tweets, and construct corresponding hard mismatches. We design our approach based on the CLIP model with several important design choices, e.g. multiplying the embeddings for multimodal fusion and increasing the percentage of hard negatives in our training data. This approach substantially improves over a powerful baseline, an off-the-shelf CLIP model, when evaluated on human-curated in-the-wild mismatches. We hope our work and insights will benefit multimedia forensics practitioners.

\begin{table}
\begin{tabular}{l|c c c} \hline \hline  & pD @ 0.1 FAR & pD@ EER & Acc @ EER \\ \hline
**OCR Coverage** & & & \\ =0\% & 0.7588 & 0.8329 & 0.8329 \\
0-10\% & 0.8192 & 0.8575 & 0.8575 \\
15-50\% & 0.8367 & 0.8709 & 0.8710 \\
2-50\% & 0.8412 & 0.8588 & 0.8588 \\ \hline \multicolumn{4}{l}{**Text-Image Relationship**} \\ Image does not add & 0.7908 & 0.8471 & 0.8470 \\ Image adds & 0.8308 & 0.8675 & 0.8674 \\ \hline Text not represented & 0.7696 & 0.8401 & 0.8401 \\ Text represented & 0.8518 & 0.8745 & 0.8745 \\ \hline \multicolumn{4}{l}{**Tweet Text Clustering**} \\ Climate Change & & & \\ Cross-cluster & 0.7214 & 0.8268 & 0.8268 \\ Within-cluster & 0.6571 & 0.8055 & 0.8055 \\ \hline COVID-19 & & & \\ Cross-cluster & 0.6837 & 0.8099 & 0.8103 \\ Within-cluster & 0.6013 & 0.7758 & 0.7753 \\ \hline Military Vehicles & & & \\ Cross-cluster & 0.7826 & 0.8634 & 0.8618 \\ Within-cluster & 0.6000 & 0.7539 & 0.7545 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Balanced binary classification accuracy at varying thresholds on Dev set broken down by: % of image covered by text (top), various text-image relationships (middle) and within- vs. cross-cluster status of the hard falsifications (bottom). The latter results are obtained on the subset of hard falsified samples and their corresponding pristine samples.

\begin{table}
\begin{tabular}{l l|c c c} \hline \hline  & & pD @ 0.1 FAR & pD @ EER & Acc @ EER \\ \hline
**Dev** & Zero Shot & 0.7396 & 0.8287 & 0.8286 \\  & Ours & **0.8044** & **0.8546** & **0.8546** \\ \hline
**hNews** & Zero Shot & 0.2852 & 0.6133 & 0.6133 \\  & Ours & **0.4219** & **0.6836** & **0.6840** \\ \hline
**hTwitter** & Zero Shot & 0.7623 & 0.8279 & 0.8306 \\  & Ours & **0.8771** & **0.8771** & **0.8771** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Balanced binary classification accuracy at varying thresholds on Dev, hNews and hTwitter sets. We report based on Probability of Detection (pD), False Alarm Rate (FAR), and Equal Error Rate (EER).

 

Ethical Considerations

Here, we discuss ethical considerations regarding our work. Image repurposing is a prominent societal issue that lacks sufficient training data in general, and in particular for content on social media platforms such as Twitter. Even more, our work aims to be proactive in studying the threat of out-of-context media and proposes an approach for detecting such misinformation. By presenting a dataset, a detection approach, and several key observations about falsified out-of-context Tweets, we hope that our work serves as a net benefit for society.

**How was the data collected?** We collected data using the Twitter Search API v2. Our methodology is described in detail in Appendix A.1.

**What are the intellectual property rights?** Twitter owns the intellectual property for the Tweets in our **Twitter-COMMs** dataset. We adhere to the restrictions they place on Tweets downloaded via their API, namely that we may not share the content downloaded from the API, but we have released the Tweet ID's -- which others can use to download the Tweets and images from the API.

**How did we address participant privacy rights?** N/A

**Were annnotators treated fairly? Did we require review from a review board?** N/A

**Which populations do we expect our dataset to work for?** Our dataset is specific to social media posts from Twitter that are written in English; it will primarily be useful for audiences from English speaking countries, such as the US, UK, Australia, and India. The biases inherent to the short text style (280 characters or less) and of Tweets with images will be useful for those interested in researching multimodal misinformation on Twitter.

**What is the generalizability of our claims?** Our results apply primarily to Tweets on our three topics of interest (COVID-19, Climate Change, Military Vehicles) written in English and having at least one attached image.

**How did we ensure dataset quality?** Our data collection methodology is described in detail in Appendix A.1. To address data quality for Military Vehicles we created an image classifier to filter out tweets that did not have images of military vehicles or aircraft (Appendix A.1.2). Additionally, the sub-topic clustering we performed (Section 4.3, Appendix A.3.4) reveals that most of the text falls into clusters that are related to the three main topics. We also provide some statistics for tweets with possibly sensitive content as flagged by Twitter in Table 14 (Appendix).

**What is the climate impact?** Our final model used 8 days of training on 10 GPUs. Additional experiments such as the investigation of text image relationships used 4 days on a single GPU, and tweet text clustering used 10 hours on a single GPU. The GPU used for all experiments were GeForce 2080 RTX Ti's. In total we used 2,026 GPU hours, and total emissions are estimated to be 218.81 kgCO\({}_{2}\)eq of which 0 percents were directly offset. Estimations were conducted using the MachineLearning Impact calculator presented in Lacoste et al. (2019).

**What are the potential dataset biases?** Here, we focus on our method used to generate hard falsified samples to understand the potential biases learned during training. Specifically, we note potential age, race, and gender biases present in CLIP, the underlying model used to generate our mismatches. Radford et al. (2021) find the CLIP exhibits significant performance differences when classifying individuals of different races and ages into categories related to crime or animals. Agarwal et al. (2021) also find gender biases in the CLIP embeddings when classifying occupations. These biases primarily affect the synthetically generated training set, not the pristine data. However, we can not rule out that the pristine Twitter data may also capture some human biases or harmful stereotypes.

## 7 Acknowledgements

We would like to thank PAR Tech, Syracuse University, and the University of Colorado, Denver for creating the evaluation data. We thank the SRI team, including John Cadigan and Martin Graciarena, for providing the WikiData-sourced news organization Twitter handles. We would also like to thank Dong Huk (Seth) Park, Sanjay Subramanian, and Reuben Tan for helpful discussions on finetuning CLIP. This work was supported in part by DoD including DARPA's LwLL, and/or SemaFor programs, and Berkeley Artificial Intelligence Research (BAIR) industrial alliance programs.

## References

* Agarwal et al. (2021) Sandhini Agarwal, Gretchen Krueger, Jack Clark, Alec Radford, Jong Wook Kim, and Miles Brundage. 2021. Evaluating clip: Towards characterization of broader capabilities and downstream implications. arXiv:2108.02818 

Juan M. Banda, Ramya Tekumalla, Guanyu Wang, Jingyuan Yu, Tuo Liu, Yuning Ding, Ekaterina Artemova, Elena Tutubilain, and Gerardo Chowell. 2021. A large-scale covid-19 twitter chatter dataset for open scientific research--an international collaboration. _Epidemiologia_, 2(3):315-324.
* Fazio [2020] Lisa Fazio. 2020. Out-of-context photos are a powerful low-tech form of misinformation. the conversation.com/out-of-context-photos-are-a-powerful-low-tech-form-of-misinformation-129959.
* Grootendorst [2020] Maarten Grootendorst. 2020. Bertopic: Leveraging bert and c-tf-idf to create easily interpretable topics.
* Hossain et al. [2020] Tamanna Hossain, Robert L. Logan IV, Arjuna Ugarte, Yoshitomo Matsubara, Sean Young, and Sameer Singh. 2020. COVIDLies: Detecting COVID-19 misinformation on social media. In _Proceedings of the 1st Workshop on NLP for COVID-19 (Part 2) at EMNLP 2020_, Online. Association for Computational Linguistics.
* Jaiswal et al. [2017] Ayush Jaiswal, Ebraam Sabir, Wael AbdAlmageed, and Premkumar Natarajan. 2017. Multimedia semantic integrity assessment using joint embedding of images and text. In _Proceedings of the 25th ACM international conference on Multimedia_.
* Kumar et al. [2022] Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. 2022. Fine-tuning can distort pretrained features and underperform out-of-distribution. In _The International Conference on Learning Representations (ICLR)_.
* Lacoste et al. [2019] Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. 2019. Quantifying the carbon emissions of machine learning. _arXiv:1910.09700_.
* Littman and Wrubel [2019] Justin Littman and Laura Wrubel. 2019. Climate Change Tweets Ids.
* Luo et al. [2021] Grace Luo, Trevor Darrell, and Anna Rohrbach. 2021. Newsclippings: Automatic generation of out-of-context multimodal media. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_.
* Malzer and Baum [2020] Claudia Malzer and Marcus Baum. 2020. A hybrid approach to hierarchical density-based cluster selection. In _2020 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI)_, pages 223-228. IEEE.
* McInnes and Healy [2017] Leland McInnes and John Healy. 2017. Accelerated hierarchical density based clustering. _2017 IEEE International Conference on Data Mining Workshops (ICDMW)_.
* McInnes et al. [2020] Leland McInnes, John Healy, and James Melville. 2020. Umap: Uniform manifold approximation and projection for dimension reduction. _arXiv:1802.03426_.
* Muller-Budack et al. [2020] Eric Muller-Budack, Jonas Theiner, Sebastian Diering, Maximilian Idahl, and Ralph Ewerth. 2020. Multimodal analytics for real-world news using measures of cross-modal entity consistency. In _ACM ICMR_.
* Patwa et al. [2021] Parth Patwa, Shivam Sharma, Srinivas Pylkal, Vineeth Gupta, Gitnjali Kumari, Md Shad Akhtar, Asif Ebbal, Amitava Das, and Tanmyoy Chakraborty. 2021. Fighting an infodemic: Covid-19 fake news dataset. _Communications in Computer and Information Science_, page 21-29.
* Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. _arXiv:2103.00020_.
* Reimers and Gurevych [2019] Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_.
* Tan and Le [2019] Mingxing Tan and Quoc Le. 2019. Efficientnet: Rethinking model scaling for convolutional neural networks. In _International Conference on Machine Learning_, pages 6105-6114. PMLR.
* Vempala and Preotiuc-Pietro [2019] Alakananda Vempala and Daniel Preotiuc-Pietro. 2019. Categorizing and inferring the relationship between the text and image of Twitter posts. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 2830-2840, Florence, Italy. Association for Computational Linguistics.

 

Appendix

In Section A.1 we provide additional details about data collection, including our strategy and search keywords. Section A.2 provides dataset statistics, including information on tweet counts, geographical information, possibly sensitive content, and image availability. We include additional experiments in Section A.3.

### Data Collection

#### a.1.1 COVID-19 and Climate Change

Our data collection consisted of three stages. The first employed simple topic, keyword, and hashtag filters, the second stage used more specific keyword and topic combinations, while the third focused on collecting topical data from Twitter accounts of various news organizations.

In the first stage we collected roughly 100,000 tweets each for COVID-19 and Climate Change topics. We used the "COVID-19" topic of the Twitter API's Entity Annotations feature8, which allows users to find tweets related to predetermined topics. For Climate Change we filtered with an OR clause on keywords "climate change", "global warming", and (#globalwarming, #climatechange) hashtags. Inspection of the stage \(1\) results revealed a lot of off-topic tweets. For example, a Twitter user might post a tweet about working from home during the pandemic and tag the tweet with a COVID-related hashtag. While this type of content is somewhat related to COVID-19, we wanted to focus on data where misinformation/disinformation might be more relevant, such as more topical/newsworthy tweets (e.g. bad actors may spread propaganda related to the COVID-19 pandemic by making false or misleading claims). To that end, in stage \(2\) we filtered by combining each topic phrase with one of the 19 topical search terms (e.g. "agriculture", "crops", "death", "vaccination"). The resulting data appeared much more relevant than the initial collection effort. Table 7 contains a list of the search terms we used to collect data for COVID-19 and Climate Change tweets. Finally, related to the argument above, in the third collection stage we focused on tweets authored by news organizations, as opposed to random users. For that, 7k news organization Twitter handles were sourced from WikiData9.

Footnote 8: [https://developer.twitter.com/en/docs/labs/annotations](https://developer.twitter.com/en/docs/labs/annotations)

Footnote 9: [https://www.wikidata.org/](https://www.wikidata.org/)

#### a.1.2 Military Vehicles

Collecting data about the Military Vehicles topic proved more challenging than the other two topics. We initially tried simple keyword filters such as "military", "aircraft", "tank", etc, but found that those resulted in a lot of irrelevant content such as tweets related to video games, or tweets where "tank" took a different meaning (e.g., "fish tank" or "tank tops"). This initial approach did not return many relevant results. The WikiDData news organization approach used in the other two topics also did not provide enough usable data. As a result we crafted two different, highly customized stages for Military Vehicles. We gathered a list of both civilian and military vehicles and aircraft from eight different publicly available datasets (see Table 8). The datasets were annotated either for image classification or object detection tasks. We queried the Twitter Search API using the vehicle and aircraft names from this set, but returned a lot of off-topic data. We then trained an EfficientNet [Tan and Le, 2019] image classifier that categorized images as either civilian ground vehicle, civilian aircraft, military ground vehicle, military aircraft, or other. (The "other" category training set consisted of several thousand manually annotated images from the initial data collection effort that did not contain any military or civilian vehicles or aircraft.) We trained the classifier

\begin{table}
\begin{tabular}{l} \hline \hline Search Terms \\ \hline Agriculture, COVID, COVID-19, Climate Change, Crops, Death, Death Toll, Floods, Harvest, Hurricane, ICBM, Military, Military Parade, Military Vehicles, Show of Force, Tank, Troops, Typhoon, Vaccination \\ \hline \hline \end{tabular}
\end{table}
Table 7: Search Terms Used in Stage 2 of the Data Collection for COVID-19 and Climate Change 

to 97% accuracy and used it to filter out any tweets predicted to be in the "other" category. For the second collection stage we combined the military vehicle and aircraft names with custom keywords (Table 9).

### Dataset Statistics

Table 10 shows a summary of the dataset. The "Geo-tagged" column refers to the geolocation data provided by tweet authors. This property is empty in most cases, but when present, can be in the form of a Twitter "place" which contains a display name, a geo polygon (which in some cases is as broad as an entire country), as well as other fields, such as country name. It is also possible for the geo data to be in the form of latitude and longitude, but that is rarer. The "Countries" columns is extracted from the geo location data and because of the small amount of geo-tagged tweets we can only report countries for a small fraction of tweets in the dataset (Table 11).

One oddity to note is that although we included an English-only search filter ("lang:en") in all API calls, the API still returned a small number of non-English tweets (Table 12). We are not sure why this is, but manual inspection of some of these examples shows that a good portion of them are in fact in English.

Figure 3 shows high-level "word cloud" summaries for the hashtags in the tweets for each topic.

Table 14 shows the number of tweets that Twitter flagged as containing possibly sensitive material, i.e., samples that may contain adult content or graphic violence. We encourage users to be aware of such tweets, which account for about 1% of the data, and may be undesirable for certain downstream tasks.

\begin{table}
\begin{tabular}{l c c c} \hline \hline Topic & Tweets & Geo-tagged & Countries & Captions \\ \hline COVID & 569,982 & 4,637 & 112 & 569,982 \\ Climate & 212,665 & 3,696 & 138 & 212,662 \\ Military & 101,684 & 3,913 & 105 & 101,640 \\ All & 884,331 & 13,404 & 172 & 884,284 \\ \hline \hline \end{tabular}
\end{table}
Table 10: Full Dataset Summary

Figure 3: Word Cloud Summaries for Each Topic



The total number of images/tweets is shown in Table 15. Twitter allows users to include 1-4 images in a tweet. As seen in Table 16, 90% of the tweets have a single image. In cases where a tweet contained

\begin{table}
\begin{tabular}{l r r} \hline \hline Pass. Sensitive & Tweets & \% of Total \\ \hline True & 9,151 & 1.03 \\ False & 875,180 & 98.97 \\ \hline \hline \end{tabular}
\end{table}
Table 15: Media Summary

\begin{table}
\begin{tabular}{l c c c} \hline \hline Country & Tweets & Geo-tagged & Captions \\ \hline India & 2,399 & 6,692 & 2,399 \\ United Kingdom & 2,127 & 6,020 & 2,127 \\ United States & 707 & 2,338 & 707 \\ Canada & 519 & 1,476 & 519 \\ Australia & 339 & 1,024 & 339 \\ Pakistan & 203 & 606 & 203 \\ Germany & 146 & 454 & 146 \\ Kenya & 130 & 360 & 130 \\ Ireland & 128 & 394 & 128 \\ South Africa & 118 & 342 & 118 \\ Nigeria & 116 & 352 & 116 \\ Uganda & 115 & 298 & 115 \\ Republic of the Philippines & 107 & 320 & 107 \\ France & 100 & 298 & 100 \\ The Netherlands & 95 & 290 & 95 \\ Indonesia & 81 & 238 & 81 \\ Malaysia & 77 & 224 & 77 \\ Spain & 75 & 194 & 75 \\ New Zealand & 68 & 212 & 68 \\ Belgium & 67 & 182 & 67 \\ \hline \hline \end{tabular}
\end{table}
Table 11: Totals by Country (Top 20 Only)

\begin{table}
\begin{tabular}{l c c c} \hline \hline Language & Tweets & Total geo-tagged & Countries & Unique Captions \\ \hline English & 883,310 & 9,268 & 172 & 883,263 \\ Non-English & 618 & 3 & 3 & 618 \\ \hline \hline \end{tabular}
\end{table}
Table 13: Totals for Country=“US”, by Topic

\begin{table}
\begin{tabular}{l r r} \hline \hline Point & Tweets & geo-tagged & Countries \\ \hline Military Vehicles & 705 & 705 & 1 \\ COVID-19 & 0 & 0 & 0 \\ Climate Change & 2 & 2 & 1 \\ \hline \hline \end{tabular}
\end{table}
Table 14: Possibly Sensitive Tweets 

more than one image, we only used the first image (according to the order of the images returned by the Twitter API).

### Additional Experiments

All experiments reported in this paper are for a single run, as we find that variance across multiple runs is low. All ROC curves and metrics are computed using sklearn's roc_curve function. All models are implemented in PyTorch. For our experiments, we make the following design choices:

* We use the RN50x16 backbone. We find that this backbone consistently yields a 2-3% improvement compared to other released backbones, such as ViT/B-32. Our final CLIP model contains \(\sim\)300M parameters initialized from the RN50x16 backbone and \(\sim\)600k parameters randomly initialized for our classifier.
* We tune the upper layers and keep CLIP's lower layers frozen10. We find that this scheme is more memory efficient and yields more stable convergence than tuning all the layers. Footnote 10: We fine-tune the layers “visual_layer4”, “visual_attnpool”, “transformer.resolocks.11”, “ln_final”, “text_projection”, “logit_scale”.
* We use a learning rate of 5e-08 for CLIP and 5e-05 for the classifier. From our hyperparameter sweeps we find this setting to be the most appropriate, as CLIP is pretrained while the classifier is randomly initialized.
* We multiply CLIP image and text embeddings before passing that as an input to the classifier. This is different from Luo et al. (2021), who used a simple feature concatenation.

#### a.3.1 Expert vs. Joint Training

Here we study whether training a joint model on all three topics at once may be inferior to training three topic-specific experts, see Figure 4. We find that the joint model performs on par with or better than the expert models, thus we use a joint model in all the other experiments.

#### a.3.2 Fine-Tuning Scheme

Since we only know the high-level topics but not the precise composition of samples in our hidden set hTwitter, we investigate methods for out-of-domain robustness. Specifically, we try the scheme from Kumar et al. (2022), where the authors first optimize the classifier while keeping the pretrained feature extractor frozen (linear probing), then optimize the entire network (fine-tuning). The intuition behind this method is that a good initialization from linear probing minimizes the chance of feature distortion, i.e. when the pretrained model overfits to in-domain data. We report the results in Table 17. In fact, we find that direct fine-tuning (FT) achieves slightly better performance on both in-domain Twitter data and out-of-domain news data (hNews). Thus, in other experiments we use direct fine-tuning.

Figure 4: ROC Curves by Expert vs. Joint Training (Section A.3.1). The model is trained on 1M samples with 75% hard negatives.

 



#### a.3.3 Training Set Size

We also investigate the influence of training set size on performance. We report the binary classification accuracy as we use 500k, 1M, and 2M samples, as seen in Table 18. We observe that increasing training data size generally leads to improved performance, with most of the gains coming from higher accuracy on hard negatives.

#### a.3.4 Tweet Text Clustering

We investigate the sub-topical clusters of the tweet text, and also evaluate the performance of the final fine-tuned model in terms of how well it performs on a set of the hard falsified samples and their corresponding pristine samples.

We use the method of (Grootendorst, 2020) to generate clusters, which entails computing SentenceBERT (Reimers and Gurevych, 2019) embeddings for each Tweet text, using UMAP (McInnes et al., 2020) to reduce the number of embedding dimensions from 768 to 20, and then running the HDBSCAN hierarchical clustering algorithm (McInnes and Healy, 2017) on the UMAP output. We compute the ten most important words for each cluster using the TF-IDF scores, and use this word list to gain insight into the concepts present in the texts of each cluster.

For UMAP we use the 10 nearest neighbors. For Climate Change HDBSCAN hyperparameters are: minimum topic size=400, and a cluster selection distance threshold = 0.56. For COVID-19 HDBSCAN: minimum topic size=1200, and cluster selection distance threshold = 0.65. For Military Vehicles HDBSCAN: minimum topic size = 100, cluster selection distance threshold = 0.60. The cluster selection size setting determines when clusters are merged, clusters within a smaller distance than this threshold setting will get merged together (see HDBSCAN(\(\hat{e}\)) parameter in section IV of (Malzer and Baum, 2020)).

As discussed in the main paper, we are interested in analyzing model performance on within-cluster vs. cross-cluster hard samples. First, the training data statistics per topic are presented in Table 19. Next, Figure 6 shows the ROC curves for the within-cluster and cross-cluster samples.

Figure 5: ROC Curves by Fine-Tuning Scheme (Section A.3.2), Percentage of Hard Negatives (Section 4.2), and Training Set Size (Section A.3.3) on the hNews set.

\begin{table}
\begin{tabular}{l|c c|c c|c c|c} \hline  & \multicolumn{3}{c|}{Cimate Change} & \multicolumn{3}{c|}{COVID-19} & \multicolumn{3}{c|}{Military Vehicles} & Nnews \\  & Random & Hard & Random & Hard & Random & Hard & \\ \hline LP & 0.9178 & 0.7548 & 0.9013 & 0.7359 & 0.9224 & 0.7071 & 0.5870 \\ LP+FT & **0.9346** & 0.7877 & 0.9195 & 0.7752 & 0.9440 & 0.7387 & 0.6188 \\ FT & 0.9344 & **0.7960** & **0.9247** & **0.7807** & **0.9440** & **0.7467** & **0.6339** \\ \hline \end{tabular}
\end{table}
Table 17: Balanced binary classification accuracy at EER by fine-tuning scheme. LP (linear probe) or FT (fine-tune) on 500k samples, 50% hard negatives.



To gain insight into the sub-topics, we concatenate the \(3\) top scoring words from each cluster to obtain the cluster "names", as seen in the Tables 20, 21, 22 with cluster names and word scores. We get between 20 and 30 clusters for each topic. We observe such sub-topics as ocean-sea-flood-flooding, plastic-recycling-recycle-sustainability for Climate Change, vaccine-vaccination-clinic-appointment, school-student-education-university for COVID-19, tank-abrams-army-m1, drone-ai-uav-drones for Military Vehicles. The hierarchy visualizations in Figures 7, 8, 9 provide further insight into the sub-topic structure.

 



\begin{table}
\begin{tabular}{l|p{142.3pt}} \hline Cluster Name & Word Scores \\ \hline
6\_china\_chinese\_carbon\_coal & (cinina, 0.23), (chinese, 0.04), (carbon, 0.02), (coal, 0.02), (world, 0.02), (beijing, 0.02), (ccp, 0.01), (emission, 0.01), (taiwan, 0.01), (emissions, 0.01) \\
7\_god\_pope church\_catholic & (god, 0.08), (pope, 0.05), (church, 0.04), ( catholic, 0.03), (religion, 0.03), (religious, 0.02), (christian, 0.02), (vatican, 0.02), (earth, 0.02), (bible, 0.01) \\
8\_hurricane\_harvey\_storm\_cyclone & (hurricane, 0.22), (harvey, 0.11), (storm, 0.09), (cyclone, 0.05), (hurricanes, 0.04), (tropical, 0.03), (irma, 0.02), (climatechange, 0.02), (stroms, 0.01), (hurricaneharycy, 0.01) \\
9\_island\_caribbean\_fiji\_maldives & (siland, 0.09), (caribbean, 0.06), (fiji, 0.04), (malloves, 0.02), (jamacia, 0.02), (islands, 0.02), (country, 0.02), (region, 0.02), (kiribati, 0.01), (figinews, 0.01), (figinews, 0.01), (figinews, 0.02), (tokechain, 0.10), (crowaut, 0.10), (cryptocurrency, 0.09), (green-crypto, 0.08), (bitcoin, 0.07), (btc, 0.06), (crypton, 0.06), (decentralization, 0.04), (climatechange, 0.02), (nfts, 0.02) \\
11\_space\_bezos\_nasa\_earth & (specac, 0.12), (bezos, 0.06), (nasa, 0.06), (earth, 0.04), (jeff, 0.03), (billionaire, 0.03), (musk, 0.02), (climatechange, 0.02), (elon, 0.01), (spacex, 0.01) \\
12\_migration\_displacement\_refugee\_displaced & (migration, 0.12), (displacement, 0.08), (refugee, 0.07), (dissplaced, 0.05), (refugees, 0.03), (migrant, 0.02), (climatechange, 0.02), (unherc, 0.02), (disasters, 0.01), (border, 0.01) \\
13\_ military\_threat\_pentagon\_dod & (military, 0.13), (threat, 0.05), (pentangon, 0.04), (dot, 0.03), (war, 0.02), (climatesecurity, 0.02), (army, 0.01), (mayv, 0.01), (millistarism, 0.01) \\
14\_indigenous\_indigenouspeopleS\_infigenous-peoplesday\_tribal & (miderosup, 0.03), (tribal, 0.03), (native, 0.02), (ribleo, 0.02), (iodiversity, 0.02), (indigenousday, 0.01), (culture, 0.01), (adaptation, 0.01) \\
15\_aviation\_flight\_plane\_airline & ( aviation, 0.09), (flight, 0.08), (plane, 0.06), (airline, 0.05), (flying, 0.04), (aiircraft, 0.03), (airplane, 0.02), (industry, 0.02), (elimestones, 0.01) \\
16\_olympics\_sort\_tokyo\_olympic & (olympics, 0.14), (sport, 0.09), (tokyo, 0.08), ( Olympic, 0.04), (tokyo2020, 0.03), (climatecomeback, 0.02), ( sports, 0.02), (clitacentechnage, 0.02), (rio2016, 0.01), (climatearcsin, 0.01) \\
17\_ai\_artificialintelligence\_machinelearning\_ intelligence & (ui, 0.26), (artificialinlineiligeence, 0.07), (ml, 0.05), (datascience, 0.05), (climatechange, 0.03), (nlp, 0.02), (sustainability, 0.02), (python, 0.01) \\
18\_nuclear\_nuclearenergy\_ uranium\_reactor & (nuclear, 0.33), (nuclearenergy, 0.04), (uranium, 0.03), (reactor, 0.03), (nuclearpower, 0.02), (electricity, 0.02), (timatechange, 0.02), (cleaninger, 0.01), (throshima, 0.01), (renewables, 0.01) \\
19\_moon\_orbit\_wobbble\_earth & (monon, 0.31), (orbit, 0.22), (wobble, 0.13), (earth, 0.07), (loading, 0.07), (nasa, 0.06), (congressman, 0.03), ( lunar, 0.02), (flood, 0.02), (wobbling, 0.02) \\
20\_air\_pollution\_cleanairday\_interp pollution & (air, 0.18), (pollition, 0.10), (cleanariday, 0.07), (airpollution, 0.07), (clearaintochearn, 0.03), (climatechange, 0.02), (breathe, 0.02), (delhi, 0.02), (smog, 0.02), (breathing, 0.01) \\ \hline \end{tabular}
\end{table}
Table 21: Cluster Names and Word Scores - COVID-19

\begin{table}
\begin{tabular}{l|p{142.3pt}} \hline Cluster Name & Word Scores \\ \hline
0\_coronavirus\_covid19\_India\_pandemic & (cornavirus, 0.02), (covid19, 0.01), (india, 0.01), (pandemic, 0.01), (corona, 0.01), (health, 0.01), (outbreak, 0.01), (news, 0.01), (hospital, 0.01), (uk, 0.01) \\ -1\_county\_covid19\_coronavirus\_health & (country, 0.01), (covid19, 0.01), (coronavirus, 0.01), (health, 0.01), (state, 0.01), (2021, 0.01), (pandemic, 0.01), (covid\_19, 0.01), (vaccination, 0.01), (death, 0.01) \\
1\_vaccine\_vaccination\_clinic\_appointment & (vaccine, 0.03), (vaccination, 0.03), (clinic, 0.02), (appointment, 0.02), (vaccinated, 0.01), (pfizer, 0.01), (walk, 0.01), (health, 0.01), (visit, 0.01), (shot, 0.01) \\
2\_ nigeria\_africa\_lagos\_nigerian & (nigeria, 0.04), (affrica, 0.03), (lagos, 0.02), (nigerian, 0.02), (sahara, 0.01), (uganda, 0.01), (buhai, 0.01), (aftican, 0.01), (ghanana, 0.01) \\ \hline \end{tabular}
\end{table}
Table 20: Cluster Names and Word Scores - Climate Change

* [16] J.-P., _et al._, (2013), (2014), (2015), (2017), (2018), 



\begin{table}
\begin{tabular}{l|p{142.3pt}} \hline Cluster Name & Word Scores \\ \hline
0\_flying\_aircraft\_helicopter\_aviation & ( flying, 0.02), (aircraft, 0.02), (helicopter, 0.02), (aviation, 0.01), (spitire, 0.01), (flight, 0.01), (raf, 0.01), (jet, 0.01), (plane, 0.01), (fly, 0.01) \\ l\_tank\_abrams\_army\_m1 & ( tank, 0.07), (abrams, 0.04), (army, 0.03), (ml, 0.02), (m1a2, 0.02), ( tanks, 0.01), (m1a1, 0.01), (armed, 0.01), (turret, 0.01), (armor, 0.01) \\ z\_rafe\_india\_frame\_iaf & (rafale, 0.08), (India, 0.04), (France, 0.03), (af, 0.03), (mirage2000, 0.02), (jet, 0.02), (jet, 0.02), (dassault, 0.02), (aimecraft, 0.02), (combat, 0.01), (greece, 0.01) \\ -1\_whatshappeninginnyanmar\_ military\_wa\_jet & (whashingtonynammar, 0.02), ( military, 0.02), (va, 0.02), (jet, 0.01), (carraff, 0.01), (helicopter, 0.01), (plane, 0.01), (landed, 0.01), (flight, 0.01) \\
3\_crashed\_jet\_plane\_plilot & (crashed, 0.05), (jet, 0.03), (plane, 0.03), (gilot, 0.03), (abija, 0.02), (killed, 0.02), (nigerian, 0.02), (airport, 0.02), (aircraft, 0.02), (niega, 0.02) \\
4\_syria\_iran\_syrian\_israel & (syria, 0.06), (iran, 0.04), (syrian, 0.04), (srisael, 0.02), (turkey, 0.02), (trussia, 0.02), (ryemen, 0.02), (libya, 0.01), (gaza, 0.01), (traq, 0.01) \\
5\_typhoon\_eurofighter\_raf\_aviation & (typhoon, 0.21), (erurofibter, 0.20), (raf, 0.03), ( Aviation, 0.02), (warlapempor, 0.02), (gt, 0.01), (dtwaffe, 0.01), (uiatiophotography, 0.01), (torado, 0.01), (squadron, 0.01) \\
6\_drone\_ai\_uaw\_drones & (drone, 0.18), (ai, 0.05), (tav, 0.04), ( drones, 0.03), (unmanned, 0.02), (tech, 0.02), (hecker, 0.02), (usas, 0.02), (intelligence, 0.01), (artificial, 0.01) \\
7\_whatshappenninginnyammar\_myam



\begin{table}
\begin{tabular}{l|p{142.3pt}} \hline Cluster Name & Word Scores \\ \hline
10\_ag600\_amplibious\_china\_flight & (ag600, 0.31), (amphibious, 0.29), (chita, 0.23), (flight, 0.08), (zhuhai, 0.05), (avic, 0.04), (qingdao, 0.03), (sinarcraft, 0.03), (shandong, 0.03), (gungadong, 0.03), (yangdong, 0.02) \\
11\_wallpaper\_f22\_aircraft\_eagle & (wallpaper, 0.49), (722, 0.07), (arircraft, 0.06), (eagle, 0.06), (walkpapers, 0.04), (falcon, 0.04), (walpaper, 0.04), (eurofighter, 0.03), (walpapers, 0.03), (backgrounds, 0.03) \\
12\_woman\_piot\_fly\_wwii & (woman, 0.15), (piplot, 0.08), (fly, 0.04), (wwii, 0.03), (airforce, 0.02), (gl pilots, 0.02), (lying, 0.02), (squadron, 0.02), (aircraft, 0.01), (flight, 0.01) \\
13\_mar23coup\_dawei\_thaung\_bike & (mar23coup, 0.23), (dawei, 0.23), (htaug, 0.22), (bike, 0.22), (road, 0.19), (whatshoppeninginmyanmar, 0.14), (death, 0.06), (dead, 0.01), (crimegasianthmanningum, 0.01), (veneed2ipinnyaman, 0.01) \\
14\_kenya\_ziphbabwe\_nigeria\_ghana & (kenya, 0.07), (ziembabwe, 0.07), (niegrain, 0.06), (ghana, 0.04), (bribery, 0.04), (bruizil, 0.03), (harare, 0.03), (chiopia, 0.02), (africa, 0.02), (buhari, 0.02), (qubitradig, 0.24), (stockmarket, 0.24), (stocks, 0.24), (investigating, 0.24), (astellites, 0.23), (investment, 0.23), (stock, 0.22), (boeing, 0.15), (shares, 0.04), (aurcfraft, 0.02) \\
16\_korea\_forean\_northkorea\_southkorea & (korea, 0.21), (Korean, 0.10), (northkorea, 0.06), (southkorea, 0.03), (war, 0.03), (nuclear, 0.02), (pyongyang, 0.02), (japan, 0.02), (seoul, 0.02) \\
17\_south\_korea\_russiana\_airspace & (south, 0.31), (Morea, 0.26), (Russian, 0.19), (airspace, 0.17), (korean, 0.14), (southkorea, 0.04), ( military, 0.03), (trsusia, 0.03), (seoul, 0.03), (ainecraft, 0.03) \\
18\_squawking\_circling\_mph\_mile & (squawking, 0.25), (circling, 0.24), (mpi, 0.08), (mile, 0.07), (creek, 0.07), (nsw, 0.06), (qld, 0.06), (county, 0.05), (marin, 0.03), (lake, 0.02) \\
19\_fl\_audi\_lap\_racing & (f1, 0.14), (audi, 0.05), (lap, 0.05), (racing, 0.04), (fermati, 0.03), (vetrel, 0.03), (melcaren, 0.03), (laps, 0.02), (traced, 0.02) \\
20\_concorde\_mach\_mph\_raf & (concorde, 0.44), (mand, 0.25), (mpi, 0.14), (raf, 0.11), (flying, 0.09), (rapidly, 0.06), (fuel, 0.06), (jet, 0.06), (supersonic, 0.05), (speed, 0.03) \\
21\_turkish\_naval\_seahawk\_escue & (turksis, 0.26), (naval, 0.17), (seahawk, 0.08), (rescue, 0.05), (hawk, 0.05), (turkey, 0.04), (sea, 0.04), (anatomical, 0.04), (total, 0.04), (quadrom, 0.04) \\
22\_actions\_feb21coup\_un\_whatshappeninglnmyanmar & (actions, 0.28), (few21coup, 0.23), (un, 0.23), (whatshopspeninglnmyanmar, 0.20), (news, 0.14), (terrorism, 0.08), (whatshopspeninginmyanmar, 0.01), (colombia, 0.01), (whatishappeninginmyanmar, 0.01), (coup, 0.00) \\ \hline \end{tabular}
\end{table}
Table 22: Cluster Names and Word Scores – Military

Figure 7: Cluster Hierarchy for Climate Change



Figure 8: Cluster Hierarchy for COVID-19



Figure 9: Cluster Hierarchy for Military Vehicles