SlowGuess/NAACL_2022_OCR huggingface dataset:
What are the potential dataset biases? Here, we focus on our method used to generate hard falsified samples to understand the potential biases learned during training. Specifically, we note potential age, race, and gender biases present in CLIP, the underlying model used to generate our mismatches. Radford et al. (2021) find the CLIP exhibits significant performance differences when classifying individuals of different races and ages into categories related to crime or animals. Agarwal et al. (2021) also find gender biases in the CLIP embeddings when classifying occupations. These biases primarily affect the synthetically generated training set, not the pristine data. However, we can not rule out that the pristine Twitter data may also capture some human biases or harmful stereotypes.
We would like to thank PAR Tech, Syracuse University, and the University of Colorado, Denver for creating the evaluation data. We thank the SRI team, including John Cadigan and Martin Graciarena, for providing the WikiData-sourced news organization Twitter handles. We would also like to thank Dong Huk (Seth) Park, Sanjay Subramanian, and Reuben Tan for helpful discussions on finetuning CLIP. This work was supported in part by DoD including DARPA's LwLL, and/or SemaFor programs, and Berkeley Artificial Intelligence Research (BAIR) industrial alliance programs.
Sandhini Agarwal, Gretchen Krueger, Jack Clark, Alec Radford, Jong Wook Kim, and Miles Brundage. 2021. Evaluating clip: Towards characterization of broader capabilities and downstream implications. arXiv:2108.02818.
Juan M. Banda, Ramya Tekumalla, Guanyu Wang, Jingyuan Yu, Tuo Liu, Yuning Ding, Ekaterina Artemova, Elena Tutubalina, and Gerardo Chowell. 2021. A large-scale pandemic-19 twitter chatter dataset for open scientific research—an international collaboration. Epidemiologia, 2(3):315-324.
Lisa Fazio. 2020. Out-of-context photos are a powerful low-tech form of misinformation. theconversation.com/out-of-context-photos-area-powerful-low-tech-form-of-misinformation-129959.
Maarten Grootendorst. 2020. Bertopic: Leveraging bert and c-tfidf to create easily interpretable topics.
Tamanna Hossain, Robert L. Logan IV, Arjuna Ugarte, Yoshitomo Matsubara, Sean Young, and Sameer Singh. 2020. COVID Lies: Detecting COVID-19 misinformation on social media. In Proceedings of the 1st Workshop on NLP for COVID-19 (Part 2) at EMNLP 2020, Online. Association for Computational Linguistics.
Ayush Jaiswal, Ekraam Sabir, Wael AbdAlmageed, and Premkumar Natarajan. 2017. Multimedia semantic integrity assessment using joint embedding of images and text. In Proceedings of the 25th ACM international conference on Multimedia.
Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. 2022. Fine-tuning can distort pretrained features and underperform out-of-distribution. In The International Conference on Learning Representations (ICLR).
Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. 2019. Quantifying the carbon emissions of machine learning. arXiv:1910.09700.
Justin Littman and Laura Wrubel. 2019. Climate Change Tweets Ids.
Grace Luo, Trevor Darrell, and Anna Rohrbach. 2021. Newsclippings: Automatic generation of out-of-context multimodal media. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.
Claudia Malzer and Marcus Baum. 2020. A hybrid approach to hierarchical density-based cluster selection. In 2020 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI), pages 223-228. IEEE.
Leland McInnes and John Healy. 2017. Accelerated hierarchical density based clustering. 2017 IEEE International Conference on Data Mining Workshops (ICDMW).
Leland McInnes, John Healy, and James Melville. 2020. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv:1802.03426.
Eric Müller-Budack, Jonas Theiner, Sebastian Diering, Maximilian Idahl, and Ralph Ewerth. 2020. Multi-modal analytics for real-world news using measures of cross-modal entity consistency. In ACM ICMR.
Parth Patwa, Shivam Sharma, Srinivas Pykl, Vineeth Guptha, Gitanjali Kumari, Md Shad Akhtar, Asif Ekbal, Amitava Das, and Tanmoy Chakraborty. 2021. Fighting an infodemic: Covid-19 fake news dataset. Communications in Computer and Information Science, page 21-29.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. arXiv:2103.00020.
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing.
Mingxing Tan and Quoc Le. 2019. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pages 6105-6114. PMLR.
Alakananda Vempala and Daniel Preoticiuc-Pietro. 2019. Categorizing and inferring the relationship between the text and image of Twitter posts. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2830-2840, Florence, Italy. Association for Computational Linguistics.
In Section A.1 we provide additional details about data collection, including our strategy and search keywords. Section A.2 provides dataset statistics, including information on tweet counts, geographical information, possibly sensitive content, and image availability. We include additional experiments in Section A.3.
A.1 Data Collection
A.1.1 COVID-19 and Climate Change
Our data collection consisted of three stages. The first employed simple topic, keyword, and hashtag filters, the second stage used more specific keyword and topic combinations, while the third focused on collecting topical data from Twitter accounts of various news organizations.
In the first stage we collected roughly 100,000 tweets each for COVID-19 and Climate Change topics. We used the "COVID-19" topic of the Twitter API's Entity Annotations feature<sup>8</sup>, which allows users to find tweets related to predetermined topics. For Climate Change we filtered with an OR clause on keywords "climate change", "global warming", and (#globalwarming, #climatechange) hashtags. Inspection of the stage 1 results revealed a lot of off-topic tweets. For example, a Twitter user might post a tweet about working from home during the pandemic and tag the tweet with a COVID-related hashtag. While this type of content is somewhat related to COVID-19, we wanted to focus on data where misinformation/disinformation might be more relevant, such as more topical/newsworthy tweets (e.g. bad actors may spread propaganda related to the COVID-19 pandemic by making false or misleading claims). To that end, in stage 2 we filtered by combining each topic phrase with one of the 19 topical search terms (e.g. "agriculture", "crops", "death", "vaccination"). The resulting data appeared much more relevant than the initial collection effort. Table 7 contains a list of the search terms we used to collect data for COVID-19 and Climate Change tweets. Finally, related to the argument above, in the third collection stage we focused on tweets authored by news organizations, as opposed to random users. For that, 7k news organization Twitter handles were sourced from WikiData<sup>9</sup>.
Table 7: Search Terms Used in Stage 2 of the Data Collection for COVID-19 and Climate Change
Search Terms
Agriculture, COVID, COVID-19, Climate Change, Crops, Death, Death Toll, Floods, Harvest, Hurricane, ICBM, Military, Military Parade, Military Vehicles, Show of Force, Tank, Troops, Typhoon, Vaccination
A.1.2 Military Vehicles
Collecting data about the Military Vehicles topic proved more challenging than the other two topics. We initially tried simple keyword filters such as "military", "aircraft", "tank", etc, but found that those resulted in a lot of irrelevant content such as tweets related to video games, or tweets where "tank" took a different meaning (e.g., "fish tank" or "tank tops"). This initial approach did not return many relevant results. The WikiData news organization approach used in the other two topics also did not provide enough usable data. As a result we crafted two different, highly customized stages for Military Vehicles. We gathered a list of both civilian and military vehicles and aircraft from eight different publicly available datasets (see Table 8). The datasets were annotated either for image classification or object detection tasks. We queried the Twitter Search API using the vehicle and aircraft names from this set, but returned a lot of off-topic data. We then trained an EfficientNet (Tan and Le, 2019) image classifier that categorized images as either civilian ground vehicle, civilian aircraft, military ground vehicle, military aircraft, or other. (The "other" category training set consisted of several thousand manually annotated images from the initial data collection effort that did not contain any military or civilian vehicles or aircraft.) We trained the classifier
<sup>8</sup>https://developer/twitter.com/en/docs/labs/annotations
<sup>9</sup>https://www.wikipidata.org/
to  $97\%$  accuracy and used it to filter out any tweets predicted to be in the "other" category. For the second collection stage we combined the military vehicle and aircraft names with custom keywords (Table 9).
Table 8: Datasets Used to Construct Civilian/Military Vehicle and Aircraft Classifier
Dataset	Source URL
Military Aircraft Detection Dataset	https://www.kaggle.com/a2015003713/militaryaircraftdetectiondataset
War Tanks Dataset	https://www.kaggle.com/icanerdogan/war-tank-images-dataset
Military Aircraft Dataset	https://github.com/tlkh/milair-dataset
Military Tanks Dataset	https://www.kaggle.com/antoreepjana/military-tanks-dataset-images
Military and Civilian Vehicles Classification Dataset	https://data.mendeley.com/datasets/njdjkbxdpn/1
Tau Vehicle Type Recognition	https://www.kaggle.com/c/vehicle/data?select=train
FGVC-Aircraft Benchmark	https://www.roots.ox.ac.uk/~vgg/data/fgvc-aircraft/
Stanford Cars Dataset	https://ai.stanford.edu/~jkrause/cars/car_dataset.html
Table 9: Additional Keywords used in Stage 2 Collection for Military Vehicles
Keywords
aircraft, airplane, army, battle, flying, military, soldiers, troops
Table 10: Full Dataset Summary
Topic	Tweets	Geo-tagged	Countries	Captions
COVID	569,982	4,637	112	569,982
Climate	212,665	3,696	138	212,662
Military	101,684	3,913	105	101,640
All	884,331	13,404	172	884,284
Table 10 shows a summary of the dataset. The "Geo-tagged" column refers to the geolocation data provided by tweet authors. This property is empty in most cases, but when present, can be in the form of a Twitter "place" which contains a display name, a geo polygon (which in some cases is as broad as an entire country), as well as other fields, such as country name. It is also possible for the geo data to be in the form of latitude and longitude, but that is rarer. The "Countries" columns is extracted from the geo location data and because of the small amount of geo-tagged tweets we can only report countries for a small fraction of tweets in the dataset (Table 11).
One oddity to note is that although we included an English-only search filter ("lang:en") in all API calls, the API still returned a small number of non-English tweets (Table 12). We are not sure why this is, but manual inspection of some of these examples shows that a good portion of them are in fact in English.
Figure 3: Word Cloud Summaries for Each Topic
Figure 3 shows high-level "word cloud" summaries for the hashtags in the tweets for each topic.
Table 14 shows the number of tweets that Twitter flagged as containing possibly sensitive material, i.e., samples that may contain adult content or graphic violence. We encourage users to be aware of such tweets, which account for about  $1\%$  of the data, and may be undesirable for certain downstream tasks.
Table 11: Totals by Country (Top 20 Only)
Country	Tweets	Geo-tagged	Captions
India	2,399	6,692	2,399
United Kingdom	2,127	6,020	2,127
United States	707	2,338	707
Canada	519	1,476	519
Australia	339	1,024	339
Pakistan	203	606	203
Germany	146	454	146
Kenya	130	360	130
Ireland	128	394	128
South Africa	118	342	118
Nigeria	116	352	116
Uganda	115	298	115
Republic of the Philippines	107	320	107
France	100	298	100
The Netherlands	95	290	95
Indonesia	81	238	81
Malaysia	77	224	77
Spain	75	194	75
New Zealand	68	212	68
Belgium	67	182	67
Table 12: Totals by Language
Language	Tweets	Total geo-tagged	Countries	Unique Captions
English	883,310	9,268	172	883,263
Non-English	618	3	3	618
Table 13: Totals for Country  $=$  "US",by Topic
Topic	Tweets	geo-tagged	Countries
Military Vehicles	705	705	1
COVID-19	0	0	0
Climate Change	2	2	1
Table 14: Possibly Sensitive Tweets
Poss. Sensitive	Tweets	% of Total
True	9,151	1.03
False	875,180	98.97
Table 15: Media Summary
Total Images	Tweets
1,039,296	884,331
Table 16: Distribution of # of Media Items per Tweet
# Media in Tweet	Tweets	% of Total
1	801,764	90.6%
2	36,969	4.2%
3	18,803	2.1%
4	26,795	3.0%
The total number of images/tweets is shown in Table 15. Twitter allows users to include 1-4 images in a tweet. As seen in Table 16,  $90\%$  of the tweets have a single image. In cases where a tweet contained
more than one image, we only used the first image (according to the order of the images returned by the Twitter API).
A.3 Additional Experiments
All experiments reported in this paper are for a single run, as we find that variance across multiple runs is low. All ROC curves and metrics are computed using sklearn's roc Curve function. All models are implemented in PyTorch. For our experiments, we make the following design choices:
- We use the RN50x16 backbone. We find that this backbone consistently yields a  $2 - 3\%$  improvement compared to other released backbones, such as ViT/B-32. Our final CLIP model contains  $\sim 300\mathrm{M}$  parameters initialized from the RN50x16 backbone and  $\sim 600\mathrm{k}$  parameters randomly initialized for our classifier.
- We tune the upper layers and keep CLIP's lower layers frozen<sup>10</sup>. We find that this scheme is more memory efficient and yields more stable convergence than tuning all the layers.
- We use a learning rate of 5e-08 for CLIP and 5e-05 for the classifier. From our hyperparameter sweeps we find this setting to be the most appropriate, as CLIP is pretrained while the classifier is randomly initialized.
- We multiply CLIP image and text embeddings before passing that as an input to the classifier. This is different from Luo et al. (2021), who used a simple feature concatenation.
A.3.1 Expert vs. Joint Training
Here we study whether training a joint model on all three topics at once may be inferior to training three topic-specific experts, see Figure 4. We find that the joint model performs on par with or better than the expert models, thus we use a joint model in all the other experiments.
Figure 4: ROC Curves by Expert vs. Joint Training (Section A.3.1). The model is trained on 1M samples with  $75\%$  hard negatives.
A.3.2 Fine-Tuning Scheme
Since we only know the high-level topics but not the precise composition of samples in our hidden set hTwitter, we investigate methods for out-of-domain robustness. Specifically, we try the scheme from (Kumar et al., 2022), where the authors first optimize the classifier while keeping the pretrained feature extractor frozen (linear probing), then optimize the entire network (fine-tuning). The intuition behind this method is that a good initialization from linear probing minimizes the chance of feature distortion, i.e. when the pretrained model overfits to in-domain data. We report the results in Table 17. In fact, we find that direct fine-tuning (FT) achieves slightly better performance on both in-domain Twitter data and out-of-domain news data (hNews). Thus, in other experiments we use direct fine-tuning.
10We fine-tune the layers "visual(layer4", "visual.attnpool", "transformer.resblocks.11", "In_final", "text_projection", "logit_scale".
Figure 5: ROC Curves by Fine-Tuning Scheme (Section A.3.2), Percentage of Hard Negatives (Section 4.2), and Training Set Size (Section A.3.3) on the hNews set.
Table 17: Balanced binary classification accuracy at EER by fine-tuning scheme. LP (linear probe) or FT (fine-tune) on 500k samples,  ${50}\%$  hard negatives.
Climate Change	COVID-19	Military Vehicles	hNews
Random	Hard	Random	Hard	Random	Hard
LP	0.9178	0.7548	0.9013	0.7359	0.9224	0.7071	0.5870
LP+FT	0.9346	0.7877	0.9195	0.7752	0.9440	0.7387	0.6188
FT	0.9344	0.7969	0.9247	0.7807	0.9440	0.7467	0.6339
A.3.3 Training Set Size
We also investigate the influence of training set size on performance. We report the binary classification accuracy as we use 500k, 1M, and 2M samples, as seen in Table 18. We observe that increasing training data size generally leads to improved performance, with most of the gains coming from higher accuracy on hard negatives.
Table 18: Balanced binary classification accuracy at EER by training set size. FT on varying number of samples,  $75\%$  hard negatives.
Climate Change	COVID-19	Military Vehicles	hNews
Random	Hard	Random	Hard	Random	Hard
500k	0.9356	0.7979	0.9241	0.7809	0.9410	0.7470	0.6586
1M	0.9350	0.8055	0.9270	0.7909	0.9480	0.7595	0.6741
2M	0.9348	0.8104	0.9266	0.7927	0.9475	0.7696	0.6844
A.3.4 Tweet Text Clustering
We investigate the sub-topical clusters of the tweet text, and also evaluate the performance of the final finetuned model in terms of how well it performs on a set of the hard falsified samples and their corresponding pristine samples.
We use the method of (Grootendorst, 2020) to generate clusters, which entails computing SentenceBERT (Reimers and Gurevych, 2019) embeddings for each Tweet text, using UMAP (McInnes et al., 2020) to reduce the number of embedding dimensions from 768 to 20, and then running the HDBSCAN hierarchical clustering algorithm (McInnes and Healy, 2017) on the UMAP output. We compute the ten most important words for each cluster using the TF-IDF scores, and use this word list to gain insight into the concepts present in the texts of each cluster.
For UMAP we use the 10 nearest neighbors. For Climate Change HDBSCAN hyperparameters are: minimum topic size  $= 400$ , and a cluster selection distance threshold  $= 0.56$ . For COVID-19 HDBSCAN: minimum topic size  $= 1200$ , and cluster selection distance threshold  $= 0.65$ . For Military Vehicles HDBSCAN: minimum topic size  $= 100$ , cluster selection distance threshold  $= 0.60$ . The cluster selection size setting determines when clusters are merged, clusters within a smaller distance than this threshold setting will get merged together (see HDBSCAN( $\hat{\epsilon}$ ) parameter in section IV of (Malzer and Baum, 2020)).
As discussed in the main paper, we are interested in analyzing model performance on within-cluster vs. cross-cluster hard samples. First, the training data statistics per topic are presented in Table 19. Next, Figure 6 shows the ROC curves for the within-cluster and cross-cluster samples.
Table 19: Training set statistics by topic, by sample_type
topic	sample_type	Total	% of Topic Total	# cross_cluster	% cross_cluster
COVID-19	pristine	736,539	50.00	0	0.00
hard	574,129	38.97	334,541	58.27
random	162,410	11.03	129,623	79.81
Climate Change	pristine	298,809	50.00	0	0.00
hard	214,377	35.87	109,984	51.30
random	84,432	14.13	56,035	66.37
Military Vehicles	pristine	139,213	50.00	0	0.00
hard	103,837	37.29	40,797	39.29
random	35,376	12.71	26,947	76.17
(b)
Figure 6: ROC curves on sets of pristine + falsified (hard-only) pairs, grouped by whether or not the falsified text fell within the same cluster ("within-cluster") or in a different cluster ("cross-cluster").
To gain insight into the sub-topics, we concatenate the 3 top scoring words from each cluster to obtain the cluster "names", as seen in the Tables 20, 21, 22 with cluster names and word scores. We get between 20 and 30 clusters for each topic. We observe such sub-topics as ocean-sea-flood-flooding, plastic-recycling-recycle-sustainability for Climate Change, vaccine-vaccination-clinic-appointment, school-student-education-university for COVID-19, tank-abrams-army-m1, drone-ai-uav-drones for Military Vehicles. The hierarchy visualizations in Figures 7, 8, 9 provide further insight into the sub-topic structure.
Table 20: Cluster Names and Word Scores - Climate Change
Cluster Name	Word Scores
0_climatechange_world_report_energy	(climatechange, 0.02), (world, 0.01), (report, 0.01), (energy, 0.01), (warming, 0.01), (environment, 0.01), (climatecrisis, 0.01), (change, 0.01), (read, 0.01), (weather, 0.01)
-1_climatechange_world_climatecrisis_globalwarming	(climatechange, 0.02), (world, 0.01), (climatecrisis, 0.01), (globalwarming, 0.01), (co2, 0.01), (environment, 0.01), (warming, 0.01), (2021, 0.01), (sustainability, 0.01), (climateaction, 0.01)
1_environment_nature_climatechange_biodiversity	(environment, 0.02), (nature, 0.02), (climatechange, 0.02), (biodiversity, 0.02), (agriculture, 0.02), (plant, 0.02), (worldenvvironmentday, 0.01), (earth, 0.01), (sustainability, 0.01), (ecosystem, 0.01)
2_ocean_sea_flood_flooding	(ocean, 0.04), (sea, 0.03), (flood, 0.02), (flooding, 0.02), (climate-change, 0.02), (coral, 0.01), (reef, 0.01), (coastal, 0.01), (warming, 0.01), (oceans, 0.01)
3_arctic/glacier_antarctica_antarctic	(arctic, 0.10), (glacier, 0.04), (antarctica, 0.04), (antarctic, 0.03), (warming, 0.03), (permafrost, 0.02), (climatechange, 0.02), (globalwarming, 0.01), (snow, 0.01), (climatecrisis, 0.01)
4 transportingvehicle_transportation_electricvehicles	(transport, 0.04), (vehicle, 0.04), (transportation, 0.03), (electricvehicles, 0.01), (electricvehicle, 0.01), (electrification, 0.01), (mobility, 0.01), (diesel, 0.01), (pollution, 0.01), (emissions, 0.01)
5_plastic_recycling_recycle_sustainability	(plastic, 0.10), (recycling, 0.04), (recycle, 0.03), (sustainability, 0.03), (sustainable, 0.02), (ecofriendly, 0.02), (plasticpollution, 0.02), (plasticfree, 0.02), (plasticfreejuly, 0.01), (plastics, 0.01)
Continued on next page
Table 21: Cluster Names and Word Scores - COVID-19
Table 21: Cluster Names and Word Scores - COVID-19
Cluster Name	Word Scores
24_hawaiian_hinews_bigislandnews_hawaiinews	(hawaii, 0.22), (hinews, 0.12), (bigislandnews, 0.11), (hawaiinews, 0.11), (island, 0.05), (hawaiicountynews, 0.04), (oahu, 0.04), (hon-olulu, 0.03), (maui, 0.02), (kaua, 0.02)
25_cruiseships_navy_seafarer	(cruise, 0.16), (ship, 0.12), (navy, 0.04), (seafarer, 0.03), (sailor, 0.03), (caribbean, 0.03), (vessel, 0.02), (ferry, 0.02), (maritime, 0.01), (carnival, 0.01)
26_iran_iranian-coronavirus_tehran	(iran, 0.30), (iranian, 0.09), (coronavirus, 0.04), (tehran, 0.03), (khamenei, 0.03), (country, 0.02), (covid19, 0.02), (vaccine, 0.02), (covid_19, 0.01), (azerbaijan, 0.01)
27_italy_italian_rome_covid_19	(italy, 0.24), (italian, 0.07), (rome, 0.02), (covid_19, 0.02), (coronavirus, 0.01), (sur, 0.01), (covid19, 0.01), (country, 0.01), (france, 0.01), (lombardy, 0.01)
28_singapore Imported_infection_singaporean	(singapore, 0.37), (imported, 0.07), (infection, 0.06), (singaporean, 0.03), (changi, 0.03), (dorm, 0.02), (airport, 0.02), (dormitory, 0.01), (transmitted, 0.01), (ttsh, 0.01)
29_differient_deltavariant_vaccinated	(diffusion, 0.26), (variant, 0.21), (deltavariant, 0.03), (vaccinated, 0.02), (vaccine, 0.01), (surge, 0.01), (unvaccinated, 0.01), (covid19, 0.01), (variants, 0.01), (deltaplus, 0.01)
30_jamaica_glnrtoday-coronameter_jamaican	(jamaica, 0.17), (glnrtoday, 0.05), (coronameter, 0.05), (jamaican, 0.04), (hospitalised, 0.04), (barbados, 0.02), (recoveries, 0.02), (died, 0.02), (glnroped, 0.02), (investigation, 0.02)
31_app_tracing_covidalert_tracer	(app, 0.19), (tracing, 0.12), (covidalert, 0.06), (tracer, 0.03), (apps, 0.03), (privacy, 0.02), (tracking, 0.02), (covid19, 0.02), (google, 0.01), (trace, 0.01)