SlowGuess/NAACL_2022_OCR huggingface dataset:
Kirk Roberts, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman, Kyle Lo, Ian Soboroff, Ellen Voorhees, Lucy Lu Wang, and William R Hersh. 2020. Trec-covid: rationale and structure of an information retrieval shared task for covid-19. Journal of the American Medical Informatics Association, 27(9):1431-1436.
Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: BM25 and beyond. Now Publishers Inc.
Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulieu, Mike Gatford, et al. 1995. Okapi at trec-3. Nist Special Publication Sp, 109:109.
Gautam Kishore Shahi, Anne Dirkson, and Tim A Majchrzak. 2021. An exploratory study of COVID-19 misinformation on twitter. Online social networks and media, 22:100104.
Kai Shu, Deepak Mahudeswaran, Suhang Wang, Dongwon Lee, and Huan Liu. 2020. Fakenewsnet: A data repository with news content, social context, and spatiotemporal information for studying fake news on social media. *Big data*, 8(3):171-188.
Amir Soleimani, Christof Monz, and Marcel Worring. 2020. Bert for evidence retrieval and claim verification. Advances in Information Retrieval, 12036:359.
Dominik Stammbach and Guenter Neumann. 2019. Team domlin: Exploiting evidence enhancement for the fever shared task. In Proceedings of the Second Workshop on Fact Extraction and VERIFICATION (FEVER), pages 105-109.
Motoki Taniguchi, Tomoki Taniguchi, Takumi Takahashi, Yasuhide Miura, and Tomoko Ohkuma. 2018. Integrating entity linking and evidence ranking for fact extraction and verification. In Proceedings of the First Workshop on Fact Extraction and Verification (FEVER), pages 124-126.
The Lancet COVID-19 content collection. 2020. The lancet COVID-19 content collection. https://www.thelancet.com/coronavirus/collection.
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018a. FEVER: a large-scale dataset for fact extraction and VERIFICATION. In NAACL-HLT.
James Thorne, Andreas Vlachos, Oana Cocarascu, Christos Christodoulopoulos, and Arpit Mittal. 2018b. The FEVER2.0 shared task. In Proceedings of the Second Workshop on Fact Extraction and VERIFICATION (FEVER).
Ellen Voorhees, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman, William R Hersh, Kyle Lo, Kirk Roberts, Ian Soboroff, and Lucy Lu Wang. 2021. Trec-covid: constructing a pandemic information retrieval test collection. In ACM SIGIR Forum, volume 54, pages 1-12. ACM New York, NY, USA.
David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020. Fact or fiction: Verifying scientific claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7534-7550.
Lucy Lu Wang, Kyle Lo, Yoganand Chandrasekhar, Russell Reas, Jiangjiang Yang, Doug Burdick, Darrin Eide, Kathryn Funk, Yannis Katsis, Rodney Michael Kinney, Yunyao Li, Ziyang Liu, William Merrill, Paul Mooney, Dewey A. Murdick, Devvret Rishi, Jerry Sheehan, Zhihong Shen, Brandon Stilson, Alex D. Wade, Kuansan Wang, Nancy Xin Ru Wang, Christopher Wilhelm, Boya Xie, Douglas M. Raymond, Daniel S. Weld, Oren Etzioni, and Sebastian Kohlmeier. 2020a. CORD-19: The COVID-19 open research dataset. In Proceedings of the 1st Workshop on NLP for COVID-19 at ACL 2020, Online. Association for Computational Linguistics.
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020b. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. arXiv preprint arXiv:2002.10957.
William Yang Wang. 2017. "liar, liar pants on fire": A new benchmark dataset for fake news detection. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 422-426.
Ralph Weischedel, Martha Palmer, Mitchell Marcus, Edward Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, et al. 2013. Ontonotes release 5.0 ldc2013t19. Linguistic Data Consortium, Philadelphia, PA, 23.
WHO database of publications on coronavirus. 2020. Who database of publications on coronavirus. https://search.bvsalud.org/global-literature-on-novel\coronavirus-2019-ncov/.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122. Association for Computational Linguistics.
Wei Yang, Kuang Lu, Peilin Yang, and Jimmy Lin. 2019. "Critically examining the" neural hype" weak baselines and the additivity of effectiveness gains from neural ranking models. In Proceedings of the 42nd international ACM SIGIR conference on research and development in information retrieval, pages 1129-1132.
Takuma Yoneda, Jeff Mitchell, Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018. Ucl machine reading group: Four factor framework for fact finding
(hexaf). In Proceedings of the First Workshop on Fact Extraction and VERIFICATION (FEVER), pages 97-102.
Xia Zeng, Amani S Abumansour, and Arkaitz Zubiaga. 2021. Automated fact-checking: A survey. Language and Linguistics Compass, 15(10):e12438.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations.
Wanjun Zhong, Jingjing Xu, Duyu Tang, Zenan Xu, Nan Duan, Ming Zhou, Jiahai Wang, and Jian Yin. 2020. Reasoning over semantic-level graph for fact checking. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6170-6180.
Jie Zhou, Xu Han, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. 2019. Gear: Graph-based evidence aggregating and reasoning for fact verification. In The 57th annual meeting of the Association for Computational Linguistics (ACL).
Xinyi Zhou, Apurva Mulay, Emilio Ferrara, and Reza Zafarani. 2020. Recovery: A multimodal repository for Covid-19 news credibility research. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pages 3205-3212.
A Data Sources
Here we present detailed information of the data sources introduced in section 3.1.
It is worth noting that for the construction of our dataset, we have only included sources or datasets that contain explicit veracity labels of specific claims, thus we have not included collections of tweets related to COVID that do not have veracity labels (Chen et al., 2020; Lamsal, 2021; AbdulMageed et al., 2021; Huang et al., 2020; Dimitrov et al., 2020; Kerchner and Wrubel, 2020; Qazi et al., 2020). We have not included claims without independent fact-checking sources (Memon and Carley, 2020; Shahi et al., 2021) and information sources without formulated claims such as the collections of scholarly articles (Wang et al., 2020a; Chen et al., 2021), news articles (Zhou et al., 2020), or articles obtained through specific repositories as (COVID-19 Data Portal, EU; WHO database of publications on coronavirus; Elsevier journals Novel Coronavirus Information Center; Cambridge journals Coronavirus Free Access Collection; The Lancet COVID-19 content collection; Oxford journals resources on COVID-19; MedRN medical research network SSRN Coronavirus Infectious Disease Research Hub).
The data sources that we have used for the construction of our dataset are:
- The CoronaVirusFacts/DatosCoronaVirus Alliance Database<sup>11</sup>. Published by Poynter<sup>12</sup>, this online publication combines fact-checking articles from more than 100 fact-checkers from all over the world, being the largest journalist fact-checking collaboration on the topic worldwide<sup>13</sup>. The publication is presented as an online portal, thus we had to develop scripts to crawl the content and extract the relevant claims, categories, and information sources.
- CoAID dataset<sup>14</sup>. The dataset (Cui and Lee, 2020) contains fake news from fact-checking websites and real news from health information websites, health clinics, and public institutions. Unlike most other datasets, it contains a wide selection of true claims.
- MM-COVID<sup>15</sup>. The multilingual dataset (Li et al., 2020) contains fake and true news collected from Poynter and Snopes<sup>16</sup>, being a good complement to the first data source.
- CovidLies dataset<sup>17</sup>. The dataset (Hossain et al., 2020) contains a curated list of common misconceptions about COVID appearing in social media, carefully reviewed to contain very relevant and unique claims unlike other automatically collected datasets.
- TREC Health Misinformation track<sup>18</sup>. Research challenge using claims on the health domain focused on information retrieval from general websites through the Common Crawl corpus<sup>19</sup>. This dataset is specialized in a very specific domain, and has been used for a very different application than the previous data sources.
- TREC COVID challenge $^{20}$ . Research challenge (Voorhees et al., 2021; Roberts et al., 2020) using claims on the health domain focused on information retrieval from scholarly peer-reviewed journals through the CORD19 dataset (Wang et al., 2020a), the largest existing compilation of such articles. Similar to the last source, but focused on scholarly papers unlike the other sources.
A.1 Pre-processing
A separate pre-processing step was carried out for each of the selected data sources:
The CoronaVirusFacts/CoronaVirus Alliance Database. The data was downloaded on 13 February 2021. From the 11,647 entries initially obtained, entries with no fact-checking source and categories with less than 10 entries were removed. The different fact-checkers used different categories to label the claims, although in most of the cases the difference was mainly in terms of spelling. Initially we identified the following common categories: False (including FALSE, FALSO, Fake, false, false and misleading, Two Pinocchios, Misinformation
11https://www.poynter.org/ ifcn-covid-19-misinformation/ 12www.poynter.org 13https://www.poynter.org/ coronavirusfactsalliance/ 14https://github.com/cuilimeng/CoAID
15 https://github.com/bigheiniu/MM-COVID   
16 www.snopes.com   
17 https://github.com/ucinlp/   
covid19-data   
18 https://trec-health-misinfo.github.   
io/   
19 https://commoncrawl.org/   
20 https://ir.nist.gov/covidSubmit/data.   
html
/ Conspiracy theory, Not true, false headline, MANIPULATED, Unproven), Misleading (MISleading, MISLEADING, misLEADING, MiSLEADING, misleading, Misleading/False, Misleading), Missing Context (Missing context, Needs Context, missing context), No Evidence (NO EVIDENCE, No evidence, No Evidence), Mostly False (Mostly False, Mostly false, MOSTLY FALSE, mostly false, Mainly false), Partially False (Partially False, Partly false, Partially false, partially false, partly false, partially false, Partially True (PARTLY TRUE, Partially correct, Partially true, Partly true, HALF TRUE, HALF TRUTH, half true), and Mostly True (Mostly true, MOSTLY TRUE, mainly correct). Next, we conducted a manual inspection of the different categories. We found that the categories Misleading, Missing Context, No Evidence, Mostly False, and Partially False had no homogeneous and clear definition through the different fact-checking media. Each group contains claims fitting the definition mixed with claims that are simply false (e.g. of false claims under other labels: "Misleading: Only people from South Korea have Covid-19 antibodies"; "Partially False: The vaccines contain substances such as arsenic or uranium according to scientific studies"; "Mostly False: Pope Francis contracted coronavirus"). Therefore, we decided to not use these nuanced categories but group them in the general False category. Additionally, we removed the 25 claims from the categories Partially True and Mostly True, since they contained both True and False claims.
CoAID dataset. The datasets
NewsRealCOVID-19, NewsFakeCOVID-19, and ClaimFakeCOVID-19 were selected. The additional available dataset contains claims already existing in other datasets, formulated in this case as questions, and thus was not included. The selected datasets contain True and False claims.
MM-COVID. The claims were obtained from the English_news part of the dataset since we are only interested in English claims. 3,409 claims were collected. Claims in other languages appeared in the file, therefore we did a language filtering using polyglot $^{21}$ . Additionally, claims without fact-checking sources were deleted. It contains True and False claims.
CovidLies dataset. The available claims have been manually revised by eliminating duplicates, resulting in a total of 62 misconception claims. It contains False claims.
TREC Health Misinformation track. The claims used in the track were obtained and reformulated manually by us as affirmative claims (e.g., "Can vitamin D cure COVID-19?" was changed to "Vitamin D cures COVID-19") for consistency with the rest of the data sources and to allow claim veracity assessment. True and False claims are used.
TREC COVID challenge. The claims used in the challenge were obtained and reformulated manually by us as full sentences using the explanations related to each query (e.g., for a given query "coronavirus immunity", and its explanation "will SARS-CoV2 infected people develop immunity?"), we form the following claim, "coronavirus infected people develop immunity"). True and False claims are used.
The above processed data sources were combined to provide 20,689 initial claims.
A.2 Claim Categorisation
The claims were analysed to identify types of claims that may be of particular interest, either for inclusion or exclusion depending on the type of analysis. The following types were identified: (1) Multimodal; (2) Social media references; (3) Claims including questions; (4) Claims including numerical content; (5) Named entities, including: PERSON - People, including fictional; ORGANIZATION - Companies, agencies, institutions, etc.; GPE - Countries, cities, states; FACILITY - Buildings, highways, etc. These entities have been detected using a RoBERTa base English model (Liu et al., 2019) trained on the OntoNotes Release 5.0 dataset (Weischedel et al., 2013) using Spacy $^{22}$ .
B Parameter Setting
In our veracity assessment experiments, the parameters of the initial RoBERTa models are frozen during the training. The inputs are padded and truncated to the longest sequence, and a ReLU function is used as the activation function for the hidden layer. The GCNConv outputs are padded to the longest graph size. The loss function used is cross-entropy. The size of the hidden layer is 50, the batch size is 30, and the training is performed for 100 epochs for NLI-SAN and its variants, and 200 epochs for NLI-graph and its variants. The optimizer used is AdamW (Loshchilov and Hutter, 2019) with  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.999$ , a weight decay of 0.01, and a learning rate of  $10^{-2}$  for NLI,  $10^{-4}$  for NLI+Sent and NLI-SAN, and
$^{21}$ https://github.com/aboSamoor/polyglot
$^{22}$ https://spacy.io/
Claim	Description
"Sugar causes a cytokine storm in the lungs that promotes COVID-19"	Retrieved documents are relating COVID and its cytokine storm effects, but without the specific mention of sugar, which does not cause a cytokine storm.
"Barron Trump had COVID-19, Melania Trump says"	Retrieved sentences such as "Rudy Giuliani has tested positive for COVID-19, Trump says." with a similar structure and mentions but mistaking the family members and missing the key name.
"Prince Charles tested positive for COVID-19 after meeting Bol-lywood singer Kanika Kapoor."	Documents mentioning Prince Charles positive COVID tests are obtained, but without any mentions to the singer.
"Vice President of Bharat Biotech got a shot of the indigenous COVAXIN vaccine"	Correct documents on the issue are retrieved. Similar sentences are retrieved such as "Covaxin which is being developed by Bharat Biotech is the only indigenous vaccine that is approved for emergency use." or "Bharat Biotech's Covaxin is the first Indian vaccine to receive approval to conduct Phase I/Phase II trials." However, being similar they give no information about the claimed situation. In the retrieved document, the sentence "The pharmaceutical company, has in a statement, denied the claim and said the image shows a routine blood test." contains the essential information to debunk the original claim. But it is missed by the sentence retrieval engine as it is very different from the claim.
"Masks can be sanitized in microwave"	Correct documents are retrieved with similar sentences such as "Claiming masks can be sanitized in microwave resurfaces". However, sentences such as "The study authors cautioned health care workers against trying to clean masks this way. Microwaves melted the masks, making them useless." or "He also warns people against using microwaves or ovens to heat their masks." that are present in the retrieved documents but are not similar enough to the claim are missed.
Table A1: Examples of errors in document or sentence retrieval.
a learning rates of  $10^{-4}$  for NLI-graph,  $10^{-3}$  for NLI-graph\_abl, and  $10^{-5}$  for NLI+Psent, these last three with a step size of 0.1 after 100 epochs.
C Additional Examples of Document or Sentence Retrieval Errors
Here we expand on the examples mentioned in Section 5.3 related to difficulties in the document or sentence retrieval parts of the process. Table A1 presents in more detail the cases previously mentioned, and includes new examples.