SlowGuess/NAACL_2022_OCR huggingface dataset:
where  $\mathbf{X}$  is the matrix of node feature vectors,  $\hat{\mathbf{A}} = \mathbf{A} + \mathbf{I}$  denotes the adjacency matrix with inserted self-loops,  $\hat{D}_{ii} = \sum_{j=0}^{n} \hat{A}_{ij}$  its diagonal degree matrix, and  $\mathbf{W}$  is a trainable weight matrix.
Once the node representation is updated via GCN, all the node representations are averaged
and passed to the MLP and the Softmax layer to generate the final veracity classification output.
$$
\hat {\boldsymbol {y}} = \operatorname {s o f t m a x} \left(\mathrm {M L P} _ {\text {R e L U}} \left(\mathrm {O} _ {\text {g r a p h}}\right)\right) \tag {7}
$$
In this section, we perform a twofold evaluation: We first evaluate our document retrieval methods (presented in §4.1) on obtaining information relevant to the dataset claims from a database of COVID-19 related websites. We subsequently present an evaluation of the veracity assessment approaches for the claims (described in §4.3).
5.1 Document Retrieval
In order to evaluate our document retrieval methods, we need the gold-standard relevant document for each claim. Therefore, in the documents dataset described in section 4.1 we additionally include the web content referenced in each of the information sources used to compile our claim dataset:
The CoronaVirus Alliance Database. All web pages from the websites referenced as fact-checking sources for the claims have been downloaded from 151 different domains.
CoAID dataset. We downloaded the websites used as fact-checking sources of false claims and the websites where correct information on true claims is gathered from 68 different domains.
MM-COVID. We collected both fact-checking sources and reliable information related to the claims of this dataset from 58 web domains.
CovidLies dataset. We include the web content used as fact-checking sources of the misconceptions from 39 domains.
We have not included web content from the TREC Challenges, as each of them is performed on a very large dataset specific to each challenge (CORD19 and Common crawl corpus), as explained previously. Note that in our subsequent experiments, we have excluded all fact-checking websites to avoid finding directly the claim references. The results of the document retrieval are presented in Table 5. For each claim, the precision@  $k$  is defined as 1 if the relevant result is retrieved in the top  $k$  list and 0 otherwise.
We can see that by using BM25, it is possible in many cases to retrieve the relevant results at the very top of our searches. Combining BM25 with MonoBERT did not offer any improvement. It even introduced noise to the retrieval results, leading to inferior performance compared to using BM25 only on AP@5 and AP@10. MonoT5 appears
AP@5	AP@10	AP@20	AP@100
BM25	0.54	0.56	0.58	0.62
BM25+MonoBERT	0.52	0.55	0.58	0.62
BM25+MonoT5	0.55	0.58	0.60	0.62
BM25+RM3+MonoT5	0.51	0.53	0.55	0.57
Table 5: Document retrieval results. Average precision for different cut-offs. For the MonoBERT and MonoT5 cases, 100 initial results are retrieved in the first retrieval stage before re-ranking.
to be more effective, consistently improving the retrieval results across all metrics. Moreover for this dataset the use of query expansion using RM3 pseudo-relevance feedback on the BM25 results does not improve the results.
5.2 Veracity Assessment Evaluation
Here we evaluate our proposed NLI-SAN and NLI-graph veracity assessment approaches. To gain a better insight into the benefits of the proposed architectures, we conducted additional experiments on the variants of the models including:
- NLI, using only the NLI outputs of the claim-evidence pairs. The outputs are concatenated and then passed through the final classification layer to generate veracity classification results.
- NLI+sent, this is the ablated version of NLI-SAN without the self-attention layer. Here, the RoBERTa-encoded claim-evidence representations are concatenated with the NLI results and then fed to the classification layer to produce the veracity classification output.
- NLI+PSent, this is similar to the previous ablated version, but using the pooled representation of the claim-evidence pair to concatenate with the NLI result.
- NLI-graph-abl, this is the ablated version of NLI-graph in which the node representation is the NLI result of the corresponding claim-evidence pair without its RoBERTa-encoded representation.
For NLI, NLI+sent and NLI-SAN, we consider the 5 most similar sentences for each claim, obtained from the 10 most relevant documents of the information source database. Those documents are retrieved using BM25 and MonoT5 re-ranking of the top 100 BM25 results. For NLI-graph, NLI-graph\_abl and NLI+PSent, in order to have enough nodes to benefit from the network structure, the number of retrieved sentences is increased to 30 for each claim, selected as the 3
Model	False	True	Macro F1
Precision	Recall	F1	Precision	Recall	F1
GEAR (Zhou et al., 2019)	0.81	0.60	0.69	0.85	0.94	0.89	0.79
KGAT (Liu et al., 2020)	0.89	0.96	0.92	0.98	0.95	0.97	0.94
NLI	0.48	0.24	0.31	0.75	0.90	0.82	0.56
NLI+Sent	0.91	0.87	0.89	0.95	0.97	0.96	0.92
NLI+PSent	0.87	0.72	0.79	0.90	0.96	0.93	0.86
NLI-SAN	0.93	0.89	0.91	0.96	0.97	0.97	0.94
NLI-graph-abl	0.50	0.33	0.39	0.77	0.87	0.81	0.60
NLI-graph	0.89	0.83	0.86	0.94	0.96	0.95	0.90
Table 6: Veracity classification results on the PANACEA SMALL dataset. The best result in each column is highlighted in bold.
Model	False	True	Macro F1
Precision	Recall	F1	Precision	Recall	F1
GEAR (Zhou et al., 2019)	0.88	0.88	0.88	0.93	0.94	0.94	0.91
KGAT (Liu et al., 2020)	0.95	0.98	0.96	0.99	0.98	0.98	0.97
NLI	0.52	0.27	0.36	0.69	0.86	0.76	0.56
NLI+Sent	0.94	0.94	0.94	0.97	0.97	0.97	0.95
NLI+PSent	0.89	0.77	0.82	0.88	0.95	0.91	0.86
NLI-SAN	0.95	0.95	0.95	0.97	0.98	0.97	0.96
NLI-graph-abl	0.60	0.43	0.50	0.73	0.84	0.78	0.64
NLI-graph	0.94	0.91	0.93	0.95	0.97	0.96	0.94
Table 7: Veracity classification results on the PANACEA LARGE dataset. The best result in each column is highlighted in bold.
most similar sentences from the top 10 retrieved documents. The retrieval procedure is as in sections 4.1 and 4.2. Details of parameter settings can be found in Appendix B. We compare against the SOTA methods  $\mathrm{GEAR}^9$  (Zhou et al., 2019) and  $\mathrm{KGAT^{10}}$  (Liu et al., 2020), with settings as described by the authors.
For all approaches we perform 5-fold cross-validation and report the averaged results on the SMALL dataset in Table 6. By using the NLI information alone it is possible to obtain reasonable results for the True claims, however, this is not the case for the most relevant False claims. Once we add sentence representations the efficiency of the method increases significantly. Using NLI-SAN instead of simply concatenating contextualised claim-evidence representations and NLI outputs further improves the results. A similar observation can be made in the results generated by NLI-graph and its variants; the contextualised representations of claim-evidence pairs are much more important than merely using the corresponding NLI values. We also note that using the graph version NLI-graph obtains better scores
than a non-graph model with the same information NLI+PSent, however the scores are still lower than the NLI-SAN method. Our method performs on a par with KGAT, while being simpler, and outperforms GEAR.
Complementing the results for the SMALL dataset, Table 7 presents the results for the LARGE dataset. In general, we observe improved performance for all models across all metrics for both classes compared to the results on the SMALL dataset. The previous results in the SMALL dataset constitute a more challenging case, since the uniqueness of the claims is increased and therefore the veracity assessment models are not able to learn from similar claims when performing the assessment.
Our results show that in document retrieval, we have obtained values of around 0.6 from a simple term scoring and re-ranking retrieval model. However, this baseline represents only a rough measure of quality using this technique, since we have only evaluated the retrieval of a single document specific to each claim; we have not evaluated the quality of other retrieved documents.
$^{9}$ https://github.com/thunlp/GEAR
$^{10}$ https://github.com/thunlp/KernelGAT
The distinction into True and False claims can be rather coarse-grained. We note that initially we considered a larger number of veracity labels, including more nuanced cases that could be interesting to analyse (see A.1). However, we have not found a clear separation between complex cases and it would seem that different fact checkers do not follow the same conventions when labelling such cases. The development of datasets especially focused on such nuanced cases may be therefore an important line of work in the future, together with the development of techniques for these more complex situations.
In analysing misclassified claims, we note some interesting cases. The scope and globality of the pandemic imply that similar issues are mentioned repeatedly on multiple occasions, yet claims to be verified may include nuances or specificities. This is challenging as it is easy to retrieve information that omits relevant nuances. E.g. The claim "Barron Trump had COVID-19, Melania Trump says" retrieves sentences such as "Rudy Giuliani has tested positive for COVID-19, Trump says." with a similar structure and mentions but missing the key name. This type of situation could be addressed by using Named Entity Recognition (NER) methods that prioritise matching between the entities involved in the claim and the information sources. See e.g. (Taniguchi et al., 2018; Nooralahzadeh and Øvrelid, 2018).
Other interesting cases involve claims for which documents with adequate information are retrieved, but the sentences containing evidence cannot be identified because they are too different from the original claim. E.g. The claim "Vice President of Bharat Biotech got a shot of the indigenous COVAXIN vaccine" retrieves correct documents on the issue. Similar sentences are retrieved such as "Covaxin which is being developed by Bharat Biotech is the only indigenous vaccine that is approved for emergency use". Despite being similar such retrieved sentences give no information about the claimed situation. In the retrieved document, the sentence "The pharmaceutical company, has in a statement, denied the claim and said the image shows a routine blood test." contains the essential information to debunk the original claim but is missed by the sentence retrieval engine as it is very different from the claim (See Table A1 in Appendix C for other examples).
Such cases are more difficult to deal with, as the similarity between claim and evidence is cer
tainly a good indicator of relevance. Nevertheless, these cases are very interesting for future work using more complex approaches. We have made an initial attempt to address this problem by representing claims and retrieved documents using Abstract Meaning Representation (Banarescu et al., 2013) in order to better select relevant information. Although the results were not satisfactory, it may be an interesting avenue for future exploration. Another line of future work is the design of strategies against adversarial attacks to mitigate possible risks to our system.
We have presented a novel dataset that aggregates a heterogeneous set of COVID-19 claims categorised as True or False. Aggregation of heterogeneous sources involved a careful dedduplication process to ensure dataset quality. Fact-checking sources are provided for veracity assessment, as well as additional information sources for True claims. Additionally, claims are labelled with sub-types (Multimodal, Social Media, Questions, Numerical, and Named Entities).
We have performed a series of experiments using our dataset for information retrieval through direct retrieval and using a multi-stage re-ranker approach. We have proposed new NLI methods for claim veracity assessment, attention-based NLI-SAN and graph-based NLI-graph, achieving in our dataset competitive results with the GEAR and KGAT state-of-the-art models. We have also discussed challenging cases and provided ideas for future research directions.
This work was supported by the UK Engineering and Physical Sciences Research Council (grant no. EP/V048597/1, EP/T017112/1). ML and YH are supported by Turing AI Fellowships funded by the UK Research and Innovation (grant no. EP/V030302/1, EP/V020579/1).
Nasreen Abdul-Jaleel, James Allan, W Bruce Croft, Fernando Diaz, Leah Larkey, Xiaoyan Li, Mark D Smucker, and Courtney Wade. 2004. Umass at trec 2004: Novelty and hard. Computer Science Department Faculty Publication Series, page 189.
Muhammad Abdul-Mageed, AbdelRahim Elmadany, El Moatez Billah Nagoudi, Dinesh Pabbi, Kunal
Verma, and Rannie Lin. 2021. Mega-COV: A billion-scale dataset of  $100+$  languages for COVID-19. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3402-3420, Online. Association for Computational Linguistics.
Rami Aly, Zhijiang Guo, Michael Sejr Schlichtkrull, James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Oana Cocarascu, and Arpit Mittal. 2021. The fact extraction and VERIFICATION over unstructured and structured information (FEVEROUS) shared task. In Proceedings of the Fourth Workshop on Fact Extraction and VERIFICATION (FEVER), pages 1-13, Dominican Republic. Association for Computational Linguistics.
Dzmitry Bahdanau, Kyung Hyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015.
Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitht, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In Proceedings of the 7th linguistic annotation workshop and interoperability with discourse, pages 178-186.
Cambridge journals Coronavirus Free Access Collection. 2020. Cambridge journals coronavirus free access collection. https://www.cambridge.org/core/browse-subjects/medicine/coronavirus-free-access-collection.
Emily Chen, Kristina Lerman, and Emilio Ferrara. 2020. Tracking social media discourse about the pandemic-19 pandemic: Development of a public coronavirus twitter data set. *JMIR Public Health and Surveillance*, 6(2):e19273.
Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced lstm for natural language inference. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1657-1668.
Qingyu Chen, Alexis Allot, and Zhiyong Lu. 2021. Litcovid: an open database of covid-19 literature. Nucleic acids research, 49(D1):D1534-D1540.
COVID-19 Data Portal (EU). 2020. Covid-19 data portal (eu). https://www.covid19dataportal.org/.
Fabio Crestani, Mounia Lalmas, Cornelis J Van Rijsbergen, and Iain Campbell. 1998. "is this document relevant?... probably" a survey of probabilistic models in information retrieval. ACM Computing Surveys (CSUR), 30(4):528-552.
Limeng Cui and Dongwon Lee. 2020. Coaid: Covid-19 healthcare misinformation dataset. arXiv preprint arXiv:2006.00885.
Enyan Dai, Yiwei Sun, and Suhang Wang. 2020. Ginger cannot cure cancer: Battling fake health news with a comprehensive data repository. In Proceedings of the International AAAI Conference on Web and Social Media, volume 14, pages 853-862.
Dimitar Dimitrov, Erdal Baran, Pavlos Fafalias, Ran Yu, Xiaofei Zhu, Matthaus Zloch, and Stefan Dietze. 2020. Tweetscov19-a knowledge base of semantically annotated tweets about the pandemic. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pages 2991-2998.
Arianna D'Ulizia, Maria Chiara Caschera, Fernando Ferri, and Patrizia Grifoni. 2021. Fake news detection: a survey of evaluation datasets. PeerJ Computer Science, 7:e518.
Elsevier journals Novel Coronavirus Information Center. 2020. Elsevier journals novel coronavirus information center. https://www.elsevier.com/connect/coronavirus-information-center.
William Ferreira and Andreas Vlachos. 2016. Emergent: a novel data-set for stance classification. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: Human language technologies, pages 1163-1168.
Matthias Fey and Jan E. Lenssen. 2019. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds.
Andrea Galassi, Marco Lippi, and Paolo Torroni. 2020. Attention in natural language processing. IEEE Transactions on Neural Networks and Learning Systems.
Reza Ghaeini, Sadid A Hasan, Vivek Datla, Joey Liu, Kathy Lee, Ashequl Qadir, Yuan Ling, Aaditya Prakash, Xiaoli Fern, and Oladimeji Farri. 2018. Drbilstm: Dependent reading bidirectional lstm for natural language inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1460-1469.
Andreas Hanselowski, Hao Zhang, Zile Li, Daniil Sorokin, Benjamin Schiller, Claudia Schulz, and Iryna Gurevych. 2018. Ukp-athene: Multi-sentence textual entailment for claim verification. EMNLP 2018, page 103.
Tamanna Hossain, Robert L. Logan IV, Arjuna Ugarte, Yoshitomo Matsubara, Sean Young, and Sameer Singh. 2020. COVID Lies: Detecting COVID-19 misinformation on social media. In Proceedings of the 1st Workshop on NLP for COVID-19 (Part 2) at EMNLP 2020, Online. Association for Computational Linguistics.
Xiaolei Huang, Amelia Jamison, David Broni- atowski, Sandra Quinn, and Mark Dredze. 2020. Coronavirus twitter data: A collection of Covid-19 tweets with automated annotations. http://twitterdata.covid19dataresources.org/index.
Daniel Kerchner and Laura Wrubel. 2020. Coronavirus tweet ids. Harvard Dataverse.
Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907.
Neema Kotonya and Francesca Toni. 2020. Explainable automated fact-checking for public health claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7740-7754.
Rabindra Lamsal. 2021. Design and analysis of a large-scale pandemic-19 tweets dataset. Applied Intelligence, 51(5):2790-2804.
Tianda Li, Xiaodan Zhu, Quan Liu, Qian Chen, Zhigang Chen, and Si Wei. 2019. Several experiments on investigating pretraining and knowledge-enhanced models for natural language inference. arXiv preprint arXiv:1904.12104.
Yichuan Li, Bohan Jiang, Kai Shu, and Huan Liu. 2020. Mm-covid: A multilingual and multimodal data repository for combating Covid-19 disinformation.
Jimmy Lin. 2019. The neural hype and comparisons against weak baselines. In ACM SIGIR Forum, volume 52, pages 40-51. ACM New York, NY, USA.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Zhenghao Liu, Chenyan Xiong, Maosong Sun, and Zhiyuan Liu. 2020. Fine-grained fact verification with kernel graph attention network. In The 58th annual meeting of the Association for Computational Linguistics (ACL).
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations.
Jackson Luken, Nanjiang Jiang, and Marie-Catherine de Marneffe. 2018. Qed: A fact verification system for the fever shared task. In Proceedings of the First Workshop on Fact Extraction and VERIFICATION (FEVER), pages 156-160.
MedRN medical research network SSRN Coronavirus Infectious Disease Research Hub. 2020. Medrn medical research network ssnr coronavirus infectious disease research hub. https://www.ssrn.com/index.cfm/en/coronavirus/.
Shahan Ali Memon and Kathleen M Carley. 2020. Characterizing COVID-19 misinformation communities using a novel twitter dataset. In CEUR Workshop Proceedings, volume 2699.
Yixin Nie, Haonan Chen, and Mohit Bansal. 2019. Combining fact extraction and verification with neural semantic matching networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6859-6866.
Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. 2020. Document ranking with a pretrained sequence-to-sequence model. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 708-718.
Rodrigo Nogueira, Wei Yang, Kyunghyun Cho, and Jimmy Lin. 2019. Multi-stage document ranking with bert. arXiv preprint arXiv:1910.14424.
Farhad Nooralahzadeh and Lilja Øvrelid. 2018. Sirius: An entity linking approach to fact extraction and verification. In Proceedings of the First Workshop on Fact Extraction and VERIFICATION (FEVER), pages 119-123.
Jeppe Nørregaard, Benjamin D Horne, and Sibel Adah. 2019. Nela-gt-2018: A large multi-labelled news dataset for the study of misinformation in news articles. In Proceedings of the international AAAI conference on web and social media, volume 13, pages 630-638.
Oxford journals resources on COVID-19. 2020. Oxford journals resources on COVID-19. https://academic.oup.com/journals/ pages/coronavirus.
Ankur Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. 2016. A decomposable attention model for natural language inference. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2249-2255.
Ronak Pradeep, Xueguang Ma, Rodrigo Nogueira, and Jimmy Lin. 2021. Vera: Prediction techniques for reducing harmful misinformation in consumer health search. In Proceedings of the 44th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2021).
Umair Qazi, Muhammad Imran, and Ferda Ofli. 2020. Geocov19: a dataset of hundreds of millions of multilingual Covid-19 tweets with location information. SIGSPATIAL Special, 12(1):6-15.
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982-3992.