SlowGuess/NAACL_2022_OCR huggingface dataset:
For our baseline, we use the setup of the state-of-the-art model of Shaar et al. (2020a). We use the claim as a query against the full text of the documents using BM25. We then train a reranker on the top-100 BM25 results using rankSVM (Herbrich et al., 1999) with an RBF kernel.
The reranker uses nine similarity measures that compare the InputClaim to the VerClaim, as well as the respective reciprocal ranks. In particular, we compute the BM25 score for InputClaim vs. VerClaim, title, text, VerClaim + title + text. We also compute the cosine using sentence-BERT embeddings for InputClaim vs. VerClaim, title, and the top-4 sentences from text. Using these scores, we create a vector representation of the InputClaim-VerClaim pair with dimensionality  $\mathbb{R}^{18}$ . We then scale the vectors of all InputClaim-VerClaim pairs in  $[-1; 1]$  and we train a rankSVM with default values of the hyper-parameters: KernelDegree = 3,  $\gamma = 1/\text{num_features}$ , and  $\epsilon = 0.001$ .
As shown in Figure 1, our model uses co-reference resolution on the source and on the target side, the local context (i.e., the neighboring sentences), and the global context (using Transformer-XH) as discussed below. It is still a pairwise reranker, but with a richer context representation.
4.2.1 Co-reference Resolution
We manually inspected the training transcripts and the associated verified claims, and we realized that there were many co-reference dependencies, resolving which could potentially help to obtain more representative textual and contextual similarity scores. As for the verified claims, we noticed that not all VerClaims were self-contained, and that some understanding of the context was needed<sup>9</sup> of the article's text that explains the verdict provided by the PolitiFact journalists. Therefore, our hypothesis was that resolving such co-references could improve the downstream matching scores.
For the same reason, we also performed co-reference resolution on the PolitiFact articles when they were used to compute the BM25 scores.
We experimented with various co-reference resolution tools including NeuralCoref, $^{10}$  e2e-coref, $^{11}$  and SpanBERT, $^{12}$  and we found that NeuralCoref was best on the input transcripts, while e2e-coref was best on the articles about the target VerClaims. Hence, in the rest of our experiments below, we show results using NeuralCoref for the source side, and using e2e-coref for the target side.
We resolved the co-reference in the InputClaim by performing co-reference resolution on the entire input transcript (as was suggested in the literature); we will refer to this as src-coref. As for the verified claims, we aimed to resolve the co-references both in the VerClaim and in the text of the PolitiFact articles. We also aimed to ensure that the dependencies from the text can be used for the VerClaim. Therefore, we concatenated both the text and VerClaim (in the same order), and we applied the co-reference model on the concatenated text. We chose this order of concatenation because the published text reserves the last paragraph to rephrase the VerClaim and to provide a summary of the justification; hence, there is a higher probability to resolve the co-references correctly.
4.2.2 Local Context
Resolving the pronominal co-references allows us to obtain the correct objects and the names the InputClaim refers to. However, in the process of analyzing the dataset, we noticed that different Ver-Claims, although having similar structure, could talk about different things, depending on the article text and also on the surrounding context. Therefore, it is important to understand the context of an InputClaim. In particular, we achieve this by performing a feature-level concatenation of the neighboring sentences in the transcript, i.e., we take the eighteen features  $(\mathbb{R}^{18}$ , as discussed in Section 4.1 above) for the neighboring sentences, and we concatenate them to the similarity score for the InputClaim. We then use the resulting representation as a feature vector to be fed into our reranker. For example, if we take three sentences before the InputClaim and one sentence after it, we denote this as  $FC(3,1)$ .
Let  $S_{i}$  be our InputClaim, which is the  $i$ -th sentence in the transcript. We compute the similarity measures and the reciprocal rank (as described in Section 4.1) to obtain the vector representation  $S_{i,v}$  for  $S_{i}$ . With  $k = 3$  previous and  $l = 1$  following neighbouring sentences our final feature vector is
$$
\begin{array}{l} F C (k = 3, l = 1) = S _ {i - 3, v} + S _ {i - 2, v} \\ + S _ {i - 1, v} + S _ {i, v} + S _ {i + 1, v} \tag {1} \\ \end{array}
$$
where  $\sharp$  represents concatenation.
Note that after the concatenation, the resulting dimensionality of the feature vector for  $FC(3,1)$  is  $18 \times (3 + 1 + 1) = 90$ .
For example, who is speaking or what is being discussed.
10github.com/huggingface/neuralcoref
"github.com/kentonl/e2e-coref
$^{12}$ github.com/facebookresearch/SpanBERT
Line No.	Model	Overall	clean	clean-hard	part-of	context-dep
1	Baseline	0.429	0.661	0.365	0.161	0.375
Source-Side Experiments: Co-reference Resolution, Local Context
2	FC(3, 1)	0.513	0.690	0.485	0.305	0.448
3	src-coref	0.479	0.667	0.408	0.286	0.429
4	src-coref + FC(3, 1)	0.532	0.695	0.452	0.385	0.485
Target-Side Experiments: Co-reference Resolution, Global Context
5	Transformer-XH	0.468	0.680	0.441	0.226	0.384
6	tgt-coref	0.443	0.673	0.422	0.182	0.339
7	tgt-coref + Transformer-XH	0.458	0.702	0.444	0.161	0.357
Source+Target-Side Experiments: Co-reference Resolution, Local Context, Global Context
8	src-coref + tgt-coref	0.487	0.672	0.440	0.291	0.411
9	All	0.517	0.749	0.389	0.321	0.464
Table 4: MAP scores of the reranker models on the test set using the Debate-Level Chrono split.
4.2.3 Global Context
The similarity scores that leverage the local context in the textual content of the InputClaim and the VerClaim are obtained using (i) BM25, and (ii) the cosine similarity between the Sentence-BERT embeddings of the InputClaim vs. the top-4 sentences of the VerClaim. This might miss relevant information further away from the InputClaim in the input document and further away from the VerClaim in the document accompanying the VerClaim. We refer to such scattered information as the global context. To capture it, we use Transformer-XH (Zhao et al., 2019), which is pretrained on the FEVER (Fact Extraction and VERIFICATION) dataset to predict whether a given input claim is supported/refuted by a set of target sentences (from Wikipedia), represented as a graph, or there is no enough information. We used the model from (Zhao et al., 2019). For a given InputClaim, we generate a graph for each of the top-100 VerClaims retrieved using BM25 and the normalized claim, the title, and the top-3 sentences from the text as nodes. Using the Transformer-XH model on the graph, we obtain three additional scores that correspond to the posterior probability that VerClaim supports or refutes the InputClaim, or there is no enough information.
4.3 Hyper-Parameter Values
For the baseline, we use the best values of the hyperparameters as found in (Shaar et al., 2020a). For our context-aware models, we select the values of the hyper-parameters by splitting the training dataset into train-train (debates from 2012-2017) and train-dev (debates from 2018), then we train on the former, and we test on the latter.
As we have a ranking task, we use mean average precision (MAP) for evaluation. It is a suitable measure as some InputClaims are paired with more than one VerClaim. This is why we opted for not using mean reciprocal rank (MRR), which would only pay attention to the highest-ranked match.
Below, we described the results for our source-side and target-side context modeling experiments.
5.1 Source-Side Experiments
For the source side experiments, we used coreference resolution on transcripts and variations of the local context by varying  $k$  and  $l$  in Eq. 1.
When we inspected the transcripts, we found that co-references tended to be resolved by a few sentences before the InputClaim; therefore, we tried  $FC(1, 1)$ ,  $FC(3, 1)$ ,  $FC(3, 3)$ , and  $FC(5, 1)$ . We obtained the best results on cross-validation using  $FC(3, 1)$ , which we use below. As shown in Table 4, the local context (line 2) improves over the baseline (line 1) by eight MAP points absolute.
We experimented using co-reference resolution with NeuralCoref. This yielded a sizable improvement over the baseline as shown in line 3 in Table 4, especially for part-of and context-dep pairs, as they have many co-references, which can make it hard for the model the understand the InputClaim. After combining the two methods, i.e., src-coref and  $FC(3,1)$  (see line 4), we achieved the highest MAP score of 0.532. We always see an improvement for the clean category as the resolved InputClaim can match the article text better.
5.2 Target-Side Experiments
For the target-side experiments, we tried using co-reference resolution (on the source and on the target side) for the VerClaim and the fact-checking article, as well as modeling the global context with Transformer-XH. Compared to the baseline, we see on line 5 of Table 4 a sizable improvement from 0.365 to 0.441 MAP points for clean-hard.
This is expected as the pair does not exhibit much semantic similarity, and we need to build our own understanding of the text of the VerClaim in order to capture the contextual similarity in the pair. We also experimented with co-reference resolution on the VerClaim and the text of the VerClaim and also see some improvement. Combining tgt-coref and Transformer-XH (line 7) improved the performance over tgt-coref alone, but it is worse than Transformer-XH alone. The combination outperforms other target-side experiments for clean.
5.3 Source-Side & Target-Side Experiments
Eventually, we experimented with modeling the context both on the source and on the target side. Line 8 in Table 4 shows the evaluation results when we use co-reference resolution both on the source and on the target side. We can see that this yields a higher overall MAP score of 0.487, compared to using source-side (MAP of 0.479; line 3) or target-side context only (MAP of 0.443; line 6). Moreover, co-reference resolution on both sides helps for clean-hard and part-of(compared to using co-reference on one side only) as they require better local and global context, respectively.
We further tried putting it all together, and the result is shown in line 9.13 While this yielded better results for clean, it was slightly worse compared to the source-side context modeling combination in line 4. This is probably due to the source-side context models being generally stronger than the target-side ones (compare lines 2-3 to lines 5-6).
We can conclude that modeling the context on the source side is much more important than on the target side. This is expected for political debates, which are conversational in nature. In contrast, the target side is a well-written journalistic article, where sentences are much more self-contained. Thus, features from the source side (i.e., from the debate) are more useful as can be seen in Table 4.
As mentioned above, our baseline is a reimplementation of the best system of Shaar et al. (2020a), and our context modeling extensions add additional components on top of it. Note, however, that our experimental results are not directly comparable to their published ones, as we use a more realistic and also a much harder setup, where the data is split by entire debates and also chronologically, following the Debate-Level Chrono data split, as we discussed in Section 3, i.e., training on the data from 2012 to 2018 and testing on 2019 (while they split all debates into sentences and randomly distribute them to training/testing). However, we do have comparison to their approach, as we ran their model on our data split, which is our baseline, as shown on line 1 of Table 4.
We have presented our work on the important but under-studied problem of detecting previously fact-checked claims in political debates and speeches. We studied the impact of modeling the context: both on the source side, i.e., in the debate, as well as on the target side, i.e., in the fact-checking document that explains how human fact-checkers have arrived at their decision about the factuality of the claim. In particular, we modeled the local context, the global context, and we further used coreference resolution and multi-hop reasoning over the target text using Transformer-XH. The experimental results have shown that each of these components represents a valuable information source, but modeling the source-side context is more important, and can yield  $10+$  points of absolute improvement over a context-free state-of-the-art baseline.
In future work, we want to try other multi-hop reasoning frameworks for context modeling. We also plan to experiment with other kinds of conversations, e.g., in community forums and in social media, including for other languages.
This research is part of the Tanbih mega-project, developed at the Qatar Computing Research Institute, HBKU, which aims to limit the impact of "fake news," propaganda, and media bias by making users aware of what they are reading, thus promoting media literacy and critical thinking.
<sup>13</sup>Note that in this result we did not use target-side co-reference, as adding it yielded somewhat worse results. It seems to interact badly with Transformer-XH, which can also be seen by comparing lines 5 and 7.
Biases We note that there might be some biases in the data we use, as well as in some manual judgments for claim matching. There could be also biases in the data selection and the fact-checking process of the human fact-checkers, which are beyond our control. Finally, there are known biases in the large-scale pre-trained transformer models that we experiment with.
Intended Use and Misuse Potential Our models can make it possible to put politicians on the spot in real time, e.g., during an interview or a political debate, by providing journalists with tools to do trustable fact-checking in real time. They can also save a lot of time to fact-checkers for unnecessary double-checking something that was already fact-checked. However, these models could also be misused by malicious actors. We, therefore, ask researchers to exercise caution.
Environmental Impact We would also like to warn that the use of large-scale Transformers requires a lot of computations and the use of GPUs/TPUs for training, which contributes to global warming (Strubell et al., 2019). This is a bit less of an issue in our case, as we do not train such models from scratch; rather, we fine-tune them on relatively small datasets. Moreover, running on a CPU for inference, once the model has been finetuned, is perfectly feasible, and CPUs contribute much less to global warming.
Pepa Atanasova, Lluis Marquez, Alberto Barron-Cedeno, Tamer Elsayed, Reem Suwaileh, Wajdi Zaghouani, Spas Kyuchukov, Giovanni Da San Martino, and Preslav Nakov. 2018. Overview of the CLEF-2018 CheckThat! lab on automatic identification and verification of political claims, task 1: Checkworthiness. In CLEF 2018 Working Notes. Working Notes of CLEF 2018 - Conference and Labs of the Evaluation Forum, CEUR Workshop Proceedings, Avignon, France.
Pepa Atanasova, Preslav Nakov, Georgi Karadzhov, MiTRA Mohtarami, and Giovanni Da San Martino. 2019a. Overview of the CLEF-2019 CheckThat! Lab on Automatic Identification and Verification of Claims. Task 1: Check-Worthiness. In CLEF 2019 Working Notes, Lugano, Switzerland.
Pepa Atanasova, Preslav Nakov, Lluis Marquez, Alberto Barron-Cedeno, Georgi Karadzhov, Tsvetomila Mihaylova, Mitra Mohtarami, and James Glass. 2019b.
Automatic fact-checking using context and discourse information. Journal of Data and Information Quality (JDIQ), 11(3):1-27.
Isabelle Augenstein, Christina Lioma, Dongsheng Wang, Lucas Chaves Lima, Casper Hansen, Christian Hansen, and Jakob Grue Simonsen. 2019. MultiFC: A real-world multi-domain dataset for evidence-based fact checking of claims. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP '19', pages 4685-4697, Hong Kong, China.
Alberto Barron-Cedeno, Tamer Elsayed, Preslav Nakov, Giovanni Da San Martino, Maram Hasanain, Reem Suwaileh, Fatima Haouari, Nikolay Babulkov, Bayan Hamdan, Alex Nikolov, Shade Shaar, and Zien Sheikh Ali. 2020. Overview of CheckThat! 2020: Automatic identification and verification of claims in social media. In Experimental IR Meets Multilinguality, Multimodality, and Interaction Proceedings of the Eleventh International Conference of the CLEF Association (CLEF 2020), LNCS (12260).
Alberto Barrón-Cedeno, Tamer Elsayed, Preslav Nakov, Giovanni Da San Martino, Maram Hasanain, Reem Suwaileh, and Fatima Haouari. 2020. CheckThat! at CLEF 2020: Enabling the automatic identification and verification of claims in social media. In Proceedings of the 42nd European Conference on Information Retrieval, ECIR '20, pages 499-507, Lisbon, Portugal.
Alberto Barrón-Cedeno, Tamer Elsayed, Reem Suwaileh, Lluis Marquez, Pepa Atanasova, Wajdi Zaghouani, Spas Kyuchukov, Giovanni Da San Martino, and Preslav Nakov. 2018. Overview of the CLEF-2018 CheckThat! lab on automatic identification and verification of political claims. Task 2: Factuality. In CLEF 2018 Working Notes. Working Notes of CLEF 2018 - Conference and Labs of the Evaluation Forum, CEUR Workshop Proceedings, Avignon, France.
Mostafa Bouziane, Hugo Perrin, Aurélien Cluzeau, Julien Mardas, and Amine Sadeq. 2020. Buster.AI at CheckThat! 2020: Insights and recommendations to improve fact-checking. In Working Notes of CLEF 2020—Conference and Labs of the Evaluation Forum, CLEF '2020, Thessaloniki, Greece.
Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced LSTM for natural language inference. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL '17, pages 1657-1668, Vancouver, Canada.
Anton Chernyavskiy, Dmitry Ilvovsky, Pavel Kalinin, and Preslav Nakov. 2022. Batch-softmax contrastive loss for pairwise sentence scoring tasks. In Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT '22, Seattle, Washington, USA.
Anton Chernyavskiy, Dmitry Ilvovsky, and Preslav Nakov. 2021a. Aschern at CLEF CheckThat! 2021: Lambda-calculus of fact-checked claims. In CLEF 2021 Working Notes. Working Notes of CLEF 2021–Conference and Labs of the Evaluation Forum, Bucharest, Romania (online).
Anton Chernyavskiy, Dmitry Ilvovsky, and Preslav Nakov. 2021b. WhatTheWikiFact: Fact-checking claims against Wikipedia. In Proceedings of the 30th ACM International Conference on Information and Knowledge Management, CIKM '21, pages 4690-4695.
Pepa Gencheva, Preslav Nakov, Lluís Márquez, Alberto Barrón-Cedeno, and Ivan Koychev. 2017. A context-aware approach for detecting worth-checking claims in political debates. In Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP '17, pages 267-276, Varna, Bulgaria.
Andreas Hanselowski, Hao Zhang, Zile Li, Daniil Sorokin, Benjamin Schiller, Claudia Schulz, and Iryna Gurevych. 2018. UKP-Athene: Multi-sentence textual entailment for claim verification. In Proceedings of the First Workshop on Fact Extraction and VERIFICATION, FEVER '18, pages 103-108, Brussels, Belgium.
Maram Hasanain, Fatima Haouari, Reem Suwaileh, Zien Sheikh Ali, Bayan Hamdan, Tamer Elsayed, Alberto Barron-Cedeño, Giovanni Da San Martino, and Preslav Nakov. 2020. Overview of CheckThat! 2020 Arabic: Automatic identification and verification of claims in social media. In Working Notes of CLEF 2020—Conference and Labs of the Evaluation Forum, CLEF '2020, Thessaloniki, Greece.
Naeemul Hassan, Gensheng Zhang, Fatma Arslan, Josue Caraballo, Damian Jimenez, Siddhant Gawsane, Shohedul Hasan, Minumol Joseph, Aaditya Kulkarni, Anil Kumar Nayak, Vikas Sable, Chengkai Li, and Mark Tremayne. 2017. ClaimBuster: The first-ever end-to-end fact-checking system. Proc. VLDB Endow., 10(12):1945-1948.
Ralf Herbrich, Thore Graepel, and Klaus Obermayer. 1999. Support vector learning for ordinal regression. In Proceedings of the 1999 Ninth International Conference on Artificial Neural Networks, ICANN '99, pages 97-102, Edinburgh, UK.
Christopher Hidey and Mona Diab. 2018. Team SWEeper: Joint sentence extraction and fact checking with pointer networks. In Proceedings of the First Workshop on Fact Extraction and VERIFICATION (FEVER), pages 150-155, Brussels, Belgium.
David M.J. Lazer, Matthew A. Baum, Yochai Benkler, Adam J. Berinsky, Kelly M. Greenhill, Filippo Menczer, Miriam J. Metzger, Brendan Nyhan, Gordon Pennycook, David Rothschild, Michael Schudson, Steven A. Sloman, Cass R. Sunstein, Emily A.
Thorson, Duncan J. Watts, and Jonathan L. Zittrain. 2018. The science of fake news. Science, 359(6380):1094-1096.
Yaliang Li, Jing Gao, Chuishi Meng, Qi Li, Lu Su, Bo Zhao, Wei Fan, and Jiawei Han. 2016. A survey on truth discovery. ACM SIGKDD Explorations Newsletter, 17(2):1-16.
Zhenghao Liu, Chenyan Xiong, Maosong Sun, and Zhiyuan Liu. 2020. Fine-grained fact verification with kernel graph attention network. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7342-7351, Online.
Jackson Lumen, Nanjiang Jiang, and Marie-Catherine de Marneffe. 2018. QED: A fact verification system for the FEVER shared task. In Proceedings of the First Workshop on Fact Extraction and VERIFICATION (FEVER), pages 156-160, Brussels, Belgium.
Simona Mihaylova, Iva Borisova, Dzhovani Chemishanov, Preslav Hadzhitsanev, Momchil Hardalov, and Preslav Nakov. 2021. DIPS at CheckThat! 2021: Verified claim retrieval. In CLEF 2021 Working Notes. Working Notes of CLEF 2021-Conference and Labs of the Evaluation Forum, Bucharest, Romania (online).
Tsvetomila Mihaylova, Georgi Karadzhov, Pepa Atanasova, Ramy Baly, Mitra Mohtarami, and Preslav Nakov. 2019. SemEval-2019 task 8: Fact checking in community question answering forums. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 860–869, Minneapolis, Minnesota, USA.
Preslav Nakov, Alberto Barrón-Cedeño, Giovanni Da San Martino, Firoj Alam, Rubén Miguez, Tommaso Caselli, Mucahid Kutlu, Wajdi Zaghouani, Chengkai Li, Shaden Shaar, Gautam Kishore Shahi, Hamdy Mubarak, Alex Nikolov, Yavuz Selim Kartal, and Javier Beltrán. 2022a. Overview of the CLEF-2022 CheckThat! lab task 1 on identifying relevant claims in tweets. In Working Notes of CLEF 2022—Conference and Labs of the Evaluation Forum, CLEF '2022, Bologna, Italy.
Preslav Nakov, Alberto Barrón-Cedeño, Giovanni Da San Martino, Firoj Alam, Julia Maria Struß, Thomas Mandl, Rubén Miguez, Tommaso Caselli, Mucahid Kutlu, Wajdi Zaghouani, Chengkai Li, Shaden Shaar, Gautam Kishore Shahi, Hamdy Mubarak, Alex Nikolov, Nikolay Babulkov, Yavuz Selim Kartal, and Javier Beltran. 2022b. The CLEF-2022 CheckThat! lab on fighting the COVID-19 infodemic and fake news detection. In Proceedings of the 44th European Conference on IR Research: Advances in Information Retrieval, ECIR '22, pages 416-428, Stavanger, Norway.
Preslav Nakov, Alberto Barrón-Cedeño, Giovanni Da San Martino, Firoj Alam, Julia Maria Struß, Thomas Mandl, Rubén Miguez, Tommaso Caselli, Mucahid Kutlu, Wajdi Zaghouani,