SlowGuess/ACL_2020_OCR huggingface dataset:
A simple baseline is to use BM25 (Robertson and Zaragoza, 2009), which is classical approach in information retrieval. BM25 assigns a score to each query-document pair based on exact matching between the words in the query and the words in a target document, and it uses this score for ranking. We experiment with BM25 using the input claim as a query against different representations of the verified claims:
- IR (Title): the article titles;
- IR (VerClaim): the verified claims;
- IR (Body): the article bodies;
- Combinations of the above.
6.2 BERT-based Models
The BM25 algorithm focuses on exact matches, but as lines 2-5 in Table 1 and line 3 in Table 2 show, the input claim can use quite different words. Thus, we further try semantic matching using BERT.
Initially, we tried to fine-tune BERT (Devlin et al., 2019), but this did not work well, probably because we did not have enough data to perform the fine-tuning. Thus, eventually we opted to use BERT (and variations thereof) as a sentence encoder, and to perform max-pooling on the penultimate layer to obtain a representation for an input piece of text. Then, we calculate the cosine similarity between the representation of the input claim and of the verified claims in the dataset, and we use this similarity for ranking.
- BERT:base, uncased: the base, uncased model of BERT;
RoBERTa:base: the base, cased model of RoBERTa (Liu et al., 2019);
- sentence-BERT:base: BERT, specifically trained to produce good sentence representations (Reimers and Gurevych, 2019); this is unlike BERT and RoBERTa, for which we found the cosine similarity between totally unrelated claims often to be quite high;
- sentence-BERT:large: the large version of sentence-BERT.
Experiment	MRR	MAP@k	HasPositives@k
1	3	5	10	20	all	1	3	5	10	20	50
IR (BM25; full database)
IR:Title	.288	.216	.259	.261	.268	.272	.276	.220	.330	.346	.401	.464	.535
IR:VerClaim	.435	.366	.404	.413	.415	.419	.422	.378	.472	.511	.527	.574	.629
IR:Body	.565	.484	.538	.546	.552	.556	.560	.488	.614	.653	.700	.740	.811
IR:Title+VerClaim+Body	.526	.425	.504	.507	.513	.516	.519	.433	.614	.630	.661	.717	.772
Semantic Matching - BERT & co. (matching against VerClaim only; full db)
BERT:base,uncased	.268	.204	.242	.251	.259	.260	.264	.204	.299	.338	.393	.409	.496
RoBERTa:base	.209	.173	.194	.198	.203	.205	.207	.173	.220	.236	.283	.315	.346
sentence-BERT:base	.377	.311	.352	.354	.361	.366	.370	.315	.417	.425	.480	.551	.614
sentence-BERT:large	.395	.354	.367	.372	.381	.382	.386	.362	.393	.417	.480	.496	.582
BERT on Full Articles (sent.BERT:large matching against VerClaim + Title + top-n article body sent.; full db)
sentence-BERT: n = 3	.515	.441	.478	.493	.498	.501	.505	.457	.528	.598	.638	.693	.756
sentence-BERT: n = 4	.517	.441	.487	.497	.500	.505	.508	.457	.551	.598	.622	.685	.756
sentence-BERT: n = 5	.515	.433	.484	.491	.498	.502	.505	.449	.551	.583	.638	.685	.748
sentence-BERT: n = 6	.509	.429	.480	.485	.491	.497	.500	.441	.543	.567	.614	.693	.740
Reranking (IR & sent.BERT:large matching against VerClaim + Title + top-4 article body sent.; top-N from IR)
Rerank-IR-top-10	.586	.528	.572	.578	.583	—	—	.512	.638	.693	.701	—	—
Rerank-IR-top-20	.586	.521	.572	.577	.580	.583	—	.512	.646	.685	.709	.740	—
Rerank-IR-top-50	.600	.519	.568	.584	.590	.590	.594	.520	.638	.717	.772	.780	.811
Rerank-IR-top-100	.608	.531	.580	.588	.597	.599	.602	.535	.654	.685	.740	.787	.819
Rerank-IR-top-200	.605	.529	.575	.585	.594	.598	.599	.535	.646	.685	.756	.803	.811
Table 5: PolitiFact: evaluation results on the test set.
- BERT on full articles: We further extend the above models to match against the body of the document, borrowing and further developing an idea from (Yang et al., 2019). We use sentence-BERT to encode each sentence in the Body, and then we compute the cosine similarity between the input claim and each of those sentences. Next, we collect scores for each claim-document pair, as opposed to having only a single score representing the similarity between the input and a verified claim. These scores include the cosine similarity for (i) claim vs. VerClaim, (ii) claim vs. Title, and (iii) top- $n$  scores of the claim vs. Body sentences. Finally, we train a binary classifier that takes all these scores and predicts whether the claim-document pair is a good match.
6.3 Reranking
Since BM25 and BERT capture different types of information, they can be combined to create a set of features based on the rankings returned by BM25 and the similarity scores computed on the embedding of the claim pairs. Following (Nogueira et al., 2019), we use a reranking algorithm, namely rankSVM with an RBF kernel, which learns to rank using a pairwise loss.
Below we describe our experiments on the PolitiFact and the Snopes datasets. We start with IR-based models, followed by BERT-based semantic similarity on claims and articles, and finally we experiment with pairwise learning-to-rank models.
7.1 Politifact Experiments
For the PolitFact dataset, we perform experiments with all models from Section 6, and we report the results in Table 5.
7.1.1 Experiment 1: BM25-based Baselines
We ran experiments matching the Input against Title, VerClaim, Body and Title+VerClaim+Body. We can see in Table 5 that using the Title yields the lowest results by a large margin. This is because the Title is only a summary, while VerClaim and Body contain more details and context. We can further see that the best representation, on all measures, is to use the Body, which performs better than using VerClaim by 0.12-0.14 in terms of MAP@k and MAP, and by 0.09 on MRR. This is probably because the article body is longer, which increases the probability of having more words matching the input claim. Finally, matching against all three targets is slightly worse than using Body only.
Experiment	MRR	MAP@k	HasPositives@k
1	3	5	10	20	all	1	3	5	10	20	50
IR (BM25; full database)
IR:Title	.619	.538	.573	.583	.587	.590	.592	.538	.673	.724	.759	.804	.844
IR:VerClaim	.655	.555	.589	.598	.600	.602	.605	.558	.729	.769	.784	.809	.864
IR:VerClaim+Title	.664	.555	.592	.600	.605	.608	.609	.558	.739	.774	.814	.849	.879
Semantic Matching - BERT & co. (matching against VerClaim only; full database)
sent.BERT-base:Title	.474	.397	.417	.425	.430	.434	.437	.397	.528	.563	.598	.658	.729
sent.BERT-base:VerClaim	.515	.402	.489	.504	.510	.512	.515	.402	.593	.653	.698	.739	.784
sent.BERT-large:VerClaim	.543	.457	.518	.527	.533	.535	.538	.457	.603	.648	.693	.724	.794
Reranking (IR & sentence-BERT:large matching against VerClaim + Title; top-  $N$  from IR)
Rerank-IR-top-10	.764	.687	.762	.764	.764	—	—	.673	.859	.869	.869	—	—
Rerank-IR-top-20	.781	.686	.773	.780	.781	.781	—	.678	.869	.905	.920	.920	—
Rerank-IR-top-50	.788	.691	.780	.782	.784	.784	.787	.693	.874	.894	.915	.925	.930
Rerank-IR-top-100	.775	.669	.758	.760	.760	.760	.774	.673	.859	.889	.910	.925	.930
Rerank-IR-top-200	.778	.672	.762	.764	.764	.764	.777	.678	.864	.884	.910	.930	.950
Table 6: Snopes: evaluation results on the test set.
7.1.2 Experiment 2: Semantic Matching
Next, we experimented with cosine similarity between the Input claim and VerClaim, as the BM25 experiments above have shown that using VerClaim is better than using Title.
We can see in Table 5 that BERT:uncased is better than RoBERTa (which is case sensitive) on all measures, which suggests that casing might not matter. We further see that the best semantic model is sentence-BERT: both the base and the large variants of sentence-BERT beat BERT and RoBERTa by at least  $13\%$  absolute across all measures (and in some cases, by a much larger margin).
7.1.3 Experiment 3: BERT on Full Articles
Next, we performed full article experiments, where we used the large model of sentence-BERT, as it outperformed the rest of the BERT models shown in Table 5. We extracted similarity scores for each claim-document pair using sentence-BERT:large. We then trained a simple neural network (20-relu10-relu) for classification. We trained the model for 15 epochs with a batch size of 2,048 using the Adam optimizer with a learning rate of 1e-3. We further used class weighting because the data was severely imbalanced: there were 614 positive examples out of 10M claim-document pairs, as we paired each of the 614 input claims with each of the 16,636 verified claims in the database.
We ran the experiment for various numbers of top- $n$  cosine scores obtained from the Body, as we wanted to investigate the relationship between the model performance and the information it uses.
In the BERT on Full Articles section in Table 5, we can see that using the scores for the top-4 best-matching sentences from the article body, together with scores for VerClaim and for the article title, yielded the best performance. Moreover, the results got closer to those for BM25, even though overall they still lag a bit behind.
7.1.4 Experiment 4:Reranking
Finally, we trained a pairwise RankSVM model to re-rank the top-  $N$  results retrieved using IR:Body. For each claim-document pair in the top-  $N$  list, we collected the scores for IR:Title, IR:VerClaim, IR:Body, as well as from sentence-BERT:large for  $n = 4$  with their corresponding reciprocal ranks for the rankings they induce. As described in Section 6.3, using both methods yields better predictions as this combines exact matching and semantic similarities.
We can see in Table 5 that the re-ranker yielded consistent and sizable improvement over the models from the previous experiments, by 0.04-0.05 points absolute across the different measures, which is remarkable as it is well-known from the literature that BM25 is a very strong baseline for IR tasks. This is because our reranker is able to use both exact and semantic matching to target the different kinds of pairs that are found in the dataset. We also notice that the performance of the re-ranker improves as we increase the length of the list that is being re-ranked until a length of 100, and it starts degrading after that.
7.2 Experiments on Snopes
On the Snopes dataset, we performed experiments analogous to those for the PolitiFact dataset, but with some differences, the most important being that this time we did not perform matching against the article body as the tweets that serve as input claims in our Snopes dataset were extracted from the article body. Note that this was not an issue for the PolitiFact dataset, as the input claim in a debate/speech required a lot of normalization and could not be found in the article body verbatim. Table 6 reports the evaluation results.
7.2.1 Experiment 1: BM25-based Baselines
We ran three experiments using BM25 to match the Input against Title, VerClaim, and Title+VerClaim. We can see in Table 6 that, just like for PolitiFact, using VerClaim performed better than using the article title, which is true for all evaluation measures; however, this time the margin was much smaller than it was for PolitiFact. We further noticed a small improvement for all MAP@k measures when matching against both the article Title and the VerClaim. Overall, BM25 is a very strong baseline for Snopes due to the high word overlap between the input claims and the verified claims (also, compared to PolitiFact, as we have seen in Table 4 above).
7.2.2 Experiment 2: Semantic Matching
Based on the lessons learned from PolitiFact, for semantic matching, we only experimented with sentence-BERT. We can see in Table 6 that this yielded results that were lower than for BM25 by a margin of at least 0.10 absolute for almost every reported measure; yet, this margin is smaller than for PolitiFact. For these experiments, once again matching against the verified claim outperformed matching against the article title by a sizable margin.
7.2.3 Experiment 3: BERT on Full Articles
As mentioned above, we did not perform matching of the input tweet against the article body, as this would easily give away the answer: the tweet can be found verbatim inside the target article.
For the purpose of comparison, we tried to filter out the text of the input tweet from the text of the article body before attempting the matching, but we still got unrealistically high results. Thus, ultimately we decided to abandon these experiments.
7.2.4 Experiment 4:Reranking
Finally, we trained a pairwise RankSVM model to re-rank the top- $N$  results from IR:VerClaim+Title. For each claim-document pair in the top- $N$  list, we extracted the scores from IR:Title, IR:VerClaim, IR:VerClaim+Title, sentence-BERT:large:Title, and sentence-BERT:large:VerClaim, as well as the corresponding reciprocal ranks for all target documents according to each of these scores. This is the same as for PolitiFact, except that now we do not use scores for matching the input to a document body. We can see in Table 6 that the best re-ranking model yielded sizable improvements over the best individual model by 0.09-0.18 points absolute on all evaluation measures.
Comparing the best re-ranking models for Snopes and PolitiFact, we can see that Snopes performed best when using a top-50 list, compared to top-100 for PolitiFact. We believe that this is due to the difference in performance of the retrieval models used to extract the top- $N$  pairs: for Snopes, IR:VerClaim+Title has an MMR score of 0.664, while the best PolitiFact model, IR:Body, has an MRR score of 0.565. Thus, for Snopes we rerank an  $N$ -best list extracted by a stronger IR model, and thus there is no need to go that deep in the list.
We have argued for the need to address detecting previously fact-checked claims as a task of its own right, which could be an integral part of automatic fact-checking, or a tool to help human fact-checkers or journalists. We have created specialized datasets, which we have released, together with our code, to the research community in order to enable further research. Finally, we have presented learning-to-rank experiments, demonstrating sizable improvements over state-of-the-art retrieval and textual similarity approaches.
In future work, we plan to extend this work to more datasets and to more languages. We further want to go beyond textual claims, and to take claim-image and claim-video pairs as an input.
This research is part of the Tanbih project, $^{14}$  which aims to limit the effect of 'fake news," disinformation, propaganda, and media bias by making users aware of what they are reading.
14http://tanhbih.qcri.org/
Zeynep Akkalyoncu Yilmaz, Shengjin Wang, Wei Yang, Haotian Zhang, and Jimmy Lin. 2019a. Applying BERT to document retrieval with Birch. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP '19, pages 19-24, Hong Kong, China.
Zeynep Akkalyoncu Yilmaz, Wei Yang, Haotian Zhang, and Jimmy Lin. 2019b. Cross-domain modeling of sentence-level evidence for document retrieval. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP '19, pages 3488-3494, Hong Kong, China.
Pepa Atanasova, Lluis Marquez, Alberto Barron-Cedeno, Tamer Elsayed, Reem Suwaileh, Wajdi Zaghouani, Spas Kyuchukov, Giovanni Da San Martino, and Preslav Nakov. 2018. Overview of the CLEF-2018 CheckThat! Lab on automatic identification and verification of political claims, Task 1: Check-worthiness. In Working Notes of the Conference and Labs of the Evaluation Forum, CLEF '18, Avignon, France.
Pepa Atanasova, Preslav Nakov, Georgi Karadzhov, Mitra Mohtarami, and Giovanni Da San Martino. 2019. Overview of the CLEF-2019 CheckThat! Lab on Automatic Identification and Verification of Claims. Task 1: Check-Worthiness. In Working Notes of the Conference and Labs of the Evaluation Forum, CLEF '19, Lugano, Switzerland.
Isabelle Augenstein, Christina Lioma, Dongsheng Wang, Lucas Chaves Lima, Casper Hansen, Christian Hansen, and Jakob Grue Simonsen. 2019. MultiFC: A real-world multi-domain dataset for evidence-based fact checking of claims. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP '19, pages 4684-4696, Hong Kong, China.
Ramy Baly, Georgi Karadzhov, Dimitar Alexandrov, James Glass, and Preslav Nakov. 2018a. Predicting factuality of reporting and bias of news media sources. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP '18, pages 3528-3539, Brussels, Belgium.
Ramy Baly, Mitra Mohtarami, James Glass, Lluis Marquez, Alessandro Moschitti, and Preslav Nakov. 2018b. Integrating stance detection and fact checking in a unified corpus. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT '18, pages 21-27, New Orleans, LA, USA.
Alberto Barron-Cedeno, Tamer Elsayed, Giovanni Da San Martino, Maram Hasanain, Reem Suwaileh, Fatima Haouari, and Preslav Nakov. 2020. CheckThat! at CLEF 2020: Enabling the automatic identification and verification of claims on social media. In Proceedings of the European Conference on Information Retrieval, ECIR '20, pages 499-507, Lisbon, Portugal.
Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. 2009. The fifth PASCAL recognizing textual entailment challenge. In Proceedings of the Text Analysis Conference, TAC '09, Gaithersburg, MD, USA.
Kevin R. Canini, Bongwon Suh, and Peter L. Pirolli. 2011. Finding credible information sources in social networks based on content and social structure. In Proceedings of the IEEE International Conference on Privacy, Security, Risk, and Trust, and the IEEE International Conference on Social Computing, SocialCom/PASSAT '11, pages 1-8, Boston, MA, USA.
Carlos Castillo, Marcelo Mendoza, and Barbara Poblete. 2011. Information credibility on Twitter. In Proceedings of the 20th International Conference on World Wide Web, WWW '11, pages 675-684, Hyderabad, India.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation, SemEval '17, pages 1-14, Vancouver, Canada.
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Brian Strope, and Ray Kurzweil. 2018. Universal sentence encoder for English. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP '18, pages 169-174, Brussels, Belgium.
Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2019. TabFact: A large-scale dataset for table-based fact verification.
Giovanni Luca Ciampaglia, Prashant Shiralkar, Luis M. Rocha, Johan Bollen, Filippo Menczer, and Alessandro Flammini. 2015. Computational fact checking from knowledge networks. PLOS ONE, 10(6):1-13.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT '19, pages 4171-4186, Minneapolis, MN, USA.
William B. Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing, IWP '05, Jeju Island, Korea.
Sebastian Dungs, Ahmet Aker, Norbert Fuhr, and Kalina Bontcheva. 2018. Can rumour stance alone predict veracity? In Proceedings of the 27th International Conference on Computational Linguistics, COLING '18, pages 3360-3370, Santa Fe, NM, USA.
Tamer Elsayed, Preslav Nakov, Alberto Barron-Cedeno, Maram Hasanain, Reem Suwaileh, Pepa Atanasova, and Giovanni Da San Martino. 2019. CheckThat! at CLEF 2019: Automatic identification and verification of claims. In Proceedings of the 41st European Conference on Information Retrieval, ECIR '19, pages 309-315, Cologne, Germany.
Mohamed H. Gad-Elrab, Daria Stepanova, Jacopo Urbani, and Gerhard Weikum. 2019a. ExFaKT: A framework for explaining facts over knowledge graphs and text. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, WSDM '19, pages 87-95, Melbourne, Australia.
Mohamed H. Gad-Elrab, Daria Stepanova, Jacopo Urbani, and Gerhard Weikum. 2019b. Tracy: Tracing facts over knowledge graphs and text. In *The World Wide Web Conference*, WWW '19, pages 3516–3520, San Francisco, CA, USA.
Pepa Gencheva, Preslav Nakov, Lluis Marquez, Alberto Barron-Cedeno, and Ivan Koychev. 2017. A context-aware approach for detecting worth-checking claims in political debates. In Proceedings of the International Conference on Recent Advances in Natural Language Processing, RANLP '17, pages 267-276, Varna, Bulgaria.
Genevieve Gorrell, Elena Kochkina, Maria Liakata, Ahmet Aker, Arkaitz Zubiaga, Kalina Bontcheva, and Leon Derczynski. 2019. SemEval-2019 task 7: RumourEval, determining rumour veracity and support for rumours. In Proceedings of the 13th International Workshop on Semantic Evaluation, SemEval '19, pages 845-854, Minneapolis, MN, USA.
Naeemul Hassan, Chengkai Li, and Mark Tremayne. 2015. Detecting check-worthy factual claims in presidential debates. In Proceedings of the 24th ACM International Conference on Information and Knowledge Management, CIKM '15, pages 1835-1838, Melbourne, Australia.
Naeemul Hassan, Gensheng Zhang, Fatma Arslan, Josue Caraballo, Damian Jimenez, Siddhant Gawsane, Shohedul Hasan, Minumol Joseph, Aaditya Kulkarni, Anil Kumar Nayak, Vikas Sable, Chengkai Li, and Mark Tremayne. 2017. ClaimBuster: The first-ever end-to-end fact-checking system. Proc. VLDB Endow., 10(12):1945-1948.