SlowGuess/NAACL_2021_OCR huggingface dataset:
In contrast, our perplexity-based classifiers manage to perform fairly well, even in the 2-shot setting, because our "classifier" is a single parameter (i.e., threshold value), which requires no complex learning or optimization. We would like to emphasize that ours consistently outperform the strong Transformer-based baselines across all dataset on the F1-Macro metric by absolute  $10\sim 20\%$ . We argue that these results demonstrate the strength of our approach in low-resource few-shot settings.
BERT vs. GPT2 for Perplexity Scores Most of the time, PPLGPT2-B outperforms PPLBERT-B. For instance, in the 50-shot setting for the FEVER dataset, performance differences are  $10.04\%$  and  $7.76\%$  for accuracy and F1-Macro
LM Type	Parameter Size	Covid-Scientific	Covid-Social	FEVER
Acc	F1 Macro	Acc	F1 Macro	Acc	F1 Macro
PPLGPT2-B	117M	74.73%	73.83%	73.63%	59.91%	67.48%	64.70%
PPLGPT2-M	345M	75.11%	73.93%	75.43%	60.23%	69.02%	66.39%
PPLGPT2-L	774M	76.19%	75.53%	73.29%	59.30%	71.66%	69.99%
PPLGPT2-XL	1558M	78.23%	77.63%	72.80%	59.88%	73.67%	71.71%
Table 4: Effect of LM parameter size on the performance of proposed perplexity-based approach in 50-shot setting. All the results are the mean value of three trials.
Shot #	Ablation	Covid-Scientific	Covid-Social	FEVER
Acc	F1 Macro	Acc	F1 Macro	Acc	F1 Macro
2	PPLGPT2-XL	68.52%	66.21%	66.62%	52.68%	62.37%	56.35%
-evidence-conditioning	62.92%	59.53%	64.32%	51.37%	54.72%	45.65%
50	PPLGPT2-XL	78.23%	77.63%	72.80%	59.88%	73.67%	71.71%
-evidence-conditioning	73.35%	70.21%	71.97%	56.08%	56.69%	47.58%
Table 5: Ablation study - Effect of the evidence-conditioning on the classification performance.
scores respectively. Based on this observation, we can speculate that the perplexity from a unidirectional LM is more suitable for our proposed method than from a masked LM. This is most likely because the BERT perplexity score is only an estimation based on the "pseudo-perplexity" proposed by Salazar et al.
In this section, we conduct multiple analysis to further evaluate and understand aspects of our perplexity-based approach.
6.1 Scaling the Language Model Size
Generally, scaling the model size helps to also improve the model performance, because more parameters mean a stronger learning capability during fine-tuning or training. Also, Roberts et al. have demonstrated that increasing the parameter size allows for more knowledge to be packed into the LM's parameters. Therefore, we experiment with the model size to see if such findings also extend to our proposed methodology. The following model sizes of GPT2 are investigated: base (PPLGPT2-B), medium (PPLGPT2-M), large (PPLGPT2-L) and xl (PPLGPT2-XL).
Results are reported in Table 4. As expected, we can observe the trend that the performance increases with parameter size. For instance,  $\mathsf{PPL}_{\mathsf{GPT2 - XL}}$  is the best performing compared to the other, smaller, models for Covid-Scientific and FEVER, achieving the new state-of-the-art few
shot results by gaining absolute  $\sim 4\%$  on Covid-Scientific and  $\sim 2\%$  on FEVERfor accuracy/F1-Macro.
We carry out an ablation study on the effect of evidence-conditioning in respect of the final perplexity scores and the corresponding final classification performance. In Table 5, we can observe the performance drops when evidence-conditioning is ablated – the biggest drop is  $\sim 15\%$  on F1-Macro for the FEVER task in the 50-shot setting. This implies that the perplexity score is assigned in relation to the context of the provided evidence.
6.3 Negation Analysis
In fact-checking, negation is one of the most difficult challenges, and many state-of-the-art models are brittle against it. Thorne and Vlachos show that the winning fact-checking systems from the FEVER workshop are brittle against negations, experiencing a huge performance drop when given negated test sets, up to absolute  $-29\%$  in accuracy. Therefore, we also conduct analysis regarding the negation handling of our proposed methods by augmenting our dataset with negated examples.
Template-based Data Negation We create our negated dataset by replacing all the auxiliary verbs (e.g., is, can) with their corresponding negated forms (e.g., is not, can not), and vice versa. We apply this approach to the Covid-Scientific dataset and obtain a new version that contains {original-
(a) Covid-Scientific
(b) Covid-Social
Figure 2: Precision at top-k. Blue-colored marks indicate precision at each of k when claims are assigned with perplexity scores from GPT2-base to rank claims in reverse order (i.e., higher to lower score). Orange-colored marks indicate mean precision value of 10 trials when random scores are assigned to each claim to rank.
Shot #	Test Set	Models	Acc	F1-Macro
2	Original	RoBERTaft	52.66%	34.29%
PPLGPT2-B	66.75%	64.39%
Negation-Augmented	RoBERTaft	46.75%	31.66%
PPLGPT2-B	52.98%	50.99%
Table 6: Negation Analysis - Comparison between finetuned RoBERTa baseline classifier (RoBERTa<sub>ft</sub>) and our perplexity-based classifier (PPL<sub>GPT2-B</sub>) on original Covid-Scientific dataset and its negation-augmented version in 2-shot setting.
sentence  $(S_{\mathrm{original}})$ , negated-sentence  $(S_{\mathrm{negated}})$  pairs. Note that the evidence is kept the same, but the veracity label of  $S_{\mathrm{original}}$  is negated (i.e., Supported is negated toUnsupported and vice versa). To illustrate with an example,  $S_{\mathrm{original}} = \{\text{"claim": "5g helps covid-19 spread.", }$  "evidence":  $evidence_1$  , "label":Unsupported} is negated into  $S_{\mathrm{negated}} = \{\text{"claim": "5g does not help covid-19 spread.", "evidence": }evidence_1$  "label":Supported}.
Q1: Can the LM distinguish negation? We use the new augmented Covid-Scientific dataset to investigate whether the LM manages to differentiate between the original-sentence  $S_{\mathrm{original}}$  and negated-sentence  $S_{\mathrm{negated}}$ . The average of the absolute difference between the perplexities assigned to  $S_{\mathrm{original}}$  and  $S_{\mathrm{negated}}$  is 122 and the maximum absolute difference value is 2800.
Q2: Performance on negation-augmented dataset? We evaluate the performance of the perplexity-based classifier (PPLGPT2-B) on the "negation-augmented" Covid-Scientific dataset in reference to its original. Unsurprisingly,
$\mathsf{PPL}_{\mathsf{GPT2 - B}}$  does experience a drop in performance of  $13.77\%$  and  $13.40\%$  in accuracy and F1-Macro, respectively. However, it still outperforms the fine-tuned RoBERTa<sub>ft</sub> baseline, the best performing baseline in the 2-shot setting, as shown in Table 6.
6.4 Comparison with existing FEVER System in Few-shot Setting
For all three tasks, we compare our perplexity models against different fine-tune baselines in Section 5.4. Unlike two newly proposed COVID-19-related tasks, FEVER is a well-established task studied by many existing works. In order to understand how our perplexity-based method compares against the literature, we conduct an additional experiment with the publicly available system from the runner-up team of the FEVER workshop, HexaF (Yoneda et al., 2018b).
We fine-tune HexaF's veracity classification modules in few-shot settings. In the 2-shot setting, HexaF shows accuracy of  $49.99\%$  and F1-Macro score of  $33.33\%$ . In the 50-shot setting, it shows accuracy of  $53.53\%$  and F1-Macro score of  $49.27\%$ . In general, machine learning models require sufficient amounts of training data, and this "sufficient amount" normally differs depending on the model being used. However, as demonstrated earlier in our main experimental results (Section 5.4),  $2 \sim 50$  samples are insufficient data to properly train one of the winning fact-checking systems.
6.5 Potential Application: Ranking of Candidate Claims for Fact-Checking
Here, we discuss another way of leveraging the evidence-conditioned perplexity score. It can be
used for prioritizing false-claim candidates for human fact-checkers, instead of doing hard prediction on the veracity of the given claims. By ranking the claims-to-be-fact-checked in descending order of perplexity, we can increase the chance that the first  $k$  claims checked by a human fact-checker areUnsupported false claims. This will be beneficial since fact-checkers can efficiently allocate their time and resources on fact-checking claims that are more likely to be false and harmful to society.
In Figure 2, we compare the precision at the top-k  $(\mathrm{P}@\mathrm{k})$  between the perplexity-based ranking and random-score-based ranking. We can view  $\mathrm{P}@\mathrm{k}$  to measure how manyUnsupported pieces are prioritized in the first  $k$  of the ranked claims. Across all datasets, perplexity-based ranking (blue marks) exhibits higher precision scores over random-score-based ranking (orange marks). Moreover, for both Covid-Scientific and Covid-Social, our  $\mathrm{P}@\mathrm{k}$  is over  $80\%$  for all  $k$  values.
7 Future Research Directions
In this work, we conduct the FEVER experiments in a binary set-up to keep all the experimental settings consistent across all three datasets. However, the original FEVER task has three classes - Support, Refute, and Not Enough Info (NEI). Since the distinction between NEI and Refute cases is also an important problem, it would be important future work to extend our binary-class setting to the three-class setting.
Moreover, we believe our method can easily be augmented into other existing approaches, for instance, leveraging the perplexity score in the final step of the FEVER fact-checkers as additional input. It would be a useful future direction to explore and discover the most effective way of incorporating the perplexity-based approach into other existing fact-checking systems.
In this paper, we propose a novel way of leveraging the perplexity score from LMs for the few-shot fact-checking task. Through experimental analysis from an ablation study to the discussion of potential applications, we further explore and evaluate the capability of the perplexity score to act as an indicator of unsupported claims. We hope our proposed approach encourages future research to continue developing LM-based methodologies as well as the few-shot approach for fact-checking. By doing so,
our community can move towards a data-efficient approach that is not constrained by the requirement of a large labeled dataset.
Tariq Alhindi, Savvas Petridis, and Smaranda Muresan. 2018. Where is your evidence: Improving fact-checking by justification modeling. In Proceedings of the First Workshop on Fact Extraction and Verification (FEVER), pages 85-90, Brussels, Belgium. Association for Computational Linguistics.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2019. Tabfact: A large-scale dataset for table-based fact verification. In International Conference on Learning Representations.
Maram Hasanain, Reem Suwaileh, Tamer Elsayed, Alberto Barrón-Cedeno, and Preslav Nakov. 2019. Overview of the ccef-2019 checkthat! lab: Automatic identification and verification of claims. task 2: Evidence and factuality. In CLEF (Working Notes).
Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Eisenschlos. 2020. TaPas: Weakly supervised table parsing via pre-training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4320-4333, Online. Association for Computational Linguistics.
Christopher Hidey, Tuhin Chakrabarty, Tariq Alhindi, Siddharth Varia, Kriste Krstovski, Mona Diab, and Smaranda Muresan. 2020. DeSePtion: Dual sequence prediction and adversarial examples for improved fact-checking. In Proceedings of the 58th Annual Meeting of the Association for Computational
Linguistics, pages 8593-8606, Online. Association for Computational Linguistics.
Georgi Karadzhov, Preslav Nakov, Lluis Márquez, Alberto Barrón-Cedeno, and Ivan Koychev. 2017. Fully automated fact checking using external sources. In Proceedings of the International Conference on Recent Advances in Natural Language Processing, RANLP 2017, pages 344-353, Varna, Bulgaria. INCOMA Ltd.
Nayeon Lee, Belinda Z. Li, Sinong Wang, Wen-tau Yih, Hao Ma, and Madian Khabsa. 2020. Language models as fact checkers? In Proceedings of the Third Workshop on Fact Extraction and VERIFICATION (FEVER), pages 36-41, Online. Association for Computational Linguistics.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Advances in Neural Information Processing Systems, volume 33, pages 9459-9474. Curran Associates, Inc.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Jing Ma, Wei Gao, Shafiq Joty, and Kam-Fai Wong. 2019. Sentence-level evidence embedding for claim verification with hierarchical attention networks. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2561-2571, Florence, Italy. Association for Computational Linguistics.
Andrea Madotto, Zihan Liu, Zhaojiang Lin, and Pascale Fung. 2020. Language models as few-shot learner for task-oriented dialogue systems.
Yixin Nie, Haonan Chen, and Mohit Bansal. 2019. Combining fact extraction and verification with neural semantic matching networks. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):6859-6866.
Fabio Petroni, Tim Rocttäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2463-2473, Hong Kong, China. Association for Computational Linguistics.
Nina Poerner, Ulli Waltinger, and Hinrich Schütze. 2019. Bert is not a knowledge base (yet): Factual knowledge vs. name-based reasoning in unsupervised qa. ArXiv, abs/1911.03681.
Kashyap Popat, Subhabrata Mukherjee, Jannik Strögen, and Gerhard Weikum. 2017. Where the truth lies: Explaining the credibility of emerging claims on the web and social media. In Proceedings of the 26th International Conference on World Wide Web Companion, WWW '17 Companion, page 1003-1012, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee.
Kashyap Popat, Subhabrata Mukherjee, Jannik Strögen, and Gerhard Weikum. 2018a. Credeye: A credibility lens for analyzing and explaining misinformation. In *Companion Proceedings of the The Web Conference* 2018, WWW '18, page 155-158, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee.
Kashyap Popat, Subhabrata Mukherjee, Andrew Yates, and Gerhard Weikum. 2018b. DeClarE: Debunking fake news and false claims using evidence-aware deep learning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 22-32, Brussels, Belgium. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9.
Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5418-5426, Online. Association for Computational Linguistics.
Julian Salazar, Davis Liang, Toan Q. Nguyen, and Katrin Kirchhoff. 2020. Masked language model scoring. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2699-2712, Online. Association for Computational Linguistics.
Dan Su, Yan Xu, Tiezheng Yu, Farhad Bin Siddique, Elham J Barezi, and Pascale Fung. 2020. Cairecovid: A question answering and multi-document summarization system for Covid-19 research. arXiv preprint arXiv:2005.03975.
James Thorne and Andreas Vlachos. 2019. Adversarial attacks against fact extraction and verification. arXiv preprint arXiv:1903.05543.
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERIFICATION. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 809-819, New Orleans, Louisiana. Association for Computational Linguistics.
Santosh Tokala, Vishal G, Avirup Saha, and Niloy Ganguly. 2019. AttentiveChecker: A bi-directional attention flow mechanism for fact verification. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2218-2222, Minneapolis, Minnesota. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, pages 5998-6008.
Lianwei Wu, Yuan Rao, Xiong Yang, Wanzhen Wang, and Ambreen Nazir. 2020. Evidence-aware hierarchical interactive attention networks for explainable claim verification. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pages 1388-1394. International Joint Conferences on Artificial Intelligence Organization.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems, volume 32, pages 5753-5763. Curran Associates, Inc.
Takuma Yoneda, Jeff Mitchell, Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018a. UCL machine reading group: Four factor framework for fact finding (HexaF). In Proceedings of the First Workshop on Fact Extraction and VERIFICATION (FEVER), pages 97-102, Brussels, Belgium. Association for Computational Linguistics.
Takuma Yoneda, Jeff Mitchell, Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018b. UCL machine reading group: Four factor framework for fact finding (HexaF). In Proceedings of the First Workshop on Fact Extraction and VERIFICATION (FEVER), pages 97-102, Brussels, Belgium. Association for Computational Linguistics.
Wanjun Zhong, Jingjing Xu, Duyu Tang, Zenan Xu, Nan Duan, Ming Zhou, Jiahai Wang, and Jian Yin. 2020. Reasoning over semantic-level graph for fact checking. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6170-6180, Online. Association for Computational Linguistics.
Jie Zhou, Xu Han, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. 2019. GEAR: Graph-based evidence aggregating and reasoning for fact verification. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 892-901, Florence, Italy. Association for Computational Linguistics.