SlowGuess/NAACL_2021_OCR huggingface dataset:
Nayeon Lee†* Yejin Bang†* Andrea Madotto† Madian Khabsa§ Pascale Fung† †Hong Kong University of Science and Technology §Facebook AI {nyleea, yjbang}@connect.ust.hk
Few-shot learning has drawn researchers' attention to overcome the problem of data scarcity. Recently, large pre-trained language models have shown great performance in few-shot learning for various downstream tasks, such as question answering and machine translation. Nevertheless, little exploration has been made to achieve few-shot learning for the fact-checking task. However, fact-checking is an important problem, especially when the amount of information online is growing exponentially every day. In this paper, we propose a new way of utilizing the powerful transfer learning ability of a language model via a perplexity score. The most notable strength of our methodology lies in its capability in few-shot learning. With only two training samples, our methodology can already outperform the Major Class baseline by more than an absolute  $10\%$  on the F1-Macro metric across multiple datasets. Through experiments, we empirically verify the plausibility of the rather surprising usage of the perplexity score in the context of fact-checking and highlight the strength of our few-shot methodology by comparing it to strong fine-tuning-based baseline models. Moreover, we construct and publicly release two new fact-checking datasets related to COVID-19.
Few-shot learning is being actively explored to overcome the heavy dependence on large-scale labeled data that serves as a crucial bottleneck to machine learning models. Recently, researchers have explored few-shot learning that leverages the powerful transfer learning ability of pre-trained large language models (LMs) in various NLP tasks. Petroni et al. demonstrated that an LM serves as a good zero-shot learner on the question-answering task due to its encoded commonsense knowledge.
Figure 1: Illustration of our simple yet effective perplexity-based approach. Few-shot data samples are used to find the optimal perplexity threshold  $th$  that separatesUnsupported claims from Supported claims.
Going further, Brown et al. illustrated the impressive potential of LMs as strong zero-shot and few-shot learners across translation, commonsense reasoning and natural language inference (NLI). However, little or no exploration has been made on few-shot learning in the fact-checking domain, which is a timely and important task in which data-scarcity is particularly problematic.
Previous works have proposed different ways of leveraging LMs to conduct zero- or few-shot learning. One common approach is to query the LM for the missing token (i.e., "answer") for the zero-shot question-answering task (Petroni et al., 2019; Brown et al., 2020) by transforming questions into a form of statement. Another approach is to adopt an in-context learning approach where the input context of the LM is carefully crafted to control the output. For example, a natural language task instruction (e.g., "Translate English to French:") or training sample (e.g., "sea otter => loutre de mer") is provided as the context for zero-shot/few shot translation (Brown et al., 2020).
In this work, we explore a new way of leveraging LMs for few-shot learning in the fact-checking task. This is done by leveraging a perplexity score from
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1971-1981 June 6-11, 2021. ©2021 Association for Computational Linguistics
Unsupported Claims	Perplexity	Supported Claims	Perplexity
5G network can spread diseases.	826.70	Beyonce is one of the most famous singers in the world.	23.03
All dogs speak English fluently.	328.23	Chicago is one of the cities in the United States.	43.92
Washing hands helps the spread of diseases.	201.10	Washing hands prevents the spread of diseases.	96.74
Table 1: Relations between veracity of claim and perplexity.Unsupported claims have higher perplexity compared to Supported claims. Note that the perplexity score listed here is using GPT2-base on each of the claims.
evidence-conditioned LMs. Fact-checking is the task of verifying a claim based on its corresponding evidence, and one of its most important objectives is to correctly model the relationship between the given claim and evidence. We hypothesize that a perplexity score from evidence-conditioned LMs is helpful for such purpose since perplexity measures the likelihood of a given sentence with reference to previously encountered text (i.e., given the evidence prefix and the LM's training corpus). Therefore, this paper attempts to investigate this hypothesis and proposes a novel perplexity-based few-shot learning methodology for fact-checking.
Through experimental analysis, we empirically demonstrate the effectiveness of our proposed methodology in few-shot learning, and we compare it to strong fine-tuning-based baselines. Moreover, we compare different LMs (BERT and GPT2) in different sizes, from small to XL, to unveil interesting insights on which model is more suitable for this task. Finally, we discuss the potential application of evidence-conditioned perplexity for ranking candidate claims in priority order of the most urgent to be fact-checked to the least.
Our contribution is three-fold: First, we propose an effective way of leveraging the perplexity score in the context of fact-checking. We would like to emphasize that our approach is a simple yet effective way of leveraging large pre-trained LMs. Second, we demonstrate the effectiveness of the perplexity-based approach in the few-shot setting by outperforming strong fine-tuned baselines, such as BERT (Devlin et al., 2019), RoBERTA (Liu et al., 2019), and XLNet (Yang et al., 2019), by an absolute  $10 \sim 20\%$  F1-Macro scores in the 2-,10-, and 50-shot settings. Third, we construct two new fact-checking datasets related to COVID-19, which has caused the problem of an "infodemic".
Fact-checking is a complex task that is split into many sub-tasks. First, credible sources of evidence need to be identified. Second, a set of relevant evi
dence needs to be retrieved from the identified credible sources. Last, veracity classification of claims can be made based on the retrieved evidence.
Some works have focused on full-pipeline systems that handle all sub-tasks and provide real working web prototypes (Karadzhov et al., 2017; Popat et al., 2017, 2018a; Hasanain et al., 2019; Tokala et al., 2019). These works use the entire Web as a knowledge source to confirm or reject a claim taking the credibility or reliability of the Web source into account. Another common setting for fact-checking is to assume a credible evidence source is given (e.g., Wikipedia), and to focus on the evidence retrieval and veracity verification steps only. FEVER (Thorne et al., 2018) and Tabfact (Chen et al., 2019) are two large datasets for this setting, and there are many follow-up studies working on them (Yoneda et al., 2018a; Nie et al., 2019; Zhong et al., 2020; Herzig et al., 2020; Zhou et al., 2019; Hidey et al., 2020).
Our work follows the latter group of works and uses the following setting: given a tuple consisting of claims and relevant evidence, we classify the final fact-checking veracity label of the given claim (Popat et al., 2018b; Ma et al., 2019; Wu et al., 2020). By doing this, we focus on the methodology for the veracity classification task without worrying about the propagated errors from earlier modules, such as source credibility profiling and evidence retrieval.
Leveraging LMs as a knowledge base, zero-shot learner or a few-shot learner has been gaining popularity within the NLP field. It was discovered that large pre-trained LMs can store factual knowledge in their parameters (Petroni et al., 2019; Roberts et al., 2020; Madotto et al., 2020), and that this stored knowledge can help LM to be good at zero-shot and few-shot learning in various NLP tasks, such as question answering, summarization, textual entailment, translation and commonsense reasoning (Brown et al., 2020). For the task of fact-checking, Lewis et al. and Lee et al. attempted to
leverage such LMs. However, they mainly use the model to replace the evidence retriever of the fact-checking pipeline, and they still require training of final veracity classifier. Our work, in contrast, focuses on the few-shot ability of LMs for veracity classification.
3 Preliminary Exploration of Hypothesis
In this section, we conduct a preliminary investigation to validate the potential of our hypothesis that the perplexity score from an evidence-conditioned LM can provide a signal for claims unsupported by evidence.
For our exploration, we first collect a small set of Supported and Unsupported claims that can be verified based on the training corpus of the target LM (namely, Wikipedia which is used in the training of many pre-trained LMs). Then, we compare the perplexity scores between them.
To recap, perplexity is a commonly used metric for measuring the performance of LMs. It is defined as the inverse of the probability of the test set normalized by the number of words:
$$
P P L (X) = \sqrt [ n ]{\prod_ {i = 1} ^ {n} \frac {1}{p \left(x _ {i} \mid x _ {0} , \dots , x _ {i - 1}\right)}}. \tag {1}
$$
Another way of interpreting perplexity is as a measure of the likelihood of a given test sentence with reference to the training corpus.
From Table 1, we can observe that Unsupported claims on average have higher perplexity than Supported claims. For example, Supported claim "Washing hands prevents the spread of diseases," has a perplexity value of 96.74, whereas theUnsupported claim "All dogs speak English fluently," has a much higher perplexity value of 328.23. We believe these observations support our hypothesis. Thus, we proceed to build our approach based on this hypothesis (Section 4), and conduct experiments (Section 5) and analysis (Section 6) to verify the validity of our perplexity-based fact-checking approach.
In this work, we define our task to be: Given a {claim, evidence} pair, determine the veracity of a claim against the evidence - i.e.,
Supported vs.Unsupported claims. The label Supported is assigned when there exists relevant evidence that supports the claim, whileUnsupported is assigned when there does not exist any supporting evidence. Note that this existence of refuting evidence also places a claim into this latter category.
4.2 Evidence Conditioned Perplexity
Although previous works have shown that an LM can encode knowledge from its training corpus, there are a few limitations to solely relying on the pre-trained weights. First, we cannot easily check and guarantee whether the LM has already seen the evidence that is required for verification, and the LM would definitely not have seen the evidence related to newly emerging events after the LM pretraining. For instance, the event of COVID-19 emerged after the release of the GPT2 pre-trained model. Second, although LMs have shown surprising ability in memorizing some knowledge, they are not perfect, as pointed out by previous works (Poerner et al., 2019; Lee et al., 2020). Therefore, we propose to incorporate evidence into the perplexity calculation by using it as a prefix of the claim.
There are two popular kinds of LMs: i) unidirectional LMs that are trained with the conventional next token prediction task, and ii) masked LMs that are trained with the masked token prediction token, resulting in a bidirectional LM. We briefly describe how to obtain the evidence-conditioned perplexity for both types of LM:
Unidirectional Language Model Perplexity For a unidirectional LM, first we concatenate the evidence and claim to obtain the input to the LM:  $X = \{x_{e_0},\ldots ,x_{e_E},x_{c_0}\ldots ,x_{c_C}\}$  , where  $E$  and  $C$  denote the number of evidence tokens and claim tokens, respectively. Then, we obtain the evidenceconditioned perplexity by
$$
P P L (X) = \sqrt [ C ]{\prod_ {i = 1} ^ {C} \frac {1}{p \left(x _ {c _ {i}} \mid x _ {e _ {0}} , \dots , x _ {e _ {E}} , \dots , x _ {c _ {i - 1}}\right)}}.
$$
Note that the evidence tokens are used to condition the perplexity, yet their conditional probabilities  $p(x_{e_i} | x_{e_0}, \ldots, x_{e_{i-1}})$  do not contribute to the  $PPL(X)$ , which is the main difference from Eq. (1).
Data sets	Unsupported claims	Supported claims	Total
Covid19-Scientific	101	71	172
Covid19-Social	263	77	340
FEVER	3333	3333	6666
Table 2: Dataset Statistics
Masked Language Model Pseudo Perplexity A masked LM (MLM) is a type of LM, first proposed by Devlin et al., which is trained with the masked token prediction task instead of the next token prediction task. The "perplexity" score from the MLM does not mean the same as the conventional perplexity score. Therefore, we use the "pseudo perplexity" score proposed by Salazar et al., which is computed by summing all the log probabilities obtained by sequentially masking each token in the input sentence.
4.3 Leveraging Perplexity
Once we obtain the evidence-conditioned perplexity scores for each claim, we find the best threshold  $th$  that separates Supported claims fromUnsupported claims. We would like to emphasize that our approach does not involve any parameter update of the LM. We only do inference with the LM, and leverage the few-shot samples as the "validation set" to find the optimal single threshold parameter,  $th$ . Throughout our paper, we refer to our methodology as the "perplexity-based classifier".
Given a set of a claim and evidence, if the evidence-conditioned perplexity score is less than the threshold (i.e.  $< th$ ), the claim is Supported by the evidence; otherwise it is Unsupported.
5 Few-shot Experiment
All datasets used in the experiment are in English, and we report the data statistics in Table 2.
Covid19-Scientific A new test set is constructed by collecting COVID-19-related myths and scientific truths labelled by reliable sources like MedicalNewsToday, the Centers for Disease Control and Prevention (CDC), and the World Health Organization (WHO). It consists of the most com
mon scientific or medical myths about COVID-19, which must be debunked correctly to ensure the safety of the public (e.g., "drinking a bleach solution will prevent you from getting COVID-19"). The set contains 172 claims with labels (Supported, Unsupported) obtained from the aforementioned reliable sources. Note that myths that are unverifiable from current findings are also assigned theUnsupported label. $^{2}$
The gold evidence is obtained from the winning system of the Kaggle Covid-19 challenge (Su et al., 2020). This system retrieves the evidence from 59,000 scholarly articles about COVID-19, SARS-CoV-2, and other related corona viruses.3
Covid19-Social Another test set is constructed by crawling 340 COVID-19-related claims fact-checked by journalists from a website called Politi-fact.com. Unlike the Covid19-Scientific dataset, it contains non-scientific and socially-related claims, such as "For the coronavirus, the death rate in Texas, per capita of 29 million people, we're one of the lowest in the country." Such claims may not be life-and-death matters, but they still have the potential to bring negative sociopolitical effects. Originally, these claims are labelled into six classes {pants-fire, false, barely-true, half-true, mostly-true, true}. However, we use it in a binary setup for consistency with the Covid19-Scientific setup by assigning the first three classes to Unsupported and the rest to Supported.
For evidence of each claim, we follow the Alhindi et al. to obtain the human-written evidence/justification available on the Politifact.com website, from which the claims are crawled.
FEVER (Thorne et al., 2018) Fact Extraction and Verification (FEVER) is a publicly released large-scale dataset generated by altering sentences extracted from Wikipedia to promote research on fact-checking systems. Since our few-shot experiment requires little data, we only leverage the "Paper Test Dataset" from the FEVER workshop (https://fever.ai/) resource page to speed up our experiments.
This dataset originally has three classes, {Support, Refute, Not Enough Info}. "Support" is sim
<sup>1</sup>Authors from HKUST obtained performed all experiments with the existing datasets and compiled and released the new datasets.
2Disclaimer: The data were collected during the early outbreak of COVID-19 (March 2020). The veracity may have been updated as the time evolved, but we release the original version of the dataset for future comparison
<sup>3</sup>https://www.kaggle.com/allen-institute-for-ai/CORD-19-research-challenge
ilar to our Supported label, where a claim can be supported by given evidence. "Refute" is where a claim is "refuted" by given evidence, whereas "Not Enough Info" means not enough evidence is available for verification. For our FEVER experiment, we treat "Refute" and "Not Enough Info" as one class. This is because we believe that in a real scenario both cases areUnsupported claims that need attention.
To provide further detail, the "Support" class is mapped into Supported, and "Refute"/"Not Enough Info" is mapped into Unsupported to match our task setting. Note that to balance the dataset, we obtain half the data from "Refute" and the other half from "Not Enough Info". Note that the gold evidence is included in the dataset released by Thorne et al.
Ours We consider one unidirectional LM and one masked LM for our proposed perplexity-based methodology.
- PPLGPT2-B - Our single-parameter classifier based on perplexity from GPT2-base (Radford et al., 2019) (unidirectional LM)
- PPLBERT-B - Our single-parameter classifier based on perplexity from BERT-base (Devlin et al., 2019) (Masked LM)
Baselines We finetune various pre-trained Transformer-based (Vaswani et al., 2017) models to build our baseline classifiers, which is a common approach used to achieve many state-of-the-art results in the literature.
- Major Class - A simple majority classifier which always assigns the majority class of the training set to all samples. We provide this for reference because some of our dataset classes are imbalanced.
- BERT- $B_{\text{ft}}$  - A fine-tuned BERT-base model with a feed-forward classifier trained on top.
- BERT- $\mathsf{L}_{\mathsf{ft}}$  - A fine-tuned BERT-large model with a feed-forward classifier trained on top.
- RoBERTaft - A fine-tuned RoBERTa-base model (Liu et al., 2019) with a feed-forward classifier trained on top.
- XLNet $_{ft}$  - A fine-tuned XLNet-base model (Yang et al., 2019) with a feed-forward classifier trained on top.
Few-Shot Data Setup Given  $N_{D}$  as the size of the dataset  $D$ , we do an  $n$ -shot experiment with  $n$  samples from  $D$  as a "validation set" for our perplexity-based approach or as a "training set" for the fine-tuning approach, and the remainder  $(N_{D} - n)$  as a test set. To give a concrete example, in the 2-shot experiment using the Covid19-Social dataset (340 samples), we have two training samples and 338 test samples. We use three seeds to split the datasets and train the models. For a fair comparison, all the seeds and splits are kept the same across the models.
Evaluation We mainly evaluate our experiments using accuracy and the Macro-F1 metric. Since some of our datasets are imbalanced (the ratio of Supported toUnsupported in Table 2), we prioritize the overall Macro-F1 score over accuracy.
Training Details In our methodology, no gradient update is required. Thus, there are no training details such as learning rate, batch size or maxepoch to report. We simply use a small validation set (size of 2,10,50) to find the best-performing hyper-parameter value for the threshold  $th$  from the range of  $\{0 \sim 1000\}$ . None of the samples from the test set were seen in threshold searching.
For baseline fine-tuned classifiers, we do a grid-search to find the best-performing parameters, as follows: We use a learning rate of  $5\mathrm{e} - 6$  for training the BERT- $B_{\mathrm{ft}}$ , ROBERTaft, and XLNetft models, while BERT- $L_{\mathrm{ft}}$  is trained with a rate of  $2\mathrm{e} - 5$ . All models share the same batch size of 32 and maximum input sequence length of 128. We also use early-stopping with patience 3 with a maximum of 10 training epochs. Each experiment is run on an Nvidia GTX 1080 Ti, and each epoch takes  $2 \sim 15$  seconds depending on the number of the training samples  $n$ . Note that for reproducibility, we will also publicly release the code.
Table 3 reports the few-shot performance of the fine-tuning-based baselines and our perplexity-based classifiers.
Usage of Perplexity We can observe that our perplexity-based classifiers, especially  $\mathsf{PPL}_{\mathsf{GPT2 - B}}$  outperform all Major Class baselines across all tasks in all settings. For instance,  $\mathsf{PPL}_{\mathsf{GPT2 - B}}$  outperforms the Major Class by a great margin of  $16\%$  and  $36.8\%$  on accuracy and F1-Macro scores,
Shot #	Models	Fine-tuning?	Size	Covid-Scientific	Covid-Social	FEVER
Acc	F1-Macro	Acc	F1-Macro	Acc	F1-Macro
Major Class	N/A	N/A	58.72%	37.00%	77.35%	43.62%	50.00%	33.33%
2	BERT-Bft	yes	110M	47.34%	32.21%	26.11%	23.33%	51.56%	37.34%
BERT-Lft	yes	336M	49.39%	34.80%	37.78%	27.81%	50.80%	36.49%
RoBERTaft	yes	125M	52.66%	34.29%	40.75%	26.78%	50.00%	33.33%
XLNetft	yes	110M	51.48%	48.49%	57.67%	44.35%	49.41%	44.65%
PPLGPT2-B	no	117M	66.75%	64.39%	62.61%	53.61%	61.92%	57.50%
PPLBERT-B	no	110M	47.93%	38.54%	77.74%	49.15%	52.54%	41.33%
10	BERT-Bft	yes	110M	46.27%	31.70%	43.26%	30.70%	51.56%	37.34%
BERT-Lft	yes	336M	50.00%	36.74%	60.49%	42.18%	50.80%	36.49%
RoBERTaft	yes	125M	52.64%	40.28%	40.73%	26.73%	50.00%	33.33%
XLNetft	yes	110M	49.69%	42.44%	59.68%	39.45%	49.41%	44.65%
PPLGPT2-B	no	117M	72.98%	68.57%	71.23%	55.11%	62.82%	57.04%
PPLBERT-B	no	110M	63.15%	60.77%	61.90%	46.35%	57.59%	57.11%
50	BERT-Bft	yes	110M	56.75%	53.61%	60.21%	36.91%	52.18%	38.82%
BERT-Lft	yes	336M	56.75%	39.15%	64.94%	44.07%	51.14%	39.99%
RoBERTaft	yes	125M	56.40%	38.97%	73.13%	45.30%	50.44%	38.15%
XLNetft	yes	110M	63.22%	51.98%	77.62%	43.70%	49.18%	48.42%
PPLGPT2-B	no	117M	74.73%	73.83%	73.63%	59.91%	67.48%	64.70%
PPLBERT-B	no	110M	62.53%	61.11%	71.11%	54.72%	57.44%	56.94%
Table 3: Results comparison among perplexity-based classifiers and fine-tuned classifiers in 2-shot, 5-shot and 10-shot settings across three different tasks. Models whose names start with PPL are our proposed perplexity-based classifiers. Major Class is a reference to evaluate classifier performance. All test results reported are mean values of three trials with randomly selected n-shot training samples from the dataset, where  $n = \{2,10,50\}$ .
for the Covid-Scientific dataset in the 50-shot setting. This supports our hypothesis that evidence-conditioned perplexity scores are capable of providing signals regarding the veracity of the given claim.
Intuitively, we can consider the perplexity score to be mimicking the role of the "logits" from a classifier, and we are trying to find the best threshold to map this pseudo-logit-like perplexity score into a veracity label. The classification performance of our perplexity-based approach increases as the shot size increases. As the shot size increases from 2 to 50,  $\mathsf{PPL}_{\mathsf{GPT2 - B}}$  shows an average gain of  $8.19\pm 2.74\%$  and  $7.64\pm 1.61\%$  in accuracy and Macro-F1 score, respectively, across all tasks. This is because a greater number of data samples means more anchor perplexity points for threshold searching, and thus, a better threshold to determine the veracity of claims.
Few-shot Comparison to Fine-tuned Baselines Except for the Covid-Social accuracy in the 50-shot setting, both of our proposed classifiers (PPLGPT2-B, PPLBERT-B) outperform the fine-tuned baseline classifiers across all tasks in all of the 2-, 10- and 50-shot settings. For the 2-shot and
10-shot settings, many of the baseline classifiers underperform the Major Class baseline regardless of the task. This implies their failure to learn anything from the fine-tuning step with a limited number of samples. Only after 50-shot do these baselines start to learn and outperform the Major Class baselines. This is not surprising, since the pre-trained models are known to perform well in a full-shot scenario, but they do not guarantee good performance when they are shown few samples.