SlowGuess/NAACL_2022_OCR huggingface dataset:
Mubashara Akhtar, Oana Cocarascu and Elena Simperl
Department of Informatics, King's College London
{mubashara.akhtar,oana.cocarascu,elena.simperl1}@kcl.ac.uk
Inspired by human fact checkers, who use different types of evidence (e.g. tables, images, audio) in addition to text, several datasets with tabular evidence data have been released in recent years. Whilst the datasets encourage research on table fact-checking, they rely on information from restricted data sources, such as Wikipedia for creating claims and extracting evidence data, making the fact-checking process different from the real-world process used by fact checkers. In this paper, we introduce PubHealthTab, a table fact-checking dataset based on real-world public health claims and noisy evidence tables from sources similar to those used by real fact checkers. We outline our approach for collecting evidence data from various websites and present an in-depth analysis of our dataset. Finally, we evaluate state-of-the-art table representation and pre-trained models fine-tuned on our dataset, achieving an overall  $F_{1}$  score of 0.73.
Fact-checking is the task of establishing the veracity of factual information, commonly performed manually by journalists. In addition to classifying how truthful claims are, human fact checkers also provide evidence for their judgements. To support this process with computational tools, researchers have compiled several datasets for evidence-based automated fact-checking (AFC), which include information about the sources supporting or refuting the claims alongside veracity labels (Thorne et al., 2018; Chen et al., 2020b; Aly et al., 2021; Schuster et al., 2021; Nørregaard and Derczynski, 2021).
While a large share of the datasets used in evidence-based AFC focus on textual evidence (e.g. (Thorne et al., 2018; Augenstein et al., 2019; Diggelmann et al., 2020; Schuster et al., 2021)), some recent datasets also cover structured data, for instance in the form of web tables (Chen et al.,
2020b; Aly et al., 2021). This is useful, as human fact checkers often need to consider a range of data modalities to verify claims. However, two main limitations remain. First, existing table fact-checking datasets consist largely of claims which have been 'artificially' created via online crowdsourcing, starting from randomly selected evidence tables. Second, the datasets use single sources of evidence, for instance Wikipedia; this is different from how human fact checkers go about the task - more often than not, they consult multiple primary sources, including websites, databases, and public reports.
To overcome these limitations, we propose PubHealthTab<sup>1</sup>, a new table fact-checking dataset, using the PubHealth dataset (Kotonya and Toni, 2020) as a seed. PubHealth has a number of advantages. It contains public health claims that human fact-checkers work on. The authors compared the complexity of these claims to real-world political claims, as well as to claims created by crowdworkers (Kotonya and Toni, 2020). As a proxy for complexity, they determined the reading skills needed to understand the claims. They established that public health claims are much more challenging, requiring high school levels of reading of 10 to 12 rather than 6 to 8 for political and crowdsourced claims. PubHealth also includes multiple sources of evidence for the claims, however, the evidence is purely text-based. In our dataset, we include web tables as evidence, extracted from different websites, similar to those used by human fact-checkers.
We designed a hybrid dataset pipeline, which takes PubMed claims and links them, via Wikipedia articles, to other websites containing potential evidence tables. We used crowdsourcing in three ways: to establish the relevance of the extracted tables; to adjust PubMed claims to support or refute the tables; and finally to assess the
https://github.com/mubasharaak/ PubMedTab
Findings of the Association for Computational Linguistics: NAACL 2022, pages 1-16 July 10-15, 2022 ©2022 Association for Computational Linguistics
quality of the new claims. The result is a dataset of 1,942 claim-table pairs about public health, drawing on evidence from more than 300 websites.
We analysed the dataset to spot potential biases in the way we collected the data and compared PubHealthTab with other table-based fact-checking datasets. Moreover, we experimented with several BERT-based models and table representations to understand how our dataset performs on state-of-the-art AFC, achieving an overall  $F_{1}$  score of 0.73. Both allowed us to identify areas of future improvement, in particular to refute claims against evidence consisting of mostly numerical data or with noisy text headers.
2.1 Evidence-based Fact-Checking
Evidence-based AFC requires one to predict a veracity label against the evidence. While most datasets focus on textual sources of evidence (Thorne et al., 2018; Jiang et al., 2020; Diggelmann et al., 2020; Schuster et al., 2021), human fact checkers use a wider range of modalities (Nakov et al., 2021). To verify factual information, they commonly ask experts, search in databases, and consult text, tables, and graphics from a multitude of sources, including scholarly literature, public reports, and official statistics.
2.2 Table Fact-Checking Datasets
There is a small number of datasets that consider tables in AFC. However, in all cases, the claims are created by crowdworkers given evidence from Wikipedia. For instance, TabFact (Chen et al., 2020b) contains tables extracted from Wikipedia and considers two classes for the claim veracity: entailment and contradiction. The InfoTabs dataset (Gupta et al., 2020) has claims that can be verified using information from Wikipedia info-boxes, with an additional "neutral" class. In FEVERIOUS (Aly et al., 2021), claims are verified using text, tables, and lists from Wikipedia. Finally, the recent Sem-Eval fact-checking challenge, Sem-Tab-Facts (Wang et al., 2021), released a table fact-checking dataset with tables extracted from scientific articles. Claims were created by crowd workers based on sentences in the article describing these tables.
2.3 Tables in Other NLP Tasks
There is an increasing body of literature looking at tables alongside text for NLP tasks such as table question answering (tableQA) or table-to-text natural language generation (NLG). The former aims to find answers to natural language questions in tabular data (Pasupat and Liang, 2015; Zhong et al., 2017; Iyyer et al., 2017) and inspired the first table fact-checking dataset (Chen et al., 2020b). Researchers later introduced variations of the task with additional modalities (Chen et al., 2020c; Hannan et al., 2020) or sub-tasks such as table retrieval (Chen et al., 2021). There are also several table-to-text NLG datasets, for instance numericNLG (Suadaa et al., 2021) with tables extracted from scientific papers, and LogicNLG (Chen et al., 2020a) with Wikipedia tables. We used some of the methods proposed by the numericNLG team (Suadaa et al., 2021) to represent tables in our experiments.
2.4 The PubMed Dataset
As noted earlier, we used PubMed (Kotonya and Toni, 2020) as a starting point for creating our table fact-checking dataset. PubMed consists of real-world claims about public health extracted from fact-checking and news review websites. The authors comment that the majority of fact-checking datasets either concentrate on politics (Wang, 2017; Augenstein et al., 2019) or are built for research purposes (Thorne et al., 2018; Chen et al., 2020b). Each record in the PubMed dataset consists of a claim, the full text of the fact-checking or news article, which discusses its veracity, and the article summary or a justification for the veracity label.
3 The PubMedTab Dataset
Figure 1 shows an overview of the data construction pipeline. In the top half, we automatically create pairs of claims and tables. We start from the PubHealth claims, assess them for relevance and then match the remaining ones with web tables (see Section 3.1). In the bottom half, we use crowdsourcing to filter tables, adjust claims to tables, and check for quality (see Section 3.1.2).
3.1.1 Steps 1 to 3: From Claims to Tables
In Step 1 we removed ambiguous and out-of-domain claims from the PubMed dataset using a
$^{2}$ https://ballotpedia.org/The methodologies_of_fact-checking
Figure 1: Dataset creation process.
lexicon of 4132 medical terms from: Wikipedia;<sup>3</sup> medical dictionaries from Harvard University,<sup>4</sup> University of Michigan<sup>5</sup>, and Schulich School of Medicine and Dentistry<sup>6</sup>; as well as the ConceptNet knowledge graph.<sup>7</sup> We retained the claims that contained at least one token matching the lexicons. For the other claims, we carried out NER to detect medical entities that the lexicons might have missed, using SciSpacy (Neumann et al., 2019). We kept the claims for which we could find an entity in the claim text whose ConceptNet node was liked to a lexicon term via the "hasContext" relation.<sup>8</sup>
In Step 2 we linked the claims from Step 1 to Wikipedia articles using two entity linking services: ELQ (Li et al., 2020) and WAT, for better coverage. We then took the websites referenced by the articles as a source of evidence tables. In Step 3, from all Wikipedia references, we kept those in English that could be scraped and which contained at least one table HTML tag ( $\langle \text{table} \rangle$ ). We heuristically removed all tables that were used purely for formatting reasons, and then ranked the remaining tables based on their BM25 similarity to the claim text. The result of this step was a set of 1915 claimtable pairs (1010 claims and 1422 tables from 1196 websites), which was fed to the crowdsourced half
of the pipeline.
3.1.2 Steps 4 to 6: Crowdsourcing
We ran three crowdsourcing tasks on Amazon Mechanical Turk (MTurk) in May-June 2021: table relevance, claim adjustment, and verification, loosely following the "find-fix-verify" crowdsourcing workflow for text processing by Bernstein et al. (2015). For each of the three tasks, we checked for quality, evaluated worker agreement, and aggregated the results before feeding them to the subsequent task.
Recruitment and training of workers. We allocated each task to three crowdworkers. Only workers with minimum 1000 previously-approved tasks and an approval rate of  $95\%$  or above were eligible to work on the tasks. Moreover, all workers had to pass a table literacy qualification test (see appendix). To train the workers, we followed the recommendations from Gadiraju et al. (2015); Doroudi et al. (2016) and included examples of expert-labelled tasks in the instructions, including the rationales for the chosen labels.
Tasks design. The tasks were designed as follows (see appendix for instructions and interfaces):
1. Task 1 - table relevance: We asked crowdworkers if claims and tables were related to each other. This was needed to evaluate the ranked list of tables from Step 3 (Figure 1), where we matched claims to tables using BM25. For each claim-table pair, workers could choose between four options: table supports, refutes, is related but more information is needed, and is unrelated to the claim. In addition, we also asked the crowd to name the columns which contributed to their choice. Each task had seven claim-table pairs, of which two were from the gold standard (see quality assurance below). We used majority voting to aggregate the answers.
2. Task 2 - claim adjustment: The input for this task were only the claim-table pairs which were judged as related but not enough information in the previous step. We asked crowd-workers to adjust a claim so that they could be supported or refuted by the table. The workers also had to flag whether the table supported or refuted the claim. Each task consisted of five claim-table pairs. As this was an open-
$^{3}$ https://en.wikipedia.org/wiki/Glossary_of_medicine $^{4}$ https://www.health.harvard.edu/a-through-c $^{5}$ https://apps.lib.umich.edu/medical-dictionary/ $^{6}$ https://www.schulich.uwo.ca/pathol/about_us/resources/glossary_of_medicalterms.html $^{7}$ https://conceptnet.io/ $^{8}$ https://github.com/commonsense/conceptnet5/wiki/Relations $^{9}$ https://sobigdata.d4science.org/web/tagme/wat-api
K-α	F-κ	R-κ
Table relevance	0.26	0.38	0.65
Verification	0.60	0.60	0.67
Table 1: Inter-annotator agreement scores for the table relevance task and the verification task.
ended task, we evaluated the results in the third crowdsourcing task.
3. Task 3 - verification: We asked crowdworkers to verify the adjusted claims. Again, each task had seven pairs of claims and tables, with two gold pairs. Workers could choose between four labels: supports, refutes, related but not enough information, and unrelated. We performed majority voting to aggregate the answers.
For the final dataset (see Section 3.2), we discarded the pairs of adjusted claims and tables labelled as unrelated by the majority of workers.
Quality assurance. For each task, we followed best practices to maintain annotation quality and detect malicious behaviour. One of the authors created a gold standard of 30 claim-table pairs for the close-ended tasks (table relevance and verification); we used two gold pairs per task. Workers who failed those two gold pairs could not submit their work. For the remaining submissions, we computed the inter-annotator agreement.
Table 1 shows the inter-annotator agreement scores using Krippendorff's alpha  $(\mathrm{K} - \alpha)$ , Fleiss' kappa  $(\mathrm{F} - \kappa)$ , and Randolph's kappa  $(\mathrm{R} - \kappa)$ .  $\mathrm{F} - \kappa$  is prone to the high agreement but low kappa phenomenon when the dataset is imbalanced (Feinstein and Cicchetti, 1990); this was the case for the table relevance task: after aggregating the answers with majority voting, we had the following distribution: less than  $1\%$  support, less than  $1\%$  refute,  $22\%$  related but not enough information, and  $77\%$  unrelated. This is why we used  $\mathrm{R} - \kappa$ , which yields more accurate results for imbalanced data. For the verification task, the data was more balanced, which is reflected in the similar scores. For both tasks, we obtained a  $\mathrm{R} - \kappa$  value of at least 0.65, which indicates substantial agreement according to Landis and Koch (1977).
The claim adjustment task was open-ended. We allowed only submissions which met a set of criteria, for instance by looking at the time spent per task and comparing the original and adjusted claim;
Claim: Measles outbreak in Quebec carries a different strain than that in Ontario in recent years.
Province	Year	Number of cases	Duration (weeks)	Strain
Quebec	2007	94	24	D4
Ontario	2008	53	11	D8
British Columbia	2010	82	7	D8 and H1
Quebec	2011	20	11	D4
Quebec	2011	678	33	D4
Caption: Table 1. Measles Outbreaks in Canada, by province, 2007 to 2011
Website title: Guidelines for measles outbreak in Canada - Canada.ca
Veracity label: SUPPORTS
Source: https://www.canada.ca/en/public-health/services/reports-publications/canada-communicable-disease-report-ccdr/monthly-issue/2013-39/guidelines-prevention-control-measles-outbreaks-canada.html
Figure 2: A support example from PubMedTab.
the full list of criteria is in the appendix. We also manually inspected the adjusted claims before accepting them. We randomly sampled one claim for each submission and accepted the work if its quality was sufficient. After a first pilot round, we banned workers with malicious behaviour, e.g. workers who did not adjust the claims, but only added or removed one token.
Our PubMedTab dataset comprises 1,942 claimtable pairs. A claim is a natural language sentence checked against a table. Each pair is labelled as support, refute, or not enough information (NEI), following Thorne et al. (2018); Gupta et al. (2020); Diggelmann et al. (2020); Aly et al. (2021). The dataset has 1,019 supported claims, 462 refuted claims, and 461 NEI claims. Figure 2 shows an example.
The evidence table is organised as a list of  $n$  rows. Each row is a list of cells, where  $m$ , the number of cells, can vary across rows. If the first row is a header, it is instead saved as "header-horizontal". Similarly, if the first column is a header, it is saved as "header_vertical". For each table, we provide the source website and, if available, the table caption. Moreover, each record also includes the original PubMed claim text, which was adjusted by crowdworkers in Step 5 (Figure 1).
Table 2 compares the original PubHealth dataset with our dataset, PubMedTab.
PubMedTab	PubMed
Entries	1,942	11,832
Evidence type	Table	Text
Claim length	20 - 194	25 - 400
Veracity labels	{supports, refutes, NEI}	{true, mixture, false, unproven}
We analysed the PubMedTab dataset for biases and correlations, and compared it to other table fact-checking datasets. We applied three methods: (i) correlation analysis of table attributes; (ii) Local Mutual Information (LMI) on adjusted claims; and (iii) claim-only veracity prediction.
4.1 Correlation analysis of table attributes
While correlations between claims and veracity labels in fact-checking datasets have been previously explored (Schuster et al., 2019; Aly et al., 2021; Thorne et al., 2021), such underlying relationships might also be present in the evidence data. Thus, we examined correlations related to tables in the PubMedTab dataset. We analysed if the veracity labels and the length of adjusted claims were correlated with the following table attributes that were visible to crowdworkers during annotation: table length (i.e. number of rows), availability of table captions, and availability of table headers.
Depending on the type of the attribute analysed, we used: the Pearson correlation coefficient, the  $\chi^2$  test, and the Anova F-test and a significance level  $\alpha$  of 0.05 to examine correlations. The p-values for all attribute pairs are shown in Table 3. No significant correlations were found between the adjusted claim length and the table attributes' length, caption availability, and header availability. Given p-values  $\geq \alpha$ , the hypothesis of independence holds for these pairs of variables. Similarly, the veracity labels were not significantly correlated with the table length, caption availability, and adjusted claim length. For the correlation between veracity labels and header availability, we calculated a p-value of 0.03 indicating an underlying relationship between the variables. Examining the attributes in detail, we found that tables with headers were more prominent for supported and refuted claims than for NEI claims in the PubHealthTab dataset.
Table 2: Comparison between our dataset and PubHealth (Kotonya and Toni, 2020).
Adj. claim length	Veracity label
Table length	0.05 (Pearson)	0.35 (F-test)
Adj. claim length	-	0.47 (F-test)
Caption available	0.36 (F-test)	0.05 (χ2test)
Header available	0.16 (F-test)	0.03 (χ2test)
Table 3: Calculated p-values for the significance tests.
Bigram b	LMI	p(l,b)	count
Supported claims	the highest	1009	0.86	44
has the	989	0.8	60
percentage of	579	0.88	24
had a	423	0.88	17
highest number	418	0.93	14
there is	376	0.79	24
more than	364	0.73	37
Refuted claims	found on	1030	0.61	28
breast cancer	617	0.46	35
is found	599	0.48	29
be found	493	0.62	13
on page	471	0.42	36
is about	450	0.64	11
has a	433	0.34	86
NEI claims	the table	675	0.46	13
of domestic	621	0.8	5
health care	584	0.25	36
domestic violence	564	0.67	6
in a	516	0.57	7
for health	398	0.6	5
to the	365	0.28	18
Table 4: Top LMI-ranked bigrams for support, refute and NEI claims (including probability and count).
4.2 Local Mutual Information
Following Schuster et al. (2019), we analysed the correlation between frequently occurring phrases in adjusted claims and their veracity labels. We computed the Local Mutual Information (LMI) score (Evert, 2005) between a bigram  $b$  and the claim's veracity label  $l$ :  $LMI(b,l) = p(b,l)*\log (\frac{p(l|b)}{p(l)})$ . Unlike the Point-wise Mutual Information (PMI) score,  $PMI = \log (\frac{p(l|b)}{p(l)})$ , the LMI score avoids over-weighting bigrams with no or low occurrences in the overall dataset by multiplying it with the probability  $p(b,l)$ , where  $p(b,l)$  is approximated by  $\frac{\text{count}(b,l)}{|B|}$ ,  $|B|$  is the number of all bigrams in the dataset and  $\text{count}(b,l)$  is the number of times  $b$  and  $l$  occur together.
Table 4 shows the top LMI-ranked bigrams for PubMedTab claims. We found similar bigrams in different classes, for example "has a" appears in refuted claims and "had a" in supported claims. Furthermore, no top-ranked bigram of refuted claims contains negation tokens such as "not", "never" or "false". Thus, we conclude that the top-ranked bigrams occurring in claims are not specific to their veracity labels.
4.3 Claim-only Veracity Prediction
We fine-tuned a BERT base model (Devlin et al., 2019) on PubMedTab claims to predict their veracity labels using only the text as input and ignoring evidence tables. A claim-only model that performs well could indicate underlying correlations between the claims and the veracity labels. A similar approach was used by Schuster et al. (2019) to evaluate claim-only biases in the FEVER dataset (Thorne et al., 2018). Using the fine-tuned claim-only BERT model, we obtain an  $F_{1}$  score of 0.51 on our test set. Comparing the  $F_{1}$  score of the claim-only model to the performance of models using evidence data (see Section 5), we conclude that claims alone are not sufficient for the BERT model to predict the veracity labels.
4.4 Table Analysis
We compared PubHealthTab to three fact-checking datasets that use tables, TabFact, InfoTabs, and FEVEROUS (Table 5). Whilst almost all TabFact, InfoTabs and FEVEROUS tables have headers, this is not the case in more than half  $(56.9\%)$  of PubHealthTab tables. Similarly, all TabFact and InfoTabs tables include captions and approximately only one-fifth of PubHealthTab tables  $(21\%)$  and FEVEROUS tables  $(22\%)$  have captions. While captions and headers can be useful for understanding the context of a table, these attributes are not always present in real-world tables.
The average number of characters per cell is 13.4 for PubMedTab tables, more than the average cell length of TabFact tables (8.6) and less than for InfoTabs (22.6) and FEVERIOUS (17.3). Moreover, PubMedTab tables show the highest ratio of cells with numerical content  $(59\%)$  and the smallest ratio with text-only content compared to the other datasets. Numerical content can pose a challenge for state-of-the-art NLP models as previous works have shown (Suadaa et al., 2021).
We experimented with several table representation techniques and state-of-the-art models on PubMedTab to understand related challenges.
5.1 Table Representation
To assess the impact of different table representation methods on the table fact-checking task, we used five table representation techniques. We also used the BERT-based TAPAS model which
extends the BERT model architecture with three additional embeddings to encode table structure. We describe the TAPAS model in more detail when we discuss the modelling approaches in Section 5.2. We describe the table representations in detail below:
Concatenation: transforms the entire content of a table into one flat string ignoring the table structure. The table caption, headers, and content are concatenated and used jointly as input for label prediction.
Template-based concatenation: maps table columns and cell values into a structured form using the following template applied to each row: row_1: column_1:cell_value, column_2:cell_value, [...]. The row and column tokens were replaced by the corresponding vertical header (for row) and horizontal header (for column), if available.
Template-based sentences: We defined a template to convert table content to one sentence per row. For example, given a table with headers "medicine" and "price", and two cells in the first row, we generate the following template-based sentence for this row: In row one column one (medicine) is Panadol, column two (price) is £15.
T5 (concatenation): Similarly to Suadaa et al. (2021), we used text from representation concatenation as input to the T5 text generation model (Raffel et al., 2020) to generate sentences that describe the tables.