TabFact: A Large-scale Dataset for Table-based Fact Verification

1 INTRODUCTION . Verifying whether a textual hypothesis is entailed or refuted by the given evidence is a fundamental problem in natural language understanding ( Katz & Fodor , 1963 ; Van Benthem et al. , 2008 ) . It can benefit many downstream applications like misinformation detection , fake news detection , etc . Recently , the first-ever end-to-end fact-checking system has been designed and proposed in Hassan et al . ( 2017 ) . The verification problem has been extensively studied under different natural language tasks such as recognizing textual entailment ( RTE ) ( Dagan et al. , 2005 ) , natural language inference ( NLI ) ( Bowman et al. , 2015 ) , claim verification ( Popat et al. , 2017 ; Hanselowski et al. , 2018 ; Thorne et al. , 2018 ) and multimodal language reasoning ( NLVR/NLVR2 ) ( Suhr et al. , 2017 ; 2019 ) . RTE and NLI view a premise sentence as the evidence , claim verification views passage collection like Wikipedia1 as the evidence , NLVR/NLVR2 views images as the evidence . These problems have been previously addressed using a variety of techniques including logic rules , knowledge bases , and neural networks . Recently large-scale pre-trained language models ( Devlin et al. , 2019 ; Peters et al. , 2018 ; Yang et al. , 2019 ; Liu et al. , 2019 ) have surged to dominate the other algorithms to approach human performance on several textual entailment tasks ( Wang et al. , 2018 ; 2019 ) . However , existing studies are restricted to dealing with unstructured text as the evidence , which would not generalize to the cases where the evidence has a highly structured format . Since such structured evidence ( graphs , tables , or databases ) are also ubiquitous in real-world applications like 1https : //www.wikipedia.org/ 1 Published as a conference paper at ICLR 2020 United States House of Representatives Elections , 1972 database systems , dialog systems , commercial management systems , social networks , etc , we argue that the fact verification under structured evidence forms is an equivalently important yet underexplored problem . Therefore , in this paper , we are specifically interested in studying fact verification with semi-structured Wikipedia tables ( Bhagavatula et al. , 2013 ) 2 as evidence owing to its structured and ubiquitous nature ( Jauhar et al. , 2016 ; Zhong et al. , 2017 ; Pasupat & Liang , 2015 ) . To this end , we introduce a large-scale dataset called TABFACT , which consists of 118K manually annotated statements with regard to 16K Wikipedia tables , their relations are classified as ENTAILED and REFUTED3 . The entailed and refuted statements are both annotated by human workers . With some examples in Figure 1 , we can clearly observe that unlike the previous verification related problems , TABFACT combines two different forms of reasoning in the statements , ( i ) Linguistic Reasoning : the verification requires semantic-level understanding . For example , “ John J. Mcfall failed to be re-elected though being unopposed. ” requires understanding over the phrase “ lost renomination ... ” in the table to correctly classify the entailment relation . Unlike the existing QA datasets ( Zhong et al. , 2017 ; Pasupat & Liang , 2015 ) , where the linguistic reasoning is dominated by paraphrasing , TABFACT requires more linguistic inference or common sense . ( ii ) Symbolic Reasoning : the verification requires symbolic execution on the table structure . For example , the phrase “ There are three Democrats incumbents ” requires both condition operation ( where condition ) and arithmetic operation ( count ) . Unlike question answering , a statement could contain compound facts , all of these facts need to be verified to predict the verdict . For example , the ” There are ... ” in Figure 1 requires verifying three QA pairs ( total count=5 , democratic count=2 , republic count=3 ) . The two forms of reasoning are interleaved across the statements making it challenging for existing models . In this paper , we particularly propose two approaches to deal with such mixed-reasoning challenge : ( i ) Table-BERT , this model views the verification task completely as an NLI problem by linearizing a table as a premise sentence p , and applies state-of-the-art language understanding pre-trained model to encode both the table and statements h into distributed representation for classification . This model excels at linguistic reasoning like paraphrasing and inference but lacks symbolic reasoning skills . ( ii ) Latent Program Algorithm , this model applies lexical matching to find linked entities and triggers to filter pre-defined APIs ( e.g . argmax , argmin , count , etc ) . We adopt bread-first-search with memorization to construct the potential program candidates , a discriminator is further utilized to select the most “ consistent ” latent programs . This model excels at the symbolic reasoning aspects by executing database queries , which also provides better interpretability by laying out the decision rationale . We perform extensive experiments to investigate their performances : the best-achieved accuracy of both models are reasonable , but far below human performance . Thus , we believe that the proposed table-based fact verification task can serve as an important new benchmark towards the goal of building powerful AI that can reason over both soft linguistic form and hard symbolic forms . To facilitate future research , we released all the data , code with the intermediate results . 2In contrast to the database tables , where each column has strong type constraint , the cell records in our semi-structured tables can be string/data/integer/floating/phrase/sentences . 3we leave out NEUTRAL due to its low inter-worker agreement , which is easily confused with REFUTED . 2 Published as a conference paper at ICLR 2020 2 TABLE FACT VERIFICATION DATASET . First , we follow the previous Table-based Q & A datasets ( Pasupat & Liang , 2015 ; Zhong et al. , 2017 ) to extract web tables ( Bhagavatula et al. , 2013 ) with captions from WikiTables4 . Here we filter out overly complicated and huge tables ( e.g . multirows , multicolumns , latex symbol ) and obtain 18K relatively clean tables with less than 50 rows and 10 columns . For crowd-sourcing jobs , we follow the human subject research protocols5 to pay Amazon Mechanical Turk6 workers from the native English-speaking countries “ US , GB , NZ , CA , AU ” with approval rates higher than 95 % and more than 500 accepted HITs . Following WikiTableQuestion ( Pasupat & Liang , 2015 ) , we provide the annotators with the corresponding table captions to help them better understand the background . To ensure the annotation quality , we develop a pipeline of “ positive two-channel annotation ” → “ negative statement rewriting ” → “ verification ” , as described below . 2.1 POSITIVE TWO-CHANNEL COLLECTION & NEGATIVE REWRITING STRATEGY . To harvest statements of different difficulty levels , we design a two-channel collection process : Low-Reward Simple Channel : the workers are paid 0.45 USD for annotating one Human Intelligent Task ( HIT ) that requires writing five statements . The workers are encouraged to produce plain statements meeting the requirements : ( i ) corresponding to a single row/record in the table with unary fact without involving compound logical inference . ( ii ) mention the cell values without dramatic modification or paraphrasing . The average annotation time of a HIT is 4.2 min . High-Reward Complex Channel : the workers are paid 0.75 USD for annotating a HIT ( five statements ) . They are guided to produce more sophisticated statements to meet the requirements : ( i ) involving multiple rows in the tables with higher-order semantics like argmax , argmin , count , difference , average , summarize , etc . ( ii ) rephrase the table records to involve more semantic understanding . The average annotation time of a HIT is 6.8 min . The data obtained from the complex channel are harder in terms of both linguistic and symbolic reasoning , the goal of the two-channel split is to help us understand the proposed models can reach under different levels of difficulty . As suggested in ( Zellers et al. , 2018 ) , there might be annotation artifacts and conditional stylistic patterns such as length and word-preference biases , which can allow shallow models ( e.g . bag-ofwords ) to obtain artificially high performance . Therefore , we design a negative rewriting strategy to minimize such linguistic cues or patterns . Instead of letting the annotators write negative statements from scratch , we let them rewrite the collected entailed statements . During the annotation , the workers are explicitly guided to modify the words , phrases or sentence structures but retain the sentence style/length to prevent artificial cues . We disallow naive negations by adding “ not , never , etc ” to revert the statement polarity in case of obvious linguistic patterns . 2.2 QUALITY CONTROL . To control the quality of the annotation process , we review a randomly sampled statement from each HIT to decide whether the whole annotation job should be rejected during the annotation process . Specifically , a HIT must satisfy the following criteria to be accepted : ( i ) the statements should contain neither typos nor grammatical errors . ( ii ) the statements do not contain vague claims like might , few , etc . ( iii ) the claims should be explicitly supported or contradicted by the table without requiring the additional knowledge , no middle ground is permitted . After the data collection , we re-distribute all the annotated samples to further filter erroneous statements , the workers are paid 0.05 USD per statement to decide whether the statement should be rejected . The criteria we apply are similar : no ambiguity , no typos , explicitly supported or contradictory . Through the post-filtering process , roughly 18 % entailed and 27 % refuted instances are further abandoned due to poor quality . 4http : //websail-fe.cs.northwestern.edu/wikiTables/about/ 5https : //en.wikipedia.org/wiki/Minimum_wage_in_the_United_States 6https : //www.mturk.com/ 3 Published as a conference paper at ICLR 2020 Proportion of different Higher-order Operations 2.3 DATASET STATISTICS . Inter-Annotator Agreement : After the data collection pipeline , we merged the instances from two different channels to obtain a diverse yet clean dataset for table-based fact verification . We sample 1000 annotated ( table , statement ) pairs and re-distribute each to 5 individual workers to re-label them as either ENTAILED or REFUTED . We follow the previous works ( Thorne et al. , 2018 ; Bowman et al. , 2015 ) to adopt the Fleiss Kappa ( Fleiss , 1971 ) as an indicator , where Fleiss κ = p̄c−p̄e1−p̄e is computed from from the observed agreement p̄c and the agreement by chance p̄e . We obtain a Fleiss κ = 0.75 , which indicates strong inter-annotator agreement and good-quality . Dataset Statistics : As shown in Table 1 , the amount of data harvested via the complex channel slightly outnumbers the simple channel , the averaged length of both the positive and negative samples are indistinguishable . More specifically , to analyze to which extent the higher-order operations are included in two channels , we group the common higher-order operations into 8 different categories . As shown in Figure 2 , we sample 200 sentences from two different channels to visualize their distribution . We can see that the complex channel overwhelms the simple channel in terms of the higher-order logic , among which , count and superlatives are the most frequent . We split the whole data roughly with 8:1:1 into train , validation7 , and test splits and shows their statistics in Table 1 . Each table with an average of 14 rows and 5-6 columns corresponds to 2-20 different statements , while each cell has an average of 2.1 words . In the training split , the positive instances slightly outnumber the negative instances , while the validation and test split both have rather balanced distributions over positive and negative instances .

Summary: This work proposes the problem of fact verification with semi-structured data source such as tables. Specifically, the authors created a new dataset TabFact and evaluated two baseline models with different variations. They applied two criteria and different rewards for workers to collect two subsets of different levels (“simple” and “complex”). They also applied a negative rewriting strategy to avoid exploitable cues or patterns in the annotations. They evaluated two baselines models: (1) latent program algorithm (LPA), which makes use of simple string match entity linking and systematic search and trains a neural network discriminator, and (2) Table-BERT, which linearize the table into a sequence through concatenation or template, and treat it as a classfication problem. Both showed reasonable accuracy (~68%), but still below human performance (92.1%) on a held out test set.