TabFact: A Large-scale Dataset for Table-based Fact Verification
Authors: Wenhu Chen
Hongmin Wang
Jianshu Chen
Yunkai Zhang
Hong Wang
Shiyang Li
Xiyou Zhou
William Yang Wang
Venue:  (2020)
Keywords: Fact Verification
Tabular Data
Symbolic Reasoning

Abstract: The problem of verifying whether a textual hypothesis holds based on the given evidence, also known as fact verification, plays an important role in the study of natural language understanding and semantic representation. However, existing studies are mainly restricted to dealing with unstructured evidence (e.g., natural language sentences and documents, news, etc), while verification under structured evidence, such as tables, graphs, and databases, remains unexplored. This paper specifically aims to study the fact verification given semi-structured data as evidence. To this end, we construct a large-scale dataset called TabFact with 16k Wikipedia tables as the evidence for 118k human-annotated natural language statements, which are labeled as either ENTAILED or REFUTED. TabFact is challenging since it involves both soft linguistic reasoning and hard symbolic reasoning. To address these reasoning challenges, we design two different models: Table-BERT and Latent Program Algorithm (LPA). Table-BERT leverages the state-of-the-art pre-trained language model to encode the linearized tables and statements into continuous vectors for verification. LPA parses statements into LISP-like programs and executes them against the tables to obtain the returned binary value for verification. Both methods achieve similar accuracy but still lag far behind human performance. We also perform a comprehensive analysis to demonstrate great future opportunities.

Published as a conference paper at ICLR 2020
TABFACT: A LARGE-SCALE DATASET FOR TABLE- BASED FACT VERIFICATION
Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, William Yang Wang University of California, Santa Barbara, CA, USA
Tencent AI Lab, Bellevue, WA, USA
(wenhuchen, hongmin.wang, wunkai.zzhang hongwang600, wiwl:iwwiliammslleeeeeeeee edu (shiyangli, xiyou ucsb. jiamshuchen@tencent com
ABSTRACT
The problem of verifying whether a textual hypothesis holds based on the given evidence, also known as fact verification, plays an important role in the study of natural language understanding and semantic representation. However, existing studies are mainly restricted to dealing with unstructured evidence (e.g.. natu- ral language sentences and documents, news, etc), while verification under struc- tured evidence, such as tables, graphs, and databases, remains under-explored. This paper specifically aims to study the fact verification given semi-structured data as evidence. To this end, we construct a large-scale dataset called TabFact with 16k Wikipedia tables as the evidence for 18k human-annotated natural lan- guage statements, which are labeled as either ENTAILED or REFUTED. TabFact is challenging since it involves both soft linguistic reasoning and hard symbolic reasoning. To address these reasoming challenges, we design two different mod- els: Table-BERT and Latent Program Algorithm (LPA). Table-BERT leverages the state-of-the-an pre-trained language model to encode the linearized tables and statements into continuous vectors for verification. LPA parses statements into programs and executes them against the tables to obtain the returned binary value for verification. Both methods achieve similar accuracy but still lag far be- hind human performance. We also perform comprehensive analysis to demon- strate great future opportunities. The data and code the dataset are provided pettps:/githuuee wwwp://wwcw//wcw/cceccaaammmaaae
INTRODUCTION
Verifying whether a textual hypothesis is entailed or refuted by the given evidence is a fundamental
problem in natural language understanding (Katz & Fodor [963] Van Benthem al., 2008). It
can benefit many downstream applications like misinformation detection, fake news detection, etc.
Recently, the first-ever end-to-end fact-checking system has been designed and proposed in Hassan et al. (2017). The verification problem has been extensively studied under different natural language tasks such recognizing textual entailment (RTE) (Dagan al., 2005), natural language inference
(NLI) (Bowman et al. (2015). claim verification (Popat et al. [2017] Hansclowski et al. 2018 Thorn
al. 2018) and multimodal language reasoning (NLVR/NLVR2) (Suhr et al. 2017 17 2019 RTE
and NLI view premise sentence the evidence, claim verification views passage collection like Wikipedia as the evidence, NLVR/NLVR2 views images as the evidence. These problems have been previously addressed using variety of techniques including logic rules, knowledge bases, and neural networks. Recently large-scale pre-trained language models (Devlin et al. | 2019 Peters al.)
2018 Yang et al. 2019; Liu et al. 2019) have surged to dominate the other al algorithms to approach human performance on several textual entailment tasks (Wang et al. 2018 2019).
However, existing studies restricted to dealing with unstructured text the evidence, which would not generalize to the cases where the evidence has a highly structured format. Since such structured evidence (graphs, tables, or databases) also ubiquitous in weal-world applications like
https:://// httpss///www.e www.iiiiiiiiii.i.......i...

---

Published as a conference paper at ICLR 2020
United States Hause Representatives Elections, 1972
District
Incumbent
Party
Result
Candidates
Califonii
John Mass
democratic
re elected
John Moss (d) 69.9% John Rakus (r) 30.1%
California
Philipp Burton
democratic
re-elected
Philip Burton (d) 81 8% Edio E. Powell (r) 18.2%
California
George Paul Miller
democratic
lost renomination democratic hold
Pete Stark (d) 52.9% Lew M. Warden . (r) 47.1%
California 14
Jerome R. Waldie
republican
re re-elected
Jerome R. Waldle (d) 77.6% Floyd E. Sims (f) 22.4%
Califonnia 15
John Mccall
republican
re elected
John Mcfall (6) unapposed
Entailed Statement
Refuled Statement
1. John E. Moss and Phillp Burton are both re re-elected in the house representative elaction.
1. John E. Moss and George Faul Miller are both re elected the hause
representative election.
2. John Mcfall is unoppesed during the re-election. There
John Mcfall failed to be re ee-eeected though being unopposed. There five cand dates total, them are democrats and three them are republicans.
different incumbents from democratic.
Figure I: Examples from the TABFACT dataset. The top table contains the semi-structured knowl- edge facts with caption "United...". The left and right boxes below provide several entailed and refuted statements. The error parts are highlighted with red font.
database systems, dialog systems, commercial management systems, social networks, etc, we argue that the fact verification under structured evidence forms is an equivalently important yet under- explored problem. Therefore. in this paper, we are specifically interested in studying fact verification with semi-structured Wikipedia tables (Bhagavatula et al. 2013 as evidence owing to its structured and ubiquitous nature (Jauhar et al.) 2016) Zhong et al. 2017; Pasupat & Liang 2015). To this end, we introduce a large-scale dataset called TABFACT, which consists of 118K mamually annotated statements with regard to 16K Wikipedia tables, their relations are classified
ENTAILED and REFUTED The entailed and refuted statements are both annotated by human workers. With some examples Figure I, we can clearly observe that unlike the previous verification related problems, TABFACT combines two different forms of reasoning in the statements, (i) Linguistic Reasoning: the verification requires semantic-level understanding, For example. "John Mcfall failed to be re-elected though being unopposed." requires understanding over the phrase "lost renomination in the table to correctly classify the entailment relation. Unlike the existing QA datasets (Zhong et al. 2017 Pasupat& & Liang] 2015), where the linguistic reasoning is dominated by paraphrasing, TABFACT requires more linguistic inference or common sense. (ii) Symbolic Reasoning: the verifi- cation requires symbolic execution on the table structure. For example, the phrase "There are three Democrats incumbents" requires both condition operation (where condition) and arithmetic oper- ation (count). Unlike question answering, statement could contain compound facts, all of these facts need to be verified to predict the verdict. For example, the "There are ..." in |Figure T requires verifying three QA pairs (total count=5, democratic count=2, republic count=3). The two forms of reasoning are interleaved across the statements making it challenging for existing models.
In this paper, we particularly propose two approaches to deal with such mixed-reasee reasoninsg challenge: (i) Table-BERT, this model views the verification task completely as an NLI problem by linearizing a table as premise sentence p, and applies state-of-the-an language understanding pre-trained model to encode both the table and statements h into distributed representation for classification. This model excels linguistic reasoning like paraphrasing and inference but lacks symbolic reasoning skills. (ii) Latent Program Algorithm, this model applies lexical matching to find linked entities and triggers to filter pre-defined APIs (e.g. argmax, argmin, count, etc). We adopt bread-firs -iiss---ssareee with memorization to construct the potential program candidates, discriminator is further utilized to select the most "consistent" latent programs. This model excels at the symbolic reasoning aspects by executing database queries, which also provides better interpretability by laying out the decision rationale. We perform extensive experiments to investigate their performances: the best achieved accuracy of both models reasonable, but far below human performance. Thus, we believe that the proposed table-based fact verification task can serve as an important new benchmark towards the goal of building powerful AI that can reason over both soft linguistic form and hard symbolic forms. To facilitate future research, we released all the data, code with the intermediate results.
In contrast to the database tables, where each column has strong type constraint, the cell records our semi-structured tables can be http://aaanmmmeeeeffooainnnaaaaa we leave out NEUTRAL. due to its low inter-worker agreement, which casily confused with REFUTED.
2

---

Published as a conference paper at ICLR 2020
2 TABLE FACT VERIFICATION DATASET
First, we follow the previous Table- Q& A datasets (Pasupat & Liang 2015 Zhong al. 2017) to extract web tables (Bhagavatula et al. 2013) with captions from Wiki lables Here we filter out overly complicated and huge tables (e.g. multirows, multicolumns, latex symbol) and obtain 18K relatively clean tables with less than 50 rows and 10 columns.
For crowd-sourcing jobs, we follow the human subject research protocols to pay Amazon Mechani- cal Turk workers from the native rates higher than 95% and more than 500 accepted HITs. Following WileTableueestion (Pasupat & Liang] 2015), we provide the annotators with the corresponding table captions to help them better understand the background. To ensure the annotation quality, we develop a pipeline of "positive two-channel annotation" "negative statement rewriting" + "verification" described below.
countries "US, GB. NZ. CA. AU" with approval
2.1 POSITIVE TWO-CHANNEL COLLECTION & NEGATIVE REWRITING STRATEGY
To harvest statements of different difficulty levels, we design two-channel collection process:
Low-Reward Simple Channel: the workers are paid 0.45 USD for annotating one Human Intel- ligent Task (HIT) that requires writing five statements. The workers are encouraged produce plain statements meeting the requirements: (i) corresponding to a single row/record in the table with unary fact without involving compound logical inference. (ii) mention the cell values without dra- matic modification or paraphrasing. The average annotation time of a HIT is 4.2 min.
the workers are paid 0.75 USD for annotating a HIT (five state- ments). They are guided to produce more sophisticated statements to meet the requirements: (i) involving multiple rows in the tables with higher-order semantics like argmax, argmin, count, differ- ence, average, summarize, etc. (ii) rephrase the table records to involve more semantic understand- ing. The average annotation time of a HIT is 6.8 min. The data obtained from the complex channel are harder in terms of both linguistic and symbolic reasoning, the goal of the two-channel split is to help us understand the proposed models can reach under different levels of difficulty.
High-Reward Complex Channel:
As suggested in (Zellers al. 2018), there might be annotation artifacts and conditional stylistic patterns such as length and word-prefccn cord-preference biases, which can allow shallow models (c.g. bag-of- words) to obtain artificially high performance. Therefore, we design negative rewriting strategy to minimize such linguistic cues or patterns. Instead of letting the annotators write negative statements from scratch, let them rewrite the collected entailed statements. During the annotation, the workers are explicitly guided to modify the words, phrases or sentence structures but retain the sentence style/length prevent artificial cues. We disallow naive negations by adding "not, never, etc" to revert the statement polarity in case of obvious linguistic patterns.
2.2 QUALITY CONTROL
To control the quality of the annotation process, we review randomly sampled statement from each HIT to decide whether the whole annotation job should rejected during the annotation process. Specifically, HIT must satisfy the following criteria to be accepted: (i) the statements should contain neither typos nor grammatical errors. (ii) the statements do not contain vague claims like might, few, etc. (iii) the claims should be explicitly supported contradicted by the table without requiring the additional knowledge, no middle ground is permitted. After the data collection, we re-distribute all the annotated samples to further filter erroneous statements, the workers are paid 0.05 USD per statement to decide whether the statement should be rejected. The criteria we apply similar: ambiguity, no typos, explicitly supported contradictory. Through the post-filtering process, roughly 18% entailed and 27% refuted instances are further abandoned due to poor quality.
Cctps://ww.ntturk
3

---

Published as a conference paper at ICLR 2020
Proportion of different Higher-order Operations
Simple
Complex
Overall
Percent
20
AGGREGATION
NEGATE
SUPERLATIVE
COUNT
COMPATIVE
ORDINAL
UNIQUE
ALL
Figure 2: Proportion of different oigher-order operations from the simple/complex channels.
Channel
#Sentence
#Table
Len(Ent)
Len(Ref)
Split
#Sentence
Table
Row
Col
Simple 50,244
9,189
13.2
13.1
Train
92,283
13,182
14.1
5.5
Complex 68,031
7,392
14.2
14.2
Val
12,792
1,696
14.0
5,4
Total 118,275
16,573
13.8
13.8
Test
12,779
1,695
14.2
5.4
Table I: Basic statistics of the data collected from the sinple/complex channel and the division of Train/Val/Test Split the dataset, where "Ieen" denotes the averaged sentence length.
2.3 DATASET STATISTICS
Inter-Annotator Agreement:
After the data collection pipeline, merged the instances from two different channels to obtain a diverse yet clean dataset for table-based fact verification. We sample 1000 annotated (table, statement) pairs and re-distribute each to 5 individual workers re-label them as either ENTAILER or REFUTED. We follow the previous works Thorne et al.] 2018 al. 2015) adopt the Fleiss Kappa (Fleiss 1971 as an indicator, where Fleiss Bowman
Ve is
I-F-
computed from from the observed agreement Pa and the agreement by chance Pe. We obtain Fleiss 0.75, which indicates strong inter-annotator agreement and good-quality.
Dataset Statisties: As shown in Table 1 the amount of data harvested via the complex channel slightly outnumbers the simple channel, the averaged length of both the positive and negative sam- ples are indistinguishable. More specifically, to analyze to which extent the higher-order operations are included in two channels, gories. As shown in Figure 2] we sample 200 sentences from two different channels to visualize their distribution. We can see that the complex channel overwhelms the simple channel in terms of the higher-order logic, among which, count and superlatives are the most frequent. We split the whole data roughly with 8:1:1 into train, validation and test splits and shows their statistics in Table 1 Each table with an average of 14 rows and 3-6 columns corresponds to 2-20 different statements, while each cell has average of 2.1 words. In the training split, the positive instances slightly outnumber the negative instances, while the validation and test split both have rather bal- anced distributions over positive and negative instances.
group the common higher-order operations into 8 different cate-
MODELS
With the collected dataset, we formally define the table-based fact verification task: the dataset is comprised of triple instances (T, S, L) consisting of a table T, a natural language statement s..-.. Sn and a verification label e {0. 1}. The table T {Tr,le < Rr.j < Cr has Rr rows and CT columns with the Ti being the content in the (i.j)-th cell. Tiy could be word, number, phrase, or even natural language sentence. The statement S describes a fact to be verified against the content in the table T. If it is entailed by T. then L 1, otherwise the label 0. Figure shows some entailed and refuted examples. During training, the model and the learning algorithm are presented with K instances like ST L) from the training split. In the testing stage, the model is presented with (T, S) and supposed to predict the label as L. We measure the performance by the prediction accuracy Acc K7 Σ I(L. Lx) on the test set. Before building the model, we first perform entity linking to detect all the entities in the statements. Briefly, first lemmatize the words and search for the longest sub-string matching pairs between statements and table cells/captions, where the matched phrases are denoted as the linked entities. To focus statement verification against the table, we do not feed the caption to the model and simply We filter roughly 400 sentences from abnormal tables including hyperlinks, math symbols,
4

---

Published as a conference paper at ICLR 2020
mask the phrases in the statements which link to the caption with placeholders. The details of the entity linker are listed in the Appendix. We describe our two proposed models as follows.
3.1 LATENT PROGRAM ALGORITHM (LPA)
In this approach, we formulate the table fact verification as program synthesis problem, where the latent program algorithm is not given in TABFACT. Thus, can be seen as a weakly supervised learning problem as discussed in Liang et al. (017); Lao et al. (2011). Under such setting, propose to break down the verification into two stages: (i) latent program search, (ii) discriminator ranking. In the first program synthesis step, aim to parse the statement into programs represent its semantics. We define the plausible API set to include roughly 50 different functions like min, max, count, averuge, filler, and and realize their interpreter with Python-Pandar Each API defined to take arguments of specific types (number, string, bool, and view (e.g sub-table)) to output specific- type variables. During the program execution, we store the generated intermediate variables to different-typett caches NN, R. B, V (Num, Str, Bool, View). At each execution step, the program can fetch the intermediate variable from the caches achieve semantic compositionality. shrink the search space, we follow NSM (Liang et al. (017)) to use trigger words to prune the API set and accelerate the search speed. The definitions of API, trigger words can be found in the Appendix. The comprehensive the latent program search procedure is summarized in Algorithm I
order to
Algorithm Latent Program Search with Comments
1: Initialize Number Cache N,, String Cache R., Bool Cache B, View Cache -
2: Push linked numbers, strings from the given statement S into N, R, and push T into V
3: Initialize the result collector P -> 0 and an empty program trace P' = 0
4: Initialize the Queue Q [P.N, R. B3 V)]. we
store the intermediate states
5: Use trigger words to find plausible function set FF, for example, more will trigger Greater function.
6: while loop over time f = + MAXSTEP do:
7: while (P.N R, B, V) = pop(() do:
8:
while loop over function set F FFdo: do:
if arguments of f are in the caches then
10:
Pop out required arguments argi, arga, ... arg. different cachess. Execute A - (argi. ... ,arg.) and concatenate the program trace P. if Ty)ee(A)-Bool then
11:
12:
13:
if = P. P.push(((P A)) The program is valid since it consumes all the variables. p # Collect the valid program P into set P and reset
then
14:
15:
16:
else (A))s # The intermediate boolean value is added to the bool cache Q.pash (PP,N, R, V)) # Add the refreshed state the queue again if Type(A) E [Num, Str, View) then then P 0:break # The program ends without consuming the cache, throw it. else
17:
18:
19:
20:
if S
21:
22:
23:
push A into N or S or V # Add the refreshed state to the queue for further search Q.pash( R, R, V))
24:
25:
Return the triple (T.S, # Return (Table, Statement, Program Set)
and the searching procedure illustrated in Figure
After we collected all the potential program candidates P (1),, ,P.).....,P,, A,)) for a given statement S (where A,) refers to i-th candidate), we need to learn discriminator iden- tify the "appropriate" traces from the set from many erroneous and spurious traces. Since we do not have the ground truth label about such discriminator, we use a weakly supervised training al- gorithm by viewing all the label-consistent programs as positive instances [P:l(P,, Ad); A, = L} and the label-incomsistent program as negative instances [P.[(P,, A,); A, + L} to minimize the cross-entropy of discriminator Po(S, P) with the weakly supervised label. Specifically, we build our discriminator with a Transformer-bast statement encoder encodes the input statement S as vector Enc (S) sion D. while the program encoder encodes the program P Pr. Pm as another vector E (P) E m concatenate these two vectors and feed into linear projection layer
two-way encoder (Vaswani et al. 2017). where the
R. xx with dimen-
5

---

Published as a conference paper at ICLR 2020
There are more democrats than republicans in the election.
democrati mppaiion
Search
Feeture Enity Linking
Vi=Filen(T, bounbbnttisdmmmoetll)
Sun
Vi
Waumben
regtblicann
V2+FilenTT iroumbent-reguiiiaa))
V
LSS Engine
S/b Sub
3-CountV11
Num -..e Count
3
12
2-Counl(V2)
Num
Covet
Coom
pop
Table
Greater(3,
Boot
Entalled
Figure 3: The program synthesis procedure for the table in Figure I We link the entity (e.g. demo- crafic, republican), and then composite functions the fly to return the values from the table.
Label
Game
Date
Opponent
Score
51
February 3, 2009
Aorida 3-4
12-Layer BERT- Model
52
Fabruary 4, 2009
Buffalo 0-5
53
Rabruary 2016
Monireal 5-2
Template Concat
Word
[.5
51
[SEP]
February
2009
[SEP]
Ronda is plying
Positinn
Type
TOK
TOR
dale
dale
date
date
TOK
TOK
Word
C15)
12N
game
February
Hondd
Position
10
13
Figure 4: The diagram of T'Tale-BERT with horizontal scan, two different linearizations are depicted.
to compute Po(S, ( Enc" (P))) the relevance between S and P with weight Up R.D At test time, we use the discriminator py to assign confidence Pe(S, P) to each candidate P E P, and then either aggregate the prediction from all hypothesis with the confidence weights or rank the highest-confident hypothesis and use their outputs as the prediction.
3.2 TABLE-BERT
In this approach, we view the table verification problem as two-sequence binary classification problem like NI.I or MPRC (Wang et al., 2018) by linearizing a table T into a sequence and treating the statement as another sequence. Since the linearized table can be extremely long surpassing the limit of sequence models like LSTM, Transformers, etc. We propose to shrink the sequence by only retaining the columns containing entities linked to the statement to alleviate such memory issue. In order to encode such sub-table as a sequence, we propose two different linearization methods, as is depicted in Figure 4 (i) Concatenation: we simply concatenate the table cells with [SEP] tokens in between and restart position counter at the cell boundaries; the column name is fed as another type embedding to the input layer. Such design retains the table information in its machine format. (ii) Template: we adopt simple natural language templates to transform a table into a "somewhat natural" sentence. Taking the horizontal scan as an example, we lincarize a table "row one's game is 51; the date is February; ... the score is 3.4 (ot). row 2 is ...". The isolated cells are connected with punctuations and copula verbs in a language-like format.
After obtaining the linearized sub-table T, we concatenate it with the natural language state- ment S and prefix a [CLS] token to the sentence to obtain the sequence-level representation H fEERT ([T, (TT.S) SI), with H E 7678 from pre-trained BERT (Devlin et al. 2019). The rep- resentation is further fed into multi-layer perceptron SMLP to obtain the entailment probability the sigmoid function. We finetune the model 0 (including the parameters of BERT and MLP) to minimize the binary cross entropy L(pe S). L) on the training set. At test time, (table, statement) pair, and classify it as ENTATLED statement when po (o T,, S) is greater than 0.5.
(T, S) = (fMLL((H) where
the trained BERT model to compute the matching probability between the

---

Published as a conference paper at ICLR 2020
4 EXPERIMENTS
In this section, we aim to evaluate the proposed methods on TABFACT. Besides the standard valida- tion and test sets, we also split the set into simple and a complex partition based the channel from which they were collected. This facilitates analyzing how well the model performs under dif- ferent levels of difficulty. Additionally, we also hold a small test with 2K samples for human evaluation, where we distribute each (table, statement) pair to different workers to approximate human judgments based on their majority voting, the results are reported in Table 2
Model
Val
Test
Test (simple)
Test (complex)
Small Test
BERT classifier w/o Table
50.9
50.5
$1.0
50.1
50.4
Table-BERT Forizontal-F+T Conncatnteee
50.7
50.4
50.8
50.0
50.3
Tane--BERR--eeticcl--------
56.7
56.2
59.8
55.0
56.2
Table-BRRTT-VericalTTT---T-egg
56.7
57.0
60.6
54.3
55.5
66.0
65.1
79.0
58.1
67.9
Table Taa--BBRT--oiiinnala/---tTt-l--
66.1
65.1
79.1
58.2
68.1
NSM w/ RI. (Binary Reward)
54.1
54.1
55.4
53.1
55.8
NSM w/ IPA-guided MI. RI.
63.2
63.5
77.4
56.1
66.9
LPA-Voting w/o Discriminator
57.7
58.2
68.5
53.2
61.5
LPA- P---eighteed- Voting
62.5
74.6
57.3
66.8
LPA-Ranking w/ Discriminator
65.2
65.0
78.4
58.5
68.6
LPA Ranking w/ Discriminator (Caption)
65.1
65.3
78.7
58.5
68.9
Human Performance
92.1
Table 2: The results of different models, the numbers are in percentage. T+F means table followed by fact, while F+T means fact followed by table. NSM is modified from Liang et al. 2017).
NSM We follow Liang et al. (0117) modify their approach to fit the setting of TABFACT. Specif- ically, we adopt an LSTM as an encoder and another LSTM with copy mechanism as a decoder to synthesize the program. However, without any ground truth annotation for the intermediate programs, directly training with reinforcement learning is difficult as the binary reward is under- specified, which is listed in Table ala "NSM w/ RL" Further, use LPA as teacher search the top programs for the NSM to bootstrap and then use reinforcement learning to finetune the model, which achieves reasonable performance on our dataset listed "NSM ML RL"
Table-BERT We build Beble based on the open source implementation of BERT using the pre-trained model with 12-layer, 768-hidden, 12-heads, and 1 110M parameters trained in 104 lan- guages. We use the standard BERT tokenizer to break the words in both statements and tables into subwords and join the two sequences with [SEP] token in between. The representation correspond- ing to [CLS] is fed into an MIP layer to predict the verification label. We finetune the model on a single TITAN X GPU with a mini-batch size of 6. The best performance is reached after about 3 hours of training (around 10K steps). We implement and compare the following variants of the Table-BERT model including (i) Concatenation vs. Template: whether to use natural language tem- plates during linearization. (ii) Horizontal vs. Vertical: scan direction in linearization.
LPA We run the latent program search distributed fashion on three 64-core machines to gener- ate the latent programs. The search terminates once the buffer has more than 50 traces the path length is larger than 7. The average search time for each statement is about 2.5s. For the discrimina- encoders (3 layers, 128-dimension hidden embedding. and 4 heads at each layer) to encode the programs and statements, respectively. The variants of LPA models considered include (i) Voting: assign each program with equal weight and vote without the learned discriminator. (ii) Weighted- -otinn:: compute a weighted-sum to aggregate the predictions of all latent programs with the discriminator confidence as the weights. (iii) Ranking: rank all the hypotheses by the discriminator confidence and use the top-rated hypothesis as the output. (Caption) means feeding the caption as a sequence of words to the discriminator during ranking.
tor model, design two transformer-
Preliminary Evaluation In order to test whether negative rewriting strategy eliminates the
tifacts or shallow cues, we also fine-tune a pre-trained BERT (Devlin et al.) 2019) to classify the statement S without feeding in table information. The result is reported as "BERT classifier w/o
https: thu.. http:////////toooooo//////oeeeee

---

Published as a conference paper at ICLR 2020
Table" in Table 2] which is approximately the majority guess and reflects the effectiveness of the rewriting strategy. Before presenting the experiment results, we first perform a preliminary study to evaluate how well the entity linking system, program search, and the sattement-prograae discrimi- nator perform. Since we do not have the ground truth labels for these models, we randomly sample 100 samples from the dev set perform the human study. For the entity linking, we evaluate its accuracy as the number of correctly linked sentences/ / total sentences. For the latent program search, we evaluate whether the "true" programs are included in the candidate set as recall score.
Results We report the performance of different methods as well as human performance in Table 2 First of all, we observe that the naive serialized model fails to learn anything effective (same as the Majority Guess). It reveals the importance of template when using the pre-trained BERT (Devlin (ct al., 2019) model: the "natural" connection words between individual cells is able to unleash the power of the large pre-trained language model and enable it to perform reasoning table form. Such behavior is understandable given the fact that BERT is pre-trained on purely natural language corpora. In addition, we also observe that the horizontal scan excels in the vertical scan because it better captures the convention of human expression. Among different LPA methods, we found that LPA-Ranking performs the best since can better suppress the spurious programs than the voting- based algorithm. Overall, the LPA model is on par with Table-BERT on both simple and test split without any pre-training on external corpus, which reflects the effectiveness of LPA to leverage symbolic operations in the verification process.
the structured
Through our human evaluation, we found that only 58% of sentences have been correctly linked without missing-link or over-link, while the systematic search has a recall of 51% under the cases where the sentence correctly linked. With that being said, the chance for LPA method to cover the correct program (rationale) is roughly under 30%. After the discriminator's re-ranking step, the probability of selecting these particular oracle program is even much lower. However, we still observe a final overall accuracy of 65%, which indicates that the spurious problem is quite severe in LPA, where the correct label is predicted based on the wrong reason.
Through our human evaluation, also observe that Table-BERY exhibits poor consistency it can misclassify simple cases but currecty-claassif hard cases. These two major weaknesses are yet to be solved in future studies. In contrast, LPA behaves much more consistently and provides clear latent rationale for its decision. But, such a pipeline system requires laborious handcrafting of API operations and is also very sensitive the entity linking accuracy. Both methods have pros and cons; how to combine them still remains an open question.
Program Annotation To further promote the development of different models in our dataset, we collect roughly 1400 human-annotated programs paired with the original statements. These state- ments include the most popular logical operations like superlative, counting, comparison, unique, etc. We provide these annotations in Githut parsers or provide the rationale for NLI models.
17] which can either be used to bootstrap the semantic
5 RELATED WORK
Natural Language Inference & Reasoning: Modeling reasoning and inference in human language is fundamental and challenging problem towards true natural language understanding. There has been extensive research on RTE in the early years (Dagan et al. 2005) and more recently shifted to NLI (Bowman al., 2015 Williams al., 2017). NLI seeks to determine whether natural language hypothesis h can be inferred from a natural language premise p. With the surge of deep learning. there have been many powerful algorithms like the Decomposed Model (Parikh et al., 2016). Enhanced-LSTM (Chen et al.) 2017) and BERT (Devlin et al. (201)). Besides the textual ev- idence, NLVR (Suhr et al.) (017) and NLVR2 (Suhr al.] 2019) have been proposed to use images as the evidence for statement verification on multi-modal setting. Our proposed fact verification task closely related to these inference tasks, where our stri-strut table can be seen as a collection of "premises" exhibited in a semi-structured format. Our proposed problem hence could be viewed as the generalization of NLI under the semi-structured domain.
Table Question Answering: Another line of research closely related to our task is the table-based
8

---

Published as a conference paper at ICLR 2020
United States House of Repreventatives Elections, 1972
(1)
Devnt
Ircundent
Perty
Rosh
Condittate
Calforria Men
demeraat:
ekcted
John Man 5%
Faiferia
Hallyy ku ses Powel
Calferss
mecokutirr deverrnnchlll
Peee Soark lew Wotdees ki Ierone Wolcle da 30 22:45 John & Mctillii anappoted
Cliffe Wal
Caloorsa Metil
There are five candicates total, two of them are dambcrats and three of them are repubiicans.
Confunctive
[uunstio:: many candiditos totai??
Question: How many democrats there? Answer:
Quastion: How many repubilcaan thare? Answer:
Answer:
Eu(CouniTT STT
Eq 'dem. 1). 2
Eq|Cound/ Filber parly rep 'D) 31-F
(2)
Jordi Arrese
6ife
Ixrravent
wrrtie
putterr
apponstes in Sinil
siete the Fira
siiner
1006
berciin, France
day
dovc de miqual
rohald aginct munserr batrani
7-5,6-4
1964
linid 6-4 6-4.
1666: Winner
1986
Bunnerrup
7-5 6
Linguistic Inference
Mathematicc Inference
Jardi Arrosa achiovas better Score
1385.
Jordi Amose won both the final games 1986.
Figure 5: The two uniqueness of Table-based fact verification against standard QA problems.
question answering, such as MCQ (Jauhar et al. 2016), WikiTableQuestion (Pasupat Liang. 2015), Spider (Yu et al., 2018), Sequential Q&A (lyyer et al.) 2017), and WikiSQL. (Zhong et al., 2017), , for which approaches have been extended to handle large-scale tables from Wikipedia (Bha- gavatula et al. 2013). However, in these Q&A tasks, the question types typically provide strong signals needed for identifying the type of answers, while TABFACT does not provide such speci- ficity. The uniqueness of TABFACT lies in two folds: 1) given fact is regarded as a false claim as long as any part of the statement contains misinformation. Due to the conjunctive nature of verifica- tion, fact needs to be broken down into several sub-clauses or (Q, A) pairs to separate evaluate their correctness. Such a compositional nature of the verification problem makes it more challenging than a standard QA setting. On one hand, the model needs to recognize the multiple QA pairs and their relationship. On the other hand, the multiple sub-clauses make the semantic form longer and logic inference harder than the standard QA setting. 2) some facts cannot even be handled using semantic forms, as they are driven by linguistic inference or common sense. In order to verify these state- ments, more inference techniques have to be leveraged to enable robust verification. We visualize the above two characteristics of TABFACT in Figure 5
Program Synthesis & Semantic Parsing: There have also been great interests in using program synthesis or logic forms to solve different natural language processing problems like question answering (Liang et al. 2013; Berant et al.) 2013; Berant & Liang 2014). visual navigation (Artzi 2014, Artzi & Zettlemoyer 2013). code generation (Yin & Neubig 2017: Dong & Lapata SQL synthesis (Yu et al. 2018), etc. The traditional semantic parsing papers (Artzi et al,
ct al.
2016
2014
Artzi & Zeutlemoeer 2013 Zettlmoyer & Collins| 2005 Liang et al. 2013 Berant et al.
greatly rely on rules, lexicon to parse natural language sentences into different forms like lambda calculus, DCS, etc. More recently, researchers strive propose neural models directly perform end-to-end formal reasoning like Theory Prover (Riedel et al. 2017 Rocktäschel & Riedel, 2017). Neural Turing Machine (Graves et al. (2014). Neural Programmer (Neelakantan et al.,
2013
2016 2017) and Neural-Symbolic Machines (Liang et al.) 2017 2018 Agarwal et al. (019)). The proposed TARFACT serves as a great benchmark to evaluate the reasoning ability of different neural reasoning models. Specifically, TABFACT poses the following challenges: 1) spurious programs (i.e., wrong programs with the true returned answers): since the program output is only a binary label, which can cause serious spurious problems and misguide the reinforcement learning with the under-sper seder--peei binary rewards. 2) decomposition: the model needs to decompose the statement into sub-clauses and verify the sub-clauses one by one, which normally requires the longer logic in- ference chains to infer the statement verdict. 3) linguistic reasoning like inference and paraphrasing.
Fact Checking The problem of verifying claims and hypotheses on the web has drawn significant at- tention recently due its high social influence. Different fact-checking pioneering studies have been

---

Published as a conference paper at ICLR 2020
performed including LIAR (Wang 2017). PolitiFact (Vlachos & Riedel 2014). FEVER (Thomee) et al. 2018) and AggChecker (Jo et al. 2019). etc. The former three studies are mainly based on textual evidence social media or Wikipedia, while AggChecker is closest to ours in using
lational databases as the evidence. Compared to AggChecker, our paper proposes a much larger dataset benchmark the progress in this direction.
CONCLUSION
This paper investigates very important yet previously under-explored research problem: semi- structured fact verification. We construct a large-scale dataset and proposed two methods, Table- BERT and LPA, based on the state-of-the-art pre-trained natural language inference model and pro- gram synthesis. In the future, we plan to push forward this research direction by inspiring more sophisticated architectures that can perform both linguistic and symbolic reasoning.
REFERENCES
Rishabh Agarwal, Chen Liang, Dale Schuurmans, and Mohammad Norouzi. Learning to generalize from sparse and underspecified rewards. International Conference of Machine Learning, 2019.
Yoav Artzi and Luke Zettlemoyer. Weakly supervised learning of semantic parsers for mapping instructions to actions. Transactions of the Association for Computational Linguistics, 1:49-62, 2013.
Yoav Artzi, Dipanjan Das, and Slav Petrov. Learning compact lexicons for ceg semantic parsing. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1273-1283, 2014.
Jonathan Berant and Percy Liang. Semantic parsing via paraphrasing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 1415-1425, 2014.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1533-1544, 2013.
Chandra Sekhar Bhagavatula, Thanapon Noraset, and Doug Downey. Methods for exploring and mining tables on wikipedia. In Proceedings of the ACM SIGKDD Workshop on Interactive Data Exploration and Analytics, pp. 18-26. ACM, 2013.
Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large anno- tated corpus for learning natural language inference. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 632-642, 2015.
Qian Chen, Xiaodan Zhu, Then-Hua Ling, Si Wei. Hui Jiang, and Diana Inkpen. Enhanced Istm for natural language inference. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1657-1668, 2017.
Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine Learning Challenges Workshop, pp. 177-190. Springer, 2005.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of NAACL-HLT, 2019.
Li Dong and Mirella Lapata. Language to logical form with neural attention. In Proceedings of the 54th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 33 43, 2016.
Joseph L. Fleiss. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378, 1971.
Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. arXiv preprint arXiv:1410.5400 2014.
10

---

Published as a conference paper at ICLR 2020
Andreas Hanselowski, Hao Zhang, Zile Li, Daniil Sorokin, Benjamin Schiller, Claudia Schulz, and Iryna Gurevych. Ukp-athene: Multi-sentence textual entailment for claim verification. arXiv preprint wwXh::180.001479 2018.
Naeemul Hassan, Gensheng Zhang, Fatma Arslan, Josue Caraballo, Damian Jimenez, Siddhant Gawsane, Shohedul Hasan, Minumol Joseph, Aaditya Kulkarni, Anil Kumar Nayak, et al. Claim- buster: the first-ever end-to-end fact-checking system. Proceedings of the VLDB Endowment, 10 (12):1945-1948, 2017.
Mohit lyyer, Wen-tau Yih, and Ming-Wii Wei Chang. Search-based neural structured learning for se- quential question answering. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguisties (Volume Long Papers), volume 1, pp. 1821-1831, 2017.
Sujay Kumar Jauhar, Peter Turney, and Eduard Hovy. Tables
knowledge for ques-
tion answering. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), volume 1, pp. 474 483, 2016
Saehan Jo, Immanuel Trummer, Weicheng Yu, Xuezhi Wang, Cong Yu, Daniel Liu, and Niyati Mehta. Aggchecker: A fact-checking system for text summaries of relational data sets. Proceed- ings of the VLDB Endowment, 12(12), 2019.
Jemold J Katz and Jerry A Fodor. The structure of semantic theory. language, 39(2):170-210 1963.
Ni Lao, Tom Mitchell, and William W Cohen. Random walk inference and learning in large scale knowledge base. In Proceedings of the Conference on Empirical Methods Natural Language Processing, pp. 529-53999999 Association for Computational Linguistics, 2011.
Chen Liang. Jonathan Berant, Quoc Le, Kenneth D Forbus, and Ni Lao. Neural symbolic machines: Learning semantic parsers freebase with weak supervision. International Conference of Ma- chine Learning, 2017.
Chen Liang. Mohammad Norouzi, Jonathan Berant, Quoc V Le, and Ni Lao. Memory augmented policy optimization for program synthesis and semantic parsing. In Advances in Neural Informa- fion Processing Systems, pp. 1994-10000 2018.
Percy Liang, Michael I Jordan, and Dan Klein. Learning dependency-basee compositional seman- tics. Computational Linguistics, 39(2):389 446, 2013.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. 1692, 2019.
Arvind Neelakantan, Quoc V Le, and Ilya Sutskever. Neural programmer:
Inducing latent programs with gradient descent. International Conference on Learning Representation, 2016.
Arvind Neelakantan, Quoc V Le, Martin Abadi, Andrew McCallum, and Dario Amodei. Learn- ing a natural language interface with neural programmer. International Conference on Learning Representation, 2017.
Ankur Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. decomposable attention model for natural language inference. In Proceedings of the 2016 Conference Empirical Meth- ods in Natural Language Processing, pp. 2249-2253 2016.
Panupong Pasupat and Percy Liang, Compositional semantic parsing semi-structured tables. In Proceedings of the 53d Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference Natural Language Processing (Volume 1: Long Papers), volume 1, pp. 1470-1480, 2015.
Matthew E Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of NAACL-HLT, pp. 2227-2237, 2018.
11

---

Published as a conference paper at ICLR 2020
Kashyap Popat, Subhabrata Mukherjee, Jannik Strötgen, and Gerhard Weikum. Where the truth lies: Explaining the credibility of emerging claims on the web and social media. In Proceedings of the 26th International Conference World Wide Web Companion, pp. 1003-1012. International World Wide Web Conferences Steering Committee, 2017.
Sebastian Riedel, Matko Bosnjak, and Tim Rocktäschel. Programming with a differentiable forth interpreter. ICML., 2017.
Tim Rocktäschel and Sebastian Riedel. End-to-end differentiable proving. Advances in Neural Information Processing Systems, pp. 3788-3800, 2017.
Alane Suhr, Mike Lewis, James Ych, and Yoav Artzi. A corpus of natural language for visual reasoning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 217-223, 2017.
Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi A corpus for reasoning about natural language grounded in photographs. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, 2019.
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. Fever: a large- scale dataset for fact extraction and verification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume 1 (Long Papers), volume I, pp. 809-819, 2018.
Johan Van Benthem et al. lions97819049874444 2008
brief history of natural logic.
LondonCollege Publica-
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Andreas Vlachos and Sebastian Riedel. Fact checking: Task definition and dataset construction. In Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science, pp. 18-22, 2014.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. EMNLP 2018, pp. 353, 2018
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. arXiv preprint
2019.
William Yang Wang. liar, liar pants on fire: A new benchmark dataset for fake news detection. In Proceedings ofthe the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 422-426, 2017.
Adina Williams, Nikita Nangia, and Samuel R Bowman. broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv: 2017.
Zhilin Yang, Zihang Dai, Yiming Yang. Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 2019.
Pengcheng Yin and Graham Neubig. syntactic neural model for general- purposs code generation. arXiv preprint 1:X.v:/704.016900 2017.
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al. Spider: A large-scale human-late labeled dataset for complex and domain semantic parsing and text-to-sql task. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp. 391 1-3221, 2018.
12

---

Published as a conference paper at ICLR 2020
Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. Swag: A large-scale adversarial dataset for grounded commonsense inference. Proceedings of the 2018 Conference on Empir- ical Methods in Natural Language Processing, pp. 93-104, 2018.
Luke S Zettlemoyer and Michael Collins. Learning to map sentences logical form: structured classification with probabilistic categorial grammars. In Proceedings of the Twenty-First Confer- ence on Uncertainty in Artificial Intelligence, pp. 658-666. AUAI Press, 2005.
Victor Zhong, Caiming Xiong, and Richard Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning. arXiv preprint arXiv:1709.00103, 2017.
13

---

Published as a conference paper at ICLR 2020
A APPENDIX
A.1 FUNCTION DESCRIPTION
We list the detailed function description in Figure 6
We also visualize the functionality of the most
Name
Arguments
Output
Comment
Count
View
Number
Return number of rows the View
Within
Header
Bool
Return whather the cell string/number exists under Header Column of view
String/Number
Without
View Header String, Cell StringNumber
Bool
Return whether the cell string/rumber does not exist under the Header Column the given view
Rool
Whether the string
"No", "-" information provided"
Before/After
Row, Row
Row
Returns whether row'l is before/after row?
Frst/Secon/Thii//Fouuhh
frst//wconsd/hiid position
wverage/Sum//Max/Mii
String
Number
average/ summattor//mx///
the given
view
Argmin/
View Header String
Row
Returns the row with the maximum/miiingg /mnnimum value under the Header Column the view
Hop
row
String
Difi/Add
Number Number
Number
Perform artthmetic operations on two numbers
Greaterv/Less
Number Number
Bool
Returns whether first number greater fless than the second number
Equal/ Unequal
String, String/
Bool
Compare numbers
whether
Number Number
Fittrr eq/
View Header String. Number
View
Returns the subwieww of aiven with the values under the Header column greater/ hts////////gg against the given number
Filterg greator/
lose/
Filter Filter_greater wgw.ttee_orrooooo ooroorr_ee q_equlll/
Filter less ors_allu
String,
All MiLless/NLLp Ltra/NNLeeeettrrrrrre pr_eeus
Number
greater/ /lll//////// esy against the given number
VAll less or or_equal
operation inputs
Figure 6: The function definition used in TabFact.
typical functions and their input/output examples
Figure
Name
Name
Count
) =
within
Name, A) Trud
B
Name
without
Name, A) Folse
None
./Not Given NA
B
Before
Name
Age
Name
Age
= True
First
Name
Age
=True
A
A
Name
Age
Age)
Argmin
Age
2
Age
2
4
Name
Age
Name
Age
All eq
Age,
Filter, og
A
Age, 2)
2
Hop
Name)
(2, =
Greater
True
True
A
Figure 7: The visualization of different functions.
14

---

Published as a conference paper at ICLR 2020
We list all the trigger words for different functions in Figure
Trigger
Function
averrage
average
difference', ggo"" tthn',, Separate'
'sum', 'summation", "combine', 'combined", 'toaall' 'add', 'all', tthere are'
ddd, sum
'nott, 'nol, 'never', "didntt" "won't", "wase't", "have,"aven"", "weren'" "wonit", 'neither', 'none', 'unable,
_ol_eq, not notwwithin Filler_not_et none
Mail,, "dffferent, "outside", "unable',
'ot, hoo, hoone'
none
"irse', 'top', 'latest', 'moos'
first
Tast,, bottom", "latese', 'mos'"
last
RRBR 'more', 'thee', 'after"
fiter_rreater greater
'RBR; (JRR) "less', 'than', 'below", 'under"
fiter_ Jess, less
'sll,, levery, 'each!
all. eq all less, all greater
['all] 'every, leach"], I'nott, 'no', 'never', "didn't, "won't", "wasn't")
all |otoot, eq
most" 'than'
all_less_ee all_greater_ee
RRBR, RRSS, UUS'
max,
'UR: ROR; R8S, 'top', 'fiss'
argmax, argmin
within' onee, of, among'
within
"ollow, Tollowing,, Yollowed', 'after' before', 'above', 'precede'
before
Yollow. following', Yollowed', 'after' before', 'above', 'precede'
after
'most'
most_freq
ordinal
First, secord, third, fourth
Figure 8: The trigger words used shrink the search space.
B HIGHER-ORDER OPERATIONS
1. Aggregation: the aggregation operation refers to sentences like "the averaged age of all "the total amount of scores obtained in ...",.tc.. etc.
2. Negation: the negation operation refers to sentences like "xxx did not get the best score", "xxx has never obtained score higher than 5"
3. Superlative: the superlative operation refers sentences like "xxx achieves the highest score in"" "xxx the lowest player in the team".
4. Comparative: the comparative operation refers sentences like "xxx has higher score than yyy".
5. Ordinal: the ordinal operation refers to sentences like "the first country to achieve xxx is xxx", "xxx is the second oldest person in the country"..
6. Unique: the unique operation refers to sentences like "there are 5 different nations in the tournament,, " "there are no two different players from U.S"
7. All: the for operation refers to sentences like "all of the trains are departing in morning". "none of the people are older than 25."
8. None:
the sentences which do not involve higher-order operations like "xxx achieves points in xxx game", "xxx player from xxx country".
C ERROR ANALYSIS
Before we quantitatively demonstrate the error analysis of the two methods, we first theoretically analyze the bottlenecks of the two methods as follows:
Symbolic We first provide a case in which the symbolic execution can not deal with theoretically in Figure 9. The fallure cases of symbolic coverage problem. For example, in the given statement below, there is no explicit mention of "7-5, 6-4" cell. Therefore, the entity linking model fails to link to this cell content. Furthermore,
either due the entity link problem function
15

---

Published as a conference paper at ICLR 2020
though we successfully link to this string, there is no defined function to parse "7-5, 6-5" 5 as "won two games" because it requires iingussti/mmathmmatii inference to understand the implication from the string. Such cases are the weakness of symbolic reasoning models.
Jordi Arrese
outcome
date
tournamennt
surface
partner
oppanents the final
the final
1985
Bologna Italy
Alberto Tous
Paolo Cané Simone Colombo
5-7,4-6
winner
1986
Bordeaux, France
David
Ronald Agénor Mansour Bahrami
7-5,6-4
Prague, Crechoslovakia
clay
Smid
6-4,6-4
1986: Winner
1986: Runner-up
Linguistic Inference
Jordi Arrese achieves
1986 1985.
7-5.6-4
Mathematic Inference
Jordi Arrese won both of the final games in 1996.
Figure 9: The error case of symbolic reasoning model
BERT contrast, Table-BERT model seems to have no coverage problem as long it can feed the whole table content. However, due to the template linearization, the table is unfolded into a long sequence as depicted in Figure IO The useful information, "clay" are separated in a very long span of unrelated words. How to grasp such a long dependency and memorize the history information poses a great challenge to the Table-BERT model.
Jordi Arrese
outcome
toumament
partner
opponents
score
up
Bologna Italy
clay
Alberto
Canè Colombo
4-6
1986
Bardeaux, France
clay
David De Miguel
Ranald Agénor Mansour Bahrami
7-5, 6-
winner
1989
Prague, Crechoslovakia clay
Horst Skoff
Petr Korda Tomáš Smid
6-4
Template
Given table titled 'Jordi Arrese", row one, the outcome is runner-up, the date is 1985, the surface clay In row two, the outcome
the surface clay. In row three, the outcome is
the surface clay.
Long Dependency
The three "Clay" are separated by more over words
Jordi played games clay surface.
Figure 10: The error case of BERT NLI model
Statistics Here we pick 200 samples from the validation set which only involve single semantic and divide them into different categories. We denote the above-mentioned cases as "linguistic in- ference", and the sentences which only describe information from one row as "Trivial", the rest are based on their logic operation like Aggregation, Superlative, Count, etc. We visualize the accuracy of LPA and Table-BERT in Figure 11] From which we can observe that the statements with linguis- tie inference are much better handled with the BERT model, while LPA achieves an accuracy barely higher than random guess. The BERT model can deal with trivial cases well as it uses horizontal scan order. contrast, the LPA model outperforms BERT on higher-order logic cases, especially when the statement involves operations like Count and Superlative.
16

---

Published as a conference paper at ICLR 2020
Error Analysis of LPA/Table-BERT
aable-BRRT LPA
65
55
45
Linguistic
Trivial
Aggregation
Superfative
Count
Compare
Negation
Figure 11: The error analysis of different models
D REASONING DEPTH
Given that our LPA has the breadth to cover a large semantic space. Here we also show the reasoning depth terms of how many logic inference steps are required tackle verify the given claims. We visualize the histogram in Figure and observe that the reasoning steps are concentrated between to 7. Such statisties indicate the difficulty of fact verification our TABFACT dataset.
Reasoning Depth Statistics in LPA
450000
300000
250000
200000
50000
Figure 12: The histogram of reasoning steps required to verify the claims
E WHETHER TO KEEP WIKIPEDIA CONTEXT
Before crowd- sourcing the annotation for the tables, we observed that the previous WikiTableQues- tion Pasupat & Liang (2015) provides context (Wikipedia title) during annotation while the Wik- Zhong et al. (2017) () does not. Therefore, we particularly design ablation annotation tasks to compare the annotation quality between w/ and w/o Wikipedia title context. We demonstrate
typical example in Figure 13 where a Wiki table 1D aims to describe the achievements of a tennis player named Dennis, but itself does not provide any explicit hint about "Tennis Player Dennis". Unsurprisingly, the sentence fluency and coherence significantly drop without such information. Actually, great portion of these Wikipedia tables requires background knowledge (like sports, celebrity, music, etc) to understand. We perform a small user study to measure the fluency of an- notated statements. Specificallyy, we collected 50 sentences from both annotation w/ and w/o title context and randomly shuffle them pairs, which are distributed to the 8 experts without telling them their source compare the language fluency. turns out that the experts ubiquitously agree that the statements with Wikipedia titles are more human-readable. Therefore, we argue that such a context necessary for annotators to understand
background knowledge to write more flu- ent sentences. On the other end, we also hope to minimize the influence of the textual context in the table-based verification task, therefore, we design an annotation criterion: the Wikipedia title
Ralaatto
17

---

Published as a conference paper at ICLR 2020
is provided to the workers during the annotation, but they explicitly banned from bringing any unrelated background information other than the title into the annotation. As illustrated in Figure 13 the title only acts placeholder in the statements make it sound more natural.
outcome
championship
surface
partner
winner
1960
Wimbledon championships
grass
Rafael Osuna
winner
1961
US Championships
grass
Chuck Mckinley
up
1962
US Championships
grass
Chuck Mckinley
winner
1963
US Championships (2)
grass
Chuck Mckinley
Context (Title]
Richard Dennis Ralston (born 27, 1942, former
No Information is provided
on American former tennis player
Annotate
From 1960 1969, Ralston five major double championships.
Winner the grass surface.
Rafael Osuna is partner in the Wimbeldon
Figure 13: Comparison of worker annotation w/ and w/o Wikipedia title as context
F ENTITY LINKING
Here we propose use the longest string match to find all the candidate entities in the table, when multiple candidates coexist, we select the one with the minimum edit distances. The visualization is demonstrated in Figure 14
Ditris
Incumbent
Party
Resut
Candsdacas
Celffrma
Califmms 5
Phälip Burron
democratic
re-slected
Philip Burton (i) $1.8% Edlo Powell (r) 29.2% Witrden
calforrie
Califmma 24
Jerome Wulde
reputiizan
re-elected
Jermma Walde (di 77.6% Hoyd E. Sima 01 22.4% John unoppesed
Calfomia
Mctill
republican
Statement: John E. Moss democratic who from California district
John E. Moes (6) john (r) 30.1%
30.1%
John
Moss
is
John Motell
Figure 14: Entity Linking System.
G THE PROGRAM CANDIDATES
Here demonstrate some program candidates in Figure 15
and show how our proposed discrimi- nator is designed to compute the matching probability between the statement and program. Specif- ically, employ
transformer-basee encoder Vassanii et al. (2017). the left one is aimed to encode the program sequence and the right one is aimed to encode the statement sequence. Their output from [CLS] position concatenated and fed into MLP to classify the verification label.
H HIT INTERFACE
We provide the human intelligent task interface on AMT in the following. Very detailed instructions on what are trivial statements and what are non-trivial statements. Comprehensive examples have been given guide the Turkers write well-formed while logically plausible statements. In order to harvest fake statements without statistical cues, we also provide detailed instructions on how to re- ee-rrite the "fake" statements. During the annotation, we hire experts to perform sanity checks on each of the HIT to make sure that the annotated dataset is clean and meets our requirements.
18

---

Published as a conference paper at ICLR 2020
Less(Coun(/iltrr(nccaa http:///iinmmmmeetedeecctii Count|filee hher//ncuumeeett-ruubiillllllli
Les(Coun((Flter(icuuum fiiee///cccmmbettteppaiiinnn Sounnf/leeiiuumbeeeddemmoree
Greater[Count|/llbeen SenaennCoounniiiiriiinmnnnne-eeo (earelll Count Fount(Filte((naat
Within[(Fikerf authinnfiterrfnmmmmee--ddd nembentt-mmmdmoo incumbent, republican)=False
(ittin((iller(innnummt republican), incumbent, democratig-falise
And(Same(all] rows, incumbent, democratic) Samefall, rows, incumbent, republican]|-Trus
Or|Same(all rews, incumbent, democratic), Samelall, rows, incumbent, republican])-Trus mollniiill countihheeliiammmmeeeddddeeeeeee
Program Encodor
Statement Encodor
Statement: more democratic than ropublican election.
Figure 15: We demonstrate the top program candidates and the discriminator rank them.
19

---

11/8/2019
HIT
Survey Instructions (Click to expand)
You are given a table with its wikipedia source, your job to compose non-trivial statements supported by the table.
the sentence can be easily generated by looking only certain row without understanding the table.
"Trivial":
the sentence requires reading multiple of the table and understanding of the table content. For example, the sentences which include summarization, comparative, negation, relational, inclusion, superlative, aggregational, rephrase or combinations them are non-trivial. But non-trvial is not limited to these types, any statement involving understanding and reasoning is accepted.
"Non-trivial":
We list two examples below to help you understand, you are encouraged to open the table wikipedia link to understand the context of the table. (Everything in the table is lower-cased, you are free to use lower or upper case in your sentence):
Table Wikipedia Link: Lale__Ruuss_Shllegggeee Island
http:://wwwkkiiioddaa/////ee/llh Thee:___h___hieee [sland)
player
original
gender
eliminated
placing
derrick kosinski
m x treme
male
winner
winner
evelyn smith
fresh meat
female
winner
winner
johnny devenanzio
key west
male
winner
winner
kenny santucci
fresh meat
male
winner
jenn grijalva
rw : denver
female
episode
runner up
paula meronek
nw : key west
female
episode
runner up
robin hibbard
san diego
female
episode
runner
ryan kehoe
fresh
male
episode
up
dunbar merrill
nw : sydney
male
episode
9th place
johanna botta
Nw austin
female
episode
10th place
kellyanne judd
sydney
female
episode
11th place
viewers' revenge
episode
12th place
colie edison
rw denver
female
episode
13th place
cohutta grindstaff
Nw sydney
male episode
14th place
tyrie ballard
denver
male
episode
15th
ashli robson
rw : sydney
female
episode
16th place
rachel robinson
m campus crawl
female
episode
17th place
abram boise
m south pacific
male
episode
18th place
dave malinosky
hollywood
male
episode (quit)
19th place
Rejected ("Trivial") examples:
1. In the TV series "The Island", Derrick Kosinski is male character. (Easy! You can simply look into first
produce this sentence.)
2. Derrick Kosinski has the placing of winner in the TV series.
3. Kenny Santucci is from original season of "Fresh Meat".
httpp://s3 www.a//wwwww..oo/moorroooooooome bulk/ N62 LAA..//LLLA2ZZAAAAAAAAAAAA

---

11/8/2019
HIT
4. Jenn Grijalva is Runner-Up of the challenge.
Accepted ("Non-Trivial") examples:
(Superlative): In the TV series "The Island", Evelyn Smith is the highest ranked female.
(Comparitive): In the TV series "The Island", Jenn Grijalva appears later than Colie Edison in the series.
(Relational): Ashli Robson appears episode later than Rachel Robinson in the TV series.
(Summarization): there are three male winners in the challenge. (Rephrase): Evelyn Smith never eliminated in any episode the TV series. (Combination): Derrick Kosinski is the winner and Jenn Grijalva is Runner-Up of the challenge. (Negation): jenn grijalva is not the female winning the challenge. (Inclusion): Evelyn smith is one of the four winner for the challenge.
Table Wikipedia Link: AFC CFCmpConssnnss League
rank
member association
points
group stage
play off
afc cup
1
saudi arabia
860.5
2
qatar
838.2
0
0
3
iran
813.5
1
0
4
750.2
2
5
uzbekistan
680.8
0
6
india
106.4
0
2
7
jordan
128.7
0
2
Rejected ("Trivial"" examples:
1. In the rank, it has 0 play off.
2. ratar is in rank 2.
3. When member association is india, the points is 106.4.
Accepted ("Non-Trivial") examples:
(Negation): iran one of the two countries getting into the 4th stage. (Average): uae and qatar have an average of play off during the champion league.
(Algorithmic): saudi arabia achieves 22.3 more points than qatar.
(Comparison): india got lower points than jordan in the league.
(Summarization): there are two team which have won the afc cup twice. (Superlative): In the Champions League, saudi arabia achieves the highest points. (Combination): saudi arabia is the group stage 4 while iran is in group stage 3.
Tips1:
We set minimum length to 9, and sentences with more complicated grammar structures are preferred.
Tips2: Do not limited to only one type of description like superlative or relative.
Tips3: Copying the records from the table is encouraged, which can help avoid typos and mis mis-spelling much as possible,
Tips4: Do not vague words like "maybe", "perhaps", "good", "excellent", "most", etc.
httpp://s3 amazonaws wwtp://wwwwwwww//m//tookkk.. bulk:hitt::775000000000 www.ANNNNNN....AAA777A7AA.......
2/4

---

11/8/2019
HIT
First Read the following table, then write five diverse non-trivial facts for this given table:
Table Source: athletics at the 1952 summer olympics men 's pole vault
hhtpss///enwwiiippdaaoogo/////// at the 1952 hatperelmmmmsppepeesee Men%27s_ppokl pal__polllllll_l_pleee_eee vault)
athlete
nationality
3.80
3.95
result
bob richards
united states
4.55 or
don laz
unitad states
4.50
ragnar lundberg
sweden
4.40
petro denysenko
soviet
valto olenius
finland
4.30
bunkichi sawada
japan
XXO
4.20
volodymyr brazhnyk
soviet union
4.20
viktor knyazev
soviet
4.20
george mattos
united states
4.20
erkki kataja
finland
4.10
tamás homonnay
sweden
4.10
lennart lind
hungary
milan milakov
yugoslavia
XO
4.10
rigas efstathiadis
greece
3.95
torfy bryngeinsson
iceland
3.95
erling kaas
norway
3.80
theodosios balafas
greece
0
XXX
3.80
jukka piironen
finland
XX
3.80
zeno dragomir
romania
3.80
Please write non-trivial statement, minimum 9 words
Please write non-trivial statement, minimum 9 words
Please write non-trivial statement, minimum 9 words
Please write non-trivial statement, minimum 9 words
Please write a non-trivial statement, minimum 9 words
httpp://s3 www.aaawwwwww./m/aaaa...... www./hits//70000000ooo.. AA..AAANNNA...LAAAAAAAAAAAAAAAA

---

11/8/2019
HIT
Survey Instructions (Click to expand)
Please first read a table to understand its content, an example is shown below, which contains the leaderboard of a competition.
Player
Original Season
Gender
Eliminated
Placing
Derrick Kosinski
RR: X-Treme
Male
Winner
Winner
Evelyn Smith
Fresh Meat
Female
Winner
Winner
Johnny Devenanzio
Key
Winner
Kenny Santucdi
Fresh Meat
Male
Winner
Winner
Jenn Grijalva
RW: Denver
Female
Episode 8
Runner-Up
Paula Meronek
RW: Key West
Female
Episode 8
Runner-Up
Robin Hibbard
RW: Diego
Female
Episode
Runner-Up
Ryan Kehoe
Fresh Meat
Episode
Runner-Up
Dunbar Merrill
RW: Sydney
Male
Episode 8
9th Place
Johanna Botta
RW: Austin
Female
Episode
10th Place
KellyAnne Judd
Sydney
Female
Episode
11th Place
Dan Walsh
RR: Viewers'
Revenge
Male
Episode
12th Place
Colle Edison
RW: Denver
Female
Episode
13th Place
Cohutta Grindstaff
RW: Sydney
Male
Episode 6
14th Place
Tyrie Ballard
RW: Denver
Male
Episode
15th Place
Ashli Robson
RW: Sydney
Female
Episode
16th Place
Rachel Robinson
RR: Campus Crawl
Female
Episode 3
17th Place
Abram Boise
RR: South Pacific
Male
Episode
18th Place
Malinosky
Hollywood
Episode (quit)
19th Place
Tonya Cooley
RW: Chicago
Female
Episode
20th Place
You are given a sentence to describe a fact the table, please follow the following two cases to finish the job:
If the given sentence is fluent and consistent with the table, then please re-write it to make it "fake" based the following criteria:
1. Contradictory: it should still be fluent and coherent, but it needs be explicitly contrdictory to the
facts in the table.
2. Do not simply add NOT to revert the sentence meaning.
3. Do not write neutral or non-verifiable sentences, you need to confirm it in the table.
3. The fake statement needs to be clear, explicit and natural, do not vague ambiguous words like
"bad", "good", "many", etc.
4. try to use diverse fake types during annotatoin.
Example Given statement: Ashli Robson was eliminated in episode 4.
Good Faking: Ashli Robson survives through episode to episode 5.
Good Faking: Ashli Robson is not the only one eliminated in episode
Bad Faking (Simply add not): Ashli Robson not eliminated episode 4.
httpp://s3 www.aa/wwwww...o/mm/ooooom http://wwwwuuukklllli////////2// htt.://ooo.......oooooaaoaaaaaaa hmml

---

11/8/2019
HIT
Bad Faking (Ambiguous, who is Ashli?): Ashli was not eliminated on episode 4. Bad Faking (Irrelevant): Ashli was born in Mexico.
Bad Faking (Too subjective, what do you mean by "early"): AshiDerrick Kosinski lost the game very early.
Bad Faking (Not verifiable): AshiDerrick Kosinski was the most popular player.
Example 2. Given statement: Tonya Cooley is in the 20th place.
Good Faking: Tonya Cooley not the last placing.
Good Faking: Tonya Cooley is eliminated in episode but not the last in placing.
Bad Faking: (There is nothing larger than 20th) Tonya Cooley is after the 20th place.
Bad Faking: (Half Wrong/half Right) When the gneder is female, the player is Tonya Colley.
Bad Faking (Introduce values outside the table): Tonya Cooley is in the 43th place.
Bad Faking (Typo): Tonya Cooler is in the 20th palace.
If the given statement is erroneous (see following), please type N/A in the input box. 1. critical grammar error like missing verbs, nouns, etc. Do not count small errors like tense, singular/plural, case errors.
2. serious typo, misspelling.
3. the described fact is contradictory to the table.
You can use the highlight button to help you find the mentions in the table, you can use either upper or lower case, not important
https://s3 www.w//wwwww.o.ooomuuuuu... www.

---

11/8/2019
HIT
First Read the given tables, then rewrite the statements to make them fake:
Table Source: 2003 04 isu junior grand prix
rank
nation
gold
silver
bronze
total
russia
10
14
32
2
united states
9
6
7
22
canada
2
Japan
5
hungary
4
0
6
czech republic
2
1
4
ukraine
3
4
Italy
0
sweden
1
2
3
8
israel
1
1
2
finland
0
0
1
france
Hightlight Mentions, Click Me!
Given Statement: russia won the most silver medals in the grand prix
Please rewrite a sentence which is contradictory to the table
Hightlight Mentions, Click Me!
Given Statement: france and finland won the least medals in the grand prix
Please rewrite a sentence which is contradictory to the table
Hightlight Mentions, Click Me!
Given Statement: hungary and finland were the only countries that idd not win any silver medals
Please rewrite a sentence which is contradictory the table
Hightlight Mentions, Click Me!
Given Statement: the united states won more gold medals than canada
Please rewrite a sentence which is contradictory to the table
Hightlight Mentions, Click Me!
Given Statement: canada won the most bronze medals in the grand prix
Please rewrite a sentence which is contradictory to the table
Submit
httpp://s3 awazonaws www.:a/wwwwwwo.oa.aaooockaaa..ea www.a//wwwwwkhhi///992999/// Stt.


Reviews:
[{"review_id": "r1lp3ZLT5S", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Updated review:\nThank you for addressing the comments and making relevant edits. Additionally, the HIT section provides a lot more insights into the data collection process. I've updated my score based on the responses/edits made.\n\n------\n\nThis paper is about a dataset (TABFACT) aimed at promoting research for fact-verification using semi-structured data as evidence. The paper highlights how the existing fact-verification studies have been restricted to work with unstructured evidence, and hence lack generalization to use-cases where the evidence is in a structured format (eg. databases). The paper also highlights how fact-verification with semi-structured evidence is challenging, since it involves both linguistic reasoning (for paraphrasing, entailment etc.) and symbolic reasoning (for operations like count, min, max etc.). To tackle this, the authors suggest two approaches as baselines on the dataset - one uses off-the-shelf BERT model for NLI; the other one focuses on symbolic reasoning and is based on program execution - which primarily uses lexical matching and a set of predefined operations (like count/max/min) to construct a program. \n\nApart from a few issues (mentioned below), the paper is well written. The authors have provided a detailed overview of their data collection/verification pipeline and related model/experiments. Overall, it seems like an interesting dataset and I'm inclined towards accepting the paper. \n\nA few remarks/concerns are:\n1. Usefulness of the dataset:\nIt seems limiting for a fact-verification dataset to restrict itself to a binary space i.e. entailed vs refuted. It is often the case, that statements are not completely true or false. For example, the 3rd refuted statement in Figure 1 is partially true (‘there are five candidates in total’). With a binary space for supervision, we don’t really know if the system is actually able to capture the linguistic and symbolic nuances present in the task. It is entirely possible for the system to “do well” without “learning well”, if the learning/output space is this coarse (as opposed to a dataset like Vlachos and Riedel, 2014).\n2. Related work:\nThe paper talks about introducing a new ‘format’ of evidence (structured text) and talks about ‘unstructured text’ as the only ‘other’ format of evidence. It misses out on a highly related task that uses image as evidence (notable datasets being: CLEVR-Humans, NLVR/2, GQA). Either these should be included in the related work, or the authors should make it explicit that this work only deals with ‘textual’ evidence.\n3. The dataset statistics in Table 1 don’t seem to add up (train+dev+test = (92,283 + 12,792 + 12,779) = 117,854 != 118,275 (=Total #Sentence)).\n4. Page 3, Section 2.3: “we further perform quality control” -> a line or two to explain quality control?\n5. Appendix C: No data/statistics have been provided to support the conclusion of the ablation study.\n\nMinor remarks:\n- Page 2: Section 2: \n1. overtly -> overly\n2. huge tables(e.g. -> huge tables (e.g.\n\n- Page 3: Section 2.3\n1. to filter 18% entailed of entailed statements -> to filter 18% entailed statements\n\n- Page 4: \n1. candidate) . we need to  -> candidate), we need to", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "H1gN3TMEcH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes the problem of fact verification with semi-structured data source such as tables. Specifically, the authors created a new dataset TabFact and evaluated two baseline models with different variations. They applied two criteria and different rewards for workers to collect two subsets of different levels (“simple” and “complex”). They also applied a negative rewriting strategy to avoid exploitable cues or patterns in the annotations. They evaluated two baselines models: (1) latent program algorithm (LPA), which makes use of simple string match entity linking and systematic search and trains a neural network discriminator, and (2) Table-BERT, which linearize the table into a sequence through concatenation or template, and treat it as a classfication problem. Both showed reasonable accuracy (~68%), but still below human performance (92.1%) on a held out test set.\n\nI would like to see the paper accepted because:\n(1) it proposes an interesting task (table fact verification) with a clean dataset, and the experiments evaluated the ability of the current neural network models, such as BERT, or hybrid models, such as the LPA baseline, to perform (symbolic) reasoning;\n(2) special care is done to ensure the dataset doesn't contain simple cues or patterns, which is a common pitfall in dataset collection, and the dataset is also validated through two reasonable baseline models. \n\nSome weakness and concerns are:\n(1) some error analysis of the baseline models are missing. For example, what types questions are hard/simple for table-BERT and what are hard/simple for LPA, and some comparison between them. This helps point out where the difficulty come from, for example, whether the difficulty is language understanding or symbolic reasoning.\n(2) \"To focus on statement verification against the table, we do not feed the caption to the model and simply mask the phrases in the statements which links to the caption with placeholders.\"\nI am not sure this is the right thing to do since even the human annotators requires the caption to understand the context, why not also feed the caption into the model? \n(3) Although the Figure 2 showed that the higher order operations are indeed used in a majority of questions, which measures the breadth of the reasoning, it is unclear about the depths of the required reasoning, i.e., how many operations / steps are required to achieve the correct answer. It would help if the average number of steps / operations required for answering the questions are shown. \n\nIf the above concerns are addressed, I will be willing to raise my score.\n\n\nMinor comments:\n\nThe paper, especially the Appendix, requires some proof reading, for example, I believe the caption for Figure 5 in Appendix A is misplaced.\n\n\"... they are explicitly banned bring ...\" -> \"banned from bringing\"\n\n“As illustrated in Figure 6, the title only acts as a placeholder in the statements to make it sound more natural.” -> From the example, it seems the name of the player is kept unchanged in the sentence, which is different from a placeholder.\n\nSuggestions:\n\nI understand this might require some works, but it would be really helpful to add more comments and maybe examples for the functions shown in Figure 5 (you might need to split the table into two pages or have another table for examples), so that it can be used by works that follows the LPA approach.\n\n“During the annotation, the workers are explicitly guided to modify the words, phrases or sentence structures but retain the sentence style/length to prevent from artificial cues”\nAvoiding simple cues or patterns are important, so it will be good if more details can be shared (for example, the instructions/guidelines you showed to the workers). \n\n==================================\n\nPost rebuttal update:\n\nThanks for the update to the paper. I think this work provides a good dataset and some reasonable baselines, thus should be accepted. \n\nHowever, I am still a bit concerned about the categorization of questions and the analysis on reasoning depth, because they are all based on the programs generated by LPA, while systematic search's recall is just 77% and the programs potentially contains a lot of spurious ones. So the categorization of the questions in Figure 11 and the number of reasoning steps in Figure 12 has to be taken with a grain of salt. It is worth annotating a few hundred, say 100-300 questions, manually with the ground truth programs for question distribution analysis or reasoning depth analysis, and confirm the numbers estimated using programs from LPA. Those manually annotated examples would also be a great addition to the dataset to aid the researchers for better analysis. A good example is the WikiTableQuestions dataset https://github.com/ppasupat/WikiTableQuestions/releases, which contains 300 manually annotated examples (annotated-all.examples), and they are used to analyze the distribution of the questions in the paper. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Bye06G7Btr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new dataset for table-based fact verification and introduces a couple of methods for the task. I think that the dataset would be a useful resource (see some comments nevertheless on its construction), however the methods proposed are not particularly interesting, and the contributions to ML and NLP are overstated in my opinion. In addition the paper needs proof reading as there are many typos, some of which make comprehension problematic. In detail:\n\n- The dataset is the main contribution of this paper. Its size is great. However I have some concerns on its construction. The guidelines described for constructing simple and complex claims are rathe vague, e.g. how does one define \"too much symbolic reasoning\" and explain it to crowdworkers? How was the linguistic complexity difference between the two channels measured? How were the claims sanity checked? In the beginning of section 2.3 it is stated that quality control filtered a substantial proportion of the statements. How was this done?\n\n- A troublesome aspect in my opinion of the dataset is that it only allows for evaluation of the Entailed/Refuted binary classification task, but not on whether the model used the right evidence to reach its conclusion. Thus it will always be possible for models to score highly without doing the right thing. Avoiding trivial re-writes helps, but it is unlikely to be enough as human crowd workers will try to optimize their earnings.\n\n- The two models discussed are OK as baselines, but not particularly interesting or appropriate. Both require substantial rule-based processing (named entity linking, latent program construction. templates) and eventually linearize structured data (the program or the table). I understand that this is not the main contribution of the paper, but given the substantial amount of work on semantic parsing and question answering I was expecting more appropriate baselines taking previous work into account. The performance of the model is not great either, especially if we consider that they are only evaluated on returning a binary label, not the correct evidence from the table.\n\n- While it is true that most of the fact verification work has focus on textual sources, the challenge of combining reasoning over continuous and discrete representations is not new. The various QA works mentioned in the related work section address this, as well as work on theorem proving: https://arxiv.org/abs/1705.11040 Furthermore, there has been at least one more previous work on table based verification against  FreeBase tables: https://www.aclweb.org/anthology/D15-1312/. Thus I believe the discussion of the challenges posed by this dataset should be re-framed, especially given that the kinds of programs that need to be constructed are of similar complexity to previous work like the WikiTableQuestions.\n\n- writing: \"the model is expected to excel... but to fall short\", \"we follow the human subject research protocols\" (which ones?), \"in case of obvious stylistic patterns\" (which ones). On the whole it is understandable, but the writing should be improved.\n\n---- Post author response\n\nI have responded to the author response and I have revised my score.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a new dataset for fact verification in text from tables. The task is to identify whether a given claim is supported by the information presented in the table. The authors have also presented two baseline models, one based on BERT and based on symbolic reasoning which have an ok performance on the dataset but still very behind the human performance. The paper is well-written and the arguments and experiments presented in the paper are sound.

After reviewer comments, the authors have incorporated major changes in the paper. I recommend an Accept for the paper in its current form.
Decision: Accept (Poster)