SlowGuess/ACL_2020_OCR huggingface dataset:
not. The primary evaluation metric of TABFACT is label accuracy. The statistics of TABFACT are given in Appendix A. Detailed hyper-parameters for model training are given in Appendix B for better reproducibility of experiments.
We compare our system with following baselines, including the textual matching based baseline Table-BERT and semantic parsing based baseline LPA, both of which are developed by Chen et al. (2019).
- Table-BERT tackles the problem as a matching problem. It takes the linearized table and the statement as the input and employs BERT to predict a binary class.
- Latent Program Algorithm (LPA) formulates the verification problem as a weakly supervised semantic parsing problem. With a given statement, it operates in two steps: (1) latent program search for searching executable program candidates and (2) transformer-based discriminator selection for selecting the most consistent program. The final prediction is made by executing the selected program.
4.1 Model Comparison
In Table 1, we compare our model (LogicalFactChecker) with baselines on the development set and test set. It is worth noting that complex test set and simple test set are partitioned based on its collecting channel, where the former involves higher-order logic and more complex semantic understanding. As shown in Table 1, our model with programs generated by Sequence-to-Action model, significantly outperforms previous systems with  $71.8\%$  label accuracy on the development set and  $71.7\%$  on the test set, and achieves the state-of-the-art performance on the TABFACT dataset.
We conduct ablation studies to evaluate the effectiveness of different components in our model.
Model	Label Acc. (%)
Val	Test
LogicalFactChecker	71.83	71.69
-w/o Graph Mask	70.06	70.13
-w/o Compositionality	69.62	69.61
As shown in Table 2, we evaluate LogicalFactChecker under following settings: (1) removing the graph-based mask described in § 3.3 (the first row); (2) removing the program-guided compositionality learning mechanism described in § 3.4 (the second row).
Table 2 shows that, eliminating the graph-based mask drops the accuracy by  $1.56\%$  on test set. Removing the program-guided compositionality learning mechanism drops the accuracy by  $2.08\%$  on test set, which reflects that the neural module network plays a more important role in our approach. This observation verifies that both mechanisms are beneficial for our task.
We conduct a case study by giving an example shown in Figure 6. From the example, we can see that our system synthesizes a semantic-consistent program of the given statement and make the correct prediction utilizing the synthesized program. This observation reflects that our system has the ability to (1) find a mapping from the textual cues to a complex function (such as the mapping from "most points" to function "argmax") and (2) derive the structure of logical operations to represent the semantic meaning of the whole statement.
Table 2: Ablation studies on the development set and the test set.
Table	Position	Pilot	Country	Points
1	Sebastian Kawa	Poland	69
2	Carlos Rocca	Chile	55
3	Mario Kiessling	Germany	47
4	Uli Schwenk	Germany	40
Statement The country with the most points is Poland.
Program eq(Poland; hop(argmax(Points); Country))
Predict ENTAILED
Figure 6: A case study of our approach.
We randomly select 400 instances and summarize the major types of errors, which can be considered as future directions for further study.
The dominant type of errors is caused by the misleading programs generated by the semantic parser. As shown in the example in Figure 7 (a), the semantic parser fails to generate a semantically correct program because it lacks the external knowledge about the date in the table and the "new year eve" in the statement. The second type of errors is caused by semantic compositionality, even though
6059
Table	Date	Visiting Team	Host Team	Score
Sep. 25	New York Giants	San Diego Chargers	23-45
Oct. 16	Houston Texans	Seattle Seahawks	10-42
Dec. 11	Detroit Lions	Green Bay Packers	13-16
Jan. 1	St. Louis Rams	Dallas Cowboys	20-10
...
Statement The visiting team is the New York Giant on new year eve and St. Louis Rams in New Year's day eq(count.filter_eq(Visiting Team;New York Giants));count.filter_eq(Visiting Team;St. Louis Ram))) (a)
Player	Country	Score
Juli Inkster	United States	65
Momoko Ueda	Japan	66
Laura Diaz	United States	66
Ji Young	South Korea	66
…
Name	Team	Best
Tristan Gommendy	Pkv Racing	1:16.776
Will Power	Team Australia	1:16.841
Neel Jani	Pkv Racing	1:16.931
Paul Tracy	Forsythe Racing	1:17.629
...
Figure 7: Examples of error types, including (a) predicting a wrong program because of the lack of background knowledge, (b) predicting a correct program but predicting a wrong label, and (c) that the logical operations required to understand the statement is not covered in the grammar.
programs are correctly predicted. As shown in Figure 7 (b), the program involves operations requiring complex reasoning, like counting the exact number of rows. Potential ways to alleviate this problem is to design more function-specific modules like Andreas et al. (2015). The third type of errors is caused by the coverage of the logical operations we used. In this work, we follow Chen et al. (2019) and use exactly the same functions. However, as shown in 7 (c), understanding this statement requires the function of difference_time, which is not covered by the current set.
There is a growing interest in fact checking in NLP with the rising importance of assessing the truthfulness of texts, especially when pre-trained language models (Radford et al., 2019; Zellers et al., 2019; Keskar et al., 2019) are more and more powerful in generating fluent and coherent texts. Previous studies in the field of fact checking differ in the genres of supporting evidence used for verification, including natural language (Thorne et al., 2018), semi-structured tables (Chen et al., 2019), and images (Zlatkova et al., 2019; Nakamura et al., 2019).
The majority of previous works deal with textual evidence. FEVER (Thorne et al., 2018) is one of the most influential datasets in this direction, where evidence sentences come from 5.4 million Wikipedia documents. Systems developed on FEVER are dominated by pipelined approaches with three separately trained models, i.e. document retrieval, evidence sentence selection, and claim verification. There also exist approaches (Yin and Roth, 2018) that attempt to jointly learn evidence selection and claim verification. More recently, the second FEVER challenge (Thorne et al., 2019) is built for studying adversarial attacks in
fact checking<sup>4</sup>. Our work also relates to fake news detection. For example, Rashkin et al. (2017) study fact checking by considering stylistic lexicons, and Wang (2017) builds LIAR dataset with six fine-grained labels and further uses meta-data features. There is a fake news detection challenge<sup>5</sup> hosted in WSDM 2019, with the goal of the measuring the truthfulness of a new article against a collection of existing fake news articles before being published. There are very recent works on assessing the factual accuracy of the generated summary in neural abstractive summarization systems (Goodrich et al., 2019; Krysciński et al., 2019), as well as the use of this factual accuracy as a reward to improve abstractive summarization (Zhang et al., 2019).
Chen et al. (2019) recently release TABFACT, a large dataset for table-based fact checking. Along with releasing the great dataset, they provide two baselines: Table-BERT and LPA. Table-BERT is a textual matching based approach, which takes the linearized table and statement as inputs and states the veracity. However, Table-BERT fails to utilize logical operations. LPA is a semantic parsing based approach, which first synthesizes programs by latent program search and then ranks candidate programs with a neural-based discriminator. However, the ranking step in LPA does not consider the table information. Our approach simultaneously utilizes the logical operations for semantic compositionality and the connections among tables, programs, and statements. Results show that our approach achieves the state-of-the-art performance on TABFACT.
4http://fever.ai/
<https://www.kaggle.com/c/
fake-news-pair-classification-challenge/
In this paper, we present LogicalFactChecker, a neural network based approach that considers logical operations for fact checking. We evaluate our system on TABFACT, a large-scale benchmark dataset for verifying textual statements over semi-structured tables, and demonstrate that our approach achieves the state-of-the-art performance. LogicalFactChecker has a sequence-to-action semantic parser for generating programs, and builds a heterogeneous graph to capture the connections among statements, tables, and programs. We utilize the graph information with two mechanisms, including a mechanism to learn graph-enhanced contextual representations of tokens with graph-based attention mask matrix, and a neural module network which learns semantic compositionality in a bottom-up manner with a fixed set of modules. We find that both graph-based mechanisms are beneficial to the performance, and our sequence-to-action semantic parser is capable of generating semantic-consistent programs.
Wanjun Zhong, Jiahai Wang and Jian Yin are supported by the National Natural Science Foundation of China (U1711262, U1611264, U1711261, U1811261, U1811264, U1911203), National Key R&D Program of China (2018YFB1004404), Guangdong Basic and Applied Basic Research Foundation (2019B1515130001), Key R&D Program of Guangdong Province (2018B010107005). The corresponding author is Jian Yin.
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2015. Deep compositional question answering with neural module networks. *ArXiv*, abs/1511.02799.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1533-1544.
Bo Chen, Le Sun, and Xianpei Han. 2018. Sequence-to-action: End-to-end semantic graph generation for semantic parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 766-777, Melbourne, Australia. Association for Computational Linguistics.
Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2019. Tabfact: A large-scale dataset for table-based fact verification. arXiv preprint arXiv:1909.02164.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Robert Faris, Hal Roberts, Bruce Etling, Nikki Bourassa, Ethan Zuckerman, and Yochai Benkler. 2017. Partisanship, propaganda, and disinformation: Online media and the 2016 us presidential election.
Ben Goodrich, Vinay Rao, Peter J Liu, and Mohammad Saleh. 2019. Assessing the factual accuracy of generated text. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 166-175. ACM.
Daya Guo, Duyu Tang, Nan Duan, Ming Zhou, and Jian Yin. 2018. Dialog-to-action: conversational question answering over a large-scale knowledge base. In Advances in Neural Information Processing Systems, pages 2942-2951.
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2018. Mapping language to code in programmatic context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1643-1652, Brussels, Belgium. Association for Computational Linguistics.
Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. 2019. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05858.
Wojciech Krysciński, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Evaluating the factual consistency of abstractive text summarization. arXiv preprint arXiv:1910.12840.
Chen Liang, Jonathan Berant, Quoc Le, Kenneth D Forbus, and Ni Lao. 2017. Neural symbolic machines: Learning semantic parsers on freebase with weak supervision. ACL.
Percy Liang. 2016. Learning executable semantic parsers for natural language understanding. arXiv preprint arXiv:1603.06677.
Dipendra Misra, Ming-Wei Chang ad Xiaodong He, and Wen tau Yih. 2018. Policy shaping and generalized update equations for semantic parsing from denotations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium. Association for Computational Linguistics.
Jeff Mitchell and Mirella Lapata. 2010. Composition in distributional models of semantics. Cognitive science, 34(8):1388-1429.
Thomas Müller, Francesco Piccinno, Massimo Nicosia, Peter Shaw, and Yasemin Altun. 2019. Answering conversational questions on structured data without logical forms. arXiv preprint arXiv:1908.11787.
Kai Nakamura, Sharon Levy, and William Yang Wang. 2019. r/fakededit: A new multimodal benchmark dataset for fine-grained fake news detection. arXiv preprint arXiv:1911.03854.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8).
Hannah Rashkin, Eunsol Choi, Jin Yea Jang, Svitlana Volkova, and Yejin Choi. 2017. Truth of varying shades: Analyzing language in fake news and political fact-checking. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2931-2937.
Richard Socher, John Bauer, Christopher D Manning, and Andrew Y Ng. 2013. Parsing with compositional vector grammars. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 455-465.
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. Fever: a large-scale dataset for fact extraction and verification. arXiv preprint arXiv:1803.05355.
James Thorne, Andreas Vlachos, Oana Cocarascu, Christos Christodoulopoulos, and Arpit Mittal. 2019. The FEVER2.0 shared task. In Proceedings of the Second Workshop on Fact Extraction and VERIFICATION (FEVER), pages 1-6, Hong Kong, China. Association for Computational Linguistics.
Vaibhav Vaibhav, Raghuram Mandyam Annasamy, and Eduard Hovy. 2019. Do sentence interactions matter? leveraging sentence level representations for fake news classification. arXiv preprint arXiv:1910.12203.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
William Yang Wang. 2017. "liar, liar pants on fire": A new benchmark dataset for fake news detection. arXiv preprint arXiv:1705.00648.
Pengcheng Yin and Graham Neubig. 2017. A syntactic neural model for general-purpose code generation. arXiv preprint arXiv:1704.01696.
Pengcheng Yin and Graham Neubig. 2018. Tranx: A transition-based neural abstract syntax parser for semantic parsing and code generation. arXiv preprint arXiv:1810.02720.
Wenpeng Yin and Dan Roth. 2018. Twowingos: A twowing optimization strategy for evidential claim verification. arXiv preprint arXiv:1808.03465.
Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. arXiv preprint arXiv:1905.12616.
Yuhao Zhang, Derek Merck, Emily Bao Tsai, Christopher D Manning, and Curtis P Langlotz. 2019. Optimizing the factual correctness of a summary: A study of summarizing radiology reports. arXiv preprint arXiv:1911.02541.
Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2sql: Generating structured queries from natural language using reinforcement learning. arXiv preprint arXiv:1709.00103.
Dimitrina Zlatkova, Preslav Nakov, and Ivan Koychev. 2019. Fact-checking meets fauxtography: Verifying claims about images. arXiv preprint arXiv:1908.11722.
A Statistic of TABFACT
Split	#Sentence	Table	Avg. Row	Avg. Col
Train	92,283	13,182	14.1	5.5
Val	12,792	1,696	14.0	5.4
Test	12,779	1,695	14.2	5.4
Table 3: Basic statistics of Train/Val/Test split in the dataset
In this part, we describe the training details of our experiments. As described before, the semantic parser and statement verification model are trained separately.
We first introduce the training process of the semantic parser. Both training and validation datasets are created in a same way as described in § 3.5. Specifically, each pair of data is labeled as true or false. Finally, the training dataset contains 495,131 data pairs, and the validation dataset contains 73,792 data pairs. We implement the approach with the LSTM-based recurrent network and use the following set of hyper parameters to train models: hidden size is 256, learning rate is 0.001, learning rate decay is 0.5, dropout is 0.3, batch size is 150. We use glove embedding to
initialize embedding and use Adam to update the parameters. We use beam search during inference and set beam size as 15. We use BLEU to select the best checkpoint by validation scores.
Then we introduce the training details of statement verification model. We employ cross-entropy loss as the loss function. We apply AdamW as the optimizer for model training. In order to directly compare with Table-BERT, we also employ BERT-Base as the backbone of our approach. The BERT network and neural module network are trained jointly. We set learning rate as 1e-5, batch size as 8 and set max sequence length as 512. The training time for one epoch is 1.2 hours by 4 P40 GPUs. We set the dimension of entity node representation as 200.
C ASDL-Grammar
In this part, we introduce the ASDL grammar (Yin and Neubig, 2018) we apply for synthesizing the programs in Seq2Action model. The definition of functions mainly follows Chen et al. (2019). Details can be found in following two pages. $^6$
The function "filter_eq" contains three arguments (subtable, column_name, value), but we ignore the first argument in the running example for a clearer illustration.
6063
Composite Type	Constructor	Fields
OutBool	Bool	pr bool bool
none	OutStr str
only	OutRow row
zero	OutNum num
after	OutRow row1, OutRow row2
before	OutRow row1, OutRow row2
first	OutRow row1, OutRow row2
second	OutRow row1, OutRow row2
third	OutRow row1, OutRow row2
fourth	OutRow row1, OutRow row2
fifth	OutRow row1, OutRow row2
last	OutRow row1, OutRow row2
greater	OutNum num1, OutNum num2
less	OutNum num1, OutNum num2
eq	OutStr str1, OutStr str2
not_eq	OutStr str1, OutStr str2
and	OutBool bool1, OutBool bool2
within	OutRow row, pr_header header, OutStr str
not_within	OutRow row, pr_header header, OutStr str
all_eq	OutRow row, pr_header header, OutStr str
all_NOT_eq	OutRow row, pr_header header, OutStr str
all-less	OutRow row, pr_header header, OutNum num
all-less_eq	OutRow row, pr_header header, OutNum num
allgreater	OutRow row, pr_header header, OutNum num
allgreater EQ	OutRow row, pr_header header, OutNum num
OutRow	Row	pr_row row
top	OutRow row
bottom	OutRow row
argmax	OutRow row, pr_header header
argmin	OutRow row, pr_header header
filter EQ	OutRow row, pr_header header, OutStr str
filter.not EQ	OutRow row, pr_header header, OutStr str
filter-less	OutRow row, pr_header header, OutNum num
filtergreater	OutRow row, pr_header header, OutNum num
filtergreater EQ	OutRow row, pr_header header, OutNum num
filter-less EQ	OutRow row, pr_header header, OutNum num
OutNum	Num	pr_number num
count	OutRow row
half	OutRow row
one THIRD	OutRow row
inc_num	OutNum num
uniq	OutRow row, pr_header header
avg	OutRow row, pr_header header
sum	OutRow row, pr_header header
max	OutRow row, pr_header header
min	OutRow row, pr_header header
diff	OutNum num1, OutNum num2
add	OutNum num1, OutNum num2
OutStr	Str	pr_str str
hop	OutRow row, pr_header header
most_freq	OutRow row, pr_header header
OutNone	dec_num	OutNum num