SlowGuess/ACL_2020_OCR huggingface dataset:
Wanjun Zhong $^{1*}$ , Duyu Tang $^{2}$ , Zhangyin Feng $^{3*}$ , Nan Duan $^{2}$ , Ming Zhou $^{2}$ , Ming Gong $^{2}$ , Linjun Shou $^{2}$ , Daxin Jiang $^{2}$ , Jiahai Wang $^{1}$  and Jian Yin $^{1}$
<sup>1</sup> The School of Data and Computer Science, Sun Yat-sen University. Guangdong Key Laboratory of Big Data Analysis and Processing, Guangzhou, P.R.China  
<sup>2</sup> Microsoft Corporation <sup>3</sup> Harbin Institute of Technology {zhongwj25@mail2,wangjiah@mail,issjyin@mail}.sysu.edu.cn {dutang,nanduan,mingzhou,migon,lisho,djiang}@microsoft.com zyfeng@ir.hit.edu.cn
Verifying the correctness of a textual statement requires not only semantic reasoning about the meaning of words, but also symbolic reasoning about logical operations like count, superlative, aggregation, etc. In this work, we propose LogicalFactChecker, a neural network approach capable of leveraging logical operations for fact checking. It achieves the state-of-the-art performance on TABFACT, a large-scale, benchmark dataset built for verifying a textual statement with semi-structured tables. This is achieved by a graph module network built upon the Transformer-based architecture. With a textual statement and a table as the input, LogicalFactChecker automatically derives a program (a.k.a. logical form) of the statement in a semantic parsing manner. A heterogeneous graph is then constructed to capture not only the structures of the table and the program, but also the connections between inputs with different modalities. Such a graph reveals the related contexts of each word in the statement, the table and the program. The graph is used to obtain graph-enhanced contextual representations of words in Transformer-based architecture. After that, a program-driven module network is further introduced to exploit the hierarchical structure of the program, where semantic compositionality is dynamically modeled along the program structure with a set of function-specific modules. Ablation experiments suggest that both the heterogeneous graph and the module network are important to obtain strong results.
Fact checking for textual statements has emerged as an essential research topic recently because of the unprecedented amount of false news and rumors spreading through the internet (Thorne et al., 2018;
Table
Year	Venue	Winner	Score
2005	Arlandastad	David Patrick	272
2004	Arlandastad	Matthew King	270
2003	Falsterbo	Titch Moore	273
2002	Halmstad	Thomas Besancenez	279
Statement In 2004, the score is less than 270.
Label REFUTED
Program lessHop filter_eq(Year;2004);Score)270
Figure 1: An example of table-based fact checking. Given a statement and a table as the input, the task is to predict the label. Program reflects the underlying meaning of the statement, which should be considered for fact checking.
Chen et al., 2019; Goodrich et al., 2019; Nakamura et al., 2019; Krysciński et al., 2019; Vaibhav et al., 2019). Online misinformation may manipulate people's opinions and lead to significant influence on essential social events like political elections (Faris et al., 2017). In this work, we study fact checking, with the goal of automatically assessing the truthfulness of a textual statement.
The majority of previous studies in fact checking mainly focused on making better use of the meaning of words, while rarely considered symbolic reasoning about logical operations (such as "count", "superlative", "aggregation"). However, modeling logical operations is an essential step towards the modeling of complex reasoning and semantic compositionality. Figure 1 shows a motivating example for table-based fact checking, where the evidence used for verifying the statement comes from a semi-structured table. We can see that correctly verifying the statement "In 2004, the score is less than 270" requires a system to not only discover the connections between tokens in the statement and the table, but more importantly understand the meaning of logical operations and how they interact in a structural way to form a whole. Under this
6053
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6053-6065 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Figure 2: An overview of our approach LogicalFactChecker. It includes a semantic parser to generate program ( $\S$  3.5), a graph construction mechanism ( $\S$  3.2), a graph-based contextual representation learning for tokens ( $\S$  3.3) and a semantic composition model over the program by neural module network ( $\S$  3.4).
consideration, we use table-based fact checking as the testbed to investigate how to exploit logical operations in fact checking.
In this paper, We present LogicalFactChecker, a neural network approach that leverages logical operations for fact checking when semi-structured tables are given as evidence. Taking a statement and a table as the input, it first derives a program, also known as the logical form, in a semantic parsing manner (Liang, 2016). Then, our system builds a heterogeneous graph to capture the connections among the statement, the table and the program. Such connections reflect the related context of each token in the graph, which are used to define attention masks in a Transformer-based (Vaswani et al., 2017) framework. The attention masks are used to learn graph-enhanced contextual representations of tokens<sup>1</sup>. We further develop a program-guided neural module network to capture the structural and compositional semantics of the program for semantic compositionality. (Socher et al., 2013; Andreas et al., 2015). Graph nodes, whose representations are computed using the contextual representations of their constituents, are considered as arguments, and logical operations are considered as modules to recursively produce representations of higher level nodes along the program.
Experiments show that our system outperforms previous systems and achieves the state-of-the-art verification accuracy. The contributions of this paper can be summarized as follows:
- We propose LogicalFactChecker, a graph-based neural module network, which utilizes logical operations for fact-checking.
- Our system achieves the state-of-the-art performance on TABFACT, a large-scale and benchmark dataset for table-based fact checking.
- Experiments show that both the graph-enhanced contextual representation learning mechanism and the program-guided semantic compositionality learning mechanism improve the performance.
We study the task of table-based fact checking in this paper. This task is to assess the veracity of a statement when a table is given as evidence. Specifically, we evaluate our system on TABFACT (Chen et al., 2019), a large benchmark dataset for table-based fact checking. With a given semi-structured table and a statement, systems are required to perform reasoning about the structure and content of the table and assess whether the statement is "ENTAILED" or "REFUTED" by the table. The official evaluation metric is the accuracy for the two-way classification (ENTAILED/REFUTED). TABFACT consists of 118,439 statements and 16,621 tables from Wikipedia. More details about the dataset are given in Appendix A.
3 LogicalFactChecker: Methodology
In this section, we present our approach LogicalFactChecker, which simultaneously considers the meaning of words, inner structure of tables and programs, and logical operations for fact-checking. One way to leverage program information is to use standard semantic parsing methods, where automatically generated programs are directly executed on
<sup>1</sup>Here, tokens include word pieces in the statement, table column names, table row names, table cells, and the program.
6054
tables to get results. However, TABFACT does not provide annotated programs. This puts the problem in a weak-supervised learning setting, which is one of the major challenges in the semantic parsing field. In this work, we use programs in a soft way that programs are represented with neural modules to guide the reasoning process between a textual statement and a table.
Figure 2 gives an overview of our approach. With a statement and a corresponding table, our system begins with program generation, which synthesizes a program. Then, we build a heterogeneous graph for capturing the inner structure of the input. With the constructed graph, we incorporate a graph-based attention mask into the Transformer for learning graph-enhanced token representations. Lastly, we learn the semantic compositionality by developing a program-guided neural module network and make the final prediction.
This section is organized as follows. We first describe the format of the program (§ 3.1) for a more transparent illustration. After that, the graph construction approach (§ 3.2) is presented first, followed by a graph-enhanced contextual representation learning mechanism (§ 3.3). Moreover, we introduce how to learn semantic compositionality over the program by neural module network (§ 3.4). At last, we describe how to synthesize programs by our semantic parsing model (§ 3.5).
3.1 Program Representation
Before presenting the technical details, we first describe the form of the program (also known as logical form) for clearer illustrations.
With a given natural language statement, we begin by synthesizing the corresponding semantic representation (LISP-like program here) using semantic parsing techniques. Following the notation defined by Chen et al. (2019), the functions (logical operations) formulating the programs come from a fixed set of over 50 functions, including "count" and "argmax", etc. The detailed description of the functions is given in Appendix C. Each function takes arguments of predefined types like string, number, bool or sub-table as input. The programs have hierarchical structure because the functions can be nested. Figure 3 shows an example of a statement and a generated program, accompanying with the derivation of the program and its semantic structure. The details of the generation of a program for a textual statement are introduced in § 3.5.
S  $\rightarrow$  less(S1,ARG0)
S1  $\rightarrow$  hop(S2,ARG1)
S2  $\rightarrow$  filter_eq (ARG2, ARG3)
$\mathrm{ARG0}\rightarrow 270$
ARG1  $\rightarrow$  Score
ARG2  $\rightarrow$  Year
ARG3  $\rightarrow$  2004
(a) Derivation with basic operations
Figure 3: An example of a program with its semantic structure and derivation with basic logical operations.
(b) The structure of compositionality
3.2 Graph Construction
In this part, we introduce how to construct a graph to explicitly reveal the inner structure of programs and tables, and the connections among statements and them. Figure 4 shows an example of the graph. Specifically, with a statement, a table and a pro
Figure 4: An example of the constructed graph.
gram, our system operates in the following steps.
- For a table, we define nodes as columns, cells, and rows, which is partly inspired by the design of the graph for table-based question answering (Müller et al., 2019). As shown in Figure 4, each cell is connected to its corresponding column node and row node. Cell nodes in the same row are fully-connected to each other.
- Program is a naturally structural representation consisting of functions and arguments. In the program, functions and arguments are represented as nodes, and they are hierarchically connected along the structure. Each node is connected to its direct parents and children.
Arguments are also linked to corresponding column names of the table.
- By default, in the statement, all tokens are the related context of each other, so they are connected. To further leverage the connections from the statement to the table and the program, we add links for tokens which are linked to cells or columns in the table, and legitimate arguments in the program.
After these processes, the extracted graph not only maintains the inner-structure of tables and programs but also explores the connections among aligned entities mentioned in different contents.
3.3 Graph-Enhanced Contextual Representations of Tokens
We describe how to utilize the graph structure for learning graph-enhanced contextual representations of tokens  ${}^{2}$  . A simple way to learn contextual representations is to concatenate all the contents  ${}^{3}$  as a single string and use the original attention mask in Transformer, where all the tokens are regarded as the contexts for each token. However, this simple way fails to capture the semantic structure revealed in the constructed graph. For example, according to Figure 4, the content "2004" exists in the statement, program and table. These aligned entity nodes for "2004" should be more related with each other when our model calculate contextual representations. To address this problem, we use the graph structure to re-define the related contexts of each token for learning a graph-enhanced representation.
Specifically, we present a graph-based mask matrix for self-attention mechanism in Transformer. The graph-based mask matrix  $G$  is a 0-1 matrix of the shape  $N \times N$ , where  $N$  denotes the total number of tokens in the sequence. This graph-based mask matrix records which tokens are the related context of the current token.  $G_{ij}$  is assigned as 1 if token  $j$  is the related context of token  $i$  in the graph and 0 otherwise.
Then, the constructed graph-based mask matrix will be feed into BERT (Devlin et al., 2018) for learning graph-enhanced contextual representations. We use the graph-based mask to control
the contexts that each token can attend in the self-attention mechanism of BERT during the encoding process. BERT maps the input  $\pmb{x}$  of length  $T$  into a sequence of hidden vectors as follows.
$$
h (\boldsymbol {x}) = \left[ h (\boldsymbol {x}) _ {1}, h (\boldsymbol {x}) _ {2}, \dots , h (\boldsymbol {x}) _ {T} \right] \tag {1}
$$
These representations are enhanced by the structure of the constructed graph.
3.4 Semantic Compositionality with Neural Module Network
In the previous subsection, we describe how our system learns the graph-enhanced contextual representations of tokens. The process mentioned above learns the token-level semantic interaction. In this subsection, we make further improvement by learning logic-level semantics using program information. Our motivation is to utilize the structures and logical operations of programs for learning logic-enhanced compositional semantics. Since the logical operations forming the programs come from a fixed set of functions, we design a modular and composable network, where each logical operation is represented as a tailored module and modules are composed along the program structure.
We first describe how we initialize the representation for each entity node in the graph (§ 3.4.1). After that, we describe how to learn semantic compositionality based on the program, including the design of each neural module (§ 3.4.2) and how these modules are composed recursively along the structure of the program (§ 3.4.3).
3.4.1 Entity Node Representation
In a program, entity nodes denote a set of entities (such as "David Patrick") from input contexts while function nodes denote a set of logical operations (such as "filterequal"), both of which may contain multiple words/word-pieces. Therefore, we take graph-enhanced contextual representations as mentioned in §3.3 to initialize the representations of entity nodes. Specifically, we initialize the representation  $h_e$  of each entity node  $e$  by averaging the projected hidden vectors of each words contained in  $e$  as follows:
$$
h _ {e} = \frac {1}{n} \sum_ {i = 0} ^ {n} r e l u \left(\boldsymbol {W} _ {\boldsymbol {e}} h (\boldsymbol {x}) _ {p _ {e} ^ {i}}\right) \tag {2}
$$
where  $n$  denotes the total number of tokens in the span of entity  $e$ ,  $p_e^i$  denotes the position of the  $i^{th}$  token,  $\mathbf{W}_e \in \mathbb{R}^{F \times D}$  is a weight matrix,  $F$  is the
2In this work, tokens include word pieces in the statement, column names and row names and contents of cells in the table, and function names in the program
3All the contents indicate texts in the concatenated sequence of the linearized table, the statement, and the sequence of the linearized program.
6056
dimension of feature vectors of arguments,  $D$  is the dimension of hidden vectors of BERT and  $relu$  is the activation function.
3.4.2 Modules
In this part, we present function-specific modules, which are used as the basic computational units for composing all the required configurations of module network structures.
Inspired by the neural module network (Andreas et al., 2015) and the recursive neural network (Socher et al., 2013), we implement each module with the same neural architecture but with different function-specific parameters. All the modules are trained jointly. Each module corresponds to a specific function, where the function comes from a fixed set of over 50 functions described before. In a program, each logical operation has the format of FUNCTION  $(ARG0, ARG1, \ldots)$ , where each function may have variable-length arguments. For example, the function hop has 2 arguments while the function count has 1 argument. To handle variable-length arguments, we develop each module as follows. We first calculate the composition for each function-argument pair and then produce the overall representation via combining the representations of items.
The calculation for each function-argument pair is implemented as matrix-vector multiplication, where each function is represented as a matrix and each argument is represented as a vector. This is inspired by vector-based semantic composition (Mitchell and Lapata, 2010), which states that matrix-vector multiplication could be viewed as the matrix modifying the meaning of vector. Specifically, the output  $y_{m}$  of module  $m$  is computed with the following formula:
$$
y _ {m} = \frac {1}{N ^ {m}} \sum_ {i = 0} ^ {N ^ {m}} \sigma \left(\boldsymbol {W} _ {\boldsymbol {m}} v _ {i} + b _ {m}\right) \tag {3}
$$
where  $W_{m}\in \mathbf{R}^{F\times F}$  is a weight matrix and  $b_{m}$  is a bias vector for a specific module  $m$ .  $N^m$  denotes the number of arguments of module  $m$ , and each  $v_{i}\in \mathbf{R}^{F}$  is the feature vector representing the  $i^{th}$  input.  $\sigma$  is the activation function.
Under the aforementioned settings, modules can compose into a hierarchical network determined by the semantic structure of the parsed program.
3.4.3 Program-Guided Semantic Compositionality
In this part, we introduce how to compose a program-guided neural module network based on the structure of programs and predefined modules. Taking the structure of the program and representations of all the entity nodes as the input, the composed neural module network learns the compositionality of the program for the final prediction. Figure 5 shows an example of a composed network based on the structure of the program.
Statement In 2004, the score is less than 270.  
Program less(hop过滤器_eq(Year; 2004); Score); 270)
Figure 5: An example of neural module network.
Along the structure of the program, each step of compositionality learning is to select a module from a fixed set of parameterized modules defined in § 3.4.2 and operate on it with Equation 3 to dynamically generate a higher-level representation. The above process will be operated recursively until the output of the top-module is generated, which is denoted as  $y_{m}^{top}$ .
After that, we make the final prediction by feeding the combination of  $y_{m}^{top}$  and the final hidden vector  $h(\pmb{x})_T$  from § 3.3 through an MLP (Multilayer Perceptron) layer. The motivation of this operation is to retain the complete semantic meaning of the whole contexts because some linguistic cues are discarded during the synthesizing process of the program.
3.5 Program Generation
In this part, we describe our semantic parser for synthesizing a program for a textual statement. We tackle the semantic parsing problem in a weakly-supervised setting (Berant et al., 2013; Liang et al., 2017; Misra et al., 2018), since the ground-truth program is not provided.
6057
Model	Val	Test	Test (simple)	Test (complex)	Small Test
Human Performance	-	-	-	-	92.1
Majority Guess	50.7	50.4	50.8	50.0	50.3
BERT classifier w/o Table	50.9	50.5	51.0	50.1	50.4
Table-BERT (Horizontal-S+T-Concatenate)	50.7	50.4	50.8	50.0	50.3
Table-BERT (Vertical-S+T-Template)	56.7	56.2	59.8	55.0	56.2
Table-BERT (Vertical-T+S-Template)	56.7	57.0	60.6	54.3	55.5
Table-BERT (Horizontal-S+T-Template)	66.0	65.1	79.0	58.1	67.9
Table-BERT (Horizontal-T+S-Template)	66.1	65.1	79.1	58.2	68.1
LPA-Voting w/o Discriminator	57.7	58.2	68.5	53.2	61.5
LPA-Weighted-Voting w/ Discriminator	62.5	63.1	74.6	57.3	66.8
LPA-Ranking w/ Discriminator	65.2	65.0	78.4	58.5	68.6
LogicalFactChecker (program from LPA)	71.7	71.6	85.5	64.8	74.2
LogicalFactChecker (program from Seq2Action)	71.8	71.7	85.4	65.1	74.3
Table 1: Performance on TABFACT in terms of label accuracy (\%). The performances of Table-BERT and LPA are reported by Chen et al. (2019). Our system is abbreviated as LogicalFactChecker, with program generated via our Sequence-to-Action model and baseline (i.e. LPA), respectively. T, S indicate the table, the statement and + means the order of concatenation. In the linearization of tables, Horizontal (Vertical) refers to the horizontal (vertical) order for concatenating the cells. Concatenate (Template) means concatenating the cells directly (filling the cells into a template). In LPA settings, (Weighted) Voting means assigning each program with (score-weighted) equal weight to vote for the final result. Ranking means using the result generated by the top program ranked by the discriminator.
As shown in Figure 3, a program in TABFACT is structural and follows a grammar with over 50 functions. To effectively capture the structure of the program and also generate legitimate programs following a grammar in the generation process, we develop a sequence-to-action approach, which is proven to be effective in solving many semantic parsing problems (Chen et al., 2018; Iyer et al., 2018; Guo et al., 2018). The basic idea is that the generation of a program tree is equivalent to the generation of a sequence of action, which is a traversal of the program tree following a particular order, like depth-first, left-to-right order. Specifically, our semantic parser works in a top-down manner in a sequence-to-sequence paradigm. The generation of a program follows an ASDL grammar (Yin and Neubig, 2018), which is given in Appendix C. At each step in the generation phase, candidate tokens to be generated are only those legitimate according to the grammar. Parent feeding (Yin and Neubig, 2017) is used for directly passing information from parent actions. We further regard column names of the table as a part of the input (Zhong et al., 2017) to generate column names as program arguments.
We implement the approach with the LSTM-based recurrent network and Glove word vec
tors (Pennington et al., 2014) in this work, and the framework could be easily implemented with Transformer-based framework. Following Chen et al. (2019), we employ the label of veracity to guide the learning process of the semantic parser. We also employ programs produced by LPA (Latent Program Algorithm) for comparison, which is provided by Chen et al. (2019).
In the training process, we train the semantic parser and the claim verification model separately. The training of semantic parser includes two steps: candidate search and sequence-to-action learning. For candidate search, we closely follow LPA by first collecting a set of programs which could derive the correct label and then using the trigger words to reduce the number of spurious programs. For learning of the semantic parser, we use the standard way with back propagation, by treating each (claim, table, positive program) as a training instance.
We evaluate our system on TABFACT (Chen et al., 2019), a benchmark dataset for table-based fact checking. Each instance in TABFACT consists of a statement, a semi-structured Wikipedia table and a label ("ENTAILED" or "REFUTED") indicates whether the statement is supported by the table or
6058