SlowGuess/ACL_2020_OCR huggingface dataset:
Wenhu Chen $^{1}$ , Jianshu Chen $^{2}$ , Yu Su $^{3}$ , Zhiyu Chen $^{1}$  and William Yang Wang $^{1}$
University of California, Santa Barbara, CA, USA
Tencent AI Lab, Bellevue, WA, USA
The Ohio State University, Columbus, Ohio, USA
{wenhuchen, zhiyuchen, william}@cs.ucsb.edu
jiangshuchen@tencent.com, su.809@osu.edu
Neural natural language generation (NLG) models have recently shown remarkable progress in fluency and coherence. However, existing studies on neural NLG are primarily focused on surface-level realizations with limited emphasis on logical inference, an important aspect of human thinking and language. In this paper, we suggest a new NLG task where a model is tasked with generating natural language statements that can be logically entailed by the facts in an open-domain semi-structured table. To facilitate the study of the proposed logical NLG problem, we use the existing TabFact dataset (Chen et al., 2019) featured with a wide range of logical/symbolic inferences as our testbed, and propose new automatic metrics to evaluate the fidelity of generation models w.r.t. logical inference. The new task poses challenges to the existing monotonic generation frameworks due to the mismatch between sequence order and logical order. In our experiments, we comprehensively survey different generation architectures (LSTM, Transformer, Pre-Trained LM) trained with different algorithms (RL, Adversarial Training, Coarse-to-Fine) on the dataset and made following observations: 1) Pre-Trained LM can significantly boost both the fluency and logical fidelity metrics, 2) RL and Adversarial Training are trading fluency for fidelity, 3) Coarse-to-Fine generation can help partially alleviate the fidelity issue while maintaining high language fluency. The code and data are available at https://github.com/wenhuchen/LogicNLG.
Neural network models, especially the recent wave of massive models like BERT (Devlin et al., 2019) and GPT-2 (Radford et al., 2019), have shown the ability to generate natural language text at an astonishing level of fluency and coherence. For the generated text to fulfill its purpose, however, a crit
Medal Table from Tournament
Nation	Gold Medal	Silver Medal	Bronze Medal	Sports
Canada	3	1	2	Ice Hockey
Mexico	2	3	1	Baseball
Colombia	1	3	0	Roller Skating
Surface-level Generation
Sentence: Canada has got 3 gold medals in the tournament.
Sentence: Mexico got 3 silver medals and 1 bronze medal.
Logical Natural Language Generation
Sentence:	Canada obtained 1 more gold medal than Mexico.
Sentence:	Canada obtained the most gold medals in the game.
Figure 1: Table-to-text generation examples with and without implicit logical inference. Logical NLG requires a generation model to generate natural language statements that can be logically entailed by the facts in the table instead of simply restating certain superficial facts in natural language.
ical property that is necessary but often overlooked is fidelity, i.e., what is generated should be faithful to the underlying data, knowledge, or meaning representation. A line of recent work has started to address the surface-level fidelity issue of natural language generation (NLG) by encouraging the model to learn to reuse the verbatim of certain inputs through copy mechanism (See et al., 2017; Gu et al., 2016; Wiseman et al., 2017; Liu et al., 2018), structured attention (Liu et al., 2018), or planning and selection/entity modeling (Puduppully et al., 2019a,b). While shown to be effective, most such methods so far are primarily focused on surface-level realization and simply restate the facts in the underlying data (Figure 1).
However, humans have the ability to generalize beyond superficial facts (e.g., "Canada has got 3 gold medals.") by inferring and communicating with new statements that can be entailed from these facts (e.g., "Canada obtained the most gold medals."). We believe it is important for NLG models to be able to generalize beyond the superficiela facts given to them as well. Therefore, we propose a new task, logical NLG, where a model is tasked
7929
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7929-7942
Figure 2: When making the decision at the third step, the model needs to foresee the future tokens to ensure logical consistency. There is no back-tracking once the model makes a wrong decision like "5".
with generating natural language statements that can be logically entailed by the given data (i.e., the premises). The new task requires a model to jointly reason and generate sentences that are consistent both linguistically and logically. Since there are a variety of reasoning/inference tasks such as natural language inference (Bowman et al., 2015) and commonsense reasoning (Talmor et al., 2019), to avoid confusion, this paper is specifically focused on inferences involving symbolic operations over the given table (Pasupat and Liang, 2015).
To empower research in this direction, we collect a new corpus LOGICNLG based on the existing TabFact (Chen et al., 2019), which brings two major renovations to the existing NLG paradigm: 1) the text involves diversified types of logical inferences including math operations like max/min/sum/add, comparison operations like same/different, and counting operations like total/only. A more detailed description of logical inference is listed in the Appendix. 2) while existing datasets are often restricted to a specific domain such as weather (Liang et al., 2009), restaurant (Dusek et al., 2019), NBA (Wiseman et al., 2017), etc, LOGICNLG uses open-domain tables without prior knowledge about their schema. As such, existing methods based on surface-level copying (See et al., 2017; Gu et al., 2016; Puduppully et al., 2019a) becomes insufficient, so are the existing fidelity evaluation based on the surface-level information extraction (Wiseman et al., 2017; Rohrbach et al., 2018; Dhingra et al., 2019), which extracts surface triples in a certain pre-defined form (i.e. subj-pred-object, n-gram) and compare them with the surface content given in the knowledge.
Most neural generation models follow a monotonic generation schema from left to right with the current prediction only depending on the preceding words. Logical NLG poses unique challenges to the traditional generation scheme due to the mismatch between sequence order and logical order. As illustrated in Figure 2, the word “2” is derived from the logical inference of
diff(Silver medal of Colombia, Silver medal of Canada)  $\rightarrow 2$ . In other words, the logical order of word “2” should be after “more”, “silver”, and “Canada”, while the sequence order of “2” is before those words. Since the monotonic generation scheme is purely based on sequence order while agnostic to logical order, existing NLG models struggle to maintain the fidelity as they cannot model the logical dependency on future tokens. To alleviate such an order mismatch, an NLG model must have the capability to plan ahead for the next few steps before generation. In this context, we believe LOGICNLG to be an important testbed to study such a planing/inference ability in generation models (Ford et al., 2018; Welleck et al., 2019). In this paper, we further propose a non-monotonic coarse-to-fine generation model and show that it is able to alleviate the order mismatch problem and achieve better performance. The contribution of this work is three-fold:
i) We propose a new research problem of logical natural language generation, and provide novel metrics to approximately evaluate the logical fidelity of generation models.
ii) We justify the mismatch problem between sequence order and logical order of the traditional monotonic generation scheme in logical NLG.
iii) We conduct comprehensive experiments with state-of-the-art neural generation models under both automatic and human evaluation, which demonstrates the challenges and opportunities for future research on logic NLG.
2 Dataset and Problem Definition
Existing NLG datasets (Chen and Mooney, 2008; Dusek et al., 2019; Lebret et al., 2016; Liang et al., 2009) are mainly composed of surface-level description over the given records. Though ROTOWIRE (Wiseman et al., 2017) involves sporadic inference in the long document, and the inference is restricted to domain-specific knowledge (e.g. double-double, smash, triple-double and other NBA-related terms). Hence, we need a better testbed for studying the proposed problem.
Statistics We construct a dataset based on TabFact (Chen et al., 2019), which is a table-based fact-checking dataset with rich logical inferences in the annotated statements. Specifically, we took their positive statements (the sentences which are en
7930
Vocab	Examples	Vocab/Sent	Tables	Domain	Source	Inference	Schema
WEATHERGOV	394	22.1K	0.01	22.1K	Weather	Crawled	No	Known
WikiBIO	400K	728K	0.54	728K	Biography	Crawled	No	Limited
ROTOWIRE	11.3K	4.9K	0.72	4.9K	NBA	Annotated	Few	Known
LOGICNLG	122K	37.0K	3.31	7.3K	Open	Annotated	Rich	Unlimited
Table 1: Comparison of LOGiCNLG against existing NLG datasets in different aspects.
Figure 3: Evaluation of surface-level generation vs. logical natural language generation. It suffices to use IE-based evaluation (Wiseman et al., 2017; Rohrbach et al., 2018) to verify surface-level generation, but it causes either "empty triple" or "false negative" problems to verify logical NLG.
tailed by the knowledge in the table) collected from "complex channel" (required to annotate sentences with logical inference) as our target text. To prevent confusion with the original dataset, we name this table-to-text dataset as LOGICNLG, which contains 28,450 training, 4,260 validation and 4,305 test examples based on 7,392 open-domain tables crawled from Wikipedia. Each table has 5 different examples covering diverse types of logical inference. More detailed statistics and comparisons are listed in Table 1. LOGICNLG is distinguished from the existing datasets due to:
i) It involves very rich logical inference, every annotated sentence involves certain types of inference with minimum domain-specific knowledge. The open-domain characteristic simulates a realistic setting, where we cannot enumerate the possible inference based on the scheme, which poses great challenges to the model's generalization capability.
ii) It is mainly composed of short sentences with an average length of 11 and a simple syntactic structure, which isolates from other linguistic complexity to focus on the problem of logical inference.
The dataset contains tables with open schema crawled from diversified domains Figure 4. The major categories are sports, politics, and entertainment. The schema diversity of the tables make the rule-based system infeasible to apply. Besides, most of the tables have very rich numeral records, which provide a great testbed for logical inference.
Problem Definition Here, we formally define our proposed table-to-text generation task. The input is a table  $\mathbf{T}$  with its title denoted as a natural language sequence  $W$ . The table  $\mathbf{T} = \{T_{i,j}|i\leq R_T,j\leq C_T\}$  has  $R_{T}$  rows and  $C_T$  columns with
Figure 4: The domain distribution of LOGICNLG.
the  $T_{ij}$  being the content in the  $(i,j)$ -th cell.  $T_{ij}$  could be a word, a number, a phrase or even a natural language sentence. The annotated statement is a sentence  $Y = y_{1},y_{2},\dots ,y_{n}$ , we aim to train a neural generation model  $p(Y|\mathbf{T})$  to generate statement  $\hat{Y}$  which are both fluent and logically (numerically) supported by the given table  $\mathbf{T}$ .
3 Automatic Evaluation
In this section, we discuss the evaluation of our proposed NLG task. The fluency evaluation is simply based on the standard metrics like Perplexity (Bengio et al., 2003) and BLEU-1,2,3 (Papineni et al., 2002) based on NLTK (Bird, 2006). The most challenging problem is to evaluate the logical fidelity of the generated sentences, which is also the core problem of our paper. The existing IE-based extractive evaluation (Wiseman et al., 2017) leads to two issues as shown in Figure 3: 1) Empty Extraction: the sentence can not be formulated as (subject, predicate, object) structure, thus the IE system fail to extract triples for verification. 2) False Negative: the sentence is a logical composition (instead of surface form) of the fact from the table, the IE system cannot match it against the table. For these reasons, we test two approximate automatic metrics:
7931
Figure 5: The parsing-based and adversarial evaluation to measure model's correctness in logical reasoning.
Parsing-based Evaluation We first propose a model-based evaluation method, which aims to directly extract the meaning representation from the generated sentence and execute it against the table to verify its correctness. Our evaluation is based on weakly-supervised semantic parsing (Liang et al., 2009, 2013), the basic idea is to first link entities and predicates in the sentence, and then use linked entities to perform a breadth-first search to synthesize potential logical forms, finally, a scorer is used to re-rank these logical forms and filter out spurious ones. The logical form returns a binary value of True to indicate whether its logic is supported by the knowledge. The basic idea is shown in the upper part of Figure 5, the implementation details are in the Appendix. We pre-train the semantic parser  $f_{\gamma}$  on the training set  $(\mathbf{T}, Y) \in D_{train}$  with weakly supervised algorithm, at test time, we use it to parse a sentence  $Y$  into a set of logical forms, which is re-ranked to obtain the highest logical form  $P_{best}$ . We compute the ratio of  $P_{best}$  returning "true" on  $D_{test}$  to approximate model's fidelity.
$$
\mathrm {S P - A c c} = \underset {(\mathbf {T}, \hat {Y}) \in D _ {t e s t}} {\mathbb {E}} \mathbb {I} (P _ {b e s t} \to T r u e | P _ {b e s t} = f _ {\gamma} (\hat {Y}))
$$
where  $\mathbb{I}$  is the indicator function.
NLI-based Evaluation We then propose another model-based evaluation method to complement the parsing-based evaluation (which is sensitive to semantic variation), the basic idea follows (Krysciński et al., 2019) to evaluate the entailment score between the table and the generated sentence. The NLI model is based on TableBERT (Chen et al., 2019), which linearizes the table into textual form and uses it as the evidence for natural language inference. The model is trained with TabFact (Chen et al., 2019) dataset containing both positive/negative samples. During the evaluation, we use this NLI model to predict the entailment relationship based on the likelihood of
$p_{NLI}(Y|T)$ . Finally, we compute the ratio of "entailed" to approximate model's fidelity:
$$
\mathrm {N L I - A c c} = \underset {(\mathbf {T}, \hat {Y}) \in D _ {t e s t}} {\mathbb {E}} \underset {Y} {\mathbb {I}} (p _ {N L I} (Y | \mathbf {T}) > 0. 5)
$$
Adversarial Evaluation Adversarial evaluation (Goodfellow et al., 2014; Kannan and Vinyals, 2017) is used to study the generation model's robustness in logical reasoning. Specifically, we hire human workers from Amazon Mechanical Turk<sup>1</sup> to annotate adversarial examples for the test/validation set by simply changing minimum words to revert the logic of the sentence. Such adversarial examples preserve linguistic components like length and style except the logic-related words to specifically disentangle the generation model's reasoning skill. As drawn in the lower part of Figure 5, the original sentence modifies its word "more" into "less" as an adversarial example. There are two principles the workers need to follow to make their jobs accepted: 1) the modified words/phrases should be roughly equally frequent to balance the language prior, for example, the number "1" is better swapped with "2,3" rather than "9999" which rarely appears in the corpus. 2) the perturbation should be diverse enough to cover different aspects of logical reasoning skills. We use the generation model  $p(Y|\mathbf{T};\beta)$  to score the original sentence  $Y$  and the adversarial sentence  $Y_{adv}$ . If the confidence of the original example is higher than its adversarial counterpart, we count it as a successful defense, otherwise as a failed defense. We use the success rate to approximate model's logical reasoning capability.
$$
\operatorname {A d v} - \operatorname {A c c} = \underset {\left(\mathbf {T}, Y, Y _ {a d v}\right) \in D _ {t e s t}} {\mathbb {E}} \left[ \mathbb {I} \left(p \left(Y \mid \mathbf {T}\right) > p \left(Y _ {a d v} \mid \mathbf {T}\right)\right) \right]
$$
<sup>1</sup>https://www.mturk.com/
7932
Discussion Both types of metrics have pros and cons, the SP-Acc and NLI-Acc are two metrics unbiased as it measures the peak samples in the model's likelihood, however, both metrics are based on imperfect models and thus their evaluation scores are inaccurate. SP-Acc is more sensitive to number/calculation errors, and NLI-Acc is more sensitive to semantic errors, therefore, we report both of them to help increase the metrics' robustness. In contrast, the adversarial evaluation score is accurate in terms of reflecting the model's reasoning capability on the given samples. However, as the provided samples might not lie in the high-confidence area of the model's distribution, it is biased in reflecting the model's general reasoning capability. Though these fidelity metric models are prone to errors, in section 6, we show their consistency with human judgment, which reveals their potential to assist human evaluation.
In this section, we design comprehensive baseline models to perform logical NLG. Specifically, we consider the following two cases: non-pretrained models (LSTM/Transformer) with copy mechanism and pre-trained models (GPT-2 and BERT) with sub-word unit. We train these models with three different algorithms: Maximum Likelihood, Adversarial Training, and Reinforcement Learning.
4.1 Non-pretrained Models
Here we mainly consider two table encoding methods, namely field-infusing and field-gating. These two methods differ in their strategies to coalesce the field information into cells. After the table is represented as a sequence of vectors, a decoder based on LSTM (Hochreiter and Schmidhuber, 1997) or Transformer (Vaswani et al., 2017) is applied to generate text token by token. The two methods are depicted in the upper part of Figure 6:
Field-Infusing This strategy is inspired by Lebret et al. (2016). We first use an LSTM (Hochreiter and Schmidhuber, 1997) to encode the table field text word by word and then use the last output  $\mathbf{z_i}$  as field representation. This representation is concatenated with the embedding of row index  $\# j$  and word embedding at each cell to obtain a position-aware cell embedding  $\mathbf{e_k}$  for each word inside the cell. We stack transformers layers on top of the cell embedding to obtain the table representation as  $\mathbf{h_i} \in \mathbb{R}^D$  with  $D$  as the dimension.
Field-Gating This strategy is inspired by Liu et al. (2018). Like the previous strategy, we first use an LSTM (Hochreiter and Schmidhuber, 1997) to obtain field representation  $\mathbf{z_i}$ . The field representation is concatenated with ending distance information as the input to an additional field gate built inside the LSTM as suggested in Liu et al. (2018), such a field gate is used to control whether the current cell is already encoded. Such a mechanism can help LSTM to identify the boundary between different cells to grasp local information.
4.2 Pre-trained Models
To further enhance the fluency and resolve the out-of-vocabulary problem, we use pre-trained language models and finetune them on LOGICNLG. Specifically, we consider two models based on GPT-2 (Radford et al., 2019) and BERT (Devlin et al., 2019), respectively, and name them as GPT-TableGen and BERT-TableGen.
Table Linearization We follow previous work on linearizing knowledge base as natural language (Liu et al., 2019; Zhang et al., 2019) to propose "table linearization", which uses template to flatten the table  $\mathbf{T}$  as a document  $P_{T} = w_{1},\dots ,w_{|T|}$  fed into pre-trained language models to generate statement  $Y$ , where we use  $w_{i}$  to denote the  $i$ -th word in the generated paragraph  $P_{T}$  and  $|T|$  to denote the length of the paragraph (the word  $w_{i}$  is either a table entry or a functional word in the template). As depicted in the left bottom part of Figure 6, the original table  $\mathbf{T}$  is transformed into a paragraph by horizontally scanning each cell  $T_{11}\rightarrow T_{1,C_T}\rightarrow T_{R_T,C_T}$  in the table.
GPT-TabGen we directly feed the paragraph  $P_T$  as the input to the pre-trained GPT-2 model and generate the output sentence  $Y$ . We finetune the model on LOGICNLG by maximizing the likelihood of  $p(Y|P_T; \beta)$ , with  $\beta$  denoting the parameters of GPT-2 model (Radford et al., 2019).
BERT-TabGen 1) we encode the linearized paragraph  $P_T$  using the pre-trained BERT model into the source representation  $\mathbf{h}_1, \dots, \mathbf{h}_{|\mathbf{T}|}$ . 2) at the  $i$ -th time step, we replace all the words in the groundtruth statement  $Y$  after  $i$ -th time step by  $\langle \mathrm{MASK} \rangle$  token and use BERT to encode the partially masked  $Y^i$  as  $\mathbf{g}_1^i, \dots, \mathbf{g}_n^i$ . 3) we use an attention layer  $f_\theta$  to obtain the output hidden states  $\hat{\mathbf{g}}_1^i, \dots, \hat{\mathbf{g}}_n^i$ , where  $\hat{\mathbf{g}}_i^i$  is used to predict the word  $\hat{y}_i$ . We jointly optimize  $\beta$  of BERT and  $\theta$  to maximize
7933
Figure 6: The Non-pretrained and Pre-trained generation models, the detailed table is shown in Figure 1.
the likelihood of generating text  $Y$  conditioned on the table and the masked partial sentence. As BERT is a bidirectional model, we need to re-encode the target sentence at each step to get  $\mathbf{g}_{\mathbf{1}:\mathbf{n}}^{\mathrm{i}}$ . Therefore, the generation is finished with  $n$  passes.
Except for the standard maximum likelihood training, we also use the following training algorithms:
Adversarial Regularization To encourage the model to ground on the table rather than relying on artificial language priors (Ramakrishnan et al., 2018), we use an adversarial regularization to enhance the maximum likelihood training. Specifically, we first perform entity resolution to locate all the numbers, count, entities in the sentence and then randomly replace them with entities or numbers appearing in the table  $\mathbf{T}$ . These perturbed samples  $Y_{adv}$  are used as adversarial examples to regularize the model's behavior. Formally, we optimize  $\beta$  to maximize the objective:
$$
\operatorname *{argmax}_{\beta}\log p(Y|\mathbf{T};\beta) - \lambda \log p(Y_{adv}|\mathbf{T};\beta)
$$
where  $\lambda$  is the controlling hyper-parameter.
Reinforcement Learning The maximum likelihood training is a fluency-driven objective, which is inconsistent with the goal of logical consistency. To bridge the gap, we view the generation problem from the reinforcement learning perspective to optimize the long-term fidelity. We use the
trained semantic parser to assign reward to the policy  $p(y_{i}|y_{1:i - 1};\beta)$ . At  $i$ -th step, the generator will sample different actions  $y_{i}$  and roll-out from  $i + 1$ -th step to produce a full sequence starting from  $y_{i}$  using greedy search. The full sentence receives a binary score  $r(Y,\mathbf{T})$  from the semantic parser as reward. Formally, we optimize the objective:
$$
\operatorname * {a r g m a x} _ {\beta} \underset {y _ {i} \sim p (y _ {i} | y _ {1: i - 1})} {\mathbb {E}} \left[ \underset {y _ {i + 1: n}} {\mathbb {E}} \left[ r \left(y _ {1: n}, \mathbf {T}\right) \right] \right] \log p \left(y _ {i} | y _ {1: i - 1}; \beta\right)
$$
where we only use one trajectory to approximate the inner roll-out expectation for efficiency.
5 Coarse-to-Fine Generation
As discussed before, the baseline models follow the monotonic generation scheme and suffer from the mismatch between sequence order and logical order (Figure 2). In this section, we propose an imperfect remedy for such a situation based on the coarse-to-fine generation paradigm.