SlowGuess/ACL_2020_OCR huggingface dataset:
Zhenyi Wang†§, Xiaoyang Wang§, Bang An†, Dong Yu§, Changyou Chen†
†State University of New York at Buffalo, §Tencent AI Lab, Bellevue, WA
{zhenyiwa, anbang, changyou}@buffalo.edu
{shawnxywang, dyu}@tencent.com
Text generation from a knowledge base aims to translate knowledge triples to natural-language descriptions. Most existing methods ignore the faithfulness between a generated text description and the original table, leading to generated information that goes beyond the content of the table. In this paper, for the first time, we propose a novel Transformer-based generation framework to achieve the goal. The core techniques in our method to enforce faithfulness include a new table-text optimal-transport matching loss and a tableau-text embedding similarity loss based on the Transformer model. Furthermore, to evaluate faithfulness, we propose a new automatic metric specialized to the table-to-text generation problem. We also provide detailed analysis on each component of our model in our experiments. Automatic and human evaluations show that our framework can significantly outperform state-of-the-art by a large margin.
Understanding structured knowledge, e.g., information encoded in tables, and automatically generating natural-language descriptions is an important task in the area of Natural Language Generation. Table-to-text generation helps making knowledge elements and their connections in tables easier to comprehend by human. There have been a number of practical application scenarios in this field, for example, weather report generation, NBA news generation, biography generation and medical-record description generation (Liang et al., 2009; Barzilay and Lapata, 2005; Lebret et al., 2016a; Cawsey et al., 1997).
Most existing methods for table-to-text generation are based on an encoder-decoder framework (Sutskever et al., 2014; Bahdanau et al.,
Zhenyi Wang was a research intern student at Tencent AI Lab in Bellevue, WA when doing this work.
Slot type	Slot value
Name_ID	Willie Burden
Date of birth	July 21 1951
Educated at	North Carolina State University
Educated at	Ohio University
Place of birth	Raleigh, North Carolina
Date of death	December 4 2015
Place of death	Atlanta
Occupation	Football player
Member of sports team	Calgary Stampeders
Willie Burden ( July 21 1951 – December 4 2015 ) was a professional Canadian football player with the Calgary who subsequently became an academic and sports administrator. The Calgary Stampeders would be Burden 's home for eight seasons between 1974 and 1981. He died in Atlanta on December 4 2015 at the age of 64. Following an outstanding high school football career at William G. Enloe High School, Burden returned to Ohio University to serve as Assistant Athletic Director.
Figure 1: An example of table-to-text generation. This generation is unfaithful because there exists information in table not covered by generated text (marked in blue); At the same time, hallucinated information in text does not appear in table (marked in red).
2015), most of which are RNN-based Sequenceto-Sequence (Seq2Seq) models (Lebret et al., 2016b; Liu et al., 2018; Wiseman et al., 2018; Ma et al., 2019; Wang et al., 2018; Liu et al., 2019a). Though significant progress has been achieved, we advocate two key problems in existing methods. Firstly, because of the intrinsic shortage of RNN, RNN-based models are not able to capture longterm dependencies, which would lose important information reflected in a table. This drawback prevents them from being applied to larger tables, for example, a table describing a large Knowledge Base (Wang et al., 2018). Secondly, little work has focused on generating faithful text descriptions, which is defined, in this paper, as the level of matching between a generated text sequence
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1072-1086 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
and the corresponding table content. An unfaithful generation example is illustrated in Figure 1. The training objectives and evaluation metrics of existing methods encourage generating texts to be as similar as possible to reference texts. One problem with this is that the reference text often contains extra information that is not presented in the table because human beings have external knowledge beyond the input table when writing the text, or it even misses some important information in the table (Dhingra et al., 2019) due to the noise from the dataset collection process. As a result, unconstrained training with such mis-matching information usually leads to hallucinated words or phrases in generated texts, making them unfaithful to the table and thus harmful in practical uses.
In this paper, we aim to overcome the above problems to automatically generate faithful texts from tables. In other words, we aim to produce the writing that a human without any external knowledge would do given the same table data as input. In contrast to existing RNN-based models, we leverage the powerful attention-based Transformer model to capture long-term dependencies and generate more informative paragraph-level texts. To generate descriptions faithful to tables, two content-matching constraints are proposed. The first one is a latent-representation-level matching constraint encouraging the latent semantics of the whole text to be consistent with that of the whole table. The second one is an explicit entity-level matching scheme, which utilizes Optimal-Transport (OT) techniques to constrain key words of a table and the corresponding text to be as identical as possible. To evaluate the faithfulness, we also propose a new PARENT-T metric evaluating the content matching between texts and tables, based on the recently proposed PARENT (Dhingra et al., 2019) metric. We train and evaluate our model on a large-scale knowledge base dataset (Wang et al., 2018). Automatic and human evaluations both show that our method achieve the state-of-the-art performance, and can generate paragraph-level descriptions much more informative and faithful to input tables.
The task of text generation for a knowledge base is to take the structured table,  $T = \{(t_1, v_1), (t_2, v_2), (t_m, v_m)\}$ , as input, and outputs a natural-language description consisting of a
Figure 2: The architecture of our proposed model for table-to-text generation. To enhance the ability of generating multi-sentence faithful texts, our loss consists of three parts, including a maximum-likelihood loss (green), a latent matching disagreement loss (orange), and an optimal-transport loss (blue).
sequence of words  $\mathbf{y} = \{y_1, y_2, \dots, y_n\}$  that is faithful to the input table. Here,  $t_i$  denotes the slot type for the  $i^{th}$  row, and  $v_i$  denotes the slot value for the  $i^{th}$  row in a table.
Our model adopts the powerful Transformer model (Vaswani et al., 2017) to translate a table to a text sequence. Specifically, the Transformer is a Seq2Seq model, consisting of an encoder and a decoder. Our proposed encoder-to-decoder Transformer model learns to estimate the conditional probability of a text sequence from a source table input in an autoregressive way:
$$
P (\boldsymbol {y} | \boldsymbol {T}; \boldsymbol {\theta}) = \prod_ {i = 1} ^ {n} P \left(y _ {i} | \boldsymbol {y} _ {<   i}, \boldsymbol {T}; \boldsymbol {\theta}\right), \tag {1}
$$
where  $\pmb{\theta}$  is the Transformer parameters and  $\pmb{y}_{< i}$  denotes the decoded words from previous steps.
Existing models for table-to-text generation either only focus on generating text to match the reference text (Liu et al., 2018; Ma et al., 2019), or only require a generated text sequence to be able to cover the input table (Wang et al., 2018). However, as the only input information is the table, the generated text should be faithful to the input table as much as possible. Therefore, we propose two constraint losses, including a table-text disagreement constraint loss and a constrained content matching loss with optimal transport, to encourage the model to learn to match between the generated text and the input table faithfully. Figure 2 illustrates the overall architecture of our model. In summary, our model loss contains three
parts: 1) a maximum likelihood loss (green) that measures the matching between a model prediction and the reference text sequence; 2) a latent feature matching disagreement loss (orange) that measures the disagreement between a table encoding and the corresponding reference-text encoding; and 3) an optimal-transport loss (blue) matching the key words of an input table and the corresponding generated text.
2.1 Table Representation
The entities of a table simply consists of Slot Type and Slot Value pairs. To apply the Transformer model, we first linearize input tables into sequences. Slot types and slot values are separated by special tokens “<” and “>”. As an example, the table in Figure 1 is converted into a sequence:  $\{<$  Name_ID  $>$ , Willie Burden,  $<$  date of birth  $>$ , July 21 1951,  $\cdots\}$ . We note that encoding a table in this way might lose some high-order structure information presented in the original knowledge graph. However, our knowledge graph is relatively simple. According to our preliminary studies, a naive combination of feature extracted with graph neural networks (Beck et al., 2018) does not seem helpful. As a result, we only rely on the sequence representation in this paper.
2.2 The Base Objective
Our base objective comes from the standard Transformer model, which is defined as the negative log-likelihood loss  $\mathcal{L}_{mle}$  of a target sentence  $\pmb{y}$  given its input  $\pmb{T}$ , i.e.,
$$
\mathcal {L} _ {m l e} = - \log P (\boldsymbol {y} | \boldsymbol {T}; \boldsymbol {\theta}) \tag {2}
$$
with  $P(\pmb {y}|\pmb {T};\pmb {\theta})$  defined in (1).
2.3 Faithfulness Modeling with a Table-Text Disagreement Constraint Loss
One key element of our model is to enforce a generated text sequence to be consistent with (or faithful to) the table input. To achieve this, we propose to add some constraints so that a generated text sequence only contains information from the table. Our first idea is inspired by related work in machine translation (Yang et al., 2019). Specifically, we propose to constrain a table embedding to be close to the corresponding target sentence embedding. Since the embedding of a text sequence (or the table) in our model is also represented as a
sequence, we propose to match the mean embeddings of both sequences. In fact, the mean embedding has been proved to be an effective representation for the whole sequence in machine translation (Yang et al., 2019; Wang et al., 2017). Let  $\hat{V}_{\text {table }}$  and  $\hat{V}_{\text {text }}$  be the mean embeddings of a table and the target text embeddings in our Transformer-based model, respectively. A table-target sentence disagreement loss  $\mathcal{L}_{\text {disagree }}$  is then defined as
$$
\mathcal {L} _ {\text {d i s a g r e e}} = \left\| \hat {\boldsymbol {V}} _ {\text {t a b l e}} - \hat {\boldsymbol {V}} _ {\text {t e x t}} \right\| ^ {2} \tag {3}
$$
2.4 Faithfulness Modeling with Constrained Content Matching via Optimal Transport
Our second strategy is to explicitly match the key words in a table and the corresponding generated text. In our case, key words are defined as nouns, which can be easily extracted with existing tools such as NLTK (Loper and Bird, 2002). To match key words, a mis-matching loss should be defined. Such a mis-matching loss could be non-differentiable, e.g., when the loss is defined as the number of matched entities. In order to still be able to learn by gradient descent, one can adopt the policy gradient algorithm to deal with the non-differentiability. However, policy gradient is known to exhibit high variance. To overcome this issue, we instead propose to perform optimization via optimal transport (OT), inspired by the recent techniques in (Chen et al., 2019a).
Optimal-Transport Distance In the context of text generation, a generated text sequence,  $\pmb{y} = (y_{1},\dots ,y_{n})$ , can be represented as a discrete distribution  $\pmb{\mu} = \sum_{i = 1}^{n}u_{i}\delta_{y_{i}}(\cdot)$ , where  $u_{i}\geq 0$  and  $\sum_{i}u_{i} = 1$ ,  $\delta_x(\cdot)$  denotes a spike distribution located at  $x$ . Given two discrete distributions  $\pmb{\mu}$  and  $\pmb{\nu}$ , written as  $\pmb{\mu} = \sum_{i = 1}^{n}\pmb{u}_{i}\delta_{\pmb{x}_{i}}$  and  $\pmb{\nu} = \sum_{j = 1}^{m}\pmb{v}_{j}\delta_{\pmb{y}_{j}}$ , respectively, the OT distance between  $\pmb{\mu}$  and  $\pmb{\nu}$  is defined as the solution of the following maximum network-flow problem:
$$
\mathcal {L} _ {\mathrm {O T}} = \min  _ {\boldsymbol {U} \in \Pi (\boldsymbol {\mu}, \boldsymbol {\nu})} \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {m} \boldsymbol {U} _ {i j} \cdot d \left(\boldsymbol {x} _ {i}, \boldsymbol {y} _ {j}\right), \tag {4}
$$
where  $d(\mathbf{x},\mathbf{y})$  is the cost of moving  $\mathbf{x}$  to  $\mathbf{y}$  (matching  $\mathbf{x}$  and  $\mathbf{y}$ ). In this paper, we use the cosine distance between the two word-embedding vectors of  $\mathbf{x}$  and  $\mathbf{y}$ , defined as  $d(\mathbf{x},\mathbf{y}) = 1 - \frac{\mathbf{xy}}{\|\mathbf{x}\|_2\|\mathbf{y}\|_2}$ .  $\Pi (\pmb {\mu},\pmb {\nu})$  is the set of joint distributions such that the two marginal distributions equal to  $\pmb{\mu}$  and  $\pmb{\nu}$ , respectively.
Figure 3: Illustration of the OT loss, which is defined with OT distance to only match key words in both the table and the generated sentence. Left: the generated sentence not only contains extra information not presented in the table (shown as orange), but also lacks some information presented in the table (shown as red). This is unfaithful generation. The OT lost is thus high. Right: all information in the table is covered in the generated sentence, and the generated sentence does not contain extra information not presented in the table. This is faithful generation. The OT cost is thus low. This example is borrowed and modified from (Dhingra et al., 2019).
Exact minimization over  $\mathbf{U}$  in the above problem is in general computational intractable (Genevay et al., 2018). Therefore, we adopt the recently proposed Inexact Proximal point method for Optimal Transport (IPOT) (Xie et al., 2018) as an approximation. The details of the IPOT algorithm are shown in Appendix C.
Constrained Content Matching via OT To apply the OT distance to our setting, we need to first specify the atoms in the discrete distributions. Since nouns typically are more informative, we propose to match the nouns in both an input table and the decoded target sequence. We use NLTK (Loper and Bird, 2002) to extract the nouns that are then used for computing the OT loss. In this way, the computational cost can also be significantly reduced comparing to matching all words.
The OT loss can be used as a metric to measure the goodness of the match between two sequences. To illustrate the motivation of applying the OT loss to our setting, we provide an example illustrated in Figure 3, where we try to match the table with the two generated text sequences. On the left plot, the generated text sequence contains "California brand Grateful Dead", which is not presented in the input table. Similarly, and the phrases "Seattle, Washington" and "Skokie Illinois" in the table are not covered by the generated text. Consequently, the resulting OT loss will be high. By contrast, on the right plot, the table contains all information in the text, and all the phrases in the table are also covered well by the generated text, leading to a low OT loss. As a result, optimizing over the OT loss in (4) would enforce faithful matching be
tween a table and its generated text.
Optimization via OT When optimizing the OT loss with the IPOT algorithm, the gradients of the OT loss is required to be able to propagate back to the Transformer component. In other words, this requires gradients to flow back from a generated sentence. Note that a sentence is generated by sampling from a multinomial distribution, whose parameter is the Transformer decoder output represented as a logit vector  $S_{t}$  for each word in the vocabulary. This sampling process is unfortunately non-differentiable. To enable backpropagation, we follow Chen et al. (2019a) and use the Soft-argmax trick to approximate each word with the corresponding soft-max output.
To further reduce the number of parameters and improve the computational efficiency, we adopt the factorized embedding parameterization proposed recently (Lan et al., 2019). Specifically, we decompose a word embedding matrix of size  $V \times D$  into the product of two matrices of sizes  $V \times H$  and  $H \times D$ , respectively. In this way, the parameter number of the embedding matrices could be significantly reduced as long as  $H$  is to be much smaller than  $D$ .
2.5 The Final Objective
Combing all the above components, the final training loss of our model is defined as:
$$
\mathcal {L} = \mathcal {L} _ {m l e} + \lambda \mathcal {L} _ {\text {d i s a g r e e}} + \gamma \mathcal {L} _ {O T}, \tag {5}
$$
where  $\lambda$  and  $\gamma$  controls the relative importance of each component of the loss function.
2.6 Decoder with a Copy Mechanism
To enforce a generated sentence to stick to the words presented in the table as much as possible, we follow (See et al., 2017) to employ a copy mechanism when generating an output sequence. Specifically, let  $P_{\mathrm{vocab}}$  be the output of the Transformer decoder.  $P_{\mathrm{vocab}}$  is a discrete distribution over the vocabulary words and denotes the probabilities of generating the next word. The standard methods typically generate the next word by directly sampling from  $P_{\mathrm{vocab}}$ . In the copy mechanism, we instead generate the next word  $y_i$  with the following discrete distribution:
$$
P (y _ {i}) = p _ {g} P _ {\mathrm {v o c a b}} (y _ {i}) + (1 - p _ {g}) P _ {\mathrm {a t t}} (y _ {i}),
$$
where  $p_{g} = \sigma (W_{1}\pmb{h}_{i} + \pmb{b}_{1})$  is the probability of switching sampling between  $P_{\mathrm{vocab}}$  and  $P_{\mathrm{att}}$ , with learnable parameters  $(W_{1}, b_{1})$  and  $h_i$  as the hidden state from the Transformer decoder for the  $i$ -th word.  $P_{\mathrm{att}}$  is the attention weights (probability) returned from the encoder-decoder attention module in the Transformer. Specifically, when generating the current word  $y_{i}$ , the encoder-decoder attention module calculates the probability vector  $P_{\mathrm{att}}$  denoting the probabilities of attending to each word in the input table. Note that the probabilities of the words not presented in the table are set to zero.
We conduct experiments to verify the effectiveness and superiority of our proposed approach against related methods.
Our model is evaluated on the large-scale knowledge-base Wikiperson dataset released by Wang et al. (2018). It contains 250,186, 30,487, and 29,982 table-text pairs for training, validation, and testing, respectively. Compared to the WikiBio dataset used in previous studies (Lebret et al., 2016b; Liu et al., 2018; Wiseman et al., 2018; Ma et al., 2019) whose reference text only contains one-sentence descriptions, this dataset contains multiple sentences for each table to cover as many facts encoded in the input structured knowledge base as possible.
For automatic evaluation, we apply the widely used evaluation metrics including the standard BLEU-4 (Papineni et al., 2002), METEOR
Figure 4: Example input for different models
(Denkowski and Lavie, 2014) and ROUGE (Lin, 2004) scores to evaluate the generation quality. Since these metrics rely solely on the reference texts, they usually show poor correlations with human judgments when the references deviate too much from the table. To this end, we also apply the PARENT (Dhingra et al., 2019) metric that considers both the reference texts and table content in evaluations. To evaluate the faithfulness of the generated texts, we further modify the PARENT metric to measure the level of matching between generated texts and the corresponding tables. We denote this new metric as PARENT-T. Please see Appendix A for details. Note that the precision in PARENT-T corresponds to the percentage of words in a text sequence that co-occur in the table; and the recall corresponds to the percentage of words in a table that co-occur in the text.
3.3 Baseline Models
We compare our model with several strong baselines, including
- The vanilla Seq2Seq attention model (Bahdanau et al., 2015).
- The method in (Wang et al., 2018): The state-of-art model on the Wikiperson dataset.
- The method in (Liu et al., 2018): The state-of-the-art method on the WikiBio dataset.
- The pointer-generator (See et al., 2017): A Seq2Seq model with attention, copying and coverage mechanism.
Our implementation is based on OpenNMT (Klein et al., 2017). We train our models end-to-end to minimize our objective function with/without the copy mechanism. The vocabulary is limited to
BLEU	METEOR	ROUGE	PARENT	PARENT-T
(Wang et al., 2018)	16.20	19.01	40.10	51.03	54.22
Seq2Seq (Bahdanau et al., 2015)	22.24	19.50	39.49	43.41	44.55
Pointer-Generator (See et al., 2017)	19.32	19.88	40.68	49.52	52.62
Structure-Aware Seq2Seq (Liu et al., 2018)	22.76	20.27	39.32	46.47	48.47
Ours	24.56	22.37	42.40	53.06	56.10
Table 1: Comparison of our model and baseline. PARENT and PARENT-T are the average of PARENT and PARENT-T scores of all table-text pairs.
P-recall	P-precision	PT-recall	PT-precision
(Wang et al., 2018)	44.83	63.92	84.34	41.10
Seq2Seq (Bahdanau et al., 2015)	41.80	49.09	76.07	33.13
Pointer-Generator (See et al., 2017)	44.09	61.73	81.65	42.03
Structure-Aware Seq2Seq (Liu et al., 2018)	46.34	51.18	83.84	35.99
Ours	48.83	62.86	85.21	43.52
Table 2: Comparison of our model and baseline. P-recall and P-precision refer to the average of PARENT precisions and recalls of all table-text pairs. Similarly, PT-recall and PT-precision are the average of PARENT-T precisions and recalls of all table-text pairs.
Copy	EF	OT (N/W)	latent	BLEU	METEOR	ROUGE	PARENT	PARENT-T	params
✘	✘	✘	✘	24.49	22.01	40.98	48.31	49.89	98.92M
✓	✘	✘	✘	24.57	22.43	42.26	51.87	54.29	98.92M
✓	✓	✘	✘	25.07	22.38	42.37	51.76	54.36	45.94M
✓	✓	✘	✓	23.86	22.08	42.65	52.72	55.30	45.94M
✓	✓	W	✘	24.64	22.39	42.52	52.77	55.46	45.94M
✓	✓	N	✘	25.29	22.60	42.25	52.74	55.80	45.94M
✓	✓	N	✓	24.56	22.37	42.40	53.06	56.10	45.94M
Table 3: Ablation study of our model components.  $\checkmark$  means the corresponding column component is used.  $\times$  means do not use the corresponding column component. Specifically, "Copy" means using copy mechanism, "EF" means using embedding factorization, "OT" means using optimal transport constraint loss, "N" means extracting nouns from both the table and text, and "W" means using the whole table and text to compute OT. Lastly, "latent" means using latent similarity loss.
the 50,000 most common words in the training dataset. The hidden units of the multi-head component and the feed-forward layer are set to 2048. The baseline embedding size is 512. Following (Lan et al., 2019), the embedding size with embedding factorization is set to be 128. The number of heads is set to 8, and the number of Transformer blocks is 3. Beam size is set to be 5. Label smoothing is set to 0.1.
For the optimal-transport based regularizer, we first train the model without OT for about 20,000 steps, then fine tune the network with OT for about 10,000 steps. We use the Adam (Kingma and Ba, 2015) optimizer to train the models. We set the hyper-parameters of Adam optimizer accordingly, including the learning rate  $\alpha = 0.00001$ , and the two momentum parameters, batch size  $= 4096$  (tokens) and  $\beta_{2} = 0.998$ .