SlowGuess/ACL_2020_OCR huggingface dataset:
Table 1 and 2 show the experiment results in terms of different evaluation metrics compared with different baselines. "Ours" means our proposed model with components of copy mechanism, embedding factorization, OT-matching with nouns, and latent similarity loss<sup>1</sup>. We can see that our model outperforms existing models in all of the automatic evaluation scores, indicating high quality of the generated texts. The superiority of the PARENT-T scores (in terms of precision and recall) indicates that the generated text from our model is more faithful than others. Example out
The result of the method by (Wang et al., 2018) is different from the score reported in their paper, as we use their publicly released code https://github.com/EagleW/Describing_a_Knowledge_Base and data that is three times larger than the original 106,216 table-text pair data used in the paper. We have confirmed the correctness of our results with the author.
Precision	Recall	F-1 measure	Fluency	Grammar
(Wang et al., 2018)	76.3	62.1	68.02	2.98	3.06
Seq2Seq (Bahdanau et al., 2015)	70.3	60.8	66.16	2.86	2.88
Pointer-Generator (See et al., 2017)	76.6	61.5	67.95	3.03	3.02
Structure-Aware Seq2Seq (Liu et al., 2018)	75.2	61.7	67.69	2.92	2.83
Ours	79.8	65.3	71.56	3.01	3.10
Table 4: Human Evaluation of various aspects of generated text.
Model	Miss	Generated texts
(Wang et al., 2018)	9	William Edward Ayrton Fellow of the Royal Society (14 September 1847 – 8 November 1908) was a British Physicist. Brompton Cemetery he was born in London the son of Sir Thomas and his wife Mary (née Fleming). he was educated at University College School and University College London.
Pointer generator	2, 9	William Edward Ayrton-Gould Fellow of the Royal Society (14 September 1847 – 8 November 1908) was an English Physicist who was born in London and was educated at Brompton College and University College London. he died in London on 8 November 1908. William was elected a Fellow of the Royal Society in 1902.
Seq2Seq	1, 2, 3, 9	William Edward Sandys Fellow of the Royal Society (14 September 1847 – 8 November 1908) was a British Physicist. he was educated at the University College London and the University College London. he was a Fellow of the Royal Society and a Fellow of the Royal Society.
Structure-Aware	1, 2, 9	William Edward Keeler Fellow of the Royal Society (14 September 1847 – 8 November 1908) was a British Physicist and Physicist. he was elected a Fellow of the Royal Society in 1889 and was a member of the Royal Society of London and the Royal Society of London and the Royal Society of London. he was educated at the University College London and at the University College London where he was a pupil of the chemist William.
Ours	None	William Edward Ayrton Fellow of the Royal Society (14 September 1847 – 8 November 1908) was an English Physicist. William was born in London and educated at University College London. he is buried in Brompton Cemetery London. he was elected a Fellow of the Royal Society in 1901. he was the father of Barbara Ayrton-Gould.
Table 5: Example outputs from different methods with an input table shown in Figure 4. The blue color indicates the corresponding row appears in the input table, but not in the output generation text. The red color indicates that these entities appear in the text but do not appear in the input table.
puts from different models are shown in Table 5 with an input table shown in Figure 4. In this example, our model covers all the entities in the input, while all other models miss some entities. Furthermore, other models hallucinate some information that does not appear in the input, while our model generates almost no extra information other than that in the input. These results indicate the faithfulness of our model. More examples are shown in Appendix E.
We also conduct extensive ablation studies to better understand each component of our model, including the copy mechanism, embedding factorization, optimal transport constraint loss, and latent similarity loss. Table 3 shows the results in different evaluation metrics.
Effect of copy mechanism The first and second rows in Table 3 demonstrate the impacts of the copy mechanism. It is observed that with the copy mechanism, one can significantly improve the performance in all of the automatic metrics, especially on the faithfulness reflected by the PARENT-T score.
Effect of embedding factorization We compare our model with the one without embedding factorization. The comparisons are shown in the second and third rows of Table 3. We can see that with embedding factorization, around half of the parameters can be reduced, while comparable performance can still be maintained.
Effect of table-text embedding similarity loss We also test the model by removing the table-text embedding similarity loss component. The third
and fourth rows in Table 3 summarize the results. With the table-text embedding similarity loss, the BLEU and METEOR scores drop a little, but the PARENT and PARENT-T scores improve over the model without the loss. This is reasonable because the loss aims at improving faithfulness of generated texts, reflected by the PARENT-T score.
Effect of the OT constraint loss We further compare the performance of the model (a) without using OT loss, (b) with using the whole table and text to compute OT, and (c) with using the extracted nouns from both table and text to compute OT. Results are presented in the third, fifth, and sixth rows of Table 3, respectively. The model with the OT loss improve performance on almost all scores, especially on the PARENT-T score. Furthermore, with only using the nouns to compute the OT loss, one can obtain even better results. These results demonstrate the effectiveness of the proposed OT loss on enforcing the model to be faithful to the original table.
3.7 Human Evaluation
Following (Wang et al., 2018; Tian et al., 2019), we conduct extensive human evaluation on the generated descriptions and compare the results to the state-of-the-art methods. We design our evaluation criteria based on (Wang et al., 2018; Tian et al., 2019), but our criteria differs from (Tian et al., 2019) in several aspects. Specifically, for each group of generated texts, we ask the human raters to evaluate the grammar, fluency, and faithfulness. The human evaluation metrics of faithfulness is defined in terms of precision, recall and F1-score with respect to the reconstructed Knowledge-base table from a generated text sequence. To ensure accurate human evaluation, the raters are trained with word instructions and text examples of the grading standard beforehand. During evaluation, we randomly sample 100 examples from the predictions of each model on the Wikiperson test set, and provide these examples to the raters for blind testing. More details about the human evaluation are provided in the Appendix B. The human evaluation results in Table 4 clearly show the superiority of our proposed method.
Table-to-text generation has been widely studied, and Seq2Seq models have achieved promising performance. (Lebret et al., 2016b; Liu et al., 2018;
Wiseman et al., 2018; Ma et al., 2019; Wang et al., 2018; Liu et al., 2019a). For Transformer-based methods, the Seq2Seq Transformer is used by Ma et al. (2019) for table-to-text generation in low-resource scenario. Thus, instead of encoding an entire table as in our approach, only the predicted key facts are encoded in (Ma et al., 2019). Extended transformer has been applied to game summary (Gong et al., 2019) and E2E NLG tasks (Gehrmann et al., 2018). However, their goals focus on matching the reference text instead of being faithful to the input.
Another line of work attempts to use external knowledge to improve the quality of generated text (Chen et al., 2019b). These methods allow generation from an expanded external knowledge base that may contain information not relevant to the input table. Comparatively, our setting requires the generated text to be faithful to the input table. Nie et al. (2018) further study fidelity-data-to-text generation, where several executable symbolic operations are applied to guide text generation. Both models do not consider the matching between the input and generated output.
Regarding datasets, most previous methods are trained and evaluated on much simpler datasets like WikiBio (Lebret et al., 2016b) that contains only one sentence as a reference description. Instead, we focus on the more complicated structured knowledge base dataset (Wang et al., 2018) that aims to generate multi-sentence texts. Wang et al. (2018) propose a model based on the pointer network that can copy facts directly from the input knowledge base. Our model uses a similar strategy but obtains much better performance.
In terms of faithfulness, one related parallel work is Tian et al. (2019). However, our method is completely different from theirs. Specifically, Tian et al. (2019) develop a confidence oriented decoder that assigns a confidence score to each target position to reduce the unfaithful information in the generated text. Comparatively, our method enforces faithfulness by including the proposed table-text optimal-transport matching loss and table-text embedding similarity loss. Moreover, the faithfulness of Tian et al. (2019) only requires generated texts to be supported by either a table or the reference; whereas ours constrains generated texts to be faithful only to the table.
Other related works are (Perez-Beltrachini and Lapata, 2018; Liu et al., 2019b). For (Perez-
Beltrachini and Lapata, 2018), the content selection mechanism training with multi-task learning and reinforcement learning is proposed. For (Liu et al., 2019b), they propose force attention and reinforcement learning based method. Their learning methods are completely different from our method that simultaneously incorporates optimal-transport matching loss and embedding similarity loss. Moreover, the REINFORCE algorithm (Williams, 1992) and policy gradient method used in (Perez-Beltrachini and Lapata, 2018; Liu et al., 2019b) exhibits high variance when training the model.
Finally, the content-matching constraints between text and table is inspired by ideas in machine translation (Yang et al., 2019) and Seq2Seq models (Chen et al., 2019a).
In this paper, we propose a novel Transformer-based table-to-text generation framework to address the faithful text-generation problem. To enforce faithful generation, we propose a new tablet-text optimal-transport matching loss and a tablet-text embedding similarity loss. To evaluate the faithfulness of the generated texts, we further propose a new automatic evaluation metric specialized to the table-to-text generation problem. Extensive experiments are conducted to verify the proposed method. Both automatic and human evaluations show that our framework can significantly outperform the state-of-the-art methods.
We sincerely thank all the reviewers for providing valuable feedback. We thank Linfeng Song, Dian Yu, Wei-yun Ma, and Ruiyi Zhang for the helpful discussions.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the International Conference on Learning Representations.
Regina Barzilay and Mirella Lapata. 2005. Collective content selection for concept-to-text generation. In HLT/EMNLP 2005, Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference, 6-8 October 2005, Vancouver, British Columbia, Canada.
Daniel Beck, Gholamreza Haffari, and Trevor Cohn. 2018. Graph-to-sequence learning using gated graph neural networks. In Proceedings of the Annual Meeting of the Association for Computational Linguistics.
Alison Cawsey, Bonnie L. Webber, and Ray B. Jones. 1997. Brief review: Natural language generation in health care. JAMIA, 4(6):473-482.
Liqun Chen, Yizhe Zhang, Ruiyi Zhang, Chenyang Tao, Zhe Gan, Haichao Zhang, Bai Li, Dinghan Shen, Changyou Chen, and Lawrence Carin. 2019a. Improving sequence-to-sequence learning via optimal transport. In Proceedings of the International Conference on Learning Representations.
Shuang Chen, Jinpeng Wang, Xiaocheng Feng, Feng Jiang, Bing Qin, and Chin-Yew Lin. 2019b. Enhancing neural data-to-text generation models with external background knowledge. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing.
Michael Denkowski and Alon Lavie. 2014. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the Ninth Workshop on Statistical Machine Translation.
Bhuwan Dhingra, Manaal Faruqui, Ankur Parikh, Ming-Wei Chang, Dipanjan Das, and William W. Cohen. 2019. Handling divergent reference texts when evaluating table-to-text generation. In Proceedings of 57th Annual Meeting of the Association for Computational Linguistics.
Sebastian Gehrmann, Falcon Z. Dai, Henry Elder, and Alexander M. Rush. 2018. End-to-end content and plan selection for data-to-text generation. In Proceedings of International Conference on Natural Language Generation.
Aude Geneva, Gabriel Peyre, and Marco Cuturi. 2018. Learning generative models with sinkhorn divergences. In Proceedings of the International Conference on Artificial Intelligence and Statistics.
Li Gong, Josep Crego, and Jean Senellart. 2019. Enhanced transformer model for data-to-text generation. In Proceedings of the 3rd Workshop on Neural Generation and Translation (EMNLP).
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the International Conference for Learning Representations.
Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M. Rush. 2017. OpenNMT: Open-source toolkit for neural machine translation. In Proceedings of Annual Meeting of the Association for Computational Linguistics.
Harold W. Kuhn. 1955. The hungarian method for the assignment problem. In *Naval research logistics quarterly*.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. In https://arxiv.org/abs/1909.11942.
Rémi Lebret, David Grangier, and Michael Auli. 2016a. Neural text generation from structured data with application to the biography domain. In Proceedings of the Conference on Empirical Methods in Natural Language Processing.
Rémi Lebret, David Grangier, and Michael Auli. 2016b. Neural text generation from structured data with application to the biography domain. In Proceedings of the Conference on Empirical Methods in Natural Language Processing.
Percy Liang, Michael I. Jordan, and Dan Klein. 2009. Learning semantic correspondences with less supervision. In Proceedings of the Annual Meeting of the Association for Computational Linguistics and International Joint Conference on Natural Language Processing of the AFNLP.
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Proceedings of Text Summarization Branches Out.
Tianyu Liu, Fuli Luo, Qiaolin Xia, Shuming Ma, Baobao Chang, and Zhifang Sui. 2019a. Hierarchical encoder with auxiliary supervision for neural table-to-text generation: Learning better representation for tables. In Proceedings of the AAAI Conference on Artificial Intelligence.
Tianyu Liu, Fuli Luo, Pengcheng Yang, Wei Wu, Baobao Chang, and Zhifang Sui. 2019b. Towards comprehensive description generation from factual attribute-value tables. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.
Tianyu Liu, Kexiang Wang, Lei Sha, Baobao Chang, and Zhifang Sui. 2018. Table-to-text generation by structure-aware seq2seq learning. In Proceedings of the AAAI Conference on Artificial Intelligence.
Edward Loper and Steven Bird. 2002. Nltk: The natural language toolkit. In https://arxiv.org/abs/cs/0205028.
Shuming Ma, Pengcheng Yang, Tianyu Liu, Peng Li, Jie Zhou, and Xu Sun. 2019. Key fact as pivot: A two-stage model for low resource table-to-text generation. In https://arxiv.org/pdf/1908.03067.pdf.
Feng Nie, Jinpeng Wang, Jin-Ge Yao, Rong Pan, and Chin-Yew Lin. 2018. Operation-guided neural networks for high fidelity data-to-text generation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of
the Annual Meeting of the Association for Computational Linguistics.
Laura Perez-Beltrachini and Mirella Lapata. 2018. Bootstrapping generators from noisy data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In https://arxiv.org/abs/1704.04368.
Linfeng Song, Yue Zhang, Zhiguo Wang, and Daniel Gildea. 2018. A graph-to-sequence model for amr-to-text generation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics.
Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems.
Ran Tian, Shashi Narayan, Thibault Sellam, and Ankur P. Parikh. 2019. Sticking to the facts: Confident decoding for faithful data-to-text generation. In https://arxiv.org/abs/1910.08684.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems.
Qingyun Wang, Xiaoman Pan, Lifu Huang, Boliang Zhang, Zhiying Jiang, Heng Ji, and Kevin Knight. 2018. Describing a knowledge base. In International Conference on Natural Language Generation.
Rui Wang, Andrew Finch, Masao Utiyama, and Eichiro Sumita. 2017. Sentence embedding for neural machine translation domain adaptation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics.
Ronald J. Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. In Machine Learning.
Sam Wiseman, Stuart M. Shieber, and Alexander M. Rush. 2018. Learning neural templates for text generation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing.
Yujia Xie, Xiangfeng Wang, Ruijia Wang, and Hongyuan Zha. 2018. A fast proximal point method for computing exact wasserstein distance. In https://arxiv.org/pdf/1802.04307.pdf.
Mingming Yang, Rui Wang, Kehai Chen, Masao Utiyama, Eiichiro Sumita, Min Zhang, and Tiejun Zhao. 2019. Sentence-level agreement for neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.
A PARENT-T Metric
PARENT-T evaluates each instance  $(T^i, G^i)$  separately, by computing the precision and recall of generated text  $G^i$  against table  $T^i$ . In other words, PARENT-T is a table-focused version of PARENT (Dhingra et al., 2019).
When computing precision, we want to check what fraction of the n-grams in  $G_{n}^{i}$  are correct. We consider an n-gram  $g$  to be correct if it has a high probability of being entailed by the table. We use the word overlap model for entailment probability  $w(g)$ . The precision score  $E_{p}$  for one instance is computed as follows:
$$
w (g) = \frac {\sum_ {j = 1} ^ {n} \mathbb {1} \left(g _ {j} \in \bar {T} ^ {i}\right)}{n} \tag {6}
$$
$$
E _ {p} ^ {n} = \frac {\sum_ {g \in G _ {n} ^ {i}} w (g) \# _ {G _ {n} ^ {i}} (g)}{\sum_ {g \in G _ {n} ^ {i}} \# _ {G _ {n} ^ {i}} (g)} \tag {7}
$$
$$
E _ {p} = \exp \left(\sum_ {n = 1} ^ {4} \frac {1}{4} \log E _ {p} ^ {n}\right) \tag {8}
$$
where  $\bar{T}^i$  denotes all the lexical items present in the table  $T^i$ ,  $n$  is the length of  $g$ , and  $g_j$  is the  $j$ th token in  $g$ .  $w(g)$  is the entailment probability, and  $E_p^n$  is the entailed precision score for n-grams of order  $n$ .  $\#_{G_n^i}(g)$  denotes the count of n-gram  $g$  in  $G_n^i$ . The precision score  $E_p$  is a combination of n-gram orders 1-4 using a geometric average.
For recall, we only compute it against table to ensure that texts that mention more information from the table get higher scores.  $E_r(T^i)$  is computed in the same way as in Dhingra et al. (2019):
$$
E _ {r} = E _ {r} \left(T ^ {i}\right) = \frac {1}{K} \sum_ {k = 1} ^ {K} \frac {1}{\left| \bar {r} _ {k} \right|} L C S \left(\bar {r} _ {k}, G ^ {i}\right) \tag {9}
$$
where a table is a set of records  $T^i = \{r_k\}_{k=1}^K$ ,  $\bar{r}_k$  denotes the value string of record  $r_k$ , and  $LCS(x,y)$  is the length of the longest common subsequence between  $x$  and  $y$ . Higher values of  $E_r(T^i)$  denote that more records are likely to be mentioned in  $G^i$ .
Thus, the PARENT-T score (i.e. F score) for one instance is:
$$
\mathrm {P A R E N T - T} = \frac {2 E _ {p} E _ {r}}{E _ {p} + E _ {r}} \tag {10}
$$
The system-level PARENT-T score for a model  $M$  is the average of instance-level PARENT-T scores across the evaluation set.
B Details of Human Evaluation
The following are the details for instructing our human evaluation raters how to rate each generated sentence:
We only provide the input table and the generated text for the raters. There are 20 well-trained raters participating in the evaluation.
Fluency :
4: The sentence meaning is clear and flow naturally and smoothly.
3: The sentence meaning is clear, but there are a few interruptions.
2: The sentence does not flow smoothly but people can understand its meaning.
1: The sentence is not fluent at all and people cannot understand its meaning.
Grammar :
4: There are no grammar errors.
3: There are a few grammar errors, but sentence meaning is clear.
2: There are some grammar errors, but not influencing its meaning.
1: There are many grammar errors. People cannot understand the sentence meaning.
Faithfulness A sentence is faithful if it contains only information supported by the table. It should not contain additional information other than the information provided by the table or inferred from the table. Also, the generated sentence should cover as much information in the given table as possible. The raters first manually extract entities from the generated sentences and then calculate the precision as the percentage of entities in the generated text also appear in the table; calculate the recall as the percentage of entities in the table also appear in the generated text. For each tablet-text pair, its F-1 score is then calculated according to the precision and recall.
C IPOT algorithm
Given a pair of table and its corresponding text description, we can obtain table words embedding as  $S = \{x_{i}\}_{i=1}^{i=n}$ , and the model output for sentence words embedding as  $S' = \{y_{j}\}_{j=1}^{j=m}$ . The cost matrix  $C$  is then computed as in Section 2.4. Both  $S$  and  $S'$  are used as inputs to the IPOT algorithm in Algorithm 1 to obtain the OT-matching distance.
D Details of Optimal Transport Loss
Figure 5 illustrates three matching cases from top to bottom, namely hard matching, soft bipartite matching, and optimal transport matching. The hard matching stands for exactly matching words between the table and the target sequences. This operation is non-differentiable. The soft bipartite matching, on the other hand, supposes the similarity between the word embedding  $\mathbf{v}_{i_k}$  and  $\mathbf{v}_{j_k}'$  is  $d(\mathbf{v}_{i_k},\mathbf{v}_{j_k}')$ , and finds the matching such that  $\mathcal{L} = \sum_{k}d(\mathbf{v}_{i_k},\mathbf{v}_{j_k}')$  is minimized. This minimization can be solved exactly by the Hungarian algorithm (Kuhn, 1955). But, its objective is still non-differentiable. Our proposed optimal transport matching can be viewed as the relaxed problem of the soft bipartite matching by computing the distance between the distribution over the input table and the decoded text sentence. This distance in optimal transport matching is differentiable.
E More generation examples