SlowGuess/ACL_2020_OCR huggingface dataset:
Metrics We measure the quality of system outputs from three perspectives: (1) word-level overlap with human references, which is a commonly used metric for text generation. We report the scores of BLEU-4 (Papineni et al., 2002), ROUGE-L (Lin, 2004), Meteor (Banerjee and Lavie, 2005) and CIDEr (Vedantam et al., 2015). (2) human evaluation. Word-level overlapping scores usually correlate rather poorly with human judgements on fluency and information accuracy (Reiter and Belz, 2009; Novikova et al., 2017a). Therefore, we passed the input data and generated text to human annotators to judge if the text is fluent by grammar (scale 1-5 as in Belz and Reiter (2006)), contains wrong fact inconsistent with input data, repeats or misses information. We report the averaged score for fluency and definite numbers for others. The human is conducted on a sampled subset from the test data. To ensure the subset covers inputs with all possible number of records ( $K \in [3,8]$  for E2E and  $K \in [1,7]$  for WebNLG), we sample 20 instances for every possible  $K$ . Finally, we obtain 120 test cases for E2E and 140 for WebNLG<sup>6</sup>. (3) Diversity of outputs. Diversity is an important concern for many real-life applications. We measure it by the number of unique unigrams and trigrams over system outputs, as done in Dušek et al. (2020).
In this section, we first show the effects of the granularity regularization we proposed, then compare model performance on two datasets and analyze the performance difference. Our model is compared against the neural attention-based pointer generator (PG) which does not explicit learn the segmentation and correspondence. To show the effects of the constrained decoding mentioned in §4, Decoding. we run our model with only the first constraint to prevent empty segments (denoted by ours in experiments), with the first two constraints to prevent repetition (denoted by ours  $(+R)$ ), and with all constraints to further reduce information missing (denoted by ours  $(+RM)$ ).
Segmentation Granularity We show the effects of the granularity regularization (§4, Segmentation
Figure 3: Average expected number of segments with varying hyperparameters. x-axis is the encoder/decoder hidden size and y-axis is the word embedding size. Upper two figures are without the granularity regularization and the bottom two are with regularization.
Granularity) in Fig 3. When varying the model size, the segmentation granularity changes much if no regularization is imposed. Intuitively if the generation module is strong enough (larger hidden size), it can accurately estimate the sentence likelihood itself without paying extra cost of switching between segments, then it tends to reduce the number of transitions. Vice versa, the number of transitions will grow if the transition module is stronger (larger embedding size). With the regularization we proposed, the granularity remains what we want regardless of the hyperparameters. We can thereby freely decide the model capacity without worrying about the difference of segmentation behavior.
Results on E2E On the E2E dataset, apart from our implementations, we also compare agianst outputs from the SLUG (Juraska et al., 2018), the overall winner of the E2E challenge (seq2seq-based), DANGNT (Nguyen and Tran, 2018), the best grammar rule based model, TUDA (Puzikov and Gurevych, 2018), the best template based model, and the autoregressive neural template model (N_TEMP) from Wiseman et al. (2018). SLUG uses a heuristic slot aligner based on a set of handcrafted rules and combine a complex pipeline of data augmentation, selection, model ensemble and reranker, while our model has a simple end-to-end learning paradigm with no special delexicalizing, training or decoding tricks. Table 2 reports the evaluated results. Seq2seq-based models are more diverse than rule-based models at the cost of higher chances of making errors. As rule-based systems are by design always faithful to the in-
The original human evaluation subset of WebNLG is randomly sampled, most of the inputs contain less than 3 records, so we opt for a new sample for a thorough evaluation.
7160
Metrics
Models	Word Overlap	Human Evaluation	Diversity
BLEU	R-L	Meteor	CIDEr	Fluent	Wrong	Repeat	Miss	Dist-1	Dist-3
SLUG	0.662	0.677	0.445	2.262	4.94	5	0	17	74	507
DANGNT	0.599	0.663	0.435	2.078	4.97	0	0	21	61	301
TUDA	0.566	0.661	0.453	1.821	4.98	0	0	10	57	143
N_TEMP	0.598	0.650	0.388	1.950	4.84	19	3	35	119	795
PG	0.638	0.677	0.449	2.123	4.91	15	1	29	133	822
OURS	0.647	0.683	0.453	2.222	4.96	0	1	15	127	870
OURS (+R)	0.645	0.681	0.452	2.218	4.95	0	0	13	133	881
OURS (+RM)	0.651	0.682	0.455	2.241	4.95	0	0	3	135	911
Table 2: Automatic and human evaluation results on E2E dataset. SLUG, DANGNT, TUDA and N_TEMP are from previous works and the other models are our own implementations.
Metrics
Models	Word Overlap	Human Evaluation	Diversity
BLEU	R-L	Meteor	CIDEr	Fluent	Wrong	Repeat	Miss	Dist-1	Dist-3
MELBOURNE	0.450	0.635	0.376	2.814	4.16	42	22	37	3167	13,744
UPF-FORGE	0.385	0.609	0.390	2.500	4.08	29	6	28	3191	12,509
PG	0.452	0.652	0.384	2.623	4.13	43	26	42	3,218	13,403
OURS	0.453	0.656	0.388	2.610	4.23	26	19	31	3,377	14,516
OURS (+R)	0.456	0.657	0.390	2.678	4.28	18	2	24	3,405	14,351
OURS (+RM)	0.461	0.654	0.398	2.639	4.26	23	4	5	3,457	14,981
Table 3: Automatic and human evaluation results on WebNLG dataset. MELBOURNE and UPFUPF-FORGE are from previous works and the other models are our own implementations.
Input: [name the mill][eattype restaurant][food english][pricerange moderate][customerring 1 out of 5][area riverside] ...
PG: the mill is a low - priced restaurant in the city centre that delivers take - away . it is located near café rouge.
Input: [name the mill][eattype restaurant][food english][pricerange moderate][customerring 1 out of 5][areariverside] ...
Ours: [the mill][restaurant][near café rouge][in riverside][serves english food][at moderate prices].[it is kid friendly and]...
Table 4: (E2E) Attention map when decoding the word "low" in the PG model and "moderate" in our model. Hallucinated contents are bolded. The PG model wrongly attended to other slots thereby "hallucinated" the content of "low-priced". Our model always attends to one single slot instead of averaging over the whole inputs, the chance of hallucination is largely reduced.
put information, they made zero wrong facts in their outputs. Most models do not have the fact repetition issue because of the relatively simple patterns in the E2E dataset. therefore, adding the (+R) constraint only improves the performance minorly. The (+RM) constraint reduces the number of information missing to 3 without hurting the fluency. All the 3 missing cases are because of the wrong alignment between the period and one data record, which can be easily fixed by defining a simple rule. We put the error analysis in appendix A. N_Temp performs worst among all seq2seq-based systems because of the restrictions we mentioned in §2. As also noted by the author, it trades-off the generation quality for interpretability and controllability. In contrast, our model, despite relying on no heuristics or complex pipelines, made zero wrong facts with the lowest information missing rate, even surpassing rule-based models. It also maintains interpretable and controllable without sacrificing the generation quality.
Results on WebNLG Table 3 reports the results evaluated on the WebNLG dataset. We also include
results from MELBOURNE, a seq2seq-based system achieving highest scores on automatic metrics in the WebNLG challenge and UPF-FORGE, a classic grammar-based system that wins in the human evaluation WebNLG contains significantly more distinct types of attributes than E2E, so the chance of making errors or repetitions increases greatly. Nevertheless, our model still performs on par on automatic metrics with superior information adequacy and output diversity. The (+R) decoding constraint becomes important since the outputs in WebNLG are much longer than those in E2E, neural network models have problems tracking the history generation beyond certain range. Models might repeat facts that have been already generated long back before. The (+R) constraint effectively reduces the repetition cases from 19 to 2. These 2 cases are intra-segment repetitions and failed to be detected since our model can only track inter-segment constraints (examples are in appendix A). The (+RM) constraint brings down the information missing cases to 5 with slightly more wrong and repeated facts compared with (+R). Forcing models
7161
Figure 4: Example generations from WebNLG. Relation types are underlined and repeated generations are bolded. Segments and corresponding records in our model are marked in the same color. By adding explicit constraints to the decoding process, repetition and missing issues can be largely reduced. (better viewed in color)
to keep generating until covering all records will inevitably increase the risk of making errors.
Discussions In summary, our models generate most diverse outputs, achieves similar or better performances in word-overlap automatic metrics while significantly reduces the information hallucination, repetition and missing problems. An example of hallucination is shown in Table 4. The standard PG model "hallucinated" the contents of "low-priced", "in the city center" and "delivers takeaway". The visualized attention maps reveal that it failed to attend properly when decoding the word "low". The decoding is driven mostly by language models instead of the contents of input data. In contrast, as we explicitly align each segment to one slot, the attention distribution of our model is concentrated on one single slot rather than averaged over the whole input, the chance of hallucinating is therefore largely reduced.
Figure 4 shows some example generations from WebNLG. Without adding the decoding constraints, PG and our model both suffer from the problem of information repetition and missing. However, the interpretability of our model enables us to easily avoid these issues by constraining the segment transition behavior. For the attention-based PG model, there exists no simple way of applying these constraints. We can also explicitly control the output structure similar to Wiseman et al. (2018), examples are shown in appendix B.
In this work, we exploit the segmental structure in data-to-text generation. The proposed model significantly alleviates the information hallucination, repetition and missing problems without sacrificing the fluency and diversity. It is end-to-end trainable, domain-independent and allows explicit control over the structure of generated text. As our model is interpretable in the correspondence between segments and input records, it can be easily combined with hand-engineered heuristics or user-specific requirements to further improve the performance.
This research was funded in part by the DFG collaborative research center SFB 1102. Ernie Chang is supported by SFB 248 Foundations of Perspicuous Software Systems (E2); Xiaoyu Shen is supported by IMPRS-CS fellowship. We sincerely thank the anonymous reviewers for their insightful comments that helped us to improve this paper.
Gabor Angeli, Percy Liang, and Dan Klein. 2010. A simple domain-independent probabilistic approach to generation. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 502-512. Association for Computational Linguistics.
7162
learning to align and translate. In International Conference on Learning Representations.
Anusha Balakrishnan, Jinfeng Rao, Kartikeya Upasani, Michael White, and Rajen Subba. 2019. Constrained decoding for neural NLG from compositional representations in task-oriented dialogue. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 831-844, Florence, Italy. Association for Computational Linguistics.
Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65-72.
Regina Barzilay and Mirella Lapata. 2005. Collective content selection for concept-to-text generation. In Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing, pages 331-338. Association for Computational Linguistics.
Anja Belz. 2008. Automatic generation of weather forecast texts using comprehensive probabilistic generation-space models. Natural Language Engineering, 14(4):431-455.
Anja Belz and Ehud Reiter. 2006. Comparing automatic and human evaluation of nlg systems. In 11th Conference of the European Chapter of the Association for Computational Linguistics.
Peter F Brown, Vincent J Della Pietra, Stephen A Della Pietra, and Robert L Mercer. 1993. The mathematics of statistical machine translation: Parameter estimation. Computational linguistics, 19(2):263-311.
Ernie Chang, David Ifeoluwa Adelani, Xiaoyu Shen, and Vera Demberg. 2020. Unsupervised pidgin text generation by pivoting english data and self-training. arXiv preprint arXiv:2003.08272.
Mingje Chen, Gerasimos Lampouras, and Andreas Vlachos. 2018. Sheffield at e2e: structured prediction approaches to end-to-end language generation. *arxiv*.
Chung-Cheng Chiu and Colin Raffel. 2018. Monotonic chunkwise attention. *ICLR*.
Emilie Colin, Claire Gardent, Yassine M'rabet, Shashi Narayan, and Laura Perez-Beltrachini. 2016. The WebNLG challenge: Generating text from DBPedia data. In Proceedings of the 9th International Natural Language Generation conference, pages 163-167, Edinburgh, UK. Association for Computational Linguistics.
Hal Daumé III and Daniel Marcu. 2005. Induction of word and phrase alignments for automatic document summarization. Computational Linguistics, 31(4):505-530.
Yuntian Deng, Anssi Kanervisto, Jeffrey Ling, and Alexander M Rush. 2017. Image-to-markup generation with coarse-to-fine attention. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 980-989. JMLR.org.
Yuntian Deng, Yoon Kim, Justin Chiu, Demi Guo, and Alexander Rush. 2018. Latent alignment and variational attention. In Advances in Neural Information Processing Systems, pages 9712-9724.
Ondrej Dusek, Jekaterina Novikova, and Verena Rieser. 2018. Findings of the e2e nlg challenge. In Proceedings of the 11th International Conference on Natural Language Generation, pages 322-328.
Ondrej Dusek, Jekaterina Novikova, and Verena Rieser. 2020. Evaluating the state-of-the-art of end-to-end natural language generation: The e2e nlg challenge. Computer Speech & Language, 59:123-156.
Jason Eisner. 2002. Parameter estimation for probabilistic finite-state transducers. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 1-8.
Jason Eisner. 2016. Inside-outside and forward-backward algorithms are just backprop (tutorial paper). In Proceedings of the Workshop on Structured Prediction for NLP, pages 1-17.
Thiago Castro Ferreira, Chris van der Lee, Emiel van Miltenburg, and Emiel Krahmer. 2019. Neural data-to-text generation: A comparison between pipeline and end-to-end architectures. EMNLP.
Kuzman Ganchev, Jennifer Gillenwater, Ben Taskar, et al. 2010. Posterior regularization for structured latent variable models. Journal of Machine Learning Research, 11(Jul):2001-2049.
Po-Sen Huang, Chong Wang, Sitao Huang, Dengyong Zhou, and Li Deng. 2018. Towards neural phrase-based machine translation. ICLR.
Juraj Juraska, Panagiotis Karagiannis, Kevin Bowden, and Marilyn Walker. 2018. A deep ensemble model with slot alignment for sequence-to-sequence natural language generation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 152-162.
Kazuya Kawakami, Chris Dyer, and Phil Blunsom. 2019. Unsupervised word discovery with segmental neural language models. ACL.
Joohyun Kim and Raymond J Mooney. 2010. Generative alignment and semantic parsing for learning from ambiguous supervision. In Proceedings of the 23rd International Conference on Computational Linguistics: Posters, pages 543-551. Association for Computational Linguistics.
7163
Yoon Kim, Carl Denton, Luong Hoang, and Alexander M Rush. 2017. Structured attention networks. ICLR.
Yoon Kim, Sam Wiseman, and Alexander M Rush. 2018. A tutorial on deep latent variable models of natural language. arXiv preprint arXiv:1812.06834.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. *ICLR*.
Rik Koncel-Kedziorski, Hannaneh Hajishirzi, and Ali Farhadi. 2014. Multi-resolution language grounding with weak supervision. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 386-396.
Ravi Kondadadi, Blake Howald, and Frank Schilder. 2013. A statistical nlg framework for aggregated planning and realization. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1406-1415.
Ioannis Konstas and Mirella Lapata. 2013. A global model for concept-to-text generation. Journal of Artificial Intelligence Research, 48:305-346.
Karen Kukich. 1983. Design of a knowledge-based report generator. In Proceedings of the 21st annual meeting on Association for Computational Linguistics, pages 145-150. Association for Computational Linguistics.
Rémi Lebret, David Grangier, and Michael Auli. 2016. Neural text generation from structured data with application to the biography domain. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1203-1213. Association for Computational Linguistics.
Percy Liang, Michael I Jordan, and Dan Klein. 2009. Learning semantic correspondences with less supervision. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 1-Volume 1, pages 91-99. Association for Computational Linguistics.
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.
Jeffrey Ling and Alexander Rush. 2017. Coarse-to-fine attention models for document summarization. In Proceedings of the Workshop on New Frontiers in Summarization, pages 33-42.
Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412-1421. Association for Computational Linguistics.
Kathleen McKeown. 1992. Text generation. Cambridge University Press.
Hongyuan Mei, TTI UChicago, Mohit Bansal, and Matthew R Walter. 2016. What to talk about and how? selective generation using lstms with coarse-to-fine alignment. In Proceedings of NAACL-HLT, pages 720-730.
Amit Moryossef, Yoav Goldberg, and Ido Dagan. 2019. Step-by-step: Separating planning from realization in neural data-to-text generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2267-2277.
Dang Tuan Nguyen and Trung Tran. 2018. Structurebased generation system for e2e nlg challenge. arxiv.
Jekaterina Novikova, Ondrej Dusek, Amanda Cercas Curry, and Verena Rieser. 2017a. Why we need new evaluation metrics for nlg. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2241-2252.
Jekaterina Novikova, Ondrej Dušek, and Verena Rieser. 2017b. The e2e dataset: New challenges for end-to-end generation. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pages 201-206.
Franz Josef Och, Christoph Tillmann, and Hermann Ney. 1999. Improved alignment models for statistical machine translation. In 1999 Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora.
Tatsuro Oya, Yashar Mehdad, Giuseppe Carenini, and Raymond Ng. 2014. A template-based abstractive meeting summarization: Leveraging summary and source text relationships. In Proceedings of the 8th International Natural Language Generation Conference (INLG), pages 45-53.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311-318. Association for Computational Linguistics.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, pages 8024-8035.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
7164
Ratish Puduppully, Li Dong, and Mirella Lapata. 2019. Data-to-text generation with content selection and planning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6908-6915.
Yevgeniy Puzikov and Iryna Gurevych. 2018. E2e nlg challenge: Neural models vs. templates. In Proceedings of the 11th International Conference on Natural Language Generation, pages 463-471.
Guanghui Qin, Jin-Ge Yao, Xuening Wang, Jinpeng Wang, and Chin-Yew Lin. 2018. Learning latent semantic annotations for grounding natural language to structured data. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3761-3771.
Lawrence R Rabiner. 1989. A tutorial on hidden markov models and selected applications in speech recognition. Proceedings of the IEEE, 77(2):257-286.
Lena Reed, Shereen Oraby, and Marilyn Walker. 2018. Can neural generators for dialogue learn sentence planning and discourse structuring? In Proceedings of the 11th International Conference on Natural Language Generation, pages 284-295.
Ehud Reiter and Anja Belz. 2009. An investigation into the validity of some metrics for automatically evaluating natural language generation systems. Computational Linguistics, 35(4):529-558.
Ehud Reiter and Robert Dale. 1997. Building applied natural language generation systems. Natural Language Engineering, 3(1):57-87.
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073-1083. Association for Computational Linguistics.
Shiv Shankar and Sunita Sarawagi. 2019. Posterior attention models for sequence to sequence learning. *ICLR*.
Xiaoyu Shen, Youssef Oualil, Clayton Greenberg, Mittul Singh, and Dietrich Klakow. 2017. Estimation of gap between current language models and human performance. Proc. Interspeech 2017, pages 553-557.
Xiaoyu Shen, Jun Suzuki, Kentaro Inui, Hui Su, Dietrich Klakow, and Satoshi Sekine. 2019a. Select and attend: Towards controllable content selection in text generation. arXiv preprint arXiv:1909.04453.
Xiaoyu Shen, Yang Zhao, Hui Su, and Dietrich Klakow. 2019b. Improving latent alignment in text summarization by generalizing the pointer generator. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3753-3764.