SlowGuess/ACL_2020_OCR huggingface dataset:
Table 2 shows the results of the automatic evaluation on the generation task. Our PLANENC achieves the best performance on BLEU and TER, while DUALENC performs best under METEOR. Both PLANENC and DUALENC significantly out
$^{6}$ https://github.com/OpenNMT/OpenNMT-py
That is why some of the numbers in our table are not exactly the same as those in the cited works.
BLEU (↑)	METEOR (↑)	TER (↓)
SEEN	UNSEEN	ALL	SEEN	UNSEEN	ALL	SEEN	UNSEEN	ALL
TILB-SMT	54.29	29.88	44.28	0.42	0.33	0.38	0.47	0.61	0.53
ADAPT	60.59	10.53	31.06	0.44	0.19	0.31	0.37	1.40	0.84
MELBOURNE	54.52	33.27	45.13	0.41	0.33	0.37	0.40	0.55	0.47
GTR-LSTM (2018)	54.00	29.20	37.10	0.37	0.28	0.31	0.45	0.60	0.55
GCN-EC (2018)	55.90	-	-	0.39	-	-	0.41	-	-
GRU (2019)	56.09	25.12	42.73	0.42	0.22	0.33	0.39	0.64	0.51
Transformer (2019)	56.28	23.04	42.41	0.42	0.21	0.32	0.39	0.63	0.50
Step-By-Step (2019a)	53.30	34.41	47.24	0.44	0.34	0.39	0.47	0.56	0.51
PLANENC	64.42	38.23	52.78	0.45	0.37	0.41	0.33	0.53	0.42
DUALENC	63.45	36.73	51.42	0.46	0.37	0.41	0.34	0.55	0.44
perform the previous state-of-the-art (bootstrapping (Koehn and Monz, 2006),  $p < 0.05$ ). For the SEEN part, while no existing published work performed better than ADAPT, our PLANENC achieves a 3.83 performance gain on BLEU. It also outperforms the single GCN encoder by 8.52 BLEU, which confirms the advantage of the planning stage for bridging the structural gap between the encoder and decoder. For the UNSEEN part, PLANENC and DUALENC improve BLEU by 3.82 and 2.32 compared with the previous state-of-the-art. While it is difficult to distinguish the performance of DUALENC and PLANENC by automatic measures, our human experiments (see Section 5.3.4) show that dual encoding generates better text compared with PLANENC.
When comparing with the pipeline methods, one difference from the data perspective is how to obtain the plans of each instance to train the planner. While Step-By-Step uses heuristic string matching to extract plans from the referenced sentences, other methods (GRU and transformer), as well as ours, use plans provided in the enriched WebNLG dataset (Castro Ferreira et al., 2018). However, Step-By-Step reported worse BLEU results on these plans.
5.3.3 Ablation Study
To further analyze what factors contribute to the performance gain, we conduct an ablation study by removing the following components:
- Copy mechanism: The text is generated without copying from the source;
- Triple planning: The input triples are shuffled before feeding into RNN, but the  $(s,p,o)$
Table 2: Generation results evaluated by BLEU, METEOR, and TER. We compare our methods with different generation systems (SMT, Sequential NMT, Graph NMT, Pipeline). Both of our methods outperform all the baselines on all three measures. We highlight both results if there is no significant difference.
Methods	BLEU (↑)	METEOR (↑)	TER (↓)
PLANENC	64.42 ± 0.17	0.45 ± 0.00	0.33 ± 0.00
-plan	57.81 ± 0.82	0.40 ± 0.00	0.40 ± 0.01
-copy	61.64 ± 0.53	0.43 ± 0.01	0.36 ± 0.01
-mention	61.49 ± 0.35	0.43 ± 0.00	0.36 ± 0.00
-delimiter	63.26 ± 0.33	0.44 ± 0.00	0.34 ± 0.00
Table 3: Results of the ablation study.
inside a triple are not shuffled.
- Entity mentions: We join the words in a node mention with underlines (e.g., Aston_Martin instead of Aston Martin).
- Plan delimiter: We concatenate the  $(s, p, o)$  without separating them with role delimiters.
We conduct the ablation study on the SEEN test-set using our PLANENC. Table 3 shows the average performance and standard deviations. Compared with PLANENC, replacing plans with a random sequence of triples hurts the BLEU score by 6.61 points, indicating that the accuracy of planning is essential for the quality of generation. Our planning also makes the model more stable to random seeds (by decreasing the standard deviation from 0.82 to 0.17). Removing the copy mechanism also decreases the BLEU score by 2.78 points. It demonstrates the effectiveness of copying words from the source triples rather than generating them from the vocabulary set. Removing the mention information, decreases the BLEU score by 2.93. It reflects two benefits of word mentions: to alleviate data sparsity and to coordinate with the copy mechanism. However, removing delimiters does not affect the BLEU much. Intuitively, we expected the delimiters to
Absolute(%)	Pairwise(%)
CVGE	FAITH	CVGE	FAITH	FLCY	ALL
MELBOURNE	83.0	75.2	-35.0	-42.5	-38.8	-68.8
STEP	96.1	89.3	5.0	-3.7	-45.0	-55.0
E2E-TRANS	85.5	78.0	-21.2	-32.5	-21.2	-46.3
GCN	79.8	76.8	-48.7	-50.0	-26.3	-67.5
PLANENC	92.3	88.2	-7.5	-12.5	-7.5	-21.2
DUALENC	94.5	91.8	-	-	-	-
Table 4: Results of human evaluation. DUALENC outperforms most of the baselines on all measures.
help the LSTM capture the boundaries and semantic roles of each node, but the ablation study does not support it. We provide an example in Table 5 to show that the LSTM indeed has trouble learning such semantic roles.
5.3.4 Human Evaluation
Automatic measures are based on lexical similarities and are not good measures of text quality in general. We therefore further conduct a human evaluation on Amazon Mechanical Turk to better access the quality of the generated texts. We evaluate the results for MELBOURNE, Step-By-Step, Transformer, GCN, as well as our PLANENC and DUALENC. We randomly select 80 test instances (440 triples in total) with the size of tripleset between 4 to 7, since they are more challenging than those with fewer triples. Then we evaluate the generation quality of each system with the following three measures:
- Coverage: the percentage of triples that are covered by the generated text (all  $< s, p, o >$  values in the triples are realized);
- Faithfulness: the percentage of triples that are faithfully described by the text (the text correctly expresses the predicate and also the subject and object as its arguments. No substitutions or hallucinations);
- Fluency: a measure of the fluency or naturalness of the generated text.
For coverage and faithfulness, workers are asked to check each triple of an instance, and judge whether the triple is covered and faithfully described by the generated text. For fluency, we ask another group of workers to compare between two outputs of the same instance and identify which one is more fluent. Table 5 shows examples where these qualities are compromised.
In Table 4, we report the absolute scores of
coverage and faithfulness, which range from 0 to  $100\%$ . We also provide pairwise scores of all three measures by comparing the outputs of DUALENC with each of the other five systems. We report the percentage of instances that were judged to be worse/better/same than those of DUALENC, yielding a score ranging from  $-100\%$  (unanimously worse) to  $100\%$  (unanimously better). For example, MELBOURNE performs better/worse/same than DUALENC for  $10\% /45\% /45\%$  of the instances, yielding a pairwise score as  $10\% -45\% = -0.35\%$ . We also report an overall pairwise score combining all three measures. For each instance, the overall score of one output is higher than the other iff it outperforms the other on at least one of the three measures and has a better or equal vote on the other two.
Our PLANENC and DUALENC outperform most of the baselines on all of the measures by a large margin (approximate randomization,  $p < 0.05$ ), which is consistent with the automatic results. The only exception is Step-By-Step, which has high Coverage and Faithfulness (not significant). It first separates the input triples into smaller subsets and then realizes them separately. This greatly reduces the difficulty of long-term generation but at the expense of Fluency (worst among all the baselines). GCN does not perform well on Coverage, which demonstrates that the structural gap between encoding and decoding indeed makes generation more difficult. However, it has the smallest difference between Coverage and Faithfulness among all the baselines, indicating that the fidelity of generation can benefit from the encoding of graph-level structural information. By combining GCN and PLANENC, our DUALENC incorporates the advantages of both encoders while ameliorating their weaknesses, and therefore achieves the best OVERALL performance on human evaluation.
Table 5 shows examples of generated texts by various systems for an input of six triples. Colored fonts represent missing, unfaithful, and unfluent information. For example, PLANENC misses "Buzz Aldrin" and also wrongly expresses the subject of "retirement" as "Frank Borman", indicating that LSTM is less powerful at capturing the semantic roles of entities. This disadvantage can be well complemented by GCN, which is designed to capture the graph structure and the relations between entities. Hence, by incorporating information from
Tripleset	(William Anders | birthPlace | British Hong Kong), (William Anders | was a crew member of | Apollo 8), (Apollo 8 | crewMembers | Frank Borman), (Apollo 8 | backup pilot | Buzz Aldrin), (Apollo 8 | operator | NASA), (William Anders | dateOfRetirement | 1969-09-01)
MELBOURNE	william anders (born in british hong kong) was a crew member of apollo 8's apollo 8 8 mission along with buzz aldrin as backup pilot and buzz aldrin on 1969-09-01 . [Frank Borman, NASA]
Step-by-Step	william anders was a crew member of apollo 8 operated by nasa. apollo 8's backup pilot was buzz aldrin and frank borman. william anders was born in british hong kong. william anders retired on september 01st, 1969.
PLANENC	william anders was born in british hong kong and was a crew member of nasa's apollo 8. frank borman was a crew members of apollo 8 and he retired on september 1st, 1969 . [Buzz Aldrin]
DUALENC	william anders was born in british hong kong and served as a crew member of nasa's apollo 8 along with frank borman and backup pilot buzz aldrin. he retired on september 1st, 1969 .
Reference	william anders was born in british hong kong and served as a crew member on apollo 8 along with frank borman. nasa operated apollo 8, where buzz aldrin was a back up pilot. anders retired on sept 1, 1969 .
Table 5: Sample texts generated by our methods and baselines, compared with a human-provided reference. We highlight in different color the [missing], unfaithful, and unfluent parts of each text. Only the results of our DUALENC correctly mention all the input triples.
both GCN and LSTM, DUALENC correctly expresses the subject argument of "retirement".
This paper proposes DUALENC, a dual encoding method to bridge the structural gap between encoder and decoder for data-to-text generation. We use GCN encoders to capture the structural information of the data, which is essential for accurate planning and faithful generation. We also introduce an intermediate content planning stage to serialize the data and then encode it with an LSTM network. This serialized plan is more compatible with the output sequence, making the information alignment between the input and output easier. Experiments on WebNLG dataset demonstrate the effectiveness of our planner and generator by outperforming the previous state-of-the-art by a large margin. Future work will validate the effectiveness of this method on more varied data-to-text generation tasks.
Daniel Beck, Gholamreza Haffari, and Trevor Cohn. 2018. Graph-to-sequence learning using gated graph neural networks. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 273-283.
Arianna Bisazza and Marcello Federico. 2016. A survey of word reordering in statistical machine translation: Computational models and language phenomena. Computational Linguistics, 42(2):163-205.
Thiago Castro Ferreira, Diego Moussallem, Sander Wubben, and Emiel Krahmer. 2018. Enriching the
webnlg corpus. In Proceedings of the 11th International Conference on Natural Language Generation, INLG'18, Tilburg, The Netherlands. Association for Computational Linguistics.
Nancy Chinchor. 1992. The statistical significance of the muc-4 results. In Proceedings of the 4th conference on Message understanding, pages 30-50. Association for Computational Linguistics.
Emilie Colin, Claire Gardent, Yassine M'rabet, Shashi Narayan, and Laura Perez-Beltrachini. 2016. The webnlg challenge: Generating text from dbpedia data. In Proceedings of the 9th International Natural Language Generation conference, pages 163-167.
Marco Damonte and Shay B Cohen. 2019. Structural neural encoders for amr-to-text generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3649-3658.
Thiago Castro Ferreira, Chris van der Lee, Emiel van Miltenburg, and Emiel Krahmer. 2019. Neural data-to-text generation: A comparison between pipeline and end-to-end architectures. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 552-562.
Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017. The webnlg challenge: Generating text from rdf data. In Proceedings of the 10th International Conference on Natural Language Generation, pages 124-133.
Zhijiang Guo, Yan Zhang, Zhiyang Teng, and Wei Lu. 2019. Densely connected graph convolutional networks for graph-to-sequence learning. Transactions of the Association for Computational Linguistics, 7:297-312.
Shizhu He, Cao Liu, Kang Liu, and Jun Zhao. 2017. Generating natural answers by incorporating copying and retrieving mechanisms in sequence-to-sequence learning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 199-208.
Xinyu Hua and Lu Wang. 2019. Sentence-level content planning and style specification for neural text generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 591-602.
Juraj Juraska, Panagiotis Karagiannis, Kevin Bowden, and Marilyn Walker. 2018. A deep ensemble model with slot alignment for sequence-to-sequence natural language generation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 152-162.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Thomas N. Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulouse, France, April 24-26, 2017, Conference Track Proceedings.
Philipp Koehn and Christof Monz. 2006. Manual and automatic evaluation of machine translation between european languages. In Proceedings on the Workshop on Statistical Machine Translation, pages 102-121.
Rémi Lebret, David Grangier, and Michael Auli. 2016. Neural text generation from structured data with application to the biography domain. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1203-1213.
Wei Li, Jingjing Xu, Yancheng He, Shengli Yan, Yunfang Wu, et al. 2019. Coherent comment generation for chinese articles with a graph-to-sequence model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4843-4852.
Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard S. Zemel. 2016. Gated graph sequence neural networks. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings.
Shuman Liu, Hongshen Chen, Zhaochun Ren, Yang Feng, Qun Liu, and Dawei Yin. 2018. Knowledge
diffusion for neural dialogue generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1489-1498.
Diego Marcheggiani and Laura Perez-Beltrachini. 2018. Deep graph convolutional encoders for structured data to text generation. In Proceedings of the 11th International Conference on Natural Language Generation, pages 1-9.
Seungwhan Moon, Pararth Shah, Anuj Kumar, and Rajen Subba. 2019. Opendialkg: Explainable conversational reasoning with attention-based walks over knowledge graphs. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 845-854.
Amit Moryossef, Yoav Goldberg, and Ido Dagan. 2019a. step-by-step: Separating planning from realization in neural data-to-text generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2267-2277.
Amit Moryossef, Yoav Goldberg, and Ido Dagan. 2019b. improving quality and efficiency in plan-based neural data-to-text generation. In Proceedings of the 12th International Conference on Natural Language Generation, pages 377-382.
Vinod Nair and Geoffrey E Hinton. 2010. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10), pages 807-814.
Eric W Noreen. 1989. Computer-intensive methods for testing hypotheses. Wiley New York.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311-318. Association for Computational Linguistics.
Ratish Puduppully, Li Dong, and Mirella Lapata. 2019. Data-to-text generation with content selection and planning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6908-6915.
Lena Reed, Shereen Oraby, and Marilyn Walker. 2018. Can neural generators for dialogue learn sentence planning and discourse structuring? INLG 2018, page 284.
Ehud Reiter and Robert Dale. 2000. Building natural language generation systems. Cambridge university press.
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. 2009. The graph neural network model. IEEE Transactions on Neural Networks, 20(1):61-80.
2490
Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. 2018. Modeling relational data with graph convolutional networks. In European Semantic Web Conference, pages 593-607. Springer.
Linfeng Song, Yue Zhang, Zhiguo Wang, and Daniel Gildea. 2018. A graph-to-sequence model for amr-to-text generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1616-1626.
Amanda Stent, Rashmi Prassad, and Marilyn Walker. 2004. Trainable sentence planning for complex information presentations in spoken dialog systems. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04), pages 79-86.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104-3112.
Bayu Distiawan Trisedya, Jianzhong Qi, Rui Zhang, and Wei Wang. 2018. Gtr-lstm: A triple encoder for sentence generation from rdf data. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1627-1637.
Petar Velicković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2018. Graph Attention Networks. International Conference on Learning Representations. Accepted as poster.
Lili Yao, Nanyun Peng, Ralph Weischedel, Kevin Knight, Dongyan Zhao, and Rui Yan. 2019. Plan-and-write: Towards better automatic storytelling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7378-7385.