SlowGuess/NAACL_2022_OCR huggingface dataset:
We compare our S-OSC model with other state-of-the-art methods on WebNLG and DART. Results on WebNLG are shown in Table 1. It can be observed from Table 1 that our S-OSC model outperforms all the previous methods in three evaluation metrics, i.e., B-4, R-L and Chrf++, except for the CIDEr value being in the second place compared to the best performing CIDEr result reported by the model in Li et al. (Li et al., 2021). Note that our S-OSC model contains one learning-based sorting network supervised by the ground-truth order extracted from KG and the ground-truth sentence. We also report our model's results under the ground-truth order during inference time (denoted as "S-OSC-GT") which serves as the upper bound for our sorting performance. From the comparison of our model with predicted order "S-OSC" and our model with ground-truth order "S-OSC-GT", we can see that our model results with predicted order are close to the results with ground-truth order with the result gaps less than 1.2 points in most metrics, which shows the advantage of our learning-based sorting network for generating KG description order.
Results on DART are shown in Table 2. From previous models' results, we can see that pretraining on the large corpus (e.g., "T5-Large" and "Bart-Large") brings significant result improvement compared to "Seq2Seq-Att" and "End-to-End Transformer". Our S-OSC model further achieves
Datasets	DART
Metrics	B-4	METEOR	MoverScore	BERTScore	BLEURT
Seq2Seq-Att (Nan et al., 2020)	29.66	0.27	0.31	0.90	-0.13
End-to-End Transformer (Ferreira et al., 2019)	27.24	0.25	0.25	0.89	-0.29
T5-Large (Ribeiro et al., 2020a)	50.66	0.40	0.54	0.95	0.44
BART-Large (Ribeiro et al., 2020a)	48.56	0.39	0.52	0.95	0.41
JointGT (Ke et al., 2021)	54.24	0.44	0.64	0.96	0.59
S-OSC(ours)	62.01	0.43	0.64	0.96	0.49
S-OSC(ours)-GT	64.53	0.44	0.66	0.96	0.51
new state-of-the-art results in all five metrics compared to these pre-training based models (e.g., "T5-Large") with B-4 score improved by 11.35 points, METEOR score improved by 0.03 points, Moverscore improved by 0.1 points, BERTScore improved by 0.01 points, and BLEURT improved by 0.05 points. JointGT outperforms all the previous baseline models. Compared with JointGT (Ke et al., 2021), our model can also obtain an improvement of 7.77 points in B-4 score. This result also validates the effectiveness of our model in improving the fluenthness and authenticity of the generated sentences with the proposed learning-based sorting network and consistency enhancement under the POS syntactic and semantic context constraints. Similar to the results on WebNLG, our model S-OSC with predicted order can achieve the performance close to that with ground-truth order.
In this section, we conduct extensive experiments on WebNLG to evaluate various factors in the word copy or prediction during sentence generation, e.g., the POS generator and semantic context (SC) scoring. We compare our full S-OSC model with the following variations: without word copy component and just use the predicted word from Word Decoder at each time step (w/o CP), without POS information in copy probability prediction (w/o POS), without semantic context scoring in copy probability prediction (w/o SC), without POS tag information and semantic context scoring in copy probability prediction (w/o POS and SC) and relying on the last hidden states to predict the copy probability as in previous methods (Koncel-Kedziorski et al., 2019; See et al., 2017). From the results in Table 3, we can observe that: (1) By removing the word copy component and directly taking the predicted word from Word Decoder at each time step (w/o CP), the results drop significantly by 3.95 points in
Table 2: The results of different models on DART dataset.
Methods	B-4	R-L	CIDEr	Chrf++
S-OSC	63.10	80.0	5.40	81.0
w/o CP	59.15	77.8	4.99	78.0
w/o POS and SC	59.80	77.7	5.01	78.1
w/o POS	61.00	78.5	5.10	78.6
w/o SC	61.06	78.9	5.21	79.1
Table 3: Ablation analysis for copy or prediction component in our model on WebNLG. S-OSC here is under the ground-truth order.
B-4, 2.2 points in R-L, 0.41 points in CIDEr and 3 points in Chrf++. (2) We then consider incorporating the copy prediction component, but just rely on the last hidden states to predict the copy probability as in previous methods (Koncel-Kedziorski et al., 2019; See et al., 2017) without POS tag information and semantic context scoring in copy probability prediction (w/o POS and SC). The results improve slightly in all the metrics compared with that of "w/o CP". (3) We then evaluate the effectiveness of POS syntactic information and semantic context scores in improving the quality of the generated sentences by removing each of them at a time.
Compared to full S-OSC model, without POS information in copy probability prediction (w/o POS), the results drop by 2.1 points in B-4, 1.5 points in R-L, 0.3 points in CIDEr and 2.4 points in Chrf++. Compared to full S-OSC model, without semantic context scoring in copy probability prediction (w/o SC), the results drop by 2 points in B-4, 1.1 points in R-L, 0.2 points in CIDEr and 1.9 points in Chrf++. Both "w/o POS" and "w/o SC" improve consistently compared to previous copy policy in "w/o POS and SC".
To further reveal more details about our model, we conduct extra ablation studies regarding the following questions. (1) How does triplet structure encoding in the sorting network help the model compared with direct node encoding without triplet structure context? (2) How does our model perform
Methods	B-4	R-L	CIDEr	Chrf++
GT	63.10	80.0	5.4	81.0
TS	61.90	79.3	5.3	79.7
NS	60.00	77.0	5.1	78.0
RS	55.20	72.0	4.7	76.0
Table 4: The results of sorting order on WebNLG. TS denotes the triple-level sorting order, NS is the node-level sorting order (Li et al., 2021) and RS represents the random sorting order. The same symbols are as below.
Figure 5: The B-4 results in three groups with different KG sizes on DART dataset for different sorting orders.
for different triplet numbers in KG? (3) How does the sliding window size in semantic context scoring function affect the model performance?
(1) Triplet structure encoding in sorting network. To show the effect of triplet structure encoding (triple-level) in our sorting network, we investigate the performance of direct node encoding without triplet structure (node-level), as well as our S-OSC model's results with random sorting order. Results on WebNLG are shown in Table 4 and the upper bound result is shown with ground-truth order (GT). From Table 4, sorting with triple-level sorting order "TS" outperforms random sorting order "RS" significantly, and also outperforms node-level sorting order "NS" by 1.9 points in B-4, 2.3 points in R-L, 0.2 points in CIDEr and 1.7 points in Chrf++, showing the advantage of triplet structure encoding compared to node encoding without triplet structure context. The results on DART dataset show similar trend and are reported in supplementary material.
(2) Different Knowledge Graph Sizes. To verify the effect of our S-OSC model performance on different KG sizes, we split the test dataset into three subsets according to the size of the KG, i.e., the number of triples in KG is less than 3, between 4 and 6, and more than 7, and reports results on each subset. DART is selected for experiment due to its wide distribution of KG sizes. The results of B-4 in each subset are plotted in Figure 5, showing that as
Figure 6: The B-4 results of different sliding windows on WebNLG dataset.
the number of triples in KG increases, the difficulty of sorting and KG-to-text generation increases and the B-4 results generally decrease in all three methods ("GT", "RS" and "TS"). Also, our model's superiority exhibits significant improvement in all three subsets with different KG sizes (comparing "TS" with "RS"). Other metric results show similar trend and are reported in supplementary material.
(3) Sliding window size in the semantic context scoring. We plot the B-4 results for different sliding window sizes of semantic context scoring function on WebNLG dataset, shown in Figure 6. From Figure 6, the model achieves the best performance when the sliding window size is 3. Other metric results show similar trend and are reported in supplementary material. Thus, we set the sliding window size to 3 in the experiments.
We conduct the human evaluation on WebNLG to further evaluate the generated text. In this paper, we adopt the same human evaluation criteria as Chen et al. (Chen et al., 2019), i.e., Factual correctness including Supp (counting the number of facts that co-exist in the KG and generated text) and Cont (counting the facts in the generated texts missing from or contradicting with KG), Language naturalness including NF (evaluating the accuracy and fluentness of generated sentences). In addition to using absolute score at 5-point metric in NF, we also use relative ranking scores (termed NA). We randomly select 100 knowledge graphs for human evaluation. Five native English speakers volunteer to score all the 100 knowledge graphs. Table 5 reports the results<sup>2</sup>. We can observe that: the generated texts of our S-OSC model are more authentic
$^{2}$ Cohen's kappa coefficients for the first two factors are 0.79, 0.83, 0.75, 0.76.
Datasets	WebNLG	DART
Knowledge graph	Malaysia
1
EthnicGroup
2
Malaysian
3
Abu Zahar Ujang
4
Malaysian Indian	Antares rocket
1
MANUFACTURER
3
Yuzhnoye
Design Office
2014 10 28
2013 04 21
Mid Atlantic Regional
Spaceport Launch Pad 0
Ground-Truth text	Abu zahar ujang is a leader in Malaysia, where there are ethnic groups called Malaysian Malay and Malaysian Indian.	The Antares rocket, made by Yuzhnoye Design Office, had its initial flight April 21, 2013 and final flight on 10/28/2014. It was launched at Mid-Atlantic Regional Spaceport Launch Pad 0.
Linearized KG from other methods	Li et al.: 1→3→2→4	Bart-Large: 1→3→4→5→2
Generated text from other methods	Li et al.: The leader of Malaysia is called jusuf kalla and Malaysian Malay are an ethnic group there.	Bart-Large: the Yuzhnoye Design Office manufactured the Antares rocket which was launched on april 21, 2013 at the Mid - Atlantic Regional Spaceport Launch Pad.
Linearized KG from ours	1→2→3→4	1→3→4→2→5
Generated text from ours	Abu Zehar Ujang is the leader of Malaysia where Malaysian Malay and Malaysian Indian are ethnic groups.	Yuzhnoye Design Office created the Antares rocket whose initial flight took place April 21, 2013 at Mid Atlantic Regional Spaceport Launch Pad 0 and final flight on 10/28/2014.
Figure 7: Two examples from WebNLG and DART. The text in color is corresponding to the KG nodes. The generated texts in red color indicate that the generated texts contract with KG.
Methods	Supp.↑	Cont.↓	NF↑	NA↑
ground-truth	3.82	0.10	4.75	2.93
Li et al.(Li et al., 2021)	3.48	0.33	3.7	2.10
S-OSC	3.77	0.15	4.25	2.48
Table 5: The results of human evaluation on WebNLG dataset.
and consistent with KG than the method in Li et al. (Li et al., 2021).
4.7 Qualitative Analysis
We show two qualitative examples from WebNLG and DART in Figure 7. From the example of WebNLG, we can see that the previous method in Li et al. (Li et al., 2021) generates the wrong description order with node 3 and node 2 exchanged positions. This causes the generated sentence from Li et al. (Li et al., 2021) containing the wrong text "jusuf kalla" noted in red color and missing the text "Malaysian Indian", while our S-OSC method generates the right order and consistent sentences. In the example from DART, though our S-OSC model generates relative inferior order with node 2 and node 5 exchanged positions, relying on our strong sentence generator with syntactic constraint and semantic consistency constraint, our model is able to generate semantic consistent sentence to the KG with the right words copied from KG. However, in this example, Bart-Large (Ribeiro et al., 2020a) still misses some key word description from KG,
e.g., "the final fight on 10/28/2014" and "Launch Pad 0", though conditioned on the right predicted order, showing the inferior performance of their copy or prediction module in sentence generation.
This paper proposes a learning-based sorting network to obtain the optimal description order for KG-to-text generation. Additionally, our model incorporates POS generator and semantic context scoring to selectively copy words from KG and improve the word authenticity in generated sentences. Extensive experiments show that our model outperforms previous state-of-the-art approaches. In the future, we will introduce casual inference into the model to further improve the reasoning ability.
Prof. Zhou's group is supported by The National Key R & D Program of China (Grant No. 2017YFB1302400), Jinan '20 New Colleges and Universities' Funded Scientific Research Leader Studio (2021GXRC079), Major Agricultural Applied Technological Innovation Projects of Shandong Province (SD2019NJ014), Shandong Natural Science Foundation(ZR2019MF064), Beijing Advanced Innovation Center for Intelligent Robots and Systems (2019IRS19). In addition, the authors thank the anonymous reviewers for providing valuable comments to improve this paper.
Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65-72.
Tom Braude, Idan Schwartz, Alexander Schwing, and Ariel Shamir. 2021. Towards coherent visual storytelling with ordered image attention. arXiv preprint arXiv:2108.02180.
Emanuele Bugliarello and Desmond Elliott. 2021. The role of syntactic planning in compositional image captioning. arXiv preprint arXiv:2101.11911.
Zhiyu Chen, Harini Eavani, Wenhu Chen, Yinyin Liu, and William Yang Wang. 2019. Few-shot nlg with pre-trained language model. arXiv preprint arXiv:1904.09521.
Liying Cheng, Dekun Wu, Lidong Bing, Yan Zhang, Zhanming Jie, Wei Lu, and Luo Si. 2020. Ent-description: Entity description generation by exploring knowledge graph. arXiv preprint arXiv:2004.14813.
Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. 2019. Show, control and tell: A framework for generating controllable and grounded captions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8307-8316.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Bayu Distiawan, Jianzhong Qi, Rui Zhang, and Wei Wang. 2018. Gtr-lstm: A triple encoder for sentence generation from rdf data. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1627-1637.
Thiago Castro Ferreira, Chris van der Lee, Emiel Van Miltenburg, and Emiel Krahmer. 2019. Neural data-to-text generation: A comparison between pipeline and end-to-end architectures. arXiv preprint arXiv:1908.09022.
Jeffrey Flanigan, Chris Dyer, Noah A Smith, and Jaime G Carbonell. 2016. Generation from abstract meaning representation using tree transducers. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 731-739.
Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017. The webnlg challenge: Generating text from rdf data. In Proceedings of the 10th International Conference on Natural Language Generation, pages 124-133.
Jingyi Hou, Xinxiao Wu, Wentian Zhao, Jiebo Luo, and Yunde Jia. 2019. Joint syntax representation learning and visual cue translation for video captioning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8918-8927.
Alexander Hoyle, Ana Marasović, and Noah Smith. 2020. Promoting graph awareness in linearized graph-to-text generation. arXiv preprint arXiv:2012.15793.
Pei Ke, Haozhe Ji, Yu Ran, Xin Cui, Liwei Wang, Linfeng Song, Xiaoyan Zhu, and Minlie Huang. 2021. Jointgt: Graph-text joint representation learning for text generation from knowledge graphs. arXiv preprint arXiv:2106.10502.
Rik Koncel-Kedziorski, Dhanush Bekal, Yi Luan, Mirella Lapata, and Hannaneh Hajishirzi. 2019. Text generation from knowledge graphs with graph transformers. arXiv preprint arXiv:1904.02342.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461.
Junyi Li, Tianyi Tang, Wayne Xin Zhao, Zhicheng Wei, Nicholas Jing Yuan, and Ji-Rong Wen. 2021. Few-shot knowledge graph-to-text generation with pretrained language models. arXiv preprint arXiv:2106.01623.
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.
Diego Marcheggiani and Laura Perez-Beltrachini. 2018. Deep graph convolutional encoders for structured data to text generation. arXiv preprint arXiv:1810.09995.
Linyong Nan, Dragomir Radev, Rui Zhang, Amrit Rau, Abhinand Sivaprasad, Chiachun Hsieh, Xiangru Tang, Aadit Vyas, Neha Verma, Pranav Krishna, et al. 2020. Dart: Open-domain structured data record to text generation. arXiv preprint arXiv:2007.02871.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318.
Matthew E Peters, Mark Neumann, Robert L Logan IV, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A Smith. 2019. Knowledge enhanced contextual word representations. arXiv preprint arXiv:1909.04164.
Maja Popovic. 2015. chrf: character n-gram f-score for automatic mt evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392-395.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683.
Leonardo FR Ribeiro, Martin Schmitt, Hinrich Schütze, and Iryna Gurevych. 2020a. Investigating pretrained language models for graph-to-text generation. arXiv preprint arXiv:2007.08426.
Leonardo FR Ribeiro, Yue Zhang, Claire Gardent, and Iryna Gurevych. 2020b. Modeling global and local node contexts for text generation from knowledge graphs. Transactions of the Association for Computational Linguistics, 8:589-604.
Leonardo FR Ribeiro, Yue Zhang, and Iryna Gurevych. 2021. Structural adapters in pretrained language models for amr-to-text generation. arXiv preprint arXiv:2103.09120.
Apoory Saxena, Aditay Tripathi, and Partha Talukdar. 2020. Improving multi-hop question answering over knowledge graphs using knowledge base embeddings. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4498-4507.
Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. arXiv preprint arXiv:1704.04368.
Thibault Sellam, Dipanjan Das, and Ankur P Parikh. 2020. Bleurt: Learning robust metrics for text generation. arXiv preprint arXiv:2004.04696.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with relative position representations. arXiv preprint arXiv:1803.02155.
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566-4575.
Xiang Wang, Tinglin Huang, Dingxian Wang, Yancheng Yuan, Zhenguang Liu, Xiangnan He, and Tat-Seng Chua. 2021. Learning intents behind interactions with knowledge graph for recommendation. In Proceedings of the Web Conference 2021, pages 878-887.
Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. 2014. Knowledge graph embedding by translating on hyperplanes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 28.
Chunpu Xu, Min Yang, Chengming Li, Ying Shen, Xiang Ao, and Ruifeng Xu. 2021. Imagine, reason and write: Visual storytelling with graph knowledge and relational reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 3022-3029.
Zhixian Yang and Xiaojun Wan. 2021. Neural text generation with part-of-speech guided softmax. arXiv preprint arXiv:2105.03641.
Zixiaofan Yang, Arash Einolghozati, Hakan Inan, Keith Diedrick, Angela Fan, Pinar Donmez, and Sonal Gupta. 2020. Improving text-to-text pre-trained models for the graph-to-text task. In Proceedings of the 3rd International Workshop on Natural Language Generation from the Semantic Web (WebNLG+), pages 107-116.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675.
Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M Meyer, and Steffen Eger. 2019. Moverscore: Text generation evaluating with contextualized embeddings and earth mover distance. arXiv preprint arXiv:1909.02622.
Supplementary Material for Syntax Controlled Knowledge Graph-to-Text Generation with Order and Semantic Consistency
{202120638,202114753,zhoufengyu}@mail.sdu.edu.cn,
Figure 1: The steps of obtaining the 0-1 sequence. When the KG node exists in the text, all mentions of the node are masked as 1, otherwise 0.
Datasets	WEBNLG
Metrics	B-4	R-L	CIDEr	Chrf++
Baselines†	57.00	75.20	4.20	75.00
Li et al.(Li et al., 2021)	61.88	75.74	6.03	79.10
S-OSC(ours)	61.90	79.30	5.30	79.70
Table 1: Results for different models on WebNLG dataset. B-4 and R-L are short for BLEU-4 and ROUGE-L, respectively. Baseline is the reproduced results of Li et al. (Li et al., 2021). Bold and underline represent the best and the second best performing results. (the same term is used below)
1 0-1 Generator
In order to make the generated text more consistent with the KG, we use a 0-1 generator to obtain the ground-truth of copy or prediction order, where 0 represents a token from generation and 1 from KG. The steps are as shown in Figure 1.
2 Ground-Truth Sorting Order
The description sorting order is essential for the generation process. In this paper, we sort the order according to the triple-level. Taking Figure 1 as an example, we can obtain three triplets, i.e., (AWH Engineering College, COUNTRY, India), (AWH Engineering College, ESTABLISED, 2001) and (AWH Engineering College, CITY, Kuttikkattoor). In the reference sentence, the triple (AWH Engineering College, CITY, Kuttikkattoor) appears first, (AWH Engineering College, COUNTRY, India) appears second and (AWH Engineering College, ESTABLISED, 2001) last. Thus, we can form the order 2,0,1 for the providing KG. Note that, we only record the position of the triplet where it first appeared regardless of many times.
3 Code Re-implementation
We re-implement the author's code in Li et al. (Li et al., 2021) as our baseline. It can be observed from Table 1 that our S-OSC model outperforms the baseline and the results of Li et al. reported from their paper in three evaluation metrics,B-4,R-L and Chrf++ metrics. Although Li et al. achieves the best CIDEr score in their reported results, this score is not replicable and our S-OSC still obtains 1.1 point higher than that of the replicated baseline.
4 Triplet Structure Encoding in Sorting Network on DART (Nan et al., 2020)
From Table 2, encoding with triple-level sorting order "TS" outperforms random sorting order "RS" significantly, and also outperforms node-level sorting order "NS" by 1.35 points in B-4, 0.01 points in METEOR, 0.02 points in BERTScore and 0.13 points in BLEURT, showing the advantage of triplet structure encoding compared to node encoding without triplet structure context.
5 Different Knowledge Graph Sizes
Table 4 shows that when the number of triples in KG increases, the difficulty of sorting and KG-to-text generation increases (see "GT" and "TS") and our model's superiority exhibits more relative im
Methods	B-4	METEOR	MoverScore	BERTScore	BLEURT
GT	64.53	0.44	0.66	0.96	0.51
TS	62.01	0.43	0.64	0.96	0.51
NS	60.66	0.42	0.64	0.94	0.38
RS	58.67	0.42	0.62	0.93	0.34