SlowGuess/ACL_2020_OCR huggingface dataset:
WebNLG (Gardent et al., 2017) This dataset consists of 18,102 training and 871 development KG-text pairs, where each KG is a subgraph of DBpedia that can contain up to 7 relations (triples). The testset has two parts: seen, containing 971 pairs where the KG entities and relations belong to the DBpedia categories that are seen in the training data, and unseen, where the entities and relations come from unseen categories. Same as most previous work, we evaluate our model on the seen part, and this is also more relevant to our setup.
We follow Marcheggiani and Perez-Beltrachini (2018) to preprocess the data. To obtain the alignments between a KG and a sentence, we use a method based on heuristic string matching. For more detail, we remove any abbreviations from a KG node (such as "New York (NY)" is changed to "New York"), before finding the first phrase in the
Figure 3: Development results on LDC2015E86.
sentence that matches the longest prefix of the node. As a result, we find a match for  $91\%$  KG nodes.
5.2 Settings
For model hyperparameters, we follow the setting of our baseline (Zhu et al., 2019), where 6 self-attention layers are adopted with 8 heads for each layer. Both sizes of embedding and hidden states are set to 512, and the batch token-size is 4096. The embeddings are randomly initialized and updated during training. All models are trained for 300K steps using Adam (Kingma and Ba, 2014) with  $\beta 1 = 0.1$ . Byte-pair encoding (BPE) (Sennrich et al., 2016) with 10K operations is applied to all datasets. We use 1080Ti GPUs for experiments.
For our approach, the multi-layer perceptrons for deep biaffine classifiers (Equations 6, 7, 9 and 10) take two layers of 512 units. The Transformer decoder (Equation 14) for predicting linearized graphs takes the same embedding and hidden sizes as the baseline decoder (Equation 4).
5.3 Development Results
Figure 3 shows the devset performances of using either Loss 1 (triple relations) or Loss 2 (linearized graph) under different coefficients. It shows the baseline performance when a coefficient equals to 0. There are large improvements in terms of BLEU score when increasing the coefficient of either loss from 0. These results indicate the effectiveness of our autoencoding training framework. The performance of our model with either loss slightly goes down when further increasing the coefficient. One underlying reason is that an over-large coefficient will dilute the primary signal on language modeling, which is more relevant to the BLEU metric. Particularly, we observe the highest performances when  $\alpha$  and  $\beta$  are 0.05 and 0.15, respectively, and thus we set our coefficients using these values for the remaining experiments.
$^{2}$ http://www.statmt.org/moses/
3https://amr.isi.edu/download.html
4https://nlp.stanford.edu/software-tokenizer.shtml
<sup>5</sup>https://wiki.dbpedia.org/
7992
Model	BLEU	Time
LSTM (Konstas et al., 2017)	22.00	-
GRN (Song et al., 2018)	23.28	-
DCGCN (Guo et al., 2019)	25.70	-
RA-Trans-SA (Zhu et al., 2019)	29.66	-
RA-Trans-F-ours	29.11	0.25
+ Loss 1 (triple relations)	30.47	0.38
+ Loss 2 (linearized graph)	31.13	0.52
+ Both	31.41	0.61
DCGCN (0.3M)	33.2	-
GRN (2M)	33.6	-
LSTM (20M)	33.8	-
DCGCN (ensemble, 0.3M)	35.3	-
Table 1 shows the main comparison results with existing work for AMR-to-text generation, where "Time" represents the average time (seconds) for training one step. The first group corresponds to the reported numbers of previous models on this dataset, and their main difference is the encoder for presenting graphs: LSTM (Konstas et al., 2017) applies a multi-layer LSTM on linearized AMRs, GRN (Song et al., 2018) and DCGCN (Guo et al., 2019) adopt graph neural networks to encode original AMRs, and RA-Trans-SA is the best performing model of Zhu et al. (2019), using self attention to model the relation path for each node pair.
The second group reports our systems, where the RA-Trans-F-ours baseline is our implementation of the feature-based model of Zhu et al. (2019). It shows a highly competitive performance on this dataset. Applying Loss 1 alone achieves an improvement of 1.36 BLEU points, and Loss 2 alone obtains 0.66 more points than Loss 1. One possible reason is that Loss 2, which aims to reconstruct the whole linearized graph, can provide more informative features. Using both losses, we observe roughly a 2.3-point gain in terms of BLEU, indicating that both losses are complementary.
Regarding Meteor, RA-Trans-SA reports 35.45, the highest among all previously reported numbers. The RA-Trans-F-ours baseline gets 35.0 that is slightly worse than RA-Trans-SA. Applying Loss 1 or Loss 2 alone gives a number of 35.5 and 36.1, respectively. Using both losses, our approach achieves 36.2 that is better than RA-Trans-SA.
Table 1: Main test results on LDC2015E86. Numbers such as “ $2M$ ” means the number of extra silver data being used, and “ensemble” indicates model ensemble.
Model	Recall (%)
RA-Trans-F-ours	78.00
+ Both	85.13
Table 2: Human study for the recall of input relations on LDC2015E86.
Regarding the training speed, adopting Loss 2 requires double amount of time compared with the baseline, being much slower than Loss 1. This is because the biaffine attention calculations for different word pairs are parallelizable, while it is not for producing a linearized graph. Using both losses together, we observe a moderately longer training process (1.4-times slower) than the baseline. Please note that our autoencoding framework only affects the offline training procedure, leaving the online inference process unchanged.
The last group shows additional higher numbers produced by systems that use the ensemble of multiple models and/or additional silver data. They suffer from problems such as requiring massive computation resources and taking a long time for training. We leave exploring additional silver data and ensemble for further work.
5.5 Quantitative Human Study on Preserving Input Relation
Our multi-view autoencoding framework aims at preserving input relations, thus we further conduct a quantitative human study to estimate this aspect. To this end, we first extract all interactions of a subject, a predict and an object (corresponding to the AMR fragment "pred :ARG0 subj :ARG1 obj") from each AMR graph, and then check how many interactions are preserved by the output of a model. The reason for considering this type of interaction comes from two folds: first, they convey fundamental information forming the backbone of a sentence, and second, they can be easily extracted from graphs and evaluated by human judges.
As shown in Table 2, we choose 200 AMR-sentence pairs to conduct this study and compare our model with the baseline in terms of the recall number, showing the percent of preserved interactions. To determine if a sentence preserves an interaction, we ask 3 people with NLP background to make their decisions and choose the majority vote as the human judgement. Out of the 491 interactions, the baseline only preserves  $78\%$ . With our multi-view autoencoding losses,  $7.13\%$  more
7993
interactions are preserved, which further confirms the effectiveness of our approach.
As shown in Table 3, we further demonstrate several typical examples from our human study for better understanding how our framework helps preserve structural input information. Each example includes an input AMR, a reference sentence (Ref), the baseline output (Baseline) and the generated sentence by our approach (Our approach).
For the first example, the baseline output drops the key predicate "recommend" and fails to preserve the fact that "you" is the subject of "go". The reason can be that "I should go to" occurs frequently in the training corpus. On the other hand, the extra signals produced by our multi-view framework enhance the input semantic information, guiding our model to generate a correct sentence
Table 3: Example system outputs.
Model	BLEU
RA-Trans-F-ours + Loss 1	30.47
w/o edge label	29.39
RA-Trans-F-ours + Loss 2	31.13
w/o edge label	30.36
random linearization	31.07
Table 4: Ablation study for both views.
with the exact meaning of the input AMR.
The second example shows a similar situation, where the baseline generates a natural yet short sentence that drops some important information from the input graph. As a result of the information loss, the resulting sentence conveys an opposite meaning ("with drug trafficking policy") to the input ("targeting drug trafficking aircraft"). This is a typical problem suffered by many neural graph-to-sequence models. Our multi-view framework helps recover the correct meaning: "policy of target for drug trafficking aircraft".
5.7 Ablation Study
As shown in Table 4, we conduct an ablation study on LDC2015E86 to analyze how important each part of the input graphs is under our framework. For Loss 1, we test the situation when no edge labels are available, and as a result, we observe a large performance drop of  $1.0+$  BLEU points. This is quite intuitive, because edge labels carry important relational knowledge between the two connected nodes. Therefore, discarding these labels will cause loss of significant semantic information.
For Loss 2, we also observe a large performance decrease when edge labels are dropped, confirming the observation for Loss 1. In addition, we study the effect of random graph linearization, where the order for picking children is random rather than following the left-to-right order at each stage of the depth-first traversal procedure. The motivation is to investigate the robustness of Loss 2 regarding input variances, as an organized input order (such as an alphabetical order for children) may not be available for certain graph-to-sequence tasks. We observe a marginal performance drop of less than 0.1 BLEU points, indicating that our approach is very robust for input variances. It is likely because different linearization results still indicate the same graph. Besides, one previous study (Konstas et al., 2017) shows a very similar observation.
7994
Model	BLEU	Meteor
DCGCN	27.60	-
RA-Trans-CNN	31.82	36.38
RA-Trans-F-ours	31.77	37.2
+ Loss 1	33.98	37.5
+ Loss 2	34.13	37.8
+ Both	34.21	38.0
Table 5: Main test results on LDC2017T10.
Model	BLEU	Meteor
ADAPT	60.59	44.0
GCNEC	55.90	39.0
RA-Trans-F-ours	60.51	42.2
+ Loss 1	61.78	43.6
+ Loss 2	62.29	43.5
+ Both	62.89	44.2
Table 6: Main test results on WebNLG
5.8 Main Results on LDC2017T10
Table 5 compares our results on LDC2017T10 with the highest numbers reported by single models without extra silver training data. RA-Trans-CNN is another model by Zhu et al. (2019) that adopt a convolutional neural network (LeCun et al., 1990) to model the relation path for each node pair. Again, the RA-Trans-F baseline achieves a comparable score with RA-Trans-CNN, and our approach improves the baseline by nearly 2.5 BLEU points, indicating its superiority.
Regarding Meteor score, our advantage (1.62 points) over the previous state-of-the-art system on this dataset is larger than that (0.75 points) on LDC2015E86. Since LDC2017T10 has almost one time more training instances than LDC2015E86, we may conclude that the problem of dropping input information may not be effectively reduced by simply adding more supervised data, and as a result, our approach can still be effective on a larger dataset. This conclusion can also be confirmed by comparing the gains of our approach on both AMR datasets regarding BLEU score (2.3 vs 2.5).
5.9 Main Results on WebNLG
Table 6 shows the comparison of our results with previous results on the WebNLG testset. ADAPT (Gardent et al., 2017) is based on the standard encoder-decoder architecture (Cho et al., 2014)
with byte pair encoding (Sennrich et al., 2016), and it was the best system of the challenge.  $\mathrm{GCN}_{EC}$  (Marcheggiani and Perez-Beltrachini, 2018) is a recent model using a graph convolution network (Kipf and Welling, 2017) for encoding KGs.
Our baseline shows a comparable performance with the previous state of the art. Based on this baseline, applying either loss leads to a significant improvement, and their combination brings a gain of more than 2 BLEU points. Although the baseline already achieves a very high BLEU score, yet the gains on this task are still comparable with those on AMR-to-text generation. This observation may imply that the problem of missing input structural knowledge can be ubiquitous among many graph-to-text problems, and as a result, our approach can be widely helpful across many tasks.
Following previous work, we also report Meteor scores, where our approach shows a gain of 2 points against the baseline and our final number is comparable with ADAPT. Similar with the gains on the BLEU metric, Loss 1 is comparable with Loss 2 regarding Meteor, and their combination is more useful than applying each own.
We proposed reconstructing input graphs as autoencoding processes to encourage preserving the input semantic information for graph-to-text generation. In particular, the auxiliary losses for recovering two complementary views (triple relations and linearized graph) of input graphs are introduced, so that our model is trained to retain input structures for better generation. Our training framework is general for different graph types. Experiments on two benchmarks showed the effectiveness of our framework under both the automatic BLEU metric and human judgements.
Both Te An and Jinsong Su were supported by the National Key R&D Program of China (No. 2019QY1803), National Natural Science Foundation of China (No. 61672440), and the Scientific Research Project of National Language Committee of China (No. YB135-49). Yue Zhang was supported by the joint research program between BriteDreams robotics and Westlake University. We thank the anonymous reviewers for their constructive suggestions.
7995
Roee Aharoni and Yoav Goldberg. 2017. Towards string-to-tree neural machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).
Anusha Balakrishnan, Jinfeng Rao, Kartikeya Upasani, Michael White, and Rajen Subba. 2019. Constrained decoding for neural NLG from compositional representations in task-oriented dialogue. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.
Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffith, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse.
Daniel Beck, Gholamreza Haffari, and Trevor Cohn. 2018. Graph-to-sequence learning using gated graph neural networks. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
Deng Cai and Wai Lam. 2019. Graph transformer for graph-to-sequence learning. In Thirty-Fourth AAAI Conference on Artificial Intelligence.
Kris Cao and Stephen Clark. 2019. Factorising amr generation through syntax. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers).
Kyunghyun Cho, Bart van Merrienboer, Caglar Gul-cehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rn n encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Marco Damonte and Shay B Cohen. 2019. Structural neural encoders for amr-to-text generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers).
Michael Denkowski and Alon Lavie. 2014. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the EACL 2014 Workshop on Statistical Machine Translation.
Timothy Dozat and Christopher D Manning. 2017. Deep bioffine attention for neural dependency parsing. In International Conference on Learning Representations.
Ondrej Dusek, David M Howcroft, and Verena Rieser. 2019. Semantic noise matters for neural natural language generation. In Proceedings of the 12th International Conference on Natural Language Generation.
Ondrej Dušek, Jekaterina Novikova, and Verena Rieser. 2018. Findings of the e2e nlg challenge. In Proceedings of the 11th International Conference on Natural Language Generation.
Jeffrey Flanigan, Chris Dyer, Noah A Smith, and Jaime Carbonell. 2016. Cmu at semeval-2016 task 8: Graph-based amr parsing with infinite ramp loss. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016).
Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017. The webnlg challenge: Generating text from rdf data. In Proceedings of the 10th International Conference on Natural Language Generation.
Zhijiang Guo, Yan Zhang, Zhiyang Teng, and Wei Lu. 2019. Densely connected graph convolutional networks for graph-to-sequence learning. Transactions of the Association for Computational Linguistics, 7.
Valerie Hajduk, Jan Buys, Michael Wayne Goodman, and Emily M Bender. 2019. Neural text generation from rich semantic representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers).
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. *Neural computation*, 9(8).
Xudong Hong, Ernie Chang, and Vera Demberg. 2019. Improving language generation from feature-rich tree-structured data with relational graph convolutional encoders. In Proceedings of the 2nd Workshop on Multilingual Surface Realisation (MSR 2019), pages 75–80.
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, Jayant Krishnamurthy, and Luke Zettlemoyer. 2017. Learning a neural semantic parser from user feedback. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 963-973.
Chen Jia, Xiaobo Liang, and Yue Zhang. 2019. Cross-domain ner using cross-domain language modeling. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Thomas N Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations.
7996
Rik Koncel-Kedziorski, Dhanush Bekal, Yi Luan, Mirella Lapata, and Hannaneh Hajishirzi. 2019. Text generation from knowledge graphs with graph transformers. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers).
Ioannis Konstas, Srinivasan Iyer, Mark Yatskar, Yejin Choi, and Luke Zettlemoyer. 2017. Neural amr: Sequence-to-sequence models for parsing and generation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
Yann LeCun, Bernhard E Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne E Hubbard, and Lawrence D Jackel. 1990. Handwritten digit recognition with a back-propagation network. In Advances in neural information processing systems, pages 396-404.
Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard Zemel. 2016. Gated graph sequence neural networks. In International Conference on Learning Representations.
Liyuan Liu, Jingbo Shang, Xiang Ren, Fangzheng Xu, Huan Gui, Jian Peng, and Jiawei Han. 2018a. Empower sequence labeling with task-aware neural language model. In Thirty-Second AAAI Conference on Artificial Intelligence.
Lizhen Liu, Xiao Hu, Wei Song, Ruiji Fu, Ting Liu, and Guoping Hu. 2018b. Neural multitask learning for simile recognition. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.
Yijia Liu, Wanxiang Che, Bo Zheng, Bing Qin, and Ting Liu. 2018c. An amr aligner tuned by transition-based parser. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.
Diego Marcheggiani and Laura Perez-Beltrachini. 2018. Deep graph convolutional encoders for structured data to text generation. In Proceedings of the 11th International Conference on Natural Language Generation.
Joakim Nivre. 2003. An efficient algorithm for projective dependency parsing. In Proceedings of the Eighth International Conference on Parsing Technologies.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics.
Nima Pourdamghani, Yang Gao, Ulf Hermjakob, and Kevin Knight. 2014. Aligning english strings with
abstract meaning representation graphs. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Marek Rei. 2017. Semi-supervised multitask learning for sequence labeling. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
Marek Rei and Anders Søgaard. 2019. Jointly learning to label sentences and tokens. In Proceedings of the AAAI Conference on Artificial Intelligence.
Leonardo FR, Ribeiro, Claire Gardent, and Iryna Gurevych. 2019. Enhancing amr-to-text generation with dual graph representations. In Proceedings of EMNLP-IJCNLP.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
Linfeng Song, Daniel Gildea, Yue Zhang, Zhiguo Wang, and Jinsong Su. 2019. Semantic neural machine translation using amr. Transactions of the Association for Computational Linguistics, 7:19-31.
Linfeng Song, Yue Zhang, Zhiguo Wang, and Daniel Gildea. 2018. A graph-to-sequence model for amr-to-text generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
Jinsong Su, Zhixing Tan, Deyi Xiong, Rongrong Ji, Xiaodong Shi, and Yang Liu. 2017. Lattice-based recurrent neural network encoders for neural machine translation. In Thirty-First AAAI Conference on Artificial Intelligence.
Ida Szubert, Adam Lopez, and Nathan Schneider. 2018. A structured syntax-semantics interface for english-amr alignment. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers).
Bayu Distiawan Trisedya, Jianzhong Qi, Rui Zhang, and Wei Wang. 2018. Gtr-lstm: A triple encoder for sentence generation from rdf data. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1.
Zhaopeng Tu, Yang Liu, Lifeng Shang, Xiaohua Liu, and Hang Li. 2017. Neural machine translation with reconstruction. In Thirty-First AAAI Conference on Artificial Intelligence.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems.
7997