SlowGuess/ACL_2020_OCR huggingface dataset:
Furthermore, our best single model still has better results than previous ensemble models on both two datasets. Note that additive attention based model  $\mathrm{HetGT}_{\mathrm{additive}}$  is significantly better than dot-product attention based model  $\mathrm{HetGT}_{\mathrm{dot-product}}$  in AMR-to-text generation. It may be attributed to that the additive attention has less parameters and is easier to train on the small dataset.
3.5 Results on Syntax-based NMT
Table 3 presents the results of our single model and previous single/ensemble models on the test sets for En-De and En-Cs language pairs. We can see that our Transformer baseline already outperforms all previous results even though some of them are Transformer based. It shows the effectiveness of Transformer for NMT tasks. Meanwhile, even without changing the model architecture for the NMT tasks, our single model surpasses Transformer baseline by 2.26 and 1.46 BLEU points on the En-De and En-Cs tasks, respectively, and our model surpasses previous best models by 4.14 and 2.19 BLEU points. In syntax-based NMT where the dataset is larger than AMR-to-text generation, the  $\mathrm{HetGT}_{\mathrm{dot-product}}$  gets comparable results compared to the  $\mathrm{HetGT}_{\mathrm{additive}}$ , and even outperforms the  $\mathrm{HetGT}_{\mathrm{additive}}$  in terms of METEOR and CHRF++ on the language pair En-De. We think on the larger datasets the  $\mathrm{HetGT}_{\mathrm{dot-product}}$  will get better results than the  $\mathrm{HetGT}_{\mathrm{additive}}$ .
4 Additional Experiments
4.1 Effect of Layer Aggregation Method
Firstly, we compare the performance of three layer-aggregation methods discussed in Section 2.3.3.
Method	HetGTdot-product	HetGTadditive
Residual	30.02	30.56
Jump	31.29	31.84
Dense	29.92	30.41
Table 4: Results of different layer aggregation methods on the test set of AMR15.
BLEU	METEOR
Full Model	31.84	36.89
w/ only fully-connected subgraph	25.69	33.88
w/ only connected subgraph	30.22	36.32
w/ only default subgraph	30.47	36.34
w/ only reverse subgraph	29.76	35.97
w/o fully-connected subgraph	30.84	36.35
w/o connected subgraph	31.62	36.75
w/o default subgraph	29.68	35.88
w/o reverse subgraph	29.86	36.03
w/o BPE	29.84	35.52
Table 5: Ablation results on the AMR15 test set.
The results are shown in Table 4. We can see the jump connection is the most effective method. However, the dense connection performs the worst. We think the reason is that dense connection introduce lots of extra parameters which are harder to learn.
4.2 Effect of Subgraphs
In this section, we also use AMR15 as our benchmark to investigate how each subgraph influences the final results of our best model  $\mathrm{HetGT}_{\mathrm{additive}}$ . Table 5 shows the results of removing or only keeping the specific subgraph. Only keeping the fully-connected subgraph essentially is what the Transformer baseline does. It means the model does not consider the inherent structural information in inputs. Obviously, it cannot get a good result. In addition, only keeping the connected subgraph does not perform well even it considers the structural information. It demonstrates that the heterogeneous information in the graph is helpful for learning the representation of the graph. When removing any subgraph, the performance of the model will decrease. It demonstrates that each subgraph has contributed to the final results. At last, we remove BPE, and we get 29.84 BLEU score which is still better than previous SoTA that also uses BPE. Note that when we remove the connected subgraph, the results do not have statistically significant changes  $(p = 0.293)$ . We think the reason is that the left subgraphs already contain the full information of the original graph because the connected subgraph is obtained by grouping all edge types into a single one. Except that, all the other results have statistically significant changes  $(p \leq 0.05)$ .
(p / possible-01 e.1 :polarity e.2 - e.2
:ARG1 (w / work-01 e.3,4
:ARG0 (i / i e.0)
:location e.5 (h / home e.6))
:ARG1-of (c / cause-01 e.8
:ARG0 (s / shout-01 e.10
:ARG0 (s2 / she e.9)
:ARG2 e.11 i e.12))))
REF: i can n't do work at home, because she shouts at me.
nsformer: i can n't do work at home , because she shouts at me .
$\mathbf{G}\mathbf{T}_{\mathrm{additive}}$  (ours): i can n't do work at home, because she shouts at me.
(s/say-01e.1
:ARG0 (h2 / he e.0)
:ARG1 e.2 (a / agree-01 e.4
:ARG0 h2 e.3
:ARG1 e.5 (o / opine-01 e.9
ARG0 e.8 (p2 / person :wiki "Liu_Huaqing" :name (n / name :op1 "Huaqing" e.6 :op2 "Liu" e.7))
:ARG1 e.10 (r / recommend-01 e.14
:ARG1 (d / develop-02 e.16
:ARG0 (a2 / and e.12
:op1 (c4 / country :wiki "Thailand"
:name (n2 / name :op1 "Thailand" e.11))
:op2 (c5 / country :wiki "China"
name (n3 / name :op1 "China" e.13)))
:ARG1 (a3 / and e.21
:op1 (c6 / cooperate-01 e.23
ARG2 (e/economy e.20)
:mod e.19 (f / form e.18
:mod(v/variouse.17)))
op2 (c7 / cooperate-01 e.23
ARG2 (t2 / trade-01 e.22)
:modf))
:degree (f2 / further e.15))))
REF: he said that he agreed with huaqing liu's opinion that thailand and china should further develop various forms of economic and trade cooperation.
Transformer: he said huaqing liu agreed to agree with thailand and china should further develop in various forms of economic cooperation and trade cooperation.
HetGTadditive (ours): he said he agreed to huaqing liu's opinion that thailand and china should further develop various forms of economic cooperation and trade cooperation.
Table 6: Example outputs of different systems are compared, including Transformer baseline and our HetGT.
We perform case studies for better understanding the model performance. We compare the outputs of Transformer baseline and our  $\mathrm{HetGT}_{\mathrm{additive}}$ . The results are presented in Table 6. In the first simple example, our Transformer baseline and  $\mathrm{HetGT}_{\mathrm{additive}}$  can generate the target sequence without mistakes. In the second example which is more complicated, the Transformer baseline fails to identify the possessor of "opinion" and the subject of "agreed" while our model successfully recognizes them. However, we find the there is a common problem: the sentences they generate all have some duplication. We will explore this issue further in the future work.
Early researches for Graph2Seq learning tasks are based on statistical methods and neural seq2seq
model. Lu et al. (2009) propose an NLG approach built on top of tree conditional random fields to use the tree-structured meaning representation. Song et al. (2017) use synchronous node replacement grammar to generate text. Konstas et al. (2017) linearize the input graph and feed it to the seq2seq model for text-to-AMR parsing and AMR-to-text generation. However, linearizing AMR graphs into sequences may incurs in loss of information. Recent efforts consider to capture the structural information in the encoder. Beck et al. (2018) employ Gated Graph Neural Networks (GGNN) as the encoder and Song et al. (2018) propose the graph-state LSTM to incorporate the graph structure. Their works belong to the family of recurrent neural network (RNN). In addition, there are some works are build upon the GNN. Damonte and Cohen (2019) propose stacking encoders including LSTM and GCN. Guo et al. (2019) introduce the densely connected GCN to encode richer local and non-local information for better graph representation.
Recent studies also extend Transformer to encode structure information. Shaw et al. (2018) propose the relation-aware self-attention which learns explicit embeddings for pair-wise relationships between input elements. Zhu et al. (2019) and Cai and Lam (2020) both extend the relation-aware self-attention to generate text from AMR graph. Our model is also based on Transformer. However, we do not employ the relative position encoding to incorporate structural information. Instead, we directly mask the non-neighbor nodes attention when updating each node representation. Moreover, we introduce the heterogeneous information and jump connection to help model learn a better graph representation, bringing substantial gains in the model performance.
In this paper, we propose the Heterogeneous Graph Transformer (HetGT) for Graph2Seq learning. Our proposed heterogeneous mechanism can adaptively model the different representation subgraphs. Experimental results show that HetGT strongly outperforms the state of the art performances on four benchmark datasets of AMR-to-text generation and syntax-based neural machine translation tasks.
There are two directions for future works. One is to investigate how the other graph models can benefit from our proposed heterogeneous mecha
7152
nism. On the other hand, we would also like to investigate how to make use of our proposed model to solve sequence-to-sequence tasks.
This work was supported by National Natural Science Foundation of China (61772036) and Key Laboratory of Science, Technology and Standard in Press Industry (Key Laboratory of Intelligent Press Media Technology). We thank the anonymous reviewers for their helpful comments. Xiaojun Wan is the corresponding author.
Chris Alberti, Daniel Andor, Ivan Bogatyy, Michael Collins, Daniel Gillick, Lingpeng Kong, Terry K Koo, Ji Ma, Mark Omernick, Slav Petrov, Chayut Thanapirom, Zora Tung, and David Weiss. 2017. Syntaxnet models for the conll 2017 shared task. ArXiv, abs/1703.04929.
Joost Bastings, Ivan Titov, Wilker Aziz, Diego Marcheggiani, and Khalil Sima'an. 2017. Graph convolutional encoders for syntax-aware neural machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1957-1967, Copenhagen, Denmark. Association for Computational Linguistics.
Daniel Beck, Gholamreza Haffari, and Trevor Cohn. 2018. Graph-to-sequence learning using gated graph neural networks. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 273-283, Melbourne, Australia. Association for Computational Linguistics.
Deng Cai and Wai Lam. 2020. Graph transformer for graph-to-sequence learning. In Proceedings of The Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI).
Benson Chen, Regina Barzilay, and Tommi Jaakkola. 2019. Path-augmented graph transformer network. arXiv preprint arXiv:1905.12712.
Marco Damonte and Shay B. Cohen. 2019. Structural neural encoders for AMR-to-text generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3649-3658, Minneapolis, Minnesota. Association for Computational Linguistics.
Michael Denkowski and Alon Lavie. 2014. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 376-380, Baltimore, Maryland, USA. Association for Computational Linguistics.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. 2014. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 580-587.
Zhijiang Guo and Wei Lu. 2018. Better transition-based AMR parsing with a refined search space. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1712-1722, Brussels, Belgium. Association for Computational Linguistics.
Zhijiang Guo, Yan Zhang, Zhiyang Teng, and Wei Lu. 2019. Densely connected graph convolutional networks for graph-to-sequence learning. Transactions of the Association for Computational Linguistics, 7:297-312.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Identity mappings in deep residual networks. In European conference on computer vision, pages 630-645. Springer.
G. Huang, Z. Liu, L. v. d. Maaten, and K. Q. Weinberger. 2017. Densely connected convolutional networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2261-2269.
Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M. Rush. 2017. OpenNMT: Open-source toolkit for neural machine translation. In Proc. ACL.
Ioannis Konstas, Srinivasan Iyer, Mark Yatskar, Yejin Choi, and Luke Zettlemoyer. 2017. Neural AMR: Sequence-to-sequence models for parsing and generation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 146-157, Vancouver, Canada. Association for Computational Linguistics.
Wei Lu, Hwee Tou Ng, and Wee Sun Lee. 2009. Natural language generation with tree conditional random fields. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 400-409, Singapore. Association for Computational Linguistics.
Graham Neubig, Zi-Yi Dou, Junjie Hu, Paul Michel, Danish Pruthi, Xinyi Wang, and John Wieting. 2019. compare-mt: A tool for holistic comparison of language generation systems. CoRR, abs/1903.07926.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Maja Popovic. 2015. chrF: character n-gram f-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation,
7153
pages 392-395, Lisbon, Portugal. Association for Computational Linguistics.
Michael Pust, Ulf Hermjakob, Kevin Knight, Daniel Marcu, and Jonathan May. 2015. Parsing English into abstract meaning representation using syntax-based machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1143-1154, Lisbon, Portugal. Association for Computational Linguistics.
Leonardo F. R. Ribeiro, Claire Gardent, and Iryna Gurevych. 2019. Enhancing AMR-to-text generation with dual graph representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3181-3192, Hong Kong, China. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725, Berlin, Germany. Association for Computational Linguistics.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with relative position representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 464-468, New Orleans, Louisiana. Association for Computational Linguistics.
Linfeng Song, Xiaochang Peng, Yue Zhang, Zhiguo Wang, and Daniel Gildea. 2017. AMR-to-text generation with synchronous node replacement grammar. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 7-13, Vancouver, Canada. Association for Computational Linguistics.
Linfeng Song, Yue Zhang, Zhiguo Wang, and Daniel Gildea. 2018. A graph-to-sequence model for AMR-to-text generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1616-1626, Melbourne, Australia. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30, pages 5998-6008. Curran Associates, Inc.
Petar Velicković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2018. Graph Attention Networks. International Conference on Learning Representations.
Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken ichi Kawarabayashi, and Stefanie Jegelka. 2018a. Representation learning on graphs with jumping knowledge networks. In Proceedings of the 35th International Conference on Machine Learning.
Kun Xu, Lingfei Wu, Zhiguo Wang, Yansong Feng, Michael Witbrock, and Vadim Sheinin. 2018b. Graph2seq: Graph to sequence learning with attention-based neural networks. arXiv preprint arXiv:1804.00823.
Jie Zhu, Junhui Li, Muhua Zhu, Longhua Qian, Min Zhang, and Guodong Zhou. 2019. Modeling graph structure in transformer for better AMR-to-text generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5458-5467, Hong Kong, China. Association for Computational Linguistics.
7154