SlowGuess/ACL_2020_OCR huggingface dataset:
Figure 5: BLEU variation between models with different orders  $K$  with respect to AMR graph size.
As mentioned above, if only consider the first-order neighborhood, the dependencies between distant AMR concepts cannot be fully explored when the graph size becomes larger. To verify this hypothesis, we split the test set into different parts according to the AMR graph size (i.e. number of concepts). We evaluate our models with order
Figure 6: BLEU variation between models with different  $K_{e}$  with respect to size of AMR graph and (left) and reentrancy numbers (right).
$K = 4$  and  $K = 1$  on different partitions. All models are trained on LDC2015E86 set. Figure 5 shows the result. The model with  $K = 4$  significantly outperforms the one with  $K = 1$ . Furthermore, we can find that the performance gap between the two models increases when the graph gets bigger. As a result, higher-order neighborhood information does play an important role in graph-to-sequence generation, especially for larger AMR graphs.
5.2 Ablation Study on Relationships of Labeled Edges
We are the first one to consider the relationships between labeled edges in AMR graph by integrating the line graph (relation graph)  $G_{e}$  in our system. This section will deeply analyze the effectiveness of this contribution. In previous settings, the graph neighborhood order  $K$  is the same for both  $G_{c}$  and  $G_{e}$ . To conduct the ablation test, we fix the neighborhood order  $K_{c}$  for  $G_{c}$  and vary the order  $K_{e}$  for relation graph  $G_{e}$ . We set  $K_{e} = 0, 1$  and 4, where  $K_{e} = 0$  indicates that the relation nodes in  $G_{e}$  can only interact with itself. This means the dependencies between AMR edges are completely ignored, and the edge information is simply combined with the corresponding concepts. We report the results on both test sets in Table 3.
LDC2015E86	LDC2017T10
Models	BLEU	Meteor	BLEU	Meteor
Kc=4, Ke=0	28.89*	35.00	31.08*	36.11
Kc=4, Ke=1	29.41*	35.29	31.35*	36.18
Kc=4, Ke=4	30.58	35.81	32.46	36.78
Table 3: Results of models with varied neighborhood orders of relation graph  $G_{e}$ . BLEU scores significantly different from the best model is marked with * ( $p < 0.01$ ), tested by bootstrap resampling (Koehn, 2004).
If we ignore the dependencies between AMR
edges ( $K_{e} = 0$ ), there is a significant performance degradation: 1.69 and 1.38 BLEU score decline on LDC2015E86 and LDC2017T10 respectively. The performance gets better when  $K_{e} > 0$ , which means the edge relations do bring benefits to the graph encoding and sequence generation. When  $K_{e} = 4$ , the edge relations are fully explored in varied neighborhood orders, and it reaches the best performance on both datasets. Performance test on different partitions of AMR graph size (Figure 6, left) also suggests that relationships of edges are helpful when the graph becomes larger.
We also study the effectiveness of edge relations when handling reentrancies. Reentrancies are the nodes with multiple parents. Such structures are identified as very difficult aspects in AMR graph (Damonte and Cohen, 2019). We think the relation graph  $G_{e}$  is helpful in exploring different dependencies with the same concept, which can bring benefits to those graphs containing more reentrancies. To test this hypothesis, we also split the test set into different parts according to their numbers of reentrancies and evaluate our models with  $K_{e} = 4$  and  $K_{e} = 0$  on different partitions. As shown in Figure 6 (right), the gap becomes wide when the number of reentrancies grows to 5. Also, compare to the graph size, edge relations are more important in handling graphs with reentrancies.
To gain insight into the model performance. Table 4 provides a few examples. The reentrancies in the AMR graphs is marked with bold type.
In Example (a), two different nodes have same concept - compete, but they have different forms in the corresponding natural language. According to the references, one is for "competitors" and the other is for "competition". Our model with  $K_{e} = 0$  fails to distinguish the difference and generate two
"competition" in the output. However, model with  $K_{e} = 4$  successfully recover word "competitors" from the context of the AMR graph.
In Example (b), the concept they has two parents with the same concept - want. Though our model with  $K_{e} = 0$  successfully finds they is the subject of the both two want, it fails to recognize the parallel relationship between the objects money and face and regard face as a verb. In the contrast, our model with  $K_{e} = 4$  perfectly finds the parallel structure in the AMR graph and
reconstructs the correct sentence.
In Example (c), we compare our best model with two baselines: GCNSEQ (Damonte and Cohen, 2019) and Structural Transformer (Denote as ST-Transformer) from Zhu et al. (2019). The AMR graph in Example (b) has two reentrancies, which makes it more difficult to recover the corresponding sentence. As we can see, traditional graph-based model GCNSEQ cannot predict the correct subject of the predicate can. Structural-Transformer uses the correct subject, but the recovered sentence is quite disfluent because of the redundant people. This overgeneration problem is mainly caused by reentrancies (Beck et al., 2018). However, our model can effectively handle this problem and generates a proper sentence with correct semantics.
AMR-to-text generation is a typical graph-to-sequence task. Early research employs rule-based methods to deal with this problem. Flanigan et al. (2016) use two-stage method by first split the graphs into spanning trees and use multiple tree transducers to generate natural language. Song et al. (2017) use heuristic extraction algorithm to learn graph-to-string rules. More works frame graph-to-sequence as a translation task and use either phrase-based (Ferreira et al., 2017; Pourdamghani et al., 2016) or neural-based (Konstas et al., 2017) models. These methods usually need to linearize the input graphs by means of a depth-first traversal. Cao and Clark (2019) get a better sequence-based model by leveraging additional syntactic information.
Moving to graph-to-sequence approaches, Marcheggiani and Perez-Beltrachini (2018) first show that graph neural networks can significantly improve the generation performance by explicitly encoding the structure of the graph. Since then, models with variant graph encoders have been proposed in recent years, such as graph LSTM (Song et al., 2018), gated graph neural networks (GGNN) (Beck et al., 2018) and graph convolutional neural networks (Damonte and Cohen, 2019). Guo et al. (2019) introduce dense connectivity to allow the information exchange across different layers. Ribeiro et al. (2019) learn dual representations capturing top-down and bottom-up adjuvant view of the graph, and reach the best performance in graph-based models.
Despite the great success of graph neural net
works, they all restrict the update of node representation based on only first-order neighborhood and rely on stacked layers to model the relationships between indirectly connected nodes. To solve this problem, recent studies extend the Transformer (Vaswani et al., 2017) to encode the graph structure. Zhu et al. (2019) and Cai and Lam (2019) use relation-aware self-attention to encode structural label sequences of concept pairs, which can model arbitrary concept pairs no matter whether directly connected or not. With several mechanisms such as sub-word (Sennrich et al., 2016) and shared vocabulary, Zhu et al. (2019) achieved state-of-the-art performance on this task.
Our model follows the same spirit of exploring the relations between indirectly connected nodes, but our method is substantially different: (1) we use a graph-based method integrated with higher-order neighborhood information while keeping the explicit structure of graphs. (2) we first consider the relations between labeled edges by introducing line graphs.
In this work, we presented a novel graph-to-sequence approach which uses line graph to model the relationships between labeled edges from the original AMR graph. The mix-order graph attention networks are found effective when handling indirectly connected nodes. The ablation studies also demonstrate that exploring edge relations brings benefits to graph-to-sequence modeling. Furthermore, our framework can be efficiently applied to other graph-to-sequence tasks such as WebNLG (Gardent et al., 2017) and syntax-based neural machine translation (Bastings et al., 2017). In future work we would like to do several experiments on other related tasks to test the versatility of our framework. Also, we plan to use large-scale unlabeled data to improve the performance further.
We thank the anonymous reviewers for their thoughtful comments. This work has been supported by the National Key Research and Development Program of China (Grant No. 2017YFB1002102) and Shanghai Jiao Tong University Scientific and Technological Innovation Funds (YG2020YQ01).
Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffith, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In LAW@ACL, pages 178-186. The Association for Computer Linguistics.
Satanjeev Banerjee and Alon Lavie. 2005. METEOR: an automatic metric for MT evaluation with improved correlation with human judgments. In IEEvaluation@ACL, pages 65-72. Association for Computational Linguistics.
Joost Bastings, Ivan Titov, Wilker Aziz, Diego Marcheggiani, and Khalil Sima'an. 2017. Graph convolutional encoders for syntax-aware neural machine translation. In EMNLP, pages 1957-1967. Association for Computational Linguistics.
Daniel Beck, Gholamreza Haffari, and Trevor Cohn. 2018. Graph-to-sequence learning using gated graph neural networks. In ACL (1), pages 273-283. Association for Computational Linguistics.
Deng Cai and Wai Lam. 2019. Graph transformer for graph-to-sequence learning. arXiv preprint arXiv:1911.07470.
Kris Cao and Stephen Clark. 2019. Factorising AMR generation through syntax. In *NAACL-HLT* (1), pages 2157-2163. Association for Computational Linguistics.
Marco Damonte and Shay B. Cohen. 2019. Structural neural encoders for amr-to-text generation. CoRR, abs/1903.11410.
Thiago Castro Ferreira, Iacer Calixto, Sander Wubben, and Emiel Krahmer. 2017. Linguistic realisation as machine translation: Comparing different MT models for amr-to-text generation. In INLG, pages 1-10. Association for Computational Linguistics.
Jeffrey Flanigan, Chris Dyer, Noah A. Smith, and Jaime G. Carbonell. 2016. Generation from abstract meaning representation using tree transducers. In HLT-NAACL, pages 731-739. The Association for Computational Linguistics.
Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017. The webnlg challenge: Generating text from RDF data. In INLG, pages 124-133. Association for Computational Linguistics.
Zhijiang Guo, Yan Zhang, Zhiyang Teng, and Wei Lu. 2019. Densely connected graph convolutional networks for graph-to-sequence learning. TACL, 7:297-312.
Frank Harary and Robert Z Norman. 1960. Some properties of line digraphs. Rendiconti del Circolo Matematico di Palermo, 9(2):161-168.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR.
Guillaume Klein, Yoon Kim, Yuntian Deng, Vincent Nguyen, Jean Senellart, and Alexander M. Rush. 2018. Opennmt: Neural machine translation toolkit. In AMTA (1), pages 177-184. Association for Machine Translation in the Americas.
Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In EMNLP, pages 388-395. Association for Computational Linguistics.
Ioannis Konstas, Srinivasan Iyer, Mark Yatskar, Yejin Choi, and Luke Zettlemoyer. 2017. Neural AMR: sequence-to-sequence models for parsing and generation. In ACL (1), pages 146-157. Association for Computational Linguistics.
Kexin Liao, Logan Lebanoff, and Fei Liu. 2018. Abstract meaning representation for multi-document summarization. In COLING, pages 1178-1190. Association for Computational Linguistics.
Diego Marcheggiani and Laura Perez-Beltrachini. 2018. Deep graph convolutional encoders for structured data to text generation. In INLG, pages 1-9. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In ACL, pages 311-318. ACL.
Nima Pourdamghani, Kevin Knight, and Ulf Hermjakob. 2016. Generating english from abstract meaning representations. In INLG, pages 21-25. The Association for Computer Linguistics.
Leonardo F. R. Ribeiro, Claire Gardent, and Iryna Gurevych. 2019. Enhancing AMR-to-text generation with dual graph representations. In EMNLP-IJCNLP. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In ACL. The Association for Computer Linguistics.
Linfeng Song, Daniel Gildea, Yue Zhang, Zhiguo Wang, and Jinsong Su. 2019. Semantic neural machine translation using AMR. TACL, 7:19-31.
Linfeng Song, Xiaochang Peng, Yue Zhang, Zhiguo Wang, and Daniel Gildea. 2017. Amr-to-text generation with synchronous node replacement grammar. In ACL (2), pages 7-13. Association for Computational Linguistics.
Linfeng Song, Yue Zhang, Zhiguo Wang, and Daniel Gildea. 2018. A graph-to-sequence model for amr-to-text generation. In ACL (1), pages 1616-1626. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS, pages 5998-6008.
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2018. Graph attention networks. In ICLR. Open-Review.net.
Jie Zhu, Junhui Li, Muhua Zhu, Longhua Qian, Min Zhang, and Guodong Zhou. 2019. Modeling graph structure in transformer for better AMR-to-text generation. In EMNLP-IJCNLP. Association for Computational Linguistics.