SlowGuess/ACL_2020_OCR huggingface dataset:
Yanbin Zhao, Lu Chen, Zhi Chen, Ruisheng Cao, Su Zhu, Kai Yu*
MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University SpeechLab, Department of Computer Science and Engineering
{zhaoyb, chenlusz, zhenchi713}@sjtu.edu.cn
{211314, paul2204, kai.yu}@sjtu.edu.cn
Efficient structure encoding for graphs with labeled edges is an important yet challenging point in many graph-based models. This work focuses on AMR-to-text generation – A graph-to-sequence task aiming to recover natural language from Abstract Meaning Representations (AMR). Existing graph-to-sequence approaches generally utilize graph neural networks as their encoders, which have two limitations: 1) The message propagation process in AMR graphs is only guided by the first-order adjacency information. 2) The relationships between labeled edges are not fully considered. In this work, we propose a novel graph encoding framework which can effectively explore the edge relations. We also adopt graph attention networks with higher-order neighborhood information to encode the rich structure in AMR graphs. Experiment results show that our approach obtains new state-of-the-art performance on English AMR benchmark datasets. The ablation analyses also demonstrate that both edge relations and higher-order information are beneficial to graph-to-sequence modeling.
Abstract Meaning Representation (Banarescu et al., 2013) is a sentence-level semantic representation formalized by a rooted directed graph, where nodes are concepts and edges are semantic relations. Since AMR is a highly structured meaning representation, it can promote many semantic related tasks such as machine translation (Song et al., 2019) and summarization (Liao et al., 2018). However, the usage of AMR graphs can be challenging, since it is non-trivial to completely capture the rich structural information in the graph-based data, especially when the graph has labeled edges.
Figure 1: An AMR graph (left) for sentence "He runs as fast as the wind." and its concept graph and relation graph (line graph). Two graphs are aligned with each other based on the node-edge relations in the original graph.
Generation from AMR aims to translate the AMR semantics into the surface form (natural language). It is a basic Graph-to-sequence task that directly takes AMR as input. Figure 1 (left) gives a standard AMR graph and its corresponding surface form. Early works utilize sequence-to-sequence framework by linearizing the entire graph (Konstas et al., 2017; Cao and Clark, 2019). Such representation may lose useful structural information. In recent studies, graph neural networks (GNNs) have been in a dominant position on this task and achieved state-of-the-art performance (Beck et al., 2018; Song et al., 2018; Guo et al., 2019; Damonte and Cohen, 2019). However, In these GNN-based models, the representation of each concept node is only updated by the aggregated information from its neighbors, which leads to two limitations: 1) The interaction between indirectly connected nodes heavily relies on the number of stacked layers. When the graph size becomes larger, the dependencies between distant AMR concepts cannot be fully explored. 2) They only focus on modeling the relations between concepts while ignoring edge relations and their structures. Zhu et al. (2019)
*Kai Yu is the corresponding author.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 732-741 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
and Cai and Lam (2019) use Transformer to model arbitrary concept pairs no matter whether directly connected or not, but they still ignore the topological structures of the edges in the entire AMR graph.
To address the above limitations, we propose a novel graph-to-sequence model based on graph attention networks (Velickovic et al., 2018). We transform the edge labels into relation nodes and construct a new graph that directly reflects the edge relations. In graph theory, such a graph is called a Line Graph (Harary and Norman, 1960). As illustrated in Figure 1, we thus separate the original AMR graph into two sub-graphs without labeled edges - concept graph and relation graph. The two graphs describe the dependencies of AMR concepts and edges respectively, which is helpful in modeling these relationships (especially for edges). Our model takes these sub-graphs as inputs, and the communications between the two graphs are based on the attention mechanism. Furthermore, for both graphs, we mix the higher-order neighborhood information into the corresponding graph encoders in order to model the relationships between indirectly connected nodes.
Empirical study on two English benchmark datasets shows that our model reaches state-of-the-art performance with 30.58 and 32.46 BLEU scores on LDC2015E86 and LDC2017T10, respectively. In summary, our contributions include:
- We propose a novel graph-to-sequence model, which firstly uses the line graph to model the relationships between AMR edges.
- We integrate higher-order neighborhood information into graph encoders to model the relationships between indirectly connected nodes.
- We demonstrate that both higher-order neighborhood information and edge relations are important to graph-to-sequence modeling.
2 Mix-Order Graph Attention Networks
In this section, we first introduce graph attention networks (GATs) and their mix-order extensions, which are the basis of our proposed model.
2.1 Graph Attention Networks
GAT is a special type of networks that operates on graph-structured data with attention mechanisms. Given a graph  $G = (V, E)$ , where  $V$  and  $E$  are
Figure 2: Neighborhood information in different orders.
the set of nodes  $x_{i}$  and the set of edges  $(e_{ij},\ell_e)^1$ , respectively.  $\mathcal{N}(x_i)$  denote the nodes which are directly connected by  $x_{i}$ .  $\mathcal{N}_{+}(x_{i})$  is the set including  $x_{i}$  and all its direct neighbors. we have  $\mathcal{N}_{+}(x_{i}) = \mathcal{N}(x_{i})\cup \{x_{i}\}$
Each node  $x_{i}$  in the graph has an initial feature  $\mathbf{h}_i^0 \in \mathbb{R}^d$ , where  $d$  is the feature dimension. The representation of each node is iteratively updated by the graph attention operation. At the  $l$ -th step, each node  $x_{i}$  aggregates context information by attending over its neighbors and itself. The updated representation  $\mathbf{h}_i^l$  is calculated by the weighted average of the connected nodes:
$$
\mathbf {h} _ {i} ^ {l} = \sigma \left(\sum_ {x _ {j} \in \mathcal {N} _ {+} (x _ {i})} \alpha_ {i j} \mathbf {h} _ {j} ^ {l - 1} \mathbf {W} ^ {l}\right), \tag {1}
$$
where attention coefficient  $\alpha_{ij}$  is calculated as:
$$
\alpha_ {i j} = \operatorname {s o f t m a x} _ {j} \left(\mathbf {h} _ {i} ^ {l - 1} \mathbf {W} _ {t 1} ^ {l}\right) \left(\mathbf {h} _ {j} ^ {l - 1} \mathbf {W} _ {t 2} ^ {l}\right) ^ {T} \tag {2}
$$
where  $\sigma$  is a nonlinear activation function, e.g. ReLU.  $\mathbf{W}^l$ ,  $\mathbf{W}_{t1}^l$  and  $\mathbf{W}_{t2}^l \in \mathbb{R}^{d \times d}$  are learnable parameters for projections. After L steps, each node will finally have a context-aware representation  $\mathbf{h}_i^L$ . In order to achieve a stable training process, we also employ a residual connection followed by layer normalization between two graph attention layers.
2.2 Mixing Higher Order Information
The relations between indirectly connected nodes are ignored in a traditional graph attention layer. Mix-Order GAT, however, can explore these relationships in a single-step operation by mixing the higher-order neighborhood information. We first give some notations before describing the details of the Mix-Order GAT. We use  $\mathbf{R}^K = \{\mathcal{R}^1,\dots \mathcal{R}^K\}$  to represent neighborhood information from order
${}^{1}{\ell }_{e}$  is the edge label which are not considered in the GAT layer
Figure 3: An overview of our proposed model
1 to order  $K$ .  $\mathcal{R}^k (x_i)$  denotes the  $k$ -th order neighborhood, which means all nodes in  $\mathcal{R}^k (x_i)$  are reachable for  $x_{i}$  within  $k$  hops ( $k\geq 1$ ).  $\mathcal{R}^1 (x_i) = \mathcal{N}_+(x_i)$ , and as illustrated in Figure 2, we can have:
$$
\mathcal {R} ^ {k} \left(x _ {i}\right) = \bigcup_ {x _ {j} \in \mathcal {R} ^ {k - 1} \left(x _ {i}\right)} \mathcal {N} _ {+} \left(x _ {j}\right). \tag {3}
$$
The  $K$ -Mix GAT integrates the neighborhood information  $\mathbf{R}^K$ . At the  $l$ -th update step, each  $x_{i}$  will interact with its reachable neighbors with different orders and calculate the attentive features independently. The representation  $\mathbf{h}_i^l$  is updated by the concatenated features from different orders, i.e.
$$
\begin{array}{l} \mathbf {h} _ {i} ^ {l} = \operatorname {M i x G A T} ^ {l} \left(\mathbf {h} _ {i} ^ {l - 1}, \mathbf {R} ^ {K}\right) \\ = \left\| _ {k = 1} ^ {K} \sigma \left(\sum_ {x _ {j} \in \mathcal {R} ^ {k} \left(x _ {i}\right)} \alpha_ {i j} ^ {k} \mathbf {h} _ {j} ^ {l - 1} \mathbf {W} _ {k} ^ {l}\right), \right. \tag {4} \\ \end{array}
$$
where  $||$  represents concatenation,  $\alpha_{ij}^{k}$  are the attention weights in the  $k$ -th order, and  $\mathbf{W}_k^l \in \mathbb{R}^{d \times d / K}$  are learnable weights for projections. We will use  $\mathrm{MixGAT}(\cdot)$  to denote the Mix-Order GAT layer in the following section.
The architecture of our method is illustrated in Figure 3. As mentioned above, we separate the AMR graph into two sub-graphs without labeled edges. Our model follows the Encoder-Decoder architecture, where the encoder takes the two sub-graphs as inputs, and the decoder generates corresponding text from the encoded information. We first give
some detailed explanations about the line graph and input representation.
3.1 Line Graph & Input Representation
The line graph of a graph  $G$  is another graph  $L(G)$  that represents the adjacencies between edges of  $G$ .  $L(G)$  is defined as:
Each node of  $L(G)$  represents an edge of  $G$
- Two nodes of  $L(G)$  are adjacent if and only if their corresponding edges share a common node in  $G$ .
For directed graphs, the directions are maintained in the corresponding line graphs. Redundant edges between two relation nodes are removed in the line graphs. Figure 4 provides several examples.
In our model, we use the line graph to organize labeled edges and transform the original AMR graph into two sub-graphs. Given an AMR graph  $G_{a} = (V_{a},E_{a})$ , we separate it into concept graph  $G_{c} = (V_{c},E_{c})$  and relation graph  $G_{e} = (V_{e},E_{e})$  where  $G_{e} = L(G_{a})$ . As for concept graph  $G_{c}$ , its topological structure is the same with  $G_{a}$ , but the edge labels are eliminated, i.e.
$$
V _ {c} = V _ {a}; \quad E _ {c} = \hat {E} _ {a}, \tag {5}
$$
Where  $\hat{E}_a$  is the edge set without label information. Both  $G_{c}$  and  $G_{e}$  have no labeled edges, which can be efficiently encoded by Mix-Order GAT.
We use  $\mathbf{R}_c^K$  and  $\mathbf{R}_e^K$  to denote  $1\sim K$  orders neighborhood information of  $G_{c}$  and  $G_{e}$ . We represent each concept node  $x_{i}\in V_{c}$  with an initial embedding  $\mathbf{c}_i^0\in \mathbb{R}^d$ , and each relation node  $y_{i}\in V_{e}$
Figure 4: Examples of finding line graphs. In the left part,  $e1$  and  $e2$  have opposite directions, so each direction is maintained in the line graph. In the right part,  $e1$  and  $e2$  follow the same direction, so there is only one direction in the corresponding line graph.
with an embedding  $\mathbf{e}_i^0\in \mathbb{R}^d$  . The sets of node embeddings are denoted as  $\mathbf{C}^0 = \{\mathbf{c}_i^0\}_{i = 1}^m$  and  $\mathbf{E}^{0} = \{\mathbf{e}_{i}^{0}\}_{i = 1}^{n}$  , where  $m = |V_{c}|$  and  $n = |V_e|$  denote the numbers of concept nodes and relation nodes, respectively. Thus, the inputs of our system can be formulated by  $\mathbf{I} = \{\mathbf{C}^0,\mathbf{E}^0,\mathbf{R}_c^K,\mathbf{R}_e^K\}$
3.2 Self Updating
The encoder of our system consists of  $N$  stacked graph encoding layers. As illustrated in Figure 3, each graph encoding layer has two parts: self-updating for each graph and masked cross attention. For  $G_{c}$  and  $G_{e}$ , We use  $\mathbf{C}^{l - 1} = \{\mathbf{c}_i^{l - 1}\}_{i = 1}^m$  and  $\mathbf{E}^{l - 1} = \{\mathbf{e}_i^{l - 1}\}_{i = 1}^n$  to denote the input node embeddings of the  $l$ -th encoding layer. The representations of the two graphs are updated independently by mix-order graph attention networks (MixGAT). At the  $l$ -th step (layer), we have:
$$
\vec {\mathbf {C}} _ {s e l f} ^ {l} = \operatorname {M i x G A T} _ {c 1} ^ {l} \left(\mathbf {C} ^ {l - 1}, \mathbf {R} _ {c} ^ {K}\right), \tag {6}
$$
$$
\vec {\mathbf {E}} _ {s e l f} ^ {l} = \mathrm {M i x G A T} _ {e 1} ^ {l} (\mathbf {E} ^ {l - 1}, \mathbf {R} _ {e} ^ {K}).
$$
Where  $\vec{\mathbf{C}}_{self}^{l}$  and  $\vec{\mathbf{E}}_{self}^{l}$  are updated representations according to the mix-order neighborhood information  $\mathbf{R}_c^K$  and  $\mathbf{R}_e^K$ . One thing should be noticed is that both  $G_{c}$  and  $G_{e}$  are directed graphs. This implies that the information propagation in the graph is in a top-down manner, following the pre-specified direction. However, unidirectional propagation loses the structural information in the reversed direction. To build communication in both directions, we employ Dual Graph (Ribeiro et al., 2019). Dual graph has the same node representations but reversed edge directions compared to the original graph. For example, if edge  $\mathrm{A}\rightarrow \mathrm{B}$  is in the original graph, it turns to  $\mathtt{B}\to \mathtt{A}$  in the corresponding dual graph. Since dual graphs have the same node representations, we only need to change the neighborhood information. Denote  $\widetilde{G}_c$  and  $\widetilde{G}_e$  as the dual graph of  $G_{c}$  and  $G_{e}$ .  $\widetilde{\mathbf{R}}_c^K$  and  $\widetilde{\mathbf{R}}_e^K$  are
the corresponding neighborhood information. We have:
$$
\mathbf {\tilde {C}} _ {s e l f} ^ {l} = \operatorname {M i x G A T} _ {c 2} ^ {l} \left(\mathbf {C} ^ {l - 1}, \widetilde {\mathbf {R}} _ {c} ^ {K}\right), \tag {7}
$$
$$
\mathbf {\bar {E}} _ {s e l f} ^ {l} = \mathrm {M i x G A T} _ {e 2} ^ {l} (\mathbf {E} ^ {l - 1}, \mathbf {\widetilde {R}} _ {e} ^ {K}).
$$
Since we have updated the node embeddings in two directions, the final representations of the independent graph updating process are the combination of the bi-directional embeddings, i.e.
$$
\mathbf {C} _ {s e l f} ^ {l} = \left[ \vec {\mathbf {C}} _ {s e l f} ^ {l}; \vec {\mathbf {C}} _ {s e l f} ^ {l} \right] \mathbf {W} _ {c 1} ^ {l}, \tag {8}
$$
$$
\mathbf {E} _ {s e l f} ^ {l} = \left[ \vec {\mathbf {E}} _ {s e l f} ^ {l}; \tilde {\mathbf {E}} _ {s e l f} ^ {l} \right] \mathbf {W} _ {e 1} ^ {l},
$$
where  $\mathbf{W}_{c1}^{l}$  and  $\mathbf{W}_{e1}^{1} \in R^{2d \times d}$  are trainable matrix for projections.  $\mathbf{C}_{self}^{l} \in \mathbb{R}^{m \times d}$  and  $\mathbf{E}_{self}^{l} \in \mathbb{R}^{n \times d}$  are results of the self-updating process.
3.3 Masked Cross Attention
Self updating for  $G_{c}$  and  $G_{e}$  can model the relationships of AMR concepts and edge respectively. However, it is also necessary to explore the dependencies between concept nodes and relation nodes. As a result, the cross-graph communication between  $G_{c}$  and  $G_{e}$  is very important. From the structure of the original AMR graph, we can easily build alignment between  $G_{c}$  and  $G_{e}$ . A relation node  $y_{i}$  is directly aligned to a concept node  $x_{i}$  if  $x_{i}$  is the start-point/end-point of the edge corresponding to  $y_{i}$ . As illustrated in Figure 1, ARG0 is the edge between run-02 and he. As a result, node ARG0 in  $G_{e}$  is directly connect to run-02 and he in  $G_{c}$ .
We apply the attention mechanism to complete the interaction between the two graphs, and use  $\mathbf{M} \in \mathbb{R}^{n \times m}$  to mask the attention weights of unaligned pairs between  $G_{c}$  and  $G_{e}$ . For element  $m_{ij}$  in  $\mathbf{M}$ , we let  $m_{ij} = 0$  if  $y_{i} \in V_{e}$  is aligned to  $x_{j} \in V_{c}$ , otherwise  $m_{ij} = -\infty$ . The masked cross attention is employed between the representation sets  $\mathbf{E}_{self}^{l}$  and  $\mathbf{C}_{self}^{l}$ , and the matrix of attention weights  $\mathbf{A}_l$  can be calculated as:
$$
\mathbf {A} _ {l} = \left(\mathbf {E} _ {s e l f} ^ {l} \mathbf {W} _ {a 1} ^ {l}\right) \left(\mathbf {C} _ {s e l f} ^ {l} \mathbf {W} _ {a 2} ^ {l}\right) ^ {T} + \mathbf {M}, \tag {9}
$$
where  $\mathbf{W}_{a1}^{l}$  and  $\mathbf{W}_{a2}^{l} \in \mathbb{R}^{d \times d}$  are learnable projection matrixes. The weight scores of unaligned pairs are set to  $-\infty$  according to  $\mathbf{M}$ . For nodes in  $\mathbf{E}_{self}^{l}$ , the relevant representation from  $\mathbf{C}_{self}^{l}$  is identified using  $\mathbf{A}_l$  as:
$$
\mathbf {E} _ {\text {c r o s s}} ^ {l} = \operatorname {s o f t m a x} \left(\mathbf {A} _ {l}\right) \mathbf {C} _ {\text {s e l f}} ^ {l}, \tag {10}
$$
where  $\mathbf{E}_{cross}^{l}\in \mathbb{R}^{n\times d}$  is the masked weighted summation of  $\mathbf{C}_{self}^{l}$ . The same calculation is performed for nodes in  $\mathbf{C}_{self}^{l}$  as:
$$
\mathbf {C} _ {c r o s s} ^ {l} = \operatorname {s o f t m a x} \left(\mathbf {A} _ {l} ^ {T}\right) \mathbf {E} _ {s e l f} ^ {l}. \tag {11}
$$
The final outputs of a graph encoding layer are the combination of the original embeddings and the context representations from another graph. We also employ the outputs from previous layer as residual inputs, i.e.
$$
\begin{array}{l} \mathbf {C} ^ {l} = \operatorname {F F N} \left(\left[ \mathbf {C} _ {s e l f} ^ {l}; \mathbf {C} _ {c r o s s} ^ {l} \right] \mathbf {W} _ {c 2} ^ {l} + \mathbf {C} ^ {l - 1}\right), \\ \mathbf {E} ^ {l} = \operatorname {F F N} \left(\left[ \mathbf {E} _ {\text {s e l f}} ^ {l}; \mathbf {E} _ {\text {c r o s s}} ^ {l} \right] \mathbf {W} _ {e 2} ^ {l} + \mathbf {E} ^ {l - 1}\right), \tag {12} \\ \end{array}
$$
where FFN is a feed-forward network consists of two linear transformations. After  $N$ -stacked graph encoding layers, The two graphs  $G_{c}$  and  $G_{e}$  are finally encoded as  $\mathbf{C}^{N}$  and  $\mathbf{E}^{N}$ .
3.4 Decoder
The decoder of our system is similar to the Transformer decoder. At each generation step, the representation of the output token is updated by multiple rounds of attention with the previously-generated tokens and the encoder outputs. Note that the outputs of our graph encoder have two parts: concept representations  $\mathbf{C}^N$  and the relation representations  $\mathbf{E}^N$ . For generation, concept information is more important, since the concept graph directly contains the natural words. With the multi-step cross attention,  $\mathbf{C}^N$  also caries abundant relation information. For simplicity, we only use  $\mathbf{C}^N$  as the encoder output on the decoder side<sup>2</sup>.
To address the data sparsity issue in sequence generation, we employ the Byte Pair Encoding (BPE) (Sennrich et al., 2016) following the settings of Zhu et al. (2019). We split the word nodes in AMR graphs and reference sentences into subwords, and the decoder vocabulary is shared with the encoder for concept graphs.
Data and preprocessing We conduct our experiments with two benchmark datasets: LDC2015E85 and LDC2017T10. The two datasets contain
16833 and 36521 training samples, and they use a common development set with 1368 samples and a common test set with 1371 samples. We segment natural words in both AMR graphs and references into sub-words. As a result, a word node in AMR graphs may be divided into several sub-word nodes. We use a special edge subword to link the corresponding sub-word nodes. Then, for each AMR graph, we find its corresponding line graph and generate  $G_{c}$  and  $G_{e}$  respectively.
Training details For model parameters, the number of graph encoding layers is fixed to 6, and the representation dimension  $d$  is set to 512. We set the graph neighborhood order  $K = 1,2$  and 4 for both  $G_{c}$  and  $G_{e}$ . The Transformer decoder is based on Open-NMT (Klein et al., 2018), with 6 layers, 512 dimensions and 8 heads. We use Adam (Kingma and Ba, 2015) as our optimizer and  $\beta = (0.9,0.98)$ . The learning rate is varied over the course of training, similar with Vaswani et al. (2017):
$$
l r = \gamma d ^ {- 0. 5} \cdot \min  \left(t ^ {- 0. 5}, t * w ^ {- 1. 5}\right), \tag {13}
$$
where  $t$  denotes the accumulative training steps, and  $w$  indicates the warmup steps. We use  $w = 16000$  and the coefficient  $\gamma$  is set to 0.75. As for batch size, we use 80 for LDC2015E86 and 120 for LDC2017T10.
We compare our system with several baselines, including traditional sequence-to-sequence models, several graph-to-sequence models with multiple graph encoders, and transformer-based models. All models are trained on the single dataset without ensemble or additional unlabeled data. For performance evaluation, we use BLEU (Papineni et al., 2002) as our major metric. We also use Meteor (Banerjee and Lavie, 2005), which considers the synonyms between predicted sentences and references.
The experimental results on the test sets of LDC2015E86 and LDC2017T10 are reported in Table 1. As we can see, Sequence-based models perform the worst, since they lose useful structural information in graphs. Graph-based models get better results with varied graph encoders to capture the structural information in graphs.
3 Our code is available at https://github.com/ybz79/AMR2text
2We also implement a version which considers both  $\mathbf{C}^N$  and  $\mathbf{E}^N$ , and achieve similar results
Models	LDC2015E86	LDC2017T10
BLEU	Meteor	BLEU	Meteor
Sequence-Based Model
Seq2Seq (Konstas et al., 2017)	22.00	-	-	-
Syntax+S2S (Cao and Clark, 2019)	23.50	-	26.80	-
Graph-Based Model
Graph LSTM (Song et al., 2018)	23.30	-	-	-
GCNSEQ (Damonte and Cohen, 2019)	24.40	23.60	24.54	24.07
Dual Graph (Ribeiro et al., 2019)	24.32	30.53	27.87	33.21
DCGCN (Guo et al., 2019)	25.70	31.50	27.60	34.00
Transformer-Based Model
Transformer (Zhu et al., 2019)	25.50	33.16	27.43	34.62
Graph Transformer (Cai and Lam, 2019)	27.40	32.90	29.80	35.10
Structural Transformer (SA) (Zhu et al., 2019)	29.66	35.45	31.54	36.02
Our Approach
Line Graph + MixGAT, K = 1	28.64	34.51	29.96	35.15
Line Graph + MixGAT, K = 2	29.62	35.38	31.06	36.13
Line Graph + MixGAT, K = 4	30.58	35.81	32.46	36.78
Transformer-based models reach previous state-of-the-art with structure-aware self-attention approach to better modeling the relations between indirectly connected concepts. Comparing to previous studies, our approach with  $K = 4$  order neighborhood information reaches the best BLEU scores, improving over the state-of-the-art model (Zhu et al., 2019) by 0.92 on both datasets. Similar phenomena can be found on the additional metrics of Meteor.
As mentioned above, our system has two critical points: higher-order graph neighborhood information and relationships between AMR edges. To verify the effectiveness of these two settings, we conduct a series of ablation tests based on different characteristics of graphs.
Table 1: Main results of our approaches and several baselines on the test sets of LDC2015E86 and LDC2017T10
Orders	LDC2015E86	LDC2017T10
K=1	24.91%	31.03%
K=2	33.93%	40.71%
K=4	41.67%	48.30%
Table 2: The connectivity of the concept graphs under different orders.
5.1 Ablation Study on Neighborhood information
Higher order neighborhood information includes the relationships between indirectly connected nodes. Table 2 shows the connectivity of the con
cept graphs under different orders. When  $K = 1$ , each node can reach  $24.91\%$  of the other nodes directly in the graph (LDC2015E86), and it grows to  $41.67\%$  when  $K = 4$ .
As suggested in Table 1, if graph nodes only interact with their direct neighbors ( $K = 1$ ), it performs worse than previous Transformer-based models. However, significant improvement can be observed when we integrate higher-order neighborhood information. As  $K$  grows from 1 to 4, the BLEU score increases 1.94 and 2.50 on LDC2015E86 and LDC2017T10, respectively.