SlowGuess/ACL_2020_OCR huggingface dataset:
Shaowei Yao, Tianming Wang, Xiaojun Wan
Center for Data Science, Peking University
{yaosw, wangtm, wanxiaojun}@pku.edu.cn
The graph-to-sequence (Graph2Seq) learning aims to transduce graph-structured representations to word sequences for text generation. Recent studies propose various models to encode graph structure. However, most previous works ignore the indirect relations between distance nodes, or treat indirect relations and direct relations in the same way. In this paper, we propose the Heterogeneous Graph Transformer to independently model the different relations in the individual subgraphs of the original graph, including direct relations, indirect relations and multiple possible relations between nodes. Experimental results show that our model strongly outperforms the state of the art on all four standard benchmarks of AMR-to-text generation and syntax-based neural machine translation.
Graph-to-sequence (Graph2Seq) learning has attracted lots of attention in recent years. Many Natural Language Process (NLP) problems involve learning from not only sequential data but also more complex structured data, such as semantic graphs. For example, AMR-to-text generation is a task of generating text from Abstract Meaning Representation (AMR) graphs, where nodes denote semantic concepts and edges refer to relations between concepts (see Figure 1 (a)). In addition, it has been shown that even if the sequential input can be augmented by additional structural information, bringing benefits for some tasks, such as semantic parsing (Pust et al., 2015; Guo and Lu, 2018) and machine translation (Bastings et al., 2017). Therefore, Xu et al. (2018b) introduced the Graph2Seq problems which aim to generate target sequence from graph-structured data.
The main challenge for Graph2Seq learning is to build a powerful encoder which is able to cap
ture the inherent structure in the given graph and learn good representations for generating the target text. Early work relies on statistical methods or sequence-to-sequence (Seq2Seq) models where input graphs are linearized (Lu et al., 2009; Song et al., 2017; Konstas et al., 2017). Recent studies propose various models based on graph neural network (GNN) to encode graphs (Xu et al., 2018b; Beck et al., 2018; Guo et al., 2019; Damonte and Cohen, 2019; Ribeiro et al., 2019). However, these approaches only consider the relations between directly connected nodes, ignore the indirect relations between distance nodes. Inspired by the success of Transformer (Vaswani et al., 2017) which can learn the dependencies between all tokens without regard to their distance, the current state-of-the-art Graph2Seq models (Zhu et al., 2019; Cai and Lam, 2020) are based on Transformer and learn the relations between all nodes no matter they are connected or not. These approaches use shortest relation path between nodes to encode semantic relationships. However, they ignore the information of nodes in the relation path and encode the direct relations and indirect relations without distinction. It may disturb the information propagation process when aggregate information from direct neighbors.
To solve the issues above, we propose the Heterogeneous Graph Transformer (HetGT) to encode the graph, which independently model the different relations in the individual subgraphs of the original graph. HetGT is adapted from Transformer and it also employs an encoder-decoder architecture. Following Beck et al. (2018), we first transform the input into its corresponding Levi graph which is a heterogeneous graph (contains different types of edges). Then we split the transformed graph into multiple subgraphs according to its heterogeneity, which corresponds to different representation subspaces of the graph. For updating the node representations, attention mechanisms are used for inde
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7145-7154 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
(a)
Here it is a country with the freedom of speech.
(b)
Figure 1: (a) An example of AMR graph for the sentence of Here it is a country with the freedom of speech. (b) Its corresponding extended Levi graph with three types of edges. (c) The architecture of HetGT encoder.
pendently aggregating information in different subgraphs. Finally, the representations of each node obtained in different subgraphs are concatenated together and a parameterized linear transformation is applied. In this way, HetGT could adaptively model the various relations in the graph independently, avoiding the information loss caused by mixing all of them. Moreover, we introduce the jump connection in our model, which significantly improves the model performance.
We evaluate our model on four benchmark datasets of two Graph2Seq tasks: the AMR-to-text generation and the syntax-based Neural Machine Translation (NMT). In terms of various evaluation metrics, our model strongly outperforms the state-of-the-art (SOTA) results on both two tasks. Particularly, in AMR-to-text generation, our model improves the BLEU scores of the SOTA by about 2.2 points and 2.3 points on two benchmark datasets (LDC2015E86 and LDC2017T10). In syntax-based NMT, our model surpasses the SOTA by about 4.1 and 2.2 BLEU scores for English-German and English-Czech on News Commentary v11 datasets from the WMT16 translation task. Our contributions can be summarized as follows:
- We propose the Heterogeneous Graph Transformer (HetGT) which adaptively models the various relations in different representation subgraphs.
- We analyze the shortcomings of the residual
connection and introduce a better connectivity method around encoder layers.
- Experimental results show that our model achieves new state-of-the-art performance on four benchmark datasets of two Graph2Seq tasks.
2 Neural Graph-to-Sequence Model
In this section, we will first begin with a brief review of the Transformer which is the basis of our model. Then we will introduce the graph transformation process. Finally, we will detail the whole architecture of HetGT.
2.1 Transformer
The Transformer employs an encoder-decoder architecture, consisting of stacked encoder and decoder layers. Encoder layers consist of two sublayers: a self-attention mechanism and a position-wise feed-forward network. Self-attention mechanism employs  $h$  attention heads. Each attention head operates on an input sequence  $x = (x_{1},\ldots ,x_{n})$  of  $n$  elements where  $x_{i}\in \mathbb{R}^{d_{x}}$ , and computes a new sequence  $z = (z_{1},\dots,z_{n})$  of the same length where  $z\in \mathbb{R}^{d_z}$ . Finally, the results from all the attention heads are concatenated together and a parameterized linear transformation is applied to get the output of the self-attention sublayer. Each output element  $z_{i}$  is computed as the weighted sum of
7146
Figure 2: An example of graph structure and its extension to subword units.
linearly transformed input elements:
$$
z _ {i} = \sum_ {j = 1} ^ {n} \alpha_ {i j} \left(x _ {j} W ^ {V}\right) \tag {1}
$$
where  $\alpha_{ij}$  is weight coefficient and computed by a softmax function:
$$
\alpha_ {i j} = \operatorname {s o f t m a x} \left(e _ {i j}\right) = \frac {\exp e _ {i j}}{\sum_ {k = 1} ^ {n} \exp e _ {i k}} \tag {2}
$$
And  $e_{ij}$  is computed using a compatibility function that compares two input elements:
$$
e _ {i j} = \frac {\left(x _ {i} W ^ {Q}\right) \left(x _ {j} W ^ {K}\right) ^ {T}}{\sqrt {d _ {z}}} \tag {3}
$$
Scaled dot product was chosen for the compatibility function.  $W^{V}, W^{Q}, W^{V} \in \mathbb{R}^{d_{x} \times d_{z}}$  are layer-specific trainable parameter matrices. Meanwhile, these parameter matrices are unique per attention head.
2.2 Input Graph Transformation
Following Beck et al. (2018), we transform the original graph into the Levi graph. The transformation equivalently turns edges into additional nodes so we can encode the original edge labels in the same way as for nodes. We also add a reverse edge between each pair of connected nodes as well as a self-loop edge for each node. These strategies can make the model benefit from the information propagation from different directions (See Figure 1 (b)).
In order to alleviate the data sparsity problem in the corpus, we further introduce the Byte Pair Encoding (BPE) (Sennrich et al., 2016) into the Levi Graph. We split the original node into multiple subword nodes. Besides adding default connections, we also add the reverse and self-loop edges among subwords. For example, the word country in Figure 2 is segmented into co@@, un@@, try with
three types of edges between them. Finally, we transform the AMR graph into the extended Levi Graph which can be seen as a heterogeneous graph, as it has different types of edges.
2.3 Heterogeneous Graph Transformer
Our model is also an encoder-decoder architecture, consisting of stacked encoder and decoder layers. Given a preprocessed extended Levi graph, we split the extended Levi graph into multiple subgraphs according to its heterogeneity. In each graph encoder block, the node representation in different subgraphs is updated based on its neighbor nodes in the current subgraph. Then all the representations of this node in different subgraphs will be combined to get its final representation. In this way, the model can attend to information from different representation subgraphs and adaptively model the various relations. The learned representations of all nodes at the last block are fed to the sequence decoder for sequence generation. The architecture of HetGT encoder is shown in Figure 1 (c). Due to the limitation of space the decoder is omitted in the figure. We will describe it in Section 2.3.2.
2.3.1 Graph Encoder
Unlike previous Transformer-based Graph2Seq models using relative position encoding to incorporate structural information, we use a simpler way to encode the graph structure. As Transformer treats the sentence as a fully-connected graph, we directly mask the non-neighbor nodes' attention when updating each node's representation. Specifically, we mask the attention  $\alpha_{ij}$  for node  $j \notin \mathcal{N}_i$ , where  $\mathcal{N}_i$  is the set of neighbors of node  $i$  in the graph. So given the input sequence  $x = (x_1, \ldots, x_n)$ , the output representation of node  $i$  denoted as  $z_i$  in each attention head is computed as follows:
$$
z _ {i} = \sum_ {j \in \mathcal {N} _ {i}} \alpha_ {i j} \left(x _ {j} W ^ {V}\right) \tag {4}
$$
where  $\alpha_{ij}$  represents the attention score of node  $j$  to  $i$  which is computed using scaled dot-product function as in Equation 2.
We also investigate another way to compute attention scores. We use the additive form of attention instead of scaled dot-product attention, which is similar to graph attention network (Velicković et al., 2018). The additive form of attention shows better performance and trainability in some tasks (Chen et al., 2019). The attention coefficient  $\alpha_{ij}$  is
$$
\alpha_ {i j} = \operatorname {s o f t m a x} \left(e _ {i j}\right) = \frac {\exp e _ {i j}}{\sum_ {k \in \mathcal {N} _ {i}} \exp e _ {i k}} \tag {5}
$$
$$
e _ {i j} = \text {L e a k y R e L U} \left(\mathbf {a} ^ {T} \left[ x _ {i} W ^ {V}; x _ {j} W ^ {V} \right]\right)
$$
where  $\mathbf{a} \in \mathbb{R}^{2d_z}$  is a weight vector. LeakyReLU (Girshick et al., 2014) is used as the activation function.
2.3.2 Heterogeneous Mechanism
Motivated by the success of the multi-head mechanism, we propose the heterogeneous mechanism. Considering a sentence, multi-head attention allows the model to implicitly attend to information from different representation subspaces at different positions. Correspondingly, our heterogeneous mechanism makes the model explicitly attend to information in different subgraphs, corresponding to different representation subspaces of the graph, which enhances the models' encoding capability.
As stated above, the extended Levi graph is a heterogeneous graph which contains different types of edges. For example, in Figure 1 (b), the edge type vocabulary for the Levi graph of the AMR graph is  $\mathcal{T} = \{\text{default}, \text{reverse}, \text{self}\}$ . Specifically, we first group all edge types into a single one to get a homogeneous subgraph referred to connected subgraph. The connected subgraph is actually an undirected graph which contains the complete connected information in the original graph. Then we split the input graph into multiple subgraphs according to the edge types. Besides learning the directly connected relations, we introduce a fully-connected subgraph to learn the implicit relationships between indirectly connected nodes. Finally, we get the set of subgraphs including  $M$  elements  $\mathcal{G}^{\mathrm{sub}} = \{\text{fully-connected}, \text{connected}, \text{default}, \text{reverse}\}$ . For AMR graph  $M = 4$  (For NMT  $M = 6$ , we will detail it in section 3.1). Note that we do not have a subgraph only containing selfedges. Instead, we add the self-loop edges into all subgraphs. We think it is more helpful for information propagation than constructing an independent self-connected subgraph. Now the output  $z$  in each encoder layer is computed as follows:
$$
z = \operatorname {F F N} \left(\operatorname {c o n c a t} \left(z ^ {\mathcal {G} _ {1} ^ {\text {s u b}}}, \dots , z ^ {\mathcal {G} _ {M} ^ {\text {s u b}}}\right) W ^ {O}\right)
$$
$$
z _ {i} ^ {\mathcal {G} _ {m} ^ {\mathrm {s u b}}} = \sum_ {j \in \mathcal {N} _ {i} ^ {\mathcal {G} _ {m} ^ {\mathrm {s u b}}}} \alpha_ {i j} \left(x _ {j} W ^ {V}\right), m \in [ 1, M ] \tag {6}
$$
where  $W^{O} \in \mathbb{R}^{Md_{z} \times d_{z}}$  is the parameter matrix.  $\mathcal{N}_{i}^{\mathcal{G}_{m}^{\mathrm{sub}}}$  is the set of neighbors in the  $m$ -th subgraph
Figure 3: Different layer aggregation methods: residual (left), jump (middle), dense (right).
of node  $i$ .  $\alpha_{ij}$  is computed as Equation 2 or Equation 5. FFN is a feed-forward network which consists of two linear transformations with a ReLU activation in between. We also employ the residual connections between sublayers as well as layer normalization. Note that the heterogeneous mechanism is independent of the model architecture, so it can be applied to any other graph models which may bring benefits.
For decoder, we follow the standard implementation of the sequential Transformer decoder to generate the text sequence. The decoder layers consist of three sublayers: self-attention followed by encoder-decoder attention, followed by a position-wise feed-forward layer.
2.3.3 Layer Aggregation
As stated above, our model consists of stacked encoder layers. A better information propagation between encoder layers may bring better performance. Therefore, we investigate three different layer aggregation methods, which are illustrated in Figure 3. When updating the representation of each node at  $l$ -th layer, recent approaches aggregate the neighbors first and then combine the aggregated result with the node's representation from  $(l - 1)$ -th layer. This strategy can be viewed as a form of a skip connection between different layers (Xu et al., 2018a):
$$
z _ {\mathcal {N} _ {i}} ^ {(l)} = \text {A G G R E G A T E} \left(\left\{z _ {j} ^ {(l - 1)}, \forall j \in \mathcal {N} _ {i} \right\}\right) \tag {7}
$$
$$
z _ {i} ^ {(l)} = \operatorname {C O M B I N E} \left(z _ {\mathcal {N} _ {i}} ^ {(l)}, z _ {i} ^ {(l - 1)}\right)
$$
The residual connection is another well-known skip connection which uses the identity mapping as the combine function to help signals propagate (He et al., 2016). However, these skip connections cannot adaptively adjust the neighborhood size of the final-layer representation independently. If we "skip" a layer for  $z_{i}^{(l)}$ , all subsequent units such as
$z_{i}^{(l + j)}$  using this representation will be using this skip implicitly. Thus, to selectively aggregate the outputs of previous layers at the last, we introduce the Jumping Knowledge architecture (Xu et al., 2018a) in our model. At the last layer  $L$  of the encoder, we combine all the outputs of previous encoder layers by concatenation to help the model selectively aggregate all of those intermediate representations.
$$
z _ {i} ^ {\text {f i n a l}} = \operatorname {C o n c a t} \left(z _ {i} ^ {(L)}, \dots , z _ {i} ^ {(1)}, x _ {i}\right) W _ {\text {j u m p}} \tag {8}
$$
where  $W_{\mathrm{jump}} \in \mathbb{R}^{(Ld_z + d_x) \times d_z}$ . Furthermore, to better improve information propagation, dense connectivity can be introduced as well. With dense connectivity, the nodes in  $l$ -th layer not only take input from  $(l - 1)$ -th layer but also draw information from all preceding layers:
$$
z _ {i} ^ {(l)} = \operatorname {C o n c a t} \left(z _ {i} ^ {(l - 1)}, \dots , z _ {i} ^ {(1)}, x _ {i}\right) W _ {\text {d e n s e}} ^ {(l)} \tag {9}
$$
where  $W_{\mathrm{dense}}^{(l)} \in \mathbb{R}^{d(l) \times d_z}$ .  $d^{(l)} = d_x + d_z \times (l - 1)$ . Dense connectivity are also introduced in previous researches (Huang et al., 2017; Guo et al., 2019).
We build and test our model on two typical Graph2Seq learning tasks. One is AMR-to-text generation and the other is syntax-based NMT. Table 1 presents the statistics of four datasets of the two tasks. For AMR-to-text generation, we use two standard benchmarks LDC2015E86 (AMR15) and LDC2017T10 (AMR17). These two datasets contain 16K and 36K training instances, respectively, and share the development and test set. Each instance contains a sentence and an AMR graph. In the preprocessing steps, we apply entity simplification and anonymization in the same way as Konstas et al. (2017). Then we transform each preprocessed AMR graph into its extended Levi graph as described in Section 2.2.
For the syntax-based NMT, we take syntactic trees of source texts as inputs. We evaluate our model on both English-German (En-De) and English-Czech (En-Cs) News Commentary v11 datasets from the WMT16 translation task<sup>1</sup>. Both sides are tokenized and split into subwords using BPE with 8000 merge operations. English text is parsed using SyntaxNet (Alberti et al., 2017). Then
Dataset	Train	Dev	Test
LDC2015E86 (AMR15)	16,833	1,368	1,371
LDC2017T10 (AMR17)	36,521	1,368	1,371
English-Czech (En-Cs)	181,112	2,656	2,999
English-German (En-De)	226,822	2,169	2,999
Table 1: The statistics of four datasets. The first two are datasets in AMR-to-text generation subtask, the last two are datasets in syntax-based NMT subtask.
we transform the labeled dependency tree into the extended Levi graph as described in Section 2.2. Unlike AMR-to-text generation, in NMT task the input sentence contains significant sequential information. This information is lost when treating the sentence as a graph. Guo et al. (2019) consider this information by adding sequential connections between each word node. In our model, we also add forward and backward edges in the extended Levi graph. Thus, the edge types vocabulary for the extended Levi graph of the dependency tree is  $\mathcal{T} = \{\text{default, reverse, self, forward, backward}\}$ . So the set of subgraphs for NMT is  $\mathcal{G}^{\text{sub}} = \{\text{fully-connected, connected, default, reverse, forward, backward}\}$ . Note that we do not change the model architecture in the NMT tasks. However, we still get good results, which indicates the effectiveness of our model on Graph2Seq tasks. Except for introducing BPE into Levi graph, the above preprocessing steps are following Bastings et al. (2017). We refer to them for further information on the preprocessing steps.
Both our encoder and decoder have 6 layers with 512-dimensional word embeddings and hidden states. We employ 8 heads and dropout with a rate of 0.3. For optimization, we use Adam optimizer with  $\beta_{2} = 0.998$  and set batch size to 4096 tokens. Meanwhile, we increase learning rate linearly for the first warmup_steps, and decrease it thereafter proportionally to the inverse square root of the step number. We set warmup_steps to 8000. The similar learning rate schedule is adopted in (Vaswani et al., 2017). Our implementation uses the openNMT library (Klein et al., 2017). We train the models for  $250\mathrm{K}$  steps on a single GeForce GTX 1080 Ti GPU. Our code is available at https://github.com/QAQ-v/HetGT.
<sup>1</sup> http://www.statmt.org/wmt16/translation-task.html
7149
Model	LDC2015E86 (AMR15)	LDC2017T10 (AMR17)
BLEU	CHRF++	METEOR	BLEU	CHRF++	METEOR
GGNN2Seq (Beck et al., 2018)	-	-	-	23.3	50.4	-
GraphLSTM (Song et al., 2018)	23.3	-	-	-	-	-
GCNSEQ (Damonte and Cohen, 2019)	24.40	-	23.60	24.54	-	24.07
DGCN (Guo et al., 2019)	25.9	-	-	27.9	57.3	-
G2S-GGNN (Ribeiro et al., 2019)	24.32	-	30.53	27.87	-	33.21
Transformer-SA (Zhu et al., 2019)	29.66	63.00	35.45	31.54	63.84	36.02
Transformer-CNN (Zhu et al., 2019)	29.10	62.10	35.00	31.82	64.05	36.38
GTransformer (Cai and Lam, 2020)	27.4	56.4	32.9	29.8	59.4	35.1
GGNN2Seqensemble (Beck et al., 2018)	-	-	-	27.5	53.5	-
DGCNensemble (Guo et al., 2019)	28.2	-	-	30.4	59.6	-
Transformer	25.69	60.10	33.88	27.60	61.78	35.21
HetGTdot-product (ours)	31.29	63.62	36.71	33.16	65.08	37.75
HetGTadditive (ours)	31.84	63.81	36.89	34.10	65.60	38.10
Table 2: Results for AMR-to-text generation on the test sets of AMR15 and AMR17.
Model	English-German	English-Czech
BLEU	CHRF++	METEOR	BLEU	CHRF++	METEOR
BiRNN+GCN (Bastings et al., 2017)	16.1	-	-	9.6	-	-
GGNN2Seq (Beck et al., 2018)	16.7	42.4	-	9.8	33.3	-
DGCN (Guo et al., 2019)	19.0	44.1	-	12.1	37.1	-
GTransformer (Cai and Lam, 2020)	21.3	47.9	-	14.1	41.1	-
GGNN2Seqensenmble (Beck et al., 2018)	19.6	45.1	-	11.7	35.9	-
DGCNensemble (Guo et al., 2019)	20.5	45.8	-	13.1	37.8	-
Transformer	23.18	49.54	26.00	14.83	39.27	19.12
HetGTdot-product (ours)	25.39	51.55	27.37	16.15	41.10	20.18
HetGTadditive (ours)	25.44	51.27	27.26	16.29	41.14	20.35
Table 3: Results for syntax-based NMT on the test sets of En-De and En-Cs.
3.3 Metrics and Baselines
For performance evaluation, we use BLEU (Papineni et al., 2002), METEOR (Denkowski and Lavie, 2014) and sentence-level CHRF++ (Popovic, 2015) with default hyperparameter settings as evaluation metrics. Meanwhile, we use the tools in Neubig et al. (2019) for the statistical significance tests.
Our baseline is the original Transformer  ${}^{2}$  . For AMR-to-text generation,Transformer takes linearized graphs as inputs. For syntax-based NMT, Transformer is trained on the preprocessed translation dataset without syntactic information. We also compare the performance of HetGT with previous single/ensemble approaches which can be grouped into three categories: (1) Recurrent neu
ral network (RNN) based methods (GGNN2Seq, GraphLSTM); (2) Graph neural network (GNN) based methods (GCNSEQ, DGCN, G2S-GGNN); (3) The Transformer based methods (Structural Transformer, GTransformer). The ensemble models are denoted by subscripts in Table 2 and Table 3.
3.4 Results on AMR-to-text Generation
Table 2 presents the results of our single model and previous single/ensemble models on the test sets of AMR15 and AMR17. We can see that our Transformer baseline outperforms most previous single models, and our best single model  $\mathrm{HetGT}_{\mathrm{additive}}$  outperforms the Transformer baseline by a large margin (6.15 BLEU and 6.44 BLEU) on both benchmarks. It demonstrates the importance of incorporating structural information. Meanwhile,  $\mathrm{HetGT}_{\mathrm{additive}}$  gets an improvement of 2.18 and 2.28 BLEU points over the latest SoTA results
2Parameters were chosen following the OpenNMT FAQ: http://opennmt.net/OpenNMT-py/FAQ.html#how-do-i-use-the-transformer-model
(Zhu et al., 2019) on AMR15 and AMR17, respectively. Previous models can capture the structural information but most of them ignore heterogeneous information. These results indicate that the heterogeneity in the graph carries lots of useful information for the downstream tasks, and our model can make good use of it.