# Improving Neural Machine Translation with AMR Semantic Graphs

Authors: Long H. B. Nguyen; Viet H. Pham; Dien Dinh
Journal: Mathematical Problems in Engineering
 (2021)

---

## Abstract

The Seq2Seq model and its variants (ConvSeq2Seq and Transformer) emerge as a promising novel solution to the machine translation problem. However, these models only focus on exploiting knowledge from bilingual sentences without paying much attention to utilizing external linguistic knowledge sources such as semantic representations. Not only do semantic representations can help preserve meaning but they also minimize the data sparsity problem. However, to date, semantic information remains rarely integrated into machine translation models. In this study, we examine the effect of abstract meaning representation (AMR) semantic graphs in different machine translation models. Experimental results on the IWSLT15 English-Vietnamese dataset have proven the efficiency of the proposed model, expanding the use of external language knowledge sources to significantly improve the performance of machine translation models, especially in the application of low-resource language pairs.


---

## Body

## 1. Introduction

Neural machine translation (NMT) [1–4] has proven its effectiveness and thus has gained researchers’ attention in recent years. In practical applications, the typical inputs to NMT systems are sentences in which words are represented as individual vectors in a word embedding space. This word embedding space does not show any connection among words within a sentence such as dependency or semantic role relationships. Recent studies [5–8] found that semantic information is essential to generate concise and appropriate translations in machine translation. Although these models have made a significant progress, their design and functions are limited to statistical machine translation systems only. Consequently, the tasks of surveying, analyzing, and applying additional semantic information to NMT systems have not received comprehensive attention.In this study, we present the method of integrating abstract meaning representation (AMR) graphs (https://amr.isi.edu) as additional semantic information into the current popular NMT systems such as Seq2Seq, ConvSeq2Seq, and Transformer. AMR graphs are rooted, labeled, directed, and acyclical graphs representing the entire content of a sentence. They are also abstracted from related syntactic representations in the sense that sentences with similar meanings will have the same AMR graph, even if the words used in these sentences are different. Figure 1 illustrates an AMR graph in which the nodes (e.g., want-01, girl) symbolize concepts, while the edges (e.g., ARG0 and ARG1) represent the relationship between the concepts that they connect. Compared to semantic role graphs, AMR graphs contain more relationships (e.g., between boy and girl). Besides, AMR graphs directly hold entity relations while excluding the alternating variables (i.e., using lemma) and the function words. Therefore, AMR graphs can be combined with the input text to generate better contextual representations. Moreover, the structured information from AMR graphs can help minimize the problem of data sparsity in resource-poor settings. First, the AMR graph representations are combined with the word embedding to create a better context representation for a sentence. Then, multihead attention can focus on all positions of contextual features with the outputs of the AMR graph representations.Figure 1
The AMR graph for the multiple sentences.Integrating AMR graphs into NMT yields several benefits. First, this addresses the problems of data sparsity and semantic ambiguity. Second, structured semantic information constructed from AMR graphs could help complement the input text by providing high-level abstract information, thereby improving the encoding of the input word embedding. Last, multihead attention can also take advantage of semantic information to improve the dependency among words within a sentence.Recent studies have applied semantic representation to NMT models. For instance, Marcheggiani et al. [9] exploited the semantic role labeling (SRL) information for NMT, indicating that the predicate-argument structure from SRL can help increase the quality of an attention-based sequence-to-sequence model. Meanwhile, Song et al. [10] proved that semantic information structured from AMR graphs can complement input text by incorporating high-level abstract information. In this approach, the graph recurrent network (GRN) was utilized to encode AMR graphs without breaking the original graph structure, and a sequential long short-term memory (LSTM) was used to encode the source input. The decoder was a doubly attentive LSTM, taking the encoding results of both the graph encoder and the sequential encoder as attention memories. Song et al. had also argued that the results of an AMR integration is significantly greater than those of a sole SRL integration because AMR graphs include both SRL and the relationships between the nodes (i.e., words). However, Song’s approach has encountered some drawbacks such as failed to address the problem of the correlation between nodes in AMR graphs and investigated only on the machine translation system using the recurrent neural network (RNN).The contributions of our work are as follows:(i)
First, instead of adding a node to represent an edge in the graph and assigning properties of the edge as those of the documents, we extend the node embedding algorithm [11] to use direct edge information(ii)
Second, instead of using the graph recurrent network in [10], we propose an architecture that binds an inductive graph encoder(iii)
Finally, we examined and analyzed the results on the English-Vietnamese bilingual set, which is considered a low-resource language pair. Through experiments, we demonstrate the effectiveness of integrating AMR into neural network machine translation and draw insightful conclusions for future studies.The organization for the remaining of the article is as follows.Section 2 introduces current popular machine translation architectures such as Seq2Seq, ConvSeq2Seq, and Transformer. Next, Section 3 presents the method of representing AMR graphs in the vector form as well as proposing a method to integrate AMR graphs into different NMT models. Then, Sections 4 and 5 discuss the corpus used in the experiment and the experimental configuration for the model, respectively. Afterward, Section 6 presents the experimental results of the machine translation model with an integrated AMR and analyzes the effect of an AMR on the model along with some translation errors generated by the model. Section 7 summarizes our work.


## 2. Neural Machine Translation

In this section, we provide a brief introduction about the Seq2Seq model and its variants such as ConvSeq2Seq and Transformer.


### 2.1. Seq2Seq

We take the attention-based sequence-to-sequence model of [1] as the baseline model, but we use LSTM [12] in both encoder and decoder.


#### 2.1.1. Encoder

Given a sentence,x=x1,x2,…,xn.(i) Uni-LSTM. As usual, the RNN reads an input sequence x in order starting from the first token x1 to xn and computes a sequence of hidden state h⟶1,h⟶2,…,h⟶n to generate input representation from left to right.(1)H=h⟶1,h⟶2,…,h⟶n.(ii) Bi-LSTM. Consists of forward and backward LSTM’s. The forward LSTM works similar to Uni-LSTM and the backward LSTM reads the sequence in the reverse order from the last token xn to x1, resulting a sequence of backward hidden states h←1,h←2,…,h←n. We obtain the word embedding xi by concatenating the forward h⟶i and backward h←i hidden state, hi=h⟶i,h←i.(2)H=h1,h2,…,hn.


#### 2.1.2. Decoder

The decoder predicts the next wordyt, given the context vector c and all previously predicted words y0,y1,…,yt−1. We used an attention-based LSTM decoder [1], with attention memory as the concatenation of the attention vectors among all source tokens.For each decoding stept, the decoder feeds the concatenation of the embedding of current input eyt and previous context vector ct−1 into LSTM to update the hidden state:(3)st=LSTMst−1,eyt,ct−1.Then, the new context vector is computed as(4)ϵt,i=ast,hi,αti=expϵt,i∑k=1Vexpϵt,k,ci=∑i=1nαt,ihi,where a is the alignment model which is a feed forward network, scores how well the inputs surround position i, and the input at position t match.The output probability over target vocabulary is calculated:(5)Pvocab=softmaxWost;ci+bo,where Wo and bo are the model parameters.


### 2.2. ConvSeq2Seq

#### 2.2.1. ConvSeq2Seq

This architecture is proposed by Gehring et al. [2] to completely replace the RNN with the CNN with the following components:The ConvS2S model followed the encoder-decoder architecture. Both encoder and decoder blocks share an identical structure that computes hidden states based on a fixed number of input elements. To enlarge the context size, we stack several blocks over each other. Each block comprises a one-dimensional convolution and a nonlinearity. In each convolution kernel, parameters areW∈ℛ2d×kdandbw∈ℛ2d. The input is represented as ∈ℛkd, which is a concatenation of k input elements with dimension of d and maps them to get the single output Y∈ℛ2d with dimension twice of that of the input. Then, the k output elements will be fed into subsequent layers. We leverage the gated linear unit (GLU) as nonlinearity which applied on the output of the convolution Y=AB∈ℛ2d:(6)vAB=A⊗σB,where A,B∈ℛd are the inputs to the nonlinearity, ⊗ denotes the element-wise multiplication, the output Y=AB∈ℛ2d has a half of size compared to Y, and σB is the gate that control which inputs A of the current contexts are relevant.In order to enable deep convolutional blocks, we adopt the residual connections which connect the input of each convolutional layer with the output:(7)hil=vWlhi−k/2l−1,…,hi+k/2l−1+Bwl+hil−1,where hl is the hidden state of lth layer.


#### 2.2.2. LightConvSeq2Seq

As a variant of a CNN called lightweight convolution [13] which allows computation with linear complexity, On, with n being the length of the input string.The structure of LightConvSeq2Seq consists of the elements similar to Conv2Seq but using lightweight convolution operation rather than convolution operation.Depthwise Convolution (DConv). Perform a convolution operation independently over every channel; thereby, the number of parameters reduce significantly from d2k to dk, where k is the kernel width. In general, at position i and direction c, the output Oi,c is calculated as follows:(8)Oi,c=∑j=1kWc,j⋅Xx+j−k+12,c.


### 2.3. Transformer

Transformer [4] also includes an encoder and a decoder. The encoder generates a vector representation of the input sentence. Assuming an input of the form x=x1,x2,…,xn and a representation of x of the form z=z1,z2,…,zn, the decoder produces sequentially for a translation of y=y1,y2,…,ym based on z and the previous outputs.


#### 2.3.1. The Encoder

There areN stacked similar blocks. Each of these blocks consists of 2 subblocks: a self-attention mechanism and a feed forward network. A residual connection surrounds each subblock, followed by layer normalization. The general representation formula for the encoder is as follows:(9)z^=LayerNormx+self−attentionx,z=LayerNormz^+feed forwardz^.


#### 2.3.2. The Decoder

There are alsoN blocks. However, each block consists of 3 subblocks: a self-attention block, a feed forward block, and an encoder-decoder attention block inserted between them. The residual connection and layer normalization are used similarly to the encoder. The encoder generates outputs step by step. The self-attention block only pays attention to the positions generated in the previous steps by using a mask. The mask prevents the decoder from paying attention to locations that have not been generated, so outputs can only be predicted based on the result z of the encoder and previous outputs.


#### 2.3.3. Self-Attention

There are 3 components as follows: query (Q), key (K), and value (V), defined as follows:(10)AttentionQ,K,V=softmaxQKTdkV,where Q,K,andV are the parameters with the number of dimensions dk,dk,anddv respectively.


## 2.1. Seq2Seq

We take the attention-based sequence-to-sequence model of [1] as the baseline model, but we use LSTM [12] in both encoder and decoder.


### 2.1.1. Encoder

Given a sentence,x=x1,x2,…,xn.(i) Uni-LSTM. As usual, the RNN reads an input sequence x in order starting from the first token x1 to xn and computes a sequence of hidden state h⟶1,h⟶2,…,h⟶n to generate input representation from left to right.(1)H=h⟶1,h⟶2,…,h⟶n.(ii) Bi-LSTM. Consists of forward and backward LSTM’s. The forward LSTM works similar to Uni-LSTM and the backward LSTM reads the sequence in the reverse order from the last token xn to x1, resulting a sequence of backward hidden states h←1,h←2,…,h←n. We obtain the word embedding xi by concatenating the forward h⟶i and backward h←i hidden state, hi=h⟶i,h←i.(2)H=h1,h2,…,hn.


### 2.1.2. Decoder

The decoder predicts the next wordyt, given the context vector c and all previously predicted words y0,y1,…,yt−1. We used an attention-based LSTM decoder [1], with attention memory as the concatenation of the attention vectors among all source tokens.For each decoding stept, the decoder feeds the concatenation of the embedding of current input eyt and previous context vector ct−1 into LSTM to update the hidden state:(3)st=LSTMst−1,eyt,ct−1.Then, the new context vector is computed as(4)ϵt,i=ast,hi,αti=expϵt,i∑k=1Vexpϵt,k,ci=∑i=1nαt,ihi,where a is the alignment model which is a feed forward network, scores how well the inputs surround position i, and the input at position t match.The output probability over target vocabulary is calculated:(5)Pvocab=softmaxWost;ci+bo,where Wo and bo are the model parameters.


## 2.1.1. Encoder

Given a sentence,x=x1,x2,…,xn.(i) Uni-LSTM. As usual, the RNN reads an input sequence x in order starting from the first token x1 to xn and computes a sequence of hidden state h⟶1,h⟶2,…,h⟶n to generate input representation from left to right.(1)H=h⟶1,h⟶2,…,h⟶n.(ii) Bi-LSTM. Consists of forward and backward LSTM’s. The forward LSTM works similar to Uni-LSTM and the backward LSTM reads the sequence in the reverse order from the last token xn to x1, resulting a sequence of backward hidden states h←1,h←2,…,h←n. We obtain the word embedding xi by concatenating the forward h⟶i and backward h←i hidden state, hi=h⟶i,h←i.(2)H=h1,h2,…,hn.


## 2.1.2. Decoder

The decoder predicts the next wordyt, given the context vector c and all previously predicted words y0,y1,…,yt−1. We used an attention-based LSTM decoder [1], with attention memory as the concatenation of the attention vectors among all source tokens.For each decoding stept, the decoder feeds the concatenation of the embedding of current input eyt and previous context vector ct−1 into LSTM to update the hidden state:(3)st=LSTMst−1,eyt,ct−1.Then, the new context vector is computed as(4)ϵt,i=ast,hi,αti=expϵt,i∑k=1Vexpϵt,k,ci=∑i=1nαt,ihi,where a is the alignment model which is a feed forward network, scores how well the inputs surround position i, and the input at position t match.The output probability over target vocabulary is calculated:(5)Pvocab=softmaxWost;ci+bo,where Wo and bo are the model parameters.


## 2.2. ConvSeq2Seq

### 2.2.1. ConvSeq2Seq

This architecture is proposed by Gehring et al. [2] to completely replace the RNN with the CNN with the following components:The ConvS2S model followed the encoder-decoder architecture. Both encoder and decoder blocks share an identical structure that computes hidden states based on a fixed number of input elements. To enlarge the context size, we stack several blocks over each other. Each block comprises a one-dimensional convolution and a nonlinearity. In each convolution kernel, parameters areW∈ℛ2d×kdandbw∈ℛ2d. The input is represented as ∈ℛkd, which is a concatenation of k input elements with dimension of d and maps them to get the single output Y∈ℛ2d with dimension twice of that of the input. Then, the k output elements will be fed into subsequent layers. We leverage the gated linear unit (GLU) as nonlinearity which applied on the output of the convolution Y=AB∈ℛ2d:(6)vAB=A⊗σB,where A,B∈ℛd are the inputs to the nonlinearity, ⊗ denotes the element-wise multiplication, the output Y=AB∈ℛ2d has a half of size compared to Y, and σB is the gate that control which inputs A of the current contexts are relevant.In order to enable deep convolutional blocks, we adopt the residual connections which connect the input of each convolutional layer with the output:(7)hil=vWlhi−k/2l−1,…,hi+k/2l−1+Bwl+hil−1,where hl is the hidden state of lth layer.


### 2.2.2. LightConvSeq2Seq

As a variant of a CNN called lightweight convolution [13] which allows computation with linear complexity, On, with n being the length of the input string.The structure of LightConvSeq2Seq consists of the elements similar to Conv2Seq but using lightweight convolution operation rather than convolution operation.Depthwise Convolution (DConv). Perform a convolution operation independently over every channel; thereby, the number of parameters reduce significantly from d2k to dk, where k is the kernel width. In general, at position i and direction c, the output Oi,c is calculated as follows:(8)Oi,c=∑j=1kWc,j⋅Xx+j−k+12,c.


## 2.2.1. ConvSeq2Seq

This architecture is proposed by Gehring et al. [2] to completely replace the RNN with the CNN with the following components:The ConvS2S model followed the encoder-decoder architecture. Both encoder and decoder blocks share an identical structure that computes hidden states based on a fixed number of input elements. To enlarge the context size, we stack several blocks over each other. Each block comprises a one-dimensional convolution and a nonlinearity. In each convolution kernel, parameters areW∈ℛ2d×kdandbw∈ℛ2d. The input is represented as ∈ℛkd, which is a concatenation of k input elements with dimension of d and maps them to get the single output Y∈ℛ2d with dimension twice of that of the input. Then, the k output elements will be fed into subsequent layers. We leverage the gated linear unit (GLU) as nonlinearity which applied on the output of the convolution Y=AB∈ℛ2d:(6)vAB=A⊗σB,where A,B∈ℛd are the inputs to the nonlinearity, ⊗ denotes the element-wise multiplication, the output Y=AB∈ℛ2d has a half of size compared to Y, and σB is the gate that control which inputs A of the current contexts are relevant.In order to enable deep convolutional blocks, we adopt the residual connections which connect the input of each convolutional layer with the output:(7)hil=vWlhi−k/2l−1,…,hi+k/2l−1+Bwl+hil−1,where hl is the hidden state of lth layer.


## 2.2.2. LightConvSeq2Seq

As a variant of a CNN called lightweight convolution [13] which allows computation with linear complexity, On, with n being the length of the input string.The structure of LightConvSeq2Seq consists of the elements similar to Conv2Seq but using lightweight convolution operation rather than convolution operation.Depthwise Convolution (DConv). Perform a convolution operation independently over every channel; thereby, the number of parameters reduce significantly from d2k to dk, where k is the kernel width. In general, at position i and direction c, the output Oi,c is calculated as follows:(8)Oi,c=∑j=1kWc,j⋅Xx+j−k+12,c.


## 2.3. Transformer

Transformer [4] also includes an encoder and a decoder. The encoder generates a vector representation of the input sentence. Assuming an input of the form x=x1,x2,…,xn and a representation of x of the form z=z1,z2,…,zn, the decoder produces sequentially for a translation of y=y1,y2,…,ym based on z and the previous outputs.


### 2.3.1. The Encoder

There areN stacked similar blocks. Each of these blocks consists of 2 subblocks: a self-attention mechanism and a feed forward network. A residual connection surrounds each subblock, followed by layer normalization. The general representation formula for the encoder is as follows:(9)z^=LayerNormx+self−attentionx,z=LayerNormz^+feed forwardz^.


### 2.3.2. The Decoder

There are alsoN blocks. However, each block consists of 3 subblocks: a self-attention block, a feed forward block, and an encoder-decoder attention block inserted between them. The residual connection and layer normalization are used similarly to the encoder. The encoder generates outputs step by step. The self-attention block only pays attention to the positions generated in the previous steps by using a mask. The mask prevents the decoder from paying attention to locations that have not been generated, so outputs can only be predicted based on the result z of the encoder and previous outputs.


### 2.3.3. Self-Attention

There are 3 components as follows: query (Q), key (K), and value (V), defined as follows:(10)AttentionQ,K,V=softmaxQKTdkV,where Q,K,andV are the parameters with the number of dimensions dk,dk,anddv respectively.


## 2.3.1. The Encoder

There areN stacked similar blocks. Each of these blocks consists of 2 subblocks: a self-attention mechanism and a feed forward network. A residual connection surrounds each subblock, followed by layer normalization. The general representation formula for the encoder is as follows:(9)z^=LayerNormx+self−attentionx,z=LayerNormz^+feed forwardz^.


## 2.3.2. The Decoder

There are alsoN blocks. However, each block consists of 3 subblocks: a self-attention block, a feed forward block, and an encoder-decoder attention block inserted between them. The residual connection and layer normalization are used similarly to the encoder. The encoder generates outputs step by step. The self-attention block only pays attention to the positions generated in the previous steps by using a mask. The mask prevents the decoder from paying attention to locations that have not been generated, so outputs can only be predicted based on the result z of the encoder and previous outputs.


## 2.3.3. Self-Attention

There are 3 components as follows: query (Q), key (K), and value (V), defined as follows:(10)AttentionQ,K,V=softmaxQKTdkV,where Q,K,andV are the parameters with the number of dimensions dk,dk,anddv respectively.


## 3. The Proposed Method

In this section, we present the graph embedding algorithm and propose our method to integrate the AMR graph embedding representation to various well-known NMT systems such as Seq2Seq, ConvSeq2Seq, and Transformer.


### 3.1. Graph-Level Information Representation

Figure2 depicts the graph encoder architecture based on the model of Xu et al. [11], with some enhancements to integrate more information about the edge of the graph.Figure 2
The graph encoder architecture.The directional graphG=V,E with the label on the edge eu,v∈E presents the relationship between the nodes u and v to which it connects. The process of learning to represent the node v∈V is as follows:(1)
We first transform the text attribute of nodev into a feature vector av by looking up the embedding matrix WE(2)
Next, we categorize the neighbors ofv into two subsets: forward neighbors, N⟶v and backward neighbors, N←v. Particularly, N⟶v returns the nodes that v directs to and vice versa. The information about the edge eu,v associated between the node v and the adjacent node u is combined as follows:(11)u←=u←+eu,v,∀u∈N←v,u⟶=u⟶+eu,v,∀u∈N⟶v.(3)
We aggregate the forward information ofv’s forward neighbors hu⟶k−1,∀u∈N⟶v into a single vector, hN⟶vk, where k∈1,…,K is the iteration index. We do this by using one of three AGG⟶ mentioned.(4)
Then, we concatenatev’s current forward representation, hv⟶k−1, with the new neighborhood vector, hN⟶vk. The result is passed to a feed forward layer, followed by a nonlinearity activation function σ, which updates the forward representation of v, to be used in the next iteration.(5)
Update the backward representation ofv, hv←k, using similar procedure in steps (3) and (4), but this time, we utilize backward representations rather than the forward representations and use AGG← to aggregate neighbor information.(6)
Repeat steps (3)∼(5) K times, and the concatenation of the final forward and backward representation is used as the final bidirectional representation of v.(12)zv=CONCAThv⟶K,hv←K,∀v∈V.As mentioned in steps (3) and (5), the representation association operation of nodev is performed with one of the following aggregation functions:(i)
Mean aggregator: performs the average calculation on each element ofhu⟶k−1,∀u∈N⟶v và hu←k−1,∀u∈N←v(ii)
GCN aggregator: it is quite similar to mean aggregator, except that the result is fed into a fully connected layer and a nonlinear activation function [14].(13)AGGk⟶=σWMEANhu⟶k+b,u∈N⟶v,AGGk←=σWMEANhu←k+b,u∈N←v,with MEAN as the function returning the average value, andσ as the nonlinear activation function.(iii)
Pooling aggregator: each node embedding vector is passed through a feed forward layer followed by the pooling operation (which can be max, min, and average):(14)AGGk⟶=maxσWphu⟶k+b,u∈N⟶v,AGGk←=maxσWphu←k+b,u∈N←v,with max as the maximum operation, andσ as the nonlinear activation function.


#### 3.1.1. Graph Embedding

Graph embeddingZ contains all the information on the graph and is calculated by one of the following two methods:(i)
Pooling based: the node embeddingszv,v∈V are passed through a linear transform network and performs pooling.(15)Z=poolingzv,∀v∈V.(ii)
Adding a super node: nodevs is pointed by all nodes in the graph. Using the algorithm in Section 3.1, the representation of vs is zvs. The representation of vs contains all information of the nodes that should be considered as representations of the graph or graph embedding.


### 3.2. Dual Attention Mechanism

The architecture of an integrated AMR machine translation model is illustrated in Figure3 with an English sentence input and a corresponding AMR graph. The proposed architecture consists of an encoder for the input sentence and a decoder with the input value resulting from the encoder. The main difference from the traditional decoder-encoder model is that there is an additional graph encoder to process information on graphs and to represent this information in a vector format. This vector is then combined with the hidden states of the encoder and fed into the decoder to find the corresponding representation in Vietnamese.Figure 3
Recommended architecture for AMR integration.We propose a specific integration method for the Seq2Seq model with sequential processing inSection 3.2.1 and focus on models with parallel processing such as ConvSeq2Seq, LightConvSeq2Seq, and Transformer in Section 3.2.2.


#### 3.2.1. Seq2Seq Model with the Sequential Processing Mechanism

The model (Figure4(a)) consists of two attention mechanisms operating independently: the original attention (left) learns the alignment between the result yi−1 and the hidden states hj,j∈1,n of the encoder and the graph attention learns to align between the output and the nodes in the AMR graph, yielding a context vector c^i−1. In particular, the computation of c^i−1 is as follows:(16)e^ij=asi−1,zj,α^ij=expe^ij∑k=1Vexpe^ikc^i=∑j=1Vα^ijzj,where a is a feed forward network, evaluating the matching between the nodes surrounding the position j and the input i.Figure 4
Integrating AMR to models with a parallel processing mechanism. (a) Seq2Seq (LSTM) model with the AMR. (b) ConvSeq2Seq-AMR model. (c) LightConvSeq2Seq-AMR model. (d) Transformer-AMR model.
(a)(b)(c)(d)These two context vectors are then combined with the decoder’s statesi and the embedding vector of yi−1 to calculate a probability distribution that determines yi.(17)Pvocab=SoftmaxWosi,yi−1,ci,c^i+bo.


#### 3.2.2. Models with a Parallel Processing Mechanism

On the contrary, with parallel processing, the model has no information about the statesi−1 of the decoder. In other words, except zv, no information about the graph is included in the calculation of attention. Besides, using only the states zv,v∈V along with the parallel computation leaves the model with no information about the association between the output and the AMR graph in step i−1. Consequently, the model cannot effectively learn the connection between the input sentence, the output sentence, and AMR graph, with a small increase of about 0.2 (experiments with LightConvSeq2Seq and Transformer). Therefore, the use of the graph embedding of Z should help the model obtain more information about the graph before the attention calculation. This has been proven with experimental results, which show an increase of the BLUE score by 0.6.Figures4(b)–4(d) describe the proposed model that integrates AMR with a dual attention mechanism. Regarding the LightConvSeq2Seq-AMR and Transformer-AMR models, the self-attention mechanism for the graph is similar to the description of the self-attention mechanism in Section 2.3 with the input being representations of nodes zv,∀v∈V instead of the state hi,∀i∈n. Regarding the ConvSeq2Seq-AMR model, experimental results show that utilizing Luong’s attention mechanism to learn the alignment between the graph and the output produced better results than the multistep attention.


## 3.1. Graph-Level Information Representation

Figure2 depicts the graph encoder architecture based on the model of Xu et al. [11], with some enhancements to integrate more information about the edge of the graph.Figure 2
The graph encoder architecture.The directional graphG=V,E with the label on the edge eu,v∈E presents the relationship between the nodes u and v to which it connects. The process of learning to represent the node v∈V is as follows:(1)
We first transform the text attribute of nodev into a feature vector av by looking up the embedding matrix WE(2)
Next, we categorize the neighbors ofv into two subsets: forward neighbors, N⟶v and backward neighbors, N←v. Particularly, N⟶v returns the nodes that v directs to and vice versa. The information about the edge eu,v associated between the node v and the adjacent node u is combined as follows:(11)u←=u←+eu,v,∀u∈N←v,u⟶=u⟶+eu,v,∀u∈N⟶v.(3)
We aggregate the forward information ofv’s forward neighbors hu⟶k−1,∀u∈N⟶v into a single vector, hN⟶vk, where k∈1,…,K is the iteration index. We do this by using one of three AGG⟶ mentioned.(4)
Then, we concatenatev’s current forward representation, hv⟶k−1, with the new neighborhood vector, hN⟶vk. The result is passed to a feed forward layer, followed by a nonlinearity activation function σ, which updates the forward representation of v, to be used in the next iteration.(5)
Update the backward representation ofv, hv←k, using similar procedure in steps (3) and (4), but this time, we utilize backward representations rather than the forward representations and use AGG← to aggregate neighbor information.(6)
Repeat steps (3)∼(5) K times, and the concatenation of the final forward and backward representation is used as the final bidirectional representation of v.(12)zv=CONCAThv⟶K,hv←K,∀v∈V.As mentioned in steps (3) and (5), the representation association operation of nodev is performed with one of the following aggregation functions:(i)
Mean aggregator: performs the average calculation on each element ofhu⟶k−1,∀u∈N⟶v và hu←k−1,∀u∈N←v(ii)
GCN aggregator: it is quite similar to mean aggregator, except that the result is fed into a fully connected layer and a nonlinear activation function [14].(13)AGGk⟶=σWMEANhu⟶k+b,u∈N⟶v,AGGk←=σWMEANhu←k+b,u∈N←v,with MEAN as the function returning the average value, andσ as the nonlinear activation function.(iii)
Pooling aggregator: each node embedding vector is passed through a feed forward layer followed by the pooling operation (which can be max, min, and average):(14)AGGk⟶=maxσWphu⟶k+b,u∈N⟶v,AGGk←=maxσWphu←k+b,u∈N←v,with max as the maximum operation, andσ as the nonlinear activation function.


### 3.1.1. Graph Embedding

Graph embeddingZ contains all the information on the graph and is calculated by one of the following two methods:(i)
Pooling based: the node embeddingszv,v∈V are passed through a linear transform network and performs pooling.(15)Z=poolingzv,∀v∈V.(ii)
Adding a super node: nodevs is pointed by all nodes in the graph. Using the algorithm in Section 3.1, the representation of vs is zvs. The representation of vs contains all information of the nodes that should be considered as representations of the graph or graph embedding.


## 3.1.1. Graph Embedding

Graph embeddingZ contains all the information on the graph and is calculated by one of the following two methods:(i)
Pooling based: the node embeddingszv,v∈V are passed through a linear transform network and performs pooling.(15)Z=poolingzv,∀v∈V.(ii)
Adding a super node: nodevs is pointed by all nodes in the graph. Using the algorithm in Section 3.1, the representation of vs is zvs. The representation of vs contains all information of the nodes that should be considered as representations of the graph or graph embedding.


## 3.2. Dual Attention Mechanism

The architecture of an integrated AMR machine translation model is illustrated in Figure3 with an English sentence input and a corresponding AMR graph. The proposed architecture consists of an encoder for the input sentence and a decoder with the input value resulting from the encoder. The main difference from the traditional decoder-encoder model is that there is an additional graph encoder to process information on graphs and to represent this information in a vector format. This vector is then combined with the hidden states of the encoder and fed into the decoder to find the corresponding representation in Vietnamese.Figure 3
Recommended architecture for AMR integration.We propose a specific integration method for the Seq2Seq model with sequential processing inSection 3.2.1 and focus on models with parallel processing such as ConvSeq2Seq, LightConvSeq2Seq, and Transformer in Section 3.2.2.


### 3.2.1. Seq2Seq Model with the Sequential Processing Mechanism

The model (Figure4(a)) consists of two attention mechanisms operating independently: the original attention (left) learns the alignment between the result yi−1 and the hidden states hj,j∈1,n of the encoder and the graph attention learns to align between the output and the nodes in the AMR graph, yielding a context vector c^i−1. In particular, the computation of c^i−1 is as follows:(16)e^ij=asi−1,zj,α^ij=expe^ij∑k=1Vexpe^ikc^i=∑j=1Vα^ijzj,where a is a feed forward network, evaluating the matching between the nodes surrounding the position j and the input i.Figure 4
Integrating AMR to models with a parallel processing mechanism. (a) Seq2Seq (LSTM) model with the AMR. (b) ConvSeq2Seq-AMR model. (c) LightConvSeq2Seq-AMR model. (d) Transformer-AMR model.
(a)(b)(c)(d)These two context vectors are then combined with the decoder’s statesi and the embedding vector of yi−1 to calculate a probability distribution that determines yi.(17)Pvocab=SoftmaxWosi,yi−1,ci,c^i+bo.


### 3.2.2. Models with a Parallel Processing Mechanism

On the contrary, with parallel processing, the model has no information about the statesi−1 of the decoder. In other words, except zv, no information about the graph is included in the calculation of attention. Besides, using only the states zv,v∈V along with the parallel computation leaves the model with no information about the association between the output and the AMR graph in step i−1. Consequently, the model cannot effectively learn the connection between the input sentence, the output sentence, and AMR graph, with a small increase of about 0.2 (experiments with LightConvSeq2Seq and Transformer). Therefore, the use of the graph embedding of Z should help the model obtain more information about the graph before the attention calculation. This has been proven with experimental results, which show an increase of the BLUE score by 0.6.Figures4(b)–4(d) describe the proposed model that integrates AMR with a dual attention mechanism. Regarding the LightConvSeq2Seq-AMR and Transformer-AMR models, the self-attention mechanism for the graph is similar to the description of the self-attention mechanism in Section 2.3 with the input being representations of nodes zv,∀v∈V instead of the state hi,∀i∈n. Regarding the ConvSeq2Seq-AMR model, experimental results show that utilizing Luong’s attention mechanism to learn the alignment between the graph and the output produced better results than the multistep attention.


## 3.2.1. Seq2Seq Model with the Sequential Processing Mechanism

The model (Figure4(a)) consists of two attention mechanisms operating independently: the original attention (left) learns the alignment between the result yi−1 and the hidden states hj,j∈1,n of the encoder and the graph attention learns to align between the output and the nodes in the AMR graph, yielding a context vector c^i−1. In particular, the computation of c^i−1 is as follows:(16)e^ij=asi−1,zj,α^ij=expe^ij∑k=1Vexpe^ikc^i=∑j=1Vα^ijzj,where a is a feed forward network, evaluating the matching between the nodes surrounding the position j and the input i.Figure 4
Integrating AMR to models with a parallel processing mechanism. (a) Seq2Seq (LSTM) model with the AMR. (b) ConvSeq2Seq-AMR model. (c) LightConvSeq2Seq-AMR model. (d) Transformer-AMR model.
(a)(b)(c)(d)These two context vectors are then combined with the decoder’s statesi and the embedding vector of yi−1 to calculate a probability distribution that determines yi.(17)Pvocab=SoftmaxWosi,yi−1,ci,c^i+bo.


## 3.2.2. Models with a Parallel Processing Mechanism

On the contrary, with parallel processing, the model has no information about the statesi−1 of the decoder. In other words, except zv, no information about the graph is included in the calculation of attention. Besides, using only the states zv,v∈V along with the parallel computation leaves the model with no information about the association between the output and the AMR graph in step i−1. Consequently, the model cannot effectively learn the connection between the input sentence, the output sentence, and AMR graph, with a small increase of about 0.2 (experiments with LightConvSeq2Seq and Transformer). Therefore, the use of the graph embedding of Z should help the model obtain more information about the graph before the attention calculation. This has been proven with experimental results, which show an increase of the BLUE score by 0.6.Figures4(b)–4(d) describe the proposed model that integrates AMR with a dual attention mechanism. Regarding the LightConvSeq2Seq-AMR and Transformer-AMR models, the self-attention mechanism for the graph is similar to the description of the self-attention mechanism in Section 2.3 with the input being representations of nodes zv,∀v∈V instead of the state hi,∀i∈n. Regarding the ConvSeq2Seq-AMR model, experimental results show that utilizing Luong’s attention mechanism to learn the alignment between the graph and the output produced better results than the multistep attention.


## 4. The Corpus

The corpus used to evaluate the model is IWSLT15 [15], which includes approximately 130,000 English-Vietnamese bilingual sentences taken from TED Talks presentations for the training set. For fine-tuning, we use the set called tst2012, which includes 1553 parallel pairs language. Besides, the test sets consist of tst2013 and tst2015, which include 1268 and 1080 English-Vietnamese bilingual pairs, respectively. The statistical information is given in Table 1.Table 1
Statistics on the corpus.
Corpus#Sentences#Tokens (English)#Tokens (Vietnamese)Training133K2.44 M2.87 MFine tuning (tst2012)155328 K34 KTest 1 (tst2013)126826.7 K33.6 KTest 2 (tst2015)108021 K26.2 KFor the preprocessing phase, byte-pair encoding (BPE) (https://github.com/rsennrich/subword-nmt) [16] with 8000 operations is utilized to deal with rare words and compound words for both English and Vietnamese, thereby significantly reducing the vocabulary size in English from 54111 to 5208 and in Vietnamese from 25335 to 3336.For AMR parsing, we use NeuralAmr toolkit (https://github.com/sinantie/NeuralAmr) [17] which implements the sequence-to-sequence models to the tasks of AMR parsing and AMR generation. Their model achieves competitive results of 62.1 SMATCH [18], the current best score (at the time doing this work, Jan 2020) reported without the significant use of external semantic resources. This tool produces AMR graphs represented in the PENMAN notation (https://www.isi.edu/natural-language/penman/penman.html) and in a linear form, as demonstrated in the AMR preprocessing example.


## 5. Experimental Configuration

The models are implemented in Python 3 and use the library Fairseq (https://fairseq.readthedocs.io/en/latest/#) [19].The configuration of the base models is as follows:(i)
Seq2Seq: we investigate the MT model with two types of LSTM which are uni-LSTM (one-directional) and bi-LSTM (two-directional). There are 512-word embedding dimensions, which utilize 512 LSTM hidden units in both the encoder and the decoder.(ii)
ConvSeq2Seq: it comprises 4 convolutional blocks and 512 hidden units for both the encoder and the decoder. The kernel size is 3.(iii)
LightConvSeq2Seq: it consists of 4 convolutional blocks with the kernel size of 3, 7, 15, and 31 for each block and applies to both the encoder and the decoder. Self-attention is adopted withH=8 heads.(iv)
Transformer: it hasN=6 blocks for both the encoder and the decoder. The word embedding dim is set to 512 and 2048 for the feed forward network. Self-attention used with the number of heads was 8.The proposed models have the same configuration as the base model. Besides, the graph encoder used 128-dimensional embedding for the representation of both edge and node. We stacked 2 layers of the graph encoder and aggregating information from neighboring nodes with the mean aggregator for LSTM and max pooling with the rest of the models.During training, Adam optimizer [20] is used with a fixed learning rate of 0.001 for LSTM and ConvSeq2Seq, 0.0002 for LightConvSeq2Seq, and 0.0005 for Transformer.Besides the basic models presented above, the results of the proposed model are also compared with the method of Song et al. [10]. To make a fair comparison, we have retrained Song’s model with the same preprocessed dataset and tuned hyperparameters.After the models are trained, the BLEU score [21] was used to evaluate the translation quality. We also apply the bootstrap resampling method [22] to measure the statistical significance p<0.05 of BLEU score differences between translation outputs of proposed models compared to the baseline.


## 6. Results and Discussion

In this section, we present our experimental results and our analyzes on the results.


### 6.1. Results

Once the models have been trained, a beam search with the size of 5 is utilized to find a translation that maximizes the conditional probabilities.With both the test sets tst2013 and tst2015, the proposed models are proven to be superior to the corresponding base model. In particular, as given in Table2, with uni-LSTM-AMR-F and bi-LSTM-AMR, the BLEU scores are 27.21 and 29.29, respectively, which are 1.09 and 3.17 higher than Song’s method [10]. Similarly, with the set tst2015, bi-LSTM-AMR improved BLEU by 2.83, compared to Song’s method. This shows that despite using the double attention mechanism, bi-LSTM-AMR and uni-LSTM-AMR can integrate the information from AMR more effectively, thereby producing better translation results.Table 2
Experimental results on Seq2Seq using one- and two-directional LSTMs.
ModelBLEUtst2013tst2015Song’s method26.1223.58Uni-LSTM-AMR26.9724.80Uni-LSTM-AMR-F27.2124.86Uni-LSTM-AMR-B26.6124.66Bi-LSTM-AMR29.2926.41Bi-LSTM-AMR-F28.6726.20Bi-LSTM-AMR-B28.3626.04The bold values are the highest results for each group of models.Meanwhile, when LightConvSeq2Seq is run on tst2013 and tst 2015, the BLEU scores are only 27.47 and 25.09, respectively. However, when integrating the AMR into the system, the BLEU score increased significantly by 1.0 and 0.58 on tst2013 and tst2015, respectively. Besides, LightConvSeq2Seq-AMR-F and LightConvSeq2Seq-AMR-B, which were integrated graph information from one direction, also outperform LightConvSeq2Seq, as given in Table3.Table 3
Experimental results on LightConvSeq2Seq.
ModelBLEUtst2013tst2015LightConvSeq2Seq27.4725.09LightConvSeq2Seq-AMR-F27.7125.05LightConvSeq2Seq-AMR-B27.8425.27LightConvSeq2Seq-AMR28.4625.67The bold values are the highest results when evaluating each model for the “tst2013” and “tst2015” testsets.As given in Table4, ConvSeq2Seq also shows an improvement in machine translation quality with an increase in the BLEU score to about 0.3 for ConvSeq2Seq-AMR with tst2013. However, there is a BLUE decrease of 0.08 with tst2015. However, the ConvSeq2Seq-AMR-F model achieves the best results when integrating information from the forward neighbors. An increase of 0.1 in BLEU is observed with tst2013 and 0.5 with tst2015. Similar to Transformer, integrating information from the forward and backward neighbors in Transformer-AMR is not effective, with only an increase of 0.09 over the base model with tst2013. Only combining information from the forward neighbors in Transformer-AMR-F achieves a noticeable BLEU score of 28.88 and 26.28 with tst2013 and tst2015, respectively, which signal an increase of 0.28 and 0.52 compared to Transformer.Table 4
Experimental results on ConvSeq2Seq and Transformer.
ModelBLEUtst2013tst2015ConvSeq2Seq26.9824.78ConvSeq2Seq-AMR27.3024.70ConvSeq2Seq-AMR-F27.4025.20ConvSeq2Seq-AMR-B26.7324.53Transformer28.6025.76Transformer-AMR28.6925.91Transformer-AMR-F28.8826.28Transformer-AMR-B28.6926.01The bold values are the highest results for each group of models.


### 6.2. The Effect of AMR on the NMT Model

According to the results presented inSection 6.1, the bi-LSTM-AMR and LightConvSeq2Seq-AMR models improve BLEU more than the other two models, ConvSeq2Seq and Transformer. Therefore, to analyze the impact of AMR on the machine translation system, bi-LSTM-AMR and LightConvSeq2Seq-AMR models are selected for further training to examine graph elements such as information integration directions, graph encoding layers, and aggregators.


#### 6.2.1. Bi-LSTM-AMR

(i). Direction and Depth. Figure 5 depicts the change in performance when adjusting the number of graph encoding layers. The mean aggregator is used to combine information from neighbors. In general, bi-LSTM-AMR and uni-LSTM-AMR-B show the highest translation quality throughout the 30 examined layers. However, an increase in the number of layers does not always help the model achieve a higher BLEU. A decrease in BLEU scores is also observed. The more stacked layers there are, the greater the amount of information the model could learn, which ultimately leads to the overfitting problem due to saturated information. All models obtain the best results with only 2 or 3 graph coding layers. As the number of layers increases, the BLEU scores decrease. Nevertheless, the results seem more consistent and less fluctuating with bi-LSTM than with uni-LSTM.Figure 5
Experimental results on tst2013 with a change in the number of graph encoding layers.
(a)(b)There are three aggregators used for aggregating information from neighboring nodes: mean aggregator (MA), max-pooling (MP) aggregator, and GCN aggregator (GCN-A). The strategy of using information from one direction (forward or backward) is also considered to make more accurate statements about the effect of the aggregator on the effectiveness of the model. The results in Table5 show that Bi-LSTM-AMR-MA achieved the highest result on the two test sets with the BLEU scores of 29.29 and 26.41, respectively. Meanwhile, uni-LSTM-AMR-MA, which uses information from both sides, achieved lower BLEU scores than the variants uni-LSTM-AMR-F and uni-LSTM-AMR-B, which only combines information from the forward and the backward neighbors, respectively. Moreover, bi-LSTM-AMR-MA outperforms bi-LSTM-AMR-F and bi-LSTM-AMR-B due to its ability to capture information from two directions during the node embedding learning and combine with information from the bi-LSTM encoder. Therefore, the LSTM decoder can leverage information from the graph more efficiently to improve the machine translation quality. This shows that bidirectional aggregation is more useful when combined with a bidirectional LSTM encoder. Accordingly, uni-LSTM-AMR-F-MP and uni-LSTM-AMR-B-MP, which only combine information from one direction, achieve good results when used with a unidirectional LSTM encoder.Table 5
The effect of aggregators on the machine translation quality.
Modeltst2013tst2015MAMPGCN-AMAMPGCN-AUni-LSTM-AMR26.9726.5426.5524.8124.2224.47Uni-LSTM-AMR-F26.8527.2126.7624.5924.8624.48Uni-LSTM-AMR-B26.6127.2226.4224.6624.8524.47Bi-LSTM-AMR29.2928.9428.3826.4126.2425.59Bi-LSTM-AMR-F28.6728.9928.4326.2025.8125.70Bi-LSTM-AMR-B28.3628.4028.4126.0425.9025.81


#### 6.2.2. LightConvSeq2Seq-AMR

Similar to bi-LSTM-AMR, the LightConvSeq2Seq-AMR model is also affected by different aggregators. In particular, as given in Table6, the mean aggregator (MA) yields better results on average values than the rest. The results with tst2015 show that all the three modes with MA both achieve much higher results than the rest of models. On the contrary, the GCN-A results are the lowest, similar to Seq2Seq. This proves that the information combination of GCN-A is not as efficient as those of MA and MP.Table 6
Experimental results of aggregators on LightConvSeq2Seq-AMR.
Modeltst2013tst2015MAMPGCN-AMAMPGCN-ALightConvSeq2Seq-AMR28.2028.4627.7625.4925.0525.33LightConvSeq2Seq-AMR-F27.8227.7127.5925.9625.2725.39LightConvSeq2Seq-AMR-B28.2527.8428.2725.6625.6725.52The bold values are the highest results when evaluating each model on aggregators (i.e., MA, MP, and GCN-A) for testset (i.e., tst2013 or tst2015).Figure6 shows the change of BLEU when stacking convolutional blocks in the encoder and the decoder and the effect of heads H in self-attention. On both sides, the BLEU scores increase when the number of heads increases. In particular, the figure on the left shows the LightConvSeq2Seq-AMR model with the configuration 4,4, which stacked 4 convolutional blocks at the encoder and 4 convolutional blocks at the decoder, and 6,6 yields the best results. The BLEU scores are approximately 28 and 27.6 with just 1 head and then increases to 28.46 and 28.2 when H=8. However, with an additional graph encoding layer, the 4,4 configuration is inferior to the 6,6 configuration. This configuration yields the highest results at H=1 with BLEU approximately 28.4 and observes a slight decrease as H approaches 8. Meanwhile, 5,4 and 5,5 configurations tend to decline sharply as H increases from 1 to 2 ≈−0.3 and continues to decline slightly until H=8. Meanwhile, the two reconfigurations tend to be the opposite when adding a graph encoding layer, as shown on the right figure in Figure 6. The remaining 4,3 configuration yields the lowest results for the 2 graph encoding layer options. The results also fluctuate more with 3 layers, as opposed to being nearly constant at 2 layers.Figure 6
Experimental results on the effect of headsH of the graph attention and the depth of the LightConvSeq2Seq-AMR model. (a) LightConvSeq2Seq-AMR with two graph coding layers. (b) The model with three layers.
(a)(b)


## 6.1. Results

Once the models have been trained, a beam search with the size of 5 is utilized to find a translation that maximizes the conditional probabilities.With both the test sets tst2013 and tst2015, the proposed models are proven to be superior to the corresponding base model. In particular, as given in Table2, with uni-LSTM-AMR-F and bi-LSTM-AMR, the BLEU scores are 27.21 and 29.29, respectively, which are 1.09 and 3.17 higher than Song’s method [10]. Similarly, with the set tst2015, bi-LSTM-AMR improved BLEU by 2.83, compared to Song’s method. This shows that despite using the double attention mechanism, bi-LSTM-AMR and uni-LSTM-AMR can integrate the information from AMR more effectively, thereby producing better translation results.Table 2
Experimental results on Seq2Seq using one- and two-directional LSTMs.
ModelBLEUtst2013tst2015Song’s method26.1223.58Uni-LSTM-AMR26.9724.80Uni-LSTM-AMR-F27.2124.86Uni-LSTM-AMR-B26.6124.66Bi-LSTM-AMR29.2926.41Bi-LSTM-AMR-F28.6726.20Bi-LSTM-AMR-B28.3626.04The bold values are the highest results for each group of models.Meanwhile, when LightConvSeq2Seq is run on tst2013 and tst 2015, the BLEU scores are only 27.47 and 25.09, respectively. However, when integrating the AMR into the system, the BLEU score increased significantly by 1.0 and 0.58 on tst2013 and tst2015, respectively. Besides, LightConvSeq2Seq-AMR-F and LightConvSeq2Seq-AMR-B, which were integrated graph information from one direction, also outperform LightConvSeq2Seq, as given in Table3.Table 3
Experimental results on LightConvSeq2Seq.
ModelBLEUtst2013tst2015LightConvSeq2Seq27.4725.09LightConvSeq2Seq-AMR-F27.7125.05LightConvSeq2Seq-AMR-B27.8425.27LightConvSeq2Seq-AMR28.4625.67The bold values are the highest results when evaluating each model for the “tst2013” and “tst2015” testsets.As given in Table4, ConvSeq2Seq also shows an improvement in machine translation quality with an increase in the BLEU score to about 0.3 for ConvSeq2Seq-AMR with tst2013. However, there is a BLUE decrease of 0.08 with tst2015. However, the ConvSeq2Seq-AMR-F model achieves the best results when integrating information from the forward neighbors. An increase of 0.1 in BLEU is observed with tst2013 and 0.5 with tst2015. Similar to Transformer, integrating information from the forward and backward neighbors in Transformer-AMR is not effective, with only an increase of 0.09 over the base model with tst2013. Only combining information from the forward neighbors in Transformer-AMR-F achieves a noticeable BLEU score of 28.88 and 26.28 with tst2013 and tst2015, respectively, which signal an increase of 0.28 and 0.52 compared to Transformer.Table 4
Experimental results on ConvSeq2Seq and Transformer.
ModelBLEUtst2013tst2015ConvSeq2Seq26.9824.78ConvSeq2Seq-AMR27.3024.70ConvSeq2Seq-AMR-F27.4025.20ConvSeq2Seq-AMR-B26.7324.53Transformer28.6025.76Transformer-AMR28.6925.91Transformer-AMR-F28.8826.28Transformer-AMR-B28.6926.01The bold values are the highest results for each group of models.


## 6.2. The Effect of AMR on the NMT Model

According to the results presented inSection 6.1, the bi-LSTM-AMR and LightConvSeq2Seq-AMR models improve BLEU more than the other two models, ConvSeq2Seq and Transformer. Therefore, to analyze the impact of AMR on the machine translation system, bi-LSTM-AMR and LightConvSeq2Seq-AMR models are selected for further training to examine graph elements such as information integration directions, graph encoding layers, and aggregators.


### 6.2.1. Bi-LSTM-AMR

(i). Direction and Depth. Figure 5 depicts the change in performance when adjusting the number of graph encoding layers. The mean aggregator is used to combine information from neighbors. In general, bi-LSTM-AMR and uni-LSTM-AMR-B show the highest translation quality throughout the 30 examined layers. However, an increase in the number of layers does not always help the model achieve a higher BLEU. A decrease in BLEU scores is also observed. The more stacked layers there are, the greater the amount of information the model could learn, which ultimately leads to the overfitting problem due to saturated information. All models obtain the best results with only 2 or 3 graph coding layers. As the number of layers increases, the BLEU scores decrease. Nevertheless, the results seem more consistent and less fluctuating with bi-LSTM than with uni-LSTM.Figure 5
Experimental results on tst2013 with a change in the number of graph encoding layers.
(a)(b)There are three aggregators used for aggregating information from neighboring nodes: mean aggregator (MA), max-pooling (MP) aggregator, and GCN aggregator (GCN-A). The strategy of using information from one direction (forward or backward) is also considered to make more accurate statements about the effect of the aggregator on the effectiveness of the model. The results in Table5 show that Bi-LSTM-AMR-MA achieved the highest result on the two test sets with the BLEU scores of 29.29 and 26.41, respectively. Meanwhile, uni-LSTM-AMR-MA, which uses information from both sides, achieved lower BLEU scores than the variants uni-LSTM-AMR-F and uni-LSTM-AMR-B, which only combines information from the forward and the backward neighbors, respectively. Moreover, bi-LSTM-AMR-MA outperforms bi-LSTM-AMR-F and bi-LSTM-AMR-B due to its ability to capture information from two directions during the node embedding learning and combine with information from the bi-LSTM encoder. Therefore, the LSTM decoder can leverage information from the graph more efficiently to improve the machine translation quality. This shows that bidirectional aggregation is more useful when combined with a bidirectional LSTM encoder. Accordingly, uni-LSTM-AMR-F-MP and uni-LSTM-AMR-B-MP, which only combine information from one direction, achieve good results when used with a unidirectional LSTM encoder.Table 5
The effect of aggregators on the machine translation quality.
Modeltst2013tst2015MAMPGCN-AMAMPGCN-AUni-LSTM-AMR26.9726.5426.5524.8124.2224.47Uni-LSTM-AMR-F26.8527.2126.7624.5924.8624.48Uni-LSTM-AMR-B26.6127.2226.4224.6624.8524.47Bi-LSTM-AMR29.2928.9428.3826.4126.2425.59Bi-LSTM-AMR-F28.6728.9928.4326.2025.8125.70Bi-LSTM-AMR-B28.3628.4028.4126.0425.9025.81


### 6.2.2. LightConvSeq2Seq-AMR

Similar to bi-LSTM-AMR, the LightConvSeq2Seq-AMR model is also affected by different aggregators. In particular, as given in Table6, the mean aggregator (MA) yields better results on average values than the rest. The results with tst2015 show that all the three modes with MA both achieve much higher results than the rest of models. On the contrary, the GCN-A results are the lowest, similar to Seq2Seq. This proves that the information combination of GCN-A is not as efficient as those of MA and MP.Table 6
Experimental results of aggregators on LightConvSeq2Seq-AMR.
Modeltst2013tst2015MAMPGCN-AMAMPGCN-ALightConvSeq2Seq-AMR28.2028.4627.7625.4925.0525.33LightConvSeq2Seq-AMR-F27.8227.7127.5925.9625.2725.39LightConvSeq2Seq-AMR-B28.2527.8428.2725.6625.6725.52The bold values are the highest results when evaluating each model on aggregators (i.e., MA, MP, and GCN-A) for testset (i.e., tst2013 or tst2015).Figure6 shows the change of BLEU when stacking convolutional blocks in the encoder and the decoder and the effect of heads H in self-attention. On both sides, the BLEU scores increase when the number of heads increases. In particular, the figure on the left shows the LightConvSeq2Seq-AMR model with the configuration 4,4, which stacked 4 convolutional blocks at the encoder and 4 convolutional blocks at the decoder, and 6,6 yields the best results. The BLEU scores are approximately 28 and 27.6 with just 1 head and then increases to 28.46 and 28.2 when H=8. However, with an additional graph encoding layer, the 4,4 configuration is inferior to the 6,6 configuration. This configuration yields the highest results at H=1 with BLEU approximately 28.4 and observes a slight decrease as H approaches 8. Meanwhile, 5,4 and 5,5 configurations tend to decline sharply as H increases from 1 to 2 ≈−0.3 and continues to decline slightly until H=8. Meanwhile, the two reconfigurations tend to be the opposite when adding a graph encoding layer, as shown on the right figure in Figure 6. The remaining 4,3 configuration yields the lowest results for the 2 graph encoding layer options. The results also fluctuate more with 3 layers, as opposed to being nearly constant at 2 layers.Figure 6
Experimental results on the effect of headsH of the graph attention and the depth of the LightConvSeq2Seq-AMR model. (a) LightConvSeq2Seq-AMR with two graph coding layers. (b) The model with three layers.
(a)(b)


## 6.2.1. Bi-LSTM-AMR

(i). Direction and Depth. Figure 5 depicts the change in performance when adjusting the number of graph encoding layers. The mean aggregator is used to combine information from neighbors. In general, bi-LSTM-AMR and uni-LSTM-AMR-B show the highest translation quality throughout the 30 examined layers. However, an increase in the number of layers does not always help the model achieve a higher BLEU. A decrease in BLEU scores is also observed. The more stacked layers there are, the greater the amount of information the model could learn, which ultimately leads to the overfitting problem due to saturated information. All models obtain the best results with only 2 or 3 graph coding layers. As the number of layers increases, the BLEU scores decrease. Nevertheless, the results seem more consistent and less fluctuating with bi-LSTM than with uni-LSTM.Figure 5
Experimental results on tst2013 with a change in the number of graph encoding layers.
(a)(b)There are three aggregators used for aggregating information from neighboring nodes: mean aggregator (MA), max-pooling (MP) aggregator, and GCN aggregator (GCN-A). The strategy of using information from one direction (forward or backward) is also considered to make more accurate statements about the effect of the aggregator on the effectiveness of the model. The results in Table5 show that Bi-LSTM-AMR-MA achieved the highest result on the two test sets with the BLEU scores of 29.29 and 26.41, respectively. Meanwhile, uni-LSTM-AMR-MA, which uses information from both sides, achieved lower BLEU scores than the variants uni-LSTM-AMR-F and uni-LSTM-AMR-B, which only combines information from the forward and the backward neighbors, respectively. Moreover, bi-LSTM-AMR-MA outperforms bi-LSTM-AMR-F and bi-LSTM-AMR-B due to its ability to capture information from two directions during the node embedding learning and combine with information from the bi-LSTM encoder. Therefore, the LSTM decoder can leverage information from the graph more efficiently to improve the machine translation quality. This shows that bidirectional aggregation is more useful when combined with a bidirectional LSTM encoder. Accordingly, uni-LSTM-AMR-F-MP and uni-LSTM-AMR-B-MP, which only combine information from one direction, achieve good results when used with a unidirectional LSTM encoder.Table 5
The effect of aggregators on the machine translation quality.
Modeltst2013tst2015MAMPGCN-AMAMPGCN-AUni-LSTM-AMR26.9726.5426.5524.8124.2224.47Uni-LSTM-AMR-F26.8527.2126.7624.5924.8624.48Uni-LSTM-AMR-B26.6127.2226.4224.6624.8524.47Bi-LSTM-AMR29.2928.9428.3826.4126.2425.59Bi-LSTM-AMR-F28.6728.9928.4326.2025.8125.70Bi-LSTM-AMR-B28.3628.4028.4126.0425.9025.81


## 6.2.2. LightConvSeq2Seq-AMR

Similar to bi-LSTM-AMR, the LightConvSeq2Seq-AMR model is also affected by different aggregators. In particular, as given in Table6, the mean aggregator (MA) yields better results on average values than the rest. The results with tst2015 show that all the three modes with MA both achieve much higher results than the rest of models. On the contrary, the GCN-A results are the lowest, similar to Seq2Seq. This proves that the information combination of GCN-A is not as efficient as those of MA and MP.Table 6
Experimental results of aggregators on LightConvSeq2Seq-AMR.
Modeltst2013tst2015MAMPGCN-AMAMPGCN-ALightConvSeq2Seq-AMR28.2028.4627.7625.4925.0525.33LightConvSeq2Seq-AMR-F27.8227.7127.5925.9625.2725.39LightConvSeq2Seq-AMR-B28.2527.8428.2725.6625.6725.52The bold values are the highest results when evaluating each model on aggregators (i.e., MA, MP, and GCN-A) for testset (i.e., tst2013 or tst2015).Figure6 shows the change of BLEU when stacking convolutional blocks in the encoder and the decoder and the effect of heads H in self-attention. On both sides, the BLEU scores increase when the number of heads increases. In particular, the figure on the left shows the LightConvSeq2Seq-AMR model with the configuration 4,4, which stacked 4 convolutional blocks at the encoder and 4 convolutional blocks at the decoder, and 6,6 yields the best results. The BLEU scores are approximately 28 and 27.6 with just 1 head and then increases to 28.46 and 28.2 when H=8. However, with an additional graph encoding layer, the 4,4 configuration is inferior to the 6,6 configuration. This configuration yields the highest results at H=1 with BLEU approximately 28.4 and observes a slight decrease as H approaches 8. Meanwhile, 5,4 and 5,5 configurations tend to decline sharply as H increases from 1 to 2 ≈−0.3 and continues to decline slightly until H=8. Meanwhile, the two reconfigurations tend to be the opposite when adding a graph encoding layer, as shown on the right figure in Figure 6. The remaining 4,3 configuration yields the lowest results for the 2 graph encoding layer options. The results also fluctuate more with 3 layers, as opposed to being nearly constant at 2 layers.Figure 6
Experimental results on the effect of headsH of the graph attention and the depth of the LightConvSeq2Seq-AMR model. (a) LightConvSeq2Seq-AMR with two graph coding layers. (b) The model with three layers.
(a)(b)


## 7. Conclusions

We proposed a method to integrate the AMR graphs into popular machine translation architectures such as Seq2Seq, ConvSeq2Seq, and Transformer. Structured semantic information from AMR graphs can supplement the context information in the translation model for a better representation of abstract information. Experimental results show that AMR graphs yield better results than other representations such as dependency trees or semantic roles.For future studies, we plan to examine other methods to integrate more complex semantic graphs, such as Prague Semantic Dependencies, Elementary Dependency Structures, and Universal Conceptual Cognitive Annotation, and investigate different encoding methods suitable for a range of semantic graphs.



---

*Source: 9939389-2021-07-08.xml*