Venue: EMNLP 2017

# Towards Bidirectional Hierarchical Representations

for Attention-Based Neural Machine Translation

Baosong Yang\({}^{\dagger}\) Derek F. Wong\({}^{\dagger}\)1 Tong Xiao\({}^{\ddagger}\) Lidia S. Chao\({}^{\dagger}\) Jingbo Zhu\({}^{\ddagger}\)

\({}^{\dagger}\)NLP\({}^{2}\)CT Lab, Department of Computer and Information Science, University of Macau, Macau, China

\({}^{\ddagger}\)NiuTrans Lab, Northeastern University, Shenyang, China nlp2ct.baosong@gmail.com, {derekfw,lidiasc}@umac.mo, {xiaotong,zhujingbo}@mail.neu.edu.cn

Footnote 1: Corresponding author

###### Abstract

This paper proposes a hierarchical attentional neural translation model which focuses on enhancing source-side hierarchical representations by covering both local and global semantic information using a bidirectional tree-based encoder. To maximize the predictive likelihood of target words, a weighted variant of an attention mechanism is used to balance the attentive information between lexical and phrase vectors. Using a tree-based rare word encoding, the proposed model is extended to sub-word level to alleviate the out-of-vocabulary (OOV) problem. Empirical results reveal that the proposed model significantly outperforms sequence-to-sequence attention-based and tree-based neural translation models in English-Chinese translation tasks.

## 1 Introduction

Neural machine translation (NMT) automatically learns the abstract features of and semantic relationship between the source and target sentences, and has recently given state-of-the-art results for various translation tasks Kalchbrenner and Blunsom (2013); Sutskever et al. (2014); Bahdanau et al. (2015). The most widely used model is the encoder-decoder framework Sutskever et al. (2014), in which the source sentence is encoded into a dense representation, followed by a decoding process which generates the target translation. By exploiting the attention mechanism Bahdanau et al. (2015), the generation of target words is conditional on the source hidden states, rather than on the context vector alone. From a model architecture perspective, prior studies of the attentive encoder-decoder translation model are mainly divided into two types.

The **sequence-to-sequence model** treats a sentence as a sequence of tokens. The most fundamental approaches transform the source sentence sequentially into a fixed-length context vector, and the annotation vector of each word summarizes the preceding words Sutskever et al. (2014); Cho et al. (2014). Although Bahdanau et al. (2015) used a bidirectional recurrent neural network RNN Schuster and Paliwal (1997) to consider preceding and following words jointly, these sequential representations are insufficient to fully capture the semantics of a sentence, due to the fact that they do not account for the syntactic interpretations of sentence structure Eriguchi et al. (2016); Tai et al. (2015). By incorporating additional features into a sequential model, Sennrich and Haddow (2016) and Stahlberg et al. (2016) suggest that a greater amount of linguistic information can improve the translation performance.

The **tree-to-sequence model** encodes a source sentence according t

Figure 1: Induction of phrase and sentence representations over the syntactic structure of a sentence.



over the sentence. The existing tree-based encoders [14, 20] recursively generate phrase (sentence) representations in a bottom-up fashion, whereby the annotation vector of each phrase is derived from its constituent sub-phrases. As a result, the learned representations are limited to local information, while failing to capture the global meaning of a sentence. As illustrated in Figure 1, the phrases "take up"1 and "a position"2 have different meanings in different contexts. However, in composing the representations \(h_{\text{VP}_{3}}\) and \(h_{\text{NP}_{7}}\) for phrases VP\({}_{3}\) and NP\({}_{7}\), the current approaches do not account for the differences in meaning which arise as a result of ignoring the neighboring context as well as the remote context, i.e. \(h_{\text{NP}^{*}_{7}}\gets h_{\text{PP${}_{8}$}}\) (sibling) and \(h_{\text{VP${}_{3}$}}\gets h_{\text{NP${}_{7}$}}\) (child of sibling). More specifically, at the encoding step \(t\), the generated phrase is based on the results at the previous time steps \(h_{t-1}\) and \(h_{t-2}\), but has no information about the parent phrases \(h_{t^{\prime}}\) for \(t^{\prime}>t\).

Footnote 1: Take up has the meanings of _start doing something new_, _use space/time_, _accept an offer_, etc.

To address the above problems, we propose a novel architecture, a bidirectional hierarchical encoder, which extends the existing attentive tree-structured models [5]. In contrast to the model of Eriguchi et al. eriguchi2016efficient, we first use a bidirectional RNN [9] at lexical level to concatenate the forward and backward states as the hidden states of source words, to capture the preceding and following contexts (described in Section 3.1). Secondly, we propose a bidirectional tree-based encoder (described in Section 3.2), in which the original bottom-up encoding model is extended using an additional top-down encoding process. In the bidirectional hierarchical model, the vector representations of the sentence, phrases as well as words, are therefore based on the global context rather than local information.

To effectively leverage hierarchical representations in generating the target words, we adopt a variant weighted tree-based attention mechanism (described in Section 3.4) in which a time-dependent gating scalar is used to control the proportion of conditional information between the word and phrase vectors. To alleviate the out-of-vocabulary (OOV) problem, we further extend the proposed tree-based model to the sub-word level by integrating byte-pair encoding (BPE) [10] into the tree-based model (as described in Section 3.3). Experimental results for the NIST English-to-Chinese translation task reveal that the proposed model significantly outperforms the vanilla tree-based [5] and sequential NMT models [1] (Section 4.1).

## 2 Tree-Based Neural Machine Translation

A neural machine translation system (NMT) aims to use a single neural network to build a translation model, which is trained to maximize the conditional distribution of sentence pairs using a parallel training corpus [1, 13, 14]. By incorporating syntactic information, the tree-based NMT exploits an additional syntactic structure of the source sentence to improve the translation. Since most existing NMTs generate one target word at a time, given a source sentence \(\textbf{x}=(x_{1},...,x_{N})\) and its corresponding syntactic tree **tr**, the conditional probability of a target sentence \(\textbf{y}=(y_{1},..,y_{M})\) is formally expressed as:

\[p(\textbf{y}~{}|~{}\textbf{x},\textbf{tr})=\prod_{1}^{M}p(y_{j}~{}|~{}y_{1}, ...,y_{j-1},\textbf{x},\textbf{tr};\theta),\]

where \(\theta\) represents the model parameters. A tree-based NMT consists of a tree-based encoder and a decoder.

### Tree-Based Encoder

In a tree-based encoder, the source language **x** is encoded according to a given syntactic structure

Figure 2: The tree-based model of Eriguchi et al. eriguchi2016efficient comprising a structured and sequential encoder.

 



**tr** of the sentence. As shown in Figure 2, Eriguchi et al. (2016) employed a forward Long Short-Term Memory (LSTM) Hochreiter and Schmidhuber (1997); Gers et al. (2000) recurrent neural network (RNN) to encode the lexical nodes and a tree-LSTM Tai et al. (2015) to generate the phrase representations in a bottom-up fashion. In the present study, we utilize the gated recurrent unit (GRU) Cho et al. (2014b) instead of an LSTM, in view of its comparable performance Chung et al. (2014) and since it yields even better results for certain tasks Jozefowicz et al. (2015). The lexical annotation vectors \((h_{1}^{l},...,h_{N}^{l})\) are sequentially generated by using a GRU. The \(i\)-th leaf node vector is calculated as:

\[h_{i}^{l}=f_{GRU}^{l}(x_{i},h_{i-1}^{l}), \tag{1}\]

where \(x_{i}\) is the \(i\)-th source word embedding and \(h_{i-1}^{l}\) denotes the previous hidden state. The parent hidden state \(h_{i,j}^{\uparrow}\) summarizes its left child \(h_{i,k}^{\uparrow}\) and right child \(h_{i+1,j}^{\uparrow}\) (\(i<k<j\)) by applying the tree-GRU Zhou et al. (2016) as follows:

\[z_{i,j}^{\uparrow} =\sigma(U_{(z)}^{L}h_{i,k}^{\uparrow}+U_{(z)}^{R}h_{k+1,j}^{ \uparrow}+b_{(z)}^{\uparrow})\] \[r_{i,k}^{\uparrow} =\sigma(U_{(rL)}^{L}h_{i,k}^{\uparrow}+U_{(rL)}^{R}h_{k+1,j}^{ \uparrow}+b_{(rL)}^{\uparrow})\] \[r_{k+1,j}^{\uparrow} =\sigma(U_{(rvR)}^{L}h_{i,k}^{\uparrow}+U_{(vR)}^{R}h_{k+1,j}^{ \downarrow}+b_{(rR)}^{\uparrow})\] \[\widetilde{h}_{i,j}^{\uparrow} =\,\tanh(U_{(h)}^{L}(r_{i,k}^{\uparrow}\odot h_{i,k}^{\uparrow})\] \[~{}~{}~{}~{}~{}+U_{(h)}^{R}(r_{k+1,j}^{\uparrow}\odot h_{k+1,j}^{ \uparrow})+b_{(h)}^{\uparrow})\] \[h_{i,j}^{\uparrow} =z_{i,j}^{\uparrow}\widetilde{h}_{i,j}^{\uparrow}+(1-z_{i,j}^{ \uparrow})(h_{i,k}^{\uparrow}+h_{k+1,j}^{\uparrow}),\]

where \(z_{i,j}^{\uparrow}\) is the update gate; \(r_{i,k}^{\uparrow}\), \(r_{k+1,j}^{\uparrow}\) are the reset gates for the left and right children; \(\widetilde{h}_{i,j}^{\uparrow}\) denotes the candidate activation; \(U_{(\cdot)}^{L}\) and \(U_{(\cdot)}^{R}\) represent weight matrices; \(b_{(\cdot)}^{\uparrow}\) denote bias vectors; \(\sigma\) is the logistic sigmoid function; and the operator \(\odot\) denotes element-wise multiplication between vectors. The phrase representations are recursively built in an upward direction.

### Decoding with a Tree-Based Attention Mechanism

In generating the target words, we employ a sequential decoder with an input-feeding method Luong et al. (2015) and attention mechanism Bahdanau et al. (2015). The conditional probability of the \(j\)-th target word \(y_{j}\) is calculated using a non-linear function \(f_{softmax}\):

\[p(y_{j}\mid y_{1},...,y_{j-1},\textbf{x},\textbf{tr};\theta)=f_{ softmax}(c_{j}),\]

where \(c_{j}\) is the composite hidden state, which consists of a target hidden state \(s_{j}\) and a context vector \(d_{j}\):

\[c_{j}=f_{\tanh}([s_{j},d_{j}]).\]

Given the previous target word \(y_{j-1}\), the concatenation of the previous hidden state \(s_{j-1}\) and the previous context vector \(c_{j-1}\) (input-feeding) Luong et al. (2015), \(s_{j}\), is calculated using a standard sequential GRU network:

\[s_{j}=f_{gra}^{dec}(y_{j-1},[s_{j-1},c_{j-1}]).\]

The context vector \(d_{j}\) is computed using an attention model which is used to softly summarize the attended part of the source-side representations. Eriguchi et al. (2016) adopted a tree-based attention mechanism to consider both the word and phrase vectors:

\[d_{j}=\sum_{i=1}^{N}\alpha_{j}(i)h_{i}^{l}+\sum_{k=1}^{N-1}\alpha_{j}(k)h_{k}^ {p}, \tag{2}\]

where \(h_{i}^{l}\) is the \(i\)-th hidden state of the source word at leaf level, and \(h_{k}^{p}\) is the \(k\)-th hidden state of the source phrase. The weight \(\alpha_{j}(t)\) of node \(t\) is computed by:

\[\alpha_{j}(t) = \frac{\exp(e_{t})}{\sum_{i=1}^{N}\exp(e_{i}^{l})+\sum_{k=1}^{N-1} \exp(e_{k}^{p})}\] \[e_{t} = (V_{a})^{T}\tanh(U_{a}s_{j}+W_{a}h_{t}+b_{a}),\]

where \(h_{t}\) is the hidden state of the node. \(V_{a}\), \(U_{a}\), \(W_{a}\) and \(b_{a}\) are the model parameters.

## 3 The Bidirectional Hierarchical Model

Although the tree-based encoder of Eriguchi et al. (2016) has shown certain advantages in translation tasks involving distant language pairs, e.g. English-Japanese, the representation of a phrase relies solely on its child nodes, and the word representation at leaf level only takes into account the sequential information. We argue that the incorporation of more hierarchical information into the representations may contribute to an improvement in the translation. In particular, the use of global information can help in distinguishing the differences between word meanings. Based on this hypothesis, we propose an alternative architecture, the bidirectional hierarchical model, to enhance the source-side representations



### Bidirectional Leaf-Node Encoding

As discussed in Section 1, the unidirectional recurrent neural network reads an input sequence in order, from the first symbol to the last. In order to generate leaf node annotation vectors which jointly take into account both preceding and following annotations, we exploit a bidirectional RNN encoder Bahdanau et al. (2015). The hidden state of the \(i\)-th leaf node \(h^{l}_{i}\) is the concatenation of the forward and backward vectors:

\[h^{l}_{i}=[\overrightarrow{h}^{l}_{i},\overleftarrow{h}^{l}_{i}],\]

where \(\overrightarrow{h}^{l}_{i}\) is obtained by a rightward GRU, as shown in Equation 1, and a leftward GRU calculates \(\overleftrightarrow{h}^{l}_{i}\), as follows:

\[\overleftrightarrow{h}^{l}_{i}=f^{\leftarrow}_{GRU}(x_{i},\overleftarrow{ h}^{l}_{i-1}),\]

where \(\overleftrightarrow{h}^{l}_{i-1}\) is the previous hidden state.

### Bidirectional Tree-Node Encoding

Since the hidden states of leaf nodes are derived in a sequential, context-sensitive way, by generating phrase annotations in a bottom-up fashion, the sequential context can be propagated to tree nodes. However, the learned annotation vectors still fail to capture global information from the upper nodes. To enhance the representations with global semantic information, we propose to use a standard GRU recurrent network to update representations in a **top-down fashion**, as shown in Figure 3. The annotation vectors, which are learned by the previous encoding steps, are fed to the updating process.

First, we treat the bottom-up hidden state of root \(h^{\uparrow}_{root}\), which covers the global meaning as well as the syntactic information of the source sentence, as the initial state of the top-down GRU network:

\[h^{\downarrow}_{root}=h^{\uparrow}_{root}.\]

Given an updated hidden state of the parent node \(h^{\downarrow}_{i,j}\), the hidden states of left and right children \(h^{\downarrow}_{i,k}\) and \(h^{\downarrow}_{k+1,j}\) are calculated as:

\[h^{\downarrow}_{i,k} = f^{ld}_{GRU}(h^{\uparrow}_{i,k},h^{\downarrow}_{i,j})\] \[h^{\downarrow}_{k+1,j} = f^{rd}_{GRU}(h^{\uparrow}_{k+1,j},h^{\downarrow}_{i,j}),\]

where \(h^{\uparrow}_{i,k}\) and \(h^{\uparrow}_{k+1,j}\) are the left and right child annotation vectors generated via the bottom-up tree-GRU network. Contrary to the similar top-down encoding for sentiment classification Kokkinos and Potamianos (2017), which uses same weighting parameters to handle both left and right child nodes, \(f^{ld}_{GRU}\) and \(f^{rd}_{GRU}\) with different parameters are applied in the proposed model to distinguish the left and right structural information. According to the definition of a GRU Cho et al. (2014b), \(f^{ld}_{GRU}\) uses an update gate \(z^{\uparrow}_{i,k}\), a reset gate \(r^{\downarrow}_{i,k}\) and a candidate activation \(\widetilde{h}^{\downarrow}_{i,k}\) to generate \(h^{\downarrow}_{i,k}\), as follows:

\[z^{\downarrow}_{i,k} =\sigma(W^{ld}_{(z)}h^{\uparrow}_{i,k}+U^{ld}_{(z)}h^{\downarrow} _{i,j}+b^{ld}_{(z)})\] \[r^{\downarrow}_{i,k} =\sigma(W^{ld}_{(r)}h^{\uparrow}_{i,k}+U^{ld}_{(r)}h^{\downarrow} _{i,j}+b^{ld}_{(r)})\] \[\widetilde{h}^{\downarrow}_{i,k} =\tanh(W^{ld}_{(dh)}h^{\uparrow}_{i,k}+U^{ld}_{(h)}(r^{\downarrow} _{i,k}\odot h^{\downarrow}_{i,j})+b^{ld}_{(h)})\] \[h^{\downarrow}_{i,k} =(1-z^{\downarrow}_{i,k})h^{\downarrow}_{i,j}+z^{\downarrow}_{i, k}\widetilde{h}^{\downarrow}_{i,k}, \tag{3}\]

where \(W^{ld}_{(\cdot)}\) and \(U^{ld}_{(\cdot)}\) represent weight matrices, and \(b^{ld}_{(\cdot)}\) denote bias vectors. \(f^{rd}_{GRU}\) is defined in a similar way.

From a linguistic point of view, in the top-down GRU network, the reset gate is able to retain the useful global information and drop irrelevant information from the parent state \(h^{\downarrow}_{i,j}\), while the proportions of the global context from the top-down state \(h^{\downarrow}_{i,j}\), and the local context from the bottom-up state \(h^{\uparrow}_{i,k}\) are controlled by the update gate. As it covers both the partial meaning of the phrase and the whole meaning of the sentence, \(h^{\downarrow}_{i,k}\) is regarded as the final representation of \(node_{i,k}\):

\[h^{p}_{i,k}=h^{\downarrow}_{i,k}.\]

With the propagation of information from root to leaf nodes, the \(i\)-th leaf node representation is updated as:

Figure 3: A top-down encoding process updates the hidden states recursively from root to leaf nodes. The red and blue lines denote the use of different learning parameters.



As each source-side hidden state of the leaf nodes and tree nodes carries the hierarchical information of the sentence, we interpret such an encoded state as a **hierarchical representation**.

### Handling Out-of-Vocabulary: Tree-Based Rare Word Encoding

In NMT, the translation of rare words and unknown words is an open problem, since the computational cost increases with the size of the vocabulary. Sennrich et al. (2016) proposed a simple and effective approach to handling out-of-vocabulary by representing rare words as a sequence of sub-word units, which are segmented using byte-pair encoding (BPE) [5].

We propose a variant tree-based rare word encoding approach which extends the tree-based model to the sub-word level. Sub-word units are encoded following an additional binary lexical tree. For a sentence \({\textbf{x}=(x_{1},...,x_{i},...,x_{N})}\), BPE segments the word \(x_{i}\) into a sequence of sub-word units \((x^{1}_{i},...,x^{n}_{i})\). The binary lexical tree is simply built by composing two nodes in a rightwards fashion, \((((x^{1}_{i},x^{2}_{i}),x^{3}_{i})...),x^{n}_{i})\), as shown in Figure 4. From the \(i\)-th leaf node, the original syntactic tree is extended downwards using the binary lexical tree, and the set of leaf nodes are replenished as \({\textbf{x}=(x_{1},...,x^{1}_{i},x^{2}_{i},...,x^{n}_{i},...,x_{N})}\). Sub-word units can therefore be regarded as leaf nodes, and can be encoded using the proposed encoder, as illustrated in Figure 5. The experimental results in Section 4.1 demonstrate the effectiveness of this simple approach.

### Decoder with Weighted Variant of Attention Mechanism

Since each representation carries both local and global information, in this case, attending fairly to the lexical and phrase representations in each decoding step may cause the problem of overtranslation (repeatedly attending and translating the same constituent of a sentence). An alternative approach is to balance the attentive information between the lexical and phrase vectors in the context vector. To effectively leverage these hierarchical representations, we propose a weighted variant of the tree-based attention mechanism (the original is defined in Equation 2). Formally, the calculation of the context vector \(d_{j}\) at step \(j\) is modified as:

\[d_{j}=(1-\beta_{j})\sum_{i=1}^{n}\alpha_{j}(i)h^{l}_{i}+\beta_{ j}\sum_{k=1}^{n-1}\alpha_{j}(k)h^{p}_{k} \tag{4}\]

where \(\beta_{j}\in[0,1]\) is used to weight the expected importance of the representations. Inspired by work on a multi-modal NMT [7] which exploits a gating scalar [9] to weight the image context vector, we use such a scalar in our model in order to dynamically adapt the weighting scalar. The gating scalar \(\beta_{j}\) at step \(j\) is calculated by :

\[\beta_{j}=\sigma(W_{\beta}c_{j-1}+b_{\beta}),\]

where \(W_{\beta}\) and \(b_{\beta}\) represent the model parameters. In contrast with \(\alpha\), which denotes the correspondence between each source annotation and the current target hidden state, \(\beta\) is dominated by the target composite hidden state alone. In other words, \(\beta\) is a time-dependent scalar in relation to the current target word, and therefore enables the attention model to explicitly quantify how far the

Figure 4: Encoding sub-word units with an additional binary lexical tree, where \(x^{1}_{3},x^{2}_{3},x^{3}_{3}\) are the sub-units of word \(x_{3}\).

Figure 5: Illustration of the bidirectional hierarchical encoder: representations are enhanced by a bidirectional leaf-node encoding and a bidirectional tree-node encoding. The green nodes indicate the sub-word representations.

 

leaf and no-leaf states contribute to the word prediction at each time step. In the proposed model, the phrase and lexical context vectors are learned by a single attention model, meaning that they are dependent, and the gating scalar weights the phrase and lexical context vectors in complementary fashion, as shown in Equation 4. This distinguishes the model from that introduced by Calixto et al. (2017), in which the context vectors of the source sentence and image (bi-modal) are measured using two independent attention models and the gating scalar is merely used to weight the image context vector.

## 4 Experiments

### Data

We evaluate the proposed model on an English-to-Chinese translation task. For reasons of computational efficiency, we extracted 1.4M sentence pairs, in which the maximum length of the sentence was 40, from the LDC parallel corpus3 as our training data. The models were developed using NIST mt08 data and were examined using NIST mt04, mt05, and mt06 data. The number of sentences in each dataset is shown in Table 1. On the English side, we used the _constituent parser_Zeng et al. (2014, 2015) to produce a binary syntactic tree for each sentence, in constrast to the use of the _HPSG parser_by Eriguchi et al. (2016). On the Chinese side, the sentences are segmented using the Chinese word segmentation toolkit of _NiuTrans_Xiao et al. (2012).

Footnote 3: Our training data was selected from _LDC2000T46_, _LDC2000T50_, _LDC2003E14_, _LDC2004T08_, _LDC2004T08_ and _LDC2005T10_.

To avoid data sparsity, words referring to time, date and number, which are low in frequency, are generalized as 'Stime', '$date' and '$number'. In addition, as described in Section 3.3, the vocabularies are further compressed by segmenting the rare words into sub-word units using BPE.

### Experimental Settings

As shown in Table 2, which gives the statistics of the token types, we limit the source and target vocabulary size to 40,000, in order to cover all the English and Chinese tokens. The dimensions of word embedding and hidden layer are respectively set as 620 and 1,000. Due to the concatenation in the bidirectional leaf-node encoding, the dimensions of the forward and backward vectors, which are half of those of the other hidden states, are set to 500. In order to prevent over-fitting, the training data is shuffled following each epoch. Moreover, the model parameters are optimized using AdaDelta Zeiler (2012), due to its capability for dynamically adapting the learning rate. We set the mini-batch size to 16 and the beam search size to 5. The accuracy of the translation relative to a reference is assessed using the BLEU metric Papineni et al. (2002). In order to give an equitable comparison, all the NMT models used for comparison are implemented or re-implemented using GRU in our code, based on \(dl4m^{d}\).

### Enhanced Hierarchical Representations

Firstly, the effectiveness of the enhanced hierarchical representations is evaluated through a set of experiments, the results of which are summarized in Table 3.

Compared with the original **tree-based encoder**Erriguchi et al. (2016), the model with bidirectional leaf-node encoding (described in Section 3.1) shows better performance. This also reveals that the future context at leaf level can contribute to word prediction. Secondly, although the representations of leaf nodes are learned in a sequential, context-sensitive way, the translation quality is further improved by considering the global semantic information in the top-down encoding (Section 3.2).

By incorporating the above enhancements into the model, the proposed **hierarchical encoder** yields significant improvements over both the sequential and the tree-based models. The problem of OOV is alleviated by further extending the tree

\begin{table}
\begin{tabular}{|c|c|c c c|} \hline Training & Dev & \multicolumn{3}{c|}{Test} \\ \hline LDC En-Zn & mt08 & mt04 & mt05 & mt06 \\ \hline
1,435,575 & 1,357 & 1,788 & 1,082 & 1,664 \\ \hline \end{tabular}
\end{table}
Table 1: Data used in the experiments.

\begin{table}
\begin{tabular}{|c|c c c|} \hline Training Set & Original & Generalization & BPE \\ \hline \(|V|\) in En & 159k & 120k & 40k \\ \hline \(|V|\) in Zh & 198k & 125k & 40k \\ \hline \end{tabular}
\end{table}
Table 2: The vocabulary size of the training set before and after applying the generalization and BPE segmentation.

 

based model to sub-word level (Section 3.3). In addition, we evaluate our tree-based rare word encoding method against the conventional rare word encoding [10] using the **sequential encoder**[2]. The empirical results confirm that our proposed tree-based BPE method achieves performance comparable to that of the standard BPE in the sequential model, but is applicable to the tree-based NMT model.

Overall, the proposed **hierarchical encoder** has demonstrated the ability to effectively model source-side representations from both the sequential and structural context. The NMT systems based on the proposed model significantly outperform those of conventional models using the **sequential encoder** and the **tree-based encoder**.

### Weighted Attention Model

As discussed in Section 3.4, in order to effectively leverage hierarchical representations in generating the target word, we adopt a variant weighted tree-based attention mechanism which incorporates a scalar to control the proportion of conditional information between the word and phrase vectors. By manually or automatically varying the weight \(\beta\), the utilization of the weighted attention model is assessed for four cases:

* \({\bf\beta=0.0}\): We manually set the weight of phrase vectors to \(0.0\); in other words, the decoder is forced to ignore the phrase vectors. The final translation is therefore generated by merely summarizing the leaf vectors.
* \({\bf\beta=0.5}\): The representations of non-leaf nodes and leaf nodes participate equally in the translation process. The decoder of this case therefore employs the same attention mechanism as that of the original model (Section 2.2).
* \({\bf\beta=1.0}\): In the reverse of the first case, the weight of the leaf nodes is manually set to \(0.0\). Thus, only the phrase vectors are used to predict the target words.
* Gating scalar (GS): A gating scalar is used for dynamically learning to control the proportion in which the lexical and phrase contexts contribute to the generation of the target words (Section 3.4).

The experimental results are shown in Table 4. The model which attends only to lexical annotation vectors (\(\beta=0.0\)) gives slightly better performance than that whi

\begin{table}
\begin{tabular}{|c||c|c|c|} \hline Model & BLEU & Perplexity & Avg. Length \\ \hline \(\beta=1.0\) & 17.16 & 98.65 & 21.13 \\ \(\beta=0.5\) & 19.41 & 94.73 & 23.08 \\ \(\beta=0.0\) & 19.83 & 94.68 & 23.33 \\ GS & **20.10** & **94.18** & 23.24 \\ \hline \end{tabular}
\end{table}
Table 4: Translation results for the development set. The last column indicates the average length of translation sentences, and the average length of reference sentences is 23.19.

\begin{table}
\begin{tabular}{|l||c|c|c c c c|} \hline Model & BPE & \# of params & MT04 & MT05 & MT06 & Dev. \\ \hline
**sequential encoder** & no & 86.8M & 31.26 & 23.98 & 24.02 & 17.20 \\  + sequential rare word encoding & yes & 86.8M & 32.54 & 25.09 & 25.07 & 18.19 \\  + tree-based rare word encoding & yes & 104.1M & 32.56 & 25.30 & 24.96 & 18.33 \\ \hline
**tree-based encoder** & no & 95.0M & 31.90 & 24.68 & 24.40 & 17.63 \\  + bidirectional leaf-node encoding & no & 92.0M & 32.13 & 24.94 & 25.02 & 18.12 \\  + top-down encoding & no & 101.1M & 32.85 & 25.37 & 25.30 & 18.26 \\  + tree-based rare word encoding & yes & 95.0M & 33.02 & 25.62 & 25.24 & 18.59 \\ \hline
**hierarchical encoder** (\(\beta=0.5\)) & no & 104.1M & 32.91\(\dagger\) & 25.55\(\ddagger\) & 25.52\(\dagger\) & 18.46\(\dagger\) \\
**hierarchical encoder** (\(\beta=0.5\)) & yes & 104.1M & 33.81\(\dagger\) & 26.47\(\dagger\) & 26.31\(\dagger\) & 19.41\(\dagger\) \\  + gating scalar & yes & 105.1M & **34.33\(\dagger\)** & **26.72\(\dagger\)** & **26.58\(\dagger\)** & **20.10\(\dagger\)** \\ \hline \end{tabular}
\end{table}
Table 3: Translation results for the various models. The first column shows the models; the second column indicates whether the corresponding experiment uses BPE data. The number of parameters (M = millions) in each model is given in the third column. The remaining columns are the translation accuracies for the test sets and development set, evaluated using BLEU scores (%). “\(\uparrow/\diamond\)”: indicates that the hierarchical encoder is significantly better than the vanilla tree-based encoder (\(p<0.05/p<0.01\)).

 

lexical and phrase vectors (\(\beta=0.5\)). The use of global information contributes to distinguishing the differences between word meanings, although the similar semantic information in the lexical and phrase representations aggravates the over-translation problem observed in the translation results. However, we found that the model which attends only to phrase representations tends to generate shorter translation of an average of 21.13 words in length, as shown in the last column of the first row of Table 4. Furthermore, the model that neglects the leaf representations (\(\beta=1.0\)) is likely to underperform the others that are also conditioned on the leaf nodes. Even though the phrase representations are derived from the lexical level via a bottom-up encoding, we believe it is unable to fully capture the lexical information of the source sentence. Through the use of the gating scalar, the hierarchical model achieves progressive improvements, as shown in Tables 3 and 4, the problem of over-translation is also alleviated. The representations of non-leaf nodes can be regarded as supplements in the translation process.

## 5 Qualitative Analysis

Figure 6 shows an English sentence and its binary tree representation, together with the corresponding Chinese translations produced by the different NMT models. All the models successfully give the correct Chinese translation "X \(\nexists\) \(\neg{}\)X \(\neg{}\)X" for the first three words of the English sentence "the organization wouldn't". Differences appear in the translation of the fourth word, and these lead to markedly different meanings. The translation "X \(\neg{}\)X



word units are likely to represent the word morphemes. The words are segmented into sub-word units, which are to some extent close to the linguistic word stems and suffixes. For example, the word "adventurer" is segmented into "adventur/er", which is correctly translated into the Chinese translations "\({}^{+}

 

* Chung et al. (2014) Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling. In _Advances in neural information processing systems (NIPS 2014) Deep Learning and Representation Learning Workshop_.
* Eriguchi et al. (2016) Akiko Eriguchi, Kazuma Hashimoto, and Yoshimasa Tsuruoka. 2016. Tree-to-Sequence Attentional Neural Machine Translation. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics_, pages 823-833.
* Gage (1994) Philip Gage. 1994. A New Algorithm for Data Compression. _The C Users Journal_, 12(2):23-38.
* Gers et al. (2000) Felix A Gers, Jurgen Schmidhuber, and Fred Cummins. 2000. Learning to Forget: Continual Prediction with LSTM. _Neural Computation_, 12(10):2451-2471.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long Short-Term Memory. _Neural Computation_, 9(8):1735-1780.
* Jozefowicz et al. (2015) Rafal Jozefowicz, Wojciech Zaremba, and Ilya Sutskever. 2015. An Empirical Exploration of Recurrent Network Architectures. In _Proceedings of the 32nd International Conference on Machine Learning_, pages 2342-2350.
* Kalchbrenner and Blunsom (2013) Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent Continuous Translation Models. In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_, pages 1700-1709.
* Kokkinos and Potamianos (2017) Filippos Kokkinos and Alexandros Potamianos. 2017. Structural Attention Neural Networks for Improved Sentiment Analysis. _CoRR_, abs/1701.01811.
* Luong et al. (2015) Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective Approaches to Attention-based Neural Machine Translation. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 1412-1421.
* Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A Method for Automatic Evaluation of Machine Translation. In _Proceedings of the 40th Annual Meeting on Association for Computational Linguistics_, pages 311-318.
* Schuster and Paliwal (1997) Mike Schuster and Kuldip K Paliwal. 1997. Bidirectional Recurrent Neural Networks. _IEEE Transactions on Signal Processing_, 45(11):2673-2681.
* Sennrich and Haddow (2016) Rico Sennrich and Barry Haddow. 2016. Linguistic Input Features Improve Neural Machine Translation. In _Proceedings of the First Conference on Machine Translation_, pages 83-90.
* Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural Machine Translation of Rare Words with Subword Units. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics_, pages 1715-1725.
* Stahlberg et al. (2016) Felix Stahlberg, Eva Hasler, Aurelien Waite, and Bill Byrne. 2016. Syntactically Guided Neural Machine Translation. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics_, pages 299-305.
* Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to Sequence Learning with Neural Networks. In _Advances in neural information processing systems (NIPS 2014)_, pages 3104-3112.
* Tai et al. (2015) Kai Sheng Tai, Richard Socher, and Christopher D. Manning. 2015. Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing_, pages 1556-1566.
* Xiao et al. (2012) Tong Xiao, Jingbo Zhu, Hao Zhang, and Qiang Li. 2012. NiuTrans: An Open Source Toolkit for Phrase-based and Syntax-based Machine Translation. In _Proceedings of the 52th Annual Meeting of the Association for Computational Linguistics, System Demonstrations_, pages 19-24.
* Xu et al. (2015) Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C Courville, Ruslan Salakhutdinov, Richard S Zemel, and Yoshua Bengio. 2015. Show, Attend and Tell!: Neural Image Caption Generation with Visual Attention. In _Proceedings of the 32nd International Conference on Machine Learning_, pages 77-81.
* Zeiler (2012) Matthew D. Zeiler. 2012. ADADELTA: An Adaptive Learning Rate Method. _CoRR_, abs/1212.5701.
* Zeng et al. (2015) Xiaodong Zeng, Derek F. Wong, Lidia S. Chao, and Isabel Trancoso. 2015. Graph-Based Lexicon Regularization for PCFG With Latent Annotations. _IEEE/ACM Transactions on Audio, Speech, and Language Processing_, 23(3):441-450.
* Zeng et al. (2014) Xiaodong Zeng, Derek F. Wong, Lidia S. Chao, Isabel Trancoso, Liangye He, and Qiuping Huang. 2014. Lexicon Expansion for Latent Variable Grammars. _Pattern Recognition Letters_, (42):47-55.
* Zhou et al. (2016) Yao Zhou, Cong Liu, and Yan Pan. 2016. Modelling Sentence Pairs with Tree-structured Attentive Encoder. In _Proceedings of 26th International Conference on Computational Linguistics_, pages 2912-2922.