Venue: ACL 2020

# Regularized Context Gates on Transformer for Machine Translation

Xintong Li1, Lemao Liu2, Rui Wang3, Guoping Huang2, Max Meng1

###### Abstract

Context gates are effective to control the contributions from the source and target contexts in the recurrent neural network (RNN) based neural machine translation (NMT). However, it is challenging to extend them into the advanced Transformer architecture, which is more complicated than RNN. This paper first provides a method to identify source and target contexts and then introduce a gate mechanism to control the source and target contributions in Transformer. In addition, to further reduce the bias problem in the gate mechanism, this paper proposes a regularization method to guide the learning of the gates with supervision automatically generated using pointwise mutual information. Extensive experiments on 4 translation datasets demonstrate that the proposed model obtains an averaged gain of 1.0 BLEU score over a strong Transformer baseline.

The Chinese University of Hong Kong \(~{}^{2}\)Tencent AI Lab

3National Institute of Information and Communications Technology znculee@gmail.com redmondliu@tencent.com wangrui@nict.go.jp donkeyhuang@tencent.com max.meng@cuhk.edu.hk

## 1 Introduction

An essence to modeling translation is how to learn an effective context from a sentence pair. Statistical machine translation (SMT) models the source context from the source-side of a translation model and models the target context from a target-side language model Koehn et al. (2003); Koehn (2009); Chiang (2005). These two models are trained independently. On the contrary, neural machine translation (NMT) advocates a unified manner to jointly learn source and target context using an encoder-decoder framework with an attention mechanism, leading to substantial gains over SMT in translation quality Sutskever et al. (2014); Bahdanau et al. (2014); Gehring et al. (2017); Vaswani et al. (2017). Prior work on attention mechanism Luong et al. (2015); Liu et al. (2016); Mi et al. (2016); Chen et al. (2018); Li et al. (2018); Elbayad et al. (2018); Yang et al. (2020) have shown a better context representation is helpful to translation performance.

However, a standard NMT system is incapable of effectively controlling the contributions from source and target contexts He et al. (2018) to deliver highly adequate translations as shown in Figure 1. As a result, Tu et al. (2017) carefully designed context gates to dynamically control the influence from source and target contexts and observed significant improvements in the recurrent neural network (RNN) based NMT. Although Transformer Vaswani et al. (2017) delivers significant gains over RNN for translation, there are still one third translation errors related to context control problem as described in Section 3.3. Obviously, it is feasible to extend the context gates in RNN based NMT into Transformer, but an obstacle to accomplishing this goal is the complicated archi

Figure 1: A running example to raise the context control problem. Both original and context gated Transformer obtain an unfaithful translation by wrongly translate “_ti qiu_” into “_play golf_” because referring too much target context. By regularizing the context gates, the purposed method corrects the translation of “_it qiu_” into “_play soccer_”. The light font denotes the target words to be translated in the future. For original Transformer, the source and target context are added directly without any rebalancing.



tecture in Transformer, where the source and target words are tightly coupled. Thus, it is challenging to put context gates into practice in Transformer.

In this paper, under the Transformer architecture, we firstly provide a way to define the source and target contexts and then obtain our model by combining both source and target contexts with context gates, which actually induces a probabilistic model indicating whether the next generated word is contributed from the source or target sentence (Li et al., 2019). In our preliminary experiments, this model only achieves modest gains over Transformer because the context selection error reduction is very limited as described in Section 3.3. To further address this issue, we propose a probabilistic model whose loss function is derived from external supervision as regularization for the context gates. This probabilistic model is jointly trained with the context gates in NMT. As it is too costly to annotate this supervision for a large-scale training corpus manually, we instead propose a simple yet effective method to automatically generate supervision using pointwise mutual information, inspired by word collocation (Bouma, 2009). In this way, the resulting NMT model is capable of controlling the contributions from source and target contexts effectively.

We conduct extensive experiments on 4 benchmark datasets, and experimental results demonstrate that the proposed gated model obtains an averaged improvement of 1.0 BLEU point over corresponding strong Transformer baselines. In addition, we design a novel analysis to show that the improvement of translation performance is indeed caused by relieving the problem of wrongly focusing on the source or target context.

## 2 Methodology

Given a source sentence \({\bf x}=\langle{\rm x_{1},\cdots,x_{|{\bf x}|}}\rangle\) and a target sentence \({\bf y}=\langle{\rm y_{1},\cdots,y_{|{\bf y}|}}\rangle\), our proposed model is defined by the following conditional probability under the Transformer architecture: 1

Footnote 1: Throughout this paper, a variable in bold font such as \({\bf x}\) denotes a sequence while regular font such as \({\bf x}\) denotes an element which may be a scalar \(x\), vector \({\mathbf{x}}\) or matrix \({\mathbf{X}}\).

\[P\left({\bf y}~{}|~{}{\bf x}\right)=\prod_{i=1}^{|{\bf y}|}P\left({\rm y}_{i}~ {}|~{}{\bf y}_{<i},{\bf x}\right)=\prod_{i=1}^{|{\bf y}|}P\left({\rm y}_{i}~{} |~{}c_{i}^{L}\right), \tag{1}\]

where \({\bf y}_{<i}=\langle{\rm y}_{1},\ldots,{\rm y}_{i-1}\rangle\) denotes a prefix of \({\bf y}\) with length \(i-1\), and \(c_{i}^{L}\) denotes the \(L^{\text{th}}\) layer context in the decoder with \(L\) layers which is obtained from the representation of \({\bf y}_{<i}\) and \({\bf h}^{L}\), i.e., the top layer hidden representation of \({\bf x}\), similar to the original Transformer. To finish the overall definition of our model in equation 1, we will expand the definition \(c_{i}^{L}\) based on context gates in the following subsections.

### Context Gated Transformer

To develop context gates for our model, it is necessary to define the source and target contexts at first. Unlike the case in RNN, the source sentence \({\bf x}\) and the target prefix \({\bf y}_{<i}\) are tightly coupled in our model, and thus it is not trivial to define the source and target contexts.

Suppose the source and target contexts at each layer \(l\) are denoted by \(s_{i}^{l}\) and \(t_{i}^{l}\). We recursively define them from \({\bf c}_{<i}^{l-1}\) as follows. 2

Footnote 2: For the base case, \({\bf c}_{<i}^{0}\) is word embedding of \({\bf y}_{<i}\).

\[\begin{split}& t_{i}^{l}={\rm rn}\circ\ln\circ{\rm att}\left({c }_{i}^{l-1},{\bf c}_{<i}^{l-1}\right),\\ & s_{i}^{l}=\ln\circ{\rm att}\left({t}_{i}^{l},{\bf h}^{L} \right),\end{split} \tag{2}\]

where \(\circ\) is functional composition, \({\rm att}\left({\rm q,kv}\right)\) denotes multiple head attention with \({\sf q}\) as query, \({\sf k}\) as key, \({\sf v}\) as value, and \({\rm rn}\) as a residual network (He et al., 2016), \(\ln\) is layer normalization (Ba et al., 2016), and all parameters are removed for simplicity.

In order to control the contributions from source or target side, we define \(c_{i}^{l}\) by introducing a context gate \({\mathbf{z}}_{i}^{l}\) to combine \(s_{i}^{l}\) and \(t_{i}^{l}\) as following:

\[{\bf c}_{i}^{l}={\rm rn}\circ\ln\circ{\rm ff}\left(({\bf 1}-{\mathbf{z}}_{i}^{ l})\otimes{t}_{i}^{l}+{\mathbf{z}}_{i}^{l}\otimes{s}_{i}^{l}\right) \tag{3}\]

with

\[{\mathbf{z}}_{i}^{l}=\sigma\left({\rm ff}\left({t}_{i}^{l}\|s_{i}^{l}\right)\right), \tag{4}\]

where \({\rm ff}\) denotes a feedforward neural network, \(\parallel\) denotes concatenation, \(\sigma(\cdot)\) denotes a sigmoid function, and \(\otimes\) denotes an element-wise multiplication. \({\mathbf{z}}_{i}^{l}\) is a vector (Tu et al. (2017) reported that a gating vector is better than a gating scalar). Note that each component in \({\mathbf{z}}_{i}^{l}\) actually induces a probabilistic model indicating whether the next generated word \({\rm y}_{i}\) is mainly contributed from the source (\({\bf x}\)) or target sentence (\({\bf y}_{<i}\)) , as shown in Figure 1.

RemarkIt is worth mentioning that our proposed model is similar to the standard Transformer with boiling down to replacing a residual connection 

with a high way connection (Srivastava et al., 2015; Zhang et al., 2018): if we replace \(({\bf 1}-{\mathbf{z}_{i}^{l}})\otimes{\mathfrak{t}_{i}^{l}}+{\mathbf{z}_{i}^{l}} \otimes{\mathfrak{s}_{i}^{l}}\) in equation 3 by \({\mathbf{t}_{i}^{l}}+{\mathbf{s}_{i}^{l}}\), the proposed model is reduced to Transformer.

### Regularization of Context Gates

In our preliminary experiments, we found learning context gates from scratch cannot effectively reduce the context selection errors as described in Section 3.3.

To address this issue, we propose a regularization method to guide the learning of context gates by external supervision \(z_{i}^{*}\) which is a binary number representing whether \({\mathsf{y}_{i}}\) is contributed from either source (\(z_{i}^{*}=1\)) or target sentence (\(z_{i}^{*}=0\)). Formally, the training objective is defined as follows:

\[\ell=-\log P({\mathbf{y}}~{}|~{}{\mathbf{x}})+\lambda\sum_{l,i} \bigg{(}z_{i}^{*}\max({\bf 0.5}-{\mathbf{z}_{i}^{l}},{\bf 0})\\ +(1-z_{i}^{*})\max({\mathbf{z}_{i}^{l}}-{\bf 0.5},{\bf 0})\bigg{)}, \tag{5}\]

where \({\mathbf{z}_{i}^{l}}\) is a context gate defined in equation 4 and \(\lambda\) is a hyperparameter to be tuned in experiments. Note that we only regularize the gates during the training, but we skip the regularization during inference.

Because golden \(z_{i}^{*}\) are inaccessible for each word \({\mathsf{y}_{i}}\) in the training corpus, we ideally have to annotate it manually. However, it is costly for human to label such a large scale dataset. Instead, we propose an automatic method to generate its value in practice in the next subsection.

### Generating Supervision \(z_{i}^{*}\)

To decide whether \({\mathsf{y}_{i}}\) is contributed from the source (**x**) or target sentence (\({\mathsf{y}_{<i}}\)) (Li et al., 2019), a metric to measure the correlation between a pair of words (\(\left\langle{\mathsf{y}_{i},\mathrm{x}_{j}}\right\rangle\) or \(\left\langle{\mathsf{y}_{i},\mathrm{y}_{k}}\right\rangle\) for \(k<i\)) is first required. This is closely related to a well-studied problem, i.e., word collocation (Liu et al., 2009), and we simply employ the pointwise mutual information (PMI) to measure the correlation between a word pair \(\left\langle{\mu,\nu}\right\rangle\) following Bouma (2009):

\[\begin{split}\operatorname{pmi}\left(\mu,\nu\right)& =\log\frac{P(\mu,\nu)}{P(\mu)P(\nu)}\\ &=\log Z+\log\frac{C(\mu,\nu)}{C(\mu)C(\nu)},\end{split} \tag{6}\]

where \(C\left(\mu\right)\) and \(C\left(\nu\right)\) are word counts, \(C\left(\mu,\nu\right)\) is the co-occurrence count of words \(\mu\) and \(\nu\), and \(Z\) is the normalizer, i.e., the total number of all possible \(\left(\mu,\nu\right)\) pairs. To obtain the context gates, we define two types of PMI according to different \(C\left(\mu,\nu\right)\) including two scenarios as follows.

PMI in the Bilingual ScenarioFor each parallel sentence pair \(\left\langle{\mathbf{x},\mathbf{y}}\right\rangle\) in training set, \(C\left({\mathsf{y}_{i},\mathrm{x}_{j}}\right)\) is added by one if both \({\mathsf{y}_{i}}\in{\mathbf{y}}\) and \({\mathsf{x}_{j}}\in{\mathbf{x}}\).

PMI in the Monolingual ScenarioIn the translation scenario, only the words in the preceding context of a target word should be considered. So for any target sentence \({\mathbf{y}}\) in the training set, \(C\left({\mathsf{y}_{i},\mathrm{y}_{k}}\right)\) is added by one if both \({\mathsf{y}_{i}}\in{\mathbf{y}}\) and \({\mathsf{y}_{k}}\in{\mathbf{y}_{<i}}\).

Given the two kinds of PMI for a bilingual sentence \(\left\langle{\mathbf{x},\mathbf{y}}\right\rangle\), each \(z_{i}^{*}\) for each \({\mathsf{y}_{i}}\) is defined as follows,

\[z_{i}^{*}={\mathbb{1}}_{\max_{j}\operatorname{pmi}\left({\mathsf{y}_{i}},{ \mathsf{x}_{j}}\right)>\max_{k<i}\operatorname{pmi}\left({\mathsf{y}_{i}},{ \mathsf{y}_{k}}\right)}, \tag{7}\]

where \({\mathbb{1}}_{b}\) is a binary function valued by 1 if \(b\) is true and 0 otherwise. In equation 7, we employ \(\max\) strategy to measure the correlation between \({\mathsf{y}_{i}}\) and a sentence (\({\mathbf{x}}\) or \({\mathbf{y}_{<i}}\)). Indeed, it is similar to use the average strategy, but we did not find its gains over \(\max\) in our experiments.

## 3 Experiments

The proposed methods are evaluated on NIST ZH\(\Rightarrow\)EN 3, WMT14 EN\(\Rightarrow\)DE 4, IWSLT14 DE\(\Rightarrow\)EN 5 and IWSLT17 FR\(\Rightarrow\)EN 6 tasks. To make our NMT models capable of open-vocabulary translation, all datasets are preprocessed with Byte Pair Encoding (Sennrich et al., 2015). All proposed methods are implemented on top of Transformer Vaswani et al. (2017) which is the state-of-the-art NMT system. Case-insensitive BLEU score Papineni et al. (2002) is used to evaluate translation quality of ZH\(\Rightarrow\)EN, DE\(\Rightarrow\)EN and FR\(\Rightarrow\)EN. For the fair comparison with the related work, EN\(\Rightarrow\)DE is evaluated with case-sensitive BLEU score. Setup details are described in Appendix A.

Footnote 3: LDC2000T50, LDC200L2I7, LDC200ZT01, LDC200E18, LDC2003E07, LDC2003E14, LDC2003T17, LDC2004T07

Footnote 4: WMT14: [http://www.statmt.org/wmtl4/](http://www.statmt.org/wmtl4/)

Footnote 5: IWSLT14: [http://workshop2014.iwslt.org/](http://workshop2014.iwslt.org/)

### Tuning Regularization Coefficient

In the beginning of our experiments, we tune the regularization coefficient \(\lambda\) on the DE\(\Rightarrow\)EN task. Table 2 shows the robustness of \(\lambda\), because the translation performance only fluctuates slightly over various \(\lambda\). In particular, 

is achieved when \(\lambda=1\), which is the default setting throughout this paper.

### Translation Performance

Table 1 shows the translation quality of our methods in BLEU. Our observations are as follows:

1) The performance of our implementation of the Transformer is slightly higher than Vaswani et al. (2017), which indicates we are in a fair comparison.

2) The proposed Context Gates achieves modest improvement over the baseline. As we mentioned in Section 2.1, the structure of RNN based NMT is quite different from the Transformer. Therefore, naively introducing the gate mechanism to the Transformer without adaptation does not obtain similar gains as it does in RNN based NMT.

3) The proposed Regularized Context Gates improves nearly 1.0 BLEU score over the baseline and outperforms all existing related work. This indicates that the regularization can make context gates more effective in relieving the context control problem as discussed following.

### Error Analysis

To explain the success of Regularized Context Gates, we analyze the error rates of translation and context selection. Given a sentence pair \({\bf x}\) and \({\bf y}\), the forced decoding translation error is defined as \(P\left({\rm y}_{i}~{}|~{}{\bf y}_{<i},{\bf x}\right)<P\left({\hat{y}}_{i}~{}|~ {}{\bf y}_{<i},{\bf x}\right)\), where \({\hat{y}}_{i}\triangleq\arg\max_{\forall}P\left({\bf v}~{}|~{}{\bf y}_{<i},{ \bf x}\right)\) and \({\bf v}\) denotes any token in the vocabulary. The context selection error is defined as \(z^{*}_{i}({\rm y}_{i})\neq z^{*}_{i}({\hat{Y}}_{i})\), where \(z^{*}_{i}\) is defined in equation 7. Note that a context selection error must be a translation error but the opposite is not true. The example shown in Figure 1 also demonstrates a context selection error indicating the translation error is related with the bad context selection.

As shown in Table 3, the Regularized Context Gates significantly reduce the translation error by avoiding the context selection error. The Context Gates are also able to avoid few context selection error but cannot make a notable improvement in translation performance. It is worth to note that there is approximately one third translation error is related to context selection error. The Regularized Context Gates indeed alleviate this severe problem by effectively rebalancing of source and target context for translation.

### Statistics of Context Gates

Table 4 summarizes the mean and variance of each context gate (every dimension of the context gate vectors) over the MT08 test set. It shows that learning context gates freely from scratch tends to pay more attention to target context (\(0.38<0.5\)), which

\begin{table}
\begin{tabular}{c|c|c|c|c||c|c|c} \multirow{2}{*}{**Models**} & \multicolumn{2}{c|}{**params**} & \multicolumn{3}{c||}{ZH\(\Rightarrow\)EN} & \multicolumn{1}{c}{EN\(\Rightarrow\)DE} & DE\(\Rightarrow\)EN & FR\(\Rightarrow\)EN \\ \cline{3-9}  & \(\times 10^{6}\) & MT05 & MT06 & MT08 & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \hline RNN based NMT & 84 & 30.6 & 31.1 & 23.2 & – & – & – \\ Tu et al. (2017) & 88 & 34.1 & 34.8 & 26.2 & – & – & – \\ Vaswani et al. (2017) & 65 & – & – & – & 27.3 & – & – \\ Ma et al. (2018) & – & 36.8 & 35.9 & 27.6 & – & – & – \\ Zhao et al. (2018) & – & 43.9 & 44.0 & 33.3 & – & – & – \\ Cheng et al. (2018) & – & 44.0 & 44.4 & 34.9 & – & – & – \\ \hline \hline \end{tabular} 
\begin{tabular}{c|c|c|c|c|c|c|c} \multirow{2}{*}{This Work} & \multicolumn{2}{c|}{Transformer} & \multicolumn{1}{c}{74} & \multicolumn{1}{c}{46.9} & \multicolumn{1}{c}{47.4} & \multicolumn{1}{c}{39.1} & \multicolumn{1}{c}{27.9} & \multicolumn{1}{c}{32.5} & \multicolumn{1}{c}{37.7} \\  & \multicolumn{1}{c|}{Regularized Context Gates} & \multicolumn{1}{c|}{92} & \multicolumn{1}{c}{**47.7**} & \multicolumn{1}{c}{**48.3**} & \multicolumn{1}{c}{**39.7**} & \multicolumn{1}{c}{**28.1**} & \multicolumn{1}{c}{**33.0**} & \multicolumn{1}{c}{**38.3**} \\ \end{tabular}
\end{table}
Table 1: Translation performances (BLEU). The RNN based NMT (Bahdanau et al., 2014) is reported from the baseline model in Tu et al. (2017). “params” shows the number of parameters of models when training ZH\(\Rightarrow\)EN except Vaswani et al. (2017) is for EN\(\Rightarrow\)DE tasks.

\begin{table}
\begin{tabular}{c|c|c|c} \multirow{2}{*}{**Models**} & **FER** & **CER** & **CE/FE** \\ \hline Transformer & 40.5 & 13.8 & 33.9 \\ Context Gates & 40.5 & 13.7 & 33.7 \\ Regularized Context Gates & **40.0** & **13.4** & **33.4** \\ \end{tabular}
\end{table}
Table 2: Translation performance over different regularization coefficient \(\lambda\).

 

means the model tends to trust its language model more than the source context, and we call this context imbalance bias of the freely learned context gate. Specifically, this bias will make the translation unfaithful for some source tokens. As shown in Table 4, the Regularized Context Gates demonstrates more balanced behavior (0.51\(\approx\)0.5) over the source and target context with similar variance.

### Regularization in Different Layers

To investigate the sensitivity of choosing different layers for regularization, we only regularize the context gate in every single layer. Table 5 shows that there is no significant performance difference, but all single layer regularized context gate models are slightly inferior to the model, which regularizes all the gates. Moreover, since nearly no computation overhead is introduced and for design simplicity, we adopt regularizing all the layers.

### Effects on Long Sentences

In Tu et al. (2017), context gates alleviate the problem of long sentence translation of attentional RNN based system Bahdanau et al. (2014). We follow Tu et al. (2017) and compare the translation performances according to different lengths of the sentences. As shown in Figure 2, we find Context Gates does not improve the translation of long sentences but translate short sentences better. Fortunately, the Regularized Context Gates indeed significantly improves the translation for both short sentences and long sentences.

## 4 Conclusions

This paper transplants context gates from the RNN based NMT to the Transformer to control the source and target context for translation. We find that context gates only modestly improve the translation quality of the Transformer, because learning context gates freely from scratch is more challenging for the Transformer with the complicated structure than for RNN. Based on this observation, we propose a regularization method to guide the learning of context gates with an effective way to generate supervision from training data. Experimental results show the regularized context gates can significantly improve translation performances over different translation tasks even though the context control problem is only slightly relieved. In the future, we believe more work on alleviating context control problem has the potential to improve translation performance as quantified in Table 3.

## References

* Ba et al. (2016) Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. _arXiv preprint arXiv:1607.06450_.
* Bahdanau et al. (2014) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. _arXiv preprint arXiv:1409.0473_.
* Bouma (2009) Gerlof Bouma. 2009. Normalized (pointwise) mutual information in collocation extraction. _Proceedings of GSCL_, pages 31-40.
* Chen et al. (2018) Kehai Chen, Rui Wang, Masao Utiyama, Eiichiro Sumita, and Tiejun Zhao. 2018. Syntax-directed attention for neural machine translation. In _Thirty-Second AAAI Conference on Artificial Intelligence_.
* Cheng et al. (2018) Yong Cheng, Zhaopeng Tu, Fandong Meng, Junjie Zhai, and Yang Liu. 2018. Towards ro

Figure 2: Translation performance on MT08 test set with respect to different lengths of source sentence. Regularized Context Gates significantly improves the translation of short and long sentences.

\begin{table}
\begin{tabular}{c|c|c}
**Models** & **Mean** & **Variance** \\ \hline Context Gates & 0.38 & 0.10 \\ Regularized Context Gates & 0.51 & 0.13 \\ \end{tabular}
\end{table}
Table 4: Mean and variance of context gates 

bust neural machine translation. _arXiv preprint arXiv:1805.06130_.
* Chiang (2005) David Chiang. 2005. A hierarchical phrase-based model for statistical machine translation. In _Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics_, pages 263-270. Association for Computational Linguistics.
* Elbayad et al. (2018) Maha Elbayad, Laurent Besacier, and Jakob Verbeek. 2018. Pervasive attention: 2d convolutional neural networks for sequence-to-sequence prediction. _arXiv preprint arXiv:1808.03867_.
* Gehring et al. (2017) Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional sequence to sequence learning. In _Proceedings of the 34th International Conference on Machine Learning-Volume 70_, pages 1243-1252. JMLR. org.
* He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In _Proceedings of the IEEE conference on computer vision and pattern recognition_, pages 770-778.
* He et al. (2018) Tianyu He, Xu Tan, Yingce Xia, Di He, Tao Qin, Zhibo Chen, and Tie-Yan Liu. 2018. Layer-wise coordination between encoder and decoder for neural machine translation. In _Advances in Neural Information Processing Systems_, pages 7944-7954.
* Koehn (2009) Philipp Koehn. 2009. _Statistical machine translation_. Cambridge University Press.
* Koehn et al. (2003) Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In _Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology-Volume 1_, pages 48-54. Association for Computational Linguistics.
* Li et al. (2019) Xintong Li, Guanlin Li, Lemao Liu, Max Meng, and Shuming Shi. 2019. On the word alignment from neural machine translation. In _Proceedings of the 57th Conference of the Association for Computational Linguistics_, pages 1293-1303.
* Li et al. (2018) Xintong Li, Lemao Liu, Zhaopeng Tu, Shuming Shi, and Max Meng. 2018. Target foresight based attention for neural machine translation. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 1380-1390.
* Liu et al. (2016) Lemao Liu, Masao Utiyama, Andrew Finch, and Eiichiro Sumita. 2016. Neural machine translation with supervised attention. _arXiv preprint arXiv:1609.04186_.
* Liu et al. (2009) Zhanyi Liu, Haifeng Wang, Hua Wu, and Sheng Li. 2009. Collocation extraction using monolingual word alignment method. In _Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 2-Volume 2_, pages 487-495. Association for Computational Linguistics.
* Luong et al. (2015) Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015. Effective approaches to attention-based neural machine translation. _arXiv preprint arXiv:1508.04025_.
* Ma et al. (2018) Shuming Ma, Xu Sun, Yizhong Wang, and Junyang Lin. 2018. Bag-of-words as target for neural machine translation. _arXiv preprint arXiv:1805.04871_.
* Mi et al. (2016) Haitao Mi, Zhiguo Wang, and Abe Ittycheriah. 2016. Supervised attentions for neural machine translation. _arXiv preprint arXiv:1608.00112_.
* Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In _Proceedings of the 40th annual meeting on association for computational linguistics_, pages 311-318. Association for Computational Linguistics.
* Sennrich et al. (2015) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural machine translation of rare words with subword units. _arXiv preprint arXiv:1508.07909_.
* Srivastava et al. (2015) Rupesh Kumar Srivastava, Klaus Greff, and Jurgen Schmidhuber. 2015. Highway networks. _arXiv preprint arXiv:1505.00387_.
* Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In _Advances in neural information processing systems_, pages 3104-3112.
* Tu et al. (2017) Zhaopeng Tu, Yang Liu, Zhengdong Lu, Xiaohua Liu, and Hang Li. 2017. Context gates for neural machine translation. _Transactions of the Association for Computational Linguistics_, 5:87-99.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in Neural Information Processing Systems_, pages 5998-6008.
* Yang et al. (2020) Mingming Yang, Min Zhang, Kehai Chen, Rui Wang, and Tiejun Zhao. 2020. Neural machine translation with target-attention model. _IEICE TRANSACTIONS ON Information and Systems_, 103(3):684-694.
* Zhang et al. (2017) Jiacheng Zhang, Yanzhuo Ding, Shiqi Shen, Yong Cheng, Maosong Sun, Huanbo Luan, and Yang Liu. 2017. Thumt: an open source toolkit for neural machine translation. _arXiv preprint arXiv:1706.06415_.
* Zhang et al. (2018) Jiacheng Zhang, Huanbo Luan, Maosong Sun, Feifei Zhai, Jingfang Xu, Min Zhang, and Yang Liu. 2018. Improving the transformer translation model with document-level context. _arXiv preprint arXiv:1810.03581_ 

* Zhao et al. (2018) Yang Zhao, Jiajun Zhang, Zhongjun He, Chengqing Zong, and Hua Wu. 2018. Addressing troublesome words in neural machine translation. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 391-400.

 

Details of Data and Implementation

The training data for ZH\(\Rightarrow\)EN task consists of 1.8M sentence pairs. The development set is chosen as NIST02 and test sets are NIST05, 06, 08. For EN\(\Rightarrow\)DE task, its training data contains 4.6M sentences pairs. Both FR\(\Rightarrow\)EN and DE\(\Rightarrow\)EN tasks contain around 0.2M sentence pairs. For ZH\(\Rightarrow\)EN and EN\(\Rightarrow\)DE tasks, the joint vocabulary is built with 32K BPE merge operations, and for DE\(\Rightarrow\)EN and FR\(\Rightarrow\)EN tasks it is built with 16K merge operations.

Our implementation of context gates and the regularization are based on Transformer, implemented by THUMT [Zhang et al., 2017]. For ZH\(\Rightarrow\)EN and EN\(\Rightarrow\)DE tasks, only the sentences of length up to 256 tokens are used with no more than \(2^{15}\) tokens in a batch. The dimension of both word embeddings and hidden size are 512. Both encoder and decoder have 6 layers and adopt multi-head attention with 8 heads. For FR\(\Rightarrow\)EN and DE\(\Rightarrow\)EN tasks, we use a smaller model with 4 layers and 4 heads, and both the embedding size and the hidden size is 256. The training batch contains no more than \(2^{12}\) tokens. For all tasks, the beam size for decoding is 4, and the loss function is optimized with Adam, where \(\beta_{1}=0.9\), \(\beta_{2}=0.98\) and \(\epsilon=10^{-9}\).