Venue: NAACL 2019

# Improving Neural Machine Translation with Neural Syntactic Distance

Chunpeng Ma\({}^{1,3}\), Akihiro Tamura\({}^{2}\), Masao Utiyama\({}^{3}\), Tiejun Zhao\({}^{1}\); Eiichiro Sumita\({}^{3}\)

\({}^{1}\)Harbin Institute of Technology, Harbin, China

\({}^{2}\)Ehime University, Matsuyama, Japan

\({}^{3}\)National Institute of Information and Communications Technology, Kyoto, Japan {cpma, tjzhao}@hit.edu.cn tamura@cs.ehime-u.ac.jp

{mutiyama, eiichiro.sumita}@nict.go.jp

Corresponding author

###### Abstract

The explicit use of syntactic information has been proved useful for neural machine translation (NMT). However, previous methods resort to either tree-structured neural networks or long linearized sequences, both of which are inefficient. Neural syntactic distance (NSD) enables us to represent a constituent tree using a sequence whose length is identical to the number of words in the sentence. NSD has been used for constituent parsing, but not in machine translation. We propose five strategies to improve NMT with NSD. Experiments show that it is not trivial to improve NMT with NSD; however, the proposed strategies are shown to improve translation performance of the baseline model (+2.1 (En-Ja), +1.3 [4], +1.2 (En-Ch), and +1.0 (Ch-En) BLEU).

## 1 Introduction

In recent years, neural machine translation (NMT) has been developing rapidly and has become the de facto approach for machine translation. To improve the performance of the conventional NMT models Sutskever et al. (2014); Bahdanau et al. (2014), one effective approach is to incorporate syntactic information into the encoder and/or decoder of the baseline model.

Based on how the syntactic information is represented, there are two categories of syntactic NMT methods: (1) those that use tree-structured neural networks (NNs) to represent syntax structures Eriguchi et al. (2016); Hashimoto and Tsuruoka (2017), and (2) those that use linear-structured NNs to represent linearized syntax structures Li et al. (2017); Ma et al. (2017, 2018). For the first category, there is a direct corresponding relationship between the syntactic structure and the NN structure, but the complexity of NN structures usually makes training inefficient. In contrast, for the second category, syntactic structures are linearized and represented using linear-structured recurrent neural networks (RNNs), but the linearized sequence can generally be quite long and therefore training efficiency is still a problem. Although using a shorter sequence may improve the efficiency, some syntactic information is lost.

We propose a method of using syntactic information in NMT that overcomes the disadvantages of both methods. The basis of our method is the neural syntactic distance (NSD), a recently proposed concept used for constituent parsing Shen et al. (2018); Gomez-Rodriguez and Vilares (2018). NSD makes it possible to represent a constituent tree as a sequence whose length is identical to the number of words in the sentence (almost) without losing syntactic information. However, there are no previous studies that use NSD in NMT. Moreover, as demonstrated by our experiments, using NSD in NMT is far from straightforward, so we propose five strategies and verify the effects empirically. The strategies are summarized below.

* Extend NSD to dependency trees, which is inspired by the dependency language model Shen et al. (2010).
* Use NSDs as input sequences1, where an NSD is regarded as a linguistic input feature Sennrich and Haddow (2016). Footnote 1: Throughout this paper, “input” means the input of an encoder or a decoder rather than the input of the NMT model (i.e., only source sentences), and ”output” is similar.
* Use NSDs as output sequences, where the NMT and prediction of the NSD are simultaneously trained through multi-task learning Firat et al. (2016).
* Use NSD as positional encoding (PE), which is a syntactic extension of the PE of the Transformer Vaswani et al. (2017).



* Add a loss function for NSD to achieve distance-aware training (Shen et al., 2018).

## 2 Neural Syntactic Distance (NSD)

The NSD was firstly proposed by Shen et al. (2018). This is the first method of linearizing a constituent tree with a sequence of length \(n\), without loss of information, where \(n\) is the number of words in the sentence.

Formally, given the sentence \({\bf w}=(w_{1},\ldots,w_{n})\), for any pairs of contiguous words \((w_{i},w_{i+1})\), we can define an NSD \(d(w_{i})\),2 where \(i\in[1,n-1]\). In Shen et al. (2018), the NSD \(d_{S}(w_{i})\) is defined as the height of the lowest common ancestor (LCA) of the words.3 Subsequently, in Gomez-Rodriguez and Vilares (2018), the NSD \(d_{G}(w_{i})\) was defined as the number of the common ancestors of the words. To make the definition complete, we define \(d(w_{n})\) as follows:4

Footnote 2: Note that NSD is defined between two contiguous words. For convenience of notation, we use \(d(w_{i})\) rather than \(d(w_{i},w_{i+1})\) to denote an NSD.

Footnote 3: In Shen et al. (2018), NSD is defined as a real number that is a function of LCA. However, in practice, NSD is simply identical to the depth of the LCA.

Footnote 4: \(d_{S}(w_{n})\) and \(d_{G}(w_{n})\) are undefined in both of the original papers. We give the definitions here to enable the use of NSD in NMT later.

\[d_{S}(w_{n})=H,~{}~{}d_{G}(w_{n})=0, \tag{1}\]

where \(H\) is the height of the constituent tree. It is easy to prove that

\[d_{S}(w_{i})+d_{G}(w_{i})=H,~{}~{}~{}i\in[1,n]. \tag{2}\]

We call \(d_{S}\) and \(d_{G}\) the _absolute_ NSD.

Furthermore, Gomez-Rodriguez and Vilares (2018) define the _relative_ NSD as follows:

\[d_{R}(w_{i})=\begin{cases}d_{G}(w_{1}),&i=1,\\ d_{G}(w_{i})-d_{G}(w_{i-1}),&i\in[2,n].\end{cases} \tag{3}\]

Figure 1 illustrates these NSDs. It is easy to see the one-to-one correspondence relationship between the constituent tree and the (absolute or relative) NSDs.

The effectiveness of all different NSDs has been proven on constituent parsing. However, there has been no attempt to use NSD in machine translation.

## 3 Strategies to improve NMT with NSD

### Dependency NSD

There are many previous studies on using dependency trees to improve NMT (Nguyen Le et al., 2017; Wu et al., 2017). Therefore, we extend NSD to dependency trees. Formally, the dependency NSD between two nodes is defined as follows:

\[d_{D}(w_{i})=i-h(i), \tag{4}\]

where \(h(i)\) is the index of the head of \(w_{i}\), and we let the index of root be \(0\). Note that \(d_{D}(w_{i})\) can be either positive or negative, representing the directional information. Figure 2 gives an example.

### NSDs as Input Sequences

It is easy to see that for \({\bf w}=(w_{1},\dots,w_{n})\), the lengths of \(d_{S},d_{G},d_{R}\) and \(d_{D}\) are all \(n\). Denoting the NSD sequence as \({\bf d}=(d_{1},\dots,d_{n})\), we can see that \(d_{i}\in\mathbb{Z},i\in[1,n]\), so we can obtain a sequence of embedding vectors \({\bf e}^{d}=(e^{d}_{1},\dots,e^{d}_{n})\) as follows:

\[e^{d}_{i}=E_{d}[d^{d}_{i}+(\max({\bf d})-\min({\bf d})+1)]. \tag{5}\]

We call \(E_{d}\) the distance embedding matrix and call \({\bf e}^{d}\) the syntactic embedding sequence. Note that \({\bf d}\) can be the NSD on either the source side or the target side, so there are two possible \(E_{d}\), which are

Figure 1: Example of different NSDs. This example is from Shen et al. (2018).

Figure 2: Example of dependency NSDs. “#” is the root. Dependency labels are omitted.

 

denoted as \(E_{d}^{s}\) and \(E_{d}^{t}\), respectively. The embeddings are calculated as follows:

\[x_{i}^{s}=f_{emb}(E_{w}^{s}[u_{i}^{s}],e_{i}^{ds}), \tag{6}\] \[x_{i}^{t}=f_{emb}(E_{w}^{t}[w_{i}^{t}],e_{i}^{dt}), \tag{7}\]

where \(e_{i}^{ds}\) and \(e_{i}^{dt}\) are defined in Eq. 5 on the source side and target side, respectively, and \(E_{w}^{s}\) and \(E_{w}^{t}\) are the word embedding matrices on both sides, respectively. Inspired by Sennrich2016, function \(f_{emb}\) is used to combine two vectors. This function has many different options, such as:

\[f_{emb}^{\parallel}(x,e)=x\|e, \tag{8}\] \[f_{emb}^{+}(x,e)=x+e,\] (9) \[f_{emb}^{Wb}(x,e)=W_{f}\cdot(x\|e)+b_{f}, \tag{10}\]

where \(x,e,b_{f}\in\mathbb{R}^{d}\) and \(W_{f}\in\mathbb{R}^{d\times 2d}\). The operator "\(\parallel\)" is the concatenation of two vectors.

When NSD is used as the input sequence on the target side, there is one problem: \(\mathbf{e}^{dt}\) is unknown during testing. For this case, we use NSDs for both the input and output sequences, let the decoder predict NSD on-the-fly using the strategy introduced in Section 3.3, and use the predicted NSD to calculate \(\mathbf{e}^{dt}\).

### NSDs as Output Sequences

An NSD can be used to form the output sequence to improve NMT using the idea of multi-task learning. Specifically, we train the model to predict the NSD sequence. When NSD is used as the output sequence of the _encoder_, we minimize the distance (e.g., cross entropy \(\mathcal{L}_{dist}^{ent}\), see Section 3.5 for details) between the predicted and the golden NSD sequences. When NSD is used as the output sequence of the _decoder_, besides minimizing the distance, we use the predicted NSD as the input of the next time step.

Denote the hidden vector as \(\mathbf{h}=(h_{1},\dots,h_{n})\). For the encoder, \(h_{i}=h_{i}^{s}\) and \(n=n_{s}\), while for the decoder, \(h_{i}=h_{i}^{t}\) and \(n\) is the current time step of decoding. Then, we can obtain a sequence of predicted syntactic distance \(\hat{\mathbf{d}}=(\hat{d}_{1},\dots,\hat{d}_{n})\), which is calculated as follows:

\[p(\hat{d}_{i}\mid h_{i})=\mathrm{softmax}(W_{d}\cdot h_{i}+b_{d}), \tag{11}\]

where \(W_{d}\) and \(b_{d}\) are parameters to be learned. By minimizing the distance between \(\hat{d}_{i}\) and \(d_{i}\), NSD can be used to enhance NMT.

### NSD as Positional Encoding (PE)

PE is used by the Transformer Vaswani et al. (2017) to encode the positions of words. Formally, it is defined as follows:

\[x_{i}^{\prime}=x_{i}+PE(i), \tag{12}\] \[PE(i)_{2k}=\sin(i/10000^{2k/d}),\] (13) \[PE(i)_{2k+1}=\cos(i/10000^{2k/d}), \tag{14}\]

where \(x_{i}\) can be either \(x_{i}^{s}\) or \(x_{i}^{t}\), and \(d\) is the dimension of the embedding vector. Similarly, we define syntactic PE as follows:

\[PE(i)_{2k}=\sin\Big{(}\frac{i+\max(\mathbf{d})-\min(\mathbf{d})}{{\lambda_{ SPE}}^{2k/d}}\cdot 2\pi\Big{)}, \tag{15}\] \[PE(i)_{2k+1}=\cos\Big{(}\frac{i+\max(\mathbf{d})-\min(\mathbf{ d})}{{\lambda_{SPE}}^{2k/d}}\cdot 2\pi\Big{)}, \tag{16}\]

where \(\lambda_{SPE}\) is a hyperparameter to be tuned. In this way, the periods of these two functions vary from \(1\) to \(\lambda_{SPE}\). We define syntactic PE in this way because (1) according to a quantitative analysis of the experimental datasets, we found that the ranges of possible values are quite different between NSD and word positions, so we tuned \(\lambda_{SPE}\) instead of fixed it to \(10000\) as in Eqs. 13 and 14, and (2) \(d_{i}\) may be negative, so we adjust it to be positive.

### Distance-aware Training

Instead of using conventional cross-entropy loss function during training, we use the following loss function to make the NMT model learn NSD better:

\[\mathcal{L}=\mathcal{L}_{NMT}+\mathcal{L}_{dist}+\mathcal{L}_{ dist}^{ent}. \tag{17}\]

The first item is the cross-entropy loss of the NMT model, which is

\[\mathcal{L}_{NMT}=-\sum_{\langle\textbf{w}^{s},\textbf{w}^{t}\rangle\in \mathcal{D}}\log p(\textbf{w}^{t}\mid\textbf{w}^{s}), \tag{18}\]

where \(\mathcal{D}\) is the training dataset. The second item is the distance-aware loss, which is inspired by Shen2018 and is as follows:

\[\mathcal{L}_{dist}= \sum_{\langle\textbf{w}^{s},\textbf{w}^{t}\rangle\in\mathcal{D}} (\mathcal{L}_{dist}^{s}(\textbf{w}^{s})+\mathcal{L}_{dist}^{t}(\textbf{w}^{t})),\] \[\mathcal{L}_{dist}^{s}(\textbf{w}^{s})= \sum_{i=1}^{n_{s}}(d_{i}-\hat{d}_{i})^{2}+\] \[\sum_{i,j>i}[1-\operatorname{sign}(d_{i}-d_{j})(\hat{d}_{i}-\hat {d}_{j})]^{+},\] 

and \({\cal L}_{dist}^{t}\) can be defined similarly. The third item is the cross-entropy loss for NSD, which is as follows:

\[\begin{split}&{\cal L}_{dist}^{ent}=\sum_{\langle{\bf w}^{s},{\bf w} ^{t}\rangle\in{\cal D}}({\cal L}_{dist}^{ent(s)}({\bf w}^{s})+{\cal L}_{ dist}^{ent(t)}({\bf w}^{t})),\\ &{\cal L}_{dist}^{ent(s)}({\bf w}^{s})=-\sum_{d_{i}\in{\bf d}^{ s}}p(d_{i}~{}|~{}h_{i})\log p(\hat{d}_{i}~{}|~{}h_{i}),\end{split} \tag{20}\]

and \({\cal L}_{dist}^{ent(t)}\) can be defined similarly.

## 4 Experiments

### Configuration

We experimented on two corpora: (1) ASPEC Nakazawa et al. (2016), using the top \(100\)K sentence pairs for training En-Ja models and top \(1\)M sentence pairs for training Ja-En models, and (2) LDC,5 which contains about 1.2M sentence pairs, for training En-Ch and Ch-En models. To tackling the problem of memory consumption, sentences longer than 150 were filtered out, so that models can be trained successfully. Chinese sentences were segmented by the Stanford segmentation tool.6 For Japanese sentences, we followed the preprocessing steps recommended in WAT 2017.7

Footnote 5: LDC200E18, LDC2003E07, LDC2003E14, Hansards portion of LDC2004T07, LDC2004T08, and LDC2005T06.

Footnote 6: [https://nlp.stanford.edu/software/stanford-segmenter-2017](https://nlp.stanford.edu/software/stanford-segmenter-2017)–06-09.zip

Footnote 7: [http://lotus.kuee.kyoto-u.ac.jp/WAT/WAT2017/baseline/dataPreparationJE.html](http://lotus.kuee.kyoto-u.ac.jp/WAT/WAT2017/baseline/dataPreparationJE.html)

The test set is a concatenation of NIST MT 2003, 2004, and 2005. Constituent trees are generated by the parser of Kitaev and Klein (2018)8, and dependency trees are generated by the parser of dyer2015automatic. Note that although we only used syntactic information of English in our experiments, our method is also applicable to other languages.

Footnote 8: [https://github.com/nikitakit/self-attentive-parser](https://github.com/nikitakit/self-attentive-parser)

Footnote 9: [https://github.com/clab/lstm-parser](https://github.com/clab/lstm-parser)

We implemented our method on OpenNMT10Klein et al. (2017), and used the Transformer as our baseline. As far as we know, there are no previous studies on using syntactic informations in the Transformer.

Footnote 10: [http://opennmt.net](http://opennmt.net)

The vocabulary sizes for all languages are \(50,000\). Both the encoder and decoder have \(6\) layers. The dimensions of hidden vectors and word embeddings are \(512\). The multi-head attention has \(8\) heads, and the dropout probability is \(0.1\)Srivastava et al. (2014). The number of training epochs was fixed to \(50\), and we used the model which performs the best on the development set for testing.

As for optimization, we used the Adam optimizer Kingma and Ba (2014), with \(\beta_{1}=0.9,\beta_{2}=0.998\), and \(\epsilon=10^{-9}\). Warmup and decay strategy for learning rate of vaswani2017attention are also used, with \(8,000\) warmup steps. We also used the label smoothing strategy Szegedy et al. (2016) with \(\epsilon_{ls}=0.1\).

### Experimental Results

Table 1 compares the effects of the strategies. We evaluate the proposed strategies using character-level BLEU Papineni et al. (2002) for Chinese and Japanese, and case-insensitive BLEU for English.

**Comparison of different NSDs.** The first five rows of Table 1 compare the results of using different NSDs. When NSD was used at the source side (En-Ja/En-Ch), all kinds of NSDs improved translation performance. This indicates that NSD can be regarded as a useful linguistic feature to improve NMT. In contrast, when NSD was used at the target side (Ja-En/Ch-En), \(d_{S}\) and \(d_{G}\) hurt the performance. This is because the values of \(d_{S}\) and \(d_{G}\) are volatile. A tiny change of syntactic structure often causes a big change of \(d_{S}\) and \(d_{G}\). Since the model has to predict the NSD during decoding, once there is one error, the subsequent predictions will be heavily influenced. The use of \(d_{R}\) and \(d_{D}\) remedies this problem. Furthermore, the effects of \(d_{S}\) and \(d_{G}\) are similar, because they are equivalent in nature (refer to Eq. 2).

**NSD as PE.** Rows 5 to 8 of Table 1 evaluate the use of dependency NSD (\(d_{D}\)) as syntactic PE. Note that for all the experiments, we used not only the syntactic PE but the conventional PE. Experiment results show that this strategy is indeed useful. When the dominators of Eqs. 15 and 16, \(\lambda_{SPE}\), were set to \(10^{4}\), there was no improvement. When they were set to \(40\), the improvement was remarkable. This indicates that our design of syntactic PE is reasonable.

**NSD as input/output and source/target sequences.** Rows 8 to 12 of Table 1 are the results of using dependency NSD (i.e., \(d_{D}\)) as the input and/or output sequences on both sides. First, for the choice of \(f_{embb}\), we can see that \(f_{emb}^{||}\) and \(f_{emb}^{+}\) are similar, while \(f_{emb}^{Wb}\) yields better performance. This is because the model has to learn \(W_{f}\) and 

\(b_{f}\), which increases the model capacity. Second, performance improved for using NSDs both as input and output sequences, and combining both obtained further improvement. Third, NSDs improved the performance both on the source and the target sides. All these results indicate the robustness of NSDs.

**Effects of distance-aware training.** The last three rows compare the different effects of the items in the loss function. When only \(\mathcal{L}_{NMT}\) are used, the performance is extremely poor. This is within expectations, because with only \(\mathcal{L}_{NMT}\), weights related to NSDs are kept to the initial values and were not updated, and hence detrimental to learning. Adding \(\mathcal{L}^{ent}_{dist}\) improves the results significantly, but the improvement is lower than that of \(\mathcal{L}_{dist}\). This is because training with \(\mathcal{L}^{ent}_{dist}\) treats different values of NSDs equally, while \(\mathcal{L}_{dist}\) penalizes larger differences between the predicted NSD and the golden NSD more severely.

## 5 Conclusion

We proposed five strategies to improve NMT with NSD. We found relative NSDs and dependency NSDs are able to improve the performance consistently, while absolute NSDs hurt the performance for some cases. The improvement obtained by using NSDs is general in that NSDs can be used at both the source side and target side, both as input sequences and output sequences. Using NSDs as syntactic PE is also useful, and training with a distance-aware loss function is quite important.

## Acknowledgements

We are grateful to the anonymous reviewers for their insightful comments and suggestions. Akihiro Tamura is supported by JSPS KAKENHI Grant Number JP18K18110. Tiejun Zhao is supported by the National Key Research and Development Program of China via grant 2017YFB1002102.

## References

* Bahdanau et al. (2014) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. _arXiv preprint arXiv:1409.0473_.
* Dyer et al. (2015) Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A. Smith. 2015. Transition-based dependency parsing with stack long short-term memory. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 334-343.
* Eriguchi et al. (2016) Akiko Eriguchi, Kazuma Hashimoto, and Yoshimasa Tsuruoka. 2016. Tree-to-sequence attentional neural machine translation. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 823-833.
* Firat et al. (2016) Orhan Firat, Bakaran Sankaran, Yaser Al-Onaizan, Fatos T. Yarmal Vural, and Kyunghyun Cho. 2016. Zero-resource translation with multi-lingual neural machine translation. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 268-277.

 

Cited by: SS2.
* C. Gomez-Rodriguez and D. Vilares (2018)Constitent parsing as sequence labeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1314-1324. Cited by: SS5.
* K. Hashimoto and Y. Tsuruoka (2017)Neural machine translation with source-side latent graph parsing. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 125-135. Cited by: SS6.
* D. P Kingma and J. Ba (2014)Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980. Cited by: SS4.2.
* N. Kitaev and D. Klein (2018)Constitency parsing with a self-attentive encoder. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2676-2686. Cited by: SS1, SS2.
* G. Klein, Y. Kim, Y. Deng, J. Senellart, and A. M. Rush (2017)OpenNMT: open-source toolkit for neural machine Translation. arXiv preprint arXiv:1701.02810. Cited by: SS1, SS2.
* J. Li, D. Xiong, Z. Tu, M. Zhu, M. Zhang, and G. Zhou (2017)Modeling source syntax for neural machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 688-697. Cited by: SS1, SS2.
* C. Ma, L. Liu, A. Tamura, T. Zhao, and E. Sumita (2017)Deterministic attention for sequence-to-sequence constituent parsing. In Proc. of AAAI-2017, pp. 3237-3243. Cited by: SS1, SS2.
* C. Ma, A. Tamura, M. Utiyama, T. Zhao, and E. Sumita (2018)Forest-based neural machine translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1253-1263. Cited by: SS1, SS2.
* T. Nakazawa, M. Yaguchi, K. Uchiimoto, M. Utiyama, E. Sumita, S. Kurohashi, and H. Isahara (2016)ASPEC: Asian scientific paper excerpt corpus. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC 2016), pp. 2204-2208. Cited by: SS1, SS2.
* An. Nguyen Le, A. Martinez, A. Yoshimoto, and Y. Matsumoto (2017)Improving sequence to sequence neural machine translation by utilizing syntactic dependency information. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 21-29. Cited by: SS1, SS2.
* K. Papineni, S. Roukos, T. Ward, and W. Zhu (2002)BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311-318. Cited by: SS1, SS2.
* R. Sennrich and B. Haddow (2016)Linguistic input features improve neural machine translation. In Proceedings of the First Conference on Machine Translation: Volume 1, Research Papers, pp. 83-91. Cited by: SS1, SS2.
* L. Shen, J. Xu, and R. Weischedel (2010)String-to-dependency statistical machine translation. Computational Linguistics36 (4). External Links: Link Cited by: SS1, SS2.
* Y. Shen, Z. Lin, A. Paul Jacob, A. Sordoni, A. Courville, and Y. Bengio (2018)Straight to the tree: constituency parsing with neural syntactic distance. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1171-1180. Cited by: SS1, SS2.
* N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov (2014)Dropout: a simple way to prevent neural networks from overfitting. Journal of machine learning research15 (1), pp. 1929-1958. Cited by: SS1, SS2.
* I. Sutskever, O. Vinyals, and Q. V. Le (2014)Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pp. 3104-3112. Cited by: SS1, SS2.
* C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna (2016)Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826. Cited by: SS1, SS2.
* A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin (2017)Attention is all you need. In Advances in Neural Information Processing Systems, pp. 5998-6008. Cited by: SS1, SS2.
* S. Wu, D. Zhang, N. Yang, M. Li, and M. Zhou (2017)Sequence-to-dependency neural machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 698-707.