Venue: ACL 2017

# Hybrid Neural Network Alignment and Lexicon Model in Direct HMM

for Statistical Machine Translation

 Weiyue Wang, Tamer Alkhouli, Derui Zhu, Hermann Ney

Human Language Technology and Pattern Recognition, Computer Science Department

RWTH Aachen University, 52056 Aachen, Germany

<surname>@i6.informatik.rwth-aachen.de

###### Abstract

Recently, the neural machine translation systems showed their promising performance and surpassed the phrase-based systems for most translation tasks. Retreating into conventional concepts machine translation while utilizing effective neural models is vital for comprehending the leap accomplished by neural machine translation over phrase-based methods. This work proposes a direct hidden Markov model (HMM) with neural network-based lexicon and alignment models, which are trained jointly using the Baum-Welch algorithm. The direct HMM is applied to rerank the \(n\)-best list created by a state-of-the-art phrase-based translation system and it provides improvements by up to \(1.0\%\) Bleu scores on two different translation tasks.

+
Footnote †: Proceedings of the 53th Annual Meeting of the Association for Computational Linguistics (Short Papers), pages 125–131 Vancouver, Canada, July 30 - August 4, 2017. ©2017 Association for Computational Linguistics [https://doi.org/10.18653/v1/P17-2020](https://doi.org/10.18653/v1/P17-2020)

## 1 Introduction

The hidden Markov model (HMM) was first introduced to statistical machine translation for addressing the word alignment problem Vogel et al. (1996). Then the HMM-based approach was widely used along with the IBM models Brown et al. (1993) for aligning the source and target words. In the conventional approach, the Bayes' theorem is used and the HMM is applied to the inverse translation model

\[\begin{split}\Pr(e_{1}^{I}|f_{1}^{J})&=\Pr(e_{1}^{ I})\cdot\Pr(f_{1}^{J}|e_{1}^{I})\\ &=\sum_{a_{1}^{J}}\Pr(f_{1}^{J},a_{1}^{J}|e_{1}^{I})\end{split} \tag{1}\]

In this case, as a part of a noisy channel model, the marginalisation becomes intractable for every \(e\).

This work proposes a novel concept focusing on direct HMM for \(\Pr(e_{1}^{I}|f_{1}^{J})\), in which the alignment direction is from target to source positions. This specific property allows us to introduce dependencies into the translation model that take the full source sentence into account. This aspect will be important for the future decoder to be developed. The lexicon and alignment probabilities in the HMM are modeled using feed-forward neural networks (FFNN) and they are trained jointly. The trained HMM is then applied for reranking the \(n\)-best lists created by a state-of-the-art open source phrase-based translation system. The experiments are conducted on the IWSLT 2016 German\(\rightarrow\)English and BOLT Chinese\(\rightarrow\)English translation tasks. The FFNN-based hybrid HMM provides improvements by up to \(1.0\%\) Bleu scores.

## 2 Related Work

In order to discuss related work, we will consider the following two key concepts that are essential for the work to be presented:

* **Neural lexicon and alignment models** The idea of using neural networks for lexicon modeling is not new Schwenk (2012); Sundermeyer et al. (2014); Devlin et al. (2014). Apart from differences in the neural network architecture, the important difference to this work is that those approaches did not include the concepts of HMM models and end-to-end training. In addition to neural lexicon modeling, Alkhouli et al. (2016) also applied a neural network for alignment modeling like this work, but their training procedure was based on the maximum approximation and on predefined GIZA++ Och and Ney (2003) alignments.



There were other studies that focused on feature-rich alignment models (Blunsom and Cohn, 2006; Berg-Kirkpatrick et al., 2010; Dyer et al., 2011), but those studies did not use a neural network to automatically learn features (as we do in this work). (Yang et al., 2013) used neural network-based lexicon and alignment models inside the HMM alignment model, but they model alignments using a simple distortion model that has no dependence on lexical context. Their goal was to improve the alignment quality in the context of a phrase-based translation system. However, apart from (Dyer et al., 2011), no results on translation were reported. The idea of using neural networks is the basis of the state-of-the-art attention-based approach to machine translation (Bahdanau et al., 2015; Luong et al., 2015). However, that approach is not based on the principle of an explicit and separate lexicon model.
* **End-to-end training** The HMM in combination with the neural translation model lends itself to what is usually called end-to-end training. The training criterion is the logarithm of the target sentence posterior probability. This criterion results in a specific training algorithm that can be interpreted as a combination of forward-backward algorithm (as in EM style training of HHMs) and backpropagation. To the best of our knowledge, this end-to-end training has not been considered before for machine translation. In the context of signal processing and recognition, the connectionist temporal classification (CTC) approach (Graves et al., 2006) leads to a similar training procedure. (Tran et al., 2016) studied neural networks for unsupervised training for a part-of-speech tagging task. In their approach, the training criterion for this problem results in a combination of EM framework and backpropagation, which has a certain similarity to the training algorithm for translation as presented in this work.

## 3 Definition of neural network-based HMM

Similar to hidden alignments \(a_{j}=j\to i\) between the source string \(f^{J}_{1}=f_{1}...f_{j}....f_{J}\) and the target string \(e^{I}_{1}=e_{1}...e_{i}...e_{I}\) in the conventional HMM, we define the alignments in direct HMM as \(b_{i}=i\to j\). Then the model can be defined as:

\[\Pr(e^{I}_{1}|f^{J}_{1})=\sum_{b^{I}_{1}}\Pr(e^{I}_{1},b^{I}_{1}|f^{J}_{1}) \tag{2}\]

\[\begin{split}&\Pr(e^{I}_{1},b^{I}_{1}|f^{J}_{1})\\ &=\prod_{i=1}^{I}p(e_{i},b_{i}|b^{i-1}_{1},e^{i-1}_{1},f^{J}_{1}) \\ &=\prod_{i=1}^{I}\underbrace{p(e_{i}|b^{i}_{1},e^{i-1}_{1},f^{J}_ {1})}_{\text{lexicon model}}\cdot\underbrace{p(b_{i}|b^{i-1}_{1},e^{i-1}_{1}, f^{J}_{1})}_{\text{alignment model}}\end{split} \tag{3}\]

Our feed-forward alignment model has the same architecture (Figure 1) as the one proposed in (Alkhouli et al., 2016). Thus the alignment probability can be modeled by:

\[p(b_{i}|b^{i-1}_{1},e^{i-1}_{1},f^{J}_{1})=p(\Delta_{i}|f^{b_{i-1}+\gamma_{m }}_{b_{i-1}-\gamma_{m}},e^{i-1}_{i-n}) \tag{4}\]

where \(\gamma_{m}=\frac{m-1}{2}\) and \(m\) indicates the window size. \(\Delta_{i}=b_{i}-b_{i-1}\) denotes the jump from the predecessor position to the current position. Thus, the jump over the source is estimated based on a \(m\)-words source context window and \(n\) predecessor target words.

For the lexicon model, we assume a similar dependence as in the alignment model with a shift, namely on the source words within a window centred on the aligned source word and \(n\) predecessor target words. To overcome the high costs of the softmax function for large vocabularies, we adopt the class-factored output layer consisting of a class layer and a word layer (Goodman, 2001; Morin

Figure 1: A feed-forward alignment neural network with 3 target history words, 5-gram source window, a projection layer, 2 non-linear hidden layers and a small output layer to predict jumps.

 

and Bengio, 2005). The model in this case is defined as

\[p(e_{i}|b^{i}_{1},e^{i-1}_{1},f^{J}_{I})\] \[=p(e_{i}|f^{b_{i}+\gamma_{m}}_{b_{i}-\gamma_{m}},e^{i-1}_{i-n})\] \[=p(e_{i}|c(e_{i}),f^{b_{i}+\gamma_{m}}_{b_{i}-\gamma_{m}},e^{i-1}_ {i-n})\cdot p(c(e_{i})|f^{b_{i}+\gamma_{m}}_{b_{i}-\gamma_{m}},e^{i-1}_{i-n}) \tag{5}\]

where \(c\) denotes a word mapping that assigns each target word to a single class, where the number of classes is chosen to be much smaller than the vocabulary size. The lexicon model architecture is shown in Figure 2.

## 4 Training

The training data of the direct HMM are the source and target sequences, without any alignment information. In the training of direct HMM including neural network-based models, the weights have to be updated along with the posterior probabilities calculated by the Baum-Welch algorithm. Similar to the training procedure used in Bergkirkpatrick et al. (2010), we apply the EM algorithm and define the auxiliary function as

\[Q(\theta;\hat{\theta})\\ =\sum_{b^{l}_{1}}p(b^{l}_{1}|f^{j}_{1},e^{l}_{1},\theta)\log p( e^{l}_{1},b^{l}_{1}|f^{j}_{1},\hat{\theta})\\ =\sum_{b^{l}_{1}}p(b^{l}_{1}|f^{j}_{1},e^{l}_{1},\theta)\sum_{i=1 }^{I}\hskip-36.98858pt\int\limits_{0}^{P(e_{i}|f^{b_{i}+\gamma_{m}}_{b_{i}- \gamma_{m}},e^{i-1}_{i-n},\hat{\alpha})}\\ +\log p(\Delta_{i}|f^{b_{i-1}+\gamma_{m}}_{b_{i-1}-\gamma_{m}},e^ {i-1}_{i-n},\hat{\beta})]\\ =\sum_{i}\sum_{j}p_{i}(j|e^{l}_{1},f^{j}_{1},\theta)\log p(e_{i} |f^{j+\gamma_{m}}_{j-\gamma_{m}},e^{i-1}_{i-n},\hat{\alpha})\\ +\sum_{i}\sum_{j^{\prime}}\sum_{j}p_{i}(j^{\prime},j|e^{l}_{1},f^ {J}_{1},\theta)\log p(\Delta_{i}|f^{j^{\prime}+\gamma_{m}}_{j^{\prime}-\gamma_ {m}},e^{i-1}_{i-n},\hat{\beta}) \tag{6}\]

where \(\hat{\theta}=\{\hat{\alpha},\hat{\beta}\}\), \(j^{\prime}=b_{i-1}\) and

\[p_{i}(j|e^{I}_{1},f^{J}_{1},\theta)=\sum_{b^{l}_{1}:b_{i}=j}p(b^{l}_{1}|e^{l} _{1},f^{J}_{1},\theta) \tag{7}\]

Then the parameters can be separated for lexicon model and alignment model:

\[Q(\theta;\hat{\theta})=Q_{\text{lex}}(\theta;\hat{\alpha})+Q_{\text{align}}( \theta;\hat{\beta}) \tag{8}\]

where

\[\frac{\partial Q_{\text{lex}}(\theta,\hat{\alpha})}{\partial\hat{ \alpha}}=\sum_{i}\sum_{j}\sum_{j}\overbrace{p_{i}(j|e^{I}_{1},f^{J}_{1}, \theta)}^{\text{forward-backward algorithm}}\\ \cdot\underbrace{\frac{\partial}{\partial\hat{\alpha}}\log p(e_{ i}|f^{j+\gamma_{m}}_{j-\gamma_{m}},e^{i-1}_{i-n},\hat{\alpha})}_{\text{ backpropagation}} \tag{9}\]

\[\begin{split}\frac{\partial Q_{\text{align}}(\theta,\hat{\beta})}{\partial\hat{\beta}}&= \sum_{i}\sum_{j^{\prime}}\sum_{j}\overbrace{p_{i}(j^{\prime},j|e^{I}_{1},f^{ J}_{1},\theta)}^{\text{forward-backward algorithm}}\\ &\qquad\quad\cdot\underbrace{\frac{\partial}{\partial\hat{\beta}} \log p(\Delta_{i}|f^{j^{\prime}+\gamma_{m}}_{j^{\prime}-\gamma_{m}},e^{i-1}_{i -n},\hat{\beta})}_{\text{backpropagation}}\end{split} \tag{10}\]

From Equations (9) and (10) we can observe that the marginalisation of hidden alignments (\(\sum_{j}p_{i}(j|e^{I}_{1},f^{J}_{1},\theta)\)) is the only difference compared to the derivative of neural network training based on word-aligned data. In this approach we iterate over all source positions and the word alignment toolkit such as GIZA++ is not required. Furthermore, the word-aligned data generated e.g. by GIZA++ might contain unaligned and multiply aligned words, which make the data difficult to use for training neural networks. Thus the heuristic-based approaches Sundermeyer et al. (2014); Devlin et al. (2014) have to be used in order to guarantee the one-on-one alignments, which may negatively influence the quality of the alignments. By contrast, the neural network-based HMM is not constrained by these heuristics. In addition, even though the training process of the direct HMM takes more time than the neural network training on the word-aligned data, we should note that generating the word-aligned data using GIZA++ is also a time-consuming process.

In general, our training procedure can be summarized as follows:

Figure 2: A feed-forward lexicon neural network with the same structure as the alignment model, except a class-factored output layer.

 

1. One iteration IBM-1 model training to create lexicon table for initializing the forward-backward table.
2. In the first epoch, for each sentence pair calculate and save the entire table of posterior probabilities \(p_{i}(b|e^{I}_{1},f^{J}_{1})\) (also \(p_{i}(b^{\prime},b|e^{I}_{1},f^{J}_{1})\) for alignment model) using forward-backward algorithm based on the results of IBM-1 model.
3. Training neural network lexicon and alignment models based on the posterior probabilities.
4. From the second epoch onwards: 1. For each sentence pair, calculating the posterior probabilities based on the lexicon and alignment probabilities estimated by neural network models. 2. Updating weights of neural networks based on the posterior probabilities. 3. Repeating step 4 until the perplexity converges.

In this work the IBM-1 initialization is required. We tried to train neural network models from scratch, but the perplexity converges towards a bad local minimum and gets stuck in it. We also attempted other heuristics for initialization, such as assigning probability \(0.9\) to diagonal alignments and spreading the left \(0.1\) evenly among other source positions. The resulted perplexity is much higher compared to initializing using IBM-1.

## 5 Experimental Results

The experiments are conducted on the IWSLT 2016 German\(\rightarrow\)English and BOLT Chinese\(\rightarrow\)English translation tasks, which consist of \(20M\) and \(4M\) parallel sentence pairs respectively. The feed-forward neural network alignment and lexicon models are jointly trained on the subset of about \(200K\) sentence pairs. As an initial research of this topic, our new model is only applied for reranking \(n\)-best lists created by a phrase-based decoder. The maximum size of the \(n\)-best lists is 500. The translation quality is evaluated by case-insensitive BleuPapineni et al. (2002) and TerSnover et al. (2006) metrics using MultEval Clark et al. (2011). The scaling factors are tuned with MERT Och (2003) with Bleu as optimization criterion on the development sets. For the translation experiments, the averaged scores are presented on the development set from three optimization runs.

Our direct HMM consists of a feed-forward neural network lexicon model with following configuration:

* Five one-hot input vectors for source words and three for target words
* Projection layer size \(100\) for each word
* Two non-linear hidden layers with \(1000\) and \(500\) nodes respectively
* A class-factored output layer with \(1000\) singleton classes dedicated to the most frequent words, and 1000 classes shared among the rest of the words.

and a feed-forward neural network alignment model with the same configuration as the lexicon model, except a small output layer with \(201\) nodes, which reflects that the aligned position can jump within the scope from \(-100\) to \(100\)Alkhouli et al. (2016).

We conducted experiments on the source and target window size of both network models. Larger source and target windows could not provide significant improvements on Bleu scores, at least for rescoring experiments.

The model is applied for reranching the \(n\)-best lists created by the Jane toolkit Vilar et al. (2010); Wuebker et al. (2012) with a log-linear framework containing phrasal and lexical smoothing models for both directions, word and phrase penalties, a distance-based reordering model, enhanced low frequency features Chen et al. (2011), a hierarchical reordering model Galley and Manning (2008), a word class language mode Wuebker et al. (2013) and an \(n\)-gram language model. The word alignments used for the training of phrase-tables are generated by GIZA++, which performs the alignment training sequentially for IBM-1, HMM and IBM-4. More details about our phrase-based baseline system can be found in Peter et al. (2015).

The experimental results are demonstrated in Table 1. The rescoring experiments are conducted by adding HMM probability as feature and tuned with MERT. The applied attention-based neural network is a neural machine translation system similar to Bahdanau et al. (2015). The decoder and encoder word embeddings are of size 620, the encoder uses a bidirectional layer with 1000 LSTMs Hochreiter and Schmidhuber (1997) to encode the source side. A layer with 1000 LSTMs 

is used by the decoder. The data is converted into subword units using byte pair encoding with 20000 operations (Sennrich et al., 2016). During training a batch size of 50 is used. More details about our neural machine translation system can be found in Peter et al. (2016).

With \(n\)-best rescoring, all neural network-based systems achieve significant improvements over the phrase-based system. The neural network-based HMMs provide promising performance, even with simple feed-forward neural networks. The direct HMM trained by the EM procedure with marginalizing the hidden alignments outperformed the same model trained on the word-aligned data. For the rescoring tasks, it provides comparable performance with the attention-based network. The neural network-based HMM also helps the phrase-based system achieve comparable results with the stand-alone attention-based system on the German\(\rightarrow\)English task.

## 6 Conclusion and Future Work

This work aims to close the gap between the conventional word alignment models and the novel neural machine translation. The proposed direct HMM consists of neural network-based alignment and lexicon models, both models are trained jointly and without any alignment information. With the simple feed-forward neural network models, the HMM model already provides promising results and significantly improves the strong phrase-based translation system.

As future work, we are searching for alternatives to initialize the training instead of using IBM-1. We will investigate recurrent model structures, such as the LSTM representation for source and target word embeddings (Luong et al., 2015). In addition to the network structure, we will implement a stand-alone decoder based on this novel model. The first step would be to apply maximum approximation for the search problem as elucidated in Yu et al. (2017). Then we plan to investigate heuristics for marginalizing the hidden alignment during search.

The work reported in this paper results from two projects, SEQCLAS and QT21. SEQCLAS has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme under grant agreement no 694537. QT21 has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement no 645452. The work reflects only the authors' views and neither the European Commission nor the European Research Council Executive Agency are responsible for any use that may be made of the information it contains.

Tamer Alkhouli was partly funded by the 2016 Google PhD Fellowship for North America, Europe and the Middle East.

The authors would like to thank Jan-Thorsten Peter for providing the attention-based neural network models.

\begin{table}
\begin{tabular}{l|c c c c|c c c} \hline  & \multicolumn{4}{c|}{IWSLT} & \multicolumn{4}{c}{BOLT} \\  & \multicolumn{2}{c}{TEMX.ist.2014} & MSLT.dev2016 & \multicolumn{2}{c}{DEV12} & PIR6 \\  & Bleu[\%] & Ter[\%] & Bleu[\%] & Ter[\%] & Bleu[\%] & Ter[\%] & Bleu[\%] & TeR[\%] \\ \hline Phrase-based translation system & 25.9 & 55.7 & 39.7 & 39.8 & 17.9 & 68.3 & 17.1 & 67.4 \\ + NN-based direct HMM (Viterbi) & 26.6 & 54.9 & 40.2 & 39.3 & 18.5 & 67.6 & 17.8 & 66.8 \\ + NN-based direct HMM (sum) & 26.9 & **54.7** & **40.6*** & **38.9*** & 18.9* & 67.3*† & 18.3* & 66.4 \\ + attention-based system \({}^{1}\) & 26.8 & 55.0 & 40.4 & 39.3 & 18.9* & 67.6 & 18.0 & 66.4* \\ \hline Stand-alone attention-based system \({}^{2}\) & **27.0** & 54.8 & 40.4 & 39.2 & **19.6*** & **67.0*** & **18.5*** & **66.1*** \\ \hline \end{tabular}
\end{table}
Table 1: Experimental results of rescoring using neural network-based direct HMM. The model with _sum_ denotes the system proposed in this work, while the model with _Viterbi_ denotes the model with the same neural network structure, which was trained based on the word-aligned data (alignments generated by GIZA++) Alkhouli et al. (2016). Improvements by systems marked by \(*\) have a \(95\%\) statistical significance from the _NN-based direct HMM (Viterbi)_ system, whereas † denotes the \(95\%\) statistical significant improvements with respect to the _attention-based system_ in rescoring. \({}^{1}\) was used in reranking the \(n\)-best lists, while \({}^{2}\) denotes the stand-alone attention-based decoder.

 



## References

* Alkhouli et al. (2016) Tamer Alkhouli, Gabriel Bretschner, Jan-Thorsten Peter, Mohammed Hethnawi, Andreas Guta, and Hermann Ney. 2016. Alignment-Based Neural Machine Translation. In _Proceedings of the ACL 2016 First Conference on Machine Translation (WMT 2016)_. Berlin, Germany, pages 54-65.
* Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. In _Proceedings of the 3rd International Conference on Learning Representations_. San Diego, CA, USA.
* Berg-Kirkpatrick et al. (2010) Taylor Berg-Kirkpatrick, Alexandre Bouchard-Cote, John DeNero, and Dan Klein. 2010. Painless Unsupervised Learning with Features. In _Proceedings of the 2010 Annual Conference of the North American Chapter of the ACL_. Los Angeles, CA, USA, pages 582-590.
* Blunsom and Cohn (2006) Phil Blunsom and Trevor Cohn. 2006. Discriminative Word Alignment with Conditional Random Fields. In _Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the ACL_. Sydney, Australia, pages 65-72.
* Brown et al. (1993) Peter F. Brown, Vincent J. Della Pietra, Stephen A. Della Pietra, and Robert L. Mercer. 1993. The Mathematics of Statistical Machine Translation: Parameter Estimation. _Computational Linguistics_ 19(2):263-311.
* Chen et al. (2011) Boxing Chen, Roland Kuhn, George Foster, and Howard Johnson. 2011. Unpacking and Transforming Feature Functions: New Ways to Smooth Phrase Tables. In _Proceedings of MT Summit XIII_. Xiamen, China, pages 269-275.
* Clark et al. (2011) Jonathan H. Clark, Chris Dyer, Alon Lavie, and Noah A. Smith. 2011. Better Hypothesis Testing for Statistical Machine Translation: Controlling for Optimizer Instability. In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics_. Portland, OR, USA, pages 176-181.
* Devlin et al. (2014) Jacob Devlin, Rabih Zbib, Zhongqiang Huang, Thomas Lamar, Richard Schwartz, and John Makhoul. 2014. Fast and robust neural network joint models for statistical machine translation. In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics_. Baltimore, MD, USA, pages 1370-1380.
* Dyer et al. (2011) Chris Dyer, Jonathan Clark, Alon Lavie, and Noah A. Smith. 2011. Unsupervised Word Alignment with Arbitrary Features. In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics_. Portland, OR, USA, pages 409-419.
* Galley and Manning (2008) Michel Galley and Christopher D. Manning. 2008. A simple and effective hierarchical phrase reordering model. In _Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing_. Honolulu, HI, USA, pages 848-856.
* Goodman (2001) Joshua Goodman. 2001. Classes for fast maximum entropy training. In _Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing_. Salt Lake City, UT, USA, pages 561-564.
* Graves et al. (2006) Alex Graves, Santiago Fernandez, Faustino Gomez, and Jurgen Schmidhuber. 2006. Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks. In _Proceedings of the 23rd International Conference on Machine Learning_. Pittsburgh, PA, USA, pages 369-376.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long Short-Term Memory. _Neural Computation_ 9(8):1735-1780.
* Luong et al. (2015) Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In _Conference on Empirical Methods in Natural Language Processing_. Lisbon, Portugal, pages 1412-1421.
* Morin and Bengio (2005) Frederic Morin and Yoshua Bengio. 2005. Hierarchical probabilistic neural network language model. In _Proceedings of the international workshop on artificial intelligence and statistics_. Barbados, pages 246-252.
* Och (2003) Franz Josef Och. 2003. Minimum Error Rate Training in Statistical Machine Translation. In _Proceedings of the 41st Annual Meeting on Association for Computational Linguistics_. Sapporo, Japan, pages 160-167.
* Och and Ney (2003) Franz Josef Och and Hermann Ney. 2003. A Systematic Comparison of Various Statistical Alignment Models. _Computational Linguistics_ 29:19-51.
* Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A Method for Automatic Evaluation of Machine Translation. In _Proceedings of the 40th Annual Meeting on Association for Computational Linguistics_. Philadelphia, PA, USA, pages 311-318.
* Peter et al. (2016) Jan-Thorsten Peter, Andreas Guta, Nick Rossenbach, Miguel Graca, and Hermann Ney. 2016. The RWTH Aachen Machine Translation System for IWSLT 2016. In _International Workshop on Spoken Language Translation_. Seattle, WA, USA.
* Peter et al. (2015) Jan-Thorsten Peter, Farzad Toutounchi, Joern Wuebker, and Hermann Ney. 2015. The RWTH Aachen German-English Machine Translation System for WMT 2015. In _EMNLP 2015 Tenth Workshop on Statistical Machine Translation_. Lisbon, Portugal, pages 158-163.
* Schwenk (2012) Holger Schwenk. 2012. Continuous Space Translation Models for Phrase-Based Statistical Machine Translation. In _Proceedings of the 24th International Conference on Computational Linguistics_. Mumbai, India, pages 1071-1080.



Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural Machine Translation of Rare Words with Subword Units. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics_. Berlin, Germany, pages 1715-1725.
* Snover et al. (2006) Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Miccilla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In _Proceedings of the Conference of the Association for Machine Translation in the Americas_. Cambridge, MA, USA, pages 223-231.
* Sundermeyer et al. (2014) Martin Sundermeyer, Tamer Alkhouli, Joern Wuebker, and Hermann Ney. 2014. Translation Modeling with Bidirectional Recurrent Neural Networks. In _Conference on Empirical Methods in Natural Language Processing_. Doha, Qatar, pages 14-25.
* Tran et al. (2016) Ke Tran, Yonatan Bisk, Ashish Vaswani, Daniel Marcu, and Kevin Knight. 2016. Unsupervised Neural Hidden Markov Models. In _Proceedings of the Workshop on Structured Prediction for NLP_. Austin, TX, USA, pages 63-71.
* Vilar et al. (2010) David Vilar, Daniel Stein, Matthias Huck, and Hermann Ney. 2010. Jane: Open Source Hierarchical Translation, Extended with Reordering and Lexicon Models. In _ACL 2010 Joint Fifth Workshop on Statistical Machine Translation and Metrics MATR_. Uppsala, Sweden, pages 262-270.
* Volume 2_. Copenhagen, Denmark, COLING '96, pages 836-841.
* Wuebker et al. (2012) Joern Wuebker, Matthias Huck, Stephan Peitz, Malte Nuhn, Markus Freitag, Jan-Thorsten Peter, Saab Mansour, and Hermann Ney. 2012. Jane 2: Open Source Phrase-based and Hierarchical Statistical Machine Translation. In _International Conference on Computational Linguistics_. Mumbai, India, pages 483-491.
* Wuebker et al. (2013)Joern Wuebker, Stephan Peitz, Felix Rietig, and Hermann Ney. 2013. Improving Statistical Machine Translation with Word Class Models. In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_. Seattle, WA, USA, pages 1377-1381.
* Yang et al. (2013) Nan Yang, Shujie Liu, Mu Li, Ming Zhou, and Nenghai Yu. 2013. Word Alignment Modeling with Context Dependent Deep Neural Network. In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics_. Sofia, Bulgaria, pages 166-175.
* Yu et al. (2017) Lei Yu, Phil Blunsom, Chris Dyer, Edward Grefenstette, and Tomas Kocisky. 2017. The Neural Noisy Channel. In _Proceedings of the 5th International Conference on Learning Representations_. Toulon, France.