SlowGuess/ACL_2020_OCR huggingface dataset:
Features. NMT features use the parallel data to train the baseline NMT models. The new-domain-discerning NMT feature  $q_{N}$  uses WMT10-11 (5500 pairs) as in-domain data  $\widehat{D}_N$ . The TED NMT feature  $q_{T}$  uses the TED subtitle training data (22k pairs) as in-domain data  $\widehat{D}_{T}$ . NLM features use the English half of parallel data to train the baseline NLMs. The news-domain-discerning NLM feature  $d_{N}$  uses the 28 million English sentences from WMT14. The TED subtitle NLM feature  $d_{T}$  uses the English side of IWSLT15 in-domain parallel training data. The training of the cross-lingual embedding model follows Yang et al. (2019) with a 3-layer transformer (Vaswani et al., 2017) (more details in Appendix A). For the BERT feature, we sample positive pairs from the same data to train the cross-lingual embedding model. The negatives are generated using the cross-lingual embedding model, via 10-nearest neighbor retrieval in the embedding space, excluding the true translation. We pick the nearest neighbor to form a hard negative pair with the English sentence, and a random neighbor to form another negative pair. We sample 600k positive pairs and produce 1.8M pairs in total.
Model. We use LSTM NMT (Wu et al., 2016) as our models, but with the Adam optimizer (Kingma and Ba, 2015). The batch size is  $10k$  averaged over 8 length-bucket (with synchronous training). NLM/NMT features uses 512 dimensions by 3 layers-NLM shares the same architecture as NMT by using dummy source sentences (Sennrich et al., 2016). The final models are of 1024 dimensions by 8 layers, trained for  $55k$  max steps. Training on WMT data uses a dropout probability of 0.2. Transformer results are in Appendix B.
Curriculum optimization. In Eq. 5 (Section 4.5), we launch 30 trials (candidate curricula). BayesOpt spends 25 trials in exploration
10Randomly sampled from www.epo.org
$^{11}$ Signature: BLEU+case.mixed+numrefs.1+ smooth.exp+tok.13a+version.1.4.2
$^\mathrm{7E.g.}$  ,https://github.com/tobegit3hub/ advisor 8When the training data is small, we can, in practice, let a model warm up before applying the schedule. 9https://paracrawl.eu
Curriculum	N14	T15	PA	D15	Avg
P1: B	33.4	35.7	29.8	30.4	32.3
P2: ˆC6-feats	37.0	38.1	48.3	35.7	39.8
W1: B	38.039.2	37.9	45.6	34.5	39.0
(Wu et al., 2016)	39.2	-	-	-	-
W2: ˆC6-feats	39.3	38.8	46.1	36.1	40.1
and the last 5 in exploitation. Each trial trains for  $2k$  steps $^{12}$  by fine-tuning a warmed-up model with the candidate curriculum. The curriculum decays  $(\rho(t))$  from  $100\%$  and plateaus at  $20\%$  at step  $2k$ . We simply and heuristically set a range of  $[0.0, 1.0]$  for all feature weights. We don't normalize feature values when weighting them.
We evaluate if the multi-domain curriculum meets requirements (i) and (ii) in Section 4.1.
5.2.1 Compared to no curriculum
We compare:
-  $\mathcal{B}$ : baseline that does not use curriculum learning.
-  $\widehat{\mathcal{C}}_{6}$ -feats: multi-domain curriculum with 6 features,  $d_N$ ,  $d_T$ ,  $q_N$ ,  $q_T$ , BERT, emb, weights learned by BayesOpt.
Table 3 shows  $\widehat{\mathcal{C}}_{6}$  -feats improves  $\mathcal{B}$  on all testsets, especially on noisy ParaCrawlrequirement (i) is met. It is important to note that our WMT baseline (W1) matches Wu et al. (2016) on N14, as shown by re-computed tokenized BLEU (italics).
5.2.2 Compared to single-domain curricula
We examine the following individual curricula, by training NMT models with each, respectively:
-  $\mathcal{C}_{d_N}$ , uses news NLM feature  $d_N$  (Eq. 9).
-  ${\mathcal{C}}_{{d}_{T}}$  ,uses TED subtitle NLM feature  ${d}_{T}$  .
-  $\mathcal{C}_{q_N}$ , uses news NMT feature  $q_{N}$  (Eq. 6).
-  $\mathcal{C}_{q_T}$ , uses TED NMT feature  $q_T$ .
-  $\mathcal{C}_{\mathrm{BERT}}$ , uses BERT quality feature.
-  $\mathcal{C}_{\mathrm{emb}}$ , uses cross-lingual embedding feature.
Table 3: English  $\rightarrow$  French multi-domain curriculum improves no-curriculum baseline (B) over all testsets. Avg: averaged score per row, for ease of reading. P: ParaCrawl data. W: WMT14 training data. BLEUs in italics are tokenized BLEU. Other scores are de-tokenized SacreBLEU.
Curriculum	N14	T15	PA	D15	Avg
P1: B	33.4	35.7	29.8	30.4	32.3
P3: CdN	34.7	36.2	32.6	32.6	34.0
P4: CdT	34.8	36.3	30.1	32.4	33.4
P5.1: CBERT	36.8	37.3	47.9	35.0	39.3
P5.2: Cemb	36.9	37.7	46.0	35.2	39.0
P6: CqN	36.8	37.1	47.7	34.9	39.1
P7: CqT	35.6	38.3	46.6	34.9	38.9
P2: C6-feats	37.0	38.1	48.3	35.7	39.8
P2-P*	+0.1	-0.2	+0.4	+0.5	+0.2
W1: B	38.0	37.9	45.6	34.5	39.0
W3: CdN	38.3	38.1	39.1	35.1	37.7
W4: CdT	38.1	38.4	43.0	36.1	38.9
W5.1: CBERT	38.5	37.8	45.9	35.9	39.5
W5.2: Cemb	38.5	37.8	45.8	35.9	39.5
W6: CqN	37.8	38.0	45.9	35.3	39.3
W7: CqT	38.5	38.8	45.0	36.1	39.6
W2: C6-feats	39.3	38.8	46.1	36.1	40.1
W2-W*	+0.8	0.0	+0.2	0.0	+0.3
Table 4: English  $\rightarrow$  French multi-domain curriculum (P2, W2) vs. single-domain curricula (P3-7, W3-7). Frame boxes mark best per-testset BLEU  $(\mathrm{W}^{*},\mathrm{P}^{*})$  over all single-domain curricula. Bold color denotes multi-domain curriculum has best BLEU  $(\mathrm{W2 - W^{*}}\geq 0)$
In Table 4, frame boxes mark the best BLEUs  $(\mathbb{P}^*$  or  $\mathrm{W^{*}}$  ) per column, across P3-P7 or W3-W7. The last column shows averaged BLEU over all testsets. Bold font indicates  $C_{6}$  -feats matches or improves  $\mathbf{W}^{*}$  .As shown,  $C_{6}$  -feats matches or slightly outperforms the per-domain curricula across testsets. Therefore,  $\widehat{C}_{6}$  -feats meets requirement (ii).
5.3.1 Features
Strengths and weaknesses of a feature. Table 4 also reveals the relative strengths and weaknesses of each type of features. The peak BLEU (in a frame box) on each testset is achieved by one of  $\mathcal{C}_{\mathrm{BERT / emb}}$ ,  $\mathcal{C}_{q_N}$  and  $\mathcal{C}_{q_T}$ , less by NLM features  $d_N$ ,  $d_T$ . This contrast seems bigger on the noisy ParaCrawl, but the NLM features do bring gains over  $\mathcal{B}$ . Overall,  $\mathcal{C}_{\mathrm{BERT / emb}}$  (P5, W5) perform well, attributed to their denoising power, but lose to the NMT features (P7, W7) on T15, due to lack of explicit capturing of domain. The NMT features seem to subtly compensate in domains, and the domain features in denoising, but working with other features improves the model.
BERT and emb features. Both BERT and emb use knowledge external to the experiment setup. For a fair comparison to baselines and a better understanding of them, we drop them by building
$^{12}2\mathrm{k}$  is empirically chosen to be practical. We use a number of fine-tuning trials in Eq. 5. NMT training is expensive so we don't want a trial to tune for many steps. NMT is very adaptive on domain data, so each trial does not need many steps. We find no significant difference among  $1\mathrm{k}, 2\mathrm{k}, 6\mathrm{k}$ .
7717
Curriculum	N14	T15	PA	D15	Avg
P2: C6-feats	37.0	38.1	48.3	35.7	39.8
P8: C4-feats	36.6	38.1	46.7	35.5	39.2
W2: C6-feats	39.3	38.8	46.1	36.1	40.1
W8: C4-feats	38.9	38.9	46.5	36.1	40.1
Figure 3: BayesOpt learns to weight features adaptively on ParaCrawl and WMT, respectively.
-  $\widehat{\mathcal{C}}_{4}$ -feats, multi-domain curriculum that excludes BERT and emb and uses 4 features.
Table 5 shows BERT and emb features in  $\widehat{\mathcal{C}}_{6}$  -feats improve  $\widehat{\mathcal{C}}_{4}$  -feats with ParaCrawl, adding to the intu-tuition that they have a denoising effect.
Learned feature weights. Figure 3 shows BayesOpt learns to weight features adaptively in  $\widehat{\mathcal{C}}_{6\text{-feats}}$  on ParaCrawl (grey) and WMT (white), respectively. ParaCrawl is very noisy thus noise non-descerning features  $d_N$  and  $d_T$  do not have a chance to help, but their weights become stronger on the cleaner WMT training data. It is surprising that BERT feature is still useful to the WMT training. We hypothesize this may suggest BERT feature have additional strength to just denoising, or that data noise could be subtle and exist in cleaner data.
5.3.2 BayesOpt vs. random search
We compare BayesOpt (BO) and Random Search (RS) (Bergstra and Bengio, 2012) to solve Eq. 5, as well as uniform weighting (Uniform). In Table 6, all improve baselines, especially on ParaCrawl (P). RS does surprisingly well on ParaCrawl, but BayesOpt appears better overall.[13]
5.3.3 Mixing validation sets
Eq. 5 evaluates  $\mathcal{P}$  using the concatenated validation set (Section 4.4). Table 7 shows that the news-vs-TED mixing ratios can affect the per-domain
Table 5: BERT and emb features positively contribute to  $\widehat{\mathcal{C}}_{6\text{-feats}}$  on ParaCrawl (P).
Curriculum	N14	T15	PA	D15	Avg
P1: B	33.4	35.7	29.8	30.4	32.3
P2: \(\widehat{C}_{6\text{-feats}}\) (BO)	37.0	38.1	48.3	35.7	39.8
P9: \(\widehat{C}_{6\text{-feats}}\) (RS)	36.7	38.4	48.0	35.5	39.7
P10: \(\widehat{C}_{6\text{-feats}}\) (Uniform)	35.4	36.9	48.3	34.1	38.7
W1: B	38.0	37.9	45.6	34.5	39.0
W2: \(\widehat{C}_{6\text{-feats}}\) (BO)	39.3	38.8	46.1	36.1	40.1
W9: \(\widehat{C}_{6\text{-feats}}\) (RS)	39.0	38.2	43.7	36.4	39.3
W10: \(\widehat{C}_{6\text{-feats}}\) (Uniform)	38.8	39.1	43.0	36.0	39.2
Table 6: On average, BayesOpt (BO) performs better than Random Search (RS) and uniform weighting (Uniform), for learning feature weights of a multi-domain curriculum.
Figure 4: The multi-domain curriculum dynamically balances multi-domain-relevant and noise-reduced data, as validated by human ratings.
BLEUs. For example, on ParaCrawl, when news sentences are absent from the validation set, N14 drops by 0.7 BLEU (P8 vs. P13). We use the four feats as in  $\widehat{C}_{4\text{-feats}}$  in this examination.
5.3.4 Dynamic data balancing
We simulate dynamic data selection with a random sample of 2000 pairs from the WMT data and annotate each pair by human raters with 0 (nonsense) - 4 (perfect) quality scale (following Wang et al. (2018b)). We sort the pairs by  $f(x, y)$  (Eq. 4). A threshold selects a subset of pairs, for which we average the respective NMT feature values as the domain relevance. Figure 4 shows that the multi-domain curriculum ( $\widehat{\mathcal{C}}_{6\text{-feats}}$ ) learns to dynamically increase quality and multi-domain relevance. Therefore, our idea (Section 4.1) works as intended. Furthermore, training seems to gradually increase quality or domain in different speeds, determined by Eq. 5.
5.3.5 Weighting loss vs. curriculum
With the learned weights, we compute a weight for each example to sort data to form a curriculum. Alternatively, we could weight the cross-entropy loss for that sentence during training (Wang et al., 2017; Chen et al., 2017). Table 8 shows that curriculum yields improvements over weighing per
$^{13}$ RS uses 30 trials, as BO (Section 5.1), so the results show their comparison given the same number of trials.
7718
Mixing Ratio	N14	T15	PA	D15	Avg
P11:	1.0:0.0	36.3	37.8	47.3	35.3	39.2
P12:	0.8:0.2	36.4	38.2	47.7	35.4	39.4
P8:	0.5:0.5	36.6	38.1	46.7	35.5	39.2
P13:	0.0:1.0	35.9	38.1	47.0	35.2	39.1
W11:	1.0:0.0	39.1	38.6	46.4	36.0	40.0
W12:	0.8:0.2	39.0	38.7	46.3	35.7	39.9
W8:	0.5:0.5	38.9	38.9	46.5	36.1	40.1
W13:	0.0:1.0	39.1	38.6	46.4	36.0	40.0
Table 7: Guiding multi-domain curriculum learning by mixing validation sets. Experiments use 4 features as in  $\widehat{C}_{4\text{-feats}}$
Model	N14	T15	PA	D15	Avg
P8: Curriculum	36.6	38.1	46.7	35.5	39.2
P14: Weight Loss	35.3	37.8	39.3	32.6	36.3
W8: Curriculum	38.9	38.9	46.5	36.1	40.1
W14: Weight Loss	38.6	37.6	45.7	35.3	39.3
Table 8: Forming a curriculum with learned weights performs better than weighting instance loss in training. Experiments use 4 features (as in  $\hat{C}_{4\text{-feats}}$ ).
sentence loss, in particular on noisy training data, confirming previous findings (van der Wees et al., 2017).
5.3.6 In-domain fine-tuning
$\mathcal{C}_{q_N}$  and  $\mathcal{C}_{q_T}$  each use a small in-domain parallel dataset, but we can simply fine-tune the final models on either dataset  $(+\mathrm{N}, + \mathrm{T})$  or their concatenation  $(+N + T)$ . Table 9 shows that  $\widehat{\mathcal{C}}_{6\text{-feats}}$  can be further improved by in-domain fine-tuning<sup>14</sup> and that both  $\widehat{\mathcal{C}}_{6\text{-feats}}$  and its fine-tuning still improve the fine-tuned baselines, in particular on ParaCrawl.
5.4 Discussion: Feature Dependency
One potential issue with using multiple per-domain features  $(q_{Z}(x,y)$ 's in Eq. 6) is scores are not shared across domains and linear weighting may not capture feature dependency. For example, we need two NMT features if there are two domains. We replace the two NMT features,  $q_{N}$  and  $q_{T}$ , in  $\widehat{\mathcal{C}}_{4\text{-feats}}$  with a single two-domain feature  $q_{Z = \{N,T\}}$  (Eq. 8), but with the two corresponding NLM features unchanged (so the new experiment has 3 features). Table 10 shows multi-domain feature contributes slightly better than linear combination of per-domain features (P19 vs. P8). The per-domain features, however, have the advantage of efficient feature weighting. In case of many features, learning to compress them seems to be an interesting future investigation.
Model	N14	T15	PA	D15	Avg
P15: B+N	35.8	37.1	41.2	32.8	36.7
P16: B+T	35.8	38.7	45.4	34.6	38.6
P17: B+N+T	35.9	38.7	44.8	34.4	38.4
P2: C6-feats (BO)	37.0	38.1	48.3	35.7	39.8
P18: C6-feats +N+T	38.1	39.7	48.6	36.6	40.8
W15: B+N	38.7	37.4	46.4	34.6	39.3
W16: B+T	36.8	38.9	44.8	36.5	39.3
W17: B+N+T	38.6	39.1	46.1	35.8	39.9
W2: C6-feats (BO)	39.3	38.8	46.1	36.1	40.1
W18: C6-feats +N+T	39.3	39.8	46.0	36.6	40.4
Table 9: The multi-domain curricula still bring improvements, even after models are fine-tuned on in-domain parallel data. +N: fine-tune on news parallel data  $\widehat{D}_N$  (Section 5.1); +T: fine-tune on TED parallel data  $\widehat{D}_T$ ; +N+T on concatenation.
Model	N14	T15	PA	D15	Avg
P8: per-dom.	36.6	38.1	46.7	35.5	39.2
P19: multi-dom.	36.6	38.6	46.8	35.9	39.5
Table 10: Multi-domain/task feature (Eq. 8) seems to contribute slightly better than linear combination of multiple per-domain features (Eq. 6).
Existing curriculum learning research in NMT focuses on a single domain. We present a multidomain curriculum learning method. We carefully introduce instance-level features and learn a training curriculum to gradually concentrate on multi-domain relevant and noise-reduced data batches. End-to-end experiments and ablation studies on large datasets at different noise levels show that the multi-domain curriculum simultaneously reaches or outperforms the individual performance and brings solid gains over no-curriculum training, on in-domain and out-of-domain testsets.
The authors would like to thank David Grangier for Eq. 7 and derivation, the three anonymous reviewers for their insightful reviews, Yuan Cao for his technical suggestions, Jason Smith, Markus Freitag, Pidong Wang and Reid Pryzant for comments on an earlier draft, Quoc V. Le for suggestions in a related thread.
14We fine-tune with SGD for 20k steps, with batch size 16, learning rate 0.0001.
7719
Milton Abramowitz and Irene A. Stegun. 1964. Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables, ninth dover printing, tenth gp0 printing edition. Dover, New York.
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. 2019. Gradient based sample selection for online continual learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 11816-11825. Curran Associates, Inc.
Amittai Axelrod, Xiaodong He, and Jianfeng Gao. 2011. Domain adaptation via pseudo in-domain data selection. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 355-362.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In Proceedings of the 26th International Conference on Machine Learning, pages 86-96, Montreal, Canada.
James Bergstra and Yoshua Bengio. 2012. Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13:281-305.
Denny Britz, Quoc Le, and Reid Pryzant. 2017. Effective domain mixing for neural machine translation. In Proceedings of the Second Conference on Machine Translation, pages 118-126. Association for Computational Linguistics.
Boxing Chen, Colin Cherry, George Foster, and Samuel Larkin. 2017. Cost weighting for neural machine translation domain adaptation. In Proceedings of the First Workshop on Neural Machine Translation, pages 40-46, Vancouver. Association for Computational Linguistics.
Boxing Chen and Fei Huang. 2016. Semi-supervised convolutional networks for translation adaptation with tiny amount of in-domain data. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning (CoNLL), pages 314-323.
Boxing Chen, Roland Kuhn, George Foster, Colin Cherry, and Fei Huang. 2016. Bilingual methods for adaptive training data selection for machine translation. In AMTA.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805.
Kevin Duh. 2018. Multi-objective hyperparameter search for fast and accurate neural machine translation - progress report. Technical report, Johns Hopkins University.
M. Amin Farajian, Marco Turchi, Matteo Negri, and Marcello Federico. 2017. Multi-domain neural machine translation through unsupervised adaptation. In Proceedings of the Second Conference on Machine Translation, pages 127-137, Copenhagen, Denmark. Association for Computational Linguistics.
Markus Freitag and Yaser Al-Onaizan. 2016. Fast domain adaptation for neural machine translation. CoRR, abs/1612.06897.
Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John Elliot Karro, and D. Sculley, editors. 2017. Google Vizier: A Service for Black-Box Optimization.
IJ Goodfellow, M Mirza, X Da, Aaron Courville, and Yoshua Bengio. 2014. An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks. TR arXiv:1312.6211v2.
David Grangier. 2019. Distribution matched contrastive data selection. In Google internal report.
Alex Graves, Marc G. Bellemare, Jacob Menick, Rémi Munos, and Koray Kavukcuoglu. 2017. Automated curriculum learning for neural networks. CoRR, abs/1704.03003.
Marcin Junczys-Dowmunt. 2018. Dual conditional cross-entropy filtering of noisy parallel corpora. In Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Task Papers, pages 901–908, Belgium, Brussels. Association for Computational Linguistics.
Huda Khayrallah, Brian Thompson, Kevin Duh, and Philipp Koehn. 2018. Regularized training objective for continued training for domain adaptation in neural machine translation. In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pages 36-44, Melbourne, Australia. Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Tom Kocmi and Ondrej Bojar. 2017. Curriculum learning and minibatch bucketing in neural machine translation. In Proceedings of the International Conference on Recent Advances in Natural Language Processing, RANLP 2017, pages 379-386. INCOMA Ltd.
Philipp Koehn, Francisco Guzman, Vishrav Chaudhary, and Juan Pino. 2019. Findings of the wmt 2019 shared task on parallel corpus filtering for low-resource conditions. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational Linguistics.
Philipp Koehn, Huda Khayrallah, Kenneth Heafield, and Mikel L. Forcada. 2018. Findings of the wmt 2018 shared task on parallel corpus filtering. In Proceedings of the Third Conference on Machine Translation, Belgium, Brussels. Association for Computational Linguistics.
Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66-75. Association for Computational Linguistics.
Gaurav Kumar, George Foster, Colin Cherry, and Maxim Krikun. 2019. Reinforcement learning based curriculum optimization for neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2054-2061, Minneapolis, Minnesota. Association for Computational Linguistics.
Robert C. Moore and William Lewis. 2010. Intelligent selection of language model training data. In Proceedings of the ACL 2010 Conference, pages 220-224.
Emmanouil Antonios Platanios, Otilia Stretcu, Graham Neubig, Barnabás Póczos, and Tom M. Mitchell. 2019. Competence-based curriculum learning for neural machine translation. CoRR, abs/1903.09848.
Matt Post. 2018. A call for clarity in reporting bleu scores. Computing Research Repository, arXiv:1804.08771v1. Version 2.
Lutz Prechelt. 1997. Automatic early stopping using cross validation: Quantifying the criteria. Neural Networks, 11:761-767.
Sebastian Ruder and Barbara Plank. 2017. Learning to select data for transfer learning with bayesian optimization. CoRR, abs/1707.05246.
Hassan Sajjad, Nadir Durrani, Fahim Dalvi, Yonatan Belinkov, and Stephan Vogel. 2017. Neural machine translation training in a multi-domain scenario. arXiv preprint arXiv:1708.08712v2.
Danielle Saunders, Felix Stahlberg, Adrià de Gispert, and Bill Byrne. 2019. Domain adaptive inference for neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 222-228, Florence, Italy. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving Neural Machine Translation Models with Monolingual Data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86-96, Berlin, Germany. Association for Computational Linguistics.