Revisiting Low-Resource Neural Machine Translation: A Case Study	It has been shown that the performance of neural machine translation (NMT) drops starkly in low-resource conditions, underperforming phrase-based statistical machine translation (PBSMT) and requiring large amounts of auxiliary data to achieve competitive results. In this paper, we re-assess the validity of these results, arguing that they are the result of lack of system adaptation to low-resource settings. We discuss some pitfalls to be aware of when training low-resource NMT systems, and recent techniques that have shown to be especially helpful in low-resource settings, resulting in a set of best practices for low-resource NMT. In our experiments on German--English with different amounts of IWSLT14 training data, we show that, without the use of any auxiliary monolingual or multilingual data, an optimized NMT system can outperform PBSMT with far less data than previously claimed. We also apply these techniques to a low-resource Korean-English dataset, surpassing previously reported results by 4 BLEU.

Introduction	While neural machine translation (NMT) has achieved impressive performance in high-resource data conditions, becoming dominant in the field  ,  ,  , recent research has argued that these models are highly data-inefficient, and underperform phrase-based statistical machine translation (PBSMT) or unsupervised methods in low-data conditions  ,  . In this paper, we re-assess the validity of these results, arguing that they are the result of lack of system adaptation to low-resource settings. Our main contributions are as follows:

Low-Resource Translation Quality Compared Across Systems	Figure FIGREF4 reproduces a plot by  which shows that their NMT system only outperforms their PBSMT system when more than 100 million words (approx. 5 million sentences) of parallel training data are available. Results shown by  are similar, showing that unsupervised NMT outperforms supervised systems if few parallel resources are available. In both papers, NMT systems are trained with hyperparameters that are typical for high-resource settings, and the authors did not tune hyperparameters, or change network architectures, to optimize NMT for low-resource conditions.

Sample Translations	Table TABREF24 shows some sample translations that represent typical errors of our PBSMT and NMT systems, trained with ultra-low (100k words) and low (3.2M words) amounts of data. For unknown words such as blutbefleckten (`bloodstained') or Spaniern (`Spaniards', `Spanish'), PBSMT systems default to copying, while NMT systems produce translations on a subword-level, with varying success (blue-flect, bleed; spaniers, Spanians). NMT systems learn some syntactic disambiguation even with very little data, for example the translation of das and die as relative pronouns ('that', 'which', 'who'), while PBSMT produces less grammatical translation. On the flip side, the ultra low-resource NMT system ignores some unknown words in favour of a more-or-less fluent, but semantically inadequate translation: erobert ('conquered') is translated into doing, and richtig aufgezeichnet ('registered correctly', `recorded correctly') into really the first thing.

Hyperparameters	Table TABREF23 lists hyperparameters used for the different experiments in the ablation study (Table 2). Hyperparameters were kept constant across different data settings, except for the validation interval and subword vocabulary size (see Table 1).

For example, the effectiveness of unsupervised methods is impaired when languages are morphologically different, or when training domains do not match   More broadly, this line of research still accepts the premise that NMT models are data-inefficient and require large amounts of auxiliary data to train. In this work, we want to re-visit this point, and will focus on techniques to make more efficient use of small amounts of parallel training data. Low-resource NMT without auxiliary data has received less attention; work in this direction includes  ,  .

Mainstream Improvements	We consider the hyperparameters used by  to be our baseline. This baseline does not make use of various advances in NMT architectures and training tricks. In contrast to the baseline, we use a BiDeep RNN architecture  , label smoothing  , dropout  , word dropout  , layer normalization  and tied embeddings  .

Language Representation	Subword representations such as BPE  have become a popular choice to achieve open-vocabulary translation. BPE has one hyperparameter, the number of merge operations, which determines the size of the final vocabulary. For high-resource settings, the effect of vocabulary size on translation quality is relatively small;  report mixed results when comparing vocabularies of 30k and 90k subwords. In low-resource settings, large vocabularies result in low-frequency (sub)words being represented as atomic units at training time, and the ability to learn good high-dimensional representations of these is doubtful.  propose a minimum frequency threshold for subword units, and splitting any less frequent subword into smaller units or characters. We expect that such a threshold reduces the need to carefully tune the vocabulary size to the dataset, leading to more aggressive segmentation on smaller datasets.

Hyperparameter Tuning	Due to long training times, hyperparameters are hard to optimize by grid search, and are often re-used across experiments. However, best practices differ between high-resource and low-resource settings. While the trend in high-resource settings is towards using larger and deeper models,  use smaller and fewer layers for smaller datasets. Previous work has argued for larger batch sizes in NMT  ,  , but we find that using smaller batches is beneficial in low-resource settings. More aggressive dropout, including dropping whole words at random  , is also likely to be more important. We report results on a narrow hyperparameter search guided by previous work and our own intuition.

Lexical Model	Finally, we implement and test the lexical model by  , which has been shown to be beneficial in low-data conditions. The core idea is to train a simple feed-forward network, the lexical model, jointly with the original attentional NMT model. The input of the lexical model at time step INLINEFORM0 is the weighted average of source embeddings INLINEFORM1 (the attention weights INLINEFORM2 are shared with the main model). After a feedforward layer (with skip connection), the lexical model's output INLINEFORM3 is combined with the original model's hidden state INLINEFORM4 before softmax computation. INLINEFORM5   Our implementation adds dropout and layer normalization to the lexical model.

Data and Preprocessing	We use the TED data from the IWSLT 2014 German INLINEFORM0 English shared translation task  . We use the same data cleanup and train/dev split as  , resulting in 159000 parallel sentences of training data, and 7584 for development. As a second language pair, we evaluate our systems on a Korean–English dataset with around 90000 parallel sentences of training data, 1000 for development, and 2000 for testing. For both PBSMT and NMT, we apply the same tokenization and truecasing using Moses scripts. For NMT, we also learn BPE subword segmentation with 30000 merge operations, shared between German and English, and independently for Korean INLINEFORM0 English. To simulate different amounts of training resources, we randomly subsample the IWSLT training corpus 5 times, discarding half of the data at each step. Truecaser and BPE segmentation are learned on the full training corpus; as one of our experiments, we set the frequency threshold for subword units to 10 in each subcorpus (see SECREF7 ).

Table TABREF14 shows statistics for each subcorpus, including the subword vocabulary. Translation outputs are detruecased, detokenized, and compared against the reference with cased BLEU using sacreBLEU  ,  . Like  , we report BLEU on the concatenated dev sets for IWSLT 2014 (tst2010, tst2011, tst2012, dev2010, dev2012).

PBSMT Baseline	We use Moses  to train a PBSMT system. We use MGIZA  to train word alignments, and lmplz  for a 5-gram LM. Feature weights are optimized on the dev set to maximize BLEU with batch MIRA  – we perform multiple runs where indicated. Unlike  , we do not use extra data for the LM. Both PBSMT and NMT can benefit from monolingual data, so the availability of monolingual data is no longer an exclusive advantage of PBSMT (see SECREF5 ).

NMT Systems	We train neural systems with Nematus  . Our baseline mostly follows the settings in  ; we use adam  and perform early stopping based on dev set BLEU. We express our batch size in number of tokens, and set it to 4000 in the baseline (comparable to a batch size of 80 sentences used in previous work). We subsequently add the methods described in section SECREF3 , namely the bideep RNN, label smoothing, dropout, tied embeddings, layer normalization, changes to the BPE vocabulary size, batch size, model depth, regularization parameters and learning rate. Detailed hyperparameters are reported in Appendix SECREF7 .

Results	Table TABREF18 shows the effect of adding different methods to the baseline NMT system, on the ultra-low data condition (100k words of training data) and the full IWSLT 14 training corpus (3.2M words). Our "mainstream improvements" add around 6–7 BLEU in both data conditions. In the ultra-low data condition, reducing the BPE vocabulary size is very effective (+4.9 BLEU). Reducing the batch size to 1000 token results in a BLEU gain of 0.3, and the lexical model yields an additional +0.6 BLEU. However, aggressive (word) dropout (+3.4 BLEU) and tuning other hyperparameters (+0.7 BLEU) has a stronger effect than the lexical model, and adding the lexical model (9) on top of the optimized configuration (8) does not improve performance. Together, the adaptations to the ultra-low data setting yield 9.4 BLEU (7.2 INLINEFORM2 16.6).

The model trained on full IWSLT data is less sensitive to our changes (31.9 INLINEFORM3 32.8 BLEU), and optimal hyperparameters differ depending on the data condition. Subsequently, we still apply the hyperparameters that were optimized to the ultra-low data condition (8) to other data conditions, and Korean INLINEFORM4 English, for simplicity. For a comparison with PBSMT, and across different data settings, consider Figure FIGREF19 , which shows the result of PBSMT, our NMT baseline, and our optimized NMT system. Our NMT baseline still performs worse than the PBSMT system for 3.2M words of training data, which is consistent with the results by  . However, our optimized NMT system shows strong improvements, and outperforms the PBSMT system across all data settings. Some sample translations are shown in Appendix SECREF8 . For comparison to previous work, we report lowercased and tokenized results on the full IWSLT 14 training set in Table TABREF20 . Our results far outperform the RNN-based results reported by  , and are on par with the best reported results on this dataset.

Table TABREF21 shows results for Korean INLINEFORM0 English, using the same configurations (1, 2 and 8) as for German–English. Our results confirm that the techniques we apply are successful across datasets, and result in stronger systems than previously reported on this dataset, achieving 10.37 BLEU as compared to 5.97 BLEU reported by gu-EtAl:2018:EMNLP1.

Conclusions	Our results demonstrate that NMT is in fact a suitable choice in low-data settings, and can outperform PBSMT with far less parallel training data than previously claimed. Recently, the main trend in low-resource MT research has been the better exploitation of monolingual and multilingual resources. Our results show that low-resource NMT is very sensitive to hyperparameters such as BPE vocabulary size, word dropout, and others, and by following a set of best practices, we can train competitive NMT systems without relying on auxiliary resources. This has practical relevance for languages where large amounts of monolingual data, or multilingual data involving related languages, are not available. Even though we focused on only using parallel data, our results are also relevant for work on using auxiliary data to improve low-resource MT. Supervised systems serve as an important baseline to judge the effectiveness of semisupervised or unsupervised approaches, and the quality of supervised systems trained on little data can directly impact semi-supervised workflows, for instance for the back-translation of monolingual data.

Acknowledgments	Rico Sennrich has received funding from the Swiss National Science Foundation in the project CoNTra (grant number 105212_169888). Biao Zhang acknowledges the support of the Baidu Scholarship.

Improving Low-Resource Neural Machine Translation	The bulk of research on low-resource NMT has focused on exploiting monolingual data, or parallel data involving other language pairs. Methods to improve NMT with monolingual data range from the integration of a separately trained language model  to the training of parts of the NMT model with additional objectives, including a language modelling objective  ,  ,  , an autoencoding objective  ,  , or a round-trip objective, where the model is trained to predict monolingual (target-side) training data that has been back-translated into the source language  ,  ,  . As an extreme case, models that rely exclusively on monolingual data have been shown to work  ,  ,  ,  . Similarly, parallel data from other language pairs can be used to pre-train the network or jointly learn representations  ,  ,  ,  ,  ,  ,  . While semi-supervised and unsupervised approaches have been shown to be very effective for some language pairs, their effectiveness depends on the availability of large amounts of suitable auxiliary data, and other conditions being met.

Question: what pitfalls are mentioned in the paper?
Answer: highly data-inefficient, underperform phrase-based statistical machine translation