Question: How do they conduct their training step?
Evidence: Neural machine translation (NMT) BIBREF0 is a promising paradigm for extracting translation knowledge from parallel text. NMT systems have achieved competitive accuracy rates under large-data training conditions for language pairs such as English-French. However, neural methods are data-hungry and learn poorly from low-count events. This behavior makes vanilla NMT a poor choice for low-resource languages, where parallel data is scarce. Table 1 shows that for 4 low-resource languages, a standard string-to-tree statistical MT system (SBMT) BIBREF1 , BIBREF2 strongly outperforms NMT, even when NMT uses the state-of-the-art local attention plus feed-input techniques from luong2015effective.
In this paper, we give a method for substantially improving NMT results on these languages. Neural models learn representations of their input that are often useful across tasks. Our key idea is to first train a high-resource language pair, then use the resulting trained network (the parent model) to initialize and constrain training for our low-resource language pair (the child model). We find that we can optimize our results by fixing certain parameters of the parent model, letting the rest be fine-tuned by the child model. We report NMT improvements from transfer learning of 5.6 Bleu on average, and we provide an analysis of why the method works. The final NMT system approaches strong SBMT baselines in in all four language pairs, and exceeds SBMT performance in one of them. Furthermore, we show that NMT is an exceptional re-scorer of `traditional' MT output; even NMT that on its own is worse than SBMT is consistently able to improve upon SBMT system output when incorporated as a re-scoring model.
We start by a brief description of our NMT model in Section "NMT Background" . Section "Transfer Results" gives some background on transfer learning and explains how we use it to improve machine translation performance. Our main experiments translating Hausa, Turkish, Uzbek and Urdu into English with the help of a French-English parent model are presented in Section "Experiments" . Section "Analysis" explores alternatives to our model to enhance understanding. We find that the choice of parent language pair affects performance, and provide an empirical upper bound on transfer performance using an artificial language. We experiment with English-only language models, copy models, and word-sorting models to show that what we transfer goes beyond monolingual information and using a translation model trained on bilingual corpora as a parent is essential. We show the effects of freezing, fine-tuning, and smarter initialization of different components of the attention-based NMT system during transfer. We compare the learning curves of transfer and no-transfer models showing that transfer solves an overfitting problem, not a search problem. We summarize our contributions in Section "Conclusion" .
In this paper, we give a method for substantially improving NMT results on these languages. Neural models learn representations of their input that are often useful across tasks. Our key idea is to first train a high-resource language pair, then use the resulting trained network (the parent model) to initialize and constrain training for our low-resource language pair (the child model).
Answer: They first train a high-resource language pair, then use the resulting trained network (the parent model) to initialize and constrain training for our low-resource language pair (the child model).