Decoupled Vocabulary Learning Enables Zero-Shot Translation from Unseen Languages (2024)

Abstract: Multilingual neural machine translation systems learn to map sentences of different languages into a common representation space. Intuitively, with a growing number of seen languages the encoder sentence representation grows more flexible and easily adaptable to new languages. In this work, we test this hypothesis by zero-shot translating from unseen languages. To deal with unknown vocabularies from unknown languages we propose a setup where we decouple learning of vocabulary and syntax, i.e. for each language we learn word representations in a separate step (using cross-lingual word embeddings), and then train to translate while keeping those word representations frozen. We demonstrate that this setup enables zero-shot translation from entirely unseen languages. Zero-shot translating with a model trained on Germanic and Romance languages we achieve scores of 42.6 BLEU for Portuguese-English and 20.7 BLEU for Russian-English on TED domain. We explore how this zero-shot translation capability develops with varying number of languages seen by the encoder. Lastly, we explore the effectiveness of our decoupled learning strategy for unsupervised machine translation. By exploiting our model’s zero-shot translation capability for iterative back-translation we attain near parity with a supervised setting.

heading: 1 Introduction
text: In order to extend Machine Translation from a hundred languages to the 7,000 languages of the world new methods have to be developed with more economical use of data from individual languages. This requires better use of abstractions and representations of languages to facilitate transfer learning to low-resource languages. In machine translation, it had been attempted by developing linguistic structures explicitly in classical Interlingua systems (Levin et al., 1998) or imposing structure on statistical models (Suhm, 1994; Wang and Waibel, 1998).
However, developing such structures was labor intensive and did not achieve acceptable performance or required domain restrictions. First attempts toward learning latent representations of language (Wang and Waibel, 1991, 1995) still required linguistic structures or specific design. Successful performance on open-domain translation was only feasible later using multilingually trained end-to-end neural translation models (Bahdanau et al., 2015; Johnson et al., 2017; Ha et al., 2016). These models share hidden representations between languages and allow for multilingual translation that benefits low-resource languages by transfer learning from high-resource languages and enable zero-shot translation on new language directions.
The multilingual encoder in such models will ideally learn to map semantically similar sentences onto similar latent representations – even across different languages. Ongoing research in the field of zero-shot translation shows that multilingual NMT models exhibit this property to a certain extent (Duquenne et al., 2023) and that enforcing the similarity between sentence representations across different languages in turn also improves the zeroshot capabilities (Pham et al., 2019; Arivazhagan et al., 2019; Zhang et al., 2020; Liu et al., 2021; Pan et al., 2021). At the same time, cross-lingual transfer learning research shows that massively multilingual systems can be rapidly extended to new languages on very little data (Neubig and Hu, 2018; Artetxe et al., 2020; Garcia et al., 2021; Marchisio et al., 2023; Chen et al., 2023). This suggests that exposing the NMT model to a higher number of languages increases the plasticity of its sentence representation and enables easier adaptation to unseen languages.
When extending a multilingual model by an unseen language, however, naturally the question arises of how to deal with the unknown vocabulary words. The field of incremental learning devised several strategies for adapting to new vocabularies.
6693
Artetxe et al. (2020) show that simply relearning the embedding layer can be done efficiently, compared to training the model from scratch. Garcia et al. (2021) find that retraining the shared BPE codes once the new language data become available allows one to keep most of the pre-trained vocabulary, making even more efficient adaptation possible.
In this work, we go one step further and show that by designing a system where word representations are learned in a separate step we can zero-shot translate from an unknown language without any adaptation whatsoever. We therefore employ crosslingual word embeddings (Artetxe et al., 2016; Lample et al., 2018b) – a technique that has extensively been used in unsupervised machine translation, but hasn’t so far been given much consideration for incremental learning – to align monolingual word representations into a common space, and then use these as our regular word embedding layer in the NMT system. Crucially, keeping these embeddings frozen in the NMT training allows us to align new language vocabularies into the common embedding space post-training.
Furthermore, we show that this zero-shot translation capability allows us to easily generate synthetic parallel data for unknown languages and enables simple and efficient unsupervised machine translation without the need for denoising autoencoding on massive amounts of monolingual data.
Finally, we also experiment on continuous output NMT (Kumar and Tsvetkov, 2019), which – while underperforming the softmax models in terms of BLEU scores – perfectly fits our setup and attains vastly superior performance in terms of training times.
We make our code and training configurations available under https://github.com/ cmullovisl/clwe-transfer and we release our pre-trained multilingual models and embedding alignments.

heading: 2 Multilingual Neural Machine Translation
text: In training the multilingual NMT system we aim to estimate the probability P(Yℓtgt = y | Xℓsrc = x) that the sentence y in the target language ℓtgt is a suitable translation of the source sentence x in the source language ℓsrc. We describe the distribution of the sentences in different natural languages ℓ through random variables Xℓ, Yℓ. A universal en-
coder maps the sentences from the different input distributions onto a single, shared latent distribution Henc = enc(Xℓ). A decoder is then tasked to model the probability distribution from this latent variable: P(Yℓtgt = y | Henc = c). Identically to a regular bilingual NMT system, the multilingual translation system consists of a neural network, which, in a supervised setting, is trained to maximize the likelihood of a dataset of parallel sentences D = {(x1, y1), . . . , (xn, yn)}, where sentence pairs (xi, yi) are translations of each other:
min ϑ∈Θ
{ − n∑
i=1
logP ( Xℓitgt = yi | Xℓisrc = xi, ϑ
)}
Note that – aside from the added target language selection mechanism – the only difference to bilingual translation is the nature of the input and output distribution. Thus, the main practical difference in the training a multilingual system is that instead of using a bilingual training corpus, D consists of a concatenation of several parallel corpora.

heading: 3 Related Work
text: Cross-Lingual Word Embeddings Qi et al. (2018) look into the effectiveness of pre-trained embeddings for regular supervised NMT and find them to be helpful in a multilingual setting. Kim et al. (2019) conduct unsupervised MT by transferring the parameters learned on a high resource language pair to a new source language. Similar to our work they therefore align the new language embeddings to the parent model source embeddings, but they focus on a bilingual setting. Compared to their work we show that this setup not only allows for unsupervised MT but zero-shot translation from unseen languages, especially when combined with a multilingual setup.
Unsupervised MT Our work is closely related to unsupervised NMT (Lample et al., 2018a,c; Artetxe et al., 2017, 2019; Conneau et al., 2020; Song et al., 2019; Liu et al., 2020b). Initial works train unsupervised NMT systems by bootstrapping a common representation space between two languages through a combination of cross-lingual word embeddings and denoising autoencoding, and then exploit these common representations for iterative back-translation (Lample et al., 2018a). Through unsupervised means they thus induce a shared sentence representation, essentially bootstrapping a multilingual NMT system from monolingual data.
Later works do away with the cross-lingual word embeddings and create this shared representation space through large scale denoising autoencoding on massive amounts of monolingual data (Song et al., 2019; Liu et al., 2020b) and multilinguality (Sen et al., 2019; Siddhant et al., 2020; Conneau et al., 2020; Liu et al., 2020b). In contrast to this development, we show that cross-lingual word embeddings remain valuable when combined with regular NMT training, allowing to bootstrap high-quality multilingual representations even on small-scale datasets.
New Language Learning Multiple works look into adding unseen languages to multilingual encoder-decoder systems. Neubig and Hu (2018) concerned about the possibility of rapidly extending a multilingual NMT model by an unseen lowresource language. They compare between bilingual training, multilingual training alongside a similar source language and multilingual pre-training with as many languages as possible. They come to the conclusion that pre-training in a highly multilingual setting – in their case 58 source languages – significantly improves transferability to low-resourced languages. In particular, they achieve significant performance (up to 15.5 BLEU) on an entirely unseen language, provided parallel training data on a closely related language. In this work we use the same parallel data for NMT training and significantly improve upon their result through the employment of cross-lingual word embeddings. Escolano et al. (2019) devise an approach to multilingual NMT with independent encoders and decoders that enables addition of new languages. Li et al. (2023) fine-tune a model based on XLM-R and use it to zero-shot translate from languages seen in pre-training, but unseen in the supervised downstream task. Tang et al. (2020) look into extending mBART25 (Liu et al., 2020b) by 25 new languages and show that given enough modelling capacity one can add new languages without loss in quality on existing ones. In their work, however, they only consider new languages sufficiently represented in the base model vocabulary, and they leave the original vocabulary unmodified.
Vocabulary Adaptation Kim et al. (2019), Garcia et al. (2021), Marchisio et al. (2023) and Chen et al. (2023) look into vocabulary adaptation for new languages. Garcia et al. (2021) show that rebuilding the subword vocabulary once the new language data is available results in most of the
vocabulary matching the old one. Retaining the embedding vectors for common subword units allows for quick adaptation and minimal forgetting of the existing languages. In our work we directly compare to this method and show that a vocabulary based on aligned pre-trained embeddings not only presents a viable alternative to a subword-based one, but also allows for more efficient adaptation to and zero-shot translation from unseen languages.

heading: 4 Generalizing to Unseen Languages
text: To learn a multilingual NMT system that easily generalizes to unseen languages we propose a decoupled learning approach, divided into multiple steps:
1. train word embeddings on monolingual data for each of the base languages ℓ ∈ Lbase
2. align the word embeddings into a common embedding space
3. initialize the encoder embedding layer of a standard NMT model with the learned word representations
4. supervised NMT training on Lbase parallel data with frozen encoder word embeddings
In this setup we may later extend the translation model by an unseen language ℓnew /∈ Lbase by learning word representations for ℓnew and aligning them into the common word embedding space (see Section 4.1). We show that this alignment alone enables us to start translating from ℓnew in a plug-and-play fashion, even without any additional updates to the NMT model’s main body (i.e. the Transformer layers).
Optionally, the learned word representations may also be employed on the decoder-side, which will further enable the back-translation-based adaptation detailed in Section 5.3. Figure 1 describes our decoupled learning NMT architecture.

heading: 4.1 Decoupled Vocabulary Learning
text: In conventional training of a universal multilingual NMT system the ℓnew word vectors are randomly initialized and then trained end-to-end on the NMT objective. In training the model then learns to represent its words in a shared multilingual embedding space, by learning cross-lingual word correlations from the parallel data. In a monolingual data only setting we aim to emulate these multilingual word representations through cross-lingual word embeddings. These word embeddings are then integrated
into our encoder-decoder NMT model by simply using them in the word embedding layers.
For each language ℓ ∈ Lbase ∪ {ℓnew} we use pre-trained monolingual word embeddings learned with fastText (Bojanowski et al., 2017), which we then align into one common space through crosslingual embedding alignment (Joulin et al., 2018). For a common alignment between multiple languages we choose to align into a single hub space (SHS Heyman et al., 2019), i.e. we pick one of our base languages as a pivot and align each of the embedding spaces Eℓ to Epivot.

heading: 4.2 Zero-Shot Translation from Unseen Languages
text: To provide an indication of how well the universal encoder generalizes an unseen language ℓnew we let our multilingual model decode from ℓnew
sentences, without any sorts of prior exposure to ℓnew. As outlined in Figure 2 we therefore employ the cross-lingual embeddings to look up the ℓnew word vectors and then simply decode from the resulting sequence. While the encoder may not know the syntax of the unseen language, many syntactical concepts in language – such as grammatical case, part-of-speech, grammatical genders, or tense – are encoded at the word level1. As such we hypothesize that the word vectors trained on the new language alone provide enough syntactic level information to perform the language comprehension task to a certain degree.
To test our hypothesis that a high degree of multilinguality will help in making the sentence representations more general we can then vary the number of languages the encoder gets exposed to in training. We may then observe how this affects languages closely related to languages in Lbase as well as languages that are distant to any of the languages in Lbase.

heading: 4.3 Unsupervised Machine Translation
text: Iterative back-translation (Sennrich et al., 2016; Lample et al., 2018a; Artetxe et al., 2018) is a common approach in unsupervised NMT, where a bidirectional model is used to generate synthetic parallel data and then subsequently adapted on this data to then generate even better synthetic data. To iteratively improve translation between two languages ℓ1 and ℓ2 one would
1. translate ℓ1 monolingual data to ℓ2 to generate ℓ2 → ℓ1 parallel data 2. adapt the model on the ℓ2 → ℓ1 data 3. translate ℓ2 monolingual data to ℓ1 to generate
ℓ1 → ℓ2 parallel data 4. adapt the model on the ℓ1 → ℓ2 data
The resulting model will be able to produce better ℓ2 → ℓ1 synthetic data than in Step 1 and the process may be repeated for iterative improvements until convergence.
To obtain usable translations for Step 1 the model, however, must first be bootstrapped, which is usually done through denoising autoencoding or language modelling on massive amounts of monolingual data. This process is both costly and oftentimes fails for languages that are either too distant
1Take German for example, a noun would have different surface forms for different grammatical cases, which the fastText model should take into account and appropriately assign different word vectors.
or have too little monolingual data available (Kim et al., 2020; Marchisio et al., 2020). We show that in our setup we may skip this step entirely by zeroshot translating from an unknown language ℓnew to kick-start the iterative back-translation process. Furthermore, we show that through our model’s improved ability to generalize to unseen languages we can robustly improve ℓnew translation quality, even when ℓnew is so distant to any other language ℓ ∈ Lbase that the zero-shot generated synthetic data is barely intelligible, and even on relatively small amounts of monolingual ℓnew data.

heading: 5 Experiments

heading: 5.1 General Experiment Setup
text: Cross-lingual word embeddings As the basis for all of our monolingual word embeddings we use pre-trained fastText models2 trained on Common Crawl (Wenzek et al., 2020). These models provide 300-dimensional high-quality embeddings for full word units, with pre-trained models available in 157 languages. Using the pivot approach described in section 4.1 we align these embeddings into a common embedding space using direct supervised optimization on the RCSLS criterion (Joulin et al., 2018) using the bilingual dictionaries provided by the MUSE project (Lample et al., 2018b). To reduce the vocabulary size, we subsequently regenerate our vocabulary and the corresponding embedding vectors using only the words in our NMT training corpus. For alignment accuracies and vocabulary sizes see Appendix Table 3.
Full word-units While using a joint BPE lets us share information between vocabularies and reduces the size of the multilingual vocabulary, for our approach we opt for full word units. Our preliminary experiments have shown full-word units to outperform our BPE models when used in combination with frozen cross-lingual embeddings. Furthermore, this gives us better compatibility with distant languages with separate alphabets and avoids the problem of having to extend the learned BPE codes post-training when we add a new language to the model. To alleviate the OOV word problem we use fastText as the basis for our monolingual word embeddings, which, by using subword level information, gives us the ability to map unseen words into the embedding space.
2https://fasttext.cc/docs/en/crawl-vectors. html
From an implementation perspective we merge our monolingual vocabularies and concatenate their embedding matrices, while dealing with duplicate words across different languages by encoding each word with a language specific prefix, e.g. we encode the English word bank as en@bank. At inference-time we limit the target vocabulary to the language we decode to, thereby forcing the model to only choose tokens from our desired target language, which is not trivial in a model based on BPE units shared across multiple languages. Due to the large vocabulary sizes using full word-units increases the number of parameters immensely. However, since we keep the embedding layers frozen the amount of trainable parameters and GPU memory consumption3 decreases in practice. Furthermore, our approach results in lower sequence lengths, thus partially making up for higher computational complexity of the larger softmax embedding layer. At inference-time (and potentially also training time) the computational complexity may further be reduced through dynamic vocabulary sub-selection (Senellart et al., 2018). We leave this line of research for future work.
Von Mises-Fisher loss Since the fastText embeddings we employ in the output vocabulary are normalized to unit norm, the matrix multiplication in the final projection layer is similar to a cosinesimilarity search through the fastText embedding space. Due to our large vocabulary sizes this makes the computation of the output layer during training and test time more costly. To counter this issue we conduct experiments with von Mises-Fisher loss for NMT (Kumar and Tsvetkov, 2019), where model outputs are optimized toward frozen pre-trained word embedding targets, which perfectly fits our setup. This loss is similar to the cosine-loss and it makes the computational complexity of training entirely independent of the vocabulary size, and we observe up to 5.8× speedups in training throughput. We find this approach to consistently under-perform the traditional softmax-based models (Appendix C), but in light of the speedup we keep this approach for the Section 5.2 experiments. In the following we refer to this model variant as vMF.
Datasets As our training, development and evaluation data sets we use multilingual transcriptions
3for non-trainable parameters we do not have to save the Adam optimizer states
of TED talks in 60 languages (Qi et al., 2018). For our main body of experiments we include English, German, Spanish, French and Italian as the supervised languages, which we refer to as RomanceGermanic languages. We train the basesystem on 20 translation directions, and a parallel corpus size of 3, 576, 046 sentences. For our experiments on unsupervised MT we focus on monolingual indomain data obtained from the same corpus of TED talks. To prevent contamination this corpus is split into train, validation and test data using the same TED talks across all languages (Qi et al., 2018). For larger scale out-of-domain experiments we use monolingual data from the Wiki-40B dataset (Guo et al., 2020) and newscrawl data (Bojar et al., 2014) and we further evaluate on Flores (Goyal et al., 2022) and Tatoeba (Tiedemann, 2020).
NMT model For our NMT model we use a Transformer model (Vaswani et al., 2017) with relative position encodings (Shaw et al., 2018) and post-layernorm. In accordance with our 300- dimensional fastText embeddings we train 9-layer models with embedding dimension 300 unless otherwise specified. Our implementation is based on the repository provided by Kumar and Tsvetkov (2019)4, which is based on OpenNMT-py (Klein et al., 2017). We share embedding parameters across the encoder and the decoder wherever applicable, and use tied output embeddings. Refer to Appendix B for the full set of training parameters.
We train our softmax-based models using Adam (Kingma and Ba, 2015) and our vMF model (Kumar and Tsvetkov, 2019) using RAdam (Liu et al., 2020a) for 160,000 updates or until early stopping. For estimates of training times and GPU hours see Appendix G. We evaluate our models using the ChrF++ metric (Popović, 2017) and list BLEU (Papineni et al., 2002) scores in the appendix. The supervised direction evaluation scores for our Romance-Germanic models are listed in Table 5.
Finally, to verify the viability of our setup – namely a full word-unit translation system with frozen pre-trained word embeddings – we also train a standard subword-unit translation system which matches our proposed model as closely as possible. We train the BPE model on the concatenated multilingual corpus for 50k merge operations. The results show an average score of 55.1/51.5/55.3 ChrF++ for the softmax-based model, the vMF
4https://github.com/Sachin19/seq2seq-con
model and the BPE baseline (detailed in Table 5), demonstrating that while the vMF model lags behind in terms of translation performance, the softmax-based model closely matches the baseline despite the word embeddings not being optimized for translation.
To ensure better reproducibility we base our data and training pipelines on DVC (Kuprieiev et al., 2024). To avoid data leakage of unseen languages into supervised training we structure our experiments in two stages, where the choice of ℓnew may be postponed until the second stage.

heading: 5.2 Zero-Shot
text: In the following we present our results for translating from unseen languages after aligning their word embeddings into the common word embedding space and plugging them into the NMT model’s encoder without any further adaptation. We report the results in Table 1. For comparison, we also evaluate the translation performance of a regular BPE-based model on unseen languages and observe that it fails to produce reasonable translations. Similar to the findings of Neubig and Hu (2018) we observe up to 12.9 BLEU (34.9 ChrF++) on Portuguese-English, but find the translations to be unintelligible.
For additional comparison we employ Garcia et al.’s (2021) method of adapting the model via vocabulary substitution. Matching their findings we observe that rebuilding the subword vocabulary with the new language results in 77 - 95 % overlap with the old vocabulary and after adaptation the translation performance nearly matches the oracle setup. We further improve on their method by freezing the decoder (but training the crossattention weights) when adding a new source language, improving by +0.8 ChrF++ (+0.6 BLEU) on Portuguese-English. Compared to the resulting 62.4 ChrF++ after adaptation we achieve up to 61.0 ChrF++ translating zero-shot on our in-domain test set. On the out-of-domain test sets we even outperform the adapted system by +3.1 ChrF++ on Flores (27.5 vs 30.6) and +1.8 ChrF++ on Tatoeba (36.9 vs 38.7). Note, that this best performing model is our Romance-Germanic model with several supervised languages close to Portuguese. For the same model we observe much higher gaps between supervised and zero-shot performance: For Russian the baseline out-performs our method by 4 ChrF++, which is, however, reasonable considering that our model has at the time of decoding never been exposed to any Slavic language.
With growing distance to the supervised languages this gap grows higher. For Korean the English translations end up being barely intelligible5. Nevertheless, the translations are still good enough to kick-start the iterative back-translation process, as we show in Section 5.3. We observe that on more distant languages the model oftentimes gets stuck in decoding loops, which we address in a post-processing step to remove repeated n-grams.
Effect of Multilinguality To study the effect of multilinguality we train multilingual models on varying numbers of languages and families and compare the translation performance on unseen languages in Table 1. On the supervised languages we see a clear trend of performance degradation the more languages we include in our model, e.g. for Spanish→English we see clear degradation when going from bilingual to multilingual. For the zeroshot directions on the other hand we see a clear boost in performance when going from bilingual to multilingual. As we increase the number of languages, however, we do not see a clear trend of im-
5Note, that we have observed some tokenization issues with Korean, leading to large vocabulary size of 330k. Generally, we observe drops in performance when fastText requires special tokenization, i.e. for Chinese, Japanese, Vietnamese.
provement. Rather than the number of languages, the relatedness of the added languages plays the more important role, i.e. adding non-Romance to the model continuously degrades performance on Portuguese, and on Slavic languages the performance increases once we include Slavic languages in training and decreases once we include Asiatic languages.
Finally, for the three languages aren’t directly related to any of the supervised languages (i.e. Arabic, Turkish, Korean) we see a jump in performance upon inclusion of Slavic languages, but no consistent improvement on inclusion of Asiatic languages.

heading: 5.3 Unsupervised Machine Translation
text: In Section 5.2 we show that our setup enables translation from unknown languages, albeit with lower quality the further the distance from our training languages. In our final experiment we exploit this capability for zero-shot iterative back-translation (Lample et al., 2018a). We therefore simply perform beam search to translate the monolingual target language portion of our TED corpus to each of the known languages to generate synthetic parallel data. On this synthetic data we then train
our softmax-based Romance-Germanic model for 16,000 steps or until saturation. In training, we freeze the encoder in each odd iteration, the decoder in each even iteration. Table 2 reports the ChrF++ scores after each iteration of training on back-translated data (Table 9 for BLEU scores), alongside the supervised fine-tune on real bilingual data. We observe that for the closest language (Portuguese) a single round of back-translation closes the gap, and on the out-of-domain test set even outperforms the supervised fine-tune. On the more distant languages we see a rapid closing of gap between the supervised and the unsupervised finetune.
Note that our primary goal is to demonstrate that our setup enables us to skip the expensive bootstrapping process involving denoising autoencoding on massive amounts of monolingual data exhibited in most approaches to unsupervised MT. As such we don’t continue the iterative back-translation process until saturation, but execute at most 6 iteration. We believe that more iterations will further close the gap between supervised and unsupervised performance.
Effect of Domain Many analyses of unsupervised MT have shown the importance of the consid-
ered domain. As the Table 2 bottom row shows the monolingual data we use for our in-domain experiments ranges from 52k to 208k sentences, which can be considered extremely small amounts when it comes to monolingual data. To study the effect of domain and amount of data we perform training on Wikipedia and news domain in Appendix F. We observe that changing the domain helps in some cases, for improving the overall translation performance a larger number of iterations on TED domain, however, seem to be most effective.
Finally, we compare to the approaches detailed in Song et al. (MASS 2019) and Artetxe et al. (2019). For both systems we use their provided implementations to train an English-Turkish unsupervised MT system, using 1947k sentences of Turkish Wikipedia data for bootstrapping. Both systems end up failing to produce any reasonable translations on any of the test sets. To verify our setup we further use MASS to train an EnglishFrench unsupervised MT model on the French portion of our Wikipedia data. This model ends up achieving 52.4 (54.1) ChrF++ from (to) English on the Flores test data.

heading: 6 Conclusion
text: In this work, we have looked into the generalization ability of a multilingual NMT system on unseen languages. We therefore propose a decoupled learning setup for MT training, where we learn the word representations of each language in a separate step. This enables easier generalization to new languages as this approach enables extension of the vocabulary post-training. We show that through simple alignment of an unseen language word representations into the common vocabulary space, we can produce intelligible translations without any further adaptation. We further show that on languages similar to the ones the NMT system has already been exposed to, this zero-shot translation setup even approaches supervised translation performances.
Lastly we show that this capability to produce translations for unseen languages can be exploited for efficient unsupervised machine translation.
Future Work In the future we would like to explore the efficacy of our presented setup for an incremental learning system, which would entail in-depth experiments on catastrophic forgetting.

heading: 7 Limitations
text: The method described in this work relies on high quality word embeddings and alignments between them. Training such high-quality word embeddings requires access to large amounts of monolingual data, which may be available in abundance for high and mid-resource languages, but may be unavailable in the amounts necessary for endangered languages, or languages which are primarily spoken, such as dialects or languages without an official orthography. This issue might be dealt with through data-efficient means of training word embeddings, which might entail an adaptation of word embeddings from a related high-resource language. Furthermore, we rely on bilingual dictionaries for learning high-quality alignments between embedding spaces. This may present a limitation for lowresource scenarios, in practice, however, bilingual dictionaries are less of an issue than the availability of monolingual data (see Kamholz et al., 2014; Vulić et al., 2019).
In part due to the availability of high-quality multi-parallel data and in part due to a limit in computational resources our work focuses on relatively small scale experiments. Furthermore, we observe that our models are highly tuned to the
TED domain and perform poorly on out-of-domain test sets in the considered supervised, as well as unsupervised settings.
Lastly, our setup operates on word-level vocabularies, which presents a computational bottleneck in the output vocabulary. This makes the model difficult to train and deploy in its current implementation. In this work we look into accelerating training using the approach presented by Kumar and Tsvetkov (2019) but we observe a drop in translation quality in doing so. These issues may, however, be addressed with more engineering work, such as using language specific mini-batches (in combination with gradient accumulation), and a dynamic vocabulary sub-selection at training time and at inference time (Senellart et al., 2018).

heading: 8 Acknowledgements
text: This work is supported from the European Union’s Horizon research and innovation programme under grant agreement No 101135798, project Meetween (My Personal AI Mediator for Virtual MEETtings BetWEEN People). Part of this work was supported by funding from the pilot program CoreInformatics of the Helmholtz Association (HGF).

heading: B Training Parameters

heading: C Multilingual Basesystem Scores
text: We evaluate all our systems using SacreBLEU (Post, 2018). Following the best practice we list the SacreBLEU signature for our BLEU and ChrF++ evalations as follows: nrefs:1|case:mixed|eff:no|tok:13a| ↪→smooth:exp|version:2.4.0 nrefs:1|case:mixed|eff:yes|nc:6|nw:2| ↪→space:no|version:2.4.0
Table 5 details the ChrF++ scores of our different multilingual models averaged over their 20 supervised translation directions. Table 6 and 7 detail the individual scores.

heading: D Supplementary Zero-Shot Translation Scores
text: Table 8 details additional zero-shot translation scores, for languages and translation directions not listed in Table 1. The scores are produced with our main Romance-Germanic model, so the seen languages are English, German, Spanish, French and Italian and model dimension dmodel = 300.

heading: E Unsupervised MT BLEU Scores
text: Table 9 details the BLEU scores for the Table 2 evaluations.

heading: F Effect of Domain
text: To study the effect of domain and amount of data we perform iterative back-translation training on Wikipedia and news domain. For domain specific test data we use Flores, to evaluate the Wikipedia domain, and wmt16 test data for evaluation of the news domain. We detail the results in Table 10. When trained on Wikipedia we observe a slight increase in ChrF++ scores on English-Portuguese Wikipedia domain, but a large drop on the original TED domain. In all other cases we observe drops in performance on all considered domains.

heading: G Estimate of GPU Hours
text: We conduct our preliminary experiments on an Nvidia GTX 1080Ti GPU. The main body of our experiments if conducted on a cluster of Nvidia Titan RTX GPUs. Our largest softmax-based fullword vocabulary model (with a count of 53M trainable parameters, 317M non-trainable) is trained on an Nvidia A6000 GPU. The GPU hours roughly sum up to 1039 GPU hours, 197h out of these on an Nvidia A6000 GPU.

heading: Future Work
text: Future Work In the future we would like to explore the efficacy of our presented setup for an incremental learning system, which would entail in-depth experiments on catastrophic forgetting