Refining Low-Resource Unsupervised Translation by Language Disentanglement of Multilingual Translation Model
Authors: Xuan-Phi Nguyen
Shafiq Joty
Kui Wu
AiTi Aw
Venue: NeurIPS 2022 Accept (2022)
Keywords: machine translation
unsupervised machine translation
low-resource translation

Abstract: Numerous recent work on unsupervised machine translation (UMT) implies that competent unsupervised translations of low-resource and unrelated languages, such as Nepali or Sinhala, are only possible if the model is trained in a massive multilingual environment, where these low-resource languages are mixed with high-resource counterparts. Nonetheless, while the high-resource languages greatly help kick-start the target low-resource translation tasks, the language discrepancy between them may hinder their further improvement. In this work, we propose a simple refinement procedure to separate languages from a pre-trained multilingual UMT model for it to focus on only the target low-resource task. Our method achieves the state of the art in the fully unsupervised translation tasks of English to Nepali, Sinhala, Gujarati, Latvian, Estonian and Kazakh, with BLEU score gains of 3.5, 3.5, 3.3, 4.1, 4.2, and 3.3, respectively. Our codebase is available at https://github.com/nxphi47/refine_unsup_multilingual_mt

Refining Low-Resource Unsupervised Translation by Language Disentanglement of Multilingual Model
Xuan-Phi Nguyen 1.3 Shafiq Joty 1:2 Wu Kui Ai Ti Aw
*Nanyang Technological University
Salesforce Research
3 Institute for Infocomm Research (I²R), A*STAR Singapore
meguyexuu022e.nttnn ntu, s srjetyentu) edu. sg (wuk, wwa.aiaitii@ii00 stt.sttareeee edu.
Abstract
Numerous recent work unsupervised machine translation (UMT) implies that competent unsupervised translations of low-resource and unrelated languages, such Nepali Sinhala, only possible if the model is trained in massive multilingual environment, where these low-resource languages are mixed with high-resource counterparts. Nonetheless, while the high-resoure languages greatly help ktck-start the target low-resource translation tasks, the language discrepancy between them may hinder their further improvement. In this work, propose simple refinement procedure to separate languages from a pre-trained multilingual UMT model for it to focus on only the target low-resource task. Our method achieves the state of the art in the fully unsupervised translation tasks of English Nepali, Sinhala, Gujarati. Latvian, Estonian and Kazakh, with BLEU score gains of 3.5, 3.5, 3.3, 4.1, 4.2, and 3.3, respectively. Our codebase is available at github.com/mxxhii47/rhiiin unsuu multilingual mt
Introduction
Fully unsupervised machine translation (UMT), where absolutely only unlabeled monolingual corpora are used the entire model training process, has gained significant traction recent years. Early success in UMT has been built the foundation of cross-lingual initialization and iterative back- translation [T 13
6 25 18]. While achieving outstanding performance on empirically high-resource languages, such German and French, these UMT methods fail to produce any meaningful translation (near zero BLEU scores) when adopted in low-resource cases, like the FLoRes Nepali and Sinhala unsupervised translation tasks [10] Arguably, such low-resource languages need UMT the most.
Liu al. discovered that low-resource UMT models benefit significantly when they are initialized with multilingual generative model (mBART), which is pre-trained on massive monolingual corpora from 25 languages (CC25). Not only does this multilingual pre-training set contain data of the low-resource languages of interest (e.g., Nepali), it also comprises sentences from related languages (e.g.. Hindi), which presumably boosts the performance of their low-resource siblings. Tran et al. [26] and Nguyen et al. [99 later propose ways to mine pseudo-paralle, data from multiple directions to improve low-resource UMT. On the other hand, Conneau et al. 7 find the curse of multilinguality, which states that performance of multilingual model with fixed model capacity tends to decline after a point as the number of languages in the training data increases. Thus, adding more languages may hinder further improvement in low-resource UMT, which is intuitively due incompatibility between the linguistic structures of different languages. Furthermore, these models have to tell languages apart by using only single-token language specifier prepended into the input sequence,
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

which is shown to be insufficient M. Our analysis also shows that the models sometimes predict words of the wrong language (see Appendix). Meanwhile, as shown later in our analyses ($4.4). other complementary techniques such as pseudo-parallel data mining may have reached their limits in improving low-resource UMT.
To alleviate the curse of multilinguality and work around the linguistic incompatibility issue in low-resource UMT, we propose a simple refinement strategy aimed at disentangling, or separating. irrelevant languages from a pre-trained multilingual unsupervised model, namely CRISS [26. and focusing exclusively on number of low-resource UMT directions. Briefly, our method consists of four stages. In the first stage, we use modified back-transsltion technique [23] to finetune an initial pre-trained multilingual UMT model on English (En) and a family of rowrresuueee languages C with a different set of feed-forward layers in the decoder for each En ++ L pair. This step aims at discarding irrelevant languages and separating the languages L from each other. The second stage separates the resulting En ++ L model into two En - L and L + En models. This stage motivated by the fact that English often vastly distinct from low-resource languages L, thus requiring greater degree of disentanglement and less parameter sharing. The third stage boosts the performance by using the second-stage models as fixed translators for the back-translation process. The final stage, which is only applied to from-English directions, is aimed to separate the target low-resource languages between themselves into different models. Overall, our method prioritises to maximize the individual performance of each low-resource task, though the rafe-offf is that it can no longer translate multiple languages in a one-for-all manner.
In our experiments, our method establishes the state of the art in fully unsupervised translation tasks of English (En) to Nepali (Ne), Sinhala (Si), Gujarati (Gu), Latvian (Lv), Estonian (Et) and Kazakh (Kk) with BLEU scores of 9.0, 9.5, 17.5, 18.5, 21.0 and 10.0 respectively, and vice versa. This is up to 4.5 BLEU improvement from the previous state of the art [26]. We also show that the method outperforms other related alternatives that attempt to achieve language separation in various low-resource unsupervised tasks. Plus, our ablation analyses demonstrate the importance of different stages of our method, especially the English separation stage (stage 2).
2 Background
Recent advances in fully unsupervised machine translation (UMT) are often built on the foundation of iterative back-translation [23
3 13 It is clever technique where the model back-translates monolingual data from one language to another, and the resulting pair is used to train the model itself via standard back- proeppropaggtioo. To make UMT successfully work, iterative
11
must be accompanied with some forms of cross- or multi-lingual initialization of the model, either through an unsupervised bilingual dictionary 112 11, phrase-based statistical MT [33, or language model pre-training 6 25 17. Apart from that, cross-model back-tranllated distillation [T8], where two distinct models are used to complement the target model, can also boost UMT performance. Plus. pseudo- -atoollaalllllll data mining, where sentences from two monolingual corpora are paired to form training samples through certain language-agnostic representations 28 26 19. significantly improve UMT performance.
has been shown to
Nonetheless, when it comes to fully unsupervised translation of distant and low-resoure, languages 10. the aforementioned techniques fail to translate their success in high nigh-resource languages to low-resource counterparts, unless "multilinguality is involved. mBART [17] is among the first multilingual model to boost the performance greatly low-resource UMT, by pre-training the model on the CC25 dataset, which is a 1,4 1.4-terabyte collection of monolingual data from 25 languages. The CC25 dataset contains not only data from English and the low-resource languages of interest, like Nepali, but also data from their related but higher-resource siblings, like Hindi.
later advances the state of the art in low-resource UMT by finetuning mBART with pseudo-paral data of more than 180 directions mined from the model's 's encoder representations. This type of UMT models handles translations in multiple directions by a few principles that are designed to maximize parameter sharing. First, it builds a large shared vocabulary that covers wordpieces from all languages. Second, the encoder encodes input sentences without knowledge about their corresponding languages to promote manguage-agnostie representations in its outputs. Lastly, the decoder receives the encoder outputs and decodes the translation in the target language by prepending a language-speccfie token at the beginning of the output sequence. Training the model
CRISS 266

---

in such a multilingual environment helps the encoder learn language-agnostic latent representations that are shared across multiple languages, allowing the decoder to translate from any language. The vast availability of high-resource siblings (e.g., Hindi for Indic languages) may help improve the performance of low-resource MT significantly [9 [ Meanwhile, LAgSwAV [19] improve unsupervised MT by mining meeuoo-prallle data from monolingual data with cluster assignments.
Despite its outstanding success, multilinguality may conversely hinder the model's further improvement in low-resource unsupervised translation due to the curse of multilinguality M 77. multilingual model with a fixed capacity may perform suboptimally on individual languages as it is forced to simultaneously handle conflicting structural reorderings of distant languages [16]. The single-token language specifier is also not enough ensure the language consistency of its predictions [. Our proposed method aims to gradually separate the languages and focus on the target low-resource directions only. One prominent feature of our method is that it prioritises to separate English from the remaining low-resource languages first, as these languages are often much more distant from the common language English than from themselves 27
There are several efforts with similar objectives as ours, albeit of different contexts and scenarios. decoder model to be trained with back-translation for unsupervised MT, which is slightly similar to stage I of our proposed four-stage refinement process, as explained in $ Our stages 2, 3 and 4 improve the first stage, or equivalently Sen et al. (22), further by refining the model in ways that are not possible in stage Meanwhile, Sachan and Neubig (2T). Zhang et al. (29] Li and Gong [16] propose methods to cause the model to implicitly "select" relevant language- specific parameters by gating mechanisms or gradient readjustments, in the context of supervised multilingual translation. However, these methods, as found in U struggle with unsupervised MT' as signals from back-translated synthetic training samples are often much noisier. On the implementation side, the design of our language-specific FFN
Specifically, Sen al. (2 propose shared encoder and language-speciin
layers 3.1 is also largely inspired by sparsely-distrbuus mixture of experts [24 14 30).
Low-resource MT can also benefit greatly from the presence of supervised parallel data, or auxiliary data, in related language (e.g., English-Hindi for English-Nepali which is not fully unsupervised setup but zero-shot scenario. Garcia et al. 8 [ propose multi-stage training procedures that combine both monolingual data and auxiliary parallel data to improve the performance of low- low-resource translation. Meanwhile, Chen et al. 4 5
suggest to scale their experiments up to 100 languages to attain better performance boosts. Nonetheless, while such line of work offers considerable insight into low-resource MT, it is not within the scope of this paper, as we focus on fully unsupervised machine translation where absolutely no parallel data is available, regardless of languages.
3 Language disentanglement refinement
In this section, we describe our multilingual language disentanglement refinement procedure that consists of four stages. Before the training process starts, first choose to restrict the model, e.g., CRISS (26), to only translate English (En) to (and from) small group L = (4,,2, l2,.....vy low-resource languages based their genealogical, demographic, areal or linguistic similarities, such as Nepali (Ne), Sinhala (Si) and Gujarati (Gu) for Indic family (a part of Indo-European). There no guarantee, however, that these languages are indeed significantly close due such heuristies. In the following, we first propose a language-specine gradual language separation process (Figure []. Then, in $3.2 $3.5 we explain the four stages of our language separation procedure, which is also visually demonstrated in Figure 2
Iv) ) of N
feed-forward layer (s3.J). which helps achieve a
3.1 Lsneuage-speeiffcc sharded foed-forward layer
While our method starts with finetuning a fully parameter-share multilingual MT model, we desire to gradually and progressively increase the level of language separation by reducing the degree of parameter sharing of the model. Thus, for every layer j of an H-layer Transformer decoder such that j is divisible by a constant a (e.g., 3) and < j < H, we propose to replace its vanilla feed-forward layer (FFN,) with N separate layers FFN,, such that each corresponds to a language 7, E L.
During the training process, each of the separate FFN,,; layers is sharded and reside individually on a distinct GPU-i accelerator, where their back-propagated gradients are not aggregated or averaged during the data parallelism process. Each GPU-i device is only fed with data from its
3

---

Separate
GPU-2
GPU-3
Self-Attention
Sell-Attention
Sell-Attention
Self-Attention
Decoder Layer
FFN-Ne
FFN-SI
FFN-HI
FFN-Gu
Altention
Attention
Attention
Attention
Shared
Data: En, Ne
Data:
Data: En, Hi
Data: Gu
En,
Figure 1: Illustration of a decoder layer with language-specific sharded FFN layers, where the FFN parameters for each language pair are separate and reside on specific GPU and only data streams from such language pair (e.g., En, Ne for GPU-1) are fed into the model replica in that GPU.
respective language l and the common language English. Meanwhile, the remaining parameters (e.g., embedding and attention layers) are still shared and their gradients are averaged across all GPU devices during back-propas prockpropagation. Figure T visualizes a sharded FFN layer of our model and how its parameters are allocated on each GPU. During inference, FFN,, is used to decode the translation to and from 4.
Formally, let θ (0m, be fully parameter-shared multilingual model (e.g., CRISS) initialization, where 0m the set parameters intended to shared and 0, the initial set of FFN parameters intended to be disentangled in our proposed model. We define our model as {Om. f). 8................. O,,0,, where On denotes the shared parameters while 03 denotes the separate parameters of all FFN layers (1 S < H) for language 4 E L. Before we begin training the model, initialize On Om and 03 0,, Vi NJ, Then during each update step, the gradients of fy and 6m are computed as follow:
V (ee.. Vol..... Voy
(1)
0
E
Vo (Jo(z:l)) + Jo (x
(2)
R. Te re
V
N
Σ LEC
(J.(..l))) + (o(oll))) IINK
E
(3)
where X, and X, are monolingual corpora (or data distribution) of language 4 and English respectively, (xly)) = log (=(x))) and yf P(-l,,, {0m, 02 }) is the back-translation into English from æ;; while v P(- xe, {6m, 0; is the Equations 2 and show that gradients are not aggregated for the language-specific sharded FFN layers, while the remaining are aggregated and averaged across all data streams. However, note that our model only separates the FFN layers of the decoder, while all encoder parameters are fully shared to ensure the language-agnostic representations from the encoder 26
into 4 from English by the model {θ... }-)
Implementation aspect. The rationale behind the design of our language-specife sharded FFNs is take full advantage of multi-GPU environment to maximize training efficiency both in speed and batch size. First, each sharded FFN is placed on separate GPU and only the respective language- specific data streams are fed into that GPU, which enables us achieve the same maximal batch size as the regular Transformer without running into out-of than other types of layers (e.g., attention), are chosen to be separated because they require minimal additional memory (only 50Mb for 8.8Gb model) to achieve noticeable performance gain (see $4.2). If have access to only one GPU, we can allocate all FFNs into it, reduce the batch size and increase gradient accumulation achieve the same multi-GPU effect.
eттоr. Second, FFN layers, rather

---

Stage
Stage
Stage 3
Stage 4
g00 back ineviatee data
014)
Head
((2)
( -
-
g(4)
0 (() d
CRISS
(i initialieed with gft
((1)
(11) back- trmattttee for d
Back-translate
0/2)
Initialize Self-back-translate
-En
-.n
Figure 2: Overview of the stages of our language disentanglement refinement procedure. Stage 1 finetunes our model with language-speciftt FFNs using multilingual tuck-translation Then, stage 2 and stage 3 continue by separating the model into to-En and from- En models. Finally, stage separates low-resource target language from the resulting from-En model.
3.2 First stage
With the aforementioned language-specific FFN layers, we enter the first stage of refinement. The purpose of this stage is to begin differentiating the language-specifii FFN layers so that each FFN can specialize in its designated language. That is, after initializing our model A = {8m, 0 /..... ON with
the pre-trained multilingual UMT model i = (6.. 0y) as 6.. = 6. and 8y θ vi E (1.., N1. we finetune it with iterative back-translation [6, except with some noteworthy modifications. Specifically, we uniformly sample monolingual sentences x, and z; from English and each low-resource language 4, E L, respectively. Then, we use 0, = {mm. (;: 07.}, a subset of f, to back translate øe and x; into yi and yf. respectively. After that, we train the model with y + Ze and yf + pairs. Note that each direction of back-translation, forward and backward directions, the appropriate FFN layers 8y are used with respect to the data stream involving language I, and English on a separate GPU- i. Furthermore, similarly to Conneau and Lample [6], the tack-translation model is continuously updated after each training step. The resulting model ((1) (ol)) 01.(11) ..... 0 (N,(1) will be used to initialize the second stage of refinement.
3.3 Second stage
Alleviating the curse of multilinguality may involve separation of unrelated languages. Yet, for many low-resource translation tasks, English is often significantly distant from the target low-resource counterparts, such as Nepali and Sinhala, which in fact share certain similarities between themselves. Thus, English may not share similar structural patterns with any of the target languages (27. Despite that, most existing UMT models are bidirectional 6 25 18 This means that they have to handle both translations to English and their target low-resource language l, causing them to endure reordering discrepancy and perform sub-optimally for both directions.
The second stage is aimed disentangle English. It builds two separate models 6(2) and g from ((1), which are specialised to translate to target low-resource languages respectively. Specifically, the second stage starts by initializing both g
and to English, -6 and g(2) with g(1), We then finetune them with iterative nack-translation exclusively in Fn + and - Fn directions, respectively. Different from the first stage, we a fixed model ((1) to back-translate the monolingual data. In other words, to finetune (2) we first initialize with ((1), Then we sample monolingual data H, uniformly from all E C and use the fixed model ((1) to back-translate them into English vff ~ P(-2.,, (0.)) (2(0))) }). The resulting pair yf - is used to train g(2) direction, we finetune 0 0(2)
-Cc- In the opposite
En by first initializing it with g(1), then sample English data ze and
5

---

randomly choose an 4 E L to back-translate de into v/ ~ P(- 10m/0,000 and finally train 8 with v22 ze pairs.
3.4 Third stage
The third stage improves the performance further by continuing finetuning g(2) and ((2) -6 En into 0
and E> respectively. Specifically, we build ( by initializing it with 013 and use 0 as fixed trassaaannlttoon model and train ( in the same manner as in $3.3 Similarly, 6 is trained in the same fashion in the opposite direction.
3.5 Fourth stage
The last stage is aimed achieve the maximal degree of language separation not only from English, but also between all languages in the target group. Specifically, we seek to train N separate models 044), for each language 4 L. Each model 0'4) is initialized with ( with the corresponding language-specific FFN layers, i.e., {0m) (0)())))) and ((+) is trained by back-translation with fixed backward model ( -n and exclusively Fn -> direction. Because only one direction is trained in this stage, we no longer use the above language-speciffs FFN layers for our end models. Instead, we revert back to the vanilla Transformer, whose FFN layers are initialized with the appropriate language-specific layers from the previous stage. Note that we do not perform this stage of refinement for the to-English (-> En) direction, due to the fact that the encoder is fully shared across all languages and there is no difference in separating 4 + En directions for the encoder, while the decoder only decodes the common English language.
4 Experiments
In this section, we tested our method on the standard low-resource unsupervised machine translation [..I. We then empirically analyze various aspects of our method and compare it with other possible variants (4.2) as well as other related alternatives in the literature ($4.))
4.1 Low-resource unsupervised translation
Setup. We evaluate our method on the FLoRes [1] low-resource unsupervised translation tasks on English (En) to and from Nepali (Ne), Sinhala (Si) and Gujarati (Gu) from the Indic family, as well as other low-resource languages: Latvian (Lv) and Estonian (Et) from the Uralic family and Kazakh (KK) from the Turkic family. Although Hindi (Hi) and Finnish (Fi) are relatively high-resourced compared their respective siblings, we still add them into the mix for Indic and Uralic families respectively to help the learning process of their respective low-resource siblings, following 99 5).
We start finetuning our model using the exact same codebase and the published pre-trained CRISS model 266). We also use the same sentencepiece pre-processing and vocabulary. Specifically, we group the tested languages into groups: Indic (Ne, Gu)),, Uralic (E).(E)i.v..t)) and Turkic (En-Kk). For each group, we use monolingual corpora from the relevant languages of the Common Crawl dataset, which contains data from totally 25 languages [17]. We limit the amount of monolingual data per language to up to 100M sentences. Similarly to CRISS, our model is 12-layer Transformer, whose decoder's 's FFN layers are replaced with our language-specifiic sharded FFN layers with 3 (see 1). during the first 3 stages. For each group of N languages, we use N GPUs to train the model and shard each language-specific FFN layer to a single GPU, where we channel only data streams of the respective languages, according our formulations in $3.2 We use a batch size of 1024 tokens with gradient accumulation factor of 2 (20). Similarly to the baselines to evaluate the BLEU scores with beam size of 5. In each
26 19), use the multi-bleu. stage, we finetune the models for 20K updates. Further details, such as corresponding SacreBLEU scores for reference, provided in the Appendix.
'To ensure the scores are comparable, we matched the entire evaluation pipeline (test sets. tokenzation, scripts, etc) with the baselines (26 [9] and consistently reproduced baselines' results with pipeline.

---

Table Comparison of BLEU scores for different methods on fully unsupervised translation tasks of
various low-resource languages from the Indic, Uralic and Turkic language families.
Indic
Uralic
Turkic
Method
En-Ne
Ne-En
En-Si
Si-En En-Hi Hi-En En-Gu Gu-En En-Fi En-Fi Fi-En En-Ly Lv-En En-Et Et-En En-Kk Kk-En
mBART
4.4
10.0
3.9
8.2
LAgSwAV
5.3
12.8
5.4
CRISS
5.5
14.5
6.0
14,5
19.4
23.6
14.2
23.7
20.2
26.7
14.4
19.2
16.8
25.0
6.7
14.5
Ours
9.0
18.2
9.5
15.3
20.8
23.8
17.5
29.5
22,9
28.2
18.5
19.3
21.0
25.7
10.0
20.0
+A
3.5
3.7
3.5
3.3
2.7
4.2
3.3
5.5
Table 2: Stage-wise ablations of our method: BLEU scores for different refinement stages on the fully unsupervised translation tasks of various low-resource languages from the Indic, Uralic and Turkic language families.
Indic
Uralic
Turkic
Method
En-Ne
Ne-En
En-Si
Si-En
En-Gu
Gu-En
En-Ly
Ly-En
En-Et
Et-En
En-Kk
Kk-En
CRISS
5.5
14.5
6.0
14.5
14.2
23.7
14.4
19.2
16.8
25.0
6.7
14.5
+Stage
7.8
16.9
7.1
14.5
15.8
26.2
15.2
19.0
17.4
24.9
7.9
16.9
+Stage
8.6
17.7
8.7
15.0
16.4
28.8
16.3
19.2
19.7
25.6
9.2
18.5
+Stage
8.8
18.2
9.0
15.3
16.8
29.5
17.9
19.3
20.0
25.7
9.7
20.0
+Stage
9.0
18.3
9.5
15.3
17.5
29.1
18.5
19.2
21.0
25.7
10.0
19.9
Results. Table T shows the performances of our method against recent best baselines, namely mBART [17]. LAgSwAV [19] and CRISS [26), on the fully unsupervised translation tasks between
English and the low-resource languages from the Indic, Uralic and Turkic families. As shown, our method achieves the state of the art in En-Ne and En-Si FLoRes low-resource UMT tasks, with 9.0 and 9.5 BLEU, as well as other languages across the three families, such as 21.0 and 10.0 BLEU for En-Et and En-Kk tasks. It surpasses the previous best (26] by up to 5.8 BLEU and achieves a mean gain of 2.6 BLEU across the 16 directions. The results also indicate that method achieves more gain margin for low-resuurce languages (Ne, Si, Gu) compared to their high-resource siblings (Hi).
4.2 Ablation study
In this section, we conduct comprehensive ablation study to gain more insights of our method and the importance of its components. In particular, we examine the contribution of each stage of our procedure to the overall performance gain across various low-resource languages (Table 2). We also investigate how our decoder language-specifie FFN layers perform in comparison to other possible variants, such as hanguage-specifer attention layers.
Stage-wise wise ablation study. Table 2alle shows the performances at different refinement stages of our method in low-resource fully unsupervised translation between English to and from Nepali, Sinhala, Gujarati, Latvian, Estonian and Kazakh. As it can be seen, our first and second stages achieve significant kick-off gain from the CRISS baseline, with up 5. BLEU score for Gu-En direction. We also observe considerable improvement from stage to stage 2, with up to 2.7 BLEU. This indicates that separation of English from other low-resource languages play crucial role. Meanwhile, the third and fourth stages improve the performances further, though with less margin. In addition, it noteworthy that stage does not noticeably outperforms stage for to-English (- En) direction, due to the fact that the encoder is fully shared across all low-resource languages and there is no difference in disentangling I + En directions from the encoder's 's perspective. Thus, there is no point training stage 4 for to-English direction, as formulated in our stage 4 (see 3.5
Laneuage-speeiffe variants. Other than our decoder language-specific FFN layers explained in 3.. there could be other possible ways to shard model parameters into language-specific components. We compare some of these variants against our own in Table examine their effects on the FLoRes low-resource unsupervised translation tasks with Nepali, Sinhala and Gujarati. Specifically, we compare our method (Lg. FFN/Dec) with a similar variant without the canguage-specinee FFN layers (w/o Lg. FFN/Dec), with decoder language-specifir self- and cross-attention layers (Lgg At/Dec), with language-specific FFN layers in the encoder (Lg. FFN/Enc), and finally with language-specific FFN layers in both encoder and decoder (I.g. FFN/Enc+Dee). To ensure consistency, we only show results up to stage 3 of our procedure and skip the fourth stage, where language-specife elements
7

---

Table BLEU scores for different variants of our method in the Indic languages: without decoder language-specitg (LS) FFN (w/o Lg. FFN/Dec), with decoder LS attention layers (Lg. Att/Dec) instead, with L.S FFNs in encoder instead of decoder (Igg. FFN//nc), with LS FFNs in both encoder and decoder (I.g. FFN/Enc+Dec). Results reported after stage 3 of our refinement procedure because stage no longer contains language-specific elements.
Method
Mem
En-Ne
Ne-En
En-Si
Si-En
En-Gu
Gu-En
CRISS
0%
5.5
14.5
6.0
14.5
14.2
23.7
Ours (Lg. FFN/DDec)
4%
8.8
18.2
9.0
15.3
16.8
29.5
w/o Lg. FFN/Dec
0%
7.9
17.3
8.3
14.7
15.7
27.8
Lg. Att/Dec
4%
8.4
17.7
8.6
15.0
15.7
28.7
Lg. FFN/Enc
4%
6.5
15.1
7.1
13.9
14.9
26.0
Lg. FFN/Enc+Dec
8%
8.9
18.2
8.8
15.3
17.0
29.1
no longer apply. In addition to the BLEU scores, we also report the percentage of excess memory overhead needed for each method compared to the bascline CRISS (0% overhead).
As shown in Table 3 the variant without any language-specific parameters (w/o Lg.FFN/dec) underperforms our method by up to 1.0 BLEU, even though English is separated in stage 2 and stage This indicates that is essential for the method have gradual language disentanglement process. Furthermore, placing the language-spese specific FFN layers on the encoder only (Lg. FFN/Enc) causes even more detrimental performance drop. The results imply that is not advisable linguistically separate the encoder only. Similarly, having language-specifeg parameters on the attention layers (I-g. Att/Dec) does not perform as good as our FFN counterpart, despite having the same memory footprint overhead of 4%. Meanwhile, placing language-specific FFN layers on both encoder and decoder (Lg. FFN/Enc+Dec) yields relatively similar results as our own method, although requires double the memory overhead. Thus, choice of separate FFN parameters on the decoder only is more favorable given its outperforming capability while consuming less memory. The Appendix provides further ablation analysis this issue.
4.3 Comparison with related methods
In this section, we compare refinement procedure against other related methods, well those that share a similar language disentanglement objective as ours. Particularly, we reproduce the method proposed by Sen al. 22 in low-resource unsupervised MT. This method is equivalent to stage 1 of our refinement procedure, except that the entire decoder is linguistically disentangled with separate decoders for different languages, not just FFN parameters, Meanwhile, Sachan and Neubig (21 and Zhang et al. [29] both share the same goal of alleviating the curse of multilinguality, but in the context of supervised multilingual translation, where the model teaches itself implicitly "select" relevant tunguage-specifii layers by gating mechanisms or gradient readjustments. We adapt these methods to unsupervised MT by substituting supervised training data with multilingual back-translated samples by the models. Lastly, Zuo et al. [30] recently propose a mixture-don of of-experts (MoE) variant [14] 24). where experts (FFNs) are selected randomly to perform the forward pass. In the context of language disentanglement, these separate experts act as implicit language-specifts components. We also adapt this method to low-resource unsupervised tasks with multilingual back translatonn and set the number of experts as the number of languages. For all methods, we use CRISS the initial model.
The comparison results are shown in Table 4 As shown, the method proposed by Sen et al. [22 performs slightly above stage 1 of our method, while it underperforms our complete procedure by up to 2.7 BLEU in Gu-En task. This in line with the fact that it almost equivalent to our first stage. The approaches suggested by Sachan and Neubig (21) and Zhang et al. 229]. on the other hand, yield considerably lower BLEU scores in various low-resource unsupervised tasks. A possible reason for this underperformance is that these methods are designed for supervised multilingual translation, where accurate parallel data plays a crucial role. When adapting to the unsupervised regime, the synthetic training samples created by back-translation may be too noisy and inaccurate. Meanwhile, the MoE candidate (30) also produce lower BLEU scores than our method, which can be due to its lack of explicit language-specific components to enforce language disentanglement naturally.
8

---

Table 4: Comparison with related methods. indicates the cited method exclusively applied to supervised MT. and it is adapted to unsupervised MT using multilingual iterative back-translation products as training samples.
Method CRISS
En-Ne 5.5
Ne-En 14.5
En-Si
Si-En
En-Gu
Gu-En
6.0
14.5
14.2
23.7
CRISS finetuned with
Sen al. [22]
7.7
17.0
7.0
14.5
15.7
26.8
Sachan and Neubig 21
7.1
16.9
7.2
14.0
15.1
26.1
Zhang et al. 29
6.8
16.1
6.4
14.0
14.9
25.2
Zuo et al. 30
8.2
17.4
7.8
14.6
16.4
25.6
Ours
9.0
18.2
9.5
15.3
17.5
29.5
Table 5: Comparison with popular UMT techniques.
Table 6:
Back-translated pseudo-
parallel dataset sizes.
Method
En-Ne
Ne-En En-Si
Si-En
En-Gu
Gu-En 23.7
CRISS
5.5
14.5
6.0
14.5
14.2
Mined data size
En-Ne
En-Si
CRISS finetuned with
Unfiltered Filtered
100..8
~088
CBD
7.6
16.9
7.3
14.6
15.8
26.0
~3000
~6000
Mined data
4.6
10.5
4.8
9.8
11.2
19.2
Mined data + BT
6.6
16.1
6.7
13.1
14.7
24.5
Ours
9.0
18.2
9.5
15.3
17.5
29.5
4.4 Limit of other unsupervised techniques
Apart from iterative back-translation, there are other techniques that have been shown improve unsupervised machine translation. One of them is cross-model back-translated distillation [18]. or CBD, where distinct UMT teachers used to distill the final model. Another is pseudo-paralla data mining [26] 19 where language- aggossic representations are built for sentences across different languages, which are then used to map unlabeled sentences from one language another create synthetic parallel data (2). CRISS is also itself meeuoo-paalleel data mining method. In this segment, we demonstrate that the aforementioned techniques not adapt well with low-resource unsupervised tasks, which shifts our attention to other areas, like the curse of multilinguality Specifically, we apply CBD on CRISS by finetuning two distinct models from pre-trained CRISS and using them to distill final model. For pseudo- parallel mining, we use LASER (2) to mine ceudo-paraalel data in only the low-resource directions of interest using the encoder outputs from CRISS itself.
The results are reported in Table 5 As shown, CBD only performs relatively similar to our stage 1. This indicates that the CBD only improves due Nonetheless, CBD faces disadvantage that the two teachers are not considerably distinct as they are both finetuned from the same pre-traue traited CRISS model, which is not advisable the original paper. This is nonideal as there is no existing well-performing pre- pre-trained model other than CRISS.
and not its own distillation effect.
Meanwhile, the mined pseudo- aeeudo-parallel data contributes little the performance improvement due to the reality that the amount and quality of mined data is too low for low-resource pairs, which is less than 5000 pairs for En-Ne and En-Si. This forces the model to be trained on such small mined datasets by progressively decreasing loss weight to prevent overfitting. While many may attribute this the model's failure in mining more high-quality pseudo-parallel data, empirically show that it is more likely that there are in fact not enough real parallel data in low-resource corpora to be mined! Specifically, we use our outperfoming model to back-translate the entire English monolingual corpus to Nepali. For each resulting back-trans -ranssltedd Ne sentence, we search for its closest real Ne sentence by token-based Levenshtein distance [15] in the Ne monolingual corpus, and then filter pairs whose distance is more than 20% the sentence length. We repeat the process for En-Si well. As shown in Table out of >100M possible unfiltered pairs, only <6000 samples satisfy the filtering criterion. We provide an in-depth analysis on this issue in the Appendix.

---

Conclusion
We proposed a simple refinement procedure to alleviate the curse of multilinguality low-resource unsupervised machine translation. Our method achieves the state of the in the FLoRes low-resource task of Beglish-Nepall and Inglish-Sinhala, as well as other language low-resource pairs across 10 language directions in Indic, Uralic and Turkic families. Further analyses show that the method outperforms other related alternatives when adapting to low-resource unsupervised translation.
References
[1] Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, et al. Massively multilingual neural machine translation in the wild: Findings and challenges. arXiv preprint 2019.
[2] Mikel Artetxe and Holger Schwenk. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. Transactions of the Association for Computational Linguistics. 7:597-610, March 2019.
doi: 10.1 (0.1162/tacl_a_000 URL https: / ogy
o org/019-1038
[3] Mikel Artetxe, Gorka Labaka, Encko Agirre, and Kyunghyun Cho. Unsupervised neural machine translation. In International Conference on Learning Representations, 2018. URL
https://openreviiee net. wewt:/formmiicco For.forrunnnddyyyyebbaaa
[4] Guanhua Chen, Shuming Ma, Yun Chen, Li Dong, Dongdong Zhang, Jia Pan, Wenping Wang. and Furu Wei. Zero-shot cross-lingual transfer of neural machine translation with multilingual pretrained encoders. In Proceedings of the 2021 Conference on Empirical Methods Natural Language Processing, pages 1 15-26, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/2021 121.66//1122211.mmmmmm...... emnll-main.22 URL
https: //acl antho ogy or..rr///......... 2021 emn 1-mmiin.2
[5] Guanhua Chen, Shuming Ma, Yun Chen, Dongdong Zhang, Jia Pan, Wenping Wang, and Furu Wei. Towards making the most of multilingual pretraining for zero-shot neural machine translation. arXiv preprint arXiv:2 10.08547, 2021.
[6] Alexis Conneau and Guillaume Lample. Cross-lingual language model pretraining. In H. Wallach, H. Larochelle, A. Beygelzimer, F. dÁlché-Bue Buc. E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 7059-7069. Curran Associates. Inc., 2019.
[7] Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pages 8440 8451, Online, July 2020. Association for Computational Linguistics. doi: 10.18865/vv1 122030/1220..mmmmmm https://aclantholl..
URL
org/ 2020. acl.main.. 747
[8] Xavier Garcia, Pierre Foret, Thibault Sellam, and Ankur Parikh. multilingual view of unsupervised machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3160-3170, Online, November 2020. Association for Computational Linguistics. doi: 10. 10.18653/112220 1tt.://55//11222..nddn.di. www...innnnnnggee..22...... URL
erg/2020. wtw.::00...ndddng--emmmm.... 283
[9] Xavier Garcia, Aditya Siddhant, Orhan Firat, and Ankur Parikh. Harnessing multilinguality in unsupervised machine translation for rare languages. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1126-11311 Online, June 2021. Association for Computational Linguistiess. doi: 10. 18653/v1/2221 htw.//lllln.aalllemeeeaaaaaaalaa main.89 URL https ac anthology.org 2021 maaci-main 89
10

---

[10] Francisco Guzmán, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample. Philipp Koehn, Vishrav Chaudhary, and Marc' Aurelio Ranzato. The FLORES evaluation datasets for low- resource machine translation: Nepali- and Sinhala-Fngliste In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Hong Kong. China, November 2019, Association for Computational Linguistics. doi: 18.186/3/vI/
Joint Conference on Natural Language Processing (EMNLP-JJCNLP)) pages
119-1632. BtLhhtts:://
www. acwweb. wwg.:/eeebooooeoe.ooe..e ottg/anthooogy/111 1632
[11] Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc' Aurelio Ranzato. Unsupervised machine translation using monolingual corpora only. International Conference on Learning Representations, 2018. URL https: https:://ppnreviiee net Forum?
IKYTTf-A2
[12] Guillaume Lample, Alexis Conneau, Marc Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. Word translation without parallel data. In International Conference on Learning Representations, 2018.
[13] Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Mare' Aurelio Ranzato. Phrase-based & neural unsupervised machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5039-5049, 5049, Brussels, Belgium,
2018. Association for Computational Linguistiss. doi: 10.18653/v1/D18. 15449. Utthttps:: www. ac acc..ebb..r///attoooooo//.......
[14) Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006. 16668, 2020.
[15] Vladimir I Levenshtein al. Binary codes capable of correcting deletions, insertions, and reversals. In Soviel physics doklady, volume 10, pages 707-710. Soviet Union, 1966.
[16] Xian Li and Hongyu Gong. Robust optimization for multilingual translation with imbalanced data. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL https: owtp://ippereeve...ommoo / n
Forum? iieetidd--- -d= JDUSJ
[17) Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. Multilingual denoising pre-training for neural machine translation. arXiv preprint arXiv:2001.0822 2020.
[18] Xuan-Phi Nguyen, Shafiq Joty. Thanh-Tung Nguyen, Kui Wu, and Ai Ti Aw. Cross-model back-translated distillation for unsupervised machine translation. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8073-8083, PMLR, 18-2 24 Jul 2021. URL htttps::pproceedinngss www....sm....r...m..........mm.m pttp....eeee/////////////ooe///
[19] Xuan-Phi Nguyen, Hongyu Gong, Yun Tang, Changhan Wang, Philipp Koehn, and Shafiq Joty. Contrastive clustering to mine pseudo parallel data for unsupervised translation. In International Conference on Learning Representations, 2022. URL
[20] Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. Scaling neural machine translation.
In Proceedings of the Third Conference on Machine Translation (WMT), 2018.
[21] Devendra Sachan and Graham Neubig. Parameter sharing methods for multilingual self- attentional translation models. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 261-271, Brussels, Belgium, October 2018. Association for Computational Linguistics. doi: 10. 11111663/11/11166330
accll antho
633//
[22] Sukanta Sen, Kamal Kumar Gupta, Asif Ekbal, and Pushpak Bhattacharyya. Multilingual unsupervised NMT using shared encoder and language-specific decoders. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3083-3089, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 18.13v/3/v P19-1297, URL https: httpp//:///aaaahhooaso//ooo wttp//g----- 1297

---

[23] Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Anmual Meeting of Association for Computational Linguistics (Volume 1: Long Papers), pages 86-96, Berlin, Germany, August 2016. Association for Computational Linguisties. doi: 10.18633/1//11610000 URL https
7 / www. ac web org/ woythooooyyy...o 6- 1-01///6661009
[24] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv: /wX.v:/00.... 10001000.....3. 2017.
[25] Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Liu. Mass: Masked sequence to sequence pre-training for language generation. arXiv preprint arXiv:1905.0244 02450, 2019.
[26] Chau Tran, Yuqing Tang, Xian Li, and Jiatao Gu. Cross-lingual retrieval for iterative self- supervised training. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Newral Information Processing Systems, volume 33, pages 2207-2219. Curran Associates, Inc., 2020. URL https::/ 72dd 2eddd//ee6607332ddaaaaeaa.. Pttp:/dddaaaaaaaPaappeppppe pdf
www.neuurip... cct cc/paper paper/
3020/F11e/1763eat 7et
[27] Chau Tran, Shruti Bhosale, James Cross, Philipp Koehn, Sergey Edunov, and Angela Fan. Facebook ai wmt21 news translation task submission. arXiv preprint arXiv:2/08.0322 2021.
[28] Lijun Wu, Jinhua Zhu, Di He, Fei Gao, Tao Qin, Jianhuang Lai, and Tie-Yan Liu. Machine translation with weakly paired documents. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP) pages 4375 4384, Hong Kong, China, November 1446. htthttps?
2019. Association for Computational Linguistics. doi: 10.18653/v
//aclantho hww.laanttooooo.oogggo.... 1446
[29] Biao Zhang, Ankur Bapna, Rico Sennrich, and Orhan Firat. Share or not? learning to schedule language-specific capacity for multilingual translation. In International Conference on Learning Representations, 2021. URL
[30] Simiao Zuo, Xiaodong Liu, Jian Jiao, Young Jin Kim, Hany Hassan, Ruofei Zhang, Tuo Zhao, and Jianfeng Gao. Taming sparsely activated transformer with stochastic experts. arXiv preprini arXiv:2110.04226, 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Our method trades multilinguality off for superior individual performance. Our models are no longer one-for-all multilingual model, but instead bilingual models in the to from-En directions and
multilingual in the to-En directions.
(c) Did you discuss any potential negative societal impacts of your work? [N/A] We are not aware of any potential negative societal impact of this work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or URL.)? [Yes]
12

---

(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No]
4. If you are using existing assets (c.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "oSdywGXasRA", "reviewer": "Reviewer_hAiK", "summary": "This paper presents a method to improve unsupervised neural machine . They propose to disentangle languages learned by multilangual language modes (eg. mBART). It is done through 4 different steps all explained in dedicated subsections. The method is evaluated on several language pairs from different language families, but always involving English. The paper claims to achieve the state-of-the-art on some of them.\n\nThe proposed method is very interesting, intuitive, innovative, and well-motivated. I especially find the step one very clever, though a bit complicated, giving each language pair its own set of layers, fine-tuned on individual GPUs. It is well engineered to remain efficient, and the authors even discussed the case where only 1 GPU is available. It is clear why it should work to improve machine translation for these language pairs.\n\nHowever, the evaluation is extremely sloppy... Most of the comaprisons with previous work is done without checking whether all the BLEU scores compared are truly comparable. Actually, they are not, and this makes half of the paper incorrect. \nI recommend to reject the paper unless the authors can convince us that all (not juste one or two) the systems compared used: the same preprocessed data, the same reference translations, the exact same tokenizer, and the same multi-bleu.perl. This is still surprising to see this kind of evaluation in 2022, especially for authors using the script multi-bleu.perl that clearly warns when executed to not publish multi-bleu scores. Why not using sacreBLEU? or, even better, COMET?\nI recommend the authors to read Marie et al. (2021)'s meta-evaluation of MT to fully understand what make their evaluation not credible.\n\nThe proposed method is very interesting an I enjoyed reading the paper, but please correct your evaluation.", "questions": "1. it would be nice to have one figure presenting the 4 first steps and their connection to help the reader understands how it works.\n2. Section 4.1: where did you get the common-crawl data exactly and how did you extract it? Is this the same common-crawl data used by all the baselines systems? If not, how can you be sure, following the scientific methodology, that the improvement comes from your method and not from the data?\n3. How does the proposed method compares to 'Multilingual Unsupervised Neural Machine Translation with Denoising Adapters'? It should be your baseline.\n\nTo be clear, if you correct the evaluation and add Bérard et al. (2021) as baseline (or at least explain why it is not a relevant baseline), I will change my score to a 7 or 8. For now, I can only recommend to reject the paper.", "limitations": "- yes the authors addresses the limitations of their work in a dedicated subsections.", "rating": 7, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n- well-written\n- perfectly motivated: I fully understand why the authors worked on this method\n- well-engineered: the proposed architecture seems to remain efficent with some clever engineering tricks\n- probably works to improve machine translation quality for the targeted language pairs\n\nWeaknesses:\n\n- the evaluation is incorrect. BLEU scores from different papers using different data, tokenization, etc are compared.\n- the method looks complicated (but is not complicated), mostly due to its decomposition into 4 different steps that are not that easy to connect. I think it should and can be reformulated as a one single step method.\n- a very close work is not cited: Multilingual Unsupervised Neural Machine Translation with Denoising Adapters (Bérard et al, 2021) and should probably be the baseline of this paper. It aims at achieving the same goal but through a different method. Bérard et al, (2021) looks much simpler.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "a44DXj7Zyp_", "reviewer": "Reviewer_5Niy", "summary": "The paper proposes a 4-stage refinement of a multi-lingual unsupervised NMT model into direct NMT models for low-resource languages, in a restricted setting when there is zero parallel data. While the procedure and its implementation is rather complicated, it delivers large BLEU gains.  \n", "questions": "- 327: in what way LASER uses the encoded sentences from CRISS?\n- If possible, please consider replacing \"disentanglement\" with \"refinement\" or other word for the final version, since in ML disentanglement has a specific meaning of learning causal factors in latent spaces of generative models. \n", "limitations": "Focus on unrealistic setup with exactly zero parallel data available for all languages", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- High improvements in the zero-parallel-data setting \n- Solid comparisons to SOTA and ablation experiments\n- Very well written paper\n\nWeaknesses:\n- While the authors make sure to carefully delineate their work from UMT that uses some quasi-parallel data, this reduces the value of the work by constraining it into an unrealistic setup -- in practice, even if public corpora like CC have too little parallel data to be mined, there are always e.g. religious texts that have more than 6000 sentences. Along the same lines, a more relevant to practice comparison would be to compare systems with existing parallel texts added.\n- The paper may be a weak fit for NeurIPS, as it focuses on a niche MT topic\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "NYIJ9bRPQkj", "reviewer": "Reviewer_me6V", "summary": "This paper presents a 4 stage training regime for multilingual unsupervised machine translation. The key idea is to separate the training of a particular component of the  model for each language, and to separate the model training for the into and out of English models. \n", "questions": "Spellcheck please:\n4: where theses low-resource languages \n", "limitations": "They discuss the limitations of work that rely on data mining, but do not explicitly address limitations of their own work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nThey show good improvements over SOTA on 6 low-resource langauge pairs.\nThey compare with numerous baselines and do ablation studies. \n\nWeaknesses\n\nComplicated strategy involving four stages. \nUnsupervised machine translation is an interesting theoretical exercise but in practise you would always use any limited parallel data that you had access to and so it would have been more useful if they had shown their methods benefit low-resource results instead of the unsupervised results. \nThe authors do not explain their baseline model  LAgSwAV - it is important to understand what we are improving over. They don't make it clear if these baselines (CRISS and LAgSwAV) and their own approach are trained on just the languages shows in their languages families (ie. 3, 2, and 1 langauge pairs) or whether they were trained on the data from all three language groups (ie. 6 language pairs) or on all the data in Commoncrawl 25 langauges. It is therefore hard to interpret the results. \nI don't like to have to look at the appendix to see the training data - this is a core part of the paper - but even looking the appendix it was not clear what model was trained on what data. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ADeg3UIe2zN", "reviewer": "Reviewer_yLsx", "summary": "This work proposes a low resource machine translation in unsupervised setting in which languages in the target side is explicitly disentangled from a multilingual model. Basic idea is to employ target language specific feed-forward layers in the decoder side in order to isolate the language specific representation. The model is trained in multiple stages:\n1. Starting from a multilingual model trained in an unsupervised way, e.g., CRISIS, the model is trained jointly by augmenting training data using back-translation.\n2. The model is split into two, English-to-X and X-to-English, and they are trained individually starting from the parameters learned in the first step. This step is iterated twice.\n3. The model for English-to-X is further split into target language specific models so that the representation is further isolated in the target side.\n\nThis procedure results in a single X-to-English model and multiple English-to-X models. Experimental results on standard unsupervised machine translation benchmark shows significant gains when compared with multilingual machine translation baselines.", "questions": "* Table 2 is strange in that no training happens in stage 4 for X-to-English direction, but BLEU scores are slightly different. I feel something wrong in the description in this submission or a bug in experiments. Or, there's somethings I've completely missed.", "limitations": "* The proposed method does not train a multilingual model in English-to-X directions, and, thus, spuriously many parameters must be learned for each direction.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* The idea itself is not novel in that there exists prior work on isolating target language specific representation by feed-forward network in the decoder side. However, its application for unsupervised setting is a new contribution.\n* The training algorithm is carefully designed to empirically avoid poor local optima by splitting into multiple stages.\n* This work also proposes efficient training by exploiting multiple GPUs so that the parameters for language specific feed-forward layers do not have to be shared among multiple devices.\n\nWeaknesses\n* ~~The efficient training strategy is applied only in the first stage of training, i.e., training a single multilingual model with target language specific parameters.~~ Thanks for the clarification, but it is applicable when multiple languages are involved, and loosing its merit in the later steps when tuning for bilingual model.\n* It is not clear whether the comparison is meaningful in that the proposed model is basically learning language pair specific model for English-to-X direction. Other models, e.g., CRISSS, are basically multilingual models so that a single model is employed to translate multiple directions. This work should be compared with other bilingual baselines that rely on unsupervised method. Or, it would be meaningful to iterate the first stage multiple times to see if there exists any gains without explicitly separating parameters.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept