Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work proposes a low resource machine translation in unsupervised setting in which languages in the target side is explicitly disentangled from a multilingual model. Basic idea is to employ target language specific feed-forward layers in the decoder side in order to isolate the language specific representation. The model is trained in multiple stages:
1. Starting from a multilingual model trained in an unsupervised way, e.g., CRISIS, the model is trained jointly by augmenting training data using back-translation.
2. The model is split into two, English-to-X and X-to-English, and they are trained individually starting from the parameters learned in the first step. This step is iterated twice.
3. The model for English-to-X is further split into target language specific models so that the representation is further isolated in the target side.

This procedure results in a single X-to-English model and multiple English-to-X models. Experimental results on standard unsupervised machine translation benchmark shows significant gains when compared with multilingual machine translation baselines.
strengths_and_weaknesses: Strengths
* The idea itself is not novel in that there exists prior work on isolating target language specific representation by feed-forward network in the decoder side. However, its application for unsupervised setting is a new contribution.
* The training algorithm is carefully designed to empirically avoid poor local optima by splitting into multiple stages.
* This work also proposes efficient training by exploiting multiple GPUs so that the parameters for language specific feed-forward layers do not have to be shared among multiple devices.

Weaknesses
* ~~The efficient training strategy is applied only in the first stage of training, i.e., training a single multilingual model with target language specific parameters.~~ Thanks for the clarification, but it is applicable when multiple languages are involved, and loosing its merit in the later steps when tuning for bilingual model.
* It is not clear whether the comparison is meaningful in that the proposed model is basically learning language pair specific model for English-to-X direction. Other models, e.g., CRISSS, are basically multilingual models so that a single model is employed to translate multiple directions. This work should be compared with other bilingual baselines that rely on unsupervised method. Or, it would be meaningful to iterate the first stage multiple times to see if there exists any gains without explicitly separating parameters.
questions: * Table 2 is strange in that no training happens in stage 4 for X-to-English direction, but BLEU scores are slightly different. I feel something wrong in the description in this submission or a bug in experiments. Or, there's somethings I've completely missed.
limitations: * The proposed method does not train a multilingual model in English-to-X directions, and, thus, spuriously many parameters must be learned for each direction.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a method to improve unsupervised neural machine . They propose to disentangle languages learned by multilangual language modes (eg. mBART). It is done through 4 different steps all explained in dedicated subsections. The method is evaluated on several language pairs from different language families, but always involving English. The paper claims to achieve the state-of-the-art on some of them.

The proposed method is very interesting, intuitive, innovative, and well-motivated. I especially find the step one very clever, though a bit complicated, giving each language pair its own set of layers, fine-tuned on individual GPUs. It is well engineered to remain efficient, and the authors even discussed the case where only 1 GPU is available. It is clear why it should work to improve machine translation for these language pairs.

However, the evaluation is extremely sloppy... Most of the comaprisons with previous work is done without checking whether all the BLEU scores compared are truly comparable. Actually, they are not, and this makes half of the paper incorrect. 
I recommend to reject the paper unless the authors can convince us that all (not juste one or two) the systems compared used: the same preprocessed data, the same reference translations, the exact same tokenizer, and the same multi-bleu.perl. This is still surprising to see this kind of evaluation in 2022, especially for authors using the script multi-bleu.perl that clearly warns when executed to not publish multi-bleu scores. Why not using sacreBLEU? or, even better, COMET?
I recommend the authors to read Marie et al. (2021)'s meta-evaluation of MT to fully understand what make their evaluation not credible.

The proposed method is very interesting an I enjoyed reading the paper, but please correct your evaluation.
strengths_and_weaknesses: Strength:

- well-written
- perfectly motivated: I fully understand why the authors worked on this method
- well-engineered: the proposed architecture seems to remain efficent with some clever engineering tricks
- probably works to improve machine translation quality for the targeted language pairs

Weaknesses:

- the evaluation is incorrect. BLEU scores from different papers using different data, tokenization, etc are compared.
- the method looks complicated (but is not complicated), mostly due to its decomposition into 4 different steps that are not that easy to connect. I think it should and can be reformulated as a one single step method.
- a very close work is not cited: Multilingual Unsupervised Neural Machine Translation with Denoising Adapters (Bérard et al, 2021) and should probably be the baseline of this paper. It aims at achieving the same goal but through a different method. Bérard et al, (2021) looks much simpler.
questions: 1. it would be nice to have one figure presenting the 4 first steps and their connection to help the reader understands how it works.
2. Section 4.1: where did you get the common-crawl data exactly and how did you extract it? Is this the same common-crawl data used by all the baselines systems? If not, how can you be sure, following the scientific methodology, that the improvement comes from your method and not from the data?
3. How does the proposed method compares to 'Multilingual Unsupervised Neural Machine Translation with Denoising Adapters'? It should be your baseline.

To be clear, if you correct the evaluation and add Bérard et al. (2021) as baseline (or at least explain why it is not a relevant baseline), I will change my score to a 7 or 8. For now, I can only recommend to reject the paper.
limitations: - yes the authors addresses the limitations of their work in a dedicated subsections.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a 4-stage refinement of a multi-lingual unsupervised NMT model into direct NMT models for low-resource languages, in a restricted setting when there is zero parallel data. While the procedure and its implementation is rather complicated, it delivers large BLEU gains.  

strengths_and_weaknesses: Strengths:
- High improvements in the zero-parallel-data setting 
- Solid comparisons to SOTA and ablation experiments
- Very well written paper

Weaknesses:
- While the authors make sure to carefully delineate their work from UMT that uses some quasi-parallel data, this reduces the value of the work by constraining it into an unrealistic setup -- in practice, even if public corpora like CC have too little parallel data to be mined, there are always e.g. religious texts that have more than 6000 sentences. Along the same lines, a more relevant to practice comparison would be to compare systems with existing parallel texts added.
- The paper may be a weak fit for NeurIPS, as it focuses on a niche MT topic

questions: - 327: in what way LASER uses the encoded sentences from CRISS?
- If possible, please consider replacing "disentanglement" with "refinement" or other word for the final version, since in ML disentanglement has a specific meaning of learning causal factors in latent spaces of generative models. 

limitations: Focus on unrealistic setup with exactly zero parallel data available for all languages
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper presents a 4 stage training regime for multilingual unsupervised machine translation. The key idea is to separate the training of a particular component of the  model for each language, and to separate the model training for the into and out of English models. 

strengths_and_weaknesses: Strengths

They show good improvements over SOTA on 6 low-resource langauge pairs.
They compare with numerous baselines and do ablation studies. 

Weaknesses

Complicated strategy involving four stages. 
Unsupervised machine translation is an interesting theoretical exercise but in practise you would always use any limited parallel data that you had access to and so it would have been more useful if they had shown their methods benefit low-resource results instead of the unsupervised results. 
The authors do not explain their baseline model  LAgSwAV - it is important to understand what we are improving over. They don't make it clear if these baselines (CRISS and LAgSwAV) and their own approach are trained on just the languages shows in their languages families (ie. 3, 2, and 1 langauge pairs) or whether they were trained on the data from all three language groups (ie. 6 language pairs) or on all the data in Commoncrawl 25 langauges. It is therefore hard to interpret the results. 
I don't like to have to look at the appendix to see the training data - this is a core part of the paper - but even looking the appendix it was not clear what model was trained on what data. 


questions: Spellcheck please:
4: where theses low-resource languages 

limitations: They discuss the limitations of work that rely on data mining, but do not explicitly address limitations of their own work.
review_title: null

Meta-review: metareview: This paper presents a four-stage process for training completely unsupervised machine translation models. The results are fairly strong. After some discussion all reviewers are convinced that the evaluation is sound.
The reviewers are somewhat split about the novelty, some say:
+ "The proposed method is very interesting, intuitive, innovative, and well-motivated."
Others say:
- "The idea itself is not novel in that there exists prior work on isolating target language specific representation by feed-forward network in the decoder side."
Some reviewers find the approach with its four steps cumbersome, while others don't have an issue with this. 
On balance, just above the decision boundary.
confidence: Certain
Decision: Accept