Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper describes a first working approach for fully unsupervised neural machine translation. The core ideas being this method are: (1) train in both directions (French to English and English to French) in tandem; (2) lock the embedding table to bilingual embeddings induced from monolingual data; (3) share the encoder between two languages; and (3)  alternate between denoising auto-encoder steps and back-translation steps. The key to making this work seems to be using a denoising auto-encoder where noise is introduced by permuting the source sentence, which prevents the encoder from learning a simple copy operation. The paper shows real progress over a simple word-to-word baseline for WMT 2014 English-French and English-German. Preliminary results in a semi-supervised setting are also provided.

This is solid work, presenting a reasonable first working system for unsupervised NMT, which had never been done before now. That alone is notable, and overall, I like the paper. The work shares some similarities with He et al.’s NIPS 2016 paper on “Dual learning for MT,” but has more than enough new content to address the issues that arise with the fully unsupervised scenario. The work is not perfect, though. I feel that the paper’s abstract over-claims to some extent. Also, the experimental section shows clearly that in getting the model to work at all, they have created a model with a very real ceiling on performance. However, to go from not working to working a little is a big, important first step. Also, I found the paper’s notation and prose to be admirably clear; the paper was very easy to follow.

Regarding over-claiming, this is mostly an issue of stylistic preference, but this paper’s use of the term “breakthrough” in both the abstract and the conclusion grates a little. This is a solid first attempt at a new task, and it lays a strong foundation for others to build upon, but there is lots of room for improvement. I don’t think it warrants being called a breakthrough - lots of papers introduce new tasks and produce baseline solutions. I would generally advise to let the readers draw their own conclusions.

Regarding the ceiling, the authors are very up-front about this in Table 1, but it bears repeating here: a fully supervised model constrained in the same way as this unsupervised model does not perform very well at all. In fact, it consistently fails to surpass the semi-supervised baseline (which I think deserved some further discussion in the paper). The poor performance of the fully supervised model demonstrates that there is a very real ceiling to this approach, and the paper would be stronger if the authors were able to show to what degree relaxing these constraints harms the unsupervised system and helps the supervised one.

The semi-supervised experiment in Sections 2.3 and 4 is a little dangerous. With BLEU scores failing to top 22 for English-French, there is a good chance that a simple phrase-based baseline on the same 100k sentence pairs with a large target language model will outperform this technique. Any low-resource scenario should include a Moses baseline for calibration, as NMT is notoriously weak with small amounts of parallel data.

Finally, I think the phrasing in Section 5.1 needs to be softened, where it states, “... it is not possible to use backtranslation alone without denoising, as the initial translations would be meaningless sentences produced by a random NMT model, ...” This statement implies that the system producing the sentences for back-translation must be a neural MT system, which is not the case. For example, a related paper co-submitted to ICLR, called “Unsupervised machine translation using monolingual corpora only,” shows that one can prime back-translation with a simple word-to-word system similar to the word-to-word baseline in this paper’s Table 1.
review_title: A straightforward first step toward unsupervised NMT

final_score: rating: 5: Marginally below acceptance threshold
review_content: unsupervised neural machine translation

This is an interesting paper on unsupervised MT. It trains a standard architecture using:

1) word embeddings in a shared embedding space, learned using a recent approach that works with only tens of bilingual word papers.

2) A encoder-decoder trained using only monolingual data (should cite http://www.statmt.org/wmt17/pdf/WMT15.pdf). Training uses a “denoising” method which is not new: it uses the same idea as contrastive estimation (http://www.aclweb.org/anthology/P05-1044, a well-known method which should be cited). 

3) Backtranslation.

All though none of these ideas are new, they haven’t been combined in this way before, and that what’s novel here. The paper is essentially a neat application of (1), and is an empirical/ systems paper. It’s essentially a proof-of-concept that it is that it’s possible to get anything at all using no parallel data. That’s surprising and interesting, but I learned very little else from it. The paper reads as preliminary and rushed, and I had difficulty answering some basic questions:

* In Table (1), I’m slightly puzzled by why 5 is better than 6, and this may be because I’m confused about what 6 represents. It would be natural to compare 5 with a system trained on 100K parallel text, since the systems would then (effectively) differ only in that 5 also exploits additional monolingual data. But the text suggests that 6 is trained on much more than 100K parallel sentences; that is, it differs in at least two conditions (amount of parallel text and use of monolingual text). Since this paper’s primary contribution is empirical, this comparison should be done in a carefully controlled way, differing each of these elements in turn.

* I’m very confused by the comment on p. 8 that “the modifications introduced by our proposal are also limiting” to the “comparable supervised NMT system”. According to the paper, the architecture of the system is unchanged, so why would this be the case? This comment makes it seem like something else has been changed in the baseline, which in turn makes it somewhat hard to accept the results here.

Comment:
* The qualitative analysis is not really an analysis: it’s just a few cherry-picked examples and some vague observations. While it is useful to see that the system does indeed generate nontrivial content in these cases, this doesn’t give us further insight into what the system does well or poorly outside these examples. The BLEU scores suggest that it also produces many low-quality translations. What is different about these particular examples? (Aside: since the cross-lingual embedding method is trained on numerals, should we be concerned that the system fails at translating numerals?)

Questions:
* Contrastive estimation considers other neighborhood functions (“random noise” in the parlance of this paper), and it’s natural to wonder what would happen if this paper also used these or other neighborhood functions. More importantly, I suspect the the neighborhood functions are important: when translating between Indo-European languages as in these experiments, local swaps are reasonable; but in translating between two different language families (as would often be the case in the motivating low-resource scenario that the paper does not actually test), it seems likely that other neighborhood functions would be important, since structural differences would be much larger.

Presentational comments (these don’t affect my evaluation, they’re mostly observations but they contribute to a general feeling that the paper is rushed and preliminary):

* BPE does not “learn”, it’s entirely deterministic.

* This paper is at best tangentially related to decipherment. Decipherment operates under two quite different assumptions: there is no training data for the source language ciphertext, only the ciphertext itself (which is often very small); and the replacement function is deterministic rather than probabilistic (and often monotonic). The Dou and Knight papers are interesting, but they’re an adaptation of ideas rather than decipherment per se. Since none of those ideas are used here this feels like hand-waving.

* Future work is vague: “we would like to detect and mitigate the specific causes…” “we also think that a better handling of rare words…” That’s great, but how will you do these things? Do you have specific reasons to think this, or ideas on how to approach them? Otherwise this is just hand-waving.
review_title: Interesting but preliminary proof-of-concept

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors present a model for unsupervised NMT which requires no parallel corpora between the two languages of interest. While the results are interesting I find very few original ideas in this paper. Please find my comments/questions/suggestions below:

1) The authors mention that there are 3 important aspects in which their model differs from a standard NMT architecture. All the 3 differences have been adapted from existing works. The authors clearly acknowledge and cite the sources. Even sharing the encoder using cross lingual embeddings has been explored in the context of multilingual NER (please see https://arxiv.org/abs/1607.00198). Because of this I find the paper to be a bit lacking on the novelty quotient. Even backtranslation has been used successfully in the past (as acknowledged by the authors). Unsupervised MT in itself is not a new idea (again clearly acknowledged by the authors).

2) I am not very convinced about the idea of denoising. Specifically, I am not sure if it will work for arbitrary language pairs. In fact, I think there is a contradiction even in the way the authors write this. On one hand, they want to "learn the internal structure of the languages involved" and on the other hand they deliberately corrupt this structure by adding noise. This seems very counter-intuitive and in fact the results in Table 1 suggest that it leads to a drop in performance. I am not very sure that the analogy with autoencoders holds in this case.

3) Following up on the above question, the authors mention that "We emphasize, however, that it is not possible to use backtranslation alone without denoising". Again, if denoising itself leads to a drop in the performance as compared to the nearest neighbor baseline then why use backtranslation in conjunction with denoising and not in conjunction with the baseline itself. 

4) This point is more of a clarification and perhaps due to my lack of understanding. Backtranslation to generate a pseudo corpus makes sense only after the model has achieved a certain (good) performance. Can you please provide details of how long did you train the model (with denoising?) before producing the backtranslations ?

5) The authors mention that 100K parallel sentences may be insufficient for training a NMT system. However, this size may be decent enough for  a PBSMT system. It would be interesting to see the performance of a PBSMT system trained on 100K parallel sentences. 

6) How did you arrive at the beam size of 12 ? Was this a hyperparameter? Just curious.

7) The comparable NMT set up is not very clear. Can you please explain it in detail ? In the same paragraph, what exactly do you mean by "the supervised system in this paper is relatively small?"
review_title: Very few original ideas

Meta-review: This work presents new results on unsupervised machine translation using a clever combination of techniques. In terms of originality, the reviewers find that the paper over-claims, and promises a breakthrough, which they do not feel is justified.
However there is "more than enough new content" and "preliminary" results on a new task. The experimental quality also has some issues, there is a lack of good qualitative analysis, and reviewers felt the claims about semi-supervised work had issues. Still the main number is a good start, and the authors are correct to note that there is another work with similarly promising results. Of the two works, the reviewers found the other more clearly written, and with better experimental analysis, noting that they both over claim in terms of novelty. The most promising aspect of the work, will likely be the significance of this task going forward, as there is now more interest in the use of multi-lingual embeddings and nmt as a benchmark task. 
Decision: Accept (Poster)