Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper performs an ablative study on the two components involved in training unsupervised MT systems: 1) back-translation loss, 2) denoising autoencoding loss. It links the reconstruction loss to ELBO (where the q distribution is a back-translation model). It shows that the original loss with both the components is important for unsupervised MT and ELBO needs to be augmented with denoising autoencoding loss to be effective at training unsupervised MT models.

-- The graphs show comparisons on ELBO across different models. However, ELBO is not really comparable across models by definition.

-- Many findings in the paper are unsurprising and add little to our current understanding of unsupervised MT systems. For example, DAE-only collapsing to copying and BT-only collapsing to language modeling is rather expected. This can be followed from the proposed conceptualization of mutual information in the paper. The MI is highest when Y' = X! Therefore, unregularized training (with just one of the loss components) is expected to result in degenerate behavior.

-- The only slightly surprising results is that ELBO on its own collapses and fails to learn anything. Technically, it does have the two components in its loss functions that make unsupervised MT work. The reasons could range from optimization issues to poor Monte-Carlo sample based approximation of expectations in ELBO (reparametrization trick or score matching is not explored in this paper for better approximation).
review_title: Official review

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper attempts to explain why UNMT using back-translation (BT) and denoising autoencoding (DAE) has been successful.

I generally found this paper difficult to follow. Both the language and logic of its arguments were often not very clear to me.

Section 2 presents some theoretical arguments, or analogies. 

Section 2.2 draws an analogy between UNMT and ELBO; the reconstruction-error terms are similar but the regularization terms are different.

Section 2.3 tries to connect UNMT to mutual information, but I didn’t understand why Definition 2.1 tries to define the “success” of UNMT in terms of MI (wouldn’t setting M to the identity function achieve maximal MI?).

Both of these analogies are very rough, and I feel that this section falls short of providing understanding of UNMT.

Section 3 presents some experiments to support the arguments made in Section 2.

Section 3.2 shows that UNMT minimizes the ELBO term that is similar to a term in the UNMT loss and doesn’t minimize the ELBO term that isn’t similar to a term in the UNMT loss. 

Section 3.3 argues that using only the BT loss will cause UNMT to degenerate to generating fluent target-language sentences, while using only the DAE loss will cause UNMT to degenerate to copying the source to the target.

Section 3.4.1 argues that BT is more important than DAE because UNMT performs best when BT is weighted 10 times more heavily than DAE.

Section 3.4.2 argues that DAE’s job is to ensure that the encoder does not lose information.

Overall, it seems to me that these findings agree with the intuitive understanding of UNMT that was already present in the UNMT literature, with the exception of the finding in 3.4.1 that the BT term should be given higher weight than the DAE term. 

While I think that explaining why UNMT works is an excellent research goal and there are some interesting ideas here, I do not think that this paper is ready for publication yet. 
review_title: Interesting, but sometimes unclear, discussion of unsupervised neural machine translation

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper aims to explain the learning of unsupervised neural machine translation(UNMT) theoretically from two perspectives: 
1)` marginal likelihood maximization, where the objective of UNMT is analogous to ELBO and it's two terms are visualized during the training of standard UNMT. 
2) information-theoretic view, where the authors derive the success of UNMT training as maximizing the mutual information between source and target translation (sufficient condition), as well as source input and its encoder output (necessary condition). Later in the experiments, the paper presents the MI during training to support their arguments.

Pros
1. This paper attempts to explain UNMT theoretically, which is a significant step for understanding UNMT inspiring the future direction of UNMT. 
2. The experiments are properly aligned with the theoretical arguments made in the paper and give good empirical explanations.


Cons:
1. It seems that we can't really evaluate if a model meets the real sufficient condition using the practical sufficient condition.
2. Although this paper provides propositions that what is a success UNMT training, it could've been better if showed how we can create a better UNMT system based on these definitions. 
3. ELBO-only protocol fails to train, which makes the analogy b/w *standard* and ELBO in the perspective of marginal likelihood maximization not pretty much hold thus less convincing. 

----

Questions:
1. fig 4: I don't really understand why dae-only always preserves a high(est) MI during training. The authors barely explained anything in terms of this.
2. though c in the def 2.1 & the sufficient condition is a conceptual quantity, to what extend we can consider a UNMT model is successfully learned? Even an intuitive thought is welcome.


------
Reason for score: This paper gives a theoretical view of understanding UNMT, which is a good contribution. Some of the analogy and propositions are not well supported by the empirical results. This paper could have given some future directions based on their theoretical understanding. 

review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper takes a closer look at the inner workings of unsupervised MT training.

The authors provide two alternate views on the backtranslation+DAE objectives used in unsupervised-MT. This interpretation sheds more light on the relationship between the two: for example, it appears that the DAE loss is critical to preserve mutual information between input and encodings, thus preventing the translation models from degenerating into unconditional language models. The authors' conclusions are well supported by experiments. However, I find that the paper falls short by being purely descriptive. While the many observations made by the authors are interesting, the reader is left hanging trying to figure out what to do with them. What is the point of these observations? Why are they useful? This paper is missing a second, more "prescriptive" half answering these questions.

**Pros**

- Sheds light on the role of the two loss terms
- Convincing experiments: the authors claims are well supported by their experiments

**Cons**

- Purely descriptive: what is the takeaway? How can we improve unsupervised MT based on these findings? Better model selection? No need for denoising if the autoencoder is trained less?
- The paper is very hard to read. I attribute this to the abundance of notations and acronyms, as well as the unwieldy phrasing. In addition, the progression of the experimental section is hard to follow. This could be remedied by summarizing and clarifying the contributions better. Right now, it reads like a list of facts about UNMT (this is also related to my first point)
- I am not 100% convinced that the mutual information between model prediction and input is a good measure of MT success. Wouldn't that make a model that just copies its input successful (if H(x)>c )? Could the authors provide more intuition on why that is a good metric?

**Remarks**
- Grammar errors and typos: there are a lot of grammar errors and typos. I understand that the authors may not be native speakers of English. Nevertheless, the amount of errors is high enough to be distracting. I highly encourage the authors to seek proof-reading. Some examples:
   * "why this standard training protocol is ~possible to be~ successful"
   * "some fundamental questions are still remained unknown" -> "some fundamental questions remain unanswered"
   *"~If~ We denote Y 0 = M(X) as the random variable (r.v.) generated by the model M over X."
   * "repoitory" -> "repository"
   * "BT LOSS IS THE MAIN TASK WHILE DAE LOSS THE AUXILIARY" -> "BT IS THE MAIN TASK WHILE DAE IS AUXILIARY"
- Style: I understand that this is a personal preference to some extent (which is why I didn't count it as a con), but the writing style is not very "academic": Examples: addressing the reader in the second person (2.2: "As you can see"), awkward/informal phrasing (1: "we present a formal definition on what does it mean to [...], "3: "As promised in Section [...]")
- There are vague statements that should either be made more precise or removed altogether. For example in Sec. 2.3: "Therefore, if (Y 0;X) gradually contains more and more bilingual knowledge," what does "containing more bilingual knowledge" mean for a pair of sentences?
- I didn't quite follow the point of the fine-tuning experiments in 3.1 (the +bt loss and +standard rows in table 3). Specifically, I'm not sure I understand what the authors mean by saying that this shows that bt-only "poisons" the model. What is meant by "poisoning"? I'm not sure how this fits in the overall narrative of the paper, and I found this observation confusing more than anything else.

---

Post rebuttal:

I thank the authors for responding to my questions. While some minor points are cleared up, I am still not completely satisfied with the rather vague notion of "bilingual knowledge" (what even is the "correct axiomatic translation correspondence", is it the (idealistic) true data distribution one tries to model?). Similarly, the use of the MI as a measure of success is still unsatisfying to me since it relies on a "trick" (mismatch of vocabulary between source and target) which might not even be relevant in practice (since most models use sub-words anyway).

Overall, I think this research direction is promising, but I keep my recommendation the same. The paper would greatly benefit from another round of revision to clear up these points and clarify the presentation if it is to be useful to the research community.
review_title: Interesting observations but unclear takeaways and confusing writing

Meta-review: This paper attempts to explain why popular UNMT training objective components (back-translation and denoising autoencoding) are effective. The paper provides experimental analysis and draws connections with ELBO and mutual information. Reviewers generally agree that the paper's goal is worthy: trying to form a better theoretical understanding of successful approaches to UNMT. 
However, most reviewers raised serious concerns about the current draft and suggested another round of revision and resubmission. Specifically, reviewers were concerned that some of the analogies used to explain UNMT are underdeveloped. Further, reviewers pointed to issues with clarity that made some of the arguments hard to follow. Finally, one reviewer argued that many of the results are expected and agree with common understanding of UNMT in the literature, thus undermining their value to some extent. I tend to agree with reviewers that this paper is not ready for publication in its current form. Thus I recommend rejection. 
Decision: Reject