Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a joint training strategy that combines supervised learning on parallel data and self-supervised learning on monolingual data for NMT. The monolingual sentences are corrupted with word order shuffling and masking. And a cross encoder-decoder is introduced to fuse the parallel source-side sentence and the corrupted monolingual sentence. The NMT models are optimized jointly with cross-entropy loss of parallel data, reconstruction loss for monolingual data, and cross-entropy of the virtual fused data.

The proposed method is simple and straightforward, with seemingly good results for rich-resource languages; can be easily reproduce in any existing toolkits.

Main results (table 2) should compare with BT and noised BT (Edunov et al. 2018) using the same amount of monolingual data as well. Current comparisons with previous BT/pre-training approaches in the main results (table 2) are not exactly fair, since lots of them are not leveraging large-scale monolingual data of all available newscrawl combined. (correct me if I'm wrong). 

I'm also a little skeptical about the fusion of source-side sentence x and target-side sentence y* into the input sequence \tilde x. Would this potentially hampers self-attention in the encoder or the enc-dec attention? Empirically, the last row of table 3 (+BT) shows that replacing n(y) with BT sentence x' leads to significantly better performance, which might be caused by this? 
This also leads to the question, what is the best fusing scheme (i.e. input (x, y), (x', y')) for the proposed method? Is this the most data efficient way of combing parallel and mono data compared with previous objectives (e.g. noisy BT, DAE, MASS, etc.)?

How is the training and convergence speed for the proposed method, compared with standard Transformer, with BT/noised BT with same amount of mono data?
review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper presents a method, inspired from genetics algorithms, to trained jointly self-supervised and supervised NMT. The claims are this new method and state-of-the-art BLEU scores for En-Fr WMT14


Strengths:
- the method inspired from genetics is original and interesting
- the method improves over vanilla Transformer
- the analysis is interesting by pointing out the key parameters of the proposed method


Weaknesses
- the main weakness of this paper is by far its evaluation: the comparisons of BLEU scores presented in Table 2 and 3 are meaningless. For instance, Zhu et al. (2020) uses sacreBLEU while this paper use multi-bleu. On En-Fr, it is well-known that the differences between these different BLEU may be within 5 BLEU points (!). It is then impossible to assess whether this work is really better than previous work from these results. sacreBLEU must be reported instead of, or in addition to, multi-bleu scores. Moreover, the work compared in Table 2 used different training data and different pre-processing, the difference in BLEU may then only come from these data differences and not from the frameworks. I actually expect this method to be worse than Zhu et al. (2020) considering that their sacreBLEU scores is less than 2 BLEU points below the multi-bleu scores reported in this paper. The second half of the paper has to be rewritten almost entirely to make a meaningful evaluation of this approach.
- the motivation for using cross-over encoder-decode is difficult to understand, probably because it is only done very briefly in Section 1 (I would recommend to make separate section for motivation)

Questions:
- Section 4.1: how reporting tokenized BLEU scores is fair while previous work, mentioned in Table 2, report sacreBLEU scores? You probably misread Section 5.1 from Zhu et al. (2020). For Nguyen et al. (2019), this is unclear what are the reported BLEU scores, but for xx-to-English it seems to be also sacreBLEU scores. Ott et al (2018) also report sacreBLEU scores in addition to multi-bleu. I think you have all you need to replace the scores in Table 2 by sacreBLEU scores. It would dramatically improve the paper.
- Some of the methods compared in Table 2 uses different sets of training data and preprocessing, maybe I misunderstood something, but even with comparable BLEU scores, how can we assess that this is the proposed method that is better and not the training data or preprocessing? To make a scientifically valid conclusion I think there is no other way than reproducing previous work. There is no need to reproduce all the work from Table 2.  I think the most relevant one to reproduce is probably the work of Zhu et al. (2020).
review_title: An interesting method that remains to be fairly evaluated

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:

This submission proposes a joint training of self-supervised training and supervised training for neural machine translation (NMT), especially for the rich-resource language datasets. The method proposed, F$_2$-XEnDec, exploits the "crossover" operation of the monolingual sentences and bilingual data pairs in the encoder-decoder framework, and train the self-supervised (on monolingual data) objective and supervised (on bilingual data) objective, and the mixed version (crossover) together, without a clear per-train and fine-tuning stage like previous BERT related works. Experiments are conducted on two rich-resource language pairs: WMT14 English-German and WMT14 English-French translation tasks. The results set a new state-of-the-art record for English-French, and superior performances are achieved on English-German language pairs.  The authors also analyze the robustness and ablation studies of their method. 

Overall comments:

The method proposed in this submission is interesting, the crossover operation conducted on the monolingual dataset is similar to the previous works like MASS and BART, but different in the bilingual sentences mixed version. The authors give clear descriptions and comparisons with previous methods. Another difference is that previous works only adopt pre-train and fine-tune stages separately, while this method is to jointly train the self-supervised and supervised objectives, on both monolingual data and bilingual dataset. The crossover operation of the source sentence is natural to be straight forward, while it is interesting for the target side sentences, both on monolingual and target sentences. The most impressive part is the experimental results, the authors give very strong performances on two widely-acknowledged benchmark datasets. The results are very competitive and demonstrate the method is quite effective. 

Detailed comments:

There are many good points of this submission, as I talked about in the last paragraph. Besides, the writing is clear and easy to follow. The overall presentation is high-quality in terms of writing, e.g., the formulations, the figures. Therefore, I would like to turn to point out some concerns from my side, and I still want to know more about this work, please see the follows:

* The first point is about the crossover operation performed on the second stage, the monolingual data and the bilingual data. The source sentences ($x$, $x'$) are directly mixed together through a mask operation to be a new source sentence, and the target sentences are mixed through a weighted version, as the equation (6) and (7) show. This releases some questions to me: (1) the source sentence $x^p$ and the noised source sentence $y^{\diamondsuit}$ is combined through equation (5), but this mix the space from both source and target, and also interrupts the meaningful space. This augmentation operation is different from the previous works, therefore, how to guarantee this operation is reasonable and effective. It seems more explanations need to be shown. (2) Similarly, as for the target sentences, the mixed version of the monolingual target $y$ and bilingual $y^\{\diamondsuit}$ is most probably a non-sense sentence. Therefore, it is hard to think of the reason or the motivation for such mixing, especially in a weighted version as equation (7) and (6). More explanations and interpretations are needed for those operations. (3) Of course, the operation is similar to the mixup operations proposed previously, but the relationship is not clearly discussed in this submission. Mixup is the most related work since the method in this paper, or jointly training method can be regarded as a data augmentation method with two loss objectives from both monolingual and bilingual sets. Therefore, I feel okay about the story presented and motivated by the pre-train methods in this work, but I do feel the most related work is mixup and related data augmentation methods for NMT. 
* Specifically for the crossover operation, besides the relations and explanations for the mixed operation, the weighted label vectors are computed in an attention-based approach by leveraging the attention matrices. As $A$ and $A'$ are not good at the first training steps, the authors add the trick of linearly increasing temperature for them. Hence, the related question is to see the performance of label vector without the weighted version, with a similar operation as equation (5). The advantage of this weighted label vector compared to others need to be discussed since this is more complex than other methods, the training cost is also increased. The importance of the normalization factor $Z$, the necessity of current formulation. More motivation for this modeling is expected. 
* As for the training tricks, for computing $h$, I do not clearly understand why predicted $f$ should be added to the target labels. Is it related to scheduled sampling, or what else? Why the label vector is formulated in such a way? 
* Since the authors compared with other pre-train methods, MASS, BART, therefore, the experimental results are expected to make a clear comparison. 
* For the details of the experiments, the authors use $128$ GPUs to run for the experiments, which is a huge number that is not easy for others to reproduce. The authors are encouraged to open-source the code and settings. A small question is for the dropout values, is attention dropout and relu dropout adopted? It seems to be not clear. The training cost should also be discussed since the algorithm contains several dependencies in the three objectives.
* In analyses, the authors see the effect of monolingual corpora size for English-German task, which contains $4.5M$ bilingual datasets, therefore, the monolingual size of $3\times$, $5\times$ and $10\times$ are less than $89M$ and $90M$ as used in the experimental settings, but the results $30.46$ is from $3\times$, is it correct? If it is like this, the authors should give more descriptions of this point to give people a clear data setting. One another small question is about the mixup replacement mentioned in ablation study, can you give more words on how you perform mixup in this setting, $x=x_1 + x_2$ corresponds to what operations for two sentences with several words. Is it similar to the masked replacement as equation (5)?

In a word, I want to hear more about the details of this submission and I am willing to increase the score if questions are answered. 

----post-update----

Hi, I thank the authors to give useful feedback about several concerns. But I am still questioned about the cross-sentence operation, even the authors give an example of the different tasks, which is hard to convince me for such kind of learning can achieve such results. The 5 tasks as the author described are much harder, which somehow indicates that bilingual data is not so necessary for machine translation (but it should be aware, this is not unsupervised machine translation, the training spirit is totally different).  Random data augmentation could be very very strong. The authors are expected to give a code implementation and the trained model to give a more convincing result. 
review_title: interesting and impressive performances

final_score: rating: 7: Good paper, accept
review_content: summary:
--------

This paper introduces a new approach to semi-supervised training of neural machine translation. During training, the traditional supervised loss is complemented with two auxiliary losses: a denoising autoencoder and what the authors call cross-breeding: shuffling together the source side of a sentence pair with an unrelated, noised monolingual sentence, and predicting a virtual target sentence whose embedings and labels are a linear interpolation of the paired target sentence and the monolingual sentence, with interpolation weights being based on an attention matrix of the input words selected during shuffling. Authors report an improvement of around 2 BLEU on WMT14 EN-DE and EN-FR over a purely supervised Transformer, and improvements of 1 BLEU (EN-DE) and 0.4 BLEU (EN-FR) over a system with back-translation. Additionally, the system shows robustness against a type of "code-switching" noise.

strengths:
----------

+ novelty: while there has been some work trying to combine autoencoding and supervised learning for NMT, the idea to create training instances with virtual outputs that combine both tasks is new to my knowledge.

+ strong empirical results: the training objectives compare favourably to previous attempts to use autoencoding as an auxiliary objective for high-resource NMT. It is compatible with back-translation (using back-translations as noise function), and in this setup, obtains a new SOTA on English-French.

+ ablation studies: the paper demonstrates the contribution of the different training objectives, and that joint training is superior to the more commonly used pre-train+fine-tune strategy.

weaknesses:
-----------

- obscure description. The paper makes heavy use of metaphors from biology, describing the approach in terms of "interbreeding" or "marrying" sentences and creating "offspring" over two "filial generations". I found these terms obscuring rather than informative. For example, phrase "we use XEnDec to marry monolingual sentences" is a really roundabout way of expressing that you apply masking to a sentence. Describing a denoising autoencoder as a special case of the crossover encoder-decoder (section 3.1-3.2) also doesn't work for me, because what makes the model special, the creation of a virtual target sentence based on an alignment matrix, isn't applied here.

- robustness results are overclaimed. I was expecting robustness to similar types of noise than in previous work, but the "code-switching" noise employed (randomly replacing source words with the aligned words from the reference translation) is unrealistic and eerily similar to the training scenario (including the reliance on an alignment matrix). There is no strong improvement in the face of "drop-word" noise, and if this noise were of practical relevance, the much simpler strategy of applying word dropout at training time is likely to remedy it.

- open questions about efficiency. Authors use model predictions as decoder inputs (3.3), which prevents parallelisation across time steps at training time. Can authors quantify the training speed difference between the vanilla objective and their combined objective?

- open questions about where improvement comes from. The main novelty is L_F2, which shuffles two inputs and creates a virtual target sentence for prediction. Why this objective is effective is still a bit of a mystery to me, and I would have liked more insights into this. Most pressingly, using the predictions as decoder inputs reminds me of work on scheduled sampling, and could have a substantial effect that may be unrelated to self-supervised learning. How much does using the prediction vector as input embedding contribute to the quality? (Ablation studies indicate that dropout on A and using the predictions together contributes 0.6 BLEU, which is about half of the benefit of L_F2) Does the vanilla objective improve when using model predictions instead of (or in addition to) gold inputs?

- related to the above question (where does improvement come from?), back-translation. Best results are reported when using back-translation instead of a denoising autoencoder. It's unclear how this result fits into the narrative. If authors still consider back-translation "self-supervised learning", then this weakens the contribution that "we show that self-supervised learning is complementary to supervised learning" (which is well-known for back-translation already), and also doesn't fit the narrative in the introduction that self-supervised learning "provides a much weaker learning signal that is easily dominated by the signal coming from supervised learning". If back-translation experiments are seen as separate from the self-supervised learning experiments, it is misleading to imply that self-supervised learning achieves a new state of the art.

recommendation:
---------------

I'm currently borderline on this paper. The main strength of the paper are the strong empirical results, but I have misgivings about some of the central claims of the paper, most importantly the robustness results and positioning the paper as a way of strengthening the self-supervised learning signal, when the reason for the performance improvement is still obscure.

additional questions to the authors and minor points:
-------------

- some implementation details were unclear to me. When using back-translation as a noise function n(·), do you still apply masking in addition? If so, is there a good rationale why masking would only be helpful for back-translated data, but not for real parallel data? I also don't understand what you mean with "applying XEnDec over parallel data". Are you mixing two sentence pairs (shuffling the source sentences together and predicting an interpolated target sentence)?

- it was not made clear until 3.3 that L_F1 is the same as L_U.

- equation 7: the second sum should be \sum_{i=1}^I

- I note that English-German results with back-translation are not directly comparable to Edunov et al. (2018) because they are based on Transformer base. Why did you not report results with Transformer big? 

UPDATE
-----------

the author response has addressed some concerns (efficiency concerns; unclarities about some ablations studies; reason for not comparing to Edunov for EN-DE) well, and I have slightly increased my rating.

I maintain that the description is needlessly obscure in places, even in the revised version (while it is technically correct that a denoising autoencoder could be described as a crossover encoder where the two parent outputs are identical, this is needlessly convoluted), and a major concern about the framing of the paper was not resolved in the response (if back-translation is self-supervised learning, then some of the claims about limitations of self-supervised learning are untrue, and the findings lose novelty. If back-translation isn't self-supervised learning, then it's misleading to imply that self-supervised learning led to a new SOTA). 
review_title: strong empirical results, but where does improvement come from?

Meta-review: in this submission, the authors propose a sophisticated pretraining strategy for neural machine translation based on the paradigm of self-supervised learning. despite some interesting and potentially significant improvement in various machine translation settings, the reviewers as well as i myself could not determine where specifically those improvements come from. is it their particular strategy of pretraining or is it just self-supervised learning in general? in order for this question, which i believe is a key question to be answered, more thorough ablative experiments and/or comparison to other self-supervised learning based pretraining algorithms, such as MASS & BART which were discussed as similar and motivational in the submission, must be done. when these are done, the submission will be much stronger and attract much more interest.
Decision: Reject