Cross-model Back-translated Distillation for Unsupervised Machine Translation

Abstract: Recent unsupervised machine translation (UMT) systems usually employ three main principles: initialization, language modeling and iterative back-translation, though they may apply them differently. Crucially, iterative back-translation and denoising auto-encoding for language modeling provide data diversity to train the UMT systems. However, these diversification processes may have reached their limit. We introduce a novel component to the standard UMT framework called Cross-model Back-translated Distillation (CBD), that is aimed to induce another level of data diversification that existing principles lack. CBD is applicable to all previous UMT approaches. In our experiments, it boosts the performance of the standard UMT methods by 1.5-2.0 BLEU. In particular, in WMT'14 English-French, WMT'16 German-English and English-Romanian, CBD outperforms cross-lingual masked language model (XLM) by 2.3, 2.2 and 1.6 BLEU, respectively. It also yields 1.5-3.3 BLEU improvements in IWSLT English-French and English-German tasks. Through extensive experimental analyses, we show that CBD is effective because it embraces data diversity while other similar variants do not.

Reviews and author responses:

In this paper, two unsupervised agents are utilized at cross-model by using the dual nature of the unsupervised machine translation model, in which forward translation of agent_1 is combined with the backward translation of agent_2, more synthetic translation pairs are obtained to train a new supervised machine translation model. The result is improved on multiple unsupervised machine translation, and this paper claims that more diversity is brought to the synthetic data, so a better translation model can be trained. This paper uses a reconstruction BLEU or BT BLEU [1] metric to compare the effect of the inside-model with that of cross-model, and finds that cross model translation has a lower back-translation effect, which shows that the diversity is enhanced. Furthermore, CBD is compared with the ensemble method and achieves better performance. The proposed method is quite simple yet effective, but it is also a kind of data enhancement.

In addition to these contributions, the paper also has some shortcomings

1. The evidence in this paper can not support the claim that the current performance bottleneck of UMT is due to the lack of diversity: the performance upper limit of UMT is still due to the lack of clear supervision signal, which limits the further performance growth. Because the training of CBD is divided into two stages, the diversity of the second stage only brings more training data to enhance the supervised machine translation model, rather than unsupervised machine translation effect.

2. Source of promotion: the second stage of CBD method adopts (x_s, y_t), (z_s, y_t), (y_s, x_t), (y_s, z_t) synthetic translation pairs, it is not clear how much performance growth comes from increased data and how much growth comes from the new model implementation (ott et al., 2018). It is not appropriate to attribute all contributions to the diversity brought by CBD. I suggest that the author should use (y_s, x_t) data to train based on the (ott et al., 2018) model, and report the effect comparison (In my experiments, the second stage model implemented with fairseq trained only on (y_s, x_t) surpass both agents trained with XLM due to more efficient implementation in fairseq).

3. Unfair comparison with the enable distillation: authors need to compare CBD with the model trained on the synthetic data (y_s, x_t) of the ensemble of agent_1 and agent_ 2. In the training data (x_s, y_t), (z_s, y_t), (y_s, x_t), (y_s, z_t) for the second stage of CDB,  x_t, golden language sequences as translation target is stronger than synthetic language sequences (silver) as target. Therefore, It is necessary to report the real result of ensembled distillation. The current results are very unreliable. In addition, it is necessary to compare the training time of the CBD method and ensemble distillation training (including the decoding process after the first stage of training) to show the efficiency of CBD.

4. The non-golden language sequence as a translation target is called pseudo-NMT (PNMT). The author adopts a variety of model structures, which is slightly redundant. They can directly add the synthetic data decoded by cross model to continually train the original XLM model with a supervised translation objective (which is naturally supported in XLM from my experience), and report the effect comparison between them.

5. The essence of the CDB approach is a process of self-supervised training, so it is necessary to compare self-training/tri-training introduced in [2].


In general, the CBD method in this paper is a simple and effective data enhancement method to improve the performance of the model. However, due to the lack of many important details of the implementation, despite the promotion, the source of promotion is unknown. In addition, the unreasonable comparison of the baseline models deepens my concern about the real promotion of this CBD method.

[1] Li, Zuchao, et al. "Reference Language based Unsupervised Neural Machine Translation." arXiv preprint arXiv:2004.02127 (2020).

[2] Sun, Haipeng, et al. "Self-Training for Unsupervised Neural Machine Translation in Unbalanced Training Data Scenarios." arXiv preprint arXiv:2004.04507 (2020).

Dear Reviewers:

Thanks for your insightful reviews! Now the discussion stage is open and the authors have posted their responses. We will appreciate that the following things-to-do can be done by Tues, Nov 24.

1 Acknowledge explicitly that you have read the responses.

2 Modify your review if necessary.

3 Communicate with the authors/reviewers/AC by adding/responding to the comments if necessary.

Thanks a lot!


Summary: The paper proposes an additional stage of training for unsupervised NMT models utilizing synthetic data generated from multiple independently trained models. The generated synthetic data uses two stages of back-translation, with different models, in order to "diversify" the set of training data used for fine-tuning the models. This is similar to the approach in [1], but uses an additional stage of back-translation with a different model. The authors add this additional stage of training to unsupervised NMT models using different pipelines (PB unsupervised MT, Neural Unsupervised MT, XLM) and show that their approach improves all of these approaches by 1.5-2 Bleu on WMT En-Fr, De-En and En-Ro.

Strengths:
1. The paper is well written, the approach is simple and seems to improve quality by significant amounts in a variety of experimental settings.
2. The authors do a great job of comparing against several relevant approaches (sampling during back-translation, ensembling, multi-agent dual learning). The paper compares against most of the relevant approaches I could think of while reading the paper.

Weaknesses / Questions for authors:
1. As with any NMT model trained with synthetic data, it would be better to report results on source and target original splits of the test data to provide a clearer evaluation [2,3]. Also clarify the Bleu scripts, tokenization and other post-processing used for evaluation.
2. The datasets used for experimentation are much smaller than the ones used for the baseline unsupervised-NMT approaches. It would be great to report results in the original training conditions (this is not a major limitation however, since the proposed approach seems to improve over baselines trained with more data).
3. Did the authors try any experiments with unsupervised models utilizing parallel data in unrelated languages, similar to [4,5] or in real low resource settings [6]? These are more practical conditions for unsupervised MT in true low resource languages.

Recommendation: Overall, this is a good paper and I would recommend acceptance. While I would have also liked to see experiments in more realistic low-resource settings, the current paper does a good enough job of evaluating the approach in standard unsupervised NMT settings on related high resource languages.

References:
[1] Data Diversification: A Simple Strategy For Neural Machine Translation, Nguyen et al.
[2] APE at Scale and its Implications on MT Evaluation Biases, Freitag et al.
[3] On The Evaluation of Machine Translation Systems Trained With Back-Translation, Edunov et al.
[4] Multilingual Denoising Pre-training for Neural Machine Translation, Liu et al.
[5] Leveraging Monolingual Data with Self-Supervision for Multilingual Neural Machine Translation, Siddhant et al.
[6] When Does Unsupervised Machine Translation Work?, Marchisio et al.

This paper describes a method to enhance unsupervised machine translation through data augmentation. The idea is pretty straight-forward, if not altogether intuitive, you begin by training two bidirectional (i.e.: they can translate source to target and target to source) unsupervised MT systems A and B. The tested scenarios always have A and B be identical architectures trained with different initializations. They then produce synthetic source-target pairs by first having A (source->target) translate the provided source sentence x to y, and then having B (target->source) translate y back to z. They then train supervised MT on both x,y and z,y. The same procedure can be repeated with source and target reversed. The authors show substantial (1-2) BLEU improvements with 3 different UMT systems in 5 low-data scenarios (En-Fr, Fr-En, En-De, En-Ro, Ro-En), all subsampled to 5M monolingual sentences for each language. In En-Fr and Fr-En and En-De, they are able to match reported XLM results from Conneau and Lample 2019, despite using much less data.

This simple idea is explored extremely thoroughly. The paper reads more like a journal paper that has undergone several stages of review than a conference paper. The authors make connections to and compare against a number of relevant ensembling strategies (to account for two systems being used) and back-translation-diversification strategies (to account for multiple sources being produced for the same target), and consistently show that only their specific recipe leads to the same levels of improvement. The authors really leave no stone unturned.

The biggest knock against this paper is the relatively small data scenario. Having two UMT systems allows them to provide two source sentences (one original and one synthetic) for each target sentence (always synthetic), but how important is this when we have 25x more original source sentences? I can imagine arguments for why the high-data UMT scenario is unrealistic (many monolingual sentences implies the likely presence of parallel data), but those arguments aren’t presented in the paper. It would be greatly strengthened by a full-data experiment for even just one or two of the language pairs.

Beyond that, I have few concerns. The paper is clear, easy to follow, and as I said, very thorough. But I’ll do my best to make some constructive criticisms:

I think the Related Work section feels a little superfluous after all of the comparisons made to related work in the Background and in the Experiments. I think I would like to have seen more discussion of the highlighly related work in sections 5.3 and 5.4. In particular, a more detailed discussion of this method’s relation to multi-agent dual learning would be worth giving up parts of Related work that are already mentioned in Background (like pre-neural statistical unsupervised MT).

It would be useful to specify how BLEU is calculated, to help readers understand just how useful the cross-paper BLEU comparisons in Table 1 are.

This paper introduces a new component to the unsupervised machine translation framework called cross-model back-translated distillation. The proposed approach is applicable to the other unsupervised methods. Experimental results in several translation tasks show that the proposed approach improves the translation accuracy of the standard unsupervised machine translation models, outperforming the cross-lingual masked language model. 

- The analyses are interesting to understand the proposed approach. Table 4 reports the diverse of synthetic data, but what about the quality as parallel data? Can you use parallel data instead and conduct the same analyses so that you can use BLEU score as an evaluation metric?
- Is it possible to apply the proposed approach to supervised NMT training, by creating BT data from monolingual data? 
- Table 1 reports that the results from your experiments show that the equivalent/better performance agains the existing models with much fewer data. What about scaling up the monolingual data size 5x/10x more? Will the performance be improved better and better?
- "the translated products (x-y) of the UMT teachers." at p.6.  What does this "x-y" mean?

Typo:
p.3 5.2.In Appendix -> 5.2. In Appendix

Meta-review: This paper proposed an additional training objective for unsupervised neural machine translation (UNMT). They first train two UNMT models and use these models to generate pseudo parallel corpora.  These parallel corpora are used to optimize the UNMT training objective. The experiments are conducted on several language pairs and they also compared with several alternative works. 

All the reviewers admit that the proposed method is straightforward and effective. The authors claim that the new training objective is used to enhance the "data diversification". This point has been questioned by the reviewers. Some reviewers are convinced by the response and some still have different opinions.  From my point of view, the proposed method can also be considered as a kind of combination of  (pseudo) supervised NMT and unsupervised NMT. 

The presentation and description of its key contributions seem unclear. However, we encourage the authors to modify their paper and we believe this proposed method can inspire the MT community for further research. At the moment, the paper is seen as not yet ready for publication at this time.

Decision: withdrawn-rejected-submissions