Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: 

This paper discusses a method that effectively incorporates a (large) pre-trained LM, such as BERT and XMN, for improving the performance of NMT.
 
The motivation of this paper is rather straightforward and not novel; many researchers can quickly think of such an idea of incorporating the power of the recent (rapid) development of pre-training LMs into NMT.
From this perspective, this paper is not very exciting. 
However, as described in the paper, we often fail to improve (or even degrade) the performance of NMT when we straightforwardly incorporate a pre-trained LM.
Thus, many researchers/developers might want to know a practical approach to integrate a pre-trained LM into NMT.
This paper provides a straightforward but smart way to incorporate pre-trained LMs, which is not trivial in the community.
In this sense, this paper might have a considerable influence on the community.
I was a bit surprised by the apparent effectiveness of the proposed method since I also have attempted to apply pre-trained LMs to NMT and have not obtained a good result.
 
 
Experimental results are mostly convincing; the authors conducted comprehensive and extensive experiments on many settings, such as supervised NMT with low- and hi-resource settings, a semi-supervised NMT setting by back-translation, document-level MT, and unsupervised NMT.
The results were also promising; the proposed method consistently outperformed conventional methods. 
I think these results are useful for many readers.
Moreover, such findings also offer further insights for many researchers who aim to apply BERT to many other tasks, especially for text generation tasks.
 

Here are my concerns about this paper.

1, unclear explanations
The writing can be much improved. Readers might be able to guess, but several descriptions are hard to follow, or detailed explanations are missing.
For example, what is the exact operation of "function cascade"?  
What is the difference between the "Training NMT module from scratch" and "Standard Transformer" in Table 6?  What is the main reason for the lower performance of (Miculicich et al. (2018)) than that of sentence-level NMT in Table 4?

2, better comparisons
I think the authors need to confirm another model setting for a fairer comparison, something like "The proposed architecture with (fixed) random vectors instead of the BERT's contextualized embeddings.
It is because we sometimes observe the improved performance for the above model comparing with the original one.
We can interpret this improvement by the effect of increasing the weight parameters for injecting the additional random vectors to the original architecture.
Therefore, I think the above model settings can improve the performance of standard Transformers, which can be a preferable counterpart of the proposed method.
Moreover, the proposed method is closely related to the model ensembling since the method utilizes two separate models.
Therefore, the authors should also report the results of model ensembling for better comparisons.
 
3, less discussion for the experimental results
I found minimal discussions about the results.
For example, in the ablation study, the authors only show (list) the observations of their results and no discussions.
The authors should provide discussions about how and why their method (architecture) can improve the performance compared with a similar (and current de facto standard) approach, like the fine-tuning setting that can often improve most of the other NLP tasks.


review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper explores the use of BERT to improve Neural Machine Translation (NMT) both in supervised, semi-supervised and unsupervised settings. The authors first show that using BERT to initialize the encoder and/or the decoder does not bring any clear improvement, while using it as a feature extractor performs better. Based on this finding, the authors propose a new approach to integrate BERT in NMT, named BERT-fused NMT, which incorporates BERT representations from the input sequence into the encoder and decoder attention mechanisms.

I am ambivalent about this paper. On the one hand, the paper presents a thorough experimental evaluation, with strong baselines (often outperforming their original implementation) and results that can be interesting from different angles, and the reported improvements are consistent. However, the paper is rather poorly written and some important details are not adequately described, which left me with some concerns and an overall negative impression as I read through the paper. More concretely:

- The paper is rather poorly written. There are many expressions that sound ungrammatical or otherwise unnatural to me (although I am not a native speaker myself) and, more importantly, the overall exposition of ideas is not sufficiently clear. I found the paper difficult to follow, and I was left with many doubts as I read through it. In addition, the style in which some results are presented is inappropriate for an academic paper (e.g. "Obviously, our proposed BERT-fused NMT can improve the BLEU scores"), although I understand that this was probably not intentional.

- To make things worse, the paper is 10 pages long, and according to the CFP reviewers are "instructed to apply a higher standard to papers in excess of 8 pages". I think that the paper could be fit in the regular 8 page limit.

- The pre-trained BERT models that the authors use were trained on different (and generally larger) training data than what they use for the NMT training (e.g. they all use Wikipedia). As such, the models that build on BERT are indirectly using this additional training data. How can we make sure that the reported improvements are not due to this additional data? What would happen if the same data was used for the baseline systems (e.g. through back-translation)? Also, please clearly state which pre-trained model you use for each specific experiment.

- The treatment of subword tokenization is not given sufficient attention and raises some concerns to me. It seems clear that the authors combine different subword tokenizations for their proposed system (i.e. BERT and the NMT encoder/decoders use a different subword vocabulary). However, it is not clear to me how this is handled in the baseline systems that use BERT for initialization only, for which a mismatch in tokenization would be problematic.

- I often find it difficult to understand what the authors did exactly for each of the reported systems. For instance, what is the difference between "Standard transformer" and "Training NMT module from scratch" in Table 6? I cannot see any yet the difference in BLEU is 1.5.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The paper proposes an approach to incorporate BERT pretrained sentence representations within a NMT architecture.
It shows that simply pretraining the encoder of a NMT model with BERT does not necessarily provide gains (and can even be detrimental) and proposes instead to add a new attention mechanism, both in the encoder and in the decoder. The modification is relatively simple, but provides significant improvements in supervised and unsupervised MT, although it makes the model slower and computationally more expensive. The paper contains a lot of experiments, and a detailed ablation study.

===

I'm very surprised by the results in Table 1, i.e. the fact that pretraining can decrease the performance significantly. The provided explanation "Our conjecture is that the XLM model is pre-trained on news data, which is out-of-domain for IWSLT dataset mainly about spoken languages" is not satisfactory to me. The domain mismatch is also there in the majority of GLUE tasks, SQUAD, etc. and yet pretraining with BERT significantly improves the performance on these tasks. When the encoder is pretrained with a BERT/XLM model, I assume the encoder is not frozen, but finetuned?

The description of the algorithm in Section 4 could be simplified a lot I feel. Overall, the attention in the encoder is simply replaced by two attention layers: one over the previous layer like in a standard setting, and one on top of the BERT representation. Also I don't understand why the attention over the BERT sequence is also necessary in the decoder. Shouldn't this information already be captured by the encoder output?

The Drop-Net Trick is interesting. But the fact that 1.0 gives the best performance (Section 6.2) is very unintuitive to me. This means that the model will never consider the setting with two attentions at training time, although this is what it does at test time.

In Table 6, you propose experiments with 12 and 18 layers for fair comparison, because as you mention, your model with BERT-fused has more parameters. But IWSLT is a very small dataset and it would have been surprising that using 18 layers actually helps (overfitting is much more likely in that setting). Instead, I think something like an ensemble model would be a more fair comparison. In fact, the BERT-fused is essentially an ensemble model of the encoder.
Could you try the following experiment on IWSLT, where you do not pretrain the BERT model with the BERT objective, but with a NMT encoder trained in a regular supervised setting (i.e. do not reload a BERT model, but a NMT encoder that you previously trained without the fused architecture)?

Overall, I think the gains are nice, but I would really like to see the comparison I mentioned just above, and comparisons with ensemble models. The proposed model is significantly larger / slower than the baseline models considered, and I wonder if you could not achieve the same gains with ensemble models.

Something I like about the approach is that is it quite generic in the sense that you can provide any external sequence of vectors as input to your encoder. As a result, it is possible to leverage a model pretrained with a different tokenization. Tokenization is often an issue with pretraining in NLP (how do you leverage a model trained without BPE if you actually want to use BPE in your new model). The proposed approach does not has this constraint and I think this is something you should highlight more in the paper.

===

Small details in the related work section:
- I would cite "Sutskever et al, 2014" for the LSTM encoder, along with "Hochreiter & Schmidhuber", and not only "Wu et al, 2016"
- Removing the NSP task was proposed in "Lample & Conneau, 2019", not in "Liu et al, 2019"
review_title: Official Blind Review #1

Meta-review: The authors propose a novel way of incorporating a large pretrained language model (BERT) into neural machine translation using an extra attention model for both the NMT encoder and decoder.   The paper presents thorough experimental design, with strong baselines and consistent positive results for supervised, semi-supervised and unsupervised experiments. The reviewers all mentioned lack of clarity in the writing and there was significant discussion with the authors. After improvements and clarifications, all reviewers agree that this paper would make a good contribution to ICLR and be of general use to the field. 
Decision: Accept (Poster)