Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This submission integrates the encoder-only and encoder-decoder Transformer for both understanding and generation tasks through parameter sharing. The authors present a variable encoder-decoder pre-training approach to unify the two mainstreams in pre-training tasks. To be specific, the model shares the self-attention layer and feed-forward layer for both encoder and decoder, while the cross-attention layer is trained in an alternative way. The VECO approach delivers strong results on various cross-lingual tasks of XTREME benchmark for understanding tasks, and generation tasks for WMT14 En-Fr and WMT14 En-De. 

Comments:
First of all, the VECO approach provides impressive results on the different benchmark datasets including both language understanding and generation. It outperforms the previous methods in a non-trivial margin, which demonstrates the effectiveness of the approach. This is deeply acknowledged. Also, the studies and the detailed statements are mostly enough to make evaluations.  Despite its effectiveness, I do have several concerns and problems:
* In terms of the technical contribution, the relation between VECO and previous works is hard to make a strong difference, for example, BART, UniLM. Though the authors mentioned different language pairs (multilingual), personally, I acknowledge the parameter sharing of self-attention and feed-forward layer between the encoder and decoder is the main difference, while this is also widely used in machine translation models. Therefore, I feel a little bit unsatisfied with the contributions. 
* As for the shared pertaining method, it is a little bit confused about the parameter updating. For the CS-MLM loss, the decoder will reuse the hidden states of encoder output, are these hidden states fixed during the decoder update? Or they will still be updated to the encoder (I know the parameters are shared)? The implementation detail is not so clear. If the encoder continues updating, how can the GPU memory cost differ from the MT fine-tuning procedure? 
* The results are compared in a clear way, however, the parameters are not clearly compared with previous works. It seems the 662M model size is bigger than previous works. This is not the main weakness.

Generally speaking, this work is good for benchmark tasks to achieve strong results or a good technical report. But for the ICLR conference, it may need more. 

----------

Post-update:
Thanks to the authors for your response. I deeply acknowledge the promising results achieved from your work, which is impressive. I still feel hard about the contribution, though I also acknowledge the difference and it is an excellent practical trick. 
review_title: acknowledge the promising results

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: Summary:

The paper presents VECO: a pre-trained encoder-decoder model which is capable of both cross-lingual understanding and generation. They present two losses for inner-sequence and cross-sequence understanding. These components are then used to either build a encoder-only model or an encoder-decoder model. They present results on XTREME for cross-lingual understanding and on MT for generation. The results on both tasks are quite competitive and clearly shows the benefit of using both monolingual and parallel data with IS_MLM and CS-MLM.


Reasons for score:

I vote for accepting the paper. The paper initializes from XLM-R and then fine-tunes it on monolingual and parallel data in 50 languages. The authors get +3 average gain over the previous best system on XTREME and were ranked #1 at the time of submission. They also get handy gains in the MT benchmarks. The two ablation experiments show that the new CS-MLM task is indeed beneficial and improves the performance. One of the concerns I have is that the authors are not upfront about their model being in just 50 languages and hence might not be comparable to other models like XLM-R which support 100+ languages.x


Cons:
- As mentioned above, please be upfront about training only on 50 languages. It's only mentioned in the Appendix. This needs to be mentioned in the main text and maybe even in the results table.
- I would like to see more information about the amount of data (both monolingual and parallel) listed clearly in the Appendix.
- What would happen if VECO was trained on 100 languages from XLM-R? That result would be interesting to see in Table 1.
- I would personally like to see more ablation experiments: experiments where the amount of pre-trained monolingual and parallel data were reduced independently to see the impact it has performance.


Minor comments:
In Section 4.1, kindly cite all the representative tasks in XTREME as suggested here:
https://github.com/google-research/xtreme#paper

- It would be great if the authors stated the number of monolingual and parallel examples explicitly in Section 3. Appendix A doesn't provide the number of examples but just the size (1TB) of parallel data used.

- Please provide breakdown of amount of monolingual data and bilingual data per language/language-pair. How many languages do you get monolingual data in? It's not clear from the paper.

- Also, please be explicit about the number of languages supported in the model (50). This is hidden in Appendix A. One can argue that this is not a fair comparison against XLM-R since it's trained on 100+ languages.

- Change "Ours implementation" to "Our implementation" everywhere.

Section 7:
- Change "targeting at initializing both..." to "targeted at ..."



review_title: Good paper demonstrating strong cross-lingual transfer learning capabilities

final_score: rating: 4: Ok but not good enough - rejection
review_content: In this paper, the authors propose variable encoder-decoder (VECO), a pre-training strategy for both NLU and NLG tasks. In VECO, two masked language model (MLM) are leveraged for pre-training, (a) inner sentence MLM with encoder only, and (b) cross sentence MLM with both encoder and decoder. The parameters of self-attention and feedforward layers are shared in encoder and decoder. Improvements are observed for multiple NLU and NMT tasks.

The paper is mostly well written and nicely presented. The authors show extensive results in various downstream understanding and generation tasks, improves performances in most cases. The proposed method is simple and straightforward; and can be readily reproduced in any existing toolkit.

I have some doubts on the motivation of separate pre-training tasks for encoder and encoder-decoder. The authors claims that for previous encoder-decoder pre-training like MASS and mBART, 
"it usually requires more computation and memory to match the performance of the encoder-only models". What exactly is the additional computation and memory required here? And how is the empirical comparison on these approaches against the proposed one in terms of both model performance and computation/memory cost?
Actually in MMTE [1], the authors show that pre-training a NMT (encoder-decoder) model and extract the encoder is effective for various encoder-only downstream tasks. 

Some more comparison with previous pre-training approaches should be presented as well, e.g. MASS, BART, MMTE[1], etc. 

In general, I think the authors did an excellent job validating their method on various different NLU/NMT datasets. However, I'm skeptical about the novelty and the general contribution/impact of the paper.

Misc:

* Table 3, why comparing with different number of encoder/decoder layers for previous methods (e.g. 24/6 for XLM-R, 12/12 for mBART)?
* Table 3, do you also share parameters (self-attn, ffn) for baseline methods? How much does parameter sharing contribute in terms of performance and training efficiency?

[1] Siddhant, Aditya, et al. "Evaluating the Cross-Lingual Effectiveness of Massively Multilingual Neural Machine Translation." AAAI. 2020.
review_title: Extensive experiments and results, but weak contribution

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a **unified** cross-lingual pretraining method that works well for both natural language *understanding* (NLU)---typically done using *encoder-only* architectures like mBERT and XLM---and conditional natural language *generation* (NLG) tasks like machine translation---typically done using *encoder-decoder* architectures like mBART. More concretely, let $\mathbf{x}$ be the "source" sequence and $\mathbf{y}$ the "target" sequence. In machine translation, $\mathbf{x}$ and $\mathbf{y}$ are the source and target sentences, while in the monolingual case, $\mathbf{x}$ and $\mathbf{y}$ are two contiguous sequences in the corpus. The pretraining loss consists of the following four terms (Eq. 8): (i) masked language modelling only on $\mathbf{x}$, (ii) masked language modelling only on $\mathbf{y}$, (iii) autoregressively decoding $\mathbf{y}$ conditional on $\mathbf{x}$ in a left-to-right fashion, and (iv) the reverse of the third loss term, i.e. autoregressively decoding $\mathbf{x}$ given $\mathbf{y}$ in a left-to-right fashion.

Compared to prior approaches, this proposed approach has a key benefit of *sharing* the parameters between the encoder and the decoder during pretraining (hence eliminating the need to train a separate decoder component from scratch), and also enables the pretrained model to be used for both NLU (where the unused cross-attention module is simply discarded) and NLG tasks. Experiments demonstrate the strong empirical performance of the proposed approach on both cross-lingual NLU (through the XTREME benchmark) and two machine translation datasets, surpassing the previous state of the art results in many cases. Further analysis suggests that the improvements are not simply due to training on more bilingual data.

**Pros:**
1. The proposed approach demonstrates strong empirical performance on both cross-lingual NLU and machine translation tasks, outperforming various strong baselines and achieving new state of the art results in many cases. 

2. The paper does a great job of analysing whether the improvements come from the proposed approach---which benefits from the encoder-decoder parameter sharing and a loss function that takes into account both intra-sequence masked language modelling and sequence-to-sequence mapping---or simply from more bilingual data. Disentangling the gains from the proposed approach vs more data is really important to understand and enable better progress in the field, which unfortunately is not always done in prior work. So I appreciate the fact that this paper takes a step towards addressing this question, and it is encouraging that the improvements do not merely come from more data (Table 2).

3. The paper is overall well-written and draws extensive connections to the relevant prior work.

**Cons:**
1. In my understanding, the difference between the proposed approach and mBART (which also unifies cross-lingual NLU and NLG) is pretty minimal---the second paragraph of page 4 mentions that the key difference here is the parameter sharing between the encoder and decoder. Could the stronger performance of the model compared to mBART be simply due to training on more bilingual data? Repeating the ablation experiments as done in Table 2, but comparing with mBART rather than only XLM, would help make the empirical setup much stronger.

2. The pretraining objective in Eq. (8) includes a "flipped" sequence-to-sequence mapping term that  predicts $\mathbf{x}$ autoregressively conditional on $\mathbf{y}$. In the monolingual case, this runs the risk of ignoring *discourse information* that comes from the sentence ordering. Imagine a very simple example where $\mathbf{x}=$"Clyde is an elephant." and $\mathbf{y}$="He is very gentle.". The pronoun "He" in $\mathbf{y}$ is allowed since there is already an antecedent, Clyde, in the previous sentence $\mathbf{x}$. But this ceases to be true if we flip the ordering of the sentences. Could you please say more about this?

3. In practice, the model parameters seem to be initialised from XLM-R (except for the cross-attention part that XLM-R does not have). This multi-stage training process can potentially benefit from an implicit model combination effect, which the baseline models may not necessarily be able to benefit from. Having equally positive results where the model is trained from scratch can help make the empirical results stronger.

4. Some of the presentational aspects can be improved. More details below.

**Presentation / grammatical mistakes / typos:**
1. In page 3, "... is that $\mathbf{Q} = \mathbf{K} = \mathbf{V} ...$". I am not sure what this means, since it implies that the matrices $\mathbf{Q, K, V}$ are equivalent.

2. In page 3, the bottom of Eq. (2) says that $\text{MultiHead}(\mathbf{Q}=\mathbf{x}, \mathbf{K}=\mathbf{y}, \mathbf{V}=\mathbf{y})$. I am not sure if this is correct in the standard case where $\mathbf{x}$ is the source sentence and $\mathbf{y}$ is the target sentence. Shouldn't the query $\mathbf{Q}$ come from the target sequence $\mathbf{y}$ while $\mathbf{K}$ and $\mathbf{V}$ come from source sequence $\mathbf{x}$?

3. In page 3, "... and FFN modules to *corporate*" -> "cooperate".

4. At the top of page 4, it is mentioned that "...we detach $\mathbf{X}^{(N)}$...to let the two objectives optimized in isolation". What does "detach" mean here, and how does it relate to optimising the two objectives in isolation?

5. In Table 3, "Randomly *Initialize*" and "Cross-Lingual Models *Initialize*" -> "Initialized".

6. In page 7, "*Figure 4* (left) contrasts two ways ..." -> "Table 4". 

7. In page 8, "... and then *continue-train* ..." -> "continue to train".

**Bottom line**
Overall, I think the pros outweigh the cons and this paper can be useful for the community, although addressing some of the concerns I raised above can make the paper stronger. I would be willing to reconsider my score based on the response.

**Update after the authors' response**
The authors have provided a comprehensive response that address most of my comments, and clearly clarified the novelty and key differences with prior work such as mBART (which also seems to be a key concern in the other reviews). Given the satisfactory authors' response, I am therefore raising my score to "7". Overall, I believe this paper presents a good contribution to the field.
review_title: Official Blind Review #3

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper targets to unify the advantages of encoder-only model and encoder-decoder model for multilingual pre-training. Given three types of parameters in transformer blocks including Self-Attention, Cross-Attention and FFN, this paper proposes to train the model for understanding and generation tasks at the same time, where Self-Attention and FFN weights are shared and trained for both understanding and generation tasks and Cross-Attention weights are trained for generation tasks only. Evaluations are performed on XTREME and WMT datasets, respectively, with good improvements obtained.

Strengths: (1) The approaches are clear and easy to reproduce. (2) The performance on XTREME and WMT are good.

Weakness: (1) Based on my understanding, this approach equals to sharing Self-Attention and FFN weights between MLM encoder, MT encoder and MT decoder, which is not innovative enough. (2) More baseline settings could be added for a more solid comparison. For example, for XTREME, what's the performance of IS-MLM + TLM, which is pre-trained using the full training corpus? (3) what kind of training data is used in the CS-MLM task? I cannot find the data detail from the paper. Is  it the MT data? (4) Do Table 1 and Table 2 use the same training corpus?

- Additional question. (1) For task IS-MLM and CS-MLM, do they use both monolingual data and bilingual data? If yes, which type of data is more useful?  (2) In table 2, do XLM and VECO use the same vocabulary and the same code for MLM? (3) If the answer to question (2) is yes, then the first line (XLM_{SMALL}) in Table 2 corresponds to XLM-R in Table 1 and the fourth line (VECO_{SMALL}) corresponds to VECO in table 2. For XNLI, the gap between VECO_{SMALL} and XLM_{SMALL} in Table 2 is 7.9, but the gap between XLM_R and VECO in Table 1 is only 0.7. What caused this difference? (4) In Table 3, different models use different hyperparameter settings, especially encoder/decoder with different layers. Could the paper use the same layer setting when compare with a specific baseline? This would help to verify that the gain comes from the new pre-training strategy, instead of a deeper model.

- Suggestions: (1) Add more baselines and ablation study, this will help readers to understand where the gains come from. For example, add a baseline model pre-trained by XLM-R and TLM in Table 1. This could prove the gain of VECO is not from TLM. (2) XGLUE, a concurrent work of XTREMEM, includes both multilingual understanding and generation tasks. I suggest the paper can provide the results of VECO on the multilingual NLG tasks in XGLUE.
review_title: A multilingual pre-trained model with shared Self-Attention and FFN weights

Meta-review: This paper proposes a unified cross-lingual pretraining method that works well for both natural language understanding (NLU)—typically done using encoder-only architectures like mBERT and XLM—and conditional natural language generation (NLG) tasks like machine translation—typically done using encoder-decoder architectures like mBART.

This paper clearly split reviewers, with 2 quite or very positive on it, and 3 thinking or leaning towards thinking that it didn't have enough novelty to merit publication.

Pro
- The model produces good SoTA results
- The method is easily replicable
- It is good for the community for leading systems on benchmark tasks to have published papers describing how they work.

Con

- The work is not groundbreaking in technical novelty
- The work has to do a better job of communicating its contributions: It's hard to understand how it differs from other methods

On balance, the overall assessment is that the paper is not yet ready in its current form. The hope is that authors find the reviewer comments useful for preparing a future submission:

- The paper **has** to do a better job of communicating its contributions. All that most researchers got from the first version was that there was parameter sharing and that helped. The revised version starts to do a better job of explaining the value of having the IS-MLM and CS-MLM objectives to doing well on NLU and NLG tasks, but much more is needed, as the discussion here shows. Indeed, even the discussion here is often opaque. In describing the key contribution of the paper, in both the revised paper and discussion, the authors fall back on phrases like "elaborately designed" and "exquisite cooperation of parameter sharing and pre-training tasks". **What do "elaborately designed" and "exquisite cooperation" mean?!?** I think you can minimally clearly explain the benefits of having an objective like IS-MLM for doing better on NLU tasks than the approach taken in mBART. You could argue for the advantages of MLM vs LM generation, which has been shown in other papers, including the original BERT paper and ELECTRA. Concretely, I wonder if you should reverse the contents of section 2 and start with equation (8) and explain why that is a good objective for your system, and better than ones that have been used previously. This discussion should be at a higher level than the current discussion under (8) which tends to be in the weeds. I haven't worked all the details, but I think you could then describe the objectives of section 2.2 before describing the implementation in section 2.1, and the result might be clearer? It would certainly emphasize the importance of these loss functions.
- The initial version didn't have important details like the number of languages covered in the main paper; the current version fixes this to the extent of saying you have 50, but still doesn't give the context of how this compares with XLM-R and mBART. And several reviewers had questions about the number of parameters of different models. I think you could fix a lot of these concerns by moving Table 8 to the main paper in a future resubmission. It doesn't take up much space and helps a lot in providing these details and easy to find citations for the models compared in other papers. 
Decision: Reject