VECO: Variable Encoder-decoder Pre-training for Cross-lingual Understanding and Generation
Authors: Fuli Luo
Wei Wang
Jiahao Liu
Yijia Liu
Bin Bi
Songfang Huang
Fei Huang
Luo Si
Venue:  (2021)
Keywords: Natural Language Processing
Representation Learning

Abstract: Recent studies about learning multilingual representations have achieved significant performance gains across a wide range of downstream cross-lingual tasks. They train either an encoder-only Transformer mainly for understanding tasks, or an encoder-decoder Transformer specifically for generation tasks, ignoring the correlation between the two tasks and frameworks. In contrast, this paper presents a variable encoder-decoder (VECO) pre-training approach to unify the two mainstreams in both model architectures and pre-training tasks. VECO splits the standard Transformer block into several sub-modules trained with both inner-sequence and cross-sequence masked language modeling, and correspondingly reorganizes certain sub-modules for understanding and generation tasks during inference. Such a workflow not only ensures to train the most streamlined parameters necessary for two kinds of tasks, but also enables them to boost each other via sharing common sub-modules. As a result, VECO delivers new state-of-the-art results on various cross-lingual understanding tasks of the XTREME benchmark covering text classification, sequence labeling, question answering, and sentence retrieval. For generation tasks, VECO also outperforms all existing cross-lingual models and state-of-the-art Transformer variants on WMT14 English-to-German and English-to-French translation datasets, with gains of up to 1~2 BLEU.

Under review as a conference paper at ICLR 2021
VECO:
VARIABLE ENCODER-DECODER PRE- TRAINING FOR CROSS-LINGUAL UNDERSTANDING AND GENERATION
Anonymous authors Paper under double-biee blind review
ABSTRACT
Recent studies about learning multilingual representations have achieved signif- icant performance gains across a wide range of downstream cross-lingual tasks. They train either an encoder-only Transformer mainly for understanding tasks, or an encoder-decoder Transformer specifically for generation tasks, ignoring the correlation between the two tasks and frameworks. In contrast, this paper presents a variable encoder-decodee (VECO) pre-training approach to unify the mainstreams in both model architectures and pre-training tasks. VECO splits the standard Transformer block into several sub-modules trained with both inner- sequence and cross-sequence masked language modeling, and correspondingly reorganizes certain sub-modules for understanding and generation tasks during inference. Such workflow not only ensures train the most streamlined param- eters necessary for two kinds of tasks, but also enables them to boost each other via sharing common modules. As result, VECO delivers new state-of-the the-art results on various cross- lonssll understanding tasks of the XTREME benchmark covering text classification, sequence labeling, question answering, and sentence retrieval. For generation tasks, VECO also outperforms all existing cross-lingual models and state-of-the-art Transformer variants on WMT14 English-to-German and Aeglish-to-Frencee translation datasets, with gains of up to I~2 BLEU,
INTRODUCTION
Driven by the striking success of pre-trained language models (Devlin et al. 2019), recent cross- lingual pre-training (Lample & Conncau 2019 Liu et al.) (020b) has attracted an increasing of attention. It provides cross-lingual contextualized representations for the inputs of different lan- guages, which significantly advances performance both natural language understanding (NLU) and generation (NLG) tasks.
There are two mainstream architectures in current cross-lingual pre-training literature: encoder- only and encoder-decoder. The former like XLM (Lample & Conneau 2019) focuses conduct- ing masked language modeling (MLM) with single Transformer (Vaswani et al. 2017) encoder. This paradigm naturally compatible with various NLU tasks, but tends suffer from limited gains on cross-lingual generation tasks (e.g., machine translation) due to the lack of effective de- coder initialization. In contrast, the latter like mBART (Liu et al., 2020b) attempts to pre-train the encoder-decoder Transformer via denoising auto-encoding tasks to provide complete initialization for downstream generation tasks. However, when applied NLU scenarios, it usually requires more computation and memory to match the performance of the encoder-only models.
In light of the above pros and cons, this work presents Variable deCode (VECO) pre- training. which targets at providing pre- pre-trained model initialization for both the encoder-only and encoder-decoder Transformer with the most streamlined parameters. We observe that Transformer encoder and decoder blocks have two common modules: SelfAttenti network), with the main difference that the latter introduces an extra CrossAttention (attention across from the encoder the decoder) module. Inspired by the lottery ticket hypothesis (Fran- kle & Carbin 2018), we split the standard Transformer block into three independent modules
and FFN (feed-forwanl
"Equal contribution.

---

Under review as a conference paper at ICLR 2021
y
FFN
FFN
FFN
FFN
stop grad
IS-MLM
Cross-attention
x
N
aess-attention
CS-MLM
Self-sttention
Self-atlention
Salf aettattenton
(a) Pre-training
(b) NLU Fine-tuning
(c) NLG Fine-tuning
Figure 1: The overview of VECO. During pre-training, feed masked segments 4: and into different modules to perform inner-sentence mask language modeling (IS-MLM) and cross-sentence mask language modeling (CS- (S-MLM). More specifically, the masked segment can only attend its context via self-attention to recover the original tokens E (IS-MLM), while masked segment ý can attend its preceding tokens via self-attention and the context via cross-attention predict the original tokens (CS-MLM), For downstream NLU tasks, we throw out the cross-attention module and only fine-tune on the self-attention and FFN modules acted as an encoder. For NLG tasks, we keep all modules to initialize the corresponding encoder and decoders.
(SelfAttente Atelftttenionn, CrossAttention, FFM) to be collaboratively trained via two specific MLM tasks. After that, we rebuild the desired complete architecture applicable for NLU or NLG with different specific combinations of these modules during fine-tuning Specifically, to be equipped with the ability of language understanding during pre-training. SelfAttention and FFN are assem- bled into a standard Transformer encoder for conducting inner-sequence masked language modeling (IS-MLM). In terms of generation, SelfAttention CrossAttentions and FFN act together the decoder in the standard sequence-to-sequene model, and are trained by the elaborately designed cross-sequence masked language modeling (CS-MLM) task. When applied to downstream fine- tuning, both Self At Attention and FFN modules constitute the Transformer encoder for contextual modeling in NLU or NLG, or cooperate with additional CrossAttention initialization of Transformer decoder. With such kind of workflow, VECO can be applied to both NLU and NLG tasks with the most streamlined parameters, which significantly reduces computa- tional overhead and memory costs. Moreover, IS-MLM is specifically designed for understanding of individual sequences, while both understanding and generation tasks can benefit from CS-MLM. With such parameter sharing, VECO enables Self Atteftiteiiioo on and FFN modules to be jointly trained by the two MLMs, which boosts both NLU and NLG performance.
provide the effective
We validate VECO variety of representative cross-lingual NLU and NLG benchmarks. For cross-lingual understanding tasks, we conduct experiments on the XTREME benchmark consisting of 9 moos-linggull tasks, including text classification, sequence labeling, question answering, and sentence retrieval. VECO ranks first at the XTREME leaderboard at the submission deadline and obtains state-of-the-art results most of the tasks. For cross-lingual generation tasks, val- idate VECO on the widely used WMT14 English-German and English-Frende machine translation benchmarks. VECO obtains 44.4 and 31.5 BLEU scores, consistently outperforming existing cross- lingual pre-training approaches and
Transformer variants by around 1~2 BLEU.
VARIABLE ENCODER-DECODER PRE- TRE-TRAINING
2.1 BACKBONE NETWORK
The backbone network of VECO is composed of a stack of N identical layers. Each layer has three modules, consisting of required self-attention module, an optional cross-attention module, and required feed-forward linear module. Both self-attention and cross-attention modules are based on
"Thus the word variahle means that the backbome Transformer varies during pre-training and fine-tuning. httpss:/stts htt.///itteerreerrtt..eeeee
research www.ll.h..oooooooooooooooo....o r/xtrree
2

---

Under review as a conference paper at ICLR 2021
the multi-heat attention (Vaswani et al. 2017):
MultiHead Q, K, V) Concat(head), head,, W
head; = Attention Nattentio(QWW4 KWk, vW
where Wo, WQ, wk and WV are parameter matrices. Attention(4 (a, b, c) represents the atten- tion operation with as query, b as key, and c as value. We refer the readers to Vaswani et al. (2017) for more details.
The main difference of self-attention module and cross-attention module is that Q = K V holds in the self-attention module while only K V exists in the cross-attention module. We formalize these two modules as:
V = x)) CrossAttentionnx, 0s) AddNorm (MultiHead y, K y,
SelfAttention(x: 0g) = AddNorm MaaloormMultiHHeall =
(2)
x, V x))
where 0g and 0g are the corresponding parameters, and AddNorm denotes a residual connection (He et al. 2016) with a post layer normalization (Ba et al. 2016).
After that, a fully connected feed-forward network is applied each element of input independently:
(x; 0F) AddNorm W GeLU (W1x)
(3)
where 0r (WI; W2] are matrices of parameters.
2.2 PRE-TNAINING OBJECTIVES
In cross-lingual pre-training scenarios, we can utilize both monolingual and bilingual data widely used in previous works (Lample Conncau 2019 Chi et al., [220bb Yang et al. (020). We formal- ize both the two adjacent segments in the monolingual corpus and pair of parallel sentence in the bilingual corpus as (æ:, y). We firstly adopt the same mask strategy like BERT (Devlin et al. 2019) to construct the masked input (æt, ÿ) Then, the backbone Transformer takes the input to perform the inner-sequence and cross-sequence masked language modeling, which enables the model to be optimized jointly for cross-lingual language understanding and cross-sequence generation.
IS-MLM: Inner-Sequence Masked Language Modeling To be equipped with the ability of lan- guage understanding, we perform masked language modeling of a single-sequence on the self- attention and FFN modules, while skipping the cross-attention modules. As shown in Figure I the purple lines show the forward process of the IS-MLM task in each layer. To be specific, the embeddings of the masked input sequence è are fed into the self-attention and FFN modules in each layer to get contextual presentation (
(+) (olfAAttention(((1))) (111);
(4)
x (6) (F 0,,
The contextual presentations X (N) of the last layer used recover the masked tokens T. Thus the training loss of inner-sequence masked language modeling can be formalized as:
GIS-MLM(22) = -lgg 0,.07
(5)
CS-MLM: Cross-Sequence Masked Language Modeling In order to fully train the cross- attention module that plays primary role in semantic mapping between sentences in cross-lingual generation tasks (c.g., machine translation), we simulate decoder by reusing the self-attention and FFN modules cooperate with the cross-attention module. As shown in Figure T the green lines depict the forward process of the CS-MLM task each layer. Specifically, we first extract the contextual representation ý via the SelfAttention modulds) then a cross-attention module is employed to model an interactive representation of (æ, g):
( SelfAttention Y (1-1);;0 1:0.)
(6) CrossAttention (1) (MM)
Y FFN(Z 00)
'A triangular attention mask matrix used only attend to the preceding tokens of each masked token.
3

---

Under review as a conference paper at ICLR 2021
Finally, Y considering both the context of the semantic-related sequence è and its left segments yet. is used to predict the masked tokens y::
Lcs- CCS-MLM(E,,) y) - log Dctio
Note that when optimizing the CS-MLM objective, detach ( in Eq. (6 from the computation graph (i.e.. stop the gradients back-propas propagation from "virtual" decoder to "virtual" encoder) to let the two objectives optimized in isolation. also speeds up and stabilizes the training of this "virtual"
decoder model, since very deep encoder-decoded are typically hard to train.
To conclude, the total MLM loss for a training instance (æ. y), by exchanging the & and ÿ Eq. (5)
and Eq. (7 can be further formalized as:
CMLM 3) Lis C MLM(M (z) + (IS-ML(()) (LS-MLM((E y) Las --s--Cs-MMM MLM
(8)
(log(())) + logP(glù) + log (yll, (er) + log ( D 2tt))
Several monolingual pre-training models such as MASS (Song et al., 2019) BART Lewis et al., 2019) and PALM (Bi et al., 2020) also present similar unidirectional language modeling tasks. Ex- cept that VECO focuses on the multilingual scenario, there are some major differences in terms of both model architecture and task design: 1) In terms of model architecture, the core difference is that share the self-attention and FFN modules in the encoder and decoder. We find that such parameter sharing can enhance the semantic mapping between different languages not only at the embedding-leved (e.g, shared BPE vocabulary), but also at the module-level. Moreover, it also acts as form of regularization that stabilizes the training and helps with generalization (Xia et al. 2019 terms of task design, differ in several ways. The proposed IS-MLM task forces the model to bidirectionally comprehend the source input (good for NLU), which is shortage of mBART. Meanwhile, CS-MLM predicts the masked words other than generating the next word (adopted by MASS and mBART), thus keeping in line with IS-MLM towards a more consistent optimization direction (predicting masked words) on the shared parameters.
Lan al., 2019).
In total, the core contribution of this article is the exquisite cooperation of parameters sharing and pre-training strategy, making VECO flexibly initialize any downstream framework.
2.3 FINE-TUNING ON DOWNSTREAM NLU AND NLG TASKS
When fine-tuning various downstream tasks, one advantage of VECO is its flexibility for initial- izing both the encoder-only Transformer and encoder-decoder Transformer.
VECO for Cross-lingual Natural Language Understanding Since the mainstream framework for NLU is an encoder-only Transformer, we only keep the self-attention and FFN modules while throwing out the cross-attention module in each layer (Figure [b)). Note that the cross-attention modules only occupy less than 20% of the total parameters, which is smaller than the discarded generator of ELECTRA (Clark al. 2020b).
VECO for Cross-lingual Natural Language Generation Considering the most popular back- bone network for cross-lingual generation tasks like machine translation is the standard Transformer macoder-decoder model, we reorganize the VECO modules to act like that. As shown in Figure Tc), the self-attention and FFN modules constitute the Transformer encoder for contextual modeling, and the three modules work together to act as a decoder for both inner-sequenc, and cross-sequence contextual modeling. Due to the training difficulty and inference speed of the deep network, we can choose a subset (e.g., 6 layers) of all layers to assemble the decoder. More in-depth analysis can be found in Table and Section 5.2
PRE-TRAINING SETUP
Model Configuration We pre-train a 24-lay model with 1024 embedding/hidden size and 4096
feed-forward size (~ 662M parameters). We do not use language embeddings to allow our model to
Flipping the monolingual sentence order can create "harder" example pairs, thus pushing the model toward stronger ability of language modeling and understanding.
4

---

Under review as a conference paper at ICLR 2021
better deal with downstream tasks of unseen languages. We adopt the same 250K vocabulary that is also used by XLM-R (Conneau et al. 2019) and mBART (Liu et al. 2020b).
Pre-Training Datasets For monolingual training datasets, we reconstruct Common-Crawe Corpus used in XLM-R (Conneau et al., 2019). We extract 1.36TB data in 50 languages, which contains 6.5G sentences and 0.4G documents. We up/down-sample the monolingual text like XLM from each language with a smoothing parameter = 0.5. For bilingual data, we collect from the OPUS website] previous works (Lmple & Conneau 2019; Chi et al. 2020b). There are 6.4G parallel sentences, covering 879 language pairs across 50 languages. More details about the languages and statistics of our training corpus can be founded in Appendix
Optimization Settings For each training iteration, we alternately sample batch of adjacent seg- ments from monolingual corpus and batch of parallel sentences from bilingual datasets to conduct a pair of masked input (æ, ÿ). We firstly perform IS-MLM for both & and y. Then, we reuse their hidden states from the last layer to perform CS-MLM. Meanwhile, we also adopt the translation masked language modeling (TLM) proposed in XLM (Lample & Conneaul (019) when the inputs parallel bilingual sentences. Thus the overall training objective is the sum of three MLM objec- tives. During training, the model parameters except for cross-attention are initialized by XLM-R. We first freeze the parameters of XLM-R and only update the cross-attention parameters for faster convergence. Then, we jointly train the whole model. We pre-train our model with mixed- precision training using 64 Nvidia Telsa V V100 32GB GPUs. More hyperparameters can be founded in Table
EXPERIMENTS ON LANGUAGE UNDERSTANDING
4.1 EXPERIMENTAL SETUP
Downstream Tasks We conduct NLU evaluations on XTREME (Hu et al. 2020). representa- tive massively multilingual multi-task benchmark. consists of various NLU tasks over 40 lan- guages. XTREME tasks can be classified into four different categories: (1) sentence-pair classifi-
cation: XNLI (Conneau et al.) [2188) PAWS-X (Yang et al. 2019); (2) structured prediction: UD- POS Nivre al. [018). Wikiann NER (Pan al.] 2017); (3) question answering: XQuAD (Artetxe)
et al. 2020), MLQA (Lewis et al. 2020). TyDIQA (Clark et al. 2020a); (4) sentence retrieval:
BUCC 2018 Zweigenbaum al. 2018). Tatoeba (Artetxe & Schwenk 2019). Tasks in the first three categories only provide golden training corpus in English and dev/test set in different target languages. For the two zero-shot sentence retrieval tasks, no training datasets provided. We refer the reader to Hu et al. (200)) for additional details about the datasets.
Fine-tuning Setting Following previous works (Conneau et al. 2019 Hu al. 2020), we con- sider two typical fine-tuning settings: (1) Cross-linguad Transfer which fine-tunes the pre- trained model only using English golden data and directly performs inference on the test data of different target languages; (2) Translate-Traiin-a
which first automatically translates the English golden data to the remaining target languages and fine-tunes a multilingual model on the concatenation of all data. We use the machine-transe translated data released by XTREME, except for two lequence-labeling tasks (POS, NER) since the golden token labels in the target language are unavailable. To have fair comparison with the strong baseline XLM-R (Conneau al. 2019) under the translate-train-all setting, we also show the results of XLM-R using the same fine-tuning hyperparameters as VECO.
4.2 EXPERIMENTAL RESULTS
The detailed test results of 9 tasks on the XTREME benchmark are shown in Table [] It demon- strates that the proposed VECO outperforms previous cross-lingual models on most of the datasets. Compared to XLM-R, it averagely scores 5.0 and 2.8 points higher under the cross-lingual transfer and donsslatioottrain--l settings. respectively. It is worth noting that, VECO delivers large im- provement on zero-shot sentence retrieval tasks (BUCC, Tatoeba). This phenomenon reflects that our model has strong cross- loossll modeling ability, thus it can better mine parallel sentences in a multilingual corpus. The reasons are two-folds. First is the introduction of more bilingual data
http://opus.nlpl..o
5

---

Under review as a conference paper at ICLR 2021
Sentence Classification
Structured Prediction
Question Answering
Sentence Retrieval
Model
Avg.
XNLI
PAWS-X
UD-POS
NER
XQUAD FI/EM
MLQA FI/EM
TyDiQA
BUCC
Tatoeba Acc
Ace
Acc
FI
FIÆM
Cross-lingual Transfer: Fine-tume model on English training set and test languages
MMTE
67.4
81.3
73.5
58.3
64,4/46.2
60.3/41.4
58.1/43.8
59.8
37.9
59.5
MBERT!
65.4
81.9
70.3
62.2
64.5/49.4
61.4/44,2
59.7/43.0
56.7
38.7
59.6
XLM
69.1
80.9
70.1
61.2
59.8/44.3
48.5/32.6
43.6/29.1
56.8
32.6
55.5
XLM-R
79.2
86.4
72.6
65.4
76.660.88
71.6/53.2
1/4.0
66.0
57.3
68.1
VECO
79.9
88.7
75.1
65.7
77.366.8
71.7/53.2
67.6/49.1
85.0
75.1
73.1
Translate-Tran
model English training tronslated data
target langwage
XLM-R 82.6
90.4
80.2/65.9
12.8/54.3
66.5/47.7
XLM-R"
82.8
90.2
72.6
65.4
80.0/65.8 8
73.0/54.3
74.5/58.3
80.2
75.2
74.4
VECO
83.0
91.1
75.1
65.7
79. 79.9/66.3
73.1/54.9
75.0N58.9
89.3
86.9
77.2
Table 1: XTREME results on each dataset (as of Oct 02, 2020). Averaged results on the four categories can be found at leaderboard: https: /opps://sitee //sttsstss.s.//. researchh google/xtrr and "f" indicates results from Hu et al. (2020) and Fang et al. (2020), respectively. "*" indicates the results obtained by our implementation. The detailed results for each language are in Appendix C
"
Models
Datasets
Tasks
Languages
Avg.
Mono.
Bill.
MS.MI
CS-MIM
TIM
.n
de
es
1
tr
zh
MEMAL
76.6
56.1
55.5
VECO
77.1
60.9
64.7
62.5
56.1
55.3
54.7
57.5
56.6
60.6
XLMs
78.3
55.1
72.1
72.5
68.2
51.1
48.8
51.6
52.9
61.2
VECOsHaL
80.0
66.4
75.6
75.6
70.9
58.7
56.1
63.3
62.9
67.7
Table 2: XNLI accuracy scores for each language under the cross-lingual transfer setting. Both XLM and VECO are small-sized models trained from scratch on the same monolingual (Mono.) and bilingual (Bili.) corpus using the same hyperparameters. For this set of experiments, we only use subset of the full training corpus and report the results of languages appeared in them.
during pre-training, which is a direct and effective way to enhance the cross-lingual ability of the model. Second is the mutual improvement between two pre-training tasks and the superiority of model design.
To analyze whether bilingual data plays leading role in improving performance, conduct set more fair experiments. We train small-sized XLM and VECO models from scratch using a subset of full training data and hyperparameters (see Appendix A for details). Table shows the sults of XNLI, the most widely used cross-lingual evaluation dataset in the XTREME. leaderboard. We observe that, when using monolingual corpus only, VECO can outperform XLM by 0.8 points. It suggests that our models can benefit from adjacent sentences used by the CS-MLM task to be equipped with a stronger ability of contextual modeling. Moreover, when trained on both the mono- lingual and bilingual corpus, VECO can achieve larger improvement compared to XLM. It reveals that VECO can better utilize the bilingual corpus, compared to only-optimized translation language modeling (TLM) in XLM.
5 EXPERIMENTS ON LANGUAGE GENERATION
5.1 EXPERIMENTAL SETUP
Datasets We choose machine translation (MT) task, a typical cross-lingual generation scenario. In order to illustrate the generality of our approach and have fair comparison with the most recent state-of-the-art Transformer works (Liu et al. 2020a), we choose two most widely used datasets: (En-De) and English FFrenhh (En-Fr) translation. WMT14 En-De is medium- resource dataset which provides 4.5M pairs for training and validation. We adopt standard newstest2014 the test set. WMT14 En-Fr is high- nigh-resource dataset which contains 36M pairs of parallel sentences. We use newstest201 2 2+newstest2013 for validation and newstest2016 for test. We measure case-insensitive tokenized BLEU with mul t tibblee per1 have fair compari-
WMT14 English-+

---

Under review as a conference paper at ICLR 2021
#Layers
WMT14 En-Fr sacreBLEU
WMT14 En-De
Model
Enc
Dec
BLEU
BLEU
sacreBLEU
Randomly Initialized
Transformer- (Vaswani et al.
2017
6
6
41.0
28.4
Transformer-xBie mmp.)
impl.)
24
42.8
40.3
28.6
27.7
Deep- Transfromer (Liu et al. 2020a
60
12
43.8
41.8
30.1
29.5
Cross- manssllngull Models Inirialized
XLM (Lample & Conncau
2019
6
6
28.9
XLM-R (Conneau al. 2019
24
6
43.7
41.1
30.8
29.9
mBART Liu et al. 2020b
12
43.2
41.0
30.0
29.1
VECO
24
6
44.4
42.0
31.5
30.5
Table 3: Results on WMT14 En-Fr and WMT14 En-De Machine Translation.
son with previous Transformer variants and de-tokenized SacreBLEU avoid the influence of different tokenization and normalization between models (Post 2018).
Fine-tuning Setting In theory, VECO can initialize 24-layer encoder and 24-layer decoder Transformer model. However, since VECO follows the post-layernorm design of vanilla Trans- former (Vaswani et al. 2017) like XLM-R, we find it is hard to train such a deep post-layernorm Transformer variant without careful parameter searching. This phenomenon is consistent with the findings in recent researches about deep transformer (Liu et al. 2020a Bachlechner et al. 2020). Since these works also show that deeper encoders are more worthwhile than deeper decoders, thus the main results of VECO and XLM-R are based on the most normal 6-layer decoder with the full 24-layer encoder. The batch size is 64k and 256k for En-De and En-Fr respectively. The total training updates are set to 100k. The learning rate is le-4/2e-4, with linear warmup over the first 16k steps and linear decay. We run En-De and En-Fr MT experiments 16 and 32 V100 GPUs respectively. We average the last 10 checkpoints and use beam search with a beam size 5.
Baselines We consider two types of Transformer baselines: randomly initialized and cross-ling loos-lln models initialized. For random initialization, we take the original Transformer-big and the state- of-the-art Deep Transformer (Liu et al., 2020a) into consideration. Besides, we also reproduce a Transformer baseline which adopts the same fine-tur minettming hyperparametens as VECO but with ran- dom initialization. For cross-lingual encoder-decoder models, we include mBART, which shows impressive results MT. We also conduct the WMT experiments for XLM-R, following the tally same fine-tuning settings as VECO. Note that the layer settings are not totally same, due to the distinct characteristic decide the layer of MT experiments. Thus the layer of mBART is set the same as pre-training (ie, 12-layer encoder and 12-layer decoder); (2) The layer of pre-trained encoder models only decides the layer of the MT encoder, while the decoder layer can be any value. Thus XLM-R initialized model fixed as 24-layer encoder during fine-tuning. In order to minimize the layer gap between XLM-R and mBART initialized MT model, we choose the most common 6-layer decoder in Table and 3-layer decoder in Table 4. To have a fair comparison to XLM-R, VECO also adopts the same layer settings. We also reproduce same-sized randomly initialized Transformer-xBig model. In conlusion, we tried our best to use the same layer setting among most the models if possible (Transformer-xxiigg XLM-R, and VECO).
pre-trained models: (1) The layer of pre-trained encoder-decoder models
5.2 EXPERIMENTAL RESULTS
Table shows the comparison between VECO and baselines. Compared to existing cross-lingual models. VECO consistently outperform the best models about BLEU. We can also observe that VECO can largely outperform randomly initialized same-sized Transformer-xBig model by BLEU. And even beats the state-of-the-ar Deep-Trane Trap-frraffrmmr with 60-layer encoder and 12-layer decoder on both datasets.
Hash: hww.:/ccwwmmiiceellnnnnneeeaeeee htwp://wwww......o........oooo/o 1atversson..

---

Under review as a conference paper at ICLR 2021
#Layers
WMT14 En-De
Method
Enc
Dec
BLEU
SacreBLEU
Randomly Initialize
24
3
28.5
27.6
29
24
6
28.6
27.7
acreBll LEU
28
Randomly Initialize
More Bilingual Data"
24
6
30.4
29.4
27
VECO Initialize
24
First-3
30.8
29.8
VECO Init.
24
Last-3
31.2
30.3
XLM-R Init.
24
First-6
31.1
30.1
Random Init.
24
Last-6
31.5
30.5
10
15
20
25
30
35
Epochs
Table BLEU scores (left) and learning curves (right) different initialization methods.
Table (left) contrasts two ways initializing a Transformer with n(< 24) decoder layers via se- lecting: (1) the first n layers; (2) the last n layers from a 24-layer pre-trained VECO model. We consider n = {3, 6} conduct experiments. The predominant method adopted this paper, the straightforward strategy of selecting the last layers, exhibits better performance. It is possibly because the last several layers play a more important role in making predictions over the whole vocabulary. Moreover, we can find that there is 0.2~0.3 BLEU improvement when increasing the decoder layers from 6 Regardless of the initialization method, the VECO-initializee model gain consistent 1~2 BLEU improvement
the randomly initialized uundollyinitialized model.
Moreover, Table 4 (right) displays the sacreBLEU scores of same-sized (24-layer encoder and 6- layer decoder) models during training. We find that VECO-initialized model can get surprising more than 28 sacreBLEU score just after 10 epochs, which is better than the final score of the randomly initialized model at 35 epochs. It reveals that VECO can provide fairly good initialization for the machine translation model, which can converge quickly and further boost the results.
To investigate whether the exciting improvement in MT mainly comes from the of parallel corpus during pre-training or 2) the superiority of the designed model and pre-training tasks, we also conduct more comparable experiment. We first train an out-of-domain Transformer-xie using the whole En-De parallel data (~~ 68M) used in VECO pre-training, and then continue to train the model on the in-domain WMT14 14 En-De training dataset. Results are shown in Table 4 (left) marked with #. Under this set of a totally fair comparison, VECO still maintains lead of 1.1 BLEU score. This directly confirms that the improvement on MT is not only due the use of bilingual data. More importantly, reasonable pre-training tasks and model design ensure betler use of bilingual and large-scale unlabeled multilingual corpus.
model
RELATED WORK
Encoder-only Cross-lingual Pre-training mBERT (Devlin et al. 2019) is the first work to pre- train a Transformer encoder over multiple languages. There have been several extensions that use the same encoder-only backbone mBERT, with the main difference is the introduction of more training corpus and pre- pre-training tasks. XLM (Lample & Conneaul 2019) utilizes both monolingual and bilingual corpus to perform mask language modeling. XLM-R (Conneau et al. 2019) extends to be built on RoBERTa (Liu et al. 2019) using larger monolingual training data. Unicoder (Huang al., 2019), ALM (Yang et al. 2020). and InfoXLM (Chi et al., 2020b) propose new pre-training tasks to better utilize the bilingual data. These works deliver impressive performance on cross- lingual understanding tasks, while only marginal improvement has been gained on cross-lingual generation tasks like machine translation, especially on high-resource languages.
Encoder-Decoder Cross-lingual Pre-training BART (Lewis et al.) 2019) pre-trains denois- ing auto-encoder built with a standard Transformer-bas based encoder-decoder architecture. And
'However, we observe that only marginal improvement can he gained when further increasing the decoder layers 12 24 our primary experiments, which is also line with the findings in Liu (020a).
8

---

Under review as a conference paper at ICLR 2021
mBART (Luu et al. 2020b) extends it to the multilingual setting, demonstrating significant gains machine translation, but with a decrease in high resource languages. Chi et al. (2020a) first trains encoder via MLM and then frozen the encoder to train the decoder only via two generative tasks. A similar approach is also proposed in Liang et al.] (2020) that extends Unicoder (Huang et al., 2019) generation tasks, with the main difference exists in the joint train- ing of encoder-decoder. All these cross-lingual models emphasize training a dedicated model for NLG, thus these didn't 't improve or even hurt the NLU capabilities of the encoder. Or they require more computation and memory to match the performance of the encoder-only models when using comparable training resources. For example, BART, when fine-tuming tinettnnig on NLU tasks, the same input is fed into the encoder and decoder, and the final output from the decoder is used. Thus it would cost more memory due to extra cross-attention modules (roughly 10%~~20% more parameters) than encoder-only RoBERTa model, but BART still can't 't performs better than RoBERTa.
in low.meddim-resoure
Unified Language Representation for NLU and NLG BART (Lewis et al. 2019) and UNILM (Dong et al.) 2019) also endeavor to build unified model for NLU and NLG tasks, with the core idea that allows the model to see both unidirectional and bidirectional context. UNILM pre-trains Transformer encoder with ensemble of attention masks, while BART pre-trains Transformer mucoder-decoded model with arbitrary noising functions. Our work differs from them in several ways. Firstly, they only foeus on the monolingual (English) domain, while we dedi- cate to the more challenging multilingual domain. Secondly, since multiple languages cannot be completely mapped to the same space every layer only via self-attention like UNILM, the cross- attention module is important to model the cross-lingual mapping. Thus we drop the way to pre-train an encoder-only model under the multilingual scenario. Thirdly, previous work (Xia et al. 2019) have shown that sharing the encoder and decoder of the Transformer can strengthen the semantic correlation between languages and regularize the encoder-decoder with high capacity on machine translation. Due to the above reasons, VECO varies during training to train specific modules via bidirectional task (IS-MLM) and unidirectional task (CS-MLM).
7 CONCLUSION
We present VECO, variable cross-lingual pre-training model, targeted at initializing both NLU preferred encoder-only and NLG specialized encoder-decoder Transformer. We analyze three core modules in standard Transformer and propose two masked language modeling tasks to train the reasonable combinations of them. The two tasks jointly optimize for inner-sequence understanding and cross-sequence generation, enabling them boost each other via strong regularization from module sharing. On this account, VECO achieves consistent improvements on various language understanding and generation tasks compared existing cross-lingual encoder-only and encoder- decoder approaches, opening up new ways of thinking about pre-trained backbone architecture.
REFERENCES
Mikel Artetxe and Holger Schwenk. Massively Multilingual Sentence Embeddings for Zero-Shot Cross- Transfer and Beyond. Transactions of the ACL 2019, 2019.
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. On the Cross-lingual Transferability of Mono- lingual Representations. In Proceedings of ACL 2020, 2020.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv: 1607.06400 2016. URLhhttss://aaxxi wttp:vvwwwgg/pd//e/ee/ 06450.p6664550ppp
Thomas Bachlechner, Bodhisattwa Prasad Majumder, Huanru Henry Mao, Garrison W Cottrell, and Julian McAuley. Rezero is all you need: Fast convergence at large depth. arXiv preprint 2rXi1:2003.04887, 2020. URL
04887.
Bin Bi, Chenliang Li, Chen Wu, Ming Yan, and Wei Wang- PALM: Pre-training an autoen- coding&aautoregressivee language model for context-conditionee generation. arXiv preprint arXiv:2004.07155, 2020. URL
07159. pddf
Zewen Chi, Li Dong. Furu Wei, Wenhui Wang. Xian-Ling Mao, and Heyan Huang. Cross-lingual natural language generation via pra-iinining. In Proceedings of the AAAI Conference on Artificial Intelligence, 2020a. https::::: //txx/aivvvv wttp:///www...///////////////0/0
10481, f

---

Under review as a conference paper at ICLR 2021
Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, and Ming Zhou. InfoXLM: An information thhooreticc framework for cross- lingual language model pre-training arXiv preprint arXiv:2007.07834 2020b. URL https:
//arxiv. //arxiv.org/pda
Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. TyDi QA: A Benchmark for Ceermattio-Seekinne Question Answer- ing in Typologically Diverse Languages. In Transactions of the Association of Computational Linguistics, 2020a.
Kevin Clark, Minh- Thang Luong, Quoe V Le, and Christopher D Manning. ELECTRA: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555, 2020b.
Minh-Thang
URL http::tttt///eto / htt.://w.eeepeeeeeeeeeeeeee wwtppedfdddddaae 11xr:BtvV1
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of EMNLP 2018, pp. 2475-2448 2018.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Un- supervised crosssiingulleppeeettiiiion learning at scale. arXiv preprint 22w.6v1....i.i
2019.
URL
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019. URL https:
www. www.eb..wbb..o org wawpr///nnhologyy//ao 1423. pdf
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Husn Hon. Unified language model pre-training for natural language understanding and generation. In Advances in Neural Information Processing Systems, 2019. URL https:
Yuwei Fang, Shuohang Wang. Zhe Gan, Siqi Sun, and Jingjing Liu. FILTER: An enhanced fusion method for cross-lingual language understanding. arXiv preprint arXiv:2009.05/66, 2020. URL
wttp:///pp.fotooo pd1
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Training pruned neural net-
works. CoRR, abs/1803.03635,, 2018 URL https:::
03635.
pdf
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016.
URL
03385. pdd
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. XTREME: A massively multilingual multi-task benchmark for evaluating cross lingull general- ization. arXiv preprint arXiv:2003. 1080, 2020. URL https: http//xxvvvv.......
11080.pcl
Haoyang Huang, Yaobo Liang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, and Ming Zhou. Unicoder: A universal language encoder by pre-training with multiple cross-lingual tasks. arXiv preprint 190Xvv/:0900000
2019. httthttss///aaae wwwllvorr//ppoe http////g////eoeeeo http://ff/19999// 00964. pdf
Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv: 0622608886666,88 2018.
BRRL/tttp:://cciv.ooo http://wppppppp/...o..oo 05t.:/
Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. arXiv preprint 19011. 07291.12900.. .pdf
arXiv:/901.07291, 2019,
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. ALBERT: A lite bert for sulf-super learning of language representations arXiv preprint arXiv: 1942, 2019. /R/.htttsss///rriv..orrr.
10

---

Under review as a conference paper at ICLR 2021
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. BART: Denoising eqquence-to-sqquene pre- training for natural language generation, translation, and comprchension. arXiv preprint arXiv: 13461, 2019.
13461.pp
Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. MLQA: Evaluat- ing Cross-lingual Extractive Question Answering. In Proceedings of ACL 2020, 2020.
Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, et al. XGLUE: A new benchmark dataset for cross-lingual pre- training, understanding and generation. arXiv preprint arXiv:2004.01401, 2020. URL https:
201220004..111.11 01401.11111..1 pdf
Xiaodong Liu, Kevin Duh, Liyuan Liu, and Jianfeng Gao. Very deep transformers for neural ma- chine translation. arXiv preprint arXiv:2008.0777: 2020a. URL https://azxiv.orry
pdf 220080077722...
Yinhan Liu, Myle Ou, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pre- training approach. arXiv preprint vrxiv:1907.11000 2019. URL bttps::/aaxxti org/
@ddf/190.. 692.6..... 1
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. Multilingual denoising pre-training for neural machine translation. arXiv preprint arXiv:2001.082/1 2020b. URL /atp:::/t///att //tprii//rxiivoo/ wttp:o/gg//p//....///// 13461,1461.1011 pdf
Joakim Nivre, Mitchell Abrams, Zeljko Agié, Lars Ahrenberg, Lene Antonsen, Maria Jesus Aran- zabe, Gashaw Arutie, Masayuki Asahara, Luma Ateyah, Mohammed Attia, et al. Universal de- pendencies 2.2. 2018
Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. Cross- lingual name tagging and linking for 282 languages. In Proceedings of ACL 2017, pp. 1946-1958, 2017.
Matt Post. A call for clarity in reporting BLEU scores. Proceedings of the Third Conference on Machine Translation, 2018.
Aditya Siddhant, Melvin Johnson, Henry Tsai, Naveen Ari, Jason Riesa, Ankur Bapna, Orhan Fi- rat, and Karthik Raman. Evaluating the cross-lingual effectiveness of massively multilingual neural machine translation. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, pp. 8854-8861, 2020. URL https:/ /aaai, wtt.o///g//o///o//////e// php/AAAI/ artic veticle/view/5ad
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Yan Liu. MASS: Masked sequence to se- quence pre-training for language generation. 2019. ptttpss//arxivv.rrrt 02450.pdE
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, łukasz. Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neu- ral Information Processing Systems, 2017. URL httts::/pappers ni nips cc/paper////cccccccccc
is all you need.pd.
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzman, Armand Joulin, and Edouard Grave. Ccnet: Extracting high quality monolingual datasets from web crawl data, arXiv preprint arXiv:1911.00359 2019. URL httpt::////e/ //tppps://rxvv.../ www./rr.gtp.....
1911. 003000359.00 fff
Yingce Xia, Tianyu He, Xu Tan, Fei Tian, Di He, and Tao Qin. Tied transformers: Neural machine translation with shared encoder and decoder. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, 2019. URL https: http::::/wwwww org///oorr//// /tsgg////////// httpo//ss//n/////////i index.iix.. php/AAAI/

---

Under review as a conference paper at ICLR 2021
Jian Yang, Shuming Ma, Dongdong Zhang, Shuangzhi Wu, Zhoujun Li, and Ming Zhou. Alternating language modeling for cross- loss-lingual pre pre-trrining, In Proceedings the AAAI Conference on Arti- ficial Intelligence, 2020. URL vi view/ fiew/64800/33 6336
Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. PAWS-X: A cross-lingual adversarial dataset for paraphrase identification. In Proceedings of EMNLP 2019, pp. 3685-3690, 2019.
Pierre Zweigenbaum, Serge Sharoff, and Reinhard Rapp. Overview of the third buce shared task: Spotting parallel sentences in comparable corpora. In Proceedings of 1Ith Workshop on Building and Using Comparable Corpora, pp. 39-42, 2018.
A PRE-TRAINING DETAILS
For monolingual data, following XLM-R (Conneau 2019), we build a clean CommonCrawl Corpus using an open-source tool CCNet (Wenzek et al. 2019). rrports the language codes and statistics of datasets. There are 1.36TB monolingual data in 50 languages before up/down- sampling.
We collect bilingual corpus in 50 languages from the OPUS website including MultiUN, UNPC, Bombay, EU-bookshop, OpenSubritles201.. Tanzil, Vlobel ParaCrawl, MultiPararrawl, DGT, Tilde, Europarl, Wikipedia, ECB, TED2013, infopankki-vl, EUconst, and Bianet. In total, there are 1TB bilingual training corpus before pre- processing, covering 879 language pairs. Table lists the statistics for each language pair.
wwwsCoomenttary Ubuntu, Books, UN,
We then apply subword tokenization directly raw text data using Sentence Piece Model (Kudo & Richardson 2018) without any additional preprocessing.
We use the whole corpus to train VECO, while using subset (~ 1/4) that contains 33 languages to train XLMSMALL and VECOSMALL- The full set of pre-training hyperparameters for small-sized and large-sized VECO (default) are listed in Table 7 The comparisons between VECO and other cross-lingual models are shown in Table
B NLU
DETAILS
We consider two typical fine-tuning settings:
Cross-lingual Transfer: For all tasks except sentence retrieve tasks (BUCC and Tatoeba), we select the model with the best average result over all the languages on the dev sets, by searching the learning rate over [le-5,2e-5,3e-5]] training epoch over [3,5,10], and batch size over [16,32,64]. As for sentence retrieve tasks without fine-tuning any parallel sentences at all, we use the average word embeddings in the middle layer to extract the sentence representations.
Trunslate-Truin-AAlt We found that the large-sized training datasets can benefit from smaller learning rate. Therefore, for translatettrainall) we select the model on the dev sets, by searching the learning rate over [3c-6.5c-6.lc-5]. For XNLI, PAWS-X, XQuAD and MLQA, we fine-tune our model directly on the translation data provided by the official XTREME repc"l Following the participants (FILTER and Anonymous on the XTREME leaderboard, we use several methods to further improve the scores. For TyDiQA, we start fine-tuning based on the best XQuAD translate-triin-all model. For the sentence retrieve tasks, BUCC and Tatoeba, we use the averaged representation in the middle layer of the best XNLI model. Note that we only use these tricks under the munslate-train-ut setting, wishing to have possible fair comparison on the XTREME leaderboard. However, we still use less fine-tuned data compared to concurrent works FILTER and Anonymous Specifically, FILTER, model-agnostic fine-tune method, uses more translation data on POS and NER tasks. Anonymous uses other labeled data outside of XTREME.
httso://qpus.nlll..al https://github www.shubb.oommeeeo wtt. goog/ researchh
12

---

Under review as a conference paper at ICLR 2021
Language
#Decument(M)
#Sentence(M)
Size(GB)
af
0.023
0.522
0.107
ar
2.823
42.659
11776
bg
0.919
14.743
5.217
bn
0.750
9.217
4.264
es
3,980
55.754
9.668
de
21.410
310.942
66,333
el
1.740
24.334
9.737
130.087
2,215.534
479.099
es
17.569
267.764
58.774
ct
0.347
5.252
0.877
0.342
5,216
0.613
fr
15,819
267.888
58,023
fa
2.506
43.570
13.831
fi
1.530
23.790
3,940
fy
0.027
0.537
0.054
gu
0.039
0.519
0.228
gd
0.009
0.126
0.020
he
0.755
12.338
3.073
hi
0.536
7.303
3.762
hu
1.816
29.962
6,421
3,417
60,908
11.528
it
9.336
133.006
30.854
27.967
588.926
71.785
jy
0.002
0.138
0.030
0.141
1,756
0.766
kk
0.061
1.545
0.448
ko
11.609
227.396
27.837
It
0.552
7.996
1.480
0.281
4.159
0.798
ms
0.334
3.762
0,455
ml
0.162
2.615
1.025
my
0.045
0.893
0.306
mr
0.059
0.708
0.365
pl
6.642
93.760
19,082
pt
8.623
128.107
25.612
ne
0.080
0.829
0.429
nl
6,513
85,997
16.648
35,887
580.291
203. 105
1.944
31.929
7,056
0.132
2,927
0.902
sw
0.057
0,945
0.179
ta
0.876
20.376
6.422
0,288
4,995
1.721
18.547
291.081
40.321
th
6,278
117.826
27.941
tl
0.166
5.611
0.679
vi 12.183
234.071
37.919
0.460
7.509
2.003
yo 0,0002
0.003
0.0005
zh 27.067
497.408
87.005
Total
382.735
6,475.444
1,360.526
Table 5: The statistics of monolingual pre-training corpus.
13

---

Under review as a conference paper at ICLR 2021
Peir
(Sent/k]
Pair
RbcanA)
Pair
Pheauk)
Fair
(Seat(k)
Pire
Pheat13)
Pile
Palr
Pair
Pale
11.0%
de-be
12751 81
-6384.32
40-sh
19:20
tymm
34.00
0025
kn-w
6.54
pi-x
48061
18.19
17.03
55063
01:11
41-31
00-te
24009
106
06-07
3555.15
791.00
13-6
400
02-0
21.62
12.25
15:78
7944
365
13:14
12857
af-le
30000
3505550
de-1
35656 02
18000.55
5201
155
15.90
10.35
kodl
1266.5
1.23
134 18
1050 57
34.31
3.73
09-00
5696.70
6-66
4245.5
94
204
24-3
0:34
130
22144
11101
16-4
03-8
00115-20
1072 14
11
05:44
1.9
16.40
36-43
10/70
11:50-00
2155.84
266
16
12:00
6221
12:13
1962
1583.2
303.99
2547
1531
615
140055
1--1
14
25.5
16.37
1115
16.37
1000
12-5
1555
13
3109
0.38
19
1959.11
1073
100
40
25
1094.5
137161
1
150
2:55
0925
S756.3 21
94
13:20
1601
14
1922
148
4.9
100 11
706-2 27
17610 18
24
1336
191.3
7.00
19
500054
451.81
1
1121
0.05
17
066
128 99
2300.00
1006
19
255
11.04
2341
138
131
<3
(6
68
16155
12.1
12.50
05260
0200
2:50
1951
19
2-d
112
27
1-2
1:10
1377,40
152.35
10/00
1972
1119
022 11
216.00
1228
12:4
1381
1011
10
16-06
10-00
2/9
25007
8
10:00
456.3
60-00
13:11
1284 [1)
225
1760
11
25:27
10
100 1
19-13
368005
11-01
34th
23095.2
210
19-11
301 31
7601 1
5.6
115783
57
21
-2-1
11:11
191
20
2517255
231533
3.5
2651
17:34
13.00
0.09
1--0
11-p1
455.73
10:00
371
05 00
13:55
00
25
24,42
01 10
Q-1
20
2542
2542.1
14-11
4.50
and
41
20
351
00-11
2
4h24
11:00
473/04
0.00
19:42
400
12-41
TESS
53.5
9.11
1.21
576.30
$76.31
10/
13-9
12051
21.01
5.3n
301 31
221 97
11-41
225/40
12-13
5001
50011
17169
13
555
W011 15
25h
115100
16/9
11:33
25
5
14-10
1055
50.54
191.55
266.5
10
14
47.0 19
13
(0)
491 11
4/1
ania
20755
122
56
11
12-11
120 K0
14
35401
10000
154128
130
1415
4157-2
1:34
0.0
12:00
5154
141.5
$1.85
081.05
67.45
141.3
11.4
16.09
11-1
11-51
1701
162.5
1943
1759.87
162.5
6.24
14
1926:
15:01
1530
10:47
14:0
210
6.24
6.34
405
1261/0
15
2
23
2016
41.05
11-13
11:00
1.6
2
411
1
15322 38
11
18-14
(378.15
11
25.06
617.91
1
15 12
cur
114.11
112201
11
181
139-90
191
11.57
3-1
10
15-1h
14.77
60-03
19
1991
0.1
16
1646635
35
138.54
61
1555.97
0.11
12-11
101.00
1144
10-th
(312.4
1210.55
1112.4
13-2
vale
1500 47
F
15.5
15.51
17.15
16-4
1540
30539.35
26187111
029177
73
155
56
0
2:469.2
99 56
29
02
111
0100
3700265
30.40
31.96
10.71
19496
15:24
5.95
1
16:3
16.21
1214
1808
19
13-1
2
12
1
10510
155 11
10
1754.22
7.25
12:2
[1-41
161.70
204h
2:28
1:30
21111
7.06
158-00
$5.50
12:23
1
BIN
186
43:57
0.16
1451
00
1/01
0
10-10
147
11
175.59
196
190 19
0.8
0153
11:1
3124 m
4710
17
14555
$36.1
30
1
51.58
160-99
011
21
56301
000
1667
255
033
18544
1023
19:21
Wits
47247
30725.5 55
1961881
15181.54
16236
$1500
19715
01
39
45
11
mm
110166
00
301-36
1214
2013
235122
11.51
77.38
3.55
1415
1.55
18:00
0004
1.30
6009
11-23
1977 51
13.11
441
12-11
1
3.20
EX-17
hg-8
2119101
18
36.41
1641
Par
2117
0.95
22857-10
2.1
71.0
1.18
12-13
17775
2764985
714721
71 38
12297
3000
504 57
61861 36
65918.54
7.89
610.8
641.30
435.12
[:39
25008.7
25008.7
31K2
4012
012
127122
5111
15
103
291 25
140.54
$7.10
al-rs
16557
410.2
1655.0
dod
301
0140
60415.4
04-1
34.56
6-1k
4416
605.75
0.12
di-t
39.21
Jo-01
09-6
6-11
18-1c
1.40
4155180
AM1 16
0.71
3761
15hc
342.55
In-al
113075
254K.54
1055.00
15198 (1
17
40-th
10.75
19-0
19.37
18-m
33067
knnll
2221-7
31-8
K:15
hg-k
165111
49-01
23-1a
123 W
21-41
3,50
99.11
15-m4
1514
kn-21
24513.22
1y-m
teml
351/7
15
07
2.00
1337
14-z
(12)
950-4
18-11
2246.2
1503.30
de-t
44302
16.05
40/16
200.28
45.40
15
6495-31
40-6
35.66
15-th
$36/75
Toea 6,22111214
Table 6: The statistics of bilingual (parallel) pre-training corpus.
14

---

Under review as a conference paper at ICLR 2021
Pre-training Hyperparameters
Large
Small
Number of layers
24
6
Hidden Size
1024
768
FFN inner hidden size
4096
3072
Attention heads
16
12
Attention head size
64
64
Embedding Size
1024
768
Mask percent (monolingual/ bilingual)
15%/25%
15%/25%
Learning Rate Decay
Linear
Linear
Warmup steps
12k
12k
Learning Rate
2e-4
3e-4
Adam e
le-6
le-6
Adam 81
0.9
0.9
Adam S2
0.98
0.999
Attention Dropout
0.1
0.1
Dropout
0.1
0.1
Weight Decay
0.01
0.01
Max Sequence Length (monolingua//bilinuaal)
512/128
512/128
Batch Size (monolingual/bilingaa))
1024/4096
1024/4096
Train Steps
240k
240k
Total Parameters
662M
247M
Table 7: The pre-training hyperparameters.
Model
Architccture
Params
Enc Layers
Layers
#Languages
Vocab Size
mBERT
Devlin
2019
Encoder-only
110M
110k
XLM [
Lammpe &
2019
Envoder-only
570M
24
100
200k
XLM-K (Comnen
2019
Encoder-only
550M
24
100
250k
MMTE
Sidchant ct a
2020
Encoder-decoder
375M
6
103
64k
mBART
L.u ct al. 20206
Encoder-dccoder
680M
12
250k
VECO
Variable
662M
24
50
250k
Table 8: Comparison large cross cross-lingual models. denotes encoder and decoder share all self- attention and feed-forward sub-layers.
The full set fine-tuning hyperparameters listed Table 9
[1e-5,2e-5,3e-5] for Cross-lingual Transfer
Learning Rate
[3e-6.5e-6.1e-5] for Translate-Train-a.
Adam e
1e-8
Adam 81
0.9
Adam B2
0.999
Batch Size
[16,32,64]
Train Epochs
[3,5,10]
Table 9: The fine-tuning hyperparameters
C DETAILED RESULTS ON XTREME
The detailed results of each XTREME task under the cross-lingual transfer and manslate-train-aa settings on all languages are listed the following tables.
15

---

Under review as a conference paper at ICLR 2021
Model
bg
de
el
fr
hi
TU
SW
th
tr
ur
Vi
zh
Avg.
Cross-lingual Transfer
MBERT
80.8
64.3
68.0
70.0
65.3
73.5
73.4
58.9
67.8
49.7
54.1
60.9
57.2
69.3
67.8
65.4
MMTE!
79.6
64.9
70.4
68.2
67.3
71.6
69.5
63.5
66.2
61.9
66.2
63.6
60.0
69.7
69.2
67.5
82.8
66.0
71.9
72.7
70.4
74.3
62.5
69.9
58.1
66.4
59.8
70.7
70.2
XIM-RT
77.2
82.5
75.6
79.1
78.0
71.7
79.3
79.2
VECO
88.2
79.2
83.1
82.9
81.2
84.2
82.8
76.2
80.3
74.3
77.0
78.4
71.3
80.4
79.1
79.9
Translate- Translate-Train-All
XLM!
85.0
80.8
81.3
80.3
79.1
80.9
78.3
75.6
77.6
78.5
76.0
79.5
72.9
72.8
68.5
77.8
XLM-R
88.6
82.2
85.2
84.5
84,5
85.7
84.2
80.8
81.8
77.0
80.2
82.1
77.7
82.6
82.7
82.6
86.0
75.3
Table 10:
XNLI accuracy scores for each language. "f" and results are provided by Hu et al. 2020 and Conneau al. (2019), respectively.
Model
de
fr
ja
ko
zh
Avg.
Model
de
Avg.
Cross-Fingual Transfer
Cross-lingual Transfer
mBERT
94.0
85.7
87.4
87.0
73.0
69.6
77.0
81.9
mBERT
62.5
62.6
50.0
56.7
XT.M
94.0
85.9
88.3
87.4
69.3
64.8
76.5
80.9
XLM
56.3
63.9
46.6
56.8
MMTE
93.1
85.1
87.2
86.9
72.0
69.2
75.9
81.3
MMTE
67.9
63.9
54.3
53.3
59.8
XLM-R
94.7
89.7
90.1
90.4
78.7
79.0
82.3
86.4
XLM-R
67.5
66.5
73.5
56.7
66.0
VECO
96.2
91.3
91.4
92.0
81.8
82,9
85.1
88.7
VECO
89.6
84.6
87.4
78.5
85.0
Translate- Train-All
-yanslate-rriin-All
Translate- Train-All
VECO
96.4
93.0
93.0
93.5
87.2
86.8
87,9
91.1
VECO
93.0
88.7
89.9
85.7
89.3
Table 11: PAWS-X X accuracy scores for each language.
Table 12: BUCC results (FI scores).
Model
40
de
25
th
Vi
2b
Avg:
Crossdingnal Tranffer
miRKK
63.5/222
61.5/4511
7067 56.
69.67449
755/566
99.21460
713/5133
42.7/335
55.41401
19.65
5K.04883
64.57490
XIM
742/63.1
6147447
6687499
57.57 W11
354/3445
57.9/4122
147/
49/7/1977
50.1768.1
63.3/463
6887500 50.
61.3/772
$8.1/409
55.8/364
86.577577
6866490
76.7/997
$9.3/00
57.6/765
56.1
79.666.6
81.2/616
78.31 74.7/57.6
75.1/50.
69.2/992
88,3777.9
76.95111
77,86622
92.500.7
71.699.4
79.9666.
Table 13: XQuAD results (FI / EM) for each language.
Model
en
es
zh
Avg.
Cross-lingual Transfer
mBERT
80.2 /
34.3/
59.0/43.8 43.8
67.4/49.2
50.2/35.3
61.2/40.7
59.6 38.6
61.4/
68.6/55.2
42.5/25.2
40.8/37.2
34.4/21.1 /3.7/37.9
48.3/30.2
40.5/21.9
/
MMTE
78.51
56.1
58.4/-
64.9
46.2 /-
59,4/
$8.37-
60.3
XLM-R
83.5/70.6
66.6/477
70.1
7
74.0/52.9 70.6/53.1
62.1/37.0
73.6/
VFCO
83.6 33.6/70.5
65.0/
69.8/554
73.8/ 55.6
/5.4
73.1 /518
67.3//3.6
71.7/53.2
Translate-Train-n Train-All
VECO
84.1/71.3
67.8/47.1
70.7/55.8
74.6/56.6
71.1/53.4
74.8/54.4
68.8/45.8
73.1/54.9
Table 14: MLQA results (F1 EM) for each language.
Model
xr
bn
ko
nu
SW
te
Avg.
Cross-lingual Transfer
73.3/63.6
60.0
49.6 38.4
XLM
65.9/53.9
59.4/41.2
15.0
48.2/414
62,5 45.8
14.2/5.1
49.2 30.7
39/4/216
15.5
4 29.1
MMTE
62.9/49.8
63.1/39.1
55.8/41.9
53.9/42.1
60.9/ 47.6
49.9/42.6
38.9
63.1/472
54.2 45.8
58.1 /43.5
10.9
VECO
71.3/58.2
73.1/52.8
58.9/42.5
70.9/55.1
77.2 60.0
54,2/39.9
/3..6
65.8 45.77
70.6 50.7
6
TramstatePrrain-Al/
VECO
77.2/64.8
77.0/57.5
72.2/56.6
76.6/59.3
80.0/64.4
63.4/52.2
72.8/50.5
79.467.11
76.0/58.0
75.0/58.9
Table 15: TyDiQA-GolP results (FI /M) for each language.
16

---

Under review as a conference paper at ICLR 2021
Model
af
de
cl
fa
fI
fr
bc
hi
hu
id
it
Cross-Rinuull Thansfer
mBERT
86.6
56.2
85.0
85.2
81.1
95.5
86.9
79.
60.7
66.7
78.9
84.2
56.2
67.2
78.3
71.0
88.4
XLM
88.5
63.1
85.0
85.8
84.3
95.
85.8
78.3
62.8
64.7
78.4
82.8
63.9
66.2
77.3
70.2
87.4
МMТE
86.2
85.8
84.3
57.3
89.2
XLM R
89.8
67.5
88.1
88.5
86.3
96.1
88.3
86.5
72.5
70.6
85.8
87.2
68.3
76.4
82.6
72.4
89.4
95.99
88.9
nl
pt
zh
Avg.
mBERT
49.2
70.5
49.6
69.4
88.6
86.2
85.5
59.0
75.9
41.7
81.4
68.5
$7.0
53.2
55.7
61.6
71.5
76.3
66.4
61.2
MMTE
48.6
70.5
59.3
74.4
83.2
86.1
88.1
63.7
81.9
43.1
80.3
71.8
61.1
56.2
51.9
68.1
73.5
XLM-R
15.9
78.1
53,9
80.8
89.5
87.6
89.5
65.2
86.6
47.2
92.2
76.3
70.3
56.8
24.6
25.7
73.8
VECO
31.4
79.3
53.1
84.3
89.8
88.3
90.2
64.3
85.8
48.0
93.7
77.2
69.2
58.1
26.2
39.4
75.1
Table 16: POS results (Accuracy) for each language.
ja
jy
Crovs- Gross-ingnal Transer
mBERT
65.2
77.4
41.1
77.0
70.0
78.0
72.5
77.4
75.4
66.3
46.2
77.2
79.6
56.6
65.0
76.4
53.5
81.5
29.0
66.4
XLM
82.6
74.9
44.8
76.7
70.0
78.1
73.5
74.8
74.8
62.11
49.2
79.6
78.5
57.7
66.1
76.5
53.11
80.7
23.6
63.0
MMTE
77.9
74.9
75.1
68.3
71.8
74.9
62.6
45.6
73.9
73.8
74.1
31.2
63.9
VECO
83.8
77.5
48.2
83.9
77.2
79.4
79.3
75.4
90.4
68.3
68.2
30.6
80.1
55.0
71.0
80.9
52.9
81.7
19.4
63.2
km
km
mn
ms
my
m1
g6
SW
EA
te
1F
Vi
yo
2h
MBBETT
64.6
45.8
59.6
52.3
58.2
72.7
45.2
81.8
80.8
64.0
67.5
50.7
48.5
3.6
71.7
71.8
36.9
71.8
44.9
42.7
XLM
57.2
59.4
63.55
70.1
26.5
32.4
28.9
XLMR
71.6
56.2
60.0
67.8
68.1
57.1
54.3
84.0
$1.9
69.1
70.5
59.5
55.8
73.2
76.1
56.4
79.4
33.6
33.1
VECO
67.1
51.2
59.9
61.4
65.0
70.0
56.1
81.4
83.1
71.3
70.5
60.5
56.2
1.4
71.3
M0.4
69.3
76.0
37.4
29.1
Table 17: NER results (F1) for each language.
Model
bit
de
fr
be
hi
hu
id
Cross-liguual Transfer
BERT
42.7
25.8
49.3
17
77.2
29.8
68.7
29.3
25.5
46.1
39
66.3
41.9
34.8
38.7
54.6
58.4
42
XLM
43.2
18.2
13.5
66.2
25.6
58.4
24.8
17.1
32.2
32.2
54.5
32.1
26.5
45.9
40
XLMR
58.2
47.5
71.6
43
88.8
61.8
75.7
52.2
35.8
70.5
71.6
73.7
66.4
72.2
65.4
77
68.3
60.6
Translate-
Translate-Triin-Al Train-All
78.1
79.8
93.7
94,2
93.0
ka
lik
ko
ml
mr
nl
tn
th
tl
zh
Crous-linuaa Transfer
BERT
17.6
20.5
27.1
38.5
19.8
20.9
68
69.9
61.2
11.5
14.3
16.2
13.7
16
34,8
31.6
62
71.6
22.9
59.6
XLMR
14.1
52.1
48.5
61.4
65.4
56.8
80.8
82.2
74.1
20.3
26.4
35.9
29.4
36.7
65.7
24.3
74.7
68.3
Pranslae-Traii--ll Train-All
82.4
Table 18: Tatoeba results (Accuracy) for each language
17


Reviews:
[{"review_id": "a68JSwGY3U3", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This submission integrates the encoder-only and encoder-decoder Transformer for both understanding and generation tasks through parameter sharing. The authors present a variable encoder-decoder pre-training approach to unify the two mainstreams in pre-training tasks. To be specific, the model shares the self-attention layer and feed-forward layer for both encoder and decoder, while the cross-attention layer is trained in an alternative way. The VECO approach delivers strong results on various cross-lingual tasks of XTREME benchmark for understanding tasks, and generation tasks for WMT14 En-Fr and WMT14 En-De. \n\nComments:\nFirst of all, the VECO approach provides impressive results on the different benchmark datasets including both language understanding and generation. It outperforms the previous methods in a non-trivial margin, which demonstrates the effectiveness of the approach. This is deeply acknowledged. Also, the studies and the detailed statements are mostly enough to make evaluations.  Despite its effectiveness, I do have several concerns and problems:\n* In terms of the technical contribution, the relation between VECO and previous works is hard to make a strong difference, for example, BART, UniLM. Though the authors mentioned different language pairs (multilingual), personally, I acknowledge the parameter sharing of self-attention and feed-forward layer between the encoder and decoder is the main difference, while this is also widely used in machine translation models. Therefore, I feel a little bit unsatisfied with the contributions. \n* As for the shared pertaining method, it is a little bit confused about the parameter updating. For the CS-MLM loss, the decoder will reuse the hidden states of encoder output, are these hidden states fixed during the decoder update? Or they will still be updated to the encoder (I know the parameters are shared)? The implementation detail is not so clear. If the encoder continues updating, how can the GPU memory cost differ from the MT fine-tuning procedure? \n* The results are compared in a clear way, however, the parameters are not clearly compared with previous works. It seems the 662M model size is bigger than previous works. This is not the main weakness.\n\nGenerally speaking, this work is good for benchmark tasks to achieve strong results or a good technical report. But for the ICLR conference, it may need more. \n\n----------\n\nPost-update:\nThanks to the authors for your response. I deeply acknowledge the promising results achieved from your work, which is impressive. I still feel hard about the contribution, though I also acknowledge the difference and it is an excellent practical trick. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "Y1dtS-_FGJ", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose variable encoder-decoder (VECO), a pre-training strategy for both NLU and NLG tasks. In VECO, two masked language model (MLM) are leveraged for pre-training, (a) inner sentence MLM with encoder only, and (b) cross sentence MLM with both encoder and decoder. The parameters of self-attention and feedforward layers are shared in encoder and decoder. Improvements are observed for multiple NLU and NMT tasks.\n\nThe paper is mostly well written and nicely presented. The authors show extensive results in various downstream understanding and generation tasks, improves performances in most cases. The proposed method is simple and straightforward; and can be readily reproduced in any existing toolkit.\n\nI have some doubts on the motivation of separate pre-training tasks for encoder and encoder-decoder. The authors claims that for previous encoder-decoder pre-training like MASS and mBART, \n\"it usually requires more computation and memory to match the performance of the encoder-only models\". What exactly is the additional computation and memory required here? And how is the empirical comparison on these approaches against the proposed one in terms of both model performance and computation/memory cost?\nActually in MMTE [1], the authors show that pre-training a NMT (encoder-decoder) model and extract the encoder is effective for various encoder-only downstream tasks. \n\nSome more comparison with previous pre-training approaches should be presented as well, e.g. MASS, BART, MMTE[1], etc. \n\nIn general, I think the authors did an excellent job validating their method on various different NLU/NMT datasets. However, I'm skeptical about the novelty and the general contribution/impact of the paper.\n\nMisc:\n\n* Table 3, why comparing with different number of encoder/decoder layers for previous methods (e.g. 24/6 for XLM-R, 12/12 for mBART)?\n* Table 3, do you also share parameters (self-attn, ffn) for baseline methods? How much does parameter sharing contribute in terms of performance and training efficiency?\n\n[1] Siddhant, Aditya, et al. \"Evaluating the Cross-Lingual Effectiveness of Massively Multilingual Neural Machine Translation.\" AAAI. 2020.", "extra_scores": {}, "extra_text": {}}, {"review_id": "EjFMnc5Agg6", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a **unified** cross-lingual pretraining method that works well for both natural language *understanding* (NLU)---typically done using *encoder-only* architectures like mBERT and XLM---and conditional natural language *generation* (NLG) tasks like machine translation---typically done using *encoder-decoder* architectures like mBART. More concretely, let $\\mathbf{x}$ be the \"source\" sequence and $\\mathbf{y}$ the \"target\" sequence. In machine translation, $\\mathbf{x}$ and $\\mathbf{y}$ are the source and target sentences, while in the monolingual case, $\\mathbf{x}$ and $\\mathbf{y}$ are two contiguous sequences in the corpus. The pretraining loss consists of the following four terms (Eq. 8): (i) masked language modelling only on $\\mathbf{x}$, (ii) masked language modelling only on $\\mathbf{y}$, (iii) autoregressively decoding $\\mathbf{y}$ conditional on $\\mathbf{x}$ in a left-to-right fashion, and (iv) the reverse of the third loss term, i.e. autoregressively decoding $\\mathbf{x}$ given $\\mathbf{y}$ in a left-to-right fashion.\n\nCompared to prior approaches, this proposed approach has a key benefit of *sharing* the parameters between the encoder and the decoder during pretraining (hence eliminating the need to train a separate decoder component from scratch), and also enables the pretrained model to be used for both NLU (where the unused cross-attention module is simply discarded) and NLG tasks. Experiments demonstrate the strong empirical performance of the proposed approach on both cross-lingual NLU (through the XTREME benchmark) and two machine translation datasets, surpassing the previous state of the art results in many cases. Further analysis suggests that the improvements are not simply due to training on more bilingual data.\n\n**Pros:**\n1. The proposed approach demonstrates strong empirical performance on both cross-lingual NLU and machine translation tasks, outperforming various strong baselines and achieving new state of the art results in many cases. \n\n2. The paper does a great job of analysing whether the improvements come from the proposed approach---which benefits from the encoder-decoder parameter sharing and a loss function that takes into account both intra-sequence masked language modelling and sequence-to-sequence mapping---or simply from more bilingual data. Disentangling the gains from the proposed approach vs more data is really important to understand and enable better progress in the field, which unfortunately is not always done in prior work. So I appreciate the fact that this paper takes a step towards addressing this question, and it is encouraging that the improvements do not merely come from more data (Table 2).\n\n3. The paper is overall well-written and draws extensive connections to the relevant prior work.\n\n**Cons:**\n1. In my understanding, the difference between the proposed approach and mBART (which also unifies cross-lingual NLU and NLG) is pretty minimal---the second paragraph of page 4 mentions that the key difference here is the parameter sharing between the encoder and decoder. Could the stronger performance of the model compared to mBART be simply due to training on more bilingual data? Repeating the ablation experiments as done in Table 2, but comparing with mBART rather than only XLM, would help make the empirical setup much stronger.\n\n2. The pretraining objective in Eq. (8) includes a \"flipped\" sequence-to-sequence mapping term that  predicts $\\mathbf{x}$ autoregressively conditional on $\\mathbf{y}$. In the monolingual case, this runs the risk of ignoring *discourse information* that comes from the sentence ordering. Imagine a very simple example where $\\mathbf{x}=$\"Clyde is an elephant.\" and $\\mathbf{y}$=\"He is very gentle.\". The pronoun \"He\" in $\\mathbf{y}$ is allowed since there is already an antecedent, Clyde, in the previous sentence $\\mathbf{x}$. But this ceases to be true if we flip the ordering of the sentences. Could you please say more about this?\n\n3. In practice, the model parameters seem to be initialised from XLM-R (except for the cross-attention part that XLM-R does not have). This multi-stage training process can potentially benefit from an implicit model combination effect, which the baseline models may not necessarily be able to benefit from. Having equally positive results where the model is trained from scratch can help make the empirical results stronger.\n\n4. Some of the presentational aspects can be improved. More details below.\n\n**Presentation / grammatical mistakes / typos:**\n1. In page 3, \"... is that $\\mathbf{Q} = \\mathbf{K} = \\mathbf{V} ...$\". I am not sure what this means, since it implies that the matrices $\\mathbf{Q, K, V}$ are equivalent.\n\n2. In page 3, the bottom of Eq. (2) says that $\\text{MultiHead}(\\mathbf{Q}=\\mathbf{x}, \\mathbf{K}=\\mathbf{y}, \\mathbf{V}=\\mathbf{y})$. I am not sure if this is correct in the standard case where $\\mathbf{x}$ is the source sentence and $\\mathbf{y}$ is the target sentence. Shouldn't the query $\\mathbf{Q}$ come from the target sequence $\\mathbf{y}$ while $\\mathbf{K}$ and $\\mathbf{V}$ come from source sequence $\\mathbf{x}$?\n\n3. In page 3, \"... and FFN modules to *corporate*\" -> \"cooperate\".\n\n4. At the top of page 4, it is mentioned that \"...we detach $\\mathbf{X}^{(N)}$...to let the two objectives optimized in isolation\". What does \"detach\" mean here, and how does it relate to optimising the two objectives in isolation?\n\n5. In Table 3, \"Randomly *Initialize*\" and \"Cross-Lingual Models *Initialize*\" -> \"Initialized\".\n\n6. In page 7, \"*Figure 4* (left) contrasts two ways ...\" -> \"Table 4\". \n\n7. In page 8, \"... and then *continue-train* ...\" -> \"continue to train\".\n\n**Bottom line**\nOverall, I think the pros outweigh the cons and this paper can be useful for the community, although addressing some of the concerns I raised above can make the paper stronger. I would be willing to reconsider my score based on the response.\n\n**Update after the authors' response**\nThe authors have provided a comprehensive response that address most of my comments, and clearly clarified the novelty and key differences with prior work such as mBART (which also seems to be a key concern in the other reviews). Given the satisfactory authors' response, I am therefore raising my score to \"7\". Overall, I believe this paper presents a good contribution to the field.", "extra_scores": {}, "extra_text": {}}, {"review_id": "832mXEQon9h", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper targets to unify the advantages of encoder-only model and encoder-decoder model for multilingual pre-training. Given three types of parameters in transformer blocks including Self-Attention, Cross-Attention and FFN, this paper proposes to train the model for understanding and generation tasks at the same time, where Self-Attention and FFN weights are shared and trained for both understanding and generation tasks and Cross-Attention weights are trained for generation tasks only. Evaluations are performed on XTREME and WMT datasets, respectively, with good improvements obtained.\n\nStrengths: (1) The approaches are clear and easy to reproduce. (2) The performance on XTREME and WMT are good.\n\nWeakness: (1) Based on my understanding, this approach equals to sharing Self-Attention and FFN weights between MLM encoder, MT encoder and MT decoder, which is not innovative enough. (2) More baseline settings could be added for a more solid comparison. For example, for XTREME, what's the performance of IS-MLM + TLM, which is pre-trained using the full training corpus? (3) what kind of training data is used in the CS-MLM task? I cannot find the data detail from the paper. Is  it the MT data? (4) Do Table 1 and Table 2 use the same training corpus?\n\n- Additional question. (1) For task IS-MLM and CS-MLM, do they use both monolingual data and bilingual data? If yes, which type of data is more useful?  (2) In table 2, do XLM and VECO use the same vocabulary and the same code for MLM? (3) If the answer to question (2) is yes, then the first line (XLM_{SMALL}) in Table 2 corresponds to XLM-R in Table 1 and the fourth line (VECO_{SMALL}) corresponds to VECO in table 2. For XNLI, the gap between VECO_{SMALL} and XLM_{SMALL} in Table 2 is 7.9, but the gap between XLM_R and VECO in Table 1 is only 0.7. What caused this difference? (4) In Table 3, different models use different hyperparameter settings, especially encoder/decoder with different layers. Could the paper use the same layer setting when compare with a specific baseline? This would help to verify that the gain comes from the new pre-training strategy, instead of a deeper model.\n\n- Suggestions: (1) Add more baselines and ablation study, this will help readers to understand where the gains come from. For example, add a baseline model pre-trained by XLM-R and TLM in Table 1. This could prove the gain of VECO is not from TLM. (2) XGLUE, a concurrent work of XTREMEM, includes both multilingual understanding and generation tasks. I suggest the paper can provide the results of VECO on the multilingual NLG tasks in XGLUE.", "extra_scores": {}, "extra_text": {}}, {"review_id": "1yfknqzT4LL", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 9, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe paper presents VECO: a pre-trained encoder-decoder model which is capable of both cross-lingual understanding and generation. They present two losses for inner-sequence and cross-sequence understanding. These components are then used to either build a encoder-only model or an encoder-decoder model. They present results on XTREME for cross-lingual understanding and on MT for generation. The results on both tasks are quite competitive and clearly shows the benefit of using both monolingual and parallel data with IS_MLM and CS-MLM.\n\n\nReasons for score:\n\nI vote for accepting the paper. The paper initializes from XLM-R and then fine-tunes it on monolingual and parallel data in 50 languages. The authors get +3 average gain over the previous best system on XTREME and were ranked #1 at the time of submission. They also get handy gains in the MT benchmarks. The two ablation experiments show that the new CS-MLM task is indeed beneficial and improves the performance. One of the concerns I have is that the authors are not upfront about their model being in just 50 languages and hence might not be comparable to other models like XLM-R which support 100+ languages.x\n\n\nCons:\n- As mentioned above, please be upfront about training only on 50 languages. It's only mentioned in the Appendix. This needs to be mentioned in the main text and maybe even in the results table.\n- I would like to see more information about the amount of data (both monolingual and parallel) listed clearly in the Appendix.\n- What would happen if VECO was trained on 100 languages from XLM-R? That result would be interesting to see in Table 1.\n- I would personally like to see more ablation experiments: experiments where the amount of pre-trained monolingual and parallel data were reduced independently to see the impact it has performance.\n\n\nMinor comments:\nIn Section 4.1, kindly cite all the representative tasks in XTREME as suggested here:\nhttps://github.com/google-research/xtreme#paper\n\n- It would be great if the authors stated the number of monolingual and parallel examples explicitly in Section 3. Appendix A doesn't provide the number of examples but just the size (1TB) of parallel data used.\n\n- Please provide breakdown of amount of monolingual data and bilingual data per language/language-pair. How many languages do you get monolingual data in? It's not clear from the paper.\n\n- Also, please be explicit about the number of languages supported in the model (50). This is hidden in Appendix A. One can argue that this is not a fair comparison against XLM-R since it's trained on 100+ languages.\n\n- Change \"Ours implementation\" to \"Our implementation\" everywhere.\n\nSection 7:\n- Change \"targeting at initializing both...\" to \"targeted at ...\"\n\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a unified cross-lingual pretraining method that works well for both natural language understanding (NLU)—typically done using encoder-only architectures like mBERT and XLM—and conditional natural language generation (NLG) tasks like machine translation—typically done using encoder-decoder architectures like mBART.

This paper clearly split reviewers, with 2 quite or very positive on it, and 3 thinking or leaning towards thinking that it didn't have enough novelty to merit publication.

Pro
- The model produces good SoTA results
- The method is easily replicable
- It is good for the community for leading systems on benchmark tasks to have published papers describing how they work.

Con

- The work is not groundbreaking in technical novelty
- The work has to do a better job of communicating its contributions: It's hard to understand how it differs from other methods

On balance, the overall assessment is that the paper is not yet ready in its current form. The hope is that authors find the reviewer comments useful for preparing a future submission:

- The paper **has** to do a better job of communicating its contributions. All that most researchers got from the first version was that there was parameter sharing and that helped. The revised version starts to do a better job of explaining the value of having the IS-MLM and CS-MLM objectives to doing well on NLU and NLG tasks, but much more is needed, as the discussion here shows. Indeed, even the discussion here is often opaque. In describing the key contribution of the paper, in both the revised paper and discussion, the authors fall back on phrases like "elaborately designed" and "exquisite cooperation of parameter sharing and pre-training tasks". **What do "elaborately designed" and "exquisite cooperation" mean?!?** I think you can minimally clearly explain the benefits of having an objective like IS-MLM for doing better on NLU tasks than the approach taken in mBART. You could argue for the advantages of MLM vs LM generation, which has been shown in other papers, including the original BERT paper and ELECTRA. Concretely, I wonder if you should reverse the contents of section 2 and start with equation (8) and explain why that is a good objective for your system, and better than ones that have been used previously. This discussion should be at a higher level than the current discussion under (8) which tends to be in the weeds. I haven't worked all the details, but I think you could then describe the objectives of section 2.2 before describing the implementation in section 2.1, and the result might be clearer? It would certainly emphasize the importance of these loss functions.
- The initial version didn't have important details like the number of languages covered in the main paper; the current version fixes this to the extent of saying you have 50, but still doesn't give the context of how this compares with XLM-R and mBART. And several reviewers had questions about the number of parameters of different models. I think you could fix a lot of these concerns by moving Table 8 to the main paper in a future resubmission. It doesn't take up much space and helps a lot in providing these details and easy to find citations for the models compared in other papers. 
Decision: Reject