Self-supervised and Supervised Joint Training for Resource-rich Machine Translation
Authors: Yong Cheng
Wei Wang.
Lu Jiang
Wolfgang Macherey
Venue:  (2021)
Keywords: resource-rich machine translation
neural machine translation
pre-training
self-supervised learning
joint training

Abstract: Self-supervised pre-training of text representations has been successfully applied to low-resource Neural Machine Translation (NMT). However, it usually fails to achieve notable gains on resource-rich NMT. In this paper, we propose a joint training approach, $F_2$-XEnDec, to combine self-supervised and supervised learning to optimize NMT models. To exploit complementary self-supervised signals for supervised learning,  NMT models are trained on examples that are interbred from monolingual and parallel  sentences through a new process called crossover encoder-decoder. Experiments on two resource-rich translation bench-marks, WMT’14 English-German and WMT’14 English-French, demonstrate that our approach achieves substantial improvements over a vanilla Transformer and obtains a new state of the art of 46 BLEU on English-French. Results also show that our approach is capable of improving model robustness against input perturbations which is known as a key weakness in contemporary NMT systems.

Under review as a conference paper at ICLR 2021
SELF-SUPERVISED AND SUPERVISED JOINT TRAINING FOR RESOURCE-RICH MACHINE TRANSLATION
Anonymous authors Paper under double-blind review
ABSTRACT
Self-supervised pre-training of text representations has been successfully applied to low-resource Neural Machine Translation (NMT). However, it usually fails to achieve notable gains on resource-rich NMT. In this paper, we propose a joint training approach, Fg-XEnDec, to combine self-supervised and supervised learn- ing to optimize NMT models. To exploit complementary seff-supervised sig- nals for supervised learning, NMT models are trained on examples that are in- terbred from monolingual and parallel sentences through a new process called crossover encoder-decoder. Experiments two resource-rich translation bench- demonstrate
marks, WMT 14 English-German and WMT 14 Baglish-Frennee that our approach achieves substantial improvements over several strong baseline methods and obtains a new state the art of 46 BLEU on English-Fremn Frenchh when incorporating back translation. Results also show that our approach is capable of improving model robustness to input perturbations, particularly to code-switching noise which usually appears on the social media.
INTRODUCTION
Self-supervised pre-training of text representations (Peters al. 2018 Radford et al., 2018) has
achieved tremendous success in natural language processing applications. Inspired by BERT (De- vlin et al.] 2019), recent works attempt to leverage sequence-to to sequenne model pre-training for Neural Machine Translation (NMT). Generally, these methods comprise two stages: pre-training and finetuning. During the pre-training stage, a proxy task, e.g. the Cloze task (Devlin et al.) 2019), used to learn the model parameters the full or partial model is finetuned on a downstream translation task labeled parallel sentences. When the amount of labeled data is limited, studies have demonstrated the benefit of pre-training for low-resource translation tasks (Lewis et al., 2019 Song et al.) 2019).
abundant unlabeled monolingual data. In the second stage,
In many NMT applications, we confronted with resource-rich languages which character- ized by millions of labeled parallel sentences. However, for these resource-rich tasks, pre-training representations rarely endows the NMT model with superior quality and, even worse, it sometimes can undermine the model's performance if improperly utilized (Zhu et al.) 2020). This is partly due to catastrophic forgetting (French, 1999) where prolonged finetuning on large corpora causes the
learning to overwhelm the knowledge learned during pre-training. Several mitigation methods have been proposed for resource-rich machine translation (Edunov et al. 20199 Yang al., 2019 Zhu et al. 2020), such as freezing the pre- pre-trained representations during the finetuning stage.
In this paper, we study resource-rich machine translation through a different perspective of joint training where in contrast to the conventional two-stage approaches, we train NMT models in single stage using the self-supervised objective (on unlaheled monolingual sentences) addition to the supervised objective (on labeled parallel sentences). The biggest challenge for this single-stage training paradigm is that self-supervised learning is less useful in joint training because it provides much weaker learning signal that is easily dominated by the signal obtained through supervised learning. As a result, plausible approaches such as simply combining self-supervised and supervised learning objectives perform not much better than the supervised learning objective by itself.
To this end. we introduce an approach to exploit complementary self-supervised learning sig- nals facilitate supervised learning in joint training framework. Inspired by chromosomal

---

Under review as a conference paper at ICLR 2021
erossovers (Rieger al. 2012). propose new task called crossover encoder-decoder (or XEn- Dec) which takes two training examples as inputs (called parents), shuffles their source sentences, and produces "virtual" sentence (called offspring) by mixture decoder model. The key to approach is to "interbreed" monolingual (unlabeled) and parallel (labeled) sentences through second filial generation with a crossover encoder-decoder, which we call F2-XEnDec, and train NMT mod- els on the F2 offispinng As the F22 offispring exhibits combinations of traits that differ from those found in either parent, it turns out to be a meaningful objective to learn NMT models from both labeled and unlabeled sentences in joint training framework.
To the best of our knowledge, the proposed F "e-ee is among the first joint training approaches that substantially improve resource-rich machine translation. Closest to our work are two-stage approaches by Zhuu et al. (2020) and Yang et al. (2019) who designed special finetuning objectives. Compared to their approaches, our focus lies on addressing a different challenge which is making self-supervised learning complementary to joint training of supervised NMT models on large labeled parallel corpora. Our experimental results substantiate the competitiveness of the proposed joint training approach. Furthermore, our results suggest that the approach improves the robustness of NMT models (Belinkov & Bisk 2018 Cheng al., 2019). Contemporary NMT systems often lack robustness and therefore suffer from dramatic performance drops when they are exposed to input perturbations, even though these perturbations may not be strong enough alter the meaning of the input sentence. Our improvement in robustness is interesting as none of the two-stage training approaches have ever reported this behavior.
We empirically validate our approach on the WMT 14 English-German and WMT 14 English- French translation benchmarks which yields an improvement of 2.13 and 1.78 BLEU points over a vanilla Transformer model baseline. We also achieve a new state of the art of 46 BLEU on the WMT' 14 English-Frenct French translation task when further incorporating the back translation technique into our approach. In summary, our contributions are as follows:
We propose a crossover encoder-decoder (XEnDec) that generates "virtual" examples over pairs of training examples. We discuss its relation to the standard self-supervised learning objective that can be recovered by XEnDec.
2. We combine supersuper and supervised losses in a joint training framework using our proposed Fy-XEnDec and show that self-supervised learning is complementary to super- vised learning for resource-rich NMT.
3. Our approach achieves significant improvements on resource-rich translation tasks and ex- hibits higher robustness against input perturbations, particularly to code-switching noise.
2 BACKGROUND
2.1 NEURAL MACHINE TRANSLATION
Under the encoder-decoder paradigm (Bahdanau al. 2015 Gehring al. 2017 Vaswani al., 2017], the conditional probability P(y x; 0) of target- languae sentence y 31. yJ given source-language sentence æ .i.... 27 modeled as follows: The encoder maps the source sentence æ onto a sequence of / word embeddings e(æ) = ((fr),..,,(x1)) Then the word em- beddings are encoded into their corresponding continuous hidden representations. The decoder acts as a conditional language model that reads embeddings e(y) for a shifted copy of y along with the aggregated contextual representations e. For clarity, we denote the input and output in the de- coder as and y. i.e. = (8), "", ,WJ-1, where (s) is a start symbol. Conditioned on the aggregated contextual representation ej and its partial target input æı₁, the decoder generates y as P(ylæ:; 0) = I[= P(y, [[o] Cj; 0). The aggregated contextual representation is often calculated by summarizing the sentence x with an attention mechanism (Bahdanau et al. (015)). A byproduct of the attention computation is a noisy alignment matrix A x which roughly captures the translation correspondence between target and source words (Garg et al. 2019).
Generally, NMT optimizes the model parameters by minimizing the empirical risk over a parallel training set (æ, y) E S:
Ls(0)=
E (n.
[[(f(x, y: 0), h(y))),
(1)
2

---

Under review as a conference paper at ICLR 2021
where l is the cross entropy loss between the model prediction f(x, y. 0) and h(y), and h(y) de- notes the sequence of one-hot label vectors for y with label smoothing in the Transformer (Vaswani et al. 2017).
2.2 PRE-TRAINING FOR NEURAL MACHINE TRANSLATION
Pre-trainin training sequence coeeenceett-seeeeeee sequence models for language generation is receiving increasing attention in the machine translation community (Song al. 2019; Lewis al. 2019). These methods gen- erally comprise two stages: pre-training and finetuning. The pre-training takes advantage of the abundant monolingual corpus = { learn representations through a self-supervised objective called denoising autoencoder (Vincent et al. 20088) The denoising autoencoder aims at reconstruct- ing the original sentence y from one of its corrupted counterparts. Let y be obtained by corrupting with noise function (=)) and masking words.
Then the pseudo parallel data (y²; yy) is fed into the NMT model to compute the reconstruction loss. The self- supervisd loss over the monolingual corpus U/ is defined as:
Lu ( = E [C((v' y: 0), h(y)))
(2)
E
The optimal model parameters 0 are learned via a self-supervised loss (i) and used to initialize downstream models during the finetuning the parallel training set S.
CROSS-BREEDING: Fz-XEnDec
n
( y
OroesDver
V
<maska token
sentences
of y
Cn.
monoiingual sentences
Le.
(1) FyXEEDDec
FxXEnDec
Figure 1: Illustration of the first filial generation (left) and the second filial generation (right) with XEnDec. Fi is used to generate (y². and produces self-supervised proxy loss Lp,- Fa-XEnDec adopts an additional round of XEnDec to incorporate parallel data resulting in virtual data (Ø. 9) for Equation 3
For resource-ries -ich translation tasks in which a large parallel corpus S and (virtually) unlimited mono- lingual corpora U4 are available, our goal is to improve translation performance by exploiting self- supervised signals to complement the supervised learning.
In F2 F2-XEnDec jointly train NMT models with supervised and self sulf-supe learning objectives in a single stage. We design a new objective Lr, and construct virtual data (æ, .) to bridge the parallel data (in supervised learning) and the pseudo parallel data (in self-supervised learning). The training loss over the virtual data (z, ÿ) computed as:
Cr, (0)
E E yeu (mF yD)e
[e(f(z 9: h()))
(3)
where generating (#. 9) depends on the parallel data (x", 3d and the pseudo parallel data (y², 3).
We propose a method called crossover decoderr (XEnDec) that operates on two sentence pairs. illustrated in Fig. [ the first generation (Fig. 1()) uses XEnDec to combine monolingual sentences, thereby incurring sulf-supe proxy loss Lr, which is equivalent to Lu. The second generation (Fig. 1(2)) applies XEnDec between the offspring of the first generation (y², y) and the parallel sentence (x", y²) to introduce LF,, The final NMT models are optimized jointly on the original translation loss and the above two auxiliary losses.
L()) (s( Lr. (0) Cry (0).
3

---

Under review as a conference paper at ICLR 2021
CFy in Equation 4 is used to deeply fuse superviee and supervised training at instance level, rather than mixing them across instances mechanically. In the remainder of this section, we first detail XEnDec. We then discuss the relation between approach, pre-training, and adversarial training objectives. Finally, we summarize some important techniques used in our approach and present the overall algorithm.
3.1 CROSSOVER
This section introduces the crossover encoder-decoder (XEnDec) an essential subtask for the proposed method. Different from con- ventional encoder-decoder, XEnDec takes two training examples as inputs (called parents), shuffles the parents' source sentences and produces virtual example (called offspring) through mixture de- coder model. Fig. iillstrrae this process. Formally, let (zz, y) de- represents a source
//
T
00O
note training example where = 21," sentence of words and y = -..... Mr is the corresponding target sentence. In supervised training. æ and y are parallel sentences. As we shall see in Section 3.4 XEnDec can be carried out with and without supervision, although we do not distinguish both cases for now.
Figure
2: CrossOver Encoder-Decoder.
XEnDec:
Given pair of examples (æ, y) and (æ',y') y') called parents, the crossover encoder shuffles the two source sequences into a new source sentence æ calculated from:
(5) where m m., my E {0, 1) stands for series of Bernoulli random variables with each taking the value with probability p. If m, = 1, then the i-th word æ will be substituted with the word in x' at the same position. For convenience, the lengths of the two sequences are aligned by padding tokens to the end of the shorter sentence. The crossover decoder employs a mixture model to generate the virtual target sentence. The embedding of the decoder's input & is computed as:
ã m, + (1 - m,)x(:
e(2,) Z [[(U)--)
Σ Au-ijam + +(U/5-) Σ Alg-Dj (1:))) i=1
(6)
where e(-) is the embedding function. 2 2= Z=E --E A/ term where A and A' are the alignment matrices for the source sequences æ and x', respectively. Equation 6 averages embeddings of and y' through the latent weights computed by m. A and A'. The alignment matrix measures the contribution of the source words for generating a specific target word (Och & Ney] 2004 Bahdanau et al., 2015). For example, Aj: indicates the contribution scores of each source word for the j-th word in the target sentence. For simplicity, this paper uses the attention matrix learned in the NMT model as noisy alignment matrix (Garg et al. 2019).
is the normalization
Likewise, we compute label vectors for the crossover decoder by:
h(ū;)
[h(yg) 7
Ayam + h(v/)) Σ A/,(1- m.)]]
A=1
The 1((-) function projects a word onto label vector, e.g. one-hot vector. The loss XEnDec over (x, y, x', yy) is computed as the negative ligeliiel of generating the virtual sentence
e(f(@, ē ÿ. 0), h()) log P(j 0) Σ EyEn(E)) log c(
(8)
Notice that even though we do not directly observe the "virtual sentences" and y. we are still able compute the loss using their embeddings. In practice, the length of z is x whereas lengths of both ÿ and ū are max( ( (v")).
3.2 RELATION WITH OTHER WORKS
Before introducing the proposed method, this subsection shows that the crossover encoder-decoder XEnDec (when fed with appropriate inputs) yields learning objectives identical to two recently pro- posed self- superviee learning approaches, MASS (Song et al., 2019) and BART (Lewis et al.
4

---

Under review as a conference paper at ICLR 2021
Table 1: Comparison with other objectives from the perspective of XEnDec with the inputs. Each row shows an input to XEnDec and its relation to existing work. Vmask is sentence of length lv/ containing only "(mask)" tokens. n(y) is a sentence obtained by corrupting all the words in y through randomly shuffling and dropping. Dodv and Yods are adversarial sentences in which all the words are substituted with adversarial words.
y
y
Objectives
y
y
Yinakk
MASS
(ong al. 2019
71
y
Yinnsk
BART
Lewis et al
2019
a
y
Tadv
Yadv
Doubly Adversarial
Cheng et al. 2019
Algorithm 1: Proposed F2-XEnDec function.
Input: Parallel corpus S. Monolingual corpus 24, Shuffling ratios pı and Pa
Output: Batch Loss L(0).
Function F3 Pa-XEnDec (S. (S.4.pi.Pa)
foreach (x" y²) S do
Sample y with similar length as x"; // done offline.
(y² y) +- XEnDec over (n(g), y) and (ymook. y) with the shuffling ratio pı;
Ls + compute l in Equation usinn (x* (z") and obtain A;
CF. compute Equation using (y y) and obtain A';
(#t t) + XłnDec over (x", and (y/: y) with A, A' and the shuffling ratio pa;
LF22 compute Equation
end
return C(0) (0) + Lr, (0) (o, // Equatien
2019) as well supervised learning approach called Doubly Adversarial (Cheng et al. 2019). Table summarizes the inputs of XEnDec to recover the learning objectives of these existing ap- proaches.
XEnDec can be used for self-supervised learning. Given arbitrary alignment matrices, if we set x' ='= yy y y, and æ to be a corrupted y, then XEnDec equivalent to the denoising autoencoder which is commonly used to pre-train
models such as in MASS (Song al. 2019) and BART (Lewis et al. 2019). In particular, if we allow x' to be a dummy sentence of length y containing only "(maask)) tokens, Equation ylields the learning objective defined in the MASS model (Song et al.) (219)) except that losses over unmasked words are not counted. Likewise, as shown Table we can recover BART's objective by setting = n(y) where n²-) is noise func- tion that alters by shuffling tokens or dropping them. In both cases, the crossover encoder-decoder trained with self-supervised proxy objective reconstruct the original sentence from one of its corrupted sentences. Conceptually, denoising autoencoder can be regarded Dee in which the imputs are two types of source correspondences for a monolingual sentence, e.g. n(y) and Ymask for y. Even though this connection is trivial, it helps illustrate the power of the XEnDec when
seguencc-to-sequeee
Equation&Jyields
degenerated XEn-
fed with different kinds of input sentences.
XEnDec can also be used in supervised learning. We can achieve the translation loss proposed in (Cheng et al. (2119) by letting x' and y be two "adversarial inputs", Badv and yodur both which consist of adversarial words each position. For the construction of Vades refer Algorithm in (Cheng et al.) 2019). In this case, the crossover encoder-decoder is trained with a supervised objective over parallel sentences.
3.3
WHAT DOES Lr, LEARN?
The core term LF, introduced in our training objective ( Equation 4 aims to combine parallel and monolingual training examples respectively in supervised and self-supervised learning, Over simply combining Es and Lr, which mechanically mixes the losses across instances, Ly, further enhances the combination by deeply fusing them at instance level. Let take example in Figure 3 to explore what Lr, really learns. Cr, on the virtual data (ø, g is optimized by performing the following tasks simultaneously:
5

---

Under review as a conference paper at ICLR 2021
x? Jintian Tianqi feichang hao
yf Stock MASK reaches MASK height
y The weather today is very good.
y Stock market reaches new height
Stock Tianqi feichang MASK height..
0 is
0000
ú Stock
market
reaches
new
height
PAD
The weather
today is very good
Figure 3: Illustration of Fb D--XDDDec which is used to incorporate parallel data resulting in virtual
data (ă, y) for Equation 3 "0.85/0.15" indicates the soft combination ratio to generate the virtual data y at each position, e.g. E/= An; ZZ = 0.85 in Equation
1. The NMT model is required to factor the fused source sentences out into two sentences at the target side guided by the alignment.
2. The NMT model is required to predict the tokens corresponding to "MASK" tokens and copy unmasked tokens in the source sentences y once is factored out from æ. For example, predict "MASK" in I to "new".
3. The NMT model is required to translate words x? once it is factored out. For example, translate "Tianqi" to "whether".
4. The NMT model is required to predict tokens that are not picked in the source sentence of the virtual data but can be inferred from the left contextual words. For example, although "market" does not correspond to any word in æ, it can still be speculated to some extent from the words "Stock", "MASK" and "height". "today" is more likely to be predicted based on "Tianqi" and "feichang".
5. The NMT model required to separate the loss portion of each above task from the mixed target side.
Integrating all the above five tasks into one task Cr, via the fused virtual data (œ; 9) would expect have the following advantages: (1) each of the above tasks integrated in L harder task than the individually designed task because integration introduces confusions; (2) the virtual points have more intelligent regularization effect than simply summing up the losses of individual tasks, This is because a virtual data point which deeply fuses the training examples from two specific tasks -self-supervised and supervised tasks) enables the model to try its best decouple this integration; (3) training the model over the loss LF, by integrating the above five tasks performs much more efficiently than simply training five individual tasks.
3.4 TRAINING
Algorithm delineates the entire procedure to compute the final loss L()). Specifically, each time we sample a monolingual sentence to avoid the expensive enumeration Equation B) To speed up the training, we group sentences by length during offline computation. For the noise function n(-) in Step 4, we follow follow[Lampee et al. (2017) and locally shuffle words while keeping the distance between the original and new position at < There are two techniques to boost the final performance.
Computing A The alignment matrix A is obtained by averaging the attention weights across all layers and heads. We also add a temperature control the sharpness of the attention distribution, the reciprocal of which was linearly increased from 0 to during the first 20K steps. To avoid overfitting A, we apply dropout A. Notice that we stop pooc-proppggting gradients through A.
Computing hw)) Instead of interpolating one-hot labels in Equation 7 we use the prediction vector f(x, y; 0) estimated by the model where Ô indicates no gradients are back-propagated over it. How- ever, the predictions made at early stages are usually unreliable. We propose to linearly combine the ground-trus -trut one-hot label with the model prediction using parameter e, which is computed as (x y: 0) (1 v))((1)) where is gradually annealed from to during the first 20K steps.

---

Under review as a conference paper at ICLR 2021
Table 2: Experiments on WMT' 14 English-German and WMT 14 English- Frenihh translation.
Models
Methods
En-+De
De-+En
En-+Fr
Fn
Reproduced Transformer
28.70
32.23
Base
Fa-XEnDec
30.46
34.06
Reproduced Transformer
29.47
33.12
43.37
39.82
Ott et al. 2018)
29.30
43.20
Big
Cheng el al. | (2019)
30.01
Yang ct al. 2019)
30,10
42.30
Nguyen et (2019)
30.70
43.70
Zhu et al. (2020)
30.75
43.78
F2-XXenDec
31.60
34.94
45.15
41.60
4 EXPERIMENTS
4.1 SETTINGS
Datasets We evaluate our approach on two representative, resource-rich translation datasets, WM English-German and 14 English-French across four translation directions, (De->En), English~>Frende (En->Fr), and French-Finglish (Fr-xEn). To fairly compare with previous state-of-the-arr results on these two tasks, we report the case-sensitive tokenized BLEU scores calculated by the multi-bleu.perl script. The English-German and English-French datasets consist of 4.5M and 36M sentence pairs, respec- tively. The English, German and French monolingual corpora our experiments come from the WMT' 14 translation tasks. We concatenate all the newscraw107- 13 data for English and German, and newscraw107-14 for French which result in 90M English sentences, 89M German sentences, and 42M French sentences. We kenized words into sub-word units. For English-German, we build a shared vocabulary of 32K sub-words units. The validation set is newstest2013 and the test set newstes(2014. The vocabu- lary for the English-French dataset is also jointly split into 44K sub-word units. The concatenation of newstest2012 and newstest2013 is used as the validation set while newstest2014 is the test set.
English~>Genne (En->De), German->Englss
word piece model (Schuster & Nakajima 2012) to split to-
Model and Hyperparameters We implement our approach on top of the Transformer model Vaswaani al. 2017) using the Lingvo toolkit (Shen et al., 2019). The Transformer models follow the original network settings /Vaswani et al., (017). In particular, the layer normalization employed after each residual connection rather than before each sub-layer. The dropout ratios are all set to 0.1 for all the Transformer models except for the Transformer-bip model on English- German where 0.3 is used. We search the hyperparameters using the Transformer-base model on English-German. In our method, the shuffling ratio p1 is set to 0.50. P2 is sampled from Beta distribution Beta(2, 6). The dropout ratio for A 0.2. For decoding, we use a beam size of and length penalty of 0.6 for English-Germin for English-Frencle We carry out our experiments on a cluster with 128 GPUs and update gradi- ents synchronously. The model is optimized with Adam (Kingma & Ba 2014) following the same learning rate schedule used (Vaswani et al. 2017), except for warmup which set to 4000.
and beam size of 5 and length penalty of 1.0
4.2
MAIN RESULTS
Table 2 shows results comparing our method with several baseline methods on the English- German and English-French datasets across four translation directions. Ott et al.) (2018) investigated how to train a strong Transformer model scale. Our reproduced Transformer model performs comparably with their reported results. Cheng et al. (2019) improved NMT by introducing adversarial attack and defense mechanisms in supervised learning. Nguyen et al. (2019) presented an effective method to boost NMT performance by adopting multiple rounds of translaaddd sentences. Both Zhu et al. 2020) and Yang et al.) (2019) incorporated the knowledge of pre-trained models into NMT models by treating them as frozen input representations for NMT.
For English-Gern German, our approach achieves significant improvements in both translation directions over the standard Transformer model (up to +2.13 BLEU points the English- German trans-

---

Under review as a conference paper at ICLR 2021
Table 3: Results on Fe-XEnDec + Back
Table 4: Results on artificial noisy inputs. "CS": code-switching noise. "DW": drop-words noise.
Translation. English-German is based on the Transformer- base model and Maglish-Frence is based on the Transformer-big model.
Transfomer 12MEEDDe
2545
Methods
En De
En FF
Transformer
28.70
43.37
20
Back Translation
31.38
35.90
164
Edunov al. (2018)
45.60
F2 Pe--XEnDec
30.46 32.41
45.15
+ Back Translation
46.01
64
10%
10%
325
Noise type
Table 5: Effect of monolingual corpora sizes,
Table 6: Finetuning vs. Joint Training.
Methods
Mono. Size
En En->De De
Methods
En DD
x0
28.70
Transformer
28.70
29.84
+ Pretrain Finetune
28.77
F2-XEnDec
x3
30.36
x5
30.46
Fa-XEnDec (Joint Training)
30.46
x10
30.22
+ Pretrain Finetune
29.70
lation). Even compared with the strongest baseline on English our approach obtains +0.85 BLEU gain. More importantly, when we apply our approach to a significantly larger dataset, English-Frenne French with 36M sentence pairs (vs. English-Gorn German with 4.5M sentence pairs), still yields consistent and notable improvements over the standard Transformer model. The gains over the best baseline (Zhu et al. 2020)) on the larger dataset extend to +1.37 BLEU points, which further corroborates the superiority of our approach resource-rich language pairs.
4.3 ANALYSES
Back Translation as Noise Function One widely applicable method to leverage monolingual data in NMT is back translation (Sennrich et al. 2016b). A straightforward way to incorporate back translation into our approach is to treat back-translated corpora parallel corpora. However, back translation can also regarded as a noise function n²-) ( in Xx-XeDDec (shown in Fig. which can increase the noise diversity. As shown in Table 3 for Eeermahn trained Transformer- base, our approach yields an additional +1.95 BLEU gain when using back translation to substitute n(.) and also outperforms the teekstrtnnltiionn baseline. When applied to the English-French French dataset, we achieve a new state of the art result over the best baseline (Edunov et al. 2018). This is interesting because, consistent with what found in previous works, e.g. (Caswell et 2019), the standard back translation for English- French hurts the performance of Transformer. These results show that our approach complementary to the mack-translation method and performs more robustly when back-translated corpora are less informative.
Robustness to Noisy Inputs Following (Cheng al. 2019), we validate the robustness of NMT models by word perturbation. Specifically, we design two types of noise to perturb the clean dataset. The first one is sode-switching noise (CS) that randomly replaces words in the source sentences with their corresponding target-language words. Alignment matrices are employed to find the target- language words the target sentences. The other one drop-words noise (DW) that randomly discards some words in the source sentences. Table 4 shows that approach exhibits higher robustness than the standard Transformer across all noise types and noise fractions. In particular, approach performs remarkably stable when exposed to the code-switching noise.
Effect of Monolingual Corpora Sizes Table 3 shows the impact of monolingual corpora sizes on the performance for our approach. We find that approach already yields improvements over the baseline when using a monolingual corpus with size comparable to the bilingual corpus (1x). As we increase the size of the monolingual corpus to 5x, obtain the best performance with 30.46 BLEU
8

---

Under review as a conference paper at ICLR 2021
scores. However, continuing increase the data size fails to improve the performance any further, and a bigger model with increased capacity might be needed which we leave for future work.
Finetuning vs. Joint Training To study the effect of pre-trained models the Transformer model and our approach, we use Equation| 2 to pre-train an NMT model on the entire English and Ger- man monolingual corpora. Then finetune the pre-trained model the parallel English-German corpus. Models finetuned on pre-trained models usually perform better than models trained from scratch at the early stage of training. However, this advantage gradually vanishes training pro- gresses (cf. Figure in in the Appendix). As shown in Table 6 Transformer with finetuning achieves virtually identical results as a Transformer trained from scratch. Using the pre-trained model over our approach impairs performance. We believe this to be caused by a discrepancy between the pre-trained loss and our joint training loss.
Ablation Study Table 7 studies the effect of different components and verifies the design in our approach. When LF, is removed, we can- not obtain A' Algorithm which leads to the failure of calculating LEz We design a prior alignment matrix to handle this issue (cf. [A.3). We find that removing any component includ- ing training techniques (such as dropout A and using model predictions to substitute h(v)) can result in slightly lower performance. We also tried using Mixup as proposed by (Zhang et al. 2018)) to replace our second XEnDec but ob- serve a large drop compared Fe-XEnDec. It is worth noting that just applying XEnDec over parallel data (last row) also achieves promising results compared to the Transformer baseline.
Table 7: Ablation study on German.
Different Settings
BLEU
Transformer
28.70
Fa-XEnDec
30.46
without LF;
29.55
without LF,
29.21
without dropout A and model predictions
29.87
without model predictions
30.24
the second XEnDec is replaced by Mixup
29.67
Ls with XEnDec over parallel data
29.23
5 RELATED WORK
The recent past has witnessed an increasing interest in the research community on leveraging pre- training models to boost NMT model performance (Ramachandran et al. 2016 Lample L & Conneau
2019 (ong et al. 2019 Lewis et al., 2019 Edunov et al. 2018 Zhu et al., 20200] Yang et al. 2019)) Most successes come from low-resource and zero-resource translation tasks. Zhu et al. (2020) and Yang et al. (201)) achieve some promising results on resource-rich translations. They propose to combine NMT model representations and frozen pre-trained representations under the common two- stage framework. The bottleneck of these methods is that these two stages decoupled and sep- arately learned. Our method, on the other hand, jointly trains self-supervised and supervised NMT models to close the gap between representations learned from either of them with essential new subtask, XEnDec. We also show that our method consistently outperforms previous approaches across several translation benchmarks. Another line of research related ours is mixing the pixels in labeled image data (Zhang et al., 2018 Yun et al., 2019 Jiang et al.) 2020) to enhance generaliza- tion and robustness. Our XŁnDec shares the commonality of combining example pairs. However, XEnDec's 'ss focus is on -oreencc----seeencee learning for NLP with the aim of using self-supervised learning to complement supervised learning in a joint training framework.
6 CONCLUSION
This paper has presented a joint training approach, F2-XEnDec, to combine self-supervised and su- pervised learning in single stage. The key part is novel cross encoder-decoder which can be used to "interbreed" monolingual and parallel sentences. Experiments on two resource-rich trans- lation tasks, WMT' 14 English- Gersan and WMT' 14 English-French show that our joint training performs favorably against two-stage training approaches and improves the NMT robustness against input perturbations, particularly to code-switchin noise. In the future, we plan to further examine the effectiveness of our approach on larger-scale corpora. We also want to design more meaningful noise functions for approach.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Dzmitry Bahdanau, KyungHyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations, 2015.
Yonatan Belinkov and Yonatan Bisk. Synthetic and natural noise both break neural machine trans- lation. In International Conference on Learning Representations, 2018. 8.
Isaac Caswell, Ciprian Chelba, and David Grangier. Tagged back-translation. arXiv preprint UrXiv:1906.. 06442, 2019.
Yong Cheng, Lu Jiang, and Wolfgang Macherey. Robust neural machine translation with doubly adversarial imputs. In Association for Computational Linguistics, 2019.
Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep
bidirectional transformers for language understanding. In North American Chapter of the Associ- ation for Computational Linguistics, 2019.
Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. Understanding tack-translation at scale. In Empirical Methods in Natural Language Processing, 2018.
Sergey Edunov, Alexei Baevski, and Michael Auli. Pre-trained language model representations for language generation. arXiv preprint arXiv:1903.0972 2019.
Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 1999.
Sarthak Garg, Stephan Peitz, Udhyakumar Nallasamy, and Matthias Paulik. Jointly learning to align and translate with transformer models. arXiv preprint
2019.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. Convolutional sequence sequence learning. In International Conference on Machine Learning. 2017.
Lu Jiang, Huang, Mason Liu, and Weilong Yang. Beyond synthetic noise: Deep learning on controlled noisy labels. In International Conference on Machine Learning, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. arXiv, pp- arXiv-1901, 2019. 2019.
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Mare' Aurelio Ranzato. Unsupervised machine translation using monolingual corpora only. arXiv preprint arXiv:1711.0004, 2017.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequueee pre- training for natural language generation, translation, and comprehension. arXiv preprint 1rXiv:1910.13461, 2019.
Xuan-Phi Nguyen, Shafiq Joty, Wu Kui, and Ai Ti Aw. Data diversification: An elegant strategy for neural machine translation. arXiv preprint 1XXi::9111001980 2019.
Franz Josef Och and Hermann Ney. The alignment template approach to statistical machine trans- lation. Computational linguistics, 2004.
Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. Scaling neural machine translation. arXiv preprint 1006:01800000111 2018.
Matthew Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In North American Chapter of the Association for Computational Linguistics, 2018.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training, 2018.
10

---

Under review as a conference paper at ICLR 2021
Prajit Ramachandran, Peter J Liu, and Quoc V Le. Unsupervised pretraining for sequence quence learning. arXiv preprint arXiv:1611.0268: 2016.
Rigomar Rieger, Amnd Michaelis, and Melvin M Green. Glossary of genetics and cytogenetics: classical and molecular. Springer Science Business Media, 2012.
Mike Schuster and Kaisuke Nakajima. Japanese and korean voice search. In 2012 IEEE Interna- fional Conference on Acoustics, Speech and Signal Processing (ICASSP), 2012.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units, In Association for Computational Linguistics, 2016a.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving nerual machine translation models with monolingual data. In Association for Computational Linguistics, 2016b.
Jonathan Shen, Patrick Nguyen, Yonghui Wu, Zhifeng Chen, Mia X Chen, Ye Jia, Anjuli Kannan, Tara Sainath, Yuan Cao, Chung-Cheee Chiu, et al. Lingvo: a modular and scalable framework for gruuence-to-eqqeeme modeling. arXiv preprint arXiv: 090.: 2019.
Kaitao Song. Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Yan Liu. Mass: Masked sequence sequence pre-training for language generation. In International Conference Machine Learning, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems, 2017.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In International Conference on Machine Learning. 2008
Jiacheng Yang. Mingxuan Wang, Hao Zhou, Chengqi Zhao, Yong Yu, Weinan Zhang, and Lei Li. Towards making the most of bert neural machine translation. arXiv preprint arXiv: /00iv::9000000 1908.0562, 05672, 2019.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In IEEE International Conference on Computer Vision, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference Learning Representations, 2018.
Jinhua Zhu, Yingee Xia, Lijun Wu, Di He, Tao Qin, Wengang Zhou, Houqiang Li, and Tie- Yan Liu. Incorporating bert into neural machine translation. arXiv preprint arXiv:2002.06822 2020.
A APPENDIX
A.1 TRAINING EFFICIENCY
When training the vanilla Transformer model, each batch contains 4096 128 tokens of parallel sentences on a 128 P100 GPUs cluster. The training speed is about 0.5 steps/sec on average. As there are three losses included in our training objective |Equation 4 and the input for each of them are different, we evenly spread the GPU memory budget into these three types of data by letting each batch include 2048 x 128 tokens. Thus the total batch size is 2048 128 x 3. The training speed on is about 0.3 steps/sec on average which is about 60% of the standard training speed. The additional computation cost partly comes from implementing the noise function n(-) to construct the corrupted y", which actually can be reduced by caching noisy data in the data input pipeline. Then the training speed can accelerate to about 0.4 steps/sec.
11

---

Under review as a conference paper at ICLR 2021
20-
-.5
25-
Trandeme
5.5x00
L.ccot
1Sxiot
0.5023
1.012
20xUp
3.0K2
4.202
Figure 4: Comparison of finetuning and training from scratch using Transformer and Fx-XEnDec. In both methods, pre-training leads to faster convergence but fails to improve the final performance after the convergence. The comparison between the figures shows our joint training approach on the left (the blue curve) significantly outperforms against the two-stage training on the right. Final BLEU numbers are reported in Table
A.2 TRAINING DETAILS
Data Pre-processing We mainly follows the pre-processing pipeline httpps:///tt thub. com/
which also adopted by Ott et al. (2018), Edunov et al.) (2018) and Zhu et al. (2020), except for the sub-word tool. To verify the consistency between the word piece model (Schuster & Nakajima [012] and the BPE model (Sennrich et al.] (0)6 we conduct comparison experiment to train two standard Trans- former models using the same data set processed by the word piece model and the BPE model re- spectively. The BLEU difference between them is about =0.2, which suggests there is significant difference between them.
Batching Data Transformer groups training examples of similar lengths together with varying batch size for training efficiency (Vaswani et al.) (017)). In our approach, when interpolating two source sentences, x" and y², is better if the lengths of x. and y are similar, which reduce the chance of wasting positions over padding tokens. To this end, in the first round, we search for monolingual sentences with exactly the same length of the source sentence in a parallel sentence pair. After the first traversal of the entire parallel data set, we relax the length difference to 1. This process is repeated by relaxing the constraint until all the parallel data are paired with their own monolingual data.
Applying Mixup In Table 7 we tried substitute the basic operation XEndee with Mixup in our approach. When applying Mixup on pair of training data (æ, y) and (x'.8", Equa- tion Equation 6 and Equation 7 can be replaced by e(ū,) = de(a,) + (1 {)e(z(). e( Xe(uş-1 (1 A)e(y/_1) and h(ū;) = Xh(u;) + (1 A)h(o/), respectively. À is sampled from a Beta distribution. Then we can compute CF, accordingly.
A.3 A PRIOR ALIGNMENT MATRIX
When LF, is removed, we of calculating LF,,- Thus we propose a prior alignment to tackle this issue. For simplicity, we set n(-) to be copy function when doing the first XEnDec, which means that we just randomly mask some words in the first round of XEnDec. In the second XEnDec, we want to combine (x* yd) and (y) y). The alignment matrix A' for (y,, y) is constructed as follows.
obtain A' according Algorithm which leads to the failure
If word y, in the target sentence y is picked in the source side which indicates y is picked and 1-p its attention value A/e if m, 0 is assigned to -T otherwise it is assigned to | | m, m: = 1. Conversely, If word yy is not picked which indicates m, = 1, its attention value A is assigned |
2 if m, O, otherwise is [1-m]] if m;
12


Reviews:
[{"review_id": "wHRVt23lpr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis submission proposes a joint training of self-supervised training and supervised training for neural machine translation (NMT), especially for the rich-resource language datasets. The method proposed, F$_2$-XEnDec, exploits the \"crossover\" operation of the monolingual sentences and bilingual data pairs in the encoder-decoder framework, and train the self-supervised (on monolingual data) objective and supervised (on bilingual data) objective, and the mixed version (crossover) together, without a clear per-train and fine-tuning stage like previous BERT related works. Experiments are conducted on two rich-resource language pairs: WMT14 English-German and WMT14 English-French translation tasks. The results set a new state-of-the-art record for English-French, and superior performances are achieved on English-German language pairs.  The authors also analyze the robustness and ablation studies of their method. \n\nOverall comments:\n\nThe method proposed in this submission is interesting, the crossover operation conducted on the monolingual dataset is similar to the previous works like MASS and BART, but different in the bilingual sentences mixed version. The authors give clear descriptions and comparisons with previous methods. Another difference is that previous works only adopt pre-train and fine-tune stages separately, while this method is to jointly train the self-supervised and supervised objectives, on both monolingual data and bilingual dataset. The crossover operation of the source sentence is natural to be straight forward, while it is interesting for the target side sentences, both on monolingual and target sentences. The most impressive part is the experimental results, the authors give very strong performances on two widely-acknowledged benchmark datasets. The results are very competitive and demonstrate the method is quite effective. \n\nDetailed comments:\n\nThere are many good points of this submission, as I talked about in the last paragraph. Besides, the writing is clear and easy to follow. The overall presentation is high-quality in terms of writing, e.g., the formulations, the figures. Therefore, I would like to turn to point out some concerns from my side, and I still want to know more about this work, please see the follows:\n\n* The first point is about the crossover operation performed on the second stage, the monolingual data and the bilingual data. The source sentences ($x$, $x'$) are directly mixed together through a mask operation to be a new source sentence, and the target sentences are mixed through a weighted version, as the equation (6) and (7) show. This releases some questions to me: (1) the source sentence $x^p$ and the noised source sentence $y^{\\diamondsuit}$ is combined through equation (5), but this mix the space from both source and target, and also interrupts the meaningful space. This augmentation operation is different from the previous works, therefore, how to guarantee this operation is reasonable and effective. It seems more explanations need to be shown. (2) Similarly, as for the target sentences, the mixed version of the monolingual target $y$ and bilingual $y^\\{\\diamondsuit}$ is most probably a non-sense sentence. Therefore, it is hard to think of the reason or the motivation for such mixing, especially in a weighted version as equation (7) and (6). More explanations and interpretations are needed for those operations. (3) Of course, the operation is similar to the mixup operations proposed previously, but the relationship is not clearly discussed in this submission. Mixup is the most related work since the method in this paper, or jointly training method can be regarded as a data augmentation method with two loss objectives from both monolingual and bilingual sets. Therefore, I feel okay about the story presented and motivated by the pre-train methods in this work, but I do feel the most related work is mixup and related data augmentation methods for NMT. \n* Specifically for the crossover operation, besides the relations and explanations for the mixed operation, the weighted label vectors are computed in an attention-based approach by leveraging the attention matrices. As $A$ and $A'$ are not good at the first training steps, the authors add the trick of linearly increasing temperature for them. Hence, the related question is to see the performance of label vector without the weighted version, with a similar operation as equation (5). The advantage of this weighted label vector compared to others need to be discussed since this is more complex than other methods, the training cost is also increased. The importance of the normalization factor $Z$, the necessity of current formulation. More motivation for this modeling is expected. \n* As for the training tricks, for computing $h$, I do not clearly understand why predicted $f$ should be added to the target labels. Is it related to scheduled sampling, or what else? Why the label vector is formulated in such a way? \n* Since the authors compared with other pre-train methods, MASS, BART, therefore, the experimental results are expected to make a clear comparison. \n* For the details of the experiments, the authors use $128$ GPUs to run for the experiments, which is a huge number that is not easy for others to reproduce. The authors are encouraged to open-source the code and settings. A small question is for the dropout values, is attention dropout and relu dropout adopted? It seems to be not clear. The training cost should also be discussed since the algorithm contains several dependencies in the three objectives.\n* In analyses, the authors see the effect of monolingual corpora size for English-German task, which contains $4.5M$ bilingual datasets, therefore, the monolingual size of $3\\times$, $5\\times$ and $10\\times$ are less than $89M$ and $90M$ as used in the experimental settings, but the results $30.46$ is from $3\\times$, is it correct? If it is like this, the authors should give more descriptions of this point to give people a clear data setting. One another small question is about the mixup replacement mentioned in ablation study, can you give more words on how you perform mixup in this setting, $x=x_1 + x_2$ corresponds to what operations for two sentences with several words. Is it similar to the masked replacement as equation (5)?\n\nIn a word, I want to hear more about the details of this submission and I am willing to increase the score if questions are answered. \n\n----post-update----\n\nHi, I thank the authors to give useful feedback about several concerns. But I am still questioned about the cross-sentence operation, even the authors give an example of the different tasks, which is hard to convince me for such kind of learning can achieve such results. The 5 tasks as the author described are much harder, which somehow indicates that bilingual data is not so necessary for machine translation (but it should be aware, this is not unsupervised machine translation, the training spirit is totally different).  Random data augmentation could be very very strong. The authors are expected to give a code implementation and the trained model to give a more convincing result. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "Sb5GNSHIJeS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a method, inspired from genetics algorithms, to trained jointly self-supervised and supervised NMT. The claims are this new method and state-of-the-art BLEU scores for En-Fr WMT14\n\n\nStrengths:\n- the method inspired from genetics is original and interesting\n- the method improves over vanilla Transformer\n- the analysis is interesting by pointing out the key parameters of the proposed method\n\n\nWeaknesses\n- the main weakness of this paper is by far its evaluation: the comparisons of BLEU scores presented in Table 2 and 3 are meaningless. For instance, Zhu et al. (2020) uses sacreBLEU while this paper use multi-bleu. On En-Fr, it is well-known that the differences between these different BLEU may be within 5 BLEU points (!). It is then impossible to assess whether this work is really better than previous work from these results. sacreBLEU must be reported instead of, or in addition to, multi-bleu scores. Moreover, the work compared in Table 2 used different training data and different pre-processing, the difference in BLEU may then only come from these data differences and not from the frameworks. I actually expect this method to be worse than Zhu et al. (2020) considering that their sacreBLEU scores is less than 2 BLEU points below the multi-bleu scores reported in this paper. The second half of the paper has to be rewritten almost entirely to make a meaningful evaluation of this approach.\n- the motivation for using cross-over encoder-decode is difficult to understand, probably because it is only done very briefly in Section 1 (I would recommend to make separate section for motivation)\n\nQuestions:\n- Section 4.1: how reporting tokenized BLEU scores is fair while previous work, mentioned in Table 2, report sacreBLEU scores? You probably misread Section 5.1 from Zhu et al. (2020). For Nguyen et al. (2019), this is unclear what are the reported BLEU scores, but for xx-to-English it seems to be also sacreBLEU scores. Ott et al (2018) also report sacreBLEU scores in addition to multi-bleu. I think you have all you need to replace the scores in Table 2 by sacreBLEU scores. It would dramatically improve the paper.\n- Some of the methods compared in Table 2 uses different sets of training data and preprocessing, maybe I misunderstood something, but even with comparable BLEU scores, how can we assess that this is the proposed method that is better and not the training data or preprocessing? To make a scientifically valid conclusion I think there is no other way than reproducing previous work. There is no need to reproduce all the work from Table 2.  I think the most relevant one to reproduce is probably the work of Zhu et al. (2020).", "extra_scores": {}, "extra_text": {}}, {"review_id": "PUpszc1E-gh", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "summary:\n--------\n\nThis paper introduces a new approach to semi-supervised training of neural machine translation. During training, the traditional supervised loss is complemented with two auxiliary losses: a denoising autoencoder and what the authors call cross-breeding: shuffling together the source side of a sentence pair with an unrelated, noised monolingual sentence, and predicting a virtual target sentence whose embedings and labels are a linear interpolation of the paired target sentence and the monolingual sentence, with interpolation weights being based on an attention matrix of the input words selected during shuffling. Authors report an improvement of around 2 BLEU on WMT14 EN-DE and EN-FR over a purely supervised Transformer, and improvements of 1 BLEU (EN-DE) and 0.4 BLEU (EN-FR) over a system with back-translation. Additionally, the system shows robustness against a type of \"code-switching\" noise.\n\nstrengths:\n----------\n\n+ novelty: while there has been some work trying to combine autoencoding and supervised learning for NMT, the idea to create training instances with virtual outputs that combine both tasks is new to my knowledge.\n\n+ strong empirical results: the training objectives compare favourably to previous attempts to use autoencoding as an auxiliary objective for high-resource NMT. It is compatible with back-translation (using back-translations as noise function), and in this setup, obtains a new SOTA on English-French.\n\n+ ablation studies: the paper demonstrates the contribution of the different training objectives, and that joint training is superior to the more commonly used pre-train+fine-tune strategy.\n\nweaknesses:\n-----------\n\n- obscure description. The paper makes heavy use of metaphors from biology, describing the approach in terms of \"interbreeding\" or \"marrying\" sentences and creating \"offspring\" over two \"filial generations\". I found these terms obscuring rather than informative. For example, phrase \"we use XEnDec to marry monolingual sentences\" is a really roundabout way of expressing that you apply masking to a sentence. Describing a denoising autoencoder as a special case of the crossover encoder-decoder (section 3.1-3.2) also doesn't work for me, because what makes the model special, the creation of a virtual target sentence based on an alignment matrix, isn't applied here.\n\n- robustness results are overclaimed. I was expecting robustness to similar types of noise than in previous work, but the \"code-switching\" noise employed (randomly replacing source words with the aligned words from the reference translation) is unrealistic and eerily similar to the training scenario (including the reliance on an alignment matrix). There is no strong improvement in the face of \"drop-word\" noise, and if this noise were of practical relevance, the much simpler strategy of applying word dropout at training time is likely to remedy it.\n\n- open questions about efficiency. Authors use model predictions as decoder inputs (3.3), which prevents parallelisation across time steps at training time. Can authors quantify the training speed difference between the vanilla objective and their combined objective?\n\n- open questions about where improvement comes from. The main novelty is L_F2, which shuffles two inputs and creates a virtual target sentence for prediction. Why this objective is effective is still a bit of a mystery to me, and I would have liked more insights into this. Most pressingly, using the predictions as decoder inputs reminds me of work on scheduled sampling, and could have a substantial effect that may be unrelated to self-supervised learning. How much does using the prediction vector as input embedding contribute to the quality? (Ablation studies indicate that dropout on A and using the predictions together contributes 0.6 BLEU, which is about half of the benefit of L_F2) Does the vanilla objective improve when using model predictions instead of (or in addition to) gold inputs?\n\n- related to the above question (where does improvement come from?), back-translation. Best results are reported when using back-translation instead of a denoising autoencoder. It's unclear how this result fits into the narrative. If authors still consider back-translation \"self-supervised learning\", then this weakens the contribution that \"we show that self-supervised learning is complementary to supervised learning\" (which is well-known for back-translation already), and also doesn't fit the narrative in the introduction that self-supervised learning \"provides a much weaker learning signal that is easily dominated by the signal coming from supervised learning\". If back-translation experiments are seen as separate from the self-supervised learning experiments, it is misleading to imply that self-supervised learning achieves a new state of the art.\n\nrecommendation:\n---------------\n\nI'm currently borderline on this paper. The main strength of the paper are the strong empirical results, but I have misgivings about some of the central claims of the paper, most importantly the robustness results and positioning the paper as a way of strengthening the self-supervised learning signal, when the reason for the performance improvement is still obscure.\n\nadditional questions to the authors and minor points:\n-------------\n\n- some implementation details were unclear to me. When using back-translation as a noise function n(·), do you still apply masking in addition? If so, is there a good rationale why masking would only be helpful for back-translated data, but not for real parallel data? I also don't understand what you mean with \"applying XEnDec over parallel data\". Are you mixing two sentence pairs (shuffling the source sentences together and predicting an interpolated target sentence)?\n\n- it was not made clear until 3.3 that L_F1 is the same as L_U.\n\n- equation 7: the second sum should be \\sum_{i=1}^I\n\n- I note that English-German results with back-translation are not directly comparable to Edunov et al. (2018) because they are based on Transformer base. Why did you not report results with Transformer big? \n\nUPDATE\n-----------\n\nthe author response has addressed some concerns (efficiency concerns; unclarities about some ablations studies; reason for not comparing to Edunov for EN-DE) well, and I have slightly increased my rating.\n\nI maintain that the description is needlessly obscure in places, even in the revised version (while it is technically correct that a denoising autoencoder could be described as a crossover encoder where the two parent outputs are identical, this is needlessly convoluted), and a major concern about the framing of the paper was not resolved in the response (if back-translation is self-supervised learning, then some of the claims about limitations of self-supervised learning are untrue, and the findings lose novelty. If back-translation isn't self-supervised learning, then it's misleading to imply that self-supervised learning led to a new SOTA). ", "extra_scores": {}, "extra_text": {}}, {"review_id": "COMfBmr2Gls", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a joint training strategy that combines supervised learning on parallel data and self-supervised learning on monolingual data for NMT. The monolingual sentences are corrupted with word order shuffling and masking. And a cross encoder-decoder is introduced to fuse the parallel source-side sentence and the corrupted monolingual sentence. The NMT models are optimized jointly with cross-entropy loss of parallel data, reconstruction loss for monolingual data, and cross-entropy of the virtual fused data.\n\nThe proposed method is simple and straightforward, with seemingly good results for rich-resource languages; can be easily reproduce in any existing toolkits.\n\nMain results (table 2) should compare with BT and noised BT (Edunov et al. 2018) using the same amount of monolingual data as well. Current comparisons with previous BT/pre-training approaches in the main results (table 2) are not exactly fair, since lots of them are not leveraging large-scale monolingual data of all available newscrawl combined. (correct me if I'm wrong). \n\nI'm also a little skeptical about the fusion of source-side sentence x and target-side sentence y* into the input sequence \\tilde x. Would this potentially hampers self-attention in the encoder or the enc-dec attention? Empirically, the last row of table 3 (+BT) shows that replacing n(y) with BT sentence x' leads to significantly better performance, which might be caused by this? \nThis also leads to the question, what is the best fusing scheme (i.e. input (x, y), (x', y')) for the proposed method? Is this the most data efficient way of combing parallel and mono data compared with previous objectives (e.g. noisy BT, DAE, MASS, etc.)?\n\nHow is the training and convergence speed for the proposed method, compared with standard Transformer, with BT/noised BT with same amount of mono data?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: in this submission, the authors propose a sophisticated pretraining strategy for neural machine translation based on the paradigm of self-supervised learning. despite some interesting and potentially significant improvement in various machine translation settings, the reviewers as well as i myself could not determine where specifically those improvements come from. is it their particular strategy of pretraining or is it just self-supervised learning in general? in order for this question, which i believe is a key question to be answered, more thorough ablative experiments and/or comparison to other self-supervised learning based pretraining algorithms, such as MASS & BART which were discussed as similar and motivational in the submission, must be done. when these are done, the submission will be much stronger and attract much more interest.
Decision: Reject