Quick Back-Translation for Unsupervised Machine Translation
Authors: Benjamin Lincoln Brimacombe
Jiawei Zhou
Venue: EMNLP 2023 Findings (2023)
Keywords: unsupervised machine translation
back-translation
non-autoregressive generation
Transformer
TL;DR: A quick back-translation approach to augment standard back-translation to improve unsupervised machine translation performances efficiently, on top of Transformer models by repurposing the encoder for non-autoregressive generations.

Abstract: The field of unsupervised machine translation has seen significant advancement from the marriage of the Transformer and the back-translation algorithm. The Transformer is a powerful generative model, and back-translation leverages Transformer's high-quality translations for iterative self-improvement. However, the Transformer is encumbered by the run-time of autoregressive inference during back-translation, and back-translation is limited by a lack of synthetic data efficiency. We propose a two-for-one improvement to Transformer back-translation: Quick Back-Translation (QBT). QBT re-purposes the encoder as a generative model, and uses encoder-generated sequences to train the decoder in conjunction with the original autoregressive back-translation step, improving data throughput and utilization. Experiments on various WMT benchmarks demonstrate that a relatively small number of refining steps of QBT improve current unsupervised machine translation models, and that QBT dramatically outperforms standard back-translation only method in terms of training efficiency for comparable translation qualities.

Quick for Unsupervised Machine Translation
Benjamin Brimacombe
Jiawei Zhou Harvard University jzhou02eg. harvard.ed
Columbia University
bb2808@columbia. edu
Abstract
2018a; Artetxe et al., 2018), and more recently pair generative language modeling with iterative back-translation (Conneau and Lample, 2019; Song et al., 2019; Nguyen et al., 10.1). Iterative back- translation (BT) creates synthetic translation pairs from monolingual corpora, and uses them to train neural MT models with a reconstruction loss (Sen- nrich et al., 2016a; Conneau and Lample, 2019). Models trained with iterative back- tock-translation hold the state-of-the-ar (SOTA) performance on vari- ety of UMT scenarios (Conneau and Lample, 2019; Song et al., 2019; Roziere et al., 2020; Nguyen et al., 2021). ). Recent work suggests that increased data diversity during buck-translation improves translation performance, and further increasing di- versity remains open challenge (Wang et al., 2019; Nguyen al., 2021).
The field of unsupervised machine transla- tion has seen significant advancement from the marriage of the Transformer and the back-translation algorithm. The Trans- former is a powerful generative model, and back-translation leverages Transformer's high-quality translations for iterative self- improvement. However, the Transformer is encumbered by the run-time of autoregressive inference during translattt,,,, and back- translation limited by lack of synthetic data efficiency. We propose two-for-one improve- ment to Transformer tack-translttion: Quick Tack-Translttion (QBT). QBT re-purposes the encoder as a generative model, and uses encoder-gent groodee-generated sequences to train the de- coder in conjunction with the original autore- gressive back-translation step. improving data throughput and utilization. Experiments on var- ious WMT benchmarks demonstrate that a rel- atively small number of refining steps of QBT improve current unsupervised machine transla- tion models, and that QBT dramatically outper- forms standard mack-translation only method in terms of training efficiency for comparable translation qualities.
tack-trrnslation involves massive amounts of sequence generation, which usually follow an au- toregressive (AR) inference procedure. In AR infer- ence, tokens are generated one step at a time, which can not be parallelized (Vaswani et al., 2017; Kasai et al., 2020). Non-autoregressive (NAR) Trans- former models have been proposed as a method to improve inference speed for translation models, in which sequence transduction is done in parallel (Gu al., 2018; Wei et al., 2019; Zhou and Keung, 2020).
1 Introduction
The capabilities of neural machine translation (MT) models have seen considerable advancement in recent years (Vaswani al., 2017; Hassan al.. 2018; Nguyen et al., 2021). However, paired trans- lation data for supervision is often difficult to pro- cure without specialized expertise, especially for low-resource languages. Proposed originally in the statistical machine translation regime (Ravi and Knight, 2011; 1; Klementiev et al., 2012), unsuper- vised machine translation (UMT) aligns languages without parallel data by leveraging vocabulary- level statistics or abstract language representa- tions. Modern UMT approaches combine self- supervised learning and statistical methods (Sen- nrich et al., 2016a; He et al., 2016; Lample et al.,
In this paper, we address both autoregressive inference and data diversification limitations with novel, framework for UMT, called Quick Back- Translation (QBT). With QBT, the Transformer en- coder is trained as a bidirectional generative trans- lation model. In what we call the Encoder Back- Translation (EBT) step, the encoder back-translates its synthetic outputs, training an indepen- dent UMT model. The encoder-synthesiin transla- tions are then leveraged as synthetic signal for the decoder, providing increased data diversity in what we call an Encoder Trac-Transltted Distillation (EBTD) step. The Transformer architecture is kept

---

allowing the new optimization procedures to supplement en- hance the original back-translation objective. The QBT methods add more diverse training signals for UMT by dual usage of the encoder both as a module inside the Transformer and as a standalone NAR translation model, which offer significant speedup on top of standard BT. We conduct ex- periments on multiple translation benchmarks of different resources, and QBT is able to achieve UMT qualities on par with previous SOTA but with high efficiency in run-time, especially for long se- quences.
intact under Quick
translation, all in the fully unsupervised scenario. They find that more synthetic data diversity im- proves even high quality UMT models.
Non-autoregressive Generation The Trans- former uses an autoregressive (AR) decoder, in which each token output conditioned on pre- viously seen tokens. Inference consists of mul- tiple generation steps increasing with the sequence length. BT with Transformer relies on this ineffi- cient step, and hence is often conducted on short se- quences, using large amounts of compute (Edunnov et al., 2018; Song al., 2019; Nguyen et al., 2021). Non- non-autoregressive (NAR) models (Gu et al., 2018; Wei et al., 2019) have been proposed as solution to improve inference speed for MT models. NAR MT conducts full sequence transduction in paral- lel. However, the parallel decoding requires an assumption of conditional independence between sequence tokens, which is believed to hinder model performance due to the underlying multi-modal distribution of natural language (Gu et al., 2018: Guo et al., 2020; Song al., 2021). A number of NAR techniques leverage knowledge distilla- tion with an AR teacher to improve training signal and mitigate the independent factorization problem. Common NAR Transformer models usually mod- ify the decoder computation by removing the AR self-attention mask and feeding sequential inputs all at once (Gu al., 201 8; Zhou and Keung, 2020), whereas in our approach we directly re-purpose the Transformer encoder for the NAR generation in the context BT to improve UMT learning efficiency.
2 Background
Unsupervised Machine Translation For many language pairs there do not exist sufficient exam- ples to train a supervised translation model. It has been proposed to leverage monolingual data only to learn map between languages, called Unsu- pervised Machine Translation (UMT). Originally proposed probabilistic decipherment problem (Ravi and Knight, 2011 Klementiev et al., 2012), modern UMT now leverages representation learn- ing (Lample et al., 2018b) and pre-trained gener- ative models (Artetxe et al., 2018; Conneau and Lample, 2019; Nguyen et al., 2021). Generative- model-based UMT techniques rely on language model pre-training and self-supervised learning (Radford et al., 2018). Common pre-training objec- tives include Masked Language Modeling (MLM), Causal Language Modeling (CLM), Denoising Au- toencoding (DAE), and the Masked Sequence to Sequence (MASS) objective (Vaswani al., 2017;
We propose counter-intuitive flip of the dis- tillation from strong to weak models, and use weak, bidirectional encoder improve strong, AR decoder. Furthermore, our weak NAR model is hiding in plain sight in the Transformer architecture. Our architecture pushes the synthetic data diversity findings from Nguyen et al. (2021) to an extreme in which the distillation model is highly limited NAR architecture. We find that this is sufficient to improve upon the BT-only algorithm, improv- ing training speed and translation performance in a variety of experimental scenarios.
Devlin et al., 2019; Conneau and Lample, 2019; Song et al., 2019), which are used for initializing UMT models.
Back Translation Back- Back-Translation (BT) refers model or set of models that recursively train using reverse synthetic translations. Originally pro- posed to supplement parallel data during super- vised MT training (Sennrich et al., 2016a), BT has been extended to multi-agent scenarios (He et al., 2016), and fully unsupervised translation in the statistical machine translation regime (Artetxe et al., 2018) and recently with language model- ing (Lample et al., 2018a; Conneau and Lample, 2019; Song et al., 2019). Most recently, the Cross- model Back-translated Distillation (CBD) method (Nguyen et al., 2021) used translatnnn be- tween models pre-trdined with independent back-
Large Language Models for UMT The advent of large language models (LLMs) has greatly im- proved many tasks with zero-shot and few-shot in-context learning. However, UMT is still rele- vant, especially when the resource is limited terms computation and language data. is ar-

---

guable whether some LLMs can perform MT in an unsupervised manner, as the massive scale of web data might already contain paired language cor- pora (Magar and Schwartz, 2022). Nevertheless, in-context back-translation with pseudo parallel ex- amples has been shown to outperform other prompt- ing methods for language translation (Zhang et al.. 2023). In addition, back-translation and UMT tech- niques remain prominent for the leading large mod- els such as mBART (Liu al., 2020), and simi- lar full encodrrddecode architectures are still em- ployed such as mT5 (Xue et al., 2021) and FSMT (Ng et al., 2019). Our work aims to improve the efficiency of BT algorithms with encoder-decoder Transformers, orthogonal to the development of versatile LLMs.
There is major difference between the self- attention computation of the encoder and decoder in sequence generation. The encoder self-attention allows each token representation to attend to all the other tokens before and after it, thus bidirectional. For the decoder, as the future tokens are unknown before being generated sequentially, each token representation at can only attend to the previous tokens y<t, which is autoregressive. During con- ditional generation such as MT, the encoder only needs to be run once with full parallelization on æ, and the decoder needs to be run n steps for every t, making it a bottleneck for generation speed.
Back-Translatee Translation By translating the back- ward direction from target to source, BT (Sennrich et al., 201 6a) provides synthetic parallel corpus for data augmentation used by many UMT approaches. In particular, BT generates x. ~ DD from a reverse model (= based on target sample y Dt. and then uses (Ir y) pseudo pairs for supervision for learning the source target model (v(y)az). For UMT, BT mostly used in an iterative fashion (Hoang et al., 2018), switching between the source language s and the target language to jointly im- prove the language alignments and translation qual- ities (Lample et al., 2018a). In our setup we also tie both translation directions with the same model. essentially making Φ A.
3 Preliminaries
Let DD and DD be the monolingual corpus of sen- tences in a source domain and a target domain i, respectively, and they are not aligned. Given a source sentence x = (I1, Z2,..................... ,mm ~ D, with length m, UMT aims to learn a conditional gen- eration model (y((x) with parameters 0 so that y (y1, 92...., Un.) D; with length is closest the true translation for x after decoding.
Transformer Model The Transformer (Vaswani al., 2017) is sequence-to-sequenee model with architecture built with atten- tion mechanisms. Given the source sentence xr models the conditional target distribution via
Quick Back- Translation
an
Iterative BT with Transformer relies on sequential generation of translations from the decoder, which may not be parallelizable due to its autoregres- sive design. We propose Quick TransSTrnssttton (QBT), non-autoregressive (NAR) generation ap- proach to provide increased synthetic data and re- duced inference time during BT. In contrast com- mon NAR Transformer designs, we directly purpose the Transformer encoder as an NAR gen- eration model without any modifications. In partic- ular, the Dack-translation from target to source x is obtained via
he Enc(Emb(2;; We); 0t)
ha, Dec(Emb(y<s) Wa), , he; 0a) po(y)a:) x Softmax(Linearrha: Wao))
(1)
where he and ha are hidden states out of encoder and decoder computations, / denotes generation steps, We and Wa are embedding matrices for en- coder and decoder inputs, Wao is the weight matrix for decoder output to map ha to the vocabulary size, and 0, and f₂ are parameters for the encoder and decoder layers, respectively. Each layer of the encoder and decoder is composed of self-attention modules followed by feedforward networks, and the decoder layer also consists of additional cross- attention modules attending to encoder outputs he-
he Enc(Emb(y; Wc):
(2)
(o(xly) x Softmax((incar((M (ho Wo.))
and the argmax of the probability distribution is taken for each position independently in the se- quence. Here the only new parameter is the output transformation matrix Weo which we tie with the encoder input embedding W.. = W..
We ignore bias terms linear layers hereafter. 'There are also positional embeddings. layernorm and residual connections which we skip discussion for simplicity.

---

ENG 11
ENG Bidirectional Encoder
ENG
Train
Bidirectional
Autoregreesive
Bidirectional Encoder
Autoragresaive Decoder
Decoder
111 F FRE
En?
EN
<En> E
FRE
FRE
FRE
Generato
Bidirectional Encoder
Auteregressive Decoder
Bidirectional Encoder
Autoregressive Decoder
Bidirectional Encoder
Autoregressive Decoder
ENG
<P
FR
ENG
ENG
(a) Back-Transiatio
(b) Quick Back-Translation: EBT
Quick Back-Translation EBTD
Figure (a) Back-Tramee Translation (b) Encoder
(EBT) (c) Encoder Trcc-Trrnslated Distillation
(EBTD). BT, EBT and EBTD compose the Quick Back-Translation (QBT) procedure. The bottom row shows the generation step for back-translation,, and the top row shows the training step with pseudo pairs. Light grey boxes indicate a forward step with no gradient updates. Dark grey boxes indicate module that is not use. An upward arrow next to the box indicate forward run of the module, and downward arrow indicate gradient and parameter updates.
This essentially shorteuts the computation from input to output by ignoring the decoder. It makes the translation NAR using only the bidirectional encoder whose computation fully parallelizable with linear time complexity, with constant genera- tion steps. Note however that output length is now confined to equal input length. An illustration of QBT in comparison with BT is presented in Fig- ure As additional benefit to the fast aun-imee of encoder synthesis, QBT provides increased syn- thetic data diversity to the BT procedure, improving machine translation performance. the following we describe different components of applying QBT for improving UMT models.
decoder models (Gu et al., 2018; Gu and Kong, 2021) trained for UMT. More details are provided in Algorithm (see Appendix A).
4.2 Encoder Back-Translater Distillation
NAR generation still suffers from degeneration is- sues due to simplification of the sequential depen- dency structures (Gu et al., 2018; Ran et al., 2021; Guo et al., 2020). QBT focuses accelerating the BT sampling process and increasing synthetic data diversity during UMT training. AR decoder based BT may be used in-conjunction with QBT to further improve translation quality. We propose Encoder Back- Trkkks-raddttdd Distillation (EBTD) to use encoder generated sequences as synthetic su- pervised data to train the decoder for generation. As illustrated in Figure (c), obtain paired sequences (I, y) with fast NAR encoder by trans- lating to x from y, and then reversely feed æ the encoder-decoder Transformer model to use y as the target to update the decoder parameters {6a, Wa). During the EBTD phase, the encoder parameters are frozen to preserve encoder translation quality. The EBTD algorithm details are outlined in Algo- rithm (see Appendix A).
4.1 Encoder Back- Tack-Translatio
Following generative language modeling or Trans- former back-translation, the encoder is pri- ori good translating on its own. We propose the Encoder Back- Transsaaaslt (EBT) phase it- eratively improve the encoder generation ability, As shown in Figure (b), starting from samples y ~ Dt, we first use encoder to generate ~ D.,, and then use the pairs (Ir, y) to train the encoder, updating parameters (6., Wo), with cross-entropy loss for the translation direction
y. The same is done for the reverse translation direction well using the same encoder, and alterna- tive between these two directions similarly to Lam- ple et al. (2018a). In this way we make the en- coder stripped down version NAR encoder-
4.3 Initialization and Training Strategy
In practice, the QBT framework leverages BT, EBT and EBTD in synchrony to compose training, We
We assume tied decoder output and input embeddings in our setup throughout the paper, i.c. Was = Wa-

---

consider two configurations below, with different model initialization and training procedures apply- ing QBT for different purposes.
Algorithm 1 Quick Back- Tock-Translatio Synced (QBT-Synced): Given two monolingual corpora aa. Dr. and an encoder-decoder Transformer model = {fe, We, Ad, Wa} converged under UMT pre-training and back-translation, fine-tunc model A.
QBT-Synced In this scenario QBT is used to quickly boost translation performances of UMT models that have fully converged under standard BT training. The Transformer is initialized from a trained MT model. Then BT, EBT, and EBTD are run iteratively on the batch level, with EBT better aligning the Transformer encoder with translation objectives, EBTD adding diverse training signals from encoder generations for the Transformer de- coder, and both leveraging the constant NAR gen- eration time complexity. The algorithm description for QBT-Synced is shown in Algorithm
1: repeat
2: Sample a mini-batch B of either 2 ~ D, or
2~ Dr
Update encoder and its embeddings {0c, W.} with one EBT step starting from z (with Algorithm 2);
4: Sample a new mini-batch B of either 2 D3 or 2 ~ Dr
5: Update decoder and its embeddings {0a, Wa} with one EBTD step starting from (with Algorithm 3);
QBT-Staged Here we use QBT supplement BT in training a UMT model from scratch. We initialize the parameters from a general pre-trained model that does not have any MT capability, and then run the following four training stages for UMT: training encoder with randomly paired source and target sentences, EBT training for the Transformer encoder, EBTD training for the Transformer de- coder, and finally BT training for the full model parameters. particular, the first stage is to pre- pare the encoder with basic language translation abilities for its direct QBT generation in later stages. We randomly sample x ~ 'D2 and y D2 from two unrelated monolingual corpora to construct lan- guage pairs for warming up the encoder described in Equation (2).4 BT is still needed encoder- generated translations are typically of lower quality than decoder-gentt -eeeerted translations. The detailed algorithm for QBT-Staged is described Algo- rithm 4 (see Appendix A due to space limit here).
Sample new mini-batch B of either D3 ~ Dr
Update full model @ with BT step starting from 2;
8: until Convergence
glish and Romanian, have distinct bilingual BPE dictionaries. Following BPE, sentences with length 175 greater are removed. Models are scored on the WMT' 14 English- Frenchh, WMT' 16 English- German and WMT 16 English-Romanian parallel test sets (Koehn et al., 2007). We evaluate the mod- els with BLEU scores (Papineni et al., 2002; Post, 2018) following the same setup of previous works (Song et al., 2019; Nguyen et al., 2021).
Model Configuration We fine-tune the open- source, MASS and CBD7 UMT model check- points provided publicly by Song et al. (2019); Nguyen et al. (2021). These checkpoints have already been fined-tuned with tracsttranl in large resource settings. We provide separate experiments with the pre-trained XLM model checkpoints provided by Conneau and Lample (2019). These models have undergone generative pre- pre-training but have not yet been trained with back- translation or machine translation objectives. All models are encoder-dccoder Transformers with 6 layers and 1024 dimensions. Models are bilingual, respective to their cited language pair, and use the
5 Experimental Setup
Data and Evaluation We evaluate our proposed methods on WMT News Crawl datasets. 5 We prepare all of the monolingual data for English, French, German, and Romanian, up to year 2017. The datasets have sentence counts of 190M, 62M and 270M, 2.9M respectively. We encode data with Byte-Pair Encoding (BPE) (Sennrich et al., 2016b) with a dictionary of 60K sub-words provided by Conneau and Lample (2019). Each language pair: English and German, English and French, and En-
"https://github. http:///....oomicccooo/ooooo////
'Whenever æ and y are of different lengths, we truncate the longer one have same length. 'https: / /dataa statmt. or wtghhwws-craw/
crosstranslate httpg/hwww..mmmmmgg//////llll.a.
"https: http:ttt.cccm/ffeeborrrrrrcha//a

---

Method
en
fr
fr
en
en -
de
de
en
en ro
ro en
NMT (Lample el al., 2018c)
25.1
24.2
17.2
21.0
21.1
19.4
PBSMT (Lample et al., 2018c)
27.8
27.2
17.7
22.6
21.3
23.0
Multi-Agent (Wang et al., 2019)
19.3
23.8
XLM (Conneau Lample, 2019)
33.4
33.3
26.4
34.3
33.3
31.8
MASS (Song el al., 2019)
37.5
34.8
28.3
35.2
35.0
33.0
MASS QBT-Synced
37.8
35.2
28.7
35.5
35.2
33.2
CBD* (Nguyen et al., 2021) CBD QBT-Synced
38.2
35.5
29.6
35.5
38.2
35.5
29.7
35.9
Table BLEU scores on the WMT' English- French, WMT" 16 English German and 1M English-Rooe Remlish-ooannian unsupervised translation tasks. marks results obtained with published checkpoints. For QBT-Synced, our model is initialized with the open-source, fine-tuned UMT model MASS (Song et al., 2019) or CBD (Nguyen et al., 2021). Translations are measured with the Moses multi-bleu, perl script (Koehn al., 2007) as be comparable with previous work. We also show additional results with SacreBLEU (Post, 2018) in Appendix C for future references.
same corresponding BPE vocabularies.
en
fr
de
Initialization
37.5
34.8
28.3
35.2
Implementation Details For all configurations the Adam optimizer (Kingma and Ba, 2015) with 31 of 0.9 and 32 of 0.98 is used for training, with the
BT
37.2
33.2
26.6
34.0
EBT
30.5
28.8
19.5
24.4
32.05
29.5
26.0
learning rate of le-4. During inference, default greedy decoding for the XLM and MASS model (Conneau and Lample, 2019; Song al., 2019). For CBD (Nguyen et al., 2021) we use a beam size of 5, and length penalty of 0.6. For all encoder-generated sequences we use the argmax of the output-layer probabilities for NAR decoding. More details can be found in the Appendix B.
EBT EBTD
EBTTD
28.9
24.8
20.8
26.1
BT + EBTD
37.0
24.1
26.5
33.9
BT EBT
37.7
35.0
28.4
35.4
QBT 37.8
35.2
28.7
35.5
Table 2: Ablation study of the proposed methods for MASS fine-tuned model. BLEU scores are pre- sented the WMT" 14 English-French and WMT 16 English-German reference test-sets. QBT-Synced could otherwise labeled BT EBT + EBTD.
6 Results and Analysis
6.1 Large Scale WMT Experiments with QBT-Synced
compared to the MASS experiments. Whereas the CBD model requires two separate, high perfor- mance UMT models, method is able to improve almost all models with additional parameters and relatively little computation.
We use QBT-Synced fine-tune the state-of-th art UMT models available (Song al., 2019; Nguyen et al., 2021), and test their performance with only 10,000 training steps. We run exper- iments on an 8-GPU system. The QBT-Synced algorithm is run, using iterative batch updates of EBT, EBTD, and BT steps.
Components of QBT-Synced In order to study the impact of each individual optimization step the QBT-Synced procedure, we evaluate the effec- tiveness of EBT, EBTD, and BT individually and in combination in Table 2. All permutations of BT, EBT, and EBTD measured after 10,000 training steps. For model initialization, the UMT fine-tuned English-Frende and English- German MASS checkpoints (Song et al., 2019).
Main Results Table shows the performance of our proposed method in comparison with recent UMT methods. For MASS fine-tuning (middle sec- tion), the proposed method slightly improves the initial UMT model by an average of 0.3 BLEU points. For CBD fine-tuning (bottom section), the proposed method performs similarly (with +0.1 BLEU points on average). The CBD model is trained via distillation of the XLM and MASS fine-tuned models, and the QBT model may be experiencing diminishing returns on data diversity
We find that the back-translation (BT only) fine- tuned model scores lower on the test than the ini- tialization. This decline likely due to the change in training regime from the original, and the lower number sequences per batch in our setup (Ik

---

QBT
Encodar Aug (A40
...
QBT Long Soq.
Aug
12
32
...-
BT Long Seq.
1.6
Dacoder Aug [En. Do)
Saquences (Thausandss
BT Long Seq.
BT
Wall Clock (Seconds)
10
ET Boen
18
0.8
12-
8-
0.4
0.0
5
20
40
60
80
2
3
Wall Clock (Hours)
Sequence Length
Wall Clock (Hours)
Figure 4: Avg En-De and De-En scores. Here BT used during QBT.
Figure 2: Average forward step run- time.
Figure 3: Data throughpur on and
De train sets.
versus 2k) (Song et al., 2021). All models under- perform the initialization except BT + EBT and QBT-Syneed (BT + EBT EBTD). Without the BT objective all models decline. The model may be losing its ability to back-translate its own high quality outputs, diminishing the final quality. EBT BT improves the model within 0.1 BLEU of the full QBT-Synced procedure. We speculate that the EBT BT step serves better align the encoder and decoder to the final test objective: machine translation.
en fr
fr en
en - de
de - en
BT Baseline
32.9
32.0
25.2
31.2
QBT-Staged 34.9
32.0
25.9
31.9
Table 3: BLEU scores the limited resource WMT 14 English-Frena and WMT" 16 Boglish-German UMT tasks. XLM pre-trained model is used as our initial- ization.
English to French by 1.5 BLEU points, compared to their score of 33.4 for UMT achieved from XLM initialization.
6.2 Limited Resource WMT Experiments with QBT-Staged
QBT Efficiency To demonstrate the speed gains of using encoder for NAR generation during QBT procedures, Figure 2 presents average wall clock time of translation generation on batch-size 32 sam- ples from the WMT test set. We report the average wall clock time per batch, which is sampled be of approximately uniform sequence length. For the encoder, we average wall-clock time for all translation directions, as the difference is not vi- sually distinguishable. For the decoder, language directions are presented independently. We can see the drastically reduced run-time of the generation when using the repurposed NAR encoder compared with the AR decoder, which greatly saves time and energy for UMT training with QBT.
We use QBT-Staged to fine-tune a language mod- eling pre-trained model that has not yet undergone back-translation for MT specialization. We limit the computational resource to a total training time of up 32 hours, with single RTX 3090 GPU, The XLM pre-trained Transformer is used our model initialization, which has not yet undergone any MT or BT tuning (Conneau and Lample, 2019). The baseline fine-tuned with the BT and DAE objectives. The QBT-Staged encoder is warmed up with randomly paired monolingual samples from different language corpora for 5,000 training steps, described in section 4.3. The remaining training budget split as follows: 4 hours of EBT, 16 hours EBTD, and 12 hours of BT.
6.3 Long Sequence Translation
Main Results Results are shown in Table 3. QBT-Staged model outperforms the baseline on all language directions, excluding French English in which they tie. Our limited resource QBT-Staged model even outperforms the original 8-GPU imple- mentation from Conneau and Lample (2019) on
The WMT dataset is skewed towards short quences, with the majority of sequence shorter than 50 token length. Therefore the large resource main results in Table and limited resource results in Table presented do not portray the dramatic run- time benefits of encoder tack-srraslltioon versus

---

decoder back-translation. In this section, we fil- ter the German and English WMT train datasets to contain only sequences longer than length 120. For evaluation, we use the full WMT 16 German- English test set with no minimum sequence length, there are not enough long sequences. We run 4 independent experiments: baseline BT unfiltered data, QBT-Staged unfiltered data, BT on long-sequence data, and QBT-Staged on long-sequence data. All models are allocated hours of training time. QBT-Staged involves 0.5 hours of EBT, and uses EBTD for the remaining time.
Model
Beam Size
Python-Java
Java-Python
BT QBT-Staged
25.61
22.57
33.75
32.27
BT QBT-Staged
(top 1)
38.26
37.86
(top 1)
41.49
40.89
Table 4: BLEU scores the filtered GeeksForGeeks parallel test set.
respectively. A BPE vocabulary of size 30k is learned over the union of tokenized Python and Java datasets, Data filtered to include only static functions and code comments are removed. We fil- ter out sequences longer than length 100 as outliers. In total we keep 10 million code snippets of Python and Java each. For evaluation, forGeeks parallel test set provided by Roziere et al. (2020). 13 A Transformer model with 6 encoder layers and 4 decoder layers used. The baseline is fine-tuned with BT for 24 hours. For QBT, we use QBT-Staged and train with EBT, EBTD, then BT, for 8 hours each. Final models are evaluated with greedy decoding and with beam 10 (select top 1) search (Roziere et al., 2020).
Data Throughput In Figure 3, QBT algorithm demonstrates almost negligible run-time slowdown at longer sequence length during training, However, the BT algorithm sees only about one third of the data during training. QBT and BT data throughput are similar at base sequence length.
use the Geeks-
Training Curves In Figure 4, the BT algorithm considerably under-performs the others long sequence length. Surprisingly, the QBT model trained on only long sequences outperforms the baseline QBT model. We can also see the QBT models converge faster due to larger data through- put. The encoder generates sequences in a token- to-token fashion, and it is possible that this trans- lation configuration generalizes well to sequences of unseen length. Additionally, the long-sequence encoder has more tokens during training, and effectively more training data. Though only score decoder outputs, the QBT decoder sits on top of the FBT-trrinne encoder representation, and may be benefiting from these advantages.
Results QBT outperforms the BT baseline for both programming language directions and decod- ing setups. The performance gap between the mod- els decreases at higher beam-size. This shift may be due to the encoder-synthesis more strongly re- sembling greedy decoder generations.
6.5 Discussion
Translation Examples The qualitative differ- ences in model translations following BT or QBT is of great interest. In Table 5 we provide samples from the fine-tuned German to English limited re- source translation task in section 6.2. Here the EBT (encoder generation) and QBT (decoder generation) models provide remarkably similar translation. The BT model suffers some syntactic degeneration, likely due to the fact that the BT model may have not fully converged given the limited time budget.
6.4 Unsupervised Programming Language Translation
order to more thoroughly test our proposed method, we provide a demo of QBT on unsuper- vised programming language translation.
For dataset curation we follow the procedure from Roziere et al. (2020). Monolingual Python and Java source code is downloaded using Google's BugQuery."* Source code is pre- rrepprocessed using the Python6 t and javalang5 tokenizer
Self-BI BLU- Analysis The translation comparison raises the question: does QBT work align the lan- guage representations between the encoder and de- coder? The Self-BLEU metric, originally proposed by Zhu et al. (2018), uses a modell-general corpus
"Note that here BT is not involved to separately demon- strate the efficiency of QBT on top of BT.
tps:////d wwwgle.com/bigoggeeeoeooooo/o/oo com/bigquery
"I thttps://iithuu wttp:l/ccccc......eee//e httpt//.ccmmmcnee//aaaanngg
Due to long sequences filtered out, our results compare BT and QBT in a novel scenario, but should not he compared to the TransCoder model from Roziere et al. (2020).
12
html

---

Model
Generation
Target
It is a mixture of a huge fashion show and waxworks chamber of horrors: designer
Jean Paul Gaultier (63) presented an exhibition of is work in Munich on Wednesday.
EBT
It's mix of very big fashion show and Wacchsfiuren-gruselkabiertt -ruselkabbnettt French designer
Jean Paul Gaultier (63) opened on Wednesday in Munich exhibition celebrating his work collection.
QBT-Staged
It is a combination of big Modenschau and a Wachsfiguren grusele-skater French designer Jean Paul Gaultier 63 opened on Wednesday in Munich show about his work career.
BT
It's a combination of complete Modenschau and a Wachsfiguren-llooing Gruselka- binett Gruselkabinett: The designer Jean Paul Gaultier (63), who has collection of his own pieces, including collection of his own pieces, including collection of his own pieces, including collection of his own pieces, a collection of his own pieces, including a collection
Table 5: Comparison of translations following QBT-staged training in 6.2. From the WMT' 14 En-De test set.
as the BLEU reference text, acting as a similarity metric between any two corpora. In the following. we score the outputs of the converged EBT encoder against various references. For the XL.M model, we use the converged QBT-Staged checkpoints from Table and for MASS we use the QBT-Sync ab- lation checkpoints in Table 2. Scores are shown in Figure 5. Decoder BT denotes the encoder transla- tions scored against a separate baseline BT decoder translations. Decoder EBT uses decoder transla- tions following EBT training of the model encoder. Finally, Decoder EBTD depicts the decoder transla- tions following EBT and EBTD." 14 Across all four scenarios, EBTD increases the Self-BLEU of the decoder to the encoder translations. This relation- ship suggests that there is potentially knowledge distillation occurring during EBTD. Note as well that the XLM models presented, trained with QBT- Staged, underwent more steps of EBT and EBTD than the MASS models. This is reflected in the higher overall self-BLEEU scores of XLM versus MASS.
Teat Reference
Decoder
Decoder
Decoder
30
20
10
En-Fr (XLM)
Fr-En (XLM)
(MASS)
Figure 5: BLEU and self-BLEU scores for encoder- generated sequences against various references. The standard BLEU scores for encoder translations are given in Test Reference, Decoder BT is the elf--BBUU of the encoder vs the baseline BT decoder, and so on. All source sequences are from the WMT'14 English- French test set.
coder, and leverages encoder- -eneerrreeeeeteedd sequences for improved performance and training speed for UMT. QBT combines Back-Translatinn (BT), En- coder Back-Translation (EBT), and Encoder Back- Translated Distillation (EBTD). This optimization framework creates efficiency gains, and provides new synthetic data sources. QBT achieved sim- ilar fine-tuned performance to other statc-of-the- art UMT models with additional parameters and higher data efficiency, On long sequences, QBT considerably outperforms the original back- translation algorithm in data throughput and trans- lation quality.
7 Conclusion
We proposed Quick Back- Translattnt (QBT) for unsupervised machine translation, which endows translation capabilities to the Transformer en-
1'For MASS Decoder EBT is the EBT BT model from the ablation in Table 2. This model is used as avoids the decoder collapse in the QBT--yync setup. Decoder EBTD uses the final QBT-Staged model from Table 3, and the final QBT-Sync model as in our main results Table 1, for XLMM and MASS respectively.

---

8 Limitations
Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018 Understanding transstannn at scale. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pages 489-500, Brussels, Belgium. Association for Computational Linguistics.
Our QBT methods serve an augmentation of the standard BT algorithm for learning unsupervised machine translation models, mainly with efficiency considerations low computational resource sce- narios. They still can not fully replace the standard BT procedures, as QBT still needs BT inside the algorithms to achieve competitive translation quali- ties. Even so, our performance gain compared to previous state-of-the-art UMT models are modest, whereas we mainly gain in training speed. Due to the limited resource setups, the models that we perimented are considered relatively small in sizes compared with the large language models nowa- days. It would be interesting to explore further on the larger scales. Our methods currently apply to encoder-decoder Transformer architectures, which may limit the applications. For different model architectures, we may hope to apply similar ideas of using fast (and potentially separate) genera- tion model to generate synthetic data for BT-like algorithms. Furthermore, the use of encoder for back-translation (EBT) with NAR generation is also limited in that the output sequence length can not go beyond the input sequence length, which we hope to relax with more sophisticated NAR strategies in the future.
Jiatao Gu. James Bradbury, Caiming Xiong, Victor OK Li, and Richard Socher. 2018. Non-autoregressive neural machine translation. In International Confer- ence on Learning Representations.
Jiatao Gu and Xiang Kong. 2021. Fully non- autoregressive neural machine translation: Tricks of the trade. Findings of the Association for Com- putational Linguisties: ACL-IJCNLP 2021, pages 120-133, Online. Association for Computational Lin- guistics.
Junliang Guo, Xu Tan, Linli Xu, Tao Qin, Enhong Chen, and Tie- Yan Liu. 2020. Fine-tuning by curriculum learning for non-autoregressive neural machine trans- lation. Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7839-7846.
Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang. Marcin Junczys Lewis, Mu Li, et al. 2018. Achieving human par- ity automatic chinese english news translation. arXiv preprint 10X..v8800.05555
William
Di He, Yingce Xia, Tao Qin. Liwei Wang. Nenghai Yu, Tie-Ya Liu, and Wei-Ying Ma. 2016. Dual learning for machine translation. Advances in neural information processing systems, 29.
Acknowledgments
Vu Cong Duy Hoang, Philipp Koehn, Gholamreza Haffari, and Trevor Cohn. 2018. Iterative back- translation for neural machine translation. In Pro- ceedings of the 2nd Workshop on Neural Machine Translation and Generation, pages 18-24, Mel- bourne, Australia. Association for Computational Linguisties.
We express our gratitude to Dr. Nakul Verma for his support and insight over the course of this work.
References
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018.
Jungo Kasai, Nikolaos Pappas, Hao Peng. James Cross, and Noah Smith. 2020. Deep encoder, shallow decoder: Reevaluating non-autoregressive machine translation. In International Conference on Learning Representations.
Unsupervised statistical machine translation. Pro- ceedings of the 2018 Conference on Empirical Meth- ods in Natural Language Processing, pages 3632- 3642, Brussels, Belgium. Association for Computa- tional Linguistics.
Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference Learning Representations (ICLR), San Diega, CA, USA.
Alexis Conneau and Guillaume Lample. 2019. Cross- lingual language model pretraining. Advances in neural information processing systems, 32.
Alexandre Klementiev, Ann Irvine, Chris Callison- Burch, and David Yarowsky. 2012. Toward statis- tical machine translation without parallel corpora. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Lin- guistics, pages 130-140.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing, In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171 1171-4 Minneapolis, Minnesota. Association for Computational Linguistics.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch Marcello Federico, Nicola Bertoldi Brooke Cowan, Wade Shen, Christine Moran,

---

Richard Zens, et al. 2007. Moses: Open source toolkit for statistical machine translation. In Pro- ceedings of the 45th annual meeting of the associa- tion for computational linguistics companion volume proceedings of the demo and poster sessions, pages 177-180.
Machine Translation: Research Papers. pages 186- 191, Brussels, Belgium. Association for Computa- tional Linguisties.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language under- standing by generative pre-training.
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc' Aurelio Ranzato. 2018a. Unsupervised machine translation using monolingual corpora only. In International Conference on Learning Representa- tions.
Qiu Ran, Yankai Lin, Peng Li, and Jie Zhou. 2021. Guiding non con--utoreeressivee neural machine transla- tion decoding with reordering information. In Pro- ceedings of the AAAI Conference on Artificial Intelli- gence, volume 35, pages 13727-13735.
Guillaume Lample, Alexis Conneau, Marc' Aurelio Ran- zato, Ludovic Denoyer, and Hervé Jégou, 2018b. Word translation without parallel data. In Interna- tional Conference on Learning Representations.
Sujith Ravi and Kevin Knight. 2011. Deciphering for- eign language. In Proceedings of the 49th Annual Meeting of the Association for Computational Lin- guistics: Human Language Technologies, pages 12- 21, Portland, Oregon, USA. Association for Compu- tational Linguistics.
Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Mare' Aurelio Ranzato. 2018c. Phrase- based & neural unsupervised machine translation. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 5039-5049, Brussels, Belgium. Association for Com- putational Linguistics.
Baptiste Roziere, Maric-Anne Lachaux, Lowik Chanus- sot, and Guillaume Lample. 2020. Unsupervised translation of programming languages. Advances in Neural Information Processing Systems, 33:20601- 20611.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre- training for neural machine translation. Transac- tions of the Association for Computational Linguis- tics, 8:726-742.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 201 Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pages 86-96, Berlin, Germany. Association for Computational Lin- guistics.
Inbal Magar and Roy Schwartz. 2022. Data contamina- tion: From memorization to exploitation. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume 2: Short Pa- pers), pages 157-165, Dublin, Ireland. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural machine translation of rare words with subword units. In Proceedings of the 54th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 17 1- 1725, Berlin, Germany. Association for Computa- tional Linguistics.
Nathan Ng. Kyra Yee, Alexei Baevski, Myle Out, Michael Auli, and Sergey Edunov. 2019. Facebook FAIR's s WMT19 news translation task submission. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1). pages 314-319, Florence, Italy, Association for Computational Linguistics.
Kaitao Song. Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Yan Liu. 2019. Mass: Masked sequence to sequence pre- training for language generation. In International Conference on Machine Learning, pages PMIR.
Xuan-Ph Phi Nguyen, Shafiq Joty, Thanh-Tung Nguyen, Kui Wu, and Ai Ti Aw. 2021. Cross-model back- translated distillation for unsupervised machine trans- lation. In Proceedings of the 38th International Con- ference on Machine Learning, volume 139 of Pro- ceedings of Machine Learning Research, pages 8073- 8083. PMLR.
Zhenqiao Song, Hao Zhou, Lihua Qian, Jingjing Xu, Shanbo Cheng. Mingxuan Wang. and Lei Li. 2021. switch-glat: Multilingual parallel machine translation via code-switch decoder. In International Conference on Learning Representations.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances neural information processing systems, 30.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computa- tional Linguistics, pages 31 1-318.
Yiren Wang, Yingce Xia, Tianyu He, Fei Tian, Tao Qin, ChengXiang Zhai, and Tie-Yan Yan Liu. 2019. Multi- agent dual learning. In International Conference on Learning Representations.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on

---

Bingzhen Wei, Mingxuan Wang, Hao Zhou, Junyang Lin, and Xu Sun. 2019. Imitation learning for non- autoregressive neural machine translation. In Pro- ceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 1304 1312, Florence, Italy. Association for Computational Linguistics.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 483-498, On- line. Association for Computational Linguistics.
Biao Zhang, Barry Haddow, and Alexandra Birch. 2023. Prompting large language model for machine transla- tion: A case study. arXiv preprint arXiv:2301.0700
Jiawei Zhou and Phillip Keung. 2020. Improving non- autoregressive neural machine translation with mono- lingual data. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 1893-1898, Online. Association for Computa- tional Linguistics.
Yaoming Zhu, Sidi Lu, Lei Zheng. Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018. Texygen: benchmarking platform for text generation models. In The st International ACM SIGIR Conference on Research Development in Information Retrieval, pages 1097-1100.

---

A QBT Algorithm Descriptions
Algorithm Quick Back- Transsstannnn Staged (QBT- Staged): Given two monolingual corpora s,, Dr. and an encoder-decoder Transformer model 0 = {fe. We, fa. Wa} undergone general pre-training without MT capability, fine-tune model 0 for UMT.
We detail the algorithmic procedures for QBT com- ponents: batch EBT for encoder translation tuning in Algorithm 2, batch EBTD for decoder generation tuning in Algorithm 3. We also describe the com- plete procedure for QBT-Staged in Algorithm 4.
Stage 1: encoder warmup with random trans-
Algorithm Encoder Back- Translation (EBT) Step: Given two monolingual corpora D., Dj and a Transformer model = {0e. We, 0a, Wa). train a UMT model over the encoder (fe, We).
lation
repeat
Sample mini-batch B composed of ran- dom pairs (x, y), with x Ds. ~ Dr.
independently;
1: Given a mini-batch B sampled with either z D3 or Dr
4: Truncate a or to make the pairs have equal
length;
2: for B do
Predict translation y' conditioned on only using the encoder {0.. We} with computa- tion described in Equation (2);
3: Infer translations 2' with encoder {0e, We);
4: Predict translations z" conditioned on s' with the same encoder:
Update the encoder parameters {0e, We) with gradient computed from reconstruction loss between y' and target y;
5: Update encoder parameters {fc. W.] with gradient computed from reconstruction loss between target z and prediction distribution for z";
7: until Convergence of a fixed amount of steps/time
6: end for
8: Stage 2: EBT to train the encoder for transla- tion
repeat
10: Sample mini-batch B of either DD D:
Algorithm 3 Encoder Back-Translated Distilla- tion (EBTD) step: Given two monolingual cor-
11: Update encoder and its embeddings {0e, We} with EBT starting from z (with Algorithm 2);
pora D., Dj and a Transformer model 0 (Ge, We, 0a, Wa), train a UMT model over the 6.
1: Given a mini-batch B sampled with either
12: until Convergence of fixed amount of steps/time
D, or Dr
2: Freeze all encoder parameters (6e, W.)
13: Stage 3: EBTD to train the decoder for gener- ation
3: for B do
4: Infer translations with encoder (0c, We):
14: repeat
5: Predict translations =" conditioned on z' with the full encoder-decoder Transformer
15: Sample a mini-batch of either ~ D, or
Dr
0:
16: Update decoder and its embeddings {0a, Wa} with EBTD starting from z (with Algorithm 3);
6: Update only the decoder parameters (0a, Wa} with gradients computed from re- construction loss between target and pre- diction distribution for z";
17: until Convergence of fixed amount of
steps/time
7: end for
18: Stage 4: BT to tune the full UMT model
19: repeat
20: Sample a mini-batch of either z ~ Dz or
Dj
21: Update full model 0 with BT starting from z:
22: until Convergence of fixed amount of steps/time

---

MASS
MASS QBT-Synced
son. In order further validate our methods, we conduct experiments separately using the Scare- BLEU metric implementation (Post, 2018). This metric allows for a more controllable metric com- parisons for future works.
(Song al., 2019)
en fr
37.5
38.1
fr
34.8
35.3
en de
28.4
29.0
In Table our models are trained with QBT- Sync, following the exact same procedure as our Moses BLEU results. Notably here we differ in that the best checkpoint is selected via the highest validation on the SacreBLEU metric. Scores SacreBLEUU are reported. Similar before, we find that QBT-Sync has a distinct but slight performance bump over the MASS baseline.
de
35,4
36.0
en ro
35.1
35.5
ro
33,1
33,2
Table 6: BLEU scores on 14 En-Fr, WMT 16 En-De, WMT' En-Ro. MASS are measured from published UMT fine-tuned checkpoints from the MASS work (Song et al., 2019). Translations are scored with the SacreBLEU scripts (Post, 2018).
B Implementation Details
All methods implemented following the origi- nal codebases of relevant multilingual pre-trained models. our main UMT results in Section 6. a batch size of 16 and maximum tokens per batch are used. Our limited resource scenario in Section 6.2 uses a batch size of 32 and 2k maxi- mum tokens per batch. For all configurations the Adam optimizer (Kingma and Ba, 2015) with $1 of 0.9 and $2 of 0.98 is used for training, with learning rate of lc-4. Best model checkpoint is selected by validation BLEU scores from greedy decoding. During inference, we use the default greedy decoding for the XLM and MASS model (Conneau and Lample, 2019; Song et al., 2019). For CBD (Nguyen et al., 2021) we use a beam size of 5, temperate of and length penalty of 0.6. For all encoder-generaiee sequences use the argmax of the output-layer probabilities for NAR decod- ing. During Lincoder Back-Translation (EBT), occasionally observe mode-collapse, in which the model copies the input sequence as the output se- quence. this case, we penalize sequence copying by adding as regularization term the inverse of the negative log-likelihood against the source sentence. The penalization is applied at each EBT step.
C Complementary Results: SacreBLEU
Our main results in Table 1 use the same Moses Perl BLEU evaluation (Koehn al., 2007) and byte pair encoding vocabularies previous UMT work (Conneau and Lample, 2019; Song et al., 2019; Nguyen et al., 2021). This choice of Moses BLEU script to follow previous work for fair compari-


Reviews:
[{"review_id": "3kyvBOuXMF", "reviewer": "Reviewer_7rp5", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comprehensive review of our paper. We appreciate your recognition of the novelty in using the encoder for back translation and the various integration approaches we proposed for QBT in the training process. Your observation about the efficiency advantage aligns with our intention to enhance training procedures, making them more feasible even in resource-constrained settings. \n\nWe acknowledge your point regarding the comparable quality improvement to the state of the art. We believe that QBT's ability to efficiently approach these moderate gains, along with its advantages in scenarios with limited resources and longer sequences, emphasize its practicality and utility for UMT. \n\nWe have provided some new details in our other two responses.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Strengths above: interesting idea.", "reasons_to_reject": "The quality improvement seems to be mostly comparable to state of the art.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper introduces a new method to unsupervised machine translation: quick back translation (QBT). QBT is the result of NAR translation from the encoder. It seems to have worse quality (per experiment results), but, when used in the unsupervised MT training procedure (eg QBT synced or QBT staged), can improve the training efficiency (because of the quick translation?) and provides comparable results to the state of the art, or slightly better in some cases/tests. Paper conducted experiments to examine the QBT impact in general, or in special cases (eg, long sequence), showing the positive effect of QBT.\n\nOverall, I like the idea of using the encoder to produce back translation and like to confirm its results compared to BT. The efficiency advantage of QBT probably is more interesting than the quality improvement (which is slight based on the result tables.)\n\nStrength:\n- Using encoder to produce BT is interesting and on the novel side. \n- Paper proposing several ways of integrating QBT into the training procedure also seems interesting to me.\n- Good amount of experiments to examine the effectiveness of QBT.\n\nWeaknesses:\n- Strength of result: it seems that translation quality of the new procedure is mostly the same as state of the art.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "HnXJNNxll1", "reviewer": "Reviewer_w5DB", "summary": "", "questions": "- In the experimental evaluation, could you discuss any specific instances or examples where QBT demonstrated a notable improvement over the standard approach in terms of translation quality?\n\n- Could you provide insights into the mechanism through which QBT enhances data efficiency? What is lost in the absence of the autoregressive decoder? did you encounter any trade-offs or potential drawbacks in other aspects of the model's performance, such as encoding quality or training stability?\n\n- Are there any specific characteristics of the Transformer architecture that make it particularly amenable to being repurposed for generative purposes, as demonstrated in QBT?\n\n\n\n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the careful review and constructive comments! We provide some more details below.\n\n1. \"further analysis on the specific mechanisms through which the encoder-generated sequences contribute to decoder improvement could provide a deeper understanding of QBT.\"\n\nMore rigorously evaluating the encoder synthetic data diversity would be of great value, especially in relation to the decoder-generated translations. We made some initial progress towards this analysis with our Self-BLEU results in Appendix section C. Here we train a model with no UMT pre-training with EBT. We generate translations with this encoder, then use these translations as the BLEU reference for a number of decoder-generation models. Interestingly, models trained with AR BT have Self-BLEU similar to their BLEU scores against the test set. This score implies that the encoder-generated sequences are different, but not that different from the decoder. Another key detail of this analysis is that encoder-distillations, namely EBTD, increase the Self-BLEU between encoder generated outputs and decoder generated outputs.\n\n2. \"In the experimental evaluation, could you discuss any specific instances or examples where QBT demonstrated a notable improvement over the standard approach in terms of translation quality?\"\n\nQBT outperforms BT in translation quality in a variety of scenarios. However, the amount by which it outperforms varies, due to the compute resources involved and the characteristics of the dataset (sequence lengths, similarity between source and target languages, and so on). One scenario where QBT shines in translation quality is long sequence translation (Figure 4). In our experiment QBT nearly triples the BLEU score of the original BT method (although both are not at the level of BLEU for short sequences). This outperformance is due in part to the data throughput efficiency of the encoder, but also likely due to properties of bidirectional translation itself. \n\nBelow we provide samples from the final fine-tuned German to English translation task in Section 6.2 Limited Resource WMT Experiments. Here the EBT (encoder generation) and QBT (decoder generation) models provide a similar translation. The sequences seem more semantically aligned with the target than the BT output. In addition, the BT model suffers some syntactic degeneration, which is likely due to the fact that the BT model may have not fully converged given the limited time budget.\n\n**Target**: *It is a mixture of a huge fashion show and a waxworks chamber of horrors : designer Jean Paul Gaultier ( 63 ) presented an exhibition of is work in Munich on Wednesday .*\n\n**EBT**: *It 's a mix of very big fashion show and a Wachsfiguren-gruselkabinett : French designer Jean Paul Gaultier ( 63 ) opened on Wednesday in Munich a exhibition celebrating his work collection .*\n\n**QBT (No BT)**: *It is a combination of a big Modenschau and a Wachsfiguren grusele-skater : French designer Jean Paul Gaultier ( 63 ) opened on Wednesday in Munich a show about his work career .*\n\n**BT**: *It 's a combination of a complete Modenschau and a Wachsfiguren-looking Gruselkabinett Gruselkabinett : The designer Jean Paul Gaultier ( 63 ) , who has a collection of his own pieces , including a collection of his own pieces , including a collection of his own pieces , including a collection of his own pieces , a collection of his own pieces , including a collection*\n\n3. \"Could you provide insights into the mechanism through which QBT enhances data efficiency? What is lost in the absence of the autoregressive decoder? did you encounter any trade-offs or potential drawbacks in other aspects of the model's performance, such as encoding quality or training stability?\"\n\nThank you for these great clarifying questions. QBT enhances data efficiency by generating a large quantity of synthetic translations very quickly using the encoder as a generative model. The encoder-as-generative-model is fast to initialize, to train with BT, and scales with time-complexity linear in sequence lengths, versus the quadratic time complexity of the decoder. However, given the encoder’s fixed output length, we find the BLEU maxes out at around 20 for the models tested; the encoder is a weak translator compared to the full AR counterpart. It would be interesting to further explore enhancements to the encoder with NAR sampling techniques [1].\n\nIn addition, Encoder Back-Translation is susceptible to “translation collapse”, whereby it simply copies input sequences as its output. For example, given some French, it would output the same French. Sequence copying is a global minimum of the back-translation loss function: if it maps French to French, then when it back-translates to the original French, copying again will result in a loss of 0. To keep the model away from converging into the global minimum, it is important to coerce it into cross-lingual behavior. We accomplish this through a very brief warm-up period where we train the model to map from random French to random English sequences, and vice versa. This results in a momentary degradation of model quality, but aligns the outputs towards the modes of the target language. For example, common words like “the” in English and “le” will appear in corresponding frequencies in the random inputs and targets. As the encoder has a powerful pre-trained representation, it will bias its translation of “the” towards “le”. This technique is interestingly similar to dictionary learning in the statistical machine translation field (Ravi and 035 Knight, 2011a; Klementiev et al., 2012). \n\nAfter very brief random-MT initialization, EBT typically converges towards the desired local minima. However, in some scenarios the model can collapse to the copying strategy later during EBT training. In order to counteract this, we apply a small regularization term to the loss which penalizes copying (Appendix B). This is calculated as the negative log-likelihood of the model output distribution against the source sequence to discourage direct copying.\n\n[1] Sun, Z., Li, Z., Wang, H., He, D., Lin, Z. and Deng, Z., 2019. Fast structured decoding for sequence models. Advances in Neural Information Processing Systems, 32.\n\n4. \"Are there any specific characteristics of the Transformer architecture that make it particularly amenable to being repurposed for generative purposes, as demonstrated in QBT?\"\n\nYes, absolutely. The QBT method is reliant on the encoder-decoder Transformer architecture. The AR nature of the decoder is largely due to the casual/unidirectional mask of the self-attention, and most of the NAR decoder remove this limitation by adopting the bidirectional mask in the decoder. The encoder is naturally equipped with the bidirectional self-attention mask, which blurs its boundaries with an NAR decoder. Without the complexity of utilizing a standalone fully dedicated NAR decoder on top of the Transformer encoder-decoder architecture, repurposing the encoder as an NAR decoder turns out to be a natural choice. All of these are made possible by the flexibility of the Transformer structure with attention mechanisms and parallel processing of sequential tokens by tweaking attention masks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The clarity with which the paper presents its proposed methodology, supported by experiments and outcomes\n- Novel approach for the field of unsupervised machine translation", "reasons_to_reject": "- further analysis on the specific mechanisms through which the encoder-generated sequences contribute to decoder improvement could provide a deeper understanding of QBT.", "questions_to_authors": "- In the experimental evaluation, could you discuss any specific instances or examples where QBT demonstrated a notable improvement over the standard approach in terms of translation quality?\n\n- Could you provide insights into the mechanism through which QBT enhances data efficiency? What is lost in the absence of the autoregressive decoder? did you encounter any trade-offs or potential drawbacks in other aspects of the model's performance, such as encoding quality or training stability?\n\n- Are there any specific characteristics of the Transformer architecture that make it particularly amenable to being repurposed for generative purposes, as demonstrated in QBT?\n\n\n\n", "paper_topic_and_main_contributions": "The reviewed paper focuses on unsupervised machine translation by introducing Quick Back-Translation (QBT). The paper outlines the existing challenges in unsupervised machine translation, particularly the computational load of autoregressive inference during back-translation and the synthetic data inefficiency of the process. The authors repurpose the Transformer encoder, harnessing it as a generative model. By utilizing encoder-generated sequences to refine the decoder, QBT significantly augments data throughput and utilization.\nThe experimental evaluation conducted on various WMT benchmarks serves as a testament to the effectiveness of QBT. QBT outperforms the standard back-translation method in terms of training efficiency while maintaining comparable translation quality.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "OhZjy9f6rY", "reviewer": "Reviewer_zNyd", "summary": "", "questions": "Q1: For Table 2, why is EBTD better than EBT, but BT+EBTD is worse than BT+EBT? And BT+EBTD is generally lower than BT?\n\nQ2: For Table 3, why is the improvement (2BLEU) on En-Fr significantly larger than the improvement on the rest of the translation directions?\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the constructive review and comments! We provide some responses below based on your valuable insights.\n\n1. \"The introduction of NAR generation in QBT effectively improves the translation efficiency.\"\n\nWhile we understand the limited novelty of BT for a NAR model in general, we would like to emphasize the unique demonstration of NAR BT inside an existing AR model structure, without any extra parameters or changes to the AR capability or quality. In particular, QBT demonstrates that existing AR MT models contain a highly efficient NAR model within their architecture, which allows for efficient model training (especially at long sequence lengths) and synthetic data diversification. Previous BT and NAR methods require dramatic modifications to the Transformer encoder-decoder, and suffer in quality from the loss of AR capability [1][2].\n\n[1] Junliang Guo, Xu Tan, Linli Xu, Tao Qin, Enhong Chen, and Tie-Yan Liu. 2020. Fine-tuning by curriculum learning for non-autoregressive neural machine translation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7839–7846\\\n[2] Zhenqiao Song, Hao Zhou, Lihua Qian, Jingjing Xu, Shanbo Cheng, Mingxuan Wang, and Lei Li. 2021. switch-glat: Multilingual parallel machine translation via code-switch decoder. In International Conference on Learning Representations.\n\n2. \"Translation quality improvements are limited. The main experimental results in Table 1 show that QBT can only bring about an improvement of about 0.2 BLEU, which I think is almost no improvement. For such a subtle improvement, the authors should adopt more metrics beyond BLEU (such as COMET), as well as a significance test to prove the effectiveness of the proposed method.\"\n\nIt is a good point that more metrics, such as COMET, along with significance tests, would provide more comprehensive comparisons of the model performances. We will make these measurements to bolster the BLEU scores. Nevertheless, we are in agreement that the translation quality improvements are only modest in the large-scale WMT scenario, and we will make this clear in our paper revision. Please note that the most comparable method to ours in terms of scores and methodology, Cross-Model Back-Translated Distillation [1], requires three, large, pre-trained models. Two of these models must be independently, extensively fine-tuned with BT. In total, 3x the number of parameters are required. Whereas QBT improves BLEU without any additional model parameters or auxiliary models. The benefits of our method are more in the efficiency gain with similar translation qualities compared to previous best models. For example, in resource constrained and long sequence scenarios, the gained efficiency enables very strong BLEU scores versus the original BT algorithm.\n\n[1] Nguyen, X.P., Joty, S., Nguyen, T.T., Wu, K. and Aw, A.T., 2021, July. Cross-model back-translated distillation for unsupervised machine translation. In International Conference on Machine Learning (pp. 8073-8083). PMLR.\n\n3. \"The efficiency evaluation in Figure 2 is less standard. As far as I know, the speedup ratio in NAT is often calculated at batch-size=1, and it will give additional efficiency improvements under different batch-sizes.\"\n\nThank you for highlighting this. In conducting this experiment, we deliberated between using batch size 1 for Figure 2, versus the batch size 32 used in our full experiment (Figures 3 and 4). We opted to use the batch size from the training scenario. But we agree that using batch size 1 would more directly probe the inference runtime for these models.\n\nBelow we have calculated the results with batch size 1. Sequences are taken from the WMT test sets, as before. As there are only 3000 samples in the test set, we bucket the source sequences based on their lengths inside a window for detailed analysis. Sequences with length +/-5 of the value in the *Sequence Length* column are taken, a random sample of 5 of these are passed through the corresponding model, and the average of the run-times is taken. (En, Fr) is the average of the En-Fr and Fr-En direction, and so on for the pairs. For encoder generations, output sequence lengths are the same as source sequence lengths, thus the generation runtime for all language directions are similar so we take the average of different language pairs. Decoder means the full AR model, and we present the runtime for the two language pairs separately.\n\nOverall, we observe that the encoder is considerably faster for the encoder than the AR mode. At a batch size of 1, the encoder runtime is not as sensitive to the increasing sequence length in our sample. Meanwhile the decoder generation experiences a slowdown of up to half a second, and over 1 second / sample in the batch size 32 case.\n\nWe will also add this information to the final paper.\n\n| Batch Size = 1: Forward step runtime (seconds), averages |                      |                  |                  |\n| -------------------------------------------------------- | -------------------- | ---------------- | ---------------- |\n|       Sequence Length                        | Encoder (En, Fr, De) | Decoder (En, Fr) | Decoder (En, De) |\n| 8                                                        | 0.009                | 0.24             | 0.19             |\n| 19                                                       | 0.009                | 0.28             | 0.22             |\n| 30                                                       | 0.01                 | 0.4              | 0.36             |\n| 40                                                       | 0.012                | 0.45             | 0.41             |\n| 60                                                       | 0.01                 | 0.57             | 0.48             |\n| 85                                                       | 0.011                | 0.85             | 0.71             |\n|                                                          |                      |                  |                  |\n\n|Batch size 32: Forward step runtime (seconds), averages  |                      |                  |                  |\n| -------------------------------------------------------- | -------------------- | ---------------- | ---------------- |\n|                                                          | Encoder (En, Fr, De) | Decoder (En, Fr) | Decoder (En, De) |\n| 8                                                        | 0.015                | 0.3              | 0.234            |\n| 19                                                       | 0.029                | 0.47             | 0.3              |\n| 30                                                       | 0.03                 | 0.63             | 0.42             |\n| 40                                                       | 0.031                | 0.78             | 0.624            |\n| 60                                                       | 0.047                | 1.15             | 0.87             |\n| 85                                                       | 0.058                | 1.55             | 1.83             |\n\n4. \"Q1: For Table 2, why is EBTD better than EBT, but BT+EBTD is worse than BT+EBT? And BT+EBTD is generally lower than BT?\"\n\nThis is a great clarifying question. For EBT and EBTD in this ablation, the decoder experiences a severe degradation in translation quality, as we are changing the model weights with disregard to the already converged MT capability. This highlights how for QBT-Sync, it is important to use BT during EBT and EBTD, to at least maintain the decoder capabilities. In the EBT row, we are shifting the encoder representations away from what the frozen decoder is expecting, which seems to be detrimental to the decoder translations. Whereas in the EBTD row, the encoder stays frozen, and the decoder is made to back-translate very low-quality, noisy translations back to the original sequence. In this way, EBTD acts as a very lossy de-noising objective, whereas EBT completely corrupts the encoder-decoder model. \n\nOnce we incorporate BT, the decoder can adapt to the shifting encoder representation on each step. So EBT + BT  aligns the encoder with the translation objective, while maintaining the BT capabilities of the decoder. For EBTD + BT, note that the encoder has never been trained for MT. So EBTD + BT is similar to BT with an additional very-lossy noising objective, which we find does not improve over BT. We hope that this clarifies the roles of the optimizations encompassing QBT-Sync. We will add additional exposition to our paper on this point.\n\n5. \"Q2: For Table 3, why is the improvement (2BLEU) on En-Fr significantly larger than the improvement on the rest of the translation directions?\"\n\nThe high score on En-Fr was a bit surprising. Our score of 34.9 was higher than the original XLM score of 33.4 [1], which used 8-GPUs vs our single GPU (although both are lower than fine-tuned MASS). While the exact reason for this dramatic difference in En-Fr over other language directions needs further investigation, we conjecture that word-level statistics of different pairings of languages benefit the bidirectional encoder in certain scenarios, especially where translations are more “simple”, vs translations which require widely varying sequence lengths between source and target languages. We will investigate this more to gain a better understanding.\n\n[1] Conneau, A. and Lample, G., 2019. Cross-lingual language model pretraining. Advances in neural information processing systems, 32.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "-\tThe introduction of NAR generation in QBT effectively improves the translation efficiency.\n-\tThe article is well written and easy to follow.\n", "reasons_to_reject": "-\tQBT just changes the AR structure in BT to NAR structure, and its novelty is limited.\n-\tTranslation quality improvements are limited. The main experimental results in Table 1 show that QBT can only bring about an improvement of about 0.2 BLEU, which I think is almost no improvement. For such a subtle improvement, the authors should adopt more metrics beyond BLEU (such as COMET), as well as a significance test to prove the effectiveness of the proposed method.\n-\tThe efficiency evaluation in Figure 2 is less standard. As far as I know, the speedup ratio in NAT is often calculated at batch-size=1, and it will give additional efficiency improvements under different batch-sizes.\n", "questions_to_authors": "Q1: For Table 2, why is EBTD better than EBT, but BT+EBTD is worse than BT+EBT? And BT+EBTD is generally lower than BT?\n\nQ2: For Table 3, why is the improvement (2BLEU) on En-Fr significantly larger than the improvement on the rest of the translation directions?\n", "paper_topic_and_main_contributions": "The passage discusses recent advancements in unsupervised machine translation, specifically focusing on the marriage of the Transformer model and the back-translation algorithm. While the Transformer and back-translation are powerful techniques, they have certain limitations that the authors aim to address with their proposed method called Quick Back-Translation (QBT). QBT repurposes the Transformer's encoder as a generative model, utilizing encoder-generated sequences to train the decoder in conjunction with the traditional autoregressive back-translation. This approach improves data throughput and utilization, addressing the runtime issue of autoregressive inference during back-translation and the lack of synthetic data efficiency.\n\nThe article is well written. The main problem is that the proposed method has limited improvement in the translation quality of unsupervised MT, and the evaluation metric is relatively single.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "-\tPaper is well written."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a method (quick back-translation) that can mostly be seen as an efficiency improvement for unsupervised MT, re-using the encoder of a unsupervised MT training setup to perform non-autoregressive decoding instead of using a separate, autoregressive model.

strengths:
- on average, reviewers consider the novelty one of the strengths of the paper
- writing and extensive experimentation are convincing.

weaknesses:
- improvements in quality are minimal. The main benefit is training efficiency for unsupervised MT, which has lower significance than demonstrating quality improvements, or applicability beyond unsupervised MT.
Decision: Accept-Findings