Unsupervised Neural Machine Translation with Generative Language Models Only
Authors: Jesse Michael Han
Igor Babuschkin
Harrison Edwards
Arvind Neelakantan
Tao Xu
Stanislas Polu
Alex Ray
Pranav Shyam
Aditya Ramesh
Alec Radford
Ilya Sutskever
Venue: ICLR 2022 Submitted (2022)
Keywords: unsupervised
machine translation
language modeling
few-shot learning

Abstract: We show how to derive state-of-the-art unsupervised neural machine translation systems from generatively pre-trained language models. Our method consists of three steps: \emph{few-shot amplification}, \emph{distillation}, and \emph{backtranslation}. We first use the zero-shot translation ability of large pretrained language models to generate translations for a small set of unlabeled sentences.  We then amplify these zero-shot translations by using them as few-shot demonstrations for sampling a larger synthetic dataset. This dataset is then distilled by discarding the few-shot demonstrations and then fine-tuning. During backtranslation, we repeatedly generate translations for a set of inputs and then fine-tune a single language model on both directions of the translation task at once, ensuring cycle-consistency by swapping the roles of gold monotext and generated translations when fine-tuning. By using our method to leverage GPT-3's zero-shot translation capability, we achieve a new state-of-the-art in unsupervised translation on the WMT14 English-French benchmark, attaining a BLEU score of 42.1.  

Under review as a conference paper at ICLR 2022
UNSUPERVISED NEURAL MACHINE TRANSLATION WITH GENERATIVE LANGUAGE MODELS ONLY
Anonymous authors Paper under double-blind review
ABSTRACT
We show how to derive state-of-the-art unsupervised neural machine translation systems from generatively pre-trained language models. Our method consists three steps: few-shot amplification, distillation, and backtranslation. We first use the zero-shot translation ability of large pretrained language models to generate translations for small set of unlabeled sentences. We then amplify these zero- shot translations by using them as few-shot demonstrations for sampling a larger synthetic dataset. This dataset then distilled by discarding the few-shot demon- strations and then fine-tuning. During backtranslation, we repeatedly generate translations for a set of inputs and then fine-tune single language model on both directions of the translation task at once, ensuring cycle-consistency by swapping the roles of gold monotext and generated translations when fine-tuning. By us- ing our method to leverage GPT-3's zero-shot translation capability, we achieve a new state-of-the-ard unsupervised translation on the WMT14 English-French benchmark, attaining BLEU score of 42.1.
INTRODUCTION
Recent work on generative pre-training has shown that with sufficient data and scale (Kaplan al., 2020
Henighan et al. 2020 Radford et al.] (2019), large language models (LMs) acquire remarkable in-context metalearning abilities (Brown al. 2020). One of the most striking ways this capability manifests is via sew-shot learning. where the model picks up patterns from multiple training exam- ples placed in context. While few-shot prompting flexible and enables strong performance on a diverse suite of NLP tasks to be coaxed out of generatively pre-trained LMs, its benefits are most pronounced with larger models, with commensurate training, inference, compute, and data costs. The desire to reduce these costs motivates our present work, which allows to continue finetun- ing our models, obtaining more performance from smaller models and pushing our larger models even further, without resorting few-shot prompting at test time any additional supervision at train time.
We target the domain of unsupervised neural machine translation (NMT), which typically involves bootstrapping a weak translation model before amplifying its translation ability via backtransla- tion. Recent work in unsupervised NMT has been dominated by large encoder-decoder architectures where the bootstrap is implemented by sennossingautoeecodinng tasks (e.g., multilingual Cloze (De- vlin al. 2019 Conneau & Lample 2019), masked-span prediction (Raffel al. 2020 Xue al., 2021), reconstruction from corrupted inputs (Wang et al., 2019) Liu et al.) 2020 intended to pro- duce strong encoders and aligned multilingual representations for decoding. our present work, we show that generative language modeling alone can implement the entire unsupervised NMT pipeline, and derive state-of of-e--of-the-art unsupervised NMT systems using only generatively pre-trained language models. We implement the bootstrap by first sampling a small number of zero-shot translations from GPT-3. These are then used as few-shot prompts to sample larger dataset of synthetic translations. The few-shot prompts are then discarded and the generated samples are distilled by fine-tuning the model on these synthetic data in the zero-shot format. This produces a language model aligned to translation format and amenable to large-scall scale backtranslation. By using our method to leverage GPT-3's 's zero-shot translation capability, we achieve a new state-of-the-as in unsupervised transla- tion the WMT14 4 Eaglish-French benchmark, attaining BLEU score of 42.

---

Under review as a conference paper at ICLR 2022
2
BACKGROUND AND RELATED WORK
The modern approach to unsupervised neural machine translation typically involves encoder- decoder architectures jointly trained via denoising autoencoding reconstruction tasks (Vincent Conneau & Lample 2019; Liu et al., 2020 Ma et al. 2020 Rattel et al. |020 Xue
ct al.
2008
al.
2021
Wang et al., 2019; Liu et al. [020] Song al.) 2019) and backtranslation (Sennrich Edunov et al., 2018 Cotterell & Kreutzer 2018). This approach to unsupervised NMT
et al. 2016
codified by Artetxe et al. (2018) and Lample et al. 2018), although various ideas can be traced back further: unsupervised machine translation was framed as a deciphering task by Ravi & Knight (20IT) and backtranslation was first introduced for machine translation as a method for data aug- mentation using target-side monolingual data by SSennrich et al. (016)). Denoising autoencoding with a bilingual encoder can be viewed as a kind of latent bilingual lexicon induction, necessary for producing sufficiently aligned embeddings kick-start backtranslation; such techniques have been extensively studied in the context of machine translation (Artetxe et al. 2017 Klementiev et al.] 2012 Vulic Moens
2015 Hu et al. 2017 Goyal al. 2016 Shen et al. 2017).
At the same time, recent work on large-scale generative pre-training (Kaplan et al. 2020 Henighan al. 2020 Radford et al., 2019) has demonstrated that with sufficient data and model scale, strong performance on a diverse suite of NLP tasks can be coaxed from transformer language models using few-shot prompts. Our present work unifies these two lines of research by using generative lan- guage modeling to simplify unsupervised NMT even further: we show how with sufficient scale, pre-training, and clever prompting, single generative language model can implement the entire unsupervised neural machine translation pipeline, avoiding optimizations such as denoising autoen- coding, auxiliary/ adversarial losses latent space, or ad-hoc bilingual dictionaries.
Our reliance large-scale generative pre-trainingis similar to prior work in unsupervised NMT which uses large-scale language modeling tasks on internet data as part of the bootstrap (Conneau & Conneaan et al. 2020 Liu et al. 2020). The role of few-shot prompting and distillation in our method is related to recent work on unsupervised data augmentation using language models (Anaby-Tavon) al. 2020, Schick et al. 2021 Kumar al., 2020 Papanikolaou & Pierleoni 2020
La11le|2019 Conneau
Schick Schüttze, 2021 Yang et al., (020) and is also in the same spirit as recent work on self-
training and noisy-studen! training (Mi et al. 2021 Vu et al. [022]] Xie al., 2020). The few-shot
distillation component of our method is similar to contemporaneous work by Wang et al. (2021b) which uses few-shot prompting for unsupervised data augmentation, though they focus only on inference for text classification rather than generation for scquence-to-sequeeee tasks like machine translation and they do not study the phenomena of self-amplification nor few-shot data efficiency (ection 6) as we do.
3 BACKTRANSLATION VIA LANGUAGE MODELING
Algorithm Iterated backtranslation using a single generative language model
Source monotext Ms; target monotext My; number of iterations I; number of samples per iteration J: monotext formatter f(-); bitext formatter ((sr); parameters 0 of language model p-(-) trained to complete outputs of f to outputs of g. Output: Final model parameters 0.
Input:
1: for = 1 to I do
2:
Bhack +- 0
3:
for j = 1 to J do
4:
y ~ Ms M.T
5:
X <- f(y))
6:
Brack f Brack U ((x
7:
estimate 0 by maximizing log po of y(x, for (x, E Brack
Backtranslation was first introduced in the context of machine translation as a method for data aug- mentation using target-side monolingual data (Bojar & Tamchynal 2011 Sennrich et al. 2016 Pon- celas et al. 2018), by sampling synthetic pource-to-targe data from another target-to-souree trans- lation model. In present work, cast machine translation language modeling task and
2

---

Under review as a conference paper at ICLR 2022
jointly train and sample from single language model for both source-to-taie to-tggettt and target-to translation.
Given bitext (seql, seq2) in languages L1 and L2, format the translation task as follows:
[L1] <seql> [[RAANSAATE]] [12] <seq2>
At test-time, the LM is prompted with [L1] <seq> [[TRANSLATE]] [12] and parse a candidate translation <sampledšeq> from the sampled completion. Backtranslation is implemented by reversing the roles of seq and sampledSeq and finetuning on the bitext (sampledSeq, seq).
We remark that in contrast to the interpretation of backtranslation as a wake-sleep algorithm (Cot- terell & Kreutzer 2018), where the forwards and backwards translators are trained alternately, we use a single language model for both forwards and backwards translation and train on both directions jointly at every iteration.
There are various ways train a model using backtranslation, e.g., completely online (interleaving minibatch gradient updates and sampling) versus offline (backtranslating the entire training dataset each epoch; potentially re-training the model from scratch after sampling new backtranslations) In practice, we find that data scaling of a model's 's optimal test loss and BLEU score quickly saturates on backtranslations from previous versions of the model, and opt for semi-online setup where we synchronously sample relatively small number of L1 and L2-L1 pairs before resuming training for a single epoch on the newly sampled data. We refer to this as a single iteration of backtranslation.
Formally, Algorithm describes implemention of backtranslation using single generative language model (-).). We assume that pe() has already been trained to complete format- ted monotext ([L1] <seql> [[TRANSLATE]] (I2]) formatted bitext ([L1] <seql> [ [[RANNSATT]] [1.] <sea2>).
THE BOOTSTRAP: GENERATIVE PRE-TRAINING, FEW-SHOT AMPLIFICATION, AND DISTILLATION
[en]
[fr]
[en]
[[TRANSLAT]]]
[fr]
[en]
prompt
[fr]
[en]
[en] I'll pursue solitary pathways through twilt meadows only this one dream: come too. [[TRANSLATE]] [fr] Je m'éloigne seul dans des chemins obscurs, dans les champs fombre, seulement
[fr]
[en]
aussi.
sample
[fr]
few-shot prompting
finetuning
Figure 1: Illustration of bootstrap procedure, which call few-shot distillation. We few- shot prompts sampled from GPT-3 to generate an initial dataset of synthetic translations from a generatively pretrained language model (left). The few-shot examples then discarded and the synthetic bitext reformatted for finetuning on the autoregressive language modeling objective (right).
The modern approach to unsupervised NMT is parametrized by choice of initialization boot- strap. The bootstrap has typically relied on some form of unsupervised cross-lingual representation learning, e.g., bilingual dictionaries initialized from unsupervised cross- ling-ll word embeddings (Lample et al.) 2018 Artetxe et al., (2018) or multilingual masked language modeling followed by denoising autoencoding with shared encoder and decoder (Conneau & Lample 2019).
3

---

Under review as a conference paper at ICLR 2022
In Section 3 we formulated iterative backtranslation in terms of language modeling, assuming a lan- guage model which has already been trained to follow a particular instruction format for translation. To complete procedure, must supply such language model. Unlike previous work un- supervised NMT, we use language models from the GPT-3 family (Brown et al., 2020) which have been generatively pre-trained of work around GPT-3 is that generative pre-training at scale induces strong in-context metalcarning abilities, two special cases of which are (I) instruction following and (2) few-shot prompting: sufficiently trained large language model benefits from both detailed natural language descriptions of tasks and, when given in-context examples, can achieve strong performance on diverse suite of natural language inference, translation.) We implement the boot-
large corpus of Internet data. A key observation from the body
tasks (e.g., question-answeriine strap by exploiting both of these abilities, by using natural language instruction to produce zero-shot translations and few-shot prompting during amplification.
4.1 FEW-SHOT AMPLIFICATION AND DISTILLATION
thus remains to adapt our generatively pre-trained models' few-shot translation ability to the zero-shot format specified in Section 3 We do this two-stage process. We first sample a small number of zero-shot translations from GPT-3. Given bitext (srcšeq, tgtSeqq in srclang and tgtLaag, and a stop-sequence <sep>, we use the following format for zero-shot prompting:
<sep> Given the following passage in <ercLang>; <sep> <sreSeq> <sep> a good <tgtLang> translation is: <sep> <tgtSeq> <sep>.
At test-time, we sample completion until the stop-sequence <sep> is detected; throughout we set <sep> to be \n---(n.
We amplify these zero-shot translations by using them as few-shot prompts to sample much larger synthetic dataset from a smaller model. We then distill this dataset by discarding the few-shot prompts and fine-tuning on formatted bitext, producing a language model aligned with our task format and amenable to backtranslation. In detail, we implement the bootstrap as follows:
1. Generatively pre-train a language model po(-) on a large corpus of Internet data.
2. Sample a pool of Ns synthetic target-side translations and Ng target-side translations zero- shot from another language model q(-) for few-shot prompting. Using A few-shot examples randomly drawn from Ns (resp. Nry,, sample Cs synthetic target-side translations (resp. Cr synthetic source-side translations) from pe (-), using the monolingual source-side cor- pus Ms (resp. target-side corpus Mr).
3. Discard the sho- prompts, reformat the (gold prompt, sampled translation) data as spec- ified in /Section 3 and finetune the language model pe(-) on these data.
4. Reverse all data and continue finetuning the language model po(-) on the backtranslations (sampled translation, gold prompt).
Why amplify and distill? While few-shot prompting is flexible and enables strong performance on a diverse suite of NLP tasks to be coaxed out of generatively pre-trained LMs, its benefits are most pronounced with larger models, with commensurate training, inference, compute, and data costs. is also unclear how to iteratively finetune language model in way that preserves its few-shot ability while remaining aligned with a zero-format like in SSeSettiion 3] Few-shot amplification allows us to generate data for the bootstrap in an unsupervised fashion, possibly avoiding the overhead of few-shot sampling from GPT-3 itself by few-shot prompting smaller model po po(.), while distillation enables iterative backtranslation.
5 RESULTS
Experimental setup For our experiments, we focus the well-studied WMT14 English-French benchmark. In the notation of Algorithm I we obtain source and target monotext Ms and Mr by splitting the WMT14 English-French training set in half, each with approximately twenty mil- lion examples, and use only the English text from one half and only French text from the other to avoid implicit sentence-level alignment between source and target monotext. At each iteration of
4

---

Under review as a conference paper at ICLR 2022
backtranslation, we sample one million translations in either direction, i.e. J 2e6, and train for one epoch on the newly sampled data. For all of our results, unless otherwise specified, we run 40 iterations of backtranslation after the bootstrap and report BLEU using the final model checkpoint.
To implement the bootstrap, we additionally set aside 2048 training examples, and sample Ng = 1024 English-French (resp. Nr. 1024 French-English) translations zero-shot from GPT-3 to use as few-shot prompts. During few-shot amplification, we sample four million initial target- and source-side translations respectively using few-shot prompls, L.., Cs CT 4e6 in the notation of |Section 4.1] drawing monolingual prompts from as Ms and MM defined above. We finetune for two epochs in the forwards direction (distillation) and for another two epochs in the backwards direction (initial backtranslation). For few-shot prompting, we use k = 3 in-context examples. In Section 6.3. we will see that we can minimize the number of few-shot examples to Ng Ny with little effect on evaluation BLEU score after iterative backtranslation.
We use the same training setup and BPE tokenizer as GPT-3. During finetuning, we use a constant learning rate of 0.05 L, where & is the pre-training learning rate, weight decay of 0.1, and residual dropout 0.1. When sampling during the bootstrap or during backtranslation, we default to using temperature 0.3. We ablate other values of r in Section 6.1]
We report BLEU score on the official WMT14 English-French test set with greedy (argmax) sam- pling and sacreBLEU" Post 2018). In Table 3 we give comparison to previous work unsuper- vised NMT using mult touultitbbi titttti-eue... aleu.per. and the XLM (Conneau & Lample 2019) tokenizer.
5.1 FEW-SHOT SELF-DISTILLATION AND BACKTRANSLATION
small 1
med i um
large
x1
en-fr
1.15
7.71
13.07
14.28
few-shot (r = 0.0)
fr-en
5.04
16.87
20.25
23.0
en-fr
1.02
7.36
11.89
13.58
few-shot (r = 0.3)
fr-en
4.46
16.13
20.7
22.07
en-fr 0.25
2.12
2.68
3.38
few-shot (r 1.0)
fr-en
1.22
5.45
6.14
9.32
en-fr 0.61
9.51
17.68
22.19
distillation
fr-en
4.31
23.67
29.38
31.12
en-fr 7.94
29.84
33.59
34.71
initial backtranslation
fr-en
1.5
23.12
28.58
30.52
en-fr fr-en 27.24
30.48
36.53
37.59
39.12
after backtranslation
32.15
34.79
35.43
Table 1: English-French (top) and French-Englist (bottom) test BLBU throughout the few-shot self-distillation bootstrap across multiple model scales.
We first report results using delf-iistillation, i.e., where during the bootstrap (Section 4) we sam- ple from single model which then trained imitate and then backtranslate its own few-shot prompted generations; for these experiments, the few-shot demonstrations themselves are generated zero-shot by GPT-3. This is then followed by the iterative backtranslation procedure described in Section 3 We apply this methodology to the small amall. 1. medium, 1 large, and x1 models from the GPT-3 family (Brown al. 2020), with 125M, 350M, 760M, and 1.3B parameters respectively. Table displays test BLEU throughout our procedure for all model sizes. We see that translation out of English benefits significantly from the backtranslation part of the bootstrap alone. We also see that our models are much stronger at the translation task compared to few-shot prompting after only self-distillation. Finally, all models benefit significantly from iterative backtranslation, with Beglish-Frenche BLEU always converging slightly higher value than the reverse direction.
'SarreBI.F signature: BI.uUcaae, mi xed+numref's 1+mmooth. mmmottheexpttok int +versiionnnn... 22.20.
5

---

Under review as a conference paper at ICLR 2022
5.2
DISTILLING DEPLIIEDD GPT-3 INTO SMALLER MODELS
small
medi um
large
x1
en-fr 34.13
36.03
37.21
37.08
distillation
fr-en 32.34
34.96
36.12
36.34
en-fr 34.71
36.31
38.89
39.05
initial backtranslation
fr-en 30.95
33.73
35.16
36.51
en-fr 35.62
37.79
38.91
39.79
after backtranslation
fr-en 31.28
34.08
35.57
35.97
en-fi 39.02
41.31
41.97
42.08
after backtranslation (+CC100)
fr-en 33.43
35.69
36.85
37.09
Table 2: English-French (top) and French-English (bottom) test BLEU throughout the bootstrap and after iterative backtranslation, this time using generations from self-amplified GPT-3 for the bootstrap. We observe the best performance by mixing in monotext from the English and French components of the CC100 dataset (Wenzek et al. 2020 Conneau et al.) 2020) during hacktranslation.
Although we do not apply our full methodology to the 175B parameter GPT-3 model due compute constraints, we observe that for few-shot distillation, instead of training model on few-shot samples from itself, we can just as well distill on few-shot samples from a much larger model instead- this case, the full-size 175B parameter GPT-3 model (henceforth just "GPT-3"). That is, we GPT-3 to self-amplify its own zero-shot translations to produce an initial dataset for distillation.
We now proceed to apply the same method as in Section 5. to all model sizes, but this time using few-shot samples from GPT-3 for the bootstrap. We display the evaluation BLEU scores throughout the bootstrap and after iterative backtranslation in Table 2 Interestingly, the higher-qualij, samples from GPT-3 appear to saturate the smaller models and they improve very little. Motivated by the possibility that models are beginning overfit the WMT14 English-French training data, we attempt another experiment where 50% of the monotext for backtranslation is sampled from the English and French components the CC100 dataset (Conncau et al. 2020). The extra monolingual data significantly benefits all model scales, improving English-French BLEU by approximately points compared to iterative backtranslation on WMT data alone. With this setup, the x1 attains new unsupervised state-of-art art of 42.1 BLEU on the WMT14 English-Frende benchmark.
6 DISCUSSION AND FURTHER ABLATIONS
Bias towards English generation Previous work (Brown et al. 2020) has shown that after genera- tive pre-training corpus of English-done degeintedd Internet text, GPT-3 models are far more capable of translating into English than translating out of English. This is reflected by the disparity between English-Frent French and French-Engee backtranslation on the few-shot prompted data. Interestingly, after only two epochs backtransla- tion on the relatively scarce few-shot prompted data, this gap is reversed, with all models achieving significantly higher English-French BLEU than French-English BLEU. The data efficiency of the bootstrap suggests that coming out of pre-training, the models are merely misaligned rather than deficient in knowledge about French, and that their latent knowledge about translation out of En- glish can be surfaced using backtranslation. Relatedly, high-qualit, samples in one language in the previous round of backtranslation lead to higher-quality synthetic bitext for training the reverse di- rection the next. This turns the asymmetry towards English generation into an advantage during backtranslation. However, if the initial disparity between the quality of the translation directions is extreme (as with the self-distilled small, which achieves < 2 BLEU for English-Frennd French few-shot compared to x 10 BLEU for French-English then see that the evaluation BLEU for either direction is unstable and oscillates between iterations, though they eventually converge upwards as backtranslation continues.
BLEU scores immediately after few-shot distillation and before
Comparison to previous work In Table we compare the BLEU scores attained by our best model (an xl distilled self-amplified GPT-3 followed by 40 rounds of backtranslation) to prior

---

Under review as a conference paper at ICLR 2022
work in unsupervised neural machine translation the WMT14 English-Frence benchmark. To ensure comparability to prior work, we report tokenized BLEU using mult molti-bleuu perr and the XLM tokenizer. This used to report the few- and zero-shot performance of GPT-3 in Brown
et al. 2020), which we also include Table for completeness.
XLM
MASS
CUNMT
XLM+
CBD
GPT-3 (fs)
GPT-3 (zs)
en-fr 33.4
37.5
37.6
40.2
38.2
41.7
32.6
25.2
fr-en
33.3
34.9
35.2
36.9
35.5
38.0
39.2
21.2
Table 3: Comparison of our best model-an distilled on self-amplified GPT-3 followed by 40
rounds of iterative backtranslation-11 to prior work (Conneau & Lample 2019: Song et al.) 2019
Wang et al., 2021a| Keung et al. 2020 Nguyen et al. 2021) unsupervised NMT on the WMT14
English-Frened benchmark. Bold indicates unsupervised state-of-the-ae and underline indicates
few-shot state-of-the-art
6.1
ABLATING TEMPERATURE FOR FEW-SHOT DISTILLATION
self-distill backtrans.
0.0
backtrans.
0.3
backtrans.
1.0
en-fr 20.3
34.4
34.7
27.8
0.0
fr-en
29.9
29.3
29.6
24.7
en-fr
20.6
33.9
35.1
27.6
== 0.3
fr-en
29.2
28.9
29.9
24.4
en-fr
20.2
34.9
34.6
27.6
7 1.0
fr-en
29.0
29.2
29.2
24.9
Table 4: English-French (top) and French-English (bottom) test BLEU using few-shot prompted samples generated with temperatures 0.0, 0.3, 1.0 throughout the bootstrap. We see that the temperature used for sampling little effect on evaluation BLEU after few-shot distillation, while high-temperature samples are harmful during the backtranslation part of the bootstrap.
was shown by Edunov et al. (2018) that backtranslation is more effective when the translations are slightly noisy, i.e., sampled with nonzero temperature via noised beam search. This motivated our use of the temperature = 0.3 throughout. We ablate this choice of temperature when sampling data for few-shot distillation, and study the effect of using using a 1 large model. We display the results in Table 4 We see that lower temperatures lead marginally higher test BLEU scores during distillation while and no overfitting after two epochs of training. However, regardless of the temperature of samples used for ceff-distillation the differences in both test BLEU and test loss almost vanish after the backtranslation part of the bootstrap when training to backtranslate low temperature samples (T or 0.3).
0.0 and 1.0 during the bootstrap
1.0 results in lower test loss
6.2 FEW-SHIOT SELF-AMPLIFICATION
We observed that few-shot prompting GPT-3 with its own zero-shot translations produced better translations than zero-shot prompting alone. We investigate this further by comparing the BLEU scores of zero-shot translations (sampled using the same prompt described in(Seciion 4) to the BLEU scores of colffiilli few-shot prompted translations (i.e., where the few-shot demonstrations are the zero-shot translations sampled from the same model) for all the model sizes studied in this paper. Our results are displayed in Table We see that self-am olff-mmliiication improves translation quality at all model scales.
6.3 USING REAL FEW-SHOT EXAMPLES
So far our results have been completely unsupervised, but few-shot learning is typically studied in the context of semi-supervised learning (Wang al. 2020), where the few-shot demonstrations

---

Under review as a conference paper at ICLR 2022
small
medi um
large
x1
GPT-3
zero-shot en-fr
0.57
1.23
1.90
2.84
26.19
fr-en
2.00
13.92
8.14
19.60
25.49
en-fr
1.39
8.98
12.46
14.32
29.96
self-amplified
fr-en
5.76
16.75
21.75
23.98
31.75
Table 5: Zero-shot versus few-shot self-amplified test BLEU for all model sizes studied in this paper. For zero-shot generation we use the same prompt format described in [Section 4 For self-amplified generation, the model's own zero-shot generations as in-context few-shot examples.
small
medium
large
x1
en-fr 1.09
7.19
11.8
13.35
few-shot (r 0.0)
fr-en
3.86
14.58
20.34
23.01
en-fr 1.09
6.83
11.38
13.08
few-shot (r = 0.3)
fr-en
4.13
14.86
19.92
22.04
en-fr 0.33
1.74
2.34
2.94
few-shot (r = 1.0)
fr-en 0.94
4.18
4.64
7.25
en-fr 0.39
7.63
17.27
19.81
distillation
fr-en
3.9
20.29
27.65
30.89
en-fr
7.77
24.71
29.64
33.78
initial hacktranslation
fr-en
1.7
18.9
26.61
30.93
en-fr
31.23
34.42
37.86
39.39
after backtranslation
fr-en
27.45
29.96
34.23
34.97
Table English-Free French (top) and French-Engline (bottom) BLEU throughout the few-shot self- distillation bootstrap across multiple model scales, this time using real few-shot examples. We see that performance after backtranslation equivalent that reported in Table
small 11
large
en-fr 32.95
36.0
distillation
fr-en
32.45
36.29
en-fr 36.32
38.72
initial backtranslation
fr-en 32.43
36.61
en-fr 36.38
39.36
after backtranslation
fr-en 32.66
35.67
en-fr 39.01
42.03
after backtranslation (+CC100)
fr-en
34.17
36.94
Table 7: English-Frenne (top) and French-English (bottom) test BLEU of the small and large models throughout the bootstrap and after iterative backtranslation, where for the bootstrap we use generations from 175B GPT-3 prompted using real few-shot examples. Similarly to TTable 2 we observe a boost in final BLEU score when, after the bootstrap. additionally sample monolingual text from the English and French portions of the CC100 dataset.
real training data. In this section, ablate the usage of synthetic few-shot translations in methodology and reproduce our experiments from [Section 5] Jusing real few-shot demonstrations. We observe virtually difference in BLEU score after iterative backtranslation.
8

---

Under review as a conference paper at ICLR 2022
We modify the few-shot prompting described in Secciinn 5 as follows. Rather than sampling zero- shot translations for each half our held-out pool of N=2048 training examples, we sample from these examples directly during few-shot prompting.
Table displays test BLEU throughout the bootstrap and after iterative backtranslation for the same model sizes studied in [Section 5.1 We see that models converge the same test BLEU (c.f. [Sec- tion 5.1). Table 7 displays analogous results when distilling samples from GPT-3 with the small and large models, this time few-shot prompted using real examples. We again see that using real rather than synthetic few-shot demonstrations to sample the initial bootstrap data from GPT-3 has no effect on final BLEU score after iterative backtranslation.
6.3.1
ALMOST-UNSUPERVIISI MACHINE TRANSLATION WITH THREE EXAMPLES ONLY
N=3
N=8
N=16
N=32
N=64
N=128
N=256
N=512
N=1024
N=2048
en-fi
12.6
12.4
12.7
13.1
13.2
13.0
12.7
12.9
12.7
12.8
fr-en
21.5
21.3
22.1
22.4
21.9
22.3
22.1
22.1
22.2
22.1
Table 8: BLEU scores (calculated over 4096 random training examples) for the few-shot prompted translations from a large model, the total number of available few-shot examples varies from N = 3 to N = 2048. We see that N has minimal impact on the BLEU score of the sampled translations. Moreover, the difference in BLEU between the models bootstrapped using N = versus N = 2048 disappears after iterative backtranslation.
Finally, we show that even in the semi-supervised setting, we can minimize the supervision available from few-shot demonstrations with no difference in test BLEU after backtranslation coverges, Ta- ble displays the BLEU scores of few-shot sampled translations across various orders of magnitude of N, the number of available few-shot examples. Remarkably, even when N is decreased to 3, there is only a slight negative impact on the BLEU score of the few-shot sampled translations. We do not ablate lower values of N in order to maintain the assumption of k=3 distinct in-context examples for few-shot prompting. We then run our entire procedure with a large model, using N=3 real few-shot demonstrations for the bootstrap followed by iterative backtranslation. We observe final English-French BLEU of 38.0 and French-Engee English BLEU of 34.2, on par with the final BLEU scores reported in Table 6
7 CONCLUSION AND FUTURE DIRECTIONS
We remark that backtranslation, like reinforcement learning, simply a way of exchanging com- pute for data. Instead of grounding the model with reward signal from environment, however, backtranslation exploits the symmetry of the translation task to ground the model by training cross-lingually denoise its own samples. Our present work can be viewed part of recent trend towards data-driven architecture engineering, where task-specific inductive biases, if any, are engineered into and learned from the training data instead of being hardcoded into the model archi- tecture. In formulating the translation task in terms of language modeling, we see that the input- output inductive bias imposed by an encoder-decoder architecture can be simulated with prompt formatting. Similarly, we see that generative language modeling at sufficient scale combined with clever prompting for automated data generation can attain sfate-of-the-ar results in unsupervised translation, rendering methods intended to produce strong encoders and aligned multilingual repre- sentations unnecessary.
Although we have focused solely on the domain of machine translation in this work, our methodol- ogy is applicable to any sequence-to-sequenee task whose forwards and inverse directions are (1) to be jointly learned by autoregressive decoder-only transformer and (2) are amenable to few-shot prompting after large-scale generative pre-training. Backtranslation is simply reverse self-training (Bojar& & Tamchyna 2011) and is fundamentally untied to the translation domain; we invite the search community at large to further explore this technique, moving beyond translation and towards applications reflecting the full generality of the transformer architecture.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Ateret Anaby-Tavor, Boaz Carmeli, Esther Goldbraich, Amir Kantor, George Kour, Segev Shlo- mov, Naama Tepper, and Naama Zwerdling. Do not have enough data? deep learning to the rescue! In The Thirty-Fourth AAAI Conference Artificial Intelligence, AAAI 2020, The Thirty- Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12. 2020, pp. 7383 7390. AAAI Press, 2020. URL https: /aaaiaaaiaa..a http:////aaaaaaa////aaaaa////
index. php/ AAAI http://AAAaaaatttaaaaaaa /vvve//wwwww 6233
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. Learning bilingual word embeddings with (almost) no bilingual data. In Regina Barzilay and Min-Yes Kan (eds.), Proceedings of the 55th Anmual Meeting of the Association for Computational Linguistics, ACI. 2017, Vancouver, Canada, July 30 August Volume I: Long Papers, pp. 451-4622 Association for Computational Linguisties, 1/P1--1444
2017. doi:
Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. Unsupervised neural machine translation. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30- May 2018, Conference Track Proceedings. OpenReview.net, 2018. URL pettps://openreviee net.eeww.eee/eee....m. wtrumrrrm?mm.
www.ppaaweieeeeeettoorrr...a.. (oorumiid-Sy2ogebbA
Ondrej Bojar and Ales Tamchyna. Improving translation model by monolingual data. In Chris Callison-Buree Philipp Koehn, Christof Monz, and Omar Zaidan (eds.), Proceedings the Sixth Workshop on Statistical Machine Translation, WMT@EMMN 2011, Edinburgh, Scotland, UK, July 30-31, 2011, pp. 330 336. Association for Computational Linguistics, 2011. URL https:
o
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Anmual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL 4bt httpc/0dbff4444440 2tto://666..ooo.aaoooaaaaaa22aa0
neur. neurips. http://s.....r.oaa/////////////0
Abst html
Alexis Conneau and Guillaume Lample. Cross-lingual language model pretraining. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Bue, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 7057-7067, 2019, URL Ut..hnttpss://prcceeeddni neur heuripp ips. CC/ cc/paper/ html
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Un- supervised cross-lingual representation learning at scale. In Dan Jurafsky, Joyce Chai, Na- talie Schluter, and Joel R. Tetreault (eds.), Proceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pp. 8450- Association for Computational Linguistics, 2020. doi: 10.18653/v1/222..oo aww.:000...ammi......mm. 747. URL https::/doi..rg// www.o////....../..... www.//66533112220... aww-mmaa...........mm... 747
Ryan Cotterell and Julia Kreutzer. Explaining and generalizing gennnniiinggggn wawperaalliiinbbaaaaarraaaaaaaae back- translation through wake-sleep.
CoRR, aass/180.044/ 04400. 2018. IRtttt//://axx wwwp:/ii.orggaa..oo
org abs 1806. 04402
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HIT
10

---

Under review as a conference paper at ICLR 2022
2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pp. 4171-
4186. Association for Computational Linguistics, 2019. doi: 10.18653/v1/n19 1423. URL
wt1/////3v//// wwl.i////11ii 1423
Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. Understanding tack-translation at scale. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 4, 2018, pp. 489-500 Association for Computational Linguisties, 2018. doi: 10.18633/v1/d1.. 1045. utt.ttts::/ owtp//oorg/000
18653/v1 htt.g//....33///////.8.
Anirudh Goyal, Alex Lamb, Ying Zhang, Saizheng Zhang, Aaron C. Courville, and Yoshua Bengio. Professor forcing: A new algorithm for training recurrent networks. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 29: Annual Conference on Neu- ral Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 4601 (601-4609, 2016. URL httpppts///prrrced.ngg.... neuripp cc/paper/22l6/haaee html
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Rad- ford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei, and Sam McCandlish. Scaling laws for autoregressive generative modeling. CoRR, abs/2010.14701, 2020. URL httpthtss//axii..ooo
Zhiting Hu, Zichao Yang. Xiaodan Liang, Ruslan Salakhutdinov, and Eric P. Xing. Controllable text generation. CoRR, abs/1703.00955 2017. URL http:: /tttt://arxiv..oo www.:ii..rrrrrooooooo org a/00333./////..././// 00955
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. CoRR, abs/2001.0836) 2020. URL https: http///xxii...//ab//////// 08361
Phillip Keung, Julian Salazar, Yichao Lu, and Noah Smith. Unsupervised bitext mining and translation via self-trained contextual embeddings. Trans. Assoc. Comput. Linguisties, 8:828- 841, 2020. URL stttps::/traanaac..orrt /indeexxxxx//eeeeeeee/eeo/xxxxx phtp/tall/atticle//aae
2233
Alexandre Klementiev, Ivan Titov, and Binod Bhattarai. Inducing crosslingual distributed represen- tations of words. In Martin Kay and Christian Boitet (eds.), COLING 2012, 24th International Conference on Computational Linguisties, Proceedings of the Conference: Technical Papers, 8-15 December 2012, Mumbai, India, pp. c12-108// 2012. URL httptt////ccaaatolleeeeeee
Indian Institute of Technology Bombay,
Varun Kumar, Ashutosh Choudhary, and Eunah Cho. Data augmentation using pre-trained trans- former models. CoRR, abs/2003.02245 2020. URL
02245
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Mare' Aurelio Ranzato. Unsupervised machine translation using monolingual corpora only. In 6th International Conference on Learn- ing Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. WtRLhttps:://cc /attts:://penreviie... net wtw.rrrriiweneeefoouumi.eeeee forum?id-
EkYTTf-RR AZ
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. Multilingual denoising pre-training for neural machine transla- tion. Trans. Assoc. Comput. Linguisties, 8:726-742, 2020. URL https::/transac. oorg/ ojs/indes pttp://al/aattice///llel////////
Shuming Ma, Jian Yang, Haoyang Huang. Zewen Chi, Li Dong, Dongdong Zhang, Hany Hassan Awadalla, Alexandre Muzio, Akiko Eriguchi, Saksham Singhal, Xia Song, Arul Menezes, and Furu Wei. XLM-T: scaling up multilingual machine translation with pretrained cross-lingual transformer encoders. CoRR, a55/2012.15547, 2020. URL https: /ttp///a.vv.////// org//bs/ 201 2012.1554 15547

---

Under review as a conference paper at ICLR 2022
Fei Mi, Wanhao Zhou, Fengyu Cai, Lingjing Kong, Minlie Huang, and Boi Faltings. Self- training improves pre-training for few-shot learning in oriented dialog systems. CoRR, abs/2108.12589, 2021. URL
Xuan-Phi Nguyen, Shafiq R. Joty. Thanh-Tung Nguyen, Kui Wu, and Ai Ti Aw. Cross-model back-translated distillation for unsupervised machine translation. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Re- search, pp. 8073- 8083. PMLR, 2021. URL nquyen2lc.htm.
Yannis Papanikolaou and Andrea Pierleoni. DARE: data augmented relatii relation extraction with GPT-2. CoRR, abs/2004. 0ls/2004.13845,, 2020.
arxiv. 13344/1.0/000444555555 13845
Alberto Poncelas, Dimitar Sht. Shterionov, Andy Way, Gideon Maillette de Buy Wenniger, and Pey- man Passhan. Investigating backtranslation neural machine translation. CoRR, 0bss/180..66
2018. URL UR./tttp:///rrs...rrr////////// 8004. C6189
Matt Post. A call for clarity in reporting BLEU scores. In Ondrej Bojar, Rajen Chatterjee, Chris- tian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno- Yepes, Philipp Koehn, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana L. Neves, Matt Post, Lucia Specia, Mareo Turchi, and Karin Verspoor (eds.), Proceedings of the Third Con- ference on Machine Translation: Research Papers, WMT 2018, Belgium, Brussels, Oclober 31 November 1, 2018, pp. 186-191. Association for Computational Linguistics, 2018. doi: 1036653/11/11-631110 URL https: /ttp:://aoi.ccg//1111888333
Alec Radford, Jeffrey Wu, Rewon Chield, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 188),9 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to- text transformer. J. Mach. Learn. Res., 21:140:1-140:67, 2020. URL attp::/jmm:: org/
[papers/221/0-000 hhmm
Sujith Ravi and Kevin Knight. Deciphering foreign language. In Dekang Lin, Yuji Matsumoto, and Rada Mihalcea (eds.), The 49th Annual Meeting of the Association for Computational Lin- guistics: Human Language Technologies, Proceedings of the Conference, 19-24 June, 2011, Portland, Oregon, USA. pp. 12-21. The Association for Computer Linguisties, 2011. URL
wtt.ooggg...... org/P11-1002/ 1002/
Timo Schick and Hinrich Schütze. Generating datasets with pretrained language models. CoRR, abs/2104.07540, 2021. URL http::hs/aaw.....og////////////e 200.//44444000
Timo Schick, Sahana Udupa, and Hinrich Schütze. Self-diagnosis and self-debiasing: A proposal for reducing corpus- based bias in NLP. CoRR, abs/2103.00043, 2021. URL bttps://arxix:
orq/ hrqg/bb/// 2113...04 00453
Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving neural machine translation mod- els with monolingual data. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016. Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics, 2016. doi: 10.18653//1/116-10000 URL https: //s:://ooi... http://....1.5/////e/eee
Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi S. Jaakkola. Style transfer from non- parallel text by cross-alignment. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus. S. V. N. Vishwanathan, and Roman Gamett (eds.), Ad- vances in Neural Information Processing Systems 30: Annual Conference on Neural In- formation Processing Systems 2017, December 4-9, 2017, Long Beach, CA. USA, pp. 6830-6841, 2017. URL http:////ppcceeee....eee.......p 2d2c8394e3110188 ab Abst http:///....ppppe//////aa/aaaa ott.aaa0111111222aooaoa110011110
12

---

Under review as a conference paper at ICLR 2022
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Yan Liu. MASS: masked sequence to quence pre- pre-training for language generation. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, ICML 2019, 9- 15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 5926-5936. PMLR, 2019. URL/htt:: song19d.htm
ml www.mmm.........mm press/v97/
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In William W. Cohen, Andrew McCal- lum, and Sam T. Roweis (eds.), Machine Learning. Proceedings of the Ewenty-Fifth International Conference (ICML. 2008), Helsinki, Finland, June 5-9, 2008, volume 307 of ACM International Conference Proceeding Series, pp. 1096-1 103. ACM, 2008. doi: 04.1145/1390155 0111566..199444
URL hnttps://doi.orge 1tle://..11111111111 1145111111111 1.0556...339000 1390294
Tu Vu, Minh-Thang Luong, Quoe Le, Grady Simon, and Mohit lyyer. Strata: Self-training with task augmentation for better few-shot learning. CoRR, abs/2109.06270, 2021. URL https:
7/
Ian Vulic and Marie-Francine Moens. Bilingual word embeddings from non-parallel document- aligned data applied to bilingual lexicon induction. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, ACL 2015, July 26-31, 2015, Beijing, China, Volume 2: Short Papers, pp. 719-725, The Association for Computer Linguistics, 2015. doi: 10.3115//1//15-21188. URL BRLLhttps:/// /ectps://doi.ooo org/10. 311 3115/v1/p15 2118
Liang Wang, Wei Zhao, Ruoyu Jia, Sujian Li, and Jingming Liu. Denoising based sequence-to- sequence pre-training for text generation. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xi- aojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Lan-
guage Processing and the 9th International Joint Conference on Natural Language Process- ing. EMNLP-IJCNLP 2019, Hong Kong. China, November 3-7, 2019, pp. 4001-4013 Asso-
ciation for Computational Linguistics, 2019. doi: 10.18653/v1/D11-14422 URL https:
/tdeii.rr/// 8653/616vvvvv /D1 1412
Mingxuan Wang, Hongxiao Bai, Lei Li, and Hai Zhao. Cross-lingual supervision improves un- supervised neural machine translation. In Young-bum Kim, Yunyao Li, and Owen Rambow (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Associa- tion for Computational Linguistics: Human Language Technologies: Industry Papers, NAACL- HLT 2021, Online, June 6-11, 2021, pp. 89-96. Association for Computational Linguisties, 2021a. doi: 10. 10.18633/v1/2021. maall-industry.12 12. 2021 nwac
11t.o////1..011111111111 18653/v1
innustrr..... 12
Yaqing Wang, Quanming Yao, James T. Kwok, and Lionel M. Ni. Generalizing from few ex- amples: A survey on few-shot learning. ACM Comput. Surv, 53(3):63: 2020. doi: 10.1145/3386252. URL / //cci. wtt//oo.ggllooooo.
3386252
Zirui Wang, Adams Wei Yu, Orhan Firat, and Yuan Cao. Towards zero-label language learning. CoRR, abs/2119.0919/) 2021b. BRLLttps::/ http://////.xx...ooooo////////// 091.099.999993 09193.
Guillaume Wenzek, Marie-Anne Lachaux. Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. Cenet: Extracting high quality monolingual datasets from web crawl data. In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mar- iani, Hélène Mazo, Asunción Moreno, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of The 12th Language Resources and Evaluation Conference, LREC 2020, Marseille, France, May 11-16, 2020, pp. http:s::///aaaaahlloggoo
European Language Resources Association, 2020. URL org/2020..0020 2ttp/2222.11.e11... 494/
Qizhe Xie, Minh- Thang Luong, Eduard H. Hovy, and Quoe V. Le. Self-training with noisy student improves imagenet classification. In 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR 2020. Seattle, WA, USA, June 13-19, 2020, pp. 10684-10695. Computer Vision Foundation IEEE, 2020. doi:
13

---

Under review as a conference paper at ICLR 2022
10.1009CCYP4420000000000000
URL https://openacees: thecvf. com/
content_cvEer 2020/html/Xie Self-Training With_WWisth
Improves ImageNet Classtficatiiioinnoo..ccmiciiiti CVPR 2020 wwppp.pppppp.. html
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-texi transformer. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani- Iz Beltagy. Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Techmologies, NAACL-HLT 2021, Online, June 6-11, 2021, pp. 483-498. As- sociation for Computational Linguistics, 2021. doi: 10.18653/v1/2021 maacl-main.41. URL
https://doi.org/10 1et.....113331.0.. 22.. www.naaacllamocm www.aaaaaaaaaaaaa.
Yiben Yang, Chaitanya Malaviya, Jared Fernandez, Swabha Swayamdipta, Ronan Le Bras, Ji- Ping Wang. Chandra Bhagavatula, Yejin Choi, and Doug Downey. G-daug: Generative data augmentation for commonsense reasoning. In Trevor Cohn, Yulan He, and Yang Liu (eds.), Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16- 20 November 2020, volume EMNLP 2020 of Findings of ACL, pp. 1008-1025, Association for Computational Linguistics, 2020. doi: 10. 10.18663////2222nnndnnggg www.iingg-eeenlp.9000 URL https:
7/doi org 10, 1ttp:/10..16653/1/2220000 findin
14


Reviews:
[{"review_id": "XCc4qkWo-KU", "reviewer": "Reviewer_K8G8", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. A new approach to construct UNMT with generative pre-trained language models.\n2. The proposed method achieves the new start-of-the-art unsupervised translation performance on the WMT14 English-French benchmark.\n\nWeaknesses:\n1. This paper only verifies the effectiveness of the proposed method on the English-French language pair. It is better to conduct experiments on other language pairs, such as English-German, to make the result more reliable.\n2. Currently, the performance of UNMT model on distant language pair is very poor. It would be more exciting, if we also observe the significant improvement of the proposed method on some distant language pairs, such as English-Chinese and English-Japanese.\n\nDetailed Comments:\n\nOverall, this paper is easy to follow and understand. I like this idea of constructing UNMT system with the GPT-X model and the proposed method is very interesting. Also, the experimental results on WMT14 English-French benchmark surprise me. \n\nMy main concern is the experiment settings. This paper merely verifies the effectiveness of the proposed method on the English-French language pair, making the experiment results not convincing enough. It is better to conduct experiments on other language pairs, such as English-German. Previously, the performance of UNMT methods on distant language pair is very poor. As the proposed method replaces the bilingual dictionary with few-shot amplification, it seems to me that the proposed method may achieve better performance on distant language pairs. I am looking forward to observing the unsupervised translation performance of the proposed method on some distant language pairs. \n\nQuestions for the Author(s):\n1. As the proposed method requires large-scale model to achieve the start-of-the-art performance, what is the training cost/time? \n2. In fact, this proposed method could be applied in sequence-to-sequence pretraining model, such as mT5. Have you explored this direction? Or leveraging synthetic parallel data generated by the GPT-3 to warm up the previous UNMT models?\n\nMissing Reference:\n1. Unsupervised Neural Machine Translation with Weight Sharing. Yang et al., ACL 2018\n2. Phrase-Based & Neural Unsupervised Machine Translation. Lample et al., EMNLP 2018\n3. Unsupervised Neural Machine Translation with SMT as Posterior Regularization. Ren et al., AAAI 2019\n4. Extract and Edit: An Alternative to Back-Translation for Unsupervised Neural Machine Translation. Wu et al., NAACL 2019\n5. An Effective Approach to Unsupervised Machine Translation. Artetxe et al.,  ACL 2019\n\n", "summary_of_the_paper": "This paper investigates the way of constructing the UNMT model with a generative pre-trained language model, such as the GPT-X model. This method consists of few-shot amplification, distillation and back-translation, in which the first two steps are leveraged to warm up a UNMT model instead of bilingual dictionaries inferred in an unsupervised way. When using GPT-3 to obtain synthetic parallel data for model training through the zero-shot prompting approach, the proposed method achieves the state-of-the-art unsupervised translation performance on the WMT14 English-French benchmark.  ", "summary_of_the_review": "This paper is easy to follow, and the idea is very interesting and novel. The proposed method achieves the new start-of-the-art unsupervised translation performance on the WMT14 English-French benchmark. It is better to conduct additional experiments on other language pairs to make the experiment results more reliable.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "OhtAztCLsJ", "reviewer": "Reviewer_bUnG", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Interesting results, but I think this study is already standing at other position than so-called \"unsupervised translation\" because it uses pretrained LM as a seed and thus the model is based on a huge amount of additional resources. It would be not reasonable to say state-of-the-art by comparing its accuracy with other \"unsupervised\" models trained from only a specific corpus.\nTable 8 is also interesting: in this task we can only prepare monolingual corpora for each language/domain and only a few seed translations for prompts (if we didn't generate them from LMs).\n\nSince pretrained LMs are trained using a bunch of resources available from the Internet, its training data possibly includes the training/evaluation data used in this experiment, meaning that there would be some suspicion about data leakage. Is it checked appropriately that the training data were not leaked, or could you say the experiment is actually safe without this check for some reason?\n\nOverall, the text is hard to read to me. Many terms were introduced without definitions, the sentence frequently omitted important contexts, and syntax was often broken.", "summary_of_the_paper": "The paper proposed an iterative finetuning method to adapt pretrained LMs to translation tasks. The framework refines the LM into translation models, which accepts both languages with a predefined format, and translate them into other language bidirectionally.\nThe model is finetuned by backtranslation.\n\nTo generate translation counterparts of the given monolingual corpora during finetuning, the paper utilized prompting on LMs: gives several translations X1 Y1 X2 Y2 ... as contexts, then predict the successor Yn given the real input Xn, where Xk and Yk are souce/target sentences.\n\nThe context of the prompt X1 Y1 X2 Y2 is also generated by LM using another template, but this part can be replaced as a real examples.\n\nExperiments show that the trained system achieves the better performance than existing similar methods, showing that the effectiveness of the proposed method. The paper provided some additional studies to support the main results.", "summary_of_the_review": "- The paper employed huge LM/prompting/backtranslation techniques to construct bidirectional translation models. The technique of model construction seems remarkable, and there is some concern about data.\n- The paper is somewhat hard to read due to syntactic/semantic issues, and I guess I possibly drops some context to assess the paper correctly.\n- I am not fully familiar with recent advances in so-called \"unsupervised translation\" and huge LMs, and I didn't judge if the proposed method is novel or not, and not provided comparison with existing studies.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ENNOynusDoP", "reviewer": "Reviewer_LmFj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n - State-of-the-art unsupervised translation results for both English-French and French-English.\n\nWeakness:\n - Lack of novelty\n - Only one non-English language is considered. \n - No low-resource language pairs considered, which is the realistic setting for unsupervised MT.\n - No in-depth discussion on the aspects which separate this work from other unsupervised MT works.\n\nThe authors claim that \"with sufficient scale, pre-training, and clever prompting, a single generative language model can implement the entire unsupervised neural machine translation pipeline, avoiding optimizations such as denoising autoencoding, auxiliary / adversarial losses in latent space, or ad-hoc bilingual dictionaries”. While I can agree that removing auxiliary / adversarial losses or ad-hoc bilingual dictionaries are certainly steps in the right direction, current state-of-the-art unsupervised translation models have already taken these steps and do not rely on these tools. For example, MASS [6] pre-trains using only a denoising autoencoding objective and yet achieves strong results comparable to the ones found in this paper (accounting for model size) and additionally does not rely on clever prompting. To be explicit, the MASS scores in Table 3 are comparable to the results of GPT-3 medium in Table 1, which brings into question if the gains found in this work are truly from the approach or simply from scaling. \n\nIt thus appears that the main contributions of this work which separate it from MASS are 1) using a decoder-only model rather than the typical encoder-decoder architecture, 2) using language modeling as pre-training objective instead of a denoising autoencoding objective and 3) using an English-centric model rather than a model appropriately trained on both languages. Of these, 1) is not particularly novel since GPT-3 has already demonstrated some initial translation performance for the language pairs in question. Previous work [1] has also shown that the initial quality of the translation system for iterative back-translation has little effect on the final model's quality. These two findings make it unsurprising that the authors achieved strong unsupervised translation performance after back-translation, despite the relatively weak model that was used to generate the few-shot examples or to initialize the back-translation process.  Moreover, other contemporary work [7] has already shown that decoder-only models are sufficient to build strong translation models and provided a far more thorough analysis of this architectural bias than this work. \n\nFor 2), it seems obvious that language modeling should result in a better pre-training objective, since the pre-trained model is directly trained to generate natural text. Models trained with masking strategies tend to struggle with zero-shot generation which hinders back-translation. For example, mBART [4] relied on LangID filtering in the initial stages of their back-translation routine to overcome this issue. The authors additionally claim that \"[d]enoising autoencoding with a bilingual encoder can be viewed as a kind of latent bilingual lexicon induction, necessary for producing sufficiently aligned embeddings to kick-start back-translation.\" However, I see no reason why we can't view language modeling on multilingual data in the same light. The initial translation performance from GPT-3 already demonstrates that language model can serve as an acceptable substitute to denoising autoencoding for kick-starting back-translation.  With these thoughts in mind, we cannot consider this work’s successful removal of denoising autoencoding for training unsupervised translation models as a particularly difficult task nor a surprising finding but instead as more of a confirmation of an expected result.\n\nThe English-centric nature of the model (point 3) is an interesting aspect, however it is understudied in the paper. The choice of language pair also hinders this study. For example, French is the second most populous language in GPT-3’s data [8]. Moreover, the original authors of GPT-3 have already demonstrated that GPT-3 can perform English < - - > French translation,  further reducing the impressiveness of the result. It is also interesting to note that this English bias the authors comment on also affects traditional multilingual machine translation models (both supervised [2] and unsupervised [3]) due to the reliance on English-centric parallel data. However, in this setting, this bias is even more pronounced since the English data does not arise from translation data. It would be interesting to explore whether the overabundance of English monolingual data (or perhaps simply scale?) is enough to make up for the scarcity of data available for some of the low-resource languages which appear in GPT-3’s corpus. Similarly, it could be interesting to see if the scripts of the languages considered or their similarity with English is an important feature for this approach to succeed. For example, do we expect GPT-3 to also excel at Hindi < - - > English translation? Or Gujarati < - - > English? Both languages appear in GPT-3’s pre-training corpus, so one could test it. The performance on these languages is crucial because the true value of unsupervised translation lies in low-resource language pairs, and their omission puts into question the validity of this method in realistic settings. The absence of true low-resource language pairs for evaluating unsupervised translation is unacceptable, especially when considering most of the recent literature in unsupervised translation includes such language pairs.\n\nLastly, it would also be interesting to explore whether these models are able to generate more natural text, avoiding the pitfalls of translationese that traditionally affects models trained with supervised data. \n\nQuestions for authors:\n\n1. What is [[TRANSLATE]] in Figure 1? Is this just a stand-in for the prompt written in 4.1?\n\n2. Many approaches in multilingual MT tend to discard the source language and use only the target language when biasing the model for translation. Yet it seems like you used the source language for both prompting GPT-3 and for the zero-shot format in the distillation. Leveraging the source language name could inhibit translation of code-mixed text. Did you experiment using only the target language instead in the prompt or attempted any investigation on whether including the source language name affected the translation performance?\n\n3. Did you attempt this for non-English-centric language pairs?\n\nCitations:\n\n[1] Artetxe et. al. Do all Roads Lead to Rome? Understanding the Role of Initialization in Iterative Back-Translation\n[2] Arivazhagan et. al. Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges\n[3] Garcia et. al. Harnessing Multilinguality in Unsupervised Machine Translation for Rare Languages\n[4] Liu et. al. Multilingual Denoising Pre-training for Neural Machine Translation\n[5] Siddhant et. al. Leveraging Monolingual Data with Self-Supervision for Multilingual Neural Machine Translation\n[6] Song et. al. MASS: Masked Sequence to Sequence Pre-training for Language Generation\n[7] Wang et. al. Language Models are Good Translators\n[8] https://github.com/openai/gpt-3/blob/master/dataset_statistics/languages_by_word_count.csv", "summary_of_the_paper": "This work studies the use of a large English-centric language model (GPT-3) for the purposes of building an unsupervised machine translation system for the language pairs English-French and French-English. Given some monolingual data, the authors use GPT-3 to generate zero-shot translations, which then get used as few-shot examples for a smaller generative model to generate a synthetic translation dataset. This synthetic dataset is used to fine-tune the language model for translation through a back-translation scheme. With this approach, the authors are able to attain a new state-of-the-art result for unsupervised translation on the language pair English-French. ", "summary_of_the_review": "In summary, I don’t believe this work in its current shape is of the adequate quality to be published at ICLR. The biggest flaw is that it is unclear whether anything presented in the work is truly novel or unknown to the machine translation community. The work can be jointly summarized as “language modeling can yield an initial translation model whose performance can boosted tremendously by back-translation”, but no single aspect of this is new, nor is the success of the combination of the various pieces at all surprising to anyone who has been following the unsupervised translation literature over the past two years.\n\nCoupled with the limitations of only studying a single non-English language, no low-resource language pairs, as well no detailed discussion/experiments on the few aspects which distinguish it from other works (e.g. the overwhelming English-centric bias of GPT-3), I cannot recommend acceptance. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "1TptUe25wos", "reviewer": "Reviewer_LnSu", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- This is an interesting way to use large pre-trained language models such as GPT-3 to solve generative tasks like Machine Translation\n- Combines multiple recipes from various sources (e.g. zero shot MT, few shot MT, back-translation) and shows how to apply them to a standard left-to-right pre-trained LM to achieve impressive translation results\n\nWeaknesses:\n- While the results are interesting and illustrate the flexibility of a standard transformer language model, the key ingredient of this approach relies on having a language model that has seen a very large and diverse dataset that very likely includes some translation examples (for zero-shot translation to work). Using such a language model to boot-strap self-training (called distillation in this paper) followed by back-translation is interesting, but should be grounded in techniques that have access to the same kind of large and diverse dataset. One clear example is bi-text mining (e.g. see https://arxiv.org/abs/1911.04944) where one can simply mine a bunch of parallel text from a similar kind of dataset and then train a standard encoder-decoder machine translation model (with self-training / backtranslation if so desired). It is not clear what the advantage of the proposed method over such an approach would be except as a curious application of existing pre-trained models. Could the authors elaborate on this point and, if they agree, compare to the methods described above.\n- Are there other works on decoder-only machine translation? (i.e. ones that do not have a decoder and rely on only the LM part but trained on bi-text data). If so, this would be great to discuss in related work. ", "summary_of_the_paper": "This paper proposes to solve the machine translation task without any bi-text data and using only a pre-trained generative language model to bootstrap the process. The entire method relies on using powerful pre-trained language models that have been trained on a large amount of data to extract initial boot-strap examples, and then fine-tune the model to solve the machine translation task using specially formatted prompts. The same language model is used to translate in both directions, and back-translation is used to improve results.", "summary_of_the_review": "While this work is an interesting way to explore the power of large pre-trained language models, I am not sure it is compared properly to methods (also starting with no parallel data) that may have access to the same kind of datasets used to train these language models. I am also not sure how this work compares to supervised and/or semi-supervised techniques using the same model sizes, or what is the state of the art for using transformer LMs with prompting for NMT (where parallel data is available). Discussion of these points would be enlightening and may lead to updating my recommendation.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an alternative method to improve UNMT by using only a pre-trained generative language model to bootstrap the process. In all, the reviewers think the proposed method is reasonable. 

However, the empirical part is not convincing.  Most reviewers think that evaluating the method on one language pair (En-Fr) is not enough to show the effect of the proposed method. In addition, some reviewers argue the clarity of this paper.

In all, I think the proposed method is meaningful. However, the current version is not ready to be published in this ICLR. I hope the authors can improve their paper according to the reviews.
Decision: Reject