The Impact of Syntactic and Semantic Proximity on Machine Translation with Back-Translation
Authors: Nicolas Guerin
Emmanuel Chemla
Shane Steinert-Threlkeld
Venue: Accepted by TMLR ()

Abstract: Unsupervised on-the-fly back-translation, in conjunction with multilingual pretraining, is the dominant method for unsupervised neural machine translation. Theoretically, however, the method should not work in general.  We therefore conduct controlled experiments with artificial languages to determine what properties of languages make back-translation an effective training method, covering lexical, syntactic, and semantic properties. We find, contrary to popular belief, that (i)~parallel word frequency distributions, (ii)~partially shared vocabulary, and (iii)~similar syntactic structure across languages are not sufficient to explain the success of back-translation.  We show however that even crude semantic signal (similar lexical fields across languages) does improve alignment of two languages through back-translation. We conjecture that rich semantic dependencies, parallel across languages, are at the root of the success of unsupervised methods based on back-translation. Overall, the success of unsupervised machine translation was far from being analytically guaranteed. Instead, it is another proof that languages of the world share deep similarities, and we hope to show how to identify which of these similarities can serve the development of unsupervised, cross-linguistic tools.

Published in Transactions on Machine Learning Research (09/2024)
The Impact of Syntactic and Semantic Proximity on Ma- chine Translation with Back- Translation
Nicolas Guerin
nicolas, nicolas.p.gueree p-guerin@gmail.co com
Laboratoire de Sciences Cognitives et Paycholinguistique
École Normale Supérieure
PSL University
Emmanuel Chemla
Laboratoire de Sciences Cognitives et Paycholinguistique
École Normale Supérieure
PSL Universily
Shane Steinert-Threlle
shanest edu
Department of Linguistics University of Washington
Reviewed on OpenReview: https: /t openreview. net/ forum? id= 6DALPAP
Abstract
Unsupervised on-the-fly back-translation, in conjunction with multilingual pretraining, is the dominant method for unsupervised neural machine translation. Theoretically, however, the method should not work in general. We therefore conduct controlled experiments with artificial languages to determine what properties of languages make back-translation effective training method, covering lexical, syntactic, and semantic properties. We find, contrary to popular belief, that (i) parallel word froquency distributions, (ii) partially shared vocabulary, and (iii) similar syntactic structure across languages are not sulficient to explain the success of back-translation, We show however that even crude semantic signal (similar lexical fields across languages) does improve aligmment of two languages through back- translation. We conjecture that rich semantic dependencies, parallel across languages, at the root of the success of unsupervised methods based on back-translation. Overall, the success of unsupervised machine translation was far from being analytically guaranteed. Instead, it is another proof that languages of the world share deep similarities, and we hope to show how to identify which of these similarities can serve the development of unsupervised, cross-linguistic tools.
1 Machine translation and the role of back-translation to eliminate supervision
Supervised training for neural machine translation (NMT) requires a vast amount of parallel data (Susskevrr eet al. 2014 Wu et al. 2016: Vaswani et al.) 2017). Creating the necessary datasets aligned across languages is a complex, oncrous and somctimes impossible task. In this context, unsupervisod back-translation, and in particular on-the-fly back-translation, becomes highly valuable,
allows unsupervised training from
independent monolingual corpora (Sennrich al. 2016 Lample et al. 2018a Guzmán et al. 2019 Haddow
ct al. 2022
Lack-translatin works by using a translation model from one language (L1) to another (L2) to synthetically generate data in the following way: a given text x in L1 is passed in and a hypothetical translation ğ in L2 is gencrated. This pair then treated if it were 'gold' translation (j, x) train an Lz-to-I, system. For iterative and on-the-fly back-translation, the whole process is repeated in the other direction, and iterated, that the models improve they are generating the data. One can think of this process, in essence,

---

Published in Transactions on Machine Learning Research (09/2024)
that of an auto-encoder of one of the languages, with the hidden space being the other language: the model is trained to translate a sentence from language into language 2 and back into language 1, success being attained if the original and final sentence from language 1 match.
Back- traksraanllttion was originally used as data augmentation, ie. part of the training was also done on a parallel corpus (Sennrich et al. (016)) In the more recent literature some models are fully trained via back-translation in an unsupervised way on top of other unsupervised objectives such as denoising auto-encoding or language modeling, and obtain decent BLEU scores, with significant improvements for low-resource languages [Lapple & Conneau 2019 Liu et al. 2020 i...). Here, then, success is obtained without any training signal as to how the two languages are aligned. This empirical success, however, is puzzling: without explicit information about how to align the two languages, how does the method succeed? To our knowledge, there is no systematic understanding as to why this type of training succeeds.
In this paper, we first discuss why the success of back-translation is puzzling in general (Soction 3 We conjecture, as many do, that it works because, despite surface differences, natural languages have rich and similar structures that can promote alignment (Lample & Conneau 2019 Wu & Dredze 2019 Conneau et al. 2020b). We then describe our shared exerpimental setup (Section which uses artificial languages to systematically manipulate various similarities and differences between languages. We then report a series of experiments which analyze which properties drive the observed empirical success of back-translation Our systematic experiments suggest that shared syntactic and very simple semantic structure does not suffice for back-translation to yield quality unsupervised NMT (UNMT) systems. Hence, shared complez semantic dependencies appears to be crucial.
(Sections 5 10
Our contributions are: (i) an analysis of why back-translation should not work in general, (ii) the use of artificial languages to conduct systematic experiments on factors driving its empirical success, and (iii) the elimination of reasons (often claimed to be behind its success) such as close syntactic structure, word fre- quency, anchor-points or even a semantic structure of lexical fields.
2
Related work
Prior to unsupervised translation, several works focused on bilingual dictionary induction but using little
no parallel data. Míkolov et al. (2013) does so by relying on distributed word representation of large
monolingual corpora as well as small supervised corpus. Klementicv et al. (2012) Irvine & Callison-Burch
(2016) develop phrase-based statistical model for phrasal and lexical translation. More recently, Lample
et al.
2018bb allows dictionary inference without any parallel data.
Bojar & Tamchyna (2011) used reverse self-training to train a model in a statistical machine translation (SMT) framework. The idea is to use a small parallel corpus to train a machine translation (MT) system from target to source and use it to translate large monolingual target-side corpus to create synthetic pairs to train source to target SMT model. This is the root of Back-translation (BT) as data augmentation method. This is applied to NMT by Semnrich et al. (2016) who train two models iteratively, source to target and target to source. The idea that translation from a language to another has a dual task, which is the translation in the reverse order, that can be learned jointly to improve efficiency is largely used in several works (Xia et al. 2017: Cheng et al.] 2016 He et al., 2016 Wang et al. 2019 Hoang et al. 2018 Niu et al. 2018). For example, Cotterell& & Kreutzer (2018) use the formalism of a wake-sleep algorithm to propose an Iterative Back- Translation method, i.c. the synthetic data is regencrated after cach model training. However, in all these works training is still partially supervised and back-translation acts as data augmentation.
Artetxe et al. (2018) introduce fully Unsupervised Neural Machine Translation (UNMT) method. It uses on-tho-fly BT (OTF-BT), mcaning that synthetic sentences are generated as the model trains. They also add a denoising autoencoding objective. This paper lays the foundations for the following BT-based architectures and models. For example, Lample et al. (2018a) use the same method and leverage the intuition that the same sentence, regardless of the language, will be mapped onto the same embedding vector by encoders, and that to translate the sentence it is then sufficient to decode it into the desired language. They enforce this behavior using an additional adversarial training objective. Other works remove this adversarial objective
2

---

Published in Transactions on Machine Learning Research (09/2024)
while maintaining performance (Lample et al. 2018c Yang et al. 2018 suggesting that this embedding
space overlap emerges naturally.
Following on from this line of work, several papers formulate the idea that the quality of the cross-lingual embeddings used to initialize BT are key and therefore add a language modeling objective as pre-training
[Tample et al.] 20188a Eduuno et al.] 2019). This method works very well and has become the standard approach in UNMT. For example, [Liu al. (2020) introduces mBART for UNMT while Zan et al. (2022)
studies in depth its benefits. Numerous papers propose modifications to BT to improve its performance,
particularly in low-resource setup (Kumari et al. 2021 Kim et al.) 2019 Pourdamghani et al., 2019 Dou
ct al. 2020; Song et al.) 2019). In particular, Filtered BT (Kaarr & Bhattacharyyal [2020] Imankulova et al.
2017 filters synthetic sentences based on their quality, using a round-trip sentence similarity metrics, while Tagged BT (Caswell et al., 2019) prepends a tag in front of syuthetic sentences to indicate to the model which are from the parallel corpus and which are not.
Many studies examine the reasons for the success of BT for UNMT (Edunov el al. 2018 Poncelas al. 2018: Kim et al.) 2019 Edunov et al.) 2020 Conneau et al. 2020b). In particular, some focus the properties of languages, pursuing the thought that BT and multilingual models work due to the similar structures of natural languages. K et al. (2019) studies the impact of several factors on the cross-lingual abilities of mBERT, including the linguistic properties of languages, such as loken overlap between vocabularies, or word order. They show that the former has very little effect, but that the latter is fundamental. [Dftter & Schütze (2020) takes up the same kind of work. Finally, Kim et al. (2020) shows that UNMT performance drops drastically when languages or domains are very different. It also shows that BT cannot recover from bad initialization.
Our work continues this line of research, but differs in that we use artificial languages to maintain perfect control over our experiments and to avoid confounding factors as much as possible. This allows to investigate precise syntactic and semantics effects on OTF-BT. We also use these artificial languages to demonstrate the inaccuracy of the usual assumptions concerning the success of OTF-BT (e.g. word frequencies, anchor points, synlax, etc.). To the best of our knowledge, this has not been done in previous works. It is hoped that this work will lead to greater intuition about BT, enabling researchers to refine the method and produce better results in the future.
Several other works also use artificial languages on related tasks (Arora et al. 2016 White & Cotterell 2021 Chiang & yi Lee 2021), but none apply their methodology to UNMT. Finally, it should be noted that the model used and studied in this work (Lample et al. 2018c) is no longer state of the art, but uses OTF-BT almost exclusively, unlike more recent models which rely heavily on massive pre-training, making it extremely relevant our work.
3 Translaaann is necessary, but not sufficient in general
A fully unsupervised training with ponents of a translation system, as seen in Figure [ Tt includes encoders e, from language L42 into a hidden space, and decoders de from this hidden space into L.g. These encoders and decoders could be soparate models or the same multilingual model conditioned on language ID token (Conneau al.) 2020a Liu et al. 2020). The composition of an encoder e, and decoder dj from different languages provides a translation function Tyj-
works as follows. The model consists of classical com-
These components are first trained through a denoising auto-encoding, monolingual regime, whereby d, with g a noise function, should lead to the identity for each language. Then suppose y is an element of the language Lg. We can translate y into L, as ă dj e e2(y) := T2 (v). This creates a synthetic pair (ø. y) L2 x Lz. This pair, which was created by the model itself, is now used as a supervised datum for translation. That is, we check that the translation (now backwards) from ã in Ta to Lag is coherent: ğ dzoes (ō) Ti2(a)
3

---

Published in Transactions on Machine Learning Research (09/2024)
L
Hidden space
L
2
T
12
d
ej
d
2
21
Figure The components of a classical translation pipeline. Translation functions T12 and T21 are obtained from the composition of encoders and decoders from different languages.
should lead back to y. With a loss function C (e.g., cross entropy), this process can be summarized as:
min
Σ
Cly, de oes di e2((1)
Begidh
WEL2
Σ C(x, di oe
(1)
DE
Σ (E(1 sEL;
Σ
fiz
where Ber,d, are the parameters of the encoders/ The gray components the first lines reflect 'rozen' parts of the pipeline: in practice we sample from those d, and then pass those generations forward, previously described. Hence the formula is slight simplification. In practice, the denoising autoencoding objective (or similar; see third line in ① often happens before the back-translation objective is used; we have included them as one loss because the present arguments do not depend on this choice.
With this framework in place, we can now explain why back-translation is a necessary objective, but not a sufficient one. The back-translation objective ( could fully met, without translation being accurate at all. We illustrate this visually in Figure 2 with two extreme cases.
(a) Accurate translation
(b) Outside of language translation
(c) Shuffled translation
Figure 2: Schematic representation three cases in which the back- translation objective [ is fully met: (a) with an accurate translation, (b) with translation missing the target language entirely, but being back on the original language appropriately, (c) with translation that bijectively shuffles the target language, and space, and write for Tip and t- for T21.
translation that unshuffles it back in place. We ignore here the hidden encoding

---

Published in Transactions on Machine Learning Research (09/2024)
First, consider Figure 2b It could be that the "translation" T12 from L1 to L2 projects L1 into space X outside of the actual 1/2 (so translation will be completely off), but that the back-translation would project this space X back into L1 appropriately.
Second, consider Figure 2c It could be that the space overlaps well with L2, but that is a shuffled version of it, in such a way that translation will be completely off again. Yet, the back-translation objective will be fully met, long as the translation un-shuffles back in place onto L1.
More generally, any combination of e, and d2 such that T12 T21 verifies the back-translation component of [ this essentially requires that the translation functions Ti be inverse bijections, but such bijections do not need to correspond to accurate translation functions. The auto-encoding component of does not sulfice priori to overcome this problem.
These simple counter-cxamples show how back-translation could fail. In practice, however, back-translation has achieved significant empirical success (Lample et al., 2018a Lample & Conncau, 2019 Liu ct al. 2020: Song al. 2019 i.a.). Back-translation must. work because natural languages look like another, i.e. struc- ture in the data makes it so that the proper alignments between L1 and Lg easier to discover than the external ones or shuffled ones. But what structure in the data? The intuition often put forward is that tressatannl works because, e.g., frequent words are better mapped onto frequent words, or because systematic word orders will anchor the mappings. In the rest this paper we investigate such hypotheses in more detail.
4 Experimental Setup
We investigate which language properties make back-translation work or fail. To do this, we tested back- translation between artificial languages for which we can freely manipulate lexical, syntactic and semantic properties.
4.1 The Context Free Languages
The grammars of our artificial languages vary somewhat similarly to what is assumed for natural languages. We used the simple Context Free Grammars (CFG) introduced by White & Cotterell (2021), which are parametrized by 'switches': the order in which constituents surface. Table [ presents the rules that are switchable in these grammars. We will denote a grammar by a sequence of six binary values, corresponding to the switches. These syntax rules are designed to reflect real rules that differentiate many natural languages. For example, Japanese corresponds to the combination 000000, English to 011101 and Spanish 011111.
Rules for each switch value
Switch
0
S
NP VP
VP NP
VP
VP
NP VP
VP VP NP
Comp
SComp
S Comp
Scomp
- Comp S
NP
PP NP
NP NP PP
PP
PP
- NP Prep
PP Prep NP
NP
NP - + Adj NP
NP NP Adj
Rel
NP - VP Rel Noun
NP Noun Rel VP
Table 1: Rules that are switchable in the grammar. Table from White & Cotterell (2021).
At the lexical level, 1,374 words were created, with plausible English morphology, and distributed in several
Parts-of-Speech (POS), based on the work of Kharkwal 2014h. We used 1,374 words to obtain a replication of White & Cotterell [2021 Picking a set of switches (i.e. a grammar) and a set of such created words thus
I Many points rely fallure of OTF-BT (see experiments). hence we did not try with larger vocabulary, which would be harder task and could only confirm failure.
5

---

Published in Transactions on Machine Learning Research (09/2024)
defines one artificial language. A full translation between two such simple languages can be reconstructed with a one-to-one mapping at the lexical level, and knowledge of the relevant switches.
We provide some examples of sentences generated by such artificial languages in Table 2 illustrating the impact of the vocabulary changes and the grammatical switches.
Language
Examples
000000 POS
NounS Subj VerbCompPresS.
000000
(Lexicon 0)
burse sub lurchifies. lurchifies burse sub. lurchifies burse sub. swopceer bus rheleates.
100000 (Lexicon 0)
100010 (Lexicon 0)
000000 Lexicon 1)
000000
POS
VerbPastP Adj NounP Subj. rolveda prask autoners sub. prask autoners sub rolveda. autoners sub prask rolveda. knyfeateda wourk krarfteers bus.
000000 (Lexicon 0)
100000 (Lexicon 0)
100010 (Lexicon 0)
000000 (Lexicon 1)
Table 2: Examples of sentences generated by our artificial languages. The POS row gives the underlying structure of the sentence in each group (colors in the following rows trace these Part-of-Speech). When relevant, constituents swapped compared to the grammar from the row above underlined. The grammars and switches are described in Table
4.2 Training
Model We used the exact model introduced by Lample et al. (2018c) for NMT. The architecture is based on Transformer encoder layers and 4 Transformer decoder layers. The last encoder layers and the first 3 decoder layers are shared across languages. The embedding size is 512 and the vocabulary, of size 000, shared and built using Byte Pair Encoding. We this model because of its perfommeformance (state of the art at the time of publication) and because it allows us to isolate the study of OTF-BT.
Data For training purposes, we generated two sets of 100,000 sentence structures All unsupervised training sets are made of (i) one these sets of sentence structures for the 100,000 sentences in one language (using the language specific grammar and lexicon), and (ii) the other set to create 100,000 sentences in the other language. Hence, the data are neither labeled for supervision, nor are they even unlabelled translations from one another in principle. The test and validation sets are each composed of 10,000 parallel sentence pairs. These are generated in one language, and then transformed into the second language using the known grammar switches and lexical translations.
Procedure At the start of training, a FastTText algorithm applied to both languages simultaneously to initialize the gross-lingua embeddings (Bojanowski al. 2017). We then train the translation model for 40 epochs, with batch size of 16 and an Adam optimizer with a 10-4 learning rate. Those parameessssssesss were chosen based Lample et al. (2018c) and on our computational capabilities. We trained on Tesla M10 with 8GB memory for roughly one day.
Objectives The overall training objective is described in ① First the model docs a denoising auto- encoding (DAE) step, which can be seen as a LM step that constrains the model to output in the correct language, and updates its parameters, then it does an on-the-fly tack-translation step in both directions and updates its parameters again. The denoising function g used combination of word substitution, local shuffling and masking. We note that DAE is here interspersed with BT, not used separate pretraining objective.
Using up million sentences yielded improvement in all instances that we tried, report results for 100,000 sentences to resources.

---

Published in Transactions on Machine Learning Research (09/2024)
Evaluation In each of the next experiments the results obtained are those computed the test set by the model having obtained the best BLEU score on the validation set. Note that the BLEU score calculated on our artificial languages is not to be compared with BLEU scores obtained on natural languages with much larger vocabularies and many turns of phrase that count as correct translations. However, as a point of reference, the Lample et al. (2018c) model we use obtained on WMT monolingual News Crawl a BLEC score of 24.65 on English-French pairs, and of 19.10 on English-German pairs which was UNMT state of the art.
4.3 Experiments
In the remainder of this paper, we present six experiments, each of which will answer several research questions, some of which will emerge from the results.
The first experiment $ is very simple and acts as sanity check, the idea being ensure that the model is indeed capable of learning to translate these artificial languages into themselves.
The sccond experiment @ is to measure the impact of grammar on translation performance, keeping the lexicon equal. In this way, the translation function is only concerned with the change in grammar from one language to another.
The third experiment g adds the difficulty of a new lexicon. We show that this difficulty completely undermines the success the machine translation system. This, of course, is different from the situation with real-world languages.
The following three experiments then seek to understand what additional signal is present in natural lan- guages and is so crucial to the success of OTF-BT. Experiment four $ adds anchor words and identical word frequency. Experiment five $ shows that a weak supervised signal, such as a bilingual dictionary or a small supervised dataset, restores the model's performance. Finally, experiment six 100 adds semantic information by way of lexical fields.
4.4 Metrics
We use the BLEU score as a metric of accurate translation (Papineni et al. 2002 and in particular the Moses implementation as in Lample et al.) 2018c Some more specific metrics will be introduced for the particular of some experiments, but BLEU will serve as a standard reference many places. There are various limitations of BLEU: cannot handle paraphrases and synonyms, does not take meaning into account directly, it is invariant by n-grams permutations, has a low correlation with human judgment, etc. The limitations concerning paraphrascs and synonyms do not apply to sctup, which docs not have paraphrases nor synonyms. Similarly, meaning is not relevant in our setup. Overall, BLEU ignores certain aspects and may lead to overestimations of the quality of translation, To mitigate this, we computed BLEU scores for n-grams ranging from 1 to 4. This helps capture syntactic structure (order constraints) and this mitigates the risk bigrams permutations insensitivity, as illustrated in Callison-Burte et al. (2006). In any event, however, our arguments rely mainly on the observation of low BLEU scores, hence overestimation is mild risk.
Alternatives to BLEU include two families: the automatic language agnostic metrics like BLEU Papineni let al. 2002). METFOR (Banerjee & Lavie 2005) or Word Error Rate (WER), and the LLMebased metrics like BERTScore (Zhang et al. 2019). MetricX (Juraska et al. 2023). , Comet RRi et al. (0020). Unite Wan et al. 2022), BLEURT (Sellam el al., 2020), etc. The second family of measures is not applicable to our artificial languages, Among the first family, WER, known to be worse than BLEU, METEOR remains
imperfect (sce Saadany & Orasan
2021 and BLEU therefore seems like the right choice.

---

Published in Transactions on Machine Learning Research (09/2024)
5 Experiment 1: Identical L, and L2 languages
Our first experiment demonstrates that our hyper-parameter and training pipeline work in principle, by using BT to translate between two identical languages. Note however that, in principle, even for identical languages, back- translation is not a sufficient objective. We randomly selected 8 grammars among the 64 possible ones, and picked one lexicon, to form 8 artificial languages. For each such language L1, we trained our model to learn translation between Lı and a similarly defined language L2. Sampling of training sets was done independently for the two identical languages, hence the two monolingual corpora not aligned.
Table 3 reports BLEU scores on the test set in this paradigm. Training was succossful: all BLEU scores are above 97.
Grammars
BLEU
000000=000000
98.76
011101+:011101
98.70
011111+011111
99.08
000001
98.13
100000+100000
98.83
000101==000101
97.76
111111@111111
97.77
111110=111110
97.30
Table 3: BLEU scores obtained on the test set for within grammar and within vocabulary training. Those scores are the average between the BLEU score obtained when evaluating source target and target source respectively.
White & Cotterell 2 found that the choice of the grammar had an impact on language modeling success for the Transformer architecture (but not for LSTM language models). Here as well, we found that the choice of grammar had an effect on the result, and in the same way: our within-language BLEU scores varied and were correlated with the perplexity the same languages obtained in their language modeling task (R² = 0.62). In short, some word orders robustly lead to better performance both for language modeling and, now, for within-land lathin-language translation.
6 Experiment 2: Effect of Grammar
If similar structure is the key for back trccstrranslttin to work, then translation between two languages should be harder their grammar are more different (even we stay within the variation observed between actual languages). We can operationalize this hypothesis in terms of the Hamming distance (number of different switches) between the grammars generating two languages: we expect BLEU score to decrease as Hamming distance increases. In these tests, we keep the vocabulary constant across languages to focus the effect of grammar.
To test this hypothesis, we used the eight random grammars of $5 and trained a translation system via back-translation for each pair, resulting in 64 BLEU scores. The complete results are in Table
First, we found a correlation between the obtained BLEU scores and the Hamming distance (R² 0.35 with a coefficient of -8.35, p 0.001). In other words, back-translation does become less performant as the grammars differ more.
Second, we used as predictors not the raw Hamming distance, but individual variables S; corresponding to each switch being different or not in the translation pair at stake, thus fitting a model of the form BLEU ~ Eim3,, B,S,, We obtain a strong fit (R² = 0.94), and a significant effect at p < 0.001 for the coefficients corresponding to switch 5, (3₁ = -5.044, 54 (64 = 6.36) and Sg = -11.04.. This suggests that not all switches are created equal: some have more dramatic effects than others on mack-translation performance.
8

---

Published in Transactions on Machine Learning Research (09/2024)
000000
011101
011111
000001
100000
000101
111111
111110
000000
98.8
64.3
67.8
73.5
46.8
68.3
35.5
36.5
011101
64.7
98.7
98.6
87.4
36.8
96.5
38.7
37.9
011111
67.8
98.7
99.1
86.2
36.1
93.9
39.4
40.4
000001
73.9
85.6
83.3
98.1
39.9
91.8
29.3
34.5
100000
43.9
33.8
33.7
33.4
98.8
31.3
63.
76.0
000101
69.4
90.3
96.7
91.8
37 6
97.8
34.1
40.5
111111
41.6
40.8
41.9
36.1
67.1
40.5
97.8
86.6
111110
40.6
36.4
38.5
39.2
78.3
46.1
86.0
97.3
Table 4: BLEU scores for languages with different grammars but the same lexicon.
Because of this, we more systematically evaluated the effect of each individual switch. To do this, we use the 000000 grammar as the source language and vary the target language so that one switch only is activated each time: 100000, 010000, 000001. Conversely, we used 111111 as the source language, deactivating each switch in turn for the target language: 011111, 101111, .... 111110. The results are in Table
Grammars
BLEU
id baseline
Grammars
BLEU
id baseline
10000000000000
45.36
51.04
(-0.11)
1111110011111
41.25
48.23
(-0.17)
00000000010000
94.81
42.37
(1.24)
111111@101111
87.94
43.27
(1.03)
000000=000000
95.93
70.33
(0.36)
111111=110111
97.86
69.57
(0.41)
000000=000100
92.88
94.63
(-0.02)
111111=111011
92.59
93.95
(-0.01)
000000=0000010
98.69
82.60
(0.20)
111111e111101
97.56
79.46
(0.23)
000000>0000001
73.17
73.17
(0.00)
1111111111110
86.40
69.66
(0.24)
Table 5: The BLEU scores show that different switches have different impacts on the translation performance, with source language 000000 (left) and 111111 (right). The second column shows the baseline BLEU score that would be obtained by translation system that would just copy the initial sentence and, in parentheses, the relative distance of the learned translation to this baseline: BLEU-baselin
baseline
Different switches show different impact on the translation performance, in way mirroring the regression coefficients: switch 1 causes the largest drop in performance, followed by switch and then switch 4. Other switches, like the fifth one, have little impact on BLEU.
An intuition behind the different impacts of the switches could be as follows: the first switch governs change that occurs near the root of the parse tree while the fifth switch concerns node closer to the leaves. Thus, more words move, and they move further away in the first case than in the second. Because of this, a model that has learned the identity mapping (and therefore has not learned the translation) would get a higher BLEU score with the fifth switch than with the first.
To isolate this elfect and determine whether the drops in BLEU identity translation, computed the BLEU score of system that would just copy the source sentence is (and the relative distance between the BLEU score obtained by our model and this bascline). The results are shown on the last two columns of Table 5
result of bias towards learning an
Different (relative) effects of each switch on the translation are found again, suggesting a more subtle effect than the intuition mentioned above. In sum, grammar changes that make words move further away have more impact on the translation performance, and this is not because the identity is learned instead of proper translation.
9

---

Published in Transactions on Machine Learning Research (09/2024)
7 Experiment 3: Effect of the Lexicon
Until now, the two languages had the same lexicon but different grammars. In this section we focus on the opposite scenario, ic. translation between languages with an identical grammar, but different lexica. We thus built second, parallel lexicon of 1,374 new fictitious words with English-like morphology using the same methods before (see 4.11
We followed the same training procedure on five out of the eight grammars from In this context, we found that back-translation systematically failed to produce successful translation systems: the best BLEU score was below 7, as reported in Table Note that the round-trip BLEU score was above 70 in all trainings. This shows that it's not the back-translation objective that has failed to be optimized, but rather that it's not sufficient ensure correct translation.
Grammars
BLEU
POS BLEU
000000 + 000000
2.46
69.3
011101 <> 011101
2.81
62.4
000001 + 000001
4.31
67.3
100000 <> 100000
5.68
76.2
111111 + 111111
6.63
58.1
Table 6: BLEU scores obtained on the test set for within grammars training but with different lexica. It is clear that the training did not work as the highest BLEU score is below 7, demonstrating that back-translation is not guaranteed to succeed. The POS BLEU colomn report the average BLEU score obtained on the Part-of-Speech only. These high scores show that the failure lies in the vocabularies mapping.
the fly
To identify the source of these errors, we computed BLEU scores for part-of-speech (POS) values instoad of exact tokens. In this setting, all BLEU scores were above 60 (see Table 6). In more detail, around a third of the sentences in the test are syntactically correctly translated. The average length of correct sentences is 6.2, compared with 11.0 for the whole test set. Correct sentences is not surprising simply if shorter length means fewer opportunities of error. This relatively good syntactic performance can be partly explained by the DAE objective, which constrains the decoder to outpul sentences from the grammar. However, it is also clear that the model does not ignore imput when doing translation.
therefore shorter average, which
To further document the failure at the vocabulary level, we focussed on an analysis of sentences whose POS sequence has been correctly translated; there it is possible to do word-by-word evaluation. For any word w, from the source language, we looked at the distribution of its translations by the model across its different occurrences in this restricted test set. The perfect model would always translate w, by its correct translation, and would have entropy of O. By comparison, model choosing the translation at random (although in the correct POS) would have entropy of log, of the size of the relevant POS set. Taking singular names as an example, the average entropy of the model is 0.05, whereas a random model would have an entropy of 7.3. The results for the other POS are qualitatively identical (see Table 7 showing that the model did pick single translation per word, albeit not the correct one. This points towards the shuffled translation sccnario illustrated in Figure 2c for the vocabulary.
8 Experiment 4: Unsupervised Vocabulary Signals
tacck-transltiin alone was unable to learn the mapping between fully distinct vocabularies. Here we model two phenomena that could help overcome this difficulty in natural settings: the presence of shared lexical items across languages, or "anchor points" (Conneau al., 2020b), and similar word frequencies across languages (Piantadosi (2014), among many others).
10

---

Published in Transactions on Machine Learning Research (09/2024)
POS
Entropy
(random)
Noun
0.05
(7.3)
Adjective
0.05
(5.4)
Transitive Verb
0.06
(6.4)
Intransitive Verb
0,05
(6.8)
Verb Complementizer
0.03
(1.5)
Table 7: Entropy for each POS (in parentheses, entropy value for a random translation within the correct
POS). For each POS, the results have been averaged over the different morphological variants. For example, the Noun row groups together Singular and Plural Nouns.
Manipulation
Exp.
BLEU score
Lexical change
3,
2.3
Anchor Points
4a,-
23.0 (+20.6)
Frequency
4b, $8.2
10.1 (+ 7.6)
(ii) Ne = 2
6a,
$10.1
9.2 (+ 6.8)
(iv) Nc = 2
6b,
10.2
10.7 (+ 8.3)
N = 10
6c,
10.3
11.9 (+ 9.5)
augmented with other unsupervised signals. We start from the previous experiment with lexical change, then show results when we supplement the situation with (i) anchor points, (ii) (intra-POS) word frequencies (for the 000000 grammar and two different lexica),
Table 8: BLEU scores obtained from
(iii) No = 2 equally balanced disjoint lexical fields, (iv) Ne = 2 unbalanced disjoint lexical fields, (v) N 10 disjoint lexical fields. In hesentheses: the gain of BLEU score compared to Exp 3.
8.1 Experiment 4a: Anchor Points
In natural language corpora, & number words are identical between languages, in particular in written form. This happens because the languages have a common root, because some words have been borrowed across languages, because of identical proper nouns, or because the languages may use similar numeral systems. These identical words between two languages may models to fit. the whole mapping between two lexica. This could help prevent the rotation problem described in $3 and the disjoint lexicon problem of 7
anchor points that allow translation
To test this hypothesis, we ran trakslaranlttion on two languages with the 000000 grammar and lexicons sharing 30% of their words. We obtained BLFFU score of 23.04. Using the word-by- boorr comparison method from 7 (for syntactically correct translations), we show that 92% of the words with non-zero precision and recall are within the common words. This suggests, mirroring similar findings on multilingual language modeling due to Conneau et al. (2020b), that common words do not serve good anchors that can lead the empirical success of back-translation on the entire vocabulary.
8.2 Experiment 4b: Frequency alignment
In the experiments described thus far, all words within given POS have equal frequeney. In natural languages, however, words vary in frequency, and presumably in similar ways across languages, following power law. This information could help translation model learn the mapping between the two vocabularies.
We hence manipulated our corpora so that the probability of appearance of words within each POS follows a power law: P(w,) x -k for k = 1.1, as they roughly do in natural languages, where w, is the n-th most frequent word.
With the 000000 grammar and two different lexica following these parallel Zipfian distributions, we obtained BLEU score of 10.08, slight. increase compared to 7 Using again the word-by-word translation analysis,
11

---

Published in Transactions on Machine Learning Research (09/2024)
observed no regularity in which words were appropriately translated: it is not the case that the most most//eaatt frequent words are better translated, it is not the case that the model outputs the most frequent words. This is illustrated in Figure 3 with singular nouns: most words have very low accuracy and recall, and those that do not are spread across the x-dimension (rank).
Noun_S
shreummer
Romeician
maiy Mall
102
1.0
cooldeist
Borldeician
Count
Score
110
0.5
100
0.0
100
101 Training Corpus Ranks
102
Figure 3: The red (resp. blue) line represents the accuracy (resp. recall) the lexicon word translation. These are ranked by their frequency the training corpus. The dashed lines represent word counts in the testset (light blue) and in the model translation (grey). It can be seen that the model tracks actual frequencies well. Then, very few words have non-zero score, which is consistent with very low BLEU score. What's more surprising is that the most frequent words are not translated any better.
Thus, in the same way that anchor points did not help much (88.1 rich and realistic word frequencies cannot be the primary driver of the empirical success of back-translation. This is disappointing, as simple mapping of lexica based on word frequency in the corpus would yield a very good translation.
Finally, the POS BLEU score for this experiment is 57, with only 22% sentences being correctly translated syntactically (compared 33% before). In the case where intra-POS frequencies were uniform, word fre- quencies were therefore fixed by the POS frequency, itself fixed by the grammar. This clustering of word frequencies based solely syntax may explain the better syntactic score of previous experiments. In other words, rich word frequencies do not improve vocabulary translation, and it impacts translations at the level of syntax.
9 Experiment 5: Adding Supervision
One possible explanation for the failures above could be the rotation effect described in $ Some supervisod training could help avoid that risk, by anchoring at least. some translations into ground-truth. We test two types of supervised examples: some aligned sentence translations (as when back-translation is used not for fully unsupervised translation but for data augmentation). and the injection of a bilingual dictionary.
9.1 Small Aligned Dataset
Here, we use back-translation as data augmentation, as it was originally used, by training the model supervised way on 1,000 sentences, on top of the training with back-translation on the same 100,000 sentences from before. The training process is as follows: first step of denoising auto-encoding, then
12

---

Published in Transactions on Machine Learning Research (09/2024)
supervised step with batch from the aligned dataset and finally back-translation step with batch from the unaligned dataset. At each of these steps, the model updates its parameters.
Results presented in Table that performance greatly improves, with BLEU scores often above 80 and all significantly above the results without supervision in Table D even though the grammars are different here. As a baseline, training only with the supervised examples not sufficient (for example for 000000~0000000 the BLEU score with only 1, 5,00 parallel sentences is 31.49, not 96.03).
Grammars
aligned sentences
bilingual dictionary
000000=000000
96.0
72.1
000000~001101
67.4
59.8
000000=100000
81.5
42.7
000000<>111111
84.8
28.3
Table 9: BLEU scores for different language pairs, when augmented with supervised signal. The column aligned sentences corresponds to the addition of supervised training parallel dataset of 1,000 sentences (out of 100,000). The column bilingual dictionary to the addition of the supervised training on all word pairs (passed in as one-word sentences). Both supervisions lead to better scores than without any supervision (see Table g where top BLEU score is 23).
9.2 Bilingual Dictionary
Previous results suggest that failure arises at the lexical level, so here we add supervised training on all pairs of aligned words, i.c. the complete bilingual dictionary. Results improve compared to previous results: BLEU scores are now between 28 and 72 (see Table when they were previously below 7 (see Table 6
Overall then, back-translation works drastically more efficiently when complemented with some supervised signal, either the sentence level (previous subsection) or at the lexical level (this subsection).
10 Experiment 6: Towards Semantics via Lexical Fields
One difference between our artificial languages and natural languages concerns semantics. Currently, it appears that our models roughly generate the correct POS sequence and then each word is randomly sampled, with no regard for semantic dependencies. Some methods have been proposed to address this sampling problem (Arora et al. 2016 Hopkins 2022a).
To stay close possible to our previous experiments, we propose here first-order approximation of the semantic dependencies between different words in a sentence: each word in 'content" POS (Noun, Adj, TVerb, Werb, Verb Comp) is associated with one of Ne lexical fields (respecting morphology, so bird and binds would be associated with the same lexical field), and each sentence in the corpus is made of words from a single lexical field, or context (we use the two terms interchangeably). Words within a given POS and lexical field are sampled from a power law. This is compatible with results from Guerin et al.] (2024) showing that beyond the usual Zipf law for word frequencies, co-oceurrences of two pairs also follow a power law, and that in practice very few pairs of words actually co-occur in single sentence (which is compatible with words belonging to different lexical fields). See examples in Table 10 The idea of this experiment is to see whether the model is capable of picking up this simple semantic eue.
To measure success at capturing the lexical field information, we calculated (i) the proportion of sentences containing only words from the same lexical field, and (ii) for accurately POS-translated sentences, the proportion of words that were translated into word from the right lexical field.
10.1 Experiment 6a: Two balanced lexical fields
In a first sub-experiment, we used N: = lexical fields, and sampled the context of the sentences uniformly across the two contexts. Across 4 training seeds, we systematically obtained a proportion of mono-context
13

---

Published in Transactions on Machine Learning Research (09/2024)
Sentence
no constraint
lexical field
burse sub kurches hostician sub lurchifies burse sub lurchifies hostician sub kurches
Table 10: Examples of sentences from the vanilla corpus (without lexical fields constraints) and from the 2 lexical fields corpus. In these examples the words burse and kurches are a noun and a verb from the first lexical field, while rheleates and lurchifies are noun and a verb from the second lexical field. Without the lexical field constraint, all pairs of words can co-occur and we can form four sentences, while the lexical field constraint eliminates the mixed sentences. This constraint does not apply to function words, like sub here, which are transparent to lexical fields.
sentences of This success may come from the DAE only, which plays the role of language modeling, completing sentences within a given context.
For word-by-wool word translations, the accuracy around 35% for 2 training seeds, and around 65% for the other 2 training seeds. This corresponds to the fact that in unsupervised learning, there is no signal to map contexts correctly across languages, since they are permutable within each language. This is thus an example of the rotation situation from Figure 2c This precision yet is not at ceiling: either 0% or 100%. This indicates that the model fails to learn perfectly this partition of our languages.
10.2 Experiment 6b: Two unbalanced lexical fields
In second sub-experiment, we used N. lexical fields again, and sampled the context of sentences across the two contexts with unbalance proportion of 30/70. The idea is to give the model the material to distinguish between contexts and overcome the bi-modal behavior noted above.
The BLEU score was slightly increased compared to the same experiment without lexical fields (see Table S The proportion of mono-context sentences remains at one. The proportion of words translated in the right context are, over six training sessions: 35%, 65,, 66%, 70%, 73%, 76%.
A naive bascline consisting in choosing the most frequent context would be at 70%. But this is visibly not what the models are doing (the oulput sentences are from both contexts). Another naive baseline consisting in choosing the context according to its proportion in the corpus would obtain 58%. The models are (almost always) above that level. In addition, unlike in the case of equally frequent lexical fields, the different trained models now tend to all choose the right mapping between lexical fields, therefore taking advantage of their relative frequencies to map them onto one another. This is both a good result, and a risk: if lexical fields are present in different proportions across cultures/languagee /languages, this could create a wrong signal. In sum, the models are, some extent, able to use semantic frequency cues to map lexical fields across languages.
10.3 Experiment 6c: Ten unbalanced lexical fields
We replicated the lexical field experiment with N. 10 lexical fields, and number of sentences in each lexical field varying according to power law with parameter k = 1.1 before, thereby providing finer- grained 'semantic' information. The mono-contest sentence proportion remains perfect at 1. The BLEU score increases slightly to 11.92 (see comparative results in Table K). The proportion of words translated in the right context are: 27% 28% 38% and 39%, which above a random baseline at 20%.
10.4 Discussion
Overall, these experiments show that more and more fine- ane-grained semantic information provides key signal for translation alignment, even if it is not fully sufficient. in this simple form to make unsupervised back-
14

---

Published in Transactions on Machine Learning Research (09/2024)
translation fully work. The implementation of the semantic signal through rigid lexical field constraint surely is an oversimplification. Whether more realistic versions would lead to clearer a noisier effect on translation quality is hard to predict. We tried evaluating the effect of more complex semantics depen- dencies using the model of Hopkins (022b) Such more sophisticated processes did not lead to qualitative improvements, and they come with their own simplifications. In fut work, the use of more complex and more realistic models of semantic dependencies could help decide whether the effect of the first-order lexical field constraint is confirmed.
11 Conclusion
The back-translation objective is not sufficient align two sets without supervision, in general. This is true even if it is complemented with additional objectives such as filtering or denoising auto-encoding. The method is successful nonetheless with real languages. This success then is presumably due to similarities between natural languages, which the training method picks up But it is not clear which similarities help do that. Through controlled experimentation with artificial languages, we investigate the role of lexical, syntactic and semantic properties.
We find that, when they share the exact same lexica, languages with more similar grammars are easier to translate into one another. Hence, grammatical similarity across languages of the world could be a key the success of back-translation. But when lexica also vary, syntactic similarities are not sufficient to make back-translation align two languages. Lexical alignments are thus hard to learn by back-translation. What language properties make them learnable? We find that neither anchor points (partially shared vocabu- lary), nor rich, parallel word frequencies are enough to make back translaannnttion work. Thus, manipulating various lexical and syntactic properties only, we find that some supervision signal is critical to support back-translation: through a small set of aligned sentences, or a complete set of aligned words.
Moving to semantics then, we explored how the distribution of word cooceurrences influence the efficiency of back-translation. We used only crude form of semantics, by implementing lexical fields: different classes of words that never occur within the same sentence (think about: 'clothes', 'sock', 'shoe', 'shirt' and 'astrophysics', 'interstellar', 'electromagnetic).. We find that unsupervised back-translation models are able to pick-up on this (coarse) semantic signal to find better aligument. We conclude that the success of back-translation is probably due to an even richer semantic parallelism across languages, above and beyond their lexical and syntactic similarities.
In the future, one would like to study more subtle semantic propertics then. Currently, the semantic infor- mation we implemented is both coarse-grained, and not completely decisive. One would thus like to test whether more realistic semantic information improves the system even more, makes it collapse again. And subtle properties could be investigated. For instance, selectional restrictions on verbs may play a role in shaping text distributions, above and beyond syntactic constraints, in a way that may help induce alignment across languages in an unsupervised setting. Future work should thus explore realistic semantic distributions, while maintaining the experimental control that artificial languages provide (Hopkins) 2022a). This will help understand what in natural languages makes unsupervised sock-translation reach so much success, despite its priori theoretical insuffieiency.
Broader Impact Statement
We hope that our work can have two impacts, navigating between engineering and scientific communities. In one direction, systematic investigations we present can help evaluate and improve applied translation system, and in particular for low resource languages where supervision not an option. Conversely, we take advantage of the engineering success of back-translation, to unearth what natural languages are made of, how they spontaneously align with one another.
"The idea is to introduce hicrarchical Pitman-Ya Yor processes to generate semantic dependencies. In nutshell it makes the generation of the sentences dynamical by modifying the probability of the next constitueuts based the ones already generated. For instance after the occurrence of the verb eat, the probability to generate the word water decreases and that of the word food increases.
15

---

Published in Transactions on Machine Learning Research (09/2024)
References
Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. A latent variable model approach to PMI-based word embeddings. Transactions of the Association for Computational Linguistics, 4:385-399, 2016. doi: 10.1162/tacl a 00106. URL
Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. Unsupervised neural machine translation. In International Conference on Learning Representations, 2018. URL https::/openreviiex wttpeerrriiii.nneeffrrrrr net/forum? id=Sy2ggebAW
Satanjeev Banerjee and Alon Lavie. METEOR:
An automatic metric for MT evaluation with improved correlation with human judgments. In Jade Goldstein, Alon Lavie, Chin-Yey Lin, and Clare Voss (cds.), Proceedings of the ACL Workshop on Intrinsic and Eatrinsic Evaluation Measures for Machine Transla- tion and/or Summarization, pp. 65 72, Ann Arbor, Michigan, June 2005. Association for Computational Linguistics. URL https://aclanthooll
org/w05-090
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. Transactions of the Association for Compulational Linguistics, 5:135 146, 2017. doi: 10.1162/ tacl 00051. URT. http:///adlaaatoooooooo//////000
Ondiej Bojar and Ales Tamchyna. Improving translation model by monolingual data. In Chris Callison- Burch, Philipp Koehn, Christof Monz, and Omar F. Zaidan (eds.), Proceedings of the Sixth Workshop on Statistical Machine Translation, pp. 830-336, Edinburgh, Scotland, July 2011. Association for Computa- tional Linguistics. URL https:///cllaathooloog.. www.:tthhhooooooo/1111111111
Chris Callison-Burch, Miles Osborne, and Philipp Koehn. Re-evaluating the role of Bleu in machine trans- lation research. In Diana McCarthy and Shuly Wintner (eds.), 11th Conference of the European Chapter of the Association for Computational Linguistics, pp. 249-256, Trento, Italy, April 2006. Association for Computational Linguistics. URL
org/E06- 1032
Isaac Caswell, Ciprian Chelba, and David Grangier. Tagged back-translation. In Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Mark Fishel, Yvelle Graham, Barry Haddow, Matthias Huck, Anto- nio Jimeno Yepes, Philipp Koehn, André Martins, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Matt Post, Marco Turchi, and Karin Verspoor (eds.), Proceedings of the Fourth Conference on Ma- chine Translation (Volume 1: Research Papers), pp. 53 63, Florence, Italy, August 2019. Association for Computational Linguistics. doi: 10. 18653/v1/W111 111//119-5520 5206. URL htttss://cclattheeee .wwggg..ooog.ooooooo Srg/WW9-5206
Yong Cheng, Wei Xu, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. Semi-supervised learning for neural machine translation. In Katrin Erk and Noah A. Smith (eds.), Proceedings of the 54th Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pp. 1965-1974, Berlin, Germany, August. 2016. Association for Computational Linguisties. doi: 10.18653/ 1/tp://1PPP/ 1PP16-1185. URL httpp::/acclantaaoogg..orrgg
Cheng- Han Chiang and Hung yi Lee. On the transferability of pre-trained language models: A study from artificial datascts. In AAAI Conference on Artificial Intelligence, 2021.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation ion learning at scale. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. Online, July 2020a. Association for Computational Linguistics. doi: (1/6633/v1/2200000 w1tp2/20acclmmaii acl-main 747. URL
https://aclanthhol.gg .rg. 2020 www.mminnn..... 747
Alexis Conneau, Shijie Wu, Haoran Li, Luke Zeitlemoyer, and Veselin Stoyanov. Emerging cross-lingual structure in pretrained language models. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (ods.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pp. 6022- 6034, Online, July 2020b. Association for Computational Linguistics. doi: 10. 111166/1112220..... www.20000.mmmiii552222 536.
URL http:://aclamttaoogy..orrg org 2ttp:/22002..lmm......333.
16

---

Published in Transactions on Machine Learning Research (09/2024)
Ryan Cotterell and Julia Kreutzer. Explaining and generalizing back-translation through wako-sleep. ArXiv, abs/1806.04400 04402, 2018.
Zi-Yi Dou, Antonios Anastasopoulss, and Graham Neubig. Dynamic data selection and weighting for itera- tive back-translane transaatanll......... In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 5894 5904, Online, November 2020. Association for Computational Linguistics. doi: 10. (10/663//1/220.mmmmm/ main.45., URL https://aclanthool.o org/ emnlp-main maip. 475.
Philipp Dufter and Hinrich Schütze. Identifying elements essential for BERT's multilinguality. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4423 4437, Online, November 2020. Association for Computational Linguistics. doi: (0.18653/v1 1000000emmlpmmaaa org/ 2020. eem00mmmlllppmmm mmpppmmi...35.
URL https: htap::///alanthologyy..
Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. Understanding
at scale. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 489 500, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10. 10.18653/v1/DDL //1////////////////// 1045. URL org/D18-1045
https://aclantthol..g wwt.aaahwhh...a.or////00000
Sergey Edunov, Alexei Baevski, and Michael Auli Pre-trained language model representations for lan- guage generation. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume 1 (Long and Short Papers), pp. 4052-4059, Minneapolis, Min- nesota, June 2019. Association for Computational Linguistics. doi: 10.18833/v111199-1 1409.. URL https://aclanthologg. org/ 0rg/N19-11099
Sergey Edunov, Myle Ott, Marc' Aurelio Ranzato, and Michael Auli On the evaluation of machine translation systems trained with back-translation. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeling of the Associalion for Compulational Linguislies, pp. 2836- 2846, Online, July 2020. Association for Computational Linguistics. doi: 10. 10.188633/112002..0 aww.://.....mmm.m.mm.mmmmm 253. org 20200020.2.000000
URL
meain.255 253
Nicolas Guerin, Emmanuel Chemla, Robin Ryder, and Shane Steinert-Threlked. Second-order zipf law for word co-occurrences. submitted, 2024.
Francisco Guzmán, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaud- hary, and Marc' Aurelio Ranzato. The FLORES evaluation datasets for low-resource machine translation: and Munhala-Englise In Kentaro Imui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) pp. 6098-6111, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 10. 10.18833/11/D11 1632. URL https://aclannthology http://ccaaatallllo.oo///////-/0
Barry Haddow, Rachel Bawden, Antonio Valerio Miceli Barone, Jindřich Helel, and Alexandra Birch. Survey of Low-Resource Machine Translation. Technical Report arXiv:2109.0044 arXiv, February 2022. URL [cs] type: article.
org//bs//109 http://arxiv.orr/aa//2120 http://arxiv.
00486 arXiv:2109.0048
Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tic Ya- Liu, and Wei-Ying Ma. Dual learning for machine translation. NIPS16, pp. 820 828, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN
9781510838819.
Vu Cong Duy Hoang, Philipp Koehn, Gholamreza Haffari, and Trevor Cohn. Iterative back-translation for neural machine translation. In Alexandra Birch, Andrew Finch, Thang Luong, Graham Neubig, and Yusuke Oda (eds.), Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pp. 18 24, Melbourne, Australia, July 2018 Association for Computational Linguistics. doi: 10.18653/v1/ W18-2703. URL attpp::/aalamnhooogy.oorrgg
17

---

Published in Transactions on Machine Learning Research (09/2024)
Mark Hopkins. Towards more natural artificial languages. In Proceedings of the 26th Conference on Compu- tational Natural Language Learning (CoNLI.); pp. 55-94, Abu Dhabi, United Arab Emirates (Hybrid), Dec 2022a. Association for Computational Linguistics. URL
Mark Hopkins. Towards more natural artificial languages. In Antske Fokkens and Vivek Srikumar (eds.); Proceedings of the 26th Conference on Computational Natural Language Learning (CoNLL), pp. 85 94, Abu Dhabi, United Arab Emirates (Hybrid), Docember 2022b. Association for Computational Linguistios. doi: 10. 10.18653/v1 (16633/v1/222.com/// con URL httpps://aclanttoooog...
Aizhan Imankulova, Takayuki Sato, and Mamoru Komachi. Improving low-resource neural machine trans- lation with filtered pseudo-parallel corpus. In Toshiaki Nakazawa and Isao Goto (eds.), Proceedings of the 4th Workshop on Asion Translation (WAT2017), pp. 70 70-78, Taipei, Taiwan, November 2017. Asian Fedcration of Natural Language Processing. URL https: https: www.elaanntooooooooo/oo..
Ann Irvine and Chris Callison- End-to-end statistical machine translation with zero or small parallel texts. Natural Language Engineering, 22(4):517-548, 2016. doi: 10. 11017//11311124490000177
Juraj Juraska, Mara Finkelstein, Daniel Deutsch, Aditya Siddhant, Mehdi Mirzazadeh, and Markus Fre- itag. MetricX-23; The Google submission to the WMT 2023 metrics shared task. In Philipp Koehn, Barry Haddow, Tom Kocmi, and Christof Monz (eds.), Proceedingg of the Eighth Conference on Machine Translation, pp. 756-767, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1 22122222..222 1.63 URL httpp://acclatthology org/2023. wmt-1.69
Karthikeyan K, Zihan Wang, Stephen Mayhew, and Dan Roth. Cross-lingual ability of multilingual bert: An empirical study. ArXio, abs/1912.07844 2019.
Gaurav Kharkwal. Taming the jobberwocky: eramining sentence processing with novel words. PhD the- sis, Rutgers University - Graduate School New Brunswick, 2014. URL fttt::://rucore.ibraate
rutgers.eduu/rtgeer. http:///teeeeee////////////eeeee -ib/4530///iiitiiinnn
export)
Jyotsana Khatri and Pushpak Bhattacharyya. Filtering back translatddd data in unsupervised neural machine translation. In Donia Scott, Nuria Bel, and Chengqing Zong (cds.), Proceedings of the 28th International Barcelona, Spain (Online), December 2020. International Committee on Computational Linguistics. doi: 10.18653/v1 /218663//1/2222.coinng main.383. URL
Conference on Compulational Länguisties, pp.
Yunsu Kim, Yingbo Gao, and Hermann Ney. Effective cross-lingual transfer of neural machine translation models without shared vocabularies. In Anna Korhonen, David Traum, and Lluís Mårquez (eds.), Pro- ceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1246-1257, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 1111188/11/P1--- httpp://aclanthoolgg. www.hhhooooyy.og/111111 otg/p19- 1120
URL
Yunsu Kim, Miguel Graça, and Hermann Ney. When and why is unsupervised neural machine translation useless? In André Martins, Helena Moniz, Sara Fumega, Bruno Martins, Fernando Batista, Luisa Coheur, Carla Parra, Isabel Trancoso, Marco Turchi, Arianna Bisazza, Joss Moorkens, Ana Guerberof, Mary Nurminen, Lena Marg, and Mikel L. Forcada (eds.), Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pp. 35 44, Lisboa, Portugal, November 2020. European Association for Machine Translation. URL
wrg./020.........oooo.. eamt-1.5
Alexandre Klementiev, Ann Irvine, Chris Callison-Burch, and David Yarowsky. Toward statistical machine translation without parallel corpora. In Proceedings of the 13th Conference of the European Chapter of the Association for Compulational Linguistics, EACL '12, pp. 130 140, USA, 2012. Association for Computational Linguistics. ISBN 9781937284190.
Surabhi Kumari, Nikhil Jaiswal, Mayur Palidar, Manasi Patwardhan, Shirish Karande, Puneet Agarwal, and Lovekesh Vig. Domain adaptation for NMT via filtered iterative tack-translation. In Eyal Ben-David, Shay Cohen, Ryan McDonald, Barbara Plank, Roi Reichart, Guy Rotman, and Yftah Ziscr (ods.), ) Proceedings of the Second Workshop on Domain Adaptation for NLP, pp. 263 263-271, Kyiv, Ukraine, April 2021, Association for Computational Linguistics. URL
org/2021 addptpppp11.ppppp11111.1.0
18

---

Nima Pourdamghani, Nada Aldarrab, Marjan Ghazvininejad, Kevin Knight, and Jonathan May. Translat- ing translationese: A two-step approach to unsupervised machine translation. In Anna Korhonen, David Traum, and Lluis Mårquez (eds.), Proceedings of the 57th Annuol Meeting of the Association for Compu- tational Linguistics, pp. 3057-3062, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 111655/v1/P19-- Published in Transactions on Machine Learning Research (09/2024)
Guillaume Lample and Alexis Conneau. Cross-lingual Language Model Pretraining. In 33rd Conference Newral Information Processing Systems (NeurIPS 2019), 2019.
Guillaume Lample, Alexis Comneau, Ludovic Denoyer, and translation using monolingual corpora only. In International Conference of Learning Representations (ICLR), 2018a.
Ranzato. Unsupervised machine
Guillaume Lample, Alexis Conneau, Mare Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. Word translation without parallel data. In International Conference on Learning Representations, 2018b. URL httpps://pennreview net.:ooonnidd-iiieeeaaa.
Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc' Aurelio Ranzato. Phrase- based & neural unsupervised machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 5039 5049, Brussels, Belgium, Oct 2018c. Association for Computational Linguistics. doi: (0.18653/v1/DHH 1549. URL
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghavvininejad, Mike Lewis, and Luke Zettlemoyer. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726-742, 2020. doi: 10. 10.1162/tacl a 00343. URL https:
hwwpaoooogoooog//2202 http://ooo...g//02200.ee1111-440 tac1-1.47
Tomás Mikolov, Quoc V. Le, and Ilya Sutskever. Exploiting similarities among languages for machine translation. CoRR, abs/1339.44165 2013. URL http:://axiv..o abs/ .4688
Xing Niu, Michael Denkowski, and Marine Carpuat. Bi-directional neural machine translation with synthetic parallel data. In Alexandra Birch, Andrew Finch, Thang Luong, Graham Neubig, and Yusuke Oda (eds.), Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pp. 84 91, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18633/v1/W18. 2710. URL https://aclaatthology/ org/ wrw.looyyoorg//1--2111 W18-2710
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Pierre Isabelle, Eugene Charniak, and Dekang Lin (eds.), Proceedings of the 40th Annual Meeling of the Association for Computational Linguislics, pp. 311 318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics. doi: 10.3115/ 10.3115/1073083. 1073313,, URL https: /aaclanthology wtwp:ttthoooooooo/P222----010-01 or4/P02-1040
Steven T. Piantadosi. Zipf's word frequency law in natural language: A critical review and future di- rections. Psychonomic Bulletin and Review, 21(5):1122-1130, 2014. ISSN 15315320. doi: 10.3758/ s13423-001400885-.
Alberto Poncelas, D. Shterionov, Andy Way, Gideon Maillette de Buy Wenniger, and Peyman Passban. Investigating backtranslation in neural machine translation. In Europeon Association for Machine Trans- lation Seaffrrnnces/Workshhe 2018.
URL httpps://aclanttooogg..ooo org/p19-1293
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. COMET: neural framework for MT evaluation. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2685-2777 Online, November 2020. Association for Computational Linguistics. doi: 10.18663/11/2000ccmmpp. mmwp://mmmmmmmmmiim..........2 213.. URL httpp::/acclanttolory.ooor
emnlp main. 213.
Hadeel Saadany and Constantin Orasan. BLEU, METEOR, BERTScore: Evaluation of metrics performance in assessing critical translation errors in sentiment-oriented text. In Ruslan Mitkov, Vilelmini Sosoni,
19

---

Published in Transactions on Machine Learning Research (09/2024)
Julie Christine Giguère, Elena Murgolo, and Elizabeth Deysel (ods.), Proceedings of the Translation and Interpreting Technology Online Conference, pp. 48-56, Held Online, July 2021. INCOMA Ltd. URL.
httts::/acllantholoee wwwphhoooggoo.oo///oeeoog org. wttp://...o///...ttrrroooooo
Thibault Sellam, Dipanjan Das, and Ankur Parikh. BLEURT: Learning robust metrics for text generation. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (cds.), Proceedings of the 58th Annual Meeling of the Associalion for Compulational Lánguislics, pp. 7881 -792, Online, July 2020. Association for Computational Linguistics. doi: (00655//v12220.... acl-main. 704. URL https://aclanthology org//
2020 a2200ac1-main.77 704
Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving neural machine translation models with monolingual data. In Katrin Erk and Noah A. Smith (eds.), Proceedings of the 54th Annual Meeling of the Association for Computational Linguisties (Volume Long Papers), pp. 86-96, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18633/11/P1610000 URL https:// aclanthology.orge wwtp://gg//11111
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie Yan Liu. MASS: Masked sequence to sequence pre-training for language generation. 36th International Conference on Machine Learning, ICML, 2019, 2019-Jume:10384 10394, 2019.
llya Sutskever, Oriol Vinyals, and Quoe V. Le. Sequence to sequence learning with neural networks. In Proceedings of the 27th International Conference on Neural Information Processing Systems Volume 2, NIPS'14, pp. 3104-3112, Cambridge, MA, USA, 2014. MIT Press.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31st International Conference on Newral Information Processing Systems, NIPS'17, pp. 6000-6010, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
Yu Wan, Dayiheng Liu, Baosong Yang, Haibo Zhang, Boxing Chen, Derek Wong, and Lidia Chao. UniTE: Unified translation evaluation. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeling of the Association for Compulational Linguistics (Volume 1: Long Papers), pp. 8117-8122 8127, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: (4.1663/vv1 1/22222.2cl---22 aall-oong.556 URL https: hatpsss///laathhlogg.ee org/ 20wgg//222..c2 http//222222222112 acl-alng....5.... .58
Yiren Wang. Yingee Xia, Tianyu Hc, Fei Tian, Tao Qin, ChengXiang Zhai, and Tic-Yan Yan Liu. Multi-agent dual learning. In International Conference on Learning Representations, 2019. URL httpp:://penreevie:
Jennifer C. White and Ryan Cotterell. Examining the inductive bias of neural language models with artificial languages. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 454 463, Online, Aug 2021. Association for Computational Linguistics. doi: (0.1863//1220221acl long.38. URL httpp:://aclantaoloogy..o org. aww.1ooo...1ooooo.......
Shijie Wu and Mark Dredze. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. Tn Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (cds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Internalional Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 833 844, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 18.13///v 11/////11119//// 107- URL httts:://ccaatttooogyooo///---/1
Yonghui WU, Mike Schuster, Zhifeng Chen, Quoe V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation. Technical report, 2016.
20

---

Published in Transactions on Machine Learning Research (09/2024)
Yingee Xia, Tao Qin, Wei Chen, Jiang Bian, Nenghai Yu, and Tic-Yan Liu. Dual supervised learning. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML'17, pp- 3789 3798. JMLR.org, 2017.
Zhen Yang, Wei Chen, Feng Wang, and Bo Xu. Unsupervised neural machine translation with weight sharing. In Iryna Gurevych and Yusuke Miyao (ods.), Proceedings of the 56th Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Pupers), pp. 46 55, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 40.18653/v1/PD1 1005. URL https://acllanthooog org/P18-10 1005
Changtong Zan, Liang Ding, Li Shen, Yu Cao, Weifeng Liu, and Dacheng Tao. On the complementarity be- tween pre-training and randomm-iiiiializttion for resource-rich machine translation. ArXiv, abs/2209.03316,, 2022.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. 2019.
21


Reviews:
[{"review_id": "KOESUickEv", "reviewer": "Reviewer_juUp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n1. The paper addresses an important question of **why exactly** back-translations work in the case of unsupervised machine translation, even though there are no theoretical guarantees for their success. This line of work is important for the community to better understand the factors that drive back-translation success, and understand for what kinds of language pairs back-translation would likely succeed (or not succeed, e.g. when there is a substantial amount of lexical differences).\n\n2. The paper is rigorous and extensive in terms of its analysis and methodology, which is done through synthetic language pairs, where one can control for the different factors (e.g. degree of syntactic & lexical overlap, etc.). I appreciate that the paper covers multiple different factors that can affect back-translation performance, such as syntactic & lexical & semantic overlap, etc.\n\n3. The paper is very clear and well-written, and includes a comprehensive overview of both unsupervised machine translation and back-translation. This would be useful for readers that have not worked directly on this line of work.\n\n4. The findings would likely be of interest to the broader community.\n\n# Weaknesses\n\n1. While the findings are interesting, it is not immediately clear how to apply the findings to make back-translation work better for unsupervised machine translation. I do think that the paper can make some recommendations based on its findings (e.g. that some degree of supervision is necessary, so having high-quality bilingual dictionary or a small amount of parallel corpus would be very helpful, or that having cross-lingual word embedding that puts similar words in different languages in the same vector space is essential etc.). It would be nice to see the implications of the paper's findings spelled out more clearly, so that the broader community can build on the findings to design better-performing approaches.\n\n2. Not enough space is dedicated to the semantic similarity section through lexical fields (Section 10, Experiment 6, page 12). This is an important part of the paper, because this shared semantic information helps translation alignment to a good extent (more so than other factors like syntactic similarity, for example). A more extensive description of lexical fields and what is exactly being done, ideally along with some examples, would be very helpful for the reader.\n\n3. I am not entirely certain about how realistic the lexical field assumptions are in real languages (in particular, the assumption that each sentence in the corpus is made of words from a single lexical field, bottom of page 12). Some discussion around this would be much appreciated.\n\n4. A discussion around how pre-training and cross-lingual embeddings would affect the findings would also be very helpful. Are the problems with lexical overlap, etc. can be solved by multi-lingual pre-training on massively multi-lingual data? Or would these likely to persist?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper aims to better understand **why** on-the-fly back-translation works well in the case of unsupervised machine translation, even when there are no theoretical guarantees for their success. To that end, the paper used an artificial language setup where it is possible to disentangle the impact of several different factors on back-translation's success in the context of unsupervised machine translation: (i) the extent to which the word frequency distributions are similar across the two languages; (ii) the extent to which the vocabularies are shared / there are lexical overlaps; (iii) the extent to which the syntax between the two languages differs; and (iv) the extent to which crude semantic signals (in particular, lexical fields) are similar across the two languages.\n\nThe paper began by demonstrating that back-translation is not theoretically guaranteed to succeed in the case of unsupervised machine translation. Furthermore, while prior work has hypothesized the importance of similar syntactic structures, word frequency, and anchor points (i.e. the fact that numbers tend to have similar surface strings across different languages, and can therefore serve as anchor points for other words different languages), this paper finds evidence to the contrary. While the paper finds that shared syntactic structures improve back-translation performance, syntactic similarity alone is insufficient to make back-translation work well in the presence of strong lexical differences. Furthermore, neither anchor words nor similar word frequencies are sufficient to make back-translation succeed, although some degree of shared semantic parallelism (here in the form of **lexical fields**) is helpful. \n\nThe paper concludes that back-translation success can likely be attributed to the richer semantic parallelism across languages. Furthermore, *some* degree of supervision signal is critical for back-translation success for unsupervised machine translation, whether through a small number of aligned sentences (i.e. a small parallel corpus) or a complete set of aligned words (i.e. a good bilingual dictionary, whether provided from an existing dictionary or learned through bilingual lexical induction).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. **Recommended**: A discussion around what the paper's findings mean for designing better unsupervised machine translation system through back-translation (e.g. that a good dictionary / some small amount of supervision is important, etc.).\n\n2. **Recommended**: More discussion and space dedicated towards explaining the lexical field experiment, ideally with examples, given that the paper highlights the importance of semantic similarity across languages.\n\n3. **Recommended**: A discussion around how realistic the lexical field assumption is in real languages.\n\n4. **Recommended**: A discussion around whether or not pre-training and cross-lingual word embeddings would likely mitigate the issues highlighted here (e.g. helping to find alignment between different words in different languages, etc.).", "broader_impact_concerns": "No broader impact concerns from my side, and I am happy with the broader impact statement as it is written on the paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ceBBLzoj8Q", "reviewer": "Reviewer_5UWs", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- The paper presents a series of controlled studies to understand what may enable the success of back-translation for unsupervised NMT.\n\n- The paper challenges assumptions about the success of unsupervised NMT.\n\n**Weaknesses**\n\n- 100,000 sentences seem low for unsupervised NMT. It is not fully clear if and how some of the findings would differ with larger datasets.\n\n- Semantic properties are only explored at a coarse level, and more complex semantic properties may be difficult to simulate with artificial languages.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper uses artificial languages based on context-free grammars to inspect the behavior of back-translation for unsupervised neural machine translation. With a shared vocabulary, using different grammars leads to various degrees of success, generally with lower performance for major ordering changes between the two languages. When using identical grammars, but fully disjoint vocabularies, translation performance is very poor even if round-trip translation is often successful, with the model learning the wrong mapping between tokens. Using shared words or non-uniform word frequencies is not sufficient, while adding some supervised signal is helpful. Using lexical fields, i.e. non-overlapping word subsets, hints that richer semantic dependencies may be necessary for unsupervised NMT.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Questions**\n\nP1. Why is creating parallel datasets \"sometimes an impossible task\"?\n\nP5. Why 1374 words?\n\nP5. Why share 3 layers out of 4? In Lample et al. (2018c), it seems that all parameters are shared.\n\n**Comments**\n\nIn the related work section, Lample et al. (2018a) is presented as a follow-up to Artetxe et al. (2018), but I believe both papers were contemporaneous.\n\n**Typos**\n\nP2. exerpimental \nP13. unbalance", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xpRcg65KpW", "reviewer": "Reviewer_JW9t", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper is clearly written and explained and the hypotheses tested are well formulated and valid tests of the back translation method.\n\nThe experimental methodology is sound and carried out in detail for the artificially generated languages the authors test. These languages and experimental results allow us to a deeper understanding of the effect of variation/similarities in languages and how this can affect models that attempt to back translate between languages.\n\nThe generation procedure of artificial languages, details of grammar, sizes of datasets produced etc., are all well documented and easy to replicate.\n\nWeakness of evaluation: the main metric to score translations in this paper is BLEU scores of generations, and although this is widely and most commonly used for machine translation evaluations, there are several papers that highlight the deficiencies of this metric (see here: https://aclanthology.org/E06-1032.pdf) and alternatives to it (see here: https://aclanthology.org/2021.triton-1.6.pdf). Especially since some of the experiments hinge on lexical similarity between languages, it would be worth discussing alternative metrics for some of this paper (e.g., why the limitations of this metric might result e.g., in higher results for the lexically similar languages and highlighting potential differences that might occur if the other metrics that are less biased towards simple n-gram overlap were to be used.)\n\nWeakness of lexical/similarity hypotheses: the similarities between real languages are often due to more subtle semantically aligned properties rather than just pure lexical similarity that is evaluated here. Although the simple case of lexical overlap or anchor points is important to evaluate, it is worth also attempting to evaluate the more nuanced similarities that occur in real languages (or atleast posit how this might influence back translation methods).\n\nIn order to understand how the results form these artificial languages can be mapped to real languages that we might want to apply these translation methods to, it would be useful to have a table or paragraph that outlines how pairs of real languages fall into the categories of hypotheses framed here (e.g., which pairs of languages have some high percentage of lexical overlap, anchor points, syntactic similarity and so on). Future work that draws on this paper can then attempt to empirically evaluate if real translation datasets/models on this task also have similar effects as is seen by the languages here.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper evaluates the performance of back translation approaches for neural machine translation through controlled experiments with artificial languages as an attempt to theoretically uncover what properties of languages allow back translation to work at all as an algorithm. Specifically they construct and look at languages that differ in word frequency distributions and syntactic structure, allowing them to evaluate how the similarity of lexical fields across languages can allow better back translation. They discuss how this can help understand which languages, based on their syntactic and lexical properties might help develop cross-linguistic tools for better translation methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "A discussion section about a mapping between the grammars/artificial languages and real languages in the world would be helpful to understand the implications on real translation tasks.\n\nA table that gives an overview of this mapping with example languages would be a great help to the community and future work drawing off of this paper.\n\nHaving one more metric that is an alternative to BLEU (especially for the lexically similar languages) would be good to see to verify that the trend of results hold up.\n\nA discussion section about the implications of the metrics used and how they favour some language pairs over others would be helpful.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All the reviewers and myself concur this an interesting and well-executed paper illuminating an important empirical technique that is seldom studied in a controlled setting. The paper is scientifically rigorous, well-written and clear. Reviewers pointed out some of the limitations of the paper that should be addressed in the camera-ready, notably 1) strive to make the paper's findings spelled out more clearly, so that new methods could more easily be designed; and 2) the decision of using the lexical field assumption, which is quite restrictive for real languages. I believe the authors already promised to elaborate and even provide more experiments on 2) for the camera-ready. Overall, this is great work and deserves acceptance.
Decision: Accept (poster)