Title: A Theory of Unsupervised Translation Motivated by Understanding Animal Communication
Venue: NeurIPS 2023
Accepted: yes
Keywords: Theory

Unsupervised Machine Translation
Abstract: Neural networks are capable of translating between languages—in some cases even between two languages where there is little or no access to parallel translations, in what is known as Unsupervised Machine Translation (UMT). Given this progress, it is intriguing to ask whether machine learning tools can ultimately enable understanding animal communication, particularly that of highly intelligent
animals. We propose a theoretical framework for analyzing UMT when no parallel translations are available and when it cannot be assumed that the source and target corpora address related subject domains or posses similar linguistic structure. We
exemplify this theory with two stylized models of language, for which our framework provides bounds on necessary sample complexity; the bounds are formally proven and experimentally verified on synthetic data. These bounds show that the error rates are inversely related to the language complexity and amount of common ground. This suggests that unsupervised translation of animal communication may be feasible if the communication system is sufficiently complex.

A Theory of Unsupervised Translation

Abstract

Neural networks are capable of translating between languages-in some cases even between two languages where there is little or no access to parallel translations, in what is known as Unsupervised Machine Translation (UMT). Given this progress, it is intriguing to ask whether machine learning tools can ultimately enable understanding animal communication, particularly that of highly intelligent animals. We propose a theoretical framework for analyzing UMT when no parallel translations are available and when it cannot be assumed that the source and target corpora address related subject domains or posses similar linguistic structure. We exemplify this theory with two stylized models of language, for which our framework provides bounds on necessary sample complexity; the bounds are formally proven and experimentally verified on synthetic data. These bounds show that the error rates are inversely related to the language complexity and amount of common ground. This suggests that unsupervised translation of animal communication may be feasible if the communication system is sufficiently complex.

1 Introduction

Recent interest in translating animal communication [2, 3, 9] has been motivated by breakthrough performance of Language Models (LMs). Empirical work has succeeded in unsupervised translation between human-language pairs such as English-French [23, 5] and programming languages such as Python-Java [33]. Key to this feasibility seems to be the fact that language statistics, captured by a LM (a probability distribution over text), encapsulate more than just grammar. For example, even though both are grammatically correct, The calf nursed from its mother is more than 1,000 times more likely than The calf nursed from its father . 2

Given this remarkable progress, it is natural to ask whether it is possible to collect and analyze animal communication data, aiming towards translating animal communication to a human language description. This is particularly interesting when the source language may be of highly social and intelligent animals, such as whales, and the target language is a human language, such as English.

Challenges. The first and most basic challenge is understanding the goal , a question with a rich history of philosophical debate [38]. To define the goal, we consider a hypothetical ground-truth translator. As a thought experiment, consider a 'mermaid' fluent in English and the source language

Project CETI

UC Berkeley & Project CETI

(e.g. sperm whale communication). Such a mermaid could translate whale vocalizations that English naturally expresses. An immediate worry arises: what about communications that the source language may have about topics for which English has no specific words? For example, sperm whales have a sonar sense which they use to perform echolocation. In that case, lacking a better alternative, the mermaid may translate such a conversation as (something about echolocation) . 3

Thus, we formally define the goal to be to achieve translations similar to those that would be output by a hypothetical ground-truth translator. While this does not guarantee functional utility, it brings the general task of unsupervised translation and the specific task of understanding animal communication into the familiar territory of supervised translation, where one can use existing error metrics to define (hypothetical) error rates.

The second challenge is that animal communication is unlikely to share much, if any, linguistic structure with human languages. Indeed, our theory will make no assumption on the source language other than that it is presented in a textual format. That said, one of our instantiations of the general theory (the knowledge graph) shows that translation is easier between compositional languages.

The third challenge is domain gap , i.e., that ideal translations of animal communications into English would be semantically different from existing English text, and we have no precise prior model of this semantic content. (In contrast, the distribution of English translations of French text would resemble the distribution of English text.) Simply put: whales do not 'talk' about smartphones. Instead, we assume the existence of a broad prior that models plausible English translations of animal communication. LMs assign likelihood to an input text based not only on grammatical correctness, but also on agreement with the training data. In particular, LMs trained on massive and diverse data, including some capturing facts about animals, may be able to reason about the plausibility of a candidate translation. See Figure 1 and the discussion in Appendix H.

1.1 Framework and results

A translator 4 is a function f : X → Y that translates source text x ∈ X into the target language f ( x ) ∈ Y . We focus on the easier-to-analyze case of lossless translation, where f is invertible (one-to-one) denoted by f θ : X ↪ →Y . See Appendix I.1 for an extension to lossy translation.

We will consider a parameterized family of translators { f θ : X → Y} θ ∈ Θ , with the goal being to learn the parameters θ ∈ Θ of the most accurate translator. Accuracy (defined shortly) is measured with respect to a hypothetical ground-truth translator denoted by f ⋆ : X → Y . We make a realizability assumption that the ground-truth translator can be represented in our family, i.e., ⋆ ∈ Θ .

̸

The source language is defined as a distribution µ over x ∈ X , where µ ( x ) is the likelihood that text x occurs in the source language. The error of a model θ ∈ Θ will be measured in terms of err( θ ) := Pr x ∼ µ [ f θ ( x ) = f ⋆ ( x )] , or at times a general bounded loss function L ( θ ) . Given µ and f ⋆ , it will be useful to consider the translated language distribution τ over Y by taking f ⋆ ( x ) for x ∼ µ .

In the case of similar source and target domains, one may assume that the target language distribution ν over Y is close to τ . This is a common intuition given for the 'magic' behind why UMT sometimes

works: for complex asymmetric distributions, there may a nearly unique transformation in { f θ } θ ∈ Θ that maps µ to something close ν (namely f ⋆ which maps µ to τ ). A common approach in UMT is to embed source and target text as high-dimensional vectors and learn a low-complexity transformation, e.g., a rotation between these Euclidean spaces. Similarly, translator complexity will also play an important role in our analysis.

Priors. Rather than assuming that the target distribution ν is similar to the translated distribution τ , we will instead assume access to a broad prior ρ over Y meant to capture how plausible a translation y is, with larger ρ ( y ) indicating more natural and plausible translation. Appendix H discusses one way a prior oracle can be created, starting with an LM ≈ ν learned from many examples in the target domain, and combined with a prompt, in the target language, describing the source domain.

We define the problem of unsupervised machine translation (with a prior) to be finding an accurate θ ∈ Θ given m iid unlabeled source texts x 1 , . . . , x m ∼ µ and oracle access to prior ρ .

MLE. Our focus is on the Maximum-Likelihood Estimator (MLE), which selects model parameters θ ∈ Θ that (approximately) maximize the likelihood of translations ∏ i ρ ( f θ ( x i ) ) .

Definition 1.1 (MLE) . Given input a translator family { f θ : X → Y} θ ∈ Θ , samples x 1 , . . . , x m ∈ X and a distribution ρ over Y , the MLE outputs

If multiple θ have equal empirical loss, it breaks ties, say, lexicographically.

We note that heuristics for MLE have proven extremely successful in training the breakthrough LMs, even though MLE optimization is intractable in the worst case.

Next, we analyze the efficacy of MLE in two complementary models of language: one that is highly structured (requiring compositional language) and one that is completely unstructured. These analyses both make strong assumptions on the target language, but make few assumptions about the source language itself. In both cases, the source distributions are uniform over subsets of X , which (informally) is the 'difficult' case for UMT as the learner cannot benefit from similarity in text frequency across languages. Both models are parameterized by the amount of 'common ground' between the source language and the prior, and both are randomized. Note that these models are not intended to accurately capture natural language. Rather, they illustrate how our theory can be used to study the effect of language similarity and complexity on data requirements for UMT.

Knowledge graph model. Our first model consists of a pair of related knowledge graphs , in which edges encode knowledge of binary relations. Each edge yields a text that described the knowledge it encodes. For example, in Figure 2 edges encode which animal A eats which other animal B , and text is derived as a simple description A eats B . 5

Formally, there are two Erd˝ os-Rényi random digraphs. The target graph is assumed to contain n nodes, while the source graph has r ≤ n nodes corresponding to an (unknown) subset of the n target nodes. The model has two parameters: the average degree d in the (randomly-generated) target language graph, and the agreement α ∈ (0 , 1] between the source and the target graphs. Here α = 1 is complete agreement on edges in the subgraph, while α = 0 is complete independence. We assume the languages use a compositional encoding for edges, meaning that they encode a given edge by encoding both nodes, so we may consider only | Θ | = n ! / ( n -r )! translators consistently mapping the r source nodes to the n target nodes, which is many fewer than the number of functions f : X → Y mapping the |X| = O ( r 2 ) source edges into the Y = O ( n 2 ) target edges. Human languages as well as the communication systems of several animals are known to be compositional [39]. 6

We analyze how the error of the learned translator depends on this 'common knowledge' α :

Theorem 1.2 (Theorem 3.2, simplified) . Consider a source language µ and a prior ρ generated by the knowledge graph model over r source nodes, n target nodes, average degree d and agreement

siarc

Welsh isentences bwytaodd siarcsiarc. bwytaodd eog bennog. bwytaodd siarceog.

English isentences sharkseatsalmon. salmon neat herring sharks eat sharks. herring eatr mysidacea.

mysidacea

herring

eog

bennog

salmon

sharks

parameter α . Then, with at least 99% probability, when given m source sentences x 1 , . . . , x m and access to a prior ρ , MLE outputs a translator ˆ θ with error

The second term decreases to 0 at a O ( m -1 / 2 ) rate, similar to (noisy) generalization bounds [30]. Note that the first term does not decrease with the number of samples. The average degree d is a rough model of language complexity capturing per-node knowledge, while the agreement parameter α captures the amount of common ground. Thus, more complex languages can be translated (within a given error rate) with less common ground. Even with m = ∞ source data, there could still be errors in the mapping. For instance, there could be multiple triangles in the source and target graphs that lead to ambiguities. However, for complex knowledge relations (degree d ≫ 1 /α 2 ), there will be few such ambiguities.

Figure 2 illustrates an example of four English sentences and three sentences (corresponding to an unknown subset of three of the English sentences) in Welsh. For UMT, one might hypothesize that bwytaodd means eat because they both appear in every sentence. One might predict that siarc means shark because the word siarc appears twice in a single Welsh sentence and only the word shark appears twice in an English sentence. Next, note that eog may mean salmon because they are the only other words occurring with siarc and shark . Similar logic suggests that bennog means herring . Furthermore, the word order is consistently permuted, with subject-verb-object in English and verb-subject-object in Welsh. This translation is indeed roughly correct. This information is encoded in the directed graphs as shown, where each node corresponds to an animal species and an edge between two nodes is present if the one species eats the other.

'Common nonsense' model. The second model, the common nonsense model, assumes no linguistic structure on the source language. Here, we set out to capture the fact that the translated language τ = µ ◦ f ⋆ and the prior ρ share some common ground through the fact that the laws of nature may exclude common 'nonsense' outside both distributions' support.

Earlier work has justified alignment for UMT under the intuition that the target language distribution ν is approximated by a nearly unique simple transformation, e.g., a rotation, of the source distribution µ . However, for a prior ρ , our work suggests that UMT may also be possible if there is nearly a unique simple transformation that maps τ so that it is contained in ρ . Figure 3 illustrates such a nearly unique rotation-the UMT 'puzzle' of finding a transformation f θ of µ which is contained within ρ is subtly different from finding an alignment.

In the common nonsense model, τ and ρ are uniform over arbitrary sets T ⊆ P ⊆ Y from which a common α ∈ (0 , 1 / 2] fraction of text is removed (hence the name 'common nonsense'). Specifically, τ and ρ are defined to be uniform over ˜ T = T \ S , ˜ P = P \ S , respectively, for a set S sampled by including each y ∈ Y with probability α .

We analyze the error of the learned translator as a function of the amount of common nonsense:

Theorem 1.3 (Theorem 3.4, simplified) . Consider source language µ and a prior ρ generated by the common nonsense model over | T | source texts and common-nonsense parameter α , and a translator family parameterized by | Θ | . Then, with at least 99% probability, when given m source sentences x 1 , . . . , x m and access to a prior ρ , MLE outputs a translator ˆ θ with error

̸

x

y

h

V~T

μ

p

Theorem 3.5 gives a nearly matching lower bound. Let us unpack the relevant quantities. First, we think of α as measuring the amount of agreement or common ground required, which might be a small constant. Second, note that | T | is a coarse measure of the complexity of the source language, which requires a total of ˜ O ( | T | ) bits to encode. Thus, the bound suggests that accurate UMT requires the translator to be simple, with a description length that is an α -factor of the language description length, and again α captures the agreement between τ, ρ . Thus, even with limited common ground, one may be able to translate from a source language that is sufficiently complex. Third, for simplicity, we require X , Y ⊂ { 0 , 1 } ∗ to be finite sets of binary strings, so WLOG Θ may be also assumed to be finite. Thus, log 2 | Θ | is the description length , a coarse but useful complexity measure that equals the number of bits required to describe any model. (Neural network parameters can be encoded using a constant number of bits per parameter.) Appendix I.2 discusses how this can be generalized to continuous parameters.

Importantly, we note that (supervised) neural machine translators typically use far fewer parameters than LMs. 7 To see why, consider the example of the nursing calf (page 1) and the fact that a translator needs not know that calves nurse from mothers. On the other hand, such knowledge is essential to generate realistic text. Similarly, generating realistic text requires maintaining coherence between paragraphs, while translation can often be done at the paragraph level.

As a warm-up, we include a simplified version of the common nonsense model, called the tree-based model (Appendix B.1), in which texts are constructed word-by-word based on a tree structure.

Comparison to supervised classification. Consider the dependency on m , the number of training examples. Note that the classic Occam bound O ( 1 m log | Θ | ) is what one gets for noiseless supervised classification, that is, when one is also given labels y i = f ⋆ ( x i ) at training time, which is similar to Theorem 1.3, and give ˜ O ( m -1 / 2 ) bounds for noisy classification as in Theorem 1.2. Furthermore, these bounds apply to translation, which can be viewed as a special case of classification with many classes Y . Thus, in both cases, the data dependency on m is quite similar to that of classification.

Experiments. We validate our theorems generating synthetic data from randomly-generated languages according to each model, and evaluating translator error as a function of the number of samples and amount of common ground. The knowledge graph model (Figure 4, left) is used to generate a source graph (language) on r = 9 nodes to a target graph (language) on n = 10 nodes and average degree d ≈ 5 , while varying the agreement parameter α . We also vary r (Figure 4, right) supporting our main message: more complex languages can be translated more accurately. For the common nonsense model (Figure 5) we simulate translation of a source language of size | T | = 10 5 while varying the fraction of common nonsense α . Appendix E contains details and code.

r

4

0

.

10

±

0

.

05

7

0

.

74

±

0

.

08

10

1

.

00

±

0

.

00

Knowledge Graph

alpha:1

一

alpha:0.66

alpha:0.33

alpha:0

Translation error

0.8

0.6

0.4

0.2

0

0

20

40

60

80

Common Nonsense

alpha:0.8- alpha:0.7

alpha: 0.6

alpha:0.5- alpha:0.4

-alpha:0.3

一alpha:0.2

-alpha:0.1

一alpha:0

1

translatorerror

0.8

0.6

0.4

Average t

0.2

0

0

20

40

60

80

Contributions. The first contribution of this work is formalizing and analyzing a model of UMT. As an initial work, its value is in the opportunities which it opens for further work more than the finality and tightness/generality of its bounds. Our model applies even to low-resource source languages with massive domain gap and linguistic distance. We emphasize that this work is only a first step in the theoretical analysis of UMT (indeed, there is little theoretical work on machine translation in general). Second, we exhibit two simple complementary models for which we prove that: (a) more complex languages require less common ground, and (b) data requirements may not be significantly greater than those of supervised translation (which tends to use less data than training a large LM). These findings may have implications for the quantity and type of communication data that is collected for deciphering animal communication and for UMT more generally. They also give theoretical evidence that UMT can be successful and worth pursuing, in lieu of parallel (supervised) data, in the case of sufficiently complex languages. All of that said, we note that our sample complexity bounds are information theoretic, that is, they do not account for the computational complexity of optimizing the translator. Finally, animal communication aside, to the best of our knowledge this work is the first theoretical treatment of UMT, and may also shed light on translation between human languages.

Organization. The framework is formally described in Section 2, and is instantiated with models of language in Section 3 (proofs, experiments, and other details deferred to Appendices A to E). Key takeaways from this work are presented in Section 4. Related and future work is discussed in Appendices F and G. We illustrate how prompting LMs may give priors in Appendix H. Appendix I sketches

a generalization of our framework to the settings of lossy translation and infinite translator families. Lastly, Appendix J proves sample complexity bounds for the settings of supervised translation.

2 The General Framework

We use f : X ↪ →Y to denote a 1-1 function, in which f ( x ) = f ( x ′ ) for all x = x ′ . For S ⊆ X , we write f ( S ) := { f ( x ) | x ∈ S } . The indicator 1 P is 1 if the predicate P holds, and 0 otherwise. The uniform distribution over a set S is denoted by U ( S ) , and log = log 2 denotes base-2 logarithm.

̸

̸

Language and Prior. A source language is a distribution µ over a set of possible texts X . Similarly, a target language is a distribution ν over a set of possible texts Y . When clear from context, we associate each language with its corresponding set of possible texts. A prior distribution ρ over translations Y aims to predict the probability of observing each translation. One could naively take ρ = ν , but Appendix H describes how better priors can focus on the domain of interest. Intuitively, ρ ( y ) measures how 'plausible' a translation y is. For simplicity, we assume that X , Y ⊆ { 0 , 1 } ∗ are finite, non-empty sets of binary strings. Appendix I.2 discusses extensions to infinite sets.

Translators. A translator is a mapping f : X ↪ → Y . There is a known set of 1-1 functions { f θ : X ↪ →Y | θ ∈ Θ } with parameter set Θ . Since parameters are assumed to be known and fixed, we will omit them from the theorem statements and algorithm inputs, for ease of presentation. Like X , Y , the set Θ is assumed to be finite. Appendix I.1 considers translators that are not 1-1.

Divergence. A translator f θ and a distribution µ induce a distribution over y = f θ ( x ) , which we denote by f θ ◦ µ . The divergence between this distribution and ρ is quantified using the KullbackLeibler (KL) divergence,

Note that since D( θ ) = E x ∼ µ [ -1 m ∑ log ρ ( f θ ( x i )) ] -H( µ ) , and H( µ ) is a constant independent of θ , the MLE of Definition 1.1 approximately minimizes divergence.

Ground truth. In order to define semantic loss, we consider a ground-truth translator f ⋆ for some ⋆ ∈ Θ . We can then define the (ground-truth) translated language τ = f ⋆ ◦ µ over Y , obtained by taking f ⋆ ( x ) for x ∼ µ . This is similar to the standard realizability assumption, and some of our bounds resemble Occam bounds with training labels y i = f ⋆ ( x i ) . Of course, the ground-truth translator ⋆ is not known to the unsupervised learning algorithm. In our setting, we further require that ground-truth translations never have 0 probability under ρ :

Semantic loss. The semantic loss of a translator is defined with respect to a semantic difference function ℓ : Y × Y → [0 , 1] . This function, unknown to the learner, measures the difference between two texts from the target language Y , with ℓ ( y, y ) = 0 for all y . For a given semantic difference ℓ and ground-truth translator f ⋆ , we define the semantic loss of a translator f θ by

̸

Of particular interest to us is the semantic error err( · , · ) , obtained when ℓ is taken to be the 0-1 difference ℓ 01 = ( y, y ′ ) = 1 for all y ′ = y . Note that since any semantic difference ℓ is upper bounded by 1 , the semantic error upper-bounds any other semantic loss L . That is,

̸

Section 3 analyzes this error err( θ ) , which thus directly implies bounds on L ( θ ) .

3 Models of Language: Instantiating the General Framework

3.1 Random knowledge graphs

In this section, we define a model in which each text represents an edge between a pair of nodes in a knowledge graph. Both languages have knowledge graphs, with the source language weakly agreeing with an unknown subgraph of the target language.

We fix X = X × X = X 2 and Y = Y × Y = Y 2 with r := | X | and n := | Y | . The set of translators considered is all mappings from the r source nodes to the n target nodes, namely Θ XY = { θ : X ↪ → Y } and f θ ( ( u, v ) ) := ( θ ( u ) , θ ( v ) ) . The random knowledge graph is parametrized by the number of source nodes r , target node set Y , an edge density parameter p ∈ (0 , 1) representing the expected fraction of edges present in each graph, and an agreement parameter α ∈ (0 , 1] representing the correlation between these edges. In particular, α = 1 corresponds to the case where both graphs agree on all edges, and α = 0 corresponds to the case where edges in the graphs are completely independent. These parameters are unknown to the learner, who only knows X and Y (and thus X = X 2 , Y = Y 2 ).

Definition 3.1 (Random knowledge graph) . For a natural number r ≤ | Y | , the KG = KG( Y, r, p, α ) model determines a distribution over sets T, P ⊆ Y (which determine distributions ρ and µ ). The sets T and P are sampled as follows:

Set P ⊆ Y is chosen by including each edge y ∈ Y with probability p , independently.

Set S ⊆ Y of size | S | = r is chosen uniformly at random.

Set T ⊆ S 2 is chosen as follows. For each edge y ∈ S 2 , independently,

(a) With probability α , y ∈ T if and only if y ∈ P .

(b) With probability 1 -α , toss another p -biased coin and add y to T if it lands on 'heads'; that is, y ∈ T with probability p , independently.

It is easy to see that T ⊆ S 2 and P ⊆ Y 2 marginally represent the edges of Erd˝ os-Rényi random graphs G r,p and G n,p , respectively. Moreover, the event that y ∈ T is positively correlated with y ∈ P : for each y ∈ S 2 , since with probability α > 0 they are identical and otherwise they are independent. Formally, the equations below describe the probability of y ∈ T for each y ∈ S 2 after we fix S and choosing T ⊆ S 2 . Letting q := (1 -p ) , for each y ∈ S 2 :

̸

The last equality, shows that the probability of excluding a random y ∈ T from P is smaller than the probability of excluding a random 'incorrect translation' y ′ = y , Pr[ y ′ / ∈ P ] = q > (1 -α ) q .

We now describe how ρ, τ, are determined from T, P and how µ, ⋆ may be chosen to complete the model description. The ground-truth target translated distribution τ := U ( T ) is uniform over T . The prior ρ is uniform over P , and then 'smoothed' over the rest of the domain Y . Formally,

The ground-truth translator ⋆ ∈ Θ is obtained by sampling a uniformly random ⋆ : X ↪ → S . Lastly, we take µ = U ( f -1 ⋆ ( T )) , which agrees with the definition of τ . 8

Next, we state the main theorem for this model, formalizing Theorem 1.2 from the introduction.

Theorem 3.2 (Translatability in the KG model) . Fix any m ≥ 1 , ∅ ̸ = S ⊆ Y, δ, α, p ∈ (0 , 1) , and let r := | S | , n := | Y | , q = 1 -p . Then, with probability ≥ 1 -δ over T, P from KG( S, Y, p, α ) ,

where ˆ θ = MLE ρ ( x 1 , x 2 , . . . , x m ) is from Definition 1.1. Simply, for p < 0 . 99 , with probability ≥ 0 . 99 ,

The proof, given in Appendix D, requires generalizing our theory to priors that have full support. Experimental validation of the theorem is described in Appendix E.

3.2 Common nonsense model

We next perform a 'smoothed analysis' of arbitrary LMs µ, ρ that are uniform over sets that share a small amount of randomness, i.e., a small common random set has been removed from both. This shared randomness captures the fact that some texts are implausible in both languages and that this set has some complex structure determined by the laws of nature, which we model as random.

The α -common-nonsense distribution is a meta-distribution over pairs ( µ, ρ ) which themselves are uniform distributions over perturbed versions of P, T . This is inspired by Smoothed Analysis [36]. Recall that U ( S ) denotes the uniform distribution over the set S .

Definition 3.3 (Common nonsense) . The α -common-nonsense distribution D P,T α with respect to nonempty sets T ⊆ P ⊆ Y is the distribution over ( ρ = U ( P ∩ S ) , τ = U ( T ∩ S ) ) where S ⊆ Y is formed by removing each y ∈ Y with probability α , independently. 9

To make this concrete in terms of a distribution µ on X , for any ground-truth translator f ⋆ : X ↪ →Y , we similarly define a distribution D P,T α,⋆ over ( µ, ρ ) where µ := U ( f -1 ⋆ ( T ∩ S )) is the uniform distribution over the subset of X that translates into τ . Wenowstate the formal version of Theorem 1.3.

Theorem 3.4 (Translatability in the CN model) . Let { f θ : X ↪ →Y | θ ∈ Θ } a family of translators, ⋆ ∈ Θ , α, δ ∈ (0 , 1 / 2] , T ⊆ P ⊆ Y , and m ≥ 1 . Then with probability ≥ 1 -δ , MLE run on ρ and m ≥ 1 iid samples from µ outputs ˆ θ with,

Note that the probability is over both ( µ, ρ ) drawn from D P,T α,⋆ , and the m iid samples from µ . More simply, with probability ≥ 0 . 99 ,

When the amount of shared randomness α is a constant, then this decreases asymptotically like the bound of supervised translation (Theorem J.1) up until a constant, similar to Theorem 3.2. For very large m , each extra bit describing the translator (increase by 1 in log | Θ | ) amounts to a constant number of mistranslated x 's out of all X . The proof is deferred to Appendix C.

We also prove the following lower-bound that is off by a constant factor of the upper bound.

Theorem 3.5 ( CN lower-bound) . There exists constants c 1 , c 2 ≥ 1 such that: for any set T ⊆ Y , for any m ≥ 1 , any α ∈ (0 , 1 / 2] , and any Θ with c 1 ≤ log | Θ | ≤ α min( m, | T | ) , there exists Θ ′ of size

| Θ ′ | ≤ | Θ | such that, for any P ⊇ T and any algorithm A ρ : X m → Θ ′ , with probability ≥ 0 . 99 over ⋆ ∼ U (Θ ′ ) and ( µ, ρ ) drawn from D P,T α,⋆ and x 1 , . . . , x m ∼ µ ,

The only purpose of Θ in the above theorem is to upper-bound the description length of translators, as we replace it with an entirely different (possibly smaller ) translator family Θ ′ that still has the lower bound using log | Θ | ≥ log | Θ ′ | . Since U (Θ ′ ) is the uniform distribution over Θ ′ , the ground-truth classifier is uniformly random from Θ ′ . A requirement of the form log | Θ | = O ( α min( m, | T | ) ) is inherent as otherwise one would have an impossible right-hand side error lower-bound greater than 1, though the constants could be improved.

The proof of this theorem is given in Appendix C.2, and creates a model with O (log n ) independent 'ambiguities' that cannot be resolved, with high probability over S, x 1 , x 2 , . . . , x m . Experimental validation of the theorem is described in Appendix E.

4 Discussion

We have given a framework for unsupervised translation and instantiated it in two stylized models. Roughly speaking, in both models, the error rate is inversely related to the amount of samples, common ground, and the language complexity. The first two relations are intuitive, while the last is perhaps more surprising. All error bounds were information-theoretic , meaning that they guarantee a learnable accurate translator, but learning this translator might be computationally intensive.

In both models, the translators are restricted . In the knowledge graph, the translators must operate node-by-node following an assumed compositional language structure. 10 In the common nonsense model, the restriction is based on the translator description bit length log | Θ | . To illustrate how such restrictions can be helpful, consider block-by-block translators which operate on limited contexts (e.g., by paragraph). Consider again the hypothetical example of Figure 1. Suppose the three texts are outputs of three translators Θ = { A,B,C } . Let us suppose that translator A always produces accurate and natural translations, and further that all translators work paragraph-by-paragraph, as modern translation algorithms operate within some limited context window. In fact, one can imagine the translators of different paragraphs as a set of isolated adversaries where each adversary is trying to mistranslate a paragraph, knowing the ground-truth translation of their paragraph, while attempting to maintain the plausibility of the entire translation. If only the first-paragraph adversary mistranslates reef to ocean basin , then the translation lacks coherence and is unlikely. If the adversaries are in cahoots and coordinate to all translate reef to ocean basin , they would generate: Have you seen mom? I just returned from the ocean basin. At the basin, there were a lot of sea turtles. which has low probability ≈ 10 -25 , presumably because encoded in GPT-3's training data is the knowledge that there are no turtles deep in the ocean near the basin. While the adversary could also decide to change the word turtle to something else when it appears near basin , eventually it would get caught in its 'web of deceit.' The intuition is that, across sufficiently many translations, the prior will not 'rule out' the ground-truth translations while very incorrect translators will be ruled out.

Judging success. Our analysis sheds some light on whether it is even possible to tell if translation without parallel data (UMT) is successful. A positive sign would be if millions of translations are fluent English accounts that are consistent over time across translations. In principle, however, this is what LM likelihood should measure (excluding consistencies across translations which sufficiently powerful LMs may be able to measure better than humans). We also considered a statistical distance (KL divergence) between the translations f ˆ θ ( x ) for x ∼ µ and the prior y ∼ ρ , and µ could be estimated given enough samples. If this distance is close to zero, then one can have predictive accuracy regardless of whether the translations are correct. This raises a related philosophical quandary: a situation in which two beings are communicating via an erroneous translator, but both judge the conversation to be natural.

Acknowledgments and Disclosure of Funding

We thank Madhu Sudan, Yonatan Belinkov and the entire Project CETI team, especially Pratyusha Sharma, Jacob Andreas, Gašper Beguš, Michael Bronstein, and Dan Tchernov for illuminating discussions. This study was funded by Project CETI via grants from Dalio Philanthropies and Ocean X; Sea Grape Foundation; Rosamund Zander/Hansjorg Wyss, Chris Anderson/Jacqueline Novogratz through The Audacious Project: a collaborative funding initiative housed at TED.

References

S. Ahyong, C.B. Boyko, N. Bailly, J. Bernot, R. Bieler, S.N. Brandão, M. Daly, S. De Grave, S. Gofas, F. Hernandez, L. Hughes, T.A. Neubauer, G. Paulay, W. Decock, S. Dekeyzer, L. Vandepitte, B. Vanhoorne, R. Adlard, S. Agatha, K.J. Ahn, N. Akkari, B. Alvarez, V. Amorim, A. Anderberg, G. Anderson, S. Andrés Sánchez, Y. Ang, D. Antic, L.S.. Antonietto, C. Arango, T. Artois, S. Atkinson, K. Auffenberg, B.G. Baldwin, R. Bank, A. Barber, J.P. Barbosa, I. Bartsch, D. Bellan-Santini, N. Bergh, A. Berta, T.N. Bezerra, S. Blanco, I. Blasco-Costa, ..., and A. Zullini. World register of marine species (worms). = https://www.marinespecies.org, 2022. URL https://www.marinespecies.org . Accessed: 2022-10-22.

Jacob Andreas, Gašper Beguš, Michael M. Bronstein, Roee Diamant, Denley Delaney, Shane Gero, Shafi Goldwasser, David F. Gruber, Sarah de Haas, Peter Malkin, Nikolay Pavlov, Roger Payne, Giovanni Petri, Daniela Rus, Pratyusha Sharma, Dan Tchernov, Pernille Tønnesen, Antonio Torralba, Daniel Vogt, and Robert J. Wood. Toward understanding the communication in sperm whales. iScience , 25(6):104393, 2022. ISSN 2589-0042. doi: https://doi.org/10.1016/ j.isci.2022.104393. URL https://www.sciencedirect.com/science/article/ pii/S2589004222006642 .

Emily Anthes. The animal translators. The New York Times , Aug 2022. URL https://www.nytimes.com/2022/08/30/science/ translators-animals-naked-mole-rats.html .

Mikel Artetxe, Gorka Labaka, and Eneko Agirre. A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 789-798, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18-1073. URL https://aclanthology.org/P18-1073 .

Mikel Artetxe, Gorka Labaka, and Eneko Agirre. Unsupervised neural machine translation, a new paradigm solely based on monolingual text. Proces. del Leng. Natural , 63:151-154, 2019. URL http://journal.sepln.org/sepln/ojs/ojs/index.php/pln/ article/view/6107 .

László Babai, Paul Erdo ˝ s, and Stanley M. Selkow. Random graph isomorphism. SIAM Journal on Computing , 9(3):628-635, 1980. doi: 10.1137/0209047. URL https://doi.org/10. 1137/0209047 .

Christos Baziotis, Barry Haddow, and Alexandra Birch. Language model prior for low-resource neural machine translation. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP2020, Online, November 16-20, 2020 , pages 7622-7634. Association for Computational Linguistics, 2020. doi: 10.18653/v1/2020.emnlp-main.615. URL https://doi.org/10. 18653/v1/2020.emnlp-main.615 .

Marina Bedny, Jorie Koster-Hale, Giulia Elli, Lindsay Yazzolino, and Rebecca Saxe. There's more to 'sparkle' than meets the eye: Knowledge of vision and light verbs among congenitally blind and sighted individuals. Cognition , 189:105-115, 2019. ISSN 0010-0277. doi: https: //doi.org/10.1016/j.cognition.2019.03.017. URL https://www.sciencedirect.com/ science/article/pii/S0010027719300721 .

Mélissa Berthet, Camille Coye, Guillaume Dezecache, and Jeremy Kuhn. Animal linguistics: a primer. Biological Reviews , n/a(n/a), 2022. doi: https://doi.org/10.1111/brv.12897. URL https://onlinelibrary.wiley.com/doi/abs/10.1111/brv.12897 .

Thorsten Brants, Ashok C. Popat, Peng Xu, Franz Josef Och, and Jeffrey Dean. Large language models in machine translation. In Jason Eisner, editor, EMNLP-CoNLL 2007, Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, June 28-30, 2007, Prague, Czech Republic , pages 858-867. ACL, 2007. URL https://aclanthology.org/D07-1090/ .

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, ..., and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems , volume 33, pages 1877-1901. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/ file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf .

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, ..., and Noah Fiedel. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 , 2022. URL https://arxiv.org/abs/ 2204.02311 .

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Benjamin Doerr. Probabilistic tools for the analysis of randomized optimization heuristics. In Natural Computing Series , pages 1-87. Springer International Publishing, nov 2019. doi: 10.1007/978-3-030-29414-4_1. URL https://doi.org/10.1007% 2F978-3-030-29414-4_1 .

Daniel Edmiston, Phillip Keung, and Noah A. Smith. Domain mismatch doesn't always prevent cross-lingual transfer learning. In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Jan Odijk, and Stelios Piperidis, editors, Proceedings of the Thirteenth Language Resources and Evaluation Conference, LREC 2022, Marseille, France, 20-25 June 2022 , pages 892-899. European Language Resources Association, 2022. URL https://aclanthology.org/2022.lrec-1.94 .

Shane Gero, Hal Whitehead, and Luke Rendell. Individual, unit and vocal clan level identity cues in sperm whale codas. Royal Society Open Science , 3(1):150372, 2016.

Oded Goldreich, Brendan Juba, and Madhu Sudan. A theory of goal-oriented communication. J. ACM , 59(2):8:1-8:65, 2012. doi: 10.1145/2160158.2160161. URL https://doi.org/ 10.1145/2160158.2160161 .

Jesse Michael Han, Igor Babuschkin, Harrison Edwards, Arvind Neelakantan, Tao Xu, Stanislas Polu, Alex Ray, Pranav Shyam, Aditya Ramesh, Alec Radford, and Ilya Sutskever. Unsupervised neural machine translation with generative language models only. CoRR , abs/2110.05448, 2021. URL https://arxiv.org/abs/2110.05448 .

Zhiwei He, Xing Wang, Rui Wang, Shuming Shi, and Zhaopeng Tu. Bridging the data gap between training and inference for unsupervised neural machine translation. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022 , pages 6611-6623. Association for Computational Linguistics, 2022. doi: 10.18653/v1/2022.acl-long.456. URL https://doi.org/10.18653/v1/ 2022.acl-long.456 .

Yaofang Hu, Wanjie Wang, and Yi Yu. Graph matching beyond perfectly-overlapping erd˝ osrényi random graphs. Stat. Comput. , 32(1):19, 2022. doi: 10.1007/s11222-022-10079-1. URL https://doi.org/10.1007/s11222-022-10079-1 .

Brendan Juba and Madhu Sudan. Universal semantic communication I. In Cynthia Dwork, editor, Proceedings of the 40th Annual ACM Symposium on Theory of Computing, Victoria, British Columbia, Canada, May 17-20, 2008 , pages 123-132. ACM, 2008. doi: 10.1145/ 1374376.1374397. URL https://doi.org/10.1145/1374376.1374397 .

Yunsu Kim, Miguel Graça, and Hermann Ney. When and why is unsupervised neural machine translation useless? In Mikel L. Forcada, André Martins, Helena Moniz, Marco Turchi, Arianna Bisazza, Joss Moorkens, Ana Guerberof Arenas, Mary Nurminen, Lena Marg, Sara Fumega, Bruno Martins, Fernando Batista, Luísa Coheur, Carla Parra Escartín, and Isabel Trancoso, editors, Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, EAMT 2020, Lisboa, Portugal, November 3-5, 2020 , pages 35-44. European Association for Machine Translation, 2020. URL https://aclanthology.org/2020. eamt-1.5/ .

Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ranzato. Unsupervised machine translation using monolingual corpora only. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings . OpenReview.net, 2018. URL https://openreview.net/ forum?id=rkYTTf-AZ .

Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ranzato. Phrase-based & neural unsupervised machine translation. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun'ichi Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 4, 2018 , pages 5039-5049. Association for Computational Linguistics, 2018. URL https://aclanthology.org/D18-1549/ .

Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ranzato. Phrase-based & neural unsupervised machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 5039-5049, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1549. URL https://aclanthology.org/D18-1549 .

Lorenzo Livi and Antonello Rizzi. The graph matching problem. Pattern Analysis and Applications , 16:253-283, 08 2013. doi: 10.1007/s10044-012-0284-8.

André Lynum, Erwin Marsi, Lars Bungum, and Björn Gambäck. Disambiguating word translations with target language models. In Petr Sojka, Ales Horák, Ivan Kopecek, and Karel Pala, editors, Text, Speech and Dialogue - 15th International Conference, TSD 2012, Brno, Czech Republic, September 3-7, 2012. Proceedings , volume 7499 of Lecture Notes in Computer Science , pages 378-385. Springer, 2012. doi: 10.1007/978-3-642-32790-2\_46. URL https://doi.org/10.1007/978-3-642-32790-2_46 .

Kelly Marchisio, Kevin Duh, and Philipp Koehn. When does unsupervised machine translation work? In Loïc Barrault, Ondrej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costajussà, Christian Federmann, Mark Fishel, Alexander Fraser, Yvette Graham, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno-Yepes, Philipp Koehn, André Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa, and Matteo Negri, editors, Proceedings of the Fifth Conference on Machine Translation, WMT@EMNLP 2020, Online, November 19-20, 2020 , pages 571-583. Association for Computational Linguistics, 2020. URL https://aclanthology.org/2020.wmt-1.68/ .

Antonio Valerio Miceli Barone. Towards cross-lingual distributed representations without parallel text trained with adversarial autoencoders. In Proceedings of the 1st Workshop on Representation Learning for NLP , pages 121-126, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18653/v1/W16-1614. URL https: //aclanthology.org/W16-1614 .

Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning . MIT press, 2018.

Surangika Ranathunga, En-Shiun Annie Lee, Marjana Prifti Skenduli, Ravi Shekhar, Mehreen Alam, and Rishemjit Kaur. Neural machine translation for low-resource languages: A survey. arXiv preprint arXiv:2106.15115 , 2021.

Sujith Ravi and Kevin Knight. Deciphering foreign language. In Dekang Lin, Yuji Matsumoto, and Rada Mihalcea, editors, The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference, 19-24 June, 2011, Portland, Oregon, USA , pages 12-21. The Association for Computer Linguistics, 2011. URL https://aclanthology.org/P11-1002/ .

Baptiste Rozière, Jie Zhang, François Charton, Mark Harman, Gabriel Synnaeve, and Guillaume Lample. Leveraging automated unit tests for unsupervised code translation. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022. URL https://openreview.net/forum?id=cmt-6KtR4c4 .

Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 4222-4235, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-main.346. URL https://aclanthology.org/ 2020.emnlp-main.346 .

Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. MASS: masked sequence to sequence pre-training for language generation. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA , volume 97 of Proceedings of Machine Learning Research , pages 5926-5936. PMLR, 2019. URL http://proceedings.mlr.press/ v97/song19d.html .

Daniel A. Spielman and Shang-Hua Teng. Smoothed analysis: An attempt to explain the behavior of algorithms in practice. Commun. ACM , 52(10):76-84, oct 2009. ISSN 00010782. doi: 10.1145/1562764.1562785. URL https://doi.org/10.1145/1562764. 1562785 .

Chau Tran, Shruti Bhosale, James Cross, Philipp Koehn, Sergey Edunov, and Angela Fan. Facebook ai's WMT21 news translation task submission. In Loïc Barrault, Ondrej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Alexander Fraser, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno-Yepes, Philipp Koehn, Tom Kocmi, André Martins, Makoto Morishita, and Christof Monz, editors, Proceedings of the Sixth Conference on Machine Translation, WMT@EMNLP 2021, Online Event, November 10-11, 2021 , pages 205-215. Association for Computational Linguistics, 2021. URL https://aclanthology.org/ 2021.wmt-1.19 .

Ludwig Wittgenstein. Philosophical Investigations . Basil Blackwell, Oxford, 1953. ISBN 0631119000.

Klaus Zuberbühler. Syntax and compositionality in animal communication. Philosophical Transactions of the Royal Society B , 375(1789):20190062, 2020.

A Translator Revisions and Plausible Ambiguities

Even though ⋆ is unknown, it will be convenient to define, for each θ ∈ Θ , a revision of f ⋆ which is the permutation π ⋆ θ : Y ↪ →Y between f θ and f ⋆ , π ⋆ θ ( y ) := f θ ( f -1 ⋆ ( y )) . 11 We write π θ = π ⋆ θ when ⋆ is clear from context, and π ⋆ ( y ) ≡ y is the identity revision.

Note that the divergence D( θ ) and semantic loss L ( θ ) can equivalently be defined using revisions,

To relate divergence and loss, it is helpful to define a notion of plausible ambiguities which are θ whose revisions π θ ( y ∼ τ ) are not too unlikely under the prior. These may also be of independent interest as they capture certain types of ambiguities that can only be resolved with supervision.

Definition A.1 ( γ -Plausible ambiguities) . For any γ ∈ [0 , 1] , ⋆ ∈ Θ , and distributions τ, ρ over Y , the set of γ -plausible ambiguities A γ = A ⋆,τ,ρ γ ⊆ Θ is:

For example, left ↔ right would constitute a γ -plausible ambiguity if one could swap the two words, making a ≤ γ fraction of the translations have 0 probability. Such a revision would have low loss if such swaps are considered similar in meaning. Condition 2.1 is equivalent to ⋆ ∈ A 0 .

The quantity of interest is ε γ , the maximum loss of any γ -plausible ambiguity. 12 Next, we prove that the loss of the translator output by the Maximum Likelihood Estimator is not greater than ε γ given m ≥ 1 γ ln | Θ | δ examples.

Theorem A.2. Let µ, ρ be probability distributions over X , Y , resp., and f ⋆ satisfy Condition 2.1: Pr µ [ ρ ( f ⋆ ( x )) = 0] = 0 . Fix any δ ∈ (0 , 1) , m ≥ 1 , and let γ ≥ 1 m ln | Θ | δ . Then,

where ˆ θ = MLE ρ ( x 1 , . . . , x m ) and ε γ is from Definition A.1.

Since ε γ is non-decreasing in γ , as the number of examples increases the loss bound ε γ decreases to approach ε 0 . This bound and its proof in Appendix A are analogous to the realizable Occam bound of supervised classification, see Appendix J.

Proof of Theorem A.2. MLE minimizes ˆ v ( θ ) := 1 m ∑ i ≤ m log 1 ρ ( π θ ( y i )) over θ ∈ Θ , where y i = f θ ( x i ) . By the realizable prior assumption ˆ v ( ⋆ ) < ∞ . Thus, for the algorithm to fail, there must be a 'bad' ˆ θ ∈ B := { θ ∈ Θ | L ( θ ) > ε γ } with ˆ v ( θ ) ≤ ˆ v ( ⋆ ) < ∞ . If ρ ( π θ ( y i )) = 0 for any i , then ˆ v ( π θ ) = ∞ . Note that by Definition A.1, B ∩ A γ = ∅ , thus for all θ ∈ B : Pr[ ρ ( π θ ( y )) = 0] > γ and,

By the union bound over θ ∈ B , Pr[ ∃ θ ∈ B ˆ v ( θ ) < ∞ ] < δ since | B | ≤ | Θ | .

B The Tree-based Model

B.1 A tree-based probabilistic model of language

The last model, which we view as secondary to the previous two, is a simpler version of the common nonsense model. It can be viewed as a 'warm-up' leading up to that more general model, and we

̸

Sharks

Ocean

play

eat

swim

you

not

whales

sunfish

basins

waves

harmonicas

pool

hockey

sharks

quickly

salmon

slow

alone

fast

dance

cook

mind

%

sleep

worry

burp

daydream

chew

hatch

dive

hunt

join

quickly

change

crest

refract

break

hope it helps illuminate that model by instantiating the language therein with a simple tree-based syntax.

In the tree-based model, the nodes of a tree are labeled with random words, and plausible texts (according to the prior ρ ) correspond to paths from root to leaf. A translated language τ (or equivalently, a source language µ ) is then derived from root-to-leaf paths in a random subtree H ⊆ G . We prove that, with high probability, a prior ρ and translated language τ sampled by this process satisfy Condition 2.1 and semantic error ε γ = O ( 1 / min( m,a n ) ) . Therefore, MLE yields a semantically accurate translator for the sampled language with high probability, for sufficiently large n .

The random tree language ( RT ) model is a randomized process for generating a tree-based source language µ , translated language τ , and prior ρ . It is parameterized by a finite vocabulary set W , depth n ∈ N , and arities a, b ∈ N such that 1 ≤ a ≤ b ≤ |W| / 4 . For simplicity, the possible target texts are taken to be the set of all possible n -grams over W , namely, Y := W n . Again for simplicity, we also assume |X| = |Y| so that each f θ : X ↪ →Y is a bijection. To generate an RT , first a full b -ary, depth n +1 tree G is constructed. Labeled edges shall correspond to words, and paths shall correspond to texts (sentences), as follows:

Starting from the root node and proceeding in level-order traversal, for each node v : sample b words w 1 , . . . , w b ∈ W uniformly at random and without replacement; label each of v 's child edges with one of the sampled words.

The labels on each path from the root to a leaf ( y 1 , . . . , y n ) corresponds to a plausible text , giving a set of plausible texts

A subtree H ⊆ G is obtained by sampling uniformly at random a out of b of the children at each level of the tree, in level-order traversal. The set of translated texts is analogously defined as

The prior ρ = U ( P ) is uniform over plausible texts P , while µ = is uniform over f -1 ⋆ ( T ) . We let ( µ, ρ ) ∼ RT( W , n, a, b ) denote the sampling of a prior and translated language obtained via this randomized process. When the parameters W , n, a, b are clear from context we simply write RT .

Note that Θ and f ⋆ ∈ Θ may be arbitrary, and by definition of τ , µ = U ( f -1 ⋆ ( T )) is uniform over T 's preimage. Since we assumed |X| = |Y| above, f ⋆ is invertible.

Next, we will argue that a random tree language sampled in this process satisfies realizability (Condition 2.1) and Definition A.1 with appropriate choice of parameters. While we make no assumptions on Θ , it is important that the plausible texts P are sampled after the possible permutations (ambiguities) Π are specified. Otherwise, if an adversary were able to choose a permutation π : Y →

Y based on P , then they could arbitrarily permute P , resulting in a permutation π with high expected loss but no change in likelihood according to the prior ρ -thereby violating Definition A.1.

You can think of Theorem B.1 as pointing to the required text length n and an upper bound on the number of parameters | Θ | for which Condition 2.1 and Definition A.1 hold with high probability. As we know from Theorem A.2, and state clearly next, these in turn imply translatability of a random tree language.

Theorem B.1 (Translatability in the RT model) . Fix any m ≥ 1 , δ ∈ (0 , 1) vocabulary set W , and tree arities a ≤ b ≤ |W| / 4 . Then, for any Θ and any ⋆ ∈ Θ , with probability at least 1 -δ over ( µ, ρ ) ∼ RT( W , n, a, b ) and iid samples x 1 , x 2 , . . . , x m ∼ µ ,

where ˆ θ = MLE ρ ( x 1 , x 2 , . . . , x m ) and MLE is from Definition 1.1

Note that the probability in the corollary is over both ( µ, ρ ) ∼ RT and the m iid training samples. Again, note how the first term is similar to the 1 m log | Θ | δ term from realizable supervised learning (see Theorem J.1), and the second term is additional due to the unsupervised case. The proof, given in Appendix B, uses Theorem A.2 and a lemma stating that ε γ ≤ 16 γ for γ that are not too small. The main challenge is that there are dependencies between the paths, so one cannot directly use standard concentration inequalities.

To better understand Theorem B.1, let us consider two possible families of translators and suppose we are trying to apply Theorem B.1 to get bounds on the sample complexity m needed to translate ( µ, ρ ) ∼ RT( W , n, a, b ) with small constant loss and small constant failure probability.

First, since |X| = |Y| = |W| n , the above bound is meaningless (bigger than 1) if Θ is the family of all translators, as the set of all translators has log( |X| !) = O ( |W| n · n log |W| ) parameters which is much larger than a n . In other words, no free lunch .

On the other hand, let us consider the family of word-for-word translators Θ w, which work by translating each word in a text separately, ignoring the surrounding context. The number of such translators is the same as the number of word-to-word permutations, i.e., | Θ w | = |W| ! . So Theorem B.1 gives a sample complexity bound of m = O (log W !) = O ( |W| log |W| ) . The number of words in the English language is about N ≈ 10 5 . The quantity a is equal to what is called the perplexity of µ , the effective number of words that may typically appear after a random prefix. For a constant a , the minimum text (or communication) length n = O (log N ) needed is thus logarithmic in the vocabulary size. While word-for-word translators are poor, this analysis still sheds some light on the possible asymptotic behavior (barring computational constraints) of translation, at least in a simplistic model.

Common tree analysis. The tree model helps us demonstrate the generality of the common nonsense model. Instead of the fixed arities a ≤ b in the tree model, one can consider an arbitrary language tree over plausible texts P and an arbitrary subset T ⊆ P corresponding to a subtree of ground-truth translations. The common nonsense model implies that if the sets are perturbed by removing, say, a α = 0 . 01 common fraction of translations, then with high probability MLE's error decreases at an ˜ O (1 / min( m, | T | )) rate. Note that Theorem 3.4 does not directly apply to the random tree LM because in that LM, the choice of which branches to include are not independent due to the fixed a -arity constraint.

B.2 Proof of Theorem B.1

The proof of Theorem B.1 follows from Lemma B.2 below. In this section, we will prove this lemma and then derive the theorem from it.

Lemma B.2 ( RT conditions) . Consider any vocabulary set W , tree arities a ≤ b ≤ |W| / 4 , tree depth n ∈ N . Then, for any δ ∈ (0 , 1) and γ ≥ 4 a -n ln(4 | Θ | /δ ) , with probability ≥ 1 -δ , ( µ, ρ ) give,

Moreover, any µ, ρ sampled from RT( W , n, a, b ) satisfy Pr x ∼ µ [ ρ ( f ⋆ ( x )) = 0] = 0 as needed for the realizability Condition 2.1.

The proof of Lemma B.2 requires two additional lemmas, stated and proved next. The first is a known variant of the Chernoff bound for so-called 0-negatively correlated random variables.

Lemma B.3 (Chernoff bound for 0-negatively correlated random variables) . Let Z 1 , . . . , Z n be 0-negatively correlated Boolean random variables, that is, they satisfy

Then, letting Z := ∑ n i =1 Z i it holds that

Lemma B.3 follows from [14, Theorem 1.10.24(a)] with δ := 1 / 2 , a i := 0 , b i := 1 and Y i := X i . That theorem, in turn, is simply Theorem 1.10.10 for 0 -negatively correlated random variables.

The second lemma used for the proof of Lemma B.2 is a combinatorial argument that we prove below.

̸

Lemma B.4. Given any π : Y ↪ →Y , it is possible to partition A = { y ∈ Y | π ( y ) = y } into four disjoint sets A = A 1 ∪ A 2 ∪ A 3 ∪ A 4 such that, for each i = 1 , 2 , 3 , 4 , the following two conditions hold:

Proof. of Lemma B.4. We will partition A greedily to achieve the two conditions. Begin with four empty sets A 1 = A 2 = A 3 = A 4 = ∅ . For each y ∈ A in turn, assign it to one of the 4 sets as follows.

Let i be the index of the set A i such that π ( y ) ∈ A i has already been assigned to. If π ( y )) has not yet been assigned, let i = 1 .

Similarly, if π ( y ) ∈ A j has been assigned, let j be its index otherwise j = 1 .

Let z be the first n -1 elements of x . Let k be the index of the set A k such that |{ y ∈ A i | y begins with z }| ≥ |W| / 2 if there is such a set, otherwise k = 1 . Note that there can be at most one such k because there are at most |W| 1 other elements beginning with z that have been assigned already.

Thus S = { 1 , 2 , 3 , 4 } \ { i, j, k } ̸ = ∅ and we can assign x to any (say the minimum) element in that set. By induction, we preserve the two properties stated in the lemma.

With Lemmas B.3 and B.4 in hand, we are ready to prove Lemma B.2.

Proof. of Lemma B.2. The realizability Condition 2.1, Pr x ∼ µ [ ρ ( f ⋆ ( x )) = 0] = 0 , follows immediately from the fact that ρ is uniform over P and that τ is supported on T ⊆ P . To prove the lemma, it suffices to show that for any γ ≥ 4 a -n ln(4 | Θ | /δ ) ,

̸

Note that err( θ ) = Pr y ∼ τ [ y = π θ ( y )] , and that ρ ( π θ ( y )) = 0 if and only if π θ ( y ) / ∈ P . Therefore, by the union bound over θ ∈ Θ , it suffices to show that, for each θ ∈ Θ ,

̸

̸

We will show that, more generally, for any permutation π : Y ↪ →Y . Equation (5) can be restated in terms of ambiguous texts A := { y ∈ Y | π ( y ) = y } and implausible ambiguities B := { y ∈ Y | π ( y ) / ∈ P } :

Using Lemma B.4, partition A into four disjoint sets A = A 1 ∪ A 2 ∪ A 3 ∪ A 4 such that for each i ∈ [4] , the following two conditions hold:

By the union bound, it suffices to show that,

because τ ( A ) = ∑ 4 i =1 τ ( A i ) , so if τ ( A ) > 16 γ then it must follow that τ ( A i ) > 16 γ/ 4 for some i . It suffices to show that this holds conditioned on any value of A i ∩ T :

Thus, fix any V ⊆ A i . We proceed by analyzing two cases, based on | V | versus | T | = a n . 13

Case 1: | V | ≤ 4 γ | T | . Then Equation (8) holds with probability 0, because conditioning on A i ∩ T = V , we have

Case 2: | V | > 4 γ | T | . For each v ∈ V we define a Boolean random variable Z v = 1 v ∈ B , i.e.,

We claim that the Z v 's satisfy the condition of Lemma B.3, which follows inductively from the fact any subset of Z v 's being 0 only makes it less likely for another Z v to be 0, because the way that the Z v 's are chosen is such that there are exactly b many 1 's among the Z v 's (and other symmetric random variables which we have not named).

Therefore, if we define Z := ∑ v ∈ V Z v and ζ := E [ Z | A i ∩ T = V ] , Lemma B.3 gives,

We conclude by bounding both sides of Equation (9) to obtain Equation (8).

For the right-hand side of eq. (9), we claim that Pr[ Z v = 1 | A i ∩ T = V ] ≥ 1 / 2 for each v ∈ V : Fix v and let

So | Q | = |W| and | Q ∩ P | = b . Equation (6) implies that π ( v ) / ∈ A i , and eq. (7) implies that | Q \ A i | ≥ |W| / 2 . Thus, there are ≥ |W| / 2 elements in Q \ A i that, by symmetry, are as likely to be in Q ∩ P as π ( v ) is, and Q ∩ P contains exactly b ≤ |W| / 4 elements, thus the conditional probability that π ( v ) is in Q ∩ P (equivalently, π ( v ) ∈ P ) is at most:

Since π ( v ) / ∈ P is equivalent to Z v = 1 , this means that Pr[ Z v = 1 | A i ∩ T = V ] ≥ 1 / 2 , and hence:

Because we are in the case that | V | > 4 γ | T | , we have

Therefore the right-hand side of eq. (9) satisfies

where the rightmost inequality holds because by our assumption that γ ≥ 4 a -n ln(4 | Θ | /δ ) .

For the left-hand side of eq. (9), note that

and therefore, using eq. (10), the left-hand side of eq. (9) satisfies

̸

We claim that V ∩ B ⊆ B ∩ T : Due to the conditional event, we have V ∩ B = A i ∩ T ∩ B ⊆ A ∩ T ∩ B . However, we argue that A ∩ T ∩ B ⊆ T ∩ B , i.e., that T ∩ B ⊆ A . Indeed, by definition, if y ∈ T ∩ B then y ∈ T but π ( y ) / ∈ P , and since P ⊇ T , this implies that π ( y ) = y ; that is, that y ∈ A , as needed.

We have just shown that V ∩ B ⊆ B ∩ T , and therefore

This shows that the left-hand side of eq. (9) upper bounds that of eq. (8), and concludes the proof.

Lastly, we prove Theorem B.1 using Lemma B.2

Proof. of Theorem B.1. Let γ be,

Lemma B.2 below shows that with probability ≥ 1 -2 δ/ 3 over µ, ρ , we have that ε γ ≤ 16 γ . Theorem A.2 implies that with probability ≥ 1 -δ/ 3 over x 1 , . . . , x m , we have err( ˆ θ ) ≤ ε γ . Thus, by the union bound, with probability ≥ 1 -δ we have

C Proofs for the Common Nonsense Model

C.1 Upper bound in the common nonsense model

C.1.1 Proof overview

To convey intuition, we think of the case of say δ = 99% , and we omit constants in the following discussion. Recall that Theorem A.2 asserts that with high probability, the learned translator ˆ θ has error at most ε γ for γ = log | Θ | / min( m, | T | ) ≥ log | Θ | /m . As such, the main technical challenge

in this proof is to show that, with high probability over the generated source language µ and the prior ρ , it holds that

By definition of ε γ , we ought to show that w.h.p over µ and ρ , any θ ∈ Θ with large semantic error must have many translations f θ ( x ) deemed implausible by ρ . Slightly more formally: Since any y ∈ Y is implausible ( ρ ( y ) = 0 ) only if y / ∈ S , a union bound over θ ∈ Θ means that is suffices to show that

̸

where the right inequality is by choice of γ := log | Θ | / min( m, | T | ) . The above inequality 'looks like' it could be proven by a Chernoff bound, but a closer look reveals a subtle flaw with this argument.

̸

To use a Chernoff bound, we'd first want to fix (i.e., condition on) each supp( µ ) := f -1 ⋆ ( S ∩ T ) and then use Chernoff over the conditional random variables 1 f θ ( x ) / ∈ S for each x ∈ supp( µ ) such that f θ ( x ) = f ⋆ ( x ) . Unfortunately, these conditional random variables are not independent. To see this, consider the case that f θ ( x ) = f ⋆ ( x ′ ) for two different x = x ′ . Then, since we are considering x ′ ∈ supp( µ ) , we have f θ ( x ) = f ⋆ ( x ′ ) ∈ S with probability 1 .

̸

To avoid this dependency, we prove a combinatorial lemma showing that it is possible to partition the set

̸

into three parts A = A 1 ∪ A 2 ∪ A 3 such that f θ ( A i ) ∩ f ⋆ ( A i ) = ∅ for each i ∈ [3] . This resolves the dependency issue demonstrated above. We then proceed by applying a Chernoff bound separately for each A i , which suffices since a union bound (over i ∈ [3]) loses only a constant factor in the upper-bound.

C.1.2 The full proof

The proof of Theorem 3.4 follows from the following main lemma. We first prove this lemma, and then show how the theorem follows from it.

Lemma C.1. Let α, δ ∈ (0 , 1) and T ⊆ P ⊆ Y . Then, for any γ ≥ 8 (1 -α ) ·| T | ln 4 | Θ | δ :

The proof of Lemma C.1 relies on a simple combinatorial proposition. This proposition is a special case of Lemma B.4, but since it is much simpler we give a self-contained proof.

̸

Proposition C.2. For any finite set Y and any π : Y ↪ →Y , it is possible to partition { y ∈ Y| π ( y ) = y } = A 1 ∪ A 2 ∪ A 2 into three sets such that, π ( A i ) ∩ A i = ∅ for i = 1 , 2 , 3 .

̸

Proof. Let S := { y ∈ Y | π ( y ) = y } . We proceed iteratively, dividing each (non-trivial) cycle of π separately into the three A i 's: Fix a cycle { s 1 , s 2 , . . . , s n } such that π ( s i ) = s i +1 and π ( s n ) = s 1 . If n is even we can just partition it into two sets: put the s i for even i 's into A 1 and odd i 's into A 2 . If n is odd, we can do the same except put the last element s n into A 3 .

Proof. of Lemma C.1. Note that the lemma holds trivially for any γ > 1 / 6 because we always have ε γ ≤ 1 . Assume that γ ≤ 1 / 6 . Let a := 6 α γ . The probabilities in this proof are over the choice of S ⊆ Y . It suffices to show that,

̸

because ρ ( π θ ( y )) = 0 whenever π θ ( y ) / ∈ S , thus A γ = { θ ∈ Θ | Pr τ [ π θ ( y ) / ∈ S ] ≤ γ } . The set S determines the perturbed sets ˜ T := T ∩ S and ˜ P := P ∩ S and the distributions τ = U ( ˜ T ) and ρ = U ( ˜ P ) .

Define β := 1 -α . Since E [ | ˜ T | ] = β | T | , a multiplicative Chernoff bound 14 gives

Thus, to show eq. (11), it suffices to show that

̸

By the union bound, to show eq. (12) it thus suffices to show that for any π : Y ↪ →Y ,

̸

̸

Fix any π : Y ↪ → Y . By Lemma C.2, we can partition { y ∈ Y | π ( y ) = y } = A 1 ∪ A 2 ∪ A 3 such that π ( A i ) ∩ A i = ∅ , and hence Pr τ [ π ( y ) = y ] = ∑ i τ ( A i ) . So if Pr τ [ π ( y ) = y ] > a , then τ ( A i ) > a/ 3 for some i . Therefore, it suffices to show that

̸

̸

With a union bound over i ∈ [3] , it suffices to show that for each i ∈ [3]

Thus, now in addition to fixing π , we fix i ≤ 3 , thus fixing A i . To continue, imagine we are picking S by first selecting A i ∩ S , and subsequently selecting S \ A i . We will show that Equation (14) holds when conditioning on each possible value for the first selection, that is, each possible A i ∩ S . Formally, we fix V ⊆ A i and condition on A i ∩ S = V , claiming that

First, observe that

therefore if | V | < βa | T | / 6 then Equation (15) holds with probability 0 (due to the first two events in the conjunction). Thus, we can assume that | V | ≥ βa | T | / 6 .

Note that τ ( V ) = τ ( A i ) , and that τ ( A i ) ≥ a/ 3 implies

therefore the left-hand side of Equation (15) is upper-bounded by

We conclude the proof by upper-bounding Equation (16) with a Chernoff bound. Consider the random variables Z y = 1 π ( y ) / ∈ S for y ∈ V . These random variables are independent by definition of α -common-nonsense. Furthermore, due to the fact that π ( A i ) ∩ A i = ∅ , they remain independent even when conditioning on the event A i ∩ S = V . By linearity of expectation,

Using the same Chernoff bound as above, we have

Noting that ∑ Z y / | V | = Pr y ∈ V [ π ( y ) / ∈ S ] , we conclude that that Equation (16) is upper-bounded by

This proves the inequality in Equation (15), thereby concluding the proof.

Finally, with the main technical lemma in hand, we prove Theorem 3.4.

Proof. of Theorem 3.4. The proof is very similar to that in Appendix B. First, the realizability Condition 2.1, Pr x ∼ µ [ ρ ( f ⋆ ( x )) = 0] = 0 , follows immediately from the fact that µ, ρ are uniform distributions with f ⋆ (supp( µ )) ⊆ supp( ρ ) which follows from the fact that T ⊆ P and the definitions of µ, ρ .

Let β := 1 -α and define γ by,

Lemma C.1 below shows that with probability ≥ 1 -2 δ/ 3 over µ, ρ , we have that ε γ ≤ 6 γ/α . Theorem A.2 implies that with probability ≥ 1 -δ/ 3 over x 1 , . . . , x m , we have err( ˆ θ ) ≤ ε γ . Thus, by the union bound, with probability ≥ 1 -δ we have

C.2 Lower bound in the common nonsense model

The proof of the lower bound works by creating log n candidate 'plausible ambiguities' and arguing that a constant fraction of survive the random removal of elements.

Proof. of Theorem 3.5. The constants c 1 , c 2 will also be determined through this proof to be large enough to satisfy multiple conditions defined below. No effort has been made to minimize the constants in this proof.

Let n := | Θ | .

We will lay out two a × b grids X ⊆ X and Y ⊆ Y , for:

For integer t , denote [ t ] := { 1 , 2 , . . . , t } . For i ∈ [ a ] , j ∈ [ b ] , choose distinct elements x ij ∈ X and y ij ∈ T . To ensure this is even possible, we must make sure ab ≤ | T | , which holds because we assumed log n ≤ α min( m, | T | ) thus,

Let X := { x ij | i ∈ [ a ] , j ∈ [ b ] } and Y := { y ij | i ∈ [ a ] , j ∈ [ b ] } . Let h : X \ X ↪ →Y\ Y be a fixed 1-1 mapping, say the lexicographically smallest. The parametrized translator family is defined by,

Clearly | Θ ′ | = 2 a ≤ n = | Θ | as needed. Let x = ( x 1 , x 2 , . . . , x m ) . It suffices to show:

for some constant c 3 sufficiently large, because we can set c 2 = 10 5 c 3 . The above equation is equivalent to the following two cases based on m :

In both cases, it will be convenient to notice that, for any z ≥ 2 , ⌊ z ⌋ ≥ z -1 ≥ z 2 . Since b ≥ 2 (because α ≤ 1 / 2 ), we therefore have

Case 1: 10 5 m>T . In this case 1 2 α ≤ b ≤ 1 α by Equation (19). We will show Equation (17). Now, consider the 'full rows':

These rows will be useful to consider because nothing has been removed from the entire row, no information about θ i has been revealed and (on average) one cannot achieve error < 1 / 2 on these examples, because one cannot distinguish between the two permutations on this row.

Note that the membership of different i, i ′ ∈ C ( S ) is independent since S is chosen independently, and by definition of C and S :

since (1 -α ) 1 /α is decreasing in α and α ≤ 1 / 2 . Thus, by multiplicative Chernoff bounds (specifically, Pr[ Z ≤ E [ Z ] / 2] ≤ e -E [ Z ] / 8 ),

for sufficiently large c 1 . Thus, Pr S [ | C ( S ) | > a/ 8] ≥ 0 . 999 . Let C ′ ( S, x ) ⊆ C ( S ) be those i which ˆ θ i = ⋆ i ,

̸

Clearly, for any algorithm and any C ( S ) , E x [ | C ′ ( S, x ) | | S ] = | C ( S ) | / 2 because no information whatsoever has been revealed about θ i for any i ∈ C . Thus, by the same Chernoff bound, we have:

for sufficiently large c 1 , because a ≥ c 1 . By the union bound over this and Equation (20),

Since each row i ∈ C ′ incurs b errors on examples x , one for each j because f ⋆ ( x ij ) ∈ S :

Now, a ≥ 1 2 log n for sufficiently large c 1 and as mentioned b ≥ 1 2 α . Thus,

This establishes Equation (17) as long as c 3 ≥ 128 .

It remains to prove Equation (18).

̸

Thus,

Case 2: 10 5 m ≤ T . In this case | T | 2 α 10 5 m ≤ b ≤ | T | α 10 5 m by Equation (19). Next, consider the set of rows with at least 1/2 of the elements in S :

̸

Intuitively, any row i ∈ D ( S ) is 'dangerous' in the sense that if ˆ θ i = ⋆ i , then it causes errors on b/ 2 different x 's in the support of µ , i.e., for which f ⋆ ( x ij ) ∈ S . Observe that E [ | D ( S ) | ] ≥ a/ 2 since each size s = ∣ ∣ { j ∈ [ b ] | y ij ∈ S } ∣ ∣ ≥ b/ 2 is at least as likely as the size b -s , since α ≤ 1 / 2 . And also, membership of i, i ′ ∈ D ( S ) since S is independent. Thus, by the same Chernoff bound as above, for sufficiently large c 1 ,

Let -θ := ( -θ 1 , -θ 2 , . . . , -θ a ) . This makes it convenient to define the giveaways G ( S ) ⊆ X to be,

These are the points x ij which we might observe f ⋆ ( x ij ) which would imply that θ i = ⋆ i (and not its negative). Also let,

(Note that if for a give row i , we do not have any x ij ∈ ˆ G ( S, x ) , then we have no information about θ i . As a preview to what is to come, we now argue that with high probability | ˆ G ( S, x ) | < a/ 8 which will mean that, if | D ( S ) | > a/ 4 , then we have no information about θ i for at least a/ 8 of the rows i ∈ D ( S ) .)

For any fixed i, j , observe that Pr[ x ij ∈ G ( S )] = α (1 -α ) so E [ | G ( S ) | ] = α (1 -α ) ab ≤ αab . By the Chernoff bound that Pr[ Z ≥ 2 E [ Z ]] ≤ e -E [ Z ] / 3 ,

for sufficiently large c 1 . (We have used the fact that the above probability is smaller than if E [ | G ( S ) | ] were actually αab .)

Also, E S [ | T ∩ S | ] ≥ | T | / 2 since α ≤ 1 / 2 . So, by the Chernoff bound Pr[ Z ≤ E [ Z ] / 2] ≤ e -E [ Z ] / 8 ,

for sufficiently large c 1 since | T | ≥ log n ≥ c 1 .

Thus, by the union bound:

Thus, using Markov's inequality in the second line below,

By the union bound over the above and Equation (22), since Pr[ E ] ≤ Pr[ E | F ] + Pr[ ¬ F ]

Also,

Finally, since

Let

Clearly | F ( S, x ) | ≥ | D ( S ) | - | ˆ G ( S, x ) | because each x ∈ ˆ G ( S, x ) can remove at most one row from D ( S ) . Thus,

̸

F ( S, x ) will function exactly like C in the analysis above of Equation (17). We repeat this analysis for completeness, replacing C by F . Let F ′ ( S, x ) ⊆ F ( S, x ) be those i which ˆ θ i = ⋆ i ,

̸

For any algorithm and any F ( S, x ) , E x [ | F ′ ( S, x ) | | S ] = | F ( S, x ) | / 2 because no information whatsoever has been revealed about θ i for any i ∈ F . Thus, by the same Chernoff bound, we have:

for sufficiently large c 1 . By the union bound over this and Equation (23),

Since each row i ∈ F ′ incurs ≥ b/ 2 errors on examples x by definition of F ′ and D , since F ′ ⊆ D and thus at least b/ 2 errors on j for which f ⋆ ( x ij ) ∈ S . Thus,

Now, a ≥ 1 2 log n for sufficiently large c 1 and we also have b ≥ | T | 2 α 10 5 m by Equation (19) since | T | α 10 5 m ≥ 2 since α ≤ 1 / 2 and we are in the case where 10 5 m ≤ | T | . Thus,

This establishes Equation (18) for c 3 ≥ 256 × 10 5 .

D Proofs for random knowledge graph

Our goal in this section is to prove Theorem 3.2. The proof is based on the following main lemma. We first state and prove this lemma, and then derive the theorem from it. Recall that the sets T, P ⊆ Y = Y × Y = Y 2 represent the edges of the two knowledge graphs.

Thus,

By the union bound with Equation (21),

Lemma D.1. Fix ∅ ̸ = S ⊆ Y , π : Y 2 ↪ → Y 2 , δ, p, α ∈ (0 , 1) , q := 1 -p , and

For any ( T, P ) ∼ KG( S, Y, p, α ) chosen from the random knowledge graph distribution, we define

̸

̸

Then,

Proof. Let A ′ := { y ∈ S 2 | π ( y ) = y } , so A ⊆ A ′ . If π is the identity then the lemma holds trivially, therefore we can assume that A ′ = ∅ . If ϵ > 1 then the lemma holds trivially as well, because A ⊆ T and therefore | A | ≤ | T | , so we assume ϵ ∈ (0 , 1] .

̸

For each y ∈ A ′ , Equations (1) and (2) under Definition 3.1 imply that Pr[ y ∈ C ] = Pr[ y ∈ T \ P ] = (1 -α ) pq and Pr[ y ∈ A ] = Pr[ y ∈ T ] = p , thus Bayes rule gives

Now suppose we fix V ⊆ A ′ and condition on A := A ′ ∩ T = V . Note that the event y ∈ C is independent for different y ∈ V , therefore for any y ∈ V it holds that

Therefore, E [ | C | | A = V ] = (1 -α ) q | A | , and so a Chernoff bound gives

(Normally, Chernoff bounds would give the tight inequality that Pr[ | C | < . . . ] ≥ 1 -δ , but the ≤ in the above is necessary for the case in which A = ∅ in which case Chernoff bounds do not apply because it would be over | A | = 0 coin flips.) Since this holds for every V , we have:

By Lemma C.2, we can partition A ′ into three disjoint sets,

As above, we are going to condition on the value of A i := A ′ i ∩ T . Also, define,

Now, fix any i ∈ [3] and any set V ⊆ A ′ i . We now claim that for all i ∈ [3] , V ∈ A ′ i , and y ∈ V

̸

The rightmost equality follows from the fact that π ( y ) = y so π ( y ) / ∈ P is independent of y ∈ T . The leftmost equality follows similarly: Since π ( A ′ i ) ∩ A ′ i = ∅ , the event A i = V is independent of π ( y ) / ∈ P . Thus, again by Chernoff bounds we have

Since this holds for all V , it holds unconditionally, and by the union bound it follows that

Now, since the sets B i partition B and A i partition A , we have | B | = ∑ i | B i | , | A | = ∑ i | A i | , and also ∑ 3 i =1 √ | A i | ≤ √ 3 | A | by Cauchy-Schwartz. Thus, summing the three equations in Equation (25) probability implies

Combining with Equation (24) gives, by the union bound,

Since √ 3 / 2 + √ 1 / 2 ≤ 2 , this implies:

Since adding additional restrictions can only reduce a probability, we have:

But if p | S | 2 2 ≤ | T | ≤ | A | ϵ then 2 | A | ≥ ϵp | S | 2 and then, since ϵ ≥ 32 α 2 q 2 p | S | 2 ln 1 δ :

Thus,

Since, in general, for any two events X and Y it holds that Pr[ Y ] ≤ Pr[ X,Y ] + Pr[ X ] , we have

which is equivalent to the statement in the lemma. To see the last step above, note that E [ | T | ] = p | S | 2 and thus by multiplicative Chernoff bounds,

In the last step we have utilized the fact that α, q, δ ∈ (0 , 1] , and the fact (observed in the first paragraph of this proof) that we may assume that ϵ ∈ (0 , 1] else the lemma holds trivially.

Using the above lemma, we now prove our main theorem regarding knowledge graphs.

Or equivalently,

Proof. of Theorem 3.2. Let q := 1 -p and,

For any θ ∈ Θ define,

̸

Note that since err( θ ) = | A θ | / | T | , our goal is to show that, with probability ≥ 1 -δ , we will not output any θ with | A θ | ≥ ϵ | T | .

Recall that | Θ | ≤ n | S | . By Lemma D.1 substituting δ ′ = 1 6 n | S | δ and the union bound over θ ∈ Θ which is of size | Θ | ≤ n | S | ,

Using err( θ ) = | A θ | / | T | , this implies,

Finally, define the empirical 'errors' for any θ to be,

It is not difficult to see that the algorithm outputs a θ with minimal ˆ e θ , and thus it will not output any θ with ˆ e θ -ˆ e ⋆ > 0 . Now, it is also not difficult to see that ˆ e θ -ˆ e ⋆ is the mean of m random variables in {-1 , 0 , 1 } and

The last step above follows because π ⋆ is the identity, and because if y = π θ ( y ) then y ∈ P ⇐⇒ π θ ( y ) ∈ P . (Formally, one may define E θ := { y ∈ T | π θ ( y ) / ∈ P } and observe that B θ ⊆ E θ , C θ ⊆ E ⋆ and E θ \ B θ = E ⋆ \ C θ ). Thus, by Chernoff bounds,

By the union bound over θ ∈ Θ ,

Combining with Equation (26) gives,

Since, for our choice of ϵ ≥ 2 αq √ 2 m ln 6 | Θ | δ ,

Put another way,

̸

We claim that we are done: Observe that if MLE outputs some θ = ⋆ then, ˆ b θ ≤ ˆ b ⋆ . To see this, recall the definition of the prior ρ ,

and therefore the objective function minimized by MLE, namely, 1 m ∑ m i =1 -log( ρ ( f θ ( x i ))) , is strictly monotonic in ˆ b θ :

so the θ output by MLE necessarily minimizes b θ .

Finally, for the simplification in the theorem, note that for p < 0 . 99 , 1 /q < 100 is at most a constant and note that a maximum is never more than a sum.

It is interesting to note that it is possible to prove the same theorem using a generalization of Plausible Ambiguities, though we use the shorter proof above here because it is somewhat more involved. This generalization may be useful for other priors of full support. Many LMs, in practice, assign non-zero probability to every string due to softmax distributions or a process called 'smoothing.' A full-support prior ρ has full support, then A γ = Θ and so the parameter ε γ becomes too large to be meaningful even for γ = 0 . To address this, we refine our definition of plausible ambiguities as follows. For generality, we state them in terms of arbitrary loss L , though we only use them for the semantic error L = err .

Definition D.2 ( ( γ, κ ) -plausible ambiguities) . For any γ, κ ∈ [0 , 1] , the set of ( γ, κ ) -plausible ambiguities is:

Furthermore, A γ = A γ, 0 and ε γ = ε γ, 0 .

E Experiments

For the experiments used to generate Figures 4 and 5, we sampled random languages according to either the knowledge graph model or the common nonsense model, and then used a brute-force learning algorithm to find the optimal translator given an increasing amount of samples. A detailed description follows, and code can be found at https://github.com/orrp/theory-of-umt .

E.1 Experiments in the knowledge graph model

Recall that in the knowledge graph model, text describes relations between nodes in a directed graph. Due to computational constraints, we consider ten nodes, each corresponding to a different word in the target language. To generate edges corresponding to the target language P , two nodes are connected with a directed edge independently, with probability 0 . 5 . We then consider source languages with r ≤ 10 words. Given a ground-truth translator f ⋆ : [ r ] → [10] , the source language graph T is obtained by choosing a random subset of nodes S of size r , taking the pre-image of graph induced on S under f ⋆ , and (3) adding noise by redrawing each edge with probability 1 -α for a fixed agreement coefficient α ∈ (0 , 1) .

The prior ρ is derived from the edges of P , and the source language µ is derived from the (noisy) permuted subgraph T . We consider the translator family { f θ | θ ∈ Θ } of all node-to-node (wordto-word) injective translators, of which one is secretly chosen to be ground-truth. Similarly to the

Figure 7: Parameters for experiments in the knowledge graph model (Figure 4). For ablations on r we take α = 0 . 5 , and for ablations on α we take r = 9 . The experiments were run in parallel on an AWS r6i.4xlarge for a total of two and a half CPU-hours.

Figure 8: Parameters for experiments in the common nonsense model (Figure 5). The experiments were run in parallel on an AWS r6i.4xlarge for a total of four CPU-hours.

previous setting, we train an MLE algorithm on randomly chosen edges from T , which correspond to sentences in the source language. For each sampled edge ( x 1 , x 2 ) , we increase the "score" of each translator that agrees with the edge, that is, that ( f θ ( x 1 ) , f θ ( x 2 )) is an edge in the graph P .

To show how common ground affects translatability, we ablate the parameter α determines the fraction of edges on which the source language graph T and the target language graph P agree. Figure 4 validates the intuition that increased agreement results in lower translation error, and that as the number of samples increases, the error of the top-scoring translator decreases.

To show how language complexity affects translatability, we ablate r , which is the size of the subgraph corresponding to the source language. Figure 4 (right) validates the intuition that a larger subgraph results in lower translation error.

The error of a translator is computed as the fraction of edges whose labels are different than the ground-truth. The values with which the model is instantiated are detailed in Figure 7.

E.2 Experiments in the common nonsense model

Since in this model the structure of sentences is arbitrary, we represent sentences by integer IDs, [10 5 ] = 1 , 2 , . . . , 10 5 and [10 6 ] for the target language. We generate a prior ρ from the common nonsense model by taking the target sentence ids [10 6 ] and labeling a random α -fraction of them as nonsense; the remaining sentences are called sensical S . Given a ground-truth translator f ⋆ : [10 5 ] → [10 6 ] , the source language then distributes uniformly over the back-translation of sensical sentences, f -1 ⋆ ( S ) .

The translator family { f θ | θ ∈ Θ } is taken to be a set of 10 5 random one-to-one translators, of which one is secretely chosen to be ground-truth f ⋆ . We then train an MLE algorithm on random samples from the source language: Each sample x ∼ µ rules-out a subset of translators, namely, all θ ∈ Θ such that f θ ( x ) / ∈ S , i.e., is nonsensical.

Figure 5 shows that as the number of samples increases, the average error over the plausible translators (that have not been ruled-out) decreases. To show how language complexity / common ground affect translatability, we ablate the parameter α which determines the fraction of common nonsense. Our experiments validate the intuition that increased common nonsense results in lower translation error. The error of a translator is computed as the fraction of disagreements with the ground-truth on a hold-out validation set of size 1000. The values with which the model is instantiated are detailed in Figure 8.

F Related work

Project CETI. The sperm whale data collection effort began with a longitudinal dataset from a community of whales off the coast of Dominica that revealed interesting communication findings, such as dialects and vocal clans [16]. A recent effort by the Cetacean Translation Initiative (Project CETI) has been to collect custom-built passive bioacoustic arrays (installed in Fall 2022) covering a 20 × 20 kilometer area where these whale families reside (collecting over 3 TB/month) in tandem with on-whale robotic acoustic and video tags, underwater (robotic swimming fish) and aerial drones as well as other observation techniques in effort to augment rich contextual communication data. CETI's scientific team consists of specialists in machine learning, robotics, natural language processing, marine biology, linguistics, cryptography, signal processing and bio-acoustics. Andreas et al. [2] present CETI's initial scientific roadmap for understanding sperm whale communication, identifying the potential for unsupervised translation to be applied to whale communication. That roadmap suggests training a full generative LM for whale communication (often using trillions of bits for parameters [11, 12]). In contrast, our analysis suggests that the data requirements for translation may be similar to those of supervised translation, which is often several orders of magnitude smaller [37].

With this setting in mind, our requirements from source and target language are not symmetric: it would be unreasonable (and unnecessary) for our framework to assume that any sentence in the target language could be expressed in the source language. Put simply: whales need not understand what a smartphone is for us to gain some understanding of their communication. Also note, regarding domain gap, that some (although not all) knowledge can be inferred by training data from, e.g., online catalogs of hundreds of thousands of marine species [1]). Of course, there are also data-collection and transcription challenges, a challenge also present in the setting of low-resource (human) language translation [31]. While these challenges are outside the scope of this paper, our theoretical bounds on the data requirements may inform how much and what types of data are collected. For instance, it is less expensive to acquire textual data alone than both textual and video data. Therefore, if it is believed that an adequate translation is statistically possible using textual data alone, then greater effort may be placed on collecting this data and on UMT algorithms.

Unsupervised translation. In unsupervised machine translation [32], a translator between two languages is learned based only on monolingual corpora from each language. A body of work on UMT uses neural networks [29, 23, 5, 24, 35] or statistical methods [25, 4] for this task. Empirical evaluation of UMT found that it is outperformed by supervised machine translation, even when UMT is trained on several orders of magnitude more data [28, 22]. Among the key barriers for UMT identified in these evaluations are the domain gap and the data gap, and recent works propose techniques for bridging these gaps [15, 19]. Our theory suggests that sample complexity should remain roughly the same between the supervised and unsupervised settings, barring computational constraints. This, we hope, will encourage practitioners to bridge the remaining gaps.

Language models (LMs). In recent years, LMs such as GPT [11], BERT [13] and PaLM [12] were shown to achieve state-of-the-art performance on many tasks in natural language processing (NLP) such as text generation, summarization, or (supervised) MT. These models are indeed large, with hundreds of billions of parameters, and are pre-trained on hundreds of billions of tokens.

LMs are useful for machine translation in a variety of ways (e.g. [10, 18]). Of particular relevance are empirical works that use target LMs as priors to improve machine translation [27, 7]. To our knowledge, our work is the first theoretical work formally proving error bounds for prior-assisted translation. Appendix H discusses the use of LMs to establish priors for translation.

Goal-oriented communication. It is interesting to contrast our work with the work on goal-oriented communication, which was introduced by [21] and extended in [17]. They study the setting of two communicating parties (one of which is trying to achieve a verifiable goal) using each a language completely unknown to the other. They put forward a theory of goal-oriented communication, where communication is not an end in itself, but rather a means to achieving some goals of the communicating parties. Focusing on goals provides a way to address 'misunderstanding' during communication, as in when one can verify whether the goal is (or is not) achieved. Their theory shows how to overcome any initial misunderstanding between parties towards achieving a given goal. Our setting is different: Informally, rather than be a participant in a communication with someone

speaking a different language, we wish to translate communications between two external parties speaking in a language unknown to us and there is no verifiable goal to aid us in this process.

Subgraph isomoprhism. For simplicity, we model the knowledge graphs of Section 3.1 as a pair of correlated Erd˝ os-Rényi (ER) graphs. The computational problem of identifying a subgraph of an ER graph has been studied by Erd˝ os and others [6, 26, 20]. In particular, [20] consider a model in which two correlated graphs P, T are derived from a 'parent graph' G by independently deleting rows and edges G , and then applying a permutation π ∗ to the vertices of T . Although their model differs from our knowledge graph model, 15 they propose efficient algorithms for recovering the latent permutation π ∗ and provide an empirical evaluation on synthetic and real-world data. Given the similarity between our models, it would be interesting to see if their algorithm can be adapted to our setting, which would nicely complement our formally-proven-yet-inefficient algorithm.

G Future Work

Our initial exploration leaves plenty of room for future work. In particular, we propose the following possible directions:

In our lossless models, the target language subsumes the source language in the sense that everything that is representable in the source language can also be represented in the target language. It would be interesting to extend our work to the partially-overlapping case.

The language distribution in our models are all uniform. It would be interesting to examine non-uniform distributions such as Zipfian or power-law distributions.

As stated earlier, our analysis is purely information-theoretic and leaves the question of the efficiency of UMT open.

Agood starting point for the efficiency question would be to design efficient UMT algorithms for one of the randomized models of language presented in this paper.

H Good Priors through Prompts

The most direct way to improve a prior is to train (or fine-tune) the LM on a dataset that includes a large number of articles relevant to the source language, e.g., volumes of oceanic research, for whales. In addition, training on a wide variety of sources including multiple languages, diverse sources, and encoding systems may be helpful. It is possible that a system that has how to transfer knowledge between hundreds of languages and even programming languages, may have a better prior.

Another general strategy for creating a prior is to use prompting: Given a prompt string s , one can define ρ ( y ) ∝ ν ( s y ) , that is the prior distribution of text that that LM generates conditioned on the text beginning with s . Figure 9 illustrates some toy examples of how even a simple prompt like A sperm whale said: can help focus on translations that are more likely for a sperm whale to say, and eliminate irrelevant translations.

Background prompts. There is a natural and potentially powerful idea that an unsupervised translator, in addition to outputting translations, would automatically generate a background prompt that increases the intelligibility of many translations. 16 Suppose, for example, across numerous communications, the unsupervised translator determines that sperm whales measure time in 'number of naps' and that typical nap duration varies by age. Then, rather than having to repeatedly explain this in each translation, it can be explained once in a background prompt that is automatically inserted before each translation. For example, following the prompt s = Sperm whales measure time in numbers of naps, but each whale's typical nap duration depends on their age. So if a whale's nap duration is 9 minutes, then 4 naps is about 36 minutes (though whales tend to exaggerate times). A sperm whale said:

would make translations like Wow, I just dove for 6 naps or How many naps ago was that? both more likely and more understandable.

I Generalizations of the framework

I.1 Lossy Translation

Many things can be included in textual transcriptions suitable for translation. For instance, one can distinguish the speakers, e.g., ' Whale 1: ... Whale 2: ... Whale 1: ... ' if the source data is annotated with speaker identifiers. Some aspects of the way one is speaking can be transcribed, e.g., ' Whale 1 (fast tempo clicking): ... Whale 2 (slow, loud clicking): ... ' It may be possible to encode these textually in x .

However, if x is encoded in a more flexible format, such as arbitrary binary files, one can include as much raw information as possible, including the source audio recording, to provide the translator with as much context as possible. In that case, lossless translation will no longer be possible, because one cannot compute the raw x from a textual translation.

Given the possible benefits of such annotations, we propose an extension of our theory to the lossy setting. A natural generalization of the maximum-likelihood approach is as follows:

Here ϕ θ : X × Y → [0 , 1] is a probabilistic inverse ('randomized back translation') of f whose parameters are also encoded in θ . Note that the family { ( f θ , ϕ θ ) | θ ∈ Θ } must satisfy that for all y ∈ Y , ∑ x ∈ f -1 ( y ) ϕ θ ( x | y ) = 1 , though it is no longer required that f θ be 1-1.

As λ decreases to 0, the optimal solution would assign infinite loss to any f θ : X ↪ → Y and ϕ θ that are not perfect inverses, where there is some x (with positive probability under µ ) such that ϕ θ ( x | y = f θ ( x )) < 1 . Thus, for sufficiently small λ , the algorithm is exactly the minimum crossentropy (maximum likelihood) algorithm of Definition 1.1. For arbitrarily large λ , the algorithm will collapse to always outputting the most likely y ∈ Y under ρ . For example, everything could be translated to Hello regardless of its contents.

For intermediate values of λ , the chosen translator trades off naturalness in the form of ρ ( f θ ( x )) versus information loss which is inversely related to ϕ θ ( x i | y = f θ ( x i )) . This trade-off makes it challenging to define and analyze the success of a lossy unsupervised translation algorithm. Nonetheless, the algorithm is intuitive.

I.2 Infinite parameter sets Θ

In some cases, one can learn with many fewer examples, which is important when parameters are real-valued and | Θ | = ∞ or when the model is over-parameterized. However, one can analyze even these cases in our model using the following 'trick.' Suppose one has a supervised translation algorithm SUPER that takes m labeled examples ( x i , y i ) as input and outputs θ ∈ Θ . A simple

observation in these cases is that one could use an initial set of m unlabeled examples, x 1 , . . . , x m , to define a subset of translators:

That is, we restrict attention to the set of possible translators that we could output for any given ground-truth translations, then it is not difficult to see log | Θ | ≤ m log |Y| . If we assume that one of these is accurate and natural, then restricting the attention of MLE to this set will suffice, and Theorem A.2 means that the number of examples required is O (log | Θ | ) = O ( m log |Y| ) which is a linear blowup in the number of examples m used for supervised translation. To make this formal, one would start from only assuming that one of the translators had negligible error-this is left to future work.

J Bounds for Supervised Translation

In this work, for ease of presentation, we have stated that error decreases like O ( 1 m log | Θ | δ ) for noiseless supervised translation. This is based in the classic Occam bound for supervised learning:

Theorem J.1 (Occam bounds) . Let X , Y , be sets, D be a joint distribution over X × Y , ℓ : Y × Y → [0 , 1] be a loss, and f θ : X → Y be a family of functions parameterized by θ ∈ Θ , and L ( θ ) := E ( x,y ) ∼D [ ℓ ( y, f θ ( x ))] . For any δ > 0 ,

̸

for ˆ θ := argmin θ ∈ Θ ∑ i ℓ ( y i , f θ ( x i )) . 17

Note that supervised translation is simply classification with many classes Y .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a theoretical framework for understanding unsupervised translation and provides bounds for sample complexity. Specifically, the unsupervised machine translation setting they consider is that the translator $f_\\theta$ is the one that maximizes the plausibility $\\rho$ of the translation of examples $x_1, x_2, \\cdots, x_m$ in the source language:\n\n$$\\theta = \\mathrm{MLE}^{\\rho}(x_1, x_2, \\cdots, x_m) = \\arg \\max_{\\theta} \\sum_{i=1}^m \\log \\rho(f_\\theta(x_i))$$\n\nThe framework they propose has the following key components:\n\n- Prior $\\mu$: the distribution of the source language.\n- Prior $\\rho$: the plausibility of the translation in the target language.\n- Translator: $f_\\theta$ which they assume belongs to a finite-sized family.\n- Translation density $\\tau$: The distribution induced by the translator, i.e. $f_\\theta{x}$ where $x \\sim \\mu$.\n- A ground truth translator: $f_*$ that maps any input $x$ to a plausible translation almost surely.\n- Semantic loss: $L(\\theta)$ that evaluates how the translations by $f_\\theta$ is correct. They use exactly match (0-1 loss) as $L$ in the following analysis.\n\nAs instantiations of their framework, they propose two models of language:\n\n## Random knowledge graph model\n\nThis model makes the following assumptions:\n1. Text in this language represents an edge between a pair of nodes in a knowledge graph. \n2. The translator translates an edge by translating the two nodes respectively. \n3. The plausible edges in the source language largely agree with the plausible edges in the target language.\nBased on these assumptions the authors construct the distributions $\\rho, \\tau \\mu$, and then they provide the sample complexity bound for this model.\n\n\n## Common nonsense model\n\nThis model assumes no linguistic structure on the source text. They only assume that there is a set of translations that are implausible. They provide a sample complexity bound for this model. They also make connections to the complexity of the translator (description length).\n\nSTRENGTHS:\n1. In my opinion, this paper’s assumption about the existence of a set of plausible translations is very realistic and insightful for understanding the mechanism of unsupervised translation. As far as I know, attributing the success of unsupervised translation to this characteristic is novel and I think it is convincing (though may not be a complete explanation).\n2. This paper also demonstrates how this assumption can be used to analyze unsupervised translation.\n3. I also like the consideration about having a plausible prior in addition to the ground truth translation distribution. It handles the mismatch between the domains of the text in the source and target language. I think this kind of domain mismatch is very common in practice and thus considering this helps us understand unsupervised translation better.\n4. The knowledge graph model is inspiring to me. I think it suggests that the success of unsupervised translation is because of the interaction between some high-level structure and some low-level structure of language. In this model, the high-level structure is the plausibility of the combination of nodes, while the low-level structure is the one-to-one mapping of the nodes (if I understood correctly).\n\nIn sum, I think I learned quite a few things from this paper. I think the community can benefit from these ideas.\n\nWEAKNESSES:\n## Clarity Issues\n\nThe first major issue to me is that this paper has a very unusual structure, i.e. having 5 pages of introduction. \n1. I would expect the introduction to talk more about the motivation and high-level intuition of the theory. However, the authors tried to elaborate many definitions in the introduction but at the same time mention a few things without clear definition. For example, I couldn’t understand “broader prior (line 69)” and “agreement (line 99)” when I was reading the introduction. \n2. Another problem of this long introduction is that there seems to be a lot of overlap between the introduction and the content in the following sections. Information in the later sections does not seem to be totally aligned with the description in the introduction section. For example, at line 101, the authors assume the compositional encoding of language, but this is not mentioned in Section 3.1.\n\nThe second  issue is that this paper never provides a high-level explanation for the proofs in the main text. It’s hard to assess the correctness of their results. I found the proofs in the appendix are not easy to follow either. \n\nThe third issues is that they rarely provide explanations for the symbols or the formulas either. For example, in Definition 3.1, they define many sets (e.g. $P, S$ etc) without describing their intuitive meanings. The usage of the symbols is also not very intuitive. For example, it seems that the $T, S$ and $P$ in Section 3.1 have different meanings than the ones in Section 3.2.\n\n## Technical Issues\n\nI have two main concerns:\n\n1. I doubt whether Theorem 3.4 is correct. Intuitively speaking, based on this model, there are many translators that never map any source text to an implausible target text. However, only one of them is correct. How can it be possible that the translator learned from MLE can be a good one without the access to other information? I also checked the proof in Appendix C.1.1. Line 714 seems to be saying it is suffice to prove that “large semantic error ($A$) implies many implausible translations ($B$)”, but the formula in Line 717 is more like saying “it’s impossible to have low error ($\\neg A$) while having many implausible translations ($B$)”. If the goal is to prove $A \\to B$, then I can’t see why we should prove $( \\neg A \\wedge B)$ is impossible.\n2. I think there needs to be more justification for how the instantiations are related to real world scenarios. One apparent example is the distributions of $\\rho$ and $\\nu$ constructed based on Definition 3.1, which is very complicated and is not intuitive. I would like to know whether this is also an abstraction of real-world scenarios.\n\nIn sum, I think this work is not ready for publication in this form, despite the interesting insights it has.\n\nQUESTIONS:\n1. Please let me know if I misunderstood anything about the technical issues. I would be happy to adjust my score.\n2. Could you describe the meaning of the broader prior $\\rho$ and agreement $a$ in the context of real-world scenarios? e.g. what would we expect this $\\rho$ and $a$ to be in the real world?\n2. I wonder whether the back translation loss used in some unsupervised translation works is relevant to the one-to-one translator assumption in this work? Could it be helpful because the loss enforces the bijectivity of the model?\n3. I would also suggest the authors to include some experiments, perhaps with some toy data.\n\nLIMITATIONS:\nIn my opinion, the authors should list their assumptions more explicitly and clearly, e.g. including them in a list or indexing them with numbers. Other than that, I think the discussion section addresses the limitations well.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n### After rebuttal\n\nThanks the authors for the detailed rebuttal, especially with new experiments. Provided the condition that the authors will add the new experiments and adjust the presentation after reviewers' comments, I gladly change my scores and recommendation!\n\n----\nThe paper presents a rather interesting theory about how unsupervised machine translation (UMT) without any parallel data. The paper taps into the analysis of world knowledge graph models, which may model relationships between objects/subjects of the world, and presented by words in the data. And suggests that it is the compositionality of languages somewhat allows UMT to work.\n\nThe paper also analyze the common nonsense model concept, where UMT generally tries to rotate the source distribution so that it can fit in a unique orientation in the target distribution, given enough nonsense data.\n\nThe paper presents some conclusion about the theorical error rate of UMT in relation with data and languages, though no experiments were done.\n\nSTRENGTHS:\n* The concept and analysis of the UMT in the paper is very interesting and novel.\n* The theoretical models of random knowledge graph and common nonsense models are interesting and should be investigated further.\n* Some observation from such formulation is that error rate is inversely related with language complexity, which is indeed non-intuitive, but there is no experiment to confirm such observation.\n\nWEAKNESSES:\n* The paper is very math heavy with limited visualization or intuitive explanation. This makes it rather difficult to comprehend and and grasp the content. More intuitive explanation with figures and illustration are needed to understand. I am also unsure about the correctness of the formulation and the math, and can only assume it to be correct in the review.\n* There is no experiments to validate the theory at all, even with a toy experiments. This makes it hard to conclude the completeness of the paper.\n* The organization is not very well-structured and rather strange. There are no experiments and related work sections.\n* Despite heavily advertised in the initial part of the paper, there is limited correlation, conclusion, or elaboration about how we can model such human-animal communications, which is quite disappointing.\n* There is no clear indication and suggestion of what to do with UMT after reading this theory. The paper does not make recommendations on how to design better UMT models, how to train them, which aspects of existing UMT models need to be fixed so that it can handle low-resource languages and more ambitiously human-animal translations. This is real challenges that UMT performs very poorly when domains of unlabeled data sources are very distant.\n\nGiven the high degree of novelty of implication of this work, I would give much higher score if the rebuttal addresses some of the concerns.\n\nQUESTIONS:\nN.A\n\nLIMITATIONS:\nThere is no discussion of limitations.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a theoretical framework for unsupervised machine translation (UMT) when no parallel data available and source and target language do not share similar linguistic structures. To do this, they first define and clarify the three main challenges: understanding the goal, no linguistic structure shared and domain gap. In response to these challenges, they propose a general framework and instantiate it with two models: knowledge graph model , which is highly structured with text representing an edge between a pair of nodes, and \"Common nonsense\" model which is completely unstructured. For both models, they establish the error bounds and demonstrate that the error rate is inversely related to the amount of samples, common ground, and the language complexity.\n\nSTRENGTHS:\n* The paper presents a novel theoretical framework for UMT, which is needed but challenging in the field of UMT.\n* The authors propose two complementary models of language, one that is highly structured and one that is completely unstructured, which provides a relatively comprehensive analysis of UMT.\n* The authors provide theoretical bounds on the data requirements for UMT, which can inform how much data are collected for such tasks.\n* The authors' ideas are interesting and insightful, such as the motivating example in the knowledge graph model where one can complete the mapping based on the similar structure of the graph, and enough \"nonsense\" make a nearly unique mapping in \"common nonsense\" model.\n\nWEAKNESSES:\n* The paper makes strong assumptions about the language, which may limit the applicability of the proposed models in real-world scenarios. \n* The paper does not provide any empirical validation of the proposed theoretical framework and models. The absence of experimental results makes it difficult to assess the practical effectiveness of the proposed methods.\n* The paper does not provide a clear explanation of how the proposed models can be implemented in practice, which may limit their usability for practitioners in the field.\n\nQUESTIONS:\n* I am not sure what the \"prior\" proposed in the text refers to, but I tentatively comprehend it to be a prefix LM, where the prefix may be a prompt like \"a whale said:\".\n* Why maximizing the likelihood of the prior implies translation. After all, the source language is not involved in the calculation of this likelihood, so how to reflect the fidelity of the translation?\n\nLIMITATIONS:\nSame as Weaknesses. Or can we consider a real experiment, such as mutual translation between knowledge graphs constructed by English and Chinese respectively?", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper describes a theoretical analysis of error bounds for unsupervised machine translation, motivated by considering the unusual, and so far only speculative, case of translation between animal \"languages\" to human natural languages.\n\nThe paper consider the case where a number of examples of the source (animal) language utterances are available, as well as a prior on the target language (e.g., a conventional pretrained LLM), and attempts to investigate what error an invertible translation function trained to maximize a MLE objective may achieve compared to a hypothetical golden truth translator.\n\nThe paper considers two specific cases with different assumptions over the languages and the priors: in the first case the languages are assumed to be Erdős–Rényi random graphs parametrized by an edge agreement parameter. The authors prove a bound which does not vanish even in the limit of infinite source sentences but vanishes in the limit of perfect agreement. I do not find this model to be very realistic for natural languages.\n\nThe second case consider the two languages to be uniform distributions over a subset that excludes a \"nonsense\" set. The authors prove an bound similar to supervised learning. I am not sure why uniformity is considered a plausible hypothesis here, as actual natural languages are characterized by power law distributions.\n\nOverall, I think there is not enough justification connecting the formal models with the motivation of translating animal communication.\n\nSTRENGTHS:\n- Novel and interesting topic.\n\n- Theoretical analysis.\n\nWEAKNESSES:\n- Assumptions not well justified\n\n- Formal model not well connected with motivation\n\n- Too much background and related work is missing. Some is provided in the supplementary material, but it should be in the main paper.\n\nQUESTIONS:\nIs there any evidence that the formal models you propose can correctly capture the relevant factors of unsupervised translation even between human languages, let alone between animal and human languages?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n### Update\n\nI'm satisfied with the author's rebuttal and decided to overall improve my score.\n\n\nThis paper proposes a theoretical framework for analyzing Unsupervised MT motivated by animal communication. The paper is only the first step in the theoretical analysis of UMT. The theoretical framework was analyzed for two models: Knowledge graph and Common nonsense models. The authors show theoretically that the error rate is inversely related to the number of samples, common ground, and language complexity.\n\nSTRENGTHS:\nThe authors work on providing a theoretical framework for analyzing unsupervised MT. The task is motivated by a real-world challenge of understanding animal communication.\n\nThe paper is well-written and was not too hard to follow. The general framework was well-defined and motivated. \n\nThe paper instantiated two models with common-nonsense and knowledge graphs. The paper derived theoretical bounds for error rates and provided comprehensive proof.\n\nWEAKNESSES:\nPurely descriptive: what is the key takeaway from the paper? How can we improve unsupervised MT based on these findings? Where can we use the theoretical framework?\n\nNo experimental results? I realize the paper focuses on the theory of unsupervised translation. However, none of the author's theoretical claims are validated using experimental findings which I find a bit hard to follow. \n\nThe authors claim that the error rate is inversely proportional to language complexity. This is counterintuitive to what I would expect in Machine Translation. However, there is no in-depth discussion, reasoning, or intuition as to why this is the case. Additionally, the related work or motivation section is missing in the main paper, where I believe they should motivate UMT.\n\nI missed the connection in the paper with learning animal communication. The paper initially has a strong motivation to learn animal communication, however, the discussion section is weak and I could not find what to conclude, or whether can we actually use the unsupervised MT theoretical framework.\n\nQUESTIONS:\nCould the authors provide experimental findings for their theoretical framework? Take a simple case of language translation example and validate their theoretical findings using a few experiments. Right now, the paper focuses on the theoretical framework only. \n\nProvide more intuition on why they found error rates are inversely proportional to language complexity.\n\nImprove the connection with learning animal communication. There was very little discussion about animal communication apart from the introduction section.\n\nLIMITATIONS:\nThe authors did not provide any limitations on their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-28"}

comments: SUMMARY:
The first contribution of this work is formalizing and analyzing a model of Unsupervised Machine Translation (UMT). The model applies even to low-resource source languages with massive domain gap and linguistic distance. Second, the paper exhibits two simple complementary models for which it is proved that:
(a) more complex languages require less common ground, and 
(b) data requirements are not significantly greater than those of supervised translation (which tend to be significantly less than generative language modeling). These findings may have implications for the quantity and type of communication data that is collected for deciphering animal communication and for UMT more generally.

STRENGTHS:
This work is the first theoretical work formally proving error bounds for prior-assisted translation, which shows that the sample complexity should remain roughly the same between the supervised and unsupervised settings (specifically, the two stylized models of language -- Random sentence trees and Random knowledge graphs), barring computational constraints.

WEAKNESSES:
There isn't any empirical evaluation aside from using text-davinci-02 to evaluate sentence likelihoods

QUESTIONS:
What has changed in this submission from the arxiv version (https://arxiv.org/pdf/2211.11081.pdf)?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-08-02