SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Therefore,  $\mathbb{E}[|C| \mid A = V] = (1 - \alpha)q|A|$ , and so a Chernoff bound gives
$$
\forall V \subseteq A ^ {\prime} \Pr_ {T, P} \left[ | C | \leq (1 - \alpha) q | A | + \sqrt {\frac {1}{2} | A | \ln \frac {1}{\delta}} \mid A = V \right] \geq 1 - \delta .
$$
(Normally, Chernoff bounds would give the tight inequality that  $\operatorname*{Pr}[|C| < \ldots] \geq 1 - \delta$ , but the  $\leq$  in the above is necessary for the case in which  $A = \emptyset$  in which case Chernoff bounds do not apply because it would be over  $|A| = 0$  coin flips.) Since this holds for every  $V$ , we have:
$$
\operatorname * {P r} _ {T, P} \left[ | C | \leq (1 - \alpha) q | A | + \sqrt {\frac {1}{2} | A | \ln \frac {1}{\delta}} \right] \geq 1 - \delta . \tag {24}
$$
By Lemma C.2, we can partition  $A^{\prime}$  into three disjoint sets,
$$
A ^ {\prime} = A _ {1} ^ {\prime} \cup A _ {2} ^ {\prime} \cup A _ {3} ^ {\prime} \text {s u c h} \pi \left(A _ {i} ^ {\prime}\right) \cap A _ {i} ^ {\prime} = \emptyset .
$$
As above, we are going to condition on the value of  $A_{i}\coloneqq A_{i}^{\prime}\cap T$ . Also, define,
$$
B _ {i} := \{y \in A _ {i} \mid \pi (y) \notin P \} = B \cap A _ {i}.
$$
Now, fix any  $i \in [3]$  and any set  $V \subseteq A_i'$ . We now claim that for all  $i \in [3], V \in A_i'$ , and  $y \in V$
$$
\operatorname * {P r} _ {T, P} \left[ y \in B _ {i} \mid A _ {i} = V \right] = \operatorname * {P r} _ {T, P} \left[ \pi (y) \notin P \mid y \in T \right] = q.
$$
The rightmost equality follows from the fact that  $\pi(y) \neq y$  so  $\pi(y) \notin P$  is independent of  $y \in T$ . The leftmost equality follows similarly: Since  $\pi(A_i') \cap A_i' = \emptyset$ , the event  $A_i = V$  is independent of  $\pi(y) \notin P$ . Thus, again by Chernoff bounds we have
$$
\forall i \in [ 3 ] \forall V \subseteq A _ {i} ^ {\prime} \operatorname * {P r} _ {T, P} \left[ | B _ {i} | \geq q | A _ {i} | - \sqrt {\frac {1}{2} | A _ {i} | \ln \frac {1}{\delta}} \Bigg | A _ {i} = V \right] \geq 1 - \delta .
$$
Since this holds for all  $V$ , it holds unconditionally, and by the union bound it follows that
$$
\Pr_ {T, P} \left[ \forall i \in [ 3 ] \left| B _ {i} \right| \geq q \left| A _ {i} \right| - \sqrt {\frac {1}{2} \left| A _ {i} \right| \ln \frac {1}{\delta}} \right] \geq 1 - 3 \delta . \tag {25}
$$
Now, since the sets  $B_i$  partition  $B$  and  $A_i$  partition  $A$ , we have  $|B| = \sum_{i} |B_i|$ ,  $|A| = \sum_{i} |A_i|$ , and also  $\sum_{i=1}^{3} \sqrt{|A_i|} \leq \sqrt{3|A|}$  by Cauchy-Schwartz. Thus, summing the three equations in Equation (25) probability implies
$$
\operatorname * {P r} _ {T, P} \left[ | B | \geq q | A | - \sqrt {\frac {3}{2} | A | \ln \frac {1}{\delta}} \right] \geq 1 - 3 \delta .
$$
Combining with Equation (24) gives, by the union bound,
$$
\operatorname * {P r} _ {T, P} \left[ | B | - | C | \geq q | A | - \sqrt {\frac {3}{2} | A | \ln \frac {1}{\delta}} - (1 - \alpha) q | A | - \sqrt {\frac {1}{2} | A | \ln \frac {1}{\delta}} \right] \geq 1 - 4 \delta .
$$
Since  $\sqrt{3 / 2} +\sqrt{1 / 2}\leq 2$  , this implies:
$$
\operatorname * {P r} _ {T, P} \left[ | B | - | C | \geq \alpha q | A | - 2 \sqrt {| A | \ln \frac {1}{\delta}} \right] \geq 1 - 4 \delta .
$$
Or equivalently,
$$
\Pr_ {T, P} \left[ | B | - | C | <   \alpha q | A | - 2 \sqrt {| A | \ln \frac {1}{\delta}} \right] \leq 4 \delta .
$$
Since adding additional restrictions can only reduce a probability, we have:
$$
\operatorname * {P r} _ {T, P} \left[ \frac {p | S | ^ {2}}{2} \leq | T | \leq \frac {| A |}{\epsilon} \wedge | B | - | C | <   \alpha q | A | - 2 \sqrt {| A | \ln \frac {1}{\delta}} \right] \leq 4 \delta .
$$
But if  $\frac{p|S|^2}{2} \leq |T| \leq \frac{|A|}{\epsilon}$  then  $2|A| \geq \epsilon p|S|^2$  and then, since  $\epsilon \geq \frac{32}{\alpha^2 q^2 p|S|^2} \ln \frac{1}{\delta}$ :
$$
2 \sqrt {| A | \ln \frac {1}{\delta}} \leq 2 \sqrt {\frac {2 | A |}{\epsilon p | S | ^ {2}} \cdot | A | \ln \frac {1}{\delta}} \leq 2 | A | \sqrt {\frac {2}{p | S | ^ {2} \frac {3 2}{\alpha^ {2} q ^ {2} p | S | ^ {2}} \ln \frac {1}{\delta}} \ln \frac {1}{\delta}} = \frac {\alpha q}{2} | A |.
$$
$$
\operatorname * {P r} _ {T, P} \left[ \frac {p | S | ^ {2}}{2} \leq | T | \leq \frac {| A |}{\epsilon} \wedge | B | - | C | <   \frac {\alpha q}{2} | A | \right] \leq 4 \delta .
$$
Since, in general, for any two events  $X$  and  $Y$  it holds that  $\operatorname*{Pr}[Y] \leq \operatorname*{Pr}[X,Y] + \operatorname*{Pr}[\overline{X}]$ , we have
$$
\begin{array}{l} \operatorname * {P r} _ {T, P} \left[ | T | \leq \frac {| A |}{\epsilon} \wedge | B | - | C | <   \frac {\alpha q}{2} | A | \right] \leq \operatorname * {P r} _ {T, P} \left[ \frac {p | S | ^ {2}}{2} \leq | T | \leq \frac {| A |}{\epsilon} \wedge | B | - | C | <   \frac {\alpha q}{2} | A | \right] + \\ + \operatorname * {P r} _ {T, P} \left[ \frac {p | S | ^ {2}}{2} > | T | \right] \\ \leq 4 \delta + \operatorname * {P r} _ {T, P} \left[ \frac {p | S | ^ {2}}{2} > | T | \right] \\ \leq 4 \delta + \delta , \\ \end{array}
$$
which is equivalent to the statement in the lemma. To see the last step above, note that  $\mathbb{E}[|T|] = p|S|^2$  and thus by multiplicative Chernoff bounds,
$$
\Pr \left[ | T | <   \frac {p | S | ^ {2}}{2} \right] \leq \exp \left(- \frac {p | S | ^ {2}}{8}\right) \leq \exp \left(- \frac {p | S | ^ {2}}{8} \cdot \frac {1}{\epsilon} \cdot \frac {3 2}{\alpha^ {2} q ^ {2} p | S | ^ {2}} \ln \frac {1}{\delta}\right) = \delta^ {\frac {4}{\epsilon \alpha^ {2} q ^ {2}}} \leq \delta .
$$
In the last step we have utilized the fact that  $\alpha, q, \delta \in (0,1]$ , and the fact (observed in the first paragraph of this proof) that we may assume that  $\epsilon \in (0,1]$  else the lemma holds trivially.
Using the above lemma, we now prove our main theorem regarding knowledge graphs.
Proof. of Theorem 3.2. Let  $q \coloneqq 1 - p$  and,
$$
\epsilon := \max  \left(\frac {6 4}{\alpha^ {2} p q ^ {2} | S | ^ {2}} \ln \frac {6 n ^ {| S |}}{\delta}, \frac {2}{\alpha q} \sqrt {\frac {2}{m} \ln \frac {6 n ^ {| S |}}{\delta}}\right).
$$
For any  $\theta \in \Theta$  define,
$$
A _ {\theta} := \{y \in T \mid \pi_ {\theta} (y) \neq y \}
$$
$$
B _ {\theta} := \{y \in A _ {\theta} \mid \pi_ {\theta} (y) \notin P \}
$$
$$
C _ {\theta} := \{y \in A _ {\theta} \mid y \notin P \}
$$
Note that since  $\mathrm{err}(\theta) = |A_{\theta}| / |T|$ , our goal is to show that, with probability  $\geq 1 - \delta$ , we will not output any  $\theta$  with  $|A_{\theta}| \geq \epsilon |T|$ .
Recall that  $|\Theta| \leq n^{|S|}$ . By Lemma D.1 substituting  $\delta' = \frac{1}{6n^{|S|}}\delta$  and the union bound over  $\theta \in \Theta$  which is of size  $|\Theta| \leq n^{|S|}$ ,
$$
\Pr_ {T, P} \left[ \exists \theta \in \Theta | A _ {\theta} | \geq \epsilon | T | \wedge | B _ {\theta} | - | C _ {\theta} | \leq \frac {\alpha q}{2} | A _ {\theta} | \right] \leq \frac {5 \delta}{6}.
$$
Using  $\mathrm{err}(\theta) = |A_{\theta}| / |T|$ , this implies,
$$
\Pr_ {T, P} \left[ \exists \theta \in \Theta \operatorname {e r r} (\theta) \geq \epsilon \wedge | B _ {\theta} | - | C _ {\theta} | \leq \frac {\alpha q \epsilon | T |}{2} \right] \leq \frac {5 \delta}{6}
$$
$$
\Pr_ {T, P} \left[ \exists \theta \in \Theta \operatorname {e r r} (\theta) \geq \epsilon \wedge \frac {\left| B _ {\theta} \right|}{\left| T \right|} - \frac {\left| C _ {\theta} \right|}{\left| T \right|} \leq \frac {\alpha q \epsilon}{2} \right] \leq \frac {5 \delta}{6} \tag {26}
$$
Finally, define the empirical "errors" for any  $\theta$  to be,
$$
\hat {e} _ {\theta} = \frac {1}{m} \left\{i \mid f _ {\theta} \left(x _ {i}\right) \notin P \right\}.
$$
It is not difficult to see that the algorithm outputs a  $\theta$  with minimal  $\hat{e}_{\theta}$ , and thus it will not output any  $\theta$  with  $\hat{e}_{\theta} - \hat{e}_{\star} > 0$ . Now, it is also not difficult to see that  $\hat{e}_{\theta} - \hat{e}_{\star}$  is the mean of  $m$  random variables in  $\{-1,0,1\}$  and
$$
\mathbb {E} [ \hat {e} _ {\theta} - \hat {e} _ {\star} ] = \operatorname * {P r} _ {y \sim \tau} \left[ \pi_ {\theta} (y) \notin P \right] - \operatorname * {P r} _ {y \sim \tau} \left[ y \notin P \right] = \frac {| B _ {\theta} |}{| T |} - \frac {| C _ {\theta} |}{| T |}.
$$
The last step above follows because  $\pi_{\star}$  is the identity, and because if  $y = \pi_{\theta}(y)$  then  $y \in P \iff \pi_{\theta}(y) \in P$ . (Formally, one may define  $E_{\theta} \coloneqq \{y \in T \mid \pi_{\theta}(y) \notin P\}$  and observe that  $B_{\theta} \subseteq E_{\theta}$ ,  $C_{\theta} \subseteq E_{\star}$  and  $E_{\theta} \setminus B_{\theta} = E_{\star} \setminus C_{\theta}$ ). Thus, by Chernoff bounds,
$$
\forall \theta \in \Theta \Pr_ {x _ {1}, \dots , x _ {m}} \left[ \hat {e} _ {\theta} - \hat {e} _ {\star} \leq \frac {| B _ {\theta} |}{| T |} - \frac {| C _ {\theta} |}{| T |} + \sqrt {\frac {2}{m} \ln \frac {6 | \Theta |}{\delta}} \right] \leq \frac {\delta}{6 | \Theta |}.
$$
By the union bound over  $\theta \in \Theta$
$$
\Pr_ {x _ {1}, \dots , x _ {m}} \left[ \exists \theta \in \Theta \hat {e} _ {\theta} - \hat {e} _ {\star} \leq \frac {| B _ {\theta} |}{| T |} - \frac {| C _ {\theta} |}{| T |} + \sqrt {\frac {2}{m} \ln \frac {6 | \Theta |}{\delta}} \right] \leq \frac {\delta}{6}.
$$
Combining with Equation (26) gives,
$$
\operatorname * {P r} _ {T, P} \left[ \exists \theta \in \Theta \operatorname {e r r} (\theta) \geq \epsilon \wedge \hat {e} _ {\theta} - \hat {e} _ {\star} \leq \frac {\alpha q \epsilon}{2} - \sqrt {\frac {2}{m} \ln \frac {6 | \Theta |}{\delta}} \right] \leq \frac {5 \delta}{6} + \frac {\delta}{6} = \delta .
$$
Since, for our choice of  $\epsilon \geq \frac{2}{\alpha q}\sqrt{\frac{2}{m}\ln\frac{6|\Theta|}{\delta}}$
$$
\Pr_ {T, P} \left[ \exists \theta \in \Theta \operatorname {e r r} (\theta) \geq \epsilon \wedge \hat {e} _ {\theta} - \hat {e} _ {\star} \leq 0 \right] \leq \delta .
$$
Put another way,
$$
\operatorname * {P r} _ {T, P} \left[ \forall \theta \in \Theta \mathrm {e r r} (\theta) \leq \epsilon \vee \hat {e} _ {\theta} - \hat {e} _ {\star} > 0 \right] \geq 1 - \delta .
$$
We claim that we are done: Observe that if MLE outputs some  $\theta \neq \star$  then,  $\hat{b}_{\theta} \leq \hat{b}_{\star}$ . To see this, recall the definition of the prior  $\rho$ ,
$$
\rho (f _ {\theta} (x)) := \left\{ \begin{array}{l l} \frac {1}{2} \cdot \left(\frac {1}{| P |} + \frac {1}{| \mathcal {Y} |}\right) & \text {i f} f _ {\theta} (x) \in P \\ \frac {1}{2 | \mathcal {Y} |} & \text {i f} f _ {\theta} (x) \notin P. \end{array} \right.
$$
and therefore the objective function minimized by MLE, namely,  $\frac{1}{m}\sum_{i = 1}^{m} - \log (\rho (f_{\theta}(x_i)))$  is strictly monotonic in  $\hat{b}_{\theta}$ :
$$
\begin{array}{l} \frac {1}{m} \sum_ {i = 1} ^ {m} - \log (\rho (f _ {\theta} (x _ {i}))) = \hat {b} _ {\theta} \cdot \log \frac {2}{1 / | \mathcal {Y} |} + (1 - \hat {b} _ {\theta}) \log \frac {2}{1 / | P | + 1 / | \mathcal {Y} |} \\ = \log \frac {2}{1 / | P | + 1 / | \mathcal {Y} |} + \hat {b} _ {\theta} \cdot \log \frac {1 / | P | + 1 / | \mathcal {Y} |}{1 / | \mathcal {Y} |} \\ \end{array}
$$
so the  $\theta$  output by MLE necessarily minimizes  $b_{\theta}$
Finally, for the simplification in the theorem, note that for  $p < 0.99$ ,  $1/q < 100$  is at most a constant and note that a maximum is never more than a sum.
It is interesting to note that it is possible to prove the same theorem using a generalization of Plausible Ambiguities, though we use the shorter proof above here because it is somewhat more involved. This generalization may be useful for other priors of full support. Many LMs, in practice, assign non-zero probability to every string due to softmax distributions or a process called "smoothing." A full-support prior  $\rho$  has full support, then  $\mathcal{A}_{\gamma} = \Theta$  and so the parameter  $\varepsilon_{\gamma}$  becomes too large to be meaningful even for  $\gamma = 0$ . To address this, we refine our definition of plausible ambiguities as follows. For generality, we state them in terms of arbitrary loss  $\mathcal{L}$ , though we only use them for the semantic error  $\mathcal{L} = \mathrm{err}$ .
Definition D.2  $((\gamma, \kappa)$ -plausible ambiguities). For any  $\gamma, \kappa \in [0,1]$ , the set of  $(\gamma, \kappa)$ -plausible ambiguities is:
$$
\mathcal {A} _ {\gamma , \kappa} := \left\{\theta \in \Theta \left| \operatorname * {P r} _ {y \sim \tau} [ \rho (\pi_ {\theta} ^ {\star} (y)) \leq \kappa ] \leq \gamma \right. \right\}, a n d \varepsilon_ {\gamma , \kappa} := \max  _ {\theta \in \mathcal {A} _ {\gamma}} \mathcal {L} (\theta).
$$
Furthermore,  $\mathcal{A}_{\gamma} = \mathcal{A}_{\gamma ,0}$  and  $\varepsilon_{\gamma} = \varepsilon_{\gamma ,0}$
E Experiments
For the experiments used to generate Figures 4 and 5, we sampled random languages according to either the knowledge graph model or the common nonsense model, and then used a brute-force learning algorithm to find the optimal translator given an increasing amount of samples. A detailed description follows, and code can be found at https://github.com/orrp/theory-of-uml.
E.1 Experiments in the knowledge graph model
Recall that in the knowledge graph model, text describes relations between nodes in a directed graph. Due to computational constraints, we consider ten nodes, each corresponding to a different word in the target language. To generate edges corresponding to the target language  $P$ , two nodes are connected with a directed edge independently, with probability 0.5. We then consider source languages with  $r \leq 10$  words. Given a ground-truth translator  $f_{\star} \colon [r] \to [10]$ , the source language graph  $T$  is obtained by choosing a random subset of nodes  $S$  of size  $r$ , taking the pre-image of graph induced on  $S$  under  $f_{\star}$ , and (3) adding noise by redrawing each edge with probability  $1 - \alpha$  for a fixed agreement coefficient  $\alpha \in (0,1)$ .
The prior  $\rho$  is derived from the edges of  $P$ , and the source language  $\mu$  is derived from the (noisy) permuted subgraph  $T$ . We consider the translator family  $\{f_{\theta}|\theta \in \Theta \}$  of all node-to-node (word-to-word) injective translators, of which one is secretly chosen to be ground-truth. Similarly to the
Name	Symbol	Value
Number of source nodes	r	1,4,7,10
Number of target nodes	n	10
Number of training data	m	1,2,... up to all edges
Edge density (probability of including an edge)	p	0.5
Agreement parameter	α	0.33, 0.66, 1
Figure 7: Parameters for experiments in the knowledge graph model (Figure 4). For ablations on  $r$  we take  $\alpha = 0.5$ , and for ablations on  $\alpha$  we take  $r = 9$ . The experiments were run in parallel on an AWS r6i.4xlarge for a total of two and a half CPU-hours.
Name	Symbol	Value
Number of source sentences	|T|	10^5
Number of target sentences	|P|	10^6
Number of training data	m	1,..., 100
Number of validation data	1000
Fraction of common nonsense	α	0, 0.1, ..., 0.8
Figure 8: Parameters for experiments in the common nonsense model (Figure 5). The experiments were run in parallel on an AWS r6i.4xlarge for a total of four CPU-hours.
previous setting, we train an MLE algorithm on randomly chosen edges from  $T$ , which correspond to sentences in the source language. For each sampled edge  $(x_{1}, x_{2})$ , we increase the "score" of each translator that agrees with the edge, that is, that  $(f_{\theta}(x_{1}), f_{\theta}(x_{2}))$  is an edge in the graph  $P$ .
To show how common ground affects translatability, we ablate the parameter  $\alpha$  determines the fraction of edges on which the source language graph  $T$  and the target language graph  $P$  agree. Figure 4 validates the intuition that increased agreement results in lower translation error, and that as the number of samples increases, the error of the top-scoring translator decreases.
To show how language complexity affects translatability, we ablate  $r$ , which is the size of the subgraph corresponding to the source language. Figure 4 (right) validates the intuition that a larger subgraph results in lower translation error.
The error of a translator is computed as the fraction of edges whose labels are different than the ground-truth. The values with which the model is instantiated are detailed in Figure 7.
E.2 Experiments in the common nonsense model
Since in this model the structure of sentences is arbitrary, we represent sentences by integer IDs,  $[10^5] = 1,2,\dots ,10^5$  and  $[10^{6}]$  for the target language. We generate a prior  $\rho$  from the common nonsense model by taking the target sentence ids  $[10^{6}]$  and labeling a random  $\alpha$ -fraction of them as nonsense; the remaining sentences are called sensical  $S$ . Given a ground-truth translator  $f_{\star}\colon [10^{5}]\to [10^{6}]$ , the source language then distributes uniformly over the back-translation of sensical sentences,  $f_{\star}^{-1}(S)$ .
The translator family  $\{f_{\theta}|\theta \in \Theta \}$  is taken to be a set of  $10^{5}$  random one-to-one translators, of which one is secretly chosen to be ground-truth  $f_{\star}$ . We then train an MLE algorithm on random samples from the source language: Each sample  $x\sim \mu$  rules-out a subset of translators, namely, all  $\theta \in \Theta$  such that  $f_{\theta}(x)\notin S$ , i.e., is nonsensical.
Figure 5 shows that as the number of samples increases, the average error over the plausible translators (that have not been ruled-out) decreases. To show how language complexity / common ground affect translatability, we ablate the parameter  $\alpha$  which determines the fraction of common nonsense. Our experiments validate the intuition that increased common nonsense results in lower translation error. The error of a translator is computed as the fraction of disagreements with the ground-truth on a hold-out validation set of size 1000. The values with which the model is instantiated are detailed in Figure 8.
F Related work
Project CETI. The sperm whale data collection effort began with a longitudinal dataset from a community of whales off the coast of Dominica that revealed interesting communication findings, such as dialects and vocal clans [16]. A recent effort by the Cetacean Translation Initiative (Project CETI) has been to collect custom-built passive bioacoustic arrays (installed in Fall 2022) covering a  $20 \times 20$  kilometer area where these whale families reside (collecting over 3 TB/month) in tandem with on-whale robotic acoustic and video tags, underwater (robotic swimming fish) and aerial drones as well as other observation techniques in effort to augment rich contextual communication data. CETI's scientific team consists of specialists in machine learning, robotics, natural language processing, marine biology, linguistics, cryptography, signal processing and bio-acoustics. Andreas et al. [2] present CETI's initial scientific roadmap for understanding sperm whale communication, identifying the potential for unsupervised translation to be applied to whale communication. That roadmap suggests training a full generative LM for whale communication (often using trillions of bits for parameters [11, 12]). In contrast, our analysis suggests that the data requirements for translation may be similar to those of supervised translation, which is often several orders of magnitude smaller [37].
With this setting in mind, our requirements from source and target language are not symmetric: it would be unreasonable (and unnecessary) for our framework to assume that any sentence in the target language could be expressed in the source language. Put simply: whales need not understand what a smartphone is for us to gain some understanding of their communication. Also note, regarding domain gap, that some (although not all) knowledge can be inferred by training data from, e.g., online catalogs of hundreds of thousands of marine species [1]). Of course, there are also data-collection and transcription challenges, a challenge also present in the setting of low-resource (human) language translation [31]. While these challenges are outside the scope of this paper, our theoretical bounds on the data requirements may inform how much and what types of data are collected. For instance, it is less expensive to acquire textual data alone than both textual and video data. Therefore, if it is believed that an adequate translation is statistically possible using textual data alone, then greater effort may be placed on collecting this data and on UMT algorithms.
Unsupervised translation. In unsupervised machine translation [32], a translator between two languages is learned based only on monolingual corpora from each language. A body of work on UMT uses neural networks [29, 23, 5, 24, 35] or statistical methods [25, 4] for this task. Empirical evaluation of UMT found that it is outperformed by supervised machine translation, even when UMT is trained on several orders of magnitude more data [28, 22]. Among the key barriers for UMT identified in these evaluations are the domain gap and the data gap, and recent works propose techniques for bridging these gaps [15, 19]. Our theory suggests that sample complexity should remain roughly the same between the supervised and unsupervised settings, barring computational constraints. This, we hope, will encourage practitioners to bridge the remaining gaps.
Language models (LMs). In recent years, LMs such as GPT [11], BERT [13] and PaLM [12] were shown to achieve state-of-the-art performance on many tasks in natural language processing (NLP) such as text generation, summarization, or (supervised) MT. These models are indeed large, with hundreds of billions of parameters, and are pre-trained on hundreds of billions of tokens.
LMs are useful for machine translation in a variety of ways (e.g. [10, 18]). Of particular relevance are empirical works that use target LMs as priors to improve machine translation [27, 7]. To our knowledge, our work is the first theoretical work formally proving error bounds for prior-assisted translation. Appendix H discusses the use of LMs to establish priors for translation.
Goal-oriented communication. It is interesting to contrast our work with the work on goal-oriented communication, which was introduced by [21] and extended in [17]. They study the setting of two communicating parties (one of which is trying to achieve a verifiable goal) using each a language completely unknown to the other. They put forward a theory of goal-oriented communication, where communication is not an end in itself, but rather a means to achieving some goals of the communicating parties. Focusing on goals provides a way to address "misunderstanding" during communication, as in when one can verify whether the goal is (or is not) achieved. Their theory shows how to overcome any initial misunderstanding between parties towards achieving a given goal. Our setting is different: Informally, rather than be a participant in a communication with someone
Sentence	Probability
I just ate a giant squid.	1.5 × 10-9
I just ate a giant cheeseburger.	1.9 × 10-8
A sperm whale said: I just ate a giant squid.	6.8 × 10-14
A sperm whale said: I just ate a giant cheeseburger.	1.2 × 10-17
Figure 9: Without using a prompt, the sentence I just ate a giant cheeseburger is more likely, but using the prompt A sperm whale said:, the sentence I just ate a giant squid is much more likely. Probabilities are from the GPT-3 API.
speaking a different language, we wish to translate communications between two external parties speaking in a language unknown to us and there is no verifiable goal to aid us in this process.