SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Shafi Goldwasser*
UC Berkeley & Project CETI
shafi.goldwasser@berkeley.edu
David F. Gruber*
david@projectceti.org
Adam Tauman Kalai*
Microsoft Research & Project CETI
adam@kal.ai
Orr Paradise*
orrp@eecs.berkeley.edu
Neural networks are capable of translating between languages—in some cases even between two languages where there is little or no access to parallel translations, in what is known as Unsupervised Machine Translation (UMT). Given this progress, it is intriguing to ask whether machine learning tools can ultimately enable understanding animal communication, particularly that of highly intelligent animals. We propose a theoretical framework for analyzing UMT when no parallel translations are available and when it cannot be assumed that the source and target corpora address related subject domains or possess similar linguistic structure. We exemplify this theory with two stylized models of language, for which our framework provides bounds on necessary sample complexity; the bounds are formally proven and experimentally verified on synthetic data. These bounds show that the error rates are inversely related to the language complexity and amount of common ground. This suggests that unsupervised translation of animal communication may be feasible if the communication system is sufficiently complex.
Recent interest in translating animal communication [2, 3, 9] has been motivated by breakthrough performance of Language Models (LMs). Empirical work has succeeded in unsupervised translation between human-language pairs such as English-French [23, 5] and programming languages such as Python-Java [33]. Key to this feasibility seems to be the fact that language statistics, captured by a LM (a probability distribution over text), encapsulate more than just grammar. For example, even though both are grammatically correct, The calf nursed from its mother is more than 1,000 times more likely than The calf nursed from its father.[2]
Given this remarkable progress, it is natural to ask whether it is possible to collect and analyze animal communication data, aiming towards translating animal communication to a human language description. This is particularly interesting when the source language may be of highly social and intelligent animals, such as whales, and the target language is a human language, such as English.
Challenges. The first and most basic challenge is understanding the goal, a question with a rich history of philosophical debate [38]. To define the goal, we consider a hypothetical ground-truth translator. As a thought experiment, consider a "mermaid" fluent in English and the source language
*Authors listed alphabetically.
2 Probabilities computed using the GPT-3 API https://openai.com/api/ text-davinci-02 model.
A	B	C
Have you seen any orcas today? I just got back from the reef. p(A1) ≈ 10-18At the reef, there were a lot of sea turtles. p(A) ≈ 10-22	Have you seen mom? I just returned from the ocean basin. p(B1) ≈ 10-18At the reef, there were a lot of sea turtles. p(B) ≈ 10-26	Hat out hat dsjgh!!! p(C1) ≈ 10-48bicycle OMG and. p(C) ≈ 10-72
Figure 1: LMs identify incoherent text. The probabilities of three two-paragraph texts computed using the GPT-3 API. The probabilities of just the first paragraphs  $A_{1}$ ,  $B_{1}$ ,  $C_{1}$  are also shown. Although  $p(A_{1}) \approx p(B_{1})$  and the second paragraphs of  $A$  and  $B$  are identical, overall  $p(A) \gg p(B)$  due to coherence between the paragraphs.  $C$  is gibberish.
(e.g. sperm whale communication). Such a mermaid could translate whale vocalizations that English naturally expresses. An immediate worry arises: what about communications that the source language may have about topics for which English has no specific words? For example, sperm whales have a sonar sense which they use to perform echolocation. In that case, lacking a better alternative, the mermaid may translate such a conversation as (something about echolocation).<sup>3</sup>
Thus, we formally define the goal to be to achieve translations similar to those that would be output by a hypothetical ground-truth translator. While this does not guarantee functional utility, it brings the general task of unsupervised translation and the specific task of understanding animal communication into the familiar territory of supervised translation, where one can use existing error metrics to define (hypothetical) error rates.
The second challenge is that animal communication is unlikely to share much, if any, linguistic structure with human languages. Indeed, our theory will make no assumption on the source language other than that it is presented in a textual format. That said, one of our instantiations of the general theory (the knowledge graph) shows that translation is easier between compositional languages.
The third challenge is domain gap, i.e., that ideal translations of animal communications into English would be semantically different from existing English text, and we have no precise prior model of this semantic content. (In contrast, the distribution of English translations of French text would resemble the distribution of English text.) Simply put: whales do not "talk" about smartphones. Instead, we assume the existence of a broad prior that models plausible English translations of animal communication. LMs assign likelihood to an input text based not only on grammatical correctness, but also on agreement with the training data. In particular, LMs trained on massive and diverse data, including some capturing facts about animals, may be able to reason about the plausibility of a candidate translation. See Figure 1 and the discussion in Appendix H.
1.1 Framework and results
A translator $^4$  is a function  $f \colon \mathcal{X} \to \mathcal{Y}$  that translates source text  $x \in \mathcal{X}$  into the target language  $f(x) \in \mathcal{Y}$ . We focus on the easier-to-analyze case of lossless translation, where  $f$  is invertible (one-to-one) denoted by  $f_{\theta} : \mathcal{X} \hookrightarrow \mathcal{Y}$ . See Appendix I.1 for an extension to lossy translation.
We will consider a parameterized family of translators  $\{f_{\theta} \colon \mathcal{X} \to \mathcal{Y}\}_{\theta \in \Theta}$ , with the goal being to learn the parameters  $\theta \in \Theta$  of the most accurate translator. Accuracy (defined shortly) is measured with respect to a hypothetical ground-truth translator denoted by  $f_{\star} : \mathcal{X} \to \mathcal{Y}$ . We make a realizability assumption that the ground-truth translator can be represented in our family, i.e.,  $\star \in \Theta$ .
The source language is defined as a distribution  $\mu$  over  $x\in \mathcal{X}$ , where  $\mu (x)$  is the likelihood that text  $x$  occurs in the source language. The error of a model  $\theta \in \Theta$  will be measured in terms of  $\mathrm{err}(\theta)\coloneqq \operatorname *{Pr}_{x\sim \mu}[f_{\theta}(x)\neq f_{\star}(x)]$ , or at times a general bounded loss function  $\mathcal{L}(\theta)$ . Given  $\mu$  and  $f_{\star}$  it will be useful to consider the translated language distribution  $\tau$  over  $\mathcal{V}$  by taking  $f_{\star}(x)$  for  $x\sim \mu$ .
In the case of similar source and target domains, one may assume that the target language distribution  $\nu$  over  $\mathcal{Y}$  is close to  $\tau$ . This is a common intuition given for the "magic" behind why UMT sometimes
<sup>3</sup>A better description might be possible. Consider the fact that some people who are congenitally blind comprehend vision-related verbs as accurately as sighted people [8].
4In this work, a translator refers to the function  $f\colon \mathcal{X}\to \mathcal{Y}$  while translation refers to an output  $y = f(x)$
works: for complex asymmetric distributions, there may be a nearly unique transformation in  $\{f_{\theta}\}_{\theta \in \Theta}$  that maps  $\mu$  to something close  $\nu$  (namely  $f_{\star}$  which maps  $\mu$  to  $\tau$ ). A common approach in UMT is to embed source and target text as high-dimensional vectors and learn a low-complexity transformation, e.g., a rotation between these Euclidean spaces. Similarly, translator complexity will also play an important role in our analysis.
Priors. Rather than assuming that the target distribution  $\nu$  is similar to the translated distribution  $\tau$ , we will instead assume access to a broad prior  $\rho$  over  $\mathcal{V}$  meant to capture how plausible a translation  $y$  is, with larger  $\rho(y)$  indicating more natural and plausible translation. Appendix H discusses one way a prior oracle can be created, starting with an  $\mathrm{LM} \approx \nu$  learned from many examples in the target domain, and combined with a prompt, in the target language, describing the source domain.
We define the problem of unsupervised machine translation (with a prior) to be finding an accurate  $\theta \in \Theta$  given  $m$  iid unlabeled source texts  $x_{1},\ldots ,x_{m}\sim \mu$  and oracle access to prior  $\rho$ .
MLE. Our focus is on the Maximum-Likelihood Estimator (MLE), which selects model parameters  $\theta \in \Theta$  that (approximately) maximize the likelihood of translations  $\prod_{i}\rho \big(f_{\theta}(x_i)\big)$ .
Definition 1.1 (MLE). Given input a translator family  $\{f_{\theta} \colon \mathcal{X} \to \mathcal{Y}\}_{\theta \in \Theta}$ , samples  $x_1, \ldots, x_m \in \mathcal{X}$  and a distribution  $\rho$  over  $\mathcal{Y}$ , the MLE outputs
$$
\mathrm {M L E} ^ {\rho} \left(x _ {1}, x _ {2}, \dots , x _ {m}\right) := \underset {\theta \in \Theta} {\operatorname {a r g m i n}} \sum_ {i = 1} ^ {m} \log \frac {1}{\rho \left(f _ {\theta} \left(x _ {i}\right)\right)}
$$
If multiple  $\theta$  have equal empirical loss, it breaks ties, say, lexicographically.
We note that heuristics for MLE have proven extremely successful in training the breakthrough LMs, even though MLE optimization is intractable in the worst case.
Next, we analyze the efficacy of MLE in two complementary models of language: one that is highly structured (requiring compositional language) and one that is completely unstructured. These analyses both make strong assumptions on the target language, but make few assumptions about the source language itself. In both cases, the source distributions are uniform over subsets of  $\mathcal{X}$ , which (informally) is the "difficult" case for UMT as the learner cannot benefit from similarity in text frequency across languages. Both models are parameterized by the amount of "common ground" between the source language and the prior, and both are randomized. Note that these models are not intended to accurately capture natural language. Rather, they illustrate how our theory can be used to study the effect of language similarity and complexity on data requirements for UMT.
Knowledge graph model. Our first model consists of a pair of related knowledge graphs, in which edges encode knowledge of binary relations. Each edge yields a text that describes the knowledge it encodes. For example, in Figure 2 edges encode which animal  $A$  eats which other animal  $B$ , and text is derived as a simple description  $A$  eats  $B$ .<sup>5</sup>
Formally, there are two Erdős–Rényi random digraphs. The target graph is assumed to contain  $n$  nodes, while the source graph has  $r \leq n$  nodes corresponding to an (unknown) subset of the  $n$  target nodes. The model has two parameters: the average degree  $d$  in the (randomly-generated) target language graph, and the agreement  $\alpha \in (0,1]$  between the source and the target graphs. Here  $\alpha = 1$  is complete agreement on edges in the subgraph, while  $\alpha = 0$  is complete independence. We assume the languages use a compositional encoding for edges, meaning that they encode a given edge by encoding both nodes, so we may consider only  $|\Theta| = n! / (n - r)!$  translators consistently mapping the  $r$  source nodes to the  $n$  target nodes, which is many fewer than the number of functions  $f : \mathcal{X} \to \mathcal{Y}$  mapping the  $|\mathcal{X}| = O(r^2)$  source edges into the  $\mathcal{Y} = O(n^2)$  target edges. Human languages as well as the communication systems of several animals are known to be compositional [39].<sup>6</sup>
We analyze how the error of the learned translator depends on this "common knowledge"  $\alpha$ :
Theorem 1.2 (Theorem 3.2, simplified). Consider a source language  $\mu$  and a prior  $\rho$  generated by the knowledge graph model over  $r$  source nodes,  $n$  target nodes, average degree  $d$  and agreement
In a future work, it would be interesting to consider  $k$ -ary relations (hypergraphs) or multiple relations.
${}^{6}$  A system is compositional if the meaning of an expression is determined by the meaning of its parts.
Welsh sentences bwytaodd siarc siarc. bwytaodd eog bennog. bwytaodd siarc eog.
English sentences  
sharks eat salmon.  
salmon eat herring.  
sharks eat sharks.  
herring eat mysidacea.
Figure 2: An illustration of the knowledge graph model. In this example, the Welsh graph is an exact subgraph of the English knowledge graph, but our model allows for differences.
parameter  $\alpha$ . Then, with at least  $99\%$  probability, when given  $m$  source sentences  $x_{1},\ldots ,x_{m}$  and access to a prior  $\rho$ , MLE outputs a translator  $\hat{\theta}$  with error
$$
\operatorname {e r r} (\hat {\theta}) \leq O \left(\frac {\log n}{\alpha^ {2} d} + \frac {1}{\alpha} \sqrt {\frac {r \log n}{m}}\right).
$$
The second term decreases to 0 at a  $O(m^{-1/2})$  rate, similar to (noisy) generalization bounds [30]. Note that the first term does not decrease with the number of samples. The average degree  $d$  is a rough model of language complexity capturing per-node knowledge, while the agreement parameter  $\alpha$  captures the amount of common ground. Thus, more complex languages can be translated (within a given error rate) with less common ground. Even with  $m = \infty$  source data, there could still be errors in the mapping. For instance, there could be multiple triangles in the source and target graphs that lead to ambiguities. However, for complex knowledge relations (degree  $d \gg 1/\alpha^2$ ), there will be few such ambiguities.
Figure 2 illustrates an example of four English sentences and three sentences (corresponding to an unknown subset of three of the English sentences) in Welsh. For UMT, one might hypothesize that bwytaodd means eat because they both appear in every sentence. One might predict that siarc means shark because the word siarc appears twice in a single Welsh sentence and only the word shark appears twice in an English sentence. Next, note that eog may mean salmon because they are the only other words occurring with siarc and shark. Similar logic suggests that bennog means herring. Furthermore, the word order is consistently permuted, with subject-verb-object in English and verb-subject-object in Welsh. This translation is indeed roughly correct. This information is encoded in the directed graphs as shown, where each node corresponds to an animal species and an edge between two nodes is present if the one species eats the other.
"Common nonsense" model. The second model, the common nonsense model, assumes no linguistic structure on the source language. Here, we set out to capture the fact that the translated language  $\tau = \mu \circ f_{\star}$  and the prior  $\rho$  share some common ground through the fact that the laws of nature may exclude common "nonsense" outside both distributions' support.
Earlier work has justified alignment for UMT under the intuition that the target language distribution  $\nu$  is approximated by a nearly unique simple transformation, e.g., a rotation, of the source distribution  $\mu$ . However, for a prior  $\rho$ , our work suggests that UMT may also be possible if there is nearly a unique simple transformation that maps  $\tau$  so that it is contained in  $\rho$ . Figure 3 illustrates such a nearly unique rotation—the UMT "puzzle" of finding a transformation  $f_{\theta}$  of  $\mu$  which is contained within  $\rho$  is subtly different from finding an alignment.
In the common nonsense model,  $\tau$  and  $\rho$  are uniform over arbitrary sets  $T \subseteq P \subseteq \mathcal{V}$  from which a common  $\alpha \in (0,1/2]$  fraction of text is removed (hence the name "common nonsense"). Specifically,  $\tau$  and  $\rho$  are defined to be uniform over  $\tilde{T} = T \setminus S$ ,  $\tilde{P} = P \setminus S$ , respectively, for a set  $S$  sampled by including each  $y \in \mathcal{V}$  with probability  $\alpha$ .
We analyze the error of the learned translator as a function of the amount of common nonsense:
Theorem 1.3 (Theorem 3.4, simplified). Consider source language  $\mu$  and a prior  $\rho$  generated by the common nonsense model over  $|T|$  source texts and common-nonsense parameter  $\alpha$ , and a translator family parameterized by  $|\Theta|$ . Then, with at least  $99\%$  probability, when given  $m$  source sentences  $x_{1}, \ldots, x_{m}$  and access to a prior  $\rho$ , MLE outputs a translator  $\hat{\theta}$  with error
$$
\operatorname {e r r} \left(\hat {\theta}\right) := \Pr_ {x \in X} \left[ f _ {\theta} (x) \neq f _ {\star} (x) \right] = O \left(\frac {\ln | \Theta |}{\alpha \min  (m , | T |)}\right).
$$
Figure 3: The previous intuition behind UMT has the distributions of target language  $\nu$  (middle) close to ground-truth translations  $\tau$ , which is assumed to be a low-complexity transformation (in this example a rotation) of the source language  $\mu$  (left). When source and target are not aligned, restricting to prior  $\rho$  region (right) allows for translation, as long as there are enough "nonsense" texts (black regions) so that there is a nearly unique rotation of  $\mu$  that is contained in  $\rho$ . For example, both distributions may assign negligible probability to nonsensical texts such as I died 3 times tomorrow. (In this toy example,  $\mu$  is uniform over a two-dimensional shape that happens to look like a whale.)
Theorem 3.5 gives a nearly matching lower bound. Let us unpack the relevant quantities. First, we think of  $\alpha$  as measuring the amount of agreement or common ground required, which might be a small constant. Second, note that  $|T|$  is a coarse measure of the complexity of the source language, which requires a total of  $\tilde{O}(|T|)$  bits to encode. Thus, the bound suggests that accurate UMT requires the translator to be simple, with a description length that is an  $\alpha$ -factor of the language description length, and again  $\alpha$  captures the agreement between  $\tau, \rho$ . Thus, even with limited common ground, one may be able to translate from a source language that is sufficiently complex. Third, for simplicity, we require  $\mathcal{X}, \mathcal{Y} \subset \{0,1\}^*$  to be finite sets of binary strings, so WLOG  $\Theta$  may be also assumed to be finite. Thus,  $\log_2 |\Theta|$  is the description length, a coarse but useful complexity measure that equals the number of bits required to describe any model. (Neural network parameters can be encoded using a constant number of bits per parameter.) Appendix I.2 discusses how this can be generalized to continuous parameters.
Importantly, we note that (supervised) neural machine translators typically use far fewer parameters than LMs. To see why, consider the example of the nursing calf (page 1) and the fact that a translator needs not know that calves nurse from mothers. On the other hand, such knowledge is essential to generate realistic text. Similarly, generating realistic text requires maintaining coherence between paragraphs, while translation can often be done at the paragraph level.
As a warm-up, we include a simplified version of the common nonsense model, called the tree-based model (Appendix B.1), in which texts are constructed word-by-word based on a tree structure.
Comparison to supervised classification. Consider the dependency on  $m$ , the number of training examples. Note that the classic Occam bound  $O\left(\frac{1}{m}\log |\Theta |\right)$  is what one gets for noiseless supervised classification, that is, when one is also given labels  $y_{i} = f_{\star}(x_{i})$  at training time, which is similar to Theorem 1.3, and give  $\tilde{O} (m^{-1 / 2})$  bounds for noisy classification as in Theorem 1.2. Furthermore, these bounds apply to translation, which can be viewed as a special case of classification with many classes  $\mathcal{V}$ . Thus, in both cases, the data dependency on  $m$  is quite similar to that of classification.
Experiments. We validate our theorems generating synthetic data from randomly-generated languages according to each model, and evaluating translator error as a function of the number of samples and amount of common ground. The knowledge graph model (Figure 4, left) is used to generate a source graph (language) on  $r = 9$  nodes to a target graph (language) on  $n = 10$  nodes and average degree  $d \approx 5$ , while varying the agreement parameter  $\alpha$ . We also vary  $r$  (Figure 4, right) supporting our main message: more complex languages can be translated more accurately. For the common nonsense model (Figure 5) we simulate translation of a source language of size  $|T| = 10^5$  while varying the fraction of common nonsense  $\alpha$ . Appendix E contains details and code.
For example, a multilingual model achieves state-of-the-art performance using only 5 billion parameters [37], compared to 175 billion for GPT-3 [11].
Figure 4: Knowledge Graph model experiments, each run on twenty seeds with standard errors shown. Left: error of the top-scoring translator vs. number of source samples  $m$ . Right: effect of source language complexity (number of source nodes  $r$ ) on translator accuracy in the knowledge graph model. We report the accuracy of the top-scoring translator after all source edges were input to the learning algorithm, i.e., as the number of samples  $m \to \infty$ .
Figure 5: Common Nonsense model. The X-axis is the number of source samples  $m$ , and the Y-axis is the average error among plausible translators (that have not been ruled-out so far). Each experiment was run on five seeds, with standard error depicted by the shaded area.
Contributions. The first contribution of this work is formalizing and analyzing a model of UMT. As an initial work, its value is in the opportunities which it opens for further work more than the finality and tightness/generality of its bounds. Our model applies even to low-resource source languages with massive domain gap and linguistic distance. We emphasize that this work is only a first step in the theoretical analysis of UMT (indeed, there is little theoretical work on machine translation in general). Second, we exhibit two simple complementary models for which we prove that: (a) more complex languages require less common ground, and (b) data requirements may not be significantly greater than those of supervised translation (which tends to use less data than training a large LM). These findings may have implications for the quantity and type of communication data that is collected for deciphering animal communication and for UMT more generally. They also give theoretical evidence that UMT can be successful and worth pursuing, in lieu of parallel (supervised) data, in the case of sufficiently complex languages. All of that said, we note that our sample complexity bounds are information theoretic, that is, they do not account for the computational complexity of optimizing the translator. Finally, animal communication aside, to the best of our knowledge this work is the first theoretical treatment of UMT, and may also shed light on translation between human languages.
Organization. The framework is formally described in Section 2, and is instantiated with models of language in Section 3 (proofs, experiments, and other details deferred to Appendices A to E). Key takeaways from this work are presented in Section 4. Related and future work is discussed in Appendices F and G. We illustrate how prompting LMs may give priors in Appendix H. Appendix I sketches