SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Subgraph isomorphism. For simplicity, we model the knowledge graphs of Section 3.1 as a pair of correlated Erdős–Rényi (ER) graphs. The computational problem of identifying a subgraph of an ER graph has been studied by Erdős and others [6, 26, 20]. In particular, [20] consider a model in which two correlated graphs  $P, T$  are derived from a "parent graph"  $G$  by independently deleting rows and edges  $G$ , and then applying a permutation  $\pi^*$  to the vertices of  $T$ . Although their model differs from our knowledge graph model,[15] they propose efficient algorithms for recovering the latent permutation  $\pi^*$  and provide an empirical evaluation on synthetic and real-world data. Given the similarity between our models, it would be interesting to see if their algorithm can be adapted to our setting, which would nicely complement our formally-proven-yet-inefficient algorithm.
G Future Work
Our initial exploration leaves plenty of room for future work. In particular, we propose the following possible directions:
1. In our lossless models, the target language subsumes the source language in the sense that everything that is representable in the source language can also be represented in the target language. It would be interesting to extend our work to the partially-overlapping case.
2. The language distribution in our models are all uniform. It would be interesting to examine non-uniform distributions such as Zipfian or power-law distributions.
3. As stated earlier, our analysis is purely information-theoretic and leaves the question of the efficiency of UMT open.
4. A good starting point for the efficiency question would be to design efficient UMT algorithms for one of the randomized models of language presented in this paper.
H Good Priors through Prompts
The most direct way to improve a prior is to train (or fine-tune) the LM on a dataset that includes a large number of articles relevant to the source language, e.g., volumes of oceanic research, for whales. In addition, training on a wide variety of sources including multiple languages, diverse sources, and encoding systems may be helpful. It is possible that a system that has how to transfer knowledge between hundreds of languages and even programming languages, may have a better prior.
Another general strategy for creating a prior is to use prompting: Given a prompt string  $s$ , one can define  $\rho(y) \propto \nu(sy)$ , that is the prior distribution of text that that LM generates conditioned on the text beginning with  $s$ . Figure 9 illustrates some toy examples of how even a simple prompt like A sperm whale said: can help focus on translations that are more likely for a sperm whale to say, and eliminate irrelevant translations.
In the knowledge graph (a) the vertices of  $T$  are always a subset of the vertices of  $P$ , and (b) the deleted vertices are fixed rather than randomly chosen
Background prompts. There is a natural and potentially powerful idea that an unsupervised translator, in addition to outputting translations, would automatically generate a background prompt that increases the intelligibility of many translations.[16] Suppose, for example, across numerous communications, the unsupervised translator determines that sperm whales measure time in "number of naps" and that typical nap duration varies by age. Then, rather than having to repeatedly explain this in each translation, it can be explained once in a background prompt that is automatically inserted before each translation. For example, following the prompt
$s =$  Sperm whales measure time in numbers of naps, but each whale's typical nap duration depends on their age. So if a whale's nap duration is 9 minutes, then 4 naps is about 36 minutes (though whales tend to exaggerate times). A sperm whale said:
would make translations like Wow, I just dove for 6 naps or How many naps ago was that? both more likely and more understandable.
I Generalizations of the framework
I.1 Lossy Translation
Many things can be included in textual transcriptions suitable for translation. For instance, one can distinguish the speakers, e.g., "Whale 1: ... Whale 2: ... Whale 1: ..." if the source data is annotated with speaker identifiers. Some aspects of the way one is speaking can be transcribed, e.g., "Whale 1 (fast tempo clicking): ... Whale 2 (slow, loud clicking): ..." It may be possible to encode these textually in  $x$ .
However, if  $x$  is encoded in a more flexible format, such as arbitrary binary files, one can include as much raw information as possible, including the source audio recording, to provide the translator with as much context as possible. In that case, lossless translation will no longer be possible, because one cannot compute the raw  $x$  from a textual translation.
Given the possible benefits of such annotations, we propose an extension of our theory to the lossy setting. A natural generalization of the maximum-likelihood approach is as follows:
$$
\min  _ {\theta \in \Theta} \frac {1}{m} \sum_ {i = 1} ^ {m} - \log \rho \left(f _ {\theta} \left(x _ {i}\right)\right) - \frac {1}{\lambda} \log \phi_ {\theta} \left(x _ {i} \mid y = f _ {\theta} \left(x _ {i}\right)\right).
$$
Here  $\phi_{\theta} \colon \mathcal{X} \times \mathcal{Y} \to [0,1]$  is a probabilistic inverse ("randomized back translation") of  $f$  whose parameters are also encoded in  $\theta$ . Note that the family  $\{(f_{\theta}, \phi_{\theta}) \mid \theta \in \Theta\}$  must satisfy that for all  $y \in \mathcal{Y}$ ,  $\sum_{x \in f^{-1}(y)} \phi_{\theta}(x \mid y) = 1$ , though it is no longer required that  $f_{\theta}$  be 1-1.
As  $\lambda$  decreases to 0, the optimal solution would assign infinite loss to any  $f_{\theta} \colon \mathcal{X} \hookrightarrow \mathcal{Y}$  and  $\phi_{\theta}$  that are not perfect inverses, where there is some  $x$  (with positive probability under  $\mu$ ) such that  $\phi_{\theta}(x \mid y = f_{\theta}(x)) < 1$ . Thus, for sufficiently small  $\lambda$ , the algorithm is exactly the minimum cross-entropy (maximum likelihood) algorithm of Definition 1.1. For arbitrarily large  $\lambda$ , the algorithm will collapse to always outputting the most likely  $y \in \mathcal{Y}$  under  $\rho$ . For example, everything could be translated to Hello regardless of its contents.
For intermediate values of  $\lambda$ , the chosen translator trades off naturalness in the form of  $\rho(f_{\theta}(x))$  versus information loss which is inversely related to  $\phi_{\theta}(x_i \mid y = f_{\theta}(x_i))$ . This trade-off makes it challenging to define and analyze the success of a lossy unsupervised translation algorithm. Nonetheless, the algorithm is intuitive.
I.2 Infinite parameter sets  $\Theta$
In some cases, one can learn with many fewer examples, which is important when parameters are real-valued and  $|\Theta| = \infty$  or when the model is over-parameterized. However, one can analyze even these cases in our model using the following "trick." Suppose one has a supervised translation algorithm SUPER that takes  $m$  labeled examples  $(x_i, y_i)$  as input and outputs  $\theta \in \Theta$ . A simple
<sup>16</sup>In general, the problem of AI-based prompt generation has recently attracted attention, e.g., [34].
observation in these cases is that one could use an initial set of  $m$  unlabeled examples,  $x_{1},\ldots ,x_{m}$ , to define a subset of translators:
$$
\bar {\Theta} := \left\{\operatorname {S U P E R} \left(\left(x _ {1}, y _ {1}\right), \left(x _ {2}, y _ {2}\right), \dots , \left(x _ {m}, y _ {m}\right)\right) \mid y _ {1}, y _ {2}, \dots , y _ {m} \in \mathcal {Y} \right\}.
$$
That is, we restrict attention to the set of possible translators that we could output for any given ground-truth translations, then it is not difficult to see  $\log |\overline{\Theta}|\leq m\log |\mathcal{V}|$ . If we assume that one of these is accurate and natural, then restricting the attention of MLE to this set will suffice, and Theorem A.2 means that the number of examples required is  $O(\log |\overline{\Theta}|) = O(m\log |\mathcal{V}|)$  which is a linear blowup in the number of examples  $m$  used for supervised translation. To make this formal, one would start from only assuming that one of the translators had negligible error—this is left to future work.
J Bounds for Supervised Translation
In this work, for ease of presentation, we have stated that error decreases like  $O\left(\frac{1}{m}\log \frac{|\Theta|}{\delta}\right)$  for noiseless supervised translation. This is based in the classic Occam bound for supervised learning:
Theorem J.1 (Occam bounds). Let  $\mathcal{X},\mathcal{Y}$  be sets,  $\mathcal{D}$  be a joint distribution over  $\mathcal{X}\times \mathcal{Y}$ ,  $\ell \colon \mathcal{Y}\times \mathcal{Y}\to [0,1]$  be a loss, and  $f_{\theta}:\mathcal{X}\rightarrow \mathcal{Y}$  be a family of functions parameterized by  $\theta \in \Theta$ , and  $\mathcal{L}(\theta)\coloneqq \mathbb{E}_{(x,y)\sim \mathcal{D}}[\ell (y,f_{\theta}(x))]$ . For any  $\delta >0$
$$
\Pr_ {(x _ {1}, y _ {1}), \dots , (x _ {m}, y _ {m}) \sim \mathcal {D} ^ {m}} \left[ \mathcal {L} (\hat {\theta}) \leq \frac {1}{m} \ln \frac {| \Theta |}{\delta} \right] \geq 1 - \delta \quad i f \mathcal {L} (\star) = 0,
$$
$$
\operatorname * {P r} _ {(x _ {1}, y _ {1}), \dots , (x _ {m}, y _ {m}) \sim \mathcal {D} ^ {m}} \left[ \mathcal {L} (\hat {\theta}) \leq \mathcal {L} (\star) + \sqrt {\frac {1}{m} \ln \frac {| \Theta |}{\delta}} \right] \geq 1 - \delta \quad i f \mathcal {L} (\star) \neq 0,
$$
$$
f o r \hat {\theta} := \operatorname {a r g m i n} _ {\theta \in \Theta} \sum_ {i} \ell \left(y _ {i}, f _ {\theta} \left(x _ {i}\right)\right). ^ {1 7}
$$
Note that supervised translation is simply classification with many classes  $\mathcal{V}$ .
<sup>17</sup>As in MLE, ties can be broken arbitrarily, e.g., lexicographically. Our bounds, like the Occam bound, hold simultaneously for all minimizers. In the realizable case, the Empirical Risk Minimizer  $\hat{\theta}$  will have  $\sum \ell(y_i, f_\theta(x_i)) = 0$ .