SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathcal{A}_{\gamma}:= \left\{\theta \in \Theta \bigg|  \operatorname *{Pr}_{y\sim \tau}\left[\rho (\pi_{\theta}(y)) = 0\right]\leq \gamma \right\} ,and\varepsilon_{\gamma}:= \max_{\theta \in \mathcal{A}_{\gamma}}\mathcal{L}(\theta).
$$
For example, left  $\leftrightarrow$  right would constitute a  $\gamma$ -plausible ambiguity if one could swap the two words, making  $a \leq \gamma$  fraction of the translations have 0 probability. Such a revision would have low loss if such swaps are considered similar in meaning. Condition 2.1 is equivalent to  $\star \in \mathcal{A}_0$ .
The quantity of interest is  $\varepsilon_{\gamma}$ , the maximum loss of any  $\gamma$ -plausible ambiguity.12 Next, we prove that the loss of the translator output by the Maximum Likelihood Estimator is not greater than  $\varepsilon_{\gamma}$  given  $m \geq \frac{1}{\gamma} \ln \frac{|\Theta|}{\delta}$  examples.
Theorem A.2. Let  $\mu, \rho$  be probability distributions over  $\mathcal{X}, \mathcal{Y}$ , resp., and  $f_{\star}$  satisfy Condition 2.1:  $\operatorname*{Pr}_{\mu}[\rho(f_{\star}(x)) = 0] = 0$ . Fix any  $\delta \in (0,1)$ ,  $m \geq 1$ , and let  $\gamma \geq \frac{1}{m} \ln \frac{|\Theta|}{\delta}$ . Then,
$$
\Pr_ {x _ {1}, \dots , x _ {m} \sim \mu} \left[ \mathcal {L} (\hat {\theta}) \leq \varepsilon_ {\gamma} \right] > 1 - \delta ,
$$
where  $\hat{\theta} = \mathrm{MLE}^{\rho}(x_1,\dots,x_m)$  and  $\varepsilon_{\gamma}$  is from Definition A.1.
Since  $\varepsilon_{\gamma}$  is non-decreasing in  $\gamma$ , as the number of examples increases the loss bound  $\varepsilon_{\gamma}$  decreases to approach  $\varepsilon_0$ . This bound and its proof in Appendix A are analogous to the realizable Occam bound of supervised classification, see Appendix J.
Proof of Theorem A.2. MLE minimizes  $\hat{v}(\theta) \coloneqq \frac{1}{m}\sum_{i\leq m}\log \frac{1}{\rho(\pi_{\theta}(y_i))}$  over  $\theta \in \Theta$ , where  $y_i = f_\theta(x_i)$ . By the realizable prior assumption  $\hat{v}(\star) < \infty$ . Thus, for the algorithm to fail, there must be a "bad"  $\hat{\theta} \in B \coloneqq \{\theta \in \Theta \mid \mathcal{L}(\theta) > \varepsilon_\gamma\}$  with  $\hat{v}(\theta) \leq \hat{v}(\star) < \infty$ . If  $\rho(\pi_{\theta}(y_i)) = 0$  for any  $i$ , then  $\hat{v}(\pi_{\theta}) = \infty$ . Note that by Definition A.1,  $B \cap \mathcal{A}_\gamma = \emptyset$ , thus for all  $\theta \in B$ :  $\operatorname*{Pr}[\rho(\pi_{\theta}(y)) = 0] > \gamma$  and,
$$
\operatorname * {P r} \left[ \hat {v} (\theta) <   \infty \right] <   (1 - \gamma) ^ {m} \leq e ^ {- \gamma m} \leq \frac {\delta}{| \Theta |}.
$$
By the union bound over  $\theta \in B$ ,  $\operatorname*{Pr}[\exists \theta \in B \hat{v}(\theta) < \infty] < \delta$  since  $|B| \leq |\Theta|$ .
B The Tree-based Model
B.1 A tree-based probabilistic model of language
The last model, which we view as secondary to the previous two, is a simpler version of the common nonsense model. It can be viewed as a "warm-up" leading up to that more general model, and we
Formally,  $f_{\theta} \circ f_{\star}^{-1} \colon f_{\star}(\mathcal{X}) \hookrightarrow \mathcal{Y}$  is only defined on  $f_{\star}(\mathcal{X})$  but can be extended to a full permutation on  $\mathcal{Y}$  in many ways. For concreteness, we take the lexicographically smallest extension on  $\mathcal{Y} \subseteq \{0, 1\}^*$ .
12For intuition, note that  $\varepsilon_{\gamma}$  can be bounded using  $\gamma \colon \frac{\gamma}{\varepsilon_{\gamma}}\geq \min_{\theta \in \Theta}\operatorname{Pr}_{y\sim \tau}\left[\rho (\pi_{\theta}(y)) = 0\mid \pi_{\theta}(y)\neq y\right]$ .
Figure 6: An example of a language tree of plausible texts and the subtree of ground-truth translations illustrated in green.
hope it helps illuminate that model by instantiating the language therein with a simple tree-based syntax.
In the tree-based model, the nodes of a tree are labeled with random words, and plausible texts (according to the prior  $\rho$ ) correspond to paths from root to leaf. A translated language  $\tau$  (or equivalently, a source language  $\mu$ ) is then derived from root-to-leaf paths in a random subtree  $H \subseteq G$ . We prove that, with high probability, a prior  $\rho$  and translated language  $\tau$  sampled by this process satisfy Condition 2.1 and semantic error  $\varepsilon_{\gamma} = O\big(1 / \min(m, a^n)\big)$ . Therefore, MLE yields a semantically accurate translator for the sampled language with high probability, for sufficiently large  $n$ .
The random tree language (RT) model is a randomized process for generating a tree-based source language  $\mu$ , translated language  $\tau$ , and prior  $\rho$ . It is parameterized by a finite vocabulary set  $\mathcal{W}$ , depth  $n \in \mathbb{N}$ , and arities  $a, b \in \mathbb{N}$  such that  $1 \leq a \leq b \leq |\mathcal{W}| / 4$ . For simplicity, the possible target texts are taken to be the set of all possible  $n$ -grams over  $\mathcal{W}$ , namely,  $\mathcal{Y} := \mathcal{W}^n$ . Again for simplicity, we also assume  $|\mathcal{X}| = |\mathcal{Y}|$  so that each  $f_{\theta} \colon \mathcal{X} \hookrightarrow \mathcal{Y}$  is a bijection. To generate an RT, first a full  $b$ -ary, depth  $n + 1$  tree  $G$  is constructed. Labeled edges shall correspond to words, and paths shall correspond to texts (sentences), as follows:
1. Starting from the root node and proceeding in level-order traversal, for each node  $v$ : sample  $b$  words  $w_{1},\ldots ,w_{b}\in \mathcal{W}$  uniformly at random and without replacement; label each of  $v$ 's child edges with one of the sampled words.
2. The labels on each path from the root to a leaf  $(y_{1},\ldots ,y_{n})$  corresponds to a plausible text, giving a set of plausible texts
$$
P := \left\{y \mid y = \left(y _ {1}, \dots , y _ {n}\right) \text {l a b e l s} G \right\}.
$$
3. A subtree  $H \subseteq G$  is obtained by sampling uniformly at random  $a$  out of  $b$  of the children at each level of the tree, in level-order traversal. The set of translated texts is analogously defined as
$$
T := \left\{y \mid y = \left(y _ {1}, \dots , y _ {n}\right) \text {l a b e l s o f a p a t h i n} H \right\} \subseteq P.
$$
The prior  $\rho = \mathcal{U}(P)$  is uniform over plausible texts  $P$ , while  $\mu =$  is uniform over  $f_{\star}^{-1}(T)$ . We let  $(\mu, \rho) \sim \mathrm{RT}(\mathcal{W}, n, a, b)$  denote the sampling of a prior and translated language obtained via this randomized process. When the parameters  $\mathcal{W}, n, a, b$  are clear from context we simply write RT.
Note that  $\Theta$  and  $f_{\star} \in \Theta$  may be arbitrary, and by definition of  $\tau$ ,  $\mu = \mathcal{U}(f_{\star}^{-1}(T))$  is uniform over  $T$ 's preimage. Since we assumed  $|\mathcal{X}| = |\mathcal{Y}|$  above,  $f_{\star}$  is invertible.
Next, we will argue that a random tree language sampled in this process satisfies realizability (Condition 2.1) and Definition A.1 with appropriate choice of parameters. While we make no assumptions on  $\Theta$ , it is important that the plausible texts  $P$  are sampled after the possible permutations (ambiguities)  $\Pi$  are specified. Otherwise, if an adversary were able to choose a permutation  $\pi: \mathcal{Y} \to$
$\mathcal{V}$  based on  $P$ , then they could arbitrarily permute  $P$ , resulting in a permutation  $\pi$  with high expected loss but no change in likelihood according to the prior  $\rho$ —thereby violating Definition A.1.
You can think of Theorem B.1 as pointing to the required text length  $n$  and an upper bound on the number of parameters  $|\Theta|$  for which Condition 2.1 and Definition A.1 hold with high probability. As we know from Theorem A.2, and state clearly next, these in turn imply translatability of a random tree language.
Theorem B.1 (Translatability in the RT model). Fix any  $m \geq 1$ ,  $\delta \in (0,1)$  vocabulary set  $\mathcal{W}$ , and tree arities  $a \leq b \leq |\mathcal{W}|/4$ . Then, for any  $\Theta$  and any  $\star \in \Theta$ , with probability at least  $1 - \delta$  over  $(\mu, \rho) \sim \mathrm{RT}(\mathcal{W}, n, a, b)$  and iid samples  $x_1, x_2, \ldots, x_m \sim \mu$ ,
$$
\operatorname {e r r} (\hat {\theta}) \leq 1 6 \max  \left(\frac {1}{m}, \frac {4}{a ^ {n}}\right) \cdot \ln \frac {6 | \Theta |}{\delta},
$$
where  $\hat{\theta} = \mathrm{MLE}^{\rho}(x_1, x_2, \ldots, x_m)$  and MLE is from Definition 1.1
Note that the probability in the corollary is over both  $(\mu, \rho) \sim \mathrm{RT}$  and the  $m$  iid training samples. Again, note how the first term is similar to the  $\frac{1}{m} \log \frac{|\Theta|}{\delta}$  term from realizable supervised learning (see Theorem J.1), and the second term is additional due to the unsupervised case. The proof, given in Appendix B, uses Theorem A.2 and a lemma stating that  $\varepsilon_{\gamma} \leq 16\gamma$  for  $\gamma$  that are not too small. The main challenge is that there are dependencies between the paths, so one cannot directly use standard concentration inequalities.
To better understand Theorem B.1, let us consider two possible families of translators and suppose we are trying to apply Theorem B.1 to get bounds on the sample complexity  $m$  needed to translate  $(\mu, \rho) \sim \mathrm{RT}(\mathcal{W}, n, a, b)$  with small constant loss and small constant failure probability.
First, since  $|\mathcal{X}| = |\mathcal{Y}| = |\mathcal{W}|^n$ , the above bound is meaningless (bigger than 1) if  $\Theta$  is the family of all translators, as the set of all translators has  $\log(|\mathcal{X}|!) = O(|\mathcal{W}|^n \cdot n \log |\mathcal{W}|)$  parameters which is much larger than  $a^n$ . In other words, no free lunch.
On the other hand, let us consider the family of word-for-word translators  $\Theta_{\mathrm{w}}$ , which work by translating each word in a text separately, ignoring the surrounding context. The number of such translators is the same as the number of word-to-word permutations, i.e.,  $|\Theta_{\mathrm{w}}| = |\mathcal{W}|!$ . So Theorem B.1 gives a sample complexity bound of  $m = O(\log \mathcal{W}!) = O(|\mathcal{W}|\log |\mathcal{W}|)$ . The number of words in the English language is about  $N \approx 10^5$ . The quantity  $a$  is equal to what is called the perplexity of  $\mu$ , the effective number of words that may typically appear after a random prefix. For a constant  $a$ , the minimum text (or communication) length  $n = O(\log N)$  needed is thus logarithmic in the vocabulary size. While word-for-word translators are poor, this analysis still sheds some light on the possible asymptotic behavior (barring computational constraints) of translation, at least in a simplistic model.
Common tree analysis. The tree model helps us demonstrate the generality of the common nonsense model. Instead of the fixed arities  $a \leq b$  in the tree model, one can consider an arbitrary language tree over plausible texts  $P$  and an arbitrary subset  $T \subseteq P$  corresponding to a subtree of ground-truth translations. The common nonsense model implies that if the sets are perturbed by removing, say, a  $\alpha = 0.01$  common fraction of translations, then with high probability MLE's error decreases at an  $\tilde{O}(1 / \min(m, |T|))$  rate. Note that Theorem 3.4 does not directly apply to the random tree LM because in that LM, the choice of which branches to include are not independent due to the fixed  $a$ -arity constraint.
B.2 Proof of Theorem B.1
The proof of Theorem B.1 follows from Lemma B.2 below. In this section, we will prove this lemma and then derive the theorem from it.
Lemma B.2 (RT conditions). Consider any vocabulary set  $\mathcal{W}$ , tree arities  $a \leq b \leq |\mathcal{W}|/4$ , tree depth  $n \in \mathbb{N}$ . Then, for any  $\delta \in (0,1)$  and  $\gamma \geq 4a^{-n} \ln(4|\Theta|/\delta)$ , with probability  $\geq 1 - \delta$ ,  $(\mu, \rho)$  give,
$$
\Pr_ {(\mu , \rho) \sim \mathrm {R T}} [ \varepsilon_ {\gamma} \leq 1 6 \gamma ] \geq 1 - \delta . \tag {4}
$$
Moreover, any  $\mu, \rho$  sampled from  $\mathrm{RT}(\mathcal{W}, n, a, b)$  satisfy  $\operatorname*{Pr}_{x \sim \mu}[\rho(f_{\star}(x)) = 0] = 0$  as needed for the realizability Condition 2.1.
The proof of Lemma B.2 requires two additional lemmas, stated and proved next. The first is a known variant of the Chernoff bound for so-called 0-negatively correlated random variables.
Lemma B.3 (Chernoff bound for 0-negatively correlated random variables). Let  $Z_{1}, \ldots, Z_{n}$  be 0-negatively correlated Boolean random variables, that is, they satisfy
$$
\forall I \subseteq [ n ] \quad \Pr \left[ \forall i \in I   Z _ {i} = 0 \right] \leq \prod_ {i \in I} \Pr \left[ Z _ {i} = 0 \right].
$$
Then, letting  $Z \coloneqq \sum_{i=1}^{n} Z_i$  it holds that
$$
\operatorname * {P r} \left[ Z \leq \frac {\mathbb {E} [ Z ]}{2} \right] \leq e ^ {- \frac {\mathbb {E} [ Z ]}{8}}
$$
Lemma B.3 follows from [14, Theorem 1.10.24(a)] with  $\delta \coloneqq 1 / 2$ ,  $a_{i} \coloneqq 0$ ,  $b_{i} \coloneqq 1$  and  $Y_{i} \coloneqq X_{i}$ . That theorem, in turn, is simply Theorem 1.10.10 for 0-negatively correlated random variables.
The second lemma used for the proof of Lemma B.2 is a combinatorial argument that we prove below.
Lemma B.4. Given any  $\pi : \mathcal{Y} \hookrightarrow \mathcal{Y}$ , it is possible to partition  $A = \{y \in \mathcal{Y} \mid \pi(y) \neq y\}$  into four disjoint sets  $A = A_{1} \cup A_{2} \cup A_{3} \cup A_{4}$  such that, for each  $i = 1, 2, 3, 4$ , the following two conditions hold:
$$
\pi (A _ {i}) \cap A _ {i} = \emptyset
$$
$$
\forall z \in \mathcal {W} ^ {n - 1} | \{y \in A _ {i} \mid y b e g i n s w i t h z \} | \leq \frac {| \mathcal {W} |}{2}
$$
Proof. of Lemma B.4. We will partition  $A$  greedily to achieve the two conditions. Begin with four empty sets  $A_{1} = A_{2} = A_{3} = A_{4} = \emptyset$ . For each  $y \in A$  in turn, assign it to one of the 4 sets as follows.
1. Let  $i$  be the index of the set  $A_{i}$  such that  $\pi(y) \in A_{i}$  has already been assigned to. If  $\pi(y)$  has not yet been assigned, let  $i = 1$ .
2. Similarly, if  $\pi(y) \in A_j$  has been assigned, let  $j$  be its index otherwise  $j = 1$ .
3. Let  $z$  be the first  $n - 1$  elements of  $x$ . Let  $k$  be the index of the set  $A_{k}$  such that  $|\{y \in A_{i} \mid y \text{ begins with } z\}| \geq |\mathcal{W}| / 2$  if there is such a set, otherwise  $k = 1$ . Note that there can be at most one such  $k$  because there are at most  $|\mathcal{W}| - 1$  other elements beginning with  $z$  that have been assigned already.
Thus  $S = \{1,2,3,4\} \setminus \{i,j,k\} \neq \emptyset$  and we can assign  $x$  to any (say the minimum) element in that set. By induction, we preserve the two properties stated in the lemma.
With Lemmas B.3 and B.4 in hand, we are ready to prove Lemma B.2.
Proof. of Lemma B.2. The realizability Condition 2.1,  $\operatorname{Pr}_{x\sim \mu}[\rho (f_{\star}(x)) = 0] = 0$ , follows immediately from the fact that  $\rho$  is uniform over  $P$  and that  $\tau$  is supported on  $T\subseteq P$ . To prove the lemma, it suffices to show that for any  $\gamma \geq 4a^{-n}\ln (4|\Theta | / \delta)$
$$
\operatorname * {P r} _ {(\mu , \rho) \sim \mathrm {R T}} \left[ \exists \theta \in \Theta \mathrm {e r r} (\theta) > 1 6 \gamma , \operatorname * {P r} _ {y \sim \tau} [ \rho (\pi_ {\theta} (y)) = 0 ] \leq \gamma \right] \leq \delta .
$$
Note that  $\mathrm{err}(\theta) = \operatorname{Pr}_{y \sim \tau}[y \neq \pi_{\theta}(y)]$ , and that  $\rho(\pi_{\theta}(y)) = 0$  if and only if  $\pi_{\theta}(y) \notin P$ . Therefore, by the union bound over  $\theta \in \Theta$ , it suffices to show that, for each  $\theta \in \Theta$ ,
$$
\Pr_ {(\mu , \rho) \sim \mathrm {R T}} \left[ \Pr_ {y \sim \tau} [ y \neq \pi_ {\theta} (y) ] > 1 6 \gamma , \Pr_ {y \sim \tau} [ \pi_ {\theta} (y) \notin P ] \leq \gamma \right] \leq \frac {\delta}{| \Theta |}. \tag {5}
$$
We will show that, more generally, for any permutation  $\pi : \mathcal{V} \hookrightarrow \mathcal{V}$ . Equation (5) can be restated in terms of ambiguous texts  $A := \{y \in \mathcal{V} \mid \pi(y) \neq y\}$  and implausible ambiguities  $B := \{y \in \mathcal{V} \mid \pi(y) \notin P\}$ :
$$
\operatorname * {P r} _ {(\mu , \rho) \sim \mathrm {R T}} [ \tau (A) > 1 6 \gamma ,   \tau (B) \leq \gamma ] <   \frac {\delta}{| \Theta |}.
$$
Using Lemma B.4, partition  $A$  into four disjoint sets  $A = A_{1} \cup A_{2} \cup A_{3} \cup A_{4}$  such that for each  $i \in [4]$ , the following two conditions hold:
$$
\pi \left(A _ {i}\right) \cap A _ {i} = \emptyset \tag {6}
$$
$$
\forall z \in \mathcal {W} ^ {n - 1} \quad | \{y \in A _ {i} \mid y \text {b e g i n s w i t h} z \} | \leq \frac {| \mathcal {W} |}{2}. \tag {7}
$$
By the union bound, it suffices to show that,
$$
\forall i \in [ 4 ] \quad \Pr_ {(\mu , \rho) \sim \mathrm {R T}} \left[ \tau \left(A _ {i}\right) > 4 \gamma ,   \tau (B) \leq \gamma \right] <   \frac {\delta}{4 | \Theta |},
$$
because  $\tau(A) = \sum_{i=1}^{4} \tau(A_i)$ , so if  $\tau(A) > 16\gamma$  then it must follow that  $\tau(A_i) > 16\gamma/4$  for some  $i$ . It suffices to show that this holds conditioned on any value of  $A_i \cap T$ :
$$
\forall V \subseteq A _ {i} \Pr_ {(\mu , \rho) \sim \mathrm {R T}} [ \tau (A _ {i}) > 4 \gamma , \tau (B) \leq \gamma \mid A _ {i} \cap T = V ] \leq \frac {\delta}{4 | \Theta |}. \tag {8}
$$
Thus, fix any  $V \subseteq A_i$ . We proceed by analyzing two cases, based on  $|V|$  versus  $|T| = a^n$ .<sup>13</sup>
Case 1:  $|V| \leq 4\gamma |T|$ . Then Equation (8) holds with probability 0, because conditioning on  $A_{i} \cap T = V$ , we have
$$
\tau (A _ {i}) := \frac {| A _ {i} \cap T |}{| T |} = \frac {| V |}{| T |} \leq 4 \gamma .
$$
Case 2:  $|V| > 4\gamma |T|$ . For each  $v \in V$  we define a Boolean random variable  $Z_{v} = \mathbf{1}_{v \in B}$ , i.e.,
$$
Z _ {v} = \left\{ \begin{array}{l l} 1 & v \in B \\ 0 & v \notin B. \end{array} \right.
$$
We claim that the  $Z_{v}$ 's satisfy the condition of Lemma B.3, which follows inductively from the fact any subset of  $Z_{v}$ 's being 0 only makes it less likely for another  $Z_{v}$  to be 0, because the way that the  $Z_{v}$ 's are chosen is such that there are exactly  $b$  many 1's among the  $Z_{v}$ 's (and other symmetric random variables which we have not named).
Therefore, if we define  $Z \coloneqq \sum_{v \in V} Z_v$  and  $\zeta \coloneqq \mathbb{E}[Z \mid A_i \cap T = V]$ , Lemma B.3 gives,
$$
\Pr \left[ Z \leq \frac {\zeta}{2} \mid A _ {i} \cap T = V \right] \leq e ^ {- \zeta / 8}. \tag {9}
$$
We conclude by bounding both sides of Equation (9) to obtain Equation (8).
- For the right-hand side of eq. (9), we claim that  $\operatorname*{Pr}\left[Z_v = 1\mid A_i\cap T = V\right]\geq 1 / 2$  for each  $v\in V$ : Fix  $v$  and let
$$
Q = \{y \in \mathcal {Y} \mid y \text {a n d} \pi (v) \text {a g r e e o n t h e f i r s t} n - 1 \text {w o r d s} \}.
$$
So  $|Q| = |\mathcal{W}|$  and  $|Q \cap P| = b$ . Equation (6) implies that  $\pi(v) \notin A_i$ , and eq. (7) implies that  $|Q \setminus A_i| \geq |\mathcal{W}| / 2$ . Thus, there are  $\geq |\mathcal{W}| / 2$  elements in  $Q \setminus A_i$  that, by symmetry, are as likely to be in  $Q \cap P$  as  $\pi(v)$  is, and  $Q \cap P$  contains exactly  $b \leq |\mathcal{W}| / 4$  elements, thus the conditional probability that  $\pi(v)$  is in  $Q \cap P$  (equivalently,  $\pi(v) \in P$ ) is at most:
$$
\frac {| \mathcal {W} | / 4}{| \mathcal {W} | / 2} \leq \frac {1}{2}.
$$
Since  $\pi(v) \notin P$  is equivalent to  $Z_v = 1$ , this means that  $\operatorname*{Pr}\left[Z_v = 1 \mid A_i \cap T = V\right] \geq 1/2$ , and hence:
$$
\zeta = | V | \cdot \mathbb {E} \left[ Z _ {v} \mid A _ {i} \cap T = V \right] = | V | \cdot \Pr \left[ Z _ {v} = 1 \mid A _ {i} \cap T = V \right] \geq \frac {| V |}{2}.
$$
<sup>13</sup>Note that while  $T$  is randomly sampled,  $|T| = a^n$  always, and therefore the case-analysis is valid.
Because we are in the case that  $|V| > 4\gamma |T|$ , we have
$$
\zeta \geq \frac {| V |}{2} > \frac {1}{2} \cdot 4 \gamma \cdot | T | = 2 \gamma \cdot a ^ {n}. \tag {10}
$$
Therefore the right-hand side of eq. (9) satisfies
$$
e ^ {- \zeta / 8} \leq e ^ {- \gamma a ^ {n} / 4} \geq \frac {\delta}{4 | \Theta |},
$$
where the rightmost inequality holds because by our assumption that  $\gamma \geq 4a^{-n}\ln (4|\Theta | / \delta)$
- For the left-hand side of eq. (9), note that
$$
Z := \sum_ {v \in V} Z _ {v} := \sum_ {v \in V} \mathbf {1} _ {v \in B} = | V \cap B |.
$$
and therefore, using eq. (10), the left-hand side of eq. (9) satisfies
$$
\Pr \left[ Z <   \frac {\zeta}{2} \mid A _ {i} \cap T = V \right] \geq \Pr \left[ \frac {| V \cap B |}{| T |} \leq \gamma \mid A _ {i} \cap T = V \right].
$$
We claim that  $V \cap B \subseteq B \cap T$ : Due to the conditional event, we have  $V \cap B = A_i \cap T \cap B \subseteq A \cap T \cap B$ . However, we argue that  $A \cap T \cap B \subseteq T \cap B$ , i.e., that  $T \cap B \subseteq A$ . Indeed, by definition, if  $y \in T \cap B$  then  $y \in T$  but  $\pi(y) \notin P$ , and since  $P \supseteq T$ , this implies that  $\pi(y) \neq y$ ; that is, that  $y \in A$ , as needed.
We have just shown that  $V \cap B \subseteq B \cap T$ , and therefore
$$
\begin{array}{l} \Pr \left[ \frac {| V \cap B |}{| T |} \leq \gamma \mid A _ {i} \cap T = V \right] \geq \Pr \left[ \frac {| B \cap T |}{| T |} \leq \gamma \mid A _ {i} \cap T = V \right] \\ = \Pr [ \tau (B) \leq \gamma \mid A _ {i} \cap T = V ]. \\ \end{array}
$$
This shows that the left-hand side of eq. (9) upper bounds that of eq. (8), and concludes the proof.
Lastly, we prove Theorem B.1 using Lemma B.2
Proof. of Theorem B.1. Let  $\gamma$  be,
$$
\gamma := \max  \left(\frac {1}{m}, \frac {4}{a ^ {n}}\right) \ln \frac {6 | \Theta |}{\delta}.
$$
Lemma B.2 below shows that with probability  $\geq 1 - 2\delta /3$  over  $\mu ,\rho$  , we have that  $\varepsilon_{\gamma}\leq 16\gamma$  Theorem A.2 implies that with probability  $\geq 1 - \delta /3$  over  $x_{1},\ldots ,x_{m}$  , we have  $\mathrm{err}(\hat{\theta})\leq \varepsilon_{\gamma}$  . Thus, by the union bound, with probability  $\geq 1 - \delta$  we have
$$
\operatorname {e r r} (\hat {\theta}) \leq \varepsilon_ {\gamma} \leq 1 6 \gamma = 1 6 \max  \left(\frac {1}{m}, \frac {4}{a ^ {n}}\right) \ln \frac {6 | \Theta |}{\delta}.
$$
C Proofs for the Common Nonsense Model
C.1 Upper bound in the common nonsense model
C.1.1 Proof overview
To convey intuition, we think of the case of say  $\delta = 99\%$ , and we omit constants in the following discussion. Recall that Theorem A.2 asserts that with high probability, the learned translator  $\hat{\theta}$  has error at most  $\varepsilon_{\gamma}$  for  $\gamma = \log |\Theta| / \min(m, |T|) \geq \log |\Theta| / m$ . As such, the main technical challenge
in this proof is to show that, with high probability over the generated source language  $\mu$  and the prior  $\rho$ , it holds that
$$
\varepsilon_ {\gamma} \lesssim \frac {\gamma}{\alpha}.
$$
By definition of  $\varepsilon_{\gamma}$ , we ought to show that w.h.p over  $\mu$  and  $\rho$ , any  $\theta \in \Theta$  with large semantic error must have many translations  $f_{\theta}(x)$  deemed implausible by  $\rho$ . Slightly more formally: Since any  $y \in \mathcal{V}$  is implausible ( $\rho(y) = 0$ ) only if  $y \notin S$ , a union bound over  $\theta \in \Theta$  means that it suffices to show that
$$
\operatorname * {P r} _ {S, \mu , \rho} \left[ \operatorname * {P r} _ {x \sim \mu} [ f _ {\theta} (x) \neq f _ {\star} (x) ] \lesssim \frac {\gamma}{\alpha}, \operatorname * {P r} _ {x \sim \mu} [ f _ {\theta} (x) \notin S ] \gtrsim \gamma \right] \lesssim \exp (- \gamma | T |)) \leq \frac {1}{| \Theta |},
$$
where the right inequality is by choice of  $\gamma \coloneqq \log |\Theta | / \min (m,|T|)$ . The above inequality "looks like" it could be proven by a Chernoff bound, but a closer look reveals a subtle flaw with this argument.