SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To use a Chernoff bound, we'd first want to fix (i.e., condition on) each  $\mathrm{supp}(\mu)\coloneqq f_{\star}^{-1}(S\cap T)$  and then use Chernoff over the conditional random variables  $\mathbf{1}_{f_{\theta}(x)\notin S}$  for each  $x\in \operatorname {supp}(\mu)$  such that  $f_{\theta}(x)\neq f_{\star}(x)$ . Unfortunately, these conditional random variables are not independent. To see this, consider the case that  $f_{\theta}(x) = f_{\star}(x^{\prime})$  for two different  $x\neq x^{\prime}$ . Then, since we are considering  $x^{\prime}\in \operatorname {supp}(\mu)$ , we have  $f_{\theta}(x) = f_{\star}(x^{\prime})\in S$  with probability 1.
To avoid this dependency, we prove a combinatorial lemma showing that it is possible to partition the set
$$
A := \{x \in X \mid f _ {\theta} (x) \neq f _ {\star} (x) \}
$$
into three parts  $A = A_{1} \cup A_{2} \cup A_{3}$  such that  $f_{\theta}(A_i) \cap f_{\star}(A_i) = \emptyset$  for each  $i \in [3]$ . This resolves the dependency issue demonstrated above. We then proceed by applying a Chernoff bound separately for each  $A_{i}$ , which suffices since a union bound (over  $i \in [3]$ ) loses only a constant factor in the upper-bound.
C.1.2 The full proof
The proof of Theorem 3.4 follows from the following main lemma. We first prove this lemma, and then show how the theorem follows from it.
Lemma C.1. Let  $\alpha, \delta \in (0, 1)$  and  $T \subseteq P \subseteq \mathcal{V}$ . Then, for any  $\gamma \geq \frac{8}{(1 - \alpha) \cdot |T|} \ln \frac{4|\Theta|}{\delta}$ :
$$
\operatorname * {P r} _ {(\mu , \rho) \sim \mathcal {D} _ {\alpha} ^ {P, T}} \left[ \varepsilon_ {\gamma} \leq \frac {6 \gamma}{\alpha} \right] \geq 1 - \delta .
$$
The proof of Lemma C.1 relies on a simple combinatorial proposition. This proposition is a special case of Lemma B.4, but since it is much simpler we give a self-contained proof.
Proposition C.2. For any finite set  $\mathcal{V}$  and any  $\pi : \mathcal{V} \hookrightarrow \mathcal{V}$ , it is possible to partition  $\{y \in \mathcal{V} | \pi(y) \neq y\} = A_1 \cup A_2 \cup A_2$  into three sets such that,  $\pi(A_i) \cap A_i = \emptyset$  for  $i = 1, 2, 3$ .
Proof. Let  $S \coloneqq \{y \in \mathcal{Y} \mid \pi(y) \neq y\}$ . We proceed iteratively, dividing each (non-trivial) cycle of  $\pi$  separately into the three  $A_i$ 's: Fix a cycle  $\{s_1, s_2, \ldots, s_n\}$  such that  $\pi(s_i) = s_{i+1}$  and  $\pi(s_n) = s_1$ . If  $n$  is even we can just partition it into two sets: put the  $s_i$  for even  $i$ 's into  $A_1$  and odd  $i$ 's into  $A_2$ . If  $n$  is odd, we can do the same except put the last element  $s_n$  into  $A_3$ .
We can now prove Lemma C.1.
Proof. of Lemma C.1. Note that the lemma holds trivially for any  $\gamma > 1/6$  because we always have  $\varepsilon_{\gamma} \leq 1$ . Assume that  $\gamma \leq 1/6$ . Let  $a := \frac{6}{\alpha}\gamma$ . The probabilities in this proof are over the choice of  $S \subseteq \mathcal{V}$ . It suffices to show that,
$$
\Pr_ {S} \left[ \exists \theta \in \Theta \Pr_ {\tau} \left[ \pi_ {\theta} (y) \neq y \right] > a \wedge \Pr_ {\tau} \left[ \pi_ {\theta} (y) \notin S \right] \leq \gamma \right] \leq \delta , \tag {11}
$$
because  $\rho (\pi_{\theta}(y)) = 0$  whenever  $\pi_{\theta}(y)\notin S$ , thus  $\mathcal{A}_{\gamma} = \{\theta \in \Theta \mid \operatorname*{Pr}_{\tau}[\pi_{\theta}(y)\notin S]\leq \gamma \}$ . The set  $S$  determines the perturbed sets  $\tilde{T}\coloneqq T\cap S$  and  $\tilde{P}\coloneqq P\cap S$  and the distributions  $\tau = \mathcal{U}(\tilde{T})$  and  $\rho = \mathcal{U}(\tilde{P})$ .
Define  $\beta \coloneqq 1 - \alpha$ . Since  $\mathbb{E}\left[|\tilde{T}|\right] = \beta |T|$ , a multiplicative Chernoff bound<sup>14</sup> gives
$$
\operatorname * {P r} _ {S} \left[ | \tilde {T} | \leq \frac {\beta}{2} | T | \right] \leq \exp \left(- \frac {\beta}{8} | T |\right) \leq \exp \left(- \frac {\beta}{8} \cdot \frac {8}{\beta \gamma} \ln \frac {4 | \Theta |}{\delta}\right) <   \frac {\delta}{4 | \Theta |} \leq \frac {\delta}{4}.
$$
Thus, to show eq. (11), it suffices to show that
$$
\Pr \left[ | \tilde {T} | \geq \frac {\beta}{2} | T | \wedge \exists \theta \in \Theta \Pr_ {\tau} [ \pi_ {\theta} (y) \neq y ] > a \wedge \Pr_ {\tau} [ \pi_ {\theta} (y) \notin S ] \leq \gamma \right] \leq \frac {3 \delta}{4}. \tag {12}
$$
By the union bound, to show eq. (12) it thus suffices to show that for any  $\pi : \mathcal{V} \hookrightarrow \mathcal{V}$ ,
$$
\Pr \left[ | \tilde {T} | \geq \frac {\beta}{2} | T | \wedge \Pr_ {\tau} [ \pi (y) \neq y ] > a \wedge \Pr_ {\tau} [ \pi (y) \notin S ] \leq \gamma \right] \leq \frac {3 \delta}{4 | \Theta |}. \tag {13}
$$
Fix any  $\pi : \mathcal{V} \hookrightarrow \mathcal{V}$ . By Lemma C.2, we can partition  $\{y \in \mathcal{V} \mid \pi(y) \neq y\} = A_1 \cup A_2 \cup A_3$  such that  $\pi(A_i) \cap A_i = \emptyset$ , and hence  $\operatorname{Pr}_{\tau}[\pi(y) \neq y] = \sum_{i} \tau(A_i)$ . So if  $\operatorname{Pr}_{\tau}[\pi(y) \neq y] > a$ , then  $\tau(A_i) > a/3$  for some  $i$ . Therefore, it suffices to show that
$$
\operatorname * {P r} \left[ | \tilde {T} | \geq \frac {\beta}{2} | T | \land \exists i \in [ 3 ] \tau (A _ {i}) > \frac {a}{3} \land \operatorname * {P r} _ {\tau} [ \pi (y) \notin S ] \leq \gamma \right] \leq \frac {3 \delta}{4 | \Theta |}.
$$
With a union bound over  $i \in [3]$ , it suffices to show that for each  $i \in [3]$
$$
\Pr \left[ | \tilde {T} | \geq \frac {\beta}{2} | T | \wedge \tau \left(A _ {i}\right) > \frac {a}{3} \wedge \Pr_ {\tau} [ \pi (y) \notin S ] \leq \gamma \right] \leq \frac {\delta}{4 | \Theta |}. \tag {14}
$$
Thus, now in addition to fixing  $\pi$ , we fix  $i \leq 3$ , thus fixing  $A_i$ . To continue, imagine we are picking  $S$  by first selecting  $A_i \cap S$ , and subsequently selecting  $S \setminus A_i$ . We will show that Equation (14) holds when conditioning on each possible value for the first selection, that is, each possible  $A_i \cap S$ . Formally, we fix  $V \subseteq A_i$  and condition on  $A_i \cap S = V$ , claiming that
$$
\Pr \left[ | \tilde {T} | \geq \frac {\beta}{2} | T | \wedge \tau \left(A _ {i}\right) > \frac {a}{3} \wedge \Pr_ {\tau} [ \pi (y) \notin S ] \leq \gamma \mid V = A _ {i} \cap S \right] \leq \frac {\delta}{4 | \Theta |}. \tag {15}
$$
First, observe that
$$
\tau (A _ {i}) = \frac {| A _ {i} \cap \tilde {T} |}{| \tilde {T} |} = \frac {| A _ {i} \cap S |}{| \tilde {T} |} = \frac {| V |}{| \tilde {T} |},
$$
therefore if  $|V| < \beta a |T| / 6$  then Equation (15) holds with probability 0 (due to the first two events in the conjunction). Thus, we can assume that  $|V| \geq \beta a |T| / 6$ .
Note that  $\tau (V) = \tau (A_i)$ , and that  $\tau (A_{i})\geq a / 3$  implies
$$
\operatorname * {P r} _ {y \sim \tau} [ \pi (y) \notin S ] \geq \operatorname * {P r} _ {y \in V} [ \pi (y) \notin S ] \cdot \tau (V) \geq \operatorname * {P r} _ {y \in V} [ \pi (y) \notin S ] \cdot \frac {a}{3},
$$
therefore the left-hand side of Equation (15) is upper-bounded by
$$
\Pr \left[ \Pr_ {y \in V} [ \pi (y) \notin S ] \leq \frac {3 \cdot \gamma}{a} \mid V = A _ {i} \cap S \right] = \Pr \left[ \Pr_ {y \in V} [ \pi (y) \notin S ] \leq \frac {a}{2} \mid V = A _ {i} \cap S \right]. \tag {16}
$$
We conclude the proof by upper-bounding Equation (16) with a Chernoff bound. Consider the random variables  $Z_{y} = \mathbf{1}_{\pi(y) \notin S}$  for  $y \in V$ . These random variables are independent by definition of  $\alpha$ -common-nonsense. Furthermore, due to the fact that  $\pi(A_{i}) \cap A_{i} = \emptyset$ , they remain independent even when conditioning on the event  $A_{i} \cap S = V$ . By linearity of expectation,
$$
\mathbb {E} \left[ \sum_ {y \in V} Z _ {y} \middle |   V = A _ {i} \cap S \right] = \alpha | V |.
$$
<sup>14</sup>Specifically, that the probability that a sum of binary random variables is less than half its mean  $\beta |T|$  is at most  $\exp(-\beta |T|/8)$ .
Using the same Chernoff bound as above, we have
$$
\Pr \left[ \frac {\sum_ {y \in V} Z _ {y}}{| V |} \leq \frac {\alpha}{2} \mid V = A _ {i} \cap S \right] \leq \exp (- \alpha | V | / 8)
$$
Noting that  $\sum Z_y / |V| = \operatorname{Pr}_{y\in V}[\pi (y)\notin S]$ , we conclude that that Equation (16) is upper-bounded by
$$
\exp \left(- \frac {\alpha | V |}{8}\right) \leq \exp \left(- \frac {\alpha \beta a | T |}{4 8}\right) \leq \frac {\delta}{4 | \Theta |}.
$$
This proves the inequality in Equation (15), thereby concluding the proof.
Finally, with the main technical lemma in hand, we prove Theorem 3.4.
Proof. of Theorem 3.4. The proof is very similar to that in Appendix B. First, the realizability Condition 2.1,  $\operatorname{Pr}_{x\sim \mu}[\rho (f_{\star}(x)) = 0] = 0$ , follows immediately from the fact that  $\mu ,\rho$  are uniform distributions with  $f_{\star}(\mathrm{supp}(\mu))\subseteq \mathrm{supp}(\rho)$  which follows from the fact that  $T\subseteq P$  and the definitions of  $\mu ,\rho$
Let  $\beta \coloneqq 1 - \alpha$  and define  $\gamma$  by,
$$
\gamma := \max \left(\frac {1}{m}, \frac {8}{\beta | T |}\right) \ln \frac {6 | \Theta |}{\delta}.
$$
Lemma C.1 below shows that with probability  $\geq 1 - 2\delta /3$  over  $\mu ,\rho$  we have that  $\varepsilon_{\gamma}\leq 6\gamma /\alpha$  Theorem A.2 implies that with probability  $\geq 1 - \delta /3$  over  $x_{1},\ldots ,x_{m}$  , we have  $\mathrm{err}(\hat{\theta})\leq \varepsilon_{\gamma}$  . Thus, by the union bound, with probability  $\geq 1 - \delta$  we have
$$
\mathrm {e r r} (\hat {\theta}) \leq \varepsilon_ {\gamma} \leq \frac {6 \gamma}{\alpha} = \frac {6}{\alpha} \max \left(\frac {1}{m}, \frac {8}{\beta | T |}\right) \ln \frac {6 | \Theta |}{\delta}.
$$
C.2 Lower bound in the common nonsense model
The proof of the lower bound works by creating  $\log n$  candidate "plausible ambiguities" and arguing that a constant fraction of survive the random removal of elements.
Proof. of Theorem 3.5. The constants  $c_{1}, c_{2}$  will also be determined through this proof to be large enough to satisfy multiple conditions defined below. No effort has been made to minimize the constants in this proof.
Let  $n\coloneqq |\Theta |$
We will lay out two  $a \times b$  grids  $X \subseteq \mathcal{X}$  and  $Y \subseteq \mathcal{Y}$ , for:
$$
a := \lfloor \log n \rfloor , b := \left\lfloor \frac {1}{\alpha} \max  \left(1, \frac {| T |}{1 0 ^ {5} m}\right) \right\rfloor .
$$
For integer  $t$ , denote  $[t] := \{1, 2, \dots, t\}$ . For  $i \in [a], j \in [b]$ , choose distinct elements  $x_{ij} \in \mathcal{X}$  and  $y_{ij} \in T$ . To ensure this is even possible, we must make sure  $ab \leq |T|$ , which holds because we assumed  $\log n \leq \alpha \min(m, |T|)$  thus,
$$
a b \leq \alpha \min  (m, | T |) \frac {1}{\alpha} \max  \left(1, \frac {| T |}{m}\right) = \min  (m, | T |) \cdot \max  \left(\frac {1}{| T |}, \frac {1}{m}\right) | T | = | T |.
$$
Let  $X \coloneqq \{x_{ij} \mid i \in [a], j \in [b]\}$  and  $Y \coloneqq \{y_{ij} \mid i \in [a], j \in [b]\}$ . Let  $h: \mathcal{X} \setminus X \hookrightarrow \mathcal{Y} \setminus Y$  be a fixed 1-1 mapping, say the lexicographically smallest. The parametrized translator family is defined by,
$$
\Theta^ {\prime} = \Theta_ {m, n, Y} := \{- 1, 1 \} ^ {a}, f _ {\theta} (x _ {i j}) := \left\{ \begin{array}{l l} y _ {i j}, & \theta_ {i} = 1 \\ y _ {i (j + 1 \bmod b)}, & \theta_ {i} = - 1 \end{array} \right., \forall x \notin X   f _ {\theta} (x) = h (x).
$$
Clearly  $|\Theta'| = 2^a \leq n = |\Theta|$  as needed. Let  $\mathbf{x} = (x_1, x_2, \ldots, x_m)$ . It suffices to show:
$$
\operatorname * {P r} _ {S, \mathbf {x}} \left[ \operatorname {e r r} (\hat {\theta}) \geq \frac {\log n}{c _ {3} \alpha \min (1 0 ^ {5} m , | T |)} \right] \geq 0. 9 9,
$$
for some constant  $c_3$  sufficiently large, because we can set  $c_2 = 10^5 c_3$ . The above equation is equivalent to the following two cases based on  $m$ :
$$
\text {C a s e} 1. 1 0 ^ {5} m > | T | \Rightarrow \Pr_ {S, \mathbf {x}} \left[ \operatorname {e r r} (\hat {\theta}) \geq \frac {\log n}{c _ {3} \alpha | T |} \right] \geq 0. 9 9 \tag {17}
$$
$$
\text {C a s e} 2. 1 0 ^ {5} m \leq | T | \Rightarrow \Pr_ {S, \mathbf {x}} \left[ \operatorname {e r r} (\hat {\theta}) \geq \frac {\log n}{c _ {3} \alpha 1 0 ^ {5} m} \right] \geq 0. 9 9 \tag {18}
$$
In both cases, it will be convenient to notice that, for any  $z \geq 2$ ,  $\lfloor z \rfloor \geq z - 1 \geq \frac{z}{2}$ . Since  $b \geq 2$  (because  $\alpha \leq 1/2$ ), we therefore have
$$
\frac {1}{2 \alpha} \max  \left(1, \frac {| T |}{1 0 ^ {5} m}\right) \leq b \leq \frac {1}{\alpha} \max  \left(1, \frac {| T |}{1 0 ^ {5} m}\right) \tag {19}
$$
Case 1:  $10^{5}m > T$ . In this case  $\frac{1}{2\alpha} \leq b \leq \frac{1}{\alpha}$  by Equation (19). We will show Equation (17). Now, consider the "full rows":
$$
C (S) := \{i \in [ a ] \mid \forall j \in [ b ] y _ {i j} \in S \}.
$$
These rows will be useful to consider because nothing has been removed from the entire row, no information about  $\theta_{i}$  has been revealed and (on average) one cannot achieve error  $< 1 / 2$  on these examples, because one cannot distinguish between the two permutations on this row.
Note that the membership of different  $i, i' \in C(S)$  is independent since  $S$  is chosen independently, and by definition of  $C$  and  $S$ :
$$
\mathbb {E} [ | C (S) | ] = (1 - \alpha) ^ {b} a \geq (1 - \alpha) ^ {1 / \alpha} a \geq \frac {a}{4},
$$
since  $(1 - \alpha)^{1 / \alpha}$  is decreasing in  $\alpha$  and  $\alpha \leq 1 / 2$ . Thus, by multiplicative Chernoff bounds (specifically,  $\operatorname*{Pr}[Z\leq \mathbb{E}[Z] / 2]\leq e^{-\mathbb{E}[Z] / 8}$ ),
$$
\Pr_ {S} \left[ | C (S) | \leq \frac {a}{8} \right] \leq e ^ {- a / 3 2} \leq e ^ {- \lfloor c _ {1} \rfloor / 3 2} \leq 0. 0 0 1, \tag {20}
$$
for sufficiently large  $c_1$ . Thus,  $\operatorname{Pr}_S[|C(S)| > a/8] \geq 0.999$ . Let  $C'(S, \mathbf{x}) \subseteq C(S)$  be those  $i$  which  $\hat{\theta}_i \neq \star_i$ ,
$$
C ^ {\prime} (S, \mathbf {x}) := \left\{i \in C (S) \mid \hat {\theta} _ {i} \neq \star_ {i} \right\}.
$$
Clearly, for any algorithm and any  $C(S)$ ,  $\mathbb{E}_{\mathbf{x}}[|C'(S,\mathbf{x})|\mid S] = |C(S)| / 2$  because no information whatsoever has been revealed about  $\theta_{i}$  for any  $i\in C$ . Thus, by the same Chernoff bound, we have:
$$
\Pr_ {S, \mathbf {x}} \left[ | C ^ {\prime} (S, \mathbf {x}) | \leq \frac {1}{4} | C (S) | \mid | C (S) | > \frac {a}{8} \right] \leq e ^ {- \frac {a}{1 6} \cdot \frac {1}{8}} \leq 0. 0 0 1,
$$
for sufficiently large  $c_{1}$ , because  $a \geq c_{1}$ . By the union bound over this and Equation (20),
$$
\Pr_ {S, \mathbf {x}} \left[ | C ^ {\prime} (S, \mathbf {x}) | \geq \frac {a}{3 2} \right] \geq 0. 9 9 8.
$$
Since each row  $i \in C'$  incurs  $b$  errors on examples  $x$ , one for each  $j$  because  $f_{\star}(x_{ij}) \in S$ :
$$
\operatorname {e r r} (\hat {\theta}) \geq \frac {b \cdot | C ^ {\prime} (S , \mathbf {x}) |}{| T |}.
$$
$$
\Pr_ {S, \mathbf {x}} \left[ \operatorname {e r r} (\hat {\theta}) \geq \frac {b a}{3 2 \alpha | T |} \right] \geq 0. 9 9 8.
$$
Now,  $a \geq \frac{1}{2} \log n$  for sufficiently large  $c_{1}$  and as mentioned  $b \geq \frac{1}{2\alpha}$ . Thus,
$$
\Pr_ {S, \mathbf {x}} \left[ \operatorname {e r r} (\hat {\theta}) \geq \frac {\log n}{1 2 8 \alpha | T |} \right] \geq 0. 9 9 8.
$$
This establishes Equation (17) as long as  $c_{3} \geq 128$ .
It remains to prove Equation (18).
Case 2:  $10^{5}m \leq T$ . In this case  $\frac{|T|}{2\alpha 10^{5}m} \leq b \leq \frac{|T|}{\alpha 10^{5}m}$  by Equation (19). Next, consider the set of rows with at least  $1/2$  of the elements in  $S$ :
$$
D (S) := \left\{i \in [ a ] \left| \left| \left\{j \in [ b ] \mid y _ {i j} \in S \right\} \right| \geq \frac {b}{2} \right. \right\}.
$$
Intuitively, any row  $i \in D(S)$  is "dangerous" in the sense that if  $\hat{\theta}_i \neq \star_i$ , then it causes errors on  $b/2$  different  $x$ 's in the support of  $\mu$ , i.e., for which  $f_{\star}(x_{ij}) \in S$ . Observe that  $\mathbb{E}[|D(S)|] \geq a/2$  since each size  $s = |\{j \in [b] \mid y_{ij} \in S\}| \geq b/2$  is at least as likely as the size  $b - s$ , since  $\alpha \leq 1/2$ . And also, membership of  $i, i' \in D(S)$  since  $S$  is independent. Thus, by the same Chernoff bound as above, for sufficiently large  $c_1$ ,
$$
\Pr_ {S} \left[ | D (S) | \leq \frac {a}{4} \right] \leq e ^ {- a / 1 6} \leq e ^ {- \lfloor c _ {1} \rfloor / 1 6} \leq 0. 0 0 1. \tag {21}
$$
Let  $-\theta \coloneqq (-\theta_{1}, - \theta_{2},\dots , - \theta_{a})$  . This makes it convenient to define the giveaways  $G(S)\subseteq X$  to be,
$$
G (S) := \left\{x _ {i j} \mid i \in [ a ], j \in [ b ], f _ {\star} \left(x _ {i j}\right) \in S, f _ {- \star} \left(x _ {i j}\right) \notin S \right\}.
$$
These are the points  $x_{ij}$  which we might observe  $f_{\star}(x_{ij})$  which would imply that  $\theta_i = \star_i$  (and not its negative). Also let,
$$
\hat {G} (S, \mathbf {x}) = \left\{x _ {1}, x _ {2}, \dots , x _ {m} \right\} \cap G (S).
$$
(Note that if for a given row  $i$ , we do not have any  $x_{ij} \in \hat{G}(S, \mathbf{x})$ , then we have no information about  $\theta_i$ . As a preview to what is to come, we now argue that with high probability  $|\hat{G}(S, \mathbf{x})| < a / 8$  which will mean that, if  $|D(S)| > a / 4$ , then we have no information about  $\theta_i$  for at least  $a / 8$  of the rows  $i \in D(S)$ .)
For any fixed  $i,j$ , observe that  $\operatorname*{Pr}[x_{ij}\in G(S)] = \alpha (1 - \alpha)$  so  $\mathbb{E}[|G(S)|] = \alpha (1 - \alpha)ab\leq \alpha ab$ . By the Chernoff bound that  $\operatorname*{Pr}[Z\geq 2\mathbb{E}[Z]]\leq e^{-\mathbb{E}[Z] / 3}$ ,
$$
\operatorname * {P r} _ {S} [ | G (S) | \geq 2 \alpha a b ] \leq e ^ {- \alpha a b / 3} \leq e ^ {- a / 3} \leq 0. 0 0 1,
$$
for sufficiently large  $c_1$ . (We have used the fact that the above probability is smaller than if  $E[|G(S)|]$  were actually  $\alpha ab$ .)
Also,  $\mathbb{E}_S[|T\cap S|]\geq |T| / 2$  since  $\alpha \leq 1 / 2$ . So, by the Chernoff bound  $\operatorname *{Pr}[Z\leq \mathbb{E}[Z] / 2]\leq e^{-\mathbb{E}[Z] / 8}$
$$
\operatorname * {P r} _ {S} \left[ | T \cap S | \leq \frac {| T |}{4} \right] \leq e ^ {- | T | / 1 6} \leq 0. 0 0 1,
$$
for sufficiently large  $c_{1}$  since  $|T|\geq \log n\geq c_{1}$
Thus, by the union bound:
$$
\Pr_ {S} \left[ | T \cap S | \leq \frac {| T |}{4} \vee | G (S) | \geq 2 \alpha a b \right] \leq 0. 0 0 2. \tag {22}
$$
$$
\mathbb {E} _ {\mathbf {x}} \left[ | \hat {G} (S, \mathbf {x}) | \mid S \right] \leq m \frac {| G (S) |}{| T \cap S |}.
$$
Thus, using Markov's inequality in the second line below,
$$
\mathbb {E} _ {\mathbf {x}, S} \left[ | \hat {G} (S, \mathbf {x}) | \mid | T \cap S | > \frac {| T |}{4}, | G (S) | \leq 2 \alpha a b \right] \leq m \frac {2 \alpha a b}{| T | / 4} = \frac {8 \alpha a b m}{| T |},
$$
$$
\Pr_ {\mathbf {x}, S} \left[ | \hat {G} (S, \mathbf {x}) | > \frac {8 0 0 0 \alpha a b m}{| T |} \right. \left| | T \cap S | > \frac {| T |}{4}, | G (S) | \leq 2 \alpha a b \right] \leq 0. 0 0 1.
$$
By the union bound over the above and Equation (22), since  $\operatorname*{Pr}[E] \leq \operatorname*{Pr}[E|F] + \operatorname*{Pr}[\neg F]$
$$
\operatorname * {P r} _ {\mathbf {x}, S} \left[ | \hat {G} (S, \mathbf {x}) | > \frac {8 0 0 0 \alpha a b m}{| T |} \right] \leq 0. 0 0 1 + 0. 0 0 2.
$$
$$
b \leq \frac {| T |}{\alpha 1 0 ^ {5} m}
$$
$$
\frac {8 0 0 0 \alpha a b m}{| T |} \leq 0. 0 8 \cdot a \leq \frac {a}{8}
$$
$$
\Pr_ {\mathbf {x}, S} \left[ | \hat {G} (S, \mathbf {x}) | > \frac {a}{8} \right] \leq 0. 0 0 3
$$
By the union bound with Equation (21),
$$
\Pr_ {\mathbf {x}, S} \left[ | D (S) | \leq \frac {a}{4} \vee | \hat {G} (S, \mathbf {x}) | > \frac {a}{8} \right] \leq 0. 0 0 4
$$
$$
F (S, \mathbf {x}) := \left\{i \in D (S) \mid \forall j \in [ b ] x _ {i j} \notin \hat {G} (S) (S, \mathbf {x}) \right\} \subseteq D.
$$
Clearly  $|F(S, \mathbf{x})| \geq |D(S)| - |\hat{G}(S, \mathbf{x})|$  because each  $x \in \hat{G}(S, \mathbf{x})$  can remove at most one row from  $D(S)$ . Thus,
$$
\Pr_ {\mathbf {x}, S} \left[ | F (S, \mathbf {x}) | \leq \frac {a}{8} \right] \leq 0. 0 0 4. \tag {23}
$$
$F(S, \mathbf{x})$  will function exactly like  $C$  in the analysis above of Equation (17). We repeat this analysis for completeness, replacing  $C$  by  $F$ . Let  $F'(S, \mathbf{x}) \subseteq F(S, \mathbf{x})$  be those  $i$  which  $\hat{\theta}_i \neq \star_i$ ,
$$
F ^ {\prime} (S, \mathbf {x}) := \left\{i \in F (S, \mathbf {x}) \mid \hat {\theta} _ {i} \neq \star_ {i} \right\}.
$$
For any algorithm and any  $F(S, \mathbf{x})$ ,  $\mathbb{E}_{\mathbf{x}}[|F'(S, \mathbf{x})| \mid S] = |F(S, \mathbf{x})| / 2$  because no information whatsoever has been revealed about  $\theta_{i}$  for any  $i \in F$ . Thus, by the same Chernoff bound, we have:
$$
\Pr_ {S, \mathbf {x}} \left[ | F ^ {\prime} (S, \mathbf {x}) | \leq \frac {1}{4} | F (S, \mathbf {x}) | \mid | F (S, \mathbf {x}) | > \frac {a}{8} \right] \leq e ^ {- \frac {a}{1 6} \cdot \frac {1}{8}} \leq 0. 0 0 1,
$$
for sufficiently large  $c_{1}$ . By the union bound over this and Equation (23),
$$
\Pr_ {S, \mathbf {x}} \left[ | F ^ {\prime} (S, \mathbf {x}) | \geq \frac {a}{3 2} \right] \geq 0. 9 9 5.
$$
Since each row  $i \in F'$  incurs  $\geq b/2$  errors on examples  $x$  by definition of  $F'$  and  $D$ , since  $F' \subseteq D$  and thus at least  $b/2$  errors on  $j$  for which  $f_{\star}(x_{ij}) \in S$ . Thus,
$$
\operatorname {e r r} (\hat {\theta}) \geq \frac {b \cdot | F ^ {\prime} (S , \mathbf {x}) |}{2 | T |}.
$$
$$
\Pr_ {S, \mathbf {x}} \left[ \operatorname {e r r} (\hat {\theta}) \geq \frac {b a}{6 4 \alpha | T |} \right] \geq 0. 9 9 5.
$$
Now,  $a \geq \frac{1}{2} \log n$  for sufficiently large  $c_{1}$  and we also have  $b \geq \frac{|T|}{2\alpha 10^{5}m}$  by Equation (19) since  $\frac{|T|}{\alpha 10^{5}m} \geq 2$  since  $\alpha \leq 1/2$  and we are in the case where  $10^{5}m \leq |T|$ . Thus,
$$
\operatorname * {P r} _ {S, \mathbf {x}} \left[ \operatorname {e r r} (\hat {\theta}) \geq \frac {\log n}{2 5 6 \alpha 1 0 ^ {5} m} \right] \geq 0. 9 9 5.
$$
This establishes Equation (18) for  $c_{3} \geq 256 \times 10^{5}$ .
D Proofs for random knowledge graph
Our goal in this section is to prove Theorem 3.2. The proof is based on the following main lemma. We first state and prove this lemma, and then derive the theorem from it. Recall that the sets  $T, P \subseteq \mathcal{Y} = Y \times Y = Y^2$  represent the edges of the two knowledge graphs.
Lemma D.1. Fix  $\emptyset \neq S\subseteq Y$ $\pi :Y^{2}\hookrightarrow Y^{2}$ $\delta ,p,\alpha \in (0,1)$ $q\coloneqq 1 - p$  and
$$
\epsilon \geq 3 2 \cdot \frac {\ln (1 / \delta)}{p \alpha^ {2} q ^ {2} | S | ^ {2}}.
$$
For any  $(T,P)\sim \mathrm{KG}(S,Y,p,\alpha)$  chosen from the random knowledge graph distribution, we define
$$
A := \{y \in T \mid \pi (y) \neq y \}
$$
$$
B := \{y \in A \mid \pi (y) \notin P \}
$$
$$
C := \{y \in A \mid y \notin P \}
$$
$$
\operatorname * {P r} _ {T, P} \left[ | A | \geq \epsilon | T | \wedge | B | - | C | \leq \frac {\alpha q}{2} | A | \right] \leq 5 \delta .
$$
Proof. Let  $A' \coloneqq \{y \in S^2 \mid \pi(y) \neq y\}$ , so  $A \subseteq A'$ . If  $\pi$  is the identity then the lemma holds trivially, therefore we can assume that  $A' \neq \emptyset$ . If  $\epsilon > 1$  then the lemma holds trivially as well, because  $A \subseteq T$  and therefore  $|A| \leq |T|$ , so we assume  $\epsilon \in (0,1]$ .
For each  $y \in A'$ , Equations (1) and (2) under Definition 3.1 imply that  $\operatorname{Pr}[y \in C] = \operatorname{Pr}[y \in T \setminus P] = (1 - \alpha)pq$  and  $\operatorname{Pr}[y \in A] = \operatorname{Pr}[y \in T] = p$ , thus Bayes rule gives
$$
\forall y \in A ^ {\prime} \Pr [ y \in C \mid y \in A ] = \frac {(1 - \alpha) p q}{p} = (1 - \alpha) q.
$$
Now suppose we fix  $V \subseteq A'$  and condition on  $A \coloneqq A' \cap T = V$ . Note that the event  $y \in C$  is independent for different  $y \in V$ , therefore for any  $y \in V$  it holds that
$$
\operatorname * {P r} _ {T, P} \left[ y \in C \mid A = V \right] = \operatorname * {P r} _ {T, P} \left[ y \in C \mid y \in A \right] = (1 - \alpha) q.
$$