SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Thus, given that  $n = n_1 + n_2 \geq 2\left(\frac{162(1 + \log(\frac{5}{\delta}))}{\varepsilon^2}\right)$  with probability at least  $1 - \frac{\delta}{5}$ , we have  $n' \geq n\left(1 - \eta - \frac{\varepsilon}{9}\right)$ . For the rest of the argument we will now assume that we have indeed  $n' \geq n\left(1 - \eta - \frac{\varepsilon}{9}\right)$ . The learner now randomly partitions the sample  $S$  into  $S_1$  and  $S_2$  of sizes  $n_1$  and  $n_2$ , respectively. Now let  $S_1' = S_1 \cap S'$  and  $S_2' = S_2 \cap S'$  be the intersections of these sets with the clean set  $S'$ , and  $n_1'$  and  $n_2'$  be their respective sizes. We note that  $n_1' \sim \text{Hypergeometric}(n, n', n_1)$  and  $n_2' \sim \text{Hypergeometric}(n, n', n_2)$ . Thus, assuming  $m' \geq m(1 - \eta - \frac{\varepsilon}{9})$  using Proposition A.2, we have that
$$
\operatorname * {P r} \left[ | S _ {1} ^ {\prime} | \leq \left(1 - \eta - \frac {2 \varepsilon}{9}\right) n _ {1} \right] \leq e ^ {- \frac {2}{8 1} \varepsilon^ {2} n _ {1}}
$$
Recall that Hypergeometric  $(N, K, n)$  is the random variable of the number of "successes" when  $n$  draws are made without replacement from a set of size  $N$ , where  $K$  elements of the set are considered to be successes.
$$
\Pr \left[ | S _ {2} ^ {\prime} | \leq \left(1 - \eta - \frac {2 \varepsilon}{9}\right) n _ {2} \right] \leq e ^ {- \frac {2}{8 1} \varepsilon^ {2} n _ {2}},
$$
where the probability is over the random partition of  $S$ . We note that by our choices of  $n_1$  and  $n_2$ , with probability  $1 - \frac{2\delta}{5}$ , the clean fractions of  $S_1$  and  $S_2$  (namely  $\frac{|S_1'|}{|S_1|}$  and  $\frac{|S_2'|}{|S_2|}$ ) are each at least  $\left(1 - \eta - \frac{2\varepsilon}{9}\right)$ .
$$
\hat {\mathcal {H}} = \left\{\mathcal {A} _ {\mathcal {Q}} ^ {r e} \left(S ^ {\prime \prime}\right): S ^ {\prime \prime} \subset S _ {1} \text {w i t h} | S ^ {\prime \prime} | = \left(1 - \eta - \frac {2 \varepsilon}{9}\right) n _ {1} \right\}
$$
be the set of distributions output by the realizable learning algorithm  $\mathcal{A}_{\mathcal{Q}}^{re}$  when given as input all possible subsets of  $S_{1}$  of size exactly  $\left(1 - \eta -\frac{2\varepsilon}{9}\right)n_1$ . For  $\varepsilon < \frac{9\eta}{2}$ , we know that this set of distributions  $|\hat{\mathcal{H}} |$  is of size  $\binom{n_1}{(1-\eta-\frac{2\varepsilon}{9})n_1} \leq n_1^{2\eta n_1}$ . By the guarantee of the realizable learning algorithm  $\mathcal{A}_{\mathcal{Q}}^{re}$ , if there exists a "clean" subset  $S_1^\prime \subset S_1$  where  $|S_1^{\prime}|\geq n_1(1 - \eta -\frac{2\varepsilon}{9})$  (i.e.,  $S_1^\prime \sim p^{n_1(1 - \eta -\frac{2\varepsilon}{9})}$ ), then with probability  $1 - \frac{\delta}{5}$  there exists a candidate distribution  $q^{*}\in \hat{\mathcal{H}}$  with  $d_{\mathrm{TV}}(p,q^{*}) = \frac{\varepsilon}{9}$ .
We now define and consider the Yatracos sets. For every  $q_i, q_j \in \hat{\mathcal{H}}$ , define the Yatracos set between  $q_i$  and  $q_j$  to be  $A_{i,j} = \{x : q_i(x) \geq q_j(x)\}$ . We let
$$
\mathcal {Y} (\hat {\mathcal {H}}) = \left\{A _ {i, j} \subset \mathcal {X}: q _ {i}, q _ {j} \in \hat {\mathcal {H}} \right\}
$$
denote the set of all pairwise Yatracos sets between distributions in the set  $\mathcal{H}$ .
We now consider the  $A$ -distance [KBG04] between two distributions with respect to the Yatracos sets, i.e., we consider
$$
d_{\mathcal{Y}(\hat{\mathcal{H}})}(p^{\prime},q^{\prime}) = \sup_{B\in \mathcal{Y}(\hat{\mathcal{H}})}|p^{\prime}(B) - q^{\prime}(B)|.
$$
This distance looks at the supremum of the discrepancy between the distributions across the Yatracos sets. Consequently, for any two distributions  $p', q'$  we have  $d_{\mathrm{TV}}(p', q') \geq d_{\mathcal{Y}(\hat{\mathcal{H}})}(p', q')$ , since total variation distance is the supremum of the discrepancy across all possible sets. Furthermore, if  $q', p' \in \hat{\mathcal{H}}$ , then  $d_{\mathrm{TV}}(p', q') = d_{\mathcal{Y}(\hat{\mathcal{H}})}(p', q')$ , since either of the Yatracos sets between the two distributions serves as a set that realizes the total variation distance between them.
Suppose there is some  $q^{*}\in \hat{\mathcal{H}}$  with  $d_{\mathrm{TV}}(p,q^{*})\leq \frac{\varepsilon}{9}$ . Then for every  $q\in \hat{\mathcal{H}}$ :
$$
\begin{array}{l} d _ {\mathrm {T V}} (p, q) \leq d _ {\mathrm {T V}} (p, q ^ {*}) + d _ {\mathrm {T V}} (q ^ {*}, q) \\ \leq \frac {\varepsilon}{9} + d _ {\mathcal {Y} (\hat {\mathcal {H}})} (q ^ {*}, q) \\ \leq \frac {\varepsilon}{9} + d _ {\mathcal {Y} (\hat {\mathcal {H}})} \left(q ^ {*}, p\right) + d _ {\mathcal {Y} (\hat {\mathcal {H}})} (p, q) \\ \leq \frac {\varepsilon}{9} + d _ {\mathrm {T V}} \left(q ^ {*}, p\right) + d _ {\mathcal {Y} (\hat {\mathcal {H}})} (p, q) \\ \leq \frac {2 \varepsilon}{9} + d _ {\mathcal {Y} (\hat {\mathcal {H}})} (p, q). \\ \end{array}
$$
Lastly, we will argue that we can empirically approximate  $d_{\mathcal{Y}(\hat{\mathcal{H}})}(p,q)$ , which we can then use to select a hypothesis. We note that, since  $\mathcal{Y}(\hat{\mathcal{H}})$  is a finite set of size  $\leq \binom{n_1}{(1-\eta-\frac{2\varepsilon}{9})n_1}^2 = \binom{n_1}{(\eta+\frac{2\varepsilon}{9})n_1}^2 \leq ((n_1^{(\eta+\frac{2\varepsilon}{9})}))^2 = n_1^{2(\eta+\frac{2\eta}{9})n_1}$ , we have uniform convergence with respect to
9These sets are also sometimes called Scheffé sets in the literature.
10Note that this definition is asymmetric:  $A_{i,j}\neq A_{j,i}$
<sup>11</sup>A collection of sets  $\mathcal{W}$  has the uniform convergence property if for every  $(\varepsilon, \delta) \in (0, 1)^2$  there is a number  $m_{\mathcal{W}}(\varepsilon, \delta)$  such that for every probability distribution  $p$ , with probability  $\geq (1 - \delta)$  over samples  $S$  of size
$\mathcal{V}(\hat{\mathcal{H}})$ . Recall that, we assumed that there is "clean" subsample  $S_2'' \subset S_2$ , which is i.i.d. distributed according to  $p$  and of size  $(1 - \frac{2\varepsilon}{9} - \eta)n_1$ . We also note that the clean samples  $S_1$  and  $S_2$  are drawn independently from each other. Thus with probability  $1 - \frac{\delta}{5}$ ,  $S_2''$  is  $\frac{\varepsilon}{9}$ -representative of  $p$  with respect to  $\mathcal{V}(\hat{\mathcal{H}})$ . For a sample  $S_0$  and a set  $B \subset \mathcal{X}$ , let us denote  $S_0(B) = \frac{|S_0 \cap B|}{|S_0|}$ . Because of the  $\frac{\varepsilon}{9}$ -representativeness of  $S_2''$ , we have for every  $B \in \mathcal{V}(\hat{\mathcal{H}})$ :
$$
| p (B) - S _ {2} ^ {\prime \prime} (B) | \leq \frac {\varepsilon}{9}.
$$
$$
\begin{array}{l} | p (B) - S _ {2} (B) | \\ = \left| p (B) - \frac {\left| S _ {2} \cap B \right|}{\left| S _ {2} \right|} \right| \\ \leq \max  \left\{\left| p (B) - \frac {\left| S _ {2} ^ {\prime \prime} \cap B \right|}{\left| S _ {2} \right|} \right|, \left| p (B) - \frac {\left| S _ {2} ^ {\prime \prime} \cap B \right| + (\eta + \frac {2 \varepsilon}{9}) n _ {2}}{\left| S _ {2} \right|} \right| \right\} \\ \leq \max \left\{\left| p (B) - \frac {S _ {2} ^ {\prime \prime} (B) | S _ {2} ^ {\prime \prime} |}{n _ {2}} \right|, \left| p (B) - \frac {S _ {2} ^ {\prime \prime} (B) | S _ {2} ^ {\prime \prime} | + (\eta + \frac {2 \varepsilon}{9}) n _ {2}}{n _ {2}} \right| \right\} \\ \leq \max  \left\{\left| p (B) - \left(1 - \eta - \frac {2 \varepsilon}{9}\right) S _ {2} ^ {\prime \prime} (B) \right|, \left| p (B) - \frac {S _ {2} ^ {\prime \prime} (B) \left(1 - \eta - \frac {2 \varepsilon}{9}\right) n _ {2} + \left(\eta + \frac {2 \varepsilon}{9}\right) n _ {2}}{n _ {2}} \right| \right\} \\ \leq \max  \left\{\left| p (B) - S _ {2} ^ {\prime \prime} (B) \right| + \left| S _ {2} ^ {\prime \prime} (B) - \left(1 - \eta - \frac {2 \varepsilon}{9}\right) S _ {2} ^ {\prime \prime} (B) \right|, \left| p (B) - \left(S _ {2} ^ {\prime \prime} (B) \left(1 - \eta - \frac {2 \varepsilon}{9}\right) + \left(\eta + \frac {2 \varepsilon}{9}\right)\right) \right| \right\} \\ \leq \max  \left\{\frac {3 \varepsilon}{9} + \eta , \left| p (B) - S _ {2} ^ {\prime \prime} (B) \left(1 - \eta - \frac {2 \varepsilon}{9}\right) - \left(\eta + \frac {2 \varepsilon}{9}\right) \right| \right\} \\ \leq \max  \left\{\frac {3 \varepsilon}{9} + \eta , \left| p (B) - S _ {2} ^ {\prime \prime} (B) + S _ {2} ^ {\prime \prime} (B) \left(\eta + \frac {2 \varepsilon}{9}\right) - \left(\eta + \frac {2 \varepsilon}{9}\right) \right| \right\} \\ \leq \max  \left\{\frac {3 \varepsilon}{9} + \eta , | p (B) - S _ {2} ^ {\prime \prime} (B) | + \left| S _ {2} ^ {\prime \prime} (B) \left(\eta + \frac {2 \varepsilon}{9}\right) - \left(\eta + \frac {2 \varepsilon}{9}\right) \right| \right\} \\ \leq \max  \left\{\frac {3 \varepsilon}{9} + \eta , \frac {\varepsilon}{9} + \left| S _ {2} ^ {\prime \prime} (B) - 1 \right| \left(\eta + \frac {2 \varepsilon}{9}\right) \right\} \\ \leq \max  \left\{\frac {3 \varepsilon}{9} + \eta , \frac {\varepsilon}{9} + \left(\eta + \frac {2 \varepsilon}{9}\right) \right\} \\ \leq \frac {3 \varepsilon}{9} + \eta \\ \end{array}
$$
Let the empirical  $A$ -distance with respect to the Yatracos sets be defined by
$$
d _ {\mathcal {Y} (\hat {\mathcal {H}})} (q, S) = \sup  _ {B \in d _ {\mathcal {Y} (\hat {\mathcal {H}})}} | q (B) - S (B) |.
$$
$n > n_{\mathcal{W}}(\varepsilon, \delta)$  generated i.i.d. by  $p$ , a sample  $S$  is  $\varepsilon$ -representative for  $\mathcal{W}$  with respect to  $p$ . Namely, for every  $A \in \mathcal{W}$ ,  $\left|\frac{|A \cap S|}{|S|} - p(A)\right| \leq \varepsilon$ . If  $\mathcal{W}$  is finite then  $n_{\mathcal{W}}(\varepsilon, \delta) \leq \frac{\log(|\mathcal{W}|) + \log(1 / \delta)}{\varepsilon^2}$ . For more details see Chapter 4 in [SB14].
Now if the learner outputs  $\hat{q} \in \arg \min_{q \in \mathcal{H}} d_{\mathcal{Y}(\hat{\mathcal{H}})}(q, S)$ , then putting all of our guarantees together, we get that with probability  $1 - \delta$
$$
\begin{array}{l} d _ {\mathrm {T V}} (\hat {q}, p) \leq \frac {2 \varepsilon}{9} + d _ {\mathcal {Y} (\hat {\mathcal {H}})} (\hat {q}, p) \\ \leq \frac {5 \varepsilon}{9} + \eta + d _ {\mathcal {Y} (\hat {\mathcal {H}})} (\hat {q}, S _ {2}) \\ \leq \frac {5 \varepsilon}{9} + \eta + d _ {\mathcal {Y} (\hat {\mathcal {H}})} \left(q ^ {*}, S _ {2}\right) \\ \leq \frac {8 \varepsilon}{9} + 2 \eta + d _ {\mathcal {Y} (\hat {\mathcal {H}})} (q ^ {*}, p) \\ \leq \frac {8 \varepsilon}{9} + 2 \eta + \frac {\varepsilon}{9} \leq 2 \eta + \varepsilon . \\ \end{array}
$$
C Robust Learnability with Subtractive Contamination Implies Robust Learnability with General Contamination
In this section we will provide the proof for Theorem 1.7.
Proof. Let  $\mathcal{C}$  be a concept class that is  $\eta$ -subtractively  $\alpha$ -robust learnable. Then there exists a successful  $\eta$ -subtractive  $\alpha$ -robust learner  $\mathcal{A}_{\mathcal{C}}^{sub}$  with sample complexity  $n_{\mathcal{C}}^{sub}$  for the class  $\mathcal{C}$ . Let  $\varepsilon < \frac{9\eta}{2}$  and  $\delta$  and be arbitrary.
Let  $n_1 \geq \max \left\{2n_{\mathcal{C}}^{sub}\left(\frac{\varepsilon}{9}, \frac{\delta}{5}\right) / (1 - \eta - \frac{2\varepsilon}{9}), \frac{162(1 + \log\left(\frac{5}{\delta}\right))}{\varepsilon^2}\right\}$  and  $n_2 \geq$
$\left\{\frac{(4(\eta + \frac{2\varepsilon}{9})n_1\log(n_1) + \log(\frac{5}{\delta}))}{\varepsilon^2},\frac{162(1 + \log(\frac{5}{\delta}))}{\varepsilon^2}\right\} .$  Lastly let  $n = n_{1} + n_{2}$
Let  $p \in \mathcal{C}$  be arbitrary. The  $\alpha$ - $\eta$ -robust learner receives a sample  $S \sim p^m$  such that there is  $q \in \mathcal{C}$  such that  $d_{\mathrm{TV}}(p, q) = \eta$ . Thus there exists a distributions  $q_1, q_2, q_3$ , such that  $(1 - \eta)q_1 + \eta q_2 = p$  and  $(1 - \eta)q_1 + q_3 = q$ . We now use the same learning strategy as in Theorem 1.5: We split the sample randomly into two subsamples  $S_1$  and  $S_2$ , where we use  $S_1$  to learn candidate sets and then use  $S_2$  to select the hypothesis from the candidate set. The goal in both settings is to find as close an approximation to  $q_1$  as possible. The candidate based on  $S_1$  is created by feeding subsamples of  $S_1$  into the subtractively robust learner in such a way that with high probability one of the subsamples is guaranteed to be i.i.d. generated by  $q_1$  and thus (with high probability) yield a good hypothesis. More precisely, the learner randomly splits the sample  $S$  into  $S_1$  and  $S_2$  with  $|S_1| = n_1$  and  $|S_1| = n_2$ . We now define the "clean" part of  $S' \subset S$ , i.e. the part of  $S'$  that is i.i.d. distributed according to  $q_1$ . We note that the size of this "clean" sample  $|S'| = n'$  is a random variable and distributed according to the binomial distributions  $Binom(n, 1 - \eta)$ . Now applying Chernoff bound, with the same argument as in the proof of Theorem 1.5, we get that with probability  $1 - \frac{\delta}{5}$ , we have  $n' \geq n(1 - \eta - \frac{\varepsilon}{9})$ . Now let  $S_1' = S_1 \cap S'$  and  $S_2' = S_2 \cap S'$  be the "clean parts" of the subsamples  $S_1$  and  $S_2$  respectively. The sizes  $|S_1''| = n_1'$  and  $|S_2''| = n_2''$ . We note that  $n_1' \sim Hypergeometric(n, n(1 - \eta), n_1)$  and  $n_2' \sim Hypergeometric(n, n(1 - \eta), n_2)$ . Thus,
$$
P r _ {\text {r a n d o m s p l i t}} \left[ | S _ {1} ^ {\prime} | \leq \left(1 - \eta - \frac {2 \varepsilon}{9}\right) n _ {1} \right] \leq e ^ {- \frac {2}{8 1} \varepsilon^ {2} n _ {1}}
$$
$$
P r _ {\text {r a n d o m s p l i t}} \left[ | S _ {2} ^ {\prime} | \leq \left(1 - \eta - \frac {2 \varepsilon}{9}\right) n _ {2} \right] \leq e ^ {- \frac {2}{8 1} \varepsilon^ {2} n _ {2}}.
$$
Taking together the guarantees on our random splits and the size of  $n'$ , we note that by our choices of  $n_1$  and  $n_2$  with probability  $1 - \frac{3\delta}{5}$ , the fractions of the parts that are i.i.d. generated by  $q_1$  (namely  $\frac{|S_1''|}{|S_1'|}$  and  $\frac{|S_2''|}{|S_2'|}$ ) are at least  $(1 - \eta - \frac{2\varepsilon}{9})$ . Going forward we will assume that this is indeed the case. Let
$$
\hat {\mathcal {H}} = \left\{\mathcal {A} _ {\mathcal {Q}} ^ {s u b} (\tilde {S}): \tilde {S} \subset S _ {1} ^ {\prime} \mathrm {w i t h} | \tilde {S} | = \left(1 - \eta - \frac {2 \varepsilon}{9}\right) n _ {1} \right\}.
$$
Using our assumption that  $|S_1'| \geq (1 - \eta - \frac{2\varepsilon}{9})n_1$ , we know that there is  $S_1'' \subset S_1'$  with  $\mathcal{A}_{\mathcal{Q}}^{sub}(S_1'') \in \mathcal{H}'$ . As  $S_1'' \sim q_1^{(1 - \eta - \frac{2\varepsilon}{9})n_1}$ , by the learning guarantee of  $\mathcal{A}_{\mathcal{Q}}^{sub}$  with probability  $1 - \frac{\delta}{5}$ , there is a candidate distribution  $q^* \in \hat{\mathcal{H}}$  with  $d_{\mathrm{TV}}(p, q^*) \leq d_{\mathrm{TV}}(p, q_1) + d_{\mathrm{TV}}(q_1, q^*) = \eta + (\alpha \eta + \frac{\varepsilon}{9}) = (\alpha + 1)\eta + \frac{\varepsilon}{9}$ . We now consider the Yatracos sets. For every  $q_i, q_j \in \hat{\mathcal{H}}$ , let  $A_{i,j} = \{x : q_i(x) \geq q_j(x)\}$  and let
$$
\mathcal {Y} (\hat {\mathcal {H}}) = \left\{A _ {i j} \subset \mathcal {X}: q _ {i}, q _ {j} \in \hat {\mathcal {H}} \right\}.
$$
$$
d _ {\mathcal {Y} (\hat {\mathcal {H}})} (p ^ {\prime}, q ^ {\prime}) = \sup  _ {B \in \mathcal {Y} (\hat {\mathcal {H}})} | p ^ {\prime} (B) - q ^ {\prime} (B) |.
$$
We note, that for any two distributions  $p', q'$  we have  $d_{\mathrm{TV}}(p', q') \geq d_{\mathcal{Y}(\hat{\mathcal{H}})}(p', q')$ . Furthermore, if  $q', p' \in \hat{\mathcal{H}}$ , then  $d_{\mathrm{TV}}(p', q') = d_{\mathcal{Y}(\hat{\mathcal{H}})}(p', q')$ . Assume there is  $q^* \in \hat{\mathcal{H}}$  with  $d_{\mathrm{TV}}(p, q^*) \leq (\alpha + 1)\eta + \frac{\varepsilon}{9}$ , then for every  $q \in \hat{\mathcal{H}}$ :
$$
\begin{array}{l} d _ {\mathrm {T V}} (p, q) \leq d _ {\mathrm {T V}} (p, q ^ {*}) + d _ {\mathrm {T V}} (q ^ {*}, q) \\ \leq (\alpha + 1) \eta + \frac {\varepsilon}{9} + d _ {\mathcal {Y} (\hat {\mathcal {H}})} (q ^ {*}, q) \\ \leq (\alpha + 1) \eta + \frac {\varepsilon}{9} + d _ {\mathcal {Y} (\hat {\mathcal {H}})} \left(q ^ {*}, p\right) + d _ {\mathcal {Y} (\hat {\mathcal {H}})} (p, q) \\ \leq (\alpha + 1) \eta + \frac {\varepsilon}{9} + d _ {\mathrm {T V}} \left(q ^ {*}, p\right) + d _ {\mathcal {Y} (\hat {\mathcal {H}})} (p, q) \\ \leq 2 (\alpha + 1) \eta + \frac {2 \varepsilon}{9} + d _ {\mathcal {Y} (\hat {\mathcal {H}})} (p, q). \\ \end{array}
$$
Lastly, we will argue that we can empirically approximate  $d_{\mathcal{V}(\hat{\mathcal{H}})}(p,q)$ , which we can then use to select a hypothesis. We note that, since  $\mathcal{V}(\hat{\mathcal{H}})$  is a finite set of size  $|\mathcal{V}(\hat{\mathcal{H}})| = \left(\binom{n_1}{(1-\eta-\frac{2\varepsilon}{9})n_1}\right)^2 \leq n_1^{2(\eta+\frac{2\varepsilon}{9})n_1}$ , we have uniform convergence with respect to  $\mathcal{V}(\hat{\mathcal{H}})$ . Recall that  $S_2' \sim q_1^{n_2'}$  and by our previous assumption  $n_2' \leq n_2\left(1-\eta-\frac{2\varepsilon}{9}\right)$ . Thus by our choice of  $n_2$ , with probability  $1-\frac{\delta}{5}$ , there is  $S_2'' \subset S_2' \subset S_2$  with  $|S_2''|=\left(1-\eta-\frac{2\varepsilon}{9}\right)n_2$  such that  $S_2''$  is  $\frac{\varepsilon}{9}$ -representative of  $q_1$  with respect to  $\mathcal{V}(\hat{\mathcal{H}})$ . For a sample  $S_0$  and a set  $B \subset \mathcal{X}$ , let us denote  $S_0(B)=\frac{|S_0 \cap B|}{|S_0|}$ . Because of the  $\frac{\varepsilon}{9}$ -representativeness of  $S_2'$ , we have for every  $B \in \mathcal{V}(\hat{\mathcal{H}})$ :
$$
\left| q _ {1} (B) - S _ {2} ^ {\prime \prime} (B) \right| \leq \frac {\varepsilon}{9}
$$
$$
\begin{array}{l} \left| q _ {1} (B) - S _ {2} (B) \right| \leq \left| q _ {1} (B) - S _ {2} ^ {\prime \prime} (B) \right| + \left| S _ {2} ^ {\prime \prime} (B) - S _ {2} (B) \right| \\ \leq \frac {\varepsilon}{9} + \left| \frac {| S _ {2} \cap B |}{| S _ {2} |} - \frac {| S _ {2} ^ {\prime \prime} \cap B |}{| S _ {2} ^ {\prime \prime} |} \right| \\ \leq \frac {\varepsilon}{9} + \left| \frac {| S _ {2} \cap B |}{n _ {2}} - \frac {| S _ {2} ^ {\prime \prime} \cap B |}{n _ {2} \left(1 - \eta - \frac {2 \varepsilon}{9}\right)} \right| \\ = \frac {\varepsilon}{9} + \left| \frac {| S _ {2} \cap B | (1 - \eta - \frac {2 \varepsilon}{\eta}) - | S _ {2} ^ {\prime \prime} \cap B |}{(1 - \eta - \frac {2 \varepsilon}{9}) n _ {2}} \right| \\ \leq \frac {\varepsilon}{9} + \max  \left\{\frac {| (| S _ {2} ^ {\prime \prime} \cap B | + (\eta + \frac {2 \varepsilon}{9}) n _ {2}) (1 - \eta - \frac {2 \varepsilon}{9}) - | S _ {2} ^ {\prime \prime} \cap B | |}{(1 - \eta - \frac {2 \varepsilon}{9}) n _ {2}}, \right. \\ \left. \right. \frac {\left| \right.\left( \right.\left| S _ {2} ^ {\prime \prime} \cap B \right|\left(1 - \eta - \frac {2 \varepsilon}{9}\right) - \left| S _ {2} ^ {\prime \prime} \cap B \right|\left. \right|\left. \right.}{\left.\left(1 - \eta - \frac {2 \varepsilon}{9}\right) n _ {2}} \right\rbrace \\ \leq \frac {\varepsilon}{9} \\ + \max  \left\{ \right.\frac {\left| \right.\left(\left| S _ {2} ^ {\prime \prime} \cap B \right| + (\eta + \frac {2 \varepsilon}{9}) n _ {2}\right)\left(1 - \eta - \frac {2 \varepsilon}{9}\right) - \left( \right.\left(1 - \eta - \frac {2 \varepsilon}{9}\right) \mid S _ {2} ^ {\prime \prime} \cap B \left. \right| + \left(\eta + \frac {2 \varepsilon}{9}\right) \mid S _ {2} ^ {\prime \prime} \cap B \left. \right|\left. \right) |}{n _ {2} \left(1 - \eta - \frac {2 \varepsilon}{9}\right)}, \left. \right. \\ \frac {n _ {2} (\eta + \frac {2 \varepsilon}{9})}{n _ {2} (1 - \eta - \frac {2 \varepsilon}{9})} \} \\ \leq \frac {\varepsilon}{9} + \max  \left\{\frac {| (\eta + \frac {2 \varepsilon}{9}) n _ {2} (1 - \eta - \frac {2 \varepsilon}{9}) - | S _ {2} ^ {\prime \prime} \cap B | (\eta + \frac {2 \varepsilon}{9}) |}{(1 - \eta - \frac {2 \varepsilon}{9}) n _ {2}}, \frac {(\eta + \frac {2 \varepsilon}{9})}{(1 - \eta - \frac {2 \varepsilon}{9})} \right\} \\ \leq \frac {\varepsilon}{9} + \max  \left\{\frac {| (\eta + \frac {2 \varepsilon}{9}) (n _ {2} (1 - \eta - \frac {2 \varepsilon}{9}) |)}{(1 - \eta - \frac {2 \varepsilon}{9}) n _ {2}}, \eta + \frac {2 \varepsilon}{9}) \right\} \\ \leq \frac {\varepsilon}{9} + \eta + \frac {2 \varepsilon}{9} \leq \frac {3 \varepsilon}{9} + \eta \\ \end{array}
$$
Let us remember that the empirical  $A$ -distance with respect to the Yatracos is defined by
$$
d _ {\mathcal {Y} (\hat {\mathcal {H}})} (q, S) = \sup  _ {B \in \mathcal {Y}} | q (B) - S (B) |.
$$
Now if the learner outputs  $\hat{q} \in \arg \min_{q \in \hat{\mathcal{H}}} d_{\mathcal{Y}(\hat{\mathcal{H}})}(q, S_2)$ , then putting all of our guarantees together, with probability  $1 - \delta$  we get
$$
\begin{array}{l} d _ {\mathrm {T V}} (\hat {q}, p) \leq 2 (\alpha + 1) \eta + \frac {2 \varepsilon}{9} + d _ {\mathcal {Y}} (\hat {q}, p) \\ \leq 2 (\alpha + 1) \eta + \frac {2 \varepsilon}{9} + d _ {\mathcal {Y}} (\hat {q}, q _ {1}) + d _ {\mathcal {Y}} (q _ {1}, p) \\ \leq 2 (\alpha + 1) \eta + \frac {2 \varepsilon}{9} + \eta + \left(\eta + \frac {3 \eta}{9}\right) + d _ {\mathcal {Y}} (\hat {q}, S _ {2}) \\ \leq 2 (\alpha + 2) \eta + \frac {5 \varepsilon}{9} + d _ {\mathcal {Y}} (\hat {q}, S _ {2}) \\ \leq 2 (\alpha + 2) \eta + \frac {5 \varepsilon}{9} + d _ {\mathcal {Y}} \left(q ^ {*}, S _ {2}\right) \\ \leq 2 (\alpha + 2) \eta + \frac {5 \varepsilon}{9} + \left(\eta + \frac {3 \varepsilon}{9}\right) + d _ {\mathcal {Y}} \left(q ^ {*}, q _ {1}\right) \\ \leq (2 \alpha + 3) \eta + \frac {8 \varepsilon}{9} + \eta + d _ {\mathcal {Y}} \left(q ^ {*}, p\right) \\ \leq (2 \alpha + 4) \eta + \varepsilon + d _ {\mathrm {T V}} \left(q ^ {*}, p\right). \\ \end{array}
$$
D Learnability Does Not Imply Robust Learnability
We start with an upper bound, showing that our class  $\mathcal{Q}_g$  is realizably learnable.
Proof. Let the realizable learner  $\mathcal{A}$  be
Note that for all  $\mathcal{Q}_g$ -realizable samples this learner is well-defined. Furthermore, we note that in the realizable case, whenever  $\mathcal{A}$  outputs a distribution different from  $\delta_{(0,0)}$ , then  $\mathcal{A}(S)$  outputs the ground-truth distribution, i.e., the output has TV-distance 0 to the true distribution. Lastly, we note, that for an i.i.d. sample  $S \sim q_{i,j,g(j)}^n$ , we have the following upper bound for the learner identifying the correct distribution:
$$
\mathbb {P} _ {S \sim q _ {i, j, g (j)} ^ {n}} [ \mathcal {A} (S) = q _ {(j)} ] = \mathbb {P} _ {S \sim q _ {i, j, g (j)} ^ {n}} [ (i, 2 j + 2) \in S ] = 1 - (1 - 1 / g (j)) ^ {n}.
$$
We note, that since  $g$  is a monotone function, if  $\varepsilon \leq \frac{1}{j}$ , then  $g(j) \leq g\left(\frac{1}{\varepsilon}\right)$  and therefore,
$$
(1 - 1 / g (j)) ^ {n} \leq (1 - 1 / g (1 / \varepsilon)) ^ {n}.
$$
Furthermore for  $q_{i,j,g(j)}$ , we have that  $d_{\mathrm{TV}}(\delta_{(0,0)}, q_{i,j,g(j)}) = \frac{1}{j}$ .
Putting these two observations together, we get
$$
\mathbb{P}_{S\sim q_{i,j,g(j)}^{n}}[d_{\mathrm{TV}}(\mathbb{A}(S),q_{i,j,g(j)})\geq \varepsilon ]\leq \left\{ \begin{array}{ll}(1 - 1 / g(1 / \varepsilon))^{n} & \text{if}\frac{1}{j}\leq \varepsilon \\ 0 & \text{if}\frac{1}{j} >\varepsilon \end{array} \right..
$$
Thus, for every  $q \in \mathcal{Q}_g$ ,
$$
\mathbb {P} _ {S \sim q ^ {n}} [ d _ {\mathrm {T V}} (\mathbb {A} (S), q) \geq \varepsilon ] \leq (1 - 1 / g (1 / \varepsilon)) ^ {n} \leq \exp \left(- \frac {n}{g (1 / \varepsilon)}\right).
$$
Letting the left-hand side equal the failure probability  $\delta$  and solving for  $n$ , we get,
$$
\log \delta \geq \frac {- n}{g (1 / \varepsilon)}
$$
$$
\log (\delta) g (1 / \varepsilon) \geq - n
$$
$$
n \geq - \log (\delta) g (1 / \varepsilon) = \log (1 / \delta) g (1 / \varepsilon).
$$
Thus, we have a sample complexity bound of
$$
n _ {Q _ {g}} (\varepsilon , \delta) \leq \log (1 / \delta) g (1 / \varepsilon).
$$
Now, we show a lower bound, that our class  $\mathcal{Q}_g$  is not robustly learnable. Before we do that, we require a few more preliminaries. For a distribution class  $\mathcal{Q}$  and a distribution  $p$ , let their total variation distance be defined by
$$
d_{\mathrm{TV}}(p,\mathcal{Q}) = \inf_{q\in \mathcal{Q}}d_{\mathrm{TV}}(p,q).
$$
We also use the following lemma from [LBD23].