SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lemma D.1 (Lemma 3 from [LBD23]). Let  $\mathcal{P}_{\eta,4k} = \{(1 - \eta)\delta_{(0,0)} + \eta U_{A \times \{2j+1\}} : A \subset [4k]\}$  For  $\mathcal{Q} = \mathcal{P}_{\eta,4k}$ , we have  $n_{\mathcal{Q}}^{re}\left(\frac{\eta}{8}, \frac{1}{7}\right) \geq k$ .
Finally, we recall the definition of weak learnability, which says that a distribution class is learnable only for some particular value of the accuracy parameter.
Definition D.2. A class  $\mathcal{Q}$  is  $\varepsilon$ -weakly learnable, if there is a learner  $\mathcal{A}$  and a sample complexity function  $n:(0,1)\to \mathbb{N}$ , such that for every  $\delta \in (0,1)$  and every  $p\in \mathcal{Q}$  and every  $n\geq n(\delta)$ ,
$$
\mathbb {P} _ {S \sim p ^ {n}} [ d _ {\mathrm {T V}} (\mathcal {A} (S), p) \leq \varepsilon ] <   \delta .
$$
Learnability clearly implies  $\varepsilon$ -weak learnability for every  $\varepsilon \in (0,1)$ . While in some learning models (e.g., binary classification) learnability and weak learnability are equivalent, the same is not true for distribution learning [LBD23].
We are now ready to prove that  $\mathcal{Q}_g$  is not robustly learnable.
Proof. Consider
$$
q _ {i, j} ^ {\prime} = \left(1 - \frac {1}{\dot {j}}\right) \delta_ {(0, 0)} + \frac {1}{\dot {j}} U _ {A _ {i} \times \{2 j + 1 \}}
$$
Note that  $d_{\mathrm{TV}}(q_{i,j}^{\prime}, \mathcal{Q}_g) = \frac{1}{g(j)}$ . Therefore, in order to show that  $\mathcal{Q}_g$  is not  $\alpha$ -robustly learnable, it is sufficient to show that there are  $j$  and  $\varepsilon$ , such that the class  $\mathcal{Q}_j^\prime = \{q_{i,j}^\prime : i \in \mathbb{N}\}$  is not  $(\frac{\alpha}{g(j)} + \varepsilon)$ -weakly learnable.
We will now show that for any  $\gamma < \frac{1}{8j}$ , the class  $\mathcal{Q}_j' = \{q_{i,j}' : i \in \mathbb{N}\}$  is not  $\gamma$ -weakly learnable. Recalling notation from Lemma D.1, we note that for every  $n \in \mathbb{N}$  the class  $P_{\frac{1}{j}, 4k} \subset \mathcal{Q}_j'$ . By monotonicity of the sample complexity and Lemma D.1, we have  $n \mathcal{Q}_j'(\frac{1}{8j}, \frac{1}{7}) \geq n_{P_{1/j, 4n}}(\frac{1}{8j}, \frac{1}{7}) \geq n$ , proving that this class is not weakly learnable.
Lastly, we need to show that for every  $\alpha$ , there are  $\varepsilon$  and  $j$ , such that this claim holds for  $\gamma \leq \left(\frac{\alpha}{g(j)} + \varepsilon\right)$ . That is, we need to show that there are  $\varepsilon$  and  $j$ , such that
$$
\frac {\alpha}{g (j)} + \varepsilon <   \frac {1}{8 j}.
$$
Let  $\varepsilon = \frac{1}{16j}$ . Now let  $g$  be any superlinear function, i.e., for every  $c \in \mathbb{R}$ , there is  $t_c \in \mathbb{N}$ , such that for every  $t \geq t_c$ ,  $g(t) \geq ct$ . This implies that, for any  $\alpha \in \mathbb{R}$ , there is  $j \in \mathbb{N}$  such that  $g(j) > 16j\alpha$ . Thus for any super-linear function  $g$  and any  $\alpha \in \mathbb{R}$ , the class  $\mathcal{Q}_g$  is not  $\alpha$ -robustly learnable.
D.1 Proof of Theorem 1.6
The result of Theorem 1.6 follows directly from the construction of class  $\mathcal{Q}_g$  for Theorem 3.1, the Claim 3.2 that shows this class is realizable learnable, and an adapted version for Claim 3.3, which states the following:
Claim D.3. For every  $\alpha$ , there is  $g(t) \in O(t^2)$ , such that for every  $0 \leq \eta \leq \frac{1}{16\alpha}$  the class  $\mathcal{Q}_g$  is not  $\eta$ -subtractive  $\alpha$ -robustly learnable.
Proof. Let  $\alpha > 1$  be arbitrary. Let  $g: \mathbb{N} \to \mathbb{N}$  be defined by  $g(t) = 32\alpha t^2$  for all  $t \in \mathbb{N}$ . Now for every  $0 \leq \eta \leq \frac{1}{16\alpha}$ , there exists some  $j$ , such that  $\frac{j}{g(j)} \leq \eta \leq \frac{1}{16\alpha j}$
For such  $j$ , we consider the distributions
$$
q _ {i, j} ^ {\prime} = \left(1 - \frac {1}{j}\right) \delta_ {(0, 0)} + \frac {1}{j} U _ {A _ {i} \times \{2 j + 1 \}}
$$
as in the proof of Claim 3.2. Recall that the element of  $\mathcal{Q}_g$  are of the form
$$
q _ {i, j, g (j)} = \left(1 - \frac {1}{j}\right) \delta_ {(0, 0)} + \left(\frac {1}{j} - \frac {1}{g (j)}\right) U _ {A _ {i} \times \{2 j + 1 \}} + \frac {1}{g (j)} \delta_ {(i, 2 j + 2)}
$$
Then we have,
$$
\begin{array}{l} q _ {i, j, g (j)} = \left(1 - \frac {1}{j}\right) \delta_ {(0, 0)} + \left(\frac {1}{j} - \frac {1}{g (j)}\right) U _ {A _ {i} \times \{2 j + 1 \}} + \frac {1}{g (j)} \delta_ {(i, 2 j + 2)} = \\ = \left(1 - \frac {1}{j}\right) \delta_ {(0, 0)} + \left(\frac {g (j) - j}{j g (j)}\right) U _ {A _ {i} \times \{2 j + 1 \}} + \frac {j}{j g (j)} \delta_ {(i, 2 j + 2)} = \\ = \left(\frac {g (j) - j}{g (j)}\right) \left(\left(1 - \frac {1}{j}\right) \delta_ {(0, 0)} + \frac {1}{j} U _ {A _ {i} \times \{(2, j + 1) \}}\right) + \frac {j}{g (j)} \left(\left(1 - \frac {1}{j}\right) \delta_ {(0, 0)} + \frac {1}{j} \delta_ {(i, 2 j + 2)}\right) \\ = \left(1 - \frac {j}{g (j)}\right) q _ {i, j} ^ {\prime} + \frac {j}{g (j)} \left(\left(1 - \frac {1}{j}\right) \delta_ {(0, 0)} + \frac {1}{j} \delta_ {(i, 2 j + 2)}\right). \\ \end{array}
$$
Thus for every element  $q_{i,j}^{\prime}$  of the class  $\mathcal{Q}_j^\prime = \{q_{i,j}^\prime : i \in \mathbb{N}\}$ , there is a distribution  $p$ , such that  $(1 - \eta) q_{i,j}^{\prime} + \eta p \in \mathcal{Q}_g$ . That is, every element of  $\mathcal{Q}_j^\prime$  results from the  $\eta$ -subtractive contamination of some element in  $\mathcal{Q}_g$ . Thus, for showing that  $\mathcal{Q}_g$  is not  $\eta$ -subtractive  $\alpha$ -robustly learnable, it is sufficient to show, that  $\mathcal{Q}_j^\prime$  is not  $(\alpha \eta + \varepsilon)$ -weakly learnable for  $\varepsilon = \frac{1}{16j}$ . As we have seen in the proof of Claim 3.3, we can use Lemma D.1 to show that for every  $n$ , we have  $n \mathcal{Q}_j^\prime \left( \frac{1}{8j}, \frac{1}{7} \right) \geq n$ . Lastly, we need that  $\frac{1}{8j} \geq \alpha \eta + \varepsilon$ , or after replacing  $\varepsilon$ , we need  $\frac{1}{16j} \geq \alpha \eta$ . This follows directly from the choice of  $g$ .
E Existence of sample compression schemes
We adopt the  $\left[\mathrm{ABDH}^{+}20\right]$  definition of sample compression schemes. We will let  $\mathcal{C}$  be a class of distribution over some domain  $\mathcal{X}$ . A compression scheme for  $\mathcal{C}$  involves two parties: an encoder and a decoder.
- The encoder has some distribution  $q \in \mathcal{C}$  and receives  $n$  samples from  $q$ . They send a succinct message (dependent on  $q$ ) to the decoder, which will allow the decoder to output a distribution close to  $q$ . This message consists of a subset of size  $\tau$  of the  $n$  samples, as well as  $t$  additional bits.
- The decoder receives the  $\tau$  samples and  $t$  bits and outputs a distribution which is close to  $q$ .
Since this process inherently involves randomness (of the samples drawn from  $q$ ), we require that this interaction succeeds at outputting a distribution close to  $q$  with only constant probability.
More formally, we have the following definitions for a decoder and a (robust) compression scheme.
Definition E.1 (decoder, Definition 4.1 of  $\mathrm{[ABDH^{+}20]}$ ). A decoder for  $\mathcal{C}$  is a deterministic function  $\mathcal{I}:\bigcup_{n = 0}^{\infty}\mathcal{X}^n\times \bigcup_{n = 0}^{\infty}\{0,1\} ^n\to \mathcal{C}$ , which takes a finite sequence of elements of  $\mathcal{X}$  and a finite sequence of bits, and outputs a member of  $\mathcal{C}$ .
The formal definition of a compression scheme follows.
Definition E.2 (robust compression schemes, Definition 4.2 of  $[\mathrm{ABDH}^+20])$ . Let  $\tau, t, n: (0,1) \to \mathbb{Z}_{\geq 0}$  be functions, and let  $r \geq 0$ . We say  $\mathcal{C}$  admits  $(\tau, t, n)$ $r$ -robust compression if there exists a decoder  $\mathcal{I}$  for  $\mathcal{C}$  such that for any distribution  $q \in \mathcal{C}$  and any distribution  $p$  on  $\mathcal{X}$  with  $d_{\mathrm{TV}}(p, q) \leq r$ , the following holds:
For any  $\varepsilon \in (0,1)$ , if a sample  $S$  is drawn from  $p^{n(\varepsilon)}$ , then, with probability at least  $2/3$ , there exists a sequence  $L$  of at most  $\tau(\varepsilon)$  elements of  $S$ , and a sequence  $B$  of at most  $t(\varepsilon)$  bits, such that  $d_{\mathrm{TV}}(\mathcal{J}(L,B),\mathcal{C}) \leq r + \varepsilon$ .
Note that  $S$  and  $L$  are sequences rather than sets, and can potentially contain repetitions.
Theorem E.3 (Compression implies learning, Theorem 4.5 of  $\mathrm{[ABDH^{+}20]}$ ). Suppose  $\mathcal{C}$  admits  $(\tau ,t,n)$ $r$  -robust compression. Let  $\tau^{\prime}(\varepsilon)\tau (\varepsilon) + t(\varepsilon)$ . Then  $\mathcal{C}$  can be  $\max \{3,2 / r\}$ -learned in the agnostic setting using
$$
O \left(n \Big (\frac {\varepsilon}{6} \Big) \log \Big (\frac {1}{\delta} \Big) + \frac {\tau^ {\prime} (\varepsilon / 6) \log (n (\varepsilon / 6) \log_ {3} (1 / \delta)) + \log (1 / \delta)}{\varepsilon^ {2}}\right) = \widetilde {O} \left(n \Big (\frac {\varepsilon}{6} \Big) + \frac {\tau^ {\prime} (\varepsilon / 6) \log n (\varepsilon / 6)}{\varepsilon^ {2}}\right)
$$
samples. If  $\mathcal{Q}$  admits  $(\tau, t, n)$  non-robust compression, then  $\mathcal{Q}$  can be learned in the realizable setting using the same number of samples.
Theorem E.4. The class  $\mathcal{Q} = \mathcal{Q}_g$  from Section 3 has a compression scheme of message size 1 (i.e., using just a single sample point).
Proof. Let  $n(\varepsilon)$  be  $10 / g(\varepsilon)$  and, give a sample  $S$  of at least that size, let the encoder pick a subset  $L(S) \subseteq S$  be
$$
L (S) = \left\{ \begin{array}{l l} \{(i, 2 j + 2) \} & \text {i f} (i, 2 j + 2) \in S \\ \{(0, 0) \} & \text {o t h e r w i s e} \end{array} \right.
$$
Let the decoder output
$$
\mathcal {J} (L) = \left\{ \begin{array}{l l} q _ {i, j, g (j)} & \text {i f} (i, 2 j + 2) \in L \\ \delta_ {(0, 0)} & \text {o t h e r w i s e} \end{array} \right.
$$
With this construction established, the analysis follows very similarly to the analysis in the proof of Claim 3.2.
We note that the claim of Theorem 1.8 (and Theorem 3.4) follows directly.
F Approximate DP learnability vs robust learnability
We prove the second claim in Theorem 4.3 by showing that the learner for the class  $\mathcal{Q}$  described in Theorem 3.1 can be made differentially private by employing stability-based histograms [KKMN09, BNS16].
Proposition F.1 (Stability-based histograms [KKMN09, BNS16], Lemma 4.1 from [AAL21]). Let  $\mathcal{X}$  be a domain of examples. Let  $K$  be a countable index set, and let  $(h_k)_{k\in K}$  be a sequence of disjoint histogram bins over  $\mathcal{X}$ . For every  $(\alpha ,\beta ,\varepsilon ,\delta)\in (0,1)^4$ , there is an  $(\varepsilon ,\delta)$ -DP algorithm that takes a dataset  $S$  of size  $n$  and with probability  $\geq 1 - \beta$ , outputs bin frequency estimates  $(f_k)_{k\in K}$  such that
$$
\left| f _ {k} - \frac {\left| \{x \in S : x \in h _ {k} \} \right|}{n} \right| \leq \alpha
$$
for all  $k \in K$ , so long as
$$
n \geq \Omega \left(\frac {\log (1 / \beta \delta)}{\alpha \varepsilon}\right).
$$
Proof of Theorem 4.3. Recall the realizably-but-not-robustly learnable class of distributions  $\mathcal{Q}_g = \{q_{i,j,g(j)}:i,j\in \mathbb{N}\}$  over  $\mathbb{N}^2$  from Theorem 3.1, where  $g:\mathbb{N}\to \mathbb{N}$  is a monotone, super-linear function and  $q_{i,j,k}$  is as defined in (1).
Fix  $(\alpha, \beta, \varepsilon, \delta) \in (0,1)^4$ . We define our DP learner  $A_{DP}$  for  $\mathcal{Q}_g$  as follows: we take a sample  $S$  of size
$$
n \geq \Omega \left(\frac {\log (1 / (\beta / 2) \delta)}{(1 / 4 g (1 / \alpha)) \varepsilon}\right) + 3 2 \log (1 / (\beta / 2)) g (1 / \alpha)
$$
and run the stability-based histogram from Proposition F.1 targeting  $(\varepsilon, \delta)$ -DP, with singleton histogram buckets  $(h_{(a,b)})_{(a,b) \in \mathbb{N}^2}$ , each  $h_{(a,b)} = \{(a,b)\}$ . This yields frequency estimates  $(f_{(a,b)})_{(a,b) \in \mathbb{N}^2}$  with  $|f_{(a,b)} - |\{x \in S : x = (a,b)\}|| \leq 1/4g(1/\alpha)$  for all  $(a,b) \in \mathbb{N}^2$  with probability  $\geq 1 - \beta/2$ . Then let
$$
A _ {D P} (S) = \left\{ \begin{array}{l l} q _ {i, j, g (j)} & \text {i f} f _ {(i, 2 j + 2)} \geq 1 / 2 g (1 / \alpha) \\ \delta_ {0} & \text {o t h e r w i s e .} \end{array} \right.
$$
Note that by post-processing  $A_{DP}$  is indeed  $(\varepsilon, \delta)$ -DP. Now suppose  $q_{i,j,g(j)}$  is our unknown distribution. There are two cases:
If  $1 / j \leq \alpha$ , conditioned on the success of the histogram algorithm, the only possible outputs of  $A_{DP}$  are  $\delta_0$  and  $q_{i,j,g(j)}$ , so  $d_{\mathrm{TV}}(A_{DP}(S), q_{i,j,g(j)}) \leq \alpha$  with probability  $\geq 1 - \beta / 2$ .
If  $1 / j > \alpha$ , we have  $1 / g(1 / \alpha) < 1 / g(j)$ . By the second term in  $n$  and Chernoff bounds (Proposition A.2), we can conclude that
$$
\mathbb {P} \left[ \frac {| \{x \in S : x = (i , 2 j + 2) \} |}{n} \leq \frac {3}{4 g (1 / \alpha)} \right] \leq \beta / 2.
$$
If the above event does not occur and if the histogram algorithm does not fail,  $A_{DP}$  outputs  $q_{i,j,g(j)}$  exactly. So  $d_{\mathrm{TV}}(A_{DP}(S),q_{i,j,g(j)}) = 0$  with probability  $\geq 1 - \beta$ .