SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {P} _ {\mathcal {O}, \mathcal {U} ^ {\prime} (\mathcal {O})} \left\{\boldsymbol {X} \in C ^ {d - 1} \right\} = \quad \mathbb {P} _ {\mathcal {U} ^ {\prime} (T)} \quad \left\{\boldsymbol {X} \in C ^ {d - 1} \right\} \\ \begin{array}{l l} \mathcal {Q} \sim \mathcal {U} (\mathbb {Q} _ {k}) & t \sim \mathcal {U} (1) \\ \mathbf {X} \sim \mathcal {Q} ^ {d - 1} & u \sim \mathcal {U} (N) \end{array} \\ \boldsymbol {X} \sim G (1 / k, t, u) ^ {d - 1} \\ = c \int_ {T} \int_ {N} \mathbf {X} \sim G (1 / k, t, u) ^ {d - 1} \left\{\mathbf {X} \in C ^ {d - 1} \right\} d u d t \\ \geq c \int_ {T ^ {\prime}} \int_ {N ^ {\prime}} \mathbf {X} \sim_ {G (1 / k, t, u) ^ {d - 1}} \mathbb {P} \left\{\mathbf {X} \in C ^ {d - 1} \right\} d u d t \\ = c \int_ {T ^ {\prime}} \int_ {N ^ {\prime}} \left(\underset {X \sim G (1 / k, t, u)} {\mathbb {P}} \{X \in C \}\right) ^ {d - 1} d u d t \\ \geq c \int_ {T ^ {\prime}} \int_ {N ^ {\prime}} \left(\underset {X \sim G (1 / k, t, u)} {\mathbb {P}} \{X \in M \}\right) ^ {d - 1} d u d t \\ \geq c \int_ {T ^ {\prime}} \int_ {N ^ {\prime}} \left(\underset {X \sim G (1 / 1 0, t, u)} {\mathbb {P}} \{X \in M \}\right) ^ {d - 1} d u d t \\ =: \eta > 0, \tag {6} \\ \end{array}
$$
where  $c = f_{T}(t) \cdot f_{N}(u) > 0$  is the uniform density over  $T \times N$ . Note that the final integral is non-zero since  $T' \times N'$  has non-zero measure in  $T \times N$  and that  $\mathbb{P}_{X \sim G(1/10, t, u)}\{X \in M\}$  is indeed non-zero for all  $t \in T', u \in N'$ .
Upper bounding  $r_k$ , the weight of  $\alpha$ -TV balls. We prove the following.
Claim G.3. For  $k \geq 1$ , let
$$
r_{k}:= \sup_{p\in \mathcal{Q}_{k}}\underset {Q\sim \mathcal{U}(\mathcal{Q}_{k})}{\mathbb{P}}\big\{\mathrm{TV}(p,Q)\leq \frac{1}{400}\big\} .
$$
$$
r _ {k} = O \left(\frac {1}{k ^ {d}}\right)\rightarrow 0 \quad a s k \rightarrow \infty .
$$
We use the following three facts regarding total variation distance, Gaussians, and the surface area of hyperspherical caps.
Fact G.4 (Data-processing inequality for TV distance). Let  $p, q \in \Delta(\mathcal{X})$ . For any measurable  $f: \mathcal{X} \to \mathcal{Y}$ ,
$$
\operatorname {T V} (f (p), f (q)) \leq \operatorname {T V} (p, q),
$$
where for  $p \in \Delta(\mathcal{X})$ ,  $f(p)$  denotes the push-forward distribution assigning for all measurable  $A \subseteq \mathcal{V}$ ,  $f(p)(A) = p(f^{-1}(A))$ .
Fact G.5 (TV Distance between 1-Dimensional Gaussians [DMR18, Theorem 1.3]). Let  $\mathcal{N}(\mu_1,\sigma_1^2)$  and  $\mathcal{N}(\mu_2,\sigma_2^2)$  be Gaussians over  $\mathbb{R}$ . Then
$$
\frac {1}{2 0 0} \cdot \min  \left\{1, \max  \left\{\frac {\left| \sigma_ {1} ^ {2} - \sigma_ {2} ^ {2} \right|}{\sigma_ {1} ^ {2}}, \frac {4 0 \left| \mu_ {1} - \mu_ {2} \right|}{\sigma_ {1}} \right\} \right\} \leq \operatorname {T V} \left(\mathcal {N} \left(\mu_ {1}, \sigma_ {1} ^ {2}\right), \mathcal {N} \left(\mu_ {2}, \sigma_ {2} ^ {2}\right)\right).
$$
Fact G.6 (Surface area of hyperspherical caps [Li10]). For  $u \in S^{d - 1}$  and  $\theta \in [0, \frac{\pi}{2}]$ , define
$$
C (u, \theta) = \left\{x \in S ^ {d - 1}: \angle (x, u) \leq \theta \right\}
$$
where for  $u, v \in S^{d-1}$ ,  $\angle(u, v) \coloneqq \cos^{-1}(u \cdot v)$ . We have
$$
\operatorname {A r e a} (C (u, \theta)) = \frac {2 \pi^ {(d - 1) / 2}}{\Gamma \left(\frac {d - 1}{2}\right)} \cdot \int_ {0} ^ {\theta} \sin^ {d - 2} (x) d x.
$$
$$
\operatorname {A r e a} (S ^ {d - 1}) = \frac {2 \pi^ {d / 2}}{\Gamma (\frac {d}{2})}.
$$
Proof of Claim G.3. Let  $\sigma > 0$ . Let  $t_1, t_2 \in T$  and  $u_1, u_2, \in N$ . We will compare the total variation distance of the Gaussians defined by these parameters. Let
$$
D _ {\sigma} = \left[ \begin{array}{c c c} \sigma^ {2} & & \\ & 1 & O \\ & O & \ddots \\ & & & 1 \end{array} \right].
$$
By Fact G.4, taking  $f\colon \mathbb{R}^d\to \mathbb{R}$  to be  $f(x) = u_1^\top (x - t_1)$ ,
$$
\begin{array}{l} \operatorname {T V} (G (\sigma , t _ {1}, u _ {2}), G (\sigma , t _ {2}, u _ {2})) \geq \operatorname {T V} \left(\mathcal {N} \left(u _ {1} ^ {\top} \left(t _ {1} - t _ {1}\right), u _ {1} ^ {\top} R _ {u _ {1}} D _ {\sigma} R _ {u _ {1}} ^ {\top} u _ {1}\right), \mathcal {N} \left(u _ {1} ^ {\top} \left(t _ {2} - t _ {1}\right), u _ {1} ^ {\top} R _ {u _ {2}} D _ {\sigma} R _ {u _ {2}} ^ {\top} u _ {1}\right)\right) \\ = \operatorname {T V} \left(\mathcal {N} \left(0, \sigma^ {2}\right), \mathcal {N} \left(u _ {1} \cdot \Delta t, \sigma^ {2} \cos^ {2} (\angle \left(u _ {1}, u _ {2}\right)) + \sin^ {2} (\angle \left(u _ {1}, u _ {2}\right))\right)\right), \\ \end{array}
$$
where  $\Delta t = t_2 - t_1$ . For the last line above, we take  $R_{u_2} = [u_2,v_2,\dots ,v_d]$ , where  $\{v_{2},\ldots ,v_{d}\}$  is an orthonormal basis for  $\{u_2\}^\perp$ . Then the equality in the last line for the variance of the second Gaussian uses,
$$
\begin{array}{l} u _ {1} ^ {\top} R _ {u _ {2}} D _ {\sigma} R _ {u _ {2}} ^ {\top} u _ {1} = \sigma^ {2} \left(u _ {1} \cdot u _ {2}\right) ^ {2} + \left(v _ {2} \cdot u _ {2}\right) ^ {2} + \dots + \left(v _ {d} \cdot u _ {2}\right) ^ {2} \\ = \sigma^ {2} \left(u _ {1} \cdot u _ {2}\right) ^ {2} + \left(1 - \left(u _ {1} \cdot u _ {2}\right) ^ {2}\right) \\ = \sigma^ {2} \cos^ {2} (\angle (u _ {1}, u _ {2})) + (1 - \cos^ {2} (\angle (u _ {1}, u _ {2}))) \\ = \sigma^ {2} \cos^ {2} (\angle (u _ {1}, u _ {2})) + \sin^ {2} (\angle (u _ {1}, u _ {2})), \\ \end{array}
$$
where  $R_{u_2}$  being unitary implies that  $(u_1 \cdot u_2)^2 + (u_1 \cdot v_2)^2 + \dots + (u_2 \cdot v_d)^2 = 1$ , yielding the second equality in the above.
We show that if  $\angle (u_1, u_2) \in [\frac{\sqrt{2}\pi}{2}\sigma, \pi - \frac{\sqrt{2}\pi}{2}\sigma]$ ,  $\mathrm{TV}(G(\sigma, t_1, u_1), G(\sigma, t_2, u_2)) \geq \frac{1}{200}$ . First, we consider the case where  $\angle (u_1, u_2) \in [\frac{\sqrt{2}\pi}{2}\sigma, \frac{\pi}{2}]$ . Using that on  $[0, \frac{\pi}{2}]$ , we have  $\sin(x) \geq \frac{2}{\pi}x$  and  $\cos(x) \geq 0$ , we get
$$
\sigma^ {2} \cos^ {2} (\angle (u _ {1}, u _ {2})) + \sin^ {2} (\angle (u _ {1}, u _ {2})) \geq \frac {4}{\pi^ {2}} \angle (u _ {1}, u _ {2}) ^ {2} \geq 2 \sigma^ {2}. \tag {7}
$$
$$
\frac {\sigma_ {1} ^ {2} - \sigma_ {2} ^ {2}}{\sigma_ {1} ^ {2}} \leq \frac {\sigma^ {2} - 2 \sigma^ {2}}{\sigma^ {2}} \leq - 1,
$$
and by Fact G.5, we can conclude that  $\mathrm{TV}(G(\sigma, t_1, u_1), G(\sigma, t_2, u_2)) \geq \frac{1}{200}$ . Now, consider the case where  $\angle(u_1, u_2) \in [\frac{\pi}{2}, \pi - \frac{\sqrt{2}\pi}{2}]$ . Note that in this case, there exists  $u_2' = -u_2 \in [\frac{\sqrt{2}\pi}{2}, \frac{\pi}{2}]$  with  $G(\sigma, t_2, u_2) = G(\sigma, t_2, u_2')$ , bringing us back to the previous case.
Next, note that since  $\| u_1\| _2 = 1$  and  $|u_{1}^{(d)}| = |u_{1}\cdot e_{d}|\leq \frac{\sqrt{3}}{2}$  (by the definition of  $N$ ), letting  $r = [u_1^{(1)},\ldots ,u_1^{(d - 1)}]^{\top}\in \mathbb{R}^{d - 1}$ , we have  $\| r\| _2\geq \frac{1}{2}$ . Let  $\hat{r} = \frac{r}{\|r\|_2}$ . We have that if  $[\Delta t_1,\dots ,\Delta t_{d - 1}]^\top \cdot \hat{r}\geq \frac{1}{20}\sigma$ , then,
$$
\begin{array}{l} u _ {1} \cdot \Delta t = r \cdot [ \Delta t _ {1}, \dots , \Delta t _ {d - 1} ] ^ {\top} \\ \geq \frac {r}{2 \| r \| _ {2}} \cdot [ \Delta t _ {1}, \dots , \Delta t _ {d - 1} ] ^ {\top} \\ \geq \frac {1}{2} \hat {\boldsymbol {r}} \cdot [ \Delta t _ {1}, \dots , \Delta t _ {d - 1} ] ^ {\top} \\ \geq \frac {1}{4 0} \sigma . \\ \end{array}
$$
$$
\frac {4 0 \left(\mu_ {1} - \mu_ {2}\right)}{\sigma_ {1}} = \frac {4 0 \left(- u _ {1} \cdot \Delta t\right)}{\sigma} \leq - 1,
$$
and by Fact G.5, we can conclude  $\mathrm{TV}(G(\sigma ,t_1,u_1),G(\sigma ,t_2,u_2))\geq \frac{1}{200}$ . Therefore, for any  $u\in N$ ,  $t\in T$
$$
\begin{array}{l} \mathop{\mathbb{P}}_{Q\sim \mathcal{U}(\mathcal{Q}_{k})}\left\{\mathrm{TV}(G(\frac{1}{k},t,u),Q)\leq \frac{1}{400}\right\} = \mathop{\mathbb{P}}_{\substack{t^{\prime}\sim \mathcal{U}(T)\\ u^{\prime}\sim \mathcal{U}(N)}}\left\{\mathrm{TV}(G(\frac{1}{k},t,u),G(\frac{1}{k},t^{\prime},u^{\prime}))\leq \frac{1}{400}\right\} \\ \leq \underset { \begin{array}{c} t ^ {\prime} \sim \mathcal {U} (T) \\ u ^ {\prime} \sim \mathcal {U} (N) \end{array} } {\mathbb {P}} \left\{\mathrm {T V} (G (\frac {1}{k}, t, u), G (\frac {1}{k}, t ^ {\prime}, u ^ {\prime})) <   \frac {1}{2 0 0} \right\} \\ \leq \underset {t ^ {\prime} \sim \mathcal {U} (T)} {\mathbb {P}} \left\{\left[ \Delta t _ {1}, \dots , \Delta t _ {d - 1} \right] ^ {\top} \cdot \hat {r} <   \frac {1}{2 0 k} \right\}. \\ \underset {u ^ {\prime} \sim \mathcal {U} (N)} {\mathbb {P}} \left\{(\angle (u, u ^ {\prime}) \in [ 0, \frac {\sqrt {2} \pi}{2 k}) \cup (\pi - \frac {\sqrt {2} \pi}{2 k}, \pi ]) \right\}. \\ \end{array}
$$
For the first term, note that the event
$$
\left\{\left[ \Delta t _ {1}, \dots , \Delta t _ {d - 1} \right] \cdot \hat {r} <   \frac {1}{2 0 k} \right\} \subseteq \left\{t ^ {\prime} \in \left\{t + \left[ \begin{array}{c} x \\ 0 \end{array} \right] + \lambda \left[ \begin{array}{c} \hat {r} \\ 0 \end{array} \right]: \| x \| _ {2} \leq 1, x \cdot \hat {r} = 0, \lambda \leq \frac {1}{2 0 k} \right\} \right\},
$$
which under  $\mathcal{U}(T)$ , for some  $c_{d} > 0$  depending only on  $d$ , has probability  $\leq c_{d} \cdot \frac{1}{20k}$ .
For the second term, note that  $\angle (u,u^{\prime})\in [0,\frac{\sqrt{2}\pi}{2k})\cup (\pi -\frac{\sqrt{2}\pi}{2k},\pi ]$  means  $u^{\prime}\in C(u,\frac{\sqrt{2}\pi}{2k})\cup$ $C(-u,\frac{\sqrt{2}\pi}{2k})$ . By Fact G.6, we know that under  $\mathcal{U}(N)$ , for some  $c_{d}$  depending only on  $d$
$$
\begin{array}{l} \underset {u ^ {\prime} \sim \mathcal {U} (N)} {\mathbb {P}} \left\{u ^ {\prime} \in C (u, \frac {\sqrt {2} \pi}{2 k}) \right\} = c _ {d} \cdot \int_ {0} ^ {\sqrt {2} \pi / 2 k} \sin^ {d - 2} (x) d x \\ \leq c _ {d} \cdot \int_ {0} ^ {\sqrt {2} \pi / 2 k} x ^ {d - 2} d x \\ = \frac {c _ {d}}{d - 1} \left(\frac {\sqrt {2} \pi}{2}\right) ^ {d - 1} \frac {1}{k ^ {d - 1}}. \\ \end{array}
$$
The bound is the same for  $C(-u, \frac{\sqrt{2}\pi}{2k})$ . Plugging these into the above, we can conclude that
$$
r _ {k} = \sup  _ {p \in \mathcal {Q} _ {k}} \underset {Q \sim \mathcal {U} (\mathcal {Q} _ {k})} {\mathbb {P}} \left\{\mathrm {T V} (p, Q) \leq \frac {1}{4 0 0} \right\} \leq O \left(\frac {1}{k ^ {d}}\right)\rightarrow 0 \quad \text {a s} k \rightarrow \infty .
$$
Lower bounding  $s_k$ , the weight of alternative hypotheses. First, we note that
$$
u_{k} = \sup_{\substack{\boldsymbol {x}\in B\\ q\in \mathcal{Q}_{k}}}q^{d - 1}(\boldsymbol {x}) = \left(\frac{1}{(2\pi)^{d / 2}} k\exp (-\frac{1}{2})\right)^{d - 1},
$$
which is achieved by  $G\left(\frac{1}{k}, \mathbf{0}, e_1\right)$  (where  $\mathbf{0} \in \mathbb{R}^d$  is the origin) and  $\boldsymbol{x} = (e_d, \dots, e_d)$ . Let
$$
c = \frac {\exp (- 5) ^ {d - 1}}{\exp (- \frac {1}{2}) ^ {d - 1}} = \exp \left(\frac {9 (d - 1)}{2}\right).
$$
Claim G.7. For  $k \geq 1$ , letting  $(u_k)_{k=1}^{\infty}$  and  $c$  be defined as above, define
$$
s_{k}:= \inf_{\boldsymbol {x}\in B}\mathbb{P}_{Q\sim \mathcal{U}(\mathcal{Q}_{k})}\big\{Q^{d - 1}(\boldsymbol {x})\geq cu_{k}\big\} .
$$
$$
s _ {k} = \Omega \left(\frac {1}{k ^ {d - 1}}\right)\rightarrow 0 \quad a s k \rightarrow \infty .
$$
Proof of Claim G.7. Let  $k \geq 1$ . Fix any  $\pmb{x} = (x_{1},\dots,x_{d - 1}) \in B$ . For every  $t \in T$ , there exists  $u \in \{x_1 - t,x_2 - t,\ldots ,x_{d - 1} - t\}^{\perp}$ . We show  $\angle (u,e_d) \geq \frac{\pi}{4}$ . Suppose otherwise, that is,  $\angle (u,e_d) < \frac{\pi}{4} \Rightarrow |u\cdot e_d| = |u^{(d)}| > \frac{\sqrt{2}}{2}$ . Then,
$$
u \cdot (x _ {1} - t) = u ^ {(1)} (x _ {1} ^ {(1)} - t ^ {(1)}) + \dots + u ^ {(d - 1)} (x _ {1} ^ {(d - 1)} - t ^ {(d - 1)}) + u ^ {(d)} x _ {1} ^ {(d)}.
$$
By our assumption on  $u^{(d)}$ , and by the fact that  $x_{1} \in C$ , we have that  $|u^{(d)}x_1^{(d)}| > \frac{\sqrt{2}}{2}$ . By Cauchy-Schwarz in  $\mathbb{R}^{d - 1}$ , we have that,
$$
\begin{array}{l} \left| u ^ {(1)} \left(x _ {1} ^ {(1)} - t ^ {(1)}\right) + \dots + u ^ {(d - 1)} \left(x _ {1} ^ {(d - 1)} - t ^ {(d - 1)}\right) \right| \\ \leq \| \left[ u ^ {(1)}, \dots , u ^ {(d - 1)} \right] ^ {\top} \| _ {2} \cdot \| \left[ x _ {1} ^ {(1)}, \dots , x _ {1} ^ {(d - 1)} \right] ^ {\top} - \left[ t ^ {(1), \dots , t ^ {(d - 1)}} \right] ^ {\top} \| _ {2} \\ <   \frac {\sqrt {2}}{2} \cdot 1. \\ \end{array}
$$
The last inequality uses that  $\| u \|_2 = 1$  and  $(u^{(d)})^2 > \frac{1}{2}$ , so the norm of the first  $(d - 1)$  coordinates is  $< \frac{\sqrt{2}}{2}$ , and also the fact that the first  $(d - 1)$  coordinates of  $x$  and  $t$  are in the  $\frac{1}{2}$ -disk. This inequality, combined with the fact that  $|u^{(d)}x_1^{(d)}| > \frac{\sqrt{2}}{2}$  contradicts that that  $(x_1 - t) \cdot u = 0$ .
Now, for the  $t$  and the  $u$  from above, consider an arbitrary  $u'$  with  $\angle(u, u') \leq \frac{1}{k}$ , and the Gaussian with mean  $t$  and normal vector  $u'$ ,  $G\left(\frac{1}{k}, t, u'\right)$ . We will show that any such Gaussian assigns high mass to the point  $x$ , and furthermore that there is a high density of such Gaussians. Note that for  $k \geq 5$ ,  $\frac{1}{k} \leq \frac{\pi}{3} - \frac{\pi}{4} \Rightarrow \angle(u', e_d) \geq \frac{\pi}{3} \Rightarrow u' \in N$ . We compute the minimum density this Gaussian assigns to  $x$ . Consider, for  $i \in [d-1]$ ,
$$
\begin{array}{l} \left(x _ {i} - t\right) ^ {\top} \left(R _ {u ^ {\prime}} D _ {1 / k} R _ {u ^ {\prime}} ^ {\top}\right) ^ {- 1} \left(x _ {i} - t\right) = \left\| D _ {\sqrt {k}} R _ {u ^ {\prime}} ^ {\top} \left(x _ {i} - t\right) \right\| ^ {2} \\ = k ^ {2} \left| u ^ {\prime} \cdot \left(x _ {i} - t\right) \right| ^ {2} + \left| v _ {2} \cdot \left(x _ {i} - t\right) \right| ^ {2} + \dots + \left| v _ {d} \cdot \left(x _ {i} - t\right) \right| ^ {2} \\ \leq 5 \left(k ^ {2} \left| u ^ {\prime} \cdot \hat {r} \right| ^ {2} + 1\right), \\ \end{array}
$$
where  $\hat{r} = (x_i - t) / \| x_i - t\|$  and  $\{v_{2},\ldots ,v_{d}\}$  is an orthonormal basis of  $\{u^{\prime}\}^{\perp}$ . We have that,
$$
\begin{array}{l} \left| u ^ {\prime} \cdot \hat {r} \right| ^ {2} = \left| \left(u + \left(u - u ^ {\prime}\right)\right) \cdot \hat {r} \right| ^ {2} \\ \leq \| u - u ^ {\prime} \| _ {2} ^ {2} \cdot \| \hat {r} \| _ {2} ^ {2} \\ = u \cdot u - 2 u \cdot u ^ {\prime} + u ^ {\prime} \cdot u ^ {\prime} \\ = 2 - 2 \cos (\angle (u ^ {\prime}, u)) \\ \leq 2 - 2 \left(1 - \frac {\angle \left(u ^ {\prime} , u\right) ^ {2}}{2}\right) \\ = \angle (u ^ {\prime}, u) ^ {2} \leq \frac {1}{k ^ {2}}. \\ \end{array}
$$
Hence, the density of  $G\left( {\frac{1}{k},t,{u}^{\prime }}\right)$  on  $x$  is lower bounded by,
$$
\left(\frac {1}{(2 \pi) ^ {d / 2}} k \exp (- 5)\right) ^ {d - 1} = c u _ {k}.
$$
For every  $t \in T$ , we found a set of  $u' \in N$  such that the density  $G\left(\frac{1}{k}, t, u'\right)$  assigns to  $x$  is greater than  $cu_k$ . Since for some constant  $c_d > 0$  depending only on  $d$ ,
$$
\begin{array}{l} \mathop{\mathbb{P}}_{u^{\prime}\sim \mathcal{U}(N)}\left\{u^{\prime}\in C(u,\frac{1}{k})\right\} = c_{d}\int_{0}^{1 / k}\sin^{d - 2}(x)dx \\ \geq c _ {d} \int_ {0} ^ {1 / k} \left(\frac {2}{\pi} x\right) ^ {d - 2} d x \\ = c _ {d} \left(\frac {2}{\pi}\right) ^ {d - 2} \frac {1}{d - 1} \cdot \frac {1}{k ^ {d - 1}}, \\ \end{array}
$$
and since  $\pmb{x} \in B$  was arbitrary, we indeed have,
$$
s_{k} = \inf_{\boldsymbol {x}\in B}\mathbb{P}_{Q\sim \mathcal{U}(\mathcal{Q}_{k})}\left\{Q^{d - 1}(\boldsymbol {x})\geq cu_{k}\right\} = \Omega \left(\frac{1}{k^{d - 1}}\right).
$$
With the three claims, applying Lemma G.1 allows us to conclude that the class of all Gaussians in  $\mathbb{R}^d$  is not list learnable with  $m(\alpha, \beta) = d - 1$  samples. This implies that the class is also not public-privately learnable with  $m(\alpha, \beta, \varepsilon) = d - 1$  public samples.
H Statements and proofs for Section 7 – Learning when the Yatracos class has finite VC dimension
When the VC dimension of the Yatracos class of  $\mathcal{Q}$  is finite, the following gives an upper bound on the number of samples required to non-privately learn  $\mathcal{Q}$ .
Fact H.1 ([Yat85], [DL01, Theorem 6.4]). Let  $\mathcal{Q} \subseteq \Delta(\mathcal{X})$ . Let  $\mathcal{H}$  be the Yatracos class of  $\mathcal{Q}$ , and let  $d = \operatorname{VC}(\mathcal{H})$ .  $\mathcal{Q}$  is learnable with
$$
m = O \left(\frac {d + \log \left(\frac {1}{\beta}\right)}{\alpha^ {2}}\right)
$$
For some classes of distributions, the above bound is tight. For example, it recovers the  $\Theta\left(\frac{d^2}{\alpha^2}\right)$  sample complexity for learning Gaussians in  $\mathbb{R}^d$  [AM18].
To prove Theorem 7.2 we employ the following result on generating distribution-dependent covers for binary hypothesis classes with public data.
Fact H.2 (Public data cover [ABM19, Lemma 3.3 restated]). Let  $\mathcal{H} \subseteq 2^{\mathcal{X}}$  and  $\mathrm{VC}(\mathcal{H}) = d$ . There exists  $\mathcal{A} \colon \mathcal{X}^{*} \to \{H \subseteq 2^{\mathcal{X}} : |H| < \infty\}$ , such that for any  $\alpha, \beta \in (0,1]$  there exists
$$
m = O \left(\frac {d \log \left(\frac {1}{\alpha}\right) + \log \left(\frac {1}{\beta}\right)}{\alpha}\right)
$$
such that for any  $p \in \Delta(\mathcal{X})$ , if we draw  $\mathbf{X} = (X_1, \ldots, X_m)$  i.i.d. from  $p$ , with probability  $\geq 1 - \beta$ ,  $\mathcal{A}(\mathbf{X})$  outputs  $\widehat{\mathcal{H}} \subseteq 2^{\mathcal{X}}$  and a mapping  $f \colon \mathcal{H} \to \widehat{\mathcal{H}}$  with
$$
p (h \triangle f (h)) \leq \alpha \quad \text {f o r a l l} h \in \mathcal {H}
$$
(where for  $A, B \subseteq \mathcal{X}$ ,  $A \triangle B$  denotes the symmetric set difference  $(A \setminus B) \cup (B \setminus A)$ ). Furthermore, we have  $|\widehat{\mathcal{H}}| \leq \left(\frac{em}{d}\right)^{2d}$ .
We also use the following pure DP algorithm for answering counting queries on finite domains.
Fact H.3 (SmallDB [BLR13], [DR14, Theorem 4.5]). Let  $\mathcal{X}$  be a finite domain. Let  $\mathcal{H} \subseteq 2^{\mathcal{X}}$ . Let  $\alpha, \beta \in (0,1]$  and  $\varepsilon > 0$ . There is an  $\varepsilon$ -DP randomized algorithm, that on any dataset  $x = (x_1, \ldots, x_n)$  with
$$
n = \Omega \left(\frac {\log (| \mathcal {X} |) \log (| \mathcal {H} |) + \log (\frac {1}{\beta})}{\varepsilon \alpha^ {3}}\right)
$$
outputs estimates  $\hat{g}\colon \mathcal{H}\to \mathbb{R}$  such that with probability  $\geq 1 - \beta$
$$
\left| \hat {g} (h) - \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {1} _ {h} \left(x _ {i}\right) \right| \leq \alpha \quad f o r a l l h \in \mathcal {H}.
$$
Proof of Theorem 7.2. We use our  $m$  public samples from the unknown  $p \in \mathcal{Q}$  to generate a public data cover  $\widehat{\mathcal{H}}$  and mapping  $f \colon \mathcal{H} \to \widehat{\mathcal{H}}$  courtesy of Fact H.2, selecting  $m$  to target error  $\frac{\alpha}{6}$  and failure probability  $\frac{\beta}{3}$ . Note that this implies that with probability  $\geq 1 - \frac{\beta}{3}$ , for every  $h \in \mathcal{H}$ ,  $|p(h) - p(f(h))| \leq |p(h \triangle f(h))| \leq \frac{\alpha}{6}$ .
Next, we consider the representative domain of  $\mathcal{X}$  with respect to  $\widehat{\mathcal{H}}$ , denoted by  $\mathcal{X}_{\widehat{\mathcal{H}}}$ . In other words, for every unique behaviour  $(\mathbb{1}_{\hat{h}}(x))_{\hat{h} \in \widehat{\mathcal{H}}} \in \{0, 1\}^{\lceil \widehat{\mathcal{H}} \rceil}$  induced by a point  $x \in \mathcal{X}$  on  $\widehat{\mathcal{H}}$ , we include exactly one representative  $[x]$  in  $\mathcal{X}_{\widehat{\mathcal{H}}}$ . By Sauer's lemma we can conclude that
$$
| \mathcal {X} _ {\widehat {\mathcal {H}}} | \leq \left(\frac {e | \widehat {\mathcal {H}} |}{d ^ {*}}\right) ^ {d ^ {*}}.
$$
Then, we take our  $n$  private samples  $\mathbf{X} = (X_{1},\ldots ,X_{n})$  and map each point  $X_{i}$  to its representative  $[X_i]\in \mathcal{X}_{\widehat{\mathcal{H}}}$ , yielding a dataset of  $n$  examples  $[X]$  on the finite domain  $\mathcal{X}_{\widehat{\mathcal{H}}}$ . Note that for any  $\hat{h}\in \widehat{\mathcal{H}}$ ,  $\frac{1}{n}\sum_{i = 1}^{n}\mathbb{1}_{\hat{h}}(X_i) = \frac{1}{n}\sum_{i = 1}^{n}\mathbb{1}_{\hat{h}}([X_i])$ . Hence when we run SmallDB (Fact H.3) on the input  $[X]$  over the finite domain  $\mathcal{X}_{\widehat{\mathcal{H}}}$  with finite class  $\widehat{\mathcal{H}}$ , choosing  $n$  large enough, we obtain  $\hat{g}:\widehat{\mathcal{H}}\to \mathbb{R}$  such that with probability  $\geq 1 - \frac{\beta}{3}$ ,  $|\hat{g} (\hat{h}) - \frac{1}{n}\sum_{i = 1}^{n}\mathbb{1}_{\hat{h}}(X_i)|\leq \frac{\alpha}{6}$  for all  $\hat{h}\in \widehat{\mathcal{H}}$ .
We also ensure  $n$  is large enough so that we get the uniform convergence property on  $\widehat{\mathcal{H}}$ , which has VC dimension  $d$ , with the private samples. That is, for all  $\hat{h} \in \widehat{\mathcal{H}}$ , with probability  $\geq 1 - \frac{\beta}{3}$ ,  $|p(\hat{h}) - \frac{1}{n}\sum_{i=1}^{n}\mathbb{1}_{\hat{h}}(X_i)| \leq \frac{\alpha}{6}$ .
As a post-processing of  $\hat{g}$ , our learner outputs
$$
\hat{q} := \operatorname *{arg  min}_{q\in \mathcal{Q}}\sup_{h\in \mathcal{H}}|q(h) - \hat{g} (f(h))|.
$$
By the union bound, with probability  $\geq 1 - \beta$ , all of our good events occur. In this case, we have for all  $h \in \mathcal{H}$
$$
| p (h) - p (f (h)) | \leq \frac {\alpha}{6}
$$
$$
\left| p (f (h)) - \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {1} _ {f (h)} \left(X _ {i}\right) \right| \leq \frac {\alpha}{6}
$$
$$
\left| \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {1} _ {f (h)} \left(X _ {i}\right) - \hat {g} (f (h)) \right| \leq \frac {\alpha}{6}
$$
which implies  $|p(h) - \hat{g}(f(h))| \leq \frac{\alpha}{2}$ . So for any  $q \in \mathcal{Q}$ ,
$$
\begin{array}{l} \left| q (h) - p (h) \right| - \frac {\alpha}{2} \leq \left| q (h) - \hat {g} (f (h)) \right| \leq \left| q (h) - p (h) \right| + \frac {\alpha}{2} \\ \Rightarrow \operatorname {T V} (q, p) - \frac {\alpha}{2} \leq \sup  _ {h \in \mathcal {H}} | q (h) - \hat {g} (f (h)) | \leq \operatorname {T V} (q, p) + \frac {\alpha}{2}. \\ \end{array}
$$
$$
\sup  _ {h \in \mathcal {H}} | \hat {q} (h) - \hat {g} (f (h) | \leq \sup  _ {h \in \mathcal {H}} | p (h) - \hat {g} (f (h) | \leq \operatorname {T V} (p, p) + \frac {\alpha}{2} \leq \frac {\alpha}{2}.
$$
$$
\operatorname {T V} (\hat {q}, p) \leq \sup  _ {h \in \mathcal {H}} | \hat {q} (h) - \hat {g} (f (h) | + \frac {\alpha}{2} \leq \alpha .
$$
It can be verified that the choices of  $m$  and  $n$  in the statement of Theorem 7.2 suffice.