SlowGuess/NeurIPS_2023_OCR huggingface dataset:
It is also worth noting that the scoring function can be written in the form:
$$
\operatorname {s c r} _ {h} (\boldsymbol {z}) := \sum_ {i \in [ n ]} \operatorname {c l i p} _ {0, 1} \left(\left| \left\{\left(x, y\right) \in \boldsymbol {z} _ {i} \mid h (x) \neq y \right\} \right|\right),
$$
which is similar to what we present below for other tasks.
Finally, we remark that the standard techniques can boost the success probability from  $2/3$  to an arbitrary probability  $1 - \beta$ , with a multiplicative factor of  $\log(1/\beta)$  cost.
E Additional Results for Pure-DP
E.1 Algorithm: Pairwise Score Exponential Mechanism
In this section, we give a user-level DP version of pairwise score exponential mechanism (EM) [BKSW19]. In the preceding subsections, we will show how to apply this in multiple settings, including agnostic learning, private hypothesis selection, and distribution learning.
Suppose there is a candidate set  $\mathcal{H}$ . Furthermore, for every pair  $H, H' \in \mathcal{H}$  of candidates, there is a "comparison" function  $\psi_{H, H'}: \mathcal{Z} \to [-1, 1]$ . For every pair  $H, H'$  of candidates and any  $x \in \mathcal{Z}^*$ , let
$$
\operatorname {p s c r} _ {H, H ^ {\prime}} ^ {\psi} (\boldsymbol {x}) = \sum_ {x \in \boldsymbol {x}} \psi_ {H, H ^ {\prime}} (x).
$$
We then define  $\mathrm{scr}^{\psi}$  for each candidate  $H$  as
$$
\operatorname {s c r} _ {H} ^ {\psi} (\boldsymbol {x}) := \max  _ {H ^ {\prime} \in \mathcal {H} \setminus \{H \}} \operatorname {p s c r} _ {H, H ^ {\prime}} ^ {\psi} (\boldsymbol {x}).
$$
Traditionally, the item-level DP algorithm for pairwise scoring function is to use EM on the above scoring function  $\operatorname{scr}_H^{\psi}$  (where  $\pmb{x}$  denote the entire input) [BKSW19]; it is not hard to see that the  $\operatorname{scr}_H^{\psi}$  has (item-level) sensitivity of at most 2.
User-Level DP Algorithm. We now describe our user-level DP algorithm ClippedPairwiseEM. The algorithm computes:
$$
\operatorname {p s c r} _ {H, H ^ {\prime}} ^ {\psi , \tau} (\boldsymbol {x}) := \sum_ {i \in [ n ]} \operatorname {c l i p} _ {- \tau , \tau} \left(\operatorname {p s c r} _ {H, H ^ {\prime}} ^ {\psi} (\boldsymbol {x} _ {i})\right),
$$
$$
\operatorname {s c r} _ {H} ^ {\psi , \tau} (\boldsymbol {x}) := \max  _ {H ^ {\prime} \in \mathcal {H} \setminus \{H \}} \operatorname {p s c r} _ {H, H ^ {\prime}} ^ {\psi , \tau} (\boldsymbol {x}).
$$
Then, it simply runs EM using the score  $\mathrm{scr}_H^{\psi ,\tau}$  with sensitivity  $\Delta = 2\tau$
To state the guarantee of the algorithm, we need several additional distributional-based definitions of the score:
$$
\operatorname {p s c r} _ {H, H ^ {\prime}} ^ {\psi} (\mathcal {D}) := \mathbb {E} _ {x \sim \mathcal {D}} \left[ \psi_ {H, H ^ {\prime}} (x) \right],
$$
$$
\operatorname {s c r} _ {H} ^ {\psi} (\mathcal {D}) := \max  _ {H ^ {\prime} \in \mathcal {H} \setminus \{H \}} \operatorname {p s c r} _ {H, H ^ {\prime}} ^ {\psi} (\mathcal {D}).
$$
Note that these distributional scores are normalized, so they are in  $[-1, 1]$ .
The guarantee of our algorithm can now be stated as follows:
Theorem 37. Let  $\varepsilon, \alpha, \beta \in (0,1/2]$ . Assume further that there exists  $H^{*}$  such that  $\mathrm{scr}_{H^{*}}^{\psi}(\mathcal{D}) \leq 0.1\alpha$ . There exists an  $\varepsilon$ -DP algorithm that with probability  $1 - \beta$  (over the randomness of the algorithm and the input  $\pmb{x} \sim (\mathcal{D}^{m})^{n}$ ) outputs  $H \in \mathcal{H}$  such that  $\mathrm{scr}_{H}^{\psi}(\mathcal{D}) \leq 0.5\alpha$  as long as
$$
n \geq \tilde {\Theta} \left(\frac {\log (| \mathcal {H} | / \beta)}{\varepsilon} + \frac {\log (| \mathcal {H} | / \beta)}{\alpha \varepsilon \sqrt {m}} + \frac {\log (| \mathcal {H} | / \beta)}{\alpha^ {2} m}\right).
$$
Proof. We use ClippedPairwiseEM with parameter  $\tau = C\left(\alpha m + \sqrt{m\log(1 / \alpha)}\right)$  and assume that  $n \geq C'\left(\frac{\tau\cdot\log(|\mathcal{H}| / \beta)}{\varepsilon\alpha m} + \frac{\log(|\mathcal{H}| / \beta)}{\alpha^2m}\right) = \tilde{\Theta}\left(\frac{\log(|\mathcal{H}| / \beta)}{\varepsilon} + \frac{\log(|\mathcal{H}| / \beta)}{\alpha\varepsilon\sqrt{m}} + \frac{\log(|\mathcal{H}| / \beta)}{\alpha^2m}\right)$ , where  $C, C'$  are sufficiently large constants. The privacy guarantee follows directly from that of EM, since the (user-level) sensitivity is at most  $\Delta$  (due to clipping).
To analyze the utility, we will consider following three events:
$\mathcal{E}_1$  ..  $\operatorname {scr}_H^{\psi ,\tau}(\pmb {x})\leq \operatorname {scr}_{H^*}^{\psi ,\tau}(\pmb {x}) + 0.1\alpha nm.$
-  $\mathcal{E}_2$ : for all  $H, H' \in \mathcal{H}$ : if  $\operatorname{pscr}_{H,H'}^\psi(\mathcal{D}) \leq 0.1\alpha$ , then  $\operatorname{pscr}_{H,H'}^\psi(\boldsymbol{x}) \leq 0.2\alpha nm$ .
-  $\mathcal{E}_3$ : for all  $H, H' \in \mathcal{H}$ : if  $\operatorname{pscr}_{H,H'}^{\psi}(\mathcal{D}) \geq 0.5\alpha$ , then  $\operatorname{pscr}_{H,H'}^{\psi}(\boldsymbol{x}) \geq 0.4\alpha nm$ .
When all events hold,  $\mathcal{E}_1$  implies that  $\operatorname{scr}_{H}^{\psi, \tau}(\pmb{x}) \leq \operatorname{scr}_{H^*}^{\psi, \tau}(\pmb{x}) + 0.1\alpha nm$ . Observe that  $\mathcal{E}_2$  implies that  $\operatorname{scr}_{H^*}^{\psi, \tau}(\pmb{x}) \leq 0.2\alpha nm$ . Combining these two inequalities, we get  $\operatorname{scr}_{H}^{\psi, \tau}(\pmb{x}) \leq 0.3\alpha nm$ ; then,  $\mathcal{E}_3$  implies that  $\operatorname{pscr}_{H, H'}^{\psi}(\mathcal{D}) \leq 0.5\alpha$  for all  $H' \in \mathcal{D}$ . In turn, this means that  $\operatorname{scr}_{H}^{\psi}(\mathcal{D}) \leq 0.5\alpha$ .
Therefore, it suffices to show that each of  $\operatorname{Pr}[\mathcal{E}_1], \operatorname{Pr}[\mathcal{E}_2], \operatorname{Pr}[\mathcal{E}_3]$  is at least  $1 - \beta / 3$ .
Bounding  $\operatorname*{Pr}[\mathcal{E}_1]$ . To bound the probability of  $\mathcal{E}_1$ , recall from Theorem 9 that, with probability  $1 - \beta /3$ , the guarantee of EM ensures that
$$
\operatorname {s c r} _ {H} ^ {\psi , \tau} (\boldsymbol {x}) \geq \operatorname {s c r} _ {H *} ^ {\psi , \tau} (\boldsymbol {x}) + O \left(\frac {\log (| \mathcal {H} | / \beta)}{\varepsilon}\right) \cdot \Delta .
$$
From our assumption on  $n$ , when  $C'$  is sufficiently large, then we have  $O\left(\frac{\log(|\mathcal{H}| / \beta)}{\varepsilon}\right) \cdot \Delta = O\left(\frac{\log(|\mathcal{H}| / \beta)}{\varepsilon} \cdot \tau\right) \leq 0.1\alpha nm$  as desired.
Bounding  $\operatorname*{Pr}[\mathcal{E}_2]$  and  $\mathrm{Pr}[\mathcal{E}_3]$ . Since the proofs for both cases are analogous, we only show the full argument for  $\operatorname*{Pr}[\mathcal{E}_3]$ . Let us fix  $H, H' \in \mathcal{H}$  such that  $\mathrm{pscr}_{H,H'}^{\psi}(\mathcal{D}) \geq 0.5\alpha$ . We may assume w.l.o.g. that
Notice that  $\operatorname{pscr}_{H,H'}^{\psi,\tau}(\boldsymbol{x})$  is a 2-bounded difference function. As a result, McDiarmid's inequality (Lemma 26) implies that
$$
\Pr_ {\boldsymbol {x} \sim \mathcal {D} ^ {n m}} \left[ \left| \operatorname {p s c r} _ {H, H ^ {\prime}} ^ {\psi , \tau} (\boldsymbol {x}) - \mu_ {\mathcal {D} ^ {n m}} \left(\operatorname {p s c r} _ {H, H ^ {\prime}} ^ {\psi , \tau}\right) \right| > 0. 0 5 \alpha n m \right] \leq 2 \exp \left(- 0. 0 2 5 \alpha^ {2} n m\right) \leq \frac {\beta}{3 | \mathcal {H} | ^ {2}}, \tag {10}
$$
where the second inequality is due to our choice of  $n$  when  $C'$  is sufficiently large.
To compute  $\mu_{\mathcal{D}^{nm}}(\mathrm{pscr}_{H,H^{\prime}}^{\psi ,\tau})$  , observe further that
$$
\mu_ {\mathcal {D} ^ {n m}} \left(\operatorname {p s c r} _ {H, H ^ {\prime}} ^ {\psi , \tau}\right) = n \cdot \mu_ {\mathcal {D} ^ {m}} \left(\operatorname {c l i p} _ {- \tau , \tau} \circ \operatorname {p s c r} _ {H, H ^ {\prime}} ^ {\psi}\right). \tag {11}
$$
Now observe once again that  $\mathrm{pscr}_{H,H'}^{\psi}:\mathcal{Z}^m\to \mathbb{R}$  is also a 2-bounded difference function and  $\mu_{\mathcal{D}^m}(\mathrm{pscr}_{H,H'}^{\psi}) = \mu m$ . Therefore, McDiarmid's inequality (Lemma 26) yields
$$
\Pr_ {\boldsymbol {x} \sim \mathcal {D} ^ {m}} \left[ \operatorname {p s c r} _ {H, H ^ {\prime}} ^ {\psi} (\boldsymbol {x}) > \tau \right] \leq 2 \exp \left(- 0. 5 \left(\tau - \mu m\right) ^ {2} / m\right) \leq 1 0 ^ {- 4} \alpha^ {2}, \tag {12}
$$
where the second inequality is due to our choice of  $\tau$  when  $C$  is sufficiently large.
$$
\begin{array}{l} \mu_ {\mathcal {D} ^ {m}} \left(\operatorname {c l i p} _ {- \tau , \tau} \circ \operatorname {p s c r} _ {H, H ^ {\prime}} ^ {\psi}\right) \\ = \mathbb {E} _ {\boldsymbol {x} \sim \mathcal {D} ^ {m}} [ \operatorname {c l i p} _ {- \tau , \tau} (\operatorname {p s c r} _ {H, H ^ {\prime}} ^ {\psi} (\boldsymbol {x})) ] \\ \geq \mathbb {E} _ {\boldsymbol {x} \sim \mathcal {D} ^ {m}} \left[ \operatorname {c l i p} _ {- \infty , \tau} \left(\operatorname {p s c r} _ {H, H ^ {\prime}} ^ {\psi} (\boldsymbol {x})\right) \right] \\ = \mathbb {E} _ {\boldsymbol {x} \sim \mathcal {D} ^ {m}} \left[ \operatorname {p s c r} _ {H, H ^ {\prime}} ^ {\psi} (\boldsymbol {x}) \right] + \mathbb {E} _ {\boldsymbol {x} \sim \mathcal {D} ^ {m}} \left[ \left(\tau - \operatorname {p s c r} _ {H, H ^ {\prime}} ^ {\psi} (\boldsymbol {x})\right) \cdot \mathbf {1} \left[ \operatorname {p s c r} _ {H, H ^ {\prime}} ^ {\psi} (\boldsymbol {x}) > \tau \right] \right] \\ \geq \mu m - \mathbb {E} _ {\boldsymbol {x} \sim \mathcal {D} ^ {m}} \left[ \operatorname {p s c r} _ {H, H ^ {\prime}} ^ {\psi} (\boldsymbol {x}) \cdot \mathbf {1} \left[ \operatorname {p s c r} _ {H, H ^ {\prime}} ^ {\psi} (\boldsymbol {x}) > \tau \right] \right] \\ \geq \mu m - \sqrt {\mathbb {E} _ {\boldsymbol {x} \sim \mathcal {D} ^ {m}} [ \operatorname {p s c r} _ {H , H ^ {\prime}} ^ {\psi} (\boldsymbol {x}) ^ {2} ]} \cdot \sqrt {\mathbb {E} _ {\boldsymbol {x} \sim \mathcal {D} ^ {m}} [ \mathbf {1} [ \operatorname {p s c r} _ {H , H ^ {\prime}} ^ {\psi} (\boldsymbol {x}) > \tau ] ]} \\ \geq \mu m - \sqrt {m ^ {2}} \cdot \sqrt {\operatorname* {P r} _ {\boldsymbol {x} \sim \mathcal {D} ^ {m}} \left[ \operatorname {p s c r} _ {H , H ^ {\prime}} ^ {\psi} (\boldsymbol {x}) > \tau \right]} \\ \geq \mu m - 0. 0 1 \alpha m = 0. 4 9 \alpha m, \tag {13} \\ \end{array}
$$
where the third-to-last inequality follows from Cauchy-Schwarz.
<sup>12</sup>Otherwise, we may keep increasing  $\psi_{H,H'}(z)$  for different values of  $z$  until  $\mathrm{pscr}_{H,H'}^{\psi}(\mathcal{D}) = 0.5\alpha$ ; this operation does not decrease the probability that  $\mathrm{pscr}_{H,H'}^{\psi}(\boldsymbol{x}) < 0.4\alpha nm$ .
Combining (10), (11), and (13), we can conclude that
$$
\operatorname * {P r} _ {\boldsymbol {x} \sim \mathcal {D} ^ {n m}} [ \operatorname {p s c r} _ {H, H ^ {\prime}} ^ {\psi , \tau} (\boldsymbol {x}) <   0. 4 \alpha n m ] \leq \frac {\beta}{3 | \mathcal {H} | ^ {2}}.
$$
Taking a union bound over  $H$ ,  $H' \in \mathcal{H}$  yields  $\operatorname*{Pr}[\mathcal{E}_3] \geq 1 - \beta / 3$  as desired.
E.2 Lower Bound: User-level DP Fano's Inequality
As we often demonstrate below that our bounds are (nearly) tight, it will be useful to have a generic method for providing such a lower bound. Here we observe that it is simple to extend the DP Fano's inequality of Acharya et al. [ASZ21] to the user-level DP setting. We start by recalling Acharya et al.'s (item-level) DP Fano's inequality<sup>13</sup>:
Theorem 38 (Item-Level DP Fano's Inequality [ASZ21]). Let  $\mathfrak{T}$  be any task. Suppose that there exist  $\mathcal{D}_1, \ldots, \mathcal{D}_W$  such that, for all distinct  $i, j \in [W]$ ,
-  $\Psi_{\mathfrak{T}}(\mathcal{D}_i)\cap \Psi_{\mathfrak{T}}(\mathcal{D}_j) = \emptyset$
$d_{\mathrm{KL}}(\mathcal{D}_i,\mathcal{D}_j)\leq \tilde{\beta},$  and
-  $d_{\mathrm{tv}}(\mathcal{D}_i,\mathcal{D}_j)\leq \gamma$
Then,  $n_1^{\mathfrak{T}}(\varepsilon ,\delta = 0)\geq \Omega \left(\frac{\log W}{\gamma\varepsilon} +\frac{\log W}{\beta}\right).$
The bound for the user-level case can be derived as follows.
Lemma 39 (User-Level DP Fano's Inequality). Let  $\mathfrak{T}$  be any task. Suppose that there exist  $\mathcal{D}_1, \ldots, \mathcal{D}_W$  such that, for all distinct  $i, j \in [W]$ ,
-  $\Psi_{\mathfrak{T}}(\mathcal{D}_i)\cap \Psi_{\mathfrak{T}}(\mathcal{D}_j) = \emptyset ,and,$
$d_{\mathrm{KL}}(\mathcal{D}_i,\mathcal{D}_j)\leq \beta .$
Then,  $n_m^{\mathfrak{T}}(\varepsilon ,\delta = 0)\geq \Omega \left(\frac{\log W}{\varepsilon} +\frac{\log W}{\varepsilon\sqrt{m\beta}} +\frac{\log W}{m\beta}\right).$
Proof. Let  $\mathcal{D}_i' = (\mathcal{D}_i)^m$  for all  $i \in [W]$ ; note that  $d_{\mathrm{KL}}(\mathcal{D}_i' \parallel \mathcal{D}_j') \leq m\beta$ . By Pinsker's inequality (Lemma 23(i)),  $d_{\mathrm{tv}}(\mathcal{D}_i' \parallel \mathcal{D}_j') \leq O(\sqrt{m\beta})$ ; furthermore, we also have the trivial bound  $d_{\mathrm{tv}}(\mathcal{D}_i' \parallel \mathcal{D}_j') \leq 1$ . Finally, define a task  $\mathfrak{T}'$  by  $\Psi_{\mathfrak{T}'}(\mathcal{D}_i') := \Psi_{\mathfrak{T}}(\mathcal{D}_i)$  for all  $i \in [W]$ . Now, applying Theorem 38,
$$
n _ {1} ^ {\mathfrak {T} ^ {\prime}} (\varepsilon , \delta = 0) \geq \Omega \left(\frac {\log W}{\min \{1 , \sqrt {m \beta} \} \cdot \varepsilon} + \frac {\log W}{m \beta}\right) = \Omega \left(\frac {\log W}{\varepsilon} + \frac {\log W}{\varepsilon \sqrt {m \beta}} + \frac {\log W}{m \beta}\right).
$$
Finally, observing that  $n_m^{\mathfrak{F}}(\varepsilon ,\delta) = n_1^{\mathfrak{F}'}(\varepsilon ,\delta)$  yields our final bound.
E.3 Applications
Our user-level DP EM with pairwise score given above has a wide variety of applications. We now give a few examples.
E.3.1 Agnostic PAC Learning
We start with agnostic PAC learning. The setting is exactly the same as in PAC learning (described in the introduction; see Theorem 4) except that we do not assume that  $\mathcal{D}$  is realizable by some concept in  $\mathcal{C}$ . Instead, the task  $\mathrm{agn}(\mathcal{C};\alpha)$  seeks an output  $c:\mathcal{X}\to \{0,1\}$  such that  $\mathrm{err}_{\mathcal{D}}(c)\leq \min_{c'\in \mathcal{C}}\mathrm{err}_{\mathcal{D}}(c') + \alpha$ .
Theorem 40. Let  $\mathcal{C}$  be any concept class with probabilistic representation dimension  $d$  (i.e.,  $\mathrm{PRDim}(\mathcal{C}) = d$ ) and let  $d_{\mathrm{VC}}$  be its VC dimension. Then, for any sufficiently small  $\alpha, \varepsilon > 0$  and for all  $m \in \mathbb{N}$ ,
$$
n _ {m} ^ {\mathrm {a g n} (\mathcal {C}; \alpha)} (\varepsilon , \delta = 0) \leq \widetilde {O} \left(\frac {d}{\varepsilon} + \frac {d}{\alpha \varepsilon \sqrt {m}} + \frac {d}{\alpha^ {2} m}\right),
$$
<sup>13</sup>The particular version we use is a slight restatement of [ASZ21, Corollary 4].
$$
n _ {m} ^ {\mathrm {a g n} (\mathcal {C}; \alpha)} (\varepsilon , \delta = 0) \geq \tilde {\Omega} \left(\frac {d}{\varepsilon} + \frac {d _ {\mathrm {V C}}}{\alpha \varepsilon \sqrt {m}} + \frac {d _ {\mathrm {V C}}}{\alpha^ {2} m}\right).
$$
Observe that our bounds are (up to polylogarithmic factors) the same except for the  $d$ -vs- $d_{\mathrm{VC}}$  in the last two terms. We remark that the ratio  $d / d_{\mathrm{VC}}$  is not always bounded; e.g., for thresholds,  $d_{\mathrm{VC}} = 2$  while  $d = \Theta (\log |\mathcal{Z}|)$ . A more careful argument can show that  $d$  in the last two terms in the upper bound can be replaced by the (maximum) VC dimension of the probabilistic representation (which is potentially smaller); this actually closes the gap in the particular case of thresholds. However, in the general case, the VC dimension of  $\mathcal{C}$  and the VC dimension of its probabilistic representation are not necessarily equal. It remains an interesting open question to close this gap.
We now prove Theorem 40. The algorithm is similar to that in the proof of Theorem 4, except that we use the pairwise scoring function to compare the errors of the two hypotheses.
Proof of Theorem 40. Algorithm. For any two hypotheses  $c, c': \mathcal{X} \to \{0,1\}$ , let the comparison function between two concepts be  $\psi_{c,c'}((x,y)) = \mathbf{1}[c(x) = y] - \mathbf{1}[c'(x) = y]$ .
Let  $\mathcal{P}$  denote a  $(0.01\alpha, 1/4)$ -PR of  $\mathcal{C}$ ; by Lemma 21, there exists such  $\mathcal{P}$  with  $\mathrm{size}(\mathcal{P}) \leq \widetilde{O}(d \cdot \log(1/\alpha))$ . Our algorithm works as follows: Sample  $\mathcal{H} \sim \mathcal{P}$  and then run the  $\varepsilon$ -DP pairwise scoring EM (Theorem 37) on candidate set  $\mathcal{H}$  with the comparison function  $\psi_{c,c'}$  defined above. The privacy guarantee follows from that of Theorem 37.
As for the utility, first observe that
$$
\operatorname {p s c r} _ {c, c ^ {\prime}} ^ {\psi} (\mathcal {D}) = \operatorname {e r r} _ {\mathcal {D}} (c) - \operatorname {e r r} _ {\mathcal {D}} \left(c ^ {\prime}\right). \tag {14}
$$
This means that, if we let  $h^* \in \operatorname{argmin}_{h \in \mathcal{H}} \operatorname{err}_{\mathcal{D}}(h)$ , then  $\operatorname{scr}_{h^*}^\psi(\mathcal{D}) \leq 0$ . Thus, Theorem 37 ensures that, w.p. 0.99, if  $n \geq \tilde{\Theta}\left(\frac{\text{size}(\mathcal{P})}{\varepsilon} + \frac{\text{size}(\mathcal{P})}{\alpha\varepsilon\sqrt{m}} + \frac{\text{size}(\mathcal{P})}{\alpha^2m}\right) = \tilde{\Theta}\left(\frac{d}{\varepsilon} + \frac{d}{\alpha\varepsilon\sqrt{m}} + \frac{d}{\alpha^2m}\right)$ , then the output  $h$  satisfies  $\operatorname{scr}_{h}^\psi(\mathcal{D}) \leq 0.5\alpha$ . By (14), this also means that  $\operatorname{err}_{\mathcal{D}}(h) \leq 0.5\alpha + \operatorname{err}_{\mathcal{D}}(h^*)$ . Finally, by the guarantee of  $\mathcal{P}$ , we have that  $\operatorname{err}_{\mathcal{D}}(h^*) \leq 0.01\alpha + \min_{c' \in \mathcal{C}} \operatorname{err}_{\mathcal{D}}(c')$  with probability 3/4. By a union bound, we can conclude that w.p. more than 2/3, we have  $\operatorname{err}_{\mathcal{D}}(h) < \alpha + \min_{c' \in \mathcal{C}} \operatorname{err}_{\mathcal{D}}(c')$ .
Lower Bound. The lower bound  $\Omega(d / \varepsilon)$  was shown in [GKM21] (and holds even for realizable PAC learning). To derive the lower bound  $\Omega\left(\frac{d_{\mathrm{VC}}}{\alpha\varepsilon\sqrt{m}} + \frac{d_{\mathrm{VC}}}{\alpha^2m}\right)$ , we will use DP Fano's inequality (Lemma 39). To do so, recall that  $\mathcal{C}$  having VC dimension  $d_{\mathrm{VC}}$  means that there exist  $\{x_1, \ldots, x_{d_{\mathrm{VC}}}\}$  that is shattered by  $\mathcal{Z}$ . From Theorem 24, there exist  $u_1, \ldots, u_W \in \{0,1\}^{d_{\mathrm{VC}}}$  where  $W = 2^{\Omega(d_{\mathrm{VC}})}$  such that  $\|u_i - u_j\|_0 \geq \kappa \cdot d_{\mathrm{VC}}$  for some constant  $\kappa$ . For any sufficiently small  $\alpha < \kappa / 8$ , we can define the distribution  $\mathcal{D}_i$  for  $i \in [W]$  by
$$
\mathcal {D} _ {i} ((x _ {\ell}, y)) = \frac {1}{2 d _ {\mathrm {V C}}} \cdot \left(1 + \frac {4 \alpha}{\kappa} \cdot (2 \mathbf {1} [ (u _ {i}) _ {\ell} = y ] - 1)\right),
$$
for all  $\ell \in [W]$  and  $y \in \{0,1\}$ . Next, notice that, for any  $i \in [W]$ , we may select  $c_i$  such that  $\mathrm{err}_{\mathcal{D}}(c_i) = \frac{1}{2}\left(1 - \frac{4\alpha}{\kappa}\right)$ .
Furthermore, for any  $i \neq j \in [W]$  and any hypothesis  $h: \mathcal{X} \to \{0,1\}$ , we have
$$
\operatorname {e r r} _ {\mathcal {D} _ {i}} (h) + \operatorname {e r r} _ {\mathcal {D} _ {j}} (h) \geq \left(1 - \frac {4 \alpha}{\kappa}\right) + \frac {4 \alpha}{\kappa} \cdot \frac {\kappa \cdot d _ {\mathrm {V C}}}{d _ {\mathrm {V C}}} = \left(1 - \frac {4 \alpha}{\kappa}\right) + 4 \alpha .
$$
This means that  $\mathrm{err}_{\mathcal{D}_i}(h) > \frac{1}{2}\left(1 - \frac{4\alpha}{\kappa}\right) + 2\alpha$  or  $\mathrm{err}_{\mathcal{D}_j}(h) > \frac{1}{2}\left(1 - \frac{4\alpha}{\kappa}\right) + 2\alpha$ . In other words, we have  $\Psi_{\mathrm{agn}(\mathcal{C};\alpha)}(\mathcal{D}_i)\cap \Psi_{\mathrm{agn}(\mathcal{C};\alpha)}(\mathcal{D}_j) = \emptyset$ . Finally, we have
$$
\begin{array}{l} d _ {\mathrm {K L}} \left(\mathcal {D} _ {i} \| \mathcal {D} _ {j}\right) \leq d _ {\chi^ {2}} \left(\mathcal {D} _ {i} \| \mathcal {D} _ {j}\right) \\ = \sum_ {\ell \in [ d _ {\mathrm {V C}} ], y \in \{0, 1 \}} \frac {\left(\mathcal {D} _ {i} ((x , y)) - \mathcal {D} _ {j} ((x , y))\right) ^ {2}}{\mathcal {D} _ {j} ((x , y))} \\ \leq \sum_ {\ell \in [ d _ {\mathrm {V C}} ], y \in \{0, 1 \}} \frac {O (\alpha / d _ {\mathrm {V C}}) ^ {2}}{1 / 4 d _ {\mathrm {V C}}} \\ \end{array}
$$
$$
\begin{array}{l} = \sum_ {\ell \in [ d _ {\mathrm {V C}} ]} O \left(\alpha^ {2} / d _ {\mathrm {V C}}\right) \\ \leq O \left(\alpha^ {2}\right). \\ \end{array}
$$
Plugging this into Lemma 39, we get that  $n_m^{\mathrm{agn}(\mathcal{C};\alpha)} \geq \Omega \left( \frac{d_{\mathrm{VC}}}{\varepsilon \alpha \sqrt{m}} + \frac{d_{\mathrm{VC}}}{m \alpha^2} \right)$  as desired.
E.3.2 Private Hypothesis Selection
In Hypothesis Selection problem, we are given a set  $\mathfrak{P}$  of hypotheses, where each  $\mathcal{P} \in \mathfrak{P}$  is a distribution over some domain  $\mathcal{Z}$ . The goal is, for the underlying distribution  $\mathcal{D}$ , to output  $\mathcal{P}^* \in \mathfrak{P}$  that is the closest (in TV distance) to  $\mathcal{D}$ . We state below the guarantee one can get from our approach:
Theorem 41. Let  $\alpha, \beta \in (0, 0.1)$ . Then, for any  $\varepsilon > 0$ , there exists an  $\varepsilon$ -user-level DP algorithm for Hypothesis Selection that, when each user has  $m$  samples and  $\min_{\mathcal{P}' \in \mathfrak{P}} d_{\mathrm{tv}}(\mathcal{P}', \mathcal{D}) \leq 0.1\alpha$ , with probability  $1 - \beta$ , outputs  $\mathcal{P} \in \mathfrak{P}$  such that  $d_{\mathrm{tv}}(\mathcal{P}, \mathcal{D}) \leq \alpha$  as long as
$$
n \geq \tilde {\Theta} \left(\frac {\log (| \mathfrak {P} | / \beta)}{\varepsilon} + \frac {\log (| \mathfrak {P} | / \beta)}{\alpha \varepsilon \sqrt {m}} + \frac {\log (| \mathfrak {P} | / \beta)}{\alpha^ {2} m}\right).
$$
For  $m = 1$ , the above theorem essentially match the item-level DP sample complexity bound from [BKSW19]. The proof also proceeds similarly as theirs, except that we use the user-level DP EM rather than the item-level DP one.
Proof Theorem 41. We use the so-called Scheffé score similar to [BKSW19]: For every  $\mathcal{P},\mathcal{P}'\in \mathfrak{P}$  we define  $\mathcal{W}_{\mathcal{P},\mathcal{P}'}$  as the set  $\{z\in \mathcal{Z}\mid \mathcal{P}(z) > \mathcal{P}'(z)\}$  and let
$$
\psi_ {\mathcal {P}, \mathcal {P} ^ {\prime}} (z) = \mathcal {P} \left(\mathcal {W} _ {\mathcal {P}, \mathcal {P} ^ {\prime}}\right) - \mathbf {1} [ z \in \mathcal {W} _ {\mathcal {P}, \mathcal {P} ^ {\prime}} ].
$$
We then use the  $\varepsilon$ -user-level DP pairwise scoring EM (Theorem 37). The privacy guarantee follows immediately from the theorem.
For the utility analysis, let  $\mathcal{P}^* \in \operatorname{argmin}_{\mathcal{P} \in \mathfrak{P}} d_{\mathrm{tv}}(\mathcal{P}, \mathcal{D})$ . Recall from our assumption that  $d_{\mathrm{tv}}(\mathcal{P}^*, \mathcal{D}) \leq 0.1\alpha$ . Furthermore, observe that
$$
\operatorname {p s c r} _ {\mathcal {P} ^ {*}, \mathcal {P}} ^ {\psi} (\mathcal {D}) = \mathcal {P} ^ {*} (\mathcal {W} _ {\mathcal {P} ^ {*}, \mathcal {P}}) - \mathcal {D} (\mathcal {W} _ {\mathcal {P} ^ {*}, \mathcal {P}}) \leq d _ {\mathrm {t v}} (\mathcal {D}, \mathcal {P} ^ {*}) \leq 0. 1 \alpha .
$$
Moreover, for every  $\mathcal{P} \in \mathfrak{P}$ , we have