SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {E} _ {\left(Z, Z ^ {\prime}\right)} \left[ \left| Z - Z ^ {\prime} \right| \right] = \mathbb {E} _ {Z} \left[ \left| Z - \mu^ {*} \right| \right] \leq \mathbb {E} _ {\left(Z, \hat {Z}\right)} \left[ \left| \hat {Z} - \mu^ {*} \right| + \left| \hat {Z} - Z \right| \right] \\ \stackrel {(\star)} {=} \mathbb {E} _ {\hat {Z}} [ \hat {Z} - \mu^ {*} ] + \mathbb {E} _ {(Z, \hat {Z})} [ | \hat {Z} - Z | ] \\ = \hat {\mu} - \mu^ {*} + \mathbb {E} _ {(Z, \hat {Z})} [ | \hat {Z} - Z | ] \\ \stackrel {(3)} {\leq} \mu - \mu^ {*} + 2 \mathbb {E} _ {(Z, \hat {Z})} [ | \hat {Z} - Z | ], \\ \end{array}
$$
where  $(\star)$  follows from  $\hat{Z} \geq e^{-\varepsilon}\mu \geq \mu^{*}$ . Combining the above inequality with (2) yields the desired bound in item (iii).
- Case II:  $\mu^{*} \approx_{\varepsilon} \mu$ . In this case, we already have  $\mathrm{supp}(\hat{Z}) \subseteq [e^{-\varepsilon} \cdot \mu, e^{\varepsilon} \cdot \mu] \subseteq [e^{-2\varepsilon} \mu^{*}, e^{2\varepsilon} \mu^{*}]$ . We assume w.l.o.g. that  $\hat{\mu} \geq \mu^{*}$ ; the case  $\hat{\mu} \leq \mu^{*}$  can be handled similarly. In this case, let  $f:[\ell, r] \to \mathbb{R}$  be defined by  $f(\tau) := \mathbb{E}_{\hat{Z}}[\mathrm{clip}_{\ell, \tau}(\hat{Z})]$ . Notice that  $f$  is continuous,  $f(\ell) = \ell \leq \mu^{*}$  and  $f(r) = \hat{\mu} \geq \mu^{*}$ . Thus, there exists  $\tau^{*}$  such that  $f(\tau^{*}) = \mu^{*}$ . We then define  $Z'$  by  $Z' := \mathrm{clip}_{\ell, \tau^{*}}(\hat{Z})$ . This immediately satisfies items (i) and (ii). Furthermore, this gives a natural coupling between  $Z'$ ,  $\hat{Z}$  (and also  $Z$ ) such that
$$
\mathbb {E} _ {(\hat {Z}, Z ^ {\prime})} [ | \hat {Z} - Z ^ {\prime} | ] = \mathbb {E} _ {(\hat {Z}, Z ^ {\prime})} [ \hat {Z} - Z ^ {\prime} ] = \hat {\mu} - \mu^ {*} \leq | \mu - \mu^ {*} | + | \hat {\mu} - \mu |.
$$
Combining this with (2) and (3) yields the desired bound in item (iii).
Concentration for Bounded-Ratio Martingales. Since we will be applying Lemma 27 on the logarithm of the ratios of random variables, it is crucial to bound its expectation. We present a bound below. Note that similar statements have been used before in DP literature; we present the (simple) proof here for completeness.
Lemma 34. Let  $Z$  be any  $\mathbb{R}_{>0}$ -valued random variable such that  $\mathbb{E}[Z] = 1$ . For any sufficiently small  $\varepsilon > 0$ , if  $Z \approx_{\varepsilon} 1$ , then  $-8\varepsilon^2 \leq \mathbb{E}\left[\ln Z\right] \leq 0$ .
Proof. By the concavity of  $\ln (\cdot)$ , we get  $\mathbb{E}[\ln Z]\leq \ln \mathbb{E}[Z] = 0$ .
Note also that for  $\varepsilon \leq 1$ , we have  $\ln (x) \geq x - 1 - 2(x - 1)^2$  for all  $x \in [e^{-\varepsilon}, e^{\varepsilon}]$ . This implies that
$$
\mathbb {E} [ \ln Z ] \geq \mathbb {E} [ Z - 1 - 2 (Z - 1) ^ {2} ] = - 2 \mathbb {E} [ (Z - 1) ^ {2} ] \geq - 2 (e ^ {\varepsilon} - 1) ^ {2} \geq - 8 \varepsilon^ {2}.
$$
We are now ready to give a concentration inequality for martingales with bounded ratios. Note that the lemma below is stated in an "expected deviation from  $[e^{-\varepsilon'} \cdot \mu, e^{\varepsilon'} \cdot \mu]$ " form since it will be more convenient to use in a subsequent step.
Lemma 35. Assume that  $\varepsilon, \delta > 0$ , and  $\varepsilon \sqrt{n \log(1 / \delta)} > 0$  are sufficiently small. Let  $Y_{1}, \ldots, Y_{n}$  be a  $\mathbb{R}_{\geq 0}$ -valued martingale such that  $Y_{i} \approx_{\varepsilon} Y_{i-1}$  almost surely, and  $\mu \coloneqq \mathbb{E}[Y_1]$ . Then, we have
$$
\mathbb {E} \left[ \left[ Y _ {n} - e ^ {\varepsilon^ {\prime}} \cdot \mu \right] _ {+} + \left[ \mu - e ^ {\varepsilon^ {\prime}} \cdot Y _ {n} \right] _ {+} \right] \leq O (\mu \cdot \delta),
$$
where  $\varepsilon^{\prime} = O(\varepsilon \sqrt{n\log(1 / \delta)})$
Proof. Let  $\tau = 100\log (1 / \delta)$ . Consider  $C_1,\ldots ,C_n$  defined by  $C_i\coloneqq \ln (Y_i / Y_{i - 1})$ , where we use the convention  $Y_{0}\coloneqq \mu$ . By our assumption, we have  $|C_i|\leq \varepsilon$ . Furthermore, Lemma 34 implies that  $-8\varepsilon^{2}\leq \mathbb{E}[Y_{i}\mid Y_{i - 1} = y_{i - 1},\ldots ,Y_{1} = y_{1}]\leq 0$  for all  $y_{i - 1},\ldots ,y_{1}$ . Let  $\gamma \coloneqq -8\varepsilon^2$ . Applying Lemma 27, the following holds for all  $G > 0$ :
$$
\Pr \left[ n \gamma - \varepsilon \sqrt {n G} \leq \sum_ {i \in [ k ]} C _ {i} \leq \varepsilon \sqrt {n G} \right] \geq 1 - 2 \exp (- G / 2).
$$
$$
\Pr \left[ e ^ {n \gamma - \varepsilon \sqrt {n G}} \cdot \mu \leq Y _ {k} \leq e ^ {\varepsilon \sqrt {n G}} \cdot \mu \right] \geq 1 - 2 \exp (- G / 2). \tag {4}
$$
$$
\mathbb {E} \left[ [ \mu - e ^ {\varepsilon \sqrt {n \cdot \tau}} \cdot Y _ {n} ] _ {+} \right] \leq \mu \cdot \operatorname * {P r} [ Y _ {n} \leq e ^ {- \varepsilon \sqrt {n \cdot \tau}} \cdot \mu ] \leq \mu \cdot \operatorname * {P r} [ Y _ {n} \leq e ^ {n \gamma - \varepsilon \sqrt {n \cdot \tau} / 2} \mu ] \overset {(4)} {\leq} \mu \cdot \delta ,
$$
where the second inequality follows from the assumption that  $\varepsilon \sqrt{n\log(1 / \delta)}$  is sufficiently small.
$$
\begin{array}{l} \mathbb {E} \left[ \left[ Y _ {n} - e ^ {\varepsilon \sqrt {n \cdot \tau}} \cdot \mu \right] _ {+} \right] = \int_ {0} ^ {\infty} \Pr \left[ Y _ {n} - e ^ {\varepsilon \sqrt {n \cdot \tau}} \cdot \mu \geq y \right] d y \\ = \mu \cdot \int_ {0} ^ {\infty} \operatorname * {P r} [ Y _ {n} - e ^ {\varepsilon \sqrt {n \cdot \tau}} \cdot \mu \geq \mu \cdot y ] d y \\ = \mu \cdot \int_ {e ^ {\varepsilon \sqrt {n \cdot \tau}}} ^ {\infty} \Pr \left[ Y _ {n} \geq \mu \cdot y \right] d y \\ = \mu \cdot \int_ {\tau} ^ {\infty} \Pr \left[ Y _ {n} \geq e ^ {\varepsilon \sqrt {n G}} \cdot \mu \right] \cdot \frac {\sqrt {n} \varepsilon}{2 \sqrt {G}} \exp (\varepsilon \sqrt {G n}) d G \\ \stackrel {(4)} {\leq} \mu \cdot \int_ {\tau} ^ {\infty} 2 \exp (- G / 2) \cdot \frac {\sqrt {n} \varepsilon}{2 \sqrt {G}} \exp (\varepsilon \sqrt {G n}) d G \\ \leq \mu \cdot \int_ {\tau} ^ {\infty} 2 \exp (- G / 4) d G \\ = \mu \cdot 8 \exp (- \tau / 4) \\ \leq \mu \cdot \delta , \\ \end{array}
$$
where the second and third inequalities are from our choice of  $\tau$ .
Combining the previous two inequalities yields the desired bound for  $\varepsilon' = \varepsilon \sqrt{n\tau}$ .
Putting Things Together: Proof of Lemma 32. Our overall strategy is similar to the proof for the always-bounded case, which is to set up a martingale from suffix averages and then apply Lemma 35. The main modification is that, instead of using this argument directly on  $f$  (and its suffix averages), we use it on a different function, which is the result of applying Lemma 33 to  $f$ .
Proof of Lemma 32. For convenience, let  $\mu \coloneqq \mu_{\mathcal{D}^n}(f)$ . Let us first extend the function  $f$  to include prefixes (i.e.,  $f:\mathcal{Z}^{\leq n}\to \mathbb{R}_{\geq 0}$ ) naturally as follows:
$$
f (x _ {1}, \dots , x _ {i}) = \mathbb {E} _ {X _ {i + 1}, \dots , X _ {n} \sim \mathcal {D}} [ f (x _ {1}, \dots , x _ {i}, X _ {i + 1}, \dots , X _ {n}) ].
$$
Next, we construct a function  $g:\mathcal{Z}^{\leq n}\to \mathbb{R}_{\geq 0}$  as follows:
$g(\emptyset) = f(\emptyset)$  (which is equal to  $\mu$ )
- For  $i = 1, \dots, n$ :
- For all  $(x_{1},\ldots ,x_{i - 1})\in \mathcal{Z}^{i - 1}$
* Apply Lemma 33 with  $\mu^{*} \coloneqq g(x_{1},\ldots ,x_{i - 1})$  and  $Z = f(x_{1},\dots,x_{i - 1},X_{i})$ , where  $X_{i}\sim \mathcal{D}$  to get a random variable  $Z^{\prime}$ .
* The coupling between  $(Z, Z')$  is naturally also a coupling between  $(x_i, Z')$  for  $x_i \in \mathcal{Z}$ . Let  $g(x_1, \ldots, x_i) = Z'$  based on this coupling for all  $x_i \in \mathcal{Z}$ .
For brevity, let  $\rho = 2(1 + e^{-\varepsilon})$ . We have
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {x} \sim \mathcal {D} ^ {n}} [ | g (\boldsymbol {x}) - f (\boldsymbol {x}) | ] \\ = \mathbb {E} _ {\boldsymbol {x} \sim \mathcal {D} ^ {n}} \mathbb {E} _ {\tilde {x} _ {n} \sim \mathcal {D} ^ {n}} [ | g (\boldsymbol {x} _ {\leq n - 1}, \tilde {x} _ {n}) - f (\boldsymbol {x} _ {\leq n - 1}, \tilde {x} _ {n}) | ] \\ \leq \mathbb {E} _ {\boldsymbol {x} \sim \mathcal {D} ^ {n}} \left[ | g (\boldsymbol {x} _ {\leq n - 1}) - f (\boldsymbol {x} _ {\leq n - 1}) | + \rho \cdot \mathbb {E} _ {\tilde {x} _ {n}, \tilde {x} _ {n} ^ {\prime} \sim \mathcal {D}} \left[ f (\boldsymbol {x} _ {\leq n - 1}, \tilde {x} _ {n}) - e ^ {\varepsilon} \cdot f (\boldsymbol {x} _ {\leq n - 1}, \tilde {x} _ {n} ^ {\prime}) \right] _ {+} \right] \\ = \mathbb {E} _ {\boldsymbol {x} _ {\leq n - 1} \sim \mathcal {D} ^ {n - 1}} [ | g (\boldsymbol {x} _ {\leq n - 1}) - f (\boldsymbol {x} _ {\leq n - 1}) | ] + \rho \cdot \mathbb {E} _ {\boldsymbol {x}, \boldsymbol {x} ^ {\prime} \sim \mathcal {D} ^ {n}} [ f (\boldsymbol {x} _ {\leq n - 1}, x _ {n}) - e ^ {\varepsilon} \cdot f (\boldsymbol {x} _ {\leq n - 1}, x _ {n} ^ {\prime}) ] _ {+}, \\ \end{array}
$$
where the inequality is from Lemma 33(iii).
By repeatedly applying the above argument (to the first term), we arrive at
$$
\mathbb {E} _ {\boldsymbol {x} \sim \mathcal {D} ^ {n}} [ | g (\boldsymbol {x}) - f (\boldsymbol {x}) | ] \leq \rho \cdot \sum_ {i = 1} ^ {n} \mathbb {E} _ {\boldsymbol {x}, \boldsymbol {x} ^ {\prime} \sim \mathcal {D} ^ {n}} \left[ f \left(\boldsymbol {x} _ {\leq i - 1}, x _ {i}\right) - e ^ {\varepsilon} \cdot f \left(\boldsymbol {x} _ {\leq i - 1}, x _ {i} ^ {\prime}\right) \right] _ {+}.
$$
Recall from the definition that  $f(\pmb{x}_{\leq i - 1}, x_i) = \mathbb{E}_{\pmb{x}_{>i} \sim \mathcal{D}^{n - i}}[f(\pmb{x}_{\leq i - 1}, x_i, \pmb{x}_{>i})]$  and  $f(\pmb{x}_{\leq i - 1}, x_i') = \mathbb{E}_{\pmb{x}_{>i} \sim \mathcal{D}^{n - i}}[f(\pmb{x}_{\leq i - 1}, x_i', \pmb{x}_{>i})]$ . Thus, by the convexity of  $[.]_+$ , we further have
$$
\mathbb {E} _ {\boldsymbol {x} \sim \mathcal {D} ^ {n}} [ | g (\boldsymbol {x}) - f (\boldsymbol {x}) | ] \leq \rho \cdot \sum_ {i = 1} ^ {n} \mathbb {E} _ {\boldsymbol {x}, \tilde {\boldsymbol {x}} \sim \mathcal {D} ^ {n}} \left[ f (\boldsymbol {x}) - e ^ {\varepsilon} \cdot f \left(\tilde {\boldsymbol {x}} ^ {(i)}\right) \right] _ {+}. \tag {5}
$$
Next, let  $Y_0, Y_1, \ldots, Y_n$  be a sequence of random variables defined by sampling  $x_1, \ldots, x_n \sim \mathcal{D}^n$  and letting  $Y_i := g(x_1, \ldots, x_i)$ . By Lemma 33(ii), we have that  $Y_1, \ldots, Y_n$  is a martingale. Lemma 33(i) further implies that  $Y_{i-1} \approx_{2\varepsilon} Y_i$ . Thus, we may apply Lemma 35 to arrive at
$$
\mathbb {E} \left[ \left[ Y _ {n} - e ^ {\varepsilon^ {\prime} / 2} \cdot \mu \right] _ {+} + \left[ \mu - e ^ {\varepsilon^ {\prime} / 2} \cdot Y _ {n} \right] _ {+} \right] \leq O (\mu \delta),
$$
for  $\varepsilon' = O\left(\varepsilon \sqrt{n \log(1 / \delta)}\right)$ . Note that this is equivalent to
$$
\mathbb {E} _ {\boldsymbol {x} \sim \mathcal {D} ^ {n}} \left[ \left[ g (\boldsymbol {x}) - e ^ {\varepsilon^ {\prime} / 2} \cdot \mu \right] _ {+} + \left[ \mu - e ^ {\varepsilon^ {\prime} / 2} \cdot g (\boldsymbol {x}) \right] _ {+} \right] \leq O (\mu \delta). \tag {6}
$$
We can now derive the bound on the desired quantity as follows:
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {x}, \boldsymbol {x} ^ {\prime} \sim \mathcal {D} ^ {n}} \left[ \left[ f (\boldsymbol {x}) - e ^ {\varepsilon^ {\prime}} f \left(\boldsymbol {x} ^ {\prime}\right) \right] _ {+} \right] \\ = \mathbb {E} _ {\boldsymbol {x}, \boldsymbol {x} ^ {\prime} \sim \mathcal {D} ^ {n}} \left[ \left(\left(f (\boldsymbol {x}) - g (\boldsymbol {x})\right) + \left(g (\boldsymbol {x}) - e ^ {\varepsilon^ {\prime} / 2} \mu\right) + e ^ {\varepsilon^ {\prime} / 2} \left(\mu - e ^ {\varepsilon^ {\prime} / 2} g \left(\boldsymbol {x} ^ {\prime}\right)\right) + e ^ {\varepsilon^ {\prime}} \left(g \left(\boldsymbol {x} ^ {\prime}\right) - f \left(\boldsymbol {x} ^ {\prime}\right)\right) \right] _ {+} \right] \\ \leq \mathbb {E} [ f (\pmb {x}) - g (\pmb {x}) ] _ {+} + \mathbb {E} [ g (\pmb {x}) - e ^ {\varepsilon^ {\prime} / 2} \mu ] _ {+} + e ^ {\varepsilon^ {\prime} / 2} \mathbb {E} [ \mu - e ^ {\varepsilon^ {\prime} / 2} g (\pmb {x} ^ {\prime}) ] _ {+} + \mathbb {E} [ g (\pmb {x} ^ {\prime}) - f (\pmb {x} ^ {\prime}) ] _ {+} \\ \stackrel {(5), (6)} {\leq} O (\mu \delta) + O \left(\sum_ {i = 1} ^ {n} \mathbb {E} _ {\boldsymbol {x}, \tilde {\boldsymbol {x}} \sim \mathcal {D} ^ {n}} \left[ f (\boldsymbol {x}) - e ^ {\varepsilon} \cdot f (\tilde {\boldsymbol {x}} ^ {(i)}) \right] _ {+}\right). \\ \end{array}
$$
C.3.2 Putting Things Together: Proof of Lemma 14
Lemma 14 is now a simple consequence of Lemma 32.
Proof of Lemma 14. For every  $o \in \mathcal{O}$  and  $\pmb{x} \in \mathcal{Z}^n$ , let  $f^o(\pmb{x}) := \operatorname*{Pr}[\mathbb{A}(\pmb{x}) = o]$ . Applying Lemma 32 to  $f^o$ , we get
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {x}, \boldsymbol {x} ^ {\prime} \sim \mathcal {D} ^ {n}} \left[ \left[ f ^ {o} (\boldsymbol {x}) - e ^ {\varepsilon^ {\prime}} \cdot f ^ {o} (\boldsymbol {x} ^ {\prime}) \right] _ {+} \right] \\ \leq O \left(\mu_ {\mathcal {D} ^ {n}} \left(f ^ {o}\right) \cdot \delta\right) + O \left(\sum_ {i = 1} ^ {n} \mathbb {E} _ {\boldsymbol {x}, \tilde {\boldsymbol {x}} \sim \mathcal {D} ^ {n}} \left[ f ^ {o} (\boldsymbol {x}) - e ^ {\varepsilon} \cdot f ^ {o} \left(\tilde {\boldsymbol {x}} ^ {(i)}\right) \right] _ {+}\right), \\ \end{array}
$$
Summing this up over all  $o \in \mathcal{O}$ , we get
$$
\mathbb {E} _ {\boldsymbol {x}, \boldsymbol {x} ^ {\prime} \sim \mathcal {D} ^ {n}} [ d _ {\varepsilon^ {\prime}} (\mathbb {A} (\boldsymbol {x}) \| \mathbb {A} (\boldsymbol {x} ^ {\prime})) ] \leq O (\delta) + O \left(\sum_ {i = 1} ^ {n} \mathbb {E} _ {\boldsymbol {x}, \tilde {\boldsymbol {x}} \sim \mathcal {D} ^ {n}} \left[ d _ {\varepsilon} (\mathbb {A} (\boldsymbol {x}) \| \mathbb {A} (\tilde {\boldsymbol {x}} ^ {(i)})) \right]\right) \leq O (n \delta),
$$
where the second inequality follows from the assumption that  $\mathbb{A}$  is  $(\varepsilon, \delta)$ -item-level DP, which implies  $d_{\varepsilon}(\mathbb{A}(\boldsymbol{x}) \parallel \mathbb{A}(\tilde{\boldsymbol{x}}^{(i)})) \leq \delta$ .
C.4 Privacy Analysis of Algorithm 1
Here we prove the privacy guarantee (Lemma 18) of Algorithm 1. This mostly follows the proof for the analogous algorithm in  $\mathrm{[GKK^{+}23]}$ .
We start with the following observation.
Observation 36. For user-level neighboring datasets  $\mathbf{x}, \mathbf{x}'$ , and all  $r_1 \in \{0, \dots, n-1\}$ , if  $\mathcal{X}_{\mathrm{stable}}^{r_1}(\mathbf{x}') \neq \emptyset$ , then  $\mathcal{X}_{\mathrm{stable}}^{r_1+1}(\mathbf{x}) \neq \emptyset$ .
Proof. Let  $i \in [n]$  be such that  $\pmb{x}_{-i} = \pmb{x}_{-i}'$ , and let  $\pmb{x}_{-S'}'$  be an element of  $\mathcal{X}_{\mathrm{stable}}^{r_1}(\pmb{x}')$ . Select  $S$  such that  $|S| = r_1 + 1$  and  $S \supseteq S' \cup \{i\}$ . It is obvious from definition that  $\pmb{x}_S \in \mathcal{X}_{\mathrm{stable}}^{r_1 + 1}(\pmb{x})$ .
Proof of Lemma 18. Let  $\pmb{x}, \pmb{x}'$  be user-level neighboring datasets, and let  $\mathcal{A}$  be a shorthand for  $\text{DelStab}_{\varepsilon, \delta, \mathbb{A}}$ . First, we follow the proof of [GKK+23, Theorem 3.3]:
$$
\begin{array}{l} \operatorname * {P r} [ \mathcal {A} (\pmb {x}) = \bot ] = \sum_ {r _ {1} = 0} ^ {2 \kappa} \mathbf {1} [ \mathcal {A} (\pmb {x}) = \bot | R _ {1} = r _ {1} ] \cdot \operatorname * {P r} [ R _ {1} = r _ {1} ] \\ = \sum_ {r _ {1} = 0} ^ {2 \kappa} \mathbf {1} \left[ \mathcal {X} _ {\text {s t a b l e}} ^ {r _ {1}} (\boldsymbol {x}) = \emptyset \right] \cdot \Pr [ R _ {1} = r _ {1} ] \\ \leq \operatorname * {P r} [ R _ {1} = 0 ] + \sum_ {r _ {1} = 1} ^ {2 \kappa} \mathbf {1} [ \mathcal {X} _ {\mathrm {s t a b l e}} ^ {r _ {1}} (\pmb {x}) = \emptyset ] \cdot \operatorname * {P r} [ R _ {1} = r _ {1} ] \\ \leq \bar {\delta} + \sum_ {r _ {1} = 1} ^ {2 \kappa} \mathbf {1} \left[ \mathcal {X} _ {\text {s t a b l e}} ^ {r _ {1}} (\boldsymbol {x}) = \varnothing \right] \cdot e ^ {\bar {\varepsilon}} \cdot \Pr \left[ R _ {1} = r _ {1} - 1 \right] \\ \leq \bar {\delta} + \sum_ {r _ {1} = 1} ^ {2 \kappa} \mathbf {1} \left[ \mathcal {X} _ {\text {s t a b l e}} ^ {r _ {1} - 1} \left(\boldsymbol {x} ^ {\prime}\right) = \emptyset \right] \cdot e ^ {\bar {\varepsilon}} \cdot \Pr \left[ R _ {1} = r _ {1} - 1 \right] \\ \leq \overline {{\delta}} + e ^ {\overline {{\varepsilon}}} \cdot \sum_ {r _ {1} = 0} ^ {2 \kappa} \mathbf {1} [ \mathcal {A} (\boldsymbol {x} ^ {\prime}) = \bot | R _ {1} = r _ {1} ] \cdot \operatorname * {P r} [ R _ {1} = r _ {1} ] \\ = \bar {\delta} + e ^ {\bar {\varepsilon}} \cdot \Pr [ \mathcal {A} (\boldsymbol {x} ^ {\prime}) = \bot ], \tag {7} \\ \end{array}
$$
and for any  $U_0 \subseteq \mathcal{O}$ , we have
$$
\begin{array}{l} \Pr [ \mathcal {A} (\boldsymbol {x}) \in U _ {0} ] \leq \sum_ {r _ {1} = 0} ^ {2 \kappa} \Pr [ \mathcal {A} (\boldsymbol {x}) \in U _ {0} \mid R _ {1} = r _ {1} ] \cdot \Pr [ R _ {1} = r _ {1} ] \\ = \sum_ {r _ {1} = 0} ^ {2 \kappa - 1} \operatorname * {P r} [ \mathcal {A} (\boldsymbol {x}) \in U _ {0} \mid R _ {1} = r _ {1} ] \cdot \operatorname * {P r} [ R _ {1} = r _ {1} ] + \operatorname * {P r} [ R _ {1} = 2 \kappa ] \\ \leq \bar {\delta} + e ^ {\bar {\varepsilon}} \cdot \sum_ {r _ {1} = 0} ^ {2 \kappa - 1} \Pr [ \mathcal {A} (\boldsymbol {x}) \in U _ {0} \mid R _ {1} = r _ {1} ] \cdot \Pr [ R _ {1} = r _ {1} + 1 ], \tag {8} \\ \end{array}
$$
To bound the term  $\operatorname*{Pr}[\mathcal{A}(\pmb {x})\in U_0\mid R_1 = r_1]$  for  $r_1 < 2\kappa$  , observe that if it is non-zero, then it must be that  $\mathcal{A}(\pmb {x})\neq \bot$  or equivalently that  $\mathcal{X}_{\mathrm{stable}}^{r_1}(\pmb {x})\neq \emptyset$  . In this case, we have<sup>10</sup>
$$
\operatorname * {P r} [ \mathcal {A} (\boldsymbol {x}) \in U _ {0} \mid R _ {1} = r _ {1} ] = \mathbb {E} _ {\mathcal {S} \sim \mathcal {X} _ {\mathrm {s t a b l e}} ^ {r _ {1}} (\boldsymbol {x}), J \sim \binom {\overline {{S}}} {n - 4 \kappa}} \left[ \operatorname * {P r} [ \mathbb {A} (\boldsymbol {x} _ {J}) \in U _ {0} ] \right].
$$
Since  $\mathcal{X}_{\mathrm{stable}}^{r_1}(\pmb {x})\neq \emptyset$  , Observation 36 also implies that  $^{\mathrm{11}}\mathcal{X}_{\mathrm{stable}}^{r_1 + 1}(\pmb{x}^{\prime})\neq \emptyset$  . Thus, we have
$$
\operatorname * {P r} [ \mathcal {A} (\boldsymbol {x} ^ {\prime}) \in U _ {0} \mid R _ {1} = r _ {1} + 1 ] = \mathbb {E} _ {S ^ {\prime} \sim \mathcal {X} _ {\mathrm {s t a b l e}} ^ {r _ {1} + 1} (\boldsymbol {x} ^ {\prime}), J ^ {\prime} \sim \binom {\overline {{S}} ^ {\prime}} {n - 4 \kappa}} \left[ \operatorname * {P r} [ \mathbb {A} (\boldsymbol {x} _ {J ^ {\prime}}) \in U _ {0} ] \right].
$$
Consider an arbitrary pair  $S \in \mathcal{X}_{\mathrm{stable}}^{r_1}(\pmb{x})$ ,  $S' \in \mathcal{X}_{\mathrm{stable}}^{r_1 + 1}(\pmb{x}')$ . Let  $i \in [n]$  be such that  $\pmb{x}_{-i} = \pmb{x}'_{-i}$ . Note that  $|(\overline{S} \cap \overline{S}') \setminus \{i\}| \geq n - 4\kappa$ . Let  $J^*$  be any subset of  $(\overline{S} \cap \overline{S}')$  of size  $n - 4\kappa$ . From  $S \in \mathcal{X}_{\mathrm{stable}}^{r_1}(\pmb{x})$ , we have
$$
\Pr [ \mathbb {A} (\boldsymbol {x} _ {J}) \in U _ {0} ] \leq e ^ {\bar {\varepsilon}} \Pr [ \mathbb {A} (\boldsymbol {x} _ {J ^ {*}}) \in U _ {0} ] + \bar {\delta},
$$
for all  $J \in \left( \begin{array}{c} \overline{S} \\ n - 4\kappa \end{array} \right)$ , and, from  $S' \in \mathcal{X}_{\mathrm{stable}}^{r_1 + 1}(\pmb{x}')$ , we have
$$
\Pr \left[ \mathbb {A} \left(\boldsymbol {x} _ {J ^ {*}}\right) \in U _ {0} \right] \leq e ^ {\bar {\varepsilon}} \Pr \left[ \mathbb {A} \left(\boldsymbol {x} _ {J ^ {\prime}}\right) \in U _ {0} \right] + \bar {\delta},
$$
for all  $J^{\prime}\in \left(\frac{\overline{S}^{\prime}}{n - 4\kappa}\right)$
Therefore, by arbitrary coupling of  $S$ ,  $S^{\prime}$ , we can conclude that
$$
\operatorname * {P r} [ \mathcal {A} (\boldsymbol {x}) \in U _ {0} \mid R _ {1} = r _ {1} ] \leq e ^ {2 \overline {{\varepsilon}}} \operatorname * {P r} [ \mathcal {A} (\boldsymbol {x} ^ {\prime}) \in U _ {0} \mid R _ {1} = r _ {1} + 1 ] + (e ^ {\overline {{\varepsilon}}} + 1) \overline {{\delta}}.
$$
10Here we write  $\overline{S} \coloneqq [n] \setminus S$ .
11Again we write  $\overline{S}^{\prime}\coloneqq [n]\setminus S^{\prime}$
Plugging this back to (8), we get
$$
\begin{array}{l} \Pr [ \mathcal {A} (\boldsymbol {x}) \in U _ {0} ] \\ \leq \bar {\delta} + e ^ {\bar {\varepsilon}} \cdot \sum_ {r _ {1} = 0} ^ {2 \kappa - 1} \left(e ^ {2 \bar {\varepsilon}} \Pr [ \mathcal {A} (\boldsymbol {x} ^ {\prime}) \in U _ {0} \mid R _ {1} = r _ {1} + 1 ] + (e ^ {\bar {\varepsilon}} + 1) \bar {\delta}\right) \cdot \Pr [ R _ {1} = r _ {1} + 1 ] \\ \leq (e ^ {2 \overline {{\varepsilon}}} + e ^ {\overline {{\varepsilon}}} + 1) \overline {{\delta}} + e ^ {3 \overline {{\varepsilon}}} \cdot \sum_ {r _ {1} = 0} ^ {2 \kappa} \operatorname * {P r} [ \mathcal {A} (\pmb {x} ^ {\prime}) \in U _ {0} \mid R _ {1} = r _ {1} ] \cdot \operatorname * {P r} [ R _ {1} = r _ {1} ] \\ = \left(e ^ {2 \bar {\varepsilon}} + e ^ {\bar {\varepsilon}} + 1\right) \bar {\delta} + e ^ {3 \bar {\varepsilon}} \cdot \Pr \left[ \mathbb {A} \left(\boldsymbol {x} ^ {\prime}\right) \in U _ {0} \right]. \tag {9} \\ \end{array}
$$
Now, consider any set  $U \subseteq \mathcal{O} \cup \{\bot\}$  of outcomes. Let  $U_{\mathcal{O}} = U \cap \mathcal{O}$  and  $U_{\bot} = U \cap \{\bot\}$ . Then, we have
$$
\begin{array}{l} \Pr [ \mathcal {A} (\boldsymbol {x}) \in U ] = \Pr [ \mathcal {A} (\boldsymbol {x}) \in U _ {\mathcal {O}} ] + \Pr [ \mathcal {A} (\boldsymbol {x}) \in U _ {\perp} ] \\ \stackrel {(9), (7)} {\leq} \left(\left(e ^ {2 \bar {\varepsilon}} + e ^ {\bar {\varepsilon}} + 1\right) \bar {\delta} + e ^ {3 \bar {\varepsilon}} \cdot \Pr [ \mathbb {A} (\boldsymbol {x} ^ {\prime}) \in U _ {\mathcal {O}} ]\right) + \left(\bar {\delta} + e ^ {\bar {\varepsilon}} \cdot \Pr [ \mathcal {A} (\boldsymbol {x} ^ {\prime}) = U _ {\perp} ]\right) \\ \leq \left(e ^ {2 \bar {\varepsilon}} + e ^ {\bar {\varepsilon}} + 2\right) \bar {\delta} + e ^ {3 \bar {\varepsilon}} \Pr \left[ \mathcal {A} \left(\boldsymbol {x} ^ {\prime}\right) \in U \right] \\ \leq \delta + e ^ {\varepsilon} \cdot \Pr [ \mathcal {A} (\boldsymbol {x} ^ {\prime}) \in U ]. \\ \end{array}
$$
Therefore, the algorithm is  $(\varepsilon, \delta)$ -user-level DP as desired.
D Missing Details from Section 4
Below we give the proof of Lemma 22.
Proof of Lemma 22. Consider any  $h \in \{0,1\}^{\mathcal{X}}$ . Since  $z_{i}$ 's are drawn independently from  $\mathcal{D}$ , we have that  $\mathbf{1}[z_i$  is not realizable by  $h]$ 's are independent Bernoulli random variables with success probability
$$
p _ {h} = 1 - (1 - \operatorname {e r r} _ {\mathcal {D}} (h)) ^ {m}.
$$
Consider the two cases:
- If  $\operatorname{err}_{\mathcal{D}}(h) \leq 0.01\alpha$ , then we have
$$
p _ {h} \leq 1 - (1 - 0. 0 1 \alpha) ^ {m} \leq 0. 0 1 \alpha m.
$$
Thus, applying Lemma 25, we can conclude that
$$
\Pr \left[ \operatorname {s c r} _ {h} (z) \geq 0. 0 5 \alpha n m \right] \leq \exp (- \Theta (n \alpha m)) = \exp (- \Theta (\kappa \cdot \operatorname {s i z e} (\mathcal {P}))).
$$
Thus, when we select  $\kappa$  to be sufficiently large, we have
$$
\Pr [ \operatorname {s c r} _ {h} (z) \geq 0. 0 5 \alpha n m ] \leq \exp (- 1 0 \operatorname {s i z e} (\mathcal {P})) \leq \frac {0 . 0 1}{\exp (\operatorname {s i z e} (\mathcal {P}))} \leq \frac {0 . 0 1}{| \mathcal {H} |}.
$$
- If  $\operatorname{err}_{\mathcal{D}}(h) > \alpha$ , then we have
$$
p _ {h} \geq 1 - (1 - \alpha) ^ {m} \geq 1 - e ^ {- \alpha m} \geq 0. 2 \alpha m,
$$
where in the second inequality we use the fact that  $\alpha m\leq 1$
Again, applying Lemma 25, we can conclude that
$$
\Pr [ \operatorname {s c r} _ {h} (z) \leq 0. 2 \alpha n m ] \leq \exp (- \Theta (n \alpha m)) = \exp (- \Theta (\kappa \cdot \operatorname {s i z e} (\mathcal {P}))).
$$
$$
\Pr \left[ \operatorname {s c r} _ {h} (\boldsymbol {z}) \leq 0. 2 \alpha n m \right] \leq \exp (- 1 0 \operatorname {s i z e} (\mathcal {P})) \leq \frac {0 . 0 1}{\exp (\operatorname {s i z e} (\mathcal {P}))} \leq \frac {0 . 0 1}{| \mathcal {H} |}.
$$
Finally, taking a union bound over all  $h \in \mathcal{H}$  yields the desired result.