SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The difference between our algorithm and the DP-GD in [46] is the noise on the gradient. Note that with high probability,  $\sum_{i\in [\Gamma ]}\| \nabla_i - \nabla F(x_t)\| _2^2$  in our algorithm is controlled and small, and hence does not change the other proofs in [46]. Hence if  $F(x_0) - F(x_{\Gamma})$  is small, i.e., the function value does not decrease significantly, we know  $x_{t}$  is close to  $x_0$ .
Let  $B_{x}(r)$  be the unit ball of radius  $r$  around point  $x$ . Denote the  $(x)_{\Gamma}$  the point  $x_{\Gamma}$  after running SGD mentioned in Lemma 3.4 for  $\Gamma$  steps beginning at  $x$ . With this observation, denote  $B^{\gamma}(x_0)\coloneqq \{x\mid x\in B_{x_0}(\eta \alpha),\operatorname *{Pr}[F((x)_\Gamma) - F(x)\geq -\Phi ]\geq \omega \}$ . [46] demonstrates the following lemma:
Lemma B.2. If  $\| \nabla F(x_0)\| \leq \alpha$  and  $\mathrm{smin}(\nabla^2 F(x_0))\leq -\sqrt{\rho\gamma}$ , then the width of  $B^{\gamma}(x_0)$  along the along the minimum eigenvector of  $\nabla^2 F(x_0)$  is at most  $\frac{\omega\eta\gamma}{\log(1 / \omega)}\sqrt{\frac{2\pi}{d}}$ .
The intuition is that if two different points  $x^{1}, x^{2} \in B_{x_{0}}(\eta \alpha)$ , and  $x^{1} - x^{2}$  is large along the minimum eigenvector, then with high probability, the distance between  $\| (x^{1})_{\Gamma} - (x^{2})_{\Gamma} \|$  will be large, and either  $\| (x^{1})_{\Gamma} - x^{1} \|$  or  $\| (x^{2})_{\Gamma} - x^{2} \|$  is large, and hence either  $F(x^{1}) - F((x^{1})_{\Gamma})$  or  $F(x^{2}) - F((x^{2})_{\Gamma})$  is large. The Lemma 3.4 follows from Lemma B.2 by using the Gaussian  $\zeta_{2}$  to kick off the point.
B.3 Proof of Lemma 3.5
$$
F \left(x _ {t + 1}\right) - F \left(x _ {t}\right) \leq - \eta \| \nabla_ {t} \| ^ {2} / 6 \leq - \eta \gamma^ {2} / 6.
$$
Proof. By the assumption on smoothness, we know
$$
\begin{array}{l} F (x _ {t + 1}) \leq F (x _ {t}) + \langle \nabla F (x _ {t}), x _ {t + 1} - x _ {t} \rangle + \frac {M}{2} \| x _ {t + 1} - x _ {t} \| ^ {2} \\ = F \left(x _ {t}\right) - \eta / 2 \| \nabla_ {t} \| ^ {2} - \langle \nabla F (x _ {t}) - \nabla_ {t}, \eta \nabla_ {t} \rangle \\ \leq F (x _ {t}) + \eta \| \nabla F (x _ {t}) - \nabla_ {t} \| \cdot \| \nabla_ {t} \| - \frac {\eta}{2} \| \nabla_ {t} \| ^ {2}. \\ \end{array}
$$
By Lemma 3.3, with probability at least  $1 - \omega$ , for each  $t \in [T]$  we have  $\| \nabla F(x_t) - \nabla_t \|_2 \leq \gamma / 4$ . Hence we know if  $\nabla F(x_t) \geq \gamma$ , we have
B.4 Proof of Lemma 3.6
Lemma 3.6. Suppose  $\mathcal{O}_1$  and  $\mathcal{O}_2$  are  $\zeta_1$  and  $\zeta_2$  norm-subGaussian respectively. If one sets  $\gamma = O(1)\sqrt{(\zeta_2^2\kappa + 4\zeta_1^2)\cdot\log(Td / \omega)}$ , with probability at least  $1 - \omega$ , at least one point in the output set
$\{x_{1}, \dots, x_{T}\}$  of Algorithm 1 is  $\alpha$ -SOSP, where
Proof. By Lemma 3.5, we know if the gradient  $\| \nabla F(x_{t})\| \geq \gamma$ , then with high probability that  $F(x_{t + 1}) - F(x_{t})\leq -\eta \gamma^{2} / 6$ . By Lemma 3.4, if  $x_{t}$  is a saddle point (with small gradient norm but the Hessian has a small eigenvalue), then with high probability that  $F(x_{\Gamma +t}) - F(x_{t})\leq -\Omega (\frac{\gamma^{3 / 2}}{\sqrt{\rho}\log^3(\frac{dMB}{\rho\gamma\omega})})$ , and the function values decrease  $\Omega \left(\frac{\gamma^2}{M\log^4(\frac{dMB}{\rho\gamma\omega})}\right)$  on average for each step.
Recall the assumption that the risk is upper bounded by  $B$ , by our setting  $T = \Omega \left(\frac{BM}{\gamma^2} \log^4 \left(\frac{dMB}{\rho \gamma \omega}\right)\right)$ , the statement is proved.
B.5 Proof of Lemma 3.7
Lemma 3.7. Under the event that  $\| \nabla_t - \nabla F(x_t)\| \leq \gamma /4$  for all  $t\in [T]$  and our parameter settings, letting  $K = \{t\in [T]:\mathrm{drift}_t\geq \kappa \}$  be the set of iterations where the drift is large, we know  $|K|\leq O\bigl (\frac{B\eta}{\kappa} +T\gamma^{2}\eta^{2} / \kappa \bigr) = O\bigl (B\eta \log^4 (\frac{dMB}{\rho\gamma\omega}) / \kappa \bigr).$
Proof. By Lemma 3.5, if  $\| F(x_{t})\| \geq \gamma$ , we know  $F(x_{t + 1}) - F(x_{t})\leq -\eta \| \nabla_{t}\|^{2} / 6$ , and  $F(x_{t + 1}) - F(x_{t})\leq \eta \gamma^{2}$  otherwise. Index the items in  $K = \{t_1,t_2,\dots ,t_{|K|}\}$  such that  $t_i < t_{i + 1}$ . We know
$$
F (x _ {t _ {i + 1}}) - F (x _ {t _ {i}}) \leq - \frac {1}{6 \eta} \mathrm {d r i f t} _ {t _ {i + 1}} + (t _ {i + 1} - t _ {i}) \gamma^ {2} \eta \leq - \frac {1}{6 \eta} \kappa + (t _ {i + 1} - t _ {i}) \gamma^ {2} \eta .
$$
Recall by the assumption that  $\max_y F(y) - \min_x F(x) \leq B$ . And hence  $-B \leq F(x_{t_{|L|}}) - F(x_{t_1}) \leq -\frac{|K|}{6\eta} \kappa + T\gamma^2\eta$ , and we know
$$
| K | \leq O \left(\frac {B \eta}{\kappa} + T \gamma^ {2} \eta^ {2} / \kappa\right) = O \left(B \eta \log^ {4} \left(\frac {d M B}{\rho \gamma \omega}\right) / \kappa\right).
$$
The pseudocode of Algorithm 2 is stated below:
Algorithm 2 AboveThreshold
1: Input: A set of points  $\{x_{i}\}_{i = 1}^{T}$ , dataset  $S$ , parameters of objective function  $B, M, G, \rho$ , objective error  $\alpha$
2: Set  $\widehat{T}_1 = \alpha +\mathrm{Lap}\left(\frac{4G}{n\varepsilon}\right) + \frac{16\log(2T / \omega)G}{n\varepsilon},\widehat{T}_2 = -\sqrt{\rho\alpha} +\mathrm{Lap}\left(\frac{4M}{n\varepsilon}\right) - \frac{16\log(2T / \omega)M}{n\varepsilon}$
3: for  $i = 1, \dots, T$  do
4: if  $\| \nabla F_S(x_i)\| +\mathrm{Lap}(\frac{8G}{n\varepsilon})\leq \widehat{T}_1\bigwedge \mathrm{smin}(\nabla^2 F_S(x_i)) + \mathrm{Lap}(\frac{8M}{n\varepsilon})\geq \widehat{T}_2$  then
5: Output:  $x_{i}$
6: Halt
7: end if
8: end for
Theorem 4.2 (Empirical). For  $\varphi = O(1)$ , use Equation (2) as gradient oracles with  $\kappa = \frac{G^{4/3}B^{1/3}}{M^{5/3}}\left(\frac{\sqrt{d}}{n\sqrt{\varphi}}\right)^{2/3}$ ,  $\sigma_{1} = \frac{G\sqrt{B\eta/\kappa}\log^{2}(dB/\omega)}{n\sqrt{\varphi}}$ ,  $\sigma_{2} = \frac{M\sqrt{BM/\alpha_{1}^{2}}\log^{5}(dB/\omega)}{n\sqrt{\varphi}}$ . Running Algorithm 1, outputting the set  $\{x_{i}\}_{i\in [T]}$  if the total time to query  $\mathcal{O}_{1}$  is bounded by  $O(B\eta\log^{4}\left(\frac{dB}{\rho\gamma\omega}\right)/\kappa)$ ,
otherwise outputting a set of  $T$  arbitrary points is  $(\varphi /2)$ - $zCDP$ . With probability at least  $1 - \omega$ , at least one point in the output set is  $\alpha_{1}$ - $SOSP$  of  $F_{\mathcal{D}}$  with
Proof. The privacy guarantee can be proved by zCDP composition (Lemma A.3) Gaussian Mechanism (Theorem A.5) and Lemma 3.7. Specifically, by Assumption 3.1 and our settings of parameters, we know the sensitivity of  $\mathcal{O}_1$  and  $\mathcal{O}_2$  are bounded by  $\frac{G}{n}$  and  $\frac{M\|x - y\|}{n}$  respectively, and querying  $\mathcal{O}_1$  and  $\mathcal{O}_2$  each time are  $\frac{G^2}{n^2\sigma_1^2}$ -zCDP and  $\frac{M^2}{n^2\sigma_2^2}$ -zCDP respectively. We can apply the advanced composition to prove the privacy guarantee of the whole algorithm. As the total number of iterations  $T$  is fixed, all queries made to  $\mathcal{O}_2$  satisfy  $\frac{T M^2}{n^2\sigma_2^2}$ -zCDP. It suffices to bound the total time to query  $\mathcal{O}_1$ , which is guaranteed in the statement. That is the total times to query  $\mathcal{O}_1$  is  $T_1 := O\big(B\eta \log^4\big(\frac{dMB}{\rho\omega}\big) / \kappa \big)$ , so the queries to  $\mathcal{O}_1$  satisfy  $\frac{T_1G^2}{n^2\sigma_1^2}$ -zCDP. If the time exceeds  $O\big(B\eta \log^4\big(\frac{dMB}{\rho\omega}\big) / \kappa \big)$ , then we will output a set of arbitrary points which does not occur additional privacy cost. The overall privacy guarantee is  $(\frac{G^2T_1}{n^2\sigma_1^2} + \frac{M^2T}{n^2\sigma_2^2})$ -zCDP. To get  $(\varphi / 2)$ -zCDP, as desired we can choose:
$$
\sigma_ {1} = \frac {2 G \sqrt {T _ {1}}}{n \sqrt {\varphi}} = \frac {2 G \log^ {2} (\frac {d M B}{\rho \omega}) \sqrt {B \eta / \kappa}}{n \sqrt {\varphi}}, \qquad \sigma_ {2} = \frac {2 M \sqrt {T}}{n \sqrt {\varphi}} = \frac {2 M \log^ {5} (\frac {d M B}{\rho \omega}) \sqrt {B M}}{n \sqrt {\varphi} \alpha_ {1}}.
$$
As for the utility, we know the  $\mathcal{O}_1$  and  $\mathcal{O}_2$  constructed in Equation (2) are first kind of  $\sigma_1\sqrt{d}$  and second kind of  $\sigma_2\sqrt{d}$  norm-subGaussian gradient oracle by Fact 2.3. Hence by Lemma 3.6, the utility  $\alpha_{1}$  satisfies that
$$
\begin{array}{l} \alpha_ {1} = O \left(\sigma_ {1} \sqrt {d} + \sigma_ {2} \sqrt {d \kappa}\right) \cdot \log^ {3} (B M d / \rho \omega) \\ = O \Big (\frac {G \sqrt {d B \eta / \kappa}}{n \sqrt {\varphi}} + \frac {M \log^ {3} (\frac {d M B}{\rho \omega}) \sqrt {B M}}{n \sqrt {\varphi} \alpha_ {1}} \sqrt {d \kappa} \Big) \cdot \log^ {5} (B M d / \rho \omega). \\ \end{array}
$$
By Lemma 3.7, with probability at least  $1 - \omega$ , the total time to query  $\mathcal{O}_1$  is controlled and the final output will not be arbitrary points. Choosing the best  $\kappa$  demonstrates the bound on  $\alpha_{1}$ . The bound for  $\alpha_{2}$  follows from the value of  $\alpha_{1}$  and Lemma 4.1. Combining the two items in Lemma 4.1, we know with probability at least  $1 - \omega$ , the output point  $x$  of Algorithm 2 satisfies that
$$
\| \nabla F _ {\mathcal {D}} (x) \| \leq \alpha_ {1} + \frac {3 2 \log (2 T / \omega) G}{n \sqrt {\varphi}}, \text {a n d} \operatorname {s m i n} (\nabla^ {2} F _ {\mathcal {D}} (x)) \geq - \sqrt {\rho \alpha_ {1}} - \frac {3 2 \log (2 T / \omega) M}{n \sqrt {\varphi}}.
$$
Hence we know  $x$  is an  $\alpha_{2}$ -SOSP for  $\alpha_{2}$  stated in the statement.
C.2 Proof of Claim 4.4
Proof. For the oracle  $\mathcal{O}_1$ , we know for each  $z \in S_1$ ,  $\mathbb{E}_{z \sim \mathcal{P}}[\nabla f(x,z)] = \nabla F_{\mathcal{P}}(x)$  and  $\nabla f(x,z) - \nabla F_{\mathcal{P}}(x)$  is nSG(L) due to the Lipschitzness assumption. The statement follows from Fact 2.3 and Lemma 2.4. As for the  $\mathcal{O}_2$ , the statement follows similarly with the smoothness assumption.
C.3 Proof of Lemma 4.5
Proof. As for any  $s \in S$ ,  $\nabla f(x; s) - \nabla F_{\mathcal{P}}(x)$  is zero-mean nSG( $G$ ). Then the Hoeffding inequality for norm-subGuassians (Lemma 2.4) demonstrates with probability at least  $1 - \omega / 2$ , we have  $\| \nabla F_S(x) - \nabla F_{\mathcal{P}}(x) \|_2 \leq O\left(\frac{G \log(d / \omega)}{\sqrt{m}}\right)$ .
As for the other term, we know for any  $s \in S$ ,  $\mathbb{E}[\nabla^2 f(x;s) - \nabla^2 F_{\mathcal{P}}(x)] = 0$ , and  $\| \nabla^2 f(x;s) - \nabla^2 F_{\mathcal{P}}(x)\|_{op} \leq 2M$  almost surely. Hence applying Matrix Bernstein inequality (Theorem A.6) with  $\sigma^2 = 4M^2 m, t = O\big(\sqrt{m} M\log (d / \omega)\big)$ , we know with probability at least  $1 - \omega /2$ ,  $\| \nabla^2 F_S(x) - \nabla^2 F_{\mathcal{P}}(x)\|_{op} \leq t / m$ .
Applying the Union bound completes the proof.
C.4 Proof of Theorem 4.6
Theorem 4.6 (Population). Divide the dataset  $\mathcal{D}$  into two disjoint datasets  $\mathcal{D}_1$  and  $\mathcal{D}_2$  of size  $\lceil n/2 \rceil$  and  $\lfloor n/2 \rfloor$  respectively. Set  $b_1 = \frac{n\kappa}{B\eta}, b_2 = \frac{n\alpha_1^2}{BM}, \sigma_1 = \frac{G}{b_1\sqrt{\varphi}}, \sigma_2 = \frac{M}{b_2\sqrt{\varphi}}$  and  $\kappa = \max \left( \frac{G^{4/3} B^{1/3} \log^{1/3} d}{M^{5/3}} n^{-1/3}, \left( \frac{GB^{2/3}}{M^{5/3}} \right)^{6/7} \left( \frac{\sqrt{d}}{n\sqrt{\varphi}} \right)^{4/7} \right)$  in Equation (3) and use them as gradient oracles. Running Algorithm 1 with  $\mathcal{D}_1$  and outputting the set  $\{x_i\}_{i \in [T]}$  if the total time to query  $\mathcal{O}_1$  is bounded by  $O(B\eta \log^4 \left( \frac{dMB}{\rho\gamma\omega} \right) / \kappa)$ , otherwise outputting a set of  $T$  arbitrary points, is  $(\varphi/2) - zCDP$  and with probability at least  $1 - \omega$ , at least one point in the output is  $\alpha_1$ -SOSP of  $F_{\mathcal{P}}$  with
$$
\alpha_ {1} = O \Big ((B G M \cdot \log d) ^ {1 / 3} \frac {1}{n ^ {1 / 3}} + (G ^ {1 / 7} B ^ {3 / 7} M ^ {3 / 7}) (\frac {\sqrt {d}}{n \sqrt {\varphi}}) ^ {3 / 7} \Big) \log^ {3} (n B M d / \rho \omega) \Big).
$$
Proof. Recall that we draw the samples to construct the gradient oracles (Equation 3) without replacement, and we should have all samples to be fresh to avoid dependency, and hence we need
$$
b _ {1} \cdot | K | + b _ {2} \cdot T \leq n / 2,
$$
which is satisfied by the procedure in the statement, as if the total time to query the  $\mathcal{O}_1$  exceeds the threshold, the algorithm fails and outputs a set of arbitrary points. As we never reuse a sample, the privacy guarantee follows directly from the Gaussian Mechanism [20]. Specifically, the sensitivity of querying  $\mathcal{O}_1$  and  $\mathcal{O}_2$  are bounded by  $G / b_{1}$  and  $M\| x - y\| /b_2$  respectively, and querying  $\mathcal{O}_1$  and  $\mathcal{O}_2$  are  $(\varphi /2)$ -zCDP by Theorem A.5.
The Norm-subGaussian parameters of the oracles follow from Claim 4.4. By lemma 3.6, we have
$$
\begin{array}{l} \frac {\alpha_ {1}}{\log^ {3} (n B M d / \rho \omega)} \\ = O \left(\sigma_ {1} \sqrt {d} + \frac {G \sqrt {\log d}}{\sqrt {b _ {1}}} + \sigma_ {2} \sqrt {d \kappa} + \frac {M \sqrt {\kappa \log d}}{\sqrt {b _ {2}}}\right). \\ = O (\frac {G B \eta \sqrt {d}}{n \sqrt {\varphi} \kappa} + \frac {B M ^ {2}}{n \sqrt {\varphi} \alpha_ {1} ^ {2}} \sqrt {d \kappa} + \frac {G \sqrt {B \eta \log d}}{\sqrt {n \kappa}} + M \sqrt {\kappa} \frac {\sqrt {B M \log d}}{\sqrt {n} \alpha_ {1}}). \\ \end{array}
$$
Setting  $\kappa = \max \left(\frac{G^{4 / 3}B^{1 / 3}\log^{1 / 3}d}{M^{5 / 3}} (n)^{-1 / 3},\left(\frac{GB^{2 / 3}}{M^{5 / 3}}\right)^{6 / 7}\left(\frac{\sqrt{d}}{n\sqrt{\varphi}}\right)^{4 / 7}\right)$ , we get
$$
\alpha_ {1} = O \Big (\big ((B G M \log d) ^ {1 / 3} \frac {1}{n ^ {1 / 3}} + (G ^ {1 / 7} B ^ {3 / 7} M ^ {3 / 7}) (\frac {\sqrt {d}}{n \sqrt {\varphi}}) ^ {3 / 7} \big) \log^ {3} (n B M d / \rho \omega) \Big).
$$
Then we use the other half fresh samples  $\mathcal{D}_2$  to find the point in the set by Algorithm 2. By Lemma 4.1 and Lemma 4.5, we know with probability at least  $1 - \omega$ , for some large enough constant  $C > 0$ , the output point  $x$  of Algorithm 2 satisfies that
$$
\left\| \nabla F _ {\mathcal {P}} (x) \right\| _ {2} \leq \alpha_ {1} + G \left(\frac {3 2 \log (2 T / \omega)}{n \sqrt {\varphi}} + \frac {C \log (d n / \omega)}{\sqrt {n}}\right),
$$
$$
\operatorname {s m i n} \left(\nabla^ {2} F _ {\mathcal {P}} (x)\right) \geq - \sqrt {\rho \alpha_ {1}} - M \left(\frac {3 2 \log (2 T / \omega)}{n \sqrt {\varphi}} + \frac {C \log (d n / \omega)}{\sqrt {n}}\right)
$$
Hence we know  $x$  is an  $\alpha_{2}$ -SOSP for  $\alpha_{2}$  stated in the statement. The privacy guarantee follows from Lemmas A.3, A.4, and 4.1.
D Omitted proof of Section 5
D.1 Proof of Lemma 5.5
Proof. We know how to bound the KL divergence by LSI:
$$
\begin{array}{l} K L \left(\pi_ {\mathcal {D}}, \pi_ {\mathcal {D} ^ {\prime}}\right) := \int \log \frac {\mathrm {d} \pi_ {\mathcal {D}}}{\mathrm {d} \pi_ {\mathcal {D} ^ {\prime}}} \mathrm {d} \pi_ {\mathcal {D}} \\ \leq \frac {C _ {\mathrm {L S I}}}{2} \underset {\pi_ {\mathcal {D}}} {\mathbb {E}} \left\| \nabla \log \frac {\mathrm {d} \pi_ {\mathcal {D}}}{\mathrm {d} \pi_ {\mathcal {D} ^ {\prime}}} \right\| _ {2} ^ {2} \\ \leq 2 C _ {\mathrm {L S I}} G ^ {2} \beta^ {2} / n ^ {2}. \\ \end{array}
$$
LSI can lead to Talagrand transportation inequality [Theorem 1 in [40]], i.e.,
$$
W _ {2} \left(\pi_ {\mathcal {D}}, \pi_ {\mathcal {D} ^ {\prime}}\right) \lesssim \sqrt {C _ {\mathrm {L S I}} \cdot K L \left(\pi_ {\mathcal {D}} , \pi_ {\mathcal {D} ^ {\prime}}\right)} = C _ {\mathrm {L S I}} G \beta / n.
$$
The generalization error is bounded by  $O(C_{\mathrm{LSI}}G^2\beta /n)$ . Using Holley-Stroock perturbation, we know  $C_{\mathrm{LSI}}(\pi_{\mathcal{D}})\leq \frac{\exp(\beta GD)}{\beta\mu}$  and hence the  $W_{2}$  distance between  $\pi_{\mathcal{D}}$  and  $\pi_{\mathcal{D}'}$  can be bounded by  $O(\frac{G\exp(\beta GD)}{n\mu})$ . The statement follows the Lipschitzness constant and Lemma 5.4.
D.2 Proof of Theorem 5.6
Theorem 5.6 (Risk bound). We are given  $\varepsilon, \delta \in (0,1/2)$ . Sampling from  $\exp(-\beta(F_{\mathcal{D}}(x) + \frac{\mu}{2}\|x\|_2^2))$  with  $\beta = O\left(\frac{\varepsilon\log(nd)}{GD\sqrt{\log(1/\delta)}}\right), \mu = \frac{d}{D^2\beta}$  is  $(\varepsilon, \delta)$ -DP. The empirical risk and population risk are bounded by  $O(GD\frac{d\cdot\log\log(n)\sqrt{\log(1/\delta)}}{\varepsilon\log(nd)})$ .
Proof. Denote  $\pi(x) \propto \exp(-\beta(F_{\mathcal{D}}(x) + \frac{\mu}{2}\|x\|_2^2))$ . By Lemma 5.2, we know  $C_{\mathrm{LSI}}(\pi) \leq \frac{1}{\beta\mu} \cdot \exp(\beta GD)$ . Plugging in the parameters and applying Theorem 5.1, we get
$$
\frac {2 G \beta}{n} \cdot \sqrt {\frac {\exp (\beta G D)}{\beta \mu}} \sqrt {3 \log (1 / \delta)} = O (1) \frac {G D \beta}{n \sqrt {d}} \sqrt {\exp (\beta G D) \log (1 / \delta)} \leq 1
$$
and hence prove the privacy guarantee.
As for the empirical risk bound, by Lemma 5.3, we know
$$
\underset {x \sim \pi} {\mathbb {E}} \left(F _ {\mathcal {D}} (x) + \frac {\mu}{2} \| x \| _ {2} ^ {2}\right) - \underset {x ^ {*} \in \mathcal {K}} {\min } \left(F _ {\mathcal {D}} \left(x ^ {*}\right) + \frac {\mu}{2} \| x ^ {*} \| _ {2} ^ {2}\right) \lesssim \frac {d \log (\beta G D / d)}{\beta},
$$
and we know
$$
\underset {x \sim \pi} {\mathbb {E}} F _ {\mathcal {D}} (x) - \underset {x ^ {*} \in \mathcal {K}} {\min } F _ {\mathcal {D}} (x ^ {*}) \lesssim \frac {d \log (\beta G D / d)}{\beta} + \mu D ^ {2}.
$$
Replacing the value of  $\beta$  achieves the empirical risk bound.
As for the population risk, we have
$$
\begin{array}{l} \underset {x \sim \pi} {\mathbb {E}} F _ {\mathcal {P}} (x) - \min  _ {y ^ {*} \in \mathcal {K}} F _ {\mathcal {P}} (y ^ {*}) \\ = \underset {x \sim \pi} {\mathbb {E}} \left[ F _ {\mathcal {P}} (x) - F _ {\mathcal {D}} (x) \right] + \mathbb {E} \left[ F _ {\mathcal {D}} (x) - \min  _ {x ^ {*} \in \mathcal {K}} F _ {\mathcal {D}} \left(x ^ {*}\right) \right] + \mathbb {E} \left[ \min  _ {x ^ {*} \in \mathcal {D}} F _ {\mathcal {D}} \left(x ^ {*}\right) - \min  _ {y ^ {*} \in \mathcal {K}} F _ {\mathcal {P}} \left(y ^ {*}\right) \right] \\ \leq \underset {x \sim \pi} {\mathbb {E}} \left[ F _ {\mathcal {P}} (x) - F _ {\mathcal {D}} (x) \right] + \mathbb {E} \left[ F _ {\mathcal {D}} (x) - \min  _ {x ^ {*} \in \mathcal {K}} F _ {\mathcal {D}} \left(x ^ {*}\right) \right]. \\ \end{array}
$$
We can bound  $\mathbb{E}_{x\sim \pi}[F_{\mathcal{P}}(x) - F_{\mathcal{D}}(x)]\leq O(\frac{G^2\exp(\beta GD)}{n\mu})\leq O(\frac{GD\varepsilon\log(n)}{n^{1 - c}d\sqrt{\log(1 / \delta)}})$  by Lemma 5.5 for an arbitrarily small constant  $c > 0$ . Hence the empirical risk is dominated term compared to  $\mathbb{E}_{x\sim \pi}[F_{\mathcal{P}}(x) - F_{\mathcal{D}}(x)]$ , and we complete the proof.
D.3 Implementation
We rewrite them below: Let  $\widehat{F} (x)\coloneqq F(x) + r(x)$  where  $r(x)$  is some regularizer, and  $F = \mathbb{E}_{i\in I}f_{i}$  is the expectation of a family of  $G$  -Lipschitz functions.
Algorithm 3 AlternateSample, [35]
1: Input: Function  $\widehat{F}$ , initial point  $x_0 \sim \pi_0$ , step size  $\eta$
2: for  $t \in [T]$  do
3:  $y_{t}\gets x_{t - 1} + \sqrt{\eta}\zeta$  where  $\zeta \sim \mathcal{N}(0,I_d)$
4: Sample  $x_{t}\gets \exp (-\widehat{F} (x) - \frac{1}{2\eta}\| x - y_{t}\|_{2}^{2})$
5: end for
6: Output:  $x_{T}$
Theorem D.1 (Guarantee of Algorithm 3, [15]). Let  $\mathcal{K} \subset \mathbb{R}^d$  be a convex set of diameter  $D$ , and  $\widehat{F}: \mathcal{K} \to \mathbb{R}$ , and  $\pi \propto \exp(-\widehat{F})$  satisfies LSI with constant  $C_{\mathrm{LSI}}$ . Then set  $\eta \geq 0$ , we have
$$
R _ {q} (\pi_ {t}, \pi) \leq \frac {R _ {q} (\pi_ {0} , \pi)}{(1 + \eta / C _ {\mathrm {L S I}}) ^ {2 t / q}},
$$
where  $R_{q}(\pi^{\prime},\pi)$  is the q-th order of Renyi divergence between  $\pi^\prime$  and  $\pi$
To get a sample from  $\exp(-\widehat{F}(x) - \frac{1}{2\eta} \| x - y_t\|_2^2)$ , we use the rejection sampler from [26], whose guarantee is stated below:
Lemma D.2 (Rejection Sampler, [26]). If the step size  $\eta \lesssim G^{-2}\log^{-1}(1 / \delta_{inner})$  and the inner accuracy  $\delta_{inner} \in (0,1/2)$ , there is an algorithm that can return a random point  $x$  that has  $\delta_{inner}$  total variation distance to the distribution proportional to  $\exp(-\widehat{F}(x) - \frac{1}{2\eta}\|x - y\|_2^2)$ . Moreover, the algorithm accesses  $O(1)$  different  $f_i$  function values and  $O(1)$  samples from the density proportional to  $\exp(-r(x) - \frac{1}{2\eta}\|x - y\|_2^2)$ .
Combining Theorem 5.6, Theorem D.1 and Lemma D.2, we can get the following implementation of the exponential mechanism for non-smooth functions.
Theorem 5.7 (Implementation, risk bound). For  $\varepsilon, \delta \in (0,1/2)$ , there is an  $(\varepsilon,2\delta)$ -DP efficient sampler that can achieve the empirical and population risks  $O(GD\frac{d\cdot\log\log(n)\sqrt{\log(1/\delta)}}{\varepsilon\log(nd)})$ . Moreover, in expectation, the sampler takes  $\tilde{O}\left(n\varepsilon^3\log^3(d)\sqrt{\log(1/\delta)}/(GD)\right)$  function values query and some Gaussian random variables restricted to the convex set  $\mathcal{K}$  in total.
Proof. By Theorem 5.6, it suffices to get a good sample from  $\pi$  with density proportional to  $\exp(-\beta(F_{\mathcal{D}}(x) + \frac{\mu}{2}\|x\|_2^2))$  where  $\beta = O\left(\frac{\varepsilon\log(nd)}{GD\sqrt{\log(1/\delta)}}\right)$ ,  $\mu = \frac{d}{D^2\beta}$ . Set  $q = 1$ , which gives that  $R_q(\cdot,\cdot)$  is the KL-divergence. Suppose we let  $x_0$  be drawn from density proportional to  $\exp(-\frac{\beta}{2}\mu\|x\|_2^2)$ , then the KL divergence between  $\pi_0$  and  $\pi$  is bounded by  $\exp(q\beta GD)$ .
Now let  $\pi_T^{(i)}$  be the distribution we get over  $x_{T}$  from Algorithm 3 if we use an exact sampler for  $i$  iterations, then the sampler of Lemma D.2 for the remaining  $T - i$  iterations. The output of Algorithm 3 that we actually get is  $\pi_T^{(0)}$ . Note that  $C_\mathrm{LSI} \leq D^2 n$ , and  $\eta \lesssim \beta^{-2} G^{-2} \log^{-1}(2T / \delta)$ . Setting
$$
T = O \left(\frac {C _ {\mathrm {L S I}}}{\eta} \log (\exp (q \beta G D) / \delta^ {2})\right) = \tilde {O} \left(\frac {n \varepsilon^ {3} \log^ {3} (d) \sqrt {\log (1 / \delta)}}{G D}\right)
$$
we get  $\delta_{inner} = \delta /2T$  in Lemma D.2 and that  $R_{1}(\pi_{T}^{(T)},\pi)\leq \delta^{2} / 8$ . This implies the total variation distance between  $\pi_T^{(T)}$  and  $\pi$  is at most  $\delta /2$  by Pinsker's inequality. Furthermore, by the postprocessing inequality, the total variation distance between  $\pi_T^{(i)}$  and  $\pi_T^{(i + 1)}$  is at most  $\delta /2T$  for all  $i$ . Then by triangle inequality the total variation distance between  $\pi_T^{(0)}$  and  $\pi$  is at most  $\delta$ .
D.4 Proof of Theorem 5.8
Proof. We pick a maximal packing  $P$  of  $O((D / r)^d)$  points, such that every point in  $\mathcal{K}$  is distance at most  $r$  from some point in  $P$ . By  $G$ -Lipschitzness, the risk of any point in  $P$  for the DP-ERM/SCO problems over  $\mathcal{K}$  are at most  $Gr$  plus the risk of the same point for DP-ERM/SCO over  $P$ . The exponential mechanism over  $P$  gives a DP-ERM risk bound of  $O\left(\frac{GD}{\varepsilon n}\log |P|\right)$ . Next, note that the empirical loss of each point in  $P$  is the average of  $n$  random variables in  $[0,GD]$  wlog. So, the expected maximum difference between the empirical and population loss of any point in  $P$  is  $O\left(\frac{GD\sqrt{\log|P|}}{\sqrt{n}}\right)$ . Putting it all together we get a DP-SCO expected risk bound of:
$$
O \left(G r + G D \left(\frac {d \log (D / r)}{\varepsilon n} + \frac {\sqrt {d \log (D / r)}}{\sqrt {n}}\right)\right).
$$
This is approximately minimized by setting  $r = Dd / \varepsilon n$ . This gives a bound of:
$$
O \left(G D \left(\frac {d \log (\varepsilon n / d)}{\varepsilon n} + \frac {\sqrt {d \log (\varepsilon n / d)}}{\sqrt {n}}\right)\right).
$$
E Conclusion