SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We present a novel framework that can improve upon the state-of-the-art rates for locating second-order stationary points for both empirical and population risks. We also examine the utilization of the exponential mechanism to attain favorable excess risk bounds for both a polynomial time sampling approach and an exponential time sampling approach. Despite the progress made, several interesting questions remain. There is still a gap between the upper and lower bounds for finding stationary points. As noted in [2], it is quite challenging to beat the current  $\left(\frac{\sqrt{d}}{n}\right)^{2/3}$  empirical upper bound, and overcoming this challenge may require the development of new techniques. A potential avenue for improving the population rate for SOSP could be combining our drift-controlled framework with the tree-based private SpiderBoost algorithm in [2]. Additionally, it is worth exploring if it is possible to achieve better excess risk bounds within polynomial time, and what the optimal risk bound could be.
F Extended related work
In the convex setting, it is feasible to achieve efficient algorithms with good risk guarantees. In turn, differentially private empirical risk minimization (DP-ERM) [13, 14, 17, 28, 33, 9, 43, 41, 42] and differentially private stochastic optimization [4, 7, 6, 22, 34, 3, 32, 26, 23, 12, 27] have been two of
the most extensively studied problems in the DP literature. Most common approaches are variants of DP-SGD [14] or the exponential mechanism [38].
As for the non-convex optimization, due to the intrinsic challenges in minimizing general non-convex functions, most of the previous works [49, 50, 47, 46, 56, 42, 44, 54, 2, 51, 24] adopted the gradient norm as the accuracy metric rather than risk. Instead of minimizing the gradient norm discussed before, [8] tried to minimize the stationarity gap of the population function privately, which is defined as  $\mathrm{Gap}_{F_{\mathcal{P}}}(x) \coloneqq \max_{y \in \mathcal{K}} \langle \nabla F_{\mathcal{P}}(x), x - y \rangle$ , which requires  $\mathcal{K}$  to be a bounded domain. There are also some different definitions of the second order stationary point. We refer the readers to [37] for more details.
The risk bound achieved by algorithms with polynomial running time is weak and requires  $n \gg d$  to be meaningful. Many previous works consider minimizing risks of non-convex functions under stronger assumptions, such as, Polyak-Lojasiewicz condition [49, 55], Generalized linear model (GLM) [46] and weakly convex functions [8].
In the (non-private) classic stochastic optimization, there is a long line of influential works on finding the first and second-order stationary points for non-convex functions, [1, 29, 21, 53, 18].
First order stationary points. Progress towards privately finding a first-order stationary point is measured in (i) the norm of the empirical gradient at the solution  $x$ , i.e.,  $\| \nabla F_{\mathcal{D}}(x)\|$ , and (ii) the norm of the population gradient, i.e.,  $\| \nabla F_{\mathcal{P}}(x)\|$ . We summarize compare these first-order guarantees achieved by Algorithm 1 with previous algorithms in Table 2:
References	Empirical	Population
[49]	d1/4/n	N/A
[47]	d1/4/n	√d/n
[50]	(√d/n)2/3	N/A
[56]	d1/4/n	d1/4/n
[44]	1/√n + (√d/n)2/3	N/A
[2]	(√d/n)2/3	1/n1/3 + (√d/n)1/2
Second order stationary points. We say a point  $x$  is a Second-Order Stationary Point (SOSP), or a local minimum of a twice differentiable function  $g$  if  $\| \nabla g(x)\| _2 = 0$  and  $\mathrm{smin}(\nabla^2 g(x))\geq 0$ . Exact second-order stationary points can be extremely challenging to find [25]. Instead, it is common to measure the progress in terms of how well the solution approximates an SOSP.
Definition F.1 (approximate-SOSP, [1]). We say  $x \in \mathbb{R}^d$  is an  $\alpha$ -second order stationary point ( $\alpha$ -SOSP) for  $\rho$ -Hessian Lipschitz function  $g$ , if
$$
\| \nabla g (x) \| _ {2} \leq \alpha \bigwedge \operatorname {s m i n} \left(\nabla^ {2} g (x)\right) \geq - \sqrt {\rho \alpha}.
$$
Table 2: Previous work in finding first-order stationary points. We omit logarithmic terms and dependencies on other parameters such as Lipschitz constant. "N/A" means we do not find an explicit result in the work.
References	Empirical	Population
[46]	d1/4/n√n	N/A
[48]	(d/n)4/7	N/A
[24]	(√dn)1/2	N/A
Ours	(√dn)2/3	1/n1/3 + (√dn)3/7
Table 3: Summary of previous results in finding  $\alpha$ -SOSP, where  $\alpha$  is demonstrated in the Table. Omit the logarithmic terms and the dependencies on other parameters like Lipschitz constant. "N/A" means we do not find an explicit result in the work.
Existing works in finding approximate SOSP privately give guarantees for the empirical function  $F_{\mathcal{D}}$ . We improve upon the state-of-the-art result and give the first guarantee for the population function  $F_{\mathcal{P}}$ , which is summarized in Table 3.