SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Daogao Liu
University of Washington*
dgliu@uw.edu
University of Washington and Google Research sewoong@cs.washington.edu
We reconsider the challenge of non-convex optimization under differential privacy constraint. Building upon the previous variance-reduced algorithm SpiderBoost, we propose a novel framework that employs two types of gradient oracles: one that estimates the gradient at a single point and a more cost-effective option that calculates the gradient difference between two points. Our framework can ensure continuous accuracy of gradient estimations and subsequently enhances the rates of identifying second-order stationary points. Additionally, we consider a more challenging task by attempting to locate the global minima of a nonconvex objective via the exponential mechanism without almost any assumptions. Our preliminary results suggest that the regularized exponential mechanism can effectively emulate previous empirical and population risk bounds, negating the need for smoothness assumptions for algorithms with polynomial running time. Furthermore, with running time factors excluded, the exponential mechanism demonstrates promising population risk bound performance, and we provide a nearly matching lower bound.
Differential privacy [19] is a standard privacy guarantee for training machine learning models. Given a randomized algorithm  $\mathcal{A}: P^* \to R$ , where  $P$  is a data domain and  $R$  is a range of outputs, we say  $\mathcal{A}$  is  $(\varepsilon, \delta)$ -differentially private (DP) for some  $\varepsilon \geq 0$  and  $\delta \in [0,1]$  if for any neighboring datasets  $\mathcal{D}, \mathcal{D}' \in P^*$  that differ in at most one element and any  $\mathcal{R} \subseteq R$ , the distribution of the outcome of the algorithm, e.g., pair of models trained on the respective datasets, are similar:
$$
\operatorname * {P r} _ {x \sim \mathcal {A} (\mathcal {D})} [ x \in \mathcal {R} ] \leq e ^ {\varepsilon} \operatorname * {P r} _ {x \sim \mathcal {A} (\mathcal {D} ^ {\prime})} [ x \in \mathcal {R} ] + \delta .
$$
Smaller  $\varepsilon$  and  $\delta$  imply the distributions are closer; hence, an adversary accessing the trained model cannot tell with high confidence whether an example  $x$  was in the training dataset. Given this measure of privacy, we consider the problem of optimizing a non-convex loss while ensuring a desired level of privacy. In particular, suppose we are given a dataset  $\mathcal{D} = \{z_1,\ldots ,z_n\}$  drawn i.i.d. from underlying distribution  $\mathcal{P}$ . Each loss function  $f(\cdot ;z):\mathcal{K}\to \mathbb{R}$  is  $G$ -Lipschitz over the convex set  $\mathcal{K}\subset \mathbb{R}^d$  of diameter  $D$ . Let the population risk function be  $F_{\mathcal{P}}(x)\coloneqq \mathbb{E}_{z\sim \mathcal{P}}[f(x;z)]$  and the empirical risk function be  $F_{\mathcal{D}}(x)\coloneqq \frac{1}{n}\sum_{z\in \mathcal{D}}f(x;z)$ . We also denote  $F_{S}(x)\coloneqq \frac{1}{|S|}\sum_{z\in S}f(x;z)$  for  $S\subseteq \mathcal{D}$ .
*Most of this work was done while the author was an intern at Google.
Our focus is in minimizing non-convex (empirical and population) risk functions, which may have multiple local minima. Since finding the global optimum of a non-convex function can be challenging, an alternative goal in the field is to find stationary points: A first-order stationary point is a point with a small gradient of the function, and a second-order stationary point is a first-order stationary point where additionally the function has a positive or nearly positive semi-definite Hessian. As first order stationary points can be saddle points or even a local maximum, we focus on the problem of finding a second order stationary point, i.e., a local minimum, privately. Existing works in finding approximate SOSP privately only give guarantees for the empirical function  $F_{\mathcal{D}}$ . We improve upon the state-of-the-art result for empirical risk minimization and give the first guarantee for the population function  $F_{\mathcal{P}}$ . This requires standard assumptions on bounded Lipschitzness, smoothness, and Hessian Lipschitzness, which we make precise in Section 2 and in Assumption 3.1.
Compared to finding a local minimum, finding a global minimum can be extremely challenging. We also present two methods, polynomial and exponential time, that outperform existing guarantees measured in excess risks for respective computational complexities. Our primary results are succinctly summarized in Table 1.
Related Work. We propose a novel and simple framework based on SpiderBoost [52], and its private version [2] that achieves the current best rate for finding the first order stationary point privately. We discuss the primary difference between our framework and theirs, that is their algorithms only promise small gradient estimation errors on average, but our framework can ensure small estimation errors consistently throughout all the iterations, and the motivation behind this briefly.
In SGD and its variants, the typical approach involves obtaining an estimation  $\Delta_t$  of the gradient  $\nabla f(x_t)$ . In the stochastic variance-reduced algorithm SpiderBoost [52, 2], it queries the gradient  $\mathcal{O}_1(x_t) \approx \nabla f(x_t)$  directly every  $q$  steps with some oracle  $\mathcal{O}_1$ , and for the other  $q - 1$  steps within each period, it queries the gradient difference between two steps, that is  $\mathcal{O}_2(x_t, x_{t-1}) \approx \nabla f(x_t) - \nabla f(x_{t-1})$ , and maintain  $\Delta_t = \Delta_{t-1} + \mathcal{O}_2(x_t, x_{t-1})$ . The contrast between these two types of oracles can be perceived as  $\mathcal{O}_1$  being more accurate but also more costly, in terms of computation or privacy budget, although our framework does not strictly necessitate this assumption.
As SpiderBoost queries  $\mathcal{O}_1$  every  $q$  steps, the error on the estimation may accumulate and  $\| \Delta_t - \nabla f(x_t)\|$  can become large. Despite this, as demonstrated in [2], these estimations can, on average, suffice to find a private FOSP. However, such large deviations pose a challenge when scrutinizing behavior near a saddle point. For instance, when the current point is a saddle point, but the current estimation is unsatisfactory, it becomes uncertain whether the algorithm can escape the saddle point. It could be argued that average good estimations could achieve a SOSP, but to the best of our knowledge, there is no existing result addressing this concern.
A plausible solution to this challenge is to maintain high-quality gradient estimations throughout all iterations, a feat accomplished by our framework. We believe this feature holds promise for improving the outcomes of various other optimization problems, thus enhancing the overall appeal and significance of our work.
1.1 Main Results
SOSP. One of our main contributions is a refined optimization framework (Algorithm 1), predicated on the variance-reduced SpiderBoost [52], which guarantees consistently accurate gradient estimations. By integrating this framework with private gradient oracles, we achieve improved error rates for privately identifying SOSP of both empirical and population risks.
Advances in private non-convex optimization have focused on finding a first-order stationary point (FOSP), whose performance is measured in  $(i)$  the norm of the empirical gradient at the solution  $x$ , i.e.,  $\| \nabla F_{\mathcal{D}}(x)\|$ , and  $(ii)$  the norm of the population gradient, i.e.,  $\| \nabla F_{\mathcal{P}}(x)\|$ . We survey the recent progress in the appendix in detail.
Definition 1.1 (First-order stationary point). We say  $x \in \mathbb{R}^d$  is a First-Order Stationary Point (FOSP) of  $g: \mathbb{R}^d \to \mathbb{R}$  iff  $\nabla g(x) = 0$ .  $x$  is an  $\alpha$ -FOSP of  $g$ , if  $\| \nabla g(x) \|_2 \leq \alpha$ .
Since FOSP can be a saddle point or a local maxima, finding a second-order stationary point is desired. Exact second-order stationary points can be extremely challenging to find [25]. Instead, progress is commonly measured in terms of how well the solution approximates an SOSP.
α-SOSP	Excess population risk
empirical	population	poly-time	exp-time
SOTA	min(d1/4n2ε1/2, d4/7ε4/7)	N/A	dε2 log n	N/A
Ours	d1/n2ε2/3	1/n1/3 + (√d/nε)3/7	d log log n/ε log(n)	d/nε + √d/n
LB	√d/nε	1/√n + √d/nε	d/nε + √d/n	d/nε + √d/n
Table 1: SOTA refers to the best previously known bounds on  $\alpha$  for  $\alpha$ -SOSP by [46, 48] and on the excess population risk by [46]. We introduce algorithm 1 that finds an  $\alpha$ -SOSP (columns 2-3) with an improved rate. We show exponential mechanism can minimize the excess risk in polynomial time and exponential time, respectively (columns 4 and 5).  $\spadesuit$  requires extra assumption on bounded smoothness. The lower bounds for SOSP are from [2], and the lower bound on excess population risk is from Theorem 5.11. We omit logarithmic factors in  $n$  and  $d$  except the upper bounds for excess population risk with polynomial time.
Definition 1.2 (Second-order stationary point, [1]). We say a point  $x \in \mathbb{R}^d$  is a Second-Order Stationary Point (SOSP) of a twice differentiable function  $g: \mathbb{R}^d \to \mathbb{R}$  iff  $\| \nabla g(x) \|_2 = 0$  and  $\nabla^2 g(x) \succeq 0$ . We say  $x \in \mathbb{R}^d$  is an  $\alpha$ -SOSP for  $\rho$ -Hessian Lipschitz function  $g$ , if  $\| \nabla g(x) \|_2 \leq \alpha \wedge \nabla^2 g(x) \succeq -\sqrt{\rho \alpha} I$ .
On the empirical risk  $F_{\mathcal{D}}$ , the SOTA on privately finding  $\alpha$ -SOSP is by [46, 48], which achieves  $\alpha = \tilde{O}(\min\{(\sqrt{d}/n)^{1/2}, (d/n)^{4/7}\})$ . In Theorem 4.2, we show that applying the proposed Algorithm 1 achieves a rate bounded by  $\alpha = \tilde{O}((\sqrt{d}/n)^{2/3})$ , which improves over the SOTA in all regime. There remains a factor  $(\sqrt{d}/n)^{-1/6}$  gap to a known lower bound of  $\alpha = \Omega(\sqrt{d}/n)$  that holds even if finding only an  $\alpha$ -FOSP [2]. On the population risk  $F_{\mathcal{P}}$ , applying Algorithm 1 with appropriate private gradient oracles is the first private algorithm to guarantee finding an  $\alpha$ -SOSP with  $\alpha = \tilde{O}(n^{-1/3} + (\sqrt{d}/n)^{3/7})$  in Theorem 4.6. There is a gap to a known lower bound of  $\alpha = \Omega(1/\sqrt{n} + \sqrt{d}/n\varepsilon)$  that holds even if finding only an  $\alpha$ -FOSP [2].
Minimizing Excess Risk. In addition to the optimization framework, we present sampling-based algorithms designed to identify a private solution  $x^{priv} \in \mathbb{R}^d$  that minimizes both the excess empirical risk:  $\mathbb{E}[F_{\mathcal{D}}(x^{priv})] - \min_{x \in \mathcal{K}} F_{\mathcal{D}}(x)$ , and the excess population risk:  $\mathbb{E}[F_{\mathcal{P}}(x^{priv})] - \min_{x \in \mathcal{K}} F_{\mathcal{P}}(x)$ . Here, the expectation is over the randomness of the solution  $x^{priv}$  and the drawing of the training date over  $\mathcal{P}$ . Our method is different from [46], which Gradient Langevin Dynamics and achieves in polynomial time a bound of  $O(d\sqrt{\log(1/\delta)} / (\varepsilon^2\log n))$  for both excess empirical and population risks with a need for the smoothness assumption. In Table 1 we omit excess empirical risk, as the bounds align with those of the population risk. We introduce a sampling-based algorithm from the exponential mechanism, which runs in polynomial time and achieves excess empirical and population risks bounded by  $O(d\sqrt{\log(1/\delta)} / (\varepsilon\log(nd)))$  with improved dependence on  $\varepsilon$  (Theorem 5.6). Crucially, it achieves these results without the need for the smoothness assumption required by [46].
In the case of permitting an exponential running time, [23] demonstrated  $\tilde{O}(d / (\varepsilon n))$  upper bound for non-convex excess empirical risks alongside a nearly matching lower bound. However, establishing a tight bound for the excess population risk remained an unresolved problem. We address this open question by providing nearly matching upper and lower bounds of  $\tilde{\Theta}(d / (\varepsilon n) + \sqrt{d / n})$  for the excess population risk (Theorem 5.8).
Stationary Points. In our framework, we deviate from the traditional approach of querying  $\mathcal{O}_1$  once every  $q$  steps. Instead, we introduce a novel but simple method of monitoring the total drift we make, that is  $\mathrm{drift}_t = \sum_{i = \tau_t}^t\| x_i - x_{i - 1}\| _2^2$ , where  $\tau_{t}$  represents the last timestamp when we employed  $\mathcal{O}_1$ . As we are considering smooth functions, the maximum error to estimate  $\nabla f(x_{t}) - \nabla f(x_{t - 1})$
2We want  $\alpha = o(1)$  and hence can assume  $d\leq n^2$
is proportional to  $\| x_{t} - x_{t - 1}\|_{2}$ . If the value  $\mathrm{drift}_t$  is small, we know the current estimation should still be good enough, eliminating the need for an expensive fresh estimation from  $\mathcal{O}_1$ . Conversely, when  $\mathrm{drift}_t$  is large, the gradient estimation error may be substantial, necessitating a query to  $\mathcal{O}_1$  and thus obtaining  $\Delta_t = \mathcal{O}_1(x_t)$ . To effectively manage the total cost, it is crucial to set an appropriate threshold to decide when the drift is significant. A smaller threshold would ensure more accurate estimations but might incur higher costs due to more frequent queries to  $\mathcal{O}_1$ .
Our aim is to bound the total occurrences of the event that  $\mathrm{drift}_t$  is large, which leads to querying  $\mathcal{O}_1$ . A crucial observation is that, if  $\mathrm{drift}_t$  increases rapidly, then the gradient norms are large and hence function values decrease quickly, which we know does not happen frequently under the standard assumption that the function is bounded.
In our framework, we assume  $\mathcal{O}_1(x)$  is an unbiased estimation of  $\nabla f(x)$ , and  $\mathcal{O}_1(x) - \nabla f(x)$  is Norm-SubGaussian (Definition 2.2), and similarly  $\mathcal{O}_2(x,y)$  is an unbiased estimation of  $\nabla f(x) - \nabla f(y)$  whose error is also Norm-SubGaussian. In the empirical case, we can simply add Gaussian noises with appropriately chosen variances to the gradients of the empirical function  $\nabla F_{\mathcal{D}}$  for simplicity, and one can choose a smaller batch size to reduce the computational complexity. In the population case, we draw samples from the dataset without replacement to avoid dependence issues, and add the Gaussian noises to the sampled gradients. Hence we only need the gradient oracle complexity to be linear in the size of dataset for the population case.
Minimizing Excess Risk. Our polynomial time approach harnesses the power of the Log-Sobolev Inequality (LSI) and the classic Stroock perturbation lemma. The previous work of [39] shows that if the density  $\exp \left(-\beta F_{\mathcal{D}}(x) - r(x)\right)$  satisfies the LSI for some regularizer  $r$ , then sampling a model  $x$  from this density is DP with an appropriate  $(\varepsilon, \delta)$ . If  $r$  is a  $\mu$  strongly convex function, then the density proportional to  $\exp (-r)$  satisfies LSI with constant  $1 / \mu$ , and  $\exp (-\beta F_{\mathcal{D}}(x) - r(x))$  satisfies LSI with constant  $\exp (\max_{x,y}|F_{\mathcal{D}}(x) - F_{\mathcal{D}}(y)|) / \mu$  by the Stroock perturbation lemma. Our bound on the empirical risk follows from choosing the appropriate inverse temperature  $\beta$  and regularizer  $r$  to satisfy  $(\varepsilon, \delta)$ -DP. The final bound on the population risk also follows from LSI, which bounds the stability of the sample drawn from the respective distribution.
When running time is not a priority, we employ an exponential mechanism over a discretization of  $\mathcal{K}$  to establish the upper bound. The empirical risk bound derives from [9], and we leverage the concentration of sums of bounded random variables to bound the maximum difference over the discretizations between the empirical and population risk. We show this is nearly tight by reductions from selection to non-convex Lipschitz optimization of [23].
In Section 2, we present necessary definitions and backgrounds for our work. In Section 3, we construct the optimization framework, with guarantees on finding the SOSP with two different kinds of SubGaussian gradient oracles. It's crucial to note that this framework focuses solely on optimization and does not pertain to privacy. Section 4 explores the pursuits of finding the SOSP privately by constructing private SubGaussian gradient oracles and seamlessly integrating them into the existing framework. We bound the private excess bounds in Section 5. For other preliminaries, all omitted proofs and some further discussions on related work can be found in the Appendix.
Throughout the paper, if not stated explicitly, the norm  $\| \cdot \|$  means the  $\ell_2$  norm.
Definition 2.1 (Lipschitz, Smoothness and Hessian Lipschitz). Given a function  $f: \mathcal{K} \to \mathbb{R}$ , we say  $f$  is  $G$ -Lipschitz, if for all  $x_1, x_2 \in \mathcal{K}$ ,  $|f(x_1) - f(x_2)| \leq G\|x_1 - x_2\|$ , we say a function  $f$  is  $M$ -smooth, if for all  $x_1, x_2 \in \mathcal{K}$ ,  $\| \nabla f(x_1) - \nabla f(x_2)\| \leq M\|x_1 - x_2\|$ , and we say the function  $f$  is  $\rho$ -Hessian Lipschitz, if for all  $x_1, x_2 \in \mathcal{K}$ , we have  $\| \nabla^2 f(x_1) - \nabla^2 f(x_2)\| \leq \rho \| x_1 - x_2\|$ .
Definition 2.2 (SubGaussian, and Norm-SubGaussian). A random vector  $x\in \mathbb{R}^d$  is SubGaussian  $(\mathrm{SG}(\zeta))$  if there exists a positive constant  $\zeta$  such that  $\mathbb{E}e^{\langle v,x - \mathbb{E}x\rangle}\leq e^{\| v\|^{2}\zeta^{2} / 2},\forall v\in \mathbb{R}^{d}.x\in \mathbb{R}^{d}$  is norm-SubGaussian (nSG(ζ)) if there exists  $\zeta$  such that  $\operatorname *{Pr}[\| x - \mathbb{E}x\| \geq t]\leq 2e^{-\frac{t^2}{2\zeta^2}},\forall t\in \mathbb{R}$
Fact 2.3. For a Gaussian  $\theta \sim \mathcal{N}(0, \sigma^2 I_d)$ ,  $\theta$  is  $\mathrm{SG}(\sigma)$  and  $\mathrm{nSG}(\sigma \sqrt{d})$ .
Lemma 2.4 (Hoeffding type inequality for norm-subGaussian, [30]). Let  $x_{1}, \dots, x_{k} \in \mathbb{R}^{d}$  be random vectors, and for each  $i \in [k]$ ,  $x_{i} \mid \mathcal{F}_{i-1}$  is zero-mean nSG( $\zeta_{i}$ ) where  $\mathcal{F}_{i}$  is the corresponding filtration. Then there exists an absolute constant  $c$  such that for any  $\delta > 0$ , with probability at least  $1 - \omega$ ,  $\| \sum_{i=1}^{k} x_{i} \| \leq c \cdot \sqrt{\sum_{i=1}^{k} \zeta_{i}^{2} \log(2d / \omega)}$ , which means  $\sum_{i=1}^{k} x_{i}$  is nSG $(\sqrt{c \log(d) \sum_{i=1}^{k} \zeta_{i}^{2}})$ .
3 Convergence to Stationary Points: Framework
We present the optimization framework for finding SOSP in this section. It's important to emphasize that this framework is dedicated exclusively to optimization concerns, with privacy considerations being outside of its purview. The results about SOSP throughout the paper follow the assumptions of [46].
Assumption 3.1. Any function drawn from  $\mathcal{P}$  is  $G$ -Lipschitz,  $\rho$ -Hessian Lipschitz, and  $M$ -smooth, almost surely, and the risk is upper bounded by  $B$ .
As discussed before, we define two different kinds of gradient oracles, one for estimating the gradient at one point and the other for estimating the gradient difference at two points.
Definition 3.2 (SubGaussian gradient oracles). For a  $G$ -Lipschitz and  $M$ -smooth function  $F$ :
(1) We say  $\mathcal{O}_1$  is a first kind of  $\zeta_1$  norm-subGaussian Gradient oracle if given  $x\in \mathbb{R}^d$ ,  $\mathcal{O}(x)$  satisfies  $\mathbb{E}\mathcal{O}_1(x) = \nabla F(x)$  and  $\mathcal{O}_1(x) - \nabla F(x)$  is nSG( $\zeta_1$ ).
(2) We say  $\mathcal{O}_2$  is a second kind of  $\zeta_2$  norm-subGaussian stochastic Gradient oracle if given  $x, y \in \mathbb{R}^d$ ,  $\mathcal{O}_2(x, y)$  satisfies that  $\mathbb{E}\mathcal{O}_2(x, y) = \nabla F(x) - \nabla F(y)$  and  $\mathcal{O}_2(x, y) - (\nabla F(x) - \nabla F(y))$  is nSG( $\zeta_2\| x - y\|$ ).
Note that we should assume  $M \geq \sqrt{\rho \alpha}$  to make finding a second-order stationary point strictly more challenging than finding a first-order stationary point. We use  $\mathrm{smin}(\cdot)$  to denote the smallest eigenvalue of a matrix.
Algorithm 1 Stochastic Spider
1: Input: Objective function  $F$ , Gradient Oracle  $\mathcal{O}_1, \mathcal{O}_2$  with SubGaussian parameters  $\zeta_1$  and  $\zeta_2$ , parameters of objective function  $B, M, G, \rho$ , parameter  $\kappa$ , failure probability  $\omega$   
2: Set  $\gamma = \sqrt{4C(\zeta_2^2\kappa + 4\zeta_1^2)\cdot\log(BMd / \rho\omega)}$ ,  $\Gamma = \frac{M\log(\frac{dMB}{\rho\gamma\omega})}{\sqrt{\rho\gamma}}$   
3: Set  $\eta = 1 / M, t = 0, T = BM\log^4 (\frac{dMB}{\rho\gamma\omega}) / \gamma^2$   
4: Set  $\mathrm{drift}_0 = \kappa$ , frozen  $= 1$ ,  $\nabla_{-1} = 0$   
5: while  $t \leq T$  do  
6: if  $\|\nabla_{t-1}\| \leq \gamma \log^3 (BMd / \rho\omega) \wedge \mathrm{frozens}_{t-1} \leq 0$  then  
7:  $\mathrm{frozens}_t = \Gamma$ ,  $\mathrm{drift}_t = 0$   
8:  $\nabla_t = \mathcal{O}_1(x_t) + g_t$ , where  $g_t \sim \mathcal{N}(0, \frac{\zeta_1^2}{d} I_d)$   
9: else if  $\mathrm{drift}_{t-1} \geq \kappa$  then  
10:  $\nabla_t = \mathcal{O}_1(x_t)$ ,  $\mathrm{drift}_t = 0$ ,  $\mathrm{frozens}_t = \mathrm{frozens}_{t-1} - 1$   
11: else  
12:  $\Delta_t = \mathcal{O}_2(x_t, x_{t-1})$ ,  $\nabla_t = \nabla_{t-1} + \Delta_t$ ,  $\mathrm{frozens}_t = \mathrm{frozens}_{t-1} - 1$   
13: end if  
14:  $x_{t+1} = x_t - \eta\nabla_t$ ,  $\mathrm{drift}_{t+1} = \mathrm{drift}_t + \eta^2\|\nabla_t\|_2^2$ ,  $t = t+1$   
15: end while  
16: Return:  $\{x_1, \dots, x_T\}$
Inspired by [2] who adapted the SpiderBoost algorithm for finding private FOSPs, we give a framework based on the SpiderBoost in Algorithm 1. Our analysis of Algorithm 1 hinges on three key properties we establish in this section:  $(i)\nabla_{t}$  remains consistently close to the true gradient  $\nabla F(x_{t})$  with high probability;  $(ii)$  the algorithm is capable of escaping the saddle point with high probability, and  $(iii)$  a large drift implies significant decrease in the function value, which enables us to limit the number of queries to the more accurate but costlier first kind of gradient oracle  $\mathcal{O}_1$ .
Lemma 3.3. For any  $0 \leq t \leq T$  and letting  $\tau_t \leq t$  be the largest integer such that  $\mathrm{drift}_{\tau_t}$  is set to be  $0$ , with probability at least  $1 - \omega / T$ , for some universal constant  $C > 0$ , we have
$$
\left\| \nabla_ {t} - \nabla F \left(x _ {t}\right) \right\| ^ {2} \leq \left(\zeta_ {2} ^ {2} \cdot \sum_ {i = \tau_ {t} + 1} ^ {t} \left\| x _ {i} - x _ {i - 1} \right\| ^ {2} + 4 \zeta_ {1} ^ {2}\right) \cdot C \cdot \log (T d / \omega). \tag {1}
$$
Hence with probability at least  $1 - \omega$ , we know for each  $t \leq T$ ,  $\| \nabla_t - \nabla F(x_t) \|^2 \leq \gamma^2 / 16$ , where  $\gamma^2 := 16C(\zeta_2^2 \kappa + 4\zeta_1^2) \cdot \log(Td / \omega)$  and  $\kappa$  is a parameter we can choose in the algorithm.
As shown in Lemma 3.3, the error on the gradient estimation for each step is bounded with high probability. Then by adding the Gaussian noise in Line 8, we can show the algorithm can escape the saddle point efficiently based on previous results.
Lemma 3.4 (Essentially from [46]). Under Assumption 3.1, run SGD iterations  $x_{t+1} = x_t - \eta \nabla_t$ , with step size  $\eta = 1 / M$ . Suppose  $x_0$  is a saddle point satisfying  $\| \nabla F(x_0) \| \leq \alpha$  and  $\min(\nabla^2 F(x_0)) \leq -\sqrt{\rho \alpha}$ ,  $\alpha = \gamma \log^3(dBM / \rho \omega)$ . If  $\nabla_0 = \nabla F(x_0) + \zeta_1 + \zeta_2$  where  $\| \zeta_1 \| \leq \gamma$ ,  $\zeta_2 \sim \mathcal{N}(0, \frac{\gamma^2}{d\log(d / \omega)} I_d)$ , and  $\| \nabla_t - \nabla F(x_t) \| \leq \gamma$  for all  $t \in [\Gamma]$ , with probability at least  $1 - \omega \cdot \log(1 / \omega)$ , one has  $F(x_\Gamma) - F(x_0) \leq -\Omega\left(\frac{\gamma^{3/2}}{\sqrt{\rho\log^3\left(\frac{dMB}{\rho\gamma\omega}\right)}}\right)$ , where  $\Gamma = \frac{M\log\left(\frac{dMB}{\rho\gamma\omega}\right)}{\sqrt{\rho\gamma}}$ .