SlowGuess/NeurIPS_2023_OCR huggingface dataset:
University of Massachusetts Amherst domke@cs.umass.edu
Guillaume Garrigos
Université Paris Cité and Sorbonne Université, CNRS Laboratoire de Probabilités, Statistique et Modélisation
F-75013 Paris, France
garrigos@lpsm.paris
Robert Gower
Center for Computational Mathematics
Flatiron Institute, New York
rgower@flatironinstitute.org
Black-box variational inference is widely used in situations where there is no proof that its stochastic optimization succeeds. We suggest this is due to a theoretical gap in existing stochastic optimization proofs—namely the challenge of gradient estimators with unusual noise bounds, and a composite non-smooth objective. For dense Gaussian variational families, we observe that existing gradient estimators based on reparameterization satisfy a quadratic noise bound and give novel convergence guarantees for proximal and projected stochastic gradient descent using this bound. This provides rigorous guarantees that methods similar to those used in practice converge on realistic inference problems.
Variational inference tries to approximate a complex target distribution with a distribution from a simpler family [1, 2, 3, 4]. If  $p(z,x)$  is a target distribution with latent variables  $z\in \mathbb{R}^d$  and observed data  $x\in \mathbb{R}^n$ , and  $q_{w}$  is a variational family with parameters  $w\in \mathcal{W}$ , the goal is to minimize the negative evidence lower bound (ELBO)
$$
f (w) := \underbrace {- \underset {z \sim q _ {w}} {\mathbb {E}} \log p (z , x)} _ {l (w)} + \underbrace {\underset {z \sim q _ {w}} {\mathbb {E}} \log q _ {w} (z)} _ {h (w)}. \tag {1}
$$
For subsequent discussion, we have decomposed the objective into the free energy  $l$  and the negative entropy  $h$ . Minimizing  $f$  is equivalent to minimizing the KL-divergence between  $q_{w}$  to  $p(\cdot |x)$ , because  $\mathrm{KL}(q_w\| p(\cdot |x)) = f(w) + \log p(x)$ . Recent research has focused on "black box" variational inference, where the target distribution  $p$  is sufficiently complex that one can only access it through evaluating probabilities (or gradients) at chosen points [5, 6, 7, 8, 9, 10, 11]. Crucially, one can still get stochastic estimates of the variational objective  $f$  and of its gradient, and use these to optimize.
Still, variational inference can sometimes be unreliable [12, 13, 14, 15], and some basic questions remain unanswered. Most notably: does stochastic optimization of  $f$  converge to a minimum of the objective? There has been various progress towards answering this question. One line of research seeks to determine if the variational objective  $f$  has favorable structural properties, such as smoothness or (strong) convexity [13, 16, 17] (Sec. 2.1). Another line seeks to control the "noise" (variance or expected squared norm) of different gradient estimators [18, 19, 20, 21] (Sec. 2.2). However, few
full convergence guarantees are known. That is, there are few known cases where applying a given stochastic optimization algorithm to a given target distribution is known to converge at a given rate.
We identify two fundamental barriers preventing from analysing this VI problem as a standard stochastic optimization problem. First, the gradient noise depends on the parameters  $w$  in a non-standard way (Sec. 2.3). This adds great technical complexity and renders many traditional stochastic optimization proofs inapplicable. Second, stochastic optimization theory typically requires that the (exact) objective function  $f$  is Lipschitz continuous or Lipschitz smooth. But in our VI setting, under some fairly benign assumptions, the ELBO is neither Lipschitz continuous nor Lipschitz smooth.
We obtain non-asymptotic convergence guarantees for this problem, under simple assumptions.
Central contributions (Informal). Suppose that the target model  $\log p(\cdot, x)$  is concave and Lipschitz-smooth, and that  $q_w$  is in a Gaussian variational family parameterized by the mean and a factor of the covariance matrix (Eq. 2). Consider minimizing the negative ELBO  $f$  using either one of the two following algorithms:
- a proximal stochastic gradient method, with the proximal step applied to  $h$  and the gradient step applied to  $l$ , estimating  $\nabla l(w)$  with a standard reparameterization gradient estimator (Eq. 5), and using a triangular covariance factor;
- a projected stochastic gradient method, with the gradient applied to  $f = l + h$ , estimating  $\nabla f(w)$  using either of two common gradient estimators (Eq. 7 or Eq. 9), with the projection done over symmetric and non-degenerate (Eq. 3) covariance factors
Then, both algorithms converge with a  $1 / \sqrt{T}$  complexity rate (Cor. 12), or  $1 / T$  if we further assume that  $\log p(\cdot ,x)$  is strongly concave (Cor. 13).
We also give a new bound on the noise of the "sticking the landing" gradient estimator, which leads to faster convergence when the target distribution  $p$  is closer to Gaussian, up to exponentially fast convergence when it is exactly Gaussian (Cor. 14).
This is achieved through a series of steps, that we summarize below.
1. We analyze the structural properties of the problem. Existing results show that with a Gaussian variational family, if  $-\log p(\cdot ,x)$  is (strongly) convex or Lipschitz smooth, then so is the free energy  $l$ . This is for instance known to be the case for some generalized linear models, and we give a new proof of convexity and smoothness for some hierarchical models including hierarchical logistic regression (see Appendix 7.3). The remaining component of the ELBO, the neg-entropy  $h$ , is convex when restricted to an appropriate set. It is not smooth, but it was recently proved to be smooth over a certain non-degeneracy set.
2. We study the noise of three common gradient estimators. They do not satisfy usual noise bounds, but we show that they all satisfy a new quadratic bound (Definition 5). For the sticking-the-landing estimator, our bound formalizes the longstanding intuition that it should have lower noise when the variational approximation is strong (Thm. 4).
3. We identify and solve the key optimization challenges posed by the above issues via new convergence results for the proximal and projected stochastic gradient methods, when applied to objectives that are smooth (but not uniformly smooth) and with gradient estimators satisfying our quadratic bound.
Recently, Xu and Campbell [22] analyzed projected-SGD (stochastic gradient descent) for Gaussian VI using the gradient estimator we will later call  $\mathbf{g}_{\mathrm{ent}}(7)$ , with a particular rescaling. They show that, asymptotically in the number of observed data, their method converges locally with a rate of  $1 / \sqrt{T}$ , under mild assumptions. Our results are less general in requiring convexity, but are non-asymptotic, apply with other gradient estimators, and give a faster  $1 / T$  rate with strong convexity.
Lambert et al. [23] introduce a VI-like SGD algorithm, derived from a discretisation of a Wasserstein gradient flow, and show it converges at a  $1 / T$  rate for the 2-Wasserstein distance when the log posterior is smooth and strongly concave. This line was continued by Diao et al. [24], who propose a
proximal-SGD method based on the decomposition  $f = l + h$ . They obtain a  $1 / \sqrt{T}$  rate when the log posterior is smooth and convex, and a  $1 / T$  rate when it is smooth and strongly convex. Unlike typical black-box VI algorithms used in practice, these algorithms require computing the Hessian of the posterior. We analyze more straightforward applications of SGD to the VI problem using standard gradient estimators. Under the same assumptions, our algorithms have the same rates for KL-divergence, which imply the same rates for 2-Wasserstein distance by of Pinsker's inequality and that the total-variation norm upper-bounds Wasserstein distance [25, Remark 8.2].
In concurrent work, Kim et al. [26], consider a proximal-SGD method similar to our approach in Sec. 6. They obtain a  $1 / T$  convergence rate, similar to what Cor. 12 gives when the log posterior is strongly concave. They also consider alternative parametrizations of the scale parameters that render the ELBO globally smooth, and obtain a nonconvex result: Under a relaxed version of the Polyak-Lojasiewicz inequality, they can guarantee a  $1 / T^4$  rate.
2 Properties of Variational Inference (VI) problems
Traditionally, the ELBO was optimized using message passing methods, which essentially assume that  $p$  and  $q$  are simple enough that block-coordinate updates are possible [2, 3, 4]. However, in the last decade a series of papers developed algorithms based on a "black-box" model where  $p$  is assumed to be complex enough that one can only evaluate  $\log p$  (or its gradient) at selected points  $z$ . The key observation is that even if  $p$  is quite complex, it is still possible to compute stochastic estimates of the gradient of the ELBO, which can be deployed in a stochastic optimization algorithm [5, 6, 8, 9, 10].
This paper seeks rigorous convergence guarantees for this problem. We study the setting where the variational family is the set of (dense) multivariate Gaussian distributions, parameterized in terms of  $w = (m,C)$ , where  $m$  is the mean and  $C$  is a factor of the covariance, i.e.
$$
q _ {w} (z) = \mathcal {N} (z | m, C C ^ {\top}). \tag {2}
$$
We optimize over  $w \in \mathcal{W}$ , where  $\mathcal{W} = \{(m, C) : C \succ 0\}$  and will further require  $C$  to be either symmetric or triangular. This does not really change the problem, since for a given covariance matrix  $\Sigma$  there always exists a symmetric (or lower triangular) positive definite factor  $C$  such that  $CC^{\top} = \Sigma$ .
Now, is it possible to solve this optimization problem? Without further assumptions, it is unlikely any guarantee is possible, because it is easy to encode NP-hard problems into this VI framework [27, 28]. We discuss below the assumptions we will make to be able to solve the problem.
2.1 Structural properties and assumptions
The properties of the free energy  $l$  depend on the properties of the target distribution  $p$ . It is necessary to assume that  $p$  is somehow "nice" to ensure the problem can be solved. Titsias and Lázaro-Gredilla [17, Proposition 1] showed that if  $-\log p(\cdot, x)$  is convex, then  $l$  is convex too. Challis and Barber [16, Sec 3.2] showed that if the likelihood  $p(\cdot | z)$  is convex and the prior  $p(z)$  is Gaussian, then  $l$  is strongly-convex. Domke [13, Theorem 9] showed that if  $-\log p(\cdot, x)$  is  $\mu$ -strongly convex, then  $l$  is  $\mu$ -strongly convex as well, and that the constant is sharp. Similarly, Domke [13, Theorem 1] showed that if  $\log p(\cdot, x)$  is  $M$ -smooth, then  $l$  is also  $M$ -smooth, and that the constant is sharp.
In this paper we make two assumptions about the target distribution  $p$ : the negative log-probability  $-\log p(\cdot, x)$  must be convex (or strongly convex), and Lipschitz smooth. Section 7.3 (Appendix) gives some example models where these assumptions are satisfied. For example, if the model is Bayesian linear regression, or logistic regression with a Gaussian prior, then  $-\log p(\cdot, x)$  is smooth and strongly convex. In addition, if the target is a hierarchical logistic regression model, then  $-\log p(\cdot, x)$  is smooth and convex.
Assumptions on  $p$  also impact what a minimizer  $w^{*} = (m^{*}, C^{*})$  of  $f(w)$  can look like. Intuitively, if the target  $\log p(\cdot, x)$  is  $\mu$ -strongly concave, then we would expect the target distribution to be "peaky". This means that the optimal distribution would be close to a delta function centered at the MAP solution:  $m^{*}$  would be close to some maximum of  $\log p(\cdot, x)$  noted  $\bar{m}$ , and the covariance factor  $C^{*}$  would not be too large. This intuition can be formalized: in this context we have  $\| C^{*} \|^{2} + \| m^{*} - \bar{m} \|_{2}^{2} \leq d / \mu$  [13, Theorem 10]. Similarly, if  $\log p(\cdot, x)$  is  $M$ -Lipschitz smooth, then we expect that the target is not too concentrated, so we might expect that the optimal covariance cannot not be too
small. Formally, it can be shown that the singular values of the covariance factor  $C^*$  are greater than  $1 / \sqrt{M}$  [13, Theorem 7].
The properties of the neg-entropy  $h$  are inherited from the choice of the variational family and do not depend on  $p$ . Since we consider a Gaussian variational family,  $h$  is known in closed-form. To avoid some technical issues, we will restrict  $h$  so that the covariance factor is positive definite, so  $h(w)$  is equal to  $-\ln \det C$  up to a constant if  $C$  is positive definite (see Appendix 7.1), and to  $+\infty$  otherwise. So  $h$  inherits the properties of the negative log determinant, meaning it is a proper closed convex function whenever  $C$  is symmetric or triangular (see Appendix 7.2). From an optimization perspective, it is natural to use a gradient-based algorithm. But unfortunately  $h$  is not Lipschitz smooth on its domain, because its gradient can change arbitrarily quickly when the singular values of  $C$  become small. However, it can be shown [13, Lemma 12] that  $h$  is  $M$ -smooth over the following set of non-singular parameters (which contains the solution  $w^*$ , see the previous paragraph) given by
$$
\mathcal {W} _ {M} = \left\{w = (m, C): C \succ 0 \text {a n d} \sigma_ {\min } (C) \geq \frac {1}{\sqrt {M}} \right\}. \tag {3}
$$
Instead of computing gradients, an optimizer might want to use the proximal operator of  $h$ , which can also be computed in closed form (see Sec. 4).
2.2 Gradient estimators
This paper considers gradient estimators based on the "path" method or "reparameterization". These assume some base distribution  $s$  is known, together with some deterministic transformation  $T_{w}$ , such that the distribution of  $T_{w}(\mathsf{u})$  is equal to  $q_{w}$  when  $\mathsf{u} \sim s$ . Then, we can write
$$
\nabla_ {w} \underset {\mathbf {z} \sim q _ {w}} {\mathbb {E}} \phi (\mathbf {z}) = \underset {\mathbf {u} \sim s} {\mathbb {E}} \nabla_ {w} \phi \left(T _ {w} (\mathbf {u})\right). \tag {4}
$$
In the case of multivariate Gaussians, the most common choice is to use  $s = \mathcal{N}(0,I)$  and  $T_w(u) = Cu + m$ , which we will consider in the rest of the paper.
We will consider three different gradient estimators based on the path-type strategy (Eq. 4). We will provide new noise bounds for each of them (all the proofs are in Appendix 8). Our analysis is mostly based on the following general result [20, Thm. 3].
Theorem 1. Let  $T_w(u) = Cu + m$  for  $w = (m, C)$ . Let  $\phi : \mathbb{R}^d \to \mathbb{R}$  be M-smooth, suppose that  $\phi$  is stationary at  $\bar{m}$ , and define  $\bar{w} = (\bar{m}, 0)$ . Then
$$
\underset {\mathfrak {u} \sim \mathcal {N} (0, I)} {\mathbb {E}} \| \nabla_ {w} \phi (T _ {w} (\mathfrak {u})) \| _ {2} ^ {2} \leq (d + 1) M ^ {2} \| m - \bar {m} \| _ {2} ^ {2} + (d + 3) M ^ {2} \| C \| _ {F} ^ {2} \leq (d + 3) M ^ {2} \| w - \bar {w} \| _ {2} ^ {2}.
$$
Furthermore, the first inequality cannot be improved.
Intuitively, this result says that the noise of a gradient estimator is lower when the scale parameter  $C$  is small and the mean parameter  $m$  is close to a stationary point.
The first estimator we consider is  $\nabla l(w)$  only. It is given by taking  $\phi = -\log p$  into Eq. 4, i.e.
$$
\mathbf {g} _ {\text {e n e r g y}} (\mathfrak {u}) := - \nabla_ {w} \log p \left(T _ {w} (\mathfrak {u}), x\right), \quad \mathfrak {u} \sim \mathcal {N} (0, I). \tag {5}
$$
The next result gives a bound on the noise of this estimator, which is a direct consequence of Theorem 1. The second line uses Young's inequality to bound the noise in terms of (i) the distance of  $w$  from  $w^{*}$  and (ii) a constant determined by the distance of  $w^{*}$  from some fixed parameters  $\bar{w}$ .
Theorem 2. Suppose that  $\log p(\cdot, x)$  is  $M$ -smooth and has a maximum (or stationary point) at  $\bar{m}$ , and define  $\bar{w} = (\bar{m}, 0)$ . Then, for every  $w$  and every solution  $w^{*}$  of the VI problem,
$$
\begin{array}{l} \mathbb {E} \left\| \mathrm {g} _ {\text {e n e r g y}} (\mathrm {u}) \right\| _ {2} ^ {2} \leq (d + 3) M ^ {2} \| w - \bar {w} \| _ {2} ^ {2} \tag {6} \\ \leq 2 (d + 3) M ^ {2} \| w - w ^ {*} \| _ {2} ^ {2} + 2 (d + 3) M ^ {2} \| w ^ {*} - \bar {w} \| _ {2} ^ {2}. \\ \end{array}
$$
Second, we consider an estimator of the gradient of the full objective  $l + h$ . It is obtained by simply taking the above estimator and adding the true (known) gradient of the neg-entropy, i.e.
$$
\mathbf {g} _ {\text {e n t}} (\mathfrak {u}) := \mathbf {g} _ {\text {e n e r g y}} (\mathfrak {u}) + \nabla h (w), \quad \mathfrak {u} \sim \mathcal {N} (0, I). \tag {7}
$$
The noise of  $\mathbf{g}_{\mathrm{ent}}$  can be bounded since it only differs from  $\mathbf{g}_{\mathrm{energy}}$  by the deterministic quantity  $\nabla h(w)$ , and the fact that—provided  $w \in \mathcal{W}_L$ —the singular values of  $w$  cannot be too small and so  $\nabla h(w)$  cannot be too large. This is formalized in the following theorem, where again, the second line relaxes the result into a term based on the distance of  $w$  from  $w^*$  plus constants. (Another type of noise bound [29] for  $\mathbf{g}_{\mathrm{ent}}$  was obtained by Kim et al. [21] for diagonal Gaussians and a variety of parameterizations of the covariance.)
Theorem 3. Suppose that  $\log p(\cdot, x)$  is  $M$ -smooth, that it is maximal at  $\bar{m}$ , and define  $\bar{w} = (\bar{m}, 0)$ . Then, for every  $L > 0$ , for every  $w \in \mathcal{W}_L$  and every solution  $w^*$  of the VI problem,
$$
\begin{array}{l} \mathbb {E} \left\| \mathrm {g} _ {\text {e n t}} (\mathrm {u}) \right\| _ {2} ^ {2} \leq 2 (d + 3) M ^ {2} \| w - \bar {w} \| _ {2} ^ {2} + 2 d L \tag {8} \\ \leq 4 (d + 3) M ^ {2} \| w - w ^ {*} \| ^ {2} + 4 (d + 3) M ^ {2} \| w ^ {*} - \bar {w} \| ^ {2} + 2 d L. \\ \end{array}
$$
While  $\mathbf{g}_{\mathrm{ent}}$  used the exact gradient of  $h$ , it may be beneficial to use a stochastic estimator  $h$  instead. To derive our third estimator, write the gradient of the ELBO as<sup>1</sup>
$$
\nabla l (w) + \nabla h (w) = \nabla_ {w} \underset {\mathbf {z} \sim q _ {w}} {\mathbb {E}} \left[ - \log p (\mathbf {z}, x) + \log q _ {v} (\mathbf {z}) \right] \Big | _ {v = w},
$$
where the parameters  $v$  serve as a way to "hold  $w$  constant under differentiation". This leads to our third estimator, called the "sticking the landing" (STL) gradient estimator
$$
\mathrm {g} _ {\mathrm {S T L}} (\mathrm {u}) := \mathrm {g} _ {\text {e n e r g y}} (\mathrm {u}) + \left[ \nabla_ {w} \log q _ {v} \left(T _ {w} (\mathrm {u})\right) \right] _ {v = w}, \quad \mathrm {u} \sim \mathcal {N} (0, I). \tag {9}
$$
Intuitively, we expect that  $\mathbf{g}_{\mathrm{STL}}$  will tend to have lower noise than  $\mathbf{g}_{\mathrm{ent}}$  when the posterior is well-approximated by the variational distribution. The reason is that, as observed by Roeder et al. [30], if  $q_{w}(z)$  were a perfect approximation of  $p(z|x)$ , then the two terms in Eq. 9 would exactly cancel (for every  $u$ ) and so the estimator would have zero variance. Below we formalize this intuition in what we believe is the first noise bound on  $\mathbf{g}_{\mathrm{STL}}$ .
Theorem 4. Suppose that  $\log p(\cdot, x)$  is  $M$ -smooth. Consider the residual  $r(z) \coloneqq \log p(z, x) - \log q_{w^*}(z)$  for any solution  $w^*$  of the VI problem, assume that it has a stationary point  $\hat{m}$ , and define  $\hat{w} = (\hat{m}, 0)$ . Then  $r$  is  $K$ -smooth for some  $K \in [0, 2M]$ , and for all  $w \in \mathcal{W}_M$ ,
$$
\begin{array}{l} \mathbb {E} \left\| \mathbf {g} _ {\mathrm {S T L}} \right\| _ {2} ^ {2} \leq 8 (d + 3) M ^ {2} \| w - w ^ {*} \| _ {2} ^ {2} + 2 (d + 3) K ^ {2} \| w - \hat {w} \| _ {2} ^ {2} \tag {10} \\ \leq 4 (d + 3) \left(K ^ {2} + 2 M ^ {2}\right) \| w - w ^ {*} \| _ {2} ^ {2} + 4 (d + 3) K ^ {2} \| w ^ {*} - \hat {w} \| _ {2} ^ {2}. \\ \end{array}
$$
Moreover, if  $p(\cdot | x)$  is Gaussian then  $K = 0$ .
When the target is Gaussian then  $K = 0$  in Eq. 10, meaning that when  $w = w^{*}$ , the STL estimator has no variance. This is a distinguishing feature of  $\mathbf{g}_{\mathrm{STL}}$ , as opposed to  $\mathbf{g}_{\mathrm{energy}}$  or  $\mathbf{g}_{\mathrm{ent}}$ .
2.3 Challenges for optimization
Three major issues bar applying existing analysis of stochastic optimization methods to our VI setting: 1) non-smooth composite objective 2) lack of uniform smoothness and 3) lack of uniformly bounded noise of the gradient estimator.
The first issue is due to the non-smoothness of neg-entropy function  $h$ . This means that under the benign assumption that the target log  $p$  is smooth, the full objective  $l + h$  cannot be smooth, since a nonsmooth function plus a smooth function is always nonsmooth. This renders stochastic optimization proofs (e.g. those for the "ABC" conditions [29]) that do not use projections or proximal operators inapplicable.
One way to tackle this first issue would be to use a non-smooth proof technique, but these rely on having a uniform gradient noise bound (our third issue). Alternatively, one can overcome the non-smoothness of the neg-entropy function by either using a proximal operator, or projecting onto a set where  $h$  is smooth, namely  $\mathcal{W}_M$ . This is the strategy we pursue.
One could also build an estimator without holding  $w$  constant in this way. However, in the case of Gaussian variational families, this turns out to be mathematically equivalent to using a closed form entropy because  $\log q_w(T_w(u)) = -\frac{1}{2}\| u\| _2^2 -\frac{d}{2}\log (2\pi) - \log |C|$ , which has the same gradient (independent of  $u$ ) as  $h(w)$ .
Table 1: Table of known black-box VI properties relevant to optimization. In some cases there could be multiple valid  $w^{*}$  or  $\bar{w}$  in which case these results hold for all simultaneously.
Description	Definition
Estimator for ∇l	genergy = -∇w log p(Tw(u), x)
Estimator for ∇l + ∇h	gent = -∇w log p(Tw(u), x) + ∇h(w)
Estimator for ∇l + ∇h	gSTL = -∇w log p(Tw(u), x) + ∇w log qv(Tw(u))|v=w
Constraint set	W_M = {(m, C): C > 0, σmin(C) ≥ 1/√M}
Optimum	w* ∈ argminw l(w) + h(w)
Stationary point of l	w = (z, 0) for any z that is a stationary point of log p(·, x)
Condition on - log p(z, x)	Consequence
none	h(w) is convex when C is symmetric or triangular
h(w) is M-smooth over W_M
convex	l(w) is convex
μ-strongly convex	l(w) is μ-strongly convex
||w* - w||_2^2 ≤ d/μ
M-smooth	l(w) is M-smooth
w* ∈ W_M
genergy, gent, and gSTL are quadratically bounded
The second issue is that existing analyses for proximal/projected stochastic methods either rely on a uniform noise bound ([31, Cor. 3.6]) or uniform smoothness [32, 33, 34, 35, 36], both of which are not known to be true for VI. By uniform smoothness, we refer to the assumption that  $\log p(T_w(u),x)$  is  $M$ -smooth for every  $u$ , with  $M$  being independent of  $u$ . Instead, we can only guarantee that  $\log p(T_w(u),x)$  is smooth in expectation, i.e. that  $l(w)$  is smooth. Several works [9, Cond. 1][37, Thm. 1][38, Sec. 4][39, Assumption A1][40, Thm. 1][41, Assumption 3.2] assumed that the full objective  $l + h$  is smooth, but we are not aware of cases where this holds in practice for VI and in our parameterization (Eq. 2) this cannot be true if  $\log p$  is smooth.
The third issue is the lack of a uniform noise bound. Most non-smooth convergence guarantees within stochastic optimization [42] assume that the noise of the gradient estimator is uniformly bounded by a constant. But this does not appear to be true even under favorable assumptions—e.g. it is untrue if the target distribution is a standard Gaussian. The best that one can hope is that the gradient noise can be bounded by a quadratic that depends on the current parameters  $w$ , and—depending on the estimator—even this may only be true when the parameters are in the set  $\mathcal{W}_M$ . Thus, our main optimization contribution is to provide theoretical guarantees for stochastic algorithms under such a specific noise regime, which we make precise in the next definition.