SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Definition 5. Let  $\phi$  be a differentiable function. We say that a random vector  $\mathbf{g}$  is a quadratically bounded estimator for  $\nabla \phi$  at  $w$  with parameters  $(a,b,w^{*})$ , if it is unbiased  $\mathbb{E}[\mathbf{g}] = \nabla \phi (w)$  and if the expected squared norm is bounded by a quadratic function of the distance of parameters  $w$  to  $w^{*}$ , i.e.  $\mathbb{E}\| \mathbf{g}\| _2^2\leq a\| w - w^*\| _2^2 +b$ .
The noise bounds derived in Section 2.2 for the estimators  $\mathbf{g}_{\mathrm{energy}}$ ,  $\mathbf{g}_{\mathrm{ent}}$ , and  $\mathbf{g}_{\mathrm{STL}}$  imply that they are uniformly quadratically bounded estimators. See Appendix 8.2 for a table with the corresponding constants  $a$  and  $b$ .
3 Stochastic Optimization with quadratically bounded estimators
In this section we give new convergence guarantees for the Prox-SGD algorithm and the Proj-SGD algorithm with quadratically bounded gradient estimators. Because these may be of independent
interest, they are presented generically, without any reference to the VI setting. We specialize these results to VI in Section 4. For both algorithms, we present results assuming the problem to be strongly convex or just convex. All proofs for this section can be found in the Appendix (Sec. 9).
3.1 Stochastic Proximal Gradient Descent
Here we want to minimize a function which is the sum of two terms  $l + h$ , were both  $l$  and  $h$  are proper closed convex functions, and  $l$  is smooth. For this we will use stochastic proximal gradient.
Definition 6 (Prox-SGD). Let  $w^0$  be a fixed initial parameters and let  $\gamma_1, \gamma_2, \dots$  be a sequence of step sizes. The stochastic proximal gradient (Prox-SGD) method is given by
$$
w ^ {t + 1} = \operatorname {p r o x} _ {\gamma_ {t} h} \left(w ^ {t} - \gamma_ {t} g ^ {t}\right),
$$
where  $g^{t}$  is a gradient estimator for  $\nabla l(w^{t})$ , and the proximal operator is defined as
$$
\operatorname {p r o x} _ {\gamma h} (w) := \underset {v} {\operatorname {a r g m i n}} h (v) + \frac {1}{2 \gamma} \| w - v \| _ {2} ^ {2}.
$$
Theorem 7. Let  $l$  be a  $\mu$ -strongly convex and  $M$ -smooth function, and let  $\bar{w} = \mathrm{argmin}(l)$ . Let  $h$  be a proper closed convex function, and let  $w^{*} = \mathrm{argmin}(l + h)$ . Let  $(w^{t})_{t\in \mathbb{N}}$  be generated by the Prox-SGD algorithm, with a constant stepsize  $\gamma \in (0,\min \{\frac{\mu}{2a},\frac{1}{\mu}\}]$ . Suppose that  $g^{t}$  is a quadratically bounded estimator (Def. 5) for  $\nabla l$  with parameters  $(a,b,w^{*})$ . Then,
$$
\mathbb {E} \left\| w ^ {T + 1} - w ^ {*} \right\| _ {2} ^ {2} \leq (1 - \gamma \mu) ^ {T} \left\| w ^ {0} - w ^ {*} \right\| _ {2} ^ {2} + \frac {2 \gamma}{\mu} (b + M ^ {2} \| w ^ {*} - \bar {w} \| _ {2} ^ {2}). \tag {11}
$$
Alternatively, if we use the decaying stepsize  $\gamma_{t} = \min \left\{\frac{\mu}{2a},\frac{1}{\mu}\frac{2t + 1}{(t + 1)^{2}}\right\}$ , then
$$
\mathbb {E} \| w ^ {T} - w ^ {*} \| ^ {2} \leq \frac {1 6 \left\lfloor a / \mu^ {2} \right\rfloor^ {2}}{T ^ {2}} \| w ^ {0} - w ^ {*} \| ^ {2} + \frac {8}{\mu^ {2} T} \left(b + M ^ {2} \| w ^ {*} - \bar {w} \| _ {2} ^ {2}\right). \tag {12}
$$
In both cases,  $T = \mathcal{O}(\epsilon^{-1})$  iterations are sufficient to guarantee that  $\mathbb{E}\| w^T -w^*\| ^2\leq \epsilon$ .
The above theorem gives an anytime  $1 / T$  rate of convergence when the step sizes are chosen based on the strong convexity and gradient noise constants  $\mu$  and  $a$ . Note that we do not need to know precisely those constants: We show in Appendix 9.3 that using any constant step size proportional to  $T / \log T$  leads to a log  $T / T$  rate.
Theorem 8. Let  $l$  be a proper convex and  $M$ -smooth function. Let  $h$  be a proper closed convex function, and let  $w^{*} \in \mathrm{argmin}(l + h)$ . Let  $(w^{t})_{t \in \mathbb{N}}$  be generated by the Prox-SGD algorithm, with a constant stepsize  $\gamma \in (0, \frac{1}{M}]$ . Suppose that  $g^{t}$  is a quadratically bounded estimator (Def. 5) for  $\nabla l$  with parameters  $(a, b, w^{*})$ . Then,
$$
\mathbb {E} \left[ f (\bar {w} ^ {T}) - \inf  f \right] \leq \gamma \left(a \frac {\| w ^ {0} - w ^ {*} \| ^ {2}}{\left(1 - \theta^ {T}\right)} + b\right),
$$
where  $\theta \stackrel{\mathrm{def}}{=} \frac{1}{1 + 2a\gamma^2}$  and  $\bar{w}^T \stackrel{\mathrm{def}}{=} \frac{\sum_{t=1}^{T} \theta^{t+1} w^t}{\sum_{t=1}^{T} \theta^{t+1}}$ . In particular, if  $\gamma = \frac{1}{\sqrt{aT}}$ , then
$$
\mathbb {E} \left[ f (\bar {w} ^ {T}) \right] - \inf  f \leq \frac {1}{\sqrt {a T}} \left(2 a \| w ^ {0} - w ^ {*} \| ^ {2} + b\right) \quad \forall T \geq \max  \left\{\frac {M ^ {2}}{a}, 2 \right\}.
$$
Thus,  $T = \mathcal{O}(\epsilon^{-2})$  iterations are sufficient to guarantee that  $\mathbb{E}\left[f(\bar{w}^T) - \inf f\right] \leq \epsilon$ .
3.2 Stochastic Projected Gradient Descent
Here we want to minimize a function  $f$  over a set of constraints  $\mathcal{W}$ , where  $\mathcal{W}$  is a nonempty closed convex set and  $f$  is a proper closed convex function which is differentiable on  $\mathcal{W}$ . To solve this problem, we will consider the stochastic projected gradient algorithm.
Definition 9 (Proj-SGD). Let  $w^0$  be some fixed initial parameter, and let  $\gamma_1, \gamma_2, \dots$  be a sequence of step sizes. The projected stochastic gradient (Proj-SGD) method is given by
$$
w ^ {t + 1} = \operatorname {p r o j} _ {\mathcal {W}} \left(w ^ {t} - \gamma_ {t} g ^ {t}\right),
$$
where  $g^{t}$  is a gradient estimator for  $\nabla f(w^{t})$ , and the projection operator is defined as
$$
\operatorname {p r o j} _ {\mathcal {W}} (w) = \operatorname * {a r g m i n} _ {v \in \mathcal {W}} \| v - w \| _ {2} ^ {2}.
$$
Note that we do not require  $f$  to be smooth, meaning that this setting is not a particular case of the one considered in Section 3.1. In fact, the arguments we use in the proofs for Proj-SGD are more closely related to stochastic subgradient methods than stochastic gradient methods.
Theorem 10. Let  $\mathcal{W}$  be a nonempty closed convex set. Let  $f$  be a  $\mu$ -strongly convex function, differentiable on  $\mathcal{W}$ . Let  $w^{*} = \operatorname{argmin}_{\mathcal{W}}(f)$ . Let  $(w^{t})_{t \in \mathbb{N}}$  be generated by the Proj-SGD algorithm, with a constant stepsize  $\gamma \in (0, \min\{\frac{\mu}{2a}, \frac{2}{\mu}\}]$ . Suppose that  $g^{t}$  is a quadratically bounded estimator (Def. 5) for  $\nabla f$  with parameters  $(a, b, w^{*})$ . Then,
$$
\mathbb {E} \left\| w ^ {T} - w ^ {*} \right\| ^ {2} \leq \left(1 - \frac {\mu \gamma}{2}\right) ^ {T} \left\| w ^ {0} - w ^ {*} \right\| ^ {2} + \frac {2 \gamma b}{\mu}. \tag {13}
$$
Alternatively, if we use the decaying stepsize  $\gamma_{t} = \min \left\{\frac{\mu}{2a},\frac{2}{\mu}\frac{2t + 1}{(t + 1)^{2}}\right\}$ , then
$$
\mathbb {E} \left[ \| w ^ {T} - w ^ {*} \| ^ {2} \right] \leq \frac {3 2 a}{\mu^ {2} T ^ {2}} \| w ^ {0} - w ^ {*} \| ^ {2} + \frac {1 6 b}{\mu^ {2} T}. \tag {14}
$$
In both cases,  $T = \mathcal{O}(\epsilon^{-1})$  iterations are sufficient to guarantee that  $\mathbb{E}\| w^T -w^*\| ^2\leq \epsilon$
Note that Eq. 13 is a sum of two terms: one that decays exponentially in  $T$  and one that decreases only when the stepsize  $\gamma$  is sufficiently small. This has an important consequence: if one uses the gradient estimator  $\mathbf{g}_{\mathrm{STL}}$  and the target distribution is exactly a Gaussian, then  $b = 0$  (Appendix 8.2), meaning that the algorithm will converge at an exponential rate. This is similar to many results in the stochastic optimization literature showing faster rates hold when interpolation holds [35].
Theorem 11. Let  $\mathcal{W}$  be a nonempty closed convex set. Let  $f$  be a convex function, differentiable on  $\mathcal{W}$ . Let  $w^{*} \in \operatorname{argmin}_{\mathcal{W}}(f)$ . Let  $(w^{t})_{t \in \mathbb{N}}$  be generated by the Proj-SGD algorithm, with a constant stepsize  $\gamma \in (0, +\infty)$ . Suppose that  $g^{t}$  is a quadratically bounded estimator (Def. 5) for  $\nabla f$  at  $w^{t}$  with constant parameters  $(a, b, w^{*})$ . Then,
$$
\mathbb {E} \left[ f (\bar {w} ^ {T}) - \inf _ {\mathcal {W}} f \right] \leq \frac {\gamma}{2} \left(a \frac {\left\| w ^ {0} - w ^ {*} \right\| ^ {2}}{1 - \theta^ {T}} + b\right).
$$
where  $\theta \stackrel{\mathrm{def}}{=} \frac{1}{1 + a\gamma^2}$  and  $\bar{w}^T \stackrel{\mathrm{def}}{=} \frac{\sum_{t=0}^{T-1} \theta^{t+1} w^t}{\sum_{t=0}^{T-1} \theta^{t+1}}$ . Finally if  $\gamma = \frac{\sqrt{2}}{\sqrt{aT}}$  and  $T \geq 2$  then
$$
\mathbb {E} \left[ f \left(\bar {w} ^ {T}\right) \right] - \inf  f \leq \frac {\sqrt {2 a}}{\sqrt {T}} \| w ^ {0} - w ^ {*} \| ^ {2} + \frac {b}{\sqrt {2 a T}}. \tag {15}
$$
4 Solving VI with provable guarantees
We now specialize the optimization results of the previous section to our VI setting. We aim to minimize the negative ELBO,  $f: \mathbb{R}^d \times V \longrightarrow \mathbb{R} \cup \{+\infty\}$ , which decomposes as  $f = l + h$  where
$$
l (w) = - \underset {\mathbf {z} \sim q _ {w}} {\mathbb {E}} \log p (\mathbf {z}, x) \quad \text {a n d} \quad h (w) = \underset {\mathbf {z} \sim q _ {w}} {\mathbb {E}} \log q _ {w} (\mathbf {z}) \text {i f} C \succ 0, + \infty \text {o t h e r w i s e}, \tag {16}
$$
and where  $V$  is the vector space of matrices in which the covariance factors  $C$  belong. Under the assumption that the log-target  $\log p(\cdot, x)$  is  $M$ -smooth and concave, we propose two strategies to minimize this objective. All the proofs for this section can be found in Appendix 10.
1. Apply the Prox-SGD algorithm to the sum  $l + h$ , using a proximal step with respect to  $h$ , and a stochastic gradient step with respect to  $l$ . The gradient of  $l$  will be estimated with  $\mathbf{g}_{\mathrm{energy}}(5)$ , which is globally quadratically bounded. The prox of  $h$  admits a closed form formula if we choose that  $V$  is the space of lower triangular matrices.
2. Apply the Proj-SGD algorithm to minimize  $f$  over  $\mathcal{W}_M$ . The gradient of  $f$  will be estimated using either  $\mathbf{g}_{\mathrm{ent}}(7)$  or  $\mathbf{g}_{\mathrm{STL}}(9)$ , both of which are quadratically bounded on  $\mathcal{W}_M$ . The projection onto  $\mathcal{W}_M$  admits a closed form formula if choose that  $V$  is the space of symmetric matrices.
Corollary 12 (Prox-SGD for VI). Consider the VI problem where  $q_{w}$  is a multivariate Gaussian distribution (Eq. 2) with parameters  $w = (m, C) \in \mathbb{R}^d \times \mathcal{T}^d$ , and assume that this problem admits a solution  $w^{*}$ . Suppose that  $\log p(\cdot, x)$  is  $M$ -smooth and concave (resp.  $\mu$ -strongly concave). Generate a sequence  $w^{t}$  by using the Prox-SGD algorithm (Def. 6) applied to  $l$  and  $h$  (Eq. 16), using  $\mathbf{g}_{\mathrm{energy}}$  (5) as an estimator of  $\nabla l$ . Let the stepsizes  $\gamma_{t}$  be constant and equal to  $1 / (\sqrt{a_{\mathrm{energy}}T})$  (resp. be decaying as in Theorem 7 with  $a_{\mathrm{energy}} = 2(d + 3)M^2$ ). Then, for a certain average  $\bar{w}^T$  of the iterates, we have for  $T \geq 2$  that
$$
\mathbb {E} \left[ f (\bar {w} ^ {T}) - \inf  f \right] = \mathcal {O} (1 / \sqrt {T}) \quad (r e s p. \mathbb {E} \left[ \| w ^ {T} - w ^ {*} \| _ {2} ^ {2} \right] = \mathcal {O} (1 / T)).
$$
Corollary 13 (Proj-SGD for VI). Consider the VI problem where  $q_w$  is a multivariate Gaussian distribution (Eq. 2) with parameters  $w = (m, C) \in \mathbb{R}^d \times \mathcal{S}^d$ , and assume that this problem admits a solution  $w^*$ . Suppose that  $\log p(\cdot, x)$  is  $M$ -smooth and concave (resp.  $\mu$ -strongly concave). Generate a sequence  $w^t$  by using the Proj-SGD algorithm (Def. 9) applied to the function  $f = l + h$  (Eq. 16) and the constraint  $\mathcal{W}_M$  (Eq. 3), using  $\mathbf{g}_{\mathrm{ent}}(7)$  or  $\mathbf{g}_{\mathrm{STL}}(9)$  as an estimator of  $\nabla f$ . Let the stepsizes  $\gamma_t$  be constant and equal to  $\sqrt{2/(aT)}$  (resp. be decaying as in Theorem 10) with  $a = a_{\mathrm{ent}} = 4(d + 3)M^2$  or  $a = a_{\mathrm{STL}} = 24(d + 3)M^2$ . Then, for a certain average  $\bar{w}^T$  of the iterates, we have for  $T \geq 2$  that
Corollary 14 (Proj-SGD for VI - Gaussian target). Consider the setting of Corollary 13, in the scenario that  $\log p(\cdot ,x)$  is  $\mu$ -strongly concave, that we use the  $\mathbf{g}_{\mathrm{STL}}$  estimator, and that we take a constant stepsize  $\gamma_{t}\equiv \gamma \in \left(0,\min \{\frac{\mu}{2a_{\mathrm{STL}}},\frac{2}{\mu}\} \right]$ . Assume further that  $p(\cdot |x)$  is Gaussian. Then,
$$
\mathbb {E} \left[ \| w ^ {T} - w ^ {*} \| _ {2} ^ {2} \right] \leq \left(1 - \frac {\mu \gamma}{2}\right) ^ {T} \| w ^ {0} - w ^ {*} \| _ {2} ^ {2}.
$$
Let us now discuss the practical implementation of these two algorithms (more details and an explicit implementation of the methods are given in Appendix 10.2). These require computing either the proximal operator of the neg-entropy  $h$ , or the projection onto the set of non-degenerate covariance factors  $\mathcal{W}_M$ . These can be computed [13] for every  $w = (m, C)$  as:
-  $\operatorname{prox}_{\gamma h}(w) = (m, C + \Delta C)$ , where  $\Delta C$  is diagonal with  $\Delta C_{ii} = \frac{1}{2} (\sqrt{C_{ii}^2 + 4\gamma} - C_{ii})$ ,
-  $\operatorname{proj}_{\mathcal{W}}(w) = (m, U\tilde{D}U^{\top})$ , where  $C$  has the SVD  $C = UDU^{\top}$  and  $\tilde{D}_{ii} = \max \{D_{ii}, 1 / \sqrt{M}\}$ .
Our theory for the Prox-SGD algorithm formally assumes that the covariance factor  $C$  lives in the vector space of lower-triangular matrices. This can be implemented by letting  $C \in \mathbb{R}^{d \times d}$  and "clamping" the upper-triangular entries to zero throughout computation. Then the gradient  $\mathrm{g}_{\text{energy}}(5)$  can be computed using automatic differentiation, and the proximal operator can be computed as in the above equation. Or, one may choose  $w = (m, c)$  where  $c \in \mathbb{R}^{d \times (d + 1)/2}$  is the lower-triangular entries of the matrix so  $C = \operatorname{tril}(c)$ . Gradients with respect to  $w = (m, c)$  can again be estimated using automatic differentiation (including the tril operator), and the proximal operator can be computed by forming  $C$ , using the above formula, and then extracting the lower-triangular entries.
Similarly, our theory for the Proj-SGD algorithm formally assumes the covariance factor  $C$  lives in the vector space of symmetric matrices. This can be implemented by letting  $C \in \mathbb{R}^{d \times d}$ , computing the gradient estimator  $\mathbf{g}_{\mathrm{ent}}(7)$  or  $\mathbf{g}_{\mathrm{STL}}(9)$  over the reals using automatic differentiation, and "symmetrizing"  $C$  throughout computation. That is, set  $C \gets \frac{1}{2} C + \frac{1}{2} C^{\top}$  after applying each gradient update, right before projection. Or, one may choose  $w = (m, c)$  where  $c \in \mathbb{R}^{d \times (d + 1)/2}$  is the lower-triangular entries of the matrix, so  $C = \operatorname{symm}(c)$ , where  $\operatorname{symm}$  is a function similar to tril except creating symmetric matrices. The gradient estimator  $\mathbf{g}_{\mathrm{ent}}(7)$  or  $\mathbf{g}_{\mathrm{STL}}(9)$  can be computed
using automatic differentiation (including the symm operator). In this case, to project one would first form  $C$ , then use the above formula, and then extract the lower-triangular matrices.
In terms of cost, computing  $\log q_w(z)$  or  $T_w(u)$  needs  $\Theta(d^2)$  operations. Computing the prox of  $h$  requires  $d$  operations, but in contrast projecting onto  $\mathcal{W}_M$  requires diagonalizing a symmetric matrix, which takes  $\Theta(d^3)$  operations. We note that this is not necessarily an issue, because 1) eigenvalue decomposition has excellent computational constants in moderate dimensions; 2) computing the target  $\log p(z,x)$  may itself require  $\Theta(d^3)$  or more operations; 3) it is common to average a "minibatch" of gradient estimates, meaning each eigenvalue decomposition computation is amortized over many ELBO evaluations. Still, in high dimensions, when  $\log p(z,x)$  is inexpensive, and small minibatches are used, computing such decomposition in each iteration could become a computational bottleneck. In this scenario, the Proj-SGD algorithm is clearly cheaper, with its  $\Theta(d)$  cost for the proximal step.
While this paper has focused on convergence with dense Gaussian variational distributions, the results in Sec. 3 apply more generally. It is conceivable that the necessary smoothness, convexity, and noise bounds could be established for other variational families, in which case these optimization results would provide "plug in" guarantees. We suspect this might be fairly easy to do with, e.g., elliptical distributions or location-scale families, using similar techniques as done with Gaussians. But it may be difficult to do so for broader variational families.
If  $-\log p$  is strongly convex and smooth, another proof strategy is possible: Smoothness guarantees that  $w^{*}\in \mathcal{W}_{M}$  and strong convexity guarantees that  $w^{*}\in \{w:\| w - \bar{w}\|_{2}^{2}\leq d / \mu \}$ . So one could do projected SGD, projecting onto the intersection of these two sets. The negative ELBO would be strongly convex and smooth over that set, and since  $\| w - \bar{w}\| _2$  is bounded, gradient noise is upper-bounded by a constant, meaning classical projected SGD convergence rates that assume smoothness, strong convexity, and uniform gradient noise apply. However, projecting onto the intersection of those sets poses a difficulty and this uniform noise bound may be loose in practice.
Several variants of the gradient estimators we consider have been proposed to reduce noise, often based on control variates or sampling from a different base distribution [30, 37, 43, 44, 45, 46, 47, 48, 49, 50]. It would be interesting to establish gradient noise bounds for these estimators.
Various VI variants have been proposed that use natural gradient descent. It would also be interesting to seek convergence guarantees for these algorithms.
VI can be done with "score" or "reinforce" gradient estimators [6, 8], which use the identity
$$
\nabla_ {w} \underset {\mathbf {z} \sim q _ {w}} {\mathbb {E}} \phi (\mathbf {z}) = \underset {\mathbf {z} \sim q _ {w}} {\mathbb {E}} \phi (\mathbf {z}) \nabla_ {w} \log q _ {w} (\mathbf {z})
$$
in place of how we used Eq. 4 for the "path" estimators we considered. While path estimators often seem to have lower variance, this is not always true: Take an unnormalized log-posterior  $\phi(z)$  where  $z$  is scalar. Now, define  $\phi'(z) = \phi(z) + \sqrt{\epsilon} \sin(z / \epsilon)$ , where  $\epsilon$  is very small. Then,  $\phi$  and  $\phi'$  represent almost the same posterior, and score estimators for them will have almost the same variance. Yet, the derivative of the added term is  $\cos(z / \epsilon) / \sqrt{\epsilon}$ , so a path estimator for  $\phi'$  will have much higher variance than one for  $\phi$ . This underlines why smoothness is essential for our guarantees—it excludes posteriors like  $\phi'$ . Future work further unravel when score estimators perform better than path estimators.
One attractive feature of VI is that data subsampling can often be used when computing gradients, decreasing the cost of each iteration. While we have not explicitly addressed this, our proofs can be easily generalized, at least for target distributions of the form  $p(z,x) = p(z)\prod_{n = 1}p(x_n|z)$ . The only issue is to bound the noise of the subsampled estimator. While our gradient variance guarantees use Theorem 1 from [20], a more general result [20, Theorem 6] considers data subsampling. Very roughly speaking, with uniform data subsampling of 1 datum of a time, the gradient noise bounds in Thms. 2 and 3 would increase by a factor between 1 (no increase) and the number of data, depending on how correlated the data are—less correlation leads to a larger increase. (More precisely, the second term in Thm. 3 would not increase.) These increases would manifest as larger constants  $a$  and  $b$  in the quadratic bounds, after which exactly the same results hold. However, it is not obvious if the bound for  $\mathrm{g}_{\mathrm{STL}}$  in Thm. 4 can be generalized in this way.
We thank Kyurae Kim and an anonymous reviewer for pointing out an error in an earlier version of Lemma 32. This work was partially completed while the first two authors were visiting the Flatiron Institute and was supported in part by the National Science Foundation under Grant No. 2045900.
[1] Michael I. Jordan and Zoubin Ghahramani. An introduction to variational methods for graphical models. Machine Learning, 37(2):183-233, 1999.
[2] Zoubin Ghahramani and Matthew Beal. Propagation Algorithms for Variational Bayesian Learning. In NeurIPS, 2001.
[3] John Winn and Christopher M Bishop. Variational Message Passing. Journal of Machine Learning Research, 6:661-694, 2005.
[4] David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe. Variational Inference: A Review for Statisticians. Journal of the American Statistical Association, 112(518):859-877, 2017.
[5] Tim Salimans and David A. Knowles. Fixed-Form Variational Posterior Approximation through Stochastic Linear Regression. Bayesian Anal., 8(4):837–882, 2013.
[6] David Wingate and Theophane Weber. Automated Variational Inference in Probabilistic Programming. arXiv:1301.1299 [cs, stat], 2013.
[7] Matthew D. Hoffman, David M. Blei, Chong Wang, and John Paisley. Stochastic Variational Inference. Journal of Machine Learning Research, 14:1303-1347, 2013.
[8] Rajesh Ranganath, Sean Gerrish, and David M. Blei. Black Box Variational Inference. In AISTATS, 2014.
[9] Jeffrey Regier, Michael I Jordan, and Jon McAuliffe. Fast Black-box Variational Inference through Stochastic Trust-Region Optimization. In NeurIPS, page 10, 2017.
[10] Alp Kucukelbir, Dustin Tran, Rajesh Ranganath, Andrew Gelman, and David M. Blei. Automatic Differentiation Variational Inference. Journal of Machine Learning Research, 18(14): 1-45, 2017.
[11] C. Zhang, J. Butepage, H. Kjellström, and S. Mandt. Advances in Variational Inference. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(8):2008-2026, 2019.
[12] Yuling Yao, Aki Vehtari, Daniel Simpson, and Andrew Gelman. Yes, but Did It Work?: Evaluating Variational Inference. In ICML, 2018.
[13] Justin Domke. Provable Smoothness Guarantees for Black-Box Variational Inference. In ICML, 2020.
[14] Akash Kumar Dhaka, Alejandro Catalina, Manushi Welandawe, Michael Riis Andersen, Jonathan Huggins, and Aki Vehtari. Challenges and opportunities in high-dimensional variational inference. In NeurIPS, 2021.
[15] Manushi Welandawe, Michael Riis Andersen, Aki Vehtari, and Jonathan H. Huggins. Robust, automated, and accurate black-box variational inference, 2022.
[16] Edward Challis and David Barber. Gaussian Kullback-Leibler Approximate Inference. Journal of Machine Learning Research, 14:2239-2286, 2013.
[17] Michalis Titsias and Miguel Lázaro-Gredilla. Doubly Stochastic Variational Bayes for non-Conjugate Inference. In ICML, 2014.
[18] Kai Fan, Ziteng Wang, Jeff Beck, James Kwok, and Katherine Heller. Fast Second-Order Stochastic Backpropagation for Variational Inference. In NeurIPS, 2015.