SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(i) If  $\phi$  is linear, the energy  $f$  is  $L_{\ell}$ -smooth.
(ii) If  $\phi$  is 1-Lipschitz, the energy  $\ell$  is  $(L_{\ell} + L_{s})$ -smooth if and only if Assumption 3 holds.
Proof. See the full proof in page 27.
Combined with Lemma 1, this directly implies that the overall ELBO is smooth.
Corollary 1 (Smoothness of the ELBO). Let  $\ell$  be  $L_{\ell}$ -smooth and Assumption 3 hold. Furthermore, let the diagonal conditioner be 1-Lipschitz continuous, and  $L_{\phi}$ -log-smooth. Then, the ELBO is  $(L_{\ell} + L_{s} + L_{\phi})$ -smooth.
The increase of the smoothness constant implies that we need to use a smaller stepsize to guarantee convergence when using a nonlinear  $\phi$ . Furthermore, even on simple  $L$ -smooth examples Assumption 3 may not hold:
Example 2. Let  $\ell (\pmb {z}) = (1 / 2)\pmb{z}^{\top}\pmb {A}\pmb{z}$  and the diagonal conditioner be  $\phi (x) =$  softplus  $(x)$  . Then,
(i) if  $A$  is dense and the variational family is the mean-field family or
(ii) if  $A$  is diagonal and the variational family is the Cholesky family,
Assumption 3 holds with  $L_{s}\approx 0.26034\left(\max_{i = 1,\dots ,d}A_{il}\right)$
(iii) If  $\mathbf{A}$  is dense but the Cholesky family is used, Assumption 3 does not hold.
Proof. See the full proof in page 29.
Example 2 illustrates that establishing the smoothness of the energy becomes non-trivial under nonlinear parameterizations. Even when smoothness does hold, the increased smoothness constant implies that BBVI will be less robust to initialization and step-sizes. Furthermore, in the next section, we will show a much more grave problem: nonlinear parameterizations may affect the convergence rate.
3.4 Convexity of the Energy
The convexity of the ELBO under linear parameterizations has first been established by Titsias & Lázaro-Gredilla (2014, Proposition 1) and Domke (2020, Theorem 9). In particular, Domke (2020) show that, when  $\phi$  is linear, if  $\ell$  is  $\mu$ -strongly convex, the energy is also  $\mu$ -strongly convex. However, when using a nonlinear  $\phi$  with a co-domain of  $\mathbb{R}_+$ , which is the whole point of using a nonlinear conditioner, strong convexity of  $\ell$  never transfers to  $f$ .
ELBO with  $\phi (x) =$  softplus(x)
ELBO with  $\phi (x) = x$
Lower-Bounding Quadratic
Tangent Line at  $s_1 = -5$
Figure 2: Optimization landscape resulting from different  $\phi$  on a strongly-convex  $\ell$ .  $\ell$  is the counter-example of Proposition 1. Item (ii).  $\phi(x) = x$  preserves strong convexity as shown by the lower-bounding quadratic (red dotted line ...).  $\phi =$  softplus violates the first-order condition of convexity (black dotted line ...).
Theorem 2. Let  $\ell$  be  $\mu$ -strongly convex. Then, we have the following:
(i) If  $\phi$  is linear, the energy  $f$  is  $\mu$ -strongly convex.
(ii) If  $\phi$  is convex, the energy  $f$  is convex if and only if Assumption 4 holds.
(iii) If  $\phi$  is such that  $\phi \in C^1 (\mathbb{R},\mathbb{R}_+)$ , the energy  $f$  is not strongly convex.
Proof. See the full proof in page 33.
The following proposition provides some conditions for Assumption 4 to hold or not hold.
Proposition 1. We have the following:
(i) If  $\ell$  is convex, then for the mean-field family, Assumption 4 holds.
(ii) For the Cholesky family, there exists a convex  $\ell$  where Assumption 4 does not hold.
Proof. See the full proof in page 31.
For any continuous, differentiable nonlinear conditioner that maps only to non-negative reals, the strong convexity of  $\ell$  does lead to a strongly-convex ELBO. This phenomenon is visualized in Figure 2. The loss surface becomes flat near the optimal scale parameter. This problem becomes more noticeable as the optimal scale becomes smaller.
Nonlinear conditioners are suboptimal. As the dataset grows, Bayesian posteriors are known to "contract" as characterized by the Bernstein-von Mises theorem (van der Vaart, 1998). That is, the posterior variance becomes close to 0. This behavior also applies to misspecified variational posteriors as shown by Wang & Blei (2019). Thus, for large datasets, nonlinear conditioners mostly operate in the regime where they are suboptimal (locally less strongly convex). But linear conditioners result in a non-smooth entropy (Domke, 2020). This dilemma originally motivated Domke to consider proximal SGD, which we analyze in Section 4.2.
4 Convergence Analysis of Black-Box Variational Inference
4.1 Black-Box Variational Inference
BBVI with SGD repeats the steps:
$$
\lambda_ {t + 1} = \lambda_ {t} - \gamma_ {t} (\widehat {\nabla f} (\lambda_ {t}) + \nabla h (\lambda_ {t})) \text {, w h e r e} \quad \widehat {\nabla f} (\lambda_ {t}) = \frac {1}{M} \sum_ {m = 1} ^ {M} \nabla_ {\lambda} \ell (\mathcal {T} _ {\lambda} (\boldsymbol {u} _ {m})) \tag {1}
$$
with  $\pmb{u}_m \sim \varphi$  is the  $M$ -sample reparameterization gradient estimator and  $\gamma_t$  is the stepsize. (See Kucukelbir et al., 2017 for algorithmic details.)
With our results in Section 3 and the results of Khaled & Richtárik (2023); Kim et al. (2023), we obtain a convergence guarantee. To apply the result of Kim et al. (2023), which bounds the gradient variance, we require an additional assumption.
Assumption 5. The negative log-likelihood  $\ell_{\mathrm{like}}(\pmb {z})\triangleq -\log p(\pmb {x}\mid \pmb {z})$  is  $\mu$ -quadratically growing for all  $\pmb {z}\in \mathbb{R}^d$  such that
$$
\frac {\mu}{2} \| \boldsymbol {z} - \bar {\boldsymbol {z}} _ {\text {l i k e}} \| _ {2} ^ {2} \leq \ell_ {\text {l i k e}} (\boldsymbol {z}) - \ell_ {\text {l i k e}} ^ {*},
$$
where  $\bar{z}_{\mathrm{like}}$  is the projection of  $\mathbf{z}$  to the set of minimizers of  $\ell_{\mathrm{like}}$ , and  $\ell_{\mathrm{like}}^{*} = \inf_{\mathbf{z} \in \mathbb{R}^{d}} \ell_{\mathrm{like}}(\mathbf{z})$ .
This assumption is weaker than assuming that the likelihood satisfies the Polyak-Lojasiewicz inequality (Karimi et al., 2016).
Theorem 3. Let Assumption 2 hold, the likelihood satisfy Assumption 5, and the assumptions of Corollary 1 hold such that the ELBO  $F$  is  $L_{F}$ -smooth with  $L_{F} = L_{\ell} + L_{\phi} + L_{s}$ . Then, the iterates generated by BBVI through Equation (1) and the M-sample reparameterization gradient include an  $\epsilon$ -stationary point such that  $\min_{0 \leq t \leq T-1} \mathbb{E}\|\nabla F(\lambda_t)\|_2 \leq \epsilon$  for any  $\epsilon > 0$  if
$$
T \geq \mathcal {O} \left(\frac {\left(F \left(\lambda_ {0}\right) - F ^ {*}\right) ^ {2} L _ {F} L _ {\ell} ^ {2} C \left(d , k _ {\varphi}\right)}{\mu M \epsilon^ {4}}\right)
$$
for some fixed stepsize  $\gamma$ , where  $C(d, \varphi) = d + k_{\varphi}$  for the Cholesky family and  $C(d, \varphi) = 2k_{\varphi}\sqrt{d} + 1$  for the mean-field family.
Proof. See the full proof in page 35.
Remark 3. Finding an  $\varepsilon$ -stationary point of the ELBO has an iteration complexity of  $\mathcal{O}\left(dL_{\ell}^{2}\kappa M^{-1}\varepsilon^{-4}\right)$  for the Cholesky family and  $\mathcal{O}\left(\sqrt{d} L_{\ell}^{2}\kappa M^{-1}\varepsilon^{-4}\right)$  for the mean-field family.
4.2 Black-Box Variational Inference with Proximal SGD
Proximal SGD For a composite objective  $F = f + h$ , proximal SGD repeats the steps:
$$
\lambda_ {t + 1} = \operatorname {p r o x} _ {\gamma_ {t}, h} (\lambda_ {t} - \gamma_ {t} \widehat {\nabla f} (\lambda_ {t})) = \underset {\lambda \in \Lambda} {\arg \min } \left[ \left\langle \widehat {\nabla f} (\lambda_ {t}), \lambda \right\rangle + h (\lambda) + \frac {1}{2 \gamma_ {t}} \| \lambda - \lambda_ {t} \| _ {2} ^ {2} \right], \tag {2}
$$
where prox is known as the proximal operator and  $\gamma_{1},\ldots ,\gamma_{T}$  is a stepsize schedule.
In the context of VI, proximal SGD has previously been considered by Altosaar et al. (2018); Dao et al. (2023); Khan et al. (2016, 2015). Their overall focus has been on developing alternative algorithms by generalizing  $\| \lambda -\lambda^{*}\|$  to other metrics. In contrast, Domke (2020) considered proximal SGD with the regular Euclidean metric  $\| \lambda -\lambda^{*}\|_{2}$  for overcoming the non-smoothness of  $h$  under
linear parameterizations. Here, we prove the convergence of this scheme and show that it retrieves the fastest known convergence rates in stochastic first-order optimization.
Proximal Operator for BBVI In our context,  $h$  is the entropy of  $q_{\lambda}$  in the location-scale family. For this, Domke (2020) show that the proximal update for  $s_1, \ldots, s_d$ , is
$$
\operatorname {p r o x} _ {\gamma_ {t}, h} (s _ {i}) = s _ {i} + \frac {1}{2} \left(\sqrt {s _ {i} ^ {2} + 4 \gamma_ {t}} - s _ {i}\right).
$$
For other parameters, the proximal operator is the regular gradient descent update in Equation (1).
Gradient Variance Bound We first establish a bound on the gradient variance. In ERM, contemporary strategies do this by exploiting the finite sum structure of the objective (Section 2.4). Here, we establish a variance bound for RP estimator that does not rely on the finite sum assumption.
Lemma 3 (Convex Expected Smoothness). Let  $\ell$  be  $L_{\ell}$ -smooth and  $\mu$ -strongly convex with the variational family satisfying Assumption 2 with the linear parameterization. Then,
$$
\mathbb {E} \| \nabla_ {\lambda} f (\lambda ; \boldsymbol {u}) - \nabla_ {\lambda^ {\prime}} f (\lambda^ {\prime}; \boldsymbol {u}) \| _ {2} ^ {2} \leq 2 L _ {\ell} \kappa C (d, \varphi) \mathrm {B} _ {f} (\lambda , \lambda^ {\prime})
$$
holds, where  $\mathrm{B}_f(\lambda ,\lambda ')\triangleq f(\lambda) - f(\lambda ') - \langle \nabla f(\lambda '),\lambda -\lambda '\rangle$  is the Bregman divergence,  $\kappa = L_{\ell} / \mu$  is the condition number;  $C(d,\varphi) = d + k_{\varphi}$  for the Cholesky family, and  $C(d,\varphi) = 2k_{\varphi}\sqrt{d} +1$  for the mean-field family.
Proof. See the full proof in page 36.
Furthermore, the gradient variance at the optimum must be bounded:
Lemma 4 (Domke, 2019; Kim et al., 2023). Let  $\ell$  be  $L_{\ell}$ -smooth with the variational family satisfying Assumption 2 and a 1-Lipschitz diagonal conditioner  $\phi$ . Then, the gradient variance at the optimum  $\lambda^{*} \in \arg \min_{\lambda \in \Lambda} F(\lambda)$  is bounded as
$$
\sigma^ {2} \leq \frac {1}{M} C (d, \varphi) L _ {\ell} ^ {2} \left(\| \bar {\boldsymbol {z}} - \boldsymbol {m} ^ {*} \| _ {2} ^ {2} + \| \boldsymbol {C} ^ {*} \| _ {\mathrm {F}} ^ {2}\right),
$$
where  $\bar{\mathbf{z}}$  is a stationary point of  $\ell$ ,  $\mathbf{m}^*$  and  $\mathbf{C}^*$  are the location and scale formed by  $\lambda^*$ , the constants are  $C(d, \varphi) = d + k_{\varphi}$  for the Cholesky family and  $C(d, \varphi) = 2k_{\varphi}\sqrt{d} + 1$  for the mean-field family,  $k_{\varphi}$  is the kurtosis of  $\varphi$  as defined in Assumption 1.
Proof. The full-rank case is proven by Domke (2019, Theorem 3), while the mean-field case is a basic corollary of the result by Kim et al. (2023, Lemma 2).  $\square$
Remark 4. The dimensional dependence in the complexity of BBVI is transferred from the variance bound in Lemma 4. Unfortunately, for the Cholesky family, this dimensional dependence in the variance bound is tight (Domke, 2019).
Main Result With the gradient variance bounds, we now present our complexity result. The proof is identical to Theorem 3.2 by Gower et al. (2019), where they use a 2-stage decreasing stepsize schedule: the stepsize is initially held constant and then reduced in a  $1/t$  rate.
Theorem 4. Let  $\ell$  be  $L_{\ell}$ -smooth and  $\mu$ -strongly convex. Then, for any  $\epsilon > 0$ , BBVI with proximal SGD in Equation (2), the  $M$ -sample reparameterization gradient estimator, a variational family satisfying Assumption 2 with the linear parameterization guarantees  $\mathbb{E}\|\lambda_T - \lambda^*\|_2^2 \leq \epsilon$  if
$$
\gamma_ {t} = \left\{ \begin{array}{l l} \frac {M}{2 L _ {\ell} \kappa C (d , \varphi)} & f o r t \leq 4 T _ {\kappa} \\ \frac {2 t + 1}{(t + 1) ^ {2} \mu} & f o r t > 4 T _ {\kappa}, \end{array} \right. \quad T \geq \max \left(\frac {8 \sigma^ {2}}{\mu^ {2} \epsilon} + \frac {4 T _ {\kappa} \| \lambda_ {0} - \lambda^ {*} \| _ {2}}{\mathrm {e} \sqrt {\epsilon}}, 4 T _ {\kappa}\right)
$$
where  $\sigma^2$  is defined in Lemma 4,  $T_{\kappa} = [\kappa^{2}C(d,\varphi)M^{-1}]$ ,  $\kappa = L_{\ell} / \mu$  is the condition number, e is Euler's constant,  $\lambda^{*} = \arg \min_{\lambda \in \Lambda}F(\lambda)$ ,  $C(d,\varphi) = d + k_{\varphi}$  for the Cholesky family, and  $C(d,\varphi) = 2k_{\varphi}\sqrt{d} +1$  for the mean-field family.
Proof. See the full proof in page 38.
Remark 5. BBVI with proximal SGD on  $\mu$ -strongly convex and  $L_{\ell}$ -smooth  $\ell$  has a complexity  $\mathcal{O}\left(\kappa^{2}dM^{-1}\epsilon^{-1}\right)$  for the Cholesky family and  $\mathcal{O}\left(\kappa^{2}\sqrt{d} M^{-1}\epsilon^{-1}\right)$  for the mean-field family.
Remark 6. We also provide a similar result with a fixed stepsize in Theorem 7 of Appendix F.3.2. In this case, the complexity is  $\mathcal{O}\left(\kappa^2 dM^{-1}\varepsilon^{-1}\log \varepsilon^{-1}\right)$  for the Cholesky family and  $\mathcal{O}\left(\kappa^2\sqrt{d} M^{-1}\varepsilon^{-1}\log \varepsilon^{-1}\right)$  for the mean-field family.
Figure 3: Stepsize versus the number of iterations for vanilla SGD and proximal SGD to achieve  $\mathrm{D}_{\mathrm{KL}}(q_{\lambda}, \pi) \leq \epsilon = 1$  under different initializations for Gaussian posteriors. The initializations  $C(\lambda_0)$  are  $\mathbf{I}$ ,  $10^{-3}\mathbf{I}$ ,  $10^{-5}\mathbf{I}$  from left to right, respectively. The average suboptimality at iteration  $t$  was estimated from 10 independent runs. For each run, the target posterior was a 10-dimensional Gaussian with a covariance with a condition number  $\kappa = 10$  and a smoothness of  $L = 100$ .
Figure 4: Comparison of BBVI convergence speed (ELBO v.s. Iteration) of different optimization algorithms. The error bands are the  $80\%$  quantiles estimated from 20 (10 for AR-eeg) independent replications. The results shown used a base stepsize of  $\gamma = 10^{-3}$ , while the initial point was  $\pmb{m}_0 = \mathbf{0}, \pmb{C}_0 = \mathbf{I}$ . Details on the setup can be found in the text of Section 5.2 and Appendix G.
5.1 Synthetic Problem
Setup We first compare proximal SGD against vanilla SGD with linear and nonlinear parameterizations on a synthetic problem, which is log-smooth, strongly log-concave, and the exact solution is known. While a similar experiment was already conducted by Domke (2020), here we include nonlinear parameterizations, which were not originally considered. We run all algorithms with a fixed stepsize to infer a multivariate Gaussian with a full-rank covariance matrix. The variational approximation is a full-rank Gaussian formed by  $\varphi = \mathcal{N}(0,1)$  and the Cholesky parameterization.
Results The results are shown in Figure 3. Proximal SGD is clearly the most robust against initialization. Also, SGD with the nonlinear parameterization  $\phi(x) = \text{softplus}(x)$  is much slower to converge under all initializations. This confirms that linear parameterizations are indeed superior for both robustness against initializations and convergence speed.
5.2 Realistic Problems
Setup We now evaluate proximal SGD on realistic problems. In practice, Adam (Kingma & Ba, 2015) is observed to be robust against stepsize choices (Zhang et al., 2019). The reason why Adam performs well on non-smooth, non-convex problems is still under investigation (Kunstner et al., 2023; Reddi et al., 2023; Zhang et al., 2022). Nonetheless, to compare fairly against Adam, we implement a recently proposed variant of proximal SGD called ProxGen (Yun et al., 2021), which
includes an Adam-like update rule. The probabilistic models and datasets are fully described in Appendix G. We implement these models and BBVI on top of the Turing (Ge et al., 2018) probabilistic programming framework. Due to the size of these datasets, we implement doubly stochastic subsampling (Titsias & Lázaro-Gredilla, 2014) with a batch size of  $B = 100$  ( $B = 500$  for BT-tennis) with  $M = 10$  Monte Carlo samples. For batch subsampling, we implement random reshuffling, which is faster than independent subsampling both empirically (Bottou, 2009) and theoretically (Ahn et al., 2020; Haochen & Sra, 2019; Mishchenko et al., 2020; Nagaraj et al., 2019). We also observe that doubly stochastic BBVI benefits from reshuffling, but leave a detailed investigation to future works.
Results Representative results are shown in Figure 4, with additional results in Appendix H. Both ProxGen-Adam and Adam with linear parameterizations converge faster than Adam with nonlinear parameterization. Furthermore, for the case of election and buzz, Adam with the nonlinear parameterization converges much slower than the alternatives. When using linear parameterizations, ProxGen-Adam appears to be generally faster than Adam. We note, however, that due to the difference in the update rule between ProxGen-Adam and Adam, proximal operators alone might not fully explain the performance difference. Nevertheless, the results of our experiment do conclusively suggest that linear parameterizations are superior.
Conclusions In this work, we have proven the convergence of BBVI. Our assumptions encompass implementations that are actually used in practice, and our theoretical analysis revealed limitations in some of the popular design choices (mainly the use of nonlinear conditioners). To resolve this issue, we re-evaluated the utility of proximal SGD both theoretically and practically, where it achieved the strongest theoretical guarantees in stochastic first-order optimization.
Related Works To prove the convergence of BBVI, early works have  $a$ -priori "assumed" the regularity of the ELBO and the gradient estimator (Alquier & Ridgway, 2020; Buchholz et al., 2018; Khan et al., 2016, 2015; Liu & Owen, 2021; Regier et al., 2017). Towards a more rigorous understanding, Domke (2019); Fan et al. (2015); Kim et al. (2023); Xu et al. (2019) studied the reparameterization gradient, Xu & Campbell (2022) studied the asymptotics of the ELBO, Challis & Barber (2013); Domke (2020); Titsias & Lázaro-Gredilla (2014) established convexity, and Domke (2020) established smoothness. On the other hand, Bhatia et al. (2022); Hoffman & Ma (2020) established rigorous convergence guarantees by considering simplified variant of BBVI where only the scale is optimized, and Fujisawa & Sato (2021) assumed that the support of  $\varphi$  is bounded almost surely. Meanwhile, under similar assumptions to ours, Diao et al. (2023); Lambert et al. (2022) recently established convergence guarantees for proximal SGD BBVI with a Bures-Wasserstein metric. Their computational properties differ from BBVI as they require Hessian evaluations. Also, understanding BBVI, which is VI with a Euclidean metric, is an important problem due to its practical relevance.
**Limitations** Our work has multiple limitations: (i) Our results are restricted to the location-scale family, (ii) the reparameterization gradient, and (iii) smooth joint log-likelihoods. However, the location-scale family with the reparameterization gradient is the most widely used combination in practice, and replacing the smoothness assumption is an active area of research in stochastic optimization. For our results on proximal SGD, we further assume that the joint log-likelihood is  $\mu$ -strongly convex (equivalently strongly log-concave posteriors). It is unclear how to extend the guarantees to only smooth but non-log-concave joint log-likelihoods.
Open Problems Although we have proven that the mean-field dimensional family has a dimension dependence of  $\mathcal{O}\left(\sqrt{d}\right)$ , empirical results suggest room for improvement (Kim et al., 2023). Therefore, we pose the following conjecture:
Conjecture 1. Under mild assumptions, BBVI for the mean-field variational family converges with only logarithmic dimensional dependence or no explicit dimensional dependence at all.
This would put mean-field BBVI in a regime clearly faster than approximate MCMC (Freund et al., 2022). Also, it is unknown whether the  $\mathcal{O}(\kappa^2)$  condition number dependence dependence is tight. In fact, for proximal SGD BBVI in Bures-Wasserstien space, Diao et al. (2023) report a dependence of  $\mathcal{O}(\kappa)$ . Lastly, it would be interesting to see whether natural gradient VI (NGVI; Amari, 1998; Khan & Lin, 2017) can achieve similar convergence guarantees. While it is empirically known that NGVI often converges faster (Lin et al., 2019), theoretical evidence has yet to follow.
The authors would like to thank Justin Domke for discussions on the concurrent results, Javier Burroni for pointing out a mistake in the earlier version of this work, and the anonymous reviewers for their constructive comments.
K. Kim and J. R. Gardner were funded by the National Science Foundation Award [IIS-2145644], while Y.-A. Ma was funded by the National Science Foundation Grants [NSF-SCALE MoDL-2134209] and [NSF-CCF-2112665 (TILOS)], the U.S. Department Of Energy, Office of Science, and the Facebook Research award.
Zhang, Yushun, Chen, Congliang, Shi, Naichen, Sun, Ruoyu, & Luo, Zhi-Quan. 2022. Adam Can Converge without Any Modification on Update Rules. Pages 28386-28399 of: Advances in Neural Information Processing Systems, vol. 35. Curran Associates, Inc. (page 9)
On the Convergence of Black-Box Variational Inference Appendix
1 Introduction 1
2 Background 2
2.1 Black-Box Variational Inference 2
2.2 Variational Family 3
2.3 Scale Parameterizations 3
2.4 Problem Structure of Black-Box Variational Inference 4
3 The Evidence Lower Bound Under Nonlinear Scale Parameterizations 5
3.1 Technical Assumptions 5
3.2 Smoothness of the Entropy 5
3.3 Smoothness of the Energy 5
3.4 Convexity of the Energy 6
4 Convergence Analysis of Black-Box Variational Inference 7
4.1 Black-Box Variational Inference 7
4.2 Black-Box Variational Inference with Proximal SGD 7
5.1 Synthetic Problem 9
5.2 Realistic Problems 9
6 Discussions 10
A Computational Resources 18
B Nomenclature 18
C Definitions 19
D ProxGen Adam for Black-Box Variational Inference 20
E Detailed Comparison Against Domke et al. (2023) 21
F Proofs 22
F.1 Auxiliary Lemmas 22
F.2 Properties of the Evidence Lower Bound 24
F.2.1 Smoothness 24
F.2.2 Convexity 30
F.3 Convergence of Black-Box Variational Inference 34
F.3.1 Vanilla Black-Box Variational Inference 34
F.3.2 Proximal Black-Box Variational Inference 36
G Details of Experimental Setup 40
H Additional Experimental Results 42
A Computational Resources
Table 1: Computational Resources
Type	Model and Specifications
System Topology	2 nodes with 2 sockets each with 24 logical threads (total 48 threads)
Processor	1 Intel Xeon Silver 4310, 2.1 GHz (maximum 3.3 GHz) per socket
Cache	1.1 MiB L1, 30 MiB L2, and 36 MiB L3
Memory	250 GiB RAM
Accelerator	1 NVIDIA RTX A5000 per node, 2 GHZ, 24GB RAM
Running the experiments took approximately a week.
B Nomenclature
Symbol	Definition	Description	Section
λ	Variational parameters	2.1
z	Parameters of the target model π	2.1
Tλ	≌ C(λ)u+m	location-scale reparameterization function	2.2
u	Random vector before reparameterization	2.2
φ	Base distribution of u	2.2
m	Location parameter (part of λ)	2.2
C	Scale parameter (part of λ)	2.2 and 2.3
φ	Diagonal conditioner	2.3
kφ	Kurtosis (non-central 4th moment) of u	2.3
Dφ(s)	Diagonal of C using the diagonal conditioner φ	2.3
L	Strictly lower triangular part of C	2.3
s	Elements forming the diagonal of C	2.3
ℓ(z)	≌ -log p(z, x)	Negative joint likelihood	2.4
f(λ)	≌ Ez~qλ ℓ(z)	Energy	2.4
h(λ)	≌ -H(qλ)	Negative entropy	2.4
F(λ)	≌ f(λ) + h(λ)	Negative ELBO	2.1 and 2.4
f(λ; u)	≌ ℓ(Tλ(u))	Negative Log-Likelihood under reparameterization	2.4
g(λ; u)	≌ ∇ ℓ(Tλ(u))	Gradient of the Log-likelihood under reparameterization	2.4
M	Number of Monte Carlo samples	2.1
γt	Stepsize of (proximal) SGD at iteration t	4.1 and 4.2
∇f	Reparameterization gradient estimator of the energy	4.1
C Definitions
For completeness, we provide formal definitions for some of the terms we used throughout the paper.
Definition 5 (Smoothness). A function  $f: \mathcal{Z} \to \mathbb{R}$  is said to be  $L$ -smooth if the inequality