Title: Provable convergence guarantees for black-box variational inference
Venue: NeurIPS 2023
Accepted: yes
Keywords: optimization

variational inference
Abstract: Black-box variational inference is widely used in situations where there is no proof that its stochastic optimization succeeds. We suggest this is due to a theoretical gap in existing stochastic optimization proofs—namely the challenge of gradient estimators with unusual noise bounds, and a composite non-smooth objective. For dense Gaussian variational families, we observe that existing gradient estimators based on reparameterization satisfy a quadratic noise bound and give novel convergence guarantees for proximal and projected stochastic gradient descent using this bound. This provides rigorous guarantees that methods similar to those used in practice converge on realistic inference problems.

Provable convergence guarantees for black-box variational inference

Abstract

Black-box variational inference is widely used in situations where there is no proof that its stochastic optimization succeeds. We suggest this is due to a theoretical gap in existing stochastic optimization proofs-namely the challenge of gradient estimators with unusual noise bounds, and a composite non-smooth objective. For dense Gaussian variational families, we observe that existing gradient estimators based on reparameterization satisfy a quadratic noise bound and give novel convergence guarantees for proximal and projected stochastic gradient descent using this bound. This provides rigorous guarantees that methods similar to those used in practice converge on realistic inference problems.

1 Introduction

Variational inference tries to approximate a complex target distribution with a distribution from a simpler family [1, 2, 3, 4]. If p ( z, x ) is a target distribution with latent variables z ∈ R d and observed data x ∈ R n , and q w is a variational family with parameters w ∈ W , the goal is to minimize the negative evidence lower bound (ELBO)

For subsequent discussion, we have decomposed the objective into the free energy l and the negative entropy h . Minimizing f is equivalent to minimizing the KL-divergence between q w to p ( ·| x ) , because KL ( q w ∥ p ( ·| x )) = f ( w ) + log p ( x ) . Recent research has focused on "black box" variational inference, where the target distribution p is sufficiently complex that one can only access it through evaluating probabilities (or gradients) at chosen points [5, 6, 7, 8, 9, 10, 11]. Crucially, one can still get stochastic estimates of the variational objective f and of its gradient, and use these to optimize.

Still, variational inference can sometimes be unreliable [12, 13, 14, 15], and some basic questions remain unanswered. Most notably: does stochastic optimization of f converge to a minimum of the objective? There has been various progress towards answering this question. One line of research seeks to determine if the variational objective f has favorable structural properties, such as smoothness or (strong) convexity [13, 16, 17] (Sec. 2.1). Another line seeks to control the "noise" (variance or expected squared norm) of different gradient estimators [18, 19, 20, 21] (Sec. 2.2). However, few

full convergence guarantees are known. That is, there are few known cases where applying a given stochastic optimization algorithm to a given target distribution is known to converge at a given rate.

We identify two fundamental barriers preventing from analysing this VI problem as a standard stochastic optimization problem. First, the gradient noise depends on the parameters w in a nonstandard way (Sec. 2.3). This adds great technical complexity and renders many traditional stochastic optimization proofs inapplicable. Second, stochastic optimization theory typically requires that the (exact) objective function f is Lipschitz continuous or Lipschitz smooth. But in our VI setting, under some fairly benign assumptions, the ELBO is neither Lipschitz continuous nor Lipschitz smooth.

We obtain non-asymptotic convergence guarantees for this problem, under simple assumptions.

Central contributions (Informal). Suppose that the target model log p ( · , x ) is concave and Lipschitz-smooth, and that q w is in a Gaussian variational family parameterized by the mean and a factor of the covariance matrix (Eq. 2). Consider minimizing the negative ELBO f using either one of the two following algorithms:

a proximal stochastic gradient method, with the proximal step applied to h and the gradient step applied to l , estimating ∇ l ( w ) with a standard reparameterization gradient estimator (Eq. 5), and using a triangular covariance factor;

a projected stochastic gradient method, with the gradient applied to f = l + h , estimating ∇ f ( w ) using either of two common gradient estimators (Eq. 7 or Eq. 9), with the projection done over symmetric and non-degenerate (Eq. 3) covariance factors

Then, both algorithms converge with a 1 / √ T complexity rate (Cor. 12), or 1 /T if we further assume that log p ( · , x ) is strongly concave (Cor. 13).

We also give a new bound on the noise of the "sticking the landing" gradient estimator, which leads to faster convergence when the target distribution p is closer to Gaussian, up to exponentially fast convergence when it is exactly Gaussian (Cor. 14).

This is achieved through a series of steps, that we summarize below.

We analyze the structural properties of the problem. Existing results show that with a Gaussian variational family, if -log p ( · , x ) is (strongly) convex or Lipschitz smooth, then so is the free energy l . This is for instance known to be the case for some generalized linear models, and we give a new proof of convexity and smoothness for some hierarchical models including hierarchical logistic regression (see Appendix 7.3). The remaining component of the ELBO, the neg-entropy h , is convex when restricted to an appropriate set. It is not smooth, but it was recently proved to be smooth over a certain non-degeneracy set.

We study the noise of three common gradient estimators . They do not satisfy usual noise bounds, but we show that they all satisfy a new quadratic bound (Definition 5). For the sticking-thelanding estimator, our bound formalizes the longstanding intuition that it should have lower noise when the variational approximation is strong (Thm. 4).

We identify and solve the key optimization challenges posed by the above issues via new convergence results for the proximal and projected stochastic gradient methods, when applied to objectives that are smooth (but not uniformly smooth) and with gradient estimators satisfying our quadratic bound.

1.1 Related work

Recently, Xu and Campbell [22] analyzed projected-SGD (stochastic gradient descent) for Gaussian VI using the gradient estimator we will later call g ent (7), with a particular rescaling. They show that, asymptotically in the number of observed data, their method converges locally with a rate of 1 / √ T , under mild assumptions. Our results are less general in requiring convexity, but are non-asymptotic, apply with other gradient estimators, and give a faster 1 /T rate with strong convexity.

Lambert et al. [23] introduce a VI-like SGD algorithm, derived from a discretisation of a Wasserstein gradient flow, and show it converges at a 1 /T rate for the 2-Wasserstein distance when the log posterior is smooth and strongly concave. This line was continued by Diao et al. [24], who propose a

proximal-SGD method based on the decomposition f = l + h . They obtain a 1 / √ T rate when the log posterior is smooth and convex, and a 1 /T rate when it is smooth and strongly convex. Unlike typical black-box VI algorithms used in practice, these algorithms require computing the Hessian of the posterior. We analyze more straighforward applications of SGD to the VI problem using standard gradient estimators. Under the same assumptions, our algorithms have the same rates for KL-divergence, which imply the same rates for 2-Wasserstein distance by of Pinsker's inequality and that the total-variation norm upper-bounds Wasserstein distance [25, Remark 8.2].

In concurrent work, Kim et al. [26], consider a proximal-SGD method similar to our approach in Sec. 6. They obtain a 1 /T convergence rate, similar to what Cor. 12 gives when the log posterior is strongly concave. They also consider alternative parametrizations of the scale parameters that render the ELBO globally smooth, and obtain a nonconvex result: Under a relaxed version of the Polyak-Lojasiewicz inequality, they can guarantee a 1 /T 4 rate.

2 Properties of Variational Inference (VI) problems

Traditionally, the ELBO was optimized using message passing methods, which essentially assume that p and q are simple enough that block-coordinate updates are possible [2, 3, 4]. However, in the last decade a series of papers developed algorithms based on a 'black-box' model where p is assumed to be complex enough that one can only evaluate log p (or its gradient) at selected points z . The key observation is that even if p is quite complex, it is still possible to compute stochastic estimates of the gradient of the ELBO, which can be deployed in a stochastic optimization algorithm [5, 6, 8, 9, 10].

This paper seeks rigorous convergence guarantees for this problem. We study the setting where the variational family is the set of (dense) multivariate Gaussian distributions, parameterized in terms of w = ( m,C ) , where m is the mean and C is a factor of the covariance, i.e.

We optimize over w ∈ W , where W = { ( m,C ) : C ≻ 0 } and will further require C to be either symmetric or triangular. This does not really change the problem, since for a given covariance matrix Σ there always exists a symmetric (or lower triangular) positive definite factor C such that CC ⊤ = Σ .

Now, is it possible to solve this optimization problem? Without further assumptions, it is unlikely any guarantee is possible, because it is easy to encode NP-hard problems into this VI framework [27, 28]. We discuss below the assumptions we will make to be able to solve the problem.

2.1 Structural properties and assumptions

The properties of the free energy l depend on the properties of the target distribution p . It is necessary to assume that p is somehow 'nice' to ensure the problem can be solved. Titsias and Lázaro-Gredilla [17, Proposition 1] showed that if -log p ( · , x ) is convex, then l is convex too. Challis and Barber [16, Sec 3.2] showed that if the likelihood p ( ·| z ) is convex and the prior p ( z ) is Gaussian, then l is strongly-convex. Domke [13, Theorem 9] showed that if -log p ( · , x ) is µ -strongly convex, then l is µ -strongly convex as well, and that the constant is sharp. Similarly, Domke [13, Theorem 1] showed that if log p ( · , x ) is M -smooth, then l is also M -smooth, and that the constant is sharp.

In this paper we make two assumptions about the target distribution p : the negative log-probability -log p ( · , x ) must be convex (or strongly convex), and Lipschitz smooth . Section 7.3 (Appendix) gives some example models where these assumptions are satisfied. For example, if the model is Bayesian linear regression, or logistic regression with a Gaussian prior, then -log p ( · , x ) is smooth and strongly convex. In addition, if the target is a hierarchical logistic regression model, then -log p ( · , x ) is smooth and convex.

Assumptions on p also impact what a minimizer w ∗ = ( m ∗ , C ∗ ) of f ( w ) can look like. Intuitively, if the target log p ( · , x ) is µ -strongly concave, then we would expect the target distribution to be 'peaky'. This means that the optimal distribution would be close to a delta function centered at the MAP solution: m ∗ would be close to some maximum of log p ( · , x ) noted ¯ m , and the covariance factor C ∗ would not be too large. This intuition can be formalized: in this context we have ∥ C ∗ ∥ 2 + ∥ m ∗ -¯ m ∥ 2 2 ≤ d/µ [13, Theorem 10]. Similarly, if log p ( · , x ) is M -Lipschitz smooth, then we expect that the target is not too concentrated, so we might expect that the optimal covariance cannot not be too

small. Formally, it can be shown that the singular values of the covariance factor C ∗ are greater than 1 / √ M [13, Theorem 7].

The properties of the neg-entropy h are inherited from the choice of the variational family and do not depend on p . Since we consider a Gaussian variational family, h is known in closed-form. To avoid some technical issues, we will restrict h so that the covariance factor is positive definite, so h ( w ) is equal to -ln det C up to a constant if C is positive definite (see Appendix 7.1), and to + ∞ otherwise . So h inherits the properties of the negative log determinant, meaning it is a proper closed convex function whenever C is symmetric or triangular (see Appendix 7.2). From an optimization perspective, it is natural to use a gradient-based algorithm. But unfortunately h is not Lipschitz smooth on its domain, because its gradient can change arbitrarily quickly when the singular values of C become small. However, it can be shown [13, Lemma 12] that h is M -smooth over the following set of non-singular parameters (which contains the solution w ∗ , see the previous paragraph) given by

Instead of computing gradients, an optimizer might want to use the proximal operator of h , which can also be computed in closed form (see Sec. 4).

2.2 Gradient estimators

This paper considers gradient estimators based on the 'path' method or 'reparameterization'. These assume some base distribution s is known, together with some deterministic transformation T w , such that the distribution of T w ( u ) is equal to q w when u ∼ s . Then, we can write

In the case of multivariate Gaussians, the most common choice is to use s = N (0 , I ) and T w ( u ) = Cu + m , which we will consider in the rest of the paper.

We will consider three different gradient estimators based on the path-type strategy (Eq. 4). We will provide new noise bounds for each of them (all the proofs are in Appendix 8). Our analysis is mostly based on the following general result [20, Thm. 3].

Theorem 1. Let T w ( u ) = Cu + m for w = ( m,C ) . Let ϕ : R d → R be M -smooth, suppose that ϕ is stationary at ¯ m , and define ¯ w = ( ¯ m, 0) . Then

Furthermore, the first inequality cannot be improved.

Intuitively, this result says that the noise of a gradient estimator is lower when the scale parameter C is small and the mean parameter m is close to a stationary point.

The first estimator we consider is ∇ l ( w ) only. It is given by taking ϕ = -log p into Eq. 4, i.e.

The next result gives a bound on the noise of this estimator, which is a direct consequence of Theorem 1. The second line uses Young's inequality to bound the noise in terms of (i) the distance of w from w ∗ and (ii) a constant determined by the distance of w ∗ from some fixed parameters ¯ w .

Theorem 2. Suppose that log p ( · , x ) is M -smooth and has a maximum (or stationary point) at ¯ m , and define ¯ w = ( ¯ m, 0) . Then, for every w and every solution w ∗ of the VI problem,

Second, we consider an estimator of the gradient of the full objective l + h. It is obtained by simply taking the above estimator and adding the true (known) gradient of the neg-entropy, i.e.

The noise of g ent can be bounded since it only differs from g energy by the deterministic quantity ∇ h ( w ) , and the fact that-provided w ∈ W L -the singular values of w cannot be too small and so ∇ h ( w ) cannot be too large. This is formalized in the following theorem, where again, the second line relaxes the result into a term based on the distance of w from w ∗ plus constants. (Another type of noise bound [29] for g ent was obtained by Kim et al. [21] for diagonal Gaussians and a variety of parameterizations of the covariance.)

Theorem 3. Suppose that log p ( · , x ) is M -smooth, that it is maximal at ¯ m , and define ¯ w = ( ¯ m, 0) . Then, for every L > 0 , for every w ∈ W L and every solution w ∗ of the VI problem,

While g ent used the exact gradient of h , it may be beneficial to use a stochastic estimator h instead. To derive our third estimator, write the gradient of the ELBO as 1

where the parameters v serve as a way to 'hold w constant under differentiation'. This leads to our third estimator, called the 'sticking the landing' (STL) gradient estimator

Intuitively, we expect that g STL will tend to have lower noise than g ent when the posterior is wellapproximated by the variational distribution. The reason is that, as observed by Roeder et al. [30], if q w ( z ) were a perfect approximation of p ( z | x ) , then the two terms in Eq. 9 would exactly cancel (for every u ) and so the estimator would have zero variance. Below we formalize this intuition in what we believe is the first noise bound on g STL .

Theorem 4. Suppose that log p ( · , x ) is M -smooth. Consider the residual r ( z ) := log p ( z, x ) -log q w ∗ ( z ) for any solution w ∗ of the VI problem, assume that it has a stationary point ˆ m , and define ˆ w = ( ˆ m, 0) . Then r is K -smooth for some K ∈ [0 , 2 M ] , and for all w ∈ W M ,

Moreover, if p ( ·| x ) is Gaussian then K = 0 .

When the target is Gaussian then K = 0 in Eq. 10, meaning that when w = w ∗ , the STL estimator has no variance. This is a distinguishing feature of g STL , as opposed to g energy or g ent .

2.3 Challenges for optimization

Three major issues bar applying existing analysis of stochastic optimization methods to our VI setting: 1) non-smooth composite objective 2) lack of uniform smoothness and 3) lack of uniformly bounded noise of the gradient estimator.

The first issue is due to the non-smoothness of neg-entropy function h . This means that under the benign assumption that the target log p is smooth, the full objective l + h cannot be smooth, since a nonsmooth function plus a smooth function is always nonsmooth. This renders stochastic optimization proofs (e.g. those for the "ABC" conditions [29]) that do not use projections or proximal operators inapplicable.

One way to tackle this first issue would be to use a non-smooth proof technique, but these rely on having a uniform gradient noise bound (our third issue). Alternatively, one can overcome the non-smoothness of the neg-entropy function by either using a proximal operator, or projecting onto a set where h is smooth, namely W M . This is the strategy we pursue.

Description

Definition

Stationary point of l

¯ w = (¯ z, 0) for any ¯ z that is a stationary point of log p ( · , x )

Condition on -log p ( z, x ) Consequence

none convex

µ -strongly convex

M -smooth h ( w ) is convex when C is symmetric or triangular

h ( w ) is M -smooth over W M

l ( w ) is convex l ( w ) is µ -strongly convex

l ( w ) is M -smooth

w

∗

M

g energy , g ent , and g STL are quadratically bounded

The second issue is that existing analyses for proximal/projected stochastic methods either rely on a uniform noise bound ([31, Cor. 3.6]) or uniform smoothness [32, 33, 34, 35, 36], both of which are not known to be true for VI. By uniform smoothness, we refer to the assumption that log p ( T w ( u ) , x ) is M -smooth for every u , with M being independent of u . Instead, we can only guarantee that log p ( T w ( u ) , x ) is smooth in expectation, i.e. that l ( w ) is smooth. Several works [9, Cond. 1][37, Thm. 1][38, Sec. 4][39, Assumption A1][40, Thm. 1][41, Assumption 3.2] assumed that the full objective l + h is smooth, but we are not aware of cases where this holds in practice for VI and in our parameterization (Eq. 2) this cannot be true if log p is smooth.

The third issue is the lack of a uniform noise bound. Most non-smooth convergence guarantees within stochastic optimization [42] assume that the noise of the gradient estimator is uniformly bounded by a constant. But this does not appear to be true even under favorable assumptions-e.g. it is untrue if the target distribution is a standard Gaussian. The best that one can hope is that the gradient noise can be bounded by a quadratic that depends on the current parameters w , and-depending on the estimator-even this may only be true when the parameters are in the set W M . Thus, our main optimization contribution is to provide theoretical guarantees for stochastic algorithms under such a specific noise regime, which we make precise in the next definition.

Definition 5. Let ϕ be a differentiable function. We say that a random vector g is a quadratically bounded estimator for ∇ ϕ at w with parameters ( a, b, w ∗ ) , if it is unbiased E [ g ] = ∇ ϕ ( w ) and if the expected squared norm is bounded by a quadratic function of the distance of parameters w to w ∗ , i.e. E ∥ g ∥ 2 ≤ a ∥ w -w ∗ ∥ 2 + b.

The noise bounds derived in Section 2.2 for the estimators g energy , g ent , and g STL imply that they are uniformly quadratically bounded estimators . See Appendix 8.2 for a table with the corresponding constants a and b .

3 Stochastic Optimization with quadratically bounded estimators

In this section we give new convergence guarantees for the Prox-SGD algorithm and the Proj-SGD algorithm with quadratically bounded gradident estimators. Because these may be of independent

interest, they are presented generically, without any reference to the VI setting. We specialize these results to VI in Section 4. For both algorithms, we present results assuming the problem to be strongly convex or just convex. All proofs for this section can be found in the Appendix (Sec. 9).

3.1 Stochastic Proximal Gradient Descent

Here we want to minimize a function which is the sum of two terms l + h , were both l and h are proper closed convex functions, and l is smooth. For this we will use stochastic proximal gradient.

Definition 6 (Prox-SGD) . Let w 0 be a fixed initial parameters and let γ 1 , γ 2 , · · · be a sequence of step sizes. The stochastic proximal gradient (Prox-SGD) method is given by

where g t is a gradient estimator for ∇ l ( w t ) , and the proximal operator is defined as

Theorem 7. Let l be a µ -strongly convex and M -smooth function, and let ¯ w = argmin( l ) . Let h be a proper closed convex function, and let w ∗ = argmin( l + h ) . Let ( w t ) t ∈ N be generated by the Prox-SGD algorithm, with a constant stepsize γ ∈ ( 0 , min { µ 2 a , 1 µ } ] . Suppose that g t is a quadratically bounded estimator (Def. 5) for ∇ l with parameters ( a, b, w ∗ ) . Then,

Alternatively, if we use the decaying stepsize γ t = min { µ 2 a , 1 µ 2 t +1 ( t +1) 2 } , then

In both cases, T = O ( ϵ -1 ) iterations are sufficient to guarantee that E ∥ w T -w ∗ ∥ 2 ≤ ϵ.

The above theorem gives an anytime 1 /T rate of convergence when the stepsizes are chosen based on the strong convexity and gradient noise constants µ and a . Note that we do not need to know precisely those constants: We show in Appendix 9.3 that using any constant step size proportional to T/ log T leads to a log T/T rate.

Theorem 8. Let l be a proper convex and M -smooth function. Let h be a proper closed convex function, and let w ∗ ∈ argmin( l + h ) . Let ( w t ) t ∈ N be generated by the Prox-SGD algorithm, with a constant stepsize γ ∈ (0 , 1 M ] . Suppose that g t is a quadratically bounded estimator (Def. 5) for ∇ l with parameters ( a, b, w ∗ ) . Then,

Thus, T = O ( ϵ -2 ) iterations are sufficient to guarantee that E [ f ( ¯ w T ) -inf f ] ≤ ϵ.

3.2 Stochastic Projected Gradient Descent

Here we want to minimize a function f over a set of constraints W , where W is a nonempty closed convex set and f is a proper closed convex function which is differentiable on W . To solve this problem, we will consider the stochastic projected gradient algorithm.

Definition 9 (Proj-SGD) . Let w 0 be some fixed initial parameter, and let γ 1 , γ 2 , · · · be a sequence of step sizes. The projected stochastic gradient (Proj-SGD) method is given by

where g t is a gradient estimator for ∇ f ( w t ) , and the projection operator is defined as

Note that we do not require f to be smooth , meaning that this setting is not a particular case of the one considered in Section 3.1. In fact, the arguments we use in the proofs for Proj-SGD are more closely related to stochastic sub gradient methods than stochastic gradient methods.

Theorem 10. Let W be a nonempty closed convex set. Let f be a µ -strongly convex function, differentiable on W . Let w ∗ = argmin W ( f ) . Let ( w t ) t ∈ N be generated by the Proj-SGD algorithm, with a constant stepsize γ ∈ ( 0 , min { µ 2 a , 2 µ } ] . Suppose that g t is a quadratically bounded estimator (Def. 5) for ∇ f with parameters ( a, b, w ∗ ) . Then,

Alternatively, if we use the decaying stepsize γ t = min { µ 2 a , 2 µ 2 t +1 ( t +1) 2 } , then

In both cases, T = O ( ϵ -1 ) iterations are sufficient to guarantee that E ∥ w T -w ∗ ∥ 2 ≤ ϵ.

Note that Eq. 13 is a sum of two terms: one that decays exponentially in T and one that decreases only when the stepsize γ is sufficiently small. This has an important consequence: if one uses the gradient estimator g STL and the target distribution is exactly a Gaussian, then b = 0 (Appendix 8.2), meaning that the algorithm will converge at an exponential rate. This is similar to many results in the stochastic optimization literature showing faster rates hold when interpolation holds [35].

Theorem 11. Let W be a nonempty closed convex set. Let f be a convex function, differentiable on W . Let w ∗ ∈ argmin W ( f ) . Let ( w t ) t ∈ N be generated by the Proj-SGD algorithm, with a constant stepsize γ ∈ (0 , + ∞ ) . Suppose that g t is a quadratically bounded estimator (Def. 5) for ∇ f at w t with constant parameters ( a, b, w ∗ ) . Then,

where θ def = 1 1 + aγ 2 and ¯ w T def = ∑ T -1 t =0 θ t +1 w t ∑ T -1 θ t +1 . Finally if γ = √ 2 √ aT and T ≥ 2 then

Thus, T = O ( ϵ -2 ) iterations are sufficient to guarantee that E [ f ( ¯ w T ) -inf f ] ≤ ϵ.

4 Solving VI with provable guarantees

We now specialize the optimization results of the previous section to our VI setting. We aim to minimize the negative ELBO, f : R d × V -→ R ∪ { + ∞} , which decomposes as f = l + h where

and where V is the vector space of matrices in which the covariance factors C belong. Under the assumption that the log-target log p ( · , x ) is M -smooth and concave, we propose two strategies to minimize this objective. All the proofs for this section can be found in Appendix 10.

Apply the Prox-SGD algorithm to the sum l + h , using a proximal step with respect to h , and a stochastic gradient step with respect to l . The gradient of l will be estimated with g energy (5), which is globally quadratically bounded. The prox of h admits a closed form formula if we choose that V is the space of lower triangular matrices.

Apply the Proj-SGD algorithm to minimize f over W M . The gradient of f will be estimated using either g ent (7) or g STL (9), both of which are quadratically bounded on W M . The projection onto W M admits a closed form formula if choose that V is the space of symmetric matrices.

Corollary 12 (Prox-SGD for VI) . Consider the VI problem where q w is a multivariate Gaussian distribution (Eq. 2) with parameters w = ( m,C ) ∈ R d × T d , and assume that this problem admits a solution w ∗ . Suppose that log p ( · , x ) is M -smooth and concave (resp. µ -strongly concave). Generate a sequence w t by using the Prox-SGD algorithm (Def. 6) applied to l and h (Eq. 16), using g energy (5) as an estimator of ∇ l . Let the stepsizes γ t be constant and equal to 1 / ( √ a energy T ) (resp. be decaying as in Theorem 7 with a energy = 2( d +3) M 2 ). Then, for a certain average ¯ w T of the iterates, we have for T ≥ 2 that

Corollary 13 (Proj-SGD for VI) . Consider the VI problem where q w is a multivariate Gaussian distribution (Eq. 2) with parameters w = ( m,C ) ∈ R d ×S d , and assume that this problem admits a solution w ∗ . Suppose that log p ( · , x ) is M -smooth and concave (resp. µ -strongly concave). Generate a sequence w t by using the Proj-SGD algorithm (Def. 9) applied to the function f = l + h (Eq. 16) and the constraint W M (Eq. 3), using g ent (7) or g STL (9) as an estimator of ∇ f . Let the stepsizes γ t be constant and equal to √ 2 / ( aT ) (resp. be decaying as in Theorem 10) with a = a ent = 4( d +3) M 2 or a = a STL = 24( d +3) M 2 . Then, for a certain average ¯ w T of the iterates, we have for T ≥ 2 that

Corollary 14 (Proj-SGD for VI - Gaussian target) . Consider the setting of Corollary 13, in the scenario that log p ( · , x ) is µ -strongly concave, that we use the g STL estimator, and that we take a constant stepsize γ t ≡ γ ∈ ( 0 , min { µ 2 a STL , 2 µ } ] . Assume further that p ( ·| x ) is Gaussian. Then,

Let us now discuss the practical implementation of these two algorithms (more details and an explicit implementation of the methods are given in Appendix 10.2). These require computing either the proximal operator of the neg-entropy h , or the projection onto the set of non-degenerate covariance factors W M . These can be computed [13] for every w = ( m,C ) as:

prox γh ( w ) = ( m,C +∆ C ) , where ∆ C is diagonal with ∆ C ii = 1 2 ( √ C 2 ii +4 γ -C ii ) ,

proj W ( w ) = ( m,U ˜ DU ⊤ ) , where C has the SVD C = UDU ⊤ and ˜ D ii = max { D ii , 1 / √ M } .

Our theory for the Prox-SGD algorithm formally assumes that the covariance factor C lives in the vector space of lower-triangular matrices. This can be implemented by letting C ∈ R d × d and "clamping" the upper-triangular entries to zero throughout computation. Then the gradient g energy (5) can be computed using automatic differentiation, and the proximal operator can be computed as in the above equation. Or, one may choose w = ( m,c ) where c ∈ R d × ( d +1) / 2 is the lower-triangular entries of the matrix so C = tril( c ) . Gradients with respect to w = ( m,c ) can again be estimated using automatic differentiation (including the tril operator), and the proximal operator can be computed by forming C , using the above formula, and then extracting the lower-triangular entries.

Similarly, our theory for the Proj-SGD algorithm formally assumes the covariance factor C lives in the vector space of symmetric matrices. This can be implemented by letting C ∈ R d × d , computing the gradient estimator g ent (7) or g STL (9) over the reals using automatic differentiation, and "symmetrizing" C throughout computation. That is, set C ← 1 2 C + 1 2 C ⊤ after applying each gradient update, right before projection. Or, one may choose w = ( m,c ) where c ∈ R d × ( d +1) / 2 is the lower-triangular entries of the matrix, so C = symm( c ) , where symm is a function similar to tril except creating symmetric matrices. The gradient estimator g ent (7) or g STL (9) can be computed

using automatic differentiation (including the symm operator). In this case, to project one would first form C , then use the above formula, and then extract the lower-triangular matrices.

In terms of cost, computing log q w ( z ) or T w ( u ) needs Θ( d 2 ) operations. Computing the prox of h requires d operations, but in contrast projecting onto W M requires diagonalizing a symmetric matrix, which takes Θ( d 3 ) operations. We note that this is not necessarily an issue, because 1) eigenvalue decomposition has excellent computational constants in moderate dimensions; 2) computing the target log p ( z, x ) may itself require Θ( d 3 ) or more operations; 3) it is common to average a 'minibatch' of gradient estimates, meaning each eigenvalue decomposition computation is amortized over many ELBO evaluations. Still, in high dimensions, when log p ( z, x ) is inexpensive, and small minibatches are used, computing such decomposition in each iteration could become a computational bottleneck. In this scenario, the Proj-SGD algorithm is clearly cheaper, with its Θ( d ) cost for the proximal step.

5 Discussion

While this paper has focused on convergence with dense Gaussian variational distributions, the results in Sec. 3 apply more generally. It is conceivable that the necessary smoothness, convexity, and noise bounds could be established for other variational families, in which case these optimization results would provide "plug in" guarantees. We suspect this might be fairly easy to do with, e.g., elliptical distributions or location-scale families, using similar techniques as done with Gaussians. But it may be difficult to do so for broader variational families.

If -log p is strongly convex and smooth, another proof strategy is possible: Smoothness guarantees that w ∗ ∈ W M and strong convexity guarantees that w ∗ ∈ { w : ∥ w -¯ w ∥ 2 2 ≤ d/µ } . So one could do projected SGD, projecting onto the intersection of these two sets. The negative ELBO would be strongly convex and smooth over that set, and since ∥ w -¯ w ∥ 2 is bounded, gradient noise is upperbounded by a constant, meaning classical projected SGD convergence rates that assume smoothness, strong convexity, and uniform gradient noise apply. However, projecting onto the intersection of those sets poses a difficulty and this uniform noise bound may be loose in practice.

Several variants of the gradient estimators we consider have been proposed to reduce noise, often based on control variates or sampling from a different base distribution [30, 37, 43, 44, 45, 46, 47, 48, 49, 50]. It would be interesting to establish gradient noise bounds for these estimators.

Various VI variants have been proposed that use natural gradient descent. It would also be interesting to seek convergence guarantees for these algorithms.

VI can be done with 'score' or 'reinforce' gradient estimators [6, 8], which use the identity

in place of how we used Eq. 4 for the 'path' estimators we considered. While path estimators often seem to have lower variance, this is not always true: Take an unnormalized log-posterior ϕ ( z ) where z is scalar. Now, define ϕ ′ ( z ) = ϕ ( z ) + √ ϵ sin( z/ϵ ) , where ϵ is very small. Then, ϕ and ϕ ′ represent almost the same posterior, and score estimators for them will have almost the same variance. Yet, the derivative of the added term is cos( z/ϵ ) / √ ϵ , so a path estimator for ϕ ′ will have much higher variance than one for ϕ . This underlines why smoothness is essential for our guarantees-it excludes posteriors like ϕ ′ . Future work further unravel when score estimators perform better than path estimators.

One attractive feature of VI is that data subsampling can often be used when computing gradients, decreasing the cost of each iteration. While we have not explicitly addressed this, our proofs can be easily generalized, at least for target distributions of the form p ( z, x ) = p ( z ) ∏ n =1 p ( x n | z ) . The only issue is to bound the noise of the subsampled estimator. While our gradient variance guarantees use Theorem 1 from [20], a more general result [20, Theorem 6] considers data subsampling. Very roughly speaking, with uniform data subsampling of 1 datum of a time, the gradient noise bounds in Thms. 2 and 3 would increase by a factor between 1 (no increase) and the number of data, depending on how correlated the data are-less correlation leads to a larger increase. (More precisely, the second term in Thm. 3 would not increase.) These increases would manifest as larger constants a and b in the quadratic bounds, after which exactly the same results hold. However, it is not obvious if the bound for g STL in Thm. 4 can be generalized in this way.

References

Michael I. Jordan and Zoubin Ghahramani. An introduction to variational methods for graphical models. Machine Learning , 37(2):183-233, 1999.

Zoubin Ghahramani and Matthew Beal. Propagation Algorithms for Variational Bayesian Learning. In NeurIPS , 2001.

John Winn and Christopher M Bishop. Variational Message Passing. Journal of Machine Learning Research , 6:661-694, 2005.

David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe. Variational Inference: A Review for Statisticians. Journal of the American Statistical Association , 112(518):859-877, 2017.

Tim Salimans and David A. Knowles. Fixed-Form Variational Posterior Approximation through Stochastic Linear Regression. Bayesian Anal. , 8(4):837-882, 2013.

David Wingate and Theophane Weber. Automated Variational Inference in Probabilistic Programming. arXiv:1301.1299 [cs, stat] , 2013.

Matthew D. Hoffman, David M. Blei, Chong Wang, and John Paisley. Stochastic Variational Inference. Journal of Machine Learning Research , 14:1303-1347, 2013.

Rajesh Ranganath, Sean Gerrish, and David M. Blei. Black Box Variational Inference. In AISTATS , 2014.

Jeffrey Regier, Michael I Jordan, and Jon McAuliffe. Fast Black-box Variational Inference through Stochastic Trust-Region Optimization. In NeurIPS , page 10, 2017.

Alp Kucukelbir, Dustin Tran, Rajesh Ranganath, Andrew Gelman, and David M. Blei. Automatic Differentiation Variational Inference. Journal of Machine Learning Research , 18(14): 1-45, 2017.

C. Zhang, J. Bütepage, H. Kjellström, and S. Mandt. Advances in Variational Inference. IEEE Transactions on Pattern Analysis and Machine Intelligence , 41(8):2008-2026, 2019.

Yuling Yao, Aki Vehtari, Daniel Simpson, and Andrew Gelman. Yes, but Did It Work?: Evaluating Variational Inference. In ICML , 2018.

Justin Domke. Provable Smoothness Guarantees for Black-Box Variational Inference. In ICML , 2020.

Akash Kumar Dhaka, Alejandro Catalina, Manushi Welandawe, Michael Riis Andersen, Jonathan Huggins, and Aki Vehtari. Challenges and opportunities in high-dimensional variational inference. In NeurIPS , 2021.

Manushi Welandawe, Michael Riis Andersen, Aki Vehtari, and Jonathan H. Huggins. Robust, automated, and accurate black-box variational inference, 2022.

Edward Challis and David Barber. Gaussian Kullback-Leibler Approximate Inference. Journal of Machine Learning Research , 14:2239-2286, 2013.

Michalis Titsias and Miguel Lázaro-Gredilla. Doubly Stochastic Variational Bayes for nonConjugate Inference. In ICML , 2014.

Kai Fan, Ziteng Wang, Jeff Beck, James Kwok, and Katherine Heller. Fast Second-Order Stochastic Backpropagation for Variational Inference. In NeurIPS , 2015.

Ming Xu, Matias Quiroz, Robert Kohn, and Scott A. Sisson. Variance reduction properties of the reparameterization trick. In AISTATS , 2019.

Justin Domke. Provable Gradient Variance Guarantees for Black-Box Variational Inference. In NeurIPS , 2019.

Kyurae Kim, Kaiwen Wu, Jisu Oh, and Jacob R. Gardner. Practical and matching gradient variance bounds for black-box variational inference. In ICML , 2023.

Zuheng Xu and Trevor Campbell. The computational asymptotics of gaussian variational inference and the laplace approximation. Stat Comput , (32), 2023.

Marc Lambert, Sinho Chewi, Francis Bach, Silvère Bonnabel, and Philippe Rigollet. Variational inference via wasserstein gradient flows. In Advances in Neural Information Processing Systems , volume 35, pages 14434-14447. Curran Associates, Inc., 2022.

Michael Ziyang Diao, Krishna Balasubramanian, Sinho Chewi, and Adil Salim. Forwardbackward Gaussian variational inference via JKO in the bures-Wasserstein space. In Proceedings of the 40th International Conference on Machine Learning , volume 202, pages 7960-7991. PMLR, 23-29 Jul 2023.

Gabriel Peyre and Marco Cuturi. Computational optimal transport. Foundations and Trends in Machine Learning , 11(5-6):355-607, 2019.

Kyurae Kim, Jisu Oh, Kaiwen Wu, Yi-An Ma, and Jacob R. Gardner. On the convergence and scale parameterizations of black-box variational inference. In NeurIPS , 2023.

Gregory F. Cooper. The computational complexity of probabilistic inference using bayesian belief networks. Artificial Intelligence , 42(2):393-405, 1990.

Paul Dagum and Michael Luby. Approximating probabilistic inference in bayesian belief networks is np-hard. Artificial Intelligence , 60(1):141-153, 1993. ISSN 0004-3702.

Ahmed Khaled and Peter Richtárik. Better theory for sgd in the nonconvex world. Transactions of Machine Learning Research , 2023.

Geoffrey Roeder, Yuhuai Wu, and David K Duvenaud. Sticking the Landing: Simple, LowerVariance Gradient Estimators for Variational Inference. In NeurIPS , 2017.

Damek Davis and Dmitriy Drusvyatskiy. Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization , 29(1):207-239, 2019. ISSN 1052-6234.

Eric Moulines and Francis Bach. Non-asymptotic analysis of stochastic approximation algorithms for machine learning. In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems , volume 24. Curran Associates, Inc., 2011.

Deanna Needell, Nathan Srebro, and Rachel Ward. Stochastic gradient descent, weighted sampling, and the randomized Kaczmarz algorithm. Mathematical Programming , 155(1): 549-573, 2015. ISSN 1436-4646.

Ahmed Khaled, Othmane Sebbouh, Nicolas Loizou, Robert M. Gower, and Peter Richtárik. Unified analysis of stochastic gradient methods for composite convex and smooth optimization, 2020.

Guillaume Garrigos and Robert M. Gower. Handbook of convergence theorems for (stochastic) gradient methods, 2023.

Eduard Gorbunov, Filip Hanzely, and Peter Richtarik. A unified theory of sgd: Variance reduction, sampling, quantization and coordinate descent. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics , volume 108 of Proceedings of Machine Learning Research , pages 680-690. PMLR, 26-28 Aug 2020.

Alexander Buchholz, Florian Wenzel, and Stephan Mandt. Quasi-Monte Carlo Variational Inference. In ICML , 2018.

Mohammad E. Khan, Pierre Baqué, François Fleuret, and Pascal Fua. Kullback-Leibler proximal variational inference. In NeurIPS , pages 3402-3410, 2015.

Mohammad Emtiyaz Khan, Reza Babanezhad, Wu Lin, Mark Schmidt, and Masashi Sugiyama. Faster Stochastic Variational Inference using Proximal-Gradient Methods with General Divergence Functions. In UAI , 2016.

K. Fan, Y. Zhang, R. Henao, and K. Heller. Triply Stochastic Variational Inference for Nonlinear Beta Process Factor Analysis. In 2016 IEEE 16th International Conference on Data Mining (ICDM) , pages 121-130, 2016.

Pierre Alquier and James Ridgway. Concentration of tempered posteriors and of their variational approximations. arXiv:1706.09293 [cs, math, stat] , 2017.

Guanghui. Lan. First-order and Stochastic Optimization Methods for Machine Learning . Springer Series in the Data Sciences. Springer International Publishing, Cham, 1st ed. 2020. edition, 2020. ISBN 3-030-39568-5.

Michalis K. Titsias and Miguel Lázaro-Gredilla. Local Expectation Gradients for Black Box Variational Inference. In NeurIPS , 2015.

Francisco J. R. Ruiz, Michalis K. Titsias, and David M. Blei. The Generalized Reparameterization Gradient. In NeurIPS , 2016.

Francisco J. R. Ruiz, Michalis K. Titsias, and David M. Blei. Overdispersed Black-Box Variational Inference. arXiv:1603.01140 [stat] , 2016.

Andrew C. Miller, Nicholas J. Foti, Alexander D'Amour, and Ryan P. Adams. Reducing Reparameterization Gradient Variance. arXiv:1705.07880 [stat] , 2017.

Linda S. L. Tan and David J. Nott. Gaussian variational approximation with sparse precision matrices. Stat Comput , 28(2):259-275, 2018.

Tomas Geffner and Justin Domke. Using Large Ensembles of Control Variates for Variational Inference. In NeurIPS , 2018.

Tomas Geffner and Justin Domke. Approximation based variance reduction for reparameterization gradients. In NeurIPS , 2020.

Masahiro Fujisawa and Issei Sato. Multilevel monte carlo variational inference. Journal of Machine Learning Research , 2021.

T. M. Cover and Joy A. Thomas. Elements of Information Theory . Wiley-Interscience, Hoboken, N.J, 2nd ed edition, 2006.

Amir Beck. First-Order Methods in Optimization . SIAM-Society for Industrial and Applied Mathematics, Philadelphia, PA, USA, 2017. ISBN 1611974984.

Robert Mansel Gower, Nicolas Loizou, Xun Qian, Alibek Sailanbayev, Egor Shulgin, and Peter Richtárik. SGD: General analysis and improved rates. In Proceedings of the 36th International Conference on Machine Learning, spotlight paper , volume 97, pages 5200-5209. PMLR, 09-15 Jun 2019.

Sebastian U. Stich. Unified optimal analysis of the (stochastic) gradient method. arXiv:1907.04232 , 2019.

Amir Beck and Marc Teboulle. Gradient-based algorithms with applications to signal recovery. Convex optimization in signal processing and communications , pages 42-88, 2009.

Contents

6 Notations

We note M d the space of d × d matrices, S d the subspace of symmetric matrices, and T d the subspace of lower triangular matrices. We use W M := { ( m,C ) ∈ W : σ min ( C ) ≥ 1 √ M } to denote the set of parameters with positive definite covariance matrix whose singular values are greater or equal to 1 √ M .

7 VI problems and their structural properties

This section contains the proofs of all the claims made in Section 2.

7.1 Modeling the VI problem

We recall and detail here the setup of our problem. Given a target distribution p and observed data x , we want to solve the following Variational Inference problem

where q w is a multivariate Gaussian variational family, whose density may be written as

We make the assumption (VI) has a non-degenerated solution, i.e. a solution for which the covariance matrix is invertible.

We will impose that our parameters have the form w = ( m,C ) ∈ R d × V , where V is some vector subspace of M d , the space of d × d matrices. On top of that, we will also want the covariance factor to be positive definite. In other words, we will impose that our parameters are taken from the set

For the sake of simplicity, the reader can assume for most of the paper that V = M d . But when coming to analyze our algorithms, we will see that we should take either V = T d (the subspace of d × d lower triangular matrices) or V = S d (the subspace of d × d symmetric matrices). This choice is dictated by two reasons: it guarantees that our problem remains convex (see Lemma 19), and it lowers the computational cost of our algorithms (see Section 10.2). Fortunately, requiring C to be symmetric (or triangular) and positive definite does not change our problem, as stated in the next Lemma.

Lemma 15 (Equivalent problems for triangular/symmetric covariance factors) . Suppose that ¯ w ∈ R d ×M d is such that q ¯ w minimizes KL ( q w ∥ p ( ·| x )) over R d ×M d . Then there exists w ∗ = ( m ∗ , C ∗ ) such that q ¯ w and q w ∗ have the same distribution, with C ∗ being positive definite and symmetric (or lower triangular).

Proof. Write ¯ w = ( ¯ m, ¯ C ) , and take m ∗ = ¯ m . Let ¯ Σ = ¯ C ¯ C ⊤ . Since we assumed that (VI) has a non-degenerated solution, we can assume without loss of generality that ¯ Σ is invertible. Therefore we can apply the Cholesky decomposition and write ¯ Σ = C ∗ ( C ∗ ) ⊤ , where C ∗ is lower triangular and positive definite. We can also define C ∗ = ( ¯ Σ) 1 / 2 to obtain a symmetric and positive definite factor.

Once restricted to W , our problem (VI) becomes equivalent to

where

l ( w ) = -E z ∼ q w log p ( z , x ) is the free energy ;

This equivalence is mostly standard, but we state it formally for the sake of completeness.

Lemma 16 (VI reformulated) . Problems (VI) and (18) are equivalent.

Proof. Let w = ( m,C ) ∈ W , and let us show that KL ( q w ∥ p ( ·| x )) = l ( w ) + h ( w ) . By definition of the Kullback-Liebler divergence, and using p ( z | x ) = p ( z, x ) /p ( x ) , we can write

If we note H ( X ) the entropy of a random variable, we can see that E z ∼ q w log q w ( z ) = -H ( z ) where z ∼ q w . We can rewrite z via an affine transform as z = C u + m with u ∼ N (0 , I ) , so we can use [51, Section 8.6] to write (see also the arguments in [13])

Here H ( u ) , the entropy of u , is a constant equal to ( d/ 2)(1 + log(2 π ) . Moreover since w ∈ W , we know that C ≻ 0 , which implies that log | det C | = log det C . Because log p ( x ) and H ( u ) are both constants, we deduce that minimizing KL ( q w ∥ p ( ·| x )) over w ∈ W is equivalent to minimizing l + h , where h ( w ) = -log det C if w ∈ W and + ∞ otherwise.

We emphasize that working with a triangular or symmetric representation for C is not restrictive, and that doing computations in this setting is pretty straightforward, as we illustrate next.

Lemma 17 (Computations with triangular/symmetric covariance factors) . Let V be a vector subspace of M d . Let ϕ : R d ×M d → R and let ϕ V : R d × V → R be the restriction of ϕ to R d × V . Then:

If ϕ is convex then ϕ V is convex.

If ϕ is differentiable then ϕ V is differentiable, with ∇ ϕ V ( w ) = proj R d × V ( ∇ ϕ ( w )) . In particular, ∥∇ ϕ V ( w ) ∥ ≤ ∥∇ ϕ ( w ) ∥ .

If ϕ is M -smooth over Ω ⊂ R d ×M d , then ϕ V is M -smooth over Ω ∩ ( R d × V ) .

proj S d ( C ) = ( C + C ⊤ ) / 2 , and proj T d ( C ) = tril( C ) , where tril( C ) sets to zero all the entries of C above the diagonal.

Proof.

This is immediate due to the fact that V is convex, since it is a vector subspace of M d .

Let ι : R d × V → R d ×M d , ι ( m,C ) = ( m,C ) be the canonical injection. Then we clearly have ϕ V = ϕ ◦ ι . Moreover, ι is a linear application whose adjoint is ι ∗ is exactly the orthogonal projection proj R d × V . So the computation of the gradient follows after applying the chain rule : ∇ ϕ V ( w ) = ι ∗ ( ∇ ϕ ( ι ( w )) = proj R d × V ( ∇ ϕ ( w )) . As for the norm, it suffices to observe that the orthogonal projection is a linear operator with norm less or equal than 1 .

Using that the orthogonal projection is a contractive map gives

This is a standard linear algebra result.

7.2 Smoothness and convexity for VI problems

Now we state the main smoothness and convex properties for VI problems. We will see in our analysis that the smoothness properties often revolve around the following subset of W

where σ min ( C ) refers to the smallest singular value of C .

Lemma 18 (Smoothness for VI) . Assume that log p ( · , x ) is M -smooth. Then

l is M -smooth.

h is differentiable over W , with ∇ h ( w ) = (0 , -proj V ( C -⊤ )) for all w ∈ W .

h is M -smooth over W M = { ( m,C ) ∈ W : σ min ( C ) ≥ 1 √ M } .

Proof.

Combine [13, Theorem 1] with Lemma 17.3.

Combine the fact that the gradient of -log det at a matrix X is -X -⊤ with Lemma 17.2.

Combine [13, Lemma 12] with Lemma 17.3.

Lemma 19 (Convexity for VI) . Assume that -log p ( · , x ) is convex (resp. µ -strongly convex). Then

l is convex (resp. µ -strongly convex).

If V = S d then h is closed convex, W is convex, and W M is convex and closed. Moreover,

If V = T d then h is closed convex and W is convex. Moreover

If V = M d then h can be not convex.

Proof.

Combine Titsias and Lázaro-Gredilla [17, Proposition 1] (resp. Domke [13, Theorem 9]) with Lemma 17.1.

Convexity of h follows from [52, Example 7.13 and Theorem 7.17]. Convexity of W comes from the fact that it is essentially the set of symmetric positive definite matrices (which is equivalent to λ min ( C ) > 0 ). Let us now prove that W M is convex and closed. First we notice that, because of symmetry, we can rewrite W M as

This set is closed because λ min is a continuous function. To see that W M is convex we are going to use the fact that λ min is a concave function. Take C 1 , C 2 ∈ W M , α ∈ [0 , 1] and write

Here W = { ( m,C ) ∈ R d ×T d : C ≻ 0 } . Using Sylvester's criterion, it is easy to see that a lower triangular matrix is positive definite if and only if its diagonal has positive coefficients. In other words, W = { ( m,C ) ∈ R d ×T d : C ii > 0 } , which is clearly convex. It remains to verify that h is closed convex even if W is not closed. For all w = ( m,C ) ∈ R d ×T d , we can use the triangular structure of C to write

where we used the notation δ A for the indicator function of a set A , which is equal to zero on A and + ∞ outside. Since -log( t ) + δ (0 , + ∞ ) ( t ) is a closed convex function, we see that h is a separable sum of closed convex functions, and thus is itself closed and convex.

Let d = 2 , let I be the identity matrix, let R = ( 0 1 -1 0 ) be a rotation matrix, and define ψ : R → R with ψ ( t ) = h ( I + tR ) . Observe that ψ is well defined because I + tR ≻ 0 for all t ∈ R . On the one hand, if h was convex, then ψ would be convex too, as it is a composition of a convex function with the affine function t ↦→ I + tR . On the other hand, we can compute explicitly that ψ ( t ) = -log(1 + t 2 ) , which is not convex.

Lemma 20 (Solutions of VI) . Assume that log p ( · , x ) is M -smooth, and that V = M d , S d or T d . Then argmin( f ) ⊂ W M := { ( m,C ) ∈ W : σ min ( C ) ≥ 1 √ M } .

Proof. If w ∗ = ( m ∗ , C ∗ ) ∈ argmin( f ) , then w ∗ is in the domain of h , which is W by definition of h . So it remains to prove that σ min ( C ∗ ) ≥ 1 / √ M . When V = M d , this is proved in [13, Theorem 7]. Suppose now that V = S d , meaning that w ∗ is a minimizer of f over W ⊂ R d ×S d . We see that w ∗ must also be a minimizer of f over R d ×M d , because if there was a better solution ˆ w = ( ˆ m, ˆ C ) with ˆ C ∈ M d , we could consider ( ˆ C ˆ C ⊤ ) 1 / 2 ∈ S d which would itself be a better solution than C ∗ , which is a contradiction. So we can apply [13, Theorem 7] to w ∗ to conclude that σ min ( C ∗ ) ≥ 1 / √ M . If V = T d , we can use the same argument, using this time the Cholesky decomposition of ˆ C ˆ C ⊤ .

Lemma 21. Let w = ( m,C ) with C invertible. Then log q w is σ min ( C ) -2 -smooth.

Proof. According to the definition of q w (see (17)), the Hessian of log q w is ∇ 2 z (log q w )( z ) = -( CC ⊤ ) -1 , and so ∥∇ 2 z (log q w )( z ) ∥ = σ max (( CC ⊤ ) -1 ) = σ min ( C ) -2 .

7.3 A case study : linear models

from which it follows that

Now, take some arbitrary vector v . We have that

Definition 22. A twice differentiable function f is M -smooth if -MI ⪯ ∇ 2 f ⪯ MI .

Definition 23. A twice differentiable function f is µ -strongly convex ∇ 2 f ⪰ µI .

Theorem 24. Take a generic i.i.d. linear model defined as p ( z, x ) = p ( z ) ∏ N n =1 p ( x n | z, a n ) , where p ( z ) = N ( z | 0 , Σ) and

for some function ϕ. Suppose that the second derivative of ϕ with respect to its first (scalar) argument is bounded by θ min ≤ ϕ ′′ ≤ θ max . Let

where A is a matrix with a n in the n -th column. Then

-log p ( z, x ) is M -smooth over z for M = max( | µ | , | β | ) .

If µ ≥ 0 then -log p ( z, x ) is convex over z

If µ > 0 then -log p ( z, x ) is µ -strongly convex over z .

Proof. It is easy to show that

Putting both the above results together, we have that

This implies that all eigenvalues of -∇ 2 z log p are in the interval [ µ, β ] . All the claimed properties follow from this:

For any eigenvalue λ of ∇ 2 z log p ( z, x ) , we know that ∣ ∣ λ i ( ∇ 2 z log p ( z, x )) ∣ ∣ ≤ M = max( | µ | , | β | ) . Thus, log p ( z, x ) is M -smooth over z .

If µ ≥ 0 , this means all eigenvalues of -∇ 2 z log p ( z, x ) are positive, which implies that -log p ( z, x ) is convex.

2

If µ > 0 , then all eigenvalues of -∇ z log( z, x ) are bounded below by α , which impiles that -log p ( z, x ) is µ -strongly convex.

Corollary 25. Take a linear regression model with inputs a n ∈ R d and outputs x n ∈ R . Let p ( z ) = N ( z | 0 , Σ) and p ( x n | z ) = N ( x | z ⊤ a n , σ 2 ) for some fixed σ . Then -log p ( z, x ) is µ -strongly convex for µ = λ min ( Σ -1 + 1 4 AA ⊤ ) and M -smooth for M = λ max (Σ -1 + 1 4 AA ⊤ ) .

Proof. In this case we have that

Or, more abstractly,

Thus we have that θ min = θ max = 1 σ 2 . It follows from Theorem 24 that -log p ( z, x ) is µ -strongly convex for µ = λ min ( Σ -1 + 1 σ 2 AA ⊤ ) and M -smooth for M = λ max ( Σ -1 + 1 σ 2 AA ⊤ ) .

In the particular case where Σ = I , we get that µ = λ min ( I + 1 σ 2 AA ⊤ ) = 1 + 1 σ 2 σ min ( A ) 2 and M = λ max (1 + 1 σ 2 AA ⊤ ) = 1 + 1 σ 2 σ max ( A ) 2 .

Corollary 26. Take a logistic regression model with inputs a n ∈ R d and outputs x n ∈ {-1 , +1 } . Let p ( z ) = N ( z | 0 , Σ) and p ( x n = 1 | z ) = Sigmoid( x n z ⊤ a n ) = 1 / (1 + exp( -x n z ⊤ a n )) . Then -log p ( z, x ) is µ -strongly convex for µ = λ min ( Σ -1 ) and M -smooth for M = λ max (Σ -1 + 1 4 AA ⊤ ) .

Proof. In this case we have that

Or, more abstractly,

The second derivative is non-negative and goes to zero if α →-∞ or α → + ∞ . It is maximized at α = 0 in which case ϕ ′′ = 1 4 β 2 . But of course, in this model, the second input is β = x n ∈ {-1 , +1 } , meaning β 2 = always. Thus we have that θ min = 0 and θ max = 1 . It follows from Theorem 24 that -log p ( z, x ) is µ -strongly convex for µ = λ min ( Σ -1 ) and M -smooth for M = λ max ( Σ -1 + 1 4 AA ⊤ ) .

In the particular case where Σ = I , we get that µ = 1 and M = λ max (1 + 1 4 AA ⊤ ) = 1 + 1 4 σ max ( A ) 2 .

Theorem 27. Take a heirarchical model of the form p ( θ, z, x ) = p ( θ ) ∏ i p ( z i | θ ) p ( x i | θ, z i ) , where all log-densities are twice-differentiable. Then, log p ( θ, z, x ) is smooth joinly over ( θ, z ) if and only if the spectral norm of the second derivative matrices ∇ θθ ⊤ log p ( θ, z, x ) , ∇ θz ⊤ i log p ( θ, z i , x i ) , and ∇ z i z ⊤ i log p ( θ, z i , x i ) are all bounded uniformly over ( θ, z ) .

Proof. The following conditions are all equivalent:

log p ( θ, z, x ) is smooth as a function of ( θ, z )

∥ ∥ ∇ 2 log p ( θ, z, x ) ∥ ∥ 2 is bounded above (uniformly over ( θ, z ) , where ∇ 2 denotes the Hessian with respect to ( θ, z ) and ∥·∥ 2 denotes the spectral norm.)

∥ ∥ ∇ 2 log p ( θ, z, x ) ∥ ∥ is bounded above (uniformly over ( θ, z ) , where ∇ 2 denotes the Hessian with respect to ( θ, z ) and ∥·∥ denotes a block infinity norm on top of spectral norm inside each block.

∥ ∥ ∇ 2 θθ ⊤ log p ( θ, z, x ) ∥ ∥ 2 , ∥ ∥ ∥ ∇ 2 θz ⊤ i log p ( θ, z, x ) ∥ ∥ ∥ 2 , and ∥ ∥ ∥ ∇ 2 z i z ⊤ j log p ( θ, z, x ) ∥ ∥ ∥ 2 are all bounded above (uniformly over ( θ, z ) )

∥ ∥ ∇ 2 θθ ⊤ log p ( θ, z, x ) ∥ ∥ 2 , ∥ ∥ ∥ ∇ 2 θz ⊤ i log p ( θ, z i , x i ) ∥ ∥ ∥ 2 , and ∥ ∥ ∥ ∇ 2 z i z ⊤ j log p ( θ, z, x ) ∥ ∥ ∥ 2 are all bounded above (uniformly over ( θ, z ) )

∥ ∥ ∇ 2 θθ ⊤ log p ( θ, z, x ) ∥ ∥ 2 , ∥ ∥ ∥ ∇ 2 θz ⊤ i log p ( θ, z i , x i ) ∥ ∥ ∥ 2 , and ∥ ∥ ∥ ∇ 2 z i z ⊤ i log p ( θ, z, x ) ∥ ∥ ∥ 2 are all bounded above (uniformly over ( θ, z ) )

∥ ∥ ∇ 2 θθ ⊤ log p ( θ, z, x ) ∥ ∥ 2 , ∥ ∥ ∥ ∇ 2 θz ⊤ i log p ( θ, z i , x i ) ∥ ∥ ∥ 2 , and ∥ ∥ ∥ ∇ 2 z i z ⊤ i log p ( θ, z i , x i ) ∥ ∥ ∥ 2 are all bounded above (uniformly over ( θ, z ) )

Corollary 28. Take a hierarchical logistic regression model defined by p ( θ, z, x ) = p ( θ ) ∏ i p ( z i | θ ) ∏ j p ( x ij | z i ) where { θ, z i } ∈ R d , x ij ∈ {-1 , +1 } , and

where Sigmoid( b ) = 1 1+exp( -b ) . Then, -log p ( θ, z, x ) is convex and smooth with respect to ( θ, z ) .

Proof. It's immediate that -log p is convex with respect to ( θ, z ) . We can show that this function is smooth, by verifying the three conditions of Theorem 27.

∇ 2 θθ ⊤ log p ( θ, z, x ) = -Σ -1 -∆ -1 is constant.

∇ 2 θz ⊤ i log p ( θ, z i , x i ) = ∆ -1 is also constant.

p ( z i | θ ) is a Gaussian with constant covariance, so is smooth with a constant independent of θ . And we can write that log p ( x ij | z i ) = log ( 1 + exp ( -x ij z ⊤ i a ij )) . This function is also smooth over z i with a constant that doesn't depend on θ . Thus log p ( z i | θ ) + ∑ j log p ( x ij | z i ) is smooth over z i with a smoothness constant that is independent of θ .

8 Estimators and variance bounds

We remember that unless specified otherwise, we consider that w = ( m,C ) ∈ R d × V , where V is a vector subspace of M d .

8.1 Variance bounds for the estimators

Theorem 1. Let T w ( u ) = Cu + m for w = ( m,C ) . Let ϕ : R d → R be M -smooth, suppose that ϕ is stationary at ¯ m , and define ¯ w = ( ¯ m, 0) . Then

Furthermore, the first inequality cannot be improved.

Proof. This bound is proven by [20, Thm. 3] in the case that V = M d . We conclude that this bound holds for every subspace V of M d by using the inequality in Lemma 17.2.

Theorem 2. Suppose that log p ( · , x ) is M -smooth and has a maximum (or stationary point) at ¯ m , and define ¯ w = ( ¯ m, 0) . Then, for every w and every solution w ∗ of the VI problem,

Proof. It is a direct consequence of Theorem 1 and Young's inequality

Theorem 3. Suppose that log p ( · , x ) is M -smooth, that it is maximal at ¯ m , and define ¯ w = ( ¯ m, 0) . Then, for every L > 0 , for every w ∈ W L and every solution w ∗ of the VI problem,

Proof. The difference between g ent and g energy is the addition of the constant vector ∇ h ( w ) . Thus, we have that

It remains to bound the final term. We can do this using the closed-form expression for the entropy gradient (see Lemma 18.2), plus the assumption that w ∈ W L ⊂ W , to see that

We conclude with Young's inequality

For the next Theorem, we will need the follwing technical Lemma :

Lemma 29. Let u ∼ N (0 , I ) , A ∈ M d and b ∈ R d . Then

Proof. Develop the squares to write

Since E u = 0 , we immediately see that E ⟨ A ⊤ b, u ⟩ = 0 . Because of symmetry, the third-order moment E u ∥ u ∥ 2 is also zero, so we obtain E ⟨ A ⊤ b, u ∥ u ∥ 2 ⟩ = 0 . We also have the second-order moment E ∥ u ∥ 2 = d , which means that E ∥ b ∥ 2 ∥ u ∥ 2 = d ∥ b ∥ 2 . For the other terms, we use the identities

from [20, Lemma 9] It allows us to write

The conclusion follows after gathering all those identities.

Theorem 4. Suppose that log p ( · , x ) is M -smooth. Consider the residual r ( z ) := log p ( z, x ) -log q w ∗ ( z ) for any solution w ∗ of the VI problem, assume that it has a stationary point ˆ m , and define ˆ w = ( ˆ m, 0) . Then r is K -smooth for some K ∈ [0 , 2 M ] , and for all w ∈ W M ,

Moreover, if p ( ·| x ) is Gaussian then K = 0 .

Proof. Let w ∈ W M be fixed. For the duration of the proof, we take v to be a second copy of w (held constant under differentiation with respect to w ). Now, we can rearrange the estimator in Eq. 9 by making appear q w ∗ ( T w ( u )) , namely

Introducing ϕ ( z ) := log q ( z ) -log q ∗ ( z ) , and recalling r ( z ) = log p ( z, x ) -log q ∗ ( z ) , we have

We assume that log p ( · , x ) is M -smooth, and that V = M d , T d or S d , which implies that w ∗ ∈ W M (see Lemma 20). This in turn guarantees that log q w ∗ is also M -smooth (see Lemma 21). Thus, r is the sum of two M -smooth functions, so it is K -smooth with K ∈ [0 , 2 M ] . Using Young's inequality, together with Theorem 1 applied to r , we obtain

where ˆ w = ( ˆ m, 0) , with ˆ m being a stationary point of r . Now it remains to control the last term of inequality (19). Apply the chain rule onto ϕ ( T w ( u )) to write (with the help of Lemma 37)

This gives us directly that

where we used that ∥ vu ⊤ ∥ 2 F = ∥ v ∥ 2 ∥ u ∥ 2 for any vectors v and u . Computing ∇ z ϕ amounts to computing the gradient of the Gaussian density q w . From its definition (see Eq. 17) we have:

Recalling the definition of ϕ = log q v -log q w ∗ , and writing v = ( m,C ) , w ∗ = ( m ∗ , C ∗ ) , Σ = CC ⊤ , Σ ∗ = C ∗ C ⊤ ∗ , we obtain

In other words, we have

where A := (Σ -1 ∗ -Σ -1 ) C and b := Σ -1 ∗ m -Σ -1 ∗ m ∗ . We can now combine (20) and (21) and use Lemma 29 to write

Let us now introduce the function κ : R d ×M d → R ∪{ + ∞} , defined by κ ( w ) := E z ∼ q w log q w ( z ) -log q w ∗ ( z ) if C is invertible, + ∞ otherwise. This function is nothing but the Kullback-Liebler divergence between the two Gaussian distributions q w and q w ∗ , and can be computed in closed-form on its domain:

where we note as before w = ( m,C ) , w ∗ = ( m ∗ , C ∗ ) , Σ = CC ⊤ , Σ ∗ = C ∗ C ⊤ ∗ . This allows us to compute its gradient at w ∈ W M :

We see that ∇ m κ ( w ) = b , and ∇ C κ ( w ) = A , which means that ∥ b ∥ 2 + ∥ A ∥ 2 F = ∥∇ w κ ( w ) ∥ 2 . From (22) and the fact that ∇ w κ ( w ∗ ) = 0 (because w ∗ is a minimizer of κ ), we deduce that

Now we remind that w ∗ ∈ W M which implies that log q w ∗ is M -smooth. This means that E z ∼ q w log q w ∗ is M -smooth as well, according to Domke [13, Theorem 1]. On the other hand, we know (see the proof of Lemma 16) that E z ∼ q w log q w ∗ = h ( w ) which is M -smooth on W M (see Lemma 18.3). All this implies that κ is 2 M -smooth on W M , from which we conclude

The Theorem's main inequality (10) follows after plugging the above inequality into (19). The second inequality follows after using Young's inequality

To conclude the proof, it remains to check that K = 0 whenever p ( ·| x ) is Gaussian. In that case, because our main objective function f ( w ) is the divergence between q w and p ( ·| x ) , it is clear that f is minimized whenever q w = p ( ·| x ) . Therefore, without loss of generality, we can assume that p ( ·| x ) = q w ∗ . This implies that the residual r is constant ( r ( z ) = log p ( x ) ), and so is 0 -smooth.

8.2 Quadratically bounded estimators

Theorem 30. The estimators obey E ∥ g ∥ 2 2 ≤ a ∥ w -w ∗ ∥ 2 + b with the the following constants:

Proof. Combine the results of Theorems 2, 3 and 4.

9 Optimization proofs

This section contains all the proofs for the Theorems stated in Section 3.

9.1 Anytime Convergence Theorem for Strongly Convex

Theorem 31 ( Theorem 3.2 in [53] ) . Suppose we are given a sequence of w t iterates such that for a step size γ t ≤ 1 2 L and given constants µ > 0 and σ 2 > 0 we have that

By switching to a decaying stepsize according to

where t ∗ = 4 ⌊L /µ ⌋ , we have that

Proof. We divide the proof for steps t that are great or smaller than t ∗ . For t ≤ t ∗ , we have by unrolling (23) starting from t ∗ , . . . , 1 , with γ t ≡ γ gives

Now consider t ≥ t ∗ for which we have that γ t = 1 µ 2 t +1 ( t +1) 2 which when inserted into (23) gives

Multiplying through by ( t +1) 2 and re-arranging gives

where we used that 2 t +1 t +1 ≤ 2 . Summing up over t = t ∗ , . . . , T and using telescopic cancellation gives

Now using that for t ≤ t ∗ we have that γ = 1 L and (25) holds, thus

where we used that (1 -γµ ) ≤ 1 . Substituting γ = 1 2 L , t ∗ = 4 ⌊L /µ ⌋ and multiplying by ( T +1) 2 gives

which concludes the proof of (24).

9.2 Complexity Lemma for Convex

Lemma 32. Suppose we are given a sequence ¯ w T and constants a, b, B > 0 such that

holds for

If γ = A √ T for A ≥ √ 2 B and T ≥ 2

Thus, T = O ( ϵ -2 ) iterations are sufficient to guarantee that E [ f ( ¯ w T ) ] -inf f ≤ ϵ.

Proof. First we show that 1 1 -θ T ≤ 2 is equivalent to

Now note that log(1+ x ) ≥ x 1+ x for x ≥ 0 , which implies that 1 log(1+ x ) ≤ 1+ 1 x for x ≥ 0 . Applying this gives that

So to guarantee 1 1 -θ T ≤ 2 , from (26) it is sufficient to enforce that

Assuming A ≥ √ 2 /B , this last condition holds if

Since we also impose that BA 2 ≥ 2 we have that 1 BA 2 -1 ≤ 1 thus for (27) to hold it suffices that T ≥ 2 . Substituting γ = A √ T and the relaxation 1 1 -θ T ≤ 2 into the original bound gives the claimed result.

9.3 Proximal gradient descent with strong convexity and smoothness

We start by recalling some standard properties of the proximal operator.

Lemma 33. Let γ > 0 and w ∗ ∈ argmin w l ( w ) + h ( w ) . Assume h ( w ) is convex and that l ( w ) is continuously differentiable at w ∗ . Then

Theorem 7. Let l be a µ -strongly convex and M -smooth function, and let ¯ w = argmin( l ) . Let h be a proper closed convex function, and let w ∗ = argmin( l + h ) . Let ( w t ) t ∈ N be generated by the Prox-SGD algorithm, with a constant stepsize γ ∈ ( 0 , min { µ 2 a , 1 µ } ] . Suppose that g t is a quadratically bounded estimator (Def. 5) for ∇ l with parameters ( a, b, w ∗ ) . Then,

Indeed this follows since

Alternatively, if we use the decaying stepsize γ t = min { µ 2 a , 1 µ 2 t +1 ( t +1) 2 } , then

In both cases, T = O ( ϵ -1 ) iterations are sufficient to guarantee that E ∥ w T -w ∗ ∥ 2 ≤ ϵ.

Proof. We start by using the non-expansiveness and fixed-point properties of the proximal operator to write

Above the first line uses the fixed point property of the proximal operator that w ∗ = prox γh ( w ∗ -γ ∇ l ( w )) , while the second line uses the fact that the proximal operator is non-expansive.

Now, we apply our assumption on the gradient estimator to write

For the last term, we have by strong convexity and the fact that g t is an unbiased estimator that

where the second line follows from the fact that l is µ strongly convex.

Putting the pieces together, we get that

Now the following conditions are equivalent:

This means that

Now since γ ≤ 1 µ ⇔ 1 -γµ ≥ 0 we can apply the above recursively, which gives that

But we can bound this geometric sum by

leading to the result that

To prove the anytime result, we can now apply Theorem 31. To do so, we need to map the notation we use here to the notation used in Theorem 31. The mapping we need is L = a µ , and σ 2 = ( b + M 2 ∥ w ∗ -¯ w ∥ 2 2 ) which when inserting into Theorem 31 gives the result.

Corollary 34. Under the conditions of Theorem 7, if γ = A log T T for 1 µ ≤ A , and T is large enough that T log T ≥ 2 aA µ ,

Proof. The previous theorem can only be applied when γ ≤ µ 2 a , i.e. when A log T T ≤ µ 2 a or 2 aA µ ≤ T log T . Supposing that is true, observe that (1 -x ) T ≤ exp( -xT ) , from which it follows that (1 -γµ ) T ≤ exp( -µA log T ) = 1 T µA ≤ 1 T .

9.4 Proximal gradient descent with convexity and smoothness

Theorem 8. Let l be a proper convex and M -smooth function. Let h be a proper closed convex function, and let w ∗ ∈ argmin( l + h ) . Let ( w t ) t ∈ N be generated by the Prox-SGD algorithm, with a constant stepsize γ ∈ (0 , 1 M ] . Suppose that g t is a quadratically bounded estimator (Def. 5) for ∇ l with parameters ( a, b, w ∗ ) . Then,

where θ def = 1 1 + 2 aγ 2 and ¯ w T def = ∑ T t =1 θ t +1 w t ∑ T t =1 θ t +1 . In particular, if γ = 1 √ aT , then

Thus, T = O ( ϵ -2 ) iterations are sufficient to guarantee that E [ f ( ¯ w T ) -inf f ] ≤ ϵ.

Proof. Let us start by looking at ∥ w t +1 -w ∗ ∥ 2 -∥ w t -w ∗ ∥ 2 . Expanding the squares, we have that

Since w t +1 = prox γ t h ( w t -γ t g t ) , we know from Lemma 33 that

and therefore that where ∂h ( w ) is the subdifferential of h . Consequently there exists a subgradient b t +1 ∈ ∂h ( w t +1 ) such that

We decompose the last term of (28) as

For the first term in the above we can use that b t +1 ∈ ∂h ( w t +1 ) is a subgradient together with the defining property of subgradient to write

On the second term we can use the fact that l is M -smooth to write

On the last term we can use the convexity of f to write

By inserting (30), (31), (32) into (29) gives

Now using the above in (28), and our assumption that γ t M ≤ 1 , we obtain

We now have to control the last term of (33) in expectation. To shorten our notation we temporarily introduce the operators

Notice in particular that w t +1 = prox γ t h ( ˆ T ( w t )) . So the the last term of (33) can be decomposed as

We observe that the last term is, in expectation, is equal to zero. This is because prox γ t h ( T ( w t )) -w ∗ is deterministic when conditioned on w t . Since we will later on take expectations, we drop this term now and keep on going. As for the first term, using the nonexpansiveness of the proximal operator (Lemma 33), we have that

Using the above two bounds in (35) we have proved that (after taking expectation)

Injecting the above inequality into (33) and multiplying through by γ t , we obtain

From now on we assume that the stepsize sequence is constant γ t ≡ γ . Now, applying our assumption on the gradient estimator, we have

Inject this inequality into (36), and reordering the terms gives

where we introduced the constant C def = 1 + 2 aγ 2 . We will now do a weighted telescoping by introducing a sequence α t +1 > 0 of weights. Multiplying the above inequality by some α t +1 > 0 , and summing over 0 , . . . , T -1 gives

To be able to use a telescopic sum in the right-hand term, we need that α t +1 C = α t (an idea we borrowed from [54] ). This holds by choosing α t def = 1 C t , which allows us to write

Let us now introduce ¯ w T def = ∑ T -1 t =0 α t +1 w t +1 ∑ T -1 t =0 α t +1 , using that α 0 = 1 , and by using the convexity of f with Jensen's inequality, we obtain

To finish the proof, it remains to compute the geometric series ∑ T -1 t =0 α t +1 which is given by

Using the above and substituting for C = 1 + 2 aγ 2 and θ = 1 /C gives

Finally by applying Lemma 32 with where B = 2 a , A = √ 2 B = 1 √ a and thus γ = A √ T = 1 √ aT . Note that with this choice of γ we have that

Thus for T ≥ 2 the result of Lemma 32 gives

9.5 Projected gradient descent with strong convexity

Theorem 10. Let W be a nonempty closed convex set. Let f be a µ -strongly convex function, differentiable on W . Let w ∗ = argmin W ( f ) . Let ( w t ) t ∈ N be generated by the Proj-SGD algorithm, with a constant stepsize γ ∈ ( 0 , min { µ 2 a , 2 µ } ] . Suppose that g t is a quadratically bounded estimator (Def. 5) for ∇ f with parameters ( a, b, w ∗ ) . Then,

Alternatively, if we use the decaying stepsize γ t = min { µ 2 a , 2 µ 2 t +1 ( t +1) 2 } , then

In both cases, T = O ( ϵ -1 ) iterations are sufficient to guarantee that E ∥ w T -w ∗ ∥ 2 ≤ ϵ.

Proof. Plugging in one step of Proj-SGD we have that

where we used that projections are non-expansive. Taking expectation conditioned on w t , and expanding squares we have that

where we used E [ g t ] = ∇ f ( w t ) and definition 5, and that f is µ -strongly convex. Taking full expectation and re-arranging gives

Now using that γ t ≤ µ 2 a we have that

and thus

Re-arranging we have that

Using γ t ≡ γ constant, and since γ ≤ 2 µ ⇔ 1 -µγ 2 ≥ 0 we have by unrolling the recurrence that

Since we have that

which concludes the proof of (13).

To prove (14) we will apply Theorem 31, for which we will switch our notation to match that of Theorem 31. That is, substituting L = a µ , σ 2 = b 2 and ˆ µ = µ 2 into (39) gives

which now holds for γ t ≤ 1 2 L and fits exactly the format of (23), excluding the additional hat on µ. Thus we know from following verbatim the proof that continues after (23) that for a stepsize schedule of

for t ∗ = 4 ⌊L / ˆ µ ⌋ we have that

After switching back notation L = a , σ 2 = b and ˆ µ = µ

µ 2 2 gives (14).

9.6 Projected gradient descent with convexity

Theorem 11. Let W be a nonempty closed convex set. Let f be a convex function, differentiable on W . Let w ∗ ∈ argmin W ( f ) . Let ( w t ) t ∈ N be generated by the Proj-SGD algorithm, with a constant stepsize γ ∈ (0 , + ∞ ) . Suppose that g t is a quadratically bounded estimator (Def. 5) for ∇ f at w t with constant parameters ( a, b, w ∗ ) . Then,

where θ def = 1 1 + aγ 2 and ¯ w T def = ∑ T -1 t =0 θ t +1 w t ∑ T -1 t =0 θ t +1 . Finally if γ = √ 2 √ aT and T ≥ 2 then

Thus, T = O ( ϵ -2 ) iterations are sufficient to guarantee that E [ f ( ¯ w T ) -inf f ] ≤ ϵ.

Proof. Plugging in one step of the algorithm, we have that

where we used that projections are non-expansive. Taking expectation conditioned on w t , and expanding squares we have that

where we used that g t is an unbiased estimator with bounded squared norm. Taking full expectation and re-arranging gives

From now on we use a constant stepsize γ t ≡ γ . Let C = (1 + aγ 2 ) . Next we would like to setup a telescopic sum. To make this happen, we multiply through by α t +1 and impose that α t +1 C = α t . This holds with α t def = 1 C t . Multiplying through by α t +1 and summing from t = 0 , . . . , T -1 we have that

Using that α 0 = 1 , dropping the negative term -α T E [ ∥ ∥ w T -w ∗ ∥ ∥ 2 ] , dividing through by 2 γ ∑ T -1 k =0 α k +1 and using Jensen's inequality we have that

Finally using that

gives that

Finally we can apply Lemma 32 with where B = a , A = √ 2 /B = √ 2 / √ a , θ = 1 1+ aγ 2 and γ = A/ √ T = √ 2 / ( √ a √ T ) which gives

10 Solving VI with stochastic optimization

10.1 Convergence of the algorithms

Corollary 12 (Prox-SGD for VI) . Consider the VI problem where q w is a multivariate Gaussian distribution (Eq. 2) with parameters w = ( m,C ) ∈ R d × T d , and assume that this problem admits a solution w ∗ . Suppose that log p ( · , x ) is M -smooth and concave (resp. µ -strongly concave). Generate a sequence w t by using the Prox-SGD algorithm (Def. 6) applied to l and h (Eq. 16), using g energy (5) as an estimator of ∇ l . Let the stepsizes γ t be constant and equal to 1 / ( √ a energy T ) (resp. be decaying as in Theorem 7 with a energy = 2( d +3) M 2 ). Then, for a certain average ¯ w T of the iterates, we have for T ≥ 2 that

Proof. Our assumption on the log target imply that l is smooth and convex (resp. µ -strongly convex) according to Lemmas 18.1 and 19.1. Furthermore, since log p ( · , x ) is smooth, we know from Theorem 30 that the gradient estimator g energy is quadratically bounded at every w t with respect to constant parameters ( a, b, w ∗ ) , with a = 2( d +3) M 2 . Then, because we work with triangular scale parameters, we know from Lemma 19.3 that h is closed convex. Finally T ≥ max { M 2 /a, 2 } = 2 since

We have now verified the hypotheses of Theorem 8 (resp. Theorem 7), which gives us the desired bounds.

Corollary 13 (Proj-SGD for VI) . Consider the VI problem where q w is a multivariate Gaussian distribution (Eq. 2) with parameters w = ( m,C ) ∈ R d ×S d , and assume that this problem admits a solution w ∗ . Suppose that log p ( · , x ) is M -smooth and concave (resp. µ -strongly concave). Generate a sequence w t by using the Proj-SGD algorithm (Def. 9) applied to the function f = l + h (Eq. 16) and the constraint W M (Eq. 3), using g ent (7) or g STL (9) as an estimator of ∇ f . Let the stepsizes γ t be constant and equal to √ 2 / ( aT ) (resp. be decaying as in Theorem 10) with a = a ent = 4( d +3) M 2 or a = a STL = 24( d +3) M 2 . Then, for a certain average ¯ w T of the iterates, we have for T ≥ 2 that

Proof. Our assumption on the log target imply that l is smooth and convex (resp. µ -strongly convex) according to Lemmas 18.1 and 19.1. Regarding the entropy, since log p ( · , x ) is M -smooth, we know from Lemma 18.3 that h is M -smooth on W M . We also know that h is closed convex, since we consider symmetric scale parameters (see Lemma 19.2). Furthermore, the smoothness of log p ( · , x ) implies that the gradient estimator g ent is quadratically bounded at every w t ∈ W M , with respect to constant parameters ( a, b, w ∗ ) . This is stated in Theorem 30, where we can see that for g ent we can take a ent = 4( d +3) M 2 , and for g STL we can take a STL = 2( d +3)(2 K 2 +(2 M ) 2 ) . Since we know from Theorem 4 that K ≤ 2 M , we can consider that a STL ≤ 24( d +3) M 2 . Finally, we know from Lemma 20 that the minimizers of f belong to W M , meaning that argmin( f ) = argmin W M ( f ) . We have now verified the hypotheses of Theorem 11 (resp. Theorem 10), which gives us the desired bounds.

Corollary 14 (Proj-SGD for VI - Gaussian target) . Consider the setting of Corollary 13, in the scenario that log p ( · , x ) is µ -strongly concave, that we use the g STL estimator, and that we take a constant stepsize γ t ≡ γ ∈ ( 0 , min { µ 2 a STL , 2 µ } ] . Assume further that p ( ·| x ) is Gaussian. Then,

Proof. We use the same arguments as in the proof of Corollary 13, but this time we use a constant stepsize, which gives us the bound (13) from Theorem 10, where b = 4( d + 3) K 2 ∥ w ∗ -ˆ w ∥ 2 according to Theorem 4. All we need to conclude is to verify that b = 0 . This comes from the assumption that the target is Gaussian, from which we deduce that K = 0 (see again Theorem 4).

10.2 Computations for the algorithms

Lemma 35 (Prox of the entropy for triangular matrices) . Assume that V = T d , and let w = ( m,C ) ∈ R d ×T d . Then prox γh ( w ) = ( m, ˆ C ) , where

̸

Proof. This is essentially proved in [13, Theorem 13]. We provide a proof here for the sake of completeness, also because our entropy here enforces positive definiteness. So we need to compute

Because the objective here is separated in z and X , we can optimizae those variables separately. It is clear that the optimal z will be z = m . For the matrix component, it remains to solve

Now remember that for a triangular matrix, being positive definite is equivalent to have positive diagonal elements (Lemma 19.3). Because X is a lower triangular matrix, we have -log det X = ∑ d i =1 -log X ii and X ij = 0 when i < j . So our problem becomes

This is a gain a separated optimization problem with respect to the variables X ij . So we deduce that when i ≥ j we must take ˆ C ij = C ij , and when i = j we need to take

This is the proximal operator of -log , which can be classically computed [55, Example 6.9], and gives the desired result.

Lemma 36 (Projection onto W M for symmetric matrices) . Assume that V = S d , and let w = ( m,C ) ∈ R d ×S d . Let C = UDU ⊤ be the eigenvalue decomposition of C , with D diagonal and U orthogonal. Then proj W M ( w ) = ( m,U ˜ DU ⊤ ) , where ˜ D is the diagonal matrix defined by

Proof. Consider g = δ W M to be the indicator function of W M , which is equal to 0 on W M and + ∞ elsewhere. Then proj W M = prox g , and we can compute it with [55, Theorem 7.18]. For this, we need to write g ( C ) = F ( λ ( C )) , where λ ( C ) is the set of eigenvalues of C , and F is the indicator function of K := { z ∈ R d : z i ≥ 1 / √ M } . In that way, we see that g is a symmetric spectral function in the sense of [55, Definition 7.12], and the conclusion follows.

Lemma 37 (Adjoint of the distribution transformation) . Let u ∈ R d be fixed. Let T : R d × V → R d be such that T ( m,C ) = Cu + m . Then T is a linear operator such that its adjoint T ∗ : R d → R d × V verifies T ∗ ( z ) = ( z, proj V ( zu ⊤ )) .

Proof. The fact that T is linear is immediate. Let us consider now the extension ¯ T : R d ×M d → R d such that ¯ T ( m,C ) = Cu + m . If we define the canonical injection ι : R d × V → R d ×M d , we clearly have T = ¯ T ◦ ι . This means that T ∗ = ι ∗ ◦ ¯ T ∗ = proj R d × V ◦ ¯ T ∗ . It remains to prove that ¯ T ∗ ( z ) = ( z, zu ⊤ ) . For this, it suffices to use the properties of the trace to write that

10.3 Explicit implementation of the algorithms

Lemma 38 (Explicit computation of the estimators) . Let u ∈ R d , and let w = ( m,C ) ∈ R d × V where C ≻ 0 . Define

Then

Proof. Let w = ( m,C ) and let T be the linear operator defined by T ( m,C ) = Cu + m , as in Lemma 37. As a shorthand we will note p x := p ( · , x ) .

From its definition in (5), the energy estimator is equal to

Applying the chain rules, we obtain

The conclusion follows from the expression of T ∗ obtained in Lemma 37.

From its definition in (7), the entropy estimator g ent ( u ) is equal to g energy ( u ) + ∇ h ( w ) , where (we use the fact that we assumed that w ∈ W ) we know from Lemma 18.2 that ∇ h ( w ) = -(0 , proj V ( C -⊤ )) . The conclusion follows from the previous item and the linearity of proj V .

From its definition in (9), the STL estimator is equal to

where ϕ ( z ) = log q v ( z ) , with v being a copy of w . From the definition of a Gaussian density (17) we have that

so we deduce that

The conclusion follows after combining the first item with Lemma 37.

Lemma 39 (Pseudocode of the algorithms) .

Algorithm 1 is equivalent to the Prox-SGD algorithm (Def. 6) applied to l and h (Eq. 16), using g energy (5) as an estimator of ∇ l and using lower triangular covariance factors.

Algorithm 2 is equivalent to the Proj-SGD algorithm (Def. 9) applied to the function f = l + h (Eq. 16) and the constraint W M (Eq. 3), and using g ent (7) as an estimator of ∇ f .

Algorithm 3 is equivalent to the Proj-SGD algorithm (Def. 9) applied to the function f = l + h (Eq. 16) and the constraint W M (Eq. 3), and using g STL (9) as an estimator of ∇ f .

Proof. In this proof, we fix an iteration t ∈ N , consider u t ∼ N (0 , I ) and we note w t = ( m t , C t ) .

From (Def. 6) we know that w t +1 = prox γ t h ( w t -γ t g energy ( u t )) . According to Lemma 38, we have

where

Moreover, we know from Lemma 35 what prox γ t h is:

̸

First, we see that with respect to the m variable the proximal operator is the identity, meaning that we indeed have

Second, we see that with respect to the C variable we have

It is immediate to see by induction that since we initialize with C 0 ∈ T d , then C t ∈ T d as well. Therefore we can use the linearity of proj T d to write that

The conclusion follows after observing that

From (Def. 9) we know that w t +1 = proj W M ( w t -γ t g ent ( u t )) . According to Lemma 38, we have

Moreover, we know from Lemma 36 what proj W M is:

where ˜ R ( C ) = U ˜ DU ⊤ , with UDU ⊤ being a SVD decomposition of C , and ˜ D ( i, i ) = max { D ( i, i ); M -1 / 2 } . First, we see that with respect to the m variable the projection is the identity, meaning that we indeed have

Second, we see that with respect to the C variable we have

It is immediate to see by induction that since we initialize with C 0 ∈ S d , then C t ∈ S d as well. Therefore we can use the linearity of proj S d to write that

The conclusion follows after setting ˆ C t +1 = C t -γ t ( π t u ⊤ t -C -⊤ t ) , and using the fact that proj S d ( C ) = ( C + C ⊤ ) / 2 .

Use the same reasoning as in the previous item, with the appropriate computation for the STL estimator from Lemma 38.

Algorithm 1 Prox-SGD with energy estimator and triangular factors

Algorithm 2 Proj-SGD with entropy estimator and symmetric factors

Algorithm 3 Proj-SGD with STL estimator and symmetric factors

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper offers the first convergence results for black-box VI, a widely used and popular framework for Bayesian problems. Under assumptions on the log model, $\\log p$, and given a Gaussian variational family of distributions, convergence rates are established by utilizing recent advances in the field. The assumptions are motivated by giving practical examples of when they are satisfied.\n\nSTRENGTHS:\nThis work seems to lay the missing puzzle piece in a series of works towards establishing convergence rates for black-box VI. As such, it is clearly original, and it is significant as black-box VI is widely used in practice and is a popular research topic. Although the results build on seemingly strong assumptions on the log model, $\\log p$, the significance of the results is emphasized by exemplifying broad classes of problem settings where the assumptions are satisfied, which deserves credit. I believe this paper will be of interest to the NeurIPS community.\n\nThe paper follows a clear line of argument and builds up to its results in a pedagogical manner, giving a convincing account of the existing, related works.\n\nWEAKNESSES:\n**W1**: $C$ is a matrix (line 63), but in line 120 I read it as $C = 0$, by inspecting $\\bar{w} = (\\bar{m}, 0)$. It is not clear to me what this means. Are all elements in the covariance matrix zero (since $\\Sigma = C C^T$)?\n\n**W2**: In Sec. 2.1 I read it as, if $\\log p$ is $M$-smooth then we can expect that the optimal covariance parameter for $q_w$ should not be too small. This makes sense to me. In fact they should be at least $1 / \\sqrt{M}$ as shown in Domke's previous work. However in Theorem 2, where the log-target is assumed to be $M$-smooth, it says that $\\bar{w}=(\\bar{m}, 0)$ is the \"maximum of $\\log p$\". I am confused about this. Doesn't $\\bar{w}$ here imply that $\\log p$ is not smooth, and that the optimal parameters $w^* = \\bar{w}$, i.e. $C^* = 0 < 1 / \\sqrt{M}$?\n\nIf I am helped clear up this confusion, I am willing to raise my score.\n\n**W3:** I am not sure how the discussion about the score-type estimator in Eq. 4 contributes to the paper. The results in paper are based on the reparameterization trick (i.e. the path-type estimator in Eq. 5), no? What is the contribution of this discussion to the results in the paper?\n\n**Minor issues**\n\nIn line 16, $h$ is called the entropy, but as it is defined in Eq. 1 and 15 it is the neg-entropy. \n\nSometimes (e.g. in the Central contributions box and on line 93) $q_w$ is referred to a family of distributions and sometimes (line 243) it is used a density. In VI the \"family\" is often meant as the set of distributions in which we want to find the optimal approximation, no? So can it really be a family and a distribution simultaneously? Perhaps it is more clear if $\\mathcal{Q}_w$ is used to notate the family of Gaussian distributions parameterized by $w$?\n\nTypo: \n* line 35: objective is misspelled\n* line 141: \"that bar\" seems to be a typo\n\nQUESTIONS:\n* In Sec. 2.1 it stated that $h$ is $M$-smooth over the set $\\mathcal{W}_M$. Based on this, do you make any assumptions on $q_w$ other than it being Gaussian? I was expecting some new assumption given the title of the section, but maybe this is a \"structural property\"?\n* Out of curiosity, could you expand on the constructed problem where the variance of the path-based estimator is \"vastly\" increased and the score-based estimator is not? Where is the noise added? To $\\phi$?\n\nLIMITATIONS:\nThe assumptions are clearly highlighted and discussed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper offers a convergence proof for the stochastic optimization problem inherent in full-rank Gaussian variational inference when the log-density of the target is concave. The primary challenge of the convergence proof lies in managing the non-smoothness present in the entropy term of Gaussian VI. This issue is addressed in the current work by considering proximal and projected stochastic gradient descent. To evaluate the validity of the imposed assumptions, case studies involving Bayesian (generalized) linear regression problems are provided.\n\nSTRENGTHS:\nI appreciate the clarity of the writing of the present work; it clearly descibes the scope of the problem studied and states the main challenge of showing the convergence. The layout of each section forms a smooth flow of the proof strategies, in addition to a nice organization of the mathematical proofs, which makes the reading enjoyable.\n\nThe case study provided in appendix (sec 7.3) is helpful for reasoning imposed assaumptions. \n\nThe proximal operator introduced for optimizing the covariance matrix is novel to me. Although it's a standard techniques in optimization literature---splitting non-strongly smooth (but closed convex) part of the objective function using proximal operator, it's the first time I see in Gaussian VI literature. (This could of course be due to my lack of knowledge in this field) Additionally, the weighted telescope summation used when $-logp$ is only convex(rather than strongly convex) is very interesting, which is a setting that rarely considered in literature as far as I'm aware.   \n\nEven though the paper limits the scope to dense Gaussian VI problem, I think the present techniques can be applied to general location-scale variational families (as long as the variance quadratic bounds still holds).\n\nWEAKNESSES:\nIn my view, the principal limitation of this work is the somewhat restricted scope of the Variational Inference (VI) problem it examines. Specifically, the variational family under consideration is: (1.) Gaussian (albeit full-rank), and (2.) the target distribution is log-concave or strongly-log-concave. Additionally, (3.) data subsampling on $\\log p$ is not taken into account. The proof techniques necessitated by this setting, in my opinion, are fairly standard within the stochastic optimization literature. While I acknowledge that any relaxation of condition (2) would likely preclude anything beyond convergence to some stationary point, and that the quadratic variance bound is heavily reliant on condition (1) (which is probably not a major concern for the broader community in regards to more complex VI families), offering a convergence result when data subsampling is implemented could greatly increase the impact of this work. I would be inclined to raise the score to 8 if data subsampling was considered, or if existing proof techniques could easily address it (though I doubt the current approach is effective in this case, please correct me if I'm mistaken).\n\nRegarding originality, even though I concur that explicit results may be lacking for this specific problem setting, I don't think this is the **first** optimization theory guarantee for full-rank Gaussian VI. For instance, [Xu & Campbell, 2022] studies the convergence of full-rank Gaussian VI without the assumption of a log-concave target (even though they utilize posterior asymptotics to somewhat reduce the underlying optimization problem on a strongly-log-concave target, they also employ a special scaling operator (Eq(8)) to handle the non-Lip-smoothness of the entropy term $\\log \\text{det}$). I recommend that the authors integrate an optimization analysis of the scaled stochastic gradient descent proposed in [Xu & Campbell, 2022], and carefully compare the results to those provided in [Xu & Campbell, 2022] within a strongly convex setting at least (ignoring the data asymptotics and merely assuming that $\\log p$ is strongly-log-concave should align their work with the current setting).\n\nFinally,  I encourage the authors to provide additional case studies, perhaps involving more Bayesian GLMs or even Bayesian sparse regression with a Horseshoe prior. From what I understand, most of these models yield a log-concave target, but the application of the variance quadratic bound is unclear to me. If negative results were to appear, for example, some models not having a quadratically bounded gradient noise, these examples would still significantly benefit the community.\n\n\nAt the end, I hope to stress that despite the aforementioned weakness/limitations, I think this is a really good work for this niche research direction.  \n\nReference:\n [Xu&Campbell, 2022]: The computational asymptotics of Gaussian variational inference and the Laplace approximation, Statistics and Computing\n\nQUESTIONS:\n1. Compare to  [Xu&Campbell, 2022]: As I mentioned earlier, this is a very relavant past work on this topic. I wonder specifically, if we ignore all the asymptotic stuff and assumes $\\log p$ is strongly-log-concave, does [Xu&Campbell, 2022] achieve similar convergence rate ($O(1/T)$) as the present paper? \n\n2. I wonder if the proximal operator (line 264) has been noted in past VI literature? Or it is proposed by the author.\n\nLIMITATIONS:\nSee Weakness", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

comments: SUMMARY:
The paper proves 1/sqrt(T) respectively 1/T convergence rates for black-box variational inference methods, when implemented with a proximal stochastic gradient method.  Such rates were not available in the literature until now, due to difficulties in bounding the gradient noise. The main contribution of the paper is to prove new bounds on the gradient noise and incorporate them into the proof techniques for proximal SGD.

STRENGTHS:
The paper is well-written and clearly states its contributions.  While the topic seems rather narrow, the technical part of the paper appears to be a novel and non-trivial result.

WEAKNESSES:
1) The paper solves a quite "narrow" problem, that is fixing some gaps in existing convergence proofs for variational inference. It is unclear whether this work will be of interest to the wider ICML community, as it is a niche topic, and the theory does not seem to directly suggest any practical improvements or changes. 

2) BBVI is sometimes less preferable in practice than natural-gradient algorithms. These methods use the KL / Fisher-geometry (https://arxiv.org/abs/2107.04562). This line of works could be mentioned in the introduction.

3) There has been some recent works
https://arxiv.org/abs/2205.15902
https://proceedings.mlr.press/v202/diao23a.html
which prove convergence for BBVI-like algorithms by interpreting them as Wasserstein gradient flow. Perhaps these works could be mentioned.

QUESTIONS:
1) Which algorithm would be preferable in practice, prox-SGD or proj-SGD? From the theory side, it seems they have similar convergence rates, but maybe there are other things to take into consideration?

LIMITATIONS:
All limitations are addressed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the lack of provable convergence guarantees for black-box variational inference (VI) and proposes convergence guarantees for two stochastic optimization algorithms applied to Gaussian variational families. \n\nThe authors identify challenges in analyzing VI as a standard stochastic optimization problem, including the non-smoothness of the objective function, lack of uniform smoothness, and lack of uniformly bounded noise for gradient estimators. \n\nThey provide theoretical results and noise bounds for two gradient estimators and propose proximal and projected stochastic gradient descent algorithms.\n\nI did not find technical flaws.\n\nSTRENGTHS:\n1. The paper addresses an important problem by providing provable convergence guarantees for black-box VI, which is widely used but lacks theoretical guarantees.\n\n2. The authors clearly articulate the challenges in analyzing VI as a standard stochastic optimization problem, such as the non-smoothness of the objective function and the lack of uniform smoothness and noise bounds. The paper provides rigorous theoretical results and noise bounds for gradient estimators used in black-box VI.\n\n3. The paper is well written. It provides a comprehensive and coherent review of the literature and clearly presents the technical contributions and proof ideas. I'm not a theory guy but I enjoy reading the paper.\n\nWEAKNESSES:\n1. I wonder if is it possible to perform some simple (even toy) examples to empirically check the results and illustrate the theory in a better way. \n\n2. The dense Gaussian family is still limited (at least in my opinion). It is better to discuss whether the techniques developed in the paper can be applied to more general cases (e.g., black-box variational inference with a neural network model) or at least discuss the challenges.\n\nQUESTIONS:\nSee weakness.\n\nLIMITATIONS:\nNone.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper analyzed the convergence of Black-box VI, which has been widely used in variational inference in recent years. Under the assumption that the target joint distribution is convex or strongly convex and the variational posterior distribution is Gaussian, the convergence of the variational parameter obtained by Black-box VI to the ELBO optimal solution is first presented when using Prox-GD and Projected Gradient Descent. An important aspect of the analysis of Black-box VI is that the gradient is stochastic.\nTo deal with this, the authors introduced a class of quadratically bounded estimators as estimators of the gradient and combined them appropriately with the existing Prox-GD analysis.\n\nSTRENGTHS:\nAlthough the proposed analysis makes a number of assumptions, the results are significant in that they include an important Bayesian model that is used in practice.\n\nFor the first time, an analysis of the convergence of Black-box VI is presented, albeit with a rather limited problem, such as the assumption of variational posterior distribution as Gaussian distribution.\n\nWEAKNESSES:\nIt was very difficult to understand how novel and important the proposed analysis is from a single reading. In existing studies, each component of the presented analysis is already well known, but I am not sure how important and novel the presented  VI analysis is as the combination of those well-known results.\n\nBy assumption, since the objective function is convex (or strongly convex), the convergence using Prox SGD or other methods seems apparent, the difficulty of the problem stems from the noisy version of the gradient, as discussed in  Sec 2.3. On the other hand, as there have been many studies discussing noisy versions of prox GD, such as Stochastic Proximal Gradient Descent[1], so I am not sure how novel this analysis is.\n\nIn the proofs of Theorem 6 and 7, I understood that the novelty is mainly the technique to treat the gradient estimator that satisfies the quadratic bounded property shown in Definition 4. However, I still did not understand how important it is compared to existing studies such as the Stochastic Proximal Gradient Descent study.\n\n[1]Convergence of Stochastic Proximal Gradient Algorithm, L. Rosasco et al.\n\nQUESTIONS:\nI would like to know more about the novelty of this study in the context of Stochastic Proximal Gradient Descent in terms of technical aspects and assumptions of the proof.\n\nLIMITATIONS:\nYes, the limitations of the statements are clearly explained.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-27"}