SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[38] Mohammad E. Khan, Pierre Baqué, François Fleuret, and Pascal Fua. Kullback-Leibler proximal variational inference. In NeurIPS, pages 3402-3410, 2015.
[39] Mohammad Emtiyaz Khan, Reza Babanezhad, Wu Lin, Mark Schmidt, and Masashi Sugiyama. Faster Stochastic Variational Inference using Proximal-Gradient Methods with General Divergence Functions. In UAI, 2016.
[40] K. Fan, Y. Zhang, R. Henao, and K. Heller. Triply Stochastic Variational Inference for Nonlinear Beta Process Factor Analysis. In 2016 IEEE 16th International Conference on Data Mining (ICDM), pages 121-130, 2016.
[41] Pierre Alquier and James Ridgway. Concentration of tempered posteriors and of their variational approximations. arXiv:1706.09293 [cs, math, stat], 2017.
[42] Guanghui. Lan. First-order and Stochastic Optimization Methods for Machine Learning. Springer Series in the Data Sciences. Springer International Publishing, Cham, 1st ed. 2020. edition, 2020. ISBN 3-030-39568-5.
[43] Michalis K. Titsias and Miguel Lázaro-Gredilla. Local Expectation Gradients for Black Box Variational Inference. In NeurIPS, 2015.
[44] Francisco J. R. Ruiz, Michalis K. Titsias, and David M. Blei. The Generalized Reparameterization Gradient. In NeurIPS, 2016.
[45] Francisco J. R. Ruiz, Michalis K. Titsias, and David M. Blei. Overdispersed Black-Box Variational Inference. arXiv:1603.01140 [stat], 2016.
[46] Andrew C. Miller, Nicholas J. Foti, Alexander D'Amour, and Ryan P. Adams. Reducing Reparameterization Gradient Variance. arXiv:1705.07880 [stat], 2017.
[47] Linda S. L. Tan and David J. Nott. Gaussian variational approximation with sparse precision matrices. Stat Comput, 28(2):259-275, 2018.
[48] Tomas Geffner and Justin Domke. Using Large Ensembles of Control Variates for Variational Inference. In NeurIPS, 2018.
[49] Tomas Geffner and Justin Domke. Approximation based variance reduction for reparameterization gradients. In NeurIPS, 2020.
[50] Masahiro Fujisawa and Issei Sato. Multilevel monte carlo variational inference. Journal of Machine Learning Research, 2021.
[51] T. M. Cover and Joy A. Thomas. Elements of Information Theory. Wiley-Interscience, Hoboken, N.J, 2nd ed edition, 2006.
[52] Amir Beck. First-Order Methods in Optimization. SIAM-Society for Industrial and Applied Mathematics, Philadelphia, PA, USA, 2017. ISBN 1611974984.
[53] Robert Mansel Gower, Nicolas Loizou, Xun Qian, Alibek Sailanbayev, Egor Shulgin, and Peter Richtárik. SGD: General analysis and improved rates. In Proceedings of the 36th International Conference on Machine Learning, spotlight paper, volume 97, pages 5200-5209. PMLR, 09-15 Jun 2019.
[54] Sebastian U. Stich. Unified optimal analysis of the (stochastic) gradient method. arXiv:1907.04232, 2019.
[55] Amir Beck and Marc Teboulle. Gradient-based algorithms with applications to signal recovery. Convex optimization in signal processing and communications, pages 42-88, 2009.
1.1 Related work 2
2 Properties of Variational Inference (VI) problems 3
2.1 Structural properties and assumptions 3
2.2 Gradient estimators 4
2.3 Challenges for optimization 5
3 Stochastic Optimization with quadratically bounded estimators 6
3.1 Stochastic Proximal Gradient Descent 7
3.2 Stochastic Projected Gradient Descent 7
4 Solving VI with provable guarantees 8
5 Discussion 10
6 Notations 15
7 VI problems and their structural properties 15
7.1 Modeling the VI problem 15
7.2 Smoothness and convexity for VI problems 17
7.3 A case study : linear models 19
8 Estimators and variance bounds 22
8.1 Variance bounds for the estimators 22
8.2 Quadratically bounded estimators 25
9 Optimization proofs 25
9.1 Anytime Convergence Theorem for Strongly Convex 25
9.2 Complexity Lemma for Convex 26
9.3 Proximal gradient descent with strong convexity and smoothness 27
9.4 Proximal gradient descent with convexity and smoothness 29
9.5 Projected gradient descent with strong convexity 32
9.6 Projected gradient descent with convexity 33
10 Solving VI with stochastic optimization 35
10.1 Convergence of the algorithms 35
10.2 Computations for the algorithms 36
10.3 Explicit implementation of the algorithms 37
6 Notations
We note  $\mathcal{M}^d$  the space of  $d\times d$  matrices,  $\mathcal{S}^d$  the subspace of symmetric matrices, and  $\mathcal{T}^d$  the subspace of lower triangular matrices. We use  $\mathcal{W}_M\coloneqq \{(m,C)\in \mathcal{W}:\sigma_{\mathrm{min}}(C)\geq \frac{1}{\sqrt{M}}\}$  to denote the set of parameters with positive definite covariance matrix whose singular values are greater or equal to  $\frac{1}{\sqrt{M}}$ .
object	description
x	observed variables
z ∈ Rd	latent variables
p(z,x)	target distribution
w = (m,C)	variational parameters (m ∈ Rd and C ∈ Md, Sd or Td)
qw(z) = N(z|m, CC†)	Gaussian variational distribution
l(w) = -Ez~qw log p(z,x)	free energy
h(w) = Ez~qw log q(z)	negative entropy
f(w) = l(w) + h(w)	negative ELBO
w* = argminw l(w) + h(w)	optimal parameters
bar{z} = (bar{z},0)	MAP parameters (bar{z} ∈ argmaxz p(z,x))
V[g] = tr C[g].	variance of vector-valued random variables
7 VI problems and their structural properties
This section contains the proofs of all the claims made in Section 2.
7.1 Modeling the VI problem
We recall and detail here the setup of our problem. Given a target distribution  $p$  and observed data  $x$ , we want to solve the following Variational Inference problem
$$
\min  _ {w \in \mathcal {W}} \mathrm {K L} \left(q _ {w} \| p (\cdot | x)\right), \tag {VI}
$$
where  $q_{w}$  is a multivariate Gaussian variational family, whose density may be written as
$$
q _ {w} (z) = \frac {1}{\sqrt {(2 \pi) ^ {d} \det  (C C ^ {\top})}} \exp \left(- \frac {1}{2} \| C ^ {- 1} (z - m) \| ^ {2}\right), \quad w = (m, C). \tag {17}
$$
We make the assumption (VI) has a non-degenerated solution, i.e. a solution for which the covariance matrix is invertible.
We will impose that our parameters have the form  $w = (m, C) \in \mathbb{R}^d \times V$ , where  $V$  is some vector subspace of  $\mathcal{M}^d$ , the space of  $d \times d$  matrices. On top of that, we will also want the covariance factor to be positive definite. In other words, we will impose that our parameters are taken from the set
$$
\mathcal {W} := \{(m, C) \in \mathbb {R} ^ {d} \times V: C \succ 0 \}.
$$
For the sake of simplicity, the reader can assume for most of the paper that  $V = \mathcal{M}^d$ . But when coming to analyze our algorithms, we will see that we should take either  $V = \mathcal{T}^d$  (the subspace of  $d \times d$  lower triangular matrices) or  $V = \mathcal{S}^d$  (the subspace of  $d \times d$  symmetric matrices). This choice is dictated by two reasons: it guarantees that our problem remains convex (see Lemma 19), and it lowers the computational cost of our algorithms (see Section 10.2). Fortunately, requiring  $C$  to be symmetric (or triangular) and positive definite does not change our problem, as stated in the next Lemma.
Lemma 15 (Equivalent problems for triangular/symmetric covariance factors). Suppose that  $\bar{w} \in \mathbb{R}^d \times \mathcal{M}^d$  is such that  $q_{\bar{w}}$  minimizes  $\mathrm{KL}(q_w \| p(\cdot | x))$  over  $\mathbb{R}^d \times \mathcal{M}^d$ . Then there exists  $w^* = (m^*, C^*)$  such that  $q_{\bar{w}}$  and  $q_{w^*}$  have the same distribution, with  $C^*$  being positive definite and symmetric (or lower triangular).
Proof. Write  $\bar{w} = (\bar{m},\bar{C})$ , and take  $m^{*} = \bar{m}$ . Let  $\bar{\Sigma} = \bar{C}\bar{C}^{\top}$ . Since we assumed that (VI) has a non-degenerated solution, we can assume without loss of generality that  $\bar{\Sigma}$  is invertible. Therefore we can apply the Cholesky decomposition and write  $\bar{\Sigma} = C^{*}(C^{*})^{\top}$ , where  $C^{*}$  is lower triangular and positive definite. We can also define  $C^{*} = (\bar{\Sigma})^{1 / 2}$  to obtain a symmetric and positive definite factor.
Once restricted to  $\mathcal{W}$ , our problem (VI) becomes equivalent to
$$
\min  _ {w = (m, C) \in \mathbb {R} ^ {d} \times V} l (w) + h (w), \tag {18}
$$
-  $l(w) = -\mathbb{E}_{z \sim q_w} \log p(z, x)$  is the free energy;
-  $h(w) = \begin{cases} -\log \det C & \text{if } w \in \mathcal{W} \\ +\infty & \text{otherwise} \end{cases}$
This equivalence is mostly standard, but we state it formally for the sake of completeness.
Lemma 16 (VI reformulated). Problems (VI) and (18) are equivalent.
Proof. Let  $w = (m, C) \in \mathcal{W}$ , and let us show that  $\mathrm{KL}(q_w \| p(\cdot | x)) = l(w) + h(w)$ . By definition of the Kullback-Liebler divergence, and using  $p(z|x) = p(z,x) / p(x)$ , we can write
$$
\begin{array}{l} \operatorname {K L} \left(q _ {w} \| p (\cdot | x)\right) = \underset {z \sim q _ {w}} {\mathbb {E}} \log \frac {q _ {w} (z)}{p (z | x)} = \underset {z \sim q _ {w}} {\mathbb {E}} \log \frac {q _ {w} (z) p (x)}{p (z , x)} \\ = \underset {\mathbf {z} \sim q _ {w}} {\mathbb {E}} \log q _ {w} (\mathbf {z}) - \underset {\mathbf {z} \sim q _ {w}} {\mathbb {E}} \log p (\mathbf {z}, x) + \underset {\mathbf {z} \sim q _ {w}} {\mathbb {E}} \log p (x) \\ = \underset {\mathbf {z} \sim q _ {w}} {\mathbb {E}} \log q _ {w} (\mathbf {z}) + l (w) + \log p (x). \\ \end{array}
$$
If we note  $H(X)$  the entropy of a random variable, we can see that  $\mathbb{E}_{\mathbf{z} \sim q_w} \log q_w(\mathbf{z}) = -H(\mathbf{z})$  where  $\mathbf{z} \sim q_w$ . We can rewrite  $\mathbf{z}$  via an affine transform as  $\mathbf{z} = C\mathbf{u} + m$  with  $\mathbf{u} \sim \mathcal{N}(0, I)$ , so we can use [51, Section 8.6] to write (see also the arguments in [13])
$$
H (\mathfrak {z}) = H (\mathfrak {u}) + \log | \det  C |.
$$
Here  $H(\mathfrak{u})$ , the entropy of  $\mathfrak{u}$ , is a constant equal to  $(d / 2)(1 + \log (2\pi))$ . Moreover since  $w \in \mathcal{W}$ , we know that  $C\succ 0$ , which implies that  $\log |\operatorname*{det}C| = \log \operatorname*{det}C$ . Because  $\log p(x)$  and  $H(\mathfrak{u})$  are both constants, we deduce that minimizing  $\mathrm{KL}(q_w\| p(\cdot |x))$  over  $w \in \mathcal{W}$  is equivalent to minimizing  $l + h$  where  $h(w) = -\log \operatorname*{det}C$  if  $w \in \mathcal{W}$  and  $+\infty$  otherwise.
We emphasize that working with a triangular or symmetric representation for  $C$  is not restrictive, and that doing computations in this setting is pretty straightforward, as we illustrate next.
Lemma 17 (Computations with triangular/symmetric covariance factors). Let  $V$  be a vector subspace of  $\mathcal{M}^d$ . Let  $\phi : \mathbb{R}^d \times \mathcal{M}^d \to \mathbb{R}$  and let  $\phi_V : \mathbb{R}^d \times V \to \mathbb{R}$  be the restriction of  $\phi$  to  $\mathbb{R}^d \times V$ . Then:
1. If  $\phi$  is convex then  $\phi_V$  is convex.
2. If  $\phi$  is differentiable then  $\phi_V$  is differentiable, with  $\nabla \phi_V(w) = \mathrm{proj}_{\mathbb{R}^d\times V}(\nabla \phi (w))$ . In particular,  $\| \nabla \phi_V(w)\| \leq \| \nabla \phi (w)\|$ .
3. If  $\phi$  is  $M$ -smooth over  $\Omega \subset \mathbb{R}^d \times \mathcal{M}^d$ , then  $\phi_V$  is  $M$ -smooth over  $\Omega \cap (\mathbb{R}^d \times V)$ .
4.  $\operatorname{proj}_{S^d}(C) = (C + C^\top)/2$ , and  $\operatorname{proj}_{\mathcal{T}^d}(C) = \operatorname{tril}(C)$ , where  $\operatorname{tril}(C)$  sets to zero all the entries of  $C$  above the diagonal.
1. This is immediate due to the fact that  $V$  is convex, since it is a vector subspace of  $\mathcal{M}^d$ .
2. Let  $\iota : \mathbb{R}^d \times V \to \mathbb{R}^d \times \mathcal{M}^d$ ,  $\iota(m, C) = (m, C)$  be the canonical injection. Then we clearly have  $\phi_V = \phi \circ \iota$ . Moreover,  $\iota$  is a linear application whose adjoint is  $\iota^*$  is exactly the orthogonal projection  $\operatorname{proj}_{\mathbb{R}^d \times V}$ . So the computation of the gradient follows after applying the chain rule:  $\nabla \phi_V(w) = \iota^*(\nabla \phi(\iota(w)) = \operatorname{proj}_{\mathbb{R}^d \times V}(\nabla \phi(w))$ . As for the norm, it suffices to observe that the orthogonal projection is a linear operator with norm less or equal than 1.
3. Using that the orthogonal projection is a contractive map gives
$$
\begin{array}{l} \| \nabla \phi_ {V} (w) - \nabla \phi_ {V} \left(w ^ {\prime}\right) \| = \| \operatorname {p r o j} _ {\mathbb {R} ^ {d} \times V} (\nabla \phi (w)) - \operatorname {p r o j} _ {\mathbb {R} ^ {d} \times V} \left(\nabla \phi \left(w ^ {\prime}\right)\right) \| \\ \leq \| \nabla \phi (w) - \nabla \phi \left(w ^ {\prime}\right) \| \leq M \| w - w ^ {\prime} \|. \\ \end{array}
$$
4. This is a standard linear algebra result.
7.2 Smoothness and convexity for VI problems
Now we state the main smoothness and convex properties for VI problems. We will see in our analysis that the smoothness properties often revolve around the following subset of  $\mathcal{W}$
$$
\mathcal {W} _ {M} := \left\{(m, C) \in \mathcal {W}: \sigma_ {\min } (C) \geq \frac {1}{\sqrt {M}} \right\},
$$
where  $\sigma_{\mathrm{min}}(C)$  refers to the smallest singular value of  $C$
Lemma 18 (Smoothness for VI). Assume that  $\log p(\cdot ,x)$  is  $M$  -smooth. Then
1.  $l$  is  $M$ -smooth.
2.  $h$  is differentiable over  $\mathcal{W}$ , with  $\nabla h(w) = (0, -\mathrm{proj}_V(C^{-\top}))$  for all  $w \in \mathcal{W}$ .
3.  $h$  is  $M$ -smooth over  $\mathcal{W}_M = \{(m, C) \in \mathcal{W} : \sigma_{\min}(C) \geq \frac{1}{\sqrt{M}}\}$ .
1. Combine [13, Theorem 1] with Lemma 17.3.
2. Combine the fact that the gradient of  $-\log \det$  at a matrix  $X$  is  $-X^{-\top}$  with Lemma 17.2.
3. Combine [13, Lemma 12] with Lemma 17.3.
Lemma 19 (Convexity for VI). Assume that  $-\log p(\cdot, x)$  is convex (resp.  $\mu$ -strongly convex). Then
1.  $l$  is convex (resp.  $\mu$ -strongly convex).
2. If  $V = \mathcal{S}^d$  then  $h$  is closed convex,  $\mathcal{W}$  is convex, and  $\mathcal{W}_M$  is convex and closed. Moreover,
$$
\mathcal {W} = \left\{(m, C) \in \mathbb {R} ^ {d} \times \mathcal {S} ^ {d}: \lambda_ {\min } (C) > 0 \right\} a n d \mathcal {W} _ {M} = \left\{(m, C) \in \mathbb {R} ^ {d} \times \mathcal {S} ^ {d}: \lambda_ {\min } (C) \geq \frac {1}{\sqrt {M}} \right\}.
$$
3. If  $V = \mathcal{T}^d$  then  $h$  is closed convex and  $\mathcal{W}$  is convex. Moreover
$$
\mathcal {W} = \{(m, C) \in \mathbb {R} ^ {d} \times \mathcal {T} ^ {d}: C h a s p o s i t i v e d i a g o n a l \}.
$$
4. If  $V = \mathcal{M}^d$  then  $h$  can be not convex.
1. Combine Titsias and Lázaro-Gredilla [17, Proposition 1] (resp. Domke [13, Theorem 9]) with Lemma 17.1.
2. Convexity of  $h$  follows from [52, Example 7.13 and Theorem 7.17]. Convexity of  $\mathcal{W}$  comes from the fact that it is essentially the set of symmetric positive definite matrices (which is equivalent to  $\lambda_{\min}(C) > 0$ ). Let us now prove that  $\mathcal{W}_M$  is convex and closed. First we notice that, because of symmetry, we can rewrite  $\mathcal{W}_M$  as
$$
\mathcal {W} _ {M} = \{(m, C) \in \mathbb {R} ^ {d} \times \mathcal {S} ^ {d}: \lambda_ {\min} (C) \geq \frac {1}{\sqrt {M}} \}.
$$
This set is closed because  $\lambda_{\mathrm{min}}$  is a continuous function. To see that  $\mathcal{W}_M$  is convex we are going to use the fact that  $\lambda_{\mathrm{min}}$  is a concave function. Take  $C_1, C_2 \in \mathcal{W}_M$ ,  $\alpha \in [0,1]$  and write
$$
\begin{array}{l} \lambda_ {\min } ((1 - \alpha) C _ {1} + \alpha C _ {2}) = \min  _ {\| z \| = 1} \langle ((1 - \alpha) C _ {1} + \alpha C _ {2}) z, z \rangle \\ = \min  _ {\| z \| = 1} (1 - \alpha) \langle C _ {1} z, z \rangle + \alpha \langle C _ {2} z, z \rangle \\ \geq \min _ {\| z \| = 1} (1 - \alpha) \langle C _ {1} z, z \rangle + \min _ {\| z \| = 1} \alpha \langle C _ {2} z, z \rangle \\ = (1 - \alpha) \lambda_ {\min } (C _ {1}) + \alpha \lambda_ {\min } (C _ {2}) \geq \frac {1}{\sqrt {M}}. \\ \end{array}
$$
3. Here  $\mathcal{W} = \{(m,C)\in \mathbb{R}^d\times \mathcal{T}^d:C\succ 0\}$ . Using Sylvester's criterion, it is easy to see that a lower triangular matrix is positive definite if and only if its diagonal has positive coefficients. In other words,  $\mathcal{W} = \{(m,C)\in \mathbb{R}^d\times \mathcal{T}^d:C_{ii} > 0\}$ , which is clearly convex. It remains to verify that  $h$  is closed convex even if  $\mathcal{W}$  is not closed. For all  $w = (m,C)\in \mathbb{R}^d\times \mathcal{T}^d$ , we can use the triangular structure of  $C$  to write
$$
h (w) = - \log \det C + \delta_ {\mathcal {W}} (w) = \sum_ {i = 1} ^ {d} - \log C _ {i i} + \delta_ {(0, + \infty)} (C _ {i i}),
$$
where we used the notation  $\delta_A$  for the indicator function of a set  $A$ , which is equal to zero on  $A$  and  $+\infty$  outside. Since  $-\log(t) + \delta_{(0, +\infty)}(t)$  is a closed convex function, we see that  $h$  is a separable sum of closed convex functions, and thus is itself closed and convex.
4. Let  $d = 2$ , let  $I$  be the identity matrix, let  $R = \left( \begin{array}{cc}0 & 1\\ -1 & 0 \end{array} \right)$  be a rotation matrix, and define  $\psi :\mathbb{R}\to \mathbb{R}$  with  $\psi (t) = h(I + tR)$ . Observe that  $\psi$  is well defined because  $I + tR\succ 0$  for all  $t\in \mathbb{R}$ . On the one hand, if  $h$  was convex, then  $\psi$  would be convex too, as it is a composition of a convex function with the affine function  $t\mapsto I + tR$ . On the other hand, we can compute explicitly that  $\psi (t) = -\log (1 + t^2)$ , which is not convex.
Lemma 20 (Solutions of VI). Assume that  $\log p(\cdot, x)$  is  $M$ -smooth, and that  $V = \mathcal{M}^d$ ,  $\mathcal{S}^d$  or  $\mathcal{T}^d$ . Then  $\operatorname{argmin}(f) \subset \mathcal{W}_M := \{(m, C) \in \mathcal{W} : \sigma_{\min}(C) \geq \frac{1}{\sqrt{M}}\}$ .
Proof. If  $w^{*} = (m^{*}, C^{*}) \in \operatorname{argmin}(f)$ , then  $w^{*}$  is in the domain of  $h$ , which is  $\mathcal{W}$  by definition of  $h$ . So it remains to prove that  $\sigma_{\min}(C^{*}) \geq 1 / \sqrt{M}$ . When  $V = \mathcal{M}^d$ , this is proved in [13, Theorem 7]. Suppose now that  $V = \mathcal{S}^d$ , meaning that  $w^{*}$  is a minimizer of  $f$  over  $\mathcal{W} \subset \mathbb{R}^d \times \mathcal{S}^d$ . We see that  $w^{*}$  must also be a minimizer of  $f$  over  $\mathbb{R}^d \times \mathcal{M}^d$ , because if there was a better solution  $\hat{w} = (\hat{m}, \hat{C})$  with  $\hat{C} \in \mathcal{M}^d$ , we could consider  $(\hat{C}\hat{C}^\top)^{1/2} \in \mathcal{S}^d$  which would itself be a better solution than  $C^{*}$ , which is a contradiction. So we can apply [13, Theorem 7] to  $w^{*}$  to conclude that  $\sigma_{\min}(C^{*}) \geq 1 / \sqrt{M}$ . If  $V = \mathcal{T}^d$ , we can use the same argument, using this time the Cholesky decomposition of  $\hat{C}\hat{C}^\top$ .
Lemma 21. Let  $w = (m, C)$  with  $C$  invertible. Then  $\log q_w$  is  $\sigma_{\min}(C)^{-2}$ -smooth.
Proof. According to the definition of  $q_w$  (see (17)), the Hessian of  $\log q_w$  is  $\nabla_z^2 (\log q_w)(z) = -(CC^\top)^{-1}$ , and so  $\| \nabla_z^2 (\log q_w)(z) \| = \sigma_{\max}((CC^\top)^{-1}) = \sigma_{\min}(C)^{-2}$ .
7.3 A case study : linear models
Table 2: Table of models
Model	Smoothness constant	Strong convexity constant	Convex
Bayesian linear regression (Σ = I)	1 + σ2σmax(A)2	1 + σ2σmin(A)2	yes
Logistic regression (Σ = I)	1 + 1/4σmax(A)2	1	yes
Heirarchical logistic regression	exists	n/a	yes
Definition 22. A twice differentiable function  $f$  is  $M$ -smooth if  $-MI \preceq \nabla^2 f \preceq MI$ .
Definition 23. A twice differentiable function  $f$  is  $\mu$ -strongly convex  $\nabla^2 f \succeq \mu I$ .
Theorem 24. Take a generic i.i.d. linear model defined as  $p(z, x) = p(z) \prod_{n=1}^{N} p(x_n | z, a_n)$ , where  $p(z) = \mathcal{N}(z | 0, \Sigma)$  and
$$
p \left(x _ {n} \mid z, a _ {n}\right) = \exp \left(- \phi \left(z ^ {\top} a _ {n}, x _ {n}\right)\right)
$$
for some function  $\phi$ . Suppose that the second derivative of  $\phi$  with respect to its first (scalar) argument is bounded by  $\theta_{\min} \leq \phi'' \leq \theta_{\max}$ . Let
$$
\mu \stackrel {{d e f}} {{=}} \lambda_ {\min } \left(\Sigma^ {- 1} + \theta_ {\min } A A ^ {\top}\right)
$$
$$
\beta \stackrel {{d e f}} {{=}} \lambda_ {\max } \left(\Sigma^ {- 1} + \theta_ {\min } A A ^ {\top}\right),
$$
where  $A$  is a matrix with  $a_{n}$  in the  $n$ -th column. Then
1.  $-\log p(z,x)$  is  $M$  -smooth over  $z$  for  $M = \max (|\mu |,|\beta |)$
2. If  $\mu \geq 0$  then  $-\log p(z,x)$  is convex over  $z$
3. If  $\mu > 0$  then  $-\log p(z, x)$  is  $\mu$ -strongly convex over  $z$ .
Proof. It is easy to show that
$$
\nabla_ {z} ^ {2} \phi (z ^ {\top} a _ {n}, x _ {n}) = - \phi^ {\prime \prime} (z ^ {\top} a _ {n}, x _ {n}) a _ {n} a _ {n} ^ {\top}
$$
$$
- \nabla_ {z} ^ {2} \log p (z, x) = \Sigma^ {- 1} + \sum_ {n} \phi^ {\prime \prime} (z ^ {\top} a _ {n}, x _ {n}) a _ {n} a _ {n} ^ {\top}.
$$
Now, take some arbitrary vector  $v$ . We have that
$$
\begin{array}{l} v ^ {\top} \left(- \nabla_ {z} ^ {2} \log p (z, x)\right) v = v ^ {\top} \Sigma^ {- 1} v + \sum_ {n} b _ {n} \left\| v ^ {\top} a _ {n} \right\| ^ {2} \\ \geq v ^ {\top} \Sigma^ {- 1} v + \theta_ {\min } \sum_ {n} \left\| v ^ {\top} a _ {n} \right\| ^ {2} \\ = v ^ {\top} \left(\Sigma^ {- 1} + \theta_ {\min } \sum_ {n} a _ {n} a _ {n} ^ {\top}\right) v, \\ = v ^ {\top} \left(\Sigma^ {- 1} + \theta_ {\min } A A ^ {\top}\right) v, \\ \end{array}
$$
$$
\begin{array}{l} v ^ {\top} \left(- \nabla_ {z} ^ {2} \log p (z, x)\right) v = v ^ {\top} \Sigma^ {- 1} v + \sum_ {n} b _ {n} \left\| v ^ {\top} a _ {n} \right\| ^ {2} \\ \leq v ^ {\top} \Sigma^ {- 1} v + \theta_ {\max } \sum_ {n} \left\| v ^ {\top} a _ {n} \right\| ^ {2} \\ = v ^ {\top} \left(\Sigma^ {- 1} + \theta_ {\max } \sum_ {n} a _ {n} a _ {n} ^ {\top}\right) v, \\ = v ^ {\top} \left(\Sigma^ {- 1} + \theta_ {\max } A A ^ {\top}\right) v. \\ \end{array}
$$
Putting both the above results together, we have that
$$
\Sigma^ {- 1} + \theta_ {\min } A A ^ {\top} \preceq - \nabla_ {z} ^ {2} \log p (z, x) \preceq \Sigma^ {- 1} + \theta_ {\max } A A ^ {\top}.
$$
This implies that all eigenvalues of  $-\nabla_z^2\log p$  are in the interval  $[\mu ,\beta ]$ . All the claimed properties follow from this:
- For any eigenvalue  $\lambda$  of  $\nabla_z^2\log p(z,x)$ , we know that  $|\lambda_i(\nabla_z^2\log p(z,x))| \leq M = \max(|\mu|, |\beta|)$ . Thus,  $\log p(z,x)$  is  $M$ -smooth over  $z$ .
- If  $\mu \geq 0$ , this means all eigenvalues of  $-\nabla_z^2 \log p(z, x)$  are positive, which implies that  $-\log p(z, x)$  is convex.
- If  $\mu > 0$ , then all eigenvalues of  $-\nabla_z^2 \log(z, x)$  are bounded below by  $\alpha$ , which implies that  $-\log p(z, x)$  is  $\mu$ -strongly convex.
Corollary 25. Take a linear regression model with inputs  $a_{n} \in \mathbb{R}^{d}$  and outputs  $x_{n} \in \mathbb{R}$ . Let  $p(z) = \mathcal{N}(z|0,\Sigma)$  and  $p(x_{n}|z) = \mathcal{N}\left(x|z^{\top}a_{n},\sigma^{2}\right)$  for some fixed  $\sigma$ . Then  $-\log p(z,x)$  is  $\mu$ -strongly convex for  $\mu = \lambda_{\min}\left(\Sigma^{-1} + \frac{1}{4}AA^{\top}\right)$  and  $M$ -smooth for  $M = \lambda_{\max}(\Sigma^{-1} + \frac{1}{4}AA^{\top})$ .
Proof. In this case we have that
$$
\begin{array}{l} \phi \left(z ^ {\top} a _ {n}, x _ {n}\right) = - \log p \left(x _ {n} \mid z\right) \\ = \frac {1}{2 \sigma^ {2}} \left(x _ {n} - z ^ {\top} a _ {n}\right) ^ {2} - \frac {1}{2} \log (2 \pi \sigma^ {2}). \\ \end{array}
$$
Or, more abstractly,
$$
\phi (\alpha , \beta) = \frac {1}{2 \sigma^ {2}} (\alpha - \beta) ^ {2} - \frac {1}{2} \log (2 \pi \sigma^ {2}).
$$
$$
\phi^ {\prime} (\alpha , \beta) = \frac {1}{\sigma^ {2}} (\alpha - \beta)
$$
$$
\phi^ {\prime \prime} (\alpha , \beta) = \frac {1}{\sigma^ {2}}
$$
Thus we have that  $\theta_{\mathrm{min}} = \theta_{\mathrm{max}} = \frac{1}{\sigma^2}$ . It follows from Theorem 24 that  $-\log p(z,x)$  is  $\mu$ -strongly convex for  $\mu = \lambda_{\mathrm{min}}\left(\Sigma^{-1} + \frac{1}{\sigma^2} AA^\top\right)$  and  $M$ -smooth for  $M = \lambda_{\mathrm{max}}\left(\Sigma^{-1} + \frac{1}{\sigma^2} AA^\top\right)$ .
In the particular case where  $\Sigma = I$ , we get that  $\mu = \lambda_{\min}\left(I + \frac{1}{\sigma^2} AA^\top\right) = 1 + \frac{1}{\sigma^2}\sigma_{\min}(A)^2$  and  $M = \lambda_{\max}\left(1 + \frac{1}{\sigma^2} AA^\top\right) = 1 + \frac{1}{\sigma^2}\sigma_{\max}(A)^2$ .