SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {E} \ell \left(\mathcal {T} _ {\lambda} (\boldsymbol {u})\right) - \mathbb {E} \ell \left(\mathcal {T} _ {\lambda^ {\prime}} (\boldsymbol {u})\right) \geq \mathbb {E} \left\langle \nabla \ell \left(\mathcal {T} _ {\lambda^ {\prime}} (\boldsymbol {u})\right), \mathcal {T} _ {\lambda} (\boldsymbol {u}) - \mathcal {T} _ {\lambda^ {\prime}} (\boldsymbol {u}) \right\rangle + \frac {\mu}{2} \mathbb {E} \| \mathcal {T} _ {\lambda} (\boldsymbol {u}) - \mathcal {T} _ {\lambda^ {\prime}} (\boldsymbol {u}) \| _ {2} ^ {2} \\ \Leftrightarrow f (\boldsymbol {\lambda}) - f (\boldsymbol {\lambda} ^ {\prime}) \geq \mathbb {E} \left\langle \nabla \ell \left(\mathcal {T} _ {\boldsymbol {\lambda} ^ {\prime}} (\boldsymbol {u})\right), \mathcal {T} _ {\boldsymbol {\lambda}} (\boldsymbol {u}) - \mathcal {T} _ {\boldsymbol {\lambda} ^ {\prime}} (\boldsymbol {u}) \right\rangle + \frac {\mu}{2} \mathbb {E} \| \mathcal {T} _ {\boldsymbol {\lambda}} (\boldsymbol {u}) - \mathcal {T} _ {\boldsymbol {\lambda} ^ {\prime}} (\boldsymbol {u}) \| _ {2} ^ {2} \\ \Leftrightarrow f (\boldsymbol {\lambda}) - f (\boldsymbol {\lambda} ^ {\prime}) \geq \mathbb {E} \left\langle \nabla g (\boldsymbol {\lambda} ^ {\prime}; \boldsymbol {u}), \mathcal {T} _ {\boldsymbol {\lambda}} (\boldsymbol {u}) - \mathcal {T} _ {\boldsymbol {\lambda} ^ {\prime}} (\boldsymbol {u}) \right\rangle + \frac {\mu}{2} \mathbb {E} \| \mathcal {T} _ {\boldsymbol {\lambda}} (\boldsymbol {u}) - \mathcal {T} _ {\boldsymbol {\lambda} ^ {\prime}} (\boldsymbol {u}) \| _ {2} ^ {2} \\ \end{array}
$$
Thus, the energy is convex if and only if
$$
\mathbb {E} \left\langle g (\lambda ; \boldsymbol {u}), \mathcal {T} _ {\lambda} (\boldsymbol {u}) - \mathcal {T} _ {\lambda^ {\prime}} (\boldsymbol {u}) \right\rangle \geq \left\langle \nabla f (\lambda), \lambda - \lambda^ {\prime} \right\rangle
$$
holds. This is established by Lemma 10.
Proof of (iii) We now prove that, under the nonlinear parameterization, the energy cannot be strongly convex. When the energy is convex, it is also strongly convex if and only if
$$
\frac {\mu}{2} \mathbb {E} \| \mathcal {T} _ {\lambda} (\boldsymbol {u}) - \mathcal {T} _ {\lambda^ {\prime}} (\boldsymbol {u}) \| _ {2} ^ {2} \geq \frac {\mu}{2} \| \lambda - \lambda^ {\prime} \| _ {2} ^ {2}.
$$
From the proof of Domke (2020, Lemma 5), it follows that
$$
\mathbb {E} \left\| \mathcal {T} _ {\lambda} (\boldsymbol {u}) - \mathcal {T} _ {\lambda^ {\prime}} (\boldsymbol {u}) \right\| _ {2} ^ {2} = \left\| \boldsymbol {C} - \boldsymbol {C} ^ {\prime} \right\| _ {\mathrm {F}} ^ {2} + \left\| \boldsymbol {m} - \boldsymbol {m} ^ {\prime} \right\| _ {2} ^ {2}.
$$
Furthermore, under nonlinear parameterizations,
$$
\begin{array}{l} \left\| \boldsymbol {C} - \boldsymbol {C} ^ {\prime} \right\| _ {\mathrm {F}} ^ {2} + \left\| \boldsymbol {m} - \boldsymbol {m} ^ {\prime} \right\| _ {2} ^ {2} \\ = \left\| \left(\boldsymbol {D} _ {\phi} (\boldsymbol {s}) - \boldsymbol {D} _ {\phi} (\boldsymbol {s} ^ {\prime})\right) - (\boldsymbol {L} - \boldsymbol {L} ^ {\prime}) \right\| _ {\mathrm {F}} ^ {2} + \| \boldsymbol {m} - \boldsymbol {m} ^ {\prime} \| _ {2} ^ {2}, \\ \end{array}
$$
expanding the quadratic,
$$
= \left\| \boldsymbol {D} _ {\phi} (\boldsymbol {s}) - \boldsymbol {D} _ {\phi} (\boldsymbol {s} ^ {\prime}) \right\| _ {\mathrm {F}} ^ {2} + \| \boldsymbol {L} - \boldsymbol {L} ^ {\prime} \| _ {\mathrm {F}} ^ {2} - 2 \left\langle \boldsymbol {D} _ {\phi} (\boldsymbol {s}) - \boldsymbol {D} _ {\phi} (\boldsymbol {s} ^ {\prime}), \boldsymbol {L} - \boldsymbol {L} ^ {\prime} \right\rangle_ {\mathrm {F}} + \| \boldsymbol {m} - \boldsymbol {m} ^ {\prime} \| _ {2} ^ {2},
$$
and since  $D_{\phi}(s)$  and  $\pmb{L}$  reside in different sub-spaces, they are orthogonal. Thus,
$$
\begin{array}{l} = \left\| \boldsymbol {D} _ {\phi} (\boldsymbol {s}) - \boldsymbol {D} _ {\phi} (\boldsymbol {s} ^ {\prime}) \right\| _ {\mathrm {F}} ^ {2} + \| \boldsymbol {L} - \boldsymbol {L} ^ {\prime} \| _ {\mathrm {F}} ^ {2} + \| \boldsymbol {m} - \boldsymbol {m} ^ {\prime} \| _ {2} ^ {2} \\ = \| \phi (s) - \phi \left(s ^ {\prime}\right) \| _ {2} ^ {2} + \| L - L ^ {\prime} \| _ {\mathrm {F}} ^ {2} + \| \boldsymbol {m} - \boldsymbol {m} ^ {\prime} \| _ {2} ^ {2}. \tag {9} \\ \end{array}
$$
For the energy term to be strongly convex, Equation (9) must be bounded below by  $\| \lambda -\lambda^{\prime}\|_{2}^{2}$ . Evidently, this implies that a necessary and sufficient condition is that
$$
\left| \phi \left(s _ {i i}\right) - \phi \left(s _ {i i} ^ {\prime}\right) \right| \geq L \left| s _ {i i} - s _ {i i} ^ {\prime} \right|
$$
by some constant  $0 < L < \infty$ . Notice that the direction of the inequality is reversed from the Lipschitz condition. Unfortunately, there is no such continuous and differentiable function  $\phi : \mathbb{R} \to \mathbb{R}_+$ , as established by Lemma 11. Thus, for any diagonal conditioner  $\phi \in \mathrm{C}^1(\mathbb{R}, \mathbb{R}_+)$ , the energy cannot be strongly convex.
F.3 Convergence of Black-Box Variational Inference
F.3.1 Vanilla Black-Box Variational Inference
Theorem 5. Let the variational family satisfy Assumption 2, the likelihood satisfy Assumption 5, and the assumptions of Corollary 1 hold such that the ELBO,  $F$ , is  $L_{F}$ -smooth with  $L_{F} = L_{\ell} + L_{\phi} + L_{s}$ . Then, if the stepsize satisfy  $\gamma < 1 / L_{F}$ , the iterates of BBVI with SGD and the M-sample reparameterization gradient estimator satisfy
$$
\begin{array}{l} \min  _ {0 \leq t \leq T - 1} \mathbb {E} \| \nabla F (\lambda_ {t}) \| _ {2} ^ {2} \leq \gamma \frac {2 L _ {F} L _ {\ell} \kappa C (d , \varphi)}{M} \left(\| \bar {\mathbf {z}} _ {\text {j o i n t}} - \bar {\mathbf {z}} _ {\text {l i k e}} \| _ {2} ^ {2} + 2 \left(F ^ {*} - f _ {\mathrm {L}} ^ {*}\right)\right) \\ + \frac {2}{\gamma T} \Big (1 + \gamma^ {2} \frac {4 L _ {F} L _ {\ell} \kappa}{M} C (d, \varphi) \Big) ^ {T} (F (\lambda_ {0}) - F ^ {*}). \\ \end{array}
$$
$$
\begin{array}{l} \bar {z} _ {\text {j o i n t}} = \operatorname {p r o j} _ {\ell} (z) \quad \text {i s t h e p r o j e c t i o n o f z o n t o s e t o f m i n i m i z e r s o f} \ell \\ \bar {\boldsymbol {z}} _ {\text {l i k e}} = \operatorname {p r o j} _ {\ell_ {\text {l i k e}}} (\boldsymbol {z}) \quad \text {i s t h e p r o j e c t i o n o f} \boldsymbol {z} \text {o n t o s e t o f m i n i m i z e r s o f} \ell_ {\text {l i k e}}, \\ \kappa = ^ {L _ {\ell} / \mu} \quad \text {i s t h e c o n d i t i o n n u m b e r}, \\ F^{*} = \inf_{\lambda \in \Lambda}F\left(\lambda\right), \\ \ell_ {\text {l i k e}} ^ {*} = \inf  _ {\boldsymbol {\lambda} \in \mathbb {R} ^ {d}} \ell_ {\text {l i k e}} (\boldsymbol {z}), \\ C (d, \varphi) = d + k _ {\varphi} \quad \text {f o r t h e C h o l e s k y n o n l i n e a r}, \\ C (d, \varphi) = 2 k _ {\varphi} \sqrt {d} + 1 \quad f o r t h e m e a n - f i e l d n o n l i n e a r; \\ M \quad \text {i s t h e n u m b e r o f M o n t e C a r l o s p a m e l s}. \\ \end{array}
$$
Proof. Khaled & Richtárik (2023, Theorem 2) show that, if the objective function  $F$  is  $L_{F}$ -smooth and the stochastic gradients satisfy the  $ABC$  given as
$$
\mathbb {E} \left\| \widehat {\nabla F} (\lambda) \right\| _ {2} ^ {2} \leq A (F (\lambda) - F ^ {*}) + B \| \nabla F \| _ {2} ^ {2} + C
$$
for some  $0 < A, B, C < \infty$ , SGD guarantees
$$
\min _ {0 \leq t \leq T - 1} \mathbb {E} \| \nabla F (\boldsymbol {\lambda} _ {t}) \| _ {2} ^ {2} \leq L _ {F} C \gamma + \frac {2 (1 + L _ {F} \gamma^ {2} A) ^ {T}}{\gamma T} (F (\boldsymbol {\lambda} _ {0}) - F ^ {*}).
$$
Under the conditions of Corollary 1,  $F$  is  $L_{F}$ -smooth with  $L_{F} = L_{\ell} + L_{s} + L_{\phi}$ . Furthermore, under Assumption 5, Kim et al. (2023) show that the Monte Carlo gradient estimates satisfy
$$
\begin{array}{l} \mathbb {E} \| \widehat {\nabla F} (\boldsymbol {\lambda}) \| _ {2} ^ {2} \leq \frac {4 L _ {\ell} ^ {2} C (d , \varphi)}{\mu M} (F (\boldsymbol {\lambda}) - F ^ {*}) + B \| \nabla F \| _ {2} ^ {2} \\ + \frac {2 L _ {\ell} ^ {2} C (d , \varphi)}{\mu M} \| \bar {z} _ {\mathrm {j o i n t}} - \bar {z} _ {\mathrm {l i k e}} \| _ {2} ^ {2} + \frac {4 L _ {\ell} ^ {2} C (d , \varphi)}{\mu M} \left(F ^ {*} - \ell_ {\mathrm {l i k e}} ^ {*}\right), \\ \end{array}
$$
This means that the  $ABC$  condition is satisfied with constants
$$
A = \frac {4 L _ {\ell} ^ {2}}{\mu M} C (d, \varphi), \qquad B = 1, \qquad C = \frac {2 L _ {\ell} ^ {2}}{\mu M} C (d, \varphi) \| \bar {\boldsymbol {z}} _ {\mathrm {j o i n t}} - \bar {\boldsymbol {z}} _ {\mathrm {l i k e}} \| _ {2} ^ {2} + \frac {4 L _ {\ell} ^ {2}}{\mu M} C (d, \varphi) (F ^ {*} - \ell_ {\mathrm {l i k e}} ^ {*}).
$$
Plugging these constants in, we obtain
$$
\begin{array}{l} \min  _ {0 \leq t \leq T - 1} \mathbb {E} \| \nabla f (\lambda_ {t}) \| _ {2} ^ {2} \leq \gamma \frac {2 L _ {F} L _ {\ell} ^ {2} C (d , \varphi)}{\mu M} \left(\| \bar {z} _ {\text {j o i n t}} - \bar {z} _ {\text {l i k e}} \| _ {2} ^ {2} + 2 \left(F ^ {*} - \ell_ {\text {l i k e}} ^ {*}\right)\right) \\ + \frac {2}{\gamma T} \bigg (1 + \gamma^ {2} L _ {F} \frac {4 L _ {\ell} ^ {2}}{\mu M} C (d, \varphi) \bigg) ^ {T} (F (\lambda_ {0}) - F ^ {*}). \\ \end{array}
$$
Substituting the condition number yields the stated result.
Theorem 3. Let Assumption 2 hold, the likelihood satisfy Assumption 5, and the assumptions of Corollary 1 hold such that the ELBO  $F$  is  $L_{F}$ -smooth with  $L_{F} = L_{\ell} + L_{\phi} + L_{s}$ . Then, the iterates generated by BBVI through Equation (1) and the M-sample reparameterization gradient include an  $\epsilon$ -stationary point such that  $\min_{0 \leq t \leq T-1} \mathbb{E} \| \nabla F(\lambda_t) \|_2 \leq \epsilon$  for any  $\epsilon > 0$  if
$$
T \geq \mathcal {O} \left(\frac {\left(F \left(\lambda_ {0}\right) - F ^ {*}\right) ^ {2} L _ {F} L _ {\ell} ^ {2} C (d , k _ {\varphi})}{\mu M \epsilon^ {4}}\right)
$$
Proof. As a corollary to Theorem 5, Khaled & Richtárik (2023, Corollary 1) show that, for an  $L_{F}$ -smooth objective function  $F$ , a gradient estimator satisfying the ABC condition, an  $\varepsilon$ -stationary point can be encountered if
$$
\gamma = \min \left(\frac {1}{\sqrt {L _ {F} A T}}, \frac {1}{L _ {F} B}, \frac {\epsilon}{2 L _ {F} C}\right), \qquad T \geq \frac {1 2 (F (\lambda_ {0}) - F ^ {*}) L _ {F}}{\epsilon^ {2}} \max \left(B, \frac {1 2 (F (\lambda_ {0}) - F ^ {*}) A}{\epsilon^ {2}}, \frac {2 C}{\epsilon^ {2}}\right).
$$
Under Assumption 5, Kim et al. (2023) show that the Monte Carlo gradient estimates satisfy
$$
\begin{array}{l} \mathbb {E} \| \widehat {\nabla F} (\boldsymbol {\lambda}) \| _ {2} ^ {2} \leq \frac {4 L _ {\ell} ^ {2} C (d , \varphi)}{\mu M} (F (\boldsymbol {\lambda}) - F ^ {*}) + B \| \nabla F \| _ {2} ^ {2} \\ + \frac {2 L _ {\ell} ^ {2} C (d , \varphi)}{\mu M} \| \bar {\boldsymbol {z}} _ {\mathrm {j o i n t}} - \bar {\boldsymbol {z}} _ {\mathrm {l i k e}} \| _ {2} ^ {2} + \frac {4 L _ {\ell} ^ {2} C (d , \varphi)}{\mu M} \left(F ^ {*} - \ell_ {\mathrm {l i k e}} ^ {*}\right), \\ \end{array}
$$
$$
A = \frac {4 L _ {f} ^ {2}}{\mu M} C (d, \varphi), \qquad B = 1, \qquad C = \frac {2 L _ {f} ^ {2}}{\mu M} C (d, \varphi) \left(\| \bar {\boldsymbol {z}} _ {\mathrm {j o i n t}} - \bar {\boldsymbol {z}} _ {\mathrm {l i k e}} \| _ {2} ^ {2} + 2 (F ^ {*} - f _ {\mathrm {L}} ^ {*})\right).
$$
$$
\begin{array}{l} \bar {z} _ {\text {j o i n t}} = \operatorname {p r o j} _ {\ell} (z) \quad \text {i s} \\ \bar {z} _ {\text {l i k e}} = \operatorname {p r o j} _ {\ell_ {\text {l i k e}}} (\boldsymbol {z}) \quad \text {i s t h e p r o j e c t i o n o f} \boldsymbol {z} \text {o n t o s e t o f m i n i m i z e r s o f} \ell_ {\text {l i k e}}, \\ F ^ {*} = \inf  _ {\lambda \in \Lambda} F (\lambda), \\ \ell_ {\text {l i k e}} ^ {*} = \inf  _ {\lambda \in \mathbb {R} ^ {d}} \ell_ {\text {l i k e}} (\boldsymbol {z}), \\ C (d, \varphi) = d + k _ {\varphi} \quad \text {f o r t h e C h o l e s k y f a m i l y}, \\ C (d, \varphi) = 2 k _ {\varphi} \sqrt {d} + 1 \quad \text {f o r t h e m e a n - f i e l d f a m i l y}, \\ M \quad \text {i s t h e n u m b e r o f M o n t e C a r l o s p a m e l s}. \\ \end{array}
$$
$$
\begin{array}{l} T \geq \frac {1 2 \left(F \left(\lambda_ {0}\right) - F ^ {*}\right) L _ {F}}{\epsilon^ {2}} \max  \left(1, \frac {4 8 \left(F \left(\lambda_ {0}\right) - F ^ {*}\right) L _ {\ell} ^ {2} C (d , \varphi)}{\mu M \epsilon^ {2}}, \frac {8 L _ {\ell} ^ {2} C (d , \varphi) \left(\| \bar {z} _ {\text {j o i n t}} - \bar {z} _ {\text {l i k e}} \| _ {2} ^ {2} + \left(F ^ {*} - \ell_ {\text {l i k e}} ^ {*}\right)\right)}{\mu M \epsilon^ {2}}\right) \\ = \mathcal {O} \left(\frac {\left(F (\lambda_ {0}) - F ^ {*}\right) ^ {2} L _ {F} L _ {\ell} ^ {2} C (d)}{\mu M \epsilon^ {4}}\right), \\ \end{array}
$$
where we omitted the dependence on  $k_{\varphi}$  and the minimizers of  $\ell$  and  $\ell_{\mathrm{like}}$ .
F.3.2 Proximal Black-Box Variational Inference
$$
\begin{array}{l} \left. \mathbb {E} \| \nabla_ {\lambda} f (\lambda ; u) - \nabla_ {\lambda^ {\prime}} f (\lambda^ {\prime}; u) \| _ {2} ^ {2} = \mathbb {E} \| \nabla_ {\lambda} \ell \left(\mathcal {F} _ {\lambda} (u)\right) - \nabla_ {\lambda^ {\prime}} \ell \left(\mathcal {F} _ {\lambda^ {\prime}} (u)\right) \| _ {2} ^ {2} \right. \\ = \mathbb {E} \left\| \frac {\partial \mathcal {T} _ {\lambda} (\boldsymbol {u})}{\partial \lambda} g (\lambda , \boldsymbol {u}) - \frac {\partial \mathcal {T} _ {\lambda^ {\prime}} (\boldsymbol {u})}{\partial \lambda^ {\prime}} g (\lambda^ {\prime}, \boldsymbol {u}) \right\| _ {2} ^ {2}. \\ \end{array}
$$
For the linear parameterization, the Jacobian of  $\mathcal{T}_{\lambda}$  does not depend on  $\lambda$ . Therefore,
$$
= \mathbb {E} \left\| \frac {\partial \mathcal {T} _ {\lambda} (\boldsymbol {u})}{\partial \lambda} \left(g (\lambda , \boldsymbol {u}) - g (\lambda^ {\prime}, \boldsymbol {u})\right) \right\| _ {2} ^ {2}
$$
and Lemma 6 yields
$$
= J _ {\mathcal {T}} (\boldsymbol {u}) \mathbb {E} \| \mathrm {g} (\lambda , \boldsymbol {u}) - \mathrm {g} (\lambda^ {\prime}, \boldsymbol {u}) \| _ {2} ^ {2},
$$
$$
J _ {\mathcal {T}} (\boldsymbol {u}) = 1 + \left\| \boldsymbol {u} \right\| _ {2} ^ {2} \quad \text {f o r t h e C h o l e s k y f a m i l y a n d}
$$
$$
J _ {\mathcal {F}} (\boldsymbol {u}) = 1 + \left\| \boldsymbol {U} ^ {2} \right\| _ {\mathrm {F}} \quad \text {f o r t h e m e a n - f i e l d f a m i l y}.
$$
From now on, we apply the strategy of Domke (2019, Theorem 3) for resolving the randomness  $\pmb{u}$ . That is,
$$
\mathbb {E} J _ {\mathcal {T}} (\boldsymbol {u}) \left\| \mathrm {g} (\lambda , \boldsymbol {u}) - \mathrm {g} (\lambda^ {\prime}, \boldsymbol {u}) \right\| _ {2} ^ {2} = J _ {\mathcal {T}} (\boldsymbol {u}) \left\| \nabla \ell \left(\mathcal {T} _ {\lambda} (\boldsymbol {u})\right) - \nabla \ell \left(\mathcal {T} _ {\lambda^ {\prime}} (\boldsymbol {u})\right) \right\| _ {2} ^ {2}
$$
from the  $L_{\ell}$ -smoothness of  $f$
$$
\leq L _ {\ell} ^ {2} \mathbb {E} J _ {\mathcal {T}} (\boldsymbol {u}) \left\| \mathcal {T} _ {\lambda} (\boldsymbol {u}) - \mathcal {T} _ {\lambda^ {\prime}} (\boldsymbol {u}) \right\| _ {2} ^ {2},
$$
and applying Corollary 2,
$$
\leq L _ {\ell} ^ {2} C (d, \varphi) \| \lambda - \lambda^ {\prime} \| _ {2} ^ {2}
$$
The last step follows the approach of Kim et al. (2023), where we convert the quadratic bound into a bound involving the energy. Recall that the  $\mu$ -strongly convexity of  $\ell$  implies
$$
\frac {\mu}{2} \left\| \boldsymbol {z} ^ {\prime} - \boldsymbol {z} \right\| _ {2} ^ {2} \leq \ell (\boldsymbol {z}) - \ell \left(\boldsymbol {z} ^ {\prime}\right) - \left\langle \nabla \ell \left(\boldsymbol {z} ^ {\prime}\right), \boldsymbol {z} - \boldsymbol {z} ^ {\prime} \right\rangle . \tag {10}
$$
From Lemma 8, we have
$$
L _ {\ell} ^ {2} C (d, \varphi) \| \boldsymbol {\lambda} - \boldsymbol {\lambda} ^ {\prime} \| _ {2} ^ {2} = L _ {f} ^ {2} C (d, \varphi) \mathbb {E} \| \mathcal {T} _ {\boldsymbol {\lambda}} (\boldsymbol {u}) - \mathcal {T} _ {\boldsymbol {\lambda} ^ {\prime}} (\boldsymbol {u}) \| _ {2} ^ {2},
$$
and by  $\mu$ -strongly convexity,
$$
\leq \frac {2 L _ {\ell} ^ {2}}{\mu} C (d, \varphi) \mathbb {E} \big (\ell (\mathcal {T} _ {\lambda} (\boldsymbol {u})) - \ell (\mathcal {T} _ {\lambda^ {\prime}} (\boldsymbol {u})) - \langle \nabla \ell (\mathcal {T} _ {\lambda^ {\prime}} (\boldsymbol {u})), \mathcal {T} _ {\lambda} (\boldsymbol {u}) - \mathcal {T} _ {\lambda^ {\prime}} (\boldsymbol {u}) \rangle \big)
$$
$$
= \frac {2 L _ {\ell} ^ {2}}{\mu} C (d, \varphi) \mathbb {E} \left(f (\lambda ; \boldsymbol {u}) - f (\lambda^ {\prime}; \boldsymbol {u}) - \langle g (\lambda^ {\prime}; \boldsymbol {u}), \mathcal {T} _ {\lambda} (\boldsymbol {u}) - \mathcal {T} _ {\lambda^ {\prime}} (\boldsymbol {u}) \rangle\right)
$$
$$
= \frac {2 L _ {\ell} ^ {2}}{\mu} C (d, \varphi) \big (f (\lambda) - f (\lambda^ {\prime}) - \mathbb {E} \langle g (\lambda^ {\prime}; u), \mathcal {I} _ {\lambda} (u) - \mathcal {I} _ {\lambda^ {\prime}} (u) \rangle \big).
$$
Finally, by applying the equality in Lemma 10,
$$
= \frac {2 L _ {\ell} ^ {2}}{\mu} C (d, \varphi) \big (f (\lambda) - f (\lambda^ {\prime}) - \langle \nabla f (\lambda^ {\prime}), \lambda - \lambda^ {\prime} \rangle \big).
$$
Lemma 12 (Variance Transfer). Let  $\ell$  be  $L_{\ell}$ -smooth and  $\mu$ -strongly convex with the variational family satisfying Assumption 2 with the linear parameterization. Also, let  $\widehat{\nabla f}$  be an M-sample gradient estimator of the energy. Then,
$$
\operatorname {t r} \mathbb {V} \widehat {\nabla f} (\boldsymbol {\lambda}) \leq \frac {4 L _ {\ell} \kappa C (d , \varphi)}{M} \mathrm {B} _ {f} (\boldsymbol {\lambda}, \boldsymbol {\lambda} ^ {\prime}) + 2 \operatorname {t r} \mathbb {V} \widehat {\nabla f} (\boldsymbol {\lambda} ^ {\prime}),
$$
$\kappa = L_{\ell} / \mu$  is the condition number,  $\mathrm{B}_f$  is the Bregman divergence defined in Lemma 3,  $C(d,\varphi) = d + k_{\varphi}$  for the Cholesky family, and  $C(d,\varphi) = 2k_{\varphi}\sqrt{d} +1$  for the mean-field family.
Proof. First, the  $M$ -sample gradient estimator is defined as
$$
\widehat {\nabla f} (\lambda) = \frac {1}{M} \sum_ {m = 1} ^ {M} \nabla_ {\lambda} f (\lambda ; \boldsymbol {u} _ {m}),
$$
where  $\pmb{u}_m\sim \varphi$  . Since  $\pmb {u}_1,\dots ,\pmb {u}_m$  are independent and identically distributed, we have
$$
\operatorname {t r} \mathbb {V} \widehat {\nabla f} (\boldsymbol {\lambda}) = \frac {1}{M} \operatorname {t r} \mathbb {V} \nabla_ {\boldsymbol {\lambda}} f (\boldsymbol {\lambda}; \boldsymbol {u}).
$$
From here, given Lemma 3, the proof is identical with that of Garrigos & Gower (2023, Lemma 8.20), except for the constants.  $\square$
Theorem 6. Let  $\ell$  be  $L_{\ell}$ -smooth and  $\mu$ -strongly convex. Then, BBVI with proximal SGD in Equation (2), M-Monte Carlo samples, a variational family satisfying Assumption 2, the linear parameterization, and a fixed stepsize  $0 < \gamma \leq \frac{M}{2L_{\ell} \kappa C(d, \varphi)}$ , the iterates satisfy
$$
\mathbb {E} \| \boldsymbol {\lambda} _ {T} - \boldsymbol {\lambda} ^ {*} \| _ {2} ^ {2} \leq \left(1 - \gamma \mu\right) ^ {T} \| \boldsymbol {\lambda} _ {0} - \boldsymbol {\lambda} ^ {2} \| _ {2} ^ {2} + \frac {2 \gamma \sigma^ {2}}{\mu},
$$
where  $\kappa = L_{\ell} / \mu$  is the condition number,  $\sigma^2$  is defined in Lemma 4,  $\lambda^{*} = \arg \min_{\lambda \in \Lambda}F(\lambda)$ ,  $C(d,\varphi) = d + k_{\varphi}$  for the Cholesky family, and  $C(d,\varphi) = 2k_{\varphi}\sqrt{d} +1$  for the mean-field family.
Proof. Provided that
(A.6.1) the energy  $f$  is  $\mu$ -strongly convex,
(A.6.2) the energy  $f$  is  $L_{\ell}$ -smooth,
(A.6.3) the regularizer  $h$  is convex,
(A.6.4) the regularizer  $h$  is lower semi-continuous,
(A.6.5) the convex expected smoothness condition holds,
(A.6.6) the variance transfer condition holds, and
(A.6.7) the gradient variance  $\sigma^2$  at the optimum is finite such that  $\sigma^2 < \infty$ ,
the proof is identical to that of Garrigos & Gower (2023, Theorem 11.9), which is based on the results of Gorbunov et al. (2020, Corollary A.2).
In our setting,
(A.6.1) is established by Theorem 2,
(A.6.2) is established by Theorem 1,
(A.6.3) is trivially satisfied since  $h$  is the negative entropy,
(A.6.4) is trivially satisfied since  $h$  is continuous,
(A.6.5) is established in Lemma 3,
(A.6.6) is established in Lemma 12,
(A.6.7) is established in Lemma 4.
The only difference is that, we replace the constant  $L_{\mathrm{max}}$  in the proof of Garrigos & Gower to  $L_{\varrho}\kappa C(d,\varphi) / M$ . This stems from the different constants in the variance transfer condition.
Theorem 7. Let  $\ell$  be  $L_{\ell}$ -smooth and  $\mu$ -strongly convex. Then, for any  $\varepsilon > 0$ , BBVI with proximal SGD in Equation (2), M-Monte Carlo samples, a variational family satisfying Assumption 2, and the linear parameterization guarantees  $\mathbb{E}\|\lambda_T - \lambda^*\|_2^2 \leq \varepsilon$  if
$$
\gamma = \min \left(\frac {\epsilon}{2} \frac {\mu}{2 \sigma^ {2}}, \frac {M}{2 L _ {\ell} \kappa C (d , \varphi)}\right), \qquad T \geq \max \left(\frac {1}{\epsilon} \frac {4 \sigma^ {2}}{\mu^ {2}}, \frac {2 \kappa^ {2} C (d , \varphi)}{M}\right) \log \left(\frac {2 \| \lambda_ {0} - \lambda^ {*} \|}{\epsilon}\right),
$$
where  $\kappa = L_{\ell} / \mu$ ,  $\sigma^2$  is defined in Lemma 4,  $\lambda^{*} = \arg \min_{\lambda \in \Lambda} F(\lambda)$ ,  $C(d, \varphi) = d + k_{\varphi}$  for the Cholesky family, and  $C(d, \varphi) = 2k_{\varphi}\sqrt{d} + 1$  for the mean-field family.
Proof. This is a corollary of the fixed stepsize convergence guarantee in Theorem 6 as shown by Garrigos & Gower (2023, Corollary 11.10). They guarantee an  $\epsilon$ -accurate solution as long as
$$
\gamma = \min \left(\frac {\epsilon}{2} \frac {2}{2 \sigma_ {\mathrm {F}} ^ {*}}, \frac {1}{2 L _ {\max}}\right), \quad T \geq \max \left(\frac {1}{\epsilon} \frac {4 \sigma_ {\mathrm {F}} ^ {*}}{\mu^ {2}}, \frac {2 L _ {\max}}{\mu}\right) \log \left(\frac {2 \| \lambda_ {0} - \lambda^ {*} \|}{\epsilon}\right).
$$
In our notation,  $\sigma_{\mathrm{F}}^{*} = \sigma^{2}$  and  $L_{\max} = L_{\ell}\kappa C(d,\varphi) / M$
Theorem 8. Let  $\ell$  be  $L_{\ell}$ -smooth and  $\mu$ -strongly convex. Then, BBVI with proximal SGD in Equation (2), the M-sample reparameterization gradient estimator, a variational family satisfying Assumption 2, the linear parameterization,  $T \geq 4T_{\kappa}$ , and a stepsize schedule of
$$
\gamma_ {t} = \left\{ \begin{array}{l l} \frac {M}{2 L _ {\ell} \kappa C (d , \varphi)} & f o r t \leq 4 T _ {\kappa} \\ \frac {2 t + 1}{(t + 1) ^ {2} \mu} & f o r t > 4 T _ {\kappa}, \end{array} \right.
$$
where  $T_{\kappa} = \lceil \kappa^{2}C(d,\varphi)M^{-1}\rceil$ $\kappa = L_{\ell} / \mu$  is the condition number,  $C(d,\varphi) = d + k_{\varphi}$  for the Cholesky family, and  $C(d,\varphi) = 2k_{\varphi}\sqrt{d} +1$  for the mean-field family, then the iterates satisfy
$$
\mathbb {E} \| \boldsymbol {\lambda} _ {T} - \boldsymbol {\lambda} ^ {*} \| _ {2} ^ {2} \leq \frac {1 6 T _ {\kappa} ^ {2} \| \boldsymbol {\lambda} _ {0} - \boldsymbol {\lambda} ^ {*} \| _ {2} ^ {2}}{\mathrm {e} ^ {2} T ^ {2}} + \frac {8 \sigma^ {2}}{\mu^ {2} T}
$$
where  $\sigma^2$  is defined in Lemma 4, e is Euler's constant, and  $\lambda^{*} = \arg \min_{\lambda \in \Lambda} F(\lambda)$ .
Proof. Under our assumptions, Theorem 6 holds, of which the proof is essentially obtaining the recursion
$$
\mathbb {E} \| \boldsymbol {\lambda} _ {t + 1} - \boldsymbol {\lambda} ^ {*} \| _ {2} ^ {2} = (1 - \gamma_ {t} \mu) \mathbb {E} \| \boldsymbol {\lambda} _ {t} - \boldsymbol {\lambda} ^ {*} \| _ {2} ^ {2} + 2 \gamma_ {t} ^ {2} \sigma^ {2}.
$$
Instead of a fixed stepsize, we can apply the decreasing stepsize rule in the proof statement, then which the proof becomes identical to that of Gower et al. (2019, Theorem 3.2). We only need to replace  $\mathcal{L}$  with  $L_{\mathrm{max}}$  in the proof of Garrigos & Gower (2023, Theorem 11.9). This, in our notation, is  $L_{\mathrm{max}} = L_{\ell} \kappa C(d, \varphi) / M$ .
Proof. The computational complexity follows from the smallest number of iterations  $T$  such that