SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
2 \gamma_ {t} \mathbb {E} \left[ f (w ^ {t}) - f (w ^ {*}) \right] \leq (1 + a \gamma_ {t} ^ {2}) \mathbb {E} \left[ \left\| w ^ {t} - w ^ {*} \right\| ^ {2} \right] - \mathbb {E} \left[ \left\| w ^ {t + 1} - w ^ {*} \right\| ^ {2} \right] + b \gamma_ {t} ^ {2}.
$$
From now on we use a constant stepsize  $\gamma_t \equiv \gamma$ . Let  $C = (1 + a\gamma^2)$ . Next we would like to setup a telescopic sum. To make this happen, we multiply through by  $\alpha_{t+1}$  and impose that  $\alpha_{t+1}C = \alpha_t$ . This holds with  $\alpha_t \stackrel{\mathrm{def}}{=} \frac{1}{C^t}$ . Multiplying through by  $\alpha_{t+1}$  and summing from  $t = 0, \ldots, T-1$  we have that
$$
\begin{array}{l} 2 \gamma \sum_ {t = 0} ^ {T - 1} \alpha_ {t + 1} \mathbb {E} \left[ f (w ^ {t}) - f (w ^ {*}) \right] \leq \sum_ {t = 0} ^ {T - 1} \left(\alpha_ {t + 1} C \mathbb {E} \left[ \| w ^ {t} - w ^ {*} \| ^ {2} \right] - \alpha_ {t + 1} \mathbb {E} \left[ \| w ^ {t + 1} - w ^ {*} \| ^ {2} \right]\right) + \sum_ {t = 0} ^ {T - 1} \alpha_ {t + 1} b \gamma^ {2} \\ = \sum_ {t = 0} ^ {T - 1} \left(\alpha_ {t} \mathbb {E} \left[ \left\| w ^ {t} - w ^ {*} \right\| ^ {2} \right] - \alpha_ {t + 1} \mathbb {E} \left[ \left\| w ^ {t + 1} - w ^ {*} \right\| ^ {2} \right]\right) + \sum_ {t = 0} ^ {T - 1} \alpha_ {t + 1} b \gamma^ {2} \\ = \alpha_ {0} \left\| w ^ {0} - w ^ {*} \right\| ^ {2} - \alpha_ {T} \mathbb {E} \left[ \left\| w ^ {T} - w ^ {*} \right\| ^ {2} \right] + \sum_ {t = 0} ^ {T - 1} \alpha_ {t + 1} b \gamma^ {2}, \\ \end{array}
$$
Using that  $\alpha_0 = 1$ , dropping the negative term  $-\alpha_T \mathbb{E}\left[\| w^T - w^*\|^2\right]$ , dividing through by  $2\gamma \sum_{k=0}^{T-1} \alpha_{k+1}$  and using Jensen's inequality we have that
$$
\begin{array}{l} \mathbb {E} \left[ f (\bar {w} ^ {t}) - f (w ^ {*}) \right] \leq \frac {\sum_ {t = 0} ^ {T - 1} \alpha_ {t + 1} \mathbb {E} \left[ f (w ^ {t}) - f (w ^ {*}) \right]}{\sum_ {k = 0} ^ {T - 1} \alpha_ {k + 1}} \\ \leq \frac {\left\| w ^ {0} - w ^ {*} \right\| ^ {2}}{2 \gamma \sum_ {k = 0} ^ {T - 1} \alpha_ {k + 1}} + \frac {b \gamma}{2}. \\ \end{array}
$$
Finally using that
$$
\sum_ {k = 0} ^ {T - 1} \alpha_ {k + 1} = \sum_ {k = 0} ^ {T - 1} C ^ {- (k + 1)} = \frac {1}{C} \sum_ {k = 0} ^ {T - 1} C ^ {- k} = \frac {1}{C} \frac {1 - \frac {1}{C ^ {T}}}{1 - \frac {1}{C}} = \frac {1 - \frac {1}{C ^ {T}}}{C - 1} = \frac {1 - \frac {1}{(1 + a \gamma^ {2}) ^ {T}}}{a \gamma^ {2}},
$$
givesthat
$$
\begin{array}{l} \mathbb {E} \left[ f (\bar {w} ^ {t}) - f (w ^ {*}) \right] \leq \frac {a \gamma^ {2}}{1 - \frac {1}{(1 + a \gamma^ {2}) ^ {T}}} \frac {\left\| w ^ {0} - w ^ {*} \right\| ^ {2}}{2 \gamma} + \frac {b \gamma}{2} \\ = \frac {a \gamma}{1 - \frac {1}{(1 + a \gamma^ {2}) ^ {T}}} \frac {\left\| w ^ {0} - w ^ {*} \right\| ^ {2}}{2} + \frac {b \gamma}{2} \\ = \frac {\gamma}{2} \left(\frac {a \left\| w ^ {0} - w ^ {*} \right\| ^ {2}}{1 - \frac {1}{(1 + a \gamma^ {2}) ^ {T}}} + b\right) \\ = \frac {\gamma}{2} \left(\frac {a \left\| w ^ {0} - w ^ {*} \right\| ^ {2}}{1 - \theta^ {T}} + b\right) \\ \end{array}
$$
Finally we can apply Lemma 32 with where  $B = a$ ,  $A = \sqrt{2 / B} = \sqrt{2} /\sqrt{a}$ ,  $\theta = \frac{1}{1 + a\gamma^2}$  and  $\gamma = A / \sqrt{T} = \sqrt{2} /(\sqrt{a}\sqrt{T})$  which gives
$$
\begin{array}{l} \mathbb {E} \left[ f (\bar {w} ^ {T}) \right] - \inf  f \leq \frac {1}{2} \frac {A}{\sqrt {T}} \left(2 a \| w ^ {0} - w ^ {*} \| ^ {2} + b\right) \\ = \frac {1}{\sqrt {2}} \frac {1}{\sqrt {a} \sqrt {T}} \left(2 a \| w ^ {0} - w ^ {*} \| ^ {2} + b\right) = \Omega \left(\frac {1}{\sqrt {T}}\right). \\ \end{array}
$$
10 Solving VI with stochastic optimization
10.1 Convergence of the algorithms
Corollary 12 (Prox-SGD for VI). Consider the VI problem where  $q_{w}$  is a multivariate Gaussian distribution (Eq. 2) with parameters  $w = (m, C) \in \mathbb{R}^d \times \mathcal{T}^d$ , and assume that this problem admits a solution  $w^{*}$ . Suppose that  $\log p(\cdot, x)$  is  $M$ -smooth and concave (resp.  $\mu$ -strongly concave). Generate a sequence  $w^{t}$  by using the Prox-SGD algorithm (Def. 6) applied to  $l$  and  $h$  (Eq. 16), using  $\mathbf{g}_{\text{energy}}(5)$  as an estimator of  $\nabla l$ . Let the stepsizes  $\gamma_{t}$  be constant and equal to  $1 / (\sqrt{a_{\text{energy}} T})$  (resp. be decaying as in Theorem 7 with  $a_{\text{energy}} = 2(d + 3)M^2$ ). Then, for a certain average  $\bar{w}^T$  of the iterates, we have for  $T \geq 2$  that
Proof. Our assumption on the log target implies that  $l$  is smooth and convex (resp.  $\mu$ -strongly convex) according to Lemmas 18.1 and 19.1. Furthermore, since  $\log p(\cdot, x)$  is smooth, we know from Theorem 30 that the gradient estimator  $\mathbf{g}_{\text{energy}}$  is quadratically bounded at every  $w^t$  with respect to constant parameters  $(a, b, w^*)$ , with  $a = 2(d + 3)M^2$ . Then, because we work with triangular scale parameters, we know from Lemma 19.3 that  $h$  is closed convex. Finally  $T \geq \max \{M^2 / a, 2\} = 2$  since
$$
\frac {M ^ {2}}{a} = \frac {M ^ {2}}{2 (d + 3) M ^ {2}} = \frac {1}{2 (d + 3)} \leq 1.
$$
We have now verified the hypotheses of Theorem 8 (resp. Theorem 7), which gives us the desired bounds.
Proof. Our assumption on the log target implies that  $l$  is smooth and convex (resp.  $\mu$ -strongly convex) according to Lemmas 18.1 and 19.1. Regarding the entropy, since  $\log p(\cdot, x)$  is  $M$ -smooth, we know from Lemma 18.3 that  $h$  is  $M$ -smooth on  $\mathcal{W}_M$ . We also know that  $h$  is closed convex, since we consider symmetric scale parameters (see Lemma 19.2). Furthermore, the smoothness of  $\log p(\cdot, x)$  implies that the gradient estimator  $\mathbf{g}_{\mathrm{ent}}$  is quadratically bounded at every  $w^t \in \mathcal{W}_M$ , with respect to constant parameters  $(a, b, w^*)$ . This is stated in Theorem 30, where we can see that for  $\mathbf{g}_{\mathrm{ent}}$  we can take  $a_{\mathrm{ent}} = 4(d + 3)M^2$ , and for  $\mathbf{g}_{\mathrm{STL}}$  we can take  $a_{\mathrm{STL}} = 2(d + 3)(2K^2 + (2M)^2)$ . Since we know from Theorem 4 that  $K \leq 2M$ , we can consider that  $a_{\mathrm{STL}} \leq 24(d + 3)M^2$ . Finally, we know from Lemma 20 that the minimizers of  $f$  belong to  $\mathcal{W}_M$ , meaning that  $\operatorname{argmin}(f) = \operatorname{argmin}_{\mathcal{W}_M}(f)$ . We have now verified the hypotheses of Theorem 11 (resp. Theorem 10), which gives us the desired bounds.
Proof. We use the same arguments as in the proof of Corollary 13, but this time we use a constant stepsize, which gives us the bound (13) from Theorem 10, where  $b = 4(d + 3)K^2\| w^* - \hat{w} \|^2$  according to Theorem 4. All we need to conclude is to verify that  $b = 0$ . This comes from the assumption that the target is Gaussian, from which we deduce that  $K = 0$  (see again Theorem 4).
10.2 Computations for the algorithms
Lemma 35 (Prox of the entropy for triangular matrices). Assume that  $V = \mathcal{T}^d$ , and let  $w = (m, C) \in \mathbb{R}^d \times \mathcal{T}^d$ . Then  $\mathrm{prox}_{\gamma h}(w) = (m, \hat{C})$ , where
$$
\hat {C} _ {i j} = \left\{ \begin{array}{l l} \frac {1}{2} (C _ {i i} + \sqrt {C _ {i i} ^ {2} + 4 \gamma}) & i f i = j, \\ C _ {i j} & i f i \neq j. \end{array} \right.
$$
Proof. This is essentially proved in [13, Theorem 13]. We provide a proof here for the sake of completeness, also because our entropy here enforces positive definiteness. So we need to compute
$$
\operatorname *{argmin}_{z\in \mathbb{R}^{d},X\in \mathcal{T}^{d},X\succ 0} - \log \det X + \frac{1}{2\gamma}\| z - m\|^{2} + \frac{1}{2\gamma}\| X - C\|_{F}^{2}.
$$
Because the objective here is separated in  $z$  and  $X$ , we can optimize those variables separately. It is clear that the optimal  $z$  will be  $z = m$ . For the matrix component, it remains to solve
$$
\operatorname *{argmin}_{X\in \mathcal{T}^{d},X\succ 0} - \log \det X + \frac{1}{2\gamma}\| X - C\|_{F}^{2}.
$$
Now remember that for a triangular matrix, being positive definite is equivalent to have positive diagonal elements (Lemma 19.3). Because  $X$  is a lower triangular matrix, we have  $-\log \det X = \sum_{i=1}^{d} -\log X_{ii}$  and  $X_{ij} = 0$  when  $i < j$ . So our problem becomes
$$
\operatorname * {a r g m i n} _ {X \in \mathcal {T} ^ {d}, X _ {i i} > 0} \sum_ {i = 1} ^ {d} (- \log X _ {i i}) + \sum_ {i \geq j} \frac {1}{2 \gamma} (X _ {i j} - C _ {i j}) ^ {2}.
$$
This is a gain a separated optimization problem with respect to the variables  $X_{ij}$ . So we deduce that when  $i \geq j$  we must take  $\hat{C}_{ij} = C_{ij}$ , and when  $i = j$  we need to take
$$
\hat {C} _ {i i} = \underset {x > 0} {\operatorname {a r g m i n}} - \log x + \frac {1}{2 \gamma} (x - C _ {i i}) ^ {2}. \tag {41}
$$
This is the proximal operator of  $-\log$ , which can be classically computed [55, Example 6.9], and gives the desired result.
Lemma 36 (Projection onto  $\mathcal{W}_M$  for symmetric matrices). Assume that  $V = S^d$ , and let  $w = (m, C) \in \mathbb{R}^d \times \mathcal{S}^d$ . Let  $C = UDU^\top$  be the eigenvalue decomposition of  $C$ , with  $D$  diagonal and  $U$  orthogonal. Then  $\mathrm{proj}_{\mathcal{W}_M}(w) = (m, U\tilde{D}U^\top)$ , where  $\tilde{D}$  is the diagonal matrix defined by
$$
\tilde {D} _ {i i} = \max  \{D _ {i i}, \frac {1}{\sqrt {M}} \}.
$$
Proof. Consider  $g = \delta_{\mathcal{W}_M}$  to be the indicator function of  $\mathcal{W}_M$ , which is equal to 0 on  $\mathcal{W}_M$  and  $+\infty$  elsewhere. Then  $\operatorname{proj}_{\mathcal{W}_M} = \operatorname{prox}_g$ , and we can compute it with [55, Theorem 7.18]. For this, we need to write  $g(C) = F(\lambda(C))$ , where  $\lambda(C)$  is the set of eigenvalues of  $C$ , and  $F$  is the indicator function of  $K := \{z \in \mathbb{R}^d : z_i \geq 1/\sqrt{M}\}$ . In that way, we see that  $g$  is a symmetric spectral function in the sense of [55, Definition 7.12], and the conclusion follows.
Lemma 37 (Adjoint of the distribution transformation). Let  $u \in \mathbb{R}^d$  be fixed. Let  $T: \mathbb{R}^d \times V \to \mathbb{R}^d$  be such that  $T(m, C) = Cu + m$ . Then  $T$  is a linear operator such that its adjoint  $T^*: \mathbb{R}^d \to \mathbb{R}^d \times V$  verifies  $T^*(z) = (z, \operatorname{proj}_V(zu^\top))$ .
Proof. The fact that  $T$  is linear is immediate. Let us consider now the extension  $\bar{T}:\mathbb{R}^d\times \mathcal{M}^d\to \mathbb{R}^d$  such that  $\bar{T}(m,C) = Cu + m$ . If we define the canonical injection  $\iota :\mathbb{R}^{d}\times V\to \mathbb{R}^{d}\times \mathcal{M}^{d}$ , we clearly have  $T = \bar{T}\circ \iota$ . This means that  $T^{*} = \iota^{*}\circ \bar{T}^{*} = \mathrm{proj}_{\mathbb{R}^{d}\times V}\circ \bar{T}^{*}$ . It remains to prove that  $\bar{T}^{*}(z) = (z,zu^{\top})$ . For this, it suffices to use the properties of the trace to write that
$$
\begin{array}{l} \langle (z, z u ^ {\top}), (m, C) \rangle = \langle z, m \rangle + \langle z u ^ {\top}, C \rangle = \langle z, m \rangle + \operatorname {t r} (u z ^ {\top} C) \\ = \langle z, m \rangle + \operatorname {t r} \left(z ^ {\top} C u\right) = \langle z, m \rangle + \langle z, C u \rangle \\ = \langle z, T (m, C) \rangle . \\ \end{array}
$$
10.3 Explicit implementation of the algorithms
Lemma 38 (Explicit computation of the estimators). Let  $u \in \mathbb{R}^d$ , and let  $w = (m, C) \in \mathbb{R}^d \times V$  where  $C \succ 0$ . Define
$$
\pi := - \nabla_ {z} \log p (C u + m, x).
$$
1.  $\mathbf{g}_{\mathrm{energy}}(u) = (\pi, \operatorname{proj}_V(\pi u^\top))$
2.  $\mathbf{g}_{\mathrm{ent}}(u) = (\pi, \operatorname{proj}_V(\pi u^\top - C^{-\top}))$
3.  $\mathbf{g}_{\mathrm{STL}}(u) = (\pi - C^{-\top} u, \operatorname{proj}_V(\pi u^\top - C^{-\top} uu^\top))$ .
Proof. Let  $w = (m, C)$  and let  $T$  be the linear operator defined by  $T(m, C) = Cu + m$ , as in Lemma 37. As a shorthand we will note  $p_x \coloneqq p(\cdot, x)$ .
1. From its definition in (5), the energy estimator is equal to
$$
\mathbf {g} _ {\text {e n e r g y}} (u) = - \nabla_ {w} (\log \circ p _ {x} \circ T) (w).
$$
Applying the chain rules, we obtain
$$
\mathbf {g} _ {\text {e n e r g y}} (u) = - T ^ {*} \nabla_ {z} (\log \circ p _ {x}) (T w) = T ^ {*} \pi .
$$
The conclusion follows from the expression of  $T^{*}$  obtained in Lemma 37.
2. From its definition in (7), the entropy estimator  $\mathbf{g}_{\mathrm{ent}}(u)$  is equal to  $\mathbf{g}_{\mathrm{energy}}(u) + \nabla h(w)$ , where (we use the fact that we assumed that  $w \in \mathcal{W}$ ) we know from Lemma 18.2 that  $\nabla h(w) = -(0, \operatorname{proj}_V(C^{-\top}))$ . The conclusion follows from the previous item and the linearity of  $\operatorname{proj}_V$ .
3. From its definition in (9), the STL estimator is equal to
$$
\mathbf {g} _ {\mathrm {S T L}} (u) = \mathbf {g} _ {\text {e n e r g y}} (u) + \nabla_ {w} (\phi \circ T) (w),
$$
where  $\phi (z) = \log q_v(z)$ , with  $v$  being a copy of  $w$ . From the definition of a Gaussian density (17) we have that
$$
\nabla_ {z} \phi (z) = - \left(C ^ {- \top} C ^ {- 1}\right) (z - m),
$$
so we deduce that
$$
\nabla_ {w} (\phi \circ T) (w) = T ^ {*} \nabla_ {z} \phi (T w) = - T ^ {*} \left(C ^ {- \top} C ^ {- 1}\right) (T w - m) = - T ^ {*} C ^ {- \top} u.
$$
The conclusion follows after combining the first item with Lemma 37.
Lemma 39 (Pseudocode of the algorithms).
1. Algorithm 1 is equivalent to the Prox-SGD algorithm (Def. 6) applied to  $l$  and  $h$  (Eq. 16), using genergy (5) as an estimator of  $\nabla l$  and using lower triangular covariance factors.
2. Algorithm 2 is equivalent to the Proj-SGD algorithm (Def. 9) applied to the function  $f = l + h$  (Eq. 16) and the constraint  $\mathcal{W}_M$  (Eq. 3), and using  $\mathbf{g}_{\mathrm{ent}}$  (7) as an estimator of  $\nabla f$ .
3. Algorithm 3 is equivalent to the Proj-SGD algorithm (Def. 9) applied to the function  $f = l + h$  (Eq. 16) and the constraint  $\mathcal{W}_M$  (Eq. 3), and using  $\mathbf{g}_{\mathrm{STL}}$  (9) as an estimator of  $\nabla f$ .
Proof. In this proof, we fix an iteration  $t \in \mathbb{N}$ , consider  $u_t \sim \mathcal{N}(0, I)$  and we note  $w^t = (m_t, C_t)$ .
1. From (Def. 6) we know that  $w^{t + 1} = \mathrm{prox}_{\gamma_{t}h}(w^{t} - \gamma_{t}\mathsf{g}_{\mathrm{energy}}(u_{t}))$ . According to Lemma 38, we have
$$
w ^ {t} - \gamma_ {t} \mathrm {g} _ {\text {e n e r g y}} (u _ {t}) = \left(m _ {t}, C _ {t}\right) - \gamma_ {t} \left(\pi_ {t}, \operatorname {p r o j} _ {\mathcal {T} ^ {d}} \left(\pi_ {t} u _ {t} ^ {\top}\right)\right),
$$
$$
\pi_ {t} = - \nabla_ {z} \log p \left(C _ {t} u _ {t} + m _ {t}, x\right).
$$
Moreover, we know from Lemma 35 what  $\mathrm{prox}_{\gamma_{th}}$  is:
$$
\operatorname {p r o x} _ {\gamma_ {t} h} (m, C) = (m, \mathcal {R} _ {\gamma_ {t}} (C)), \text {w h e r e} \mathcal {R} _ {\gamma_ {t}} (C) _ {i j} = \left\{ \begin{array}{l l} \frac {1}{2} (C _ {i i} + \sqrt {C _ {i i} ^ {2} + 4 \gamma_ {t}}) & \text {i f} i = j, \\ C _ {i j} & \text {i f} i \neq j. \end{array} \right.
$$
First, we see that with respect to the  $m$  variable the proximal operator is the identity, meaning that we indeed have
$$
m _ {t + 1} = m _ {t} - \gamma_ {t} \pi_ {t}.
$$
Second, we see that with respect to the  $C$  variable we have
$$
C _ {t + 1} = \mathcal {R} _ {\gamma_ {t}} \left(C _ {t} - \gamma_ {t} \operatorname {p r o j} _ {\mathcal {T} ^ {d}} \left(\pi_ {t} u _ {t} ^ {\top}\right)\right).
$$
It is immediate to see by induction that since we initialize with  $C_0 \in \mathcal{T}^d$ , then  $C_t \in \mathcal{T}^d$  as well. Therefore we can use the linearity of  $\mathrm{proj}_{\mathcal{T}^d}$  to write that
$$
C _ {t + 1} = \mathcal {R} _ {\gamma_ {t}} \left(\operatorname {p r o j} _ {\mathcal {T} ^ {d}} \left(C _ {t} - \gamma_ {t} \pi_ {t} u _ {t} ^ {\top}\right)\right).
$$
The conclusion follows after observing that
$$
(\mathcal {R} _ {\gamma_ {t}} \circ \mathrm {p r o j} _ {\mathcal {T} ^ {d}}) (C) (i, j) = \left\{ \begin{array}{l l} C (i, j) & \text {i f i > j}, \\ \frac {1}{2} \left(C (i, i) + \sqrt {C (i , i) ^ {2} + 4 \gamma_ {t}}\right) & \text {i f i = j}, \\ 0 & \text {i f i <   j}. \end{array} \right.
$$
2. From (Def. 9) we know that  $w^{t + 1} = \mathrm{proj}_{\mathcal{W}_M}(w^t - \gamma_t \mathsf{g}_{\mathrm{ent}}(u_t))$ . According to Lemma 38, we have
$$
w ^ {t} - \gamma_ {t} \mathbf {g} _ {\text {e n t}} (u _ {t}) = \left(m _ {t}, C _ {t}\right) - \gamma_ {t} \left(\pi_ {t}, \operatorname {p r o j} _ {\mathcal {S} ^ {d}} \left(\pi_ {t} u _ {t} ^ {\top} - C _ {t} ^ {- \top}\right)\right).
$$
Moreover, we know from Lemma 36 what  $\mathrm{proj}_{\mathcal{W}_M}$  is:
$$
\operatorname {p r o j} _ {\mathcal {W} _ {M}} (m, C) = (m, \tilde {\mathcal {R}} (C)),
$$
where  $\tilde{\mathcal{R}}(C) = U\tilde{D}U^{\top}$ , with  $U D U^{\top}$  being a SVD decomposition of  $C$ , and  $\tilde{D}(i, i) = \max \{D(i, i); M^{-1/2}\}$ . First, we see that with respect to the  $m$  variable the projection is the identity, meaning that we indeed have
$$
C _ {t + 1} = \tilde {\mathcal {R}} \left(C _ {t} - \gamma_ {t} \operatorname {p r o j} _ {\mathcal {T} ^ {d}} \left(\pi_ {t} u _ {t} ^ {\top} - C _ {t} ^ {- \top}\right)\right).
$$
It is immediate to see by induction that since we initialize with  $C_0 \in S^d$ , then  $C_t \in S^d$  as well. Therefore we can use the linearity of  $\mathrm{proj}_{S^d}$  to write that
$$
C _ {t + 1} = \tilde {\mathcal {R}} \left(\operatorname {p r o j} _ {\mathcal {S} ^ {d}} \left(C _ {t} - \gamma_ {t} \left(\pi_ {t} u _ {t} ^ {\top} - C _ {t} ^ {- \top}\right)\right)\right).
$$
The conclusion follows after setting  $\hat{C}_{t + 1} = C_t - \gamma_t(\pi_t u_t^\top - C_t^{-\top})$ , and using the fact that  $\mathrm{proj}_{\mathcal{S}^d}(C) = (C + C^\top) / 2$ .
3. Use the same reasoning as in the previous item, with the appropriate computation for the STL estimator from Lemma 38.
Algorithm 1 Prox-SGD with energy estimator and triangular factors
Let  $m_0\in \mathbb{R}^d$ $C_0\in \mathcal{T}^d$  such that  $C_0\succ 0,(\gamma_t)_{t\in \mathbb{N}}\subset (0, + \infty)$    
for  $t\in \mathbb{N}$  do Sample  $u_{t}\sim \mathcal{N}(0,I)$  Compute  $\pi_t = -\nabla_z\log p(C_t u_t + m_t,x)$  Set  $m_{t + 1} = m_t - \gamma_t\pi_t$  Set  $\hat{C}_{t + 1} = C_t - \gamma_t\pi_tu_t^\top$  Set  $C_{t + 1}$  via  $C_{t + 1}(i,j) = \left\{ \begin{array}{ll}\hat{C}_{t + 1}(i,j) & \text{if} i > j\\ \frac{1}{2}\left(\hat{C}_{t + 1}(i,i) + \sqrt{\hat{C}_{t + 1}(i,i)^2 + 4\gamma_t}\right) & \text{if} i = j\\ 0 & \text{if} i <   j \end{array} \right.$
Algorithm 2 Proj-SGD with entropy estimator and symmetric factors
Let  $m_0\in \mathbb{R}^d$ $C_0\in S^d$  such that  $C_0\succ 0$ $(\gamma_{t})_{t\in \mathbb{N}}\subset (0, + \infty)$    
for  $t\in \mathbb{N}$  do Sample  $u_{t}\sim \mathcal{N}(0,I)$  Compute  $\pi_t = -\nabla_z\log p(C_t u_t + m_t,x)$  Set  $m_{t + 1} = m_t - \gamma_t\pi_t$  Set  $\hat{C}_{t + 1} = C_t - \gamma_t(\pi_t u_t^\top -C_t^{-1})$  Compute  $U_{t + 1}\hat{D}_{t + 1}U_{t + 1}^\top$  a singular value decomposition of  $(\hat{C}_{t + 1} + \hat{C}_{t + 1}^\top) / 2$  Set  $C_{t + 1} = U_{t + 1}D_{t + 1}U_{t + 1}^\top$  , where  $D_{t + 1}(i,i) = \max \{\hat{D}_{t + 1}(i,i);M^{-1 / 2}\}$
Algorithm 3 Proj-SGD with STL estimator and symmetric factors
Let  $m_0\in \mathbb{R}^d$ $C_0\in S^d$  such that  $C_0\succ 0$ $(\gamma_{t})_{t\in \mathbb{N}}\subset (0, + \infty)$    
for  $t\in \mathbb{N}$  do Sample  $u_{t}\sim \mathcal{N}(0,I)$  Compute  $\pi_t = -\nabla_z\log p(C_t u_t + m_t,x)$  Set  $m_{t + 1} = m_t - \gamma_t(\pi_t - C_t^{-1}u_t)$  Set  $\hat{C}_{t + 1} = C_t - \gamma_t(\pi_t u_t^\top -C_t^{-1}u_t u_t^\top)$  Compute  $U_{t + 1}\hat{D}_{t + 1}U_{t + 1}^\top$  a singular value decomposition of  $(\hat{C}_{t + 1} + \hat{C}_{t + 1}^\top) / 2$  Set  $C_{t + 1} = U_{t + 1}D_{t + 1}U_{t + 1}^\top$  , where  $D_{t + 1}(i,i) = \max \{\hat{D}_{t + 1}(i,i);M^{-1 / 2}\}$