SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\| \nabla f (z) - \nabla f \left(z ^ {\prime}\right) \| \leq \| z - z ^ {\prime} \|
$$
holds for all  $z,z^{\prime}\in \mathcal{Z}$
This assumption, also occasionally called Lipschitz smoothness, restricts the amount the gradient can change for a given distance. When  $f$  is twice differentiable, an equivalent condition is the Hessian to be bounded:
Definition 6 (Smoothness). A twice differentiable function  $f: \mathcal{Z} \to \mathbb{R}$  is said to be  $L$ -smooth if the inequality
$$
\| \nabla^ {2} f (\boldsymbol {z}) \| \leq L
$$
holds for all  $z \in \mathcal{Z}$ .
Remark 7. Assuming a function  $f$  is smooth is equivalent to assuming that  $f$  can be upper bounded by a quadratic function everywhere.
Remark 8. When the log-density  $\log \pi$  of a probability measure  $\Pi$  is  $L$ -smooth,  $\log \pi$  can be upper bounded everywhere by the log-density of a Gaussian.
Definition 7 (Strong Convexity). A twice differentiable function  $f: \mathbb{R}^d \to \mathbb{R}$  is said to be  $\mu$ -strongly convex if the inequality
$$
\frac {\mu}{2} \left\| \boldsymbol {z} - \boldsymbol {z} ^ {\prime} \right\| ^ {2} + \langle \nabla f (\boldsymbol {z}), \boldsymbol {z} - \boldsymbol {z} ^ {\prime} \rangle + f (\boldsymbol {z}) \leq f (\boldsymbol {z} ^ {\prime})
$$
holds for all  $z,z^{\prime}\in \mathbb{R}^{d}$  and some  $\mu >0$
Remark 9. If Definition 7 holds only for  $\mu = 0$ ,  $f$  is said to be (non-strongly) convex.
Remark 10. Assuming a function  $f$  is strongly convex is equivalent to assuming that  $f$  can be lower bounded by a quadratic.
Definition 8 (Strongly Log-Concave Measures). For a probability measure  $\Pi$  in a Euclidean measurable space  $(\mathbb{R}^d, \mathcal{B}(\mathbb{R}^d), \mathbb{P})$ , where  $\mathcal{B}(\mathbb{R}^d)$  is the  $\sigma$ -algebra of Borel-measurable subsets of  $\mathbb{R}^d$ ,  $\mathbb{P}$  is the Lebesgue measure, we say  $\Pi$  is  $\mu$ -strongly log-concave if its log-density  $\log \pi(z): \mathbb{R}^d \to \mathbb{R}$  is  $\mu$ -strongly convex for some  $\mu > 0$ .
Remark 11. If Definition 8 holds only for  $\mu = 0$ ,  $\Pi$  is said to be (non-strongly) log-concave.
Remark 12. When  $\Pi$  is  $\mu$ -strongly log-concave,  $\log \pi$  can be lower bounded everywhere by the log-density of a Gaussian.
Algorithm 1: ProxGen-Adam for Black-Box Variational Inference
Input: Initial variational parameters  $\lambda_0$ , base stepsize  $\alpha$ , second moment stepsize  $\beta_2$ , momentum stepsize  $\{\beta_{1,t}\}_{t=1}^{T}$ , small positive constant  $\epsilon$   
for  $t = 1, \ldots, T$  do  
estimate gradient of energy  $\widehat{\nabla f}$ $\mathbf{g}_t = \widehat{\nabla f}(\lambda) + \nabla h(\lambda)$ $\overline{\lambda}_{t+1} = \beta_{1,t}\overline{\lambda}_t + (1 - \beta_{1,t})\overline{\lambda}_t$ $\pmb{v}_{t+1} = \beta_2\pmb{v}_t + (1 - \beta_2)\pmb{g}_t^2$ $\Gamma_{t+1} = \mathrm{diag}\left(\frac{\alpha}{\sqrt{\pmb{v}_{t+1}} + \epsilon}\right)$ $\lambda_{t+1} = \lambda_t - \Gamma_{t+1}\overline{\lambda}_{t+1}$ $\pmb{s}_{t+1} \gets \text{getscale}(\lambda_{t+1})$ $\pmb{s}_{t+1} \gets \pmb{s}_{t+1} + \frac{1}{2}\left(\sqrt{\pmb{s}_{t+1}^2 + 4\gamma_{s,t+1}} - \pmb{s}_{t+1}\right)$ $\lambda_{t+1} \gets \text{setscale}(\lambda_{t+1}, \pmb{s}_{t+1})$   
end
(By convention, all vector operations are elementwise.)
Adaptive and matrix-valued stepsize-variants of SGD such as Adam (Kingma & Ba, 2015), AdaGrad (Duchi et al., 2011) are widely used. The matrix stepsize of Adam at iteration  $t$  is given as
$$
\boldsymbol {\Gamma} _ {t + 1} = \operatorname {d i a g} \left(\alpha / \left(\sqrt {\boldsymbol {v} _ {t + 1}} + \epsilon\right)\right),
$$
where  $\pmb{v}_t$  is the exponential moving average of the second moment,  $\alpha$  is the "base stepsize." Furthermore, the matrix stepsize is applied to the moving average of the gradients, a scheme often called the (heavy-ball) momentum, denoted here as  $\overline{\lambda}_t$ .
Recently, Yun et al. (2021) have proven the convergence for these adaptive, momentum, and matrix-valued stepsize-based SGD methods with proximal steps. Then, the proximal operator is applied as
$$
\operatorname {p r o x} _ {\boldsymbol {\Gamma} _ {t}, h} \left(\boldsymbol {\lambda} _ {t} - \boldsymbol {\Gamma} _ {t} \overline {{\boldsymbol {\lambda}}} _ {t}\right) = \underset {\boldsymbol {\lambda}} {\arg \min} \left\{\left<   \overline {{\boldsymbol {\lambda}}} _ {t}, \boldsymbol {\lambda} \right> + h \left(\boldsymbol {\lambda}\right) + \frac {1}{2} (\boldsymbol {\lambda} - \boldsymbol {\lambda} _ {t}) ^ {\top} \boldsymbol {\Gamma} _ {t} ^ {- 1} \left(\boldsymbol {\lambda} - \boldsymbol {\lambda} _ {t}\right)\right\}.
$$
For Adam, the matrix-valued stepsize is a diagonal matrix. Thus, the proximal operator of Domke (2020) for each  $s_i$  forms independent 1-dimensional quadratic problems. Thus, the proximal step is given in the closed-form
$$
\operatorname {p r o x} _ {\Gamma_ {t}, h} (s _ {i}) = s _ {i} + \frac {1}{2} \left(\sqrt {s _ {i} ^ {2} + 4 \gamma_ {s _ {i}}} - s _ {i}\right),
$$
where, dropping the index  $t$  for clarity,  $\overline{s}_i$  is the element of  $\overline{\lambda}_t$  corresponding to  $s_i$ ,  $\gamma_{s_i}$  denotes the stepsize of  $s_i$  (a diagonal element of  $\Gamma_t$ ). Combined with the Adam-like stepsize rule, the algorithm is shown in Algorithm 1.
Difference with Adam In Algorithm 1, we can see the differences with vanilla Adam. Notably, ProxGenAdam does not perform bias correction of the estimated moments. Furthermore, while some implementations of Adam decay  $\beta_{1}$ , we keep it constant. It is possible that these differences could result in a different behavior from vanilla Adam. However, in this work, we follow the original implementation by Yun et al. (2021) as closely as possible and leave the comparison with vanilla Adam to future works.
D ProxGen Adam for Black-Box Variational Inference
E Detailed Comparison Against Domke et al. (2023)
In this section, we contrast our results against those of Domke et al. (2023). First, the main challenge to establishing a convergence guarantee for BBVI has been on bounding the gradient variance. In particular, Domke (2019) proved that the variance of the reparameterization gradient for the energy,  $\widehat{\nabla f}$ , is bounded as
$$
\mathbb {E} \| \widehat {\nabla f} (\lambda) \| _ {2} ^ {2} \leq \alpha \| \lambda - \bar {\lambda} \| _ {2} ^ {2} + \beta \tag {3}
$$
for some finite positive constants  $\alpha, \beta$  depending on the problem constants  $d, L, k_{\varphi}$ . Domke et al. (2023) call a gradient estimator satisfying this bound to be a "quadratic variance" estimator. Furthermore, they prove that the closed-form entropy (CFE; Kucukelbir et al., 2017; Titsias & Lázaro-Gredilla, 2014) estimator:
$$
\widehat {\nabla F} _ {\mathrm {C F E}} (\lambda) \triangleq \widehat {\nabla f} (\lambda) + \nabla h (\lambda)
$$
and the STL estimator by Roeder et al. (2017):
$$
\widehat {\nabla F} _ {\mathrm {S T L}} (\boldsymbol {\lambda}) \triangleq \frac {1}{M} \sum_ {m = 1} ^ {M} - \nabla_ {\boldsymbol {\lambda}} \ell \left(\mathcal {F} _ {\boldsymbol {\lambda}} (\boldsymbol {u} _ {m})\right) + \nabla_ {\boldsymbol {\lambda}} \log q _ {\boldsymbol {\lambda}} \left(\mathcal {F} _ {\boldsymbol {\nu}} (\boldsymbol {u} _ {m})\right) \Big | _ {\boldsymbol {\nu} = \boldsymbol {\lambda}},
$$
where  $\pmb{u} \sim \varphi$ , also qualify as quadratic variance estimators.
Unfortunately, it has been unknown whether SGD is guaranteed to converge with a quadratic variance estimator except for strongly convex objectives (Wright & Recht, 2021, p. 85). Domke et al. (2023) expand the boundaries of SGD and prove that projected and proximal SGD with a quadratic variance estimator converges for both convex and strongly convex objectives. In particular, for the location-scale variational family, the linear parameterization, and log-concave objectives, they prove a complexity of  $\mathcal{O}\left(1 / \varepsilon^2\right)$ , and for strongly log-concave objectives, they prove a complexity of  $\mathcal{O}(1 / \varepsilon)$ .
On the other hand, Kim et al. (2023) and Lemma 12 developed the bound in Equation (3) to be of the form of
$$
\mathbb {E} \left\| \widehat {\nabla F} (\lambda) \right\| _ {2} ^ {2} \leq A (F (\lambda) - F ^ {*}) + \| \nabla F (\lambda) \| _ {2} ^ {2} + C \tag {4}
$$
for some positive finite constants  $A, B, C$ , for which the convergence of SGD for convex, strongly convex (Garrigos & Gower, 2023; Gorbunov et al., 2020), and non-convex objectives (Khaled & Richtárik, 2023) have already been proven. Applying these results to log-smooth and log-quadratically growing objectives, we prove a complexity of  $\mathcal{O}\left(1 / \epsilon^4\right)$ , while for strong log-concave objectives, we also prove a complexity of  $\mathcal{O}\left(1 / \epsilon\right)$ .
Overall, both approaches can be summarized as follows: we focused on establishing gradient variance bounds of known convergence proofs, while Domke et al. (2023) aimed to prove that the bound by Domke (2019) is sufficient to guarantee convergence. Note that, for strongly log-concave objectives, Equation (3) immediately implies Equation (4). Therefore, both approaches intersect in the case of strongly log-concave objectives. Indeed, Theorem 8 and the analogous result of Domke et al. (2023) are both based on the same proof strategy by Gower et al. (2019).
F.1 Auxiliary Lemmas
Lemma 5. Let  $\phi(x) = x$ . Then, the parameterization is linear in the sense that  $\mathcal{T}_{\lambda}$  is a bilinear function such that
$$
\mathcal {T} _ {\lambda - \lambda^ {\prime}} (\boldsymbol {u}) = \mathcal {T} _ {\lambda} (\boldsymbol {u}) - \mathcal {T} _ {\lambda^ {\prime}} (\boldsymbol {u}).
$$
for any  $\lambda, \lambda' \in \Lambda$ .
$$
\begin{array}{l} \mathcal {T} _ {\lambda - \lambda^ {\prime}} (\boldsymbol {u}) = \left(\boldsymbol {C} \left(\lambda - \lambda^ {\prime}\right)\right) \boldsymbol {u} + \left(\boldsymbol {m} - \boldsymbol {m} ^ {\prime}\right) \\ = \left(\boldsymbol {D} _ {\phi} \left(\boldsymbol {s} - \boldsymbol {s} ^ {\prime}\right) + \left(\boldsymbol {L} - \boldsymbol {L} ^ {\prime}\right)\right) \boldsymbol {u} + \left(\boldsymbol {m} - \boldsymbol {m} ^ {\prime}\right), \\ \end{array}
$$
using the fact that  $\phi$  is the identity function,
$$
\begin{array}{l} = \left(\boldsymbol {D} _ {\phi} (\boldsymbol {s}) - \boldsymbol {D} _ {\phi} \left(\boldsymbol {s} ^ {\prime}\right) + \left(\boldsymbol {L} - \boldsymbol {L} ^ {\prime}\right)\right) \boldsymbol {u} + \left(\boldsymbol {m} - \boldsymbol {m} ^ {\prime}\right) \\ = \left(\boldsymbol {C} (\lambda) - \boldsymbol {C} \left(\lambda^ {\prime}\right)\right) \boldsymbol {u} + \left(\boldsymbol {m} + \boldsymbol {m} ^ {\prime}\right) \\ = (\boldsymbol {C} (\lambda) \boldsymbol {u} + \boldsymbol {m}) - (\boldsymbol {C} (\lambda^ {\prime}) \boldsymbol {u} + \boldsymbol {m} ^ {\prime}) \\ = \mathcal {T} _ {\lambda} (\boldsymbol {u}) - \mathcal {T} _ {\lambda^ {\prime}} (\boldsymbol {u}). \\ \end{array}
$$
The linearity with respect to  $\pmb{u}$  is obvious.
Lemma 6. Let the linear parameterization be used. Then, for any  $\lambda, \lambda' \in \Lambda$ , the inner product of the Jacobian of the reparameterization function satisfies the following equalities for any  $\mathbf{u} \in \mathbb{R}^d$ .
(i) For the Cholesky family (Domke, 2019, Lemma 8),
$$
\left(\frac {\partial \mathcal {T} _ {\lambda} (\boldsymbol {u})}{\partial \lambda}\right) ^ {\top} \frac {\partial \mathcal {T} _ {\lambda} (\boldsymbol {u})}{\partial \lambda} = \left(1 + \| \boldsymbol {u} \| _ {2} ^ {2}\right) \mathbf {I}
$$
(ii) For the mean-field family (Kim et al., 2023, Lemma 1),
$$
\left(\frac {\partial \mathcal {T} _ {\lambda} (\boldsymbol {u})}{\partial \lambda}\right) ^ {\top} \frac {\partial \mathcal {T} _ {\lambda} (\boldsymbol {u})}{\partial \lambda} = \left(1 + \| \boldsymbol {U} ^ {2} \| _ {\mathrm {F}}\right) \mathbf {I},
$$
$$
w h e r e \boldsymbol {U} = \operatorname {d i a g} \left(u _ {1}, \dots , u _ {d}\right).
$$
Lemma 7. Let the linear parameterization be used. Then, for any  $\lambda \in \Lambda$  and any  $z\in \mathbb{R}^d$ , the following relationships hold.
(i) For the Cholesky family (Domke, 2019, Lemma 2),
$$
\mathbb {E} \left(1 + \| \boldsymbol {u} \| _ {2} ^ {2}\right) \| \mathcal {T} _ {\lambda} (\boldsymbol {u}) - \boldsymbol {z} \| _ {2} ^ {2} = (d + 1) \| \boldsymbol {m} - \boldsymbol {z} \| _ {2} ^ {2} + \left(d + k _ {\varphi}\right) \| \boldsymbol {C} \| _ {\mathrm {F}} ^ {2}
$$
(ii) For the mean-field family (Kim et al., 2023, Lemma 2),
$$
\mathbb {E} \left(1 + \| \boldsymbol {U} ^ {2} \| _ {\mathrm {F}}\right) \| \mathcal {T} _ {\lambda} (\boldsymbol {u}) - \boldsymbol {z} \| _ {2} ^ {2} \leq \left(\sqrt {d k _ {\varphi}} + k _ {\varphi} \sqrt {d} + 1\right) \| \boldsymbol {m} - \boldsymbol {z} \| _ {2} ^ {2} + \left(2 k _ {\varphi} \sqrt {d} + 1\right) \| \boldsymbol {C} \| _ {\mathrm {F}} ^ {2}.
$$
Corollary 2. Let the linear parameterization be used and  $\lambda, \lambda' \in \Lambda$  be any pair of variational parameters.
(i) For the Cholesky family,
$$
\mathbb {E} \left(1 + \| \pmb {u} \| _ {2} ^ {2}\right) \left\| \mathcal {T} _ {\lambda^ {\prime}} (\pmb {u}) - \mathcal {T} _ {\lambda} (\pmb {u}) \right\| _ {2} ^ {2} \leq (k _ {\varphi} + d) \| \lambda - \lambda^ {\prime} \| _ {2} ^ {2}
$$
(ii) For the mean-field family,
$$
\mathbb {E} \left(1 + \| \boldsymbol {U} ^ {2} \| _ {\mathrm {F}}\right) \| \mathcal {T} _ {\lambda^ {\prime}} (\boldsymbol {u}) - \mathcal {T} _ {\lambda} (\boldsymbol {u}) \| _ {2} ^ {2} \leq \left(2 k _ {\varphi} \sqrt {d} + 1\right) \| \lambda - \lambda^ {\prime} \| _ {2} ^ {2}
$$
Proof. The results are a direct consequence of Lemma 7 and Lemma 5.
Proof of (i) We start from Lemma 7 as
$$
\mathbb {E} \left(1 + \| \boldsymbol {u} \| _ {2} ^ {2}\right) \| \mathcal {T} _ {\lambda - \lambda^ {\prime}} (\boldsymbol {u}) - \boldsymbol {z} \| _ {2} ^ {2} = (d + 1) \| (\boldsymbol {m} - \boldsymbol {m} ^ {\prime}) - \boldsymbol {z} \| _ {2} ^ {2} + (d + k _ {\varphi}) \| \boldsymbol {C} (\lambda) - \boldsymbol {C} (\lambda^ {\prime}) \| _ {\mathrm {F}} ^ {2},
$$
setting  $z = 0$
$$
= (d + 1) \left\| \boldsymbol {m} - \boldsymbol {m} ^ {\prime} \right\| _ {2} ^ {2} + (d + k _ {\varphi}) \left\| \boldsymbol {C} (\lambda) - \boldsymbol {C} (\lambda^ {\prime}) \right\| _ {\mathrm {F}} ^ {2},
$$
and since  $k_{\varphi} \geq 3$  by the property of the kurtosis,
$$
\begin{array}{l} \leq \left(d + k _ {\varphi}\right) \left(\| \boldsymbol {m} - \boldsymbol {m} ^ {\prime} \| _ {2} ^ {2} + \| \boldsymbol {C} (\lambda) - \boldsymbol {C} (\lambda^ {\prime}) \| _ {\mathrm {F}} ^ {2}\right) \\ = \left(d + k _ {\varphi}\right) \| \lambda - \lambda^ {\prime} \| _ {2} ^ {2}. \\ \end{array}
$$
Proof of (ii) Similarly, for the mean-field family, we can apply Lemma 7 as
$$
\mathbb {E} \left(1 + \| \boldsymbol {U} ^ {2} \| _ {\mathrm {F}}\right) \| \mathcal {T} _ {\lambda - \lambda^ {\prime}} (\boldsymbol {u}) - \bar {\boldsymbol {z}} \| _ {2} ^ {2} \leq \left(\sqrt {d k _ {\varphi}} + k _ {\varphi} \sqrt {d} + 1\right) \| (\boldsymbol {m} - \boldsymbol {m} ^ {\prime}) - \bar {\boldsymbol {z}} \| _ {2} ^ {2} + \left(2 k _ {\varphi} \sqrt {d} + 1\right) \| \boldsymbol {C} - \boldsymbol {C} ^ {\prime} \| _ {\mathrm {F}} ^ {2},
$$
$$
= \left(\sqrt {d k _ {\varphi}} + k _ {\varphi} \sqrt {d} + 1\right) \| \boldsymbol {m} - \boldsymbol {m} ^ {\prime} \| _ {2} ^ {2} + \left(2 k _ {\varphi} \sqrt {d} + 1\right) \| \boldsymbol {C} - \boldsymbol {C} ^ {\prime} \| _ {\mathrm {F}} ^ {2},
$$
$$
\begin{array}{l} \leq \left(2 k _ {\varphi} \sqrt {d} + 1\right) \left(\| \boldsymbol {m} - \boldsymbol {m} ^ {\prime} \| _ {2} ^ {2} + \| \boldsymbol {C} (\lambda) - \boldsymbol {C} (\lambda^ {\prime}) \| _ {\mathrm {F}} ^ {2}\right) \\ = \left(2 k _ {\varphi} \sqrt {d} + 1\right) \| \lambda - \lambda^ {\prime} \| _ {2} ^ {2}. \\ \end{array}
$$
Lemma 8. For the linear parameterization,
$$
\mathbb {E} \lVert \mathcal {T} _ {\lambda} (\pmb {u}) - \mathcal {T} _ {\lambda^ {\prime}} (\pmb {u}) \rVert_ {2} ^ {2} = \lVert \lambda - \lambda^ {\prime} \rVert_ {2} ^ {2}
$$
Proof. First notice that, for linear parameterizations, we have
$$
\begin{array}{l} \mathbb {E} \left\| \mathcal {T} _ {\lambda} (\boldsymbol {u}) \right\| _ {2} ^ {2} = \mathbb {E} \left\| \boldsymbol {C u} + \boldsymbol {m} \right\| _ {2} ^ {2} \\ = \mathbb {E} u ^ {\top} C ^ {\top} C u + \| m \| _ {2} ^ {2} + 2 m ^ {\top} C \mathbb {E} u \\ = \mathbb {E} \operatorname {t r} \left(\boldsymbol {u} ^ {\top} \boldsymbol {C} ^ {\top} \boldsymbol {C u}\right) + \| \boldsymbol {m} \| _ {2} ^ {2} + 2 \boldsymbol {m} ^ {\top} \boldsymbol {C} \mathbb {E} \boldsymbol {u}, \\ \end{array}
$$
rotating the elements of the trace,
$$
= \operatorname {t r} \left(\boldsymbol {C} ^ {\top} \boldsymbol {C} \boldsymbol {\mathbb {E}} \boldsymbol {u u} ^ {\top}\right) + \| \boldsymbol {m} \| _ {2} ^ {2} + 2 \boldsymbol {m} ^ {\top} \boldsymbol {C} \boldsymbol {\mathbb {E}} \boldsymbol {u},
$$
applying Assumption 1
$$
\begin{array}{l} = \operatorname {t r} \left(\boldsymbol {C} ^ {\top} \boldsymbol {C}\right) + \| \boldsymbol {m} \| _ {2} ^ {2} \\ = \| \boldsymbol {C} \| _ {\mathrm {F}} ^ {2} + \| \boldsymbol {m} \| _ {2} ^ {2} \\ = \| \lambda \| _ {2} ^ {2}. \\ \end{array}
$$
Combined with Lemma 5, we have
$$
\mathbb {E} \left\| \mathcal {T} _ {\lambda} (\boldsymbol {u}) - \mathcal {T} _ {\lambda^ {\prime}} (\boldsymbol {u}) \right\| _ {2} ^ {2} = \mathbb {E} \left\| \mathcal {T} _ {\lambda - \lambda^ {\prime}} (\boldsymbol {u}) \right\| _ {2} ^ {2} = \left\| \lambda - \lambda^ {\prime} \right\| _ {2} ^ {2}.
$$
F.2 Properties of the Evidence Lower Bound
F.2.1 Smoothness
Proof. The entropic regularizer is
$$
h \left(\lambda\right) = - \mathrm {H} \left(\varphi\right) - \sum_ {i = 1} ^ {d} \log \phi \left(s _ {i}\right),
$$
and depends only on the diagonal elements  $s_1, \ldots, s_d$  of  $\mathbf{C}$ . The Hessian of  $h$  is then a diagonal matrix, where only the entries that correspond to  $s_1, \ldots, s_d$  are non-zero. The Lipschitz smoothness constant is then the constant  $L_h < \infty$  that satisfies
$$
\frac {\partial^ {2} h (\lambda)}{\partial s _ {i} ^ {2}} = - \frac {\mathrm {d} ^ {2} \log \phi}{\mathrm {d} s _ {i} ^ {2}} <   L _ {h}
$$
for all  $i = 1, \ldots, d$ , which is the smoothness constant of  $s_i \mapsto \log \phi(s_i)$ .
Lemma 2. Let  $\pmb{H}$  be a  $n \times n$  symmetric random matrix, where it is bounded as  $\| H \|_2 \leq L < \infty$  almost surely. Also, let  $\pmb{J}$  be an  $m \times n$  random matrix such that  $\| \mathbb{E} J^\top J \|_2 < \infty$ . Then,
$$
\| \mathbb {E} \boldsymbol {J} ^ {\top} \boldsymbol {H} \boldsymbol {J} \| _ {2} \leq L \| \mathbb {E} \boldsymbol {J} ^ {\top} \boldsymbol {J} \| _ {2}.
$$
Proof. By the property of the Rayleigh quotients, for a symmetric matrix  $\mathbf{A}$ , its maximum eigenvalue is given in the variational form
$$
\sup _ {\| \boldsymbol {x} \| \leq 1} \boldsymbol {x} ^ {\top} \boldsymbol {H} \boldsymbol {x} = \sigma_ {\max } (\boldsymbol {H}) \leq \sqrt {\sigma_ {\max } (\boldsymbol {H}) ^ {2}} = \| \boldsymbol {H} \| _ {2},
$$
where  $\sigma_{\mathrm{max}}(\mathbf{A})$  is the maximal eigenvalue of  $\mathbf{A}$ . Notice the relationship with the  $\ell_2$ -operator norm. The inequality is strict only if all eigenvalues are negative.
From the property above,
$$
\| \mathbb {E} J ^ {\top} H J \| _ {2} = \sup  _ {\| x \| _ {2} \leq 1} x ^ {\top} \left(\mathbb {E} J ^ {\top} H J\right) x.
$$
By reparameterizing as  $\pmb {y} = \pmb {J}\pmb{x}$
$$
= \sup  _ {\| \boldsymbol {x} \| _ {2} \leq 1} \mathbb {E} \boldsymbol {y} ^ {\top} \boldsymbol {H} \boldsymbol {y},
$$
and the property of the  $\ell_2$ -operator norm,
$$
\leq \sup  _ {\| \boldsymbol {x} \| _ {2} \leq 1} \mathbb {E} \| \boldsymbol {H} \| _ {2} \| \boldsymbol {y} \| _ {2} ^ {2} = \sup  _ {\| \boldsymbol {x} \| _ {2} \leq 1} \mathbb {E} \| \boldsymbol {H} \| _ {2} \| \boldsymbol {J} \boldsymbol {x} \| _ {2} ^ {2}.
$$
From our assumption about the maximal eigenvalue of  $\pmb{H}$ ,
$$
\leq L \sup_{\| \boldsymbol {x}\|_{2}\leq 1}\mathbb{E}\| \boldsymbol {J}\boldsymbol{x}\|_{2}^{2},
$$
denoting the  $\ell_2$  vector norm as a quadratic form as,
$$
= L\sup_{\| \boldsymbol {x}\|_{2}\leq 1}\boldsymbol{x}^{\top}\left(\mathbb{E}\boldsymbol{J}^{\top}\boldsymbol {J}\right)\boldsymbol {x},
$$
again, by the property of the  $\ell_2$ -operator norm,
$$
\begin{array}{l} \leq L \| \mathbb {E} \boldsymbol {J} ^ {\top} \boldsymbol {J} \| _ {2} \sup  _ {\| \boldsymbol {x} \| _ {2} \leq 1} \| \boldsymbol {x} \| _ {2} ^ {2} \\ = L \| \mathbb {E} \boldsymbol {J} ^ {\mathsf {T}} \boldsymbol {J} \| _ {2}. \\ \end{array}
$$
Lemma 9. For a 1-Lipschitz diagonal conditioner  $\phi$ , the Jacobian of the location-scale reparam-eterization function  $\mathcal{T}_{\lambda}$  satisfies
$$
\left\| \mathbb {E} \left(\frac {\partial \mathcal {T} _ {\lambda} (\boldsymbol {u})}{\partial \lambda}\right) ^ {\top} \frac {\partial \mathcal {T} _ {\lambda} (\boldsymbol {u})}{\partial \lambda} \right\| _ {2} \leq 1.
$$
Proof. For notational clarity, we will occasionally represent  $\mathcal{T}_{\lambda}$  as
$$
\mathcal {T} _ {\lambda} (\boldsymbol {u}) = \mathcal {T} (\lambda ; \boldsymbol {u}),
$$
such that  $\mathcal{T}_i(\lambda ;\pmb {u})$  denotes the ith component of  $\mathcal{T}_{\lambda}$
From the definition of  $\mathcal{T}_{\lambda}$ , it is straightforward to notice that its Jacobian is the concatenation of 3 block matrices
$$
\boldsymbol {J} _ {\boldsymbol {m}} = \frac {\partial \mathcal {T} _ {\lambda} (\boldsymbol {u})}{\partial \boldsymbol {m}}, \qquad \boldsymbol {J} _ {\boldsymbol {s}} = \frac {\partial \mathcal {T} _ {\lambda} (\boldsymbol {u})}{\partial \boldsymbol {s}}, \text {a n d} \boldsymbol {J} _ {\boldsymbol {L}} = \frac {\partial \mathcal {T} _ {\lambda} (\boldsymbol {u})}{\partial \operatorname {v e c} (\boldsymbol {L})}.
$$
The  $m$  block form a deterministic identity matrix
$$
\boldsymbol {J} _ {\boldsymbol {m}} = \frac {\partial \mathcal {T} _ {\lambda} (\boldsymbol {u})}{\partial \boldsymbol {m}} = \mathbf {I},
$$
which is shown by (Domke, 2020, Lemma 4).
The proof strategy is as follows: we will directly compute the squared Jacobian through block matrix multiplication. The key is that, after expectation, the resulting matrix becomes diagonal. Then, the  $\ell_2$  operator norm, or maximal eigenvalue, follows trivially as the maximal diagonal element.
First,
$$
\begin{array}{l} \mathbb {E} \left(\frac {\partial \mathcal {T} (\boldsymbol {\lambda} ; \boldsymbol {u})}{\partial \boldsymbol {\lambda}}\right) ^ {\top} \frac {\partial \mathcal {T} (\boldsymbol {\lambda} ; \boldsymbol {u})}{\partial \boldsymbol {\lambda}} = \mathbb {E} \left[ \begin{array}{c} \boldsymbol {J} _ {m} ^ {\top} \\ \boldsymbol {J} _ {s} ^ {\top} \\ \boldsymbol {J} _ {L} ^ {\top} \end{array} \right] \left[ \begin{array}{c c c} \boldsymbol {J} _ {m} & \boldsymbol {J} _ {s} & \boldsymbol {J} _ {L} \end{array} \right] \\ = \mathbb {E} \left[ \begin{array}{c c c} J _ {m} ^ {\top} J _ {m} & J _ {m} ^ {\top} J _ {s} & J _ {m} ^ {\top} J _ {L} \\ J _ {s} ^ {\top} J _ {m} & J _ {s} ^ {\top} J _ {s} & J _ {s} ^ {\top} J _ {L} \\ J _ {L} ^ {\top} J _ {m} & J _ {L} ^ {\top} J _ {s} & J _ {L} ^ {\top} J _ {L} \end{array} \right] \\ = \left[ \begin{array}{c c c} \mathbf {I} & \mathbb {E} J _ {s} & \mathbb {E} J _ {L} \\ \mathbb {E} J _ {s} ^ {\top} & \mathbb {E} J _ {s} ^ {\top} J _ {s} & \mathbb {E} J _ {s} ^ {\top} J _ {L} \\ \mathbb {E} J _ {L} ^ {\top} & \mathbb {E} J _ {L} ^ {\top} J _ {s} & \mathbb {E} J _ {L} ^ {\top} J _ {L} \end{array} \right]. \\ \end{array}
$$
For  $J_{s}$ , the entries are
$$
\frac {\partial \mathcal {T} _ {i} (\boldsymbol {u})}{\partial s _ {j}} = \phi^ {\prime} (s _ {i}) u _ {j} \mathbb {1} _ {i = j},
$$
which is a diagonal matrix. Thus, by Assumption 1,
$$
\mathbb {E} \boldsymbol {J} _ {s} = \mathbf {O}, \qquad \mathbb {E} \boldsymbol {J} _ {s} ^ {\top} \boldsymbol {J} _ {s} = \operatorname {d i a g} \left(\boldsymbol {\phi} ^ {\prime} (\boldsymbol {s})\right) ^ {2}.
$$
$$
\frac {\partial \mathcal {F} _ {i} (\boldsymbol {\lambda} ; \boldsymbol {u})}{\partial L _ {j k}} = u _ {k} \mathbb {1} _ {i = j}.
$$
To gather some intuition, the case of  $d = 4$  looks like the following: