SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Kyurae Kim
University of Pennsylvania kyrkim@seas.upenn.edu
Jisu Oh
North Carolina State University joh26@ncsu.edu
Kaiwen Wu
University of Pennsylvania kaiwenwu@seas.upenn.edu
Yi-An Ma
University of California, San Diego  
yianma@ucsd.edu
Jacob R. Gardner
University of Pennsylvania jacobrg@seas.upenn.edu
We provide the first convergence guarantee for black-box variational inference (BBVI) with the reparameterization gradient. While preliminary investigations worked on simplified versions of BBVI (e.g., bounded domain, bounded support, only optimizing for the scale, and such), our setup does not need any such algorithmic modifications. Our results hold for log-smooth posterior densities with and without strong log-concavity and the location-scale variational family. Notably, our analysis reveals that certain algorithm design choices commonly employed in practice, such as nonlinear parameterizations of the scale matrix, can result in suboptimal convergence rates. Fortunately, running BBVI with proximal stochastic gradient descent fixes these limitations and thus achieves the strongest known convergence guarantees. We evaluate this theoretical insight by comparing proximal SGD against other standard implementations of BBVI on large-scale Bayesian inference problems.
Despite the practical success of black-box variational inference (BBVI; Kucukelbir et al., 2017; Ranganath et al., 2014; Titsias & Lázaro-Gredilla, 2014), also known as stochastic gradient variational Bayes and Monte Carlo variational inference, whether it converges under appropriate assumptions on the target problem have been an open problem for a decade. While our understanding of BBVI has been advancing (Bhatia et al., 2022; Challis & Barber, 2013; Domke, 2019, 2020; Hoffman & Ma, 2020), a full convergence guarantee that extends to the practical implementations as used in probabilistic programming languages (PPL) such as Stan (Carpenter et al., 2017), Turing (Ge et al., 2018), Tensorflow Probability (Dillon et al., 2017), Pyro (Bingham et al., 2019), and PyMC (Patil et al., 2010) has yet to be demonstrated.
Due to our lack of understanding, a consensus on how we should implement our BBVI algorithms has yet to be achieved. For example, when the variational family is chosen to be the location-scale family, the "scale" matrix can be parameterized linearly or nonlinearly, and both parameterizations are used by default in popular software packages. (See Table 1 in Kim et al. 2023.) Surprisingly, as we will show, seemingly innocuous design choices like these can substantially impact the convergence of BBVI. This is critical as BBVI has been shown to be less robust (e.g., sensitive to initial points, stepsizes, and such) than competing inference methods such as Markov chain Monte Carlo (MCMC). (See Dhaka et al., 2020; Domke, 2020; Welandawe et al., 2022; Yao et al., 2018.) Instead, the evaluation of BBVI algorithms has been relying on expensive empirical evaluations (Agrawal et al., 2020; Dhaka et al., 2021; Giordano et al., 2018; Yao et al., 2018).
To rigorously analyze the design of BBVI algorithms, we establish the first convergence guarantee for the implementations precisely as used in practice. We provide results for BBVI with the reparameterization gradient (RP; Kingma & Welling, 2014; Titsias & Lázaro-Gredilla, 2014) and the location-scale variational family, arguably the most widely used combination in practice. Our results apply to log-smooth posteriors, which is a routine assumption for analyzing the convergence of stochastic optimization (Garrigos & Gower, 2023) and sampling algorithms (Dwivedi et al., 2019, §2.3). The key is to show that evidence lower bound (ELBO; Jordan et al., 1999) satisfies regularity conditions required by convergence proofs of stochastic gradient descent (SGD; Bottou, 1999; Nemirovski et al., 2009; Robbins & Monro, 1951), the workhorse underlying BBVI.
Our analysis reveals that nonlinear scale matrix parameterizations used in practice are suboptimal: they provably break strong convexity and sometimes even convexity. Even if the posterior is strongly log-concave, the ELBO is not strongly convex anymore. This contrasts with linear parameterizations, which guarantee the ELBO to be strongly convex if the posterior is strongly log-concave (Domke, 2020). Under linear parameterizations, however, the ELBO is no longer smooth, making optimization challenging. Because of this, Domke (2020) proposed to use proximal SGD, which Agrawal & Domke (2021, Appendix A) report to have better performance than vanilla SGD with nonlinear parameterizations. Indeed, we show that BBVI with proximal SGD achieves the fastest known converges rates of SGD, unlike vanilla BBVI. Thus, we provide a concrete reason for employing proximal SGD. We evaluate this insight on large-scale Bayesian inference problems by implementing an Adam-like (Kingma & Ba, 2015) variant of proximal SGD proposed by Yun et al. (2021).
Concurrently to this work, convergence guarantees on BBVI with the RP and the sticking-the-landing estimator (STL; Roeder et al., 2017) under the linear parameterization were published by Domke et al. (2023). To achieve this, they show that a quadratic bound on the gradient variance is sufficient to guarantee the convergence of projected and proximal SGD. In contrast, we focus on analyzing the ELBO under nonlinear parameterizations and connect it to existing analysis strategies. A more in-depth comparison of the two works is provided in Appendix E.
$\bullet$  Convergence Guarantee for BBVI: Theorem 3 establishes a convergence guarantee for BBVI with assumptions matching the implementations used in practice. That is, without algorithmic simplifications and unrealistic assumptions such as bounded domain or bounded support.
$\Theta$  Optimality of Linear Parameterizations: Theorem 2 shows that, for location-scale variational families, nonlinear scale parameterizations prevent the ELBO from being strongly-convex even when the target posterior is strongly log-concave.
Convergence Guarantee for Proximal BBVI: Theorem 4 guarantees that, if proximal SGD is used, BBVI on  $\mu$ -strongly log-concave posteriors can obtain a solution  $\varepsilon$ -close to the global optimum with  $\mathcal{O}(1 / \varepsilon)$  iterations.
$\bullet$  Evaluation of Proximal BBVI in Practice: In Section 5, we evaluate the utility of proximal SGD on large-scale Bayesian inference problems.
Notation Random variables are denoted in  $\operatorname{serif}(e.g., x, x)$ , vectors are in bold  $(e.g., x, x)$ , and matrices are in bold capitals  $(e.g., A)$ . For a vector  $x \in \mathbb{R}^d$ , we denote the inner product as  $\boldsymbol{x}^\top \boldsymbol{x}$  and  $\langle \boldsymbol{x}, \boldsymbol{x} \rangle$ , the  $\ell_2$ -norm as  $\| \boldsymbol{x} \|_2 = \sqrt{\boldsymbol{x}^\top \boldsymbol{x}}$ . For a matrix  $A$ ,  $\| A \|_F = \sqrt{\operatorname{tr}(A^\top A)}$  denotes the Frobenius norm.  $\mathbb{S}_{++}^d$  is the set of positive definite matrices. For some function  $f$ ,  $\mathrm{D}_i f$  denotes the  $i$ th coordinate of  $\nabla f$ , and  $\mathrm{C}^k(\mathcal{X}, \mathcal{Y})$  is the set of  $k$ -time differentiable continuous functions mapping from  $\mathcal{X}$  to  $\mathcal{Y}$ .
2.1 Black-Box Variational Inference
Variational inference (VI, Blei et al., 2017; Jordan et al., 1999; Zhang et al., 2019) aims to minimize the exclusive (or backward/reverse) Kullback-Leibler (KL) divergence as:
$$
\underset {\lambda \in \Lambda} {\text {m i n i m i z e}} \mathrm {D} _ {\mathrm {K L}} \left(q _ {\lambda}, \pi\right) \triangleq \mathbb {E} _ {\boldsymbol {z} \sim q _ {\lambda}} - \log \pi (\boldsymbol {z}) - \mathbb {H} \left(q _ {\lambda}\right),
$$
where  $\mathrm{D}_{\mathrm{KL}}(q_{\lambda},\pi)$  is the KL divergence,  $\mathbb{H}$  is the differential entropy,  $\pi$  is the (target) posterior distribution, and  $q_{\lambda}$  is the variational distribution,
While alternative approaches to VI (Dieng et al., 2017; Hernandez-Lobato et al., 2016; Kim et al., 2022; Naesseth et al., 2020) exist, so far, exclusive KL minimization has been the most successful. We thus use "exclusive KL minimization" as a synonym for VI, following convention.
Equivalently, one minimizes the negative evidence lower bound (ELBO, Jordan et al., 1999)  $F$ :
$$
\underset {\lambda \in \Lambda} {\text {m i n i m i z e}} F (\lambda) \triangleq \mathbb {E} _ {\boldsymbol {z} \sim q _ {\lambda}} - \log p (\boldsymbol {z}, \boldsymbol {x}) - \mathbb {H} (q _ {\lambda}),
$$
where  $\log p(\pmb{z},\pmb{x})$  is the joint likelihood, which is proportional to the posterior as  $\pi (\pmb {z})\propto p(\pmb {z},\pmb {x}) = p(\pmb {x}\mid \pmb {z})p(\pmb {z})$ , where  $p(x\mid z)$  is the likelihood and  $p(z)$  is the prior.
2.2 Variational Family
In this work, we focus on the following variational family.  $(\stackrel{\mathrm{d}}{=} \mathrm{is}$  equivalence in distribution.)
Definition 1 (Reparameterized Family). Let  $\varphi$  be some  $d$ -variate distribution. Then,  $q_{\lambda}$  that can be equivalently represented as
$$
\begin{array}{l l} \boldsymbol {z} \sim q _ {\lambda} & \Leftrightarrow \quad \boldsymbol {z} \stackrel {{\mathrm {d}}} {{=}} \mathcal {T} _ {\lambda} (\boldsymbol {u}); \quad \boldsymbol {u} \sim \varphi , \end{array}
$$
is said to be part of a reparameterized family generated by the base distribution  $\varphi$  and the reparameterization function  $\mathcal{T}_{\lambda}$ .
Definition 2 (Location-Scale Reparameterization Function).  $\mathcal{T}_{\lambda}:\mathbb{R}^{d}\to \mathbb{R}^{d}$  defined as
$$
\mathcal {F} _ {\lambda} (\boldsymbol {u}) \triangleq \boldsymbol {C u} + \boldsymbol {m}
$$
with  $\lambda$  containing the parameters for forming the location  $\pmb{m} \in \mathbb{R}^d$  and scale  $\pmb{C} = \pmb{C}(\lambda) \in \mathbb{R}^{d \times d}$  is called the location-scale reparameterization function.
The location-scale family enables detailed theoretical analysis, as demonstrated by (Domke, 2019, 2020; Fujisawa & Sato, 2021; Kim et al., 2023), and includes the most widely used variational families such as the Student-t, elliptical, and Gaussian families (Titsias & Lázaro-Gredilla, 2014).
Handling Constrained Support For common choices of the base distribution  $\varphi$ , the support of  $q_{\lambda}$  is the whole  $\mathbb{R}^d$ . Therefore, special treatment is needed when the support of  $\pi$  is constrained. Kucukelbir et al. (2017) proposed to handle this by applying diffeomorphic transformation denoted with  $\psi$ , often called bjectors (Dillon et al., 2017; Fjelde et al., 2020; Leger, 2023), to  $q_{\lambda}$  such that
$$
\zeta \sim q _ {\psi , \lambda} \quad \Leftrightarrow \quad \zeta \stackrel {d} {=} \psi^ {- 1} (\mathbf {z}); \quad \mathbf {z} \sim q _ {\lambda},
$$
such that the support of  $q_{\psi, \lambda}$  matches that of  $\pi$ . For example, when the support of  $\pi$  is  $\mathbb{R}_+$ , one can choose  $\psi^{-1} = \exp$ . This approach, known as automatic differentiation VI (ADVI), is now standard in most modern PPLs.
Why focus on posteriors with unconstrained supports? When bijectors are used, the entropy of  $q_{\lambda}$ ,  $\mathbb{H}(q_{\lambda})$ , needs to be adjusted by the Jacobian of  $\psi$  (Kucukelbir et al., 2017),  $J_{\phi^{-1}}$ . However, applying the transformation to  $\pi$  instead of  $q_{\lambda}$  is mathematically equivalent and more convenient. In fact, bijectors can be automatically incorporated into our notation by implicitly setting
$$
p (\boldsymbol {x} \mid \boldsymbol {z}) = \widetilde {p} (\boldsymbol {x} \mid \psi^ {- 1} (\boldsymbol {z})) \quad \text {a n d} \quad p (\boldsymbol {z}) = \widetilde {p} (\psi^ {- 1} (\boldsymbol {z})) | \mathbf {J} _ {\psi^ {- 1}} (\boldsymbol {z}) |,
$$
such that  $\widetilde{\pi} (\zeta)\propto \widetilde{p} (x\mid \zeta)\widetilde{p} (\zeta)$ , where  $\widetilde{\pi}$  is the constrained posterior that we are actually interested in. Therefore, our setup in Section 2.1, where the domain of  $\mathbf{z}$  is taken to be the unconstrained  $\mathbb{R}^d$ , already encompasses constrained posteriors through ADVI.
Lastly, we impose light assumptions on the base distribution  $\varphi$ , which are already satisfied by most variational families used in practice. (i.i.d.: independently and identically distributed.)
Assumption 1 (Base Distribution).  $\varphi$  is a  $d$ -variate distribution such that  $\pmb{u} \sim \varphi$  and  $\pmb{u} = (u_{1}, \dots, u_{d})$  with i.i.d. components. Furthermore,  $\varphi$  is (i) symmetric and standardized such that  $\mathbb{E}u_{i} = 0$ ,  $\mathbb{E}u_{i}^{2} = 1$ ,  $\mathbb{E}u_{i}^{3} = 0$ , and (ii) has finite kurtosis  $\mathbb{E}u_{i}^{4} = k_{\varphi} < \infty$ .
The assumptions on the variational family we will use throughout this work are collectively summarized in the following assumption:
Assumption 2. The variational family is the location-scale family formed by Definitions 1 and 2 with the base distribution  $\varphi$  satisfying Assumption 1.
2.3 Scale Parameterizations
For the "scale" matrix  $\mathbf{C}(\lambda)$  in the location-scale family, any parameterization that results in a positive-definite covariance  $\mathbf{CC}^{\top} \in \mathbb{S}_{++}^{d}$  is valid. However, for the ELBO to ever be convex, the entropy  $\mathbb{H}(q_{\lambda})$  must be convex, which requires the mapping  $\lambda \mapsto \mathbf{CC}^{\top}$  to be convex. To ensure this, we restrict  $\mathbf{C}$  to (lower) triangular matrices with strictly positive eigenvalues, essentially, Cholesky factors. This leaves two of the most common parameterizations:
Definition 3 (Mean-Field Family.).
$$
\boldsymbol {C} = \boldsymbol {D} _ {\phi} (\boldsymbol {s})
$$
where the  $d$  elements of  $\pmb{S}$  forms the diagonal and  $\lambda \in \Lambda$  such that
$$
\Lambda = \{(\boldsymbol {m}, \boldsymbol {s}) \mid \boldsymbol {m} \in \mathbb {R} ^ {d}, \boldsymbol {s} \in \mathcal {S} \}.
$$
Definition 4 (Full-Rank Cholesky Family).
$$
\boldsymbol {C} = \boldsymbol {D} _ {\phi} (\boldsymbol {s}) + \boldsymbol {L},
$$
where the  $d$  elements of  $\pmb{S}$  forms the diagonal,  $\pmb{L}$  is  $d$ -by- $d$  strictly lower triangular, and  $\lambda \in \Lambda$  such that
$$
\Lambda = \left\{\left(\boldsymbol {m}, \boldsymbol {s}, \boldsymbol {L}\right) \mid \boldsymbol {m} \in \mathbb {R} ^ {d}, \boldsymbol {s} \in \mathcal {S}, \operatorname {v e c} (\boldsymbol {L}) \in \mathbb {R} ^ {(d + 1) d / 2} \right\}.
$$
Here,  $S$  is discussed in the next paragraph,  $\pmb{D}_{\phi}(\pmb{s}) \in \mathbb{R}^{d \times d}$  is a diagonal matrix such that  $\pmb{D}_{\phi}(\pmb{s}) \triangleq \operatorname{diag}(\pmb{\phi}(\pmb{s})) = \operatorname{diag}(\phi(s_1), \dots, \phi(s_d))$ , and  $\phi$  is a function we call a diagonal conditioner.
Linear v.s. Nonlinear Parameterizations When the diagonal conditioner is a linear function  $\phi(x) = x$ , we say that the covariance parameterization is linear. In this case, to ensure that  $C$  is a Cholesky factor, the domain of  $s$  is set as  $S = \mathbb{R}_+^d$ . On the other hand, by choosing a nonlinear conditioner  $\phi: \mathbb{R} \to \mathbb{R}_+$ , we can make the domain of  $s$  to be the unconstrained  $S = \mathbb{R}^d$ . Because of this, nonlinear conditioners such as the softplus  $(x) \triangleq \log(1 + \exp(x))$  (Dugas et al., 2000) are frequently used in practice, especially for mean-field. (See Table 1 by Kim et al., 2023).
2.4 Problem Structure of Black-Box Variational Inference
Exclusive KL minimization VI is fundamentally a composite (regularized) optimization problem
$$
F (\lambda) = f (\lambda) + h (\lambda), \quad (\text {E L B O})
$$
where  $f(\lambda) \triangleq \mathbb{E}_{\boldsymbol{z} \sim q_{\lambda}} \ell(\boldsymbol{z})$  is the energy term,  $\ell(\boldsymbol{z}) \triangleq -\log p(\boldsymbol{z}, \boldsymbol{x})$  is the negative joint log-likelihood, and  $h(\lambda) \triangleq -\mathbb{H}\left(q_{\psi, \lambda}\right)$  is the entropic regularizer. From here, BBVI introduces more structure.
An illustration of the taxonomy is shown in Figure 1. In particular, BBVI has an infinite sum structure (IS). That is, it cannot be represented as a sum of finite subcomponents as in ERM. Furthermore,
$$
\begin{array}{l} F (\lambda) = \mathbb {E} _ {\boldsymbol {u} \sim \varphi} f (\lambda ; \boldsymbol {u}) + h (\lambda) \quad \left(\mathrm {C P} \cap \mathrm {I S}\right) \\ = \mathbb {E} _ {\boldsymbol {u} \sim \varphi} \ell \left(\mathcal {T} _ {\lambda} (\boldsymbol {u})\right) + h (\lambda), \quad \text {(C P} \cap \mathrm {I S} \cap \mathrm {R P}) \\ \end{array}
$$
where  $f(\lambda ;\pmb {u})\triangleq \ell (\mathcal{T}_{\lambda}(\pmb {u}))$
Theoretical Challenges The structure of BBVI has multiple challenges that have hindered its theoretical analysis: (i) the stochasticity of the Jacobian of  $\mathcal{T}$  and (ii) The infinite sum structure.
For Item (i), we can see that in
$$
\nabla_ {\lambda} \ell \left(\mathcal {T} _ {\lambda} (\boldsymbol {u})\right) = \frac {\partial \mathcal {T} _ {\lambda} (\boldsymbol {u})}{\partial \lambda} \nabla \ell \left(\mathcal {T} _ {\lambda} (\boldsymbol {u})\right) = \frac {\partial \mathcal {T} _ {\lambda} (\boldsymbol {u})}{\partial \lambda} g (\lambda ; \boldsymbol {u}),
$$
where  $g(\lambda; \boldsymbol{u}) \triangleq (\nabla \ell \circ \mathcal{T}_{\lambda})(\boldsymbol{u})$ , both the Jacobian of  $\mathcal{T}_{\lambda}$  and the gradient of the log-likelihood,  $g$ , depend on the randomness  $\boldsymbol{u}$ . Effectively decoupling the two is a major challenge to analyzing the properties of the ELBO and its gradient estimators (Domke, 2019, 2020).
For Item (ii), the problem is that recent analyses of SGD (Garrigos & Gower, 2023; Gower et al., 2019; Nguyen et al., 2018; Vaswani et al., 2019) have increasingly been relying on the assumption that  $f(\lambda; \boldsymbol{u})$  is smooth for all  $\boldsymbol{u}$  such that
$$
\left\| \nabla_ {\lambda} f (\boldsymbol {\lambda}; \boldsymbol {u}) - \nabla_ {\lambda} f \left(\boldsymbol {\lambda} ^ {\prime}; \boldsymbol {u}\right) \right\| \leq L \| \boldsymbol {\lambda} - \boldsymbol {\lambda} ^ {\prime} \|
$$
CP Composite
IS Infinite Sum
RP Reparameterized
FS Finite Sum
ERM Empirical Risk Minimization
Figure 1: Taxonomy of variational inference. Within BBVI, this work only considers the reparameterization gradient (BBVI  $\cap$  RP, shown in dark red). This leaves out BBVI with the score gradient (BBVI  $\backslash$  RP, shown in light red). The set VI  $\cap$  FS includes sparse variational Gaussian processes (Titsias, 2009), while the remaining set VI  $\backslash$  (FS  $\cup$  IS  $\cup$  RP) includes coordinate ascent VI (Blei et al., 2017).
for some  $L < \infty$ . This is sensible if the support of  $u$  is bounded, which is true for the ERM setting but not for the class of infinite sum (IS) problems. Previous works circumvented this issue by assuming (i) that the support of  $u$  is bounded (Fujisawa & Sato, 2021) which implicitly changes the variational family, or (ii) that the gradient  $\nabla f$  is bounded by a constant (Buchholz et al., 2018; Liu & Owen, 2021) which contradicts strong convexity (Nguyen et al., 2018).
3 The Evidence Lower Bound Under Nonlinear Scale Parameterizations
Under the linear parameterization  $(\phi(x) = x)$ , the properties of the ELBO, such as smoothness and convexity, have been previously analyzed by Challis & Barber (2013); Domke (2020); Titsias & Lázaro-Gredilla (2014). We generalize these results to nonlinear conditioners.
3.1 Technical Assumptions
Let  $g_{i}(\lambda ;\pmb{u})$  be the  $i$ th coordinate of  $\pmb{g}(\lambda ;\pmb{u})$  and recall that  $u_{i}$  denote the  $i$ th element of  $\pmb{u}$ . Establishing convexity and smoothness of the ELBO under nonlinear parameterizations depends on a pair of necessary and sufficient assumptions. To establish smoothness:
Assumption 3. The gradient of  $\ell$  under reparameterization,  $g$ , satisfies
$$
\left| \mathbb {E} g _ {i} (\boldsymbol {\lambda}; \boldsymbol {u}) u _ {i} \phi^ {\prime \prime} (s _ {i}) \right| \leq L _ {s}
$$
for every coordinate  $i = 1,\ldots d$ , any  $\lambda \in \Lambda$ , and some  $0 < L_{s} < \infty$ .
Here,  $\phi''$  is the second derivative of  $\phi$ . The next one is required to establish convexity:
Assumption 4. The gradient of  $\ell$  under reparameterization,  $g$ , satisfies
$$
\mathbb {E} \mathbf {g} _ {i} (\boldsymbol {\lambda}; \boldsymbol {u}) u _ {i} \geq 0
$$
for every coordinate  $i = 1,\ldots d$
Intuitively, these assumption control how much  $\nabla \ell$  and  $\mathcal{F}_{\lambda}$  rotate the randomness  $\pmb{u}$ . (Notice that the assumptions are closely related to the matrix  $\mathrm{Cov}(g(\lambda; \pmb{u}), \pmb{u})$ , the covariance between  $g$  and  $\pmb{u}$ .) However, the peculiar aspect of these assumptions is that they are not implied by the convexity and smoothness of  $\ell$ . Especially, Assumption 3 strongly depends on the internals of  $\nabla \ell$ .
3.2 Smoothness of the Entropy
Under the linear parameterization, Domke (2020) has previously shown that the entropic regularizer term  $h$  is not smooth. This fact immediately implies the ELBO is not smooth. However, certain nonlinear conditioners do result in a smooth regularizer.
Lemma 1. If the diagonal conditioner  $\phi$  is  $L_{h}$ -log-smooth, then the entropic regularizer  $h(\lambda)$  is  $L_{h}$ -smooth.
Proof. See the full proof in page 24.
Example 1. The following diagonal conditioners result in a smooth entropic regularizer:
1. Let  $\phi(x) = \text{softplus}(x)$ . Then,  $h$  is  $L_h$ -smooth with  $L_h \approx 0.167096$ .
2. Let  $\phi (x) = \exp (x)$ . Then,  $h$  is  $L_{h}$ -smooth for arbitrarily small  $L_{h}$ .
This might initially suggest that diagonal conditioners are a promising way of making the ELBO globally smooth. Unfortunately, the properties of the energy,  $f$ , change unfavorably.
3.3 Smoothness of the Energy
Inapplicability of Existing Proof Strategy Previously, Domke (2020, Theorem 1) have proven that the energy is smooth when  $\phi$  is linear. The key step was to use Bessel's inequality based on the observation that the partial derivatives of the reparameterization function  $\mathcal{T}$  form unit bases in expectation. That is,
$$
\mathbb {E} \left\langle \frac {\partial \mathcal {T} _ {\lambda} (\boldsymbol {u})}{\partial \lambda_ {i}}, \frac {\partial \mathcal {T} _ {\lambda} (\boldsymbol {u})}{\partial \lambda_ {j}} \right\rangle = \mathbb {1} _ {i = j},
$$
where  $\mathbb{1}_{i = j}$  is an indicator function that is 1 only when  $i = j$  and 0 otherwise.
Unfortunately, when  $\phi$  is nonlinear, the partial derivatives  $\partial \mathcal{T}_{\lambda}(u) / \partial \lambda_{i}$  for  $i = 1,\dots ,p$  no longer form unit bases: while they are still orthogonal in expectation, the lengths change nonlinearly depending on  $\lambda$ . This leaves Bessel's inequality inapplicable. To circumvent this challenge, we establish a replacement for Bessel's inequality:
Lemma 2. Let  $\pmb{H}$  be a  $n \times n$  symmetric random matrix, where it is bounded as  $\| H \|_2 \leq L < \infty$  almost surely. Also, let  $\pmb{J}$  be an  $m \times n$  random matrix such that  $\| \mathbb{E} \pmb{J}^\top \pmb{J} \|_2 < \infty$ . Then,
$$
\left\| \mathbb {E} \boldsymbol {J} ^ {\top} \boldsymbol {H} \boldsymbol {J} \right\| _ {2} \leq L \left\| \mathbb {E} \boldsymbol {J} ^ {\top} \boldsymbol {J} \right\| _ {2}.
$$
Remark 1. By assuming that the joint log-likelihood  $\ell$  is smooth and twice-differentiable, we retrieve Theorem 1 of Domke (2020) by setting  $\pmb{J}$  to be the Jacobian of  $\mathcal{T}$ , and  $\pmb{H}$  to be the Hessian of  $\ell$  under reparameterization.
Remark 2. While our reparameterization function's partial derivatives still form orthogonal bases, they need not be; unlike Bessel's inequality, Lemma 2 does not require this. This implies that Lemma 2 is a strategy more general than Bessel's inequality.
Equipped with Lemma 2, we present our main result on smoothness:
Theorem 1. Let  $\ell$  be  $L_{\ell}$ -smooth and twice differentiable. Then, the following results hold: