SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 2: Scaling with dimensionality: evolution of forward (a) and reverse KL (b) with the number of gradient evaluations of the target distribution, which here is a Gaussian distribution with a dense  $D \times D$  covariance matrix. Each panel shows the results for a different dimensionality  $D$  (indicated in the title). Translucent lines show the scatter of 10 different runs, and the solid line shows the average. GSM-VI and BBVI use batch sizes of  $B = 2$ , while the batch sizes for natural gradient descent are shown in the legend.
to implementation details; moreover, we note that in real-world problems, the computation is often bottlenecked by evaluations of the target log density and its gradient. For completeness, we also measure running times of GSM-VI and BBVI for a full rank Gaussian target with 2048 dimensions using sample size of 1 in Jax after JIT-compiler. The GSM update takes  $230~\mathrm{ms}$  /update while BBVI takes  $227~\mathrm{ms}$  /update.
In the experiments with synthetic models in Sections 3.1, and 3.2 we have access to the true distribution; hence for these experiments, we estimate the forward KL divergence (FKL) empirically by  $\sum_{\pmb{\theta}_i\sim p(\pmb{\theta})}\log p(\pmb{\theta}_i) - \log q(\pmb{\theta}_i)$ . To reduce stochasticity, we always use the same pre-generated set of 1000 samples from the target distribution. In the experiments with real-world models in Section 3.3, we do not have access to the samples from the target distribution; for these experiments we monitor the negative ELBO. In all experiments, we show the results for 10 independent runs.
3.1 GSM-VI for Gaussian approximation
We begin by studying GSM-VI where the target distribution is also a multivariate Gaussian.
Scaling with dimensions. How does GSM-VI scale with respect to the dimensionality,  $D$ , of the latent variable? Figure 2 shows the convergence of FKL for GSM-VI and BBVI for different dimensionalities. Empirically, for GSM-VI, we find that the number of iterations required for convergence increases almost linearly with dimensionality. The scaling for BBVI is worse: even for small problems ( $D < 64$ ) it requires 100 times more iterations while also converging to a sub-optimal solution (as measured by the FKL metric). The figure also shows the evolution of the reverse KL divergence, and it is noteworthy that GSM-VI also outperforms BBVI in this metric.
For completeness, we also experimented on these problems<sup>3</sup> with methods for variational inference based on natural gradient descent (NGD-VI) [2, 22]. Natural gradient descent is a stochastic gradient method that uses the Fisher matrix as a preconditioner. We find NGD-VI is slower by factor of
${}^{3}$  Specifically,we use the package GMMVI with default settings except changing the batch size.
Figure 3: Impact of condition number: evolution of FKL with the number of gradient evaluations of the target distribution. The target is a 10-dimensional Gaussian albeit with a dense covariance matrix of different condition numbers  $c$  specified in the title of different panels. Translucent lines show the scatter of 10 different runs and the solid line shows the average.
Figure 4: Impact of non-Gaussianity: evolution of FKL with the number of gradient evaluations for Sinh-arcsinh distributions with 10-dimensional dense Gaussian as the base distribution. Gaussian distribution has  $s = 0$ ,  $t = 1$ . In the left panel, we vary skewness  $s$  while fixing  $t = 1$ , and in the right panel we vary the tail-weight  $t$  with skewness fixed to  $s = 0$ . Solid lines are the results for GSM, dashed for BBVI.
5-10x than GSM-VI. While NGD-VI outperforms BBVI, its performance is sensitive to the choice of batch size, with small batch sizes prone to divergence. Furthermore, we note that that per-iteration-complexity of NGD-VI is cubic in the dimensionality  $(D^3)$  as each iteration requires inverting a Hessian approximation; this is in contrast to the quadratic  $(D^2)$  per-iteration-complexities of GSM-VI and BBVI. For these reasons, we do not investigate NGD-VI further in this paper.
Impact of condition number. What is the impact of the shape of the target distribution? We again consider a Gaussian target distribution, but vary the condition number of its covariance matrix by fixing its smallest eigenvalue to 0.1 and scaling the largest eigenvalue to  $0.1 \times c$ . Figure 3 shows the results for a 10 dimensional Gaussian where we vary the condition number  $c$  from 1 to 1000. Convergence of GSM-VI seems to be largely insensitive to the condition number of the covariance matrix. BBVI on the other hand struggles with poorly conditioned problems, and it does not converge for  $c > 100$  even with 100 times more iterations than GSM.
3.2 GSM-VI for non-Gaussian target distributions
GSM-VI was designed to solve the exact score-matching equations in Eq. 3, but these equations only have a solution when the family of variational distributions contains the target distribution (see Lemma 2.1). Here we investigate the sensitivity of GSM-VI to this assumption by fitting non-Gaussian target distributions with varying degrees on non-Gaussianity. Specifically, we suppose that
Figure 5: Models from posteriordb: Convergence of the ELBO for four models with multivariate normal posteriors. We show results for 10 runs.
Figure 6: Models from posteriordb: Coverage of ELBO for four models with non-Gaussian posteriors. We show results for 10 runs.
the target has a multivariate Sinh-arcsinh normal distribution [17]
$$
\boldsymbol {z} \sim \mathcal {N} (\boldsymbol {\mu}, \boldsymbol {\Sigma}); \quad \boldsymbol {x} = \sinh \left(\frac {1}{t} [ \sinh^ {- 1} (\boldsymbol {z}) + s ]\right) \tag {10}
$$
where the scalar parameters  $s$  and  $t$  control, respectively, the skewness and the heaviness of the tails. Note that Eq. 10 reduces as a special case to a Gaussian distribution for the choices  $s = 0$  and  $t = 1$  of these parameters.
Figure 4 shows the result for fitting the variational Gaussian to a 10-dimensional Sinh-arcsinh normal distribution for different values of  $s$  and  $t$ . As the target departs further from Gaussianity, the quality of variational fit worsens for both GSM-VI (solid lines) and BBVI (dashed lines), but they converge to a fit of similar quality in terms of average FKL. GSM converges to this solution at least 10 times faster than BBVI. For highly non-Gaussian targets ( $s \geq 1$  or  $|t - 1| \geq 0.8$ ), we have found that GSM-VI does not converge to a fixed point, and it can experience oscillations that are larger in amplitude than BBVI; see for instance  $s = 1.8$  and  $t = 0.1$  on the left and right of Figure 4, respectively.
3.3 GSM-VI on real-world data
We evaluate GSM-VI for approximate on real-world data with 8 models from the posteriordb database [24]. The database provides the Stan code, data and reference posterior samples, and we use bridgestan to access the gradients of these models [6, 30]. We study the following statistical models, each with a different complexity and dimensionality (D): diamonds (generalized linear models, D=26), hudson-lynx-hare (differential equation dynamics, D=8), bball-drive (hidden Markov models, D=8) and arK (time-series, D=7), eight-schools-centered and non-centered (hierarchical meta-analysis, D=10), gp-pois-regr (Gaussian processes, D=13), low-dim-gauss-mix (Gaussian mixture, D=5).
For each model (except hudson-lynx-hare), we initialize the variational parameter  $\mu_0$  at the mode of the distribution, and we set  $\Sigma_0 = 0.1I_d$  where  $I_{d}$  is the identity matrix of dimension  $d$ .
For hudson-lynx-hare, we initialize the variational distribution as standard normal. We also experimented with other initializations. We find that they do not qualitatively change the conclusions, but can have larger variance between different runs.
We show the evolution of the ELBO for 10 runs of these models. Four of the models have posteriors that can be fit with multivariate normal distribution: diamonds, hudson-lynx-hare, bball-drive, and arK. Figure 5 shows the result for these models. The other models have non-Gaussian posteriors: eight-schools-centered, eight-schools-non-centered, gp-pois-regr,, and low-dim-gauss-mix. Figure 6 shows the results.
Overall, GSM-VI outperforms BBVI by a factor of  $10 - 100\mathrm{x}$ . When the target posterior is Gaussian, GSM-VI leads to more stable solutions. When the target is non-Gaussian, it converges to the same quality of variational approximation as BBVI. Finally, we note that while GSM-VI yields noisy estimates of the ELBO, it does yield stable estimates of the 1-D marginals and their moments.
In this paper we have proposed Gaussian score matching VI (GSM-VI), a new approach for VI when the variational family is multivariate Gaussian. GSM-VI does not explicitly minimize a divergence between the variational and target distribution; instead, it repeatedly enforces score-matching constraints with closed-form updates for the mean and covariance matrix of the variational distribution. This is in contrast to stochastic gradient based methods, such as in BBVI, that rely on first-order Taylor approximations of their objective function.
GSM-VI is implemented in an open-source Python code at https://github.com/modichirag/GSM-VI.
Unlike approaches that are rooted in stochastic gradient descent, GSM-VI does not require the tuning of step-size hyper-parameters. Rather, it is able to adaptively make large jumps in the initial iterations (see Fig. 1a) and make smaller adjustments as the approximation converges to the target. GSM-VI has only one free parameter, the batch size, and we found a batch-size of 2 to perform competitively across all experiments. Another choice is how to initialize the variational distribution. For the experiments in this paper, we initialized the covariance matrix as the identity matrix, but additional gains could potentially be made with more informed choices derived from a Laplace approximation or L-BFGS Hessian approximation [35].
We evaluated the performance of GSM-VI on synthetic targets and real-world models from posteriordb. In general, we found that it requires 10-100x fewer gradient evaluations than BBVI for the target distribution to converge. When the target distribution is itself multivariate Gaussian, we observed that GSM-VI scales nearly linearly with dimensionality, which is significantly better than BBVI, and that GSM-VI is fairly insensitive to the condition number of the target covariance matrix. We also found that GSM-VI converges more quickly than BBVI—and also to a solution with a larger ELBO, which is surprising given that BBVI explicitly maximizes the ELBO.
An important avenue for future work is to provide a proof that GSM-VI converges. We note that good convergence results have been obtained for analogous methods that project onto interpolation equations for empirical risk minimization. For instance the Stochastic Polyak Step achieves the min-max optimal rates of convergence for SGD [23]. Note that convergence of VI is a generally challenging problem, with no known rates of convergence even for BBVI [10, 11]. This and others directions are left to future work.
[1] A. Agrawal, D. R. Sheldon, and J. Domke. Advances in black-box vi: normalizing flows, importance weighting, and optimization. Neural Information Processing Systems, 2020.
[2] O. Arenz, P. Dahlinger, Z. Ye, M. Volpp, and G. Neumann. A unified perspective on natural gradient variational inference with gaussian mixture models. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https://openreview.net/forum?id=tLBjsX4tjs.
[25] Charles C. Margossian. A review of automatic differentiation and its efficient implementation. Wiley interdisciplinary reviews: data mining and knowledge discovery, 9, 3 2019. doi: 10.1002/WIDM.1305.
[26] T. Minka. Expectation propagation for approximate Bayesian inference. In Uncertainty in Artificial Intelligence (UAI), pages 362-369, 2001.
[27] C. Naesseth, F. Lindsten, and D. Blei. Markovian score climbing: Variational inference with KL(p || q). In Neural Information Processing Systems, 2020.
[28] R. Ranganath, S. Gerrish, and D. M. Blei. Black Box Variational Inference. art. arXiv:1401.0118, December 2013.
[29] R. Ranganath, J. Altosaar, D. Tran, and D. Blei. Operator variational inference. In Neural Information Processing Systems. 2016.
[30] E. Roualdes, B. Ward, S. Axen, and B. Carpenter. BridgeStan: Efficient in-memory access to Stan programs through Python, Julia, and R, March 2023. URL https://github.com/roualdes/bridgestan.
[31] J. Salvatier, Wiecki T., and C. Fonnesbeck. Probabilistic programming in Python using PyMC3. PeerJ Computer Science, 2016.
[32] U. Seljak and B. Yu. Posterior inference unchained with EL_2O. art. arXiv:1901.04454, January 2019.
[33] M. W., M. R. Andersen, A. Vehtari, and J. H. Huggins. Robust, automated, and accurate black-box variational inference. arXiv:2203.15945, 2022.
[34] Y. Yang, R. Martin, and H. Bondell. Variational approximations using Fisher divergence. arXiv e-prints, art. arXiv:1905.05284, May 2019.
[35] L. Zhang, B. Carpenter, A. Gelman, and A. Vehtari. Pathfinder: Parallel quasi-newton variational inference. Journal of Machine Learning Research, 23(306):1-49, 2022.
A Proof of Lemma 2.1
if and only if  $\boldsymbol{w}^*$  also satisfies Eq. 1.
Proof. (1)  $\Longrightarrow$  (3): Differentiating both sides of (1) in  $\theta$  gives
$$
\begin{array}{l} \nabla_ {\theta} \log q _ {\boldsymbol {w} ^ {*}} (\boldsymbol {\theta}) = \nabla_ {\boldsymbol {\theta}} \log p (\boldsymbol {\theta} | \boldsymbol {x}) = \nabla_ {\boldsymbol {\theta}} \left(\log p (\boldsymbol {\theta}, \boldsymbol {x}) - \log p (\boldsymbol {x})\right) \\ = \nabla_ {\theta} \log p (\boldsymbol {\theta}, \boldsymbol {x}), \quad \forall \boldsymbol {\theta} \in \Theta . \\ \end{array}
$$
(3)  $\Longrightarrow$  (1): Let  $\pmb{\theta}_0$  be any arbitrary point in  $\Theta$ . Because  $\Theta$  is path-connected, every  $\pmb{\theta} \in \Theta$  is connected to  $\pmb{\theta}_0$  via a differentiable path  $\mathbf{r}(t)$  where  $\mathbf{r}(0) = \pmb{\theta}_0$  and  $\mathbf{r}(1) = \pmb{\theta}$ . Integrating both sides of (3) along this path  $\mathbf{r}(t)$ , using again that  $\nabla_{\theta} \log p(\pmb{\theta} | \pmb{x}) = \nabla_{\theta} \log p(\pmb{\theta}, \pmb{x})$ , and using the Fundamental Theorem of Calculus gives
$$
\begin{array}{l} \log q _ {\boldsymbol {w} ^ {*}} (\boldsymbol {\theta}) - \log q _ {\boldsymbol {w} ^ {*}} (\boldsymbol {\theta} _ {0}) = \int_ {0} ^ {1} \left\langle \nabla_ {\boldsymbol {\theta}} \log q _ {\boldsymbol {w} ^ {*}} (\mathbf {r} (t)), \mathbf {r} ^ {\prime} (t) \right\rangle d t \\ = \int_ {0} ^ {1} \left\langle \nabla_ {\theta} \log p ((\mathbf {r} (t)) | \boldsymbol {x}), \mathbf {r} ^ {\prime} (t) \right\rangle d t \\ = \log p (\boldsymbol {\theta} | \boldsymbol {x}) - \log p (\boldsymbol {\theta} _ {0} | \boldsymbol {x}), \quad \forall \boldsymbol {\theta} \in \Theta . \\ \end{array}
$$
Rearranging and defining  $C \coloneqq \log q_{\boldsymbol{w}^*}(\boldsymbol{\theta}_0) - \log p(\boldsymbol{\theta}_0|\boldsymbol{x})$  gives
$$
\log q _ {\boldsymbol {w} ^ {*}} (\boldsymbol {\theta}) = \log p (\boldsymbol {\theta} | \boldsymbol {x}) + C, \quad \forall \boldsymbol {\theta} \in \Theta ,
$$
By exponentiating both sides and integrating in  $\theta$  over  $\Theta$  we have that
$$
1 = \int_ {\boldsymbol {\theta} \in \Theta} q _ {\boldsymbol {w} ^ {*}} (\boldsymbol {\theta}) d \boldsymbol {\theta} = e ^ {C} \int_ {\boldsymbol {\theta} \in \Theta} p (\boldsymbol {\theta} | \boldsymbol {x}) d \boldsymbol {\theta} = e ^ {C},
$$
where we used that  $\Theta$  contains the support of both  $p(\boldsymbol{\theta}|\boldsymbol{x})$  and  $q_{\boldsymbol{w}^*}(\boldsymbol{\theta})$ . Consequently  $C = 0$ , which gives our result.
Here we give the proof for Theorem 2.2. We also re-introduce the theorem with a simplified notation, where we use  $(\pmb{\mu}_0,\pmb{\Sigma}_0)$  to denote the mean and covariance at the previous time step of the method, thus dropping the iteration counter  $t$ .
Theorem B.1. (GSM updates) Let  $p(\theta, \boldsymbol{x})$  be given for some  $\theta \in \mathbb{R}^d$ , and let  $q_0(\theta)$  and  $q(\theta)$  be the multivariate normal distributions, respectively, with means  $\mu_0$  and  $\mu$  and covariance matrices  $\Sigma_0$  and  $\Sigma$ . We seek the distribution
$$
\underset {\boldsymbol {\mu}, \boldsymbol {\Sigma} = \boldsymbol {\Sigma} ^ {\top}} {\arg \min } \left[ \mathrm {K L} \left(q _ {0}, q\right) \right] \quad \text {s u c h t h a t} \quad \nabla_ {\boldsymbol {\theta}} \log q (\boldsymbol {\theta}) = \nabla_ {\boldsymbol {\theta}} \log p (\boldsymbol {\theta}, \boldsymbol {x}). \tag {11}
$$
As shorthand, let  $\pmb{g} \coloneqq \nabla_{\pmb{\theta}} \log p(\pmb{\theta}, \pmb{x})$ , and let  $\rho$  be the positive root of the quadratic equation
$$
\rho (1 + \rho) = \boldsymbol {g} ^ {\top} \boldsymbol {\Sigma} _ {0} \boldsymbol {g} + \left[ (\boldsymbol {\mu} _ {0} - \boldsymbol {\theta}) ^ {\top} \boldsymbol {g} \right] ^ {2}. \tag {12}
$$
Then the solution is given by the following closed-form updates:
$$
\boldsymbol {\mu} = \boldsymbol {\mu} _ {0} + \frac {1}{1 + \rho} \left[ \mathbf {I} - \frac {(\boldsymbol {\mu} _ {0} - \boldsymbol {\theta}) \boldsymbol {g} ^ {\top}}{1 + \rho + (\boldsymbol {\mu} _ {0} - \boldsymbol {\theta}) ^ {\top} \boldsymbol {g}} \right] \boldsymbol {\Sigma} _ {0} (\boldsymbol {g} - \nabla_ {\boldsymbol {\theta}} \log q _ {0} (\boldsymbol {\theta})) ， \tag {13}
$$
$$
\boldsymbol {\Sigma} = \boldsymbol {\Sigma} _ {0} + (\boldsymbol {\mu} _ {0} - \boldsymbol {\theta}) (\boldsymbol {\mu} _ {0} - \boldsymbol {\theta}) ^ {\top} - (\boldsymbol {\mu} - \boldsymbol {\theta}) (\boldsymbol {\mu} - \boldsymbol {\theta}) ^ {\top}. \tag {14}
$$
Furthermore, if  $\Sigma_0$  is symmetric positive definite then so is  $\Sigma$ .
Proof. The constraint in this optimization is given by
$$
\boldsymbol {g} = \nabla_ {\boldsymbol {\theta}} \log q (\boldsymbol {\theta}) \tag {15}
$$
$$
= \nabla_ {\boldsymbol {\theta}} \left[ - \frac {1}{2} (\boldsymbol {\theta} - \boldsymbol {\mu}) \boldsymbol {\Sigma} ^ {- 1} (\boldsymbol {\theta} - \boldsymbol {\mu}) - \frac {1}{2} \log \left((2 \pi) ^ {d} | \boldsymbol {\Sigma} |\right) \right] \tag {16}
$$
$$
= - \boldsymbol {\Sigma} ^ {- 1} (\boldsymbol {\theta} - \boldsymbol {\mu}). \tag {17}
$$
The KL divergence is given by
$$
\mathrm {K L} \left(q _ {0}, q\right) = \frac {1}{2} \left\{\operatorname {t r} \left[ \boldsymbol {\Sigma} ^ {- 1} \boldsymbol {\Sigma} _ {0} \right] + \log \frac {| \boldsymbol {\Sigma} |}{| \boldsymbol {\Sigma} _ {0} |} + \left(\boldsymbol {\mu} - \boldsymbol {\mu} _ {0}\right) ^ {\top} \boldsymbol {\Sigma} ^ {- 1} \left(\boldsymbol {\mu} - \boldsymbol {\mu} _ {0}\right) - d \right\}. \tag {18}
$$
Dropping irrelevant terms from the optimization, we obtain the Lagrangian
$$
\mathcal {L} (\boldsymbol {\mu}, \boldsymbol {\Sigma}, \boldsymbol {\lambda}) = \frac {1}{2} \left\{\operatorname {t r} \left[ \boldsymbol {\Sigma} ^ {- 1} \boldsymbol {\Sigma} _ {0} \right] - \log | \boldsymbol {\Sigma} ^ {- 1} | + \left(\boldsymbol {\mu} - \boldsymbol {\mu} _ {0}\right) ^ {\top} \boldsymbol {\Sigma} ^ {- 1} \left(\boldsymbol {\mu} - \boldsymbol {\mu} _ {0}\right) \right\} + \boldsymbol {\lambda} ^ {\top} \left(\boldsymbol {g} - \boldsymbol {\Sigma} ^ {- 1} \left(\boldsymbol {\mu} - \boldsymbol {\theta}\right)\right). \tag {19}
$$
It is easier to optimize the matrix  $\Sigma^{-1}$  instead of  $\Sigma$ . We can enforce the symmetry of  $\Sigma^{-1}$  by writing
$$
\boldsymbol {\Sigma} ^ {- 1} = \frac {1}{2} \left(\boldsymbol {\Phi} + \boldsymbol {\Phi} ^ {\top}\right) \tag {20}
$$
and performing an unconstrained optimization over  $\Phi$ . With respect to the latter, the gradients of the Lagrangian are given by
$$
\frac {\partial \mathcal {L}}{\partial \Phi_ {i j}} = \sum_ {k l} \left(\frac {\partial \mathcal {L}}{\partial \Sigma_ {k l} ^ {- 1}}\right) \left(\frac {\partial \Sigma_ {k l} ^ {- 1}}{\partial \Phi_ {i j}}\right) = \sum_ {k l} \left(\frac {\partial \mathcal {L}}{\partial \Sigma_ {k l} ^ {- 1}}\right) \left(\frac {1}{2} \delta_ {k i} \delta_ {l j} + \frac {1}{2} \delta_ {k j} \delta_ {l i}\right) = \frac {1}{2} \left(\frac {\partial \mathcal {L}}{\partial \Sigma_ {i j} ^ {- 1}} + \frac {\partial \mathcal {L}}{\partial \Sigma_ {j i} ^ {- 1}}\right). \tag {21}
$$
Next we examine where the gradients of the Lagrangian vanish:
$$
\mathbf {0} = \frac {\partial \mathcal {L}}{\partial \boldsymbol {\mu}} \quad \Longrightarrow \quad \mathbf {0} = \boldsymbol {\Sigma} ^ {- 1} (\boldsymbol {\mu} - \boldsymbol {\mu} _ {0}) - \boldsymbol {\Sigma} ^ {- 1} \boldsymbol {\lambda} \quad \Longrightarrow \quad \boxed {\boldsymbol {\lambda} = \boldsymbol {\mu} - \boldsymbol {\mu} _ {0}} \tag {22}
$$
$$
\mathbf {0} = \frac {\partial \mathcal {L}}{\partial \boldsymbol {\lambda}} \quad \Longrightarrow \quad \mathbf {0} = g - \Sigma^ {- 1} (\boldsymbol {\mu} - \boldsymbol {\theta}) \quad \Longrightarrow \quad \boxed {\boldsymbol {\mu} - \boldsymbol {\theta} = \Sigma \boldsymbol {g}} \tag {23}
$$
$$
\begin{array}{l} \mathbf {0} = \frac {\partial \mathcal {L}}{\partial \boldsymbol {\Phi}} \quad \Longrightarrow \quad \mathbf {0} = \boldsymbol {\Sigma} _ {0} - \boldsymbol {\Sigma} + (\boldsymbol {\mu} - \boldsymbol {\mu} _ {0}) (\boldsymbol {\mu} - \boldsymbol {\mu} _ {0}) ^ {\top} - [ \boldsymbol {\lambda} (\boldsymbol {\mu} - \boldsymbol {\theta}) ^ {\top} + (\boldsymbol {\mu} - \boldsymbol {\theta}) \boldsymbol {\lambda} ^ {\top} ], (24) \\ \Rightarrow \boxed {\Sigma = \Sigma_ {0} + (\boldsymbol {\mu} - \boldsymbol {\mu} _ {0}) (\boldsymbol {\mu} - \boldsymbol {\mu} _ {0}) ^ {\top} - \boldsymbol {\lambda} (\boldsymbol {\mu} - \boldsymbol {\theta}) ^ {\top} - (\boldsymbol {\mu} - \boldsymbol {\theta}) \boldsymbol {\lambda} ^ {\top}} (25) \\ \end{array}
$$
We claim that these equations (though nonlinear) can be solved in closed form. The first step is to eliminate  $\lambda$  from eq. (25) using eq. (22). In this way we find
$$
\begin{array}{l} \boldsymbol {\Sigma} = \boldsymbol {\Sigma} _ {0} + (\boldsymbol {\mu} - \boldsymbol {\mu} _ {0}) (\boldsymbol {\mu} - \boldsymbol {\mu} _ {0}) ^ {\top} - (\boldsymbol {\mu} - \boldsymbol {\mu} _ {0}) (\boldsymbol {\mu} - \boldsymbol {\theta}) ^ {\top} - (\boldsymbol {\mu} - \boldsymbol {\theta}) (\boldsymbol {\mu} - \boldsymbol {\mu} _ {0}) ^ {\top} (26) \\ = \boldsymbol {\Sigma} _ {0} - \boldsymbol {\mu} \boldsymbol {\mu} ^ {\top} + \boldsymbol {\mu} \boldsymbol {\theta} ^ {\top} + \boldsymbol {\theta} \boldsymbol {\mu} ^ {\top} + \boldsymbol {\mu} _ {0} \boldsymbol {\mu} _ {0} ^ {\top} - \boldsymbol {\mu} _ {0} \boldsymbol {\theta} ^ {\top} - \boldsymbol {\theta} \boldsymbol {\mu} _ {0} ^ {\top} (27) \\ = \boldsymbol {\Sigma} _ {0} + (\boldsymbol {\mu} _ {0} - \boldsymbol {\theta}) (\boldsymbol {\mu} _ {0} - \boldsymbol {\theta}) ^ {\top} - (\boldsymbol {\mu} - \boldsymbol {\theta}) (\boldsymbol {\mu} - \boldsymbol {\theta}) ^ {\top}. (28) \\ \end{array}
$$
It is worth highlighting the form of this equation:
$$
\boldsymbol {\Sigma} = \boldsymbol {\Sigma} _ {0} + (\boldsymbol {\mu} _ {0} - \boldsymbol {\theta}) (\boldsymbol {\mu} _ {0} - \boldsymbol {\theta}) ^ {\top} - (\boldsymbol {\mu} - \boldsymbol {\theta}) (\boldsymbol {\mu} - \boldsymbol {\theta}) ^ {\top}
$$
This is a simple rank-two update for  $\Sigma$ . Note that  $\Sigma = \Sigma_0$  if  $\mu = \mu_0$ ; also, the solution for  $\Sigma$  is determined by the solution for  $\mu$ .
Ultimately we must solve for  $\pmb{\mu}$ , but first it is useful to solve for the intermediate quantity  $\pmb{g}^{\top}\pmb{\Sigma}\pmb{g} > 0$ . From eq. (28), we obtain
$$
\boldsymbol {g} ^ {\top} \boldsymbol {\Sigma} \boldsymbol {g} = \boldsymbol {g} ^ {\top} \boldsymbol {\Sigma} _ {0} \boldsymbol {g} + \left[ (\boldsymbol {\mu} _ {0} - \boldsymbol {\theta}) ^ {\top} \boldsymbol {g} \right] ^ {2} - \left[ (\boldsymbol {\mu} - \boldsymbol {\theta}) ^ {\top} \boldsymbol {g} \right] ^ {2}, \tag {29}
$$
and from eq. (23), we obtain