SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 7: The above figures show the accuracy and computation time of each method across varying components, orders, and dimensions in approximating a multivariate mixture T distribution with a GMM. Our method consistently improves accuracy significantly. Higher order approximations do not increase computation time (bottom middle), and even low order approximations provide substantial improvement (top middle). Most accuracy improvements are achieved with a small number of components, unlike NPV (top left) which continues to need higher number of components to see good accuracy return. Our computation time increases with dimension due to less salable cross-entropy approximation with Gauss-Hermite quadrature (bottom right) however the guaranteed convergence of the approximation seems to have a drastic improvement on accuracy (top right).
A.6 Nonparametric Variational Inference
Consider a target density  $p(x,\mathcal{D})$  with latent variables  $x$  and observations  $\mathcal{D}$ . The NPV approach [11] optimizes the evidence lower bound (ELBO),
$$
\log p (x) \geq \max  _ {q} H _ {q} (p (x, \mathcal {D})) - H _ {q} (q (x)) \equiv \mathcal {L} (q) \tag {37}
$$
w.r.t. a  $m$ -component GMM variational distribution  $q(x) = \frac{1}{N}\sum_{i=1}^{m}\mathcal{N}(x|\mu_i,\sigma_i^2I_d)$ . The GMM entropy lacks a closed-form so NPV applies Jensen's lower bound as an approximation,
$$
H _ {q} (q (x)) = - \mathbb {E} _ {q} [ \log (q (x)) ] \geq - \sum_ {i = 1} ^ {M} w _ {i} \log \left(\mathbb {E} _ {q _ {i}} [ q (x) ]\right) = \hat {H} _ {q} ^ {J} (q (x)) \tag {38}
$$
The cross entropy also lacks a closed-form, so NPV approximates this term using the analogous Huber et al. Taylor approximation. Specifically, NPV expands the log density around the means of each GMM component as,
$$
H _ {q} (p (x)) \approx - \sum_ {i = 1} ^ {M} w _ {i} \sum_ {n = 0} ^ {N} \frac {\nabla^ {2} \log \left(p \left(\mu_ {i}\right)\right)}{n !} \mathbb {E} _ {q _ {i}} [ (x - \mu_ {i}) ^ {n} ] = \hat {H} _ {N, q} ^ {H} (p (x)) \tag {39}
$$
However, Eqn. (20) is subject to the divergence criterion of Theorem 3.1 if  $2p(\mu_i)\leq \max (p(x))$ . This approximation is often known as the multivariate delta method for moments. The authors use these approximations of the entropy and cross entropy to create the following two approximations of the ELBO.
$$
\mathcal {L} _ {1} (q) = \hat {H} _ {1, q} ^ {H} (p (x)) - \hat {H} _ {q} ^ {J} (q (x)) \quad \mathcal {L} _ {2} (q) = \hat {H} _ {2, q} ^ {H} (p (x)) - \hat {H} _ {q} ^ {J} (q (x)) \tag {40}
$$
Gershman et al. (2012) use the two approximations because optimizing  $\hat{H}_{2,q}^{H}(p(x))$  with respect to the mean components,  $\mu_{i}$  requires computing a third order, multivariate derivative of  $\log(p(x))$  which is computationally expensive. The authors iterate between optimizing the mean components,  $\mu_{i}$ , using  $\mathcal{L}_1(q)$  and optimizing the variance components,  $\sigma_i^2$ , using  $\mathcal{L}_2(q)$  until the second approximate appropriately converges  $\delta \mathcal{L}_2(q) < .0001$ .
In our approach, we will highlight and address three problems with the NPV algorithm; the potential divergence of  $\hat{H}_{N,q}^{H}(p(x))$ , the inconsistent ELBO approximations, and the poor estimation of the GMM entropy via  $\hat{H}_q^J (q(x))$ . To address the potential divergence of  $\hat{H}_{N,q}^{H}(p(x))$ , we will take motivation from the results found in [23] and use a 2 point Gauss-Hermite quadrature method to approximate  $H_{q}(p(x))$ . This method will be a limiting factor in scaling the NPV algorithm in dimension, however it guarantees that the cross-entropy approximation will not diverge. This alteration leads to a solution for the inconsistency of the ELBO approximations. Since the quadrature method does not require any derivatives of  $\log (p(x))$  w.r.t. the mean components of the GMM, we can now optimize the means and variances simultaneously on the same ELBO approximation. Finally, Jensen's inequality is a very poor approximation for entropy in general, instead we will use the three methods we have introduced, Taylor, Taylor limit, and Legendre, as the GMM entropy approximations for higher accuracy. Fig. 4 shows an approximation of a two dimensional, three component mixture Student T distribution using a fiver component GMM in the traditional NPV, using our Taylor approximation and our Legendre approximation.
The results, as seen in Fig. 5, highlight the accuracy and computation time of each method versus the number of components used, the order of our polynomial approximation used, and the dimension of the GMM. The accuracy is the same as seen in Section 6, the new information here is the computation time of each method. We see that the order of the method has very little impact on the computation time of our algorithm. We even see most of the accuracy improvement at around order 2 or 3 so staying in a low order approximation seems advisable. We do see that the component does increase our time by a bit compared to that of traditional NPV. The source of the computation time increase in our methods comes from more iterations in the optimization. Each evaluation of the (ELBO) approximator are near equivalent but since we are converging to a better optimum, this take more iteration steps than NPV. Finally, we see that dimension does have a large impact on our method. The source of this computation increase come from our Gauss-Hermite approximation of the cross entropy. The number of quadrature points used 2 per dimension  $D$ , so we are computing with  $2^{D}$  points, which clearly scale poorly with dimension. We are seeking better ways of computing the cross-entropy of a GMM with any non-GMM function that is both convergent and computationally fast, however this was not the focus of the paper and no method was considered yet.