On Convergence of Polynomial Approximations to the Gaussian Mixture Entropy
Authors: Caleb Dahlke
Jason Pacheco
Venue: NeurIPS 2023 poster (2023)
Keywords: entropy
Gaussian mixture model
uncertainty quantification
approximate inference
TL;DR: We provide analysis on convergence/divergence criterion for polynomial approximation of GMM entropy.

Abstract: Gaussian mixture models (GMMs) are fundamental to machine learning due to their flexibility as approximating densities. However, uncertainty quantification of GMMs remains a challenge as differential entropy lacks a closed form.  This paper explores polynomial approximations, specifically Taylor and Legendre, to the GMM entropy from a theoretical and practical perspective. We provide new analysis of a widely used approach due to Huber et al.(2008) and show that the series diverges under simple conditions.  Motivated by this divergence we provide a novel Taylor series that is provably convergent to the true entropy of any GMM.  We demonstrate a method for selecting a center such that the series converges from below, providing a lower bound on GMM entropy. Furthermore, we demonstrate that orthogonal polynomial series result in more accurate polynomial approximations. Experimental validation supports our theoretical results while showing that our method is comparable in computation to Huber et al. We also show that in application, the use of these polynomial approximations, such as in Nonparametric Variational Inference by Gershamn et al. (2012), rely on the convergence of the methods in computing accurate approximations. This work contributes useful analysis to existing methods while introducing a novel approximation supported by firm theoretical guarantees.

On Convergence of Polynomial Approximations to the Gaussian Mixture Entropy
Caleb Dahlke Department of Mathematics University of Arizona
Jason Pacheco
Department of Computer Science University of Arizona
Abstract
Gaussian mixture models (GMMs) are fundamental to machine learning due their flexibility as approximating densities. However, uncertainty quantification of GMMs remains challenge differential entropy lacks closed form. This paper explores polynomial approximations, specifically Taylor and Legendre, to the GMM entropy from theoretical and practical perspective. We provide new analysis of widely used approach due to Huber et al. (2008) and show that the series diverges under simple conditions. Motivated by this divergence we provide novel Taylor series that is provably convergent to the true entropy of any GMM. We demonstrate method for selecting a center such that the series converges from below, providing lower bound on GMM entropy. Furthermore, we demonstrate that orthogonal polynomial series result in more accurate polynomial approximations. Experimental validation supports our theoretical results while showing that our method is comparable show that in application, the use of these polynomial approximations, such as in Nonparametric Variational Inference, rely on the convergence of the methods in computing accurate approximations. This work contributes useful analysis to existing methods while introducing novel approximations supported by firm theoretical guarantees.
computation to Huber et al. We also
Introduction
Entropy is a natural measure of uncertainty and is fundamental to many information--thoortic quanti- ties such mutual information (MI) and Kullback-Leibbe (KL) divergence [8]. As result, entropy plays a key role in many problems of ML including model interpretation [7], feature selection [6], and representation learning [27]. often used in the data acquisition process as active learn- ing [25, 26), Bayesian optimal experimental design [17, 5, 3J. and Bayesian optimization [13]. Yet,
despite its important role entropy is difficult to calculate in general.
One such case the Gaussian mixture model (GMM), where entropy lacks a closed form and the focus of this paper. GMMs are fundamental to machine learning and statisties due to their property as universal density approximators [18]. However, the lack of a closed-form entropy requires approxima- tion, often via Monte Carlo expectation. Such stochastic estimates can undesirable computation becomes coupled with sample size and deterministic approach is often preferred. Simple determinis- tic bounds be calculated via Jensen's 's inequality or Gaussian moment matching [14]. Such bounds are often too loose to be useful, leading to other options such as variational approximations (21, 10] and neural network-bas based approximation [2]. Yet, these deterministic estimators do not allow straightforward tradeoff' of computation and accuracy as in the Monte Carlo setting.
Polynomial series approximations are both deterministic and provide mechanism for computation- accuracy tradeoff by varying the polynomial degree. In this paper we focus on three such polynomial approximations of the GMM entropy. We begin with the widely used approximation of Huber et al. (2008). While this approximation yields good empirical accuracy in many settings, a proof of
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

convergence is lacking, In this work we show that the Huber et al. approximation in fact does not converge general, and we provide a divergence criterion (Theorem 3.1). In response to the divergent behavior, we propose two alternative polynomial approximations, a Taylor and Legendre series approximation of GMM entropy that are provably convergent. We establish Theorem 4.2 and Theorem 4.5 that each series converges everywhere under conditions on the center point or interval, respectively. In Theorem 4.4 we provide a simple mechanism for choosing a value to ensure that these series converge everywhere. We additionally establish, in Theorem 4.3, that our Taylor approximation is a convergent lower bound on the true entropy for any finite poynomial order.
The complexity of both Huber et al. and our proposed methods have similar computation largely driven by polynomial order. To address this we propose an approximation that estimates the higher- order terms by fitting polynomial regression. This approach requires the evaluation of only three consecutive polynomial orders to approximate higher order series. In this way we can obtain more accurate estimates without the computational overhead of evaluating higher order polynomial terms.
We conclude with an empirical comparison of all polynomial approximations that produce the diver- gent behavior of the Huber et al. approximation while our propsed methods maintain convergence. We also compare accuracy and compuation time for each method accross a varaitey of dimensions, number of GMM components, and polynomial orders. Finally, we show an application of our methods in Nonparametric Variational Inference [11] where the guarantees of convergence play large role in the accuracy of posterior approximation via GMMs.
2 Preliminaries
We briefly introduce required notation and concepts, beginning with definition of the Gaussian mixture entropy. We will highlight the challenges that preclude efficient computation of entropy. We conclude by defining notation that will be used for discussion of polynomial approximations.
2.1
Gaussian Mixture Entropy
The differential entropy of a continuous-valued random vector x E Rd with a probability density function p(z) is given by,
H(p(x)) =
= H(p(z))=-
p(x) logp(r)dr E[ B[-logp(x))
(1)
The differential entropy is in [- co] for continuous random variables. It is a measure of uncertainty in the random variable in the sense that its minimum is achieved when there is uncertainty in the random vector, i.e., a Dirac delta, and approaches the maximum as the density becomes uniformly distributed.
Gaussian mixtures are ubiquitous in statistics and machine learning due to their property as universal density approximators [18]. However, despite this flexibility, the entropy of Gaussian mixture requires computing the expectation of the log-sum operator, which lacks a closed form. Many approximations and bounds are used in practice. A simple upper bound is given by the entropy of single Gaussian with the same mean and covariance as the mixture [14], and a lower bound can be obtained by Jensen's 's inequality. Though efficient, these bounds are very loose in practice, leading to more accurate methods being used, such as various Monte Carlo approximations, deterministic sampling [12], and numerous variational bounds and approximations [21, 10].
2.2 Taylor Polynomials
In this paper we explore entropy approximation using Taylor polynomials. The -order Taylor polynomial of a function f(x) with evaluation point e given by,
" ff)) = Σ (())(c) (a (x-c)" n!
(2)
where (c) denotes the th derivative of f evaluated at point e. The Taylor series has region of convergence which determines the range of x-values where the series accurately represents the original function. It depends on the behavior of the function and its derivatives at the expansion point. Analyzing the region of convergence is crucial for ensuring the validity of the Taylor series approximation. Various convergence tests, such as the ratio test, help determine the x-values where the Taylor series provides an accurate approximation.

---

2.3 Orthogonal Polynomials
Taylor series are versatile approximations, however predominately behave well near the center point chosen. We ideally would like an approximation that performs well across a range of values. To achieve this, we consider series approximation via orthogonal polynomials. A set of orthogonal polynomials on the range (a, bi is an infinite sequence of polynomials Po(x), (i(x).... where P, (x) an n th degree polynomial and for any pair of polynomials satisfies
(P(()), - [" P.())))(r)d= = c..j
where 6ij is the Kronecker delta function and is some constant. Orthogonal polynomials can be used to approximate a function, f(x), on their interval, [a, bj. by finding the projection f(x) onto each polynomial in the series (z)x).
00
(J(z), P.(o)) P((x)
f(x) Σ (R(x),P(({))
Any appropriate choice of orthogonal polynomials can be used. One might be interested in consid- ering the Chebyshev polynomials for their property of minimizing interpolation or Legendre polynomials for their versatility and ease of computation.
3 Convergence of Polynomial Approximations
To estimate the entropy H(p) = F.p[-loge [og[=))(p(=))) using a polynomial approximation one may ap- proximate either the log-density log(p(z:) or just the logarithm log(v). We will show that estimating log(p(z)) has convergence issues and that it can be complicated to compute due to tensor arithmetic in higher dimensions. Both of these issues will be addressed by simply approximating log(y) and computing the exact p(x). All proof's are deferred to the Appendix for space.
3.1 Divergence of Huber et al. Approximation
We begin our exploration with widely used approximation of the GMM entropy due to [15]. Let p(x) be GMM and the IMg-GMM h(z) = log(p(x)). Huber et al. provides Taylor series approximation of the GMM entropy given by.
(im)
(ps) (x
log(p(x)) =
ps)",
n
The series is M individual Taylor series evaluated at each component mean, Pa. The equality in Eqn. (5) only holds if the series converges, which we will show is not the case in general.
w;N(x Theorem 3.1 (Divergence Criterion for Huber et al.). Let p(z) = = consider the Taylor series presented in Eqn. (5). If any mean component, Pi satisfies the condition p(u;) < )max(p(x)), then the series diverges, otherwise converges.
Ju, E.) and
Theorem 3.1 provides with the condition that Huber et al.'s approximation Eqn. (5) will diverge. This means that the entropy approximation will be inaccurate for any GMM with any of its modes less than half the probability of any other point, illustrated in Fig. 1.
3.2 Taylor Series Approximation of the Logarithm
Motivated by the divergence of the previous Taylor series we propose a different approach that is provably convergent. While Huber et al. perform Taylor decomposition of the log- log-GMM PDF, our approach decomposes only the log(y) function using Taylor series of the function centered about the point a. It is well-knownn that this series converges for values y al [28] and is given by,
(-1) nan
log(v) log(a)
a)).
(6)
Note the change of e to a as the Taylor series center. This change highlights the difference in function domains. In particular, the former series is computed on values of the random vector whereas ours computed on the PDF y = p(z). Choosing any center a > mpxxx())) will ensure that the series converges everywhere.
3

---

0.6
GMM
0.5
Huber et al.
logGMM
-2
GMM
0.4
Our Taylor
-4
0.3
Our Legendre
0.2
-8
0.1
-10
-12
-6 -4
-2
0
2
-2
0
2
Figure 1: Divergence of Huber et al. and convergence of our polynomial series approximation plotted for the Gaussian mixture, p(x) (5, (z: -3 665. 02. In the left graph, the log-GMM plotted, which is what each series is defined for. right plot the exponential how each converge in the more familiar framework of a GMM. Notice that the Huber el al. is centered on the first component mean Pa = -3 and diverges around the mean of the second component μl 0 as supported by Theorem 3.1 since the mode μn less than half the probability at the mode μ22, Both of methods convergent, the Taylor series bound (Theorem 4.3) while the Legendre series has a lower global error.
the series so we can see
Lemma 3.2 (Convergent Taylor Series of Log). Ifa max((((x)), then for all
20
(-1)a-1
n
log(p(x)) log(a) +
(-a)" ((a))~>p(x)
na"
The proof of Lemma 3.2 is a simple ratio test. The only assumption on p(z) is that it has a finite maximum, which true for any non-degenerate GMM with positive definite component covariances. As result, the Taylor series converges for all x regardless of the GMM form.
3.3 Legendre Series Approximation of the Logarithm
For the orthogonal polynomial approximation, we consider the Legendre polynomials, specifically the shifted Legendre polynomials [4] which are orthogonal on [0, a).
Σ (-1)*t* +(a+k)) k)! (n ())(k
1
(v) [oo.a) (v)
y
(8)
Lemma 3.3 (Convergent Legendre Series of Log). If a max(p(x)), and consider the shifted Legendre polynomials on the interval [0, in Eqn. (8). Then for x a.e.
(-1)++ (n + j){((j + 1) log(a) -) L L (n j)(((i + 1)!)²
log(p(x)) = Σ (2n 11
(9)
T=0
Again, all that is assumed about this approximation is that the max of GMM can be bounded, so this approximation converges for all GMMs regardless of structure.
GMM Entropy Approximations
Having established multiple polynomial approximations in Sec. 3, now consider applying them to the definition of entropy for a GMM. We can directly substitute the series approximation into the entropy definition, H (p(x)) = ! [og[((()))] and push the expectation through the summations.
4.1 Huber et al. Entropy approximation
Applying Huber et al.'s Taylor series approximation of the log(p(æ)), we see the GMM entropy can be approximated by,
((m)(1s) Byg [[x 14,
H(p(z)) =
(10)
m!
where (x) = Nix Pt, Σ,) is shorthand for the 7th Gaussian component. The attractive feature of Eqn. (10) is that it simplifies the expected value of log-Glt log-GMM the nh central moments of the 7th

---

component which, is exactly zero when n is odd and has closed form when is even. However, Theorem 3.1 shows that this approximation is not guaranteed converge, which is supported by experimental results in Sec. 6. Furthermore, in higher dimensions, h(m) (14) 0x .r.0kk
@"h(M)) where
ji + ..... + ja = n which grows rapidly, is an n dimensional tensor. This is cumbersome to compute and is difficult to deal with the tensor arithmetic required beyond Hessian. In practice, this limits Eqn. (10) to only second order approximations for random vectors.
4.2 Taylor Series Entropy Approximation
Having established the convergent Taylor series of the logarithm in Lemma 3.2, we can apply the approximation and push the expectation through the summations. This reduces the computation of the entropy to computing (x) for all k < n where n is the order of the polynomial approximation.
Lemma 4.1 (Closed form expectation of powers GMMs). Let p(x) E wiN wgN(alp 21) be GMM and k be non-negative integer. Then
Ep(p(x)*]
i =k (
Σ
(r.......N 1L.....IM
M
Wi
NO E.) M p, t=1
(I (.0(())))..))))))) (11)
where == (x, + MM )- and = - #4 + Ex 1 pe).
While Eqn. (11) may seem complicated at first glance. it is straightforward to compute. All terms Gaussian densities, polynomial functions, and binomial coefficients. Lemma 4.1 is defined for [xx))] but an analogous definition holds for [p]](r) allowing us to apply all the following results not only to entropy, but cross-entropy, KL and MI of GMMs. This is not the focus of this paper, however discussion can be found in A.4 for completeness. Using Lemma 3.2 and Eqn. (11),
we can obtain the following approximation,
(-1)-- na
A (p(x)) = log(a)
(-a)
B
(12)
k=0
To ensure the expected value can be pushed through the infinite sum of the series, we check that our finite order entropy approximation does still converge the true entropy.
Theorem 4.2 (Convergence of (p(x))) Let p(z) EE' wyN(xlu 2,) be GMM and choose a Taylor center such that a >> 1 max(p())). Then, for HT. (p(x)) defined in Eqn. (12)
lim H. (p(x)) H(p(z)) N-400
(13)
Having established convergence of our estimator, it remains to provide a method for selecting a Taylor center that meets the convergence criterion m)xx((())))- In fact, show in Theorem 4.3 that selecting a looser condition a > max(p(z)) ensures convergence from below, thus yielding a lower bound on the true entropy.
Theorem 4.3 (Taylor Series is Lower Bound of Entropy). Let p(x) EM w,N(x) E.) and a > max(p(z)). Then, for all finite N,
(p(x) < (p(x))
(14)
We have now established that the Taylor center chosen as > max(p(x)) both convergent and yields a lower bound. In fact, it is easy to find such a point by upper bounding the maximum of GMM given in Theorem 4.4.
Theorem 4.4 (Upper bound on maximum a GMM). Let p(x) = EM, waN(alu E,),, then
M
max(p(x)) <a =
Σ wi E
(15)
In our experience choosing center closer the convergence criterion mmax(((()) yields slightly more accurate estimates, but not significantly so.
5

---

4.3 Legendre Entropy Approximation
Now, starting with the convergent Legendre approximation considered in Lemma 3.3 and Eqn. (11), can obtain the following approximation,
(-1))t (n + 5)(((( 1) log(a) - Lio.)).n (LE))(())1) (16)
12
Σ
(n-j)(( + 1)!)²
Again, we verify that taking the expectation of our series does not effect convergence.
Theorem 4.5 (Convergence of H (p(x))) Let p(z) = Ei= wyN(xlm,, E,) be GMM and choose an interval such that a > max(p(x))) Then for H (x(())) defined Eqn. (16)
lim H (p(x)) (((())))((((())))
(17)
N-y00
Now having established converge criterion for the Legendre series approximation, we need choose an upper point of the interval for the Legendre series. We need to choose max(p(x)) which is satisfied by the same a found in Lemma 4.4.
4.4 Approximation of Taylor Series Limit
Computing the series Eqn. (12) for higher orders can be computationally prohibitive. In particular, the sum Σnn tn. is over M integers summing n, which is O M 1)!). In this section, provide approximation that avoids explicit computation of this sum for higher orders. We employ a method (similar to Richardson extrapolation [22]) that is based on a polynomial fit of the convergence rate for the lower bound property discussed in Theorem 4.3. From Taylor's theorem, we know that a function can be represented as f = Tn + Ra, where T, is the nh order Taylor polynomial and Rs ((a") is the remainder. Rewriting this terms of the Taylor polynomial, observe that T, = - Rr + Ba + 1/. We take B < 0 to represent the negative scale factor in front of the remainder, and < model the decay of the remainder.
fr,, (p(x)) = a"
(18)
We require three consecutive orders of our Taylor series approximation, (p(x)), ( and AT 2, (p(x)), to solve for the three unknown parameters:
fr (p(x)) =B a² + n1 (v(=)) =B a3 a+1 +1
=B -a n+2 +
Since 0 1, lim, Sa" = 0, indicating that aim solve for approximation of the limit of the Taylor series entropy.
( 2.0 (p(x))) 2ft + 2
AT = y AT (0)20((0)))
(19)
This approach assumes that the Taylor series converges according to Eqn. (18), which is not the case in general. Identifying the exact rate of convergence is topic of future work. Nevertheless, this simple approximation has shown higher accuracy in practice with negligible additional computation, as demonstrated in the experiments of Sec. 6. With slight abuse of terminology, we refer this approach as the Taylor limit. We do not apply this method to the Legendre approximation (Eqn. (16)) as it doesn't t maintain a lower bound during its convergence. Although equivalent methods have been considered to model potential oscillation convergence, in practice, we do not find increase in accuracy.
Related Work
Numerous approximation methods exist in the literature for estimating entropy and related information measures, such as mutual information and Kullback-Leible divergence, in the context Gaussian Mixture Models (GMMs), Monte Carlo estimation, deterministic bounds using Jensen's inequality,

---

4.4
1.5
4.2
4
True Entropy
X
3.8
0.5
Huber et al.
I
3.6
Our Taylor
3.4
0
Taylor Limit
3.2
Our Legendre
-0.5
0.2
0.4
0.6
0.8
1
-3
-2
-1
1 2
2nd Gaussian Variance
5th Gaussian Component Mean
Figure 2: Scalar GMM example is plotted on the left. The variance of a component of a two component GMM is varied between o E (0, 1] as in theory according to Theorem 3.1, the example will be divergent where of < .46 and convergent above. We plot the fourth order of each method and diverge where the theory predicts. Two dimensional GMM with five components is consider on the right. Here the mcan of a single component is shifted from (A, = [-3, -3 to AAs = [3, 3 We consider the third order approximation of cach method and see that Huber et al. is poorly behaved. In both examples, we see that our Taylor method is lower bound (Theorem 4.3), the Taylor limit provides higher accuracy, and Our Legendre method is highly accurate approximate.
that Huber et al.'s approximation does
best-fit moment matched Gaussians, and numerical integral approximations based on the unscented transform have been explored [16, 14, 15]. This paper focuses the Taylor approximation by Huber et al., an alternative Taylor approximation is proposed by Sebastiani [24], which assumes a shared covariance matrix among GMM components in the high variance setting. However, neither Huber et al. or Sebastiani provide theoretical analysis or convergence guarantees offered in our present work. An analysis conducted by Ru et al. [23] explores the efficiency of Huber et al.'s method and demonstrates that deterministic quadrature methods can be equally fast and accurate in a single dimension, however quadrature methods scale poorly with dimension, at ((ND) where N is the number quadrature points per dimension and D is the dimension of the problem.
Variational approximations and bounds are also widely explored for estimating entropy and mutual information (MI). Much of this work is motivated by the use of Gibbs' inequality, which leads to bounds on entropy and MI [1]. Later work explored similar techniques for upper and lower bounds on MI [21, 10]. More recent work uses artificial neural networks (ANNs) as function approximators for variety of information -theoretcccc measures based on differential entropy. The MI neural estimator (MINE) uses such an approach for representation learning via the information bottleneck [2] based on the Donsker- Varadhan (DV) lower bound KI [9]. Related methods ANNs for optimizing the convex conjugate representation of Nguyen et al. [20]. McAllester and Stratos [ [19] show that many of these distribution-fred approaches based ANN approximation rely Monte Carlo approximations
that have poor bias -arianee characteristics which they provide their own Difference of Entropies (DoB) estimator that achieves the theoretical limit on estimator confidence.
6 Experiments
We consider two experiments, synthetic GMM section where we look at divergence of Huber et al. approximation (Eqn. (10)) and convergence of our three methods, our Taylor (Eqn. (12)), Taylor limit (Eqn. (19)), and our Legendre (Eqn. (16)). Furthermore, we give comparisons accuracy and computation time across variety of setting of approximation order, number of GMM components, and dimension for all methods. We then show our how our methods can be applied in practice to Nonparametric Variational Inference [11] where the convergence guarantees of the estimators has noticeable accuracy improvement on their algorithm.
6.1 Synthetic Multivariate GMM
To highlight the theoretical properties, such as convergence, divergence, accuracy, and lower-bound of methods as discussed in Sec. 4, we will consider some synthetic GMMs. We create two GMMs similar to the example published in [15] (original experiment recreated in A.5). We consider single and multi-dimensional dimensional case that satisfy the divergence criterion Theorem 3.1. We also look at time and accuracy analysis versus dimension, components, and polynomial order.
7

---

100
4.5
Entropy
Entropy
Entropy
3.5
2.5
1
5
10
20
50
1
2
10
3
4
9
10
Dimension
Components
Order
Time
0.1
0.1
Time
0.01
0.01
0.001
0.01
Taylor
Taylor Limit
- Huber et al.
100μ
Legendre
0.001
10
2
5
Dimension
Components
Order
Figure 3: Dimension (left) a commonennnt GMM varies from zero to fifty, for the second order of each method. Our methods show comparable accuracy and computation time to Huber, regardless of dimension. Number of components (middle) in two-dimensional GMM is considered for the second order approximation of all methods. Huber et al. and our Legendre approximations are nearly equivalent in accuracy. while the Taylor series and Taylor limit Huber al., deviating only at high numbers of components. Order (right) of each approximation varied for a three- saree-dimensionll two-component GMM. Huber et al. is plotted up to order three, as higher orders are restrictive due to tensor arithmetic and Taylor limit starts at order two as it requires three consecutive terms.
lower hounds. Computation time for all methods is identical and comparable
Scalar GMM this experiment, we consider scalar GMM as fourth order and above cannot be easily computed in higher dimensions for Huber et al. due to tensor arithmetic. We use simple two-component GMM with parameters W1 = 0.35, w2 = 0.65, μ1 -2, μ12 = -1 of = 2, and of E (0, 1]. We are changing the variance of the second Gaussian, σ2. in the range (0, 1] because the condition for divergence in Theorem 3.1 (p(x μ1) f = 222) is satisfied approximately when o3 < 0.46 meaning this experiment should have regions of both convergence and divergence for Huber et al. approximation. Fig. 2 (left) shows the fourth order approximations of all methods. We see that the Huber et al. approximations diverges as expected in the range where of < .4.. Our Taylor method remains convergent and accurate for all values while maintaining lower bound. Again, our limit method gains us some accuracy and still manages to be lower bound. this case, we see that the Legendre approximation is near perfect fit for the entropy.
To demonstrate that divergence not limited single dimension or higher orders, consider five-component, two-dimensional GMM with the parameters w; .2 Vi, #1 = [0, 0) #22 = [3, 2|T [3 = [1--5]" μ4 = [2.5, 1.5]", , μt
Multivariate GMM
c[L, 1J" for c [-3,3], Σ 25.12 Dg = 31g, and Σn E4 D> 21, where 12 is the two dimensional identity matrix. This examples shifts the mean of the fifth component to show that simply the location of components make the Huber et al. approximation behave poorly. Fig. 2 (right) shows the third order approximation each method. We see that Huber et al. is clearly not well behaved this case with low order approximation. Furthermore, continue method, an increased accuracy from out limit method, and that the Legendre approximation is very close the true entropy.
lower bound by our Taylor
6.1.1 Computation Time
In this experiment empirically analyze the computation time of each method as function of Gaussian dimension, number of Gaussian components, and the order of each polynomial approxima- tion. The baseline of each method will be compared the Monte Carlo estimation of entropy using
L 1000 samples {xry]_- p. The Monte Carlo estimator given by I - Σ, log p(xy))
Dimension In Fig. (left), we evaluate the accuracy and computation time for 30 two-component GMMs per dimension in the range of [1, 50). Comparing second order approximations of all methods against the Monte Carlo estimator, polynomial approximations demonstrate similar accuracy and

---

nearly identical computation time. The results are comparable to Huber, indicating that our methods preserve accuracy and computation efficiency while providing convergence guarantees.
GMM Components In Fig. 3 (middle), accuracy and computation time are presented for 30 two-dimensional GMMs with varying numbers of components (from 1 to 20) using second order approximations. Legendre and Huber methods show slightly higher accuracy compared Taylor approximation and Taylor limit. Notice, Huber's standard deviation also increases with more components, due to the increased likelihood of satisfying the divergence condition in Theorem 3.1. Computation time remains similar for all methods, but is more prohibitive for higher components.
our
Polynomial Order Fig. 3 (right) shows as the order of the polynomial approximation increases for wo-comppnent GMMs three dimensions. Legendre and Huber methods show higher accuracy compared to Taylor approximation and Taylor limit. Huber is limited to order 3 due to Tensor arithmetic, while Taylor limit starts at order as requires multiple orders. Computation times are similar across all methods. Notice no accuracy is gained from zero to first order and from second to third order in Huber's approximation due relying on odd moments of Gaussians which are zero.
6.2 Nonparametric Variational Inference
Consider a target density p(z, D) with latent variables r and observations D. The NPV approach [11]
optimizes the evidence lower bound (ELBO), log p(z, D) 2 max, Hg (p(z, D)) Hy (9(2)) ((q)
Σ M(r w.r.t. an m--oompoentt GMM variational distribution g(x) N
The GMM
entropy lacks a closed-form so NPV applies Jensen's lower bound as an approximation, H. (q(x)).
The cross entropy also lacks closed-form so NPV approximates this term using the analogous Huber et al. Taylor approximation. Specifically, NPV expands the log density around the means of each GMM component as,
M
N
2 log(p(())) Egg [[ pa) = (p(z))
H₂(p(z)) ~
(20)
n!
However, Eqn. (20) is subject to the divergence criterion of Theorem 3.1 if 2p(nn) < max(p(x))
By replacing the entropy terms with our convergent series approximations we observe significant improvements in accuracy.
In our approach, we will highlight and address two problems with the NPV algorithm; the potential divergence of (p(z)) and the poor estimation of the GMM entropy via H.g (q(x)). To address the potential divergence of N. (p(x)), we will take motivation from the results found in [23] and use a 2 point Gauss-Hermite Hermite quadrature method to approximate (p((z))) This method will be limiting factor in scaling the NPV algorithm in dimension, however it guarantees that the cross- entropy approximation will not diverge. This alteration leads to a solution for the inconsistency of the BLBO approximations. Then, Jensen's 's inequality general, instead we will use the three methods we have introduced, our Taylor, Taylor limit, and our Legendre, the GMM entropy approximations for higher accuracy. Fig. 4 shows approximation of a two dimensional, three component mixture Student T distribution using five component GMM in the traditional NPV, our modified NPV algorithm with our Taylor and Legendre approximation.
very poor approximation for entropy in
The results, as seen in Fig. 5, highlight the accuracy of each method versus the number of components, the order of our polynomial approximation, and the dimension of the GMM. In each experiment, we are approximating a multivariate mixture distribution, p(x). We randomize the parameters of p(x) and the initialization parameters of the variational GMM, g(z), for optimization. The KL is approximated using a 100000 Monte Carlo approximation after convergence of each algorithm. We see that in all cases of components, order, and dimension, our method achieves significant accuracy improvements. We see that we can use low order approximations to receive substantial approximation improvement (Fig. 5 (middle)). We see all methods gain accuracy as number of components increase (Fig. 5 (left)) however our methods see most of the accuracy improvements with only few components, whereas NPV has substantially worse approximations with low components. Finally, we see we maintain a lower variance and KL. than NPV with all our methods as the dimension grows (Fig. 5 (right)). For further discussion of the experiment, see A.6.
7 Limitations
Our Taylor and Legendre methods ensure convergence and deliver comparable accuracy to that of Huber et al. However, the computational complexity of these methods grows with the number of

---

1,9
1.5
1,5
1.5
1
0.L
-0.5
-0.5
-0.
-0.5
0.05
-15
-1.
-15
-1.5
Distribution
NPV PDF
Legendre PDF
Figure component mixture Student distribution PDF (far-left) is approximated by five component GMM using traditional NPV (left), our algorithm using 6t order Taylor polynomial (right), and Legendre polynomial (far-right). We see that NPV both has issues with finding correct placement of means and sets the variances of the GMM components be too narrow. Our methods do better job of assigning means and the Legendre method seems set the variances slightly better than Taylor.
20
12-
NPV
10
Our Taylor
Taylor Limit
8-
Our Legendre
4
10
5
2
6
10
12
Component
Order
Dimension
The above figures show the accuracy of each method across varying components, orders, and dimensions in approximating a multivariate mixture T distribution with GMM. Our method consistently improves accuracy significantly. Low order the convergent estimators provide substantial approximation improvement (middle). Most accuracy improvements are achieved with a small number of components, unlike NPV (left) which continues to need higher number of components see good accuracy return. The guaranteed convergence of the approximation in higher dimensions
Figure 5:
have drastic improvement accuracy ().
components, M, in the GMM. following an O((n + - 1)!) 1)!) time complexity, where represents the order of the approximating polynomial. This summation comprises only scalar terms and is amenable to parallelization but may become prohibitively expensive for large M and values. Furthermore, our polynomial methods are specifically tailored the entropy, (p(x)))) and cross- entropy, (z(r))) where both (z) and q(x:) are GMMs. In contrast, Huber et al. 's approximation can be more readily extended cross-entropy scenarios where ((x) pertains to any distribution amenable the construction of second-order Taylor polynomial. This limitation hinders the broader applicability of our approximation.
8 Discussion
We have provided novel theoretical analysis of the convergence for the widely used Huber et al. Taylor approximation of GMM entropy and established that the series diverges under conditions the component means. We address this divergence by introducing multiple novel methods which provably converge. We wish to emphasize that the Huber al. approximation tends to yield accurate results when it is convergent and the intention of this work is not to dissuade the use of this approximator: Quite the contrary, this work encourages the of either Huber al. our own estimator by providing a solid theoretical foundation for both methods. We acknowledge that there are contexts in which one method may be preferred over the other, for example when bounds are preferred, or when convergence criteria are provably satisfied.
There several areas that require further investigation. For example, one limitation of both methods that they scale poorly with polynomial order and number of components. fact, Huber et al. cannot easily be calculated for fourth order and above, due to tensor arithmetic. Our approximation works well in practice, but is limited solely to GMM densities. Further work is necessary to efficiently apply convergent series situations of cross-entropy's that contain non-GMM distributions.
10

---

References
[1] D. Barber and F. Agakov. The IM algorithm: a variational approach to information maximization. NIPS, 16:201, 2004.
[2] M. I. Belghazi, A. Baratin, S. Rajeshwar, S. Ozair, Y. Bengio, A. Courville, and D. Hjelm. Mutual information neural estimation. In International conference on machine learning, pages 531-540. PMLR, 2018.
[3] M. Bernardo. Expected Information as Expected Utility, Ann. Stat., 7(3):686-690, May 1979.
[4] A. Bhrawy, E. Doha, S. Ezz-Eldien, and M. Abdelkawy. A numerical technique based on the shifted legendre polynomials for solving the time-fractionas coupled kdv equation. Calcolo, 53, 01 2015. doi: 0000077/100022-1--011222.
[5] D. Blackwell. Comparison of experiments, In J. Neyman, editor, 2nd BSMSP, pages 93-102, Berkeley, CA, August 1950. UC Berkeley.
[6] G. Brown, A. Pocock, M.-J. Zhao, and Luján. Conditional likelihood maximisation: unifying framework for information theoretic feature selection. The journal of machine learning research, 13:27-66, 2012.
[7] A. Carrington, P. Fieguth, and H. Chen. Measures of model interpretability for model selection. In A. Holzinger, P. Kieseberg, A. M. Tjoa, and E. Weippl, editors, Machine Learning and Knowledge Extraction, pages 329-349, Cham, 2018. Springer International Publishing. ISBN 978-3-319-9970077
[8] M. Cover and A. Thomas. Elements of Information Theory 2nd Edition (Wiley Series in Telecommunications and Signal Processing). Wiley-Interscience, July 2006. ISBN 0471241954.
[9] M. D. Donsker and S. S. Varadhan. Asymptotic evaluation of certain markov process ex- pectations for large time, i. Communications on Pure and Applied Mathematics, 28(1):1-47, 1975.
[10] A. Foster, M. Jankowiak, E. Bingham, P. Horsfall, Y. W. Teh, T. Rainforth, and N. Goodman. Variational bayesian optimal experimental design. In Advances in Neural Information Processing Systems 32, pages 14036-14047, 2019.
[11] S. Gershman, M. D. Hoffman, and M. Blei. Nonparametric variational inference. CoRR. abs/1206.4665, 2012.
[12] J. Goldberger, S. Gordon, H. Greenspan, al. An efficient image similarity measure based on approximations of kl-divergence between two gaussian mixtures. ICCV, volume 3, pages 487-493, 2003.
[13] J. M. Hernández-Lobato, M. W. Hoffman, and Z. Ghahramani. Predictive entropy search for efficient global optimization of black-box functions. Advances in neural information processing systems, 27, 2014.
[14] Hershey and A. Olsen. Approximating the kullback leibler divergence between gaussian mixture models. In 2007 IEEE International Conference on Acoustics, Speech and Signal Processing ICASSP '07, volume pages IV-317-IV-320, 2007. doi: 10.1109/CCASSP22007 366913.
[15] M. F. Huber, T. Bailey, H. Durrant-Whyte and U. D. Hanebeck. On entropy approximation for gaussian mixture random vectors. 2008 JEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems, pages 181 181-188, 2008. doi: 10. 10.1109/MHL.200 WWW.MMMM...00044448880000
[16] Julier and J. K. Uhlmann. general method for approximating nonlinear transformations of probability distributions. robotics research group, department of engineering science, university of oxford, 1996.
[17] D. V. Lindley. On a measure the information provided by an experiment. The Annals of Mathematical Statisties, 2764):986-1005, December 1956. ISSN 0003-4851.

---

[18] V. Maz's ya and G. Schmidt. On approximate approximations using gaussian kernels. IMA Journal of Numerical Analysis, 16(1):13-29, 1996.
[19] D. McAllester and K. Stratos. Formal limitations on the measurement mutual information. In International Conference on Artificial Intelligence and Statistics, pages 875-884. PMLR, 2020.
[20] X. Nguyen, M. J. Wainwright, and M. I. Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions on Information Theory, 56 (11):5847-5861, 2010.
[21] B. Poole, S. Ozair, A. Van Den Oord, Alemi, and G. Tucker. On variational bounds of mutual information. In International Conference on Machine Learning, pages 5171-5180, PMLR, 2019.
[22] I.. Richardson and R. Glazebrook. 1x. the approximate arithmetical solution by finite differences of physical problems involving differential equations, with an application to the stresses in a masonry dam. Philosophical Transactions of the Royal Society of London, Series A. Containing Papers of Mathematical Physical Character, (taecterr,10((5555 2106459-470:3007333,, 1911. doi: 10.1098/rsta. 191 11.0009.
[23] B. Ru, M. McLeod, D. Granziol, and M. A. Osbome. Fast information-theoreti bayesian optimisation, 2018.
[24] Sebastiani and H. P. Wynn. Maximum entropy sampling and optimal bayesian exper- imental design. Journal of the Royal Statistical Society: Series (Statistical Methodol-
ogy), 62(1):145-157, 2000. doi: https://doi.cooo http:://doi.or//O..111I.. htgp//0.1111//44778888.8.00
URL https:
/ //rss. owlssnliineebbrrn ceeerrrrrylllyy.. com/ doi abs/ 10. 1ww.11111166111 98688 086..00225.......
[25] B. Settles. Active learning literature survey. Computer Sciences Technical Report 1648, University of Wisconsin-Madison, 2009.
[26] M. C. Shewry and P. Wynn. Maximum entropy sampling. Journal of applied statistics, 14 (2):165-170, 1987.
[27] N. Tishby, F. C. Pereira, and W. Bialek. The information bottleneck method. arXiv preprint 0oyxiess0000057,20000 2000.
[28] D. Zwillinger. Cre standard mathematical tables and formulae; 30th edition. Journal of the American Statistical Association, 91:1755, 1995. URL https:/ /// //api./pppp.pp appi... semanticscholl org/Corpus /tt//orpuuDD:62700006.
12

---

A Appendix
A.1
Section 3 Proofs
Theorem 3.1 (Divergence Criterion for Huber et al.). Let = E and consider the Taylor series presented by Huber al. in Eqn. (5). If any mean component, pa satisfies the condition p(u,) < {max(p(x)), then Huber et al.' approximation diverges, otherwise it converges.
p(z)
i= wyN(x p4, E,) be a GMM
Proof. Let f(v) = log(v), g(ar) = p(x), and h(x)) [(((a)) = log(p(x)). Huber et al. creates the Taylor series in Eqn. (5) with N order approximation
M
M
N
(14)
w. Mi (x)
(x p4)
(21)
n!!
7==
Let us consider just a single one of the Taylor series in Eqn. (21)
h
(ps) (x p:)"
Th,N M4 (x)
(22)
n!!
+=0
By Theorem 3.4 in Lang', the Taylor series of composition of function is equivalent to the composition of each components Taylor series, ie.. N., (x) (x) (T,N.s())) o N. (I) where is the composition operation. Since p(x) is GMM, is the sum entire functions, and thus itself is entire, meaning it's Taylor series, N., (z), converges everywhere. We turn our attention to the Taylor approximation of log, T(NN)
(-1)R-1
T T.N log(p(u))
Σ
np(μu,) (y- -((()))
(23)
We can look at the th term in the series, bn convergence.
(-1)--
(y p(u+))" and use the ratio test define mp(pz)"
bx+1
(n + ()(())) (U - p(44) T1 npp(u))) "
(-1)"
lim nt0000
lim n-000
(U p(()))
bn
(-1)n-1)
(-1)"
np(#4))
(y p(14))++11
lim
n-+00
1-) (n 1)p(m))mt
(y p(u)))"
lim
n
(y p((s)))
n-000
(n + 1)p(μ;)
n
y p((())
y - p((s))
lim n-
=
(n + 1)
p(m))
p(p)
The ratio test states that the series converges if the limit, I., is strictly less than 1. However, setting L == and some simple manipulation, find
y p(p;) p(()
p(p(( < p(u))
y < 2p(ps)
(24)
This only converges if all y 2(p(u,). Consider the maximum, max(p( if it satisfies this condition, so will every other point, if doesn't it satisfy this point, then the series is divergent by the ratio test. So we have the convergent criterion max (px()(z)) < 2p(u:), written in terms of divergence criterion, we diverge if p(1s)) < 2 max()))))
Lemma 3.2 (Convergent Taylor Series of Log). Ifa >> }max(p(:)), then for all
log(p(x)) log(a) +
(-1)a11
n (-a) TM p(x)*
(<x)"~>p(*>
(25)
nhan
k=0
LLang. Serge. Complex Analysis. 4th ed. Springer, 2013. ISBN 978- 17757-3083-8.
13

---

Proof. Consider the n term in the sum
(-1)a=1
bn
(p(x) a)"
na
The ratio test says if the limit of the absolute value of successive terms converges value strictly less than one, then the series converges
Da+1
na (n 1)a (p(a) -a)+t
(-1)a
lim
lim
(-1) 111 (p(x) - a) a)-
On
(-1)" na"
(p(x) n+1
lim n-000
(-1)--1 (n + 1)a
(p(x) a)"
lim
n
(p(x) a)
n-000
(n + 1)a
n
p(a)
p(x) a L
lim n-000
(n + 1)
a
We see that L < Ve iff a > 1 max((((x) in which case the series converges everywhere.
Lemma 3.3 (Convergent Legendre Series of Log). Ifa > max(p(z)), and consider the th shifted Legendre polynomial on the interval [0, al in Eqn. (8). Then for a.e.
log(p(x)) Σ (2n Σ (-1)+ti + i){((i + log(()) -) I10,)),n (p(z)) (n - j)M(( + 1)!)2
(26)
Proof. Orthogonal polynomials can approximate any function a.e. that is continuous and square- integrable (see Trefethen and Bau² and Gustafson"). In out case, o (v) live on (2( a)) (referring the second Lebesgue space on the interval [o, al). This means all we have show is that log(y) lives in this domain which means [|log()][2 0o
[log(v))l J."' log(v)) = a a(log((a> 2) log(a) + 2) < xo
(27)
So we see that log(y) E L2(10, al) and therefore it's Legendre series is convergent. For completeness, we now derive the Legendre series for log(y). We will appeal Equ. (8).
Liu,al.n Ek=o (-1)x+&(n k))! (n k)))((kk))k vh and (L1o,a).. 10 (n)))))))())) 2 found in [4]
log(p(x)) (log(v), Loo.).,
(Lo,a), (3),, L10 (v)) [1.0)).(((x))))
2n + 1
log(2) (-1)m=M( (n + k)!
k))(kt)aaa y (p(().).)
2n 1
1) (n k)! [-" log(u)u A dy
n=0
k=0
2n
11) (n + k)) h+1 ((k 1) log(a) - L Lio.oj.n (p(x))
(n 1))((=)
(k 1)2
0=0
k=U
80
(-1)ath (n + k:)1 ((k 1) log(a) 1) [1000,,(((1)))
2n +
(n k)) + 1)!)2
n=0
which we know is convergent from the above discussion
Trefeethen, Lloyd N., and David Bau III. Numerical I Linear Algebra. SIAM, 1997.
GGrnnt B. Gustafson Differential Equations and Linear Algebra. 1999-2022.
14

---

A.2
Section 4 Proofs
Lemma 4.1 (Closed form expectation of powers GMMs), Ler Lerp(x) Em wi waNNalp Ei) be GMM and k be non-negative integer. Then
E (I..,,,A .MM
k
M
NOOOl E,) M p.
Ep(p(x)*]
Σ
(28)
++++ =
where E = (x,+ + Em ) and μ E p4 +
Proof.
We will prove this statement by directly expanding out each term
=B
Σ weNN (zclue Ex)
le Σ Cr....
m II
=Ep
Im Jm
ji +im mn
(r (r....sm))
M
I
Σ 31 + +Jm=k
II P=1
[ (we) pou
Σ
N(xluu Ex) II (M Ex
1=1
To combine the Gaussians under the integral, we appeal to the power of Gaussians (Lemma A.2.3) and product of Gaussians (Lemma A.2.4)
Marlu J, E() N( E.)
Σ
k
(we)
der
Σ w/
(2 - de
ji,...,
fm
)we)) )ar
M Σ ui
M. Σ.) I1 N(00u) 3,20) N(OO,,
Σ
II
Nalu, E)de
Lix (2 (de 11-11//2
ji
2m1
k Ji....,JI fimmk
M Σ W0
.N Ee)
m
NO Σ.) (wrN(0))m Ex) E()))))))))))))
Im
where μ E[2 Me Etm Je "4) defined from Lemma A.2.4. We see that are left with
no integral and a closed form of the expectation of the powers of the GMM.
O
Theorem 4.2 (Convergence of (p(x))) Let p(x) E w,N(xlu, E,) be GMM and choose a Taylor center such that a > } max()(()). Then, for ( defined in Eqn. (12)
lim
A (p(x)) H(p(x))
(29)
N
Proof. We start out with the definition of entropy and introduce the approximation discussed in Lemma 3.2
M
1(p(z)) -
Σ
wwg (z) log(p(r))dr
Σ
wi (i(x))
log(p(x))
1=1
1) (p(x) - -a)" dx
(x) log(a)) +
W/
nan
M
Wi log(a) + I (((x))
11)=-1 (p(x) - a)) dx
nan
i=1
M
wi log(a) + (-1)n-1
na (p(2(((()) a)n dr
15

---

We now wish to swap the order of integration and of the infinite summation as shown in Lemma A.2.1
M
(-1) (x)(((())))))))))))))))))))))))) a)"
Wi log(a)
+
n
M
1-1)-1
W
log(a)
+
(1))(p(()) - dz
n
==1
1-1)aー
(1)
11
Σ i=1
Wi log(a)
+
(:(x))
Σ
(p(x)) dx (-a) (p(a)))((>())++++)))))))))))))))
na
k
i=1
k=0
(-1)--1
Wi log(a)
+
11
(-a)
E 1-aan>BB
Be(()) (((()))*)
n
k
We can compute (x)(z) [((()))* using Lemma 4.1. The above term is equality for the Entropy.
simply truncating the infinite summation gives a convergent approximation.
(p((((x)) = Σ ws (1og(a)+ +
M
N
n-
n1
(-a) n- Ex((2) [(x(2))*]
(30)
n
Theorem 4.3 (Taylor Series is Lower Bound of Entropy). Let p(x) = Em wNN(x/ Σ,) and
max(p(x)) Then, for all finite N,
(p(x)) (p(x))
(31)
Proof. a >> max(p(x)), then we have the following lower bound
H(p(z)) / p(z) logp(r)dr= = - p(x) log(a) +
((()
(-1)a-1 (p(x) - a)" dr
na
#=1
(-1)" (a p(x)))² dx
= -- p(x) log(a) +
na
log(a) - f p(x))
nan (a - -p(x))) dx
log(a)
p(xx)
nan (a - p(x))) dx
2 log(aa) + / p(x)
Σ nin (a ((=v()))" dr Hx(p(x))
since every term in the summation is positive due to a > p(x) Vz, then truncating the series only removes positive terms, leaving us with a lower bound.
Theorem 4.4 (Upper bound on maximum of GMM), Let pex) MM WxNa(rl E.), then
M
max(p(x)) a Σ W/ [2T
(32)
Proof. We need to find an upper bound on max (pxx))
M Σ
max(p(x)) =max
WoNN(1/p Σx)
M
M uamax (N(z)p. E,)) = Σ
we |
16

---

We simply have bound the maximum of the combination by combining the maximum of every component in the GMM.
Theorem 4.5 (Convergence of (v(x)))))) Let p(a) - Ein waN(alu Ed) be GMM and choose an interval such that a > max(p(z)). Then for H (p(x)) defined Eqn. (16)
lim (x(()) H(p(a))
(33)
N-
Proof. We start out with the definition of entropy and introduce in the approximation discussed in Lemma 3.3
M
w.):(x) log(p(z))de = Σ wi /
M
H(r(x)) -
(=(())o(((())dd
1=1
1=1
M
(log(u), I(ooa).
Wi
4s (x(
(Lo.a) (v), I100ao.. (v)) (p))),
M
20
(log(u). Lpo.o). (v))
W
9: (z)
(L10.a), (v), Lgo.a (v)) (0.))).(((())))
We swapped the order of integration and of the infinite summation shown in Lemma A.2.2. We simplify computation that recreated in Theorem 3.3
(log(y), (w) Σ W0s
(Loaa) (V), (v))
(-1)at) (n + j)(( + 1) log(a) - 1)
(-1)a+ (n + k:)!
(2m + 1)
uy (xr)
(n k))(kk))aa p(xr)
()]) + 1)1)
1=
Σ T-0
(2m + 1)
(-1) (n + ()((( + 1) log(a) - 1) Σ (n - ()((( 1)1)²
(-1)at (n A:)!
Σ
M
w. (x) [p())*]
(n - 1))(
==
[(p(x))) using Lemma 4.1. Simply truncating the infinite summation gives We can compute (m)) the approximation.
(kko((())) - E(2no
N
(1))+ts (n + (() log(o) - 1) - (n(() + 1)1)2
(n)=+*((+++ K)! (k!
Σ an [p())]
k= (n
1=0
1=0
which soo a (p(x)) = H (p(+)) as the above series is exactly equal to the entropy.
Here, we address few of the assumptions made in the above derivations. We start with the ability to swap the order of the integral and infinite sum for the Taylor series. Lemma A.2.1 (S (Swapping Integral and Infinite Sum (Taylor)). Let a > maxx(p(;)), then
(-1)-1 na
(-1)-11
-
- a)) dx
na" (x)(((()) - -a)"
Proof. For simplicity of notation, let c = sup p(=) < since a > ma)()())) We then appeal
to Fubini-Tonelli theorem which states that if / Σ (f(()dddz < ca or if ΣJ (z) dr < co, then I Σ fn(x)de= = ΣJ (x))dr.
Σ / - (l))(((a)) - a)n do
(b(o))al al a
(x(x)
dx
-
na"
n
(x(x)) (c)" dx
/ (x(x)dx
< 00
i=1
17

---

We know that Ei=1 <º <
oo because of the ratio test again
c=+1
lim
lim 2=000 n
c=c<1
71 + 1 cr
So that Tonelli holds
(-1)--1
(-1)a-1 ((x)(((x)) - a)"de
(z))((())
- a)" =
Σ t=1
na
na
We now consider the case for the Legendre series Lemma A.2.2 (Swapping Integral and Infinite Sum (Legendre)). Let a > max(p(x)).. then
f (x(z)) (log(v), Lio.)) (v))
[io.o). (v), 4 ((0))) (p. = Σ / q((x) (Lio,a) (V), L (0000)
(log(v), L o) (v))
L (p(x)) )dx
Proof. We again appeal to Fubini-Tonelli. We will use that =pp_a)=((pz]]
as it is the
(Loo.al: (1). (p.)..)......)))
orthonormal polynomials and then use Schwartz on the remaining term
(x))) (log(v), In
(Loo.a,. ((0,)),(()) Lio.a),n (v(x)) dr
(og(()) /
(x (1.o,,
dr
(Li0,ai (n 1n
( ((())dr
Σ a((log(a) log(() + 2)
2n + ) )²/
a
/
( uss Eijdr
8
=a ((log(a)) - 2) lag(a) +
Σ
(2n 1)2 0c
So we see that since the absolute value is finite, then MoneliiTonelli applies and can swap the order of the integral and infinite summation.
The next thing we show is the relations we used for powers of Gaussians.
Lemma A.2.3. Powers of Gaussian
W(zlu, 2)" = =11(2=2)m-11- N (m/,, -
We are going to do an inductive proof and rely on the well known relation of products of Gaussians
Proof.
(a Where D (A-1+ A) B) = Do A+B)
and - B-b).
Base Case: n =
E) [x,())
(34)
=N(x|u, E)N (u/u, 22) [2 1/2 N (z|11 2
1
1 دー
(35)
In this case, we get that D -5-1 5 d d= - + -ー (u) = μ. We also
that N (ulu, 2Σ)) is being evaluated its maximum, which just leaves the scaling term out front
18

---

of the exponential in the Gaussian, |2 = Inductive step: Assume that (z Σ)" |n(2x2)a~11 N (2) then wish to show
that (z) 2)=+1 [(n + (2 1/2 /V ss||e n+I 2
(alu 2) [ Σ)
= -11(2 NV
x||, - N (
= -11(2=2)0~11 N
2|4 n + 2) Nulu,,
n+1 2)
16
7 N 2//4 n 1
(12==))-11- = =|1(2=Z))~11/--
2π
n
+ 1 Σ
1 Σ
= =I(n + 11)+22))))1--) N 22//
5
n n+ 1
Here, D = -(11)) + 1-1)
-1 1 E and d x+1
Σ
/ (++E-1) μ.
n+1
We finally show the product of Gaussians that we used. We keep the exact same notation used in the derivation of the entropy Taylor series so the terms may be more easily identified. Lemma A.2.4. Product of Gaussians
1 Σs = -N(O E,) II N (o+ ue, je Ee
Narl Σ) N(Ou,
(x Σx) II N z0lles
je
and μ > (2, #4 EE Me) where = (2'-+ + -1- )-
Proof. We will simply expand out the product of Gaussians, collect like terms, complete the square, and then recollect exponentials into Gaussians evaluated at
Ed N
II
us,
2.) 3t
[2mE,1
à exp
2 (2 m.) "T. (x m.)
25 E
exp { 2 Ho) 3125 (a m²)
1=1
|2x3,|1
2T
exp
,x i 2 1/4 u4 'm4 Σ a "..,,, lo 2x 1eF Im si m1
2
= =12
2st
exp
2 Σ Je - 22" -i 56
+ + +, a. Σ Mf 2 m1)
Now we let Σ
(E-1 + >m
Jr J:
11-1) 1 and μ = (>x Ui + Em j (i:)
=[2m2,|
1
2x
exp
D 2x To - p + Σ as ا #s
Ja
=|2nE,]
1
II
2x
Dx
exp
x 2x E
Ex n²² En + u T.. mn + Σ u 3., m1
1=1
=12mS,11
[m]
II
2xx
Je
2 1/2
|2 1/2
exp
2 (1r " 1 m0) } dae --" of t Σ Ea
m
= =I2nE,]
t exp
M4
25
exp {-1 1 (1( |l exp f (-x( (ootEa)((llll M(alu
1 -N(()p II v (olues
1/2 23
\ E) No Σ)
je
19

---

3.4
3.2
-2
5th Gaussian Component Mean
Figure 6: Well-Behaved GMM example is plotted on the left. The entropy of five-component, bivariate GMM plotted as function of the location of the fifth component JAs E [[3.3]. We show the true entropy, the 3 order of Huber al.'s approximation and methods, along with approximate limit.
A.4 Extensions to Cross-Entropy
Our main results on GMM entropy approximation also extend to the cross-entropy between different GMMs. We can instead consider Hp(g(x)) -Rp(e) [log(ι(:))] where p(x)
E WaN(al Σ,) and ((x) = Σ'' (2 (,) E,). The series representations of log(.) stay unchanged however must choose center point that allows the series to converge with respect to the inner GMM, qxx). This means Theorem 3.2, Theorem 3.3, Theorem 4.2, Theorem 4.3, and 2 m We
Theorem 4.5 need to be reformulated that > max(g( (or the respective do not formally restate these theorems here for brevity. We instead provide a sketch of how the results extend to the cross-entropy setting. Choosing the bounding of the max found in Theorem 4.4 can simply be altered so that
M Σ
max(q(x)) S
2T
The analogous proofs will all hold in this case. The final alteration that needs to be made is to Theorem 4.1. Again, following the exact same proof, just switching notation and being careful, one find that
Ep(g(2)*1=
Σ
A
M7
Wi
NO Σ No Σ)
II [w.N (1)))
T
(36)
+ tisk
M
i=
where + - same form, however has much more convoluted notation and hence dropped from the main paper as an attempt to bring clarity to the methods being discuss without unnecessary notation.
Jr
and μ Σ + Ae). The result is the
A.5 Recreation of Huber et al Experiment
Here we reproduce the experiment of [15] of five-component bivariate GMM with uniform weights
w; 0.2 for 1, .,. 5, l1 [0. 0 /22 [3, 2/a μ4 [1, 0.5 μ4 [2.5, 1.5)"" 145 c [J, Σ, diag 16, 1). Σg = diag(1, 0.16), and Σg E4 = Σ; d0ag 0.5). We vary the position of the fifth mean (115) in the range 13, 3). Fig. 6 (left) reports the third order Taylor approximations from both Taylor approximations, the Legendre approximation, as well as our approximate limiting method.
Huber al. accurate in the well-behaved case, but does not have any convergence guarantees nor is it a bound. Our proposed Taylor approximation sacrifices some accuracy, but is always lower bound (Theorem 4.3) and convergent (Theorem 4.2). We also note that our naïve limit method does gain us substantial accuracy and is still a lower for this approximation. We notice that our Legendre approximation has comparable accuracy to that of Huber et al. in this well behaved case but has the advantage that it is guaranteed to converge (Theorem 4.5) and that we can compute higher order approximations that are difficult to define for the Huber et al. approximation.
we have not proven the bound property
20

---

20
12
8
10
10
2
10
5
10
Component
Order
Dimension
600
15
NPV
500
250
Our Taylor
Time
Run Time
10
200
Taylor Limit
Our Legendre
150
200
100
2
6
10
3
5
2
8
10
12
Component
Order
Dimension
The above figures show the accuracy and computation time of each method across varying components, orders, and dimensions approximating multivariate mixture distribution with GMM. Our method consistently improves accuracy significantly. Higher order approximations (bottom middle), and even low order approximations provide substantial improvement (top middle). Most accuracy improvements are achieved with small number of component, unlike NPV (top left) which continues need higher number of components to good accuracy return. Our computation time increases with dimension due less salable cross-entropy approximation with Gauss-Hermin quadrature (bottom right) however the guaranteed convergence of the approximation seems to have a drastic improvement on accuracy (top right).
Figure 7:
not increase computation time
A.6 Nonparametric Variational Inference
Consider target density p(a,, D) with latent variables x and observations D. The NPV approach [11] optimizes the evidence lower bound (ELBO),
p(z) 2 max Hy (p(2, D)) H,(((x))) C(g)
(37)
w.r.t. m-coompoentt GMM variational distribution q(x) = Σm Wa us, da.. The GMM
entropy lacks closed-form NPV applies Jensen's 's lower bound as approximation,
M
(g((())) = E,, [og(g(>))
we log (E,, ig(=)) = H (g(x))
(38)
The cross entropy also lacks closed-form, NPV approximates this term using the analogous Huber et al. Taylor approximation. Specifically. NPV expands the log density around the means of each GMM component as,
M
2 llg(p(u)) Bgg. [(x pa))m] = (p(x)((())
H₂(p(z)) ~
(39)
n.!
1=1
n=
However, Eqn. (20) is subject to the divergence criterion of Theorem 3.1 2p(n:) S max(p(x)). This approximation is often known as the multivariate delta method for moments. The authors use these approximations of the entropy and cross entropy to create the following two approximation of the ELBO.
(1 HH - Hr (g(x))
(2(g) - (p(z)) H (g(x))
(40)
Gershman et al. (2012) use the two approximations because optimizing fr,,, (p(x)) with respect to the mean components, μi; requires computing a third order, multivariate derivative log(p(æ)) which is computationally expensive. The authors iterate between optimizing the mean components, fi. using (g) and optimizing the variance components, of, using L2(g) until the second approximate appropriately converges 6Lz(g) .0001.
21

---

In our approach, we will highlight and address three problems with the NPV algorithm; the potential divergence of H. N.q (p(x)), the inconsistent ELBO approximations, and the poor estimation of the GMM entropy via Hy (q(x)). To address the potential divergence of W, (p(x)), we will take motivation from the results found in [23] and use 2 point Gauss-Hermite quadrature method to approximate Hg (p(x)) This method will be a limiting factor in scaling the NPV algorithm in dimension, however guarantees that the cross-entropy approximation will not diverge. This alteration leads to solution for the inconsistency of the ELBO approximations. Since the quadrature method does not require any derivatives of log(p())) w.r.t. the mean components of the GMM, we can now optimize the means and variances simultaneously on the same ELBO approximation. Finally, Jensen's 's inequality a very poor approximation for entropy in general, instead we will use the three methods we have introduced, Taylor, Taylor limit, and Legendre, as the GMM entropy approximations for higher accuracy. Fig. 4 shows an approximation of two dimensional, three component mixture Student T distribution using a fiver component GMM in the traditional NPV, using our Taylor approximation and our Legendre approximation.
The results, as seen in Fig. 5, highlight the accuracy and computation time of each method versus the number of components used, the order of our polynomial approximation used, and the dimension of the GMM. The accuracy is the same as seen in Section 6, the new information here is the computation time of each method. We see that the order of the method has very little impact on the computation time of our algorithm. We even see most of the accuracy improvement at around order or 3 so staying in low order approximation seems advisable. We do see that the component does increase our time by a bit compared to that of traditional NPV. The source of the computation time increase in our methods comes from more iterations in the optimization. Each evaluation of the (ELBO) approximator are near equivalent but since we are converging to better optimum, this take more iteration steps than NPV. Finally, we see that dimension does have a large impact on our method. The source of this computation increase come from our Gauus-Hermite approximation of the cross entropy. The number of quadrature points used 2 per dimension D, so we are computing with 2D points, which clearly scale poorly with dimension. We are seeking better ways of computing the cross-entropy of a GMM with any non-GMM function that both convergent and computationally fast, however this was not the focus of the paper and no method was considered yet.
22


Reviews:
[{"review_id": "v9xlK6w5d6", "reviewer": "Reviewer_BrW6", "summary": "The authors show that a common entropy approximator, based on Taylor series approximators, for Gaussian mixture models is not necessarily convergent.  They propose an alternative approach based on the Taylor expansion of the logarithm, whose efficiency hinges on the fact that moments of Gaussians can be computed efficiently.", "questions": "-  Practical computation time is, of course, important.  However, I think you should also include a formal statement of the computational complexity of all of the methods involved.\n\n-  How does this approach perform when applying NPVI to a real data set?  The experiments are mostly synthetic -- it would have been nice to see a real application.\n\nMinor typos:\nWhile the submission is comprehensible, the language, e.g., in the introduction and elsewhere, feels a bit unnatural. I suggest a proofreading pass to smooth it out a bit.\n\n-  \"i.e.\" and \"e.g.\" must always be followed by commas.\n-  The sentence in lines 73-75 is awkward.\n-  The sentence in line 86-87 is not a sentence\n- \"h\" is not defined in (5).\n- The presentation is a bit confusing as it seems to present the univariate Taylor theorem as applying to multivariate functions...  This is fine in (6), which is a univariate Taylor series, but the GMM problem is presented in the general multivariate case.\n-  Notation changed from c to a in (6).  Maybe be consistent?\n-  Line 152 \"we can applying\".", "limitations": "Yes.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The proposed approach fills gaps in the literature and proposes novel solutions.", "weaknesses": "It isn't clear how practical this approach is.  Many of the lemmas and theorems are known results, and the authors do not do a good job differentiating their contributions.  The presentation could use some improvement.  The experiments are entirely synthetic despite the possibility of applying this approximation as part of non-parametric variational inference.  Computational complexity is not clearly stated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**\"Many of the lemmas and theorems are known results...\"**\n\nKey results of this paper are divergence of the Huber et al. approximation (Thm. 3.1) along with our Taylor and Legendre approximations and their convergence (Thm. 4.2 & 4.5).  To the best of our knowledge these results have not previously been established.  We have included some established results restated in our own context for completeness, such as convergence of Taylor (Lemma 3.2) and Legendre (Lemma 3.3) series.  We do not claim these as novel contributions of the paper and provide references where appropriate.  We can defer these results to the appendix if the reviewer feels it would improve the presentation.  We are also happy to include any references that the reviewer feels may have been overlooked if they are provided.\n\n**Practicality of approach**\n\nOur work builds on an established approximation proposed in Huber et al., which has been accepted as practical with over 300 citations at the time of this writing.  We show that this widely cited method does not converge in general and we provide two alternatives that are both convergent and applicable to higher polynomial orders than previous work.  Moreover, our approach has no additional computational overhead compared to existing work (see response below and paper Fig. 3).  We feel this firmly establishes that the work is, both, practical and beneficial to the community.\n\n**\"...I think you should also include a formal statement of the computational complexity...\"**\n\nAppendix A.5 outlines complexity of the Taylor approximation and motivates our proposed limit approximation.  We summarize here for convenience: Complexity of the approximation is dominated by the summation over additive sequences with a fixed sum in Eqn. (11).  For an $M$-component GMM our $n$-th order Taylor series this is dominated by $\\mathcal{O}((n+M-1)!)$.  We will include this statement in the main text, along with complexity of our Legendre series, which is comparable as it is dependent on the same dominating summation.  \n\nHuber et al. do not provide complexity order of their approach, but we provide a summary of our calculation here for the reviewer's convenience.  Recall that the Huber et al.'s approximation about mean $\\mu_i$ is given in Eq. (5) as, $\\sum_{n=0}^\\infty \\frac{1}{n!} \\log(p(\\mu_i))^{(n)} (x-\\mu_i)^n$.  For $D$-dimensional r.v. $x$ the term $(x-\\mu_i)^n$ requires $\\mathcal{O}(D^n)$ operations. The term $\\log(p(\\mu_i))^{(n)}$ requires Faa' di Bruno's formula, which yields a sum over all $n$-tuples of non-negative integers $(m_1,\\dots, m_n)$ s.t. $1m_1+2m_2+\\dots+nm_n=n$ which is combinatorial. We will include an extended form of this discussion in the final appendix.\n\n**Synthetic vs. Real-Data Experiments**\n\nWe agree with the reviewer that real-data experiments would be interesting, and will be a focus of future application of this methodology.  For the present work we feel that controlled synthetic evaluation is preferable for supporting our claims in this paper.  Our synthetic evaluation supports theoretical analysis by directly showing divergence of Huber et al. even in simple settings, validates our convergence results in those same settings, and provides a computational comparison along several dimensions that can easily be controlled.  \n\n\n**The presentation...seems to present the univariate Taylor theorem as applying to multivariate functions...**\n\nHuber et al. expands about the vector-valued random variable $x$.  Because of this Huber et al. cannot easily be calculated above second-order due to tensor arithmetic that arises (see computation above).  By contrast our approach expands about the density function $p(x)$, which is always a scalar function by definition. This distinction is what allows our approach to be computed to polynomial orders above 2 without tensor arithmetic.  Divergence of Huber et al. in the scalar case (Thm. 3.1) is sufficient to establish that the method is not convergent in general.\n\n**Notation changed from c to a in (6). Maybe be consistent?**\n\nThis notation change is intentional and explicitly noted following Eq. (6): \"Note the change of c to a as the Taylor series center...\"."}, {"review_id": "yD4pyF8smz", "reviewer": "Reviewer_gUEj", "summary": "This paper looks at the convergence of an existing approximation of the GMM entropy. They show conditions, which commonly occur, in which the approximation diverges. The authors then present several new approximations in which conditions can be chosen that guarantee convergence. The authors demonstrate this both theoretically and empirically.", "questions": "No questions.", "limitations": "The authors address the limitations.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "I have reviewed a previous version of this paper and this version is stronger in all regards. The theory is sound, the experiments are well-done, and the paper is clearly written. My concerns for previous versions of this paper have been addressed.", "weaknesses": "The computational complexity is a concern for this method. Also, the contribution here is somewhat limited. Other methods involving entropy often consider nonparametric estimation methods that can be adapted to different and unknown densities. Here, it is assumed that the parameters of the GMM are known. That said, the GMM is of sufficient importance in machine learning that I believe the contribution is sufficient. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**I have reviewed a previous version of this paper**\n\nWe sincerely thank the reviewer for their effort and helpful insights on an earlier submission.  The early feedback has strengthened the present work.\n\n**The computational complexity is a concern for this method**\n\nFor similar polynomial orders of our approximations to Huber et al., the computation complexity is comparable (Figure 3). The computation cost may start to become a larger focus for higher polynomial orders due to the combinatorial sum in computing Eq. (11) in Lemma 4.1. However, the multivariate normal distribution terms can be evaluated outside the summation once as their values are constant, meaning the whole summation is simply scalar terms. This computation can then easily be parallelized to utilize GPU acceleration to offset cost.  In our experiments we found that GPU parallelization was not necessary as adequate accuracy is achieved with moderate polynomial orders.\n\n**The contribution here is somewhat limited. Other methods involving entropy often consider nonparametric estimation methods that can be adapted to different and unknown densities. Here, it is assumed that the parameters of the GMM are known**\n\nOur work builds upon the well-established approximation introduced by Huber et al., which has over 300 citations indicating its widespread recognition and use. We highlight a crucial observation that the original approach lacks general convergence. In response, we present two novel methodologies that not only ensure convergence but also extend applicability to higher polynomial orders compared to previous approaches. Furthermore, our advancements come with no additional computational overhead for similar accuracy. We believe that this firmly establishes our contribution's broader impact and its potential benefits to a wider research audience."}, {"review_id": "6eZFuOv3iK", "reviewer": "Reviewer_2uqK", "summary": "The paper provides an improved approximation to the entropy of Gaussian Mixture Models (GMMs). The proposed approximation is a lower bound to the entropy and is more accurate compared to related methods.", "questions": "1. How do you choose the order of the approximating polynomials? \n2. How does the approximation improve the VI compared to using Huber et al.?", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Accurate computation of the GMM entropy has implications on variational inference (VI) as well as information theory. The proposed approximation establishes a lower bound for the GMM entropy which can be naturally adapted in VI.\n2. The authors show that the proposed approximation overcomes the divergence issue in the related methods, e.g., Huber et al. (2008), and hence provides stronger theoretical guarantees.\n3. The method utilizes an interesting idea leveraging the Legendre series approximation of the logarithm.", "weaknesses": "1. While the proposed approximation establishes a lower bound for the GMM entropy, it is not clear whether the bound is tight. Thus, applying the approximation to VI may not yield monotonically increasing ELBO.\n2. Computation of the approximation seems very intensive, e.g., Eq. 9. and Eq. 12. The increased accuracy may not justify the computational costs.\n3. The authors only provided simulation results on toy data. It could be helpful to demonstrate the benefits of the approximation in a real-life problem.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**While the proposed approximation establishes a lower bound for the GMM entropy, it is not clear whether the bound is tight. Thus, applying the approximation to VI may not yield monotonically increasing ELBO**\n\nThe Taylor series bound on GMM entropy becomes tight as the order of the approximation increases. This is a result of convergence of the series (Theorem 4.2) and the lower bound property (Theorem 4.3).\n\n**Computation of the approximation seems very intensive, e.g., Eq. 9. and Eq. 12. The increased accuracy may not justify the computational costs.**\n\nComputation is dominated by the combinatorial sum in Eq. (11) in Lemma 4.1. For an $M$-component GMM our $n$-th order Taylor series this is dominated by $\\mathcal{O}((n+M-1)!)$, see Appendix A.5 for discussion.  Computation of the series in Eq. (11) can easily be parallelized to utilize GPU acceleration, offsetting the cost.  In our experiments we achieve good accuracy with moderate polynomial orders, making GPU parallelization unnecessary.  Empirical analysis of computation time is presented in Fig. 3.\n\n**The authors only provided simulation results on toy data. It could be helpful to demonstrate the benefits of the approximation in a real-life problem.**\n\nWe recognize the desire for an application of our methods to real data and this will be a focus in future work. The focus of the present work is a convergent alternative to that of Huber et al.  To support our claims made in this paper we prefer controlled synthetic evaluation, as it allows us to demonstrate divergence of the baseline, convergence of our approaches in the same setting, and straightforward evaluation of computation time.  Evaluation in real-data scenarios becomes less straightforward as the data distribution is unknown and cannot be instrumented. \n\n**How do you choose the order of the approximating polynomials?**\n\nOur series approximations are convergent, meaning higher-order polynomials are more accurate. The user may choose the order based on computational resources and required accuracy for the problem at hand. This is not necessarily true for Huber et al. as the series may diverge and divergence is more severe at higher orders. In A.6, we reproduce the toy model from Huber et al. in which that method is well-behaved.  Figure 6 shows that our Legendre series yields comparable accuracy and Figure 3 (bottom right) shows the two methods have comparable computation time.\n\n**How does the approximation improve the VI compared to using Huber et al.?**\n\nThe accuracy of the Legendre polynomial is close to that of Huber et al. when the latter method is convergent. In this setting the benefit of Legendre over Huber et al. is limited, but is guaranteed to be convergent in all cases.  Our Taylor series benefits from being a lower bound on $H_q(q(x))$. In ELBO,\n\n$\\log(p(x))\\geq L(\\theta) = \\mathbb{E}\\_{q_\\theta(y|x)}[\\log p(x,y)] + H_{q_\\theta(y|x)}(q_\\theta(y|x))$\n\nthis lower bound on the entropy term gives a lower bound on $L(\\theta)$, and thus a lower bound on $\\log(p(x))$. This bound does not hold in Huber et al.'s approximation."}, {"review_id": "n9gxoBjMxa", "reviewer": "Reviewer_HKUU", "summary": "This paper presents new methods for approximating the entropy of a mixture of Gaussians/cross-entropy between mixtures of Gaussians. By using deterministic approximations based on power series/orthogonal polynomials expansions, the authors are able to obtain similar results as MCMC approximations for only a fraction of the computing time. Furthermore another deterministic approach (Huber) is shown to be theoretically and empirically divergent under a simple condition. The result is applied to Nonparametric Variational Inference and is shown to exhibit a better convergence behavior than the Huber approximation.", "questions": "-line 183 : what do you mean by convergence rate of \\beta \\alpha^n + \\eta ? (edit : it is introduced in the appendix but is quite obscure in the main at the first read)\n-I can't find the result on pointwise convergence for Legendre series when the function is continuous and L^2 ; can you point more directly in the book/to another reference ?", "limitations": "The authors have correctly adressed the main limitations of their results, namely the limitations to GMM and the problem of scaling to high orders of approximations as well as the number of components. Moreover, they do not indicate that their method should always be preferred to Huber's method but point out contexts where it could be the case.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper is well written and the experiments are clearly presented. The theoretical claims are solid and provide useful insights and methods for approximationg entropy of GMMs. The convergence criterions clearly show the limitations of the already used Huber method and is supported empirically. Moreover the method of Taylor approximation and Legendre polynomials are shown to perform well in practice and outperform Huber is some settings.", "weaknesses": "- I think it could be beneficial to have a presentation of NPV from the start for the sake of motivation\n- no convergence rates/error estimate is presented or at least discussed, I think this is also crucial even though harder to investigate", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**line 183 : what do you mean by convergence rate of $\\beta \\alpha^n + \\eta$ ?**\n\nFrom Taylor's theorem, a function, $f$, can be represented as $f=T_n+R_n$ where $T_n$ is the $n^{th}$ order Taylor series and $R_n=\\mathcal{O}(\\alpha^n)$ is the remainder. Rewriting this function in terms of the Taylor series, $T_n=-R_n+f= \\beta\\alpha^n+\\eta$, we assume $\\beta<0$ to force the negative on the remainder and use $0<\\alpha <1$ to model the decay of the error. We agree the main text could benefit from a clearer description of this and will move the discussion in section A.5 to the main body.\n\n**I can't find the result on pointwise convergence for Legendre series when the function is continuous and $L^2$**\n\nThe reference we cite in the paper is Bharwy et al. [4]. They define the shifted Legendre polynomials and state (without proof) that they are an orthonormal basis on $L^2((0,L))$. For a rigorous proof of this statement, one can look at \"Linear Operators in Hilbert Spaces\" by Joachim Weidmann in Section 3.2. We can add a reference to this book if the reviewer believes this should be included.\n\n**no convergence rates/error estimate is presented or at least discussed, I think this is also crucial even though harder to investigate**\n\nWe agree that having some statement of convergence rate would be a nice addition, but the reviewer is correct that this is harder to investigate. We attempted to approach the Taylor series using the Taylor theorem which says that the $n^{th}$ order Taylor series has a remainder $R_{n+1}(x)= \\dfrac{f^{(n+1)}(c)}{(n+1)!}(x-a)^k$ for some $c\\in[a,x]$. Since we are expanding the Taylor series inside the expected value of the entropy term, we must compute $\\mathbb{E}_{q}\\left[\\dfrac{f^{(n+1)}(c)}{(n+1)!}(x-a)^k\\right]$ however $c$ is dependent on $x$. Bounding this quantity is not fruitful because as $x\\rightarrow 0$, the bound diverges. Furthermore, for Legendre and Taylor, we attempted other traditional approaches to finding rates of convergence however fell into similar issues. This is still ongoing research and we hope to make this contribution in future work."}]

Author rebuttal:
We thank all reviewers for their time and useful insights.  We have provided individual responses to each reviewer below.  For brevity we do not address minor typos, all of which will be corrected in a final version.  

Meta-review: The reviewers agree that this paper makes two valuable contributions: showing that a widely used estimator of the Gaussian mixture model entropy is inconsistent and providing alternatives that are consistent. The claims are well supported by both theory and numerical results.
Decision: Accept (poster)