SlowGuess/NeurIPS_2023_OCR huggingface dataset:
<sup>3</sup>Grant B. Gustafson Differential Equations and Linear Algebra. 1999-2022.
A.2 Section 4 Proofs
$$
\mathbb {E} _ {p} \left[ p (x) ^ {k} \right] = \sum_ {j _ {1} + \dots + j _ {M} = k} \binom {k} {j _ {1}, \dots , j _ {M}} \sum_ {i = 1} ^ {M} w _ {i} \left(\frac {\mathcal {N} \left(0 \mid \mu_ {i} , \Sigma_ {i}\right)}{\mathcal {N} \left(0 \mid \mu , \Sigma\right)} \prod_ {t = 1} ^ {M} \left(w _ {t} \mathcal {N} \left(0 \mid \mu_ {t}, \Sigma_ {t}\right) ^ {j _ {t}}\right)\right) \tag {28}
$$
Proof. We will prove this statement by directly expanding out each term
$$
\begin{array}{l} \mathbb {E} _ {p} [ p (x) ^ {k} ] = \mathbb {E} _ {p} \left[ \left(\sum_ {t = 1} ^ {m} w _ {t} \mathcal {N} (x | \mu_ {t}, \Sigma_ {t})\right) ^ {k} \right] \\ = \mathbb {E} _ {p} \left[ \sum_ {j _ {1} + \dots + j _ {m} = k} \binom {k} {j _ {1}, \dots , j _ {m}} \prod_ {t = 1} ^ {m} \left(w _ {t} \mathcal {N} \left(x \mid \mu_ {t}, \Sigma_ {t}\right)\right) ^ {j _ {t}} \right] \\ = \sum_ {j _ {1} + \dots + j _ {m} = k} \binom {k} {j _ {1}, \dots , j _ {m}} \prod_ {t ^ {\prime} = 1} ^ {m} (w _ {t ^ {\prime}}) ^ {j _ {t ^ {\prime}}} \sum_ {i = 1} ^ {M} w _ {i} \int \mathcal {N} (x | \mu_ {i}, \Sigma_ {i}) \prod_ {t = 1} ^ {m} (\mathcal {N} (x | \mu_ {t}, \Sigma_ {t})) ^ {j _ {t}} d x \\ \end{array}
$$
To combine the Gaussians under the integral, we appeal to the power of Gaussians (Lemma A.2.3) and product of Gaussians (Lemma A.2.4)
$$
\begin{array}{l} = \sum_ {j _ {1} + \dots + j _ {m} = k} \binom {k} {j _ {1}, \dots , j _ {m}} \prod_ {t ^ {\prime} = 1} ^ {m} (w _ {t ^ {\prime}}) ^ {j _ {t ^ {\prime}}} \sum_ {i = 1} ^ {M} w _ {i} \int \mathcal {N} (x | \mu_ {i}, \Sigma_ {i}) \prod_ {t = 1} ^ {m} \frac {\mathcal {N} (x | \mu_ {t} , \frac {1}{j _ {t}} \Sigma_ {t})}{| j _ {t} (2 \pi \Sigma_ {t}) ^ {j _ {t} - 1} | ^ {1 / 2}} d x \\ = \sum_ {j _ {1} + \dots + j _ {m} = k} \binom {k} {j _ {1}, \dots , j _ {m}} \prod_ {t ^ {\prime} = 1} ^ {m} \frac {(w _ {t ^ {\prime}}) ^ {j _ {t ^ {\prime}}}}{| j _ {t} (2 \pi \Sigma_ {t}) ^ {j _ {t} - 1} | ^ {1 / 2}} \sum_ {i = 1} ^ {M} w _ {i} \int \frac {\mathcal {N} (0 | \mu_ {i} , \Sigma_ {i}) \prod_ {t = 1} ^ {m} \mathcal {N} (0 | \mu_ {t} , \frac {1}{j _ {t}} \Sigma_ {t})}{\mathcal {N} (0 | \mu , \Sigma)} N (x | \mu , \Sigma) d x \\ = \sum_ {j _ {1} + \dots + j _ {m} = k} \binom {k} {j _ {1}, \dots , j _ {m}} \sum_ {i = 1} ^ {M} w _ {i} \left(\frac {\mathcal {N} (0 | \mu_ {i} , \Sigma_ {i})}{\mathcal {N} (0 | \mu , \Sigma)} \prod_ {t = 1} ^ {m} \left(w _ {t} \mathcal {N} (0 | \mu_ {t}, \Sigma_ {t})\right) ^ {j _ {t}}\right) \\ \end{array}
$$
where  $\mu = \Sigma (\Sigma_{i}^{-1}\mu_{i} + \sum_{t = 1}^{M}j_{t}\Sigma_{t}^{-1}\mu_{t})$  as defined from Lemma A.2.4. We see that we are left with no integral and a closed form of the expectation of the powers of the GMM.
$$
\lim  _ {N \rightarrow \infty} \hat {H} _ {N, a} ^ {T} (p (x)) = H (p (x)) \tag {29}
$$
Proof. We start out with the definition of entropy and introduce in the approximation discussed in Lemma 3.2
$$
\begin{array}{l} H (p (x)) = - \int \sum_ {i = 1} ^ {M} w _ {i} q _ {i} (x) \log (p (x)) d x = - \sum_ {i = 1} ^ {M} w _ {i} \int q _ {i} (x) \log (p (x)) d x \\ = - \sum_ {i = 1} ^ {M} w _ {i} \int q _ {i} (x) \left(\log (a) + \sum_ {i = 1} ^ {\infty} \frac {(- 1) ^ {n - 1}}{n a ^ {n}} (p (x) - a) ^ {n}\right) d x \\ = - \sum_ {i = 1} ^ {M} w _ {i} \left(l o g (a) + \int q _ {i} (x) \sum_ {i = 1} ^ {\infty} \frac {(- 1) ^ {n - 1}}{n a ^ {n}} (p (x) - a) ^ {n} d x\right) \\ = - \sum_ {i = 1} ^ {M} w _ {i} \left(l o g (a) + \int \sum_ {i = 1} ^ {\infty} \frac {(- 1) ^ {n - 1}}{n a ^ {n}} q _ {i} (x) (p (x) - a) ^ {n} d x\right) \\ \end{array}
$$
We now wish to swap the order of integration and of the infinite summation as shown in Lemma A.2.1
$$
\begin{array}{l} = - \sum_ {i = 1} ^ {M} w _ {i} \left(l o g (a) + \sum_ {i = 1} ^ {\infty} \int \frac {(- 1) ^ {n - 1}}{n a ^ {n}} q _ {i} (x) (p (x) - a) ^ {n} d x\right) \\ = - \sum_ {i = 1} ^ {M} w _ {i} \left(l o g (a) + \sum_ {i = 1} ^ {\infty} \frac {(- 1) ^ {n - 1}}{n a ^ {n}} \int q _ {i} (x) (p (x) - a) ^ {n} d x\right) \\ = - \sum_ {i = 1} ^ {M} w _ {i} \left(l o g (a) + \sum_ {i = 1} ^ {\infty} \frac {(- 1) ^ {n - 1}}{n a ^ {n}} \int q _ {i} (x) \sum_ {k = 0} ^ {n} \binom {n} {k} (- a) ^ {n - k} (p (x)) ^ {k} d x\right) \\ = - \sum_ {i = 1} ^ {M} w _ {i} \left(l o g (a) + \sum_ {i = 1} ^ {\infty} \frac {(- 1) ^ {n - 1}}{n a ^ {n}} \sum_ {k = 0} ^ {n} \binom {n} {k} (- a) ^ {n - k} \mathbb {E} _ {q _ {i} (x)} [ (p (x)) ^ {k} ]\right) \\ \end{array}
$$
We can compute  $\mathbb{E}_{q_i(x)}[(p(x))^k]$  using Lemma 4.1. The above term is equality for the Entropy, simply truncating the infinite summation gives a convergent approximation.
$$
\hat {H} _ {N, a} ^ {T} (p (x)) = - \sum_ {i = 1} ^ {M} w _ {i} \left(\log (a) + \sum_ {i = 1} ^ {N} \frac {(- 1) ^ {n - 1}}{n a ^ {n}} \sum_ {k = 0} ^ {n} \binom {n} {k} (- a) ^ {n - k} \mathbb {E} _ {q _ {i} (x)} [ (p (x)) ^ {k} ]\right) \tag {30}
$$
$$
\hat {H} _ {N, a} ^ {T} (p (x)) \leq H (p (x)) \tag {31}
$$
Proof. If  $a > \max(p(x))$ , then we have the following lower bound
$$
\begin{array}{l} H (p (x)) = - \int p (x) \log p (x) d x = - \int p (x) \left(\log (a) + \sum_ {n = 1} ^ {\infty} \frac {(- 1) ^ {n - 1}}{n a ^ {n}} (p (x) - a) ^ {n}\right) d x \\ = - \int p (x) \left(\log (a) + \sum_ {n = 1} ^ {\infty} \frac {(- 1) ^ {n - 1} (- 1) ^ {n}}{n a ^ {n}} (a - p (x)) ^ {n}\right) d x \\ = - \log (a) - \int p (x) \left(\sum_ {n = 1} ^ {\infty} \frac {- 1}{n a ^ {n}} (a - p (x)) ^ {n}\right) d x \\ = - \log (a) + \int p (x) \left(\sum_ {n = 1} ^ {\infty} \frac {1}{n a ^ {n}} (a - p (x)) ^ {n}\right) d x \\ \geq - \log (a) + \int p (x) \left(\sum_ {n = 1} ^ {N} \frac {1}{n a ^ {n}} (a - p (x)) ^ {n}\right) d x = \hat {H} _ {N} (p (x)) \\ \end{array}
$$
since every term in the summation is positive due to  $a > p(x)\forall x$ , then truncating the series only removes positive terms, leaving us with a lower bound.
Theorem 4.4 (Upper bound on maximum of a GMM). Let  $p(x) = \sum_{i=1}^{M} w_i \mathcal{N}(x|\mu_i, \Sigma_i)$ , then
$$
\max  (p (x)) \leq a = \sum_ {i} ^ {M} w _ {i} \left| 2 \pi \Sigma_ {i} \right| ^ {- \frac {1}{2}} \tag {32}
$$
Proof. We need to find an upper bound on  $\max(p(x))$
$$
\begin{array}{l} \max  (p (x)) = \max  \left(\sum_ {i = 1} ^ {M} w _ {i} \mathcal {N} (x | \mu_ {i}, \Sigma_ {i})\right) \\ \leq \sum_ {i = 1} ^ {M} w _ {i} \max  \left(\mathcal {N} \left(x \mid \mu_ {i}, \Sigma_ {i}\right)\right) = \sum_ {i = 1} ^ {M} w _ {i} \left| 2 \pi \Sigma_ {i} \right| ^ {- \frac {1}{2}} \\ \end{array}
$$
We simply have bound the maximum of the combination by combining the maximum of every component in the GMM.
$$
\lim  _ {N \rightarrow \infty} \hat {H} _ {N, a} ^ {L} (p (x)) = H (p (x)) \tag {33}
$$
Proof. We start out with the definition of entropy and introduce in the approximation discussed in Lemma 3.3
$$
\begin{array}{l} H (p (x)) = - \int \sum_ {i = 1} ^ {M} w _ {i} q _ {i} (x) \log (p (x)) d x = - \sum_ {i = 1} ^ {M} w _ {i} \int q _ {i} (x) \log (p (x)) d x \\ = - \sum_ {i = 1} ^ {M} w _ {i} \int q _ {i} (x) \sum_ {i = 0} ^ {\infty} \frac {\left\langle \log (y) , L _ {[ 0 , a ], n} (y) \right\rangle}{\left\langle L _ {[ 0 , a ], n} (y) , L _ {[ 0 , a ], n} (y) \right\rangle} L _ {[ 0, a ], n} (p (x)) d x \\ = - \sum_ {i = 1} ^ {M} w _ {i} \sum_ {i = 0} ^ {\infty} \int q _ {i} (x) \frac {\left\langle \log (y) , L _ {[ 0 , a ], n} (y) \right\rangle}{\left\langle L _ {[ 0 , a ], n} (y) , L _ {[ 0 , a ], n} (y) \right\rangle} L _ {[ 0, a ], n} (p (x)) d x \\ \end{array}
$$
We swapped the order of integration and of the infinite summation as shown in Lemma A.2.2. We simplify computation that are recreated in Theorem 3.3
$$
\begin{array}{l} = - \sum_ {i = 0} ^ {\infty} \frac {\left\langle \log (y) , L _ {[ 0 , a ] , n} (y) \right\rangle}{\left\langle L _ {[ 0 , a ] , n} (y) , L _ {[ 0 , a ] , n} (y) \right\rangle} \sum_ {i = 1} ^ {M} w _ {i} \int q _ {i} (x) L _ {[ 0, a ], n} (p (x)) d x \\ = - \sum_ {i = 0} ^ {\infty} (2 n + 1) \sum_ {j = 0} ^ {n} \frac {(- 1) ^ {n + j} (n + j) ! ((j + 1) \log (a) - 1)}{(n - j) ! ((j + 1) !) ^ {2}} \sum_ {i = 1} ^ {M} w _ {i} \mathbb {E} _ {q _ {i} (x)} \left[ \sum_ {k = 0} ^ {n} \frac {(- 1) ^ {n + k} (n + k) !}{(n - k) ! (k !) ^ {2} a ^ {k}} p (x) ^ {k} \right] \\ = - \sum_ {i = 0} ^ {\infty} (2 n + 1) \sum_ {j = 0} ^ {n} \frac {(- 1) ^ {n + j} (n + j) ! ((j + 1) \log (a) - 1)}{(n - j) ! ((j + 1) !) ^ {2}} \sum_ {k = 0} ^ {n} \frac {(- 1) ^ {n + k} (n + k) !}{(n - k) ! (k !) ^ {2} a ^ {k}} \sum_ {i = 1} ^ {M} w _ {i} \mathbb {E} _ {q _ {i} (x)} [ p (x) ^ {k} ] \\ \end{array}
$$
We can compute  $\mathbb{E}_{q_i(x)}[(p(x))^k]$  using Lemma 4.1. Simply truncating the infinite summation gives the approximation.
$$
\hat {H} _ {N, a} ^ {L} (p (x)) = - \sum_ {i = 0} ^ {N} (2 n + 1) \sum_ {j = 0} ^ {n} \frac {(- 1) ^ {n + j} (n + j) ! ((j + 1) \log (a) - 1)}{(n - j) ! ((j + 1) !) ^ {2}} \sum_ {k = 0} ^ {n} \frac {(- 1) ^ {n + k} (n + k) !}{(n - k) ! (k !) ^ {2} a ^ {k}} \sum_ {i = 1} ^ {M} w _ {i} \mathbb {E} _ {q _ {i} (x)} [ p (x) ^ {k} ]
$$
which  $\lim_{N\to \infty}\hat{H}_{N,a}^{L}(p(x)) = H(p(x))$  as the above series is exactly equal to the entropy.
Here, we address a few of the assumptions we made in the above derivations. We start with the ability to swap the order of the integral and infinite sum for the Taylor series.
Lemma A.2.1 (Swapping Integral and Infinite Sum (Taylor)). Let  $a > \frac{1}{2} \max(p(x))$ , then
$$
\int \sum_ {i = 1} ^ {\infty} \frac {(- 1) ^ {n - 1}}{n a ^ {n}} q _ {i} (x) (p (x) - a) ^ {n} d x = \sum_ {i = 1} ^ {\infty} \int \frac {(- 1) ^ {n - 1}}{n a ^ {n}} q _ {i} (x) (p (x) - a) ^ {n} d x
$$
Proof. For simplicity of notation, let  $c = \sup \left|\frac{p(x) - a}{a}\right| < 1$  since  $a > \frac{1}{2} \max(p(x))$ . We then appeal to Fubini-Tonelli theorem which states that if  $\int \sum |f_n(x)| dx < \infty$  or if  $\int \int |f_n(x)| dx < \infty$ , then  $\int \sum f_n(x) dx = \sum \int f_n(x) dx$ .
$$
\begin{array}{l} \sum_ {i = 1} ^ {\infty} \int \left| \frac {(- 1) ^ {n - 1}}{n a ^ {n}} q _ {i} (x) (p (x) - a) ^ {n} \right| d x = \sum_ {i = 1} ^ {\infty} \frac {1}{n} \int q _ {i} (x) \left(\frac {| p (x) - a |}{a}\right) ^ {n} d x \\ \leq \sum_ {i = 1} ^ {\infty} \frac {1}{n} \int q _ {i} (x) (c) ^ {n} d x \\ = \sum_ {i = 1} ^ {\infty} \frac {c ^ {n}}{n} \int q _ {i} (x) d x = \sum_ {i = 1} ^ {\infty} \frac {c ^ {n}}{n} <   \infty \\ \end{array}
$$
We know that  $\sum_{i=1}^{\infty} \frac{c^n}{n} < \infty$  because of the ratio test again
$$
\lim  _ {n \rightarrow \infty} \left| \frac {c ^ {n + 1}}{n + 1} \frac {n}{c ^ {n}} \right| = \lim  _ {n \rightarrow \infty} \frac {n}{n + 1} c = c <   1
$$
So we see that Fubini-Tonelli holds so
We now consider the case for the Legendre series
Lemma A.2.2 (Swapping Integral and Infinite Sum (Legendre)). Let  $a > \max(p(x))$ , then
$$
\int q _ {i} (x) \sum_ {i = 1} ^ {\infty} \frac {\left\langle \log (y) , L _ {[ 0 , a ] , n} (y) \right\rangle}{\left\langle L _ {[ 0 , a ] , n} (y) , L _ {[ 0 , a ] , n} (y) \right\rangle} L _ {[ 0, a ], n} (p (x)) d x = \sum_ {i = 1} ^ {\infty} \int q _ {i} (x) \frac {\left\langle \log (y) , L _ {[ 0 , a ] , n} (y) \right\rangle}{\left\langle L _ {[ 0 , a ] , n} (y) , L _ {[ 0 , a ] , n} (y) \right\rangle} L _ {[ 0, a ], n} (p (x)) d x
$$
Proof. We again appeal to Fubini-Tonelli. We will use that  $\left|\frac{L_{[0,a],n}(p(x))}{\langle L_{[0,a],n}(y),L_{[0,a],n}(y)\rangle}\right| \leq 1$  as it is the orthonormal polynomials and then use Cauchy-Schwartz on the remaining term
$$
\begin{array}{l} \sum_ {i = 1} ^ {\infty} \int \left| q _ {i} (x) \frac {\langle \log (y) , L _ {[ 0 , a ] , n} (y) \rangle}{\langle L _ {[ 0 , a ], n} (y) , L _ {[ 0 , a ], n} (y) \rangle} L _ {[ 0, a ], n} (p (x)) \right| d x \\ = \sum_ {i = 1} ^ {\infty} \left\langle \log (y), L _ {[ 0, a ], n} (y) \right\rangle \int \left| \frac {q _ {i} (x) L _ {[ 0 , a ] , n} (p (x))}{\left\langle L _ {[ 0 , a ] , n} (y) , L _ {[ 0 , a ] , n} (y) \right\rangle} \right| d x \\ \leq \sum_ {i = 1} ^ {\infty} \| \log (y) \| ^ {2} \left\| L _ {[ 0, a ], n} (y) \right\| ^ {2} \int q _ {i} (x) d x \\ = \sum_ {i = 1} ^ {\infty} a ((\log (a) - 2) \log (a) + 2) \left(\frac {a}{2 n + 1}\right) ^ {2} \int N (x | \mu_ {i}, \Sigma_ {i}) d x \\ = a ^ {3} ((\log (a) - 2) \log (a) + 2) \sum_ {i = 1} ^ {\infty} \frac {1}{(2 n + 1) ^ {2}} <   \infty \\ \end{array}
$$
So we see that since the absolute value is finite, then Fubini-Tonelli applies and we can swap the order of the integral and infinite summation.
The next thing we show is the relations we used for powers of Gaussians.
Lemma A.2.3. Powers of a Gaussian
$$
\mathcal {N} (x | \mu , \Sigma) ^ {n} = \left| n (2 \pi \Sigma) ^ {n - 1} \right| ^ {- \frac {1}{2}} \mathcal {N} \left(x | \mu , \frac {1}{n} \Sigma\right)
$$
Proof. We are going to do an inductive proof and rely on the well known relation of products of Gaussians
$$
\mathcal {N} (x | a, A) \mathcal {N} (x | b, B) = \mathcal {N} (x | d, D) \mathcal {N} (a | b, A + B)
$$
Where  $D = (A^{-1} + B - 1)^{-1}$  and  $d = D(A^{-1}a + B^{-1}b)$ .
Base Case:  $n = 2$
$$
\begin{array}{l} \mathcal {N} (x | \mu , \Sigma) ^ {2} = \mathcal {N} (x | \mu , \Sigma) \mathcal {N} (x | \mu , \Sigma) (34) \\ = N (x \mid \mu , \frac {1}{2} \Sigma) N (\mu \mid \mu , 2 \Sigma) = | 2 \pi \Sigma | ^ {- \frac {1}{2}} N (x \mid \mu , \frac {1}{2} \Sigma) (35) \\ \end{array}
$$
In this case, we get that  $D = (\Sigma^{-1} + \Sigma - 1)^{-1} = \frac{1}{2}\Sigma$  and  $d = \frac{1}{2}\Sigma(\Sigma^{-1}\mu + \Sigma^{-1}\mu) = \mu$ . We also see that  $N(\mu|\mu, 2\Sigma)$  is being evaluated at its maximum, which just leaves the scaling term out front
of the exponential in the Gaussian,  $|2\pi \Sigma|^{-\frac{1}{2}}$ .
Inductive step: Assume that  $N(x|\mu, \Sigma)^n = \left| n(2\pi \Sigma)^{n-1} \right|^{-\frac{1}{2}} \mathcal{N}\left(x|\mu, \frac{1}{n}\Sigma\right)$ , then we wish to show that  $N(x|\mu, \Sigma)^{n+1} = |(n+1)(2\pi \Sigma)^n|^{-\frac{1}{2}} \mathcal{N}\left(x|\mu, \frac{1}{n+1}\Sigma\right)$ .
$$
\begin{array}{l} N (x | \mu , \Sigma) ^ {n + 1} = N (x | \mu , \Sigma) ^ {n} N (x | \mu , \Sigma) \\ = \left| n (2 \pi \Sigma) ^ {n - 1} \right| ^ {- \frac {1}{2}} \mathcal {N} \left(x | \mu , \frac {1}{n} \Sigma\right) N (x | \mu , \Sigma) \\ = \left| n (2 \pi \Sigma) ^ {n - 1} \right| ^ {- \frac {1}{2}} \mathcal {N} \left(x | \mu , \frac {1}{n + 1} \Sigma\right) N (\mu | \mu , \frac {n + 1}{n} \Sigma) \\ = \left| n (2 \pi \Sigma) ^ {n - 1} \right| ^ {- \frac {1}{2}} \left| 2 \pi \frac {n + 1}{n} \Sigma \right| ^ {- \frac {1}{2}} \mathcal {N} \left(x | \mu , \frac {1}{n + 1} \Sigma\right) \\ = | (n + 1) (2 \pi \Sigma) ^ {n} | ^ {- \frac {1}{2}} \mathcal {N} \left(x | \mu , \frac {1}{n + 1} \Sigma\right) \\ \end{array}
$$
Here,  $D = \left(\left(\frac{1}{n}\Sigma\right)^{-1} + \Sigma^{-1}\right)^{-1} = \frac{1}{n + 1}\Sigma$  and  $d = \frac{1}{n + 1}\Sigma \left(\left(\frac{1}{n}\Sigma\right)^{-1}\mu + \Sigma^{-1}\mu\right) = \mu$ .
We finally show the product of Gaussians that we used. We keep the exact same notation used in the derivation of the entropy Taylor series so the terms may be more easily identified.
Lemma A.2.4. Product of a Gaussians
$$
\mathcal {N} (x | \mu_ {i}, \Sigma_ {i}) \prod_ {t = 1} ^ {m} \mathcal {N} \left(x | \mu_ {t}, \frac {1}{j _ {t}} \Sigma_ {t}\right) = \mathcal {N} (0 | \mu_ {i}, \Sigma_ {i}) \prod_ {t = 1} ^ {m} \mathcal {N} \left(0 + \mu_ {t}, \frac {1}{j _ {t}} \Sigma_ {t}\right) \frac {N (x | \mu , \Sigma)}{N (0 | \mu , \Sigma)}
$$
where  $\Sigma = (\Sigma^{-1} + \sum_{t=1}^{m} j_t \Sigma_t^{-1})^{-1}$  and  $\mu = \Sigma (\Sigma_i^{-1} \mu_i + \sum_{t=1}^{m} j_t \Sigma_t^{-1} \mu_t)$ .
Proof. We will simply expand out the product of Gaussians, collect like terms, complete the square, and then recollect exponentials into Gaussians evaluated at 0.
$$
\begin{array}{l} \mathcal {N} (x | \mu_ {i}, \Sigma_ {i}) \prod_ {t = 1} ^ {m} \mathcal {N} \left(x | \mu_ {t}, \frac {1}{j _ {t}} \Sigma_ {t}\right) = | 2 \pi \Sigma_ {i} | ^ {- \frac {1}{2}} \exp \left\{- \frac {1}{2} (x - \mu_ {i}) ^ {T} \Sigma_ {i} ^ {- 1} (x - \mu_ {i}) \right\} \prod_ {t = 1} ^ {m} \left| \frac {2 \pi}{j _ {t}} \Sigma_ {t} \right| ^ {- \frac {1}{2}} \exp \left\{- \frac {1}{2} (x - \mu_ {t}) ^ {T} j _ {t} \Sigma_ {t} ^ {- 1} (x - \mu_ {t}) \right\} \\ = \left| 2\pi \Sigma_ {i} \right| ^ {- \frac {1}{2}} \prod_ {t = 1} ^ {m} \left| \frac {2 \pi}{j _ {t}} \Sigma_ {t} \right| ^ {- \frac {1}{2}} \exp \left\{- \frac {1}{2} \left(x ^ {T} \Sigma_ {i} ^ {- 1} x - 2 x ^ {T} \Sigma_ {i} ^ {- 1} \mu_ {i} + \mu_ {i} ^ {T} \Sigma_ {i} ^ {- 1} \mu_ {i} + \sum_ {t = 1} ^ {m} x ^ {T} j _ {t} \Sigma_ {t} ^ {- 1} x - 2 x ^ {T} j _ {t} \Sigma_ {t} ^ {- 1} \mu_ {t} + \mu_ {t} ^ {T} j _ {t} \Sigma_ {t} ^ {- 1} \mu_ {t}\right) \right\} \\ = \left| 2 \pi \Sigma_ {i} \right| ^ {- \frac {1}{2}} \prod_ {t = 1} ^ {m} \left| \frac {2 \pi}{j _ {t}} \Sigma_ {t} \right| ^ {- \frac {1}{2}} \exp \left\{- \frac {1}{2} \left(x ^ {T} \left(\Sigma_ {i} ^ {- 1} + \sum_ {t = 1} ^ {m} j _ {t} \Sigma_ {t}\right) x - 2 x ^ {T} \left(\Sigma_ {i} ^ {- 1} \mu_ {i} + \sum_ {t = 1} ^ {m} j _ {t} \Sigma_ {t} ^ {- 1}\right) + \mu_ {i} ^ {T} \Sigma_ {i} ^ {- 1} \mu_ {i} + \sum_ {t = 1} ^ {m} \mu_ {t} ^ {T} j _ {t} \Sigma_ {t} ^ {- 1} \mu_ {t}\right) \right\} \\ \end{array}
$$
Now we let  $\Sigma = (\Sigma^{-1} + \sum_{t=1}^{m} j_t \Sigma_t^{-1})^{-1}$  and  $\mu = \Sigma \left( \Sigma_i^{-1} \mu_i + \sum_{t=1}^{m} j_t \Sigma_t^{-1} \mu_t \right)$
$$
\begin{array}{l} = \left| 2 \pi \Sigma_ {i} \right| ^ {- \frac {1}{2}} \prod_ {t = 1} ^ {m} \left| \frac {2 \pi}{j _ {t}} \Sigma_ {t} \right| ^ {- \frac {1}{2}} \exp \left\{- \frac {1}{2} \left(x ^ {T} \Sigma^ {- 1} x - 2 x ^ {T} \Sigma^ {- 1} \mu + \mu_ {i} ^ {T} \Sigma_ {i} ^ {- 1} \mu_ {i} + \sum_ {t = 1} ^ {m} \mu_ {t} ^ {T} j _ {t} \Sigma_ {t} ^ {- 1} \mu_ {t}\right) \right\} \\ = \left| 2 \pi \Sigma_ {i} \right| ^ {- \frac {1}{2}} \prod_ {t = 1} ^ {m} \left| \frac {2 \pi}{j _ {t}} \Sigma_ {t} \right| ^ {- \frac {1}{2}} \exp \left\{- \frac {1}{2} \left(x ^ {T} \Sigma^ {- 1} x - 2 x ^ {T} \Sigma^ {- 1} \mu + \mu^ {T} \Sigma \mu - \mu^ {T} \Sigma \mu + \mu_ {i} ^ {T} \Sigma_ {i} ^ {- 1} \mu_ {i} + \sum_ {t = 1} ^ {m} \mu_ {t} ^ {T} j _ {t} \Sigma_ {t} ^ {- 1} \mu_ {t}\right) \right\} \\ = | 2 \pi \Sigma_ {i} | ^ {- \frac {1}{2}} \prod_ {t = 1} ^ {m} \left| \frac {2 \pi}{j _ {t}} \Sigma_ {t} \right| ^ {- \frac {1}{2}} \frac {| 2 \pi \Sigma | ^ {1 / 2}}{| 2 \pi \Sigma | ^ {1 / 2}} \exp \left\{- \frac {1}{2} \left((x - \mu) ^ {T} \Sigma^ {- 1} (x - \mu)\right) \right\} \exp \left\{- \frac {1}{2} \left(- \mu^ {T} \Sigma \mu + \mu_ {i} ^ {T} \Sigma_ {i} ^ {- 1} \mu_ {i} + \sum_ {t = 1} ^ {m} \mu_ {t} ^ {T} j _ {t} \Sigma_ {t} ^ {- 1} \mu_ {t}\right) \right\} \\ = | 2 \pi \Sigma_ {i} | ^ {- \frac {1}{2}} \exp \left\{- \frac {1}{2} \left(\mu_ {i} ^ {T} \Sigma_ {i} ^ {- 1} \mu_ {i}\right) \right\} \prod_ {t = 1} ^ {m} \left| \frac {2 \pi}{j _ {t}} \Sigma_ {t} \right| ^ {- \frac {1}{2}} \exp \left\{- \frac {1}{2} \left(\mu_ {t} ^ {T} j _ {t} \Sigma_ {t} ^ {- 1} \mu_ {t}\right) \right\} | 2 \pi \Sigma | ^ {1 / 2} \exp \left\{- \frac {1}{2} \left(- \mu^ {T} \Sigma \mu\right) \right\} {\mathcal N} (x | \mu , \Sigma) \\ = \mathcal {N} (0 | \mu_ {i} \Sigma_ {i}) \prod_ {t = 1} ^ {m} \mathcal {N} \left(0 | \mu_ {t}, \frac {1}{j _ {t}} \Sigma_ {t}\right) \frac {\mathcal {N} (x | \mu , \Sigma)}{\mathcal {N} (0 | \mu , \Sigma)} \\ \end{array}
$$
Figure 6: Well-Behaved GMM example is plotted on the left. The entropy of a five-component, bivariate GMM is plotted as a function of the location of the fifth component  $\mu_5 \in [-3, 3]$ . We show the true entropy, the  $3^{\mathrm{rd}}$  order of Huber et al.'s approximation and our two methods, along with our approximate limit.
A.4 Extensions to Cross-Entropy
Our main results on GMM entropy approximation also extend to the cross-entropy between different GMMs. We can instead consider  $H_{p}(q(x)) = -\mathbb{E}_{p(x)}[\log (q(x))]$  where  $p(x) = \sum_{i=1}^{\widetilde{M}} \widehat{w_i} \mathcal{N}(x|\widetilde{\mu_i}, \widehat{\Sigma_i})$  and  $q(x) = \sum_{i=1}^{\widetilde{M}} \widehat{w_i} \mathcal{N}(x|\widehat{\mu_i}, \widehat{\Sigma_i})$ . The series representations of  $\log(\cdot)$  stay unchanged however we must choose a center point that allows the series to converge with respect to the inner GMM,  $q(x)$ . This means Theorem 3.2, Theorem 3.3, Theorem 4.2, Theorem 4.3, and Theorem 4.5 need to be reformulated so that  $a > \max(q(x))$  (or the respective  $a > \frac{1}{2} \max(q(x))$ ). We do not formally restate these theorems here for brevity. We instead provide a sketch of how the results extend to the cross-entropy setting. Choosing the bounding  $a$  of the max found in Theorem 4.4 can simply be altered so that
$$
\max  (q (x)) \leq a = \sum_ {i} ^ {\widehat {M}} \widehat {w _ {i}} \left| 2 \pi \widehat {\Sigma_ {i}} \right| ^ {- \frac {1}{2}}
$$
The analogous proofs will all hold in this case. The final alteration that needs to be made is to Theorem 4.1. Again, following the exact same proof, just switching notation and being careful, one can find that
$$
\mathbb {E} _ {p} [ q (x) ^ {k} ] = \sum_ {j _ {1} + \dots + j _ {\widehat {M}} = k} \binom {k} {j _ {1}, \dots , j _ {\widehat {M}}} \sum_ {i = 1} ^ {\widehat {M}} \widetilde {w _ {i}} \left(\frac {\mathcal {N} \left(0 | \widetilde {\mu_ {i}} , \widetilde {\Sigma_ {i}}\right)}{\mathcal {N} \left(0 | \mu , \Sigma\right)} \prod_ {t = 1} ^ {\widehat {M}} \left(\widehat {w _ {t}} \mathcal {N} \left(0 | \widehat {\mu_ {t}}, \widehat {\Sigma_ {t}}\right) ^ {j _ {t}}\right)\right) \tag {36}
$$
where  $\Sigma = (\widetilde{\Sigma_i}^{-1} + \sum_{t=1}^{\widehat{M}} j_t \widehat{\Sigma_t}^{-1})^{-1}$  and  $\mu = \Sigma (\widetilde{\Sigma_i}^{-1} \widetilde{\mu_i} + \sum_{t=1}^{\widehat{M}} j_t \widehat{\Sigma_t}^{-1} \widehat{\mu_t})$ . The result is the same form, however has much more convoluted notation and hence dropped from the main paper as an attempt to bring clarity to the methods being discuss without unnecessary notation.
A.5 Recreation of Huber et al Experiment
Here we reproduce the experiment of [15] of a five-component bivariate GMM with uniform weights  $w_{i} = 0.2$  for  $i = 1, \dots, 5$ ,  $\mu_{1} = [0,0]^{T}$ ,  $\mu_{2} = [3,2]^{T}$ ,  $\mu_{3} = [1, = 0.5]^{T}$ ,  $\mu_{4} = [2.5,1.5]^{T}$ ,  $\mu_{5} = c \cdot [1,1]^{T}$ ,  $\Sigma_{1} = \mathrm{diag}(0.16,1)$ ,  $\Sigma_{2} = \mathrm{diag}(1,0.16)$ , and  $\Sigma_{3} = \Sigma_{4} = \Sigma_{5} = \mathrm{diag}(0.5,0.5)$ . We vary the position of the fifth mean  $(\mu_{5})$  in the range  $[-3,3]$ . Fig. 6 (left) reports the third order Taylor approximations from both Taylor approximations, the Legendre approximation, as well as our approximate limiting method.
Huber et al. is accurate in the well-behaved case, but does not have any convergence guarantees nor is it a bound. Our proposed Taylor approximation sacrifices some accuracy, but is always a lower bound (Theorem 4.3) and is convergent (Theorem 4.2). We also note that our naive limit method does gain us substantial accuracy and is still a lower bound though we have not proven the bound property for this approximation. We notice that our Legendre approximation has comparable accuracy to that of Huber et al. in this well-behaved case but has the advantage that it is guaranteed to converge (Theorem 4.5) and that we can compute higher order approximations that are difficult to define for the Huber et al. approximation.